Triggers a ton of queries when used with Django 1.5's prefilling of the related object cache

Don't blindy `getattr` when `setattr` is called - only do it if it's the field we care about.