Revert "Revert "performance fix""

This reverts commit 757010908ddca8f58546869544c582bc28f707b2.

Alas. Until we upgrade to Django 1.4+, this causes us a lot of grief...
An alternative "fix" will be coming shortly.
Minor code cleanup.
Backport performance fix from Django 1.4
Don't serialize many-to-many as performance sucks. Badly.