Fix regression introduced by updating taggit (27971d6eed)

django-taggit 0.11+ introduced support for prefetch_related which breaks
our taggit wrapping: alex/django-taggit@4f2e47f833