backporting prefetch_related issue #19607

See https://code.djangoproject.com/ticket/19607

Original patch: 
https://github.com/django/django/commit/00b39e0145c5b9117503b20bba74948294a30e5a