Fix KeyError in _rel_agnostic_fields_ref for uncommon `rel.to` setup

NOTE: I am not sure, if this is a bug with django-taggit instead, which should have returned `to` in its deconstruction.

django-taggit manually sets up `rel.to` and therefore there is no "to" kwarg.

Fixes https://code.djangoproject.com/ticket/22263
Ref: https://github.com/alex/django-taggit/issues/206