Fixed #20836: to_field lost when adding via raw_id_fields

Added a selenium test that demonstrated the problem.

Started from this pull request https://github.com/django/django/pull/1417 and then fixed a couple more areas.