Fixed #23862 -- Made sure m2m fields through to_field correcly work.

You're right, this was mostly a copy of the logic in `ManyToManyField._get_m2m_attr`.