Fixed #23862 -- Made sure m2m fields through to_field correcly work.

Best to make it a separate commit for clarity.