model/follower: Removed pointless assertions

There is no point in asserting the type of source/destination here. If
the either one of them was of the wrong type the query would complain
anyway.
model: Reduced model dependencies

.. by removing unnecessary primaryjoin parameters. If needed they are
replaced by the foreign_keys parameter.