Fix bugs with invites/joins across federatiom.

Both in terms of auth and not trying to fetch missing PDUs for invites,
joins etc.