Fix removal of unauthorized objects from relations

Closes #1118 but causes a problem with Kaminari's pagination on
users#show. It will require some manual work to replicate the
magic that makes the pagination happen.