Correct reference counting of remotes

We were accidentally overwriting the magic, causing the repository's
reference count to never be decremented.