Correct reference counting of remotes

We were accidently overwriting the magic, causing the repository's
reference count to never be decremented.