Only take the lock when the ref count hits zero

r? @glennw