background: Don't prematurely remove file monitors

We need to only remove file monitors when there's no other users
of the content...