Fix a flaw in the removal code that would remove a stream when it should not