Only removing our signs in buffer, no all

We used to remove all the signs in a buffer. Now we just remove the ones we set.
This prevents conflicts with other Vim plugins.