Revert "Only removing our signs in buffer, no all"

This reverts commit 182848050e412e3b46f09e928ecc23d32ae9cea1.

The reason we are reverting this is because removing signs in a loop causes
flicker. The only non-flicker approach is to remove all signs in a buffer with
"sign unplace buffer=<buffer-num>".

So no compatibility with other plugins for us.