Fast deselection

Closes #180 

The most recent comment in #180 also explains what this PR is doing.

Not sure if we stick to that solution for all times.