Fixed scroll break after drag

Removed remove listener calls since they were breaking scroll. Couldn't
properly reattach so event is persistent.