Add internal state awareness to dragpan to ensure addition and removal of correct events