Only add necessary events to optimize performance

According to this document, performance can be improved by adding event after drag start.
https://developers.google.com/web/fundamentals/input/touch/touchevents/#handle-single-element-interaction