Don't remove event listener for this.handleUp_ twice.

The listener key is in this.dragListenerKeys_