drag-action: don't mix touch and pointer events

https://bugzilla.gnome.org/show_bug.cgi?id=709762