Fix touch handling when we're not given the ACTION_DOWN event

Fixes #18