Clear all possible listeners when canceling press

When no event is provided (such as when a client uses cancelPress)
we currently have no notion of the interaction type that was used
to trigger the interaction (touch or mouse), but we need to remove
the right listeners as part of the cleanup; removing them all
works well enough for now.

It's probably worth tracking the type of interaction, but I'll leave
that for a later point in time as we address that everywhere.