Properly handle pointer down and up events