Rework button event handling

This series of commits fixes a few issues and oddities with the way button events are handled.