Fixed the issue with the button in the listview. Turns out, you have to set it to focusable="false"