Fixed button user feedback so it doesn't toggle while the user hasn't left the button