Remove click delay on side buttons

and only check the delay if it's > 0.