app: improve and clean up the canvas rotation dialog

Make the action area buttons behave less unexpected, and add and angle
dial.