Only act on left/right clicks when the mouse is grabbed.

Signed-off-by: Steven Downer <grinch@outlook.com>