display: Fix accidental inversion from 2f9c601

Commit 2f9c601 accidentally changed the logic here, changing the grab
behavior when not using raise-on-click. Fix this.

Spotted-by: Adam Goode <adam@spicenitz.org>