Only perform workspace switch on drag during a mouse drag.

Fixes #2513