Use primary mouse button for primary action

And take the mouse position on mousedown