Fix issue with non-working modifiers on mouse events