Handle mouse-button-modifier being disabled

In case 'mouse-button-modifier' is disabled the mask is 0 which means
we would always grab.