Don't check modifier keys twice