Ignore swapping for other mods

Control and Alt keys should also get normal numbers instead of symbols.