Oh, this is much cleaner than xmodmap.
Use setxkbmap instead