Use xkb instead of xmodmap.