Switch from using xmodmap + setxkbmap to using localectl (needs root).
You may need to: localectl set-x11-keymap --no-convert us pc104 dvp keypad:atm,numpad:shift3,kpdl:semi,caps:escape,terminate:ctrl_alt_bksp