Do not use xmodmap without display.