Use setxkbmap instead of xmodmap. Works in xmonad. Maybe it'll work better in Unity.