Now only care about prefix's saved-overriding-map when modifying
overriding-terminal-local-map.