Change hiliting implementation

Instead of storing "false" values in the map,
just drop them when removing the hilite.