Finish light refactor.

There was a gnarly bug where adding some object to group1 and then
group2 silently removed it from group1 :x