Fix adding and immediately removing a Layer from the map when the map is never initialized

Crazy, but broken :)
Removes a few chars, yay!