Maybe, but I don't see a way to do that because there is no hook that is called when an interaction is removed from the map. For now I'm adding some code to remove selection layers when they do not contain any selection.