Fix #11143 (remove visibility groups after removal of a layer)