Remove Layer references from GeometricLayer
Reduce memory footprint