Fixing performance regression by including too much in Scene.affectedLayers()