Ensure unused reference is removed when fill is set to none or a solid colour.