refine detection for vector masks to handle shapeLayers correctly