Remove pointer events on layers