Cache bitmap mask and reuse it when needed, only clear on invalidate