Keep track of the flip/identity matrix for the image so it only needs to be applied when it is changed