Fix inverse matrix transformation

The formula for computing the Y coordinate was wrong, causing incorrect result when there was a non-uniform scaling transformation on the input matrix.