Fix Y coordinate calculation

The formula for computing the Y coordinate was wrong, causing incorrect result when there was a non-uniform scaling transformation on the input matrix.