Determinant of a matrix NxN % mod