Calculate the det of a square matrix