Finds the determinant recursively in Matlab