Implement the diagonal sum manually or do a np.asarray