Ensure we don't test for hermiticity when not working with
an orthogonal basis