fix a sign error in 3x3 characteristic polynomial