quadratic solver: improved tolerance testing, bug#3606099