BUG: optimize: check tolerance >= 0 in brent

Fixes gh-4140.