BUG: optimize: ensure line search is not confused by float issues

Detect non-finite (nan, inf) results from _cubicmin and _quadmin so that
the rest of the line search doesn't become confused.  Avoids problems
when float overflows etc. occur.

Also replace intermediate value checks by exceptions.
Merge pull request #2864 from pv/linesearch-nan

BUG: optimize: ensure line search is not confused by float issues