fix isless and cmp/lexcmp for floating point

for now cmp() uses the total ordering, but we might change it to give a
DomainError for NaN arguments