Add a test for NaN in sorting.
Fix a bug that NaN causes wrong results in sorting.