Fix sorting on NaN values in aggregations.

Closes #5236.
Added unit tests for terms and histogram sorting on sub aggregations that return NaN.