Refactor sort by to be simpler