Only keep one item in the sorter cache

Once a new sorter is built, the old one is not needed any more, so it is just
wasting memory to keep it around.