Yes, you are right with standard sort. However Chris and I looked at it and the problem is the locale:

```
LC_ALL=C time sort -u
```

is even faster than the python sort (almost twice as fast in our test). Can you please change that to save processing time?