Use Sorts Mill directly

Because that is what I’m actually using, also delay some imports to when
they are actually needed.