(1) Fix parsing -DOCSTART- markers. (2) Ignore newlines in documents file (2) Before benchmarking do a single sort pass to create a unique entities file. We need this because MPH function generation fails after complaining the presence of duplicate entries. (3) Add a unieque flag to ExternalSort to support this. (4) Fix key flag in external sort. The end field is made explicit as it defaults to the last field if not specified. (5) Modify tests.
Refactor document parsing code.