mozilla-metrics/telemetry-toolbox — pull request #7 by espindola (2013-02-21 19:04:05+00:00)
Title: Fix parsing of lines with "special" leading charters.

The first byte in each key can have any value from 0 to 255. This can confuse
pig which was trying to handle them as utf-8. We can avoid the problem by
using a bytearray for the key.

We still have the problem that that one byte can be a character like \t or
\n. To be able to parse this we cannot simply read a line and split on tabs.
We have to remember where in the file we are and read the next expected token.

This pull request includes an extra example file with lines starting with \r,
\n, \t and a non printable byte (0xbf).

With this change I was able to process a dump (Feb 15) without any errors.

Last but not least, this pull request removes the toplevel try..except that
just logs erros. We control this part of the format, and it is better to
know right away that something is wrong than to skip entries and produce
files with the wrong key.