harelba/q — pull request #17 by JKrag (2014-01-09 14:29:45+00:00)
Title: Survive unicode chars in data when doing beautify output

Hi.
I have been using your q script the last couple of days on a 200K lines csv file of cat pedigree data :-)
I ran into a few problems along the way, pertaining to the exact nature of my data.
I will try to address these myself and send pull-requests if possible, but I might end up sending an "issues" if there is anything I can't easily fix.
The first problem i came across is that the beautify feature fails hard on (certain?) unicode characters in the data.
e.g. lines like the following:
161|M|(N)Bjørn's Cornelis|BUR e|15-08-1999|NRR LO 125821|134316|125854
fail with:
Traceback (most recent call last):
File "/usr/local/bin/q", line 497, in
max_lengths = determine_max_col_lengths(m)
File "/usr/local/bin/q", line 456, in determine_max_col_lengths
new_len = len(str(m[row_index][col_index]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf8' in position 15: ordinal not in range(128)

I found out that I can seemingly fix the problem with this very minimal fix. As you are probably vastly more experienced in Python, you might very well know a better way to do this, but it seems to work for me.