edsu/pymarc — Issue #18: unicode leader fields
Author: gugek
Created: 2011-10-28 13:33:53+00:00
State: open

I ran into an issue with serializing from marc_xml into marc21 transmission. I get a UnicodeDecode error if any of my data is beyond the default encoding (ASCII) because there is a mix of types. This could also be an issue if someone modifies the leader through record.leader and somehow puts in a unicode object instead of an ordinary string object. It doesn't show up in the other as_marc tests because there isn't any writing out of both unicode data and leaders.

I've got a test at: https://gist.github.com/1322248

```python
def test_writing_unicode_leader(self):
"""Test serializing a record with a unicode leader"""
record = Record()
record.add_field(Field(245, ['1', '0'], ['a', unichr(0x1234)]))
record.leader = u' a ' # unicode leader here.
writer = MARCWriter(open('test/foo', 'w'))
writer.write(record)
writer.close()

reader = MARCReader(open('test/foo'), to_unicode=True)
record = reader.next()
self.assertEqual(record['245']['a'], unichr(0x1234))

os.remove('test/foo')
```

Here is my traceback:

Traceback (most recent call last):
File "/.../marc8.py", line 69, in test_writing_unicode_leader
writer.write(record)
File "/...", line 40, in write
self.file_handle.write(record.as_marc())
File "/...", line 249, in as_marc
return self.leader + directory + fields
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 4: ordinal not in range(128)

I can submit a patch, but my problem is I am kind of mystified by where I should be dealing with this:

+ should it be in the ingest in marc_xml.py, I could sort of backpedal the leader into string.
+ should it be in the as_marc() serialization, where we can decode the leader if it is unicode.
+ or should it be pulled into the larger unicode issue that some pymarc conversation has talked about; adding __unicode__ to the field and record objects.