zopefoundation/zope.tal — pull request #1 by mauritsvanrees (2013-09-21 21:54:53+00:00)
Title: Fix possible UnicodeDecodeError in warning when msgid already exists.

Due to a difference in how i18n:attributes and i18n:translate are handled, a snippet such as this (in PloneSoftwareCenter) can give problems:

```

Read more…

```

i18ndude (a program for extracting msgids) uses zope.tal and zope.tal gives an error while trying to print a warning message that a msgid is already in the catalog with a different default. The i18n:attribute gets switched to unicode and the i18n:translate stays a string and they are different so zope.tal tries to display a warning, but it fails:

```python
$ ~/buildout/tools/src/i18ndude/bin/i18ndude rebuild-pot --pot ./plonesoftwarecenter.pot --create plonesoftwarecenter ../skins/plonesoftwarecenter/
There was an error processing ../skins/plonesoftwarecenter/psc_release_listing.pt
Traceback (most recent call last):
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/i18ndude/extract.py", line 508, in tal_strings
metal=False)()
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 270, in __call__
self.interpret(self.program)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 342, in interpret
handlers[opcode](self, args)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 877, in do_useMacro
self.interpret(block)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 342, in interpret
handlers[opcode](self, args)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 532, in do_optTag_tal
self.do_optTag(stuff)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 517, in do_optTag
return self.no_tag(start, program)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 512, in no_tag
self.interpret(program)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 342, in interpret
handlers[opcode](self, args)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 828, in do_loop_tal
self.interpret(block)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 342, in interpret
handlers[opcode](self, args)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 532, in do_optTag_tal
self.do_optTag(stuff)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 517, in do_optTag
return self.no_tag(start, program)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 512, in no_tag
self.interpret(program)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 342, in interpret
handlers[opcode](self, args)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talinterpreter.py", line 729, in do_insertTranslation
xlated_msgid = self.translate(msgid, default, i18ndict, obj)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talgettext.py", line 89, in translate
default=default, position=self.position)
File "/Users/mauritsvanrees/buildout/tools/src/i18ndude/src/zope.tal/src/zope/tal/talgettext.py", line 140, in translate
(msgid, self.file+':'+str(position), msgid.default.encode('utf-8'), existing_msgid.default.encode('utf-8'), references)).encode('utf-8'), file=sys.stderr)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9: ordinal not in range(128)
```

Wichert had fixed this a few years ago, but the later changes for supporting Python 3 made it fail. I have fixed it again and added a test. Tox tells me it works on python 2.6, 2.7 and 3.3 (I do not have pypy).

That does mean the tests print a warning now, which I guess is unavoidable, as the point of the added test is checking that printing the warning does not give an error.

```
$ bin/test
Running zope.testrunner.layer.UnitTests tests:
Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
Warning: msgid 'foo' in psc_release_listing.pt:42 already exists with a different default (bad: Read still more…, should be: Read more…)
The references for the existent value are:
psc_release_listing.pt:7
psc_release_listing.pt:13

Ran 219 tests with 0 failures, 0 errors and 0 skipped in 0.431 seconds.
Tearing down left over layers:
Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
```