Issue: Possible unicode issue
Author: rcarmo
State: closed
Created: 2013-06-07 16:46:10

Hello there, 

I've been trying to get the following to work right:

<pre>
import os, sys
sys.path.append('../lib')
from peewee import Model, PostgresqlDatabase, CharField

db = PostgresqlDatabase("test")

class Test(Model):
    name = CharField(null=False)

    class Meta:
        database = db

Test.create_table(True)

t = Test.create(name=u'M\u00f6rk Gryning')
</pre>


However, regardless of what I try to do with that unicode string, I always get a traceback similar to this:

<pre>
Traceback (most recent call last):
  File "unicode.py", line 15, in <module>
    t = Test.create(name=u'M\u00f6rk Gryning')
  File "../lib/peewee.py", line 2294, in create
    inst.save(force_insert=True)
  File "../lib/peewee.py", line 2383, in save
    new_pk = insert.execute()
  File "../lib/peewee.py", line 1712, in execute
    return self.database.last_insert_id(self._execute(), self.model_class)
  File "../lib/peewee.py", line 1400, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "../lib/peewee.py", line 1811, in execute_sql
    res = cursor.execute(sql, params or ())
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
</pre>


This doesn't make much sense to me, since from what I can understand the named argument should still be a Unicode string when it gets to `cursor.execute` - so I'm stumped as to the actual cause.


That's interesting... I know that there are some tests that cover unicode functionality.  Can you create a testcase to reproduce this bug?


Erm.. the snippet of code I posted reproduces it on all my machines. That is my testcase, so to speak :)

On Jun 11, 2013, at 03:48 , Charles Leifer notifications@github.com wrote:

> That's interesting... I know that there are some tests that cover unicode functionality. Can you create a testcase to reproduce this bug?
> 
> —
> Reply to this email directly or view it on GitHub.


I pushed a testcase and was unable to reproduce.  From the traceback it looks like the issue might be happening inside psycopg2.  What is the encoding of your connection?

``` python

# here is the encoding I have when running the tests
>>> test_db.get_conn().encoding
'UTF8'
```


Mine returns 'SQLASCII', which doesn't make sense - I've tried running my code with the psycopg2 unicode extensions enabled (like jokull added a year back in https://github.com/coleifer/peewee/commit/4a2ba5bd528c1ddd40d79cdd10438f3a5c270c4b) and it still fails.

On Jun 11, 2013, at 15:11 , Charles Leifer notifications@github.com wrote:

> I pushed a testcase and was unable to reproduce. From the traceback it looks like the issue might be happening inside psycopg2. What is the encoding of your connection?
> 
> # here is the encoding I have when running the tests
> 
> > > > test_db.get_conn().encoding
> > > > 'UTF8'
> > > > —
> > > > Reply to this email directly or view it on GitHub.


Cool, take a look here: http://www.postgresql.org/docs/9.2/static/multibyte.html -- there are a couple solutions:
- `initdb` when setting up your cluster
- `createdb` when creating your database
- `set client encoding`


Hmm. Perhaps this should be added to the docs? The defaults in Ubuntu and Debian are to start with SQL_ASCII.

And your tests probably ought to reflect that, since you're testing against a UTF8 cluster - and hence won't spot the general case... :)

On Jun 11, 2013, at 16:26 , Charles Leifer notifications@github.com wrote:

> Cool, take a look here: http://www.postgresql.org/docs/9.2/static/multibyte.html -- there are a couple solutions:
> 
> initdb when setting up your cluster
> createdb when creating your database
> set client encoding
> —
> Reply to this email directly or view it on GitHub.