akumoth/peewee-issues huggingface dataset:
Issue: Proposal: Peewee should provide a standard definition of database errors and wrap the database driver errors in those
Author: gciding
State: closed
Created: 2013-11-26 01:20:25

Peewee acts as an ORM and therefore it is not necessary to import the raw/underlying database python library when you create a database. However, if you want to perform tasks that rely on processing underlying errors (such as a databse Integrity Error) you will need to import the underlying database library.

To illustrate a situation where it becomes necessary to import `sqlite3` in order to process integrity violations, I have set out a use case.
#### Existing Behavior (plahouse.py:Loader)

If you wanted to import rows of data and, lets say, one of those rows violated a unique constraint on the table, Peewee will log an error as well as well as raise the _underlying_ exception.

So, just say I wanted to ignore duplicates in my import, I could subclass the `Loader` class and override the `load` method:

``` python
import sqlite3
from playhouse import Loader

class IgnoreDuplicatesLoader(Loader):
    """ A loader that will continue importing even if it sees a duplicate
    value.  Duplicate Values will be skipped.
    """
    def load(self):
        if not self.fields:
            self.analyze_csv()
        if not self.field_names and not self.has_header:
            self.field_names = [
                'field_%d' % i for i in range(len(self.fields))]

        with self.get_reader(self.filename, **self.reader_kwargs) as reader:
            if not self.field_names:
                self.field_names = map(self.clean_field_name, reader.next())
            elif self.has_header:
                reader.next()

            ModelClass = self.get_model_class(self.field_names, self.fields)

            with self.database.transaction():
                ModelClass.create_table(True)
                for row in reader:
                    try:
                        insert = {}
                        for field_name, value in zip(self.field_names, row):
                            if value:
                                insert[field_name] = value.decode('utf-8')
                        if insert:
                            ModelClass.insert(**insert).execute()
                    except sqlite3.IntegrityError, e:
                        print e
                        print "Continuing..."
                        continue

        return ModelClass
```

Note that I had to import `sqlite3` in order to catch that error.  This means that this module code is now tied to an SQLite implementation, and someone cannot change the DB backend without it breaking.  
#### Proposed Operation

Peewee should raise it's own errors which map to the underlying error raised by the database.  So in the above situation, `sqlite3.IntegrityError` would map to `peewee.IntegrityError`


Yes, I agree.


I've added standard exceptions and wrapped the connect / close / execute methods to ensure they raise the "peewee" version where possible:

2d6f43358ebdacb12a6d7319f3fc2c699f22a053
Issue: Playouse - CSV Loader - should accept file object, or handle files in universal-newline mode
Author: gciding
State: closed
Created: 2013-11-26 00:07:08

In file `playhouse/csv_loader.py`

When you use `load_csv`, it expects a filename of the CSV file. If the CSV file has been saved in a slightly different format with respect to New Lines (which is common with csv files generated by MS Excel), the standard `open(filename, 'r')` won't work.  

```
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
```

The code which opens the provided filename should read `open(filename, 'rU')` or alternatively, you should be able to provide the function a file object.


Agreed, good idea.


Added in 04129b097ceaf0a7a8f93ddff072d8f4b3131348
Issue: comments on code
Author: akaRem
State: closed
Created: 2013-11-21 20:57:45

Could you, please, add more comments on PeeWee's code?
You've made very cool and interesting framework. And I'd like to study how it works and get involved into development. But.. in many places its hard to understand which functions do what and why.

BTW, Flask-style comments are the best i've ever seen.


Noted, thank you!
Issue: proposal on QueryCompiler._parse()
Author: akaRem
State: closed
Created: 2013-11-21 20:47:53

You have very long if-elif-elif-...else in QueryCompiler._parse(), where it mostly works with sql, params..
It would be better to use this something like this construction:

```
 def non_inline_function(a,b,c):
     # do some /1
     # do some /2
     # do some /3
     return ('sql', 'pqrams')

 mapper = {
     Func: lambda a,b,c: (func_A(a,b), func_B(c)) # very simple, inline
     Clause: non_inline_function # not simple, defined above
     Param: ... # etc
 } # or, may be some variant with closure

def parse(node, alias_map, conv):
    try:
       sql, params = mapper[type(node)](node, alias_map, conv)        
    except KeyError:
        # other more common or more complicated situations
        pass
```

So peewee will not loop over all ef-elif-elif-.. and will use "cached" lambdas (or even not lambdas).
It would be speed-up (may be not big, but.. speed-up) with cost of readability (it is possible to make it even more readable).

Right now this function is actually dispatches conversions depending on node type, so why not to make it in more traditional way?


I'm not sure I agree with this proposal.  The "cost" of evaluating a SQL expression only occurs once, when the SQL is generated and handed off to the cursor, so I'm not sure I see the need to refactor this for performance.  Additionally `isinstance()` is more flexible than calling `type()` because it understands derived classes.
Issue: How to use limit()?
Author: kramer65
State: closed
Created: 2013-11-19 09:08:00

I'm trying to use limit as follows:

```
one_ticket = Ticket.select().limit(1)
print one_ticket.count()
```

This prints out 5 however. Does anybody know what's wrong here?


I think that's, oddly, correct.  I ran this in postgresql:

``` sql
# select count(*) from entry limit 10;
 count 
-------
   118
(1 row)
```

If you wrap the select you get what you expect:

``` sql
select count(*) from (select 1 from entry limit 10) _;
 count 
-------
    10
(1 row)
```

I added a commit (304c054c99277) so you can accomplish this in peewee:

``` python
Ticket.select().limit(1).wrapped_count(clear_limit=False)
```
Issue: Allow null column migrations
Author: cam-stitt
State: closed
Created: 2013-11-18 21:35:40

Just a simple option that allows people to force it to try to add the column if null is True. I need this for my migrator at https://github.com/cam-stitt/arnold.


I'm not sure I follow these changes.


I think it can be closed. It was for a specific thing I had been working on but I don't think it is actually needed :)
Issue: playhouse PostgresqlExtDatabase/PostgresqlExtCompiler
Author: akaRem
State: closed
Created: 2013-11-16 20:10:34

Im on Mac OS X 10.9 with PyPy 2.1.0 and PostgreSQL 9.3.1, having
peewee 2.1.5 and psycopg2ct 2.4.4
I study how peewee builds SQL strings

When I trying to use Postgres extension like this:

```
# -*- coding: utf-8 -*-
from psycopg2ct import compat
compat.register()

import peewee as pw
import playhouse.postgres_ext as pp

db = pp.PostgresqlExtDatabase('mydb')

class MyModel(pw.Model):
    class Meta:
        database = db
    some_value = pw.IntegerField()
    some_uuid = pp.UUIDField()

print pp.PostgresqlExtCompiler().create_table(MyModel)
```

I Have this error message:

```
/Users/user/Projects/pypy/bin/pypy /Users/user/Projects/pypy-flask/pp-test.py
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "/Users/user/Projects/pypy-flask/pp-test.py", line 17, in <module>
    print pp.PostgresqlExtCompiler().create_table(MyModel)
  File "/Users/user/Projects/pypy/site-packages/peewee.py", line 1102, in create_table
    return ' '.join(self.create_table_sql(model_class, safe))
  File "/Users/user/Projects/pypy/site-packages/peewee.py", line 1093, in create_table_sql
    columns = map(self.field_sql, meta.get_fields())
  File "/Users/user/Projects/pypy/site-packages/peewee.py", line 1061, in field_sql
    attrs['column_type'] = self.get_field(field.get_db_field())
  File "/Users/user/Projects/pypy/site-packages/peewee.py", line 795, in get_field
    return self._field_map[f]
KeyError: 'uuid'

Process finished with exit code 1
```

These errors go away if I add `'uuid'` to `PostgresqlExtCompiler.field_map`:

```
# -*- coding: utf-8 -*-
from psycopg2ct import compat
compat.register()

import peewee as pw
import playhouse.postgres_ext as pp

pp.PostgresqlExtCompiler().field_map['uuid'] = 'UUID'

db = pp.PostgresqlExtDatabase('mydb')

class MyModel(pw.Model):
    class Meta:
        database = db

    some_value = pw.IntegerField()
    some_uuid = pp.UUIDField()

print pp.PostgresqlExtCompiler().create_table(MyModel)

# but this also patches pw.QueryCompiler:    
import pprint
pprint.pprint(pw.QueryCompiler.field_map)

/Users/user/Projects/pypy/bin/pypy /Users/user/Projects/pypy-flask/pp-test.py
CREATE TABLE "mymodel" ("id" INTEGER NOT NULL PRIMARY KEY, "some_value" INTEGER NOT NULL, "some_uuid" UUID NOT NULL)
{'bigint': 'BIGINT',
 'blob': 'BLOB',
 'bool': 'SMALLINT',
 'date': 'DATE',
 'datetime': 'DATETIME',
 'decimal': 'DECIMAL',
 'double': 'REAL',
 'float': 'REAL',
 'int': 'INTEGER',
 'primary_key': 'INTEGER',
 'string': 'VARCHAR',
 'text': 'TEXT',
 'time': 'TIME',
 'uuid': 'UUID'}

Process finished with exit code 0
```

**The question is:**
Am I doing something wrong? 
Or is it really need to add custom field types to `QueryCompiler`?
Or is it a bug in `PostgresqlExtDatabase.register_fields(...)` (or, may be, somewhere else)?


The problem is in how you're instantiating the compiler.  Try this instead, it will work:

``` python
compiler = db.compiler()
print compiler.create_table(MyModel)
```


@coleifer Thanks! It works.


Awesome!
Issue: How to do an internal reference to the model itself?
Author: kramer65
State: closed
Created: 2013-11-14 16:20:41

I'm building a website in which Admins (which are of course also regular Users) can add new users. For this reason, my User model looks something like this:

```
class User(db.Model, BaseUser):
    username = CharField()
    password = CharField()
    added_by = ForeignKeyField(User, related_name='users_created')
```

Unfortunately, this throws a `NameError:  name 'User' is not defined.`

Is there a way to do this using peewee?


Check out http://peewee.readthedocs.org/en/latest/peewee/models.html#self-referential-foreign-keys

tl;dr

``` python
added_by = ForeignKeyField('self', related_name='users_created')
```
Issue: How to select on basis of ForeignKeyField.attribute == 'something'
Author: kerqicce
State: closed
Created: 2013-11-10 18:39:53

I have two models:

```
class Primary(db.Model):
    prim_attr = CharField()

class Secondary(db.Model):
    primary = ForeignKeyField(Primary, related_name='secondaries')
    sec_attr = IntegerField()
```

I want to get all secondary items where the primary.prim_attr equals 'something'. So I tried to do like this:

```
result = Secondary.select().where(Secondary.primary.prim_attr == 'something')
```

But I get an error: AttributeError: 'ForeignKeyField' object has no attribute 'prim_attr'.

I can get the Primary object first, and then insert in the second query like this:

```
result = Secondary.select().where(Secondary.primary == primaryObject)
```

But it is easier if I can do it in one line. Is that possible?


You're close, but you will need to perform a join.  Check out the docs on joining: http://peewee.readthedocs.org/en/latest/peewee/querying.html#joining-tables

``` python
result = Secondary.select().join(Primary).where(Primary.prim_attr == 'something')
```
Issue: How to return choice of a IntegerField as string?
Author: kerqicce
State: closed
Created: 2013-11-10 12:01:40

I have one table with an IntegerField and in it two choices:

```
class Something(db.Model):
    myattr = IntegerField(choices=((1, 'option one'), (2, 'option two')))
    def __unicode__(self):
        return myattr
```

The **unicode**() method obviously just return the integer 1 or 2. How can I return strings of the choices ('option one' or 'option two')?

Thank you


Typically I store my choices in a class-level constant so they are easier to access and compare against:

``` python

class Something(Model):
    FOO_YES = 1
    FOO_NO = 2
    FOO_CHOICES = (
        (FOO_YES, 'Yes'),
        (FOO_NO, 'No'))
    foo = IntegerField(choices=FOO_CHOICES)

    def __unicode__(self):
        return dict(self.FOO_CHOICES)[self.foo]
```


Thanks to you. That is a super good tip! I am going to use it immediately!!
Issue: How to create conditional transactions?
Author: kerqicce
State: closed
Created: 2013-11-09 22:07:57

Hello Coleifer. Thank you for creating so good software, we love using it!

Today we looked at code in which we need to make something like conditional transactions. I do not know if that is an official name, but for us this means that we want to check if a certain value is present in the database. If it is, we want to do some insert, and if it is not, we want to do a different insert. But in the time between the read and the insert, a different process could have inserted the searched for record, which would create inconsistencies.

My question being: we can include the select into the transaction like in the code below. This seems to work, but will that indeed make the select, the if statement, and the write all together atomic? In other wording; will this make it impossible to insert a user with username 'someUserName' between the read, and the write?

```
with db.transaction():
    result = User.select().where(User.username == 'someUserName')
    if result.count() == 0:
        insertObj = Something()
        insertObj.someattr = 123
        insertObj.save()
    elif result.count() == 1:
        insertObj = Something()
        insertObj.someattr = 456
        insertObj.save()
```

Also: is using result.count() the correct way to see if there is one occurrence, or is there a better/faster/more efficient way?


Check your database's documentation on `SELECT FOR UPDATE` ([link to peewee docs](http://peewee.readthedocs.org/en/latest/peewee/api.html#SelectQuery.for_update)).

If you're using postgres, here are some docs. http://www.postgresql.org/docs/9.0/static/sql-select.html#SQL-FOR-UPDATE-SHARE

Does this answer your question?


Thank you very much for the information. This is a perfect answers to my question!

I have never worked with FOR UPDATE before, but I will read in the PostgreSQL documentation about it. Do I need any addon to peewee for this, or will it work with PostgreSQL out of the box?

Do you know if there is any peewee example code using for_update?


It will work out of the box.  The tests are the only place there is example code for this feature, around line 3085.
Issue: Presumably internal debug message gets printed to stderr
Author: pitchforks
State: closed
Created: 2013-11-01 09:57:46

Few days ago I have upgraded from 2.1.4 to 2.1.5 and now everytime my CLI program runs, it dumps this line to stdout:

```
No handlers could be found for logger "peewee"
```

I have searched for this text in the sources for 2.1.5 but didn't find anything.

This looks related to logging module but my own program doesn't make use of logging module at all.


Peewee supports logging using python's standard library logging module.  The logger can help you debug queries and also supports logging exceptions executing queries.  Python docs have you covered: 

http://docs.python.org/3.1/library/logging.html#configuring-logging-for-a-library
Issue: Model cache being inconsistent when the value is set by id
Author: ricoboni
State: closed
Created: 2013-10-23 16:47:52

Project have a ForeignKeyField referencing Client
The output is from my projects iPython shell

```
In [1]: p = Project.get(id=581)

In [2]: p.client
Out[2]: <models.Client at 0x1f6d790>

In [3]: p.client.id
Out[3]: 115

In [4]: p.client = 123

In [5]: p.client.id
Out[5]: 115
```


Thanks!
Issue: Postgres Mixed Encoding SQLASCII Issue
Author: caelx
State: closed
Created: 2013-10-22 23:05:48

I have to deal with a postgres database which has mixed encodings (mix utf-8 and latin1) in some of the text fields.  Now I don't know what kind of monster doesn't properly encode text before it gets inserted into the database, but it is what it is.

I created a ByteArrayField to deal with the raw text, but I was still getting errors.  After tracing them, it's related to the following lines where you force everything to be unicode:

if psycopg2:
    import psycopg2.extensions
    psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
    psycopg2.extensions.register_type(psycopg2.extensions.UNICODEARRAY)

Commenting out the extensions fixes it for me, I know it's terrible practice and but it would be nice to have this as a configurable setting so that in the future I don't have to modify the peewee source.

Thank you very much!


Just out of curiosity have you tried messing with the encoding on the connection object?


Thanks for getting back to me.

Yes, the database is set for client encoding USASCII but I tried changing it to UTF8.  This did not help though because there were still unicode conversion errors for some of the characters which aren't valid UTF8 characters.

The invalid characters were \x93 and \x94


OK, cool.  Well it looks like I can register those converters on the connection, so this should be pretty straightforward to fix.


Thanks!

This project is giving me a hell of a headache, appreciate this


Should be fixed in f1607416474ae99ccc16b058172cbce9d0ebd230 .

You'll need to turn off the unicode converter registration code:

``` python
db = PostgresqlDatabase('...')
db.register_unicode = False  # disables registering the unicode handlers on a per-connection basis.
```

If for some reason this doesn't solve your problem please reopen.
Issue: Add support for MySQL connection recycling
Author: arnuschky
State: closed
Created: 2013-10-17 09:26:05

We're getting the following error on our flask project:

```
  SQLError: (OperationalError) (2006, ‘MySQL server has gone away’)
```

This problem was implicitly mentioned in https://github.com/coleifer/peewee/issues/222 for example.

SQLalchemy addresses the problem as follows: https://github.com/mitsuhiko/flask-sqlalchemy/commit/05c51933c4972b47c1420fdc29429da9b67b75d8
Associated issue: https://github.com/mitsuhiko/flask-sqlalchemy/issues/2

Would it be possible to implement this in PeeWee as well?


Now that I think of it, this issue maybe more appropriate for flask-peewee, but I am not sure.


I'm a little unclear on how your particular issue is arising.  If you're using flask-peewee's `Database` object, it should be opening and closing the connection on each request.  If you're using a multi-threaded wsgi server have you tried configuring your database with `threadlocals=True`?  That would eliminate the possibility that a second thread is closing the conn opened by the first thread.


Yes, we are using gunicorn with multiple workers and without threadlocals=True. Must have missed it in the docs (now I even found another old issue regarding this)! I guess that his issue is invalid then, but I will test first.


You could use `sql_error_handler` to handle this problem. Anyway, this problem is caused by the connection is idle for a long time(longer than timeout seconds, default 28800 seconds)


@arnuschky let me know if `threadlocals` resolved it!


It seems to resolve the problem. Thanks!


`threadlocals=True` can not resolved this issue~
after 28800 seconds, Error Msg ‘MySQL server has gone away’ appear agian!

---

read peewee doc again, use mysql_db.connect() & mysql_db.close() for each request in CherryPy.


You can also check out http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool for connection pooling and recycling.


Thanks for your big help. `playhouse.pool` is the final solution which I needed~


I use peewee with flask.

Peewee database connection : 

```
db = PooledMySQLDatabase(database,**{
            "passwd": password, "user": user,
            "max_connections":None,"stale_timeout":None,
            "threadlocals" : True
        })

@app.before_request
def before_request():
    db.connect()

@app.teardown_request
def teardown_request(exception):
    db.close()
```

After mysql error that "MySQL server has gone away (error(32, 'Broken pipe'))", select queries works without problem, but insert,update,delete queries don't work.

On insert,update,delete queries works behind(in mysql) but peewee throw this errors.

```
(2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
```


That's odd, not sure what that might mean.


@coleifer I have two different applications with peewee and flask. They have same problem. Select queries has no problem but insert,update,delete always throws error which above. I will do test with an basic peewee and flask app and I will write the result of the test.
Issue: Can't refer to model which not exists yet
Author: wtld
State: closed
Created: 2013-10-14 13:51:28

``` python
class Topic(Model):
    title = CharField()
    last_post = ForeignKeyField(Post)

class Post(Model):
    topic = ForeignKeyField(Topic)
    body = CharField()
```

How to refer 'Post' model from 'Topic' model?


By default you can use topic.post_set.
If you change  topic = ForeignKeyField(Topic) to topic = ForeignKeyField(Topic,related_name='posts')
you can use topic.posts.


Unfortunately peewee does not support this kind of relationship.  This is partly a limitation of python, but also in part due to my own feeling that these kinds of circular dependencies indicate a problem with the schema.

To work around you really only have one option:

``` python
class Topic(Model):
    title = CharField()
    last_post_id = IntegerField(null=True)

    @property
    def last_post(self):
        return Post.get(Post.id == self.last_post_id)

class Post(Model):
    topic = ForeignKeyField(Topic)
    body = CharField()
```
Issue: Is it possible to use getter in the model class?
Author: a-nanasov
State: closed
Created: 2013-10-08 10:02:47

Hi, I've tried to use @propery decorator with getter function:

``` python
class Location(db.Model):
    def __init__(self):
        self._name = CharField(unique = True)

    @property
    def name(self):
        return self._name

    class Meta:
        db_table = 'locations'
```

but got the following SQL query 
`<SELECT t1.`id`FROM`locations`AS t1 WHERE ('<property object at 0x031E0E40>' = 'loc01')>`
Could you please provide an example for getter? Thanks in advance!


That is an interesting setup. Unfortunately, due to the way peewee uses metaclasses, you cannot define your fields in `__init__`.  I assume in practice there's a reason you're using a property -- perhaps to modify the value in the database column? Try:

``` python

class Location(db.Model):
    name_column = CharField(unique=True)

    class Meta:
        db_table = 'locations'

    @property
    def name(self):
        return self.name_column
```