akumoth/peewee-issues huggingface dataset:
Issue: UTF-8/Unicode Issues
Author: rcarmo
State: closed
Created: 2012-10-12 15:58:44

There seem to be some UTF-8 issues lingering with Python 2.7.

While trying to set a text field to "Operações", I got the following stack dump:

Traceback (most recent call last):
...
  File "/home/rcarmo/test/devices.py", line 81, in activate_device
    d.save()
  File "/home/rcarmo/test/peewee.py", line 1992, in save
    update.execute()
  File "/home/rcarmo/test/peewee.py", line 1397, in execute
    result = self.database.execute(self)
  File "/home/rcarmo/test/peewee.py", line 1505, in execute
    sql, params = query.sql(self.get_compiler())
  File "/home/rcarmo/test/peewee.py", line 1394, in sql
    return compiler.parse_update_query(self)
  File "/home/rcarmo/test/peewee.py", line 837, in parse_update_query
    sets, params = self._parse_field_dictionary(query._update)
  File "/home/rcarmo/test/peewee.py", line 828, in _parse_field_dictionary
    val_params = [field.db_value(vp) for vp in val_params]
  File "/home/rcarmo/test/peewee.py", line 321, in db_value
    return value if value is None else self.coerce(value)
  File "/home/rcarmo/test/peewee.py", line 387, in coerce
    value = unicode(value or '')


Ouch -- could you paste the actual exception?


I've since worked around it by doing:

d.name = unicode(name, 'utf-8')  

In my code. I'm guessing there's a little more to this, but will try to check again later on.

On 12/10/2012, at 16:59, Charles Leifer notifications@github.com wrote:

> Ouch -- could you paste the actual exception?
> 
> —
> Reply to this email directly or view it on GitHub.


I have been putting off learning about encoding for so long, i'm not gonna change today...but maybe you might understand a bit better.  Do you know what might be different between your example and this test case?

https://github.com/coleifer/peewee/blob/master/tests.py#L918


The tests are using a unicode string literal.  If you pass in an encoding to the unicode builtin, e.g. `unicode(..., 'utf-8')` then it will try to decode the value passed in.  If this value is an 8-bit string its fine, but if you pass in unicode you get `TypeError: decoding Unicode is not supported`.


Well, I'm passing in a POST value after cleaning up from bottle.py. I suppose it's not really unicode, but haven't had time to figure out the differences yet.
Issue: Disabled autocommit on select
Author: obensonne
State: closed
Created: 2012-10-11 21:37:35

The `Database.execute_sql()` considers 2 parameters when deciding if to do a _commit_ after executing a statement:

``` python
def execute_sql(self, sql, params=None, require_commit=True):
    cursor = self.get_cursor()
    res = cursor.execute(sql, params or ())
    if require_commit and self.get_autocommit():
        self.commit()
    logger.debug((sql, params))
    return cursor
```

Two issues:
1. Shouldn't the condition be _or_-ed? At least the wording _require_ suggests this.
2. If no, why does a `SelectQuery` has set `require_commit = False`?

A continously running application which only does _selects_ opens a transaction once and keeps it open forever (tested with PostgreSQL).

Looks like a bug, isn't it?


My thinking was that we should only explicitly commit when, first of all, it is warranted (by being a query that writes something) and second of all, that the commit isn't being managed elsewhere.

I am not aware of the need to open a transaction when doing SELECTs.  That seems odd -- can you provide a little more info?


Thanks for reporting -- i didn't realize that psycopg2, as you said, opens a transaction every query.  This should be fixed now, and select queries will have their transactions handled correctly


Great, thanks for the quick fix!
Issue: Print SQL queries
Author: ripperdoc
State: closed
Created: 2012-10-10 23:47:16

I updated from 1.0 to 2.0 and got into a bit of a mess - which has mostly cleared up now. One thing though - how do I print sql queries? I used to be able to do SelectQuery.sql() but not it requires a compiler argument, but what should I put into that? I cannot find any mention in the docs...

Thanks!


I added a little helper here:
https://github.com/coleifer/peewee/commit/a5ec7bb90b1882db95c6f4eae94ea373f7f6a50f


Great, thanks!


@coleifer Can you please give an example of how to use that helper function?


That function actually went away. Now you can just call:

``` python
q = MyModel.select().where(MyModel.id == 3)
print q.sql()
sql, params = q.sql()
```


@coleifer What if the result of `q = MyModel.select().where(MyModel.id == 3)` fails and caused an error?  Is it possible to get the query that was sent to the DB and failed?

You can log all queries.

```python
import logging
logger = logging.getLogger('peewee')
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)
```

Or you can add try/except or use an interactive debugger...it's just Python, remember!

http://docs.peewee-orm.com/en/latest/peewee/database.html#logging-queries

@TimothyBramlett you can use the logging package:

```
import logging
logging.basicConfig(level=logging.DEBUG)
```

Peewee will print all queries that it sends to the DB.

As per peewee `3.13`, if you are connecting to a postgres database with peewee's underlying `psycopg2` (thanks @coleifer), you can print SQL queries by first getting the cursor, then calling the `mogrify()` method for your query.

```python
query = <YOUR PEEWEE QUERY>

cur = database.cursor()
print(cur.mogrify(*query.sql()))
```
Where `database` is the Peewee Database object representing the connection to your database.

`mogrify` may be a psycopg2-only thing, and no supported by other drivers. For example, the standard lib sqlite3 does not provide this method.
Issue: Support SelectQuery on a view
Author: jhorman
State: closed
Created: 2012-10-04 03:16:00

I am writing some queries that are beyond what peewee can/should do. Specifically I need to be able to union together a result set. I figured I would just create a view, and then somehow set that view into the SelectQuery. I think that would be a really nice simple option, where you just use the view name in place of 

```
self.model._meta.db_table
```

I ended up subclassing, but it isn't easy since the clone method always creates a new instance of SelectQuery, so each call to .where, etc, ends up getting rid of my subclass. Would also be nice if clone could use the current class name to create the instance, so I could just override clone, call super, and then copy the viewname as well.

It also isn't easy b/c sql_meta uses db_table directly

```
table = self.qn(self.model._meta.db_table)
```

vs calling a method like get_table or something that could be overridden.

I think it makes more sense though just to support an optional "view" parameter?

Thanks for any feedback.


I wonder if it would work just to create a model and have it specify its db_table attribute as the view.  If the view "looks" like one of your existing models, you should also be able to subclass the exisitng model, e.g.:

``` python

class MyModel(Model):
    # fields ...

class MyViewModel(MyModel):
    class Meta:
        db_table = 'this_is_actually_the_view'
```

If I'm missing something essential to the issue please let me know but I belive the above should work.

More to the point, you have raised some excellent points about places where the API is brittle.  I've rewritten peewee from the ground up and am finishing the work before merging it into master, but you can check the branch:
https://github.com/coleifer/peewee/tree/unstable/2.0

It provides better hooks with a lot less insanity :)


In this case I am using different views for different queries. I have a "Video" object, and I have 3 different views for pulling out videos based on some complicated rules. I forgot to mention that I was going to use RawQuery, which would have almost worked, except RawQuery and flask-peewee don't work well together. flask-peewee assumes a SelectQuery in a few places (like PaginatedQuery).

I will check out the branch in the meantime too though.


Cool, so just to clarify it won't work to do something like:

``` python

class VideoViewOne(Video):
    class Meta:
        db_table = 'video_view_one'

class VideoViewTwo(Video):
    class Meta:
        db_table = 'video_view_two'
```

And then:

``` python

VideoViewOne.select() # <--- pull video objects out of the view "video_view_one"
```


Oh... um.... yeah that might work. I think I will need to see how that plays with my flask-peewee setup. Thanks for the tip.


Just wanted to say I finally made the jump to your 2.0 release. Really nice. I really like the changes to the query language. I also was able to kill my fork and extend flask-peewee with my JSONField stuff pretty easily.

Thanks for a nice set of frameworks.
Issue: Support for group_by and count aggregate on two or more different table cloumns.
Author: zffl
State: closed
Created: 2012-10-01 16:12:48

I have this:

``` python
class A(Model):
    c1 = CharField()

class B(Model):
    c2 = ForeignKeyField(A)

class C(Model):
    c3 = ForeignKeyField(B)
    c4 = CharField()
```

I want to group by A.c1 and C.c4, and count C.id
and my query like this:

``` python
C.select({A:['c1'], C:['c4']}).filter(c3__c2__c1__in=[]).group_by('t1.c4, t3.c1')
```

Two problems here:
1.I don't know where to put the count aggregate? I try the select dict, count() function, but failed.
2.The group clause is weird. I can only use the alias. Any other solution?


Yeah...its weird, i'm not in love with the api...but here's how you do it:

``` python

C.select({A: ['c1'], C: ['c4', Count('id')]}).group_by('c4').join(B).join(A).group_by('c1')
```
1. First we select the columns -- you had that part right except you were just missing the Count('id') in the list of "C" columns.  `from peewee import Count`
2. Then while the "query context" is "C", we'll group by `c4`
3. Then join from c -> b, b -> a
4. Now the query context is "A", we'll group by `c1`

This yields:

``` sql

SELECT t1."c4", COUNT(t1."id") AS count, t3."c1"
FROM "c" AS t1 
INNER JOIN "b" AS t2 ON t1."c3_id" = t2."id"
INNER JOIN "a" AS t3 ON t2."c2_id" = t3."id" 
GROUP BY t1."c4", t3."c1"
```

I'm working on a rewrite of peewee in branch unstable/2.0 -- it will improve on some of these inconsistencies.
Issue: Access to ForeignKeyField's related_name attribute from another process?
Author: davidthewatson
State: closed
Created: 2012-09-27 14:50:24

I have several processes that share the same peewee models. One of those processes manipulates data which effects a ForeignKeyField's related_name attribute which is then later accessed by another process. However, accessing the attribute throws a missing attribute exception from the downstream process. Is it possible to instantiate this attribute in the second process? If so, how?

Thanks.


This sounds very evil.


Without sounding pedantic, could you elaborate on the ambiguous pronoun "this"? That is, are you saying that model sharing across processes is evil or that attribute level model accessor affordances existing across processes is evil? If so, why? This seemed like a perfectly reasonable use of peewee to me.


a peewee model is just a class...changes made in one python interpreter will not affect another interpreter process.  if you need to, shuffle messages around via IPC that do the attribute manipulation.

i said it was evil because a related_name is really an internal API used by code, so i was confused why code would need to change it.


when you get a chance, please update me with the status of this or provide more info so i can help resolve


In the example below, D is the model class representing the related_name d which was being used to access the values from the downstream process. This is, of course, broken, as you pointed out, because the downstream process hasn't manifested the attributes like the upstream process. Thus, this breaks because the related_name attribute d doesn't exist:

a.b.c.d[0].e

The solution is to use a get query in place of the missing related_name attribute:

D.get(c=self.a.b.c).e

which works fine.

This is, of course, writ large to show the relationships. The actual code uses functional shorthand to flatten the joins. 

I thought this might be a useful piece of documentation. Perhaps obvious, but wasn't immediately apparent to me until I thought through the related_name issue. It seemed from your earlier answer that related_name may be intended for internal use only. Should I never be using related_name attributes outside the model they're defined in?

I wanted to point this out, because, while my use of peewee maybe somewhat beyond its intended usage, these things can and do work. And I love peewee!


Hmm... I suppose I'm still not 100% sure what the related name issue is.  Can we talk about two of my favorite models, User and tweet (which Fks to user)?

``` python

class User(Model):
    username = CharField()

class Tweet(Model):
    user = ForeignKeyField(User, related_name='tweets')
    message = CharField()
```

Can you show me, using those two models, the issue?


Yeah, no problem.

The first process creates a user which has associated tweets. The second process needs to act on a user's associated tweets but the tweets related_name doesn't exist:

Process #1: 

user.tweets

Process #2:
user = User.get(username="charlie")
Tweet.get(user=user)


closing for now
Issue: Postgresql with pk_col='ID' reported "doesn't exist"
Author: widoyo
State: closed
Created: 2012-09-17 04:22:31

I have a primary column named 'ID' (UPPERCASE). Executing SQL into Postgresql "SELECT COUNT(ID) FROM inbox" postgresql reported "column doesn't exist". But it is return as expected when my SQL "SELECT COUNT('id') FROM inbox".

I found the problem when using flask-peewee that the error ofound at peewee.py line 1410.

The problem doesn't occur when I change column name from 'ID' into 'id'.


Can you share your model code with me?  Just the class definition should be fine.  If you can get me the SQL create table query as well that you ran that'd be great (ModelClass.create_table_query() i think).  Were these models autogenerated by peewee or were they pre-existing?


I'm not sure if you have tried this, but you might write:

``` python
class MyModel(db.Model):
    ID = PrimaryKeyField()
    some_field = CharField()
```

That should work correctly, and in my testing locally appears to do the right thing.


I use pre existing database, I use Gammu with postgresql database.

...sql
-- Table: sentitems

-- DROP TABLE sentitems;

CREATE TABLE sentitems
(
  "ID" serial NOT NULL,
  "UDH" text NOT NULL,
  "SMSCNumber" character varying(20) NOT NULL DEFAULT ''::character varying,
  "Class" integer NOT NULL DEFAULT (-1),
  "TextDecoded" text NOT NULL DEFAULT ''::text,
  "SenderID" character varying(255) NOT NULL,
  "SequencePosition" integer NOT NULL DEFAULT 1,
  "Status" character varying(255) NOT NULL DEFAULT 'SendingOK'::character
varying,
  "StatusError" integer NOT NULL DEFAULT (-1),
  "TPMR" integer NOT NULL DEFAULT (-1),
  "RelativeValidity" integer NOT NULL DEFAULT (-1),
  "CreatorID" text NOT NULL,
  id_folder integer NOT NULL DEFAULT 3,
)
...

And my model is

...python
class Sent(db.Model, PgModel):
    ID = pw.PrimaryKeyField()
    body = pw.TextField(db_column='TextDecoded')
    destination = pw.CharField(max_length=20, db_column='DestinationNumber')

```
class Meta:
    db_table = 'sentitems'
```

...

and the error is:

```
psycopg2.ProgrammingError

ProgrammingError: column "id" does not exist LINE 1: SELECT COUNT(ID) FROM
"sentitems" ^
```

When I rename column 'ID' into 'id' there is no "programming error".
The programming error is because of "SELECT COUNT(ID) FROM ...", it is not
error when "SELECT COUNT('ID') FROM ...", I have tried from SQL command on
Postgresql.

On Mon, Sep 17, 2012 at 11:35 PM, Charles Leifer
notifications@github.comwrote:

> Can you share your model code with me? Just the class definition should be
> fine. If you can get me the SQL create table query as well that you ran
> that'd be great (ModelClass.create_table_query() i think). Were these
> models autogenerated by peewee or were they pre-existing?

## 

Widoyo


Hmm -- something strange is going on.  Is it only SELECT COUNT() queries that don't work?  Is the SQL you pasted generated by dumping your existing database, or is it something else?  I'm curious to see exactly what your _current_ database schema looks like


The problem could be on the postgresql with column named ID. Sql generated
from pgadminIII
Issue: Default value no work in Model.insert().execute()
Author: zffl
State: closed
Created: 2012-09-09 07:40:16

I add default value for datetime field, but not work in this case, but It work by writing  m =Model(..); m.save(). Is this a problem or by design?


When you initialize a model instance, it will initialize fields to their default values, e.g.:

``` python

>>> from tests import *
>>> dv = DefaultVals() # <-- pub date defaults to datetime.datetime.now
>>> dv.pub_date
datetime.datetime(2012, 9, 9, 11, 31, 37, 797693)
```

When you call `save()` it rounds up all the field values on the instance and passes them along to the insert query.

However when using `insert` directly it will only use the values you pass in explicitly.  This is not explicitly by design.  I'll have to think on whether to keep the current behavior (insert is explicit and ignores defaults) or whether it will implicitly fill in missing values with the field defaults.


Insert will now respect defaults when they are not specified


I think this fix might have been un-fixed.

```
m = DatModel()
m.foo = None
m.save()
```

This results in an Integrity error (NOT NULL constraint failed) if `DatModel.foo` has `null=False`.

I already found out that in the current version in pip, nothing besides `Model.__init__` calls `ModelOptions.get_default_dict` (The fix introduced a call to it in `InsertQuery.__init__`).


I'm not sure I'm understanding correctly, because if you explicitly set foo to None, peewee will not override it. Given that, makes sense to me there would follow an exception if foo was not null.


Oh, but I see your point in reference to this particular issue and the way it was resolved.

So, to clarify, the fix should still be in place. Check out InsertQuery._iter_rows(). Basically it's applying defaults there. But when you are working with a model and explicitly set a field to a value, that takes precedence over a default.


Thanks for the clarification. But now I'm wondering what the correct way of setting the value of a field (of a previously existing instance, so UPDATE) back to its default. Do I have to manually evaluate `DatModel.datfield._meta.default` or is there a way to make peewee do it?


Just do: obj.field = ModelClass.field.default (add parentheses if it's a callable).
Issue: Problem with flask debug mode and automatic restart
Author: mrjoes
State: closed
Created: 2012-08-26 08:01:34

Hi,

I have following code:

``` python
from flask import Flask

import peewee

app = Flask(__name__)
app.config['SECRET_KEY'] = '123456790'

db = peewee.SqliteDatabase('test.sqlite')


class BaseModel(peewee.Model):
    class Meta:
        database = db


class User(BaseModel):
    username = peewee.CharField(max_length=80)
    email = peewee.CharField(max_length=120)

    def __unicode__(self):
        return self.username


class Post(BaseModel):
    title = peewee.CharField(max_length=120)
    text = peewee.TextField(null=False)
    date = peewee.DateTimeField()

    user = peewee.ForeignKeyField(User)

    def __unicode__(self):
        return self.title


@app.route('/')
def index():
    for p in Post.select():
        print p
    return '<a href="/admin/">Click me to get to Admin!</a>'


if __name__ == '__main__':
    try:
        User.create_table()
        Post.create_table()
    except:
        pass

    app.debug = True
    app.run('0.0.0.0', 8000)
```

and whenever I try to open index view, I get following exception: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 7052 and this is thread id 884.

This issue happens only when debug set to True and at least one restart happened.


I found this in documentation: http://peewee.readthedocs.org/en/latest/peewee/cookbook.html#multi-threaded-applications but in Flask autoreload case, I'm not using threads. Can use these options to work around it though.


Looks like the werkzeug debugger uses multiple threads:

https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/serving.py#L585

To suppress these errors you can configure your database `threadlocals=True` or use the sqlite driver's `check_same_thread=False`


`check_same_thread=False` is useful.
Issue: Migrations
Author: jamorton
State: closed
Created: 2012-08-21 08:40:33

Just curious if you have thought about migrations in peewee. If not an automated way, is there an idiomatic way to do it with peewee besides changing fields and guessing the needed SQL statements to reflect them?


If you take a look at the Database class, there are some undocumented methods that may be of help:

https://github.com/coleifer/peewee/blob/master/peewee.py#L456

Say you're renaming a field... from 'foo' to 'bar':

`database.execute(database.rename_column_sql(FooModel, 'foo', 'bar'))`

I'll see about getting this into the docs.
Issue: PeeWee doesn't play nice with pylint (or vice-versa)
Author: diwakergupta
State: closed
Created: 2012-08-15 20:56:17

PeeWee modifies the model classes at RunTime. pylint doesn't like this and I get errors like:

```
E:123,15:Benchmark.add_run_data: Class 'RunData' has no 'DoesNotExist' member
```


Peewee uses metaclasses to give you "declarative" models. Your models and the fields you define as class attributes are valuable metadata and I cribbed the notion of using a metaclass to introspect these.  It would be arguably more pythonic to do it differently, but this is the API I have chosen.  You can see the details:

https://github.com/coleifer/peewee/blob/master/peewee.py#L2566


I'm not contesting the way Peewee does things. I'm merely asking if there's
any way to make pylint + peewee happy short of disabling that error (which
can mask legit bugs in other code)

On Fri, Aug 17, 2012 at 6:42 AM, Charles Leifer notifications@github.comwrote:

> Peewee uses metaclasses to give you "declarative" models. Your models and
> the fields you define as class attributes are valuable metadata and I
> cribbed the notion of using a metaclass to introspect these. It would be
> arguably more pythonic to do it differently, but this is the API I have
> chosen. You can see the details:
> 
> https://github.com/coleifer/peewee/blob/master/peewee.py#L2566
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/issues/104#issuecomment-7818122.

## 

http://floatingsun.net


Ahh, totally -- I am not very familiar with pylint so I'm not sure what hooks it provides to say "i know what i'm doing here".  If you come up with a patch I'd consider merging.


This may be helpful http://stackoverflow.com/questions/115977/using-pylint-with-django