akumoth/peewee-issues huggingface dataset:
Issue: deferred ManyToManyField assignment
Author: ael-code
State: closed
Created: 2015-09-23 09:18:38

If the `ManyToManyField` assignment is made after class declaration you will encounter errors.
The following code represent the scenario.

``` python
from peewee import PrimaryKeyField, CharField,\
    Model, SqliteDatabase
from playhouse.fields import ManyToManyField


db = SqliteDatabase(':memory:')


class BaseModel(Model):
    class Meta:
        database = db  # Use proxy for our DB.


class User(BaseModel):
    id = PrimaryKeyField()
    name = CharField(unique=True)


class Group(BaseModel):
    id = PrimaryKeyField()
    name = CharField(unique=True)


Group.users = ManyToManyField(User, related_name='groups')
UserToGroup = Group.users.get_through_model()


def main():
    db.connect()
    db.create_tables([User,
                      Group,
                      UserToGroup])

    anons = Group.create(name='anons')
    jhondohe = User.create(name='jhondhoe')
    anons.users.add(jhondohe)
    anons.save()
    print [u.name for u in anons.users]
    print [g.name for g in jhondohe.groups]


if __name__ == '__main__':
    main()
```

Output:

``` python
  File "peewee_test.py", line 25, in <module>
    UserToGroup = Group.users.get_through_model()
  File "/home/anon/peewee_test/ve/local/lib/python2.7/site-packages/playhouse/fields.py", line 74, in get_through_model
    lhs, rhs = self.get_models()
  File "/home/anon/peewee_test/ve/local/lib/python2.7/site-packages/playhouse/fields.py", line 69, in get_models
    (self._is_backref, self.model_class),
AttributeError: 'ManyToManyField' object has no attribute 'model_class'
```

Is this normal?


> Is this normal?

Oke I've read [peewee.py#L4194](https://github.com/coleifer/peewee/blob/master/peewee.py#L4194). and I've understood that Fields are replaced on class creation, so you cannot deferred Model fields definition. In some cases you can you can use model proxies as workaround.

Am I wrong?


You should just need to call `add_to_class()`. Untested, but:

``` python
users_field = ManyToManyField(User, related_name='groups')
users_field.add_to_class(Group, 'users')
```


Given it is 2023, and API have changed, now we (as far as I get) should use:
```
users_field = ManyToManyField(User, backref='groups')
Group._meta.add_field('users', users_field)
```
Issue: return " ProgrammingError: execute() first" in sql_error_handler
Author: Funnyhao
State: closed
Created: 2015-09-23 03:35:14

class MyMYSQL(MySQLDatabase):

```
def sql_error_handler(self,exception,sql,params,require_commit):
    db.close();
    db.connect();
    db.execute_sql(sql, params or (), require_commit = require_commit);

    return False;
```

I find test sql_error_handler with kill mysql process but it gets :

```
  File "/data1/wejobssvr/handler/base.py", line 64, in get
    result = self._do(args);
  File "/data1/wejobssvr/handler/main.py", line 75, in _do
    result = getattr(op_controller, action_type, self.error_result)(params);
  File "/data1/wejobssvr/controller/oneoff_controller.py", line 489, in get_job
    job_meta = JobMetaInfo.get(id=int(params_dict["job_meta_id"]));
  File "/data1/wejobssvr/client/../util/peewee.py", line 3395, in get
    return sq.get()
  File "/data1/wejobssvr/client/../util/peewee.py", line 2383, in get
    return clone.execute().next()
  File "/data1/wejobssvr/client/../util/peewee.py", line 1686, in next
    obj = self.iterate()
  File "/data1/wejobssvr/client/../util/peewee.py", line 1667, in iterate
    row = self.cursor.fetchone()
  File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 365, in fetchone
    self._check_executed()
  File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 105, in _check_executed
    self.errorhandler(self, ProgrammingError, "execute() first")
  File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
ProgrammingError: execute() first
```


I don't see how this is a bug in peewee.


it's not possible to allow returning a new cursor to execute_sql when "Mysql gone away" happens?


Hmm, that would change the contract of the function and I'm not sure I'd like to make the change.

I do see, though, how returning the old cursor from before the reconnect is causing the issue.


See 017e4e4952f0b429dd21f0e90465a0f644cf6015 for a fix.


thx a lot
Issue: New release soon?
Author: bahadir
State: closed
Created: 2015-09-21 22:15:34

Hi, I'm happy with changes after last tagged release and I want to use peewee without messing my requirements.txt / setup.py.

Are you planning to tag a new release soon?

Thanks & keep up the great work!


Released 2.6.4: https://pypi.python.org/pypi/peewee/

Changes: https://github.com/coleifer/peewee/releases/tag/2.6.4
Issue: Backreference relationship signal
Author: josefdlange
State: closed
Created: 2015-09-15 22:28:30

Hey! I have a suggestion for an addition (and perhaps I could even undertake implementation if you think it's worthwhile and would accept changes!) to the signal support in `playhouse`. It's a little hard to articulate, but I think there might be value in a signal that fires for some model instance, let's call it `Banana`, when any other model establishes a Foreign Key relationship to that instance. I can imagine that in the save handler of a Model, we'd look at the fields for any dirty FKs, and fire a signal at them.

Does that make sense? Do you think that would be useful?

(the other option, of course, is for me to just implement that logic local to _my_ base model class and not make it part of the signal architecture)


I do not think that I would merge such functionality into peewee, but if you end up implementing it on you r own I'd love to see the diff. Maybe you could post it to this issue?


Haven't implemented yet, but I'm imagining this. This is not using the signals architecture as I'm not too well-read on your implementation of it.

```
# Save method on the base model (from which everyone inherits)
def save(self, *args, **kwargs):
    # Check all our ForeignKeyFields...
    for field in self.dirty_fields():
        if isinstance(field, ForeignKeyField):
            # If we're using signals, I think we would look it up and fire it here instead of making a
            # heavy assumption that the rel_model implements the method fk_established(self)
            try:
                field.rel_model.fk_established()
            except AttributeError:
                pass
```
Issue: SQLite migration cannot handle all CREATE TABLE queries
Author: sroebert
State: closed
Created: 2015-09-15 15:19:11

I was implementing the use of the `drop_column` method in the `SqliteMigrator` when I found a bug.

I tested the result by changing my database model using sqlitebrowser. The `CREATE TABLE` query sqlitebrowser  generates, contains newlines and ` characters around the table name. This differs from the peewee `CREATE TABLE` queries and the migration code does not support this.
Issue: Problem with playhouse.db_url.connect query params
Author: foxx
State: closed
Created: 2015-09-15 14:27:18

Due to an outstanding [bug](https://bugs.python.org/issue24887) in sqlite3, some connection flags such as `mode=memory/cache=shared` can only be passed in via URI.

However, `playhouse.db_url.connect()` [converts](https://github.com/coleifer/peewee/blob/master/playhouse/db_url.py#L68) these query string parameters into `kwargs` for the database `connect()` call and are stripped from the URI, meaning that they are never actioned on.

As a temporary workaround for Sqlite3, you can bypass `db_url.connect()` and  use `SqliteDatabase` directly with `uri=True`, however it would be nice if it handled this stuff automagically.


Well, technically I think you can specify those using PRAGMAs, can't you? Or as manual params with `connect()`...

But isn't the `uri` aspect fairly new to pysqlite, or am I mistaken?


It looks also like it's a Python3 only thing at the moment, added in 3.4, so I'm going to hold off on supporting that directly.


Oh I wasn't aware of PRAGMAs, I'll take another look at this, thanks!


After looking at this deeper, some options cannot be enabled via PRAGMAs, such as "shared cache". In this particular example, Python 2.x would have to use `sqlite3.enable_shared_cache(True)` module global, rather than specifying in the URLs. For this feature to have Python 2.x support, the `uri` module functionality would have to be [backported](https://hg.python.org/cpython/rev/f13bb1e40fbc), which is extremely difficult given that it's a `cpython` module. 

For anyone seeing this in future, if you're using Python 2.x then you're mostly outta luck. If you're using Python 3.x, then use `uri=True`.
Issue: Output null values for missing dates
Author: aidanheerdegen
State: closed
Created: 2015-09-12 12:59:28

Hi,

I've tried and I've failed.

I have a table like so 

``` python
class ShortUsage(BaseModel):
    projid = ForeignKeyField(Project, to_field="projid")
    folder = CharField(index=True)
    user = ForeignKeyField(User, to_field="username")
    # Support only dates. Don't want sub-day timescales, makes it harder to join
    # data and snapshots are only taken daily
    scandate = DateField(index=True)
    inodes = FloatField()
    size = FloatField()
    class Meta:
        # Combination of projid+folder+user+date must be unique
        indexes = ( (('projid','folder','user','scandate'), True), )
```

For each projid and user I want to output the sum of size, which I can do no problem. Not all users have entries for all dates in a given range. I want to force the output of a value (zero?) for dates where they have no data. Basically, I need to output the sum of size for a superset of all dates within a given range.

I have succeeded in making a query which produces the dates I need for a given period:

``` python
SUalias = ShortUsage.alias()
q1 = SUalias.select(fn.Distinct(SUalias.scandate).alias('alldate')).where(SUalias.projid == projid & SUalias.scandate >= startdate & SUalias.scandate <= enddate).alias('q1')
```

but when I try and join that to my existing table

``` python
    for date, diskusage in ShortUsage.select(ShortUsage.scandate,fn.SUM(ShortUsage.size))\
        .where((ShortUsage.user == user) & (ShortUsage.projid == projid))\
        .join(q1, JOIN.LEFT_OUTER, on=(q1.c.alldate == ShortUsage.scandate))\
        .group_by(ShortUsage.scandate)\
        .order_by(ShortUsage.scandate).tuples():
        dates.append(date)
        usage.append(diskusage/1000.)
```

It does not report values for all the dates in q1. Is this because it is a LEFT_OUTER join? When I try a full join or RIGHT_OUTER I get this:

```
peewee.OperationalError: RIGHT and FULL OUTER JOINs are not currently supported
```

I am using v2.6.3, and the docs seem to suggest full joins are possible 

> By default peewee will use an INNER join, but you can use LEFT OUTER or FULL joins as well

Any assistance would be greatly appreciated.


If you're using SQLite, then while Peewee supports RIGHT OUTER, SQLite itself does not. Can you simply switch the order of your joins and use a left outer?


You may need to add clauses to your where or join as well that allow for the date to be NULL. But, basically this is a SQL question not a peewee bug.
Issue: Ability to get a bare cursor
Author: alexanderad
State: closed
Created: 2015-09-11 08:41:35

Hello.

I'm trying to get a bare, out of transaction context cursor from within peewee, but for some reason this does not work for me.

``` python
from peewee import *

database = PostgresqlDatabase('test')

database.set_autocommit(False)
database.execute_sql('VACUUM FULL', require_commit=False)
database.set_autocommit(True)
```

I get 

``` python
Traceback (most recent call last):
  File "autocommit.py", line 9, in <module>
    database.execute_sql('VACUUM FULL', require_commit=False)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3015, in execute_sql
    self.commit()
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 2864, in __exit__
    reraise(new_type, new_type(*exc_value.args), traceback)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3007, in execute_sql
    cursor.execute(sql, params or ())
peewee.InternalError: VACUUM cannot run inside a transaction block
```

Any chance I'm doing this wrong (should not `set_autocommit` take care of that)?

The only one way I found to do this type of query is by operating low level cursor like this:

``` python
@contextlib.contextmanager
def get_bare_cursor(db):
    """Gets the bare cursor which has no transaction context."""
    connection = db.get_conn()
    old_isolation_level = connection.isolation_level

    connection.set_isolation_level(
        psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
    try:
        yield db.get_cursor()
    finally:
        connection.set_isolation_level(old_isolation_level)

with get_bare_cursor(database) as cursor:
    cursor.execute('VACUUM FULL')
```

Thanks.


Peewee does not alter the isolation level of the postgres connection, unfortunately the `autocommit` parameter is poorly named. When True, it means basically: "pass queries through the DB-API driver normally and call connection.commit() afterwards". If False, then you need to call commit() yourself.

Working with the psycopg2 connection seems like the best route, since you need to set the connection's isolation level. I think psycopg2 issues an implicit `BEGIN` when you are not in autocommit mode, so that may be what's causing the trouble.


For now I think it will suffice to leave the `autocommit` semantics as they are. There's quite a bit that's confusing given that postgres itself specifies AUTOCOMMIT differently than the DB-API 2.0, and peewee having it's own definition certainly adds to the confusion.

The code you have there looks good to me, so I'm going to close this.
Issue: Creating relationships using the backref
Author: kolanos
State: closed
Created: 2015-09-08 18:33:19

First off, I'd just like to say that peewee is awesome. I'm currently working on a peewee backend for potion, a  REST API library for flask. Here's what I have so far:

https://github.com/biosustain/potion/pull/28

But I'm running into an issue (or possible limitation) of peewee when creating relationships between models. I'm hoping there's an obvious way to do this...

Say I have a self-referencing model such as:

```
class User(Model):
    parent = ForeignKeyField('self', null=True, related_name='children')
    name = CharField()
```

And I'm working with the following instances:

```
father = User.create(name='foo')
son = User.create(name='bar')
```

Potion provides me with the `father` and `son` instances, as well as the
`children` attribute.

The way that the SQLAlchemy backend handles this is to:

```
getattr(father, 'children').append(son)
```

But there doesn't appear to be a way to create a one-to-many relationship
using the backref with peewee. At least I haven't found a way using
`father.children`.

Is there a simple way to do something like what ManyToManyField has like:

```
father.children.add(son)
```

Or is there a way to determine the matching foreign key field on the `son`
instance using only the `father` instance and the `children` attribute?

I would just as well do:

```
son.parent = father
```

But I don't know about the `parent` attribute or that it compliments the
`childlren` attribute. I need a way to determine this.


Good question, no I hadn't thought about a corollary to the many-to-many add, but that seems useful. I will add it.


Well, I forgot that the reverse fk descriptor exposes a plain old select query. It seems smelly to me to special case the select query returned by the fk descriptor. Similarly, I don't know that I like the idea to expose a wrapper / subclass with an `add()` method.

So, at the moment, I will say that there is no such method, but that you can do something like:

``` python
def add_child(parent, child):
    child.parent = parent.id
    child.save()
```

Or, generically:

``` python
def add_general(source, reverse_rel, child_inst):
    # source = "father", reverse_rel = "father.children"
    # reverse_rel.field.to_field.name = "id"
    target_val = getattr(source, reverse_rel.field.to_field.name)
    setattr(inst, reverse_field.field.name, target_val)
    inst.save()
```


Thanks for the response. I ended up with something like this to determine the complimentary relation name.

```
father._meta.reverse_rel['children'].name
```

Which correctly returns `parent` in this context. But your response has me questioning the use of reverse_rel for this purpose. Any issues there?


That's totally fine as well.
Issue: peewee+MySQL TimeField returns timedelta object
Author: ibushong
State: closed
Created: 2015-09-06 11:42:00

When using MySQL, the `value` that gets passed into `TimeField.python_value(value)` is a `timedelta` object, which doesn't get properly handled (to turn it into a `time` object).

Modifying `TimeField.python_value` to the following fixes the issue (note the `is None`, as `time(0)` (midnight) evaluates to `False`):

``` python
class TimeField(_BaseFormattedField):
    ...
    def python_value(self, value):
        if value and isinstance(value, basestring):
            pp = lambda x: x.time()
            return format_date_time(value, self.formats, pp)
        elif value and isinstance(value, datetime.datetime):
            return value.time()
        elif value is not None and isinstance(value, datetime.timedelta):
            return (datetime.datetime.min + value).time()
        else:
            return value
```


The fix does not work. The `elif` branch is never reached because the timedelta object evaluates to `True`, so the behavior is the same as before.
Issue: Model.update() not firing pre_save signal
Author: rvindas-gap
State: closed
Created: 2015-09-04 14:57:49

I have this import in my models: `from playhouse.signals import Model, pre_save` and have a model that inherits from that 'Model' class. I also added the pre_save signal implementation which gets fired when I execute Model.create() or Model.save().
But executing a Model.update().execute() updates the record correctly but it doesn't fire the pre_save signal.


Yes this is a known (and obvious) limitation. You'll want to look into triggers.


Not looking to resurrect the dead here, but could there perhaps be a docs update to reflect this? It says:

> Provides an additional keyword argument created, indicating whether the model is being saved for the first time or updated.

which to me made me expect it would work, and thus chase my tail until I came here.  :)

Thanks!


See 590ccac3044e13988270017b85ec85dd76161c8b


Thanks!
Issue: AttributeError: 'Database' object has no attribute 'compiler'
Author: kramer65
State: closed
Created: 2015-09-04 14:25:27

I'm trying to use the peewee migration [described here](http://peewee.readthedocs.org/en/latest/peewee/playhouse.html#schema-migrations).  But when running the migrate command I get the error below this message.

I'm on Ubuntu 14.04 with Python 2.7 with the regular MySQL from the apt repos.

Anybody any idea what could be wrong here? All tips are welcome!

```
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 575, in migrate
    operation.run()
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 144, in run
    getattr(self.migrator, self.method)(*self.args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 138, in _handle_result
    self._handle_result(item)
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 135, in _handle_result
    result.run()
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 144, in run
    getattr(self.migrator, self.method)(*self.args, **kwargs))
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 152, in inner
    return fn(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/playhouse/migrate.py", line 190, in alter_add_column
    field_clause = self.database.compiler().field_definition(field)
AttributeError: 'Database' object has no attribute 'compiler'
```


Is database the flask peewee wrapper? Have you tried database.database in that case?


Awesome! That was indeed the problem!

Thanks a million. Both for solving this, and for creating Peewee. We're already using it in several websites now. Big thumbs up!
Issue: Add sensible default from Field if Database() wasn't initialized with fields argument
Author: bdpdx
State: closed
Created: 2015-09-03 23:53:46

Modified QueryCompiler.get_column_type to return an uppercase version of the f argument if the field type doesn't exist in _field_map.

This enables a sensible default if a Field subclass defines a type supported by the underlying database that isn't passed into the Database() constructor via its fields initializer.


Cool, let's try it out!
Issue: auto_increment  differrent in  create and  create_or_get 
Author: tablecell
State: closed
Created: 2015-09-02 11:49:03

``` python
from peewee import *
from  pprint import pprint

db = SqliteDatabase('people.db')

class User(Model):
    uid = IntegerField(primary_key=True)
    name = CharField()
    password = FixedCharField()


    class Meta:
        database = db # This model uses the "people.db" database.



#db.create_tables([User])

username='xx'
try:
    user = User.get(name=username)
    #user = User.get_or_create(name=username)
    pprint(user.uid)
except:

   user=User.create(name=username,password='xx')
   print(user.uid)

```

the result is None 
by  get_or_create   the result is 1 


You should be using `PrimaryKeyField` instead of `IntegerField`. The former is auto increment, while the latter is just an arbitrary int field.
Issue: Discerning what fields "were dirty" in post_save signal
Author: josefdlange
State: closed
Created: 2015-08-29 19:55:18

Hi! I'm trying to figure out what might be the cleanest way to know what model fields "were dirty" (or, more specifically, if a given field changed) in a signal-supporting model.

I'm thinking adding some kind of field on the signal base `Model` that gets set during the `save` method which is set to the contents of `instance.dirty_fields` just before the superclass `save` method is invoked. Do you think it would be possible to subclass the signal-support `Model` and override the `save` logic to do that? Any red flags come up in your mind? Maybe just have an additional parameter that can go into a connected signal handler, `modified`, which is those values.

Model and save method in question:
https://github.com/coleifer/peewee/blob/master/playhouse/signals.py#L65


Why not just add a `pre_save` signal handler that sets an additional attribute on the class:

``` python
@pre_save(sender=SomeModel)
def pre_save_handler(sender, instance, created):
    instance._dirty_field_cache = instance.dirty_fields

@post_save(sender=SomeModel)
def post_save_handler(sender, instance, created):
    # do something with instance._dirty_field_cache
    pass
```