akumoth/peewee-issues huggingface dataset:
Issue: Query filtering does not work as expected with multiple foreign keys to the same table. 
Author: rammie
State: closed
Created: 2015-07-03 18:20:47

Thanks as always for this amazing package.

Here is a small example to recreate this issue. I'm really hoping that you don't decide to deprecate the the filter API as I frequently use the REST API's from the retired flask-peewee module that relies on it.

I think the issue is that ensure_join does not care about the on clause of the join which causes the second join to be skipped.

``` python
from peewee import *

database = SqliteDatabase(':memory:')

class Model(Model):
    class Meta:
        database = database


class User(Model):
    username = CharField(unique=True)

class Tweet(Model):
    user1 = ForeignKeyField(User, related_name='tweets1')
    user2 = ForeignKeyField(User, related_name='tweets2')
    message = CharField()


User.create_table()
Tweet.create_table()

u1 = User.create(username="foo")
u2 = User.create(username="bar")

t1 = Tweet.create(user1=u1, user2=u2, message="boo")

# Following line prints "foo" as expected
print Tweet.filter(user1__username="foo", message="boo").get().user1.username


# Following line should print "foo" as well,
# but instead it raises an DoesNotExist exception.
print Tweet.filter(user1__username="foo", user2__username="bar").get().user1.username
```


This is a limitation of the `filter()` API. To get the correct semantics (and avoid extra queries when looking up associated usernames), you would need to write:

``` python
UA = User.alias()
q = (Tweet
     .select(Tweet, User, UA)
     .join(User, on=(Tweet.user1 == User.id))
     .switch(Tweet)
     .join(UA, on=(Tweet.user2 == UA.id).alias('user2'))
     .where(
         (User.username == 'foo') &
         (UA.username == 'bar')))
tweet = q.get()
print tweet.user1.username
print tweet.user2.username
```

Unfortunately, `filter()` does not do well with multiple joins to the same table.


Hi again, and sorry to keep bothering with this. I investigated a little further and it seemed possible to make this work using aliases. I put together a pull request for this and was hoping you could take a look. I also hope that you think that the approach is reasonable.
Issue: Using bitwise and / or in a query
Author: josefdlange
State: closed
Created: 2015-07-01 08:50:28

Is it possible to use bitwise operations (&, |, #, ~) in a query and have them translate to the (flavor)SQL keywords to _actually_ do the bitwise operation?


Yes, you can use `.bin_and`, `.bin_or`, `^` (xor).


If they are not mapped to the correct operator for your database please let me know.


It looks like I may need a bitwise `NOT`, though, now that I look at it.


Awesome, thanks. My usage is like this, am I doing it right? I'm trying to select any `Thing` for which its `field` (an `IntegerField`, representing a bit mask) when ANDed with `other_integer` (also a bit mask), is a match (or better, which is why I'm not doing just integer equality comparison).

So if the binary of `Thing.field` is 11111111 and `other_integer`'s binary is 10101011, that `Thing` would be selected.

So, in essence, am I correct in the use of the `bin_and` method in the where clause?

```
things = (
    Thing
    .select()
    .where(Thing.field.bin_and(other_integer) == other_integer)
)
```


Yeah, I think that should be right.
Issue: pwiz AttributeError: 'tuple' object has no attribute 'args'
Author: shaotianchi
State: closed
Created: 2015-06-30 08:31:53

I'm using python 2.6 
when I use pwiz to generator my model, this error come

```
python -m pwiz -e mysql -H myhost -p 3306 -u user -P psw app_trends_food > mymodels.py
```


Can you paste the entire traceback?
Issue: unsigned integer  and auto_incresing
Author: AlexHeKa
State: closed
Created: 2015-06-25 16:31:10

It seems custom field only support Postgresql now，is't possible to custom an unsigned or auto_incresing field in mysql db? Thanks


I'm not sure I understand the question completely but yes, you can write custom fields. Check the docs:

http://docs.peewee-orm.com/en/latest/peewee/models.html#creating-a-custom-field
Issue: Proposal to extend IntegrityError
Author: foxx
State: closed
Created: 2015-06-25 16:06:02

As a follow up to #443, I'm proposing that IntegrityError should expose some information about the error its representing. For example, if the error was related to a unique constraint failure, perhaps we could have a subclass representing this (`UniqueConstraintError`?).

It seems this is detectable inside `exc.args`, e.g;

```
('UNIQUE constraint failed: useremail.user_id, useremail.email_address',)
```

It seems that most databases will return an error code, rather than having to do a string comparison.

```
SQLITE_CONSTRAINT_UNIQUE (2067)
MySQL: ERROR 1062 (23000): Duplicate entry '2147483647' for key 'PRIMARY'
```

I think that unique constraint failure is a good candidate for this, at the very least to reliably implement `create_or_get()` type functionality.

On a side note - it seems that there is a [bug](http://bugs.python.org/issue16379) with Python 3.4 that prevents Sqlite error codes from reaching Python

Thoughts?


Eeee, yeah, this seems like a can of worms I'd rather not open. I'm going to stick with the ones specified by the DB-API 2.0 spec.

https://www.python.org/dev/peps/pep-0249/#exceptions


Agreed, makes sense to stick by DB-API 2.0. For all other engines/versions, we can just inspect `exc.args[0]` to get the error code, but for Sqlite3 on Python 3.4 we'll just have to wait heh.
Issue: Proposal for Model.refetch()
Author: foxx
State: closed
Created: 2015-06-25 15:23:40

One common pattern is to "refetch" a model instance, returning a new instance of the model with updated database values. It's also possible to use `refresh()`, but this could result in surprising behaviour when overwriting class attributes etc. This pattern is applicable to any model with a primary or compound key.

I've been able to get this working by using;

``` py

class BaseModel(Model):
    def refetch(self):
        fields = self._meta.get_primary_key_fields()
        filters = [ field == self._data[field.name] for field in fields]
        results = list(self.__class__.select().where(*filters).execute())
        if len(results) == 0:
            raise IntegrityError("No such row")
        elif len(results) > 1:
            raise IntegrityError("Too many rows")
        return results[0]
```

``` py
>>> u = User.get(id=1234)
>>> u = u.refetch()
>>> print u.id
1234
```

If I presented this as a clean PR with tests and documentation, would it be accepted?


I think I will leave this as an application-type API. Re-fetching seems to me to be complex (do you just go bby primary key, or as you have, by values)?


So far I've only tried this by using the primary or compound key, which has worked well so far, but will fall over when used with a model that doesn't have this. I also need to raise an exception if there is no primary key fields, and add some unit tests. At the very least, this recipe is here for Google. 

Just so I can be sure, do you see any glaring flaws in the approved I've used above, or does it look sane enough to leave here for future? (ty for the feedback btw, appreciated)


The way I have always done it is just `model_inst = ModelClass.get(model_inst._pk_expr())` (undocumented API).


Yours is better though because it handles composite keys, so no, no flaws!
Issue: Added small bit in documentation about ForeignKeyField's related_name
Author: josefdlange
State: closed
Created: 2015-06-24 20:44:40

I had a lengthy argument with a co-worker about the implicit `related_name` backreference created by peewee when declaring a `ForeignKeyField`. He was convinced it didn't exist, since it wasn't in the documentation. Only after walking him through the source code and demonstrating it did he believe me. So, for the future, I added a small section in the `ForeignKeyField` docs explaining. 

I do not know how to build/test `peewee`'s documentation but would be glad to if I had some direction. Also accepting changes to what I wrote, if it doesn't match up with the original intent or design.

Also added a `.gitignore` to ignore `.pyc` files and the `build` directory. We can trim that out of here if you feel it doesn't belong, though.


Sick, I'll take the gitignore too!
Issue: get next value for pk
Author: ak4nv
State: closed
Created: 2015-06-24 04:45:21

Is there an easy way to get the next value for the primary key (ID) of a sequence?
Needs to use id in one field of model.

I have to save the object twice now.

``` python
    def save(self, *args, **kwargs):
        rows = super().save(*args, **kwargs)
        self.code = '.'.join((self.branch.code, self.type, str(self.id)))
        return super().save(only=[Dep.code])
```


Not coleifer, but I would have to say probably not. To prevent race conditions, the only reliable way to know the next id is to have it assigned to you by the DBMS, and each DBMS handles passing that information back in a different way after an ID is assigned.

https://github.com/coleifer/peewee/blob/master/peewee.py#L3389
https://github.com/coleifer/peewee/blob/master/peewee.py#L3418
https://github.com/coleifer/peewee/blob/master/peewee.py#L3132

Maybe your app is going to be single threaded forever? If so, and you're also running on Postgres you could copy and modify the Postgres implementation of last_insert_id and switch curval for nextval.


Thanks @jakedt for the quick answer, probably I'll not change my code — it runs very rarely.
I thought I can run postgres function `nextval` for sequence and get the next id guaranteed.
It's not a problem to save the object twice for me.

I use nginx+uwsgi with four processes. Anyway — modify the Postgres implementation is too expensive :smiley: 


I thought for some reason calling `nextval()` would increment the counter, at any rate @jakedt is correct that it's dicey to try and predict.
Issue: Regression: using a column alias breaks ModelQueryResultWrapper for nested joined objects
Author: jakedt
State: closed
Created: 2015-06-23 19:18:25

Repro:

``` python
import logging
import datetime

from peewee import *


logger = logging.getLogger(__name__)
database = SqliteDatabase(':memory:')


class User(Model):
    username = CharField(unique=True)
    class Meta:
        database = database


class Tweet(Model):
    user = ForeignKeyField(User, related_name='tweets')
    message = TextField()
    created_date = DateTimeField(default=datetime.datetime.now)
    is_published = BooleanField(default=True)

    class Meta:
        database = database


def generate_data():
    database.create_tables([User, Tweet])
    for username in ['1', '2', '3']:
        new_user = User.create(username=username)
        for msg in ['hi', 'hello', 'test']:
            Tweet.create(user=new_user, message=msg)    


def test_query():
    found = (Tweet.select(Tweet.message, User.username.alias('uname'))
             .join(User).where(Tweet.id == 1).get())
    logger.debug(found.user.username)


if __name__ == '__main__':
    generate_data()
    logging.basicConfig(level=logging.DEBUG)
    test_query()
```

output 2.6.1

```
DEBUG:peewee:('SELECT "t1"."message", "t2"."username" AS uname FROM "tweet" AS t1 INNER JOIN "user" AS t2 ON ("t1"."user_id" = "t2"."id") WHERE ("t1"."id" = ?) LIMIT 1', [1])
DEBUG:__main__:1
```

output master

```
DEBUG:peewee:('SELECT "t1"."message", "t2"."username" AS uname FROM "tweet" AS t1 INNER JOIN "user" AS t2 ON ("t1"."user_id" = "t2"."id") WHERE ("t1"."id" = ?) LIMIT 1', [1])
DEBUG:__main__:None
```


Actually the behavior in master is now correct:

``` python
In [2]: generate_data()

In [3]: found = (Tweet.select(Tweet.message, User.username.alias('uname'))
   ...:              .join(User).where(Tweet.id == 1).get())

In [4]: found.user
Out[4]: <__main__.User at 0x7f35f01a54d0>

In [5]: found.user.uname
Out[5]: u'1'
```

So by using an alias, Peewee will assume you want the alias to be attribute on the object. Instead of `user.username` it is `user.uname` because of the alias.

PS thanks for the test code!
Issue: int field with max length set to 1
Author: emamirazavi
State: closed
Created: 2015-06-23 13:09:30

As i knew i could not set max length of integer field to 1 for MySQL. We can not have this feature yet?!


Unfortunately peewee doesn't support int with max length natively. You can write a custom field class though if you want to. Check the docs on adding a custom field.
Issue: Playhouse proposal for optimised MySQL UUIDField
Author: foxx
State: closed
Created: 2015-06-22 19:44:13

I spent the day putting together the most optimised method of storing UUIDs in MySQL/MariaDB, and ended up implementing some research by Percona in which bits are shifted around to give better index performance.

I've since decided to switch over to ObjectIDs instead, but thought I'd share this code. If it doesn't get included in the playhouse then at least it's here for others in future. 

The only remaining performance tweak might be storing it as a BINARY rather than BLOB, as this _may_ perform better under MySQL/Maria (tbc).

I'm not particularly happy about this approach, as it feels quite dirty and inelegant, but it does give the raw performance for those that need it. Also, this has only been tested on Python 3.

``` py

class UUIDField(BlobField):
    """
    Optimized storage for UUID fields in MySQL, based on research by Percona
    https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/

    XXX: Should storage use BINARY instead of BLOB? Needs benchmark
    """

    def db_value(self, value):
        """
        Convert UUID to binary blob

        >>> db_value('12345678-1234-5678-1234-567812345678')
        >>> db_value(uuid.uuid4())
        >>> db_value('lsjfhgjg')
        Exception: ValueError
        """

        # ensure we have a valid UUID
        if not isinstance(value, UUID):
            value = UUID(value)

        # reconstruct for optimal indexing
        parts = str(value).split("-")
        reordered = ''.join([parts[2], parts[1], parts[0], parts[3], parts[4]])
        value = binascii.unhexlify(reordered)
        return super(UUIDField, self).db_value(value)

    def python_value(self, value):
        """
        Convert binary blob to UUID instance
        """
        u = binascii.b2a_hex(value)
        value = u[8:16] + u[4:8] + u[0:4] + u[16:22] + u[22:32]
        return UUID(value)


class UUIDFieldTestCase(unittest.TestCase):
    def test_random(self):
        uuids = [ uuid4() for x in range(100) ]
        for value in uuids:
            a = UUIDField()
            as_binary = a.db_value(value)
            as_result = a.python_value(as_binary)
            self.assertEquals(value, as_result)
            self.assertEquals(str(value), str(as_result))

    def test_single(self):
        value = UUID('AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE')
        a = UUIDField()
        as_binary = a.db_value(value)
        as_result = a.python_value(as_binary)
        self.assertEquals(value, as_result)
        self.assertEquals(str(value), str(as_result))
```


This is quite neat! I will pass but hopefully Google will help anyone else find this who may be interested.
Issue: Proposal for FixedCharField?
Author: foxx
State: closed
Created: 2015-06-22 14:39:14

Although Postgres apparently does not experience any performance difference between `char/varchar` [[1]](http://www.postgresql.org/message-id/C11A06F4.4B7B8%pgsql@007Marketing.com), other databases such as MySQL do have a noticeable impact [[2]](http://dba.stackexchange.com/questions/424/performance-implications-of-mysql-varchar-sizes). Also, `char` will use slightly less space than a `varchar`, which seems to be consistent across all databases.

Do you think Peewee would benefit from having `FixedCharField` in the Playhouse? Admittedly this probably isn't something that should go into the core. I'm also not entirely happy with the name `FixedCharField`.

Thoughts?


I think I will leave this alone, but if you wanted to add it to `playhouse` you could create a new `fields` module I suppose? (and maybe move the `AESEncryptedField` and `CompressedField` from `shortcuts` I believe).


That's a neat idea, it would be a good place for other common fields too. I have some spare time coming up next week for contributions, I'll get this done then.


I've refactored the code, so there is now a `fields` module in `playhouse.fields`. This module contains the encrypted, compressed and many-to-many fields.


Damn you are too fast. LGTM, thank you!
Issue: Support modification tracking by supply an event api?
Author: galeo
State: closed
Created: 2015-06-18 03:59:22

Peewee has very limited signal support in `playhouse.signals` currently. Does it have any plan to supply a variety of hooks or an event api that could support modification tracking in the future?


The limitations in Peewee are really the maximum possible. Anything more should be implemented in the database as triggers.


I'm trying to integrate Whoosh with Peewee just as the way [Flask-WhooshAlchemy](https://github.com/gyllstromk/Flask-WhooshAlchemy) uses [Flask-SQLAlchemy](https://github.com/mitsuhiko/flask-sqlalchemy/) to implement the so-called real-time indexing by tracking **CUD** actions. Maybe this is not a feasible idea.


Well, if you use the `post_save` and `post_delete` hooks, you should be fine. This will not catch bulk inserts, updates or deletes, however.


Without catching bulk operations may lead to confusion. Perhaps as you said above,

> The limitations in Peewee are really the maximum possible.
Issue: Migrations: change column
Author: klen
State: closed
Created: 2015-06-15 13:08:03

I've used the functionality in my project with Postgres and it works for me. So I thought it could be useful for other people. The request provides beta support for `change_column` functionality in migrator. I've tested this with Postgres and SQLite.


I think I will pass on this, but thank you.


What wrong?


Well, I think `ALTER TABLE / ALTER COLUMN` is just complicated enough that I'm not sure this implementation will suffice. Additionally, the MySQL syntax is quite a bit different IIRC, and is not added as part of this PR.


That the reason why I didn't include the functionality in docs and chosen such word as `beta`. But the code (uncompleted of course) already helped me with postgres and sqlite.

But it up to you, coz. Thank you for the project!


Thanks for the PR and for taking the time to submit the code. If I should decide to add this functionality in the future I will definitely use this PR as the starting place.
Issue: [Support request] No primary key for a model
Author: alexlatchford
State: closed
Created: 2015-06-15 02:42:10

Hi Charles,

I've got a table which logs keywords, schema is:

```
class Keyword(Model):
    name = CharField(null=False)
    created = DateTimeField(null=False, default=datetime.utcnow)

    class Meta:
        indexes = (
            (('created', 'name'), False)
        )
```

In short, I want to drop the auto-generated primary key column (id). How do I prevent it from being readded in subsequent db creations? I've taken a look into the manual and not found the syntax.

---

In long, in the database however it has added an id column and associated pkey index. This table now has 35 million rows and that pkey index taking up 10gb in ram. I've taken a look into the code and it looks safe to drop the column, we only perform an aggregate query on the table in any case:

`Keyword.select(Keyword.created, Keyword.name).where(Keyword.created > 3_days)` OR `Keyword.select().where(Keyword.created > 3_days).count()`

Neither of these should need a primary key as we can get exact duplicates (35mil rows are for 3 days of data). 

Many thanks,
Alex

PS. It would also be awesome to be able to name index constraints, getting a few problems when running schema migrations because of name mismatches between environments.


For models with no primary key, simply specify `primary_key=False` in the inner `Meta` class:

``` python
class Keyword(Model):
    name = CharField(null=False)
    created = DateTimeField(null=False, default=datetime.utcnow)

    class Meta:
        primary_key = False
```

If you want a multi-column primary key you can write:

``` python
class Keyword(Model):
    name = CharField(null=False)
    created = DateTimeField(null=False, default=datetime.utcnow)

    class Meta:
        primary_key = CompositeKey('name', 'created')
```


Hmm, but surely this will imply a unique constraint which isn't what I'm after because the dataset can contain duplicate values. Is there any plans for peewee to support no primary key set ups?


Oh wait my mistake I forgot that this _is_ possible!

Edited my comment above.


I've also updated the docs to make this more clear.


Awesome thanks Charles :)


Thanks @coleifer for this answer of yours - it helped me greatly now, three years later. :+1: (I was struggling with peewee trying to retrieve data from a non-existing sequence as below)

```
psycopg2.ProgrammingError: relation "issue_assignees_id_seq" does not exist
LINE 1: SELECT CURRVAL('"issue_assignees_id_seq"')
```

Applying the `primary_key = False` to the inner class fixed the issue.
Issue: Small issue with get_object_list() method
Author: PiJoules
State: closed
Created: 2015-06-13 16:50:00

I was trying out the blog example and whenever I went on the dev server (127.0.0.1:5000), the 404 page would always get displayed. I was able to get around this by creating and publishing a post, but whenever I went to the drafts page (127.0.0.1:5000/drafts/), I would also get redirected to the same 404 page. I was able to trace the problem down to this line in the get_object_list() method in the `flask_utils.py` file:

``` py
if self.check_bounds and self.get_page() > self.get_page_count():
    abort(404)
```

If I understand the overall code correctly, if no matches are found that match the query, then the page is redirected to the 404 page. I'm not sure if this was an intended feature (if this was intended, please ignore this issue), but when I first launched the app, I was expecting the same template provided in `index.html` just without posts instead of a 404 page.

I don't mean to nitpick or anything. I just wanted to point this out.
Issue: Debian package
Author: fliiiix
State: closed
Created: 2015-06-11 06:04:47

Are there any plans to pack peewee for Debian?
Or are there any reasons this is not gonna happen? 


I've started something https://github.com/fliiiix/python-peewee but like most times I have not really a clue if I'm doing this the right way. 


I do not have plans to make a debian package, nor make that part of the peewee source code. `pip` is seen as the primary way for managing python packages these days, anyways.


Okey cool thanks for your fast response. 
Just to get this right: you're not doing it and it won't be part of this repository, but there is no problem if someone else does it? 


Correct. No problem at all if someone else wanted to maintain a debian package.


:+1:
Issue: issue when field name is _data
Author: xinming90
State: closed
Created: 2015-06-08 15:58:32

For example

``` python

from peewee import *

class User(Model):
    id = IntegerField()
    _data = CharField()

u = User(id=1, _data='eleme')
print u.id, u._data

```

got error

```
(virtualenv)➜  peewee git:(master) ✗ python t.py
Traceback (most recent call last):
  File "t.py", line 7, in <module>
    u = User(id=1, _data='eleme')
  File "/Users/ming/github/peewee/peewee.py", line 3971, in __init__
    setattr(self, k, v)
  File "/Users/ming/github/peewee/peewee.py", line 705, in __set__
    instance._data[self.att_name] = value
TypeError: 'str' object does not support item assignment
(virtualenv)➜  peewee git:(master) ✗


```
### just inheritance dict to resovle this bug
### please see my pr [#619](https://github.com/coleifer/peewee/pull/619) for detail


Unfortunately I do not plan on merging your fix, as it changes some major underlying assumptions about peewee models. For now this is just a limitation of peewee. If your column is named `_data`, you can do this instead:

``` python
class User(Model):
     data = CharField(db_column='_data')
```