akumoth/peewee-issues huggingface dataset:
Issue: Please provide a way to easily check NULLness of a foreign key field
Author: maaaks
State: closed
Created: 2016-09-29 11:10:09

Currently I know only two ways to check whether there is a related instance.

First way:

``` python
try:
    other_obj = obj.other_obj
    # yes, there is

except OtherObj.DoesNotExist:
    # no, there isn't
```

This is **too** long for simple checks which I sometimes have to do.

Second way:

``` python
if obj.other_obj_id:
    # yes, there is
```

This looks good and seems to work, but PyCharm highlights this as reference to a non-existing field `other_obj_id`, so I basically lose a lot of useful IDE features when using this undeclared attribute.

Is there/will there be a third, convinient way to to this? Maybe something like this (I'm not sure how this can be implemented, just an idea):

``` python
with dont_throw_doesnotexist_here():
    if obj.other_obj:
        # yes, there is
```

Or, maybe:

``` python
if SuppressDne(obj).other_obj:
    # yes, there is
```


And — yes, I know about [`suppress()`](https://docs.python.org/3/library/contextlib.html#contextlib.suppress) from standard library, but it still makes impossible to do quick checks inline, like:

``` python
if obj.other_obj and obj.other_obj.foo == "bar":
    # yes, it is "bar"
```


> This looks good and seems to work, but PyCharm highlights this as reference to a non-existing field other_obj_id, so I basically lose a lot of useful IDE features when using this undeclared attribute.

This is really the de-facto way to do it. I'm sorry that PyCharm complains about it, the field is added dynamically. I would avoid using the foreign key directly as that can cause a query to be issued, which in most cases you don't want.


Oh, you can also try this:

``` python
obj._data.get('foreign_key_name') is None
```
Issue: UNION query with limit
Author: jimstjean
State: closed
Created: 2016-09-28 17:05:54

Thanks for your work on the peewee library.
I have just found an issue closely related to #566, this time on mySQL.

The case is:

```
query_1= Object.select().where(SQL(sql_1)).limit(10)
query_2= Object.select().where(SQL(sql_2)).limit(10)

results = (query_1 | query_2).execute()
```

Even if both query_1 and query_2 return 10 non-overlapping records, this does not return the proper 20 records. It only returns 10.

The reason is missing parenthesis around each side of the UNION clause.

peewee generates:
`SELECT ___ FROM ___ WHERE ___ LIMIT 10
UNION
SELECT ___ FROM ___ WHERE ___ LIMIT 10`
this returns only 10 records.

where it should generate:
`(SELECT ___ FROM ___ WHERE ___ LIMIT 10)
UNION
(SELECT ___ FROM ___ WHERE ___ LIMIT 10)`

which will return 20 records as it is supposed to. 

I cannot determine a way to force the parenthesis around each side of the union, however maybe I have overlooked something.

Read more about limit and union behavior here: [http://stackoverflow.com/questions/1415328/combining-union-and-limit-operations-in-mysql-query]
Issue: peewee get_or_create default use id field?
Author: rfyiamcool
State: closed
Created: 2016-09-28 09:25:10

peewee get_or_create default use id field?

can peewee use custom diff  field ?   mysql table  not has `id` field 

```
In [163]: data
Out[163]:
{'brand': 'kkkk',
 'brand_img': 'b_img',
 'extra': '1sdf',
 'pid': 5,
 'pname': 'panme',
 'price': 11,
 'product_img': 'xiaorui.cc',
 'purl': 'url',
 'secound_id': 2,
 'third_id': 3,
 'top_id': 1}
Product.get_or_create(**data)
```

Error:

```
/Library/Python/2.7/site-packages/MySQLdb/connections.pyc in defaulterrorhandler(***failed resolving arguments***)
     34     del cursor
     35     del connection
---> 36     raise errorclass, errorvalue
     37
     38 re_numeric_part = re.compile(r"^(\d+)")

OperationalError: (1054, "Unknown column 't1.id' in 'field list'")

```


This ticket is lacking in a lot of detail...What exactly is it that you're asking? The `get_or_create`  method does not depend on an "id" field, but can work with any constraint. The method itself first attempts a "get" using the data provided. If this fails, then an attempt is made to create. If the create fails with an integrity error, then the `get` is retried. That's all there is to it.


@coleifer please, explain why is it necessary to have an integrity check at get_or_create?
My use case: check an item by key. in case it doesn't exist - create new one. But know I need to provide all the fields, otherwise not null and other constains won't be met.
I would like to provide them only in case there is no matchin by id field.

Besides, current behavior with integrity check is not documented in API: http://docs.peewee-orm.com/en/latest/peewee/api.html. And I can't get how this integrity check is currently implemented. Please, at least provide some explanation of it at API.
(btw, nice ORM! I am loving its simplicity)
Issue: ModelQueryResultWrapper no longer synthesizes models for joins without FK
Author: jakedt
State: closed
Created: 2016-09-27 20:33:08

Due to #1012 we can no longer perform a query we were using and get reasonable results. Previously we were doing a many-to-one and then one-to-many join, and then eliminating duplicate rows on in code on the client side. In the previous behavior we ended up with a synthesized model instance for the incoming FK, and then just deduplicated those.

I know we could do this with the `<incoming_fk>_set()` method, but that would be an extra query per result row. What is the recommended way to accomplish this? We could use tuple results, but it was nice having the results parsed into objects for us automatically.


Oh yeah, I can see the implementation has a bad unintended consequence if the columns are aliases. Can you tell me a bit more about your use case? Its after lunch and I'm having trouble thinking abstractly. Is there an analogy example using Twitter tables? (user / tweet one to many, users to users many to many)?


``` python
class SMS(Model):
  sender = ForeignKey(User)

tweets = Tweet.select().join(User).join(SMS)
```

Now there should be one row for every tweet and SMS combination. Before I could get both tweets and smses from the query results:

```
tweet (unique)
tweet.user (often duplicated)
tweet.user.sms (often duplicated)
```

This isn't a great idea in practice, but we actually have a model where the `SMS` table usually only links to a `User` once.


@coleifer have you had a chance to give this any thought yet?


@jzelinskie -- do you work with @jakedt or has this issue arisen for you independently?


Honestly this feels a bit like a code smell to me. The query you're executing seems very much like a hack. Why not query user and join on both tweet and SMS? Closing as this seems like an edge case that only applies to a particular kind of hack.
Issue: _prefetch doesn't return results
Author: ghost
State: closed
Created: 2016-09-27 06:51:26

I have upgraded from peewee==2.6.4 to 2.8.3 and got some issues with _prefetch_ now.

```
prefetched_users = prefetch(db_users, db_emails, db_interactions)

for user in prefetched_users:
    print user.uuid
    '9533af75-a176-40eb-9b24-cdd2dc3e7e5d'
    print user.emails[0].email
    <email address>

    print user.uuid
    '9533af75-a176-40eb-9b24-cdd2dc3e7e5d'
    print user.emails_prefetch[0].email
    IndexError: list index out of range
```

Some more debugging:

```
prefetched_users = prefetch(db_users, db_emails, db_interactions)

print 'users', db_users.count()
50

print 'emails', db_emails.count()
182

    counter = 0
    for user in db_users:

        found = False

        for email in db_emails:
            if email.user_id == user.uuid:
                found = True
                break

        if found:
            print counter, user.uuid, email.email
        else:
            print 'NOT FOUND!'

        counter += 1
```

Prints 50 uuids + emails but no NOT FOUND.

**peewee==2.8.3**

Let me know if you need any other info.


Had another query where _prefetch_ didn't return expected results as before.


Can you give me something reproducible?


I don't assume you need anything else from me after @alexlatchford's comment.
Do you think you'll get to this soon?

Thanks heaps! ;-)


Should be fixed in 1c237421eaf6d1ab66218e999baa41c1811c5fc2


Brilliant thanks Charles!
Issue: AUTOINCREMENT on sqlite doesn't work properly
Author: dromer
State: closed
Created: 2016-09-21 08:56:16

It seems that AUTOINCREMENT on sqlite primary keys doesn't actually increment properly.

ie.:

```
entries with id 1 and 2
2 is removed
new is added -> id = 2
----
expected -> id = 3
```

In postgres it uses the proper SERIAL increment as expected.


That's how sqlite auto increment work if it is not a primary key: https://www.sqlite.org/autoinc.html


Except I would assume that 'id' is a primary key.


I agree, but that depends on how the table was created. 

Whatever the case I think the problem here is on the sqlite side, not peewee.


The table was created by peewee based on the model:

```
peewee will automatically add an auto-incrementing integer primary key field named id
```

http://docs.peewee-orm.com/en/latest/peewee/models.html


That's how sqlite work:

```
sqlite> create table t (id integer primary key, f);
sqlite> insert into t (f) values (100);
sqlite> insert into t (f) values (200);
sqlite> insert into t (f) values (300);
sqlite> delete from t where f=300;
sqlite> insert into t (f) values (400);
sqlite> select * from t;
1|100
2|200
3|400
sqlite>
```


The peewee documentation explicitly states it uses an auto-incrementing primary key for the row id.

But from the sqlite documentation I understand that it depends where the AUTOINCREMENT keyword is used.
Either way I would expect the same behaviour in sqlite and postgres when peewee creates these tables.


Ok, I found a specific class in peewee's sqlite implementation that should provide this:
https://github.com/coleifer/peewee/blob/master/playhouse/sqlite_ext.py#L92
So the id should be overridden to use this class in stead.


As was noted, see the:

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#PrimaryKeyAutoIncrementField
Issue: Bug with multiple foreign keys to same table and DeferredRelation
Author: miketheanimal
State: closed
Created: 2016-09-20 10:09:26

Python3 with Mysql, peewee from pip3 install --upgrade peewee. Two mysql tables:

```
mysql> desc c ;
+-------+---------+------+-----+---------+----------------+
| Field | Type    | Null | Key | Default | Extra          |
+-------+---------+------+-----+---------+----------------+
| id    | int(11) | NO   | PRI | NULL    | auto_increment |
+-------+---------+------+-----+---------+----------------+
1 row in set (0.00 sec)

mysql> desc p ;
+-------+---------+------+-----+---------+----------------+
| Field | Type    | Null | Key | Default | Extra          |
+-------+---------+------+-----+---------+----------------+
| id    | int(11) | NO   | PRI | NULL    | auto_increment |
| c1    | int(11) | YES  |     | NULL    |                |
| c2    | int(11) | YES  |     | NULL    |                |
+-------+---------+------+-----+---------+----------------+
3 rows in set (0.01 sec)
```

and the following test code:

```
from peewee import *
db = MySQLDatabase('p', user='p', password='p', host='localhost')
deferredC = DeferredRelation()

class C(Model):
    id = PrimaryKeyField()
    class Meta:
        db_table = 'c'
        database = db
class P(Model):
    id = PrimaryKeyField()
    c1 = ForeignKeyField(deferredC, db_column='c1', related_name='c1')
    c2 = ForeignKeyField(deferredC, db_column='c2', related_name='c2')
    class Meta:
        db_table = 'p'
        database = db

deferredC.set_model(C)

c1 = C.create()
c2 = C.create()
p1 = P.get(P.id == 3)
p1.c1 = c1
p1.save()
p2 = P.get(P.id == 3)
p2.c2 = c2
p2.save()
db.commit()
```

If I run this on database with table c empty, and one record in p with p.c1=null, p.c2=null, then I get two records in table c (as expected) but only p.c1 links, not p.c2:

```
mysql> select * from c ;
+----+
| id |
+----+
| 15 |
| 16 |
+----+
2 rows in set (0.00 sec)

mysql> select * from p ;
+----+------+------+
| id | c1   | c2   |
+----+------+------+
|  3 |   15 | NULL |
+----+------+------+
1 row in set (0.00 sec)
```

If I remove the DeferredRelation() and use ForeignKeyField(C, ....) then it works as expected, and p.c1 and p.c2 are both correctly set.

Thanks
Mike


Fixed! Thanks for the great ticket.


Glad to be of service :)
Issue: fix ignore & replace syntax for mysql
Author: 631068264
State: closed
Created: 2016-09-19 15:09:40

2 bugs in peewee.py
1.   `upsert_sql = 'INSERT OR REPLACE INTO'`   will have syntax error for Mysql. So I fix it like `upsert_sql = 'REPLACE INTO'`.
2.  add ignore for mysql   `INSERT OR IGNORE INTO` will have syntax error for Mysql.

Example:
Page.insert(data).ignore().execute()
Page.insert_many(data).ignore().execute()

Generate sql：
'INSERT IGNORE INTO `page` (`manga_id`, `page_num`, `img_url`) VALUES (%s, %s, %s)', [11997, 6, 'http://img.177pic66.com/uploads/2016/05a/00178.jpg']

And you will get mysql warning when you use `ignore` but it can work：

```
/Users/wyx/project/scrap/.env/lib/python3.5/site-packages/pymysql/cursors.py:166: Warning: (1062, "Duplicate entry '6' for key 'page_page_num'")
  result = self._query(query)
```


Currently ran into this. Would be awesome to see 1`INSERT IGNORE INTO` added for MySQL

@coleifer what would need to happen to get this merged in. Would be willing to create new PR if needed.

This should be working in 3.x. See:

* http://docs.peewee-orm.com/en/latest/peewee/api.html#Insert
* http://docs.peewee-orm.com/en/latest/peewee/api.html#Model.replace
Issue: Add `upsert_many`
Author: maxnoe
State: closed
Created: 2016-09-19 14:23:00

There's `insert_many` but not `upsert_many`


What do you get calling insert_many() with .upsert() as well?


You mean `insert_many(stuff).upsert()`?


Yeah, I was just away from my computer and wondered what that would output?


Ah, yes. This works. Thanks a lot.
Issue: primary keys ignored in update queries
Author: phryk
State: closed
Created: 2016-09-19 02:55:33

Primary keys are ignored (at least for `CompositeKeyField`) when `UPDATE`-ing rows.
I have a few "linker" tables that are just two `ForeignKeyField` used as `CompositeKeyField`.
They are not editable due to this, since any call to `.save()` results in a syntax error in the query.

I pasted an example illustrating this behavior here: https://paste.xinu.at/Jz87/

Does peewee just not support updating primary keys due to potential consistency problems when using `ForeignKeyField` or is this a bug? I.e. do I have to work around this in my software or do you consider this something that falls under peewee's responsibility?


When it comes to using composite primary keys, things like save() break down a bit because peewee doesn't know what exactly to do since the previous primary key values are gone from the model instance. The way it should work is that it would update, setting the new values and using the old values as the where clause. But when you change them, the old values are gone so peewee freaks out.

But changing primary keys is kind of a sign that maybe you should consider a different primary key.


I'm not editing for any real use-case, I'm just writing very generic software on top of peewee and noticed this behavior, which seemed weird to me - but it makes total sense now.
I'll look for a way to either nicely assert from a Model class whether this will happen or simply implement an immutability flag for Models like this.

Thanks for the info so early in the morning. :)
Issue: KeyError: '"title" is not a recognized field.'
Author: 631068264
State: closed
Created: 2016-09-16 15:57:12

```
class Manga(BaseModel):
    manga_id = IntegerField(unique=True, help_text="漫画id")
    cover_url = CharField(help_text="封面链接")
    view_page = CharField(help_text="漫画链接")
    title = CharField(help_text="漫画标题")
    date = DateField(help_text="发布日期")
```

What I want to save is a list of dict  like 

`{'manga_id': 1233504, 'title': '[中文][天道まさえ] 二つ結び [雙重馬尾][177P]', 'cover_url': 'http://img.177pic66.com/uploads/2016/08a/00012.jpg', 'date': datetime.date(2016, 9, 15), 'view_page': 'http://www.177pic66.com/html/2016/09/1233504.html'}`

and I use `Manga.insert_many(items).execute()` so why I can't insert it.
Issue: {AttributeError}'ModelOptions' object has no attribute 'declared_fields'
Author: smittysmee
State: closed
Created: 2016-09-16 15:26:15

Attempting a basic command, but it appears that "declared_fields" in the _meta is not a valid field.
See line in code [here](https://github.com/coleifer/peewee/blob/master/playhouse/shortcuts.py#L107). 

**Model:**

```
db = AppEngineDatabase('adb')

class CloudSQLModel(Model):
    create_date = DateField(default=datetime.datetime.now)
    last_modified = DateField(default=datetime.datetime.now)
    modified_by = CharField(null=True)  #ndb key of the user that modified it

    class Meta:
        database = db  # This model uses the "agent.db" database


# Add the initial items for insertion into our datastore
class SourcedModel(CloudSQLModel):
    # Create the available status of the property
    STATE_CHOICES = (
        (0, 'new'),
        (1, 'csg_cleaned'),
        (2, 'third_party_cleaned'),
        (3, 'ready'),
        (4, 'released'),
        (5, 'expired'),
        (9, 'deleted'))

    # Source of the location that we found... usually a URL will suffice
    source = CharField(null=True)

    # Date of the item last touched upon
    last_seen_date = DateField(null=True)

    # Status of the property... we do not delete, just disable
    state = IntegerField(default=0, choices=STATE_CHOICES)

    # Return the status of the item
    def get_status_label(self):
        return dict(self.STATUS_CHOICES)[self.status]

class NationalProducerNumber(SourcedModel):
    number = CharField()
```

**Create new item and attempt to print model.**

```
npn = NationalProducerNumber.create(
                    number="001193381901"
                )
print(model_to_dict(non))
```

When I run the code shown, I receive the following Attribute Error:
`'ModelOptions' object has no attribute 'declared_fields`

Declared fields does not seem to be an attribute of the _meta object. `fields` is present, but `declared_fields` is not. 

Am I missing a key step in this?


Issue with versions in my systems library and within the project.
Issue: Add postgresql network address types plz
Author: Q-Master
State: closed
Created: 2016-09-16 14:11:20

ALTER TABLE info ADD COLUMN ip inet;
this column type causes the pwiz.py to generate UnknownField class.

https://www.postgresql.org/docs/9.1/static/datatype-net-types.html


Peewee does not currently support that column type. To add it, see the custom field docs:

http://docs.peewee-orm.com/en/latest/peewee/models.html#custom-fields

To get pwiz to understand the field, you'll first need your field class, then you'll want to modify the `PostgresqlMetadata.column_map` mapping the OID to the new field type.
Issue: Join by field (using)
Author: ak4nv
State: closed
Created: 2016-09-15 09:12:58

I try to write query:

``` sql
select code, name
    from medservices join (
        select unnest(ms) as code from services where id = 7
    ) u using (code);
```

in peewee. And I can't understand how to do:

``` python
    qs = (MedServ
        .select(MedServ.code, MedServ.name)
        .join(
            Service.select(fn.unnest(Service.ms).alias('code')).where(Service.id == id),
            on=(MedServ.code == ???))
```

I need join table with subquery by named field. Is it possible?

P.S. Will the 3'd version of peewee use asyncio? :-)


Joining on a subquery is totally easy in peewee, you just have to use the explicit form of the join predicate:

``` python
subq = Service.select(fn.unnest(Service.ms).alias('code'))
subq = subq.alias('svc')  # Subqueries need an alias.
qs = (MedServ
      .select(MedServ.code, MedServ.name)
      .join(subq, on=(MedServ.code == subq.c.code)))
```

To access a column inside the subquery we use the magical "subq.c.code". The ".c" just allows us to create references to values inside the subquery.

Does that help?

And no, peewee itself will not, but there are already at least one Peewee extension that provide asyncio. This one seems pretty popular:

https://github.com/05bit/peewee-async


Also, docs: http://docs.peewee-orm.com/en/latest/peewee/querying.html#using-subqueries


> Does that help?

Yes, that's I need! Thanks!
Really, yesterday I tired and forgot about `.c`

I surprised about progress peewee-asyncio. Last time I saw the project it had 0.0.5 version :)
What is the main difference between 2.8 and 3 branches of peewee?


Closing as the issue seems resolved.

Differences: mostly around the SQL generation, but generally the goal is to have a more powerful sql generator. Also decoupling the returning of records from model instances
Issue: fixed a problem with TimestampField resolution
Author: akrs
State: closed
Created: 2016-09-14 03:56:17

When retrieving a value from the database, TimestampField didn't properly scale the microseconds by the resolution. Instead of using the resolution, it would always assume the fractional portion was in microseconds.

This fixes that by scaling the fractional second portion as a function of the resolution.


Also had to fix the unit test, as the unit test assumed that `654321` microseconds stored with a millisecond precision would come out as `654` microseconds instead of `654000` microseconds.
Issue: Variable non-persistent instances?
Author: johaven
State: closed
Created: 2016-09-13 17:07:43

``` python
>>> p = db.Paths.select().where(db.Paths.id == 1).get()
>>> p.lastsync
datetime.datetime(2016, 9, 13, 18, 59, 32, 463395)
>>> v = db.Paths.select().where(db.Paths.id == 1).get()
>>> id(p), id(v)
(4545364304, 4545364496)
>>> v.lastsync
datetime.datetime(2016, 9, 13, 19, 0, 41, 913038)
>>> import datetime
>>> v.lastsync = datetime.datetime.now()
>>> v.save()
1
>>> v.lastsync
datetime.datetime(2016, 9, 13, 19, 1, 28, 755457)
>>> p.lastsync
datetime.datetime(2016, 9, 13, 18, 59, 32, 463395)
>>> v.is_enabled = 0
>>> v.save()
1
>>> v.is_enabled
0
>>> p.is_enabled
True
```

Is this a standard behavior?


peewee is an activerecord ORM. Once you understand that, though, there are no surprises. An instance is a container for row values, but not a representation of the row itself in the sense that multiple instances with the same ID are somehow linked.

Does that help?  And yes it is standard behavior. Django is the same.


Please add a comment or create a new issue if you feel I haven't answered your question, or you are still not sure how this works.


Ok for me, thank you :)