akumoth/peewee-issues huggingface dataset:
Issue: Should there be an All expression?
Author: toppk
State: closed
Created: 2019-12-09 07:17:03

Just getting started with peewee, and I was looking at this pattern , to dynamically control the "where " clause.   

```
where = Expression(1,"=",1) # All
if some_condition1():
    where &= (Users.group_id != 1)
if some_condition2():
    where &= (Users.name % "%en%")
query = Users.select().where(where)
```

It seems this pattern is the best way to accomplish a dynamic where condition.  If so, it would be nice to have an "All" expression instance inside peewee, so I wouldn't have to define it myself.  

Something like:
`All = Expression(1,"=",1)`

Of course, if there's an better way to control which expressions are used in a query, I'd like to know what it is, I've looked at the examples, and didn't see anything offhand.

Just append to a list and use `reduce` to combine the list elements (expressions).

https://stackoverflow.com/questions/22238194/python-peewee-dynamically-or-clauses?r=SearchResults

That still has the same issue with an empty clauses array.  you need to have one element for this to work.

```python
clauses = [
    (User.username == 'something'),
    (User.something == 'another thing'),
    ...
]
query = User.select()
if clauses:
    query = query.where(reduce(operator.or_, clauses))
```
Issue: atomic() does not seem to have lock_type argument on 3.13.0
Author: JeanBarriere
State: closed
Created: 2019-12-06 11:27:42

Hi !

We're using peewee on our project and we are using it with an IMMEDIATE lock type, as: 
`db.atomic(lock_type="IMMEDIATE")`

Everything was working fine on the 3.12.0, but now we have this error:

`TypeError: atomic() got an unexpected keyword argument 'lock_type'`

Any idea why ?

Looks like a breaking change made here: https://github.com/coleifer/peewee/commit/36f5b4a0fef0bd236f25bf99f92f74852f219103

For now try removing the keyword (lock_type) and just pass the string 'immediate'. I will put together a fix, in any case.

I'm tagging a new release once the build passes. Thanks for reporting this so quickly.

https://github.com/coleifer/peewee/releases/tag/3.13.1

Thank you for the fast update
Issue: Regression in 3.12.0 with regard to saving model with JSONField
Author: arel
State: closed
Created: 2019-12-04 06:34:28

A change between 3.11.2 and 3.12.0 broke support for JSONFields. This seems related to #2034, but I don't think this is a duplicate.

I'm using mysql as the database.

To reproduce:

```python
import os
import json
from peewee import Model, TextField
from playhouse.db_url import connect

uri = os.environ.get('DATABASE')
db = connect(uri)

class JSONField(TextField):

    def db_value(self, value):
        json_string = json.dumps(value)
        return json_string

    def python_value(self, value):
        if value is not None:
            return json.loads(value)


class BaseModel(Model):
    class Meta:
        database = db

class Example(BaseModel):
    data = JSONField()

e = Example.create(data=['abc', 'def'])
e.save() # FAILS for version 3.12.0; works for 3.11.2.
```

I get the following exception:
```
...
~/.local/share/virtualenvs/acm-api-UVOmdwT4/lib/python3.7/site-packages/pymysql/err.py in raise_mysql_exception(data)
    107         errval = data[3:].decode('utf-8', 'replace')
    108     errorclass = error_map.get(errno, InternalError)
--> 109     raise errorclass(errno, errval)

InternalError: (1241, 'Operand should contain 1 column(s)')
```
And, if I debug, I see the problem is with the JSONField's SQL representation:

```mysql
UPDATE `example` SET `data` = ('\"abc\"', '\"def\"') WHERE (`example`.`id` = 2)
```

Thanks for reporting, this is now fixed in master.
Issue: BL-536
Author: rafae1
State: closed
Created: 2019-12-03 14:04:41

imporve JSONLookUp contains method

Hey, there's some cool stuff in here. I'll look to maybe integrate some of this. Can you tell me more about some of the changes I'm looking at? I understand the reasoning for certain of them, but others aren't clear exactly what the intent is.

* e5ad248db24210e62a2a3829ea78627202aa0d3d
* 99921b9b3d8444eec1daf58d4ce06b6210e3aefd
* 8829c9483fef222e844a092c7720c8e5f6ee1a07
Issue: Support for SuperSQLite
Author: lllama
State: closed
Created: 2019-12-03 13:43:25

Is it possible to swap out the default sqlite driver with SuperSQLite? (https://github.com/plasticityai/supersqlite). The library provides an up to date version of sqlite and adds a number of extensions which would be useful for my app.

Can I swap the underlying lib, or would it require its own backend?

You have *got* to be kidding me, this is absolute garbage. It is incredibly easy to build your own SQLite, which the default Python driver should pick up.

This should contain all the information you might need:

http://charlesleifer.com/blog/compiling-sqlite-for-use-with-python-applications/

Not seeing a single thing about Windows on there, plus specifying a single dependency with pre-built binaries would make cross-platform distribution a lot easier but thanks for the quick response, I guess?

Sorry - I don't know the first thing about how to do this on Windows.

Okay, I'll investigate what I need to do for that and how to bundle it up with my app. Given that I'll struggle to then get a Windows build working with gitlab CI, would implementing my own backend be the best way forward should I want to use that other lib? Or is some sort of monkey patch possible to get the existing one working?

You can look at the apsw playhouse extension for an example, since it looks like supersqlite is a wrapper on apsw.
Issue: delete_instance causes error if tables are not created
Author: JuleBert
State: closed
Created: 2019-12-03 12:25:47

Hi there,

we are using peewee to create a medium complex data structur. We are using inheritance since a lot of table are similar to each other. We do this by creating template classes which are not created as tables later on. These classes are picked up by delete_instance although they are created as tables. Which causes an error. 
Is this a bug and can be fixed by checking whether a table is created? Or is there a workaroud / hint for me to make this work without creating my template classes as tables?

Thanks a lot in advance

Peewee should only be traversing the graph of tables related by foreign-key, so my guess is that you have some weird stuff going on in your foreign-key inheritance because Peewee will not traverse subclasses blindly. If you can provide a minimal example showing a bug I will reopen.

A minimal example is no problem. 
As you will see the class `Child` is not created as a table and is only used as a template to be inherited from.

```
# docker run --name postgres -e POSTGRES_DB=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -d -p 5432:5432 postgres

from peewee import *

pg_db = PostgresqlDatabase('postgres', user='postgres', password='postgres',
                           host='localhost', port=5432)

class Parent(Model):
    name = CharField()
    class Meta:
        database = pg_db

class Child(Model):
    parent = ForeignKeyField(Parent)
    class Meta:
        database = pg_db

class Son(Child):
    boys_name = CharField()

class Daughter(Child):
    girls_name = CharField()

if __name__ == '__main__':
    pg_db.create_tables([Parent, Son, Daughter])
    parent = Parent.create(name='dad')
    Son.create(boys_name='jim', parent=parent)
    Daughter.create(girls_name='jane', parent=parent)
    parent.delete_instance(recursive=True)
```

Oh, I see what you mean - thank you, you are correct this does produce an error due to the missing Child table. Unfortunately, this is not something I am planning to fix, as inheritance + foreign-keys has always been rather fraught territory. Peewee models maintain an in-memory graph of their relations and there is no way to differentiate between a model that is used as a simple "base-class" only.

My recommendation would be to put the foreign-keys on the base-classes *OR* declare your foreign-keys such that they specify the appropriate on_delete/on_update behavior and then avoid using recursive delete-instance.

```python
parent = ForeignKeyField(Parent, on_delete='cascade')
```

Putting FKs on base-classes is also problematic because of the way backrefs are set-up. It's truly better to just put them where they live.
Issue: fn.IF returning True instead of values
Author: mikemill
State: closed
Created: 2019-12-02 20:49:49

I'm trying to use `fn.IF` to pick a value to use.  However, if using a model property it is returning `True` as the value instead of the proper value and this is happening even if that property is `False`.


Example script (note: this is a greatly simplified version of what I'm actually trying to do):
```python
from peewee import BooleanField, fn, Model
from playhouse.pool import PooledMySQLDatabase


db_conn = PooledMySQLDatabase(
    user='',
    password='',
    database='',
    stale_timeout=5 * 60
)

class TestModel(Model):
    flag = BooleanField()

    class Meta:
        database = db_conn

TestModel.create_table(safe=True)

TestModel.create(flag=True)
TestModel.create(flag=False)

query = (
    TestModel.select(
        TestModel.id,
        fn.IF(
            TestModel.flag,
            'Hello',
            'GoodBye'
        ).alias('test')
    )
).dicts()

print(query)

for record in query:
    print(record)
```


The output I get is:
```
SELECT `t1`.`id`, IF(`t1`.`flag`, 'Hello', 'GoodBye') AS `test` FROM `testmodel` AS `t1`
{'id': 1, 'test': True}
{'id': 2, 'test': True}
```
And if I run that query inside MySQL I get the following results:
```
mysql> SELECT `t1`.`id`, IF(`t1`.`flag`, 'Hello', 'GoodBye') AS `test` FROM `testmodel` AS `t1`;
+----+---------+
| id | test    |
+----+---------+
|  1 | Hello   |
|  2 | GoodBye |
+----+---------+
```
which is what I'd expect to get.

I can modify the query slightly to:
```python
query = (
    TestModel.select(
        TestModel.id,
        fn.IF(
            TestModel.flag == True,
            'Hello',
            'GoodBye'
        ).alias('test')
    )
).dicts()
```

And that does appear to work but it seems unnecessary to require the comparison on a boolean field.

Peewee uses a heuristic for function calls when the first argument is a field, where peewee applies that field's conversion method to the value returned by the database. This usually works, but you've found a case where it is problematic. To avoid coercing the function output, specify `coerce=False`:

```python
# Note you don't need TestModel.flag == True
fn.IF(TestModel.flag, 'hello', 'goodbye', coerce=False)
```

Confirmed.  Thanks!
Issue: How to print raw sql for Blog.select().where(Blog.id == 1)
Author: jet10000
State: closed
Created: 2019-11-29 09:11:10

```
q = Blog.select().where(Blog.id == 1)
print(q.sql())
```
I want to compile to raw sql .



Peewee returns a parameterized query, a 2-tuple of SQL string and parameters. This is to save people doing crazy stuff and getting SQL injection vulnerabilities. The driver accepts this 2-tuple and does the right thing.

What about for DB API (PEP-249) v2.0 drivers?  How can I tell peewee to not add double quotes on everything?  
ex:
from:  SELECT "t1"."item1", "t1"."item2", "t1"."item3" FROM "trex_inventory_recipe_material" AS t1
to: SELECT t1.item1, t1.item2, t1.item3 FROM trex_inventory_recipe_material AS t1

Why would you want to do that? Lots of times people unwittingly (or intentionally) use reserved words for column-names, e.g. "count". Peewee adds quotes to ensure that identifiers are always treated as-such.

I'm using Impyla which is fully DB API v2.0 compiant.  And the HiveServer2 (Hive) gives an error when it sees double quoted column-names.

On your database class, just set the quote attribute to be an empty string.

```python

class HiveDatabase(...)
    quote = ''
```

Does that address the issue?

No change - using peewee 2.9.0
I'm still getting double quotes when doing a select() on the Model,

<class 'model.dbo.item_table.ItemTable'> SELECT "t1"."rname", "t1"."mrole", "t1"."rmaterial" FROM "db1"."item_table" AS t1 []


>peewee 2.9.0

That's a very old version, from March 2017. For peewee 3.0, I think you would actually set `quote = ('', '')` instead of empty string, but since you're on such an old version I'm not sure what to advise.

Thanks @coleifer  - appreciate it.  Thanks for pointing me in the right path - using `quote = ''`
Also, you reminded me that I forgot to change the version on the documentation.  I was looking at the "latest".  Older versions (like 2.9.0) use `quote_char = ''`  
Yup; I will be upgrading it and fix any breaks along the way. Thank you!
Issue: Model.filter not generating correct SQL for join clause
Author: yifeikong
State: closed
Created: 2019-11-28 08:43:23

I have three models in a heritical order:

```python
class CrawlProject(Model):
    name = CharField()

class CrawlGroup(Model):
    name = CharField()
    project = ForeignKeyField(CrawlProject)

class CrawlRule(Model):
    name = CharField()
    group = ForeignKeyField(CrawlGroup)
```
    
Given the following code:

```python
CrawlRule.filter(group__project__id=47)
```

Then I got the following error:

```
 File "/home/tiger/repos/baelish/.venv/lib/python3.6/site-packages/MySQLdb/cursors.py", line 378, in _query
   db.query(q)
 File "/home/tiger/repos/baelish/.venv/lib/python3.6/site-packages/MySQLdb/connections.py", line 280, in query
   _mysql.connection.query(self, query)
eewee.OperationalError: (1054, "Unknown column 't3.project_id' in 'on clause'")
```

The generated SQL is like:
```SQL
SELECT `t1`.`id` FROM `crawl_rule` AS `t1`
INNER JOIN `crawl_project` AS `t2`ON (`t3`.`project_id` = `t2`.`id`)
INNER JOIN `crawl_group` AS `t3` ON (`t1`.`group_id` = `t3`.`id`)
WHERE ((`t3`.`project_id` = 47)) LIMIT 20 OFFSET 0
```
It seems that `t3` was referenced before defined.

actually, simply swapping the inner joins works:

```SQL
SELECT `t1`.`id` FROM `crawl_rule` AS `t1`
INNER JOIN `crawl_group` AS `t3` ON (`t1`.`group_id` = `t3`.`id`)
INNER JOIN `crawl_project` AS `t2`ON (`t3`.`project_id` = `t2`.`id`)
WHERE ((`t3`.`project_id` = 47)) LIMIT 20 OFFSET 0
```


Sorry, 3.12.0 seems to work, closing. I was using 3.10

OK, It's not fixed. Actually it randomly happens each time you restart the flask server.

I think the reason is in function of  peewee.ModelSelect.filter().
target code is:
```python
        dq_joins = set()
        while q:
            curr = q.popleft()
            if not isinstance(curr, Expression):
                continue
            for side, piece in (('lhs', curr.lhs), ('rhs', curr.rhs)):
                if isinstance(piece, DQ):
                    query, joins = self.convert_dict_to_node(piece.query)
                    dq_joins.update(joins)
```

I can't understand why you use set() to define dq_joins? if use list(), this error will disappear.

Thanks for the analysis. Yeah, it looks like the use of `set` is causing problems with the ordering of the joins. I will take a look!

Fixed.

Thanks! peewee has the lowest open-issue/star ratio in all the repositories I have seen on GitHub. ^_^
Issue: Join attr does get added with left outer join (3.11 ... 3.12)
Author: nad2000
State: closed
Created: 2019-11-26 00:33:15

http://docs.peewee-orm.com/en/latest/peewee/relationships.html#selecting-from-multiple-sources shows how to user **attr** with joins. It works with inner join fine. The query fails with the outer join. 

```python
>>> class User(Model):
...     name = CharField()
...     class Meta:
...         database = db

>>> class Tweet(Model):
...     user = ForeignKeyField(User, null=True)
...     message = TextField(null=True)
...     class Meta:
...         database = db


>>> for r in Tweet.select(Tweet, User).join(User, JOIN.LEFT_OUTER, attr="author"): print(r.author)
2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Tweet' object has no attribute 'author'

'Tweet' object has no attribute 'author'

>>> for r in Tweet.select(Tweet, User).join(User, attr="author"): print(r.author)
2

```

It seems you included two versions in the issue title -- are you indicating this is a regression?

At any rate -- when you are using a left outer join, all rows from the left-hand table (Tweet) are included, along with any matching from the right-hand (User). This means you can have Tweet without a User.

When you manually patch a related object onto a user-defined attribute, currently peewee does not set the join attribute when there is no data/related-object to set.

You can work around this for now by just using the regular `user` attr, or you can use `hasattr(tweet, 'author')` / `getattr(tweet, 'author', None)` instead.

> It seems you included two versions in the issue title -- are you indicating this is a regression?

I have seen this behaviour with both versions. 

yes, it's, in a way, a regression. With version 2.x (it used to be defined with an **alias** of the join condition) it was working. The value of the property was None, eg, the test was `twee.author is None`

This is probably worth documenting as I was expecting a null result. (Thinking about it, I suspect the null result would be rather difficult to implement -- so docs would be great)
Issue: dynamic table name support
Author: TaylorChen
State: closed
Created: 2019-11-25 12:59:14

In my business, some tables are divided according to time, such as table table_20191214, table_20191215, how can I do it in the current orm framework?

It's right there in the docs.

http://docs.peewee-orm.com/en/latest/peewee/models.html#table-names

> If you wish to implement your own naming convention

It has example code.
Issue: Typo in example.rst doc
Author: maghniem
State: closed
Created: 2019-11-24 17:04:48

Adding fields after the table has been created will required you to
    either drop the table and re-create it or manually add the columns
    using an *ALTER TABLE* query.

To

Adding fields after the table has been created will require you to
    either drop the table and re-create it or manually add the columns
    using an *ALTER TABLE* query.
Issue: Peewee and Postgres HA: load balancer or master/master fail-over?
Author: Andrei-Pozolotin
State: closed
Created: 2019-11-23 00:18:40

@coleifer Charles:

1. say, there is a master/master replication cluster based on `pglogical`:
https://www.2ndquadrant.com/en/resources/pglogical/pglogical-docs/

2. question: is there an easy way to reuse [PooledPostgresqlExtDatabase](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#PooledPostgresqlExtDatabase)
so that it can serve as:
* a round robin load balancer for the cluster
* a transparent master/master fail-over manager
* ?

3. any ideas are much appreciated, thank you.




I'm not familiar with that project, but I think I get the jist of your question and hopefully can provide some pointers.

Peewee implements a simple connection-per-thread model internally. The connection pool simply plugs into the connect / close steps of the usual peewee workflow to recycle connections. So it provides pooling, but not load-balancing or master/slave or anything of that nature. The pool's `_connect()` method shows how you can insert your own logic for actually creating connections.

Maybe the simplest thing you could do would be to skip the pool for now and just subclass `PostgresqlDatabase` and override the `_connect()` method (the method responsible for actually creating the psycopg2 connection). You could pass in a list of hosts and then, when the method is called, open a connection on the appropriate host. You could also add your failover logic here, e.g.:

```python

class ExampleDatabase(PostgresqlDatabase):
    def __init__(self, database, conn_info_list, **kwargs):
         # e.g., [{'host': '10.8.0.1'}, {'host': '10.8.1.1'}, ...]
         self.conn_info_list = conn_info_list
         self.conns = itertools.cycle(self.conn_info_list)
         super(ExampleDatabase, self).__init__(database, **kwargs)

    def _connect(self):
        while self.conn_info_list:
            conn_kwargs = next(self.conns)  # load balancing
            try:
                kwargs = self.connect_params.copy()  # Get a copy of anything specified in __init__()
                kwargs.update(conn_kwargs)  # Add in args specific for the next conn in list.
                conn = psycopg2.connect(database=self.database, **kwargs)
            except SomeException:
                # failover
                self.conn_info_list.remove(conn_kwargs)  # Remove the bad conn from list
                self.conns = itertools.cycle(self.conn_info_list)
            else:
                return conn
        raise Exception('out of connections to try! everything failed!')
```

Hopefully this gives you some ideas, though I think stuff like this might be better handled in a dedicated connection proxy, e.g. pgbouncer or something that sits between peewee and your database cluster.

1. Thank you for the code snippet - that is what I had in mind. Though hoped `peewee` had more hidden treasures right in the play box :-)

2. `PgPool` and `PgBouncer` both add needless complexity, while their fail-over modes are rather limited/inflexible.

3. Thanks again!
Issue: Iterating through a ManyToManyField with a DeferredThroughModel gives back Expression objects
Author: seanballais
State: closed
Created: 2019-11-13 07:31:36

I have a two models, `Class` and `TimeSlot`. `Class` has a many-to-many field, `timeslots`, to `TimeSlot`. The field uses a `DeferredThroughModel`.

Now, say I have an instance of `Class` in a variable called `class`. I want to iterate through the `timeslots` of `class`, so let's do:

    for timeslot in class.timeslots:
        print(type(timeslot))

However, the type of `timeslot` is `Expression` and not `TimeSlot`. `class.timeslots` also gives back a `ManyToManyField: (unbound)` instead of `ManyToManyQuery`. This problem only affects `Class`. Other classes that use a `ManyToManyField` without a `DeferredThroughModel` just do fine.

How do you fix this problem?

Eureka! I solved the problem.

I did not perform `.set_model()` to the `DeferredThroughModel` instance I was using for `Class`. It was only being used inside a function that only gets called during creation of the database file causing the field to be unbounded and causing my problem when the database file was not being created. Using `.set_model()` outside the function and during the initialization of the module all of my models reside in fixed the issue.

**TL;DR to those folks who might come across the same problem:** Make sure `.set_model()` for all `DeferredThroughModel` instances gets called before using the model the `DeferredThroughModel` is used in.
Issue: How to get last day of month
Author: Maheshkumar094
State: closed
Created: 2019-11-05 12:06:48

I am trying to get last day of the month for that I can find out the sql query
`SELECT (date_trunc('MONTH', now()::date) + INTERVAL '1 MONTH - 1 day')::DATE;`
but I am unable formate into peewee query can you please help me doing this.
Your comment and suggestion really appreciate.
Thank You


Couple ways, this way worked for me just now:

```python

from peewee import *
from peewee import NodeList  # Grab this explicitly.

class Foo(BaseModel):
    timestamp = DateTimeField()

ts_month = fn.date_trunc('month', Foo.timestamp)
eom = NodeList((SQL('INTERVAL'), '1 MONTH - 1 DAY'))

# For each timestamp we have stored, grab the last day of the month
# for the given timestamp.
query = Foo.select(ts_month + eom)
for row in query.tuples():
     print(row)

# Alternatively, if you just want the end of the current month:
ts_month = fn.date_trunc('month', fn.now())
eom = NodeList((SQL('INTERVAL'), '1 MONTH - 1 DAY'))
expr = ts_month + eom
```

There's probably a better way to do it, but the above should work.

@coleifer 
Thanks a lot, work for me.