akumoth/peewee-issues huggingface dataset:
Issue: Model converter behavior change
Author: vaniakov
State: closed
Created: 2021-03-18 16:16:25

Hi!
While updating peewee from 3.13.3 to 3.14.3 I noticed model converter behavior change.
If model has __str__ method defined, it is used to represent this model's value in the expression instead of primary key:

```python
import uuid
from peewee import *


db = SqliteDatabase(":memory:")


class BaseModel(Model):
    class Meta:
        database = db


class User(BaseModel):
    id = CharField(max_length=255, primary_key=True)
    name = CharField(max_length=255)

    def __str__(self):
        return f"{self.name}"


if __name__ == "__main__":
    User.create_table()

    users = []
    for i in range(10):
        users.append(User.create(id=uuid.uuid4(), name=f"User_{i}"))

    query = User.filter(User.id << users)
    print(query.sql())
    print(list(query))
```
Output:
```
('SELECT "t1"."id", "t1"."name" FROM "user" AS "t1" WHERE ("t1"."id" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?))', ['User_0', 'User_1', 'User_2', 'User_3', 'User_4', 'User_5', 'User_6', 'User_7', 'User_8', 'User_9'])
[]
```

The change was introduced in this commit: https://github.com/coleifer/peewee/commit/ebe3ad5023d60ebf2fb91528d422a01596220cde

I'm wondering is this behavior is intentional.
Thanks.

Thanks for the succinct report and isolating the problematic changeset. No, this was a regression that I didn't foresee. I've patched this and will issue a new release.

This is fixed and I've tagged a new version 3.14.4, which contains the fix and additional tests.

@coleifer Thank you for such quick response and fix!
Issue: [Q] Is it possible to mark a model object as "unsave-able"?
Author: NightMachinery
State: closed
Created: 2021-03-17 20:59:30

I have some data analysis code that reads some objects from DB, and mutates them for easier analysis; I want to make sure that these mutated objects are never accidentally saved to the database. Is there a way to enforce this?

No, this is not something that is supported in Peewee.

@coleifer I just thought about simply setting `model.save = None`. I think this will be enough to prevent me from shooting myself in the foot later ...

You would also want to lock-down the update, delete, delete_instance, etc, etc, presumably. If you're using sqlite it's possible to set the connection read-only and may be ways to do this with other databases as well. That's what I would look into. Or even adding a read-only user to your db and connecting with that user.
Issue: [Documentation] Lost link "Advanced Connection Management"
Author: aferrari94
State: closed
Created: 2021-03-17 13:43:18

Hi,
I was reading the documentation about the `connection pool` and I think I have found a missing reference caused by a reorganization of documentation with the new versions.

-------------------------

With the doc of the current version (v3.6.0), there is an error about an old link that is pointing to something that is not anymore in the structure documentation.

I'm speaking of the actual section of `Connection Pool` into the chapter `Playhous, extensions to Peewee`: at the end of the section missing a link:

<img width="1901" alt="connPoll-latest" src="https://user-images.githubusercontent.com/70535738/111475819-c434ba00-872d-11eb-9908-250952a9de6a.png">

In the `doc of the version 2.10.2` it was pointing to this section:

<img width="1917" alt="old-AdvConnMng" src="https://user-images.githubusercontent.com/70535738/111475973-efb7a480-872d-11eb-8e39-2410a0f433c1.png">

Actually, the above section doesn't exist anymore because, I think, it was move and update into this new section (actually present in the latest documentation):

<img width="1919" alt="new-AdvConnMng" src="https://user-images.githubusercontent.com/70535738/111476275-4c1ac400-872e-11eb-86db-b55dcbb1a6a4.png">

-------------------------

I hope this could be useful.

Regards,
Ale
Issue: Question about data joining to the same model
Author: hrekov
State: closed
Created: 2021-03-15 16:12:25

I have the following code:
```
from datetime import datetime, date

from peewee import *

db = SqliteDatabase(':memory:')


class Car(Model):
    model = CharField(max_length=50, unique=True)
    manufactured = DateField()

    class Meta:
        database = db

    def __str__(self):
        return f'Car: {self.model}'


class Person(Model):
    name = CharField(max_length=100)
    main_car = ForeignKeyField(Car)
    helper_car = ForeignKeyField(Car, on_delete='SET NULL')

    class Meta:
        database = db

    def __str__(self):
        return f'Person: {self.name}'


db.create_tables([Car, Person])

Person.insert(
    name="test",
    main_car=Car.create(model='Mitsubishi', manufactured=datetime.today()),
    helper_car=Car.create(model='BMW', manufactured=date(2001, 12, 12))
).execute()

MainCar = Car.alias()
HelperCar = Car.alias()

print(tuple(
    Person.
        select(Person, MainCar, HelperCar)
        .join(MainCar, on=Person.main_car)
        .switch(Person)
        .join(HelperCar, on=Person.helper_car)
    )
)
```
but when i'm trying to execute this i'm getting an empty tuple in results.
So i tried to execute generated SQL query from that and everything is OK and data returned
```
sqlite> SELECT "t1"."id", "t1"."name", "t1"."main_car_id", "t1"."helper_car_id", "t2"."id", "car" AS "t3", "t2"."manufactured", "t4"."id", "car" AS "t3", "t4"."manufactured" FROM "person" AS "t1" INNER JOIN "car" AS "t2" ON ("t1"."main_car_id" = "t2"."id") INNER JOIN "car" AS "t4" ON ("t1"."helper_car_id" = "t4"."id");
1|test|1|2|1|car|2021-03-15|2|car|2001-12-12
sqlite> 
```

What am i doing wrong?

The problem is a bad manifestation of a bug that occurs when you have a field named "model". I probably need to make Peewee raise an error if this occurs, but this fixes things:

```python

class Car(Model):
    model_ = CharField(column_name='model', max_length=50, unique=True)
    ...
```

Thanks. That's worked
Issue: [bug]or where query got a bug not expect sql where
Author: mouday
State: closed
Created: 2021-03-10 07:31:35


query:
```python
LibProductDataModel
   .select()
   .where(LibProductDataModel.product == product_or_company
          | LibProductDataModel.company == product_or_company)
   .first())
```

got:
```sql
('SELECT `t1`.`id`, `t1`.`product`, `t1`.`company`, `t1`.`ticket`, `t1`.`update_time` 
FROM `lib_product_data` AS `t1` 
WHERE ((%s OR `t1`.`company`) = %s) 
LIMIT %s', 
['腾讯', '腾讯', 1])
```

expect:
```sql
('SELECT `t1`.`id`, `t1`.`product`, `t1`.`company`, `t1`.`ticket`, `t1`.`update_time` 
FROM `lib_product_data` AS `t1` 
WHERE `t1`.`product` = %s OR `t1`.`company` = %s 
LIMIT %s', 
['腾讯', '腾讯', 1])
```

it's my logic not clean! not a bug. 

this code is ok.

```python
row = (LibProductDataModel
               .select()
               .where((LibProductDataModel.product == product_or_company)
                      | (LibProductDataModel.company == product_or_company))
               .first())

```
Issue: can i get how long one sql execute time
Author: mouday
State: closed
Created: 2021-03-10 02:35:56

I want to  known sql execute time, but logging not show.

just like :
```
10.12 ms SELECT * FROM `person` WHERE `name` = 'Tom' LIMIT 1
```

Peewee can log all queries. This is documented:

http://docs.peewee-orm.com/en/latest/peewee/database.html#logging-queries

If you want to know the sql only for a particular query you can use the `.sql()` method of the query object.

http://docs.peewee-orm.com/en/latest/peewee/api.html#BaseQuery.sql

> Peewee can log all queries. This is documented:
> 
> http://docs.peewee-orm.com/en/latest/peewee/database.html#logging-queries
> 
> If you want to know the sql only for a particular query you can use the `.sql()` method of the query object.
> 
> http://docs.peewee-orm.com/en/latest/peewee/api.html#BaseQuery.sql

I known ， but i want to see execute time and execute sql

like
```
time: 347.00 ms
```

You can try monkey-patching `Database.execute_sql()` or subclass your database implementation and override the `execute_sql()` method to add timing information.

> You can try monkey-patching `Database.execute_sql()` or subclass your database implementation and override the `execute_sql()` method to add timing information.

thank you, i impl code  as :
```python
# -*- coding: utf-8 -*-

import math
import time
from functools import wraps

from peewee import MySQLDatabase

import logging


logger = logging.getLogger('peewee')

def timer(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.time() * 1000
        ret = func(*args, **kwargs)
        end_time = time.time() * 1000
        logger.debug("time: %.2f ms" % math.ceil(end_time - start_time))
        return ret

    return wrapper


class CustomMySQLDatabase(MySQLDatabase):
    @timer
    def execute_sql(self, sql, params=None, commit=SENTINEL):
        return super().execute_sql(sql, params, commit)


```

Just a suggestion: you might try using something like this for the start/end timings:

https://docs.python.org/3.9/library/time.html#time.perf_counter

> Just a suggestion: you might try using something like this for the start/end timings:
> 
> https://docs.python.org/3.9/library/time.html#time.perf_counter

thank you!
Issue: insert_many() does not generate NULL values for fields with null=True
Author: rafal-jaworski
State: closed
Created: 2021-03-05 11:46:39

Snippet with error.
```python
import peewee

db = peewee.SqliteDatabase('people.db')


class Model(peewee.Model):
    name = peewee.CharField()
    x = peewee.IntegerField(null=True)

    class Meta:
        database = db


db.connect()
db.create_tables([Model])
query = Model.insert_many([{"name": "foo", "x": None}, {"name":"bar"}])
query.execute()
```
In a workaround i have to use callable, because of `default=None` is ommited:
```python
class Model(peewee.Model):
    name = peewee.CharField()
    x = peewee.IntegerField(null=True, default=lambda: None)

    class Meta:
        database = db
```



Thanks, this is fixed now.

@coleifer Thanks! Could you release a new version with this fix? :)

3.14.3

Hey Im still having this issue, and i dont know how to solve it. 

Works fine for me:

```python
class Reg(Model):
    key = TextField()
    value = IntegerField(null=True)
    class Meta:
        database = db

db.create_tables([Reg])

l1 = [{'key': 'k1', 'value': None}, {'key': 'k2'}, {'key': 'k3', 'value': 3}]
l2 = [{'key': 'k4'}, {'key': 'k5', 'value': None}, {'key': 'k6', 'value': 6}]

Reg.insert_many(l1, fields=['key', 'value']).execute()
Reg.insert_many(l2, fields=['key', 'value']).execute()

for r in Reg.select().order_by(Reg.key):
    print(r.key, r.value)
# k1 None
# k2 None
# k3 3
# k4 None
# k5 None
# k6 6
```

If you don't specify `fields` it will just infer the fields you want from the *first* dict in the list of rows to insert. Alternatively, you can of course provide row tuples.

http://docs.peewee-orm.com/en/latest/peewee/querying.html#bulk-inserts

You re rigth, the error was mine.
Issue: Does peewee create MySQL conn object every time a query execute?
Author: zhuyanxi
State: closed
Created: 2021-03-05 11:23:17

https://github.com/coleifer/peewee/blob/ac5126d774d89b07b3e8ccb871897e9a0bf53fde/peewee.py#L3972
I find that in the `MySQLDatabase` class , it will create MySQL conn object every time the function `_connect` executed:
```python
def _connect(self):
        if mysql is None:
            raise ImproperlyConfigured('MySQL driver not installed!')
        conn = mysql.connect(db=self.database, **self.connect_params)
        return conn
```

And now my situation is: I have a MySQL DB in AWS North America, and my frontend app is in China. After I had a test, I found that each time I send a request, there was 2 steps: first connect to the MySQL DB take about 350ms, and then second execute the sql command takes another 350ms, that make my app a little bit slow. 

So I think may it's better to hold a `conn` object in memory for a while and reuse it than to create a `conn` object every time.

And My question is:
Is there a way to make the `MySQLDatabase` reuse the `conn` object? Or should I need to extend the `MySQLDatabase` class myself?

If you had spent your time investigating rather than posting this issue, you would find:

http://docs.peewee-orm.com/en/latest/peewee/database.html#connection-management

If you want to reuse a connection, then you should use the connection pool, which is mentioned in the above document and also extensively documented:

* http://docs.peewee-orm.com/en/latest/peewee/database.html#connection-pooling
* http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool
Issue: Connection pool: How to close single connection from pool
Author: kalyanidz
State: closed
Created: 2021-03-04 11:12:13

Hello @coleifer 

First of all, thank you for developing this simple yet wonderful ORM app peewee. It made the database related coding so well. 

We are using postgres connection pool with peewee.
```
db = PooledPostgresqlExtDatabase(
            database=db_secret['name'],
            user=db_secret['uname'],
            password=db_secret['pwd'],
            host=db_secret['host'],
            port=db_secret['port'],
            max_connections=30, autocommit=True, autorollback=True)
```
Every time we are done with saving data to database, in finally of that method, we call 

```
db.close()
```
I have multiple questions/queries - 
1. Will this close the entire pool or just a single connection?
2. In the [documentation](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool) it is mentioned that, 

> For web applications, this typically means that at the beginning of a request, you will open a connection, and when you return a response, you will close the connection.

We are using FastAPI and uvicorn for making it multi-threaded. Considering the above like, do we need to explicitly close the connection?

Peewee uses a connection-per-thread, so you get:

* connect() -> get conn from pool, or if none free create new one
* use connection for a while
* close() -> return connection back to pool

Calling `db.close()` will close the connection you've been using in the current thread. You use the exact same API when you're using the connection pool as you would when using a non-pooled peewee database implementation.

I'm not very familiar with fastapi/uvicorn, but for web applications you will use the same pattern I described above:

* request comes in, thread starts handling request
* open db connection (db.connect())
* do work
* return response
* close db connection (db.close())

Most web frameworks have some hooks for before/after request. There is a section on the peewee docs about how to do this with FastAPI:

http://docs.peewee-orm.com/en/latest/peewee/database.html#fastapi

Thank you so much @coleifer 

@coleifer 
One more query, even if we are passing database instance in peewee basemodel class, do we still need to call db.connect() before the request?

You should always call `connect()` at the beginning of the connection lifecycle and `close()` at the end. Why? Read here:

http://docs.peewee-orm.com/en/latest/peewee/database.html#using-autoconnect

Please read the documentation.

> You should always call `connect()` at the beginning of the connection lifecycle and `close()` at the end. Why? Read here:
> 
> http://docs.peewee-orm.com/en/latest/peewee/database.html#using-autoconnect
> 
> Please read the documentation.

Sure. Thanks again.
Issue: Simpler upsert: `updates` on get_or_create
Author: RickMeasham
State: closed
Created: 2021-03-01 02:19:24

_I acknowledge that this functionality may already exist in a similarly simple form and I just haven't found it_

At the moment if I call `get_or_create` the `defaults` clause is only used when we create the object:

```
model, created = Model.get_or_create(
   **primary_details,
   defaults=**secondary_details
)
```

In the case of an "upsert" we don't just want to use those details on a create. We want to update an existing row that matches `primary_details`.

Right now, I have this:

```
model, created = Model.get_or_create(
   **primary_details,
   defaults=**secondary_details
)

if not created:
   Model.update(**secondary_details).where(Model.id == model).execute()
```

I know postgres has the `on_conflict` extensions but they're both cumbersome and non-transportable. Therefore I would llike to propose an addition to `get_or_create` whereby a new parameter (Eg `updates`) has the same effect on both an existing and new object that `defaults` does on a new object.

This would result in the following syntax for users needing an upsert. And as it's handled in python, it would be transportable:

```
model, created = Model.get_or_create(
   **primary_details,
   updates=**secondary_details
)
```

For this you should probably be explicit about using the database-specific UPSERT. Because this functionality is not necessarily available across all databases, nor does is it implemented in the same way, Peewee doesn't offer a "shortcut" which may mask inefficiencies or gotcha's in the underlying implementation.

Check the docs for details on how to issue upsert queries: http://docs.peewee-orm.com/en/latest/peewee/querying.html#upsert
Issue: [Q] Do I have to add indexes to peewee if I have manually added them to the table?
Author: NightMachinery
State: closed
Created: 2021-02-28 14:44:46

I have manually added some indexes using Datagrip to my peewee database. Do I have to add these indexes to the peewee model (i.e., add some peewee Python code), or do things just work?

The only reason you would need to add them to the Peewee model is if you want Peewee to create them whenever you run `database.create_tables([...])` or `MyModel.create_table()`.
Issue: Encoding issue on some weird fields
Author: joveice
State: closed
Created: 2021-02-27 00:02:38

Hey, I have some issues trying to read some weird fields. I get this encoding exception:
```
Traceback (most recent call last):
  File "D:/Projects/autocat-companion/app.py", line 275, in <module>
    main()
  File "D:/Projects/autocat-companion/app.py", line 173, in main
    for x in s:
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 6851, in __iter__
python-BaseException
    self.execute()
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 1898, in inner
    return method(self, database, *args, **kwargs)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 1969, in execute
    return self._execute(database)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 2141, in _execute
    cursor = database.execute(self)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 3142, in execute
    return self.execute_sql(sql, params, commit=commit)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 3136, in execute_sql
    self.commit()
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 2902, in __exit__
    reraise(new_type, new_type(exc_value, *exc_args), traceback)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "D:\Projects\autocat-companion\venv\lib\site-packages\peewee.py", line 3129, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: Could not decode to UTF-8 column 'RockerSettings' with text '╔T'
```

Now I got the same on sqlite3 and solved it by using [text factory](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.text_factory) as bytes
I tried to insert it on line 3403 in peewee.py and that seems to sort it out.

Is there any ways to do something like this without editing the library?
I'm trying to use it with this (which works great with the text_factory insert)
```
class LuabinField(BlobField):

    def db_value(self, value):
        return luabins.encode_luabins(value)

    def python_value(self, value):
        if value is not None:
            return luabins.decode_luabins(io.BytesIO(value))
```


```
db = SqliteDatabase(config.get('save_path'))
conn = db.connection()
conn.text_factory = bytes
```
This works, would this be a valid way?
Sorry for the dumb questions, but this is really new to me.

Sqlite uses UTF8 (or UTF16) for text. It sounds like you have a weird encoding. I don't know the provenance of this database, but if you're using peewee's `TextField` then this is expressly for storing UTF8-encoded unicode text. If you need to store arbitrary binary data you should use the `BlobField`.
Issue: Schema keyword and SQLite
Author: LVerneyPEReN
State: closed
Created: 2021-02-26 11:28:00

Hi,

I am using the `schema` keyword in my models to store the resulting tables in different PostgreSQL schemas. Typical example is:

```python
database_proxy = peewee.DatabaseProxy()


class Stock(peewee.Model):
    class Meta:
        database = database_proxy
        schema = 'financial-data'

    symbol = peewee.CharField(max_length=10, primary_key=True)
    name = peewee.CharField(max_length=255)
```

I also need this code to support SQLite (in a degraded way) to help quick development / deployment. Currently, if ran against a `database_proxy.initialize('sqlite:///data.db')`, I get the following `OperationalError`:

```
peewee.OperationalError: unknown database "financial-data"
```

Is it expected that the `schema` keyword break in this way with an SQLite database? Is it possible to have a dynamic `schema` keyword to remove it when using an SQLite backend?

Thanks!

A lot of people like to use SQLite as a peer for PostgreSQL for local development.
However, I think this approach is not always guaranteed to work, and as much as I love SQLite, PostgreSQL has tons of other features to offer.
So, my question is: if your production code is running with the latter, why are you not simply developing against it? It's easy enough to start a docker container with PostgreSQL locally even on non Linux machines (Mac and Windows ...).

This kind of namespacing works differently in sqlite than postgres. To utilize the additional layers of nampespaces in sqlite, you need to use separate database files and then load them into your main database via "attach". When you attach a database, you specify it's nampespace, which corresponds to the "schema" setting on the model metadata.

http://docs.peewee-orm.com/en/latest/peewee/api.html#SqliteDatabase.attach

You can make a custom metadata class implementation that has a dynamic schema. This should give you an idea of how to go about it:

http://docs.peewee-orm.com/en/latest/peewee/database.html#thread-safety-and-multiple-databases

Hi @coleifer, thanks for the details, this is perfect for my use case!