akumoth/peewee-issues huggingface dataset:
Issue: BlobField not converted to bytes on Python 3.3+ and Postgresql
Author: PatTheMav
State: closed
Created: 2019-10-29 23:55:04

Due to psycopg2 converting Postgres' `bytea` data to `memoryview` objects starting with Python 3.3, models with `BlobField`s require special care:

Data needs to be converted to bytes via `tobytes()` to achieve expected behavior.

Right now Peewee's core implementation doesn't seem to take this into account (the tests seem to check for this and apply `tobytes` themselves), which would require a project to either implement a Postgres-only conversion step _or_ create a custom field that handles the conversion, à la:

```
class Py3Blob(BlobField):
    def python_value(self, value):
        if isinstance(value, memoryview):
            return value.tobytes()
        else:
            return super().python_value(value)
```

IIRC the motivation behind using `memoryview` was to use a more memory-efficient data type for actually big blobs (e.g. image data) that wouldn't require copying that data around, which is a valid concern and might make automatically converting this data to bytes a no-go.

Is there a chance for Peewee nevertheless converting to bytes itself to align Py3 behavior for  `BlobField`s with Py2, especially as the Model wouldn't return different types based on backend _and_ Python version or maybe a "watch out for this, fix with custom field" warning in the docs?

A `memoryview` is just a pointer to a buffer of bytes. It can be indexed, sliced, etc without incurring an allocation and copy. This is a very important optimization to retain when the database-driver supports it (eg psycopg2). Your example field is exactly how I would work-around this if your code depends on having a `bytes` object instead.

For what it's worth, most APIs support using `memoryview` just as they would `bytes` or `bytearray`...you can write them to a file, etc., so if you are running into some kind of compatibility issue you might consider addressing it upstream.

> For what it's worth, most APIs support using memoryview just as they would bytes or bytearray...you can write them to a file, etc., so if you are running into some kind of compatibility issue you might consider addressing it upstream.
 
Yeah fair enough - hopefully the next person switching from Py2 to Py3 will find this issue and understands how to solve it without having to change lots of application logic.. 😁

Many python 2.x drivers return `buffer` type. Python 2.7 also has memoryview and bytearray, so it really just depends on the driver you're using.
Issue: ModuleNotFoundError: No module named 'peewee'
Author: 52lemon
State: closed
Created: 2019-10-21 02:51:28

i install peewee with pip3, and i got the error below
`
(logpress-tornado) F:\Projects\logpress-tornado>python3 manager.py --cmd=syncdb
Traceback (most recent call last):
  File "manager.py", line 17, in <module>
    from core import jinja_environment, smtp_server
  File "F:\Projects\logpress-tornado\core.py", line 7, in <module>
    from lib.database import Database
  File "F:\Projects\logpress-tornado\lib\database.py", line 8, in <module>
    import peewee
ModuleNotFoundError: No module named 'peewee'
`

You must have some problem in your imports. This is definitely not the appropriate place for questions like this. Try stackoverflow.
Issue: Query.returning() wrong alias implementation 
Author: dmkulazhenko
State: closed
Created: 2019-10-20 20:00:09

Hi there!

I tried to use peewee as SQLGenerator (yeah, I know that it's very strange and ... know, I just have a reason of doing this shit :trollface:)

That's OK:
```
c = User.insert(username='Petr').returning(User.username)
c.sql()
[stdout]: ('INSERT INTO "user" ("username", "password", "last_activity") VALUES (%s, %s, %s) RETURNING "user"."username"', ['Petr', 'password', 1571594114])
```
But if I will try to use alias in returning section:
```
c = User.insert(username='Petr').returning(User.username.alias('u'))
c.sql()
[stdout]: ('INSERT INTO "user" ("username", "password", "last_activity") VALUES (%s, %s, %s) RETURNING "u"', ['Petr', 'password', 1571594114])
```
That isn't OK, because I want something like:
```
('INSERT INTO "user" ("username", "password", "last_activity") VALUES (%s, %s, %s) RETURNING "user"."username" AS "u"', ['Petr', 'password', 1571594114])
```

idk, if that problem reproduces with other queries (not insert)...

Maybe, I just cannot find method, that do this staff, btw I can't find anything about this [here](http://docs.peewee-orm.com/en/latest/peewee/querying.html#returning-clause).

peewee==3.11.2
psycopg2-binary==2.8.4
(PostgreSQL)




@coleifer, any reply?)

Fixed in c2dc7ca7caaafc0840b771de6e5dfc5cbabe83a0

@coleifer, thx, have a nice day!
Issue: BinaryJSONField unescaped JSON text
Author: DucChau
State: closed
Created: 2019-10-19 22:34:11

Hello,

I'm using `PostgresqlExtDatabase` for the `JSONBField` types.

I currently have a JSONB text field in a Postgres table. When I set the field type to `TextField` in the PeeWee model, saving data in the table looks like valid JSON. Example of JSON value I'm saving to the table below.

``` {"query": "select \n    count(*) as count \nfrom\n    foo\n", "value": [21950367]}```

However when I switch the field type out to `BinaryJSONField` all values in the field becomes escaped. This makes it a pain to use the JSON parsing functions in Postgres.

```"{\"query\": \"select \\n    count(*) as count \\nfrom\\n  foo\\n\", \"value\": [21950367]}"```

Is there a way to not escape values going into the `BinaryJSONField` field when saving via a model in peewee?



I'm not clear on why you used a text-field in the first place or whether that is in any way related to the problem. Or, for that matter, what the underlying actual column type is on your table? The peewee `playhouse.postgres_ext.BinaryJSONField` will store and retrieve data as JSON. So you can store `dict`, `list`, and string/scalar values.

The docs should give you clear examples on usage:

* http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#JSONField
* http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#BinaryJSONField
Issue: Usage example with FastAPI framework
Author: aniljaiswal
State: closed
Created: 2019-10-17 10:39:04

Hi,

I'm trying to integrate peewee with FastAPI and would be using it with the Serverless lambda framework. Can you provide me an example of how I would structure and use the database in terms of Models? I need to make sure that in a single lambda invocation I'm opening the DB connection, it's closed when the response is sent.

FastAPI documentation only contains SQLAlchemy related things and I love peewee since it's small, efficient and expressive. Any pointers on code structure would be helpful. It would be great if you could share a sample app with MySQL which I can start off with.



http://docs.peewee-orm.com/en/latest/peewee/contributing.html#questions
Issue: sqlite3.Warning: You can only execute one statement at a time.
Author: luminousmen
State: closed
Created: 2019-10-14 10:41:37

Hi, I'm trying to execute some SQL scripts on the sqlite database. The SQL scripts example:
```sql
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (
    id integer,
    name VARCHAR(100)
);
INSERT INTO t1(id, name) VALUES
    (1, 'test');

```
And the code:

```python
from peewee import SqliteDatabase

database = SqliteDatabase(":memory:")

for template in templates:
    query = template.format(**args)
    database.execute_sql(query)
```
It all crashes on `execute_sql` call because internally the code calls the `execute` method of sqlite and it expects just one particular query.

How to fix that? The same code works with postgres

pysqlite does not like it. Just break it up into multiple statements or use the connection directly and call the `executescript()` method.

https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.executescript
Issue: Peewee returns trigger information, but psycopg2 does not
Author: phuvp
State: closed
Created: 2019-10-13 20:33:12

Hi, I have created a few triggers and am now trying to print their information out. When I run the below, the psyco method errors out since `result=None`, whereas the peewee method returns a list of trigger info shown below. I've already checked that the cursor for psycopg2 is in the right database. I'd like to know what causes this discrepancy.

Output:
`
[('test_db', 'public', 'executortrigger', 'INSERT', 'test_db', 'public', 'executortable', 1, None, 'EXECUTE PROCEDURE test_executor()', 'ROW', 'AFTER', None, None, None, None, None)]
`

`def list_triggers_peewee():
    return [i for i in POSTGRES.execute_sql('select * from information_schema.triggers;')]
`
`
def list_triggers_2_psyco():
    result = psyco_cursor.execute('select * from information_schema.triggers;')
    return [i for i in result.fetchall()]
`

version is peewee 3.10.0

This is not a peewee issue. This is a psycopg2 question. Please use the appropriate forum going forward.

Psycopg2 cursor methods do not return a result. So:

```
cursor = conn.cursor()
cursor.execute('select ...')
for row in cursor.fetchall():
    ...
```
Issue: [Request] field type migrations for playhouse.migrate PostgresqlMigrator
Author: mountainash
State: closed
Created: 2019-10-12 14:21:21

Can a function be added to PostgresqlMigrator to allow for field types to be changed and logged in a migration? (eg smallint to biginit)

Thank you.

Just found this as a possible workaround, but it's a fair bit of "work" :-)

https://teamtreehouse.com/community/changing-a-peewee-charfield-to-a-textfield

This is a pretty decent request, let me think about what it would entail.

Dang - that was fast. Nice work!

Thx, I appreciate the suggestion.
Issue: read from slave, write to master
Author: evstratbg
State: closed
Created: 2019-10-11 11:32:34

Hi! I'm using peewee 3.11.2

We have master-slave postgres instances, slave - for reading, master - for writing

https://github.com/coleifer/peewee/blob/fc842eaf54a5d047f9281cf1866f81ce93a98496/docs/peewee/changes.rst
I've found out, that there is no longer `read_slave` extension (link above)

Any possibilities to read from slave and write to master in peewee > 3.0?

You'll have to implement it yourself, which can actually be pretty easy.

You probably want to have two database instances (or two pools):

```python

master = PooledPostgresqlDatabase('master')
db = PooledPostgresqlDatabase('slave')
```

Then you can use the `bind_ctx()` context manager to bind your model(s) to the master when you need to do some writes:

```python

# Grab connection to master db from pool, and return it after write.
with master.connection_context():
    # Instruct user model (and related models) to use master db for write.
    with master.bind_ctx([User]):
        User.create(username='huey')
```

You can abstract this away in your own context manager:

```python

@contextmanager
def use_master(models):
    with master.connection_context():
        with master.bind_ctx(models):
            yield
```

Docs: 

* http://docs.peewee-orm.com/en/latest/peewee/api.html#Database.bind_ctx
* http://docs.peewee-orm.com/en/latest/peewee/api.html#Database.connection_context


@coleifer It looks like transaction is not maintained while using `master.bind_ctx(models)`. Can you provide some example where I can maintain transaction while being able to bind the model to specific database?

Transactions are specific to a database connection, so there's literally no way to carry the transaction across to another connection...
Issue: PostgresqlExtDatabase with RealDictCursor causes KeyError on every select
Author: schulzsebastian
State: closed
Created: 2019-10-10 09:33:41

Peewee version: 3.11.2
Postgresql version: 11
Db init:
```python
from playhouse.postgres_ext import PostgresqlExtDatabase
from psycopg2.extras import RealDictCursor

database = PostgresqlExtDatabase(
    None,
    register_hstore=False,
    autorollback = True,
    cursor_factory=RealDictCursor
)
```
Setting RealDictCursor cursor on PostgresqlExtDatabase database causes KeyError 0 on every select e.g.:
```
project-python |   File "/project/app/db/general.py", line 42, in get_all
project-python |     for row in cls.select(*fields).dicts():
project-python |   File "/root/.local/share/virtualenvs/project-nBADCc5g/lib/python3.6/site-packages/peewee.py", line 4224, in next
project-python |     self.cursor_wrapper.iterate()
project-python |   File "/root/.local/share/virtualenvs/project-nBADCc5g/lib/python3.6/site-packages/peewee.py", line 4143, in iterate
project-python |     result = self.process_row(row)
project-python |   File "/root/.local/share/virtualenvs/project-nBADCc5g/lib/python3.6/site-packages/peewee.py", line 7217, in process_row
project-python |     result[attr] = converters[i](row[i])
project-python | KeyError: 0
```
Moreover, any use of .create() method returns None without any warnings. I didn't find any information in the documentation or similar error, so I report it as a issue.

My current workaround is to use RealDictCursor only in context manager with raw SQL query, e.g.:
```python
with database.connection().cursor(cursor_factory=RealDictCursor) as cursor:
    cursor.execute("select * from table")
    print(cursor.fetchall())
```

Yeah, don't use that. If you want dicts, then use the `.dicts()` method. Peewee expects cursors to yield row tuples.

```python

q = MyModel.select().dicts()
for row_dict in q:
    # ...
```
Issue: JSONFields can't be updated using Model.bulk_update if they're top-level lists
Author: KarolTrojanowskiDL
State: closed
Created: 2019-10-09 12:23:18

DISCLAIMER: I am terribly sorry I didn't read that I should post an issue first. There's already a PR I made for this: #2033 

I noticed that using `Model.bulk_update` won't work with `JSONField`s that are top-level lists because by default `Value` will unpack the list into multiple values. Top-level lists are valid JSON objects.

Example:

```python
class Item(Model):
    json = JSONField()

item = Item.create(json=['a','b'])
item.json = ['c','d']

Item.bulk_update([item], fields=[Item.json])
```
will raise `OperationalError` because of a malformed update query.

Fixed by above commit. Thanks for the report!
Issue: Add ability to update JSON fields that are lists in Model.bulk_update
Author: KarolTrojanowskiDL
State: closed
Created: 2019-10-09 11:57:22

I noticed that using `Model.bulk_update` won't work with `JSONField`s that are top-level lists because by default `Value` will unpack the list into multiple values. Top-level lists are valid JSON objects.

Example (before):
```python
class Item(Model):
    json = JSONField()

item = Item.create(json=['a','b'])
item.json = ['c','d']

Item.bulk_update([item], fields=[Item.json])
```
will raise `OperationalError` because of a malformed update query.

Example (after):
```python
class Item(Model):
    json = JSONField()

item = Item.create(json=['a','b'])
item.json = ['c','d']

Item.bulk_update([item], fields=[Item.json], unpack_values=False)
```
will update the row fine.

This applies to both SQLite and PostgreSQL `JSONField`s.

I've patched this in a slightly different way. Am hoping to extend the code elsewhere to use the new `to_value()` interface (as opposed to manually creating Value instances w/the appropriate converter). Thx for the patch, at any rate.
Issue: Forced default value
Author: BukinPK
State: closed
Created: 2019-10-07 17:17:22

I create a model with a fild:
```
class Map(Model):
    size = IntegerField()
```
It is NOT NULL and don't have DEFAULT value by default:
```
pw.IntegerField(
    null=False,
    default=None,
    ...
)
```
But when I create new object, with no "size" argument put in automatically generated constructor, a model is created with a value of 0 in the size field, instead of throwing an exception.
```
Map.create()
/home/bukinpk/.local/lib/python3.7/site-packages/pymysql/cursors.py:170: Warning: (1364, "Field 'size' doesn't have a default value")
  result = self._query(query)
Out[12]: <Map: 5>
```
How can I deactivate this implicit behavior?


I think something is missing in your example, because this code correctly prints "None":

```python

from peewee import *


db = SqliteDatabase(':memory:')

class M(Model):
    value = IntegerField(null=True)
    class Meta:
        database = db

db.create_tables([M])

m = M.create()
m_db = M[M.id]

print(m_db.value)
# output: None
```

I see this.
But this code correctly throws error and incorrectly prints "0":

```python
from peewee import *


db = MySQLDatabase(DB_NAME, user=DB_USER, password=DB_PASSWORD)

class M(Model):
    value = IntegerField(default=None, null=False)
    class Meta:
        database = db

db.create_tables([M])

m = M.create()
m_db = M[M.id]

print(m_db.value)

# output:
# /home/bukinpk/.local/lib/python3.7/site-packages/pymysql/cursors.py:170: Warning: (1364, "Field 'value' doesn't have a default value")
#   result = self._query(query)
# 0

```

On use `SqliteDatabase(':memory:')` such problems I don't observe.

Well shyea...you specified your field with null=false, what do you expect?

I expect throwing an error on attempt to set value to NULL, same as on use SqliteDatabase(':memory:').

Looks like this has to do with the sql_mode setting. If you change it to strict_trans_tables or strict_all_tables it should take care of the problem.

Yes, it works! Thanks for the help!

UPD.:
But it will broke some string methods, such as:
https://github.com/coleifer/peewee/blob/83873b70bcdfb82320f5d3a127e4783c04531b3a/peewee.py#L4563
Becouse you use `||` as concatination operator.
And if I change `sql_mode` field, to `STRICT_TRANS_TABLES`, i will turn off actualy setted mode:
https://github.com/coleifer/peewee/blob/83873b70bcdfb82320f5d3a127e4783c04531b3a/peewee.py#L3838
And operator `||` will not work correctly.
I think, I should set `PIPES_AS_CONCAT,STRICT_TRANS_TABLES`
Or to append `PIPES_AS_CONCAT` mode to all default setted values:
`STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION`.
Issue: Use the pythonic way to filter with list comprehensions or generator expressions
Author: eltonvs
State: closed
Created: 2019-10-03 04:32:34

This is a really small refactor in order to use the pythonic way to filter in python.

References:
- https://www.oreilly.com/library/view/python-cookbook/0596001673/ch01s11.html
- https://www.artima.com/weblogs/viewpost.jsp?thread=98196

Gonna pass, thanks though.
Issue: OpenAPI Specification / SwaggerUI for REST API
Author: brandond
State: closed
Created: 2019-09-30 20:00:07

I'd love to have an autogenerated OpenAPI spec with SwaggerUI support that was easy to use with `flask_peewee.rest`. Someone hacked this together as a standalone module but the project appears to be dead:
https://github.com/hapyak/flask-peewee-swagger

I recently hacked together a translation layer to expose PynamoDB models via [Flask-RESTPlus](https://flask-restplus.readthedocs.io), so I considered taking a shot at that for Peewee, but I'd much rather use your built-in REST layer if at all possible.

That's a no then?

> flask_peewee

wrong repo

Right, sorry. I mistakenly assumed it was a submodule like the playhouse stuff. I just saw the message that you're not working on it any more. Did you ever get around to breaking out the REST bits into a separate library?
Issue: How best to get inserted row count from Model.insert_many()?
Author: grantcox
State: closed
Created: 2019-09-27 16:40:45

I would expect the `Model.insert_many().execute()` return value to the `rowcount` affected, however it appears to always be `0` (presumably it's attempting to get the primary row id inserted?).

I have found that the following approach works, hacking on how the `returning` implementation works.  But it doesn't look like the correct way:
```
multi_insert_query = MyModel.insert_many(rows).on_conflict(action='IGNORE')
multi_insert_query._return_cursor = True
cursor = multi_insert_query.execute()
total_inserted = cursor.rowcount
```

Any suggestions for how this is better implemented?  I am using MySQL 5.6, although I'm not sure that matters.

I hear you -- thanks for mentioning the database you're using. This should work with sqlite as well, though I don't believe it will work for postgres -- but I need to test it out. Let me look into this as I think it's a worthwhile request.

ab43376

Just a heads-up that this change has been rolled-back in #2556. Going forwards, to obtain the inserted row count, you may use the `Insert.as_rowcount()` chainable query method. Examples:

```python
db = MySQLDatabase(...)

query = User.insert_many([...])
# Old behavior:
#rowcount = query.execute()

# Behavior going forwards:
lastid = query.execute()

# To get the rowcount instead of the last id:
rowcount = query.as_rowcount().execute()
```
Issue: querying jsonfields
Author: Nocturem
State: closed
Created: 2019-09-24 15:15:44

Hi there,


Having a little trouble figuring out the children or json_each function in peewee

trying to replicate the basic example 
`SELECT DISTINCT user.name
  FROM user, json_each(user.phone)
 WHERE json_each.value LIKE '704-%';`

however I can't see an example in the documentation 

tried mocking up my own effort 
```
phone = User.phone.children().alias('phone_children')
query = User.select().from_(User, phone).where(phone.c.value ** '704%')
```

however had some issues trying to execute this getting TypeError: unhashable type: 'Alias'
I am fairly certain I am going about this backwards :( 

Assuming you're using sqlite, right?

yes 

This may be related to a change in 3.11.0. If you're using 3.11.0 or newer, can you try using 3.10 and letting me know if it works there?

currently on Version: 3.10.0
can try 3.11.0 if that may help

I've just added a test for the `children()` method and it appears passing locally:

ddf4ce32c5ca180b2f93d01f731aaf161be4a2da

Does that help at all?

Full code for the new test-case:

```python
    def test_children(self):
        children = KeyData.data.children().alias('children')
        query = (KeyData
                 .select(KeyData.key, children.c.fullkey.alias('fullkey'))
                 .from_(KeyData, children)
                 .where(~children.c.fullkey.contains('k'))
                 .order_by(KeyData.id, SQL('fullkey')))
        accum = [(row.key, row.fullkey) for row in query]
        self.assertEqual(accum, [
            ('a', '$.x1'),
            ('b', '$.x2'),
            ('d', '$.x1'),
            ('e', '$.l1'), ('e', '$.l2')])
```

Maybe try updating to 3.11.1 if you're still experiencing issues? I think I had the versions backwards -- that is, 3.10 had the issue which should be resolved in 3.11.

no worries I will try upgrading now and see if that resolves the issue :) 
thank you 

Oh shit, that test *is* failing on python 3... damn I was accidentally testing on python 2. I need to push a fix for this.

tested and working 
additionally may I suggest the addition of 
```
  def __enter__(self):
        return self

    def __exit__(self, *args):
        return None
```

on the Alias class 
this allows the convenience of using with statements for context handling 

thus 
```
with User.phone.children().alias('phone_children') as phone:
    for row in User.select().from_(User, phone).where(phone.c.value ** '704%'):
        ...
```

I imagine this would probably be handy on the CTE class as well.. 

Glad it's fixed. I tagged 3.11.2 release w/the fix.

I'm not a fan of implementing a context manager that does ... nothing. That is just a slower way of doing a simple assignment.

it might allow the garbage collector to recognize the transience of the assignment and toss the disposable variable after as it drops out of scope but mostly its an aesthetic/familiarity case of working similar to the SQL WITH clause where its used for virtual tables and common table expressions.
its not a major thing more a nice to have