akumoth/peewee-issues huggingface dataset:
Issue: When my program running in ubuntu 16.04, something error what is show.
Author: 0xn0ne
State: closed
Created: 2018-07-03 06:22:44

This is error detail
```
web@VM-35-97-ubuntu:~/smallprogram/smallprogram-backend$ sudo python3.6 app.py 
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
Traceback (most recent call last):
  File "app.py", line 9, in <module>
    from view import routes
  File "/home/web/smallprogram/smallprogram-backend/view/__init__.py", line 180, in <module>
    from model.user import User
  File "/home/web/smallprogram/smallprogram-backend/model/user.py", line 17, in <module>
    class User(BaseModel):
  File "/usr/local/lib/python3.6/site-packages/peewee.py", line 5044, in __new__
    cls._meta.set_primary_key(pk_name, pk)
  File "/usr/local/lib/python3.6/site-packages/peewee.py", line 4904, in set_primary_key
    self.add_field(name, field)
  File "/usr/local/lib/python3.6/site-packages/peewee.py", line 4846, in add_field
    field.bind(self.model, field_name, set_attribute)
  File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3864, in bind
    self._constructor = model._meta.database.get_binary_type()
  File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2959, in get_binary_type
    return sqlite3.Binary
AttributeError: 'NoneType' object has no attribute 'Binary'
```

And the database I use sqlite3, What's the proble?



Does your basemodel class have a database assigned?

How did you install python? Is it possible that you don't have the sqlite driver installed? You can test:

```pycon
>>> import sqlite3
>>> sqlite3
```

Closing due to unresponsiveness.
Issue: Error when attaching Sqlite db using APSW
Author: astrilchuk
State: closed
Created: 2018-07-02 17:05:34

I'm using APSW database driver(?) with peewee 3.5.1 and Sqlite and I am attempting to attach an in-memory database with the attach command:

db = apsw_ext.APSWDatabase("storage.db", pragmas=dict(foreign_keys=1, journal_mode="wal", cache_size=-1000 * 64))
...
db.attach(":memory:", "cache")

I'm getting an error on line 2812 of peewee.py stating: "'apsw.Connection' object has no attribute 'execute'".

Looking at the apsw docs, it appears that their connection object does not have have an execute() shortcut method.

If there's any more information I can provide please let me know.

Thanks


Thanks, fixed: 3475cbe0e092d9cea7f527b3a6a380b9bf714b86
Issue: Incorrect syntax for index creation in Sqlite when using schema
Author: astrilchuk
State: closed
Created: 2018-07-02 15:49:14

Creating an index on a table with a schema specified (other than the default) produces the following sql for a Sqlite database:

CREATE UNIQUE INDEX IF NOT EXISTS "programhash_program_id" ON "cache"."programhash" ("program_id")

The Sqlite definition for index creation (https://www.sqlite.org/lang_createindex.html) states it should appear as:

CREATE UNIQUE INDEX IF NOT EXISTS "cache"."programhash_program_id" ON "programhash" ("program_id")

Thanks

Nice catch, thanks for reporting.
Issue: peewee.OperationalError: (2013, 'Lost connection to MySQL server during query')
Author: cookieY
State: closed
Created: 2018-07-02 09:44:57

I find that if mysql wait_timeout  value is set too low peewee will lost connection .

so peewee can't automatically release the connection ?

how do I deal with this problem?

thanks

You probably want to more closely monitor your connection setup/teardown. These docs may be relevant: http://docs.peewee-orm.com/en/latest/peewee/database.html#error-2006-mysql-server-has-gone-away

I've not seen that particular error. Probably best off trying to ask on StackOverflow and posting some more information about what exactly your application is doing when things break.

Hi.
I have same problem after a while that my script is running. does it solve the issue that use `db.connect()` before **every** query and `db.close()` after it? or not? and is this efficient?

> and is this efficient?

No.
Issue: Prefetch same model error
Author: 1xinghuan
State: closed
Created: 2018-07-01 13:07:47

I have some tables like these:

```
class File(ModelBase):
    name = CharField(column_name='name', null=False)

class Shot(ModelBase):
    name = CharField(column_name='name', null=False)
    file= ForeignKeyField(column_name='file_id', model=File, field='id')

class Task(ModelBase):
    name = CharField(column_name='name', null=False)
    file= ForeignKeyField(column_name='file_id', model=File, field='id')
    shot = ForeignKeyField(column_name='shot_id', model=Shot, field='id')

class Version(ModelBase):
    name = CharField(column_name='name', null=False)
    task = ForeignKeyField(column_name='task_id', model=Task, field='id')
```

When I try prefetch like this:

```
query = Version.select().limit(50)
prefetch(query,
         Task, File, (Shot, Task), File
         )
```

There will be an error:

```
peewee.ProgrammingError: missing FROM-clause entry for table "t2"
LINE 1: ...hots" AS "t1" WHERE ("t1"."file_id" IN (SELECT "t2"."id" ...
```

If I use prefetch twice:

```
query = Version.select().limit(50)
prefetch(query, Task, File)
prefetch(query, Task, Shot, File)

for q in query:
    print q.task.name
    print q.task.file.name
```

There are still lots of query being executed, it seems like the first prefetch doesn't work.

I don't think you have any idea how prefetch is intended to be used... You don't need prefetch at all and you can do this all in a single query:

```python
TaskFile = File.alias()  # File reference for Task.file object.
versions = (Version
            .select(Version, Task, Shot, File, TaskFile)
            .join(Task)
            .join(Shot)
            .join(File)
            .join_from(Task, TaskFile))

for version in versions:
    print(version.name)
    task = version.task
    print('  ', task.name)
    print('  - shot=', task.shot.name, ' file=', task.shot.file.name)
    print('  - file
```
Issue: Support for Postgres 10 IDENTITY columns
Author: seamons
State: closed
Created: 2018-06-29 07:14:17

#Postgresql 10 has a new IDENTITY columns that replace SERIAL column:

> This is the SQL standard-conforming variant of PostgreSQL's serial
columns.  It fixes a few usability issues that serial columns have:
> - CREATE TABLE / LIKE copies default but refers to same sequence
> - cannot add/drop serialness with ALTER TABLE
> - dropping default does not drop sequence
> - need to grant separate privileges to sequence
> - other slight weirdnesses because serial is some kind of special macro

SQLAlchemy has partially supported for it: [PostgreSQL — SQLAlchemy 1.2 Documentation](http://docs.sqlalchemy.org/en/latest/dialects/postgresql.html#postgresql-10-identity-columns)

I have tweaked the Peewee field to use IDENTITY column as follows:
```py
class AutoField(peewee.IntegerField):
    """Auto Increment ID Field

    Suport PostgreSQL 10's IDENTITY column.
    """
    #sequence = None # Disable sequence for auto increment id
    #auto_increment = True
        
    def __init__(self, *args, **kwargs):
        kwargs['primary_key'] = True
        kwargs['unique'] = True
        kwargs['constraints'] = [peewee.SQL("GENERATED BY DEFAULT AS IDENTITY")]
        super().__init__(*args, **kwargs)
```

Do you have any plan to officially support this feature?

I slightly changed this:

```python
class IdentityField(AutoField):
    field_type = 'INT GENERATED BY DEFAULT AS IDENTITY'
```

Added support. By default Peewee will continue to use `AutoField` (which corresponds to `SERIAL`), but you now have the option of using `IdentityField` instead:

```python
class MyModel(Model):
    id = IdentityField()
    data = TextField()
```

Cool! Thanks for the rapid response
Issue: Allow arbitrary attributes on Fields
Author: conqp
State: closed
Created: 2018-06-28 08:19:11

I am maintaining an own [extension of peewee](https://gitlab.com/HOMEINFO/peeweeplus) I use in several web applications.
For JSON-serializable models I currently use their "visibility" (Attrbiute prefixed with an underscore) state to determine whether to serialize and deserialize them or not.
I'd now like to have more control over serialization and deserialization of these fields.
Hence I'd like to add properties, set by respective keyword arguments like `deserialize`, `serialize` and `cascade` (in case of FK fields).

I obviously would need to subclass all types of fields in order to achieve that.
This is possible, but would require me to change the respective field classes in a lot of my code.

So my questions is this: May it be possible to allow arbitrary custom kwargs which will set appropriate properties on the fields?

I'm thinking of something like this:

```
class Field(ColumnBase):
    …

    def __init__(self, null=False, index=False, unique=False, column_name=None,
                 default=None, primary_key=False, constraints=None,
                 sequence=None, collation=None, unindexed=False, choices=None,
                 help_text=None, verbose_name=None, db_column=None,
                 _hidden=False, **kwargs):
        …
        for key, value in kwargs.items():
            if hasattr(self, key):
                raise DontCompromiseFieldFunctionality()
            setattr(self, key, value)
```

If you think, that something like this may be reasonable to add, I'd  be happy to write and submit the respective patch.

While I understand how, with your use-case, you would want this functionality -- at present I am not interested in making such a change to how fields work. I think subclassing is going to be the way to go. You can create a mixin class and then apply it to fields, e.g.:

```python
class FieldMixin(object):
    def __init__(self, *args, **kwargs):
        if 'serialize' in kwargs:
            # do something?
        super(FieldMixin, self).__init__(*args, **kwargs)

class MyCharField(FieldMixin, CharField): pass
```

Alternatively, you could handle this at the model level or at some other level...that's probably what I would end up doing -- put it at a higher level.
Issue: get_or_create return value is wrong in documentation
Author: jouve
State: closed
Created: 2018-06-27 23:05:29

it documents `Model` instead of tuple of `Tuple[Model, bool]`.

```raw
 .. py:classmethod:: get_or_create(**kwargs)
         ...
        :returns: :py:class:`Model` instance.
```


Thanks, will fix.
Issue: Psycogreen and Peewee 
Author: mtairu
State: closed
Created: 2018-06-27 14:00:56

Following gevent, psycogreen and peewee, what is the difference btw:

```
import gevent
import gevent.monkey
gevent.monkey.patch_all()

import psycogreen.gevent
psycogreen.gevent.patch_psycopg()
```
this:
```
from gevent.socket import wait_read, wait_write
from psycopg2 import extensions

# Call this function after monkey-patching socket (etc).
def patch_psycopg2():
    extensions.set_wait_callback(_psycopg2_gevent_callback)

def _psycopg2_gevent_callback(conn, timeout=None):
    while True:
        state = conn.poll()
        if state == extensions.POLL_OK:
            break
        elif state == extensions.POLL_READ:
            wait_read(conn.fileno(), timeout=timeout)
        elif state == extensions.POLL_WRITE:
            wait_write(conn.fileno(), timeout=timeout)
        else:
            raise ValueError('poll() returned unexpected result')
```
Ref:

- https://bitbucket.org/dvarrazzo/psycogreen/src/master/tests/test_gevent.py
- http://docs.peewee-orm.com/en/latest/peewee/database.html#async-with-gevent

psycogreen is just settings a wait callback, so yeah it should be fine to use that instead.
Issue: insert_from with raw SQL query broke between Peewee 2 and 3
Author: timovwb
State: closed
Created: 2018-06-26 13:45:38

I've been using `insert_from` to migrate data from a non-Peewee table to a Peewee created table. It works as intended with Peewee 2, I'm moving to 3 but it hangs and keeps increasing memory usage.

Below is a minimal, full working example showing the issue.
```
from peewee import SqliteDatabase, Model, CharField, SQL

database = SqliteDatabase(":memory:")

class Foo(Model):
    foo = CharField()

    class Meta:
        database = database

database.create_tables([Foo])
database.execute_sql("CREATE TABLE Bar (bar TEXT);")
database.execute_sql("INSERT INTO Bar (bar) VALUES ('testvalue');")

cursor = database.execute_sql("SELECT * FROM Bar;")
print(cursor.fetchone())

# Works correctly with Peewee 2.10.2, hangs and memory leak with 3.5.0
Foo.insert_from(query=SQL("SELECT bar FROM Bar;"), fields=[Foo.foo]).execute()
print(list(Foo.select()))
```

It looks like it's entering endless recursion...will take a look.

Fixed, thanks for reporting.

Thanks for the fast fix!

Thanks for the quality bug report.
Issue: reflection got wrong mysql blob field type
Author: GrayMissing
State: closed
Created: 2018-06-26 10:15:38

using pwiz to reflect created mysql table, I found blob column got a TextField defintion, which can lead to codec error. I don't think it's a proper way to decode byte_array using utf-8.

I read the issue before #1034 , but django's TextField seems more like BlobField in peewee...

I'm confused by your issue. What exactly is the problem you are reporting? And if you read #1034, what are you confused about?

pwiz reflect mysql blob column to TextField, and TextField decode byte value using utf-8

Yeah, as it states in #1034, Peewee cannot differentiate blob from text due to a limitation in the introspection provided by pymysql. So you need to go through and change the TextFields to BlobFields.
Issue: Improve pwiz documentation with some examples
Author: willyhakim
State: closed
Created: 2018-06-22 20:13:14

Pwiz is a helluva an awesome tool. Improving the docs around it by giving some examples for postgres, mysql and sqlite would be super cool. 

Thank you for the suggestion, I'll look into it. Glad you find it helpful!

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz-a-model-generator

Now includes more examples. Thanks for the suggestion.

Hahaha. You're so fast. I was hoping to make that my first PR
Issue: Inserting pandas DataFrame
Author: GitHubAre
State: closed
Created: 2018-06-17 23:22:59

Hello,

Is there a way how to insert pandas DataFrame into a database?
Assuming that I have a class defined with exactly same attributes as the DataFrame's columns are (same data type and column name).
I have been googling and didnt find an answer for that (I found answer on how to select data from database into DataFrame, which is also useful: https://stackoverflow.com/questions/42596206/how-to-convert-a-select-query-into-a-pandas-dataframe-using-peewee).

I can think of iterating over the rows of DataFrame and bulk-insert them.

Can you convert the elements of a `DataFrame` into dictionaries or tuples? I'm not too familiar with Pandas.

It supports conversion to dictionary: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_dict.html

Which basically returns dictionary of dictionaries or a list of dictionaries.
Dataframe is basically representing multiple rows to be inserted.
Can that dictionary be inserted as a "bulk" insert?

I found your post: https://stackoverflow.com/questions/22750439/peewee-how-to-convert-a-dict-into-a-model/22753644
In that post there is a dictionary representing one row, where the DataFrame contains many rows.

Right, so you would do:

```python
# Convert a DF into a list of dicts.
list_of_dicts = convert_df_to_list_of_dicts(df)
MyModel.insert_many(list_of_dicts).execute()
```

Perfect, thank you!

Just to explicitly state the function it would be:
`MyModel.insert_many(df.to_dict(orient='records')).execute()`

Thanks for this excellent ORM!
Issue: Composable queries
Author: christianmalek
State: closed
Created: 2018-06-13 19:49:10

Hi,

I'm a elixir guy and use ecto as primary OR mapper. ecto has the nice feature to create composable queries. So you can save queries and then use them as input for other queries.

Example (from https://blog.drewolson.org/composable-queries-ecto/):
```elixir
# query2 uses query as input

query  = from p in MyApp.Post,
         select: p

query2 = from p in query,
         where: p.published == true

MyApp.Repo.all(query2)
```

Another example (same source):
```elixir
  def for_post(query, post) do
    from c in query,
     join: p in assoc(c, :post)
    where: p.id == ^post.id
  end

  def popular(query) do
    query |> where([c], c.votes > 10)
  end

  def published(query) do
    query |> where([c], c.published == true)
  end

# The composable queries make it very easy to reuse and improve the code semantics
# It's obvious that you first filter the published Posts, sort them and then return one record
last_post = Post
|> Post.published
|> Post.sorted
|> MyApp.Repo.one

end
```

Is something like this possible with peewee?



Yes and no. Peewee, being written in Python, uses method-chaining to accomplish something like what ecto is doing.

With peewee you'd write:

```python
query = Post.select()  # equiv to "select * from post"
query2 = query.where(Post.published == True)  # equiv to select * from post where published = true
```

Queries in peewee are composable in the sense, also, that they can be nested or combined to create new queries. For example:

```python
# Get all published posts.
published = Post.select().where(Post.published == True)

# Get all comments that are on posts that were published.
comments_on_published = Comment.select().where(Comment.post.in_(published))
```

Does this answer your question?

Hi @coleifer,

Thanks for your ultra-fast response!

So under-the-hood peewee is executing multiple queries (in your 2nd example). Seems like an possible performance impact.

It's absolutely not executing any queries. They are lazily evaluated when you iterate the results. That would be ridiculous.

Ah, ok. Sorry for the wrong implication.

Thanks for your fast response again. I'll look into using it. 👍
Issue: Join to the same model with different depth
Author: 1xinghuan
State: closed
Created: 2018-06-10 12:40:31

I have models like this:

```
class Status(ModelBase):
    name = CharField()

class Project(ModelBase):
    name = CharField()
    status = ForeignKeyField(Status)

class Shot(ModelBase):
    name = CharField()
    project = ForeignKeyField(Project)
    status = ForeignKeyField(Status)
```

If I want to join Status using following query, there will be an error
```
query = Shot.select()
query = query.join(Status).where(....)
query = query.switch(Shot).join(Project).join(Status).where(....)

for q in query:
    print q.id
```

`peewee.ProgrammingError: table name "t2" specified more than once`

Does that means peewee doesn't support this kind of multiple join or is there another way to do that?


You need to use an alias for the 2nd reference:

```python
Status2 = Status.alias()
query = Shot.select()
query = query.join(Status)...
query = query.switch(Shot).join(Project).join(Status2)
```

Docs: http://docs.peewee-orm.com/en/latest/peewee/querying.html#using-model-aliases
Issue: Benchmark with PostgreSQL gives worse insert results than with MySQL
Author: sapph1re
State: closed
Created: 2018-06-09 02:13:42

I'm running bench.py. Using python 3.5, peewee 3.5.0, on Debian 9 with 64 GB of RAM.

With a MySQL database (driver mysqlclient):
```
insert 0.232
insert_related 0.315
batch_insert 0.227
bulk_insert 0.027
select 0.093
select_related 0.26
select_related_left 0.237
select_related_dicts 0.09
```

With a PostgreSQL database (driver psycopg2):
```
insert 0.35
insert_related 0.52
batch_insert 0.32
bulk_insert 0.024
select 0.082
select_related 0.218
select_related_left 0.205
select_related_dicts 0.064
```

Why are results with PostgreSQL are significantly worse on inserting operations than with MySQL? PostgreSQL is supposed to be faster, isn't it?

This is such a complicated subject. It depends on many, many variables and config settings. The bench is an internal script I use to profile the code sometimes after making big changes. It's in no way an indication of how well a database is performing. Be smart. Ignore it. It is literally meaningless.
Issue: ForeignField not compatible with BigAutoField
Author: fenngwd
State: closed
Created: 2018-06-08 08:51:48

Description:

Column column_a in table B is a foreign key which references to primary_key a in table A. But if field type of primary_key a is BigAutoField, create table B will raise Exception(cann't add foreign key constraint).

How to reproduce the problem?

```python
import peewee as pw
from playhouse import db_url

db = db_url.connect('mysql+pool://root:root1234@localhost:3306/user')

class User(pw.Model):
    uid = pw.BigAutoField()
    class Meta:
        table_name = 'user'
        database = db


class UserSocial(pw.Model):
    uid = pw.ForeignKeyField(User, backref='userSocials', index=True, column_name='uid')
    class Meta:
        table_name = 'user_social'
        database = db

User.create_table()
UserSocial.create_table()

```

But if I change `BigAutoField` to `AutoField`, create_table will succeed. Or if I still want to use BigInt, I could update code like this

```python
import peewee as pw
from playhouse import db_url

db = db_url.connect('mysql+pool://root:root1234@localhost:3306/user')

class User(pw.Model):
    uid = pw.BigIntegerField(primary_key=True)
    uid.field_type = 'BigAuto'
    class Meta:
        table_name = 'user'
        database = db


class UserSocial(pw.Model):
    uid = pw.BigIntegerField()
    class Meta:
        table_name = 'user_social'
        database = db
        constraints = [pw.SQL('FOREIGN KEY(uid) '
                              'REFERENCES `user`(`uid`)')]

User.create_table()
UserSocial.create_table()

```

Fixed, thanks for the excellent report.
Issue: Force one-to-one backref?
Author: albireox
State: closed
Created: 2018-06-07 22:40:23

Is there a way to force a backref in a foreign key to be one to one? Right now all backrefs seem to create a list even if one takes measures to ensure the relationship is one-to-one (e.g., by making the backref column UNIQUE). In SQLA one can manage this by doing

```
relationship(ToModel, backref=backref(FromModel, uselist=False))
```

Is there a way to do this in peewee?

No, this isn't supported as-such, but you are on the right track by adding a unique constraint. In this case why not just add a `property` or something to look-up the one-to-one related model?

Right, I hadn't thought of using a property but that works nicely. It would be nice if this becomes a feature at some point but for now this works for my needs. Thanks.

For simplicity's sake I'll probably leave it for now, but thanks.
Issue: How to handle "Reference Cycles"
Author: JoshuaC3
State: closed
Created: 2018-06-07 12:08:05

I have an auto generated db_model using pwiz. However, there is a reference cycle such that

```
class DataFaultReview(BaseModel):
    data_fault = ForeignKeyField(column_name='data_fault_id', field='id', model=DataFault)
```

and 

```
class DataFault(BaseModel):
    current_review = ForeignKeyField(
        column_name='current_review_id', field='data_fault_review',
        model=DataFaultReview, null=True
)
```

reference one another.

How can/should I handle this?

Many thanks :)

http://docs.peewee-orm.com/en/latest/peewee/models.html#circular-foreign-key-dependencies