akumoth/peewee-issues huggingface dataset:
Issue: Peewee 2.8.3 insert_many problem
Author: MarkSeol
State: closed
Created: 2016-09-08 11:31:16

Hi Charles:
Sorry to trouble you, we are using peewee and mysql, and our model's primary key is auto increase, we found a problem when invoking the insert_many api.
The method doesn't return the first inserted record's id, it just return True, but in the early version(like 2.4.5), it does return the first id.
So, we checked the peewee.py code. The insert section starts from line 3436:

```
if not self._is_multi_row_insert:
    if self.database.insert_returning:
        pk_row = cursor.fetchone()
        meta = self.model_class._meta
        clean_data = [
            field.python_value(column)
            for field, column
            in zip(meta.get_primary_key_fields(), pk_row)]
        if self.model_class._meta.composite_key:
            return clean_data
        return clean_data[0]
    return self.database.last_insert_id(cursor, self.model_class)
elif self._return_id_list:
    return map(operator.itemgetter(0), cursor.fetchall())
else:
    return True
```
- If we use Model.insert_many(rows).execute() , it just return True.
- If we use Model.insert_many(rows).return_id_list().execute(), it return a empty array []. operator.itemgetter(0), cursor.fetchall() both return empty array.
- If we use Model.insert_many(rows).returning(Model).execute(), an exception is threw out says that database is not supported.

So we want to ask is the insert_many method's behavior is by design or a bug? And how can we get the inserted id if we still want to use the insert many for sake of good performance?

Thanks.


By the way, cursor.lastrowid is exactly the first record's id, it's what we want.
By using cursor.fetchone() or cursor.fetchall(), nothing is returned.


This is intentional. When you're inserting multiple rows with auto-incrementing primary keys, and only the first ID is returned, it seems like maybe it would be confusing.

Postgres supports `RETURNING`, which can be used to do an `INSERT` and get back the relevant IDs. MySQL and SQLite, however, do not.


Thanks for replying. 
So I think we should change db to Postgres or we modify a little code to return the cursor.lastrowid.


Could you describe why you need the id when inserting multiple records?
Issue: Set PIPES_AS_CONCAT in MySQL sessions
Author: jakedt
State: closed
Created: 2016-09-02 18:30:15

Default `||` behavior in MySQL is to treat it as logical or. The `.concat` operator in peewee always uses `||`, so for concat to work in MySQL the session must be set to use `PIPES_AS_CONCAT`.


For MySQL, I suppose that you will just need to use `fn.CONCAT('string 1', 'string 2', etc)` instead of `Model.some_field.concat()`. Is that reasonable, or do you think that the `concat()` function needs to be a bit smarter?


Due to the design of the `QueryCompiler`, making the parser switch from an expression to a function call, in the event we're dealing with a concat on MySQL, requires some very ugly hacks.


I just set up a proxy to switch between the two when I initialize the database connection. It's a fine workaround (the same as what we do for the random function across drivers) but I wish it were a little more seamless.
Issue: Syntax error in query
Author: kennedyshead
State: closed
Created: 2016-09-02 14:28:57

Python2.7

```
peewee[20118] DEBUG ('SELECT "t1"."id", "t1"."user_id", "t1"."unique", "t1"."title", "t1"."message", "t1"."created_date" FROM "event" AS t1 WHERE (((("t1"."message" = ?) AND ("t1"."unique" = ?)) AND ("t1"."user_id" = (?, ?))) AND ("t1"."title" = ?)) LIMIT 1 OFFSET 0', [u'Someone created a new project', u'97df91d41dc15590451eade6f1d16de1bb218d1e0d8a21aa2de295f1', 25, 0, u'Ruby'])
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/Users/magnusknutas/git/thefarm/app/services/mqtt_service.py", line 38, in run
    self.client.loop()
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 811, in loop
    rc = self.loop_read(max_packets)
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1073, in loop_read
    rc = self._packet_read()
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1475, in _packet_read
    rc = self._packet_handle()
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1947, in _packet_handle
    return self._handle_pubrel()
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2152, in _handle_pubrel
    self._handle_on_message(self._in_messages[i])
  File "/usr/local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2274, in _handle_on_message
    self.on_message(self, self._userdata, message)
  File "/Users/magnusknutas/git/thefarm/app/services/mqtt_service.py", line 84, in on_message
    message=text
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 4800, in get_or_create
    return query.get(), False
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3090, in get
    return next(clone.execute())
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3142, in execute
    self._qr = ResultWrapper(model_class, self._execute(), query_meta)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 2826, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3683, in execute_sql
    self.commit()
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3507, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3676, in execute_sql
    cursor.execute(sql, params or ())
OperationalError: near ",": syntax error
```


Looks like you're passing a tuple in to user id


Hi!

Its resolved, switched to get_or_create and then it told me it was a tuple and not an int as you stated.

It was just a misleading error ;)


Thank you for your quick reply


Sure thing, I think peewee could stand to get smarter about checking these types of errors, but then there will be an inevitable performance hit. So, trade-off.
Issue: Postgres extension's full text search Match raises KeyError 'T@@'
Author: ephiepark
State: closed
Created: 2016-08-30 05:08:47

Hi, I am using peewee with flask and postgresql db, and I would like to use peewee's postgres extension to do full text search (http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#TSVectorField). But when I use Match function from the link, it raises KeyError 'T@@'. 

I followed the stack trace, and the KeyError is raised in get_op() call in QueryCompiler class. When q, the parameter to get_op() call, equates to 'T@@', self._op_map does not have key 'T@@' and raises KeyError. 


My guess is that you are not using the `PostgresqlExtDatabase` class, which is a subclass of `PostgresqlDatabase` class.

How are you defining your database?


Thank you for the quick reply!

Yes, I think my application is using PostgresqlDatabase not PostgresqlExtDatabase. I am using FlaskDB class setup db connection, and during init_app() call in FlaskDB it reads from config DB url and it automatically instantiate PostgresqlDatabase by looking at the url. 

Looking at the playhouse.db_url module, it seems like there is no way to let FlaskDB to create PostgresqlExtDatabase instance. 

What should I do to make use of PostgresqlExtDatabase? 

p.s. I have tried the following to force PostgresqlExtDatabase to be used 
`DATABASE = 'postgres://....db_url....'`
 `parsed = urlparse(DATABASE)`
 `connect_kwargs = parseresult_to_dict(parsed)`
 `flask_db = FlaskDB(database=PostgresqlExtDatabase(**connect_kwargs))`
And I got the follow error
`peewee.ProgrammingError: hstore type not found in the database. please install it from your 'contrib/hstore.sql' file`


> Looking at the playhouse.db_url module, it seems like there is no way to let FlaskDB to create PostgresqlExtDatabase instance.

The docs show you how to use `db_url` to use the `PostgresqlExtDatabase`:

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#db-url

By default the hstore extension is enabled, so you can do one of two things:
1. Disable it
2. Enable it in postgres

To disable, pass in `register_hstore=False` when either instantiating the database instance, or as a parameter of the db_url, e.g. `scheme://user:password@host:port/database_name?register_hstore=false`

To enable it in postgres, open up a postgres shell `psql my_database` and run the following SQL:

``` sql
CREATE EXTENSION hstore;
```

Depending on your system, you may need to install additional packages.
Issue: [improvement] Model Definition via JSON
Author: rognoni
State: closed
Created: 2016-08-26 20:23:59

For my project I need the **Model Definition** via JSON, for example:

https://github.com/rognoni/adaptable/blob/master/Backend/resources/schemas/info.json

``` javascript
{
  "entities": {

    "group": {
      "properties": {
        "id": {"type": "int", "isPK": true},
        "desc": {"type": "str"},
        "name": {"type": "str", "isNull": false}
      }
    },

    "group_user": {
      "properties": {
        "id": {"type": "int", "isPK": true},
        "group_id": {"type": "ref", "entity": "group", "isNull": false},
        "user_id": {"type": "ref", "entity": "user", "isNull": false}
      }
    },

    "user": {
      "properties": {
        "id": {"type": "int", "isPK": true},
        "key": {"type": "str", "isNull": false, "isUnique": true},
        "password": {"type": "str", "isNull": false},
        "username": {"type": "str", "isNull": false}
      }
    }
  }
}
```

For the moment, to avoid source-code generation, I am using a dynamic way with `type(name, bases, dict)`

https://github.com/rognoni/adaptable/blob/master/Backend/Python-Flask/storage/sqlalchemy_sqlite.py


Ummmm, huh?  http://thedailywtf.com/articles/the-inner-json-effect

I'm not going to add anything like this to peewee. It can't be more than 50-60 lines of code to do what you're trying to do.

Besides the familiar declarative syntax, Peewee can introspect an existing database, modify the schema on-the-fly, and provides extensions like [playhouse.dataset](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#dataset) and apps like [my web-based sqlite manager](https://github.com/coleifer/sqlite-web/).
Issue: Materialized views?
Author: Resisty
State: closed
Created: 2016-08-26 18:34:24

Hello,

I was just trying to figure out, definitively, if Peewee supports the notion of [materialized views](https://www.postgresql.org/docs/9.3/static/sql-creatematerializedview.html). If so, any tips would be great, but if not, it's just good to know.

Thanks!


Peewee does not generate the DDL for the creation of views, but you can certainly create a model for your view and query it normally. So, the answer is yes and no. Does that answer your question?


I think so. The advice I got from a friend (who doesn't speak Python but only sql) was that having a materialized view would be good so I could have an index instead of, say, running a count across every table in the query every time. As long as I know I have to do it that way, I'm good to go. Thanks!


You could also denormalize, and add the count as a column directly on the table. Might save you some disk space if your materialized view is large, and you can ensure the counts stay up-to-date using simple triggers. Just a thought.
Issue: Peewee 2.8.2 regression
Author: mrjoes
State: closed
Created: 2016-08-25 15:10:27

Long story short - something in 2.8.2 is causing problem for one of the Flask-Admin tests.

Here's the test: https://github.com/flask-admin/flask-admin/blob/master/flask_admin/tests/peeweemodel/test_basic.py#L918

Here's the traceback:

```
ERROR: flask_admin.tests.peeweemodel.test_basic.test_form_args
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/joes/Work/flask-admin/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/joes/Work/flask-admin/flask-admin/flask_admin/tests/peeweemodel/test_basic.py", line 938, in test_form_args
    eq_(len(create_form.test.validators), 2)
AttributeError: 'ModelForm' object has no attribute 'test'
-------------------- >> begin captured logging << --------------------
peewee: DEBUG: ('CREATE TABLE "basemodel" ("id" INTEGER NOT NULL PRIMARY KEY)', [])
--------------------- >> end captured logging << ---------------------
```

Basically, `test` is not registering as a column for whatever reason.


Please ignore - it was already fixed in the master. Thanks!


Can you point me to the commit that fixed this issue? I want to be sure I understand what happened.


I don't know - I installed peewee from the master and it worked, all of a sudden.

Edit: More info

`pip install -e git+https://github.com/coleifer/peewee#egg=peewee` works
`pip install peewee==2.8.1` works
`pip install peewee==2.8.2` does not work


Just makes me nervous since that traceback doesn't really ring any bells.


The traceback is internal to flask-admin tests. 

Here's the issue:

``` python
def test_fn():
    class BaseModel(peewee.Model):
        class Meta:
            database = db

    class Model(BaseModel):
        test = peewee.CharField(null=False)

    Model.create_table()
```

produced this:

``` sql
CREATE TABLE "basemodel" ("id" INTEGER NOT NULL PRIMARY KEY)
```

For whatever the reason - it ignored the `test` column in the model definition. wtf-peewee didn't generate the field and test crashed with:

```
AttributeError: 'ModelForm' object has no attribute 'test'
```
Issue: Why is foreign object's key always accessible as *_id?
Author: maaaks
State: closed
Created: 2016-08-24 12:39:17

In the ForeignKeyField source code, I see this:

``` python
obj_id_name = name + '_id'
...
setattr(model_class, obj_id_name, self._get_id_descriptor())
```

This means that no matter what field is foreign key in, the ID of the object is always accessible as user_id, parent_id, etc. Even if the real column name, the `db_column`, is not *_id at all.

I noticed this because I made a `ForeignUuidField` for my projects. It is based on the standard `ForeignKeyField` but instead of integer `*_id` uses field `*_guid`. But, surprisingly, Peewee still creates field `*_id` with the GUID that I need to load foreign object.

I think the `obj_id_name` from the code above should be equal to `db_column`, at least for this particular situation. (And I think that there is no other possible situations to hardcode it that way, except for default behaviour when `db_column` is not defined by user.)


I understand your concerns. When the foreign key's field name is equal to it's db_column, I suppose it would be fine to fall back to just appending "_id" -- otherwise the db_column could be used. It's important to have this ID descriptor, as it provides a way to access the foreign key data without forcing resolution of the foreign key.

Would you consider the best fix, then, to use the db_column for the foreign key's ID descriptor except when it is the same as the field name (in which case append "_id")?


Yes, I just didn't think of the case without any postfix in column name. In my DB, there is always a postfix, it's just different: `_id` or `_guid`.

So yes, «use `db_column` if provided and differs from name, else use `name+"_id"`» seems to be the correct way to fix it, as far as I understand it all.


+1
Issue: Default   value weird behavior
Author: staier
State: closed
Created: 2016-08-23 14:00:24

After  last  upgrade  something  went  very  wrong .

```


class SomeModel(BaseModel):
    value=IntegerField()
    hash= FixedCharFiled(default=utils.some_hash_func)

object= SomeModel(value=3)
object.save()
object2= SomeModel.get(SomeModel.id==object.id)
```

**object1.hash ==object2.hash**  returns **False** 

it looks like  field populated with one default value ,  but  insert statement  is prepared with  another call to  default  function,  even though the value was already  generated.
 this ruined my  application completely. 


update:
setting 

```
  class Meta:
        only_save_dirty = False
```

Fixed the  issue ,   however   i  do  not  believe this is a good solution.
One more thing  i  noticed ,  default values functions calculated every  time  an  object  gets populated from database regardless of if it  was set  already  to  default value or otherwise.

This may  get  expensive  quickly


Can you tell me what version you were using previously?


I think I know -- in `Model._prepare_instance` it's clearing the dirty fields a bit too aggressively.


It was 2.6.4.


That's quite a big jump... I was wondering if e28b6e4529176a1dff6b2b6e247658ccab5bcbb4 was the issue but I'm guessing the fact that defaults are excluded from dirty fields on population is the real issue.


I believe I have a fix pushed, just going to let the tests run.


Fixed in c17d3db


Please note  ,  that with  IntegerField(default=incrementer()) default only  executes once ,   but if IntegerField(default=incrementer)  it will be callable  and treated differently .  Not  sure if it   matters .


It's still considered a callable because the return value is a function.
Issue: Entity is dirty after loading from DB (peewee==2.8.2)
Author: tuukkamustonen
State: closed
Created: 2016-08-17 06:31:38

``` python
from peewee import Model, SqliteDatabase, CharField

db = SqliteDatabase('test.db')

class Foobar(Model):
    field = CharField()

    class Meta:
        database = db
        only_save_dirty = True

db.connect()
db.create_tables([Foobar])

obj1 = Foobar.create(field='foobar')

foo = Foobar.get(id=obj1.id)
print(foo._dirty)

foo.save()  # should do nothing but actually hits db because fields are marked dirty
```

Output:

```
{'id', 'field'}
```

With 2.8.1 `_dirty` is an empty set instead.


I checked out 2.8.1 and was seeing that the instance from the db was dirty...nonetheless it looks like a bug.


Weird, I tried again (in a clean virtualenv) and with peewee==2.8.1 `_dirty` is indeed an empty `set()`. Sounds a bit weird if the result is different in your sandbox.

Well, thanks for the bugfix anyway.


I did not update PyPI (that would entail pushing a new release, 2.8.2). If you point at master branch on github you should see the fix:

```
pip install -e git+https://github.com/coleifer/peewee#egg=peewee
```


That doesn't play together with an in-house pypi (devpi) so I'll just wait for 2.8.3 as 2.8.1 works just fine for me.
Issue: GFKField with custom field names?
Author: bcthomas
State: closed
Created: 2016-08-17 04:15:25

I have a legacy Rails database that I'm writing a peewee wrapper around.  The Rails db uses a polymorphic/GFK for a "note".  I'm trying to use peewee GFKs, but the SQL it's using to find a note is using the lowercase name of the table for the associated model, but the value in the db is the actual class name.

For example, "Feature" is in the db resource_type column, but peewee is using a select query with "features".

``` SQL
SELECT `t1`.`id`, `t1`.`value`, `t1`.`resource_id`, `t1`.`resource_type`
 FROM `notes` AS t1 WHERE ((`t1`.`resource_type` = %s)
 AND (`t1`.`resource_id` = %s)) ['features', 59]
```

Is there a way to customize this?

Thanks


Unfortunately, the GFK field was never intended to interact with Rails. Perhaps you can add extra column(s) to the table to support the Peewee version of the data, or write a custom field that uses the names you need.


I figured this was the case.  Modifying the table is a great idea and probably the least work.

Thanks @coleifer !
Issue: Peewee install is broken on PyPy
Author: scitesy
State: closed
Created: 2016-08-15 15:50:43

Ubuntu 16.04
PyPy pypy2-v5.3.1-linux64

Both pip install peewee and python setup.py install from a virtualenv fail with the following terminal output: https://gist.github.com/scitesy/550fa086870c7d90ae138b8f7e209e1c

I believe it has been broken since sometime around July 14th.


Interesting. Do you have Cython installed alongside Peewee by chance? If not it's using the C source files bundled with Peewee, which appear not to be compatible.


I've reverted a particular commit and pushed the changes to `master`... Do you mind trying them out?

Should be:

```
pip install -e git+https://github.com/coleifer/peewee.git#egg=peewee
```


Thank you. I'll try it again later tonight.


That worked! To answer the question from earlier, I am running PyPy with no Cython installed.
The terminal output:
Obtaining peewee from git+https://github.com/coleifer/peewee.git#egg=peewee
  Cloning https://github.com/coleifer/peewee.git to ./test_pypy_peewee/src/peewee
Installing collected packages: peewee
  Running setup.py develop for peewee
Successfully installed peewee

Thank you!


I'll issue a new release to pypi later today.


@coleifer When do you think this will be released? Thanks!
Issue: Add pool option to db_url.connect
Author: joeyespo
State: closed
Created: 2016-08-13 18:48:31

This separates the URL concern from the pooling concern. This is also backwards-compatible, so you can still use 'scheme+pool' if you want to hardcode the decision to use pooling right in your URL.

This lets you write:

``` py
db = db_url.connect(app.config['DATABASE_URL'], pool=True)
```

Now, for example, Heroku's raw `DATABASE_URL` can be used as-is _and_ have pooling enabled.

I had this logic in my app, but it made sense to me to expose it to `db_url`. Note that you can also ensure pooling is disabled by specifying `pool=False`, which I included for completeness.

If you think adding a new parameter is better (`def connect(url, pool=None, **connect_params):`), I'll change it to that. I just didn't want to break anybody. In the far future when Peewee drops Python 2 support, this can be accomplished with `def connect(url, *, pool=None, **connect_params):`.


I will pass on this feature. The whole point of `db_url` is to allow you to specify things in the URL, and I don't see this as adding any particular value beyond what's already there.


Ok.

Seems odd that when URL comes from an external source, you have to manually inject the Peewee-specific '+pool' argument. Oh well. Thanks for the response.
Issue: Documentation error
Author: lyssdod
State: closed
Created: 2016-08-13 01:01:25

http://docs.peewee-orm.com/en/latest/peewee/database.html?highlight=sqlite#pragma-statements

Here it says "To run one or more PRAGMA statements against new connections, you can specify them as a list of 2-tuples... " but the code coming afterwards utilizes a tuple instead of list:

```
pragmas=(
    ('journal_mode', 'WAL'),
    ('cache_size', 10000),
    ('mmap_size', 1024 * 1024 * 32),
)
```

I've just experienced this myself, getting this error:

```
  File "/usr/lib64/python2.7/site-packages/peewee.py", line 3627, in _set_pragmas
    for pragma, value in self._pragmas:
ValueError: too many values to unpack
```

unless I tried to substitute `pragmas = (...)` to `pragmas = [...]`


You can specify them as a list or a tuple, so I will update the documentation to that effect.

I'm guessing you ran into the error because you were only specifying one `PRAGMA` and forgot the trailing comma Python requires to differentiate simple parentheses from a tuple declaration.


Exactly, I was specifying one `PRAGMA` and forgot the comma. Thanks!
Issue: Use six library
Author: thomdixon
State: closed
Created: 2016-08-12 16:09:23

You should use the wonderful [`six`](https://pythonhosted.org/six/) library, which would make a lot of [this](https://github.com/coleifer/peewee/blob/master/peewee.py#L118) unnecessary (especially the [`exec` for `reraise`](https://github.com/coleifer/peewee/blob/master/peewee.py#L143)).


In my opinion, the `six` library is silly and I don't want to add a useless dependency.