akumoth/peewee-issues huggingface dataset:
Issue: _is_closed check doesn't work with PooledMySQLDatabase and pymysql
Author: decoratorz
State: closed
Created: 2016-01-15 15:40:44

We encountered the same problem as described here: https://github.com/coleifer/peewee/issues/783

Our connections in the pool were closed by MySQL with the 2006 error after reaching server-side timeout. Because we don't always control the environment we run in, we didn't want to guess the my.cnf wait_timeout value and hardcode it in the `stale_timeout` argument for Pool, we just always need to restart a dead connection.

But the solution was simple. The main problem lies [here](https://github.com/coleifer/peewee/blob/7857f08a6fb70fc91c8e884d9eaa2289155d29f5/playhouse/pool.py#L168). This check doesn't work, because `pymysql.connection` also has the `open` attribute (at least in our version 0.6.6). We could use some other method to differentiate between  pymysql and MySQLdb, but there is actually no need. Even though `conn.ping` always [returns None](https://github.com/farcepest/MySQLdb1/blob/d236bb7f6b45f81f0156c4a0cfdef2319ba825d9/_mysql.c#L2011) on MySQLdb, it still does the right thing: throws exception on a dead connection.

After this fix PooledMySQLDatabase recognizes closed connections and everything works good. 


Nice, I tested this out and it appears to work correctly. Thanks!
Issue: Fallback gracefully in the face of ancient (and probably broken) Cython versions
Author: jherre
State: closed
Created: 2016-01-15 00:47:47

Some folks have dependencies that won't support easy Cython upgrades. Ignore incompatible Cython installations - same thing as not having it at all...

Tested on py27, py34 and py35 and all versions of Cython up to 0.23.4


Where did you arrive at that version number? I'm leery of merging this as I think it might cause some false positives.


I ran a matrix of all versions of Cython since 0.21 using py27, py34 and py35 using the peewee tests.

Using versions prior to 0.22.1 doesn't work.  Failures happen during cythonization.  If it's able to install, all tests pass.   Trying to install peewee with any version prior to Cython==0.21.1 will fail.


What about py26? Peewee officially supports python 2.6.


Yep, same result on python26.  Using versions of Cython earlier than 0.21.1 will cause setup to fail.

It's really about what versions of Cython you're will to support for speedups.  I think it's fair to say that folks need to upgrade to use the speedups in peewee but it's not fair to say they have to upgrade just to use peewee at all...

FWIW, Cython 0.22.1 was released 2014-10-18 which is really not that long ago in Python2.6 terms.
Issue: Fixed broken BlobField/BinaryField in MySQL
Author: foxx
State: closed
Created: 2016-01-14 17:32:59

Fixes #818


This will need sign off and review from @coleifer before merge. I'm about 90% happy with the code, but there could be some edge cases I haven't considered.


I'm leaning towards killing off the binary field altogether.


@coleifer Yeah I just updated the original issue with my conclusion, lemme know your thoughts


Closing as per #818
Issue: Proposal to rename `safe` to `fail_silently`
Author: foxx
State: closed
Created: 2016-01-14 15:45:45

There seems to be two keywords that have the same meaning, `safe` and `fail_silently`. Is there any reason not to use one, rather than both?

PS) This will need a docs update before it can be merged.
PS) This is also backwards incompatible
Issue: Support for model mixins?
Author: foxx
State: closed
Created: 2016-01-13 16:06:31

Currently it's not possible to do the following;

``` python
class BaseModelMixin(object):
    id = UUIDField(default=uuid1, primary_key=True, unique=True)

class BaseModel(BaseModelMixin, Model):
    pass

class User(BaseModel):
    pass
```

It's surprising behaviour that I can't construct models from a set of mixins, are there any plans to support this? If not, could you give some insight into why?


@foxx, try with:

```
class BaseModelMixin(Model):
```


@yohanboniface has it right.


Additional nit, you don't need `unique=True` for a primary key.


Ahh okay, so the mixins have to have subclass `Model`, I'm guessing due to metaclass magic of some sort. Looks to be working now, thanks both :) (also will remove `unique=True`, ty)
Issue: Incorrect SQL generated for OVER clause
Author: liato
State: closed
Created: 2016-01-13 10:40:33

I'm trying to implement sub ordering on my table based on the `challenge_id`field using the following peewee query:

``` python
(ChallengeResult
 .select(
   ChallengeResult, 
   fn.row_number().over(
     partition_by=[ChallengeResult.challenge_id], 
     order_by=[ChallengeResult.length.desc()]).alias("row_number"))
 .where(ChallengeResult.user == user).alias("rows"))
```

This should generate the following SQL:

``` sql
('
SELECT "t1"."id", "t1"."created_at", "t1"."user_id", "t1"."challenge_id", "t1"."length", 
  row_number() OVER (
    PARTITION BY "t1"."challenge_id" 
    ORDER BY "t1"."length" DESC) AS row_number FROM "challenge_result" AS t1 
WHERE ("t1"."user_id" = %s)', [2])
```

and return these results:

```
 id  |         created_at         | user_id | challenge_id | length | row_number 
-----+----------------------------+---------+--------------+--------+------------
 149 | 2016-01-12 08:45:30.534326 |       2 |            2 |    200 |          1
 151 | 2016-01-12 08:57:27.233171 |       2 |           11 |    101 |          1
 150 | 2016-01-12 08:57:21.597426 |       2 |           11 |    100 |          2
```

For some reason the `PARTITION BY` clause in the `OVER` clause is missing the table field and the generated SQL looks like:

``` sql
('
SELECT "t1"."id", "t1"."created_at", "t1"."user_id", "t1"."challenge_id", "t1"."length", 
  row_number() OVER (
    PARTITION BY %s 
    ORDER BY "t1"."length" DESC) AS row_number FROM "challenge_result" AS t1 
WHERE ("t1"."user_id" = %s)', [None, 2])
```

Returning these results:

```
 id  |         created_at         | user_id | challenge_id | length | row_number 
-----+----------------------------+---------+--------------+--------+------------
 149 | 2016-01-12 08:45:30.534326 |       2 |            2 |    200 |          1
 151 | 2016-01-12 08:57:27.233171 |       2 |           11 |    101 |          2
 150 | 2016-01-12 08:57:21.597426 |       2 |           11 |    100 |          3
```

Using peewee 2.7.4, python 3.4.0 and postgresql 9.3.9.


Instead of using `.challenge_id` can you use `.challenge` instead? Let me know. PS I had to reformat your code so I could read it without scrolling -- please add linebreaks for long chunks of code.


Closing. Please reopen if this does not address your concern.


Thanks! Stupid mistake by me. Using `.challenge` instead of `.challenge_id` worked perfectly.


Not stupid at all, glad that helped.
Issue: PooledMySQLDatabase has cache or buffer?.md
Author: tradaniel
State: closed
Created: 2016-01-13 03:23:31

i create a mysql pool like this:

```
database = PooledMySQLDatabase('test', max_connections=10,
                           **{'host': 'localhost', 'port': 3306,
                              'user': 'root', 'password': ''})


class BaseModel(Model):
    class Meta:
        database = database


class Tb1(BaseModel):
   point_id = PrimaryKeyField()

   class Meta:
         db_table = 'tb1'
```

however, where i insert a record or delete a record in database  manully,  in my program, select result is not change.

```
while True:
   database.connect()
   tb_list = Tb1.select()
   print(len(tb_list))
   database.close()
   import time
   time.sleep(2)
```

when i repalce  PooledMySQLDatabase to MySQLDatabase the result is normal.
In my project, I want to use multithread, so I need to create a mysql pool.

thanks a lot


You don't need a pool for multi-threaded, for what it's worth. Peewee correctly handles connections and will open a connection per thread.

The issue is related to #727 and #806 . I've reverted the change in #727 so this should resolve itself. In the meantime, make sure you are using transactions! Wrap your code in `database.atomic()` blocks where you are making changes. That should help until I create a new release that has the `commit_select` change.


See 6087199320a3ad8968add5d50f6c3a047767afdd
Issue: Transaction decorator not working with Tornado and Python 3.5's "async"
Author: csytan
State: closed
Created: 2016-01-11 15:36:07

The transaction decorator doesn't rollback correctly when used with an async function in Tornado.

```
@database.transaction()
async def test_txn():
    Test.create()
    raise Exception()
```

Results of test code:

```
$ py peeweetest.py 
<function test_asyncio at 0x10e033ea0>
Success

<function test_tornado at 0x10e24b2f0>
Failed
```

Here's some test code, using Python 3.5, Peewee, and Tornado:

```
import asyncio
import peewee
import tornado.ioloop


# Database setup
database = peewee.SqliteDatabase('testdb')
database.connect()

class Model(peewee.Model):
    class Meta:
        database = database

database.create_tables([Model], safe=True)


@database.transaction()
async def test_txn():
    Model.create()
    raise Exception()


def test_asyncio():
    # Start event loop, then block until test completes
    return (asyncio.get_event_loop()
        .run_until_complete(test_txn))


def test_tornado():
    # Start event loop, then block until test completes
    return (tornado.ioloop.IOLoop.instance()
        .run_sync(test_txn))


# Run tests
for test in (test_asyncio, test_tornado):
    print(test)

    count_before = Model.select().count()

    try:
        test()
    except Exception:
        pass

    count_after = Model.select().count()

    # Compare Model counts, they should be the same before and after
    if count_before == count_after:
        print('Success')
    else:
        print('Failed')
    print()
```

Manual transactions seem to work fine.  I suspect this may have something to do with Tornado's generator based co-routine support where you raise an Exception to return a value. I'm really not sure though.


This doesn't seem like a peewee issue to me, as peewee doesn't know about your event loop and is designed using normal python APIs. I am not familiar with tornado/twisted so I can't really offer much more in the way of advice for debugging this.


Understood. Thanks for having a look!
Issue: add DictField
Author: alexpantyukhin
State: closed
Created: 2016-01-07 21:34:23

Hello!
Is it possible to implement DictField (basing for example TextField)?
I need to store unstructed data in model and I think such feature would be useful.


You can use the `PickledField`, added just now:

https://github.com/coleifer/peewee/blob/master/playhouse/fields.py#L293-L302

If you're using a new-ish version of Postgres you can use the JSON field or the HStore field.

If you're using a new-ish version of SQLite with the JSON1 extension compiled in or loadable, you can use the JSONField from the SQLite extension module.
Issue: Model object constructor using kwargs puts FKs in weird state on local instance
Author: josefdlange
State: closed
Created: 2016-01-06 12:44:52

Hey!

Found an interesting one, and I'm curious if it's intended behavior or a bug. When I create a model object using a dictionary exploded into `kwargs`, if that dictionary has a None for the value of a FK field, I get a `(ModelClass).DoesNotExist` error on the local instance of the model when trying to access that field, as opposed to just receiving `None`.

Example:

```
class User(Model):
    name = CharField()
    email = CharField()

class Tweet(Model):
    user = ForeignKeyField(User, null=False)
    contents = TextField()

t = Tweet(**{"user": None, "contents": "Some contents"})
print t.user  # (raises User.DoesNotExist, expected to print None)
```

As I'm writing this, I'm curious if the `null=False` on the `ForeignKeyField` might have something to do with it, but I imagine the accessor pattern for a ForeignKeyField doesn't change based on that; only the database behavior would be different based on that flag.

Any thoughts on the validity of this as a bug or workarounds I should consider if this is expected behavior? I'm just `try`/`except`ing in these cases and it's good enough for now, but it feels Java-y.


Yes, its because null=False. That is normal behavior. It happens whether or not you initialize the fk with None, if there is no value. The null False tells us that a missing value is an error.
Issue: IndexError: list index out of range
Author: gjedeer
State: closed
Created: 2016-01-06 10:28:07

I have a problem that I can't figure out. Npm is a peewee model which I'm using like this:

```
package = Npm.select().where(Npm.id==package_name)[0]
```

However peewee throws an exception which is not very meaningful:

![peewee](https://cloud.githubusercontent.com/assets/315648/12140463/7b0206a0-b468-11e5-986c-2c821ab43b59.png)

Any idea what I'm doing wrong or how this can be fixed in peewee code ?


There is no object matching your query. Use .get() instead if you only need 1 object.
Issue: Adding table to existing db does not add unique constrant.
Author: vincentdavis
State: closed
Created: 2016-01-06 03:49:05

Existing SQLite DB defines and created by peewee.
Adding a new table, with a field that has unique=True.
The resulting table does not get the unique constrat added. In my case the line below is needed but not added. I had to remove the entire db to get peewee to add the table correctly.
The table had no ForeignKeyField. 

``` sql
CREATE UNIQUE INDEX addresses_IdHash ON addresses (IdHash);
```


How did you create the table, as in what method? You should have either called db.create_tables or model.create_table, but not db.create_table as this will not create indexes.


I used db.create_table.
Thanks


Yeah, it's an older API and can unfortunately cause some confusion... db.create_table() just creates the table. However create_tables() will create the tables for a number of models, resolving the foreign key dependencies to ensure a proper order of operations. Model.create_table() will just create the table and indexes for the model.
Issue: Proposal for create/destroy database
Author: foxx
State: closed
Created: 2015-12-28 23:16:24

There is a common use case of creating/destroying databases, for example when running tests.

Currently there is no support for automating detecting the source database and acting appropriately, for example;

``` python
x = SqliteDatabase("file:/tmp/lala.sql", uri=True)
x.create_database() # this would create /tmp/lala.sql
x.destroy_database() # this would unlink /tmp/lala

x = SqliteDatabase(":memory:", uri=True)
x.create_database() # this would create in-memory db space
x.destroy_database() # this would close all connections, thus destroying DB space

x = MySQLDatabase(host='127.0.0.1', database='helloworld')
x.create_database() # this would execute "CREATE DATABASE helloworld;"
x.destroy_database() # this would execute "DROP DATABASE helloworld;"
```

I've got the majority of the above working for Sqlite, however I haven't done Postgres or MySQL yet.

If this was put forward as a well formed PR, would this sort of functionality be accepted into the core, or playhouse?


I've always felt that this type of thing did not belong as a part of peewee. I will pass on this feature.


Got it, I'll put this into a separate repo, thanks for the quick reply!


So there is no way to create a database using peewee, right?


@gitttt -- just use the tool from your vendor. For sqlite you don't have to do anything, as it will create if not exists. For MySQL:
- https://dev.mysql.com/doc/refman/5.5/en/mysqladmin.html
- https://dev.mysql.com/doc/refman/5.5/en/create-database.html

For postgresql, use `createdb`: http://www.postgresql.org/docs/9.1/static/app-createdb.html


ok, thanks


for future MySQL readers, the way I chose to do this (after ending up here) was this: 

```
    db = MySQLdb.connect(host='', user='', passwd='')
    c = db.cursor()
    c.execute("drop database %s" % db_name)
```
Issue: Add support for BINARY field in MySQL
Author: foxx
State: closed
Created: 2015-12-28 16:47:07

See #801


Needs tests in order to merge.


You can add tests to the `test_fields.TestFieldTypes` or you can just create a one-off test-case. Either is fine.


Done. Apologies for the initial lack of tests, I got lazy!


Those tests don't actually test the binary field.


Ah crap, it looks like my cherry pick didn't apply properly due to lack of concentration, will get this fixed in the morning, sorry


Done, I'm pretty sure this is fine but any problems let me know


Thank you!
Issue: Patch proposal for better database routing - see #701
Author: foxx
State: closed
Created: 2015-12-28 15:46:26

After several different attempts, this seems to be the cleanest way forward. 

For this patch to be merge worthy, I'd need sign off from @coleifer on these questions about `Using()`. Also feedback from @stas and @alexanderad would be appreciated

This has minimal code impact as `_execute()` seems to handle every edge case, whilst ensuring that queries are still executed in the correct database order (to the extent of all tests passing, there /might/ be untested edge cases).

On the model there is now `determine_database`, by default uses the same SQL string inspection method as `read slave`, but in the event that users are making heavy use of raw queries with `IF()` conditionals, then they can extend to their hearts content.

There is also now `get_read_database()` and `get_write_database()`, by default these continue to use `self._meta.database` but can be modified to whatever the user wants.

The remaining problems now rests with `Using()`, as it does not support two database args. Altering the syntax with extra keywords would make it backwards incompatible, therefore I'd propose that we support the following syntax;

```
Using(database)
Using((read_database, write_database))
```

The last gotcha is whether `Using()` should override model, or visa versa. In my personal opinion, if someone has specified a `Using()` context then this should override whatever is in `get_read_database()` and `get_write_database()`.

From what I can tell, this /should/ cover all use cases from #701.


Although I agree this is a pretty clean patch, I am going to pass on merging.

I think that database routing is a complex enough topic that I do not want to include it in peewee itself (as I don't want to maintain it). One option would be to create a new package `peewee-routing` or something similar that encapsulates this logic.


Thanks for the quick reply! If I put myself forward as a maintainer for this, and submitted a fully documented PR, would you consider merging into playhouse? As a last option I could create a new repo, but would argue the use cases are common enough to justify being in the playhouse.


I appreciate the offer and don't want to seem to be discouraging, but I'm actively trying to rein in some of the sprawl in the code-base and tighten things up. Database-routing, to me, both complex and uninteresting from a maintenance perspective. It is my strong preference to keep it out of this repo.


Okay no worries, I'll look into creating a separate repo for this. I'll update the original issue with the URL once finished, if you're happy with it then (and agree with the approach used), then perhaps it might be worthy of a small section in the docs, so at least people know where to look?


> then perhaps it might be worthy of a small section in the docs, so at least people know where to look?

I'm sure it'll turn out well, and I'd be happy to link it up.
Issue: "'playhouse._speedups._ModelQueryResultWrapper' object has no attribute 'next'" with cython
Author: yohanboniface
State: closed
Created: 2015-12-24 15:12:29

With latest master, using cython, doing a `User.select().where(User.id == 1).get()`:

```
    def get(self):
        clone = self.paginate(1, 1)
        try:
>           return clone.execute().next()
E           AttributeError: 'playhouse._speedups._ModelQueryResultWrapper' object has no attribute 'next'
```

Does that ring a bell to you as is or should I investigate?


Fixed.


Thanks :)
Issue: Fix PasswordField on postgres (fix #766)
Author: yohanboniface
State: closed
Created: 2015-12-23 10:53:00

Fix #766

This also makes the PasswordField tests to be ran with postgres.


@coleifer any feedback on this? :)


LGTM. I haven't tested with Postgres, but the changes looks sane. Changing from BaseModel to TestModel allows it to be ran with `--engine postgres`, and changing from TextField to BlobField allows bytes to be stored safely. Needs sign off from @coleifer 


Thank you!
Issue: PostgresqlDatabase unable to connect with any database
Author: davidthewatson
State: closed
Created: 2015-12-18 14:02:00

I have a project with a virtualenv running CPython 2.7 and a pip installed peewee. When I do:

```
psql_db = PostgresqlDatabase('postgres://me@localhost/whatever')                                                                    
psql_db.connect()
```

on  a pre-existing database that has schema, tables, and rows of data, I get the following stacktrace:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dwatson/Environments/sensor_data/lib/python2.7/site-packages/peewee.py", line 3296, in connect
    self.initialize_connection(self.__local.conn)
  File "/Users/dwatson/Environments/sensor_data/lib/python2.7/site-packages/peewee.py", line 3212, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/Users/dwatson/Environments/sensor_data/lib/python2.7/site-packages/peewee.py", line 3294, in connect
    **self.connect_kwargs)
  File "/Users/dwatson/Environments/sensor_data/lib/python2.7/site-packages/peewee.py", line 3632, in _connect
    conn = psycopg2.connect(database=database, **kwargs)
  File "/Users/dwatson/Environments/sensor_data/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
peewee.OperationalError: FATAL:  database "postgres://me@localhost/whatever" does not exist
```

In the same virtualenv:

```
from postgres import Postgres
db = Postgres("postgres://me@localhost/whatever")
db.all('select * from some_table;')
```

And I get several thousand rows of data. 

I've tried all variants of Postgresql URI I could think of including password, port, etc. None of them will connect.

I am using Heroku Postgres.app on Mac OS X 10.10 and I can telnet to localhost:5432 and get the postgres interface.


in case anyone else stumbles upon this combination (Heroku Postgres with peewee), the trick seems to be to parse the Heroku-provided URL first, and then provide the parsed components to the PostgresqlDatabase constructor. It looks like this:

```python
url = urlparse.urlparse(os.environ["DATABASE_URL"])
db = PostgresqlDatabase(database=url.path[1:], user=url.username, password=url.password, host=url.hostname, port=url.port)
```

Source:
https://swifthorseman.com/2015/06/18/deploying-a-flask-app-with-peewee-to-heroku/

Thanks @gregsadetsky