akumoth/peewee-issues huggingface dataset:
Issue: How can I explicitly call close on all stale connections in the pool (including the ones in _in_use)
Author: confessin
State: closed
Created: 2018-07-19 07:43:39

We are using Pooled Mysql connections on appengine with peewee and flask.
Although we have written a decorator to explicitly call close on db connections after each call. There are instances when it remains open. As a result we are hitting the max connection exceeded error.
Here is the decorator

    def catchdb(dbobj):
       """TODO: Docstring for catchdb.
    
       :dbobj: TODO
       :returns: TODO
    
       """
       def decorator(func):
           @wraps(func)
           def foo(*args, **kwargs):
               result = None
               try:
                   result = func(*args, **kwargs)
               finally:
                   print "Closing current connection"
                   dbobj.close()
                   print "Number of in use connections: ", len(dbobj._in_use)
                   for conn, t in dbobj._in_use.items():
                       if (time.time() - t) > (10 * 60):  # 10 mins seconds before we close the stale connections.
                           print "Closing stale connection ", conn
                           del dbobj._in_use[conn]
               return result
           return foo
       return decorator

The solution we wanted to try is to explicitly close all the inactive connections (in_use for too much time) The problem is dbobj._in_use contains the id of the reference and not the actual reference.

It would be great if anyone can guide us in any right direction to solve this issue, for now we have removed max connection limit.

> Although we have written a decorator to explicitly call close on db connections after each call. There are instances when it remains open.

This sounds like you have a bug in your code or there is some problem in your environment. I'd strongly suggest you look into where and how you're leaking database connections. As a result, any of the mitigation strategies proposed are going to be hacks.

Given the leaky connections, perhaps you should not use the pool?

Making connections evertime is really heavy in google cloud, because of all the auth in place i guess. So no pooling is not an option. We are actively checking for the bugs, it works for most part but overtime we face this issue.

But is there anyway I can push the connections from _in_use to the usable pool?

The entire code for the connection pooling implementation is ~150 lines of code, so I'm sure if you take the time to read and understand it you will be able to subclass/extend it to more aggressively handle your unclosed connections.

I've added the requested feature:

* 22e8557e279f5174053ed711f342da8e03fae0c4
* c985636ca13bf8b712d37bf991ece8fdb9a022a8
Issue: Why read_slave extension is deleted in Peewee 3.x??
Author: zjyExcelsior
State: closed
Created: 2018-07-17 14:05:54

I want to know why read_slave is deleted in Peewee 3.x?what's the reason for doing that? I am recently using  Peewee 2.x

It is removed because I am no longer interested in maintaining it. Feel free to take the 2.x code and modify it and release it as a separate library.
Issue: Embed SSH tunnel usage and reuse existing connections
Author: imakiro
State: closed
Created: 2018-07-17 13:11:15

Because of environment constraints, and peewee lazy loading of DB properties, when using an SSH tunnel  library like 
`from sshtunnel import SSHTunnelForwarder `
I am getting rejected by the SSH server because it establishes a connection every request.

It's really hard to integrate Peewee with the SSH tunnel without running into the SSH server getting spammed with connection opening doing one request and closing.

It would improve reliability to allow peewee itself to manage it's SSH tunnels.
Or maybe there is a best practice when dealing with secure environments that I missed.


This is way outside the scope of Peewee. Depending on the database driver you are using, you may want to discuss with them, but I doubt you'll have any luck there either. Why you would ever want to pipe a db connection over SSH is a whole other question.

You can use the connection pool, which keeps connections open and recycles them -- this may be the best option for you. http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pool

Thank you for the reply, I though this would actually matter for the project, but I'll try my luck with the mylsq driver we use.

> Why you would ever want to pipe a db connection over SSH is a whole other question.
Would you suggest an alternative to avoid exposing the database and still allow peewee to connect?

> Would you suggest an alternative to avoid exposing the database and still allow peewee to connect?

Presumably your application server and database server would be on a LAN and only the application server would be exposed to the internet? I've never, in my career, found it necessary to expose a database server to the public internet.
Issue: AttributeError: 'NoneType' object has no attribute 'safe_create_index'
Author: gurland
State: closed
Created: 2018-07-15 21:54:07

I'm using `SqliteExtDatabase`. When I'm trying to create tables next error raises:

```Traceback (most recent call last):
  File "/home/gurland/Projects/mathbot/bot.py", line 3, in <module>
    from models import SubscriptionChat, Subscription, IUMCourse
  File "/home/gurland/Projects/mathbot/models.py", line 119, in <module>
    db.create_tables([User, Catalog, Book, IUMCourse, SubscriptionChat, Subscription], safe=True)
  File "/home/gurland/Projects/mathbot/venv/lib/python3.6/site-packages/peewee.py", line 2774, in create_tables
    model.create_table(**options)
  File "/home/gurland/Projects/mathbot/venv/lib/python3.6/site-packages/peewee.py", line 5604, in create_table
    if safe and not cls._meta.database.safe_create_index \
AttributeError: 'NoneType' object has no attribute 'safe_create_index'```

peewee 3.5.2
python 3.6

Your model class does not appear to have declared a database:

```python
class MyModel(Model):
    class Meta:
        database = ...  # You are missing this.
```

> Your model class does not appear to have declared a database:
> 
> ```python
> class MyModel(Model):
>     class Meta:
>         database = ...  # You are missing this.
> ```

I am facing the same issue and database is defined in Meta. Could you please suggest a fix for this please?

Share your code

My apologies, I did a terrible mistake while writing the code. Instead of passing database instance under table's META, I was passing database name itself and hence this issue. But I must say the error "'NoneType' object has no attribute 'safe_create_index'" is not developer friendly. It should elaborate more to help us. It took around 1 full day for me to identify this mistake in code.
Thanks for your response, you may close this.
Issue: TimestampField, Microseconds (Precision=10**6) and Postgres
Author: yakirmar
State: closed
Created: 2018-07-15 20:18:07

Hi,
I created a Postgres DB with "PostgresqlDatabase" object and the field was defined like this:
`timestamp = TimestampField(resolution=10**6, index=True)`

Everything works great when I did some tests with SQLite but when I moved to Postgres, it created an integer field in the DB and the numbers that I want to insert (like this 1531326087799518) is much bigger than integer/int4.

How can I change it to be BigInt (according this: https://www.postgresql.org/docs/9.2/static/datatype-numeric.html#DATATYPE-INT)?

Thanks in advance,
Yakir

Whoops, oversight on my part. I need to change the underlying field type to `BigIntegerField`. Will have a patch up shortly.

3b89d3af132d06f986642384ffbdadfcffbfa563
Issue: percent sign in column name breaks sql
Author: NCarson
State: closed
Created: 2018-07-15 00:12:25

I had a column named "win%" in one of my tables in Postgresql. When I tried to execute the query I got this error (last of the call stack):
```
File "/usr/local/lib/python3.5/dist-packages/peewee-3.5.2-py3.5-linux-x86_64.egg/peewee.py", line 2639, in execute_sql
    cursor.execute(sql, params or ())
IndexError: list index out of range
```

Anyways, after looking at the sql peewee generated I saw the percent sign, which after changing the name to win_pct fixed the problem. I have seen the other closed issue with modulo and percents and understand how psycopg2 binds params. But, I think introspection should give an error or warning that your table will be unusable. Since, the error peewee generates does not give you much hints. 

> I have seen the other closed issue with modulo and percents and understand how psycopg2 binds params.

My guess is that the problem is with the way the parameters are interpolated into the SQL string. If you understand this, then it is hopefully clear why this column name was problematic.

I agree the Peewee error isn't too helpful but in the 8 years I've maintained this project I've never seen anyone use a "%" character in a column and report a problem. So I'm going to leave the code as-is. I'd suggest you stick to well-known conventions for naming your tables and columns.

https://launchbylunch.com/posts/2014/Feb/16/sql-naming-conventions/

Furthermore this isn't a Peewee problem, as much as it is a problem with Python strings interacting with percent signs. Peewee's introspection will detect the weird column name and use a valid Python identifier, so the introspector works correctly in my opinion.
Issue: Added support for introspecting views for Postgresql.
Author: NCarson
State: closed
Created: 2018-07-14 22:32:28

A command line switch will introspect views instead of tables in pwiz.py. A
get_views function was added for Postgres and NotImplemented version for the
interface. It would probably be more convenient for users to do both tables and
views at once, but this was an extremely simple way to do it. I did not add it for
others since I am mostly familiar with Postgres, but it seems pretty easy.

I'm going to pass on this for now, but may reconsider adding this functionality (if it can be added cleanly for all supported database implementations). Thank you -- I may end up implementing this at some point and your code will certainly come in handy.

Also, for what it's worth, if you want to increase the likelihood of getting a "new feature" patch merged, it should have tests and documentation as well.

This is actually now implemented by 273008413737ff16d26b27f846a315cdf4875615

Although it will generate both regular tables *and* views if passed the `-v` option. There is no option to *just* generate views.
Issue: Fix compatibility with Python 3.7
Author: felixonmars
State: closed
Created: 2018-07-14 17:59:52

Fixes many test failures like below:

```
======================================================================
ERROR: test_iterator (tests.models.TestModelAPIs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-peewee/src/peewee-3.5.2/peewee.py", line 3613, in iterator
    yield self.iterate(False)
  File "/build/python-peewee/src/peewee-3.5.2/peewee.py", line 3597, in iterate
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/build/python-peewee/src/peewee-3.5.2/tests/base.py", line 247, in inner
    method(self)
  File "/build/python-peewee/src/peewee-3.5.2/tests/models.py", line 989, in test_iterator
    self.assertEqual([u.username for u in query], users)
  File "/build/python-peewee/src/peewee-3.5.2/tests/models.py", line 989, in <listcomp>
    self.assertEqual([u.username for u in query], users)
RuntimeError: generator raised StopIteration
```

With the patch, all tests pass under both python 3.7 and 2.7 here.

Thanks. So you can't raise `StopIteration` inside a generator according to [this PEP](https://www.python.org/dev/peps/pep-0479/)... weird. I'm guessing this is some consequence from the asyncio abomination. 

> Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.

What if it's raised *intentionally* though? @gvanrossum needs to **sit the fuck down**, I'm glad he's taking his seat and backing off...so many terrible PEPs in the last couple versions.

Merged, thanks again for taking the time to investigate and fix.
Issue: peewee error message
Author: TechComet
State: closed
Created: 2018-07-14 15:28:36

Hi,

How get error message

http://docs.peewee-orm.com/en/latest/peewee/contributing.html#bugs

I have literally no clue what you are asking.

```
try:
	table_name.function_name(...)
except IntegrityError as e:
	return e
```

You can do this:

```python
try:
    SomeModel.function_name(...)
except IntegrityError as exc:
    exc_message = str(exc)
```
Issue: Improve imports in shortcuts extension
Author: gustavi
State: closed
Created: 2018-07-11 16:04:58

Improve the imports in this file.

First statement use wildcard import (and this is bad) and second import something already imported.

Import only required objects is the good way to do.

Peewee's main module `peewee.py` defines a module-level `__all__` which controls what is exported when you `import *`. Peewee uses the convention of `import *` internally, so I am going to pass on this change, even though I do understand your reasoning behind the suggestion.
Issue: Feature: a possibility to check whether Model matches Schema
Author: pauleikis
State: closed
Created: 2018-07-10 16:45:53

I have some semi permanent data which I would be fine with losing when I need to update Schema. 
It would be nice to have a feature to check whether Model changed and recreate table if needed. I would imagine API and use case like this

```python
for model in model_list:
    if not db.validate_tables(model):
        db.drop_tables([model])
        db.create_tables([model])
```

I think this feature might be useful in early stages of many projects where it could be much simpler solution than running it manually or maintaining migration. Not sure how can this be implemented though.

I do not plan on implementing such an API, but the fundamentals are all there already.

You can use `database.get_tables()` and `database.get_columns()` and check for differences with `Model._meta.fields`. You can also use the higher-level `reflection` APIs.

* http://docs.peewee-orm.com/en/latest/peewee/api.html#Database.get_tables
* http://docs.peewee-orm.com/en/latest/peewee/api.html#Database.get_columns
* http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#reflection

I put this little function together:

```python
from collections import namedtuple

from playhouse.reflection import Introspector


ValidationResult = namedtuple('ValidationResult', (
    'valid', 'table_exists', 'add_fields', 'remove_fields', 'change_fields'))


def validate_schema(model):
    db = model._meta.database
    table = model._meta.table_name
    if not db.table_exists(table):
        return ValidationResult(False, False, None, None, None)

    introspector = Introspector.from_database(db)
    db_model = introspector.generate_models(table_names=[table])[table]

    columns = set(model._meta.columns)
    db_columns = set(db_model._meta.columns)

    to_remove = [model._meta.columns[c] for c in columns - db_columns]
    to_add = [db_model._meta.columns[c] for c in db_columns - columns]
    to_change = []
    intersect = columns & db_columns  # Take intersection and remove matches.
    for column in intersect:
        field = model._meta.columns[column]
        db_field = db_model._meta.columns[column]
        if field.field_type != db_field.field_type:
            to_change.append((field, db_field))

    is_valid = not any((to_remove, to_add, to_change))
    return ValidationResult(is_valid, True, to_add, to_remove, to_change)
```

Pushed a new branch which contains the above commit (d6cbfd3), which includes this function as well as some basic unit tests. I don't plan on merging this, as it seems like a slippery slope of folks asking for more and more sophisticated difference detection. But it might be useful for those trying to implement something like you described?

Wow, thanks for the really quick and detailed answer. 
I totally understand that this might be an overkill and out of scope of such library to implement every detail properly. 
The example code is helpful, thanks a lot!
Issue: TypeError: get_sort_key() missing 1 required positional argument: 'ctx'
Author: mindojo-victor
State: closed
Created: 2018-07-07 06:55:16

What this might mean? I have no idea how to fix this. Or is it a bug in the `peewee` code?
```python
  File "/Users/Vic/projects/brain2/brain/utils/reports/bi.py", line 449, in sync_author_tasks
    BIAuthorTask.insert_many(author_tasks).on_conflict_replace().execute()
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 1560, in inner
    return method(self, database, *args, **kwargs)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 1631, in execute
    return self._execute(database)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2274, in _execute
    return super(Insert, self)._execute(database)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2049, in _execute
    cursor = database.execute(self)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2637, in execute
    sql, params = ctx.sql(query).query()
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 549, in sql
    return obj.__sql__(self)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2258, in __sql__
    self._generate_insert(self._insert, ctx)
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2181, in _generate_insert
    columns = sorted(accum, key=lambda obj: obj.get_sort_key(ctx))
  File "/Users/Vic/projects/brain2/venv/lib/python3.6/site-packages/peewee.py", line 2181, in <lambda>
    columns = sorted(accum, key=lambda obj: obj.get_sort_key(ctx))
TypeError: get_sort_key() missing 1 required positional argument: 'ctx'
```

peewee==3.5.2
Python 3.6.5

Oh, the model contained a bad attribute:
```python
    assigned_at = peewee.DateTimeField
```
2.10.2 detected this, while 3.5.2 showed this error.
After I added the parens, it worked.

Should I close this issue?

> 2.10.2 detected this, while 3.5.2 showed this error.

What did 2.10.2 do when this happened?

Something like "Unknown field".

I think I'm going to leave this as-is for now, as I'm not sure of a clean way to implement this.

Very unintuitive error message. This issue probably saved me hours of debugging. I did the same thing @mindojo-victor did. Maybe something in `BaseModel.__new__` like this:
```
for field in dir(cls):
    if field.startswith('__'):
        pass
    if type(getattr(cls, field)) == type:
        raise ValueError('Fields must be initialized.')
```

I could work on PR

Whoo that was a fun 2 hrs trying to figure out what was wrong  -- I was missing the parens after IntegerField

Is there an easier way to detect this issue in the future?
Issue: Reject or warn on naive datetime objects for playhouse.postgres_ext.DateTimeTZField?
Author: wonderbeyond
State: closed
Created: 2018-07-06 07:38:23

When assign a naive datetime object to a model filed of DateTimeTZField, it results an utc-timezoned value into db.

This behavior is crude, because without the knowledge of the current timezone we cannot convert a naive datetime object to the right aware one.

Can we let DateTimeTZField just reject on naive datetime objects?

I'm going to leave it as-is, but you are welcome to subclass `DateTimeTZField` and add whatever validation you want in the appropriate places.
Issue: the order of importing mysql database driver is wrong
Author: elonzh
State: closed
Created: 2018-07-05 06:03:02

```python
try:
    import pymysql as mysql
except ImportError:
    try:
        import MySQLdb as mysql  # prefer the C module.
        mysql_passwd = True
    except ImportError:
        mysql = None
```
As your comment, we should prefer the C module as default because it has better performance for production but the order of importing mysql database driver is wrong.

It should be great If we can choose our database driver like sqlalchemy.



So `mysql-python` (`MySQLdb`) does not support Python 3, which is why it is no longer preferred. The comment needs to be updated.

If you would like to explicitly use a particular driver, it is quite easy:

```python
import MySQLdb

class MySQLDatabaseCustom(MySQLDatabase):
    def _connect(self):
        return MySQLdb.connect(db=self.database, **self.connect_params)
```

[farcepest/MySQLdb1](https://github.com/farcepest/MySQLdb1)  has already been abandoned, [PyMySQL/mysqlclient-python](https://github.com/PyMySQL/mysqlclient-python) has perfect Python3 support without losing performance benefit.

The code snippet I shared should work just fine. Again, peewee will continue to prefer pymysql, but you can always subclass and override the `_connect()` method. It's a bit surprising that you have both drivers installed, as you don't wish to use pymysql.

Sorry for disturbance.

I was debugging a issue on production and I installed mycli, then I found that mycli is requiring PyMySQL as its driver and your comment in code.

So I thought, hey, I can make a PR,  then two problems will be solved.

It sounds like a greate idea, right ?:smile:

No disturbance -- I recognize it's a somewhat opinionated stance and the comment was incorrect, which made it worse. Hope you're able to get things working.
Issue: vcvarsall.bat
Author: Rkauff
State: closed
Created: 2018-07-04 20:53:35

When trying to pip install peewee, I'm getting the message “unable to find vcvarsall.bat”

Are there wheels built for peewee?

Weird, I would think you would only see that if you had cython, but how would you have cython without a c compiler?

No wheels.

Currently peewee is built using distutils, so there's no support for wheels at present.
Issue: something like id__in(Django) for peewee
Author: salva-imm
State: closed
Created: 2018-07-04 09:35:41

hi , 
does peewee provide something like this on django ?? 
`Model.objects.filter(id__in=instance['model_ids'])`
the instance['model_ids'] is list of ids ([1,2,3,4,5,6])

i know i can use `on((Model.id==instance['id][0]) & |.......................)`

but my list length isnt fixed ,, can change
i want to get all of them once in single query,,


`.where(MyModel.id.in_([1, 2, 3, 4]))`

http://docs.peewee-orm.com/en/latest/peewee/query_operators.html#id1