akumoth/peewee-issues huggingface dataset:
Issue: Namespace collision with another pip module
Author: n3storm
State: closed
Created: 2016-02-01 17:50:59

Looks like if you install peewee and PluginBase from pocoo then python, pluginbase and peewee goes crazy. Took me 1 hour before realizing this.

http://pluginbase.pocoo.org/

```
   from goingpostal.contrib.group.models import Member, Queue, Archive, database
  File "/usr/local/lib/python2.7/dist-packages/pluginbase.py", line 404, in plugin_import
    fromlist, level)
  File "/usr/local/lib/python2.7/dist-packages/goingpostal/contrib/group/models.py", line 1, in <module>
    from peewee import *
  File "/usr/local/lib/python2.7/dist-packages/pluginbase.py", line 404, in plugin_import
    fromlist, level)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 147, in <module>
    import sqlite3
  File "/usr/local/lib/python2.7/dist-packages/pluginbase.py", line 404, in plugin_import
    fromlist, level)
  File "/usr/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/dist-packages/pluginbase.py", line 404, in plugin_import
    fromlist, level)
ImportError: No module named dbapi2
```

Don't know if this has any solution without breaking user API, but I wanted to write it down so somebody else my find it helpful before becoming bald pulling hair.

Cheers.


WTF... I have to say this does not appear to be a peewee issue to me, I'm at a loss. Peewee doesn't do anything weird regarding imports so I'm laying this at the doorstep of the thing that's jacking with imports and doing dark magic.


I have peeked into peewee repo and looks like you are right, so pluginbase must be messing directly with python. FYI, before using peewee this app was running ok using tinydb for storage (and pluginbase).
I will create an issue at pocoo repo.

Thanks Charles.


Well, for whatever reason it's messing up the system path. Peewee import SQLite3, which for some crazy reason goes through pluginbase? But sqlite3 is a part of the standard lib...so that's probably the source of the issue.
Issue: Documentation Idea: Alternative way to using same model with multiple databases
Author: shirish93
State: closed
Created: 2016-01-30 00:16:17

Hey @coleifer , 

I've asked this question in the mailing list, but I believe I've come up with a more elegant idea to use a model with multiple databases. The user defines a Model, but encapsulates it in a function. The function takes the name of the database to be used, and returns the Model that uses the specified database. If this question comes up again, it might be worth it to point people in this direction.

Sample:

```
global dbName  #Put this dummy value here. It'll be overwritten anyway
def makeModel(dbName):
   global dNname
  class ModelName:
    #classDef goes here.

    class Meta:
      database = dbName

  return ModelName
```

To use it:

```
model_for_db1 = makeModel(dbName1)
model_for_db2 = makeModel(dbName2)

query = model_for_db1.select().where(model_for_db1.field_1 == 'value_1')
```

What is your view with regards to this? Please feel free to remove/close the issue if you feel this does not deserve to be in the 'issues' section. I had a particularly long process trying to find a way to make this work, so I imagine others might gain from it. : )-

Update 2: Added a global variable to fix it. It appears to work now.

Update 1: Doesn't work as expected. Any sort of specific query doesn't seem to work. I'll close the issue after further testing. Sorry for the false alarm.


I don't mean to offend, but this is just basic Python. I don't think it's necessarily worth documenting, but thank you for sharing your thoughts on the issue.
Issue: Custom ManyToManyQuery?
Author: yohanboniface
State: closed
Created: 2016-01-29 09:24:12

I'd like to implement a custom result wrapper for my ManyToManyQuery (already done a custom [SelectQuery](https://github.com/etalab/ban/blob/master/ban/db/model.py#L6)), but currently the `ManyToManyQuery` is hardcoded in the [ManyToManyFieldDescriptor](https://github.com/coleifer/peewee/blob/master/playhouse/fields.py#L168) (and in `ManyToManyQuery.clone`), which itself is hardcoded in [ManyToManyField](https://github.com/coleifer/peewee/blob/master/playhouse/fields.py#L116).

Would you consider a PR that would make that more customizable? I'm thinking about making those class attributes. Something like this:

``` python
# Or maybe better at Field level directly?
class ManyToManyField(Field):
    descriptor_class = ManyToManyFieldDescriptor

    def add_to_class(self, model_class, name):
        …
        setattr(model_class, name, self.descriptor_class(self))
        …

class ManyToManyFieldDescriptor(FieldDescriptor):
    query_class = ManyToManyQuery
    …
```

Thoughts?


See f1988db for the changes.


Thanks for the quick fix @coleifer!

That fixes the main issues, though I'd still need to overwrite and copy-paste [`ManyToManyFieldDescriptor.__get__`](https://github.com/coleifer/peewee/blob/master/playhouse/fields.py#L169) and [`ManyToManyQuery.clone`](https://github.com/coleifer/peewee/blob/master/playhouse/fields.py#L189), any idea to prevent that too?


Replacing:

```
    def clone(self):
        query = ManyToManyQuery(
            self._instance,
            self._field_descriptor,
            self.model_class)
        query.database = self.database
        return self._clone_attributes(query)
```

by

```
    def clone(self):
        query = self.__class__(
            self._instance,
            self._field_descriptor,
            self.model_class)
        query.database = self.database
        return self._clone_attributes(query)
```

would help without hurting, imho :)


You could probably override `clone()` in your subclass. I think that would
actually be more appropriate.

On Tue, Feb 2, 2016 at 4:48 AM, Yohan Boniface notifications@github.com
wrote:

> Replacing:
> 
> ```
> def clone(self):
>     query = ManyToManyQuery(
>         self._instance,
>         self._field_descriptor,
>         self.model_class)
>     query.database = self.database
>     return self._clone_attributes(query)
> ```
> 
> by
> 
> ```
> def clone(self):
>     query = self.__class__(
>         self._instance,
>         self._field_descriptor,
>         self.model_class)
>     query.database = self.database
>     return self._clone_attributes(query)
> ```
> 
> would help without hurting, imho :)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/coleifer/peewee/issues/838#issuecomment-178504560.


> You could probably override `clone()` in your subclass. I think that would actually be more appropriate.

Why is that more appropriate?
IMHO, using `self.__class__` in peewee code does not hurt and make it more generic, preventing users to override `clone` just because they have a custom ManyToManyQuery class.
Issue: fix for issue #836
Author: ronyb29
State: closed
Created: 2016-01-28 03:07:29

fix for issue #836 


This is not the intended behavior of last_insert_id() -- the intended behavior is to return the last inserted `rowid`. For auto-incrementing primary key fields, this works as expected.

If you have a user-defined primary key, then presumably you know what it is beforehand so there's no need to return it (and issue an extra query into the bargain).


the thing is, currently, Model.save() sets the pk to whatever the database reports as `last_insert_id`, so when you make relationships with instances created in code (not retrieved from the DB) the foreign key for ForeignKeyField are broken.


Peewee only automatically sets the primary key for auto incremented integers. If you have a uuid primary key you presumably set that in the application. I'm confused as to what issue you're actually experiencing.


Perhaps I have failed to convey the issue. Currently it is setting the PK regardless of its kind to whatever `last_insert_id` returns. Please take a look at this tests.

``` python
from uuid import uuid1, UUID

from playhouse.apsw_ext import *
from playhouse.tests.base import ModelTestCase

db = APSWDatabase(':memory:')

class BaseModel(Model):
    class Meta:
        database = db

class Batch(BaseModel):
    id = UUIDField(primary_key=True, default=uuid1)

class Document(BaseModel):
    id = UUIDField(primary_key=True, default=uuid1)
    batch = ForeignKeyField(Batch, related_name='documents', to_field='id')

class APSWPKTest(ModelTestCase):
    requires = [Batch, Document]

    def test_non_int_pk_saving(self):
        batch = Batch()
        self.assertIsInstance(batch.id, UUID)  # passes
        batch.save(force_insert=True) # Will save the right id in the DB, but will modify the instance and set it to the rowid
        self.assertIsInstance(batch.id, UUID)  # fails, contains the ROWID returned by last_insert_id

    def test_non_int_pk_relationships(self):
        batch = Batch.create()
        self.assertIsInstance(batch.id, UUID)  # fails, contains the ROWID returned by last_insert_id

        doc = Document(batch=batch)
        self.assertIsInstance(doc.id, UUID)  # passes
        doc.save(force_insert=True)
        self.assertIsInstance(doc.id, UUID)  # fails, contains the ROWID returned by last_insert_id
```

That is because Model.save actually sets the instance pk field after inserting

``` python
class Model(with_metaclass(BaseModel)):
    ...
    def save(self, force_insert=False, only=None):
        ...
        self._populate_unsaved_relations(field_dict)
        if pk_value is not None and not force_insert:
            ...
        elif pk_field is None:
            ...
        else:
            pk_from_cursor = self.insert(**field_dict).execute()
            if pk_from_cursor is not None:
                pk_value = pk_from_cursor
            self._set_pk_value(pk_value)
            rows = 1
        self._dirty.clear()
        return rows
```

Perhaps a better solution than what I originally proposed would be for Model.save to only set the PK if it was an instance of PrimaryKeyField
Issue: apsw and non integer PK
Author: ronyb29
State: closed
Created: 2016-01-28 03:04:41

the apsw driver currently returns the last [rowid](https://www.sqlite.org/c3ref/last_insert_rowid.html) instead of the las inserted PK, this causes problems for any non int pk (like a UUID)


https://github.com/coleifer/peewee/pull/837#issuecomment-175938671


I see the issue now. Normally, last_insert_id() checks first that the model has an auto incrementing pk.

``` python
def last_insert_id(self, cursor, model):
        if model._meta.auto_increment:
            return cursor.lastrowid
```

This check is missing in the apsw driver, though. Apologies for the confusion on my part. The fix will just need to add this check to the apsw implementation.
Issue: Signal Support and Identifying Models
Author: jimjkelly
State: closed
Created: 2016-01-26 13:22:59

Hi - so I have a setup where I've abstracted my models out into a pip installable package so that they can be used in multiple places in my infrastructure.  This works great except when trying to get signals working.  Essentially, things aren't working because of the check here:

https://github.com/coleifer/peewee/blob/master/playhouse/signals.py#L39

Namely - the instance class is being referred to as `foo.Model` where as outside (i.e. when I wire up a signal handler and give it a sender) it's referred to as `models.foo.Model`.  Does that make sense?  If not I can try and put together a small example.

Given this problem - is it possible instead compare models by name?  Or is it not expected that those necessarily be globally unique?  Or is there something else obvious maybe I'm missing here to resolve that problem?


This is not a peewee problem, but a problem with the way you're importing and managing your models.

Python loads your imported modules into memory and you're double-loading them (foo.Model and models.foo.Model). Python sees this as two different modules and in fact they are essentially two different models. What's scarier is you might have two different databases.... I'd strongly consider reorganizing your code as this type of bug can produce weird issues like the one you've posted.


Given the way the models get configured I don't have to worry about two different databases, but I'll be a bit more vigilant about importing them the same way.  Thanks!
Issue: Python 3 PasswordField Bug AttributeError encode
Author: mosquito
State: closed
Created: 2016-01-24 23:11:56

The class PasswordHash was inherited from bytes. For python 3 it's wrong because python's 3 bytes not provide "encode" method and this code:

```
class PasswordField(BlobField):
    def __init__(self, iterations=12, *args, **kwargs):
        if None in (hashpw, gensalt):
            raise ValueError('Missing library required for PasswordField: bcrypt')
        self.bcrypt_iterations = iterations
        self.raw_password = None
        super(PasswordField, self).__init__(*args, **kwargs)
```

fails with AttributeError.

For me I make a fix:

```
from playhouse.fields import PasswordField as PasswordFieldBase, PasswordHash


class PasswordField(PasswordFieldBase):
    def db_value(self, value):
        if isinstance(value, PasswordHash):
            value = value.decode("utf-8")
        return super(PasswordField, self).db_value(value)
```


I believe the code was correct, but needed to just check for a unicode type rather than assuming one.
Issue: Feauture proposal for playhouse: Option to store binary blobs in compressed form
Author: shirish93
State: closed
Created: 2016-01-24 18:14:57

Hello,

I've been working with peewee lately, and since my data is a lot of repeating text information, it's being compressed. It would seem that such a feature might be useful for everyday peewee users too, considering that most common compression algorithms have much shorter decompression times than compression times (and assuming that most peewee users do database reads more often than db writes).

Would such an addition be aligned with your vision of the project? If so, I would start working on it, and the unit tests. There might be a little guidance required with the internal workings of peewee, but besides that, not too much. Not sure if the proposal would fit with playhouse, or core peewee either.

Here's an example usage of the proposed feature:

During model definition:

`
myLargeContiguousString = BlobField(compressed= True)
`

for access:

`
Model.myLargeContiguousString.select().get()
`

would return a decompressed version.

I am not sure if returning raw compressed strings would be worth it, and it would be upon you to decide on the syntax for such a feature, but such a feature would be made available too, as per your decision.

A good documentation would be needed to tell people about the encodings and as such (my proposal: default to 'utf-8' unless they ask for something else), and that  they would gain the most by compressing only larger strings, etcetera, maybe even a choice between bz2 and zlib (both available in Python core) but that's about the design decisions I can think of. The work of writing up the documentation would obviously be on me, with your guidance.

Thoughts?


You should check the `CompressedField`. It supports zlib and bz2.

http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#CompressedField


Oh dear! Totally missed out on that. Thanks!
Issue: Support for RethinkDB
Author: foxx
State: closed
Created: 2016-01-23 13:26:21

I'm looking at the possibility of adding support for [RethinkDB](https://www.rethinkdb.com/) into Peewee. It recently became stable and so far looks absolutely [amazing](http://rob.conery.io/2015/04/17/rethinkdb-2-0-is-amazing/). After looking at some of the other extensions in playhouse, this seems to be a straight forward task. However I'm struggling to decide what components should be abstracted, for example, there's a lot of functionality relating to [sharding/replication](https://www.rethinkdb.com/docs/sharding-and-replication/), which might make it difficult to use as a generic layer in Peewee. There's a cheat sheet [here](https://www.rethinkdb.com/docs/sql-to-reql/javascript/) showing comparison of SQL to ReQL.

In my use case, I've built several modules with auditing and metrics capabilities, which are re-usable between applications. For some users, they might suffice with just using Postgres but in other situations like mine, these need to scale to billions of rows. If I was able to add support into Peewee for RethinkDB, then I can reduce the amount of database specific work (e.g. having to specifically add support for both Peewee and RethinkDB API).

Anyway I'd be interested to hear your thoughts on whether RethinkDB would be a suitable candidate for playhouse. I'll be working on a prototype in `peewee-extras` over the next month, so any thoughts/feedback/hints would be much appreciated. It will likely be another 6-8 weeks until `peewee-extras` is stable and clean with documentation though (it's still a building site at the moment!)


Sure, a rethinkdb backend would be pretty neat!


Okay awesome, I'll let you know once I have something ready for review (an early preview would be appreciated, as you can spot any design flaws much sooner!)


What's going on with this issue? You know, as I'm thinking about it, this would probably benefit from being a separate package. To that effect, I think I will close this out.


Apologies for the lack of replies on this, this was shelved in favour of getting some other functionality working first, however I do plan on coming back to this eventually. I'll create a separate package for this, and we can always consider merging it into playhouse at a later date.
Issue: Unable to install in Windows + Python3
Author: shirish93
State: closed
Created: 2016-01-22 19:12:27

When trying to install peewee in Windows (with WinPython), I get the following error:

`
copying playhouse\__init__.py -> build\lib.win32-3.4\playhouse
running build_ext
cythoning playhouse/speedups.pyx to playhouse\speedups.c
building 'playhouse._speedups' extension
error: Unable to find vcvarsall.bat
`

This SO thread http://stackoverflow.com/questions/33834113/fail-to-install-peewee-under-winpython suggests that the solution would be as follows:

`
init_sqlite_ext is the wrong PyMODINIT_FUNC to call on Python 3. It should be PyInit__sqlite_ext
`

Would you be able to test this, and make the necessary change? (I could put in a pull request for it, if it is so desired...)


Just saw https://github.com/coleifer/peewee/issues/763 . Closing this as it is a duplicate.
Issue: Support for SmallInteger
Author: foxx
State: closed
Created: 2016-01-21 11:09:40

Doesn't appear to be a cross database compatible field for representing `smallint`.

[Postgres](http://www.postgresql.org/docs/9.1/static/datatype-numeric.html), [MySQL](https://dev.mysql.com/doc/refman/5.5/en/integer-types.html) and [SQLite](https://www.sqlite.org/datatype3.html) seem to handle `smallint` in the same way, with the exception of SQLite casting all `smallint` to `int`.

Any objections for adding this to the core? Happy to send a PR if needed


No, I'll whip up an implementation, seems useful.


Beautiful, thank you!
Issue:  too many clients already
Author: strongant
State: closed
Created: 2016-01-21 07:18:11

OperationalError: FATAL:  sorry, too many clients already
FATAL:  sorry, too many clients already


This just means your database is rejecting the connection because there are too many clients connected.


I don't see how this is a peewee issue. Probably check your connection management code, maybe you're not properly closing connections.

In the future please provide some shred of useful information in a ticket, otherwise you're just wasting my time and yours.
Issue: Support for sophisticated migrations
Author: foxx
State: closed
Created: 2016-01-20 11:10:59

As mentioned in [Peewee Docs](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#schema-migrations);

> This engine provides the basis on which a more sophisticated tool could some day be built.

Other framework implementations, such as South and Django Migrations, are quite infuriating to use so I'd be interested to hear your thoughts on how such a tool would be constructed @coleifer.

Would this tool be within the scope of Peewee core or playhouse? If so, are there any plans on the roadmap to start working on such a tool? I can put a certain amount of time towards helping out with this, so if it needs volunteers then count me in.


I've always maintained the view that anything beyond the APIs currently available would be best suited for a third party package, and I believe maybe one or two such exist. Any type of automatic schema migrator seems like an invitation to disaster and will never be included as part of peewee.


Thanks for the quick reply, sounds good, I'll have a think about how this might be implemented as a third party module and will follow up for feedback at a later date.


@foxx Have a look to https://github.com/klen/peewee_migrate
Issue: [Feature Req] Ability to order by column with NULL preference
Author: alexlatchford
State: closed
Created: 2016-01-20 05:21:20

Currently you can use the `order_by` metaclass property to assign default sort ordering. I have a requirement to be able to sort a column and also have it respect sort order for nulls, I've been looking through the docs and can't seem to find support for it so I'd like to propose a syntax for potentially adding the feature in lieu of a PR. Something like:

```
class MyModel(Model):
    start = DateTimeField(null=True)
    stop = DateTimeField(null=True)

    class Meta:
        order_by = (('-start', 'nulls'), ('nulls', '-stop'))
```

This extends the current syntax and uses it's tuple position to signify NULLS FIRST/LAST. I'm not 100% happy with it but it'll do the job for sure, let me know your thoughts.

PS. I'm using Postgres which supports this operation, not sure how the other supported databases handle this interaction, for reference: http://www.postgresql.org/docs/current/static/queries-order.html


I do not plan on supporting that as a Meta option, but you can write a wrapper method on your model class that sorts the table correctly. You can pass arbitrary SQL nodes into order_by(), so:

MyModel.select().order_by(SQL('the_col nulls last'))


Awesome! Thought it might be possible but wasn't sure how, thanks Charles for the answer :smile:
Issue: Using ArrayField w/ BinaryJSONField (PostgreSQL)
Author: tantona
State: closed
Created: 2016-01-18 02:17:53

I was unable to use these two postgres extensions in conjunction.  I scoured the internet for an answer but all I found was http://stackoverflow.com/questions/18099589/postgres-array-of-json-avoid-casting

My solution (which is a hack) was this 

```
def postgresJsonArrayString(array):
    return "".join(["{", ", ".join(["".join(["\"", json.dumps(m).replace('"', "\\\""), "\""])
                    for m in array]), "}"])
```

I realize it's gross but it does the job for now! I was going to circle back on this but at least this gets me back on track!

love this library - keep up the great work!


Well...since JSON fields themselves can contain arrays, I'm not sure how crucial I feel it is to address this use-case. If peewee is not supporting this, then I'll just defer and mark this wontfix.
Issue: PaginatedQuery should not throw 404 on empty query
Author: econne01
State: closed
Created: 2016-01-16 18:32:21

Working through the [Flask Blog in an hour or less tutorial](https://github.com/econne01/peewee.git), I noticed 404 errors are being thrown when the database is empty.

I think `PaginatedQuery` class should gracefully handle an empty query and return empty set when the query returns no results.


The behavior is intended


I see. Ok, thanks