akumoth/peewee-issues huggingface dataset:
Issue: Convert readthedocs links for their .org -> .io migration for hosted projects
Author: adamchainz
State: closed
Created: 2016-06-16 20:08:28

As per [their blog post of the 27th April](https://blog.readthedocs.com/securing-subdomains/) ‘Securing subdomains’:

> Starting today, Read the Docs will start hosting projects from subdomains on the domain readthedocs.io, instead of on readthedocs.org. This change addresses some security concerns around site cookies while hosting user generated data on the same domain as our dashboard.

Test Plan: Manually visited all the links I’ve modified.
Issue: Index Creation in Postgresql, create_indexes
Author: mlag
State: closed
Created: 2016-06-15 14:28:47

The first version doesn't work, i.e. creates the table but fails to create an index in postgresql.

```
db = PostgresqlDatabase()
class TestTable(Model):
    class Meta:
        database = db
    id = PrimaryKeyField()
    name = CharField(index=True)
```

whereas the following works: `db.create_indexes(TestTable, ['name'])`

peewee 2.8.1 from pip


I think the "first version" you referred to is missing from the comment. I see your table definition and that calling `db.create_indexes()` creates the indexes, but not the part that may not have been working?


Sorry, nevermind.
After having unrelatedly restarted postgres, it works.
What I meant was, that calling `create_tables()` only created the table, but not the index for the `name` field. I had to call `create_indexes()` separately.


Cool, glad to hear it's working correctly! Thanks for the update.
Issue: Join multiple foreign keys to the same model
Author: rudyryk
State: closed
Created: 2016-06-15 12:24:42

Consider the `Match` model:

``` python
class Match(peewee.Model):
    user_a = peewee.ForeignKeyField(User, index=True,
                                    related_name='matches_a')
    user_b = peewee.ForeignKeyField(User, index=True,
                                    related_name='matches_b')
```

How can we perform query joining **both** fields: `user_a` and `user_b`?

There's a chapter in docs describing the [similar case](http://docs.peewee-orm.com/en/latest/peewee/querying.html#multiple-foreign-keys-to-the-same-model) with the `Relationship` model. But it's different, the model `Match` above can be considered as the symmetrical version of the `Relationship`.

Such query can't be executed:

``` python
matches_for_user = (Match.select()
    .join(User, on=Match.user_a)
    .join(User, on=Match.user_b)
    .where((Match.user_a == user) |
           (Match.user_b == user)))
```

I get an exception:

```
AttributeError: type object 'User' has no attribute 'user_b'
```

And when trying to add `switch(Match)` statement between joins, get another error:

```
peewee.ProgrammingError: table name "t2" specified more than once
```


Aha!

It seems like we can use `alias()` to resolve "conflicts":

``` python
user_a = User.alias()
user_b = User.alias()

query = (Match.select(Match, user_a, user_b)
    .join(user_a, on=Match.user_a)
    .switch(Match)
    .join(user_b, on=Match.user_b)
    .where((Match.user_a == user) | (Match.user_b == user)))
```

Worked for me, but I'm not sure, whether this is intended usage of `alias()` :)


Yep, that's what the `alias()` helper is for -- allowing multiple table references in a single query .
Issue: Requesting item [-1] on the result of select() raises an error.
Author: HughMacdonald
State: closed
Created: 2016-06-15 09:39:24

I've found something that may or may not be considered an issue, but standard python conventions don't work.

If I do:

```
items = MyTable.select()
myItem = items[-1]
```

I get:

``` python
[code]Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 2980, in __getitem__
    res.fill_cache(index)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 2131, in fill_cache
    raise ValueError('Negative values are not supported.')
ValueError: Negative values are not supported.[/code]
```

(These lines are actually 3094 and 2233 in the current master)

I've made a change in my own fork, but I wanted to flag it up here before putting a pull request in.

I've modified the tests too to test that this does the right job, and added some tests for slices using negative indices.

Here's my branch with this in - let me know if it's okay to put in as a PR.

https://github.com/HughMacdonald/peewee/tree/select_results_negative_index

Thanks


Originally this is a design decision I borrowed from Django's ORM. I believe the idea is to that the API should reinforce the concept that the query result wrapper is not a list, and should not be treated as such. That said, I can see how this might cause confusion since the other `__getitem__` behaviors are available. I'll address this with a patch, using yours as a reference. Thanks!

Discussion on django issue tracker: https://code.djangoproject.com/ticket/13089


Thanks again @HughMacdonald


Cool - thanks for that.
Issue: peewee 2.8.1 blob error
Author: rfyiamcool
State: closed
Created: 2016-06-15 02:02:59

in peewee 2.8.1 ,  use BlobField() raise error .

my peewee model example:

```
class ArticleContent(Model):
    article_id = IntegerField()
    content = BlobField()

    class Meta:
        database = db
        db_table= 'mirro_article_content'
```

error:

```
[operation@wx-buzz-zhihu2 extractor_mirro]$ python core/mm.py
Traceback (most recent call last):
  File "core/mm.py", line 30, in <module>
    class ArticleContent(Model):
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 4479, in __new__
    field.add_to_class(cls, name)
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 1081, in add_to_class
    self._constructor = model_class._meta.database.get_binary_type()
  File "/usr/local/lib/python2.7/site-packages/peewee.py", line 3985, in get_binary_type
    return mysql.Binary
AttributeError: 'NoneType' object has no attribute 'Binary'
```

I use peewee 2.8.0 version, not raise error.   Please fix it .


So this error indicates that the `mysql` Python driver is not installed or is not import-able. Here is the code peewee executes at the top of the module:

``` python
try:
    import MySQLdb as mysql  # prefer the C module.
except ImportError:
    try:
        import pymysql as mysql
    except ImportError:
        mysql = None
```

You've triggered the "mysql=None" bit, meaning peewee could not import either driver.
Issue: Extra `t2` while generating query
Author: blueset
State: closed
Created: 2016-06-14 08:56:13

# Model

``` python
class MsgLog(Model):
    master_msg_id = CharField(unique=True, primary_key=True)
    text = CharField()
    slave_origin_uid = CharField()
    slave_origin_display_name = CharField(null=True)
    slave_member_uid = CharField(null=True)
    slave_member_display_name = CharField(null=True)
    msg_type = CharField()
    sent_to = CharField()

    class Meta:
        database = SqliteDatabase('data.db')
```
# Query

``` python
chat_id = 1
q = MsgLog.select().where(MsgLog.master_msg_id.startswith("%s." % chat_id)).order_by(MsgLog.id.desc())
print(q)
```
## Expected result

``` sql
SELECT "t1"."master_msg_id", "t1"."text", "t1"."slave_origin_uid", "t1"."slave_origin_display_name", "t1"."slave_member_uid", "t1"."slave_member_display_name", "t1"."msg_type", "t1"."sent_to" FROM "msglog" AS t1 WHERE ("t1"."master_msg_id" LIKE ?) ORDER BY "t1"."id" DESC ['1.%']
```
## Returned result

``` sql
SELECT "t1"."master_msg_id", "t1"."text", "t1"."slave_origin_uid", "t1"."slave_origin_display_name", "t1"."slave_member_uid", "t1"."slave_member_display_name", "t1"."msg_type", "t1"."sent_to" FROM "msglog" AS t1 WHERE ("t1"."master_msg_id" LIKE ?) ORDER BY "t2"."id" DESC ['1.%']
```


q = MsgLog.select().where(MsgLog.master_msg_id.startswith("%s." % chat_id)).order_by(MsgLog.id.desc())

The problem is you're ordering by MsgLog.id but MsgLog.master_msg_id is the primary key and the model (while it has an "id" attribute), does not consider it a valid field.

To fix this, just change your "order_by()" call to order on a field that's defined on the MsgLog model. Or alternatively, you could make "master_msg_id" have "unique=True" instead of being a primary key.


I think I may make some modifications to Peewee to drop the `id` attribute if a different field is used as the foreign key.


@coleifer deleting other user's comments just because you don't agree with the technical viewpoint is very uncool.


The truth is that I removed your comments because they were not constructive, and I was tired of having to defend aspects of my project because you felt entitled to a technical discussion because you happened to disagree with it. When I pointed to you that the things you disagreed with were documented and obviously intentional (via git log), you suggested I was trying to justify, post-hoc, bad design decisions, and was being an irresponsible maintainer. How is that constructive?


Reopening as I intend to fix.
Issue: Prevent the silent overwriting of model's attributes e.g. "foo_id" when "foo" foreign key added
Author: quantumdark
State: closed
Created: 2016-06-13 22:28:35

In complex structures with a large number of fields easy to miss the collision, especially if it happened with parrent's model attribute.
Considering that the python dict doesn't guarantee the order of items, there are two possibilities:
1. The *_id attr will be overwritten by ObjectIdDescriptor and will no longer available, but field (if the attr is a field) still will be in _meta.fields. This leads to unexpected and floating bugs.
2. ObjectIdDescriptor will be overwritten by custom attribute. Much less harmful and may be unnoticed for a long time.

I think it should explicitly show the possible issue, by throwing an exception.


I'm kind of curious what kind of model structure you'd have to have to run into this kind of thing... I understand that there may be a bug here, but I'm not sure how you would actually encounter it in practice?


I run into it not only once, last time we had save parameters from external API call in the model Inherited from the base model. In this case, the log of actions associated with a subscription entity.
Our deveveloper did'nt know the _id thing and just save it like that:

if v.validate(self.message):
    FortumoLog.create(
       subscription = self.subscription
       **self.message)

So about the stucture: we have several log entities for different APIs and one general subscription model.

Nobody has noticed that one of the parameters is not stored (it just stored for history), but sometimes it drove to crashes our flask-admin, only on the prod with disabled debugging.) with some error 
"Failed to find field for filter: %s"
So I think it may save a few hours for someone.


Fixed 34e56cb887751453680641695bdec1fc6518b2cf
Issue: calling len() on a query should do a database count() rather than load the whole result set into memory
Author: keredson
State: closed
Created: 2016-06-13 20:44:09

currently calling `len()` on a query loads all the records from the database, and then calls `len()` on the resulting list of objects.  this is hugely sub-optimal.

in most instances (ie if it hasn't already been cached, etc.) using `count()` on the database would be both the expected behavior and much more efficient.

i've not seen anything in the documentation that would warn users of the current behavior or any explanation as to why it would do this.


so i have a feeling that i'm going to get a response along the lines of "we preload all the data in case the user wants to access/use it later", similar to the rationale given for why `first()` doesn't use a `limit 1`.  i just would like to point out that both of these are directly counter to the stated "peewee doesn't do things based on what it 'thinks' you might want" policy.


> "peewee doesn't do things based on what it 'thinks' you might want"

Actually I think that current behavior is in line with this policy. If I wanted a database count query, I would've called the `count()` method instead of using `len`. 


It actually was, and has been documented, for some time:

45e096e94afb3c4496ac961f64ada374d27b2529
Issue: should prefetches all happen inside a transaction?
Author: keredson
State: closed
Created: 2016-06-13 15:52:58

i would argue they should, as otherwise there is no guarantee we're getting a read-consistent view of the data.  seems like a simple fix on the face, given that peewee can already handle nesting `atomic()`.


That all would depend on your databases default isolation level. Also recall please how, as a general rule, peewee doesn't use magic or do things based on what it "thinks" you might want. You have the tools to do your prefetching in an atomic block. If you always want this behavior, write a wrapper.


> That all would depend on your databases default isolation level.

i don't think that's true.  two subsequent queries outside of a transaction don't have any consistency guarantees between them at any isolation level on any database.

> peewee doesn't use magic or do things based on what it "thinks" you might want

all APIs are doing what they think you want to a certain degree, in the sense that they have reasonable defaults and behavior.  anyone who makes a single API call to a peewee method would expect it to return an internally consistent collection of objects.  i shouldn't have to know / worry about internal implementation details in order to get correct data, vs. this obvious race condition.  "it's usually right" arguments aside.


> i shouldn't have to know / worry about internal implementation details in order to get correct data

In that case you should wrap your code in an `atomic()` block.


i get that that's an option.  i'm arguing that that should be the default because as is the results are only "usually" right.  :)
Issue: exception when cloning a FK
Author: keredson
State: closed
Created: 2016-06-11 03:52:22

ran into this today...

``` python
class Post(pw.Model):
  title = pw.CharField()
  author = pw.ForeignKeyField(Person)

Post.author.alias('derek').clone()
```

throws:

```
Traceback (most recent call last):
  File "test.py", line 30, in <module>
    Post.author.alias('derek').clone()
  File "/home/derek/projects/coleifer-peewee/peewee.py", line 469, in clone
    inst = self.clone_base()
  File "/home/derek/projects/coleifer-peewee/peewee.py", line 1342, in clone_base
    related_name=self.related_name,
AttributeError: 'ForeignKeyField' object has no attribute 'related_name'
```

looks like `related_name` is only set when `add_to_class` is called.
Issue: Additive order_by
Author: josefdlange
State: closed
Created: 2016-06-08 16:17:40

Spent a few hours debugging something in a query only to find that the implementation of `order_by` on `SelectQuery` is:

```
@returns_clone
def order_by(self, *args):
    self._order_by = list(args)
```

This means that if one was composing a query based on a number of inputs (of which more than one might affect ordering) if `order_by` is called more than once, any prior `order_by` statements will be obliterated. Maybe we could toss in a `extend=False` `kwarg` to the method that, if `True`, will extend the `_order_by` list with the value of `*args`.

If you like that idea I can implement and send you a PR. Wanted to get your take before I did it, however. Right now to get around it I'm directly accessing `_order_by` if it is not `None` and appending/extending as I need.


> Maybe we could toss in a extend=False kwarg to the method that

i can see the use, but a definite +1 on not changing the default behavior if this is done.


Of course. Default behavior is critical, especially when considering that this is likely a heavily-used feature of the ORM.
Issue: Test case for RowID + insert_many bug
Author: jberkel
State: closed
Created: 2016-06-07 21:27:14

This is the problem I mentioned in #971. Not sure how to fix this properly, for now just added a bunch of tests to document the problem. The first two tests pass, only the last one fails.

```
Traceback (most recent call last):
  File "/Users/jan/projects/peewee/playhouse/tests/test_sqlite_ext.py", line 818, in test_insert_many_with_rowid_with_field_validation
    RowIDModel.insert_many([{RowIDModel.rowid: 5, 'data': 1}], validate_fields=True).execute()
  File "/Users/jan/projects/peewee/peewee.py", line 3351, in execute
    cursor = self._execute()
  File "/Users/jan/projects/peewee/peewee.py", line 2747, in _execute
    sql, params = self.sql()
  File "/Users/jan/projects/peewee/peewee.py", line 3319, in sql
    return self.compiler().generate_insert(self)
  File "/Users/jan/projects/peewee/peewee.py", line 1960, in generate_insert
    for row_dict in query._iter_rows():
  File "/Users/jan/projects/peewee/peewee.py", line 3271, in _iter_rows
    validate_field(key)
  File "/Users/jan/projects/peewee/peewee.py", line 3261, in validate_field
    raise KeyError('"%s" is not a recognized field.' % field)
KeyError: '"<playhouse.sqlite_ext.RowIDField object at 0x102dc81d0>" is not a recognized field.'
```


There's some chicanery going on with Peewee in regards to implementing the `rowid` field. I'll take another look at the implementation and see if I can clean it up.


Thanks! Yes, the field gets removed here: 
https://github.com/coleifer/peewee/blob/master/playhouse/sqlite_ext.py#L90

Do extension tests not get run by the CI? In theory the build should have failed. (I had to add `from playhouse.tests.test_sqlite_ext import *` to `tests.py` get them to run locally).


> Do extension tests not get run by the CI?

They should be getting run. The .travis.yml file specifies `./runtests.py -a` which includes all extensions (except those for which a dependency is missing, which shouldn't be the case for the sqlite extension tests).


Yes, looks like they ran there and passed. I suspect it's because some code paths don't get executed on the CI, for instance:

``` python
    insert_many = sqlite3 and sqlite3.sqlite_version_info >= (3, 7, 11, 0)
    # ...
    def execute(self):
        insert_with_loop = (
            self._is_multi_row_insert and
            self._query is None and
            self._returning is None and
            not self.database.insert_many)
        if insert_with_loop:
            return self._insert_with_loop()
```

Anyway, with the lastest master everything works as expected and I can remove my workaround, thanks!
Issue: Impl custom alias names
Author: keredson
State: closed
Created: 2016-06-07 20:24:05

implements https://github.com/coleifer/peewee/issues/972


Oof...this implementation feels very fragile and hacky, and I don't think the extra lines of code are worth the feature of being able to specify an alias name.

Going to pass on this for now. The 3.0 stuff I'm working on can do this as a matter of course, so it'll land then.


+1 on it for the rewrite.  including the alias equality?
Issue: allow the user to specify the table alias name
Author: keredson
State: closed
Created: 2016-06-07 20:23:41

currently this:

``` python
UA = User.alias()
UA.select().where(UA.username=='derek')
```

generates this:

``` sql
SELECT "t1"."id", "t1"."username" FROM "users" AS t1
WHERE ("t1"."username" = ?)
```

we should allow the user to optionally specify the table alias.  example:

``` python
UA = User.alias('u1')
UA.select().where(UA.username=='derek')
```

which would generate:

``` sql
SELECT "u1"."id", "u1"."username" FROM "users" AS u1 
WHERE ("u1"."username" = ?)
```

optionally, a little syntax sugar that i prefer:

``` python
UA = User.as_('u1')
UA.select().where(UA.username=='derek')
```

also identical aliases in the same query are handled appropriately, so aliases can be used in one-liners:

``` python
User.as_('u1').select().where(User.as_('u1').username=='derek')
```


In rewriting the SQL generation, I've got this feature covered. The implementation of this in peewee 2.8 just feels a bit too hacky to me to want to merge.
Issue: Infer alias when only one interpretation
Author: keredson
State: closed
Created: 2016-06-07 17:56:27

likewise for the alias matching version of that same basic code change for https://github.com/coleifer/peewee/issues/965

not that this is now valid:

``` python
User.alias().select().where(User.username=='charlie')
```

but this is still not:

``` python
User.alias().select().join(User.alias(), on=User.username=='charlie')
```
Issue: Throw error no alias
Author: keredson
State: closed
Created: 2016-06-07 16:30:01

for https://github.com/coleifer/peewee/issues/964.  i know you said no interest, but logging here anyway since i was doing it for my own fork regardless.

the `_join` wound up being unnecessary as far as i can tell.  i couldn't find a path in which it didn't hit the alias_map.

also added a unit test for your concat concern.

changing the unit test because it genned `t4` instead of `t3` i didn't like, but in the end the alias name used doesn't matter, and because if the way the counters are being passed around i'm not sure how fixable it is to keep the `t3`.  (and in my branch all that's gone anyway, so the personal ROI is low to continue debugging)


Thanks for sharing the changes. Closing as wontfix per earlier discussion.
Issue: pwiz generates invalid model name for tables starting with numbers
Author: kennes913
State: closed
Created: 2016-06-07 16:12:19

Modified copy-paste of process to produce error:

```
$ python -m pwiz -e mysql XXXXX -H XXXXX -P -u dbadmin -t 1337_table > db_models.py
Password: 
$ python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import db_models
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/file/location/db_models.py", line 12
    class 1337Table(BaseModel):
             ^
SyntaxError: invalid syntax
>>> 
```
Issue: primary_key on sqlite seems to make table not saveable
Author: daveselinger
State: closed
Created: 2016-06-07 07:42:13

Potentially related to my other issue log. I've found that if I set a column to primary_key=true (I've tried this on a CHAR and UUID columns), I don't get any errors, but the "save()" method returns 0:

class SimpleTest(BaseModel):
  name = CharField(primary_key=True)
  name2 = CharField()

simple = SimpleTest(name="Charlie", name2="Bucket")
print("Saving %d simples" % simple.save())

running this yields:

Saving 0 simples

No compile time errors or other messages.

I've then logged into the sqlite file directly and not surprisingly, I see no rows when selecting from the table.

Apologies if these are already logged--I did some searching and all I found was a comment from you about a year ago discussing the "primary key as only column" on a forum post...


force_insert=True
Issue: primary_key as only column on sqlite
Author: daveselinger
State: closed
Created: 2016-06-07 07:39:14

Hey there--great library, btw.

I'm running into a couple of issues (working directly on head/master):
1. A table with only a primary key as the sole field:

class SimpleTest(BaseModel):
  name = CharField(primary_key=True)

When I call:
  simple = SimpleTest(name="Charlie")
  print("Saving %d simples" % simple.save())

I get:

  File "./peewee/peewee.py", line 4862, in save
    rows = self.update(*_field_dict).where(self._pk_expr()).execute()
  File "./peewee/peewee.py", line 3224, in execute
    return self.database.rows_affected(self._execute())
  File "./peewee/peewee.py", line 2748, in _execute
    return self.database.execute_sql(sql, params, self.require_commit)
  File "./peewee/peewee.py", line 3599, in execute_sql
    self.commit()
  File "./peewee/peewee.py", line 3423, in __exit__
    reraise(new_type, new_type(_exc_args), traceback)
  File "./peewee/peewee.py", line 132, in reraise
    raise value.with_traceback(tb)
  File "./peewee/peewee.py", line 3592, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: near "WHERE": syntax error


force_insert=True
The logging helped a ton--didn't find that till I looked at https://github.com/coleifer/peewee/issues/961 but super helpful.

Thx.