akumoth/peewee-issues huggingface dataset:
Issue: Easy Python 3 compatibility
Author: zielmicha
State: closed
Created: 2013-02-11 21:46:53

This library is nearly compatible with Python 3. Just add:

```
def __hash__(self):
    return id(self)
```

to Leaf and run 2to3. Only FieldTypeTestCase test fails. It may be worth mentioning in README.


No, there is _much_ more needed to make Peewee Python 3 compatible. `Model` uses a metaclass, and Python 3 changed how those were defined. Some judicious application of the `six` library would help there.


I think I'm finally serious about py3 compatibility... I'll start a fix.


With the release of 2.1.0, peewee's test suite and extra modules are working with 3.2 and 3.3!
Issue: Add support for database modifications
Author: hobeone
State: closed
Created: 2013-02-08 18:31:23

I'd like to implement some sort of schema migration support for the app I'm working on.  It would be awesome if Peewee could support some of the basic modifications people would want to do.

Things like:
- add, rename and remove columns
- add and remove indexes (it already supports add I think)
- alter column defaults, NOT NULL etc

My pony request would be to have a standard peewee way of doing this ala the ruby ActiveRecord migrations or something like that.


Yep, I agree that would be nice.  I'm working on refactoring the way DDL is generated currently -- would like to get that in place and probably build any schema changing stuff on top of it.


Sweet.

Thanks for making peewee awesome!

On Fri, Feb 8, 2013 at 3:15 PM, Charles Leifer notifications@github.comwrote:

> Yep, I agree that would be nice. I'm working on refactoring the way DDL is
> generated currently -- would like to get that in place and probably build
> any schema changing stuff on top of it.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/issues/162#issuecomment-13319239..

## 

Daniel Hobe hobe@gmail.com


I try!  I will update this ticket with progress


So I have made two or three attempts at creating a nice API for representing DDL and every time it feels more cumbersome and clunky than just writing the `create` statements by hand.  I may have to scale back my goal and just try to provide a nice api for doing basic schema changes.  Proposals are welcome


Added some stuff in 180bf0b8e0dd22e87b216c59de4da68aa2ed1683 with some basic postgresql support.

SQLite doesn't handle much more than adding columns, and MySQL uses a completely different syntax than Postgresql.  Maybe this code could be refactored or retooled into something usable on different backends.  For sqlite it will be necessary to create temp tables and rename rather than altering or dropping columns, since this is not supported.


Cool, I'll take a look.

I've looked into how a few other orms deal with this and it does seem like
each db driver has to support their own thing.

On Sun, Mar 17, 2013 at 8:46 PM, Charles Leifer notifications@github.comwrote:

> Added some stuff in 180bf0bhttps://github.com/coleifer/peewee/commit/180bf0b8e0dd22e87b216c59de4da68aa2ed1683with some basic postgresql support.
> 
> SQLite doesn't handle much more than adding columns, and MySQL uses a
> completely different syntax than Postgresql. Maybe this code could be
> refactored or retooled into something usable on different backends. For
> sqlite it will be necessary to create temp tables and rename rather than
> altering or dropping columns, since this is not supported.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/issues/162#issuecomment-15037985
> .

## 

Daniel Hobe hobe@gmail.com
Issue: save() doesn't remember the INSERT/UPDATE state correctly
Author: koblas
State: closed
Created: 2013-02-07 20:11:50

If you have an assigned ID to a primary key Peewee doesn't remember the get_or_create state of the Model.

```
class GoogleThing(BaseModel):                                                                                                                                                   
    google_id    = peewee.CharField(max_length=40, primary_key=True)                                                                                                          
    name          = peewee.CharField(max_length=32, null=False)            
```

Workaround is to write code this - note the "force" boolean that's being tracked around.

```
    try:                                                                                                                                                                
        model = GoogleThing.get(GoogleThing.google_id==google_id)                                                                                                             
    except Channel.DoesNotExist:                                                                                                                                        
        force = True                                                                                                                                                    
        model = GoogleThing(google_id=google_id)                                                                                                                          
        model.name = "Some Interesting String"

    # random other statements... Then

    model.save(force_insert=force)                                                                                                                                                                                                                                                             
```

The problem is that save() only looks for the presense of the ID field to determine INSERT/UPDATE rather than knowing how the model was created.


This is documented: http://peewee.readthedocs.org/en/latest/peewee/models.html#non-integer-primary-keys

The reason a model behaves this way is that it is much more simple and straightforward to simply look for a primary key than to try and track the state of a model.


This bug would also apply if you sharded your ID space and used Snowflake to generate the IDs - which we are doing on some other tables.  So it's not just limited to non-integer keys.


Yes, this is somewhat discussed: http://peewee.readthedocs.org/en/latest/peewee/cookbook.html#bulk-loading-data-or-manually-specifying-primary-keys
Issue: Problems using order_by
Author: andmart
State: closed
Created: 2013-02-05 20:03:17

When running the script below, I got  

sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 0 supplied.

It seems that orm is not filling the resultant sql with fields to order.

When debugging, printing Ms of line 27 outputs :

<class '__main__.M'> SELECT t1."id", t1."number" FROM "m" AS t1 ORDER BY ? []
###### ######## Script to reproduce error

```
from peewee import IntegerField, Model, SqliteDatabase
import os
DBNAME = 'test.db'
db = SqliteDatabase(DBNAME)

class M(Model):
    number = IntegerField()
    class Meta:
        database = db

def init():
    # create table
    if not os.path.exists(DBNAME):
        M.create_table()

    #populate

    m1 = M()
    m1.number = 1
    m1.save()

    m2 = M()
    m2.number = 2
    m2.save()

def query():
    Ms = M.select().order_by('-number',)
    for m in Ms:
        print m

if __name__ == '__main__':
    init()
    query() 
```


I replaced the line 786 by 

```
parts.append('ORDER BY %s' % tuple(_)) 
```


I can understnad your confusion.  Peewee uses "-<field name>" to specify ordering in a models "Meta" attributes.

However, when querying you will use a different syntax:

``` python

M.select().order_by(M.number.desc())
```

This is covered in the docs: 
- http://peewee.readthedocs.org/en/latest/peewee/cookbook.html#sorting-records
- http://peewee.readthedocs.org/en/latest/peewee/api.html#SelectQuery.order_by


Thanks, man. Sorry about the false issue.
Issue: Makes script useable with legacy sqlite dbs which have no foreign keys
Author: jharmn
State: closed
Created: 2013-02-02 19:31:03

I had this issue specifically on this db: https://github.com/jasonh-n-austin/WheelsREST/blob/master/wheels.db
This is an export from an older MS Sql Server db, which did not have any foreign keys on tables etc. This error handling for foreign keys made it instantly quit with 'Unable to read table definition for "rides"'. 
Handling this quietly doesn't seem to hurt anything, as it generated models for all of the tables, it just didn't apply any treatment for fks.
Issue: Alled for old style mysql no FK relation
Author: rosscdh
State: closed
Created: 2013-02-01 23:11:45

Hi there, we have a very old db that has no FKs defined.

Currently its not possible (or seems not) to perform joins on elements that have no FK

This is a minor patch for it.. 


Even though your db may not use foreign key constraints, you can (and should) still use the `ForeignKeyField` to access them, unless I misunderstand the issue.

Perhaps you could send me some models that you are using, or how you use this patch.


HI Charles, Thanks for the Feedback.. let me try the FKField and Ill let
you know how it goes.. Apologies I dont know how, but I did miss this in
the documentation.

All the best
Ross

On Sat, Feb 2, 2013 at 1:52 AM, Charles Leifer notifications@github.comwrote:

> Even though your db may not use foreign key constraints, you can (and
> should) still use the ForeignKeyField to access them, unless I
> misunderstand the issue.
> 
> Perhaps you could send me some models that you are using, or how you use
> this patch.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/pull/158#issuecomment-13021749.


I rewrote some of the join code to be a little more flexible in case you want to join on something that may not be a foreignkeyfield.  Commit d1e5a4845383b4

The way you might do it is:

``` python

class User(Model):
    username= CharField()

class Comment(Model):
    username = CharField()
    message = CharField()

# Say you want to join comments with their user
Comment.select(Comment, User).join(User, on=(Comment.username == User.username))
```


Hi there!

That is perfect! and also more intuitive! Thank you very much.

I must say Im quite enjoying using peewee. It feels a little ungainly on
larger queries but that is a symptom all ORMs suffer from I think.

All the best!

Ross

On Sun, Feb 3, 2013 at 4:06 AM, Charles Leifer notifications@github.comwrote:

> I rewrote some of the join code to be a little more flexible in case you
> want to join on something that may not be a foreignkeyfield. Commit
> d1e5a48 https://github.com/coleifer/peewee/commit/d1e5a4845383b4
> 
> The way you might do it is:
> 
> class User(Model):
>     username= CharField()
> class Comment(Model):
>     username = CharField()
>     message = CharField()
> 
> # Say you want to join comments with their userComment.select(Comment, User).join(User, on=(Comment.username == User.username))
> 
>  —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/pull/158#issuecomment-13041552.
Issue: 2 issues at once
Author: ivan-kleshnin
State: closed
Created: 2013-02-01 10:33:05

Issue #1

`get()` adds `limit(1)` to `SelectQuery`. 
When we `join()` something 1-to-n, `get()` breaks selection by limiting those joined items

Issue #2 

`DeleteQuery` doesn't have `limit()` method


Do you mind putting an example of #1 -- particularly what you expect vs what you get.


I also see that Postgresql does not support limit/order by on delete and update queries, and SQLite only supports them if it was compiled with `SQLITE_ENABLE_UPDATE_DELETE_LIMIT` -- what db do you use out of curiosity?

Another option is to do the delete and select the ids to remove in a subquery where you apply the limit and ordering.


#1 nevermind, sorry. I forgot that peewee can preload with join only master-models.
#2 I use MySQL. I didn't know that PgSQL does not support that. I'ts kinda disappointing when you DELETE without LIMIT and with mistyped WHERE clause

Thus close this thread, pls.
Issue: sub query in select
Author: nesizer
State: closed
Created: 2013-01-30 18:03:25

is there a way for peewee to support sub queries like this one

SELECT g.*,
(SELECT SUM(s.viewers) FROM stats s WHERE s.game_id = g.id) AS viewers
FROM games AS g
WHERE g.id != 96
ORDER BY viewers DESC
LIMIT 5


a workaround for now is

```
Game.select(Game, fn.Sum(Stat.viewers).alias('viewers')).join(Stat, JOIN_LEFT_OUTER).group_by(Game).order_by(R('viewers DESC')).limit(5) 
```

but that runs up to 10 times slower then the sub queries on
Issue: Database is locked, can't be opened again
Author: Miserlou
State: closed
Created: 2013-01-25 00:57:00

Help!

I've been using peewee as the DB for a scraper job that's been running for about 3 months now.

I tried copying the database file to make a backup. This may have happened while the process was still running, and now it's stuck in a locked state.     

new_pk = insert.execute()
  File "/usr/local/lib/python2.6/dist-packages/peewee.py", line 1355, in execute
    result = self.database.execute(self)
  File "/usr/local/lib/python2.6/dist-packages/peewee.py", line 1446, in execute
    return self.execute_sql(sql, params, commit)
  File "/usr/local/lib/python2.6/dist-packages/peewee.py", line 1452, in execute_sql
    self.commit()
  File "/usr/local/lib/python2.6/dist-packages/peewee.py", line 1460, in commit
    self.get_conn().commit()
sqlite3.OperationalError: database is locked

I can neither load the backup nor continue the job. What should I do? Halp!


Not sure exactly what happened -- take a look here:

http://stackoverflow.com/questions/151026/how-do-i-unlock-a-sqlite-database


I love you. Sorry for bringing this up on GitHub, I couldn't get a response on IRC. This data is pretty important to me and I was having a mini panic attack.
Issue: Add support for LIMIT clause to UPDATE
Author: djt
State: closed
Created: 2013-01-24 04:26:32

First of all, this is a great library.  I have only just started using it but already am loving the implementation and direction.

I had a requirement of restricting the amount of items modified by an update so went ahead and added LIMIT support.  This has only been tested on MySQL in my case so I'm not sure if it will impact the other DBs that you support.
Issue: peewee + dbslayer
Author: rosscdh
State: closed
Created: 2013-01-18 10:30:03

Hey there,

Firstly well done on this neat little orm.

I'm considering writing a peewee Database type to interface with dbslayer (JSON response interface) (would implement using "requests")

I was wondering if you are aware of anything like this? or if you had any pointers?
Best
Ross


Thank you, I'm glad you're enjoying peewee!

I hadn't heard of dbslayer so I took a look at it... I am pretty sure i would never recommend anyone use dbslayer... but of course if you wanted to build an adapter I think it would be possible.

This is a good place to start: http://peewee.readthedocs.org/en/latest/peewee/database.html#writing-a-database-driver

You would need to implement a subclass of database, probably `MySqlDatabase`.  Since you're working with http you will need to implement or work around the various db-api 2.0 calls that peewee uses.

The apsw driver in "playhouse" has to wrap a number of db-api 2.0 functions, so that's a good place to look as well:
https://github.com/coleifer/peewee/blob/master/playhouse/apsw_ext.py

Good luck, let me know if you have any more questions.

:monkey_face: :bear: :rabbit:


Hi there,

Thanks for the great feedback. Id be interested in hearing your opinion on
why not to use dbslayer? aside from the obvious security issues (which in
our case are partially solved by a closed network and specific access
routes). Our main attraction for it being clean simply load-balancing
(using cherokee).

all the best
Ross

On Fri, Jan 18, 2013 at 3:45 PM, Charles Leifer notifications@github.comwrote:

> Thank you, I'm glad you're enjoying peewee!
> 
> I hadn't heard of dbslayer so I took a look at it... I am pretty sure i
> would never recommend anyone use dbslayer... but of course if you wanted to
> build an adapter I think it would be possible.
> 
> This is a good place to start:
> http://peewee.readthedocs.org/en/latest/peewee/database.html#writing-a-database-driver
> 
> You would need to implement a subclass of database, probably MySqlDatabase.
> Since you're working with http you will need to implement or work around
> the various db-api 2.0 calls that peewee uses.
> 
> The apsw driver in "playhouse" has to wrap a number of db-api 2.0
> functions, so that's a good place to look as well:
> https://github.com/coleifer/peewee/blob/master/playhouse/apsw_ext.py
> 
> Good luck, let me know if you have any more questions.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/coleifer/peewee/issues/151#issuecomment-12424613.


Sure thing!  And i'm happy to provide help when you start implementing.

The biggest detractors i see are:
- overhead of HTTP versus socket connections
- overhead of serialization to JSON and having to implement conversion between json & native python types
- limitations on response size or problems with large queries or responses

If it were me i would write a connection pool that also did load balancing.  This would get rid of the Http request/response cycle overhead and serialization overhead.
Issue: Certain operations not respecting the db_value()
Author: coleifer
State: closed
Created: 2013-01-15 14:45:22

I want to create a custom field that will be a list in python and is converted to text for database storage, but my custom field is generating invalid SQL. Here's an example to illustrate the problem I'm having.

```
from peewee import *

db = SqliteDatabase('test.db')

class CSVField(TextField):
    def db_value(self, value):
        if value:
            value = ','.join(value)
        return value

    def python_value(self, value):
        return value.split(',') if value else []

class Row(Model):
    values = CSVField()

    class Meta:
        database = db

Row.create_table(fail_silently=True)
row = Row(values=['first', 'second', 'third', 'fourth'])
row.save()
```

This will raise an exception with the message: sqlite3.OperationalError: near ",": syntax error

When I investigate, I find that the following SQL and params are being executed:

```
INSERT INTO "row" ("values") VALUES ((?,?,?,?)

['f,i,r,s,t', 's,e,c,o,n,d', 't,h,i,r,d', 'f,o,u,r,t,h']
```

Am I using this wrong? The db_value method of my custom field gets called for each item of the list separately, but I'd like to process the list as a whole. Is there a way to do this? Using a set instead of a list works fine.


This error was reported via google groups.  The issue is on line 776  in `_parse_field_dictionary` which parses things like updates and inserts.  It iterates the dictinary and hands off the value to `parse_expr` before sending it to db_value() and this is where the lists are getting mangled.


Fixed in 1d4290784a1ae183c -- see commit message for details, the tl;dr is that we don't know if we should call db_value or parse_expr first -- we call parse_expr because it is more general and powerful, but there is special logic for handling lists (which are used primarily to build up IN clauses).  This doesn't play nice w/the behavior we wanted for the csv field for instance.  The fix was basically, if the expression is not a "rich" object, to pass it thru the parse_expr.


Is this still fixed in the latest version?

I have made a ListField, which is pretty much the same as here:

```
class ListField(Field):    
 
   def db_value(self, value):
        out = ""
        for val in value:
            out += val
            out += " "
        return out[:-1]

    def python_value(self, value):
        return value.split(" ")
```

I get the same error: peewee.OperationalError: near ",": syntax error
db_value doesnt get a list, but gets called for each item in the list

Workaround:

Made a class that contains the list. Made a ClassField for this class, that uses the list to create the output.
Issue: pwiz: reserved python keywords
Author: Pacek
State: closed
Created: 2013-01-14 13:19:31

Pwiz script should keep in mind python reserved keywords like "import", "str", ... and generate alternative field name.


hmm...yeah it probably should...though its not really supposed to be "perfect" ya know?  you will still probably want to edit it...but maybe it can drop a warning in there or something
Issue: auto-incrementing primary keys in Sqlite
Author: markschl
State: closed
Created: 2013-01-09 16:33:31

I have a question regarding auto-incrementing primary keys. As far as I can see from tests, in MySQL, primary key columns are created with AUTO_INCREMENT by default, whereas in Sqlite databases the primary key of an insert may not be unique forever if some rows are deleted. As far as I understand, the equivalent of AUTO_INCREMENT (MySQL) is AUTOINCREMENT in Sqlite.
I haven't yet been able to figure out how to use this feature with Sqlite without modifying peewee itself (model._meta.auto_increment doesn't seem to have an effect).

Thank you


http://www.sqlite.org/autoinc.html  - i will see about a fix


Because SQLite incurs additional overhead to guarantee monotonically incrementing PKs, I will keep the default behavior which simply guarantees they are unique.

If you would like to use an autoincrementing primary key, try this in your code:

``` python

db = SqliteDatabase('mydb.db', fields={'primary_key': 'INTEGER AUTOINCREMENT'})
```

Or

``` python

db.register_fields({'primary_key': 'INTEGER AUTOINCREMENT'})
```


I thought about that, too; unfortunately Sqlite complains about a syntax error. The "invalid" SQL looks like this:

CREATE TABLE "test" ("id" INTEGER AUTOINCREMENT NOT NULL PRIMARY KEY, ...)
I believe, the AUTOINCREMENT should be positioned after PRIMARY KEY.


I'm thinking about trying to refactor all the DDL stuff.  Hopefully that might make something like this change easier...currently the ordering of the clauses is somewhat hardcoded :goberserk: 


Still not work.

1 `db = SqliteDatabase('mydb.db', fields={'primary_key': 'INTEGER AUTOINCREMENT'})` got below error:

`peewee.OperationalError: near "AUTOINCREMENT": syntax error`

SQL is:
`u'CREATE TABLE "customer" ("id" INTEGER AUTOINCREMENT NOT NULL PRIMARY KEY, "name" VARCHAR(255) NOT NULL, "upper_name" VARCHAR(255) NOT NULL, "remark" VARCHAR(255) NOT NULL)'`

2 `db.register_fields({'primary_key': 'INTEGER AUTOINCREMENT'})` has no effect at all


This is for a much older version of Peewee. Check out http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#PrimaryKeyAutoIncrementField


So, I have to define id manually? And, `PrimaryKeyAutoIncrementField` is only for Sqlite, right?

```
Class User(Model):
    id = PrimaryKeyAutoIncrementField()
    name = CharField()
```


Peewee includes an autoincrement pk by default for all databases. For sqlite, there are different types, rowid and a special autoincrement type. See:

http://www.sqlite.org/autoinc.html
Issue: Primary key value 0
Author: arielrossanigo
State: closed
Created: 2012-12-20 22:37:35

In an escenario where two tables are related and the value of the foreign key is 0, it raises an DoesNotExist exception. That's because 0  evaluates to False even when it's a real value.


I decided to fix this everywhere that a "false-y" value was used: 51328066c4c7b1d7c043c5e37fc06c519e9f168a


Great! thanks