akumoth/peewee-issues huggingface dataset:
Issue: StopIteration thrown using load_csv
Author: horkko
State: closed
Created: 2017-04-27 13:40:06

Hi,

I try to load a csv file in sqlite db using `peewee 2.8.1`.
To do so, I've been using `playhouse.csv_utils`.
However, when I try to load CSV file, I've got a `StopIteration` thrown:
```
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 309, in load_csv
    return loader.load()
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 277, in load
    next(reader)
StopIteration
```

Below are the first 3 lines of my csv file:
```
ncbi_taxid,parent_taxid,tax_name,lineage_level
1,1,root,no rank
2,131567,Bacteria,superkingdom
...
```
and the way I call `load_csv`:
```python
...
load_csv(model, data, has_header=True, delimiter=",")
...
```

where `model` is as such:
```python
class Taxa(BaseModel):

    ncbi_taxid = pw.IntegerField(null=False, primary_key=True, unique=True)
    parent_taxid = pw.IntegerField(null=False)
    tax_name = pw.CharField()
    lineage_level = pw.CharField()
```
and `BaseModel` as below:
```python
db = peewee.Proxy()


class BaseModel(peewee.Model):
    class Meta:
        database = db
```
If I use a `database` object instead of a `model` another error is then thrown:
```python
...
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 309, in load_csv
    return loader.load()
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 266, in load
    self.analyze_csv()
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 243, in analyze_csv
    **self.reader_kwargs)
  File "/home/tuco/src/workspace/taxadb/lib/python3.5/site-packages/playhouse/csv_utils.py", line 138, in extract_rows
    header, rows = rows[0], rows[1:]
IndexError: list index out of range
```

Is there something I do not use or set correctly?
Thx

It looks like the CSV reader is expecting more lines in the file than there are. The CSV code is tested and should be working, so I doubt a bug as serious as "loading a CSV file is broken"... Guessing the issue is with the file.

How many lines are needed at least for expecting CSV reader to work as expected?
Should I set `sample_size` greater than default 10? My CSV file is 15 lines long ....
Issue: playhouse dburl if password contains '@' , the parse will failed
Author: ygsun
State: closed
Created: 2017-04-25 14:47:00


IDK, it seems like it works to me:

```python
In [3]: parse('postgresql://postgres:my_p@ssword@localhost:5432/my_database')
Out[3]: 
{'database': 'my_database',
 'host': 'localhost',
 'password': 'my_p@ssword',
 'port': 5432,
 'user': 'postgres'}
```
Issue: Decorator for test_database
Author: jonathan-s
State: closed
Created: 2017-04-20 14:20:16

Using the the context manager test_database is very useful while testing. But I found myself writing that a lot in the tests which amounted to quite some boilerplate. Perhaps it would be worthwhile to make that usable as class decorator? 

Let me know what you think. 

It won't work as a class decorator, I don't think -- not without dynamically modifying the setup/teardown methods.
Issue: How should I structure my peewee postgres database for hosting on heroku
Author: rahul3103
State: closed
Created: 2017-04-19 21:08:03

http://stackoverflow.com/questions/43505529/peewee-and-postgres-on-heroku-not-able-to-create-table

I tried a lot but I was not able to run a simple code with peewee and postgres on heroku. Without database my app is running on heroku but with peewee and database its not running. If you can help me a little bit it will be great. The app [link](https://github.com/rahul3103/test-heroku)

Responded on SO. Issue had nothing to do w/your peewee setup.
Issue: Automatically creating all tables, in right order.
Author: tomvlk
State: closed
Created: 2017-04-19 10:14:01

I don't know if this was suggested before, but it would be great to have a function to create all tables, that automatically orders them in dependencies (like foreign keys etc.) so you would be able to have a dynamic application with dynamic models easily running without having to struggle with foreign keys while creating tables.

`db.create_tables()` does just that.

I'm very sorry, thanks :+1:
Issue: Return back LTreeField support
Author: spumer
State: closed
Created: 2017-04-17 11:11:53

I saw in 1.0.0 version peewee have LTreeField support, but now we have only HSTORE and JSON.
Why this feature dropped?

Previous inital support of ltree extension for PG
https://github.com/coleifer/peewee/commit/9959cd8142ecece6704fd82e9c6274de2794d8c4

I wasn't interested in supporting it. 1.0 was also released back in 2012, a relative eternity ago.
Issue: Implement __ne__ for CompositeKey.
Author: timster
State: closed
Created: 2017-04-15 13:34:30

For a model with a single primary key, you can do both of these:

```python
(obj._meta.primary_key == obj._get_pk_value())
(obj._meta.primary_key != obj._get_pk_value())
```

And both return a `peewee.Expression` object.

However if the primary key happens to be CompositeKey, the `!=` compare returns `False`.

This fixes that. I can write tests if you'd like.

Wouldn't it just be:

```python
def __ne__(self, other):
    return ~(self == other)
```

Yeah, that works too. Forgot about `~` when I wrote it.
Issue: How about make a transparent cache layer?
Author: aiportal
State: closed
Created: 2017-04-12 15:06:21

If small table, cache all data in memory or redis or memcached like this:

```
class MyModel(Cache.Model):
    Meta:
        cache_db = ':memory:'  # or 'redis' or 'memcached'
      ...
```

If big table, cache a part of data by a range define.

```
class MyModel(Cache.Model):
    Meta:
        cache_db = ':memory:'
        cache_range = CacheRange(fieldname, min_value, max_value)
      ...
```

If need dynamic cache, record select sql history, and merge result data by primary key in cache db.
If new select sql in history, hit cache, if not, select from backend database and cache it.
User can limit cache items count by cache_limit parameter.

If save or delete executed, remove cache data or reload cache data.
Issue: Allow disabling lazy loading
Author: tuukkamustonen
State: closed
Created: 2017-04-11 07:01:02

Is it possible to disable lazy loading of foreign keys somehow? I assume the magic happens in some `__getattr__` or proxy class but not sure where.

You probably don't want to add this as a built-in option, but if you have any pointers to what I should stab, that would be nice.

Use case: I am tuning my code for performance and wouldn't like peewee to do any queries that I didn't intend. I have some quite large fields in DB that I don't want to accidentally load.

Here are a couple ideas:

* Use the `<fk field>_id` accessor -- this will return the value of the foreign key column w/o triggering a lookup on the related table.
* Subclass `ForeignKeyField` and override the `_get_descriptor()` method to return a descriptor w/the desired behavior

Thanks, I'll try those out!

FYI for anyone wondering about the same: I decided to just use option 1 (using `<fk field>_id` as accessor).

Implementing custom `RelationDescriptor` (option 2) is easy, but it becomes confusing what the field should return:

```python
a = Foo.create()
b = Foo.create(parent=a)

b.parent  # has a instance in object cache so returns a's Foo() instance
b.parent.id  # works

b2 = Foo.get(id=b.id)

b2.parent  # only has PK and parent instance is not cached so should it return int or throw error or what...? if it returns int then:
b2.parent.id  # fails to exception
```

If I made descriptor always return raw PK value, that would be less confusing, but then doing something like `a.parent.id` would not be possible, even if `a` had parent instance already cached. So, going to use use `<fk field>_id` accessors for now.
Issue: The select loop is processed twice
Author: MarcSky
State: closed
Created: 2017-04-10 12:08:54

I have a strange problem, the select loop is processed twice. Ie I receive from the database 2 objects, and the cycle passes 4 times.

    for enter in Enter.select().join(PaySystem).where(
                    (Enter.en_status == 1) &
                    (Enter.en_date >= startdate) &
                    (PaySystem.ps_abbr=='btc')):

I made a variable count and derived its value, it turned out 4. Why is it so?

Are you using SQLite, and in the body of the for loop are you issuing any INSERT/UPDATE/DELETE queries?

#450 touches on this, but essentially you should do one of the following:

* wrap the for loop in a transaction using `with db.atomic():`
* coerce the select query to a list before iterating over it.
Issue: Model generator support tables with no primary key
Author: onlyice
State: closed
Created: 2017-04-07 15:47:01

Currently for tables with no primary key, pwiz would generate model files without `primary_key = False`. Thus peewee assumes that there is a `id` column as primary key in the table, and the query failed.

Add `primary_key = False` in the `Meta` definition would prevent this error.

Merged equivalent patch.
Issue: Hybrid Attributes and TextFields don't properly form an expression (SQLite)
Author: ballan91
State: closed
Created: 2017-04-06 21:00:55

Basic repro below, but just as the description says, I cannot seem to get the two to play nicely, I was going off some examples in SQLAlchemy which I believe was the basis for hybrid attributes.

Using SQLite and tested with the latest release version of peewee 2.9.1

```python
from peewee import TextField, Model
from playhouse.hybrid import hybrid_property


class TestModel(Model):

    first_name = TextField()
    last_name = TextField()

    @hybrid_property
    def full_name(self):

        return self.first_name + ' ' + self.last_name

def test():

    TestModel.create_table(fail_silently=True)

    TestModel.insert(
        first_name='John',
        last_name='Doe'
    ).execute()

    for person in TestModel.select():
        print(person.full_name)

    q = TestModel.select().where(
        TestModel.full_name == 'John Doe'
    )

    if q.count() == 0:
        print('Failed with')
        print(q.sql())
    else:
        print('Success')

if __name__ == '__main__':
    test()
```

Cool -- I think I need to modify the string based fields to use concatenation for the `+` operator overload. Nice catch.

1740cc1132ca01d3964f543eca00f1dc9532bc24

Thanks for such a swift reply! Release schedule wise, there appear to be a number of fixes since the last release, do you expect to issue a new release in the near future?

Just released 2.9.2, thanks!
Issue: Question: Can I use FlaskDB.Model in non-Flask process?
Author: devkhan
State: closed
Created: 2017-04-06 17:24:33

I have defined a model `X` by subclassing `FlaskDB.Model`, and then I want to access one of the instances in a celery worker process. Is it possible, and if so, how should I go about that?

I've already tried that, and receiving an `AttributeError: Cannot use uninitialized Proxy.`. But I guess that is because of not opening a database connection, which also sparks the question that how should I use the database in the worker as there is no Flask app context to provide?

Any help is highly appreciated.

This is do-able. You'll need to initialize the Flask application, though. Something like this:

```
with app.app_context():
    # Try using the database here.
```

This definitely isn't a bug, so I'm closing as this type of question belongs on Stackoverflow.
Issue: ValueError: too many values to unpack (expected 3)
Author: lqsgd123
State: closed
Created: 2017-04-05 13:14:52

I want to calculate the days between two dates.

```
def isLeapYear(year):
    if year%400 == 0:
            return True
    elif year%100!=0 and year%4==0:
        return True
    else:
        return False

def nextDay(year,month,day):
    if isLeapYear(year):
        daysOfMonths1 = [ 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] 
        daysOfMonths = daysOfMonths1
        return daysOfMonths
    else:
        daysOfMonths2 = [ 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]
        daysOfMonths = daysOfMonths2
        return daysOfMonths
    if day < daysOfMonths[month-1]:
        return year, month, day + 1
    else:
        if month == 12:
            return year + 1, 1, 1
        else:
            return year, month + 1, 1
        
def dateIsBefore(year1, month1, day1, year2, month2, day2):
    """Returns True if year1-month1-day1 is before year2-month2-day2. Otherwise, returns False."""
    if year1 < year2:
        return True
    if year1 == year2:
        if month1 < month2:
            return True
        if month1 == month2:
            return day1 < day2
    return False        

def daysBetweenDates(year1, month1, day1, year2, month2, day2):
    """Returns the number of days between year1/month1/day1
       and year2/month2/day2. Assumes inputs are valid dates
       in Gregorian calendar."""
    # program defensively! Add an assertion if the input is not valid!
    assert not dateIsBefore(year2, month2, day2, year1, month1, day1)
    days = 0
    while dateIsBefore(year1, month1, day1, year2, month2, day2):
        year1, month1, day1 = nextDay(year1, month1, day1)
        days += 1
    return days

    
print(daysBetweenDates(2012,1,1,2012,2,29))

```
Sadly this doesn't work:

`>>> runfile('G:/Anaconda3/ex1.py', wdir='G:/Anaconda3')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "G:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)
  File "G:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)
  File "G:/Anaconda3/ex1.py", line 56, in <module>
    print(daysBetweenDates(2012,1,1,2012,2,22))
  File "G:/Anaconda3/ex1.py", line 51, in daysBetweenDates
    year1, month1, day1 = nextDay(year1, month1, day1)
ValueError: too many values to unpack (expected 3)`
Issue: any documentation that doesn't require flask.
Author: faststare
State: closed
Created: 2017-04-03 12:34:40

it gets complicated that you mix peewee and flask. not all developer develop web app nor flask.
documentation should be simple like  ponyorm docs.
it seems that you didn't the import this mantra with your package that you develop.
peewee have potential but too much quirks in the docs therefore new developer who try peewee will stay because of messy documentation.

I'm just suggest to improve the quality of peewee documentation.
thanks

What are you talking about? Hardly anything is Flask-specific!

Maybe you're talking about the example application, which is a flask application?

At any rate, the documentation is intentionally designed to be framework- or implementation-agnostic. It doesn't matter if you're doing batch processing, ETL, web, whatever....the docs describe the APIs and mechanics of peewee and don't tie you to a particular usage scenario.

how I map specific user with name column with insert function?
example:
Message.insert(msg='example', user_id=1#<--need to replace an user name here)
example schema:
Users:
    username = CharField()
    password = CharField()
Msg:
    user_id = ForeignKey(Users, related_name='username'')
    msg = Charfield()
    timestamp = DateTimeField()
Issue: Python3: AESEncryptedField attempts to combine strings and bytes
Author: jlowin
State: closed
Created: 2017-04-02 02:40:52

This is only an error in Python 3.

In [this line](https://github.com/coleifer/peewee/blame/master/playhouse/fields.py#L320), the AESEncryptedField attempts to concatenate the `key` (a `bytes` object) with `'  '` (a `str`). This raises an error since the operation is not allowed.
Issue: Add a way to copy over where clause between queries
Author: tasn
State: closed
Created: 2017-04-01 10:56:17

Hi,

I want to be able to do:
```
a = Model.select().where(COMPLEX_CONDITION_HERE)
Model.insert(a=1).inherit(a)

```
Not sure if this is the best API, but this is a start.

There are two main places where this is useful:
1. When you want to do two manipulations on the same consecutively. For example, run a select, do some calculations, and if condition X applies, delete all of the entries that match the query without having to repeat yourself.
2. This one is more important because not having it here can introduce bugs: when the select query is abstracted away, either by a helper function that conditionally returns a different set, and then you can't know what is the set you have, so you can't easily delete it, or even in the most by sense by the built it "related_set". So for example, a.related_set.delete() is a useful operation, and at the moment people would have to resort to reimplementing the implementation of related_set to delete all related objects.

I asked on IRC and there were no database-only solutions to this problem. I.e the best solutions proposed were fetching all of the entries and then acting on them using the "in" operator, or alternatively just deleting/updating the objects one by one. Both are slow and unacceptable.

I hope this makes sense for the project.

Well, for one, INSERT queries don't take a WHERE clause.

But to your point, just put the expressions together and assign them to a variable...

```python
COND = (MyModel.foo == 'bar') & (MyModel.baz < 4)
q1 = MyModel.select().where(COND)
MyModel.delete().where(COND).execute()

# or
MyModel.delete().where(MyModel.id << q1)
```
Issue: Why "from peewee import *" doesn't import InsertQuery?
Author: TwoAnts
State: closed
Created: 2017-03-30 13:22:31

```
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from peewee import *
>>> InsertQuery()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'InsertQuery' is not defined
>>> from peewee import InsertQuery
>>> InsertQuery()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() takes at least 2 arguments (1 given)
>>> 
```
`from peewee import *` doesn't import InsertQuery, 
but `from peewee import InsertQuery` does.  
I have tested it on Python 2.7.6 and Python 2.7.12 , got the same result.  
A little curious.  


Typically you would not import an InsertQuery directly, but use the `insert()` model method.

Please read the docs.

I think I have found my answer from http://stackoverflow.com/questions/44834/can-someone-explain-all-in-python.
Maybe I'm a little stupid here, now I'm clear. I see  `__all__` is used in peewee.py and other variables will not  be imported by `from peewee import *`.
Thanks a lot for your patience.

I see - so you were less interested, then, in using `InsertQuery` and were more interested in the mechanics of the Python import statement...

😁
Issue: how to make peewee work with MysqlProxy
Author: knightliao
State: closed
Created: 2017-03-28 02:42:21

hello, peewee recommend 'read-slave module' ( http://docs.peewee-orm.com/en/latest/peewee/database.html?highlight=slave#read-slaves ) to do with master/slave mysql. 

atlas(https://github.com/Qihoo360/Atlas) is a mysql proxy, user write less code when use it.

and my question is 'how peewee can do with mysql proxy? ' ( such as Atlas)



my solution is:

1. make mysql connection with autocommit = true
     
     `connect_kwargs.update({'autocommit': True})`

2. no transaction sql without commit 
```

def execute_sql(self, sql, params=None, require_commit=True):
        logger.debug((sql, params))
        with self.exception_wrapper():
            cursor = self.get_cursor()
            try:
                cursor.execute(sql, params or ())
            except Exception as exc:
                if self.get_autocommit() and self.autorollback:
                    self.rollback()
                if self.sql_error_handler(exc, sql, params, require_commit):
                    raise
            else:
                pass
        return cursor
```


3. when using transaction, add begin statement

```
def begin(self):
        logging.debug("trans begin")
        self.get_conn().begin()

```

the total code is :

```
class ProxyPooledMySQLDatabase(PooledMySQLDatabase):
    def init(self, database, **connect_kwargs):
        connect_kwargs.update({'autocommit': True})
        super(ProxyPooledMySQLDatabase, self).init(database, **connect_kwargs)

    def execute_sql(self, sql, params=None, require_commit=True):
        logger.debug((sql, params))
        with self.exception_wrapper():
            cursor = self.get_cursor()
            try:
                cursor.execute(sql, params or ())
            except Exception as exc:
                if self.get_autocommit() and self.autorollback:
                    self.rollback()
                if self.sql_error_handler(exc, sql, params, require_commit):
                    raise
            else:
                pass
        return cursor

    def begin(self):
        logging.debug("trans begin")
        self.get_conn().begin()
```

what do you think about my solution ?

?

?
Issue: Where clause cannot handle dict_values (dict view - python3)
Author: dhiana
State: closed
Created: 2017-03-27 12:40:32

When trying to use a where clause comparing a field to a dict_value, I get this error.

peewee.ProgrammingError: syntax error at or near "'dict_values([‘’foo’’, “bar’’])’”
LINE 1: …baz = ‘quz’)) AND ("t1”.”qux” IN 'dict_valu...

https://docs.python.org/3.0/library/stdtypes.html#dict.values

My code was working on python 2, but when modifying iteritems to items to use python 3, now I need to force list() on the dict_values.

Convert it to a list explicitly:

```python
.where(SomeModel.field << list(some_dict.values()))
```
Issue: Model.drop_table disregards Database.drop_cascade
Author: nad2000
State: closed
Created: 2017-03-24 01:26:04

Attempting to drop a table in Sqlite DB with **cascade=True** fails even if `Database.drop_cascade=False` with:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File ".../peewee.py", line 5000, in drop_table
>     cls._meta.database.drop_table(cls, fail_silently, cascade)
>   File ".../site-packages/peewee.py", line 3870, in drop_table
>     model_class, fail_silently, cascade))
>   File ".../site-packages/peewee.py", line 3755, in execute_sql
>     self.commit()
>   File ".../site-packages/peewee.py", line 3578, in __exit__
>     reraise(new_type, new_type(*exc_args), traceback)
>   File ".../site-packages/peewee.py", line 135, in reraise
>     raise value.with_traceback(tb)
>   File ".../site-packages/peewee.py", line 3748, in execute_sql
>     cursor.execute(sql, params or ())
> **peewee.OperationalError: near "CASCADE": syntax error**
> **near "CASCADE": syntax error**
> 

The SQL get built (`DROP TABLE "foo" CASCADE`) without taking in account if DB supports DROP CASCADE  (`Database.drop_cascade`): https://github.com/coleifer/peewee/blob/master/peewee.py#L2187 
While the test (`TestDropTableCascade`) does get skipped for the case when DB doesn't support DROP CASCADE.

Perhaps it should be handled a bit more gracefully, either 1) with more specific exception or 2) implementing cascaded dropping for the dependant tables.

### Test code:

```python
from peewee import Model, TextField, ForeignKeyField
from peewee import drop_model_tables, SqliteDatabase

db = SqliteDatabase(":memory:")

class Foo(Model):
    foo = TextField()

    class Meta:
        database = db

Foo.create_table()
Foo.drop_table(cascade=True)

```