akumoth/peewee-issues huggingface dataset:
Issue: ManyToManyField not in Model._meta.fields
Author: yohanboniface
State: closed
Created: 2015-12-17 10:07:33

```
import peewee
from playhouse.fields import ManyToManyField

db = peewee.SqliteDatabase(':memory:')


class Student(peewee.Model):
    name = peewee.CharField()

    class Meta:
        database = db


class Course(peewee.Model):
    name = peewee.CharField()
    students = ManyToManyField(Student, related_name='courses')

    class Meta:
        database = db


if __name__ == '__main__':
    assert 'name' in Course._meta.fields
    assert 'students' in Course._meta.fields
```

```
$ python peewee_many2many.py 
Traceback (most recent call last):
  File "peewee_many2many.py", line 23, in <module>
    assert 'students' in Course._meta.fields
AssertionError
```

Is that by design?


ManyToManyField does not call `model_class.add_field`, so a "solution" may be to just add this to `ManyToManyField.add_to_class`:

```
model_class._meta.fields[name] = self
```

(It's not DRY, but we can't call `add_field` itself, because it expects the field to have a column, so maybe a bit of refactor is needed.)

Edit: it crashes in `_fields_to_index` (`AttributeError: 'ManyToManyField' object has no attribute 'index'`), so certainly a filter needs to be added there.


It's not technically a field, so it is not present in the field lists. It's just a descriptor that provides convenient access to the through table relationship.
Issue: Delete/Update Many-to-Many tables
Author: samsoft00
State: closed
Created: 2015-12-17 09:47:51

@coleifer great job, kindly help... i'm having challenge deleting and updating many to many tables, below is the model and code

```
class Job(Model):
    """docstring for Jobs"""
    url = CharField(max_length=200, unique=True)
    title = TextField()
    description = TextField()
    sponsor = CharField(max_length=100)
    location = CharField(max_length=100)
    uuid = CharField(max_length=150)
    timestamp = DateTimeField(default=datetime.datetime.now)

    class Meta:
        database = DATABASE
        order_by = ('-timestamp',)

class Category(Model):
    """docstring for JobCategory"""
    category = CharField(max_length=100, unique=True)
    cat_slug = TextField()
    jobs = ManyToManyField(Job, related_name='categories')

    class Meta:
        database = DATABASE

JobsCategory = Category.jobs.get_through_model()
```

here is the method to delete queries, actually it work but it only delete a row, how can i delete multiple rows 

```
    def updateEntries(self):
        try:
            query = models.Job.select().where(models.Job.timestamp < datetime.date.today()-datetime.timedelta(days=2)):
            return jobs.execute()  # Returns the number of rows deleted. delete_instance()

        except models.DoesNotExist:
            pass

```

Using the method below will raise error

```
query = models.Job.delete().where(models.Job.timestamp < datetime.date.today()-datetime.timedelta(days=2))
return query.execute()

Error: 
peewee.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`career`.`category_job_through`, CONSTRAINT `category_job_through_ibfk_2` FOREIGN KEY (`job_id`) REFERENCES `job` (`id`))')
```


Wait... I'm confused by your code. You're doing a select? I think you copy/pasted wrong.

This is really not appropriate for github, please ask on stackoverflow.


sorry for the mistake in updateEntries method, my intention was to delete all entries in Job table and everything that associate with it using this code 

```
query = models.Job.delete().where(models.Job.timestamp < datetime.date.today()-datetime.timedelta(days=2))  
query.execute()
```

 but generate 

```
peewee.IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (career.category_job_through, CONSTRAINT category_job_through_ibfk_2 FOREIGN KEY (job_id) REFERENCES job (id))')
```

what can i do?


You need to first delete the rows from the through table that reference the job you're deleting.
Issue: load csv into database - boolean values always convert to TRUE
Author: adsahay
State: closed
Created: 2015-12-16 08:01:20

While loading a csv with a boolean field into a db table, the boolean value is always converting to `TRUE`. 

Please find reproducible test case on this [gist](https://gist.github.com/adsahay/38d73837beccd560ad66).

Expected behaviour - `0` should map to `false`, `1` should map to `true`.

I have tried using `t/f`, `TRUE/FALSE` and `0/1` in the csv.


You need to probably specify a custom `RowConverter` and override the `Loader.get_converter` method to appropriately handle your CSV data.


You can pass in a custom converter using the `converter` parameter.


I am trying the following (no luck):

```
    # use a custom RowConverter to handle boolean values, as it is not
    # handled out of the box.
    class CustomRowConverter(RowConverter):
        def field(field_class, **field_kwargs):
            def decorator(fn):
                fn.field = lambda: field_class(**field_kwargs)
                return fn

            return decorator

        @field(BooleanField, default=0)
        def is_boolean(self, value):
            # True if value is a boolean
            # ref: http://www.postgresql.org/docs/9.4/static/datatype-boolean.html
            print('is_boolean %s' % value)
            true_literals = ['TRUE', 't', 'true', 'y', 'yes', 'on', '1']
            false_literals = ['FALSE', 'f', 'false', 'n', 'no', 'off', '0']
            return value in true_literals or value in false_literals

        def get_checks(self):
            """Return a list of functions to use when testing values."""
            return [
                self.is_date, self.is_datetime, self.is_integer, self.is_float,
                self.is_boolean, self.default
            ]

    custom_converter = CustomRowConverter(database=database)
    try:
        load_csv(MyModel,
                 'file.csv',
                 converter=custom_converter)
    except Exception as e:
        app.logger.error('Exception while loading csv %s - %s' %
                         (e.__class__, e))
```

Can you please tell me where the problem is? I can't trigger the `is_boolean` check above.


I put together a test-case and was able to verify that the custom converter could interpret a booleanfield. Maybe you can look at the new test-case and get some ideas on the implementation. It's a little bit of a crummy API, I definitely would like to take a crack at rewriting it in the future, but for now hopefully this helps.


See: 58d0fcbb75faa6200ea63db11a4c43e304f39c73


Thanks for this. I think if we supply a `model` instead of a `database` to the loader custom converter isn't working.
Issue: More useful error classes
Author: macks22
State: closed
Created: 2015-12-08 16:24:12

It would be useful to be able to access pieces of the sql query that raised the error. Right now this requires parsing the `message` instance variable of the error. For instance, assume I have some `Model` instance stored in a variable `table` and I run `Model.get` using a key that does not exist. This will raise a `Model.DoesNotExist` exception:

```
try:
    table.get(key = 'test')
except table.DoesNotExist as err:
    print(dir(err))
```

The output shows that only two instance variables are present: the standard `args` and `message`. The `message` is (table fields are "key" and "val"):

```
'Instance matching query does not exist:\nSQL: SELECT "t1"."key", "t1"."val" FROM "keyvaluepair" AS t1 WHERE ("t1"."key" = ?)\nPARAMS: [u\'test2\']'
```

For this particular error, it would be nice to have another instance variable with the key. In general, it could be a `tuple` that contains all the parameters used to fill in the "?" placeholders in the query.

Why would this be useful? Well in this case, I'm implementing a simple DB-backed dictionary. I'd like to except the `table.DoesNotExist` exception and repackage it as a `KeyError` to be raised up to the user. However, I don't know what key caused the error without parsing the `message` attribute.


I think I will pass on this functionality.


@macks22 but you know what key was passed to your function. Why not use it instead of taking this value from the exception? 


@arikfr that's basically what I was getting at...there's no new info there.


Yes, this is what I ended up doing. I guess I just have a habit of packing as much info as possible into my exception instances. In this case it's not necessary though. Thanks for the quick feedback.
Issue: Better support for UUIDField
Author: foxx
State: closed
Created: 2015-12-08 14:50:53

As an extension from #780, databases which don't support UUID natively may encounter consistency problems. 

Postgres will correctly handle multiple input variants [[1]](http://www.postgresql.org/docs/9.1/static/datatype-uuid.html), such as
`a0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11` and `a0eebc999c0b4ef8bb6d6bb9bd380a11`.

However SQLite, which now uses `TEXT` since #780, does not support multiple input types properly, and will end up with different variants being stored in the backend, rather than being normalised. Other databases, such as MySQL, are not supported at all as they don't have the necessary `field_overrides`.

The most viable fix I can see for this would be;

``` py
class UUIDField(Field):
    ...
    def db_value(self, value):
        if isinstance(value, uuid.UUID):
            return value.hex
        try:
            return uuid.UUID(value).hex
        except:
            return value
```

The above would ensure that the UUID output is always consistent, whilst also allowing non confirming values to be used with `filter()`, e.g. `0f0f0f0f-` 

I'd also like to propose that we add a `field_override` for `MySQLDatabase`, much the same as `SqliteDatabase`.

If you're happy with the above solution, I'll throw up a PR, let me know


Ahh yeah good call, and I should know cause I got bitten by just this issue.

Yeah a PR b great, thx


Awesome, PR coming shortly :)


Also there could be an argument for storing as `BINARY` instead of TEXT, but I'd need to think about the edge cases. I'll open a new issue if I can think of a compelling argument


@coleifer Any idea when this fix will make it into next release? Just got stung by it again lol


Ahh, dang have I not pushed a release? OK I'll get one out.


PS - I'm not sure if you use MySQL/MariaDB but I had to modify the UUID type to VARCHAR(255) in order to get it working. Not sure if that matters, it was LONGTEXT before.


Ah sorry that's my bad, although for indexing purposes it should probably be `CHAR(16)`. I've added `OptimizedUUIDField` and `BinaryUUIDField` into my `peewee-extras` repo, for anyone that wants to store UUIDs in MySQL on a larger scale. It'll be a couple of weeks before the repo is ready for prime time, as I haven't finished adding all the tests yet.


PS @foxx you may want to add a LICENSE file to peewee-extras, as I'd like to maybe borrow some of those tidbits :)


@coleifer Yup will do, in a few weeks it will be as clean as [bottlecap](https://github.com/foxx/bottlecap) (travis/coveralls/py2.7/py3.4/100% test coverage etc), will ping you when ready.
Issue: Support for triggers in DDL
Author: macks22
State: closed
Created: 2015-12-07 22:17:23

It would be very useful to be able to define triggers when using `create_table` to create database schemas. [I had this issue and came up with a very simple solution](http://stackoverflow.com/q/34142550/2486302). It fit my use case for a particular type of trigger using sqlite. However, based on my reading of the source, it didn't seem like supporting the full trigger API would be too difficult.


You can use the fields constraints param to specify default values: DateTimeField(constraints=(SQL('default now'),))

I do not plan on supporting triggers at this time.


Got it; thanks for the constraints tip.
Issue: how to use "insert ignore" sql in peewee?
Author: pc10201
State: closed
Created: 2015-12-05 11:11:36

how to use "insert ignore" sql in peewee?


http://docs.peewee-orm.com/en/latest/peewee/api.html#InsertQuery.on_conflict

Please search the docs before posting tickets like this. Github tickets are more appropriate for **bugs**. If you have a question, post to stackoverflow, the mailing list, or the IRC channel. All of these are described in the documentation: http://docs.peewee-orm.com/en/latest/peewee/contributing.html
Issue: pwiz password
Author: pc10201
State: closed
Created: 2015-12-05 08:35:01

when I run

```
python -m pwiz --host=127.0.0.1 --port=3306 --user=root --password=root -e mysql forex
```

result

```
Usage: pwiz.py [options] database_name

pwiz.py: error: --password option does not take a value
```

when I run

```
python -m pwiz --host=127.0.0.1 --port=3306 --user=root -P root -e mysql forex
```

result

```
Password: 
```

It need to input password again.Why?


The idea is that it is bad practice to accept passwords on the command-line, since they get written to your bash history, etc.


So you specify `-P`, then when prompted you enter your password in.


Interesting that you chose not to accept the passwords on the command line which is an explicit choice that the end user makes, however the pwiz tool itself _echos the password into the console or is piped into the file you store the models in without explicitly telling the user that their password is going to get written_.  This has led me to accidentally commit a password into a repo. You should be able to specify os.env variables to retrieve the passwords in the generated files, or else find some other paradigm to set the database on the BaseModel.

> This has led me to accidentally commit a password into a repo.

Whoops, I can add a note to the docs that the password will be outputted as a connection parameter.

As far as using an environment variable, just modify the generated code to do what you want it to.

I am using peewee in a containerized project. There is a console program that manages the project, and I just want to run pwiz in a container where my python backend lives to get an updated schema from the database when developers change it.

All the database credentials are env variables inside the container, and this is a standard practice. We are working with dev environment with pwiz, as expected.

Instead of simply running something like

`docker exec <CONTAINER_ID> 'python -m pwiz -H $DBHOST -p $DBPORT -u $DBUSER -p $DBPASSWORD -e postgresql $DBNAME'`

I have to ask for password or have to write an expect script just because there is no way for pwiz to accept a password non-interactively.

This is to show that enforcing users to enter password from the terminal is a flawed practice, and this is an issue we are facing with this tool, otherwise excellent. Maybe you reconsider your policy, thanks!


`pwiz` is not typically something intended for use outside of interactive sessions, since it is really just a generator for skeleton code. You could, for your purposes, probably just look at the code inside pwiz's main and write your own short script.
Issue: RetryOperationalError mixin failing on retry w/ error 2006
Author: bakemecookies
State: closed
Created: 2015-12-04 14:01:18

Tried using the new RetryOperationalError mixin and although it's reaching the exception block properly (known because it's going to shortcuts.py, line 208, execute_sql), the retry fails and raises the same error.

My hunch is that since we're using PooledMySQLDatabase, could it be that we're closing one stale connection but just getting another stale one on the retry?  Not that it's necessarily a solution, but in the past we've used a combo of sql_error_handler to close stale connections and recursion on execute_sql (i.e., infinite retry only in the case of 2006 errors, as opposed to the single retry in RetryOperationalError) within its exception block to handle this.


Unfortunately, I don't think I will be going back to the "sql error handler" API. I think the implementation in the `RetryOperationalError` mixin is an appropriate way to try to handle re-connect/re-try logic. Maybe you can make some changes to your pool implementation, particularly overriding the `_is_closed()` method to perform some sort of test to check that the conn is really active. Take a look at the pool module's code, particularly the `PooledMySQLDatabase._is_closed()` method...I bet you can find something that'll work for you.
Issue: provide an entry_point for third party drivers
Author: mfussenegger
State: closed
Created: 2015-12-03 21:36:30

As far as I can see there is currently no way for a third party Database drivers to hook into peewees db_url functionality. 

It would be great if there was an extension mechanism, maybe similar to how SQLAlchemy does it (see http://sqlalchemy.readthedocs.org/en/latest/core/connections.html#registering-new-dialects) so that a user can install a third party driver via pip install and then use that in his application without changing any line of code except the db uri.
Issue: Support for UUID in Sqlite3
Author: foxx
State: closed
Created: 2015-12-03 15:34:38

As per the docs, UUID is not natively supported in Sqlite3, resulting in an error when you try and use it;

```
  File "/home/vagrant/.virtualenvs/vagrant/lib/python3.4/site-packages/peewee.py", line 1824, in inner
    return self.parse_node(fn(*args, **kwargs))
  File "/home/vagrant/.virtualenvs/vagrant/lib/python3.4/site-packages/peewee.py", line 1852, in _create_table
    columns.append(self.field_definition(field))
  File "/home/vagrant/.virtualenvs/vagrant/lib/python3.4/site-packages/peewee.py", line 1802, in field_definition
    column_type = self.get_column_type(field.get_db_field())
  File "/home/vagrant/.virtualenvs/vagrant/lib/python3.4/site-packages/peewee.py", line 1431, in get_column_type
    return self._field_map[f]
KeyError: 'uuid'
```

This can be easily added by using a `VARCHAR` instead, sure it's not a native column but at least we get something rather than nothing.

Therefore I'd like to propose adding a field mapping into Sqlite3 for 'uuid=VARCHAR'. If I submit a PR, would this be accepted @coleifer?

In the mean time, I was able to get this working with a monkey patch;

```
overrides = peewee.SqliteDatabase.field_overrides.copy()
overrides['uuid'] = 'VARCHAR'
peewee.SqliteDatabase.field_overrides = overrides
```


LGTM, you're too fast!!
Issue: MyModel.DoesNotExist throws TypeError: exceptions must derive from BaseException
Author: amorphic
State: closed
Created: 2015-12-02 14:08:35

Using Python 3.4.0 and Peewee 2.7.3 I am attempting to handle the case where MyModel.get() returns no results as [described in the docs](http://docs.peewee-orm.com/en/latest/peewee/api.html#Model.get). Something like:

```
try:
    user = User.get(User.username == 'charlie')
except User.DoesNotExist:
    print "No user found"
```

However this results in: `TypeError: exceptions must derive from BaseException`.


Apologies... this had absolutely nothing to do with peewee.


What was the actual problem? I have a similar error.


I honestly can't remember but I was doing something stupid which was unrelated to peewee.

The module I was working on is [here](https://github.com/re-3D/opengb/blob/master/opengb/database.py) in case it's useful.
Issue: Deadlock restart
Author: ccario83
State: closed
Created: 2015-12-01 21:02:20

I'm running a multi-threaded app with many threads read/writing a central database at the same time. I've wrapped my queries in transactions and have tried to limit their sizes to prevent too many rows from being locked at the same time. However, I still occasionally get **deadlocks**:

`peewee.InternalError: (1213, u'Deadlock found when trying to get lock; try restarting transaction')`

Peewee recommends large bulk inserts as a performance improvement which is contrary to the deadlock solution. The obvious answer to my problem is to **catch this exception and retry the transaction**, which I will do. But wouldn't this sort of boilerplate code be better included **in peewee** itself? Can a feature be added where peewee will automatically retry a deadlocked transaction after waiting a brief amount of time? This feature could be disabled by default if not recommended as best practice, though a lot of mysql newbies would appreciate this sort of thing being handled for them behind the scenes. 


> But wouldn't this sort of boilerplate code be better included in peewee itself?

Not in my opinion, no. There are far too many things involved for me to feel comfortable implementing a canonical / library solution to the problem of deadlocks.


@lucidv01d Did you implement some generic(ish) retry logic or how did you end up tackling it?
Issue: Aliased column is not converted to attribute when selected as alias as a part of join
Author: alexanderad
State: closed
Created: 2015-12-01 12:18:43

``` python
class User(Model):
    ...

# it works in this way
for user in User.select(User, User.id.alias('some_alias_name'):
     print user.some_alias_name

# it does not in this way
for user in User.select(User, Group.id.alias('group_id')).join(Group):
     print user.group_id # AttributeError

# however
for user_dict in User.select(User, Group.id.alias('group_id')).join(Group).dicts():
     print 'group_id' in user_dict # True
```


Ah, `naive()` can help. Sorry.
Issue: Question: ability to implicitly select joined model instance
Author: alexanderad
State: closed
Created: 2015-12-01 09:26:28

Given two tables

``` python
class User(Model):
    ...

class Group(Model):
    user = FK(User)
```

and database contains

``` python
user1 = User()
group1 = Group(user=user1)
group2 = Group(user=user1)
```

Is there any way to select group related instance (resolved automatically if any) as a part of `selection`?

``` python
for user in User.select().join(Group):
    print user.group
```

I know model aliases are used differently but I imagine this could be something like `User.select(Group.alias('group')).join(Group)`


It seems pretty straightforward to me:

``` python
In [15]: with assert_query_count(1):
   ....:     q = User.select(User, Group).join(Group)
   ....:     for user in q:
   ....:         print user.username, user.group.name
   ....: 
... prints users and groups ...
```
Issue: prefetch() now loads all foreign keys to given table
Author: hedger
State: closed
Created: 2015-11-30 20:32:26


Yay, all checks have passed.


I've pushed a branch that has some of your changes integrated, 89cd2c4f74a526591f3804d04b2252c489be5432


That's great, thanks!


Pushed 2f028c32225c5aad9b97f0f451b9d238c3846e0d to the `multiprefetch` branch. Things aren't working quite right in the test-case. I think the identity map needs to differentiate things by field, but haven't dug into it too much yet.


Got things fixed and have merged into master.
Issue: prefetch() only loads the first ForeignKeyField of given type
Author: hedger
State: closed
Created: 2015-11-30 13:05:01

```
class M1(Model):
   i1 = IntegerField()
...

class M2(Model):
   i2 = IntegerField()
   f1 = ForeignKeyField(M1, related_name='origin')
   f2 = ForeignKeyField(M1, related_name='target')
...
```

In this example, `prefetch(M2.select().where(M2.i2>10), M1)` only fetches `f1` entries. `f2` is loaded with separate queries when iterating over results, since `rel_for_model` in `prefetch_add_subquery` returns single (first) foreign key of given type.


Tried to fix that, https://github.com/coleifer/peewee/pull/775.
Tested on some backrefs in my project. Still needs more testing.


Fix merged.
Issue: Full migration framework
Author: mekanix
State: closed
Created: 2015-11-29 11:14:58

In short, if anyone is working on full fledged migration framework, I'd like to contribute. If nobody is working on it, I'd like to do it. Any pointers on how to contribute/implement migrations ala Django (or maybe something better?) are welcome.


@mekanix -- https://github.com/cam-stitt/arnold is one such project I'm aware of. I don't have plans for adding anything more than what is already present in the `playhouse.migrate` module. Perhaps you can bring this up on the mailing list? https://groups.google.com/group/peewee-orm