Issue: Issue with insert_many - IndexError: pop from empty list
Author: kelso6969
State: closed
Created: 2018-01-24 20:38:37

I'm getting this error:
```
Traceback (most recent call last):
  File "test.py", line 166, in <module>
    Mtas2.insert_many(subs).execute()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3147, in __exit__
    return self._helper.__exit__(exc_type, exc_val, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3188, in __exit__
    self.db.pop_transaction()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2500, in pop_transaction
    return self._state.transactions.pop()
IndexError: pop from empty list
```

I'm passing a list of dicts like this:
```
[{'dn': '0123456789', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'pbt8rd7vy.mo.testing.net'}, 
 {'dn': '0123456780', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'tdfbu4j8k.mo.testing.net'}, 
 {'dn': '0123456781', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'bwwtujjrp.mo.testing.net'}]
```
Executing my insert like this
```
with database.atomic():
    Mtas2.insert_many(subs).execute()
```
Model if needed
```
class Mtas2(BaseModel):
    ad = CharField()
    agcid = IntegerField()
    as_ = CharField(column_name='as')
    dn = CharField()
    fqdn = CharField()
    npa = CharField()
    nxx = CharField()
    ps = CharField()
    timestamp = DateTimeField()

    class Meta:
        table_name = 'MTAs2'
```

Lastly, using peewee (3.0.0)


Note: I edited your comment to include formatting. Please take the extra 15 seconds to format code properly when submitting issues.

Regarding the issue, I am unable to replicate it:

```python

from peewee import *

db = SqliteDatabase(':memory:')

class Mtas2(Model):
    ad = CharField()
    agcid = IntegerField()
    as_ = CharField(column_name='as')
    dn = CharField()
    fqdn = CharField()
    npa = CharField()
    nxx = CharField()
    ps = CharField()
    timestamp = DateTimeField()

    class Meta:
        database = db
        table_name = 'MTAs2'

data = [{'dn': '0123456789',
         'ps': 'UNA',
         'ad': 'AALN/1',
         'timestamp': '2018-01-24 12:34:57',
         'npa': '0',
         'nxx': '0',
         'as_': 'ACT',
         'agcid': '5',
         'fqdn': 'pbt8rd7vy.mo.testing.net'},
        {'dn': '0123456780', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'tdfbu4j8k.mo.testing.net'},
        {'dn': '0123456781', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'bwwtujjrp.mo.testing.net'}]

db.create_tables([Mtas2])

with db.atomic():
    Mtas2.insert_many(data).execute()

print(Mtas2.select().count())
# prints 3
```

The issue has to do with how you're handling transactions, by the way, and doesn't seem related to `insert_many()`. Are you explicitly committing, rolling back, or otherwise doing stuff with transactions?

If I cause the `insert_many()` call to fail, by the way, the exception is raised correctly -- I do not get the "pop() from empty list" error.

Thank you and apologies.

I would do individual inserts, but on the 18 objects im looping through, this count can be 2k to 10k records being inserted, so I need to use a bulk insert method.

If I try the method to insert x rows at a time

```
with database.atomic():
    for idx in range(0, len(subs), 100):
        Mtas2.insert_many(Subs[idx:idx+100]).execute()
```
I get the following error
```
Traceback (most recent call last):
  File "test.py", line 172, in <module>
    Mtas2.insert_many(Subs[idx:idx+100]).execute()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3147, in __exit__
    return self._helper.__exit__(exc_type, exc_val, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3180, in __exit__
    self.rollback(False)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3167, in rollback
    self.db.rollback()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2528, in rollback
    return self._state.conn.rollback()
AttributeError: 'NoneType' object has no attribute 'rollback'
```

Should I see if the problem persists with peewee 2?

" Are you explicitly committing, rolling back, or otherwise doing stuff with transactions?" I'm not sure I understand. I'm essentially parsing a large soap response into XML and iterating over the results building this list.

I'm not doing anything else with that table or records, so I'm not sure. My other querys to update and insert single records are working as expected.

Oddly, I tested your example above and it works using sqlite to memory. Im using mysql otherwise as I have a PHP front end. I guess my next question would be, how to diagnose this further? I can send you my whole script if needed if you think it would help.

Can you send me a minimal example that I can use to try and reproduce this? That's very odd that it works fine with SQLite and breaks under MySQL. Peewee 3.0a (and 2.x) is tested using a combination of Python versions and database backends, including MySQL so if things were broken I would expect to see test failures: https://travis-ci.org/coleifer/peewee/builds/332047787

If you've got a **minimal** example I can run to replicate that'd be fantastic. As it is, it appears the insert is failing for some reason. Instead of finding and rolling-back the transaction, though, it's running into an error when doing the rollback...which doesn't make sense why it would only occur with MySQL.

OK, I use the example of yours above, and the only thing I change is the db string (and comment out the table create since it's there) and the error returns.

db = MySQLDatabase('myDB', **{'passwd': 'MyPass', 'charset': 'utf8', 'user': 'MyUser', 'use_unicode': True})



```

from peewee import *

db = MySQLDatabase('myDB', **{'passwd': 'MyPass', 'charset': 'utf8', 'user': 'MyUser', 'use_unicode': True})


class Mtas2(Model):
    ad = CharField()
    agcid = IntegerField()
    as_ = CharField(column_name='as')
    dn = CharField()
    fqdn = CharField()
    npa = CharField()
    nxx = CharField()
    ps = CharField()
    timestamp = DateTimeField()

    class Meta:
        database = db
        table_name = 'MTAs2'

data = [{'dn': '0123456789',
         'ps': 'UNA',
         'ad': 'AALN/1',
         'timestamp': '2018-01-24 12:34:57',
         'npa': '0',
         'nxx': '0',
         'as_': 'ACT',
         'agcid': '5',
         'fqdn': 'pbt8rd7vy.mo.testing.net'},
        {'dn': '0123456780', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'tdfbu4j8k.mo.testing.net'},
        {'dn': '0123456781', 'ps': 'UNA', 'ad': 'AALN/1', 'timestamp': '2018-01-24 12:34:57', 'npa': '0', 'nxx': '0', 'as_': 'ACT', 'agcid': '5', 'fqdn': 'bwwtujjrp.mo.testing.net'}]

#db.create_tables([Mtas2])

with db.atomic():
    Mtas2.insert_many(data).execute()

print(Mtas2.select().count())
# prints 3
```

error
```
python testmy.py
Traceback (most recent call last):
  File "testmy.py", line 36, in <module>
    Mtas2.insert_many(data).execute()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3147, in __exit__
    return self._helper.__exit__(exc_type, exc_val, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3188, in __exit__
    self.db.pop_transaction()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2500, in pop_transaction
    return self._state.transactions.pop()
IndexError: pop from empty list
```

In my other script .update().execute() works great, so does an individual .create()

```
CREATE TABLE IF NOT EXISTS `MTAs2` (
  `agcid` int(3) NOT NULL,
  `ad` varchar(7) NOT NULL,
  `fqdn` varchar(50) NOT NULL,
  `as` varchar(5) NOT NULL,
  `ps` varchar(5) NOT NULL,
  `dn` varchar(10) NOT NULL,
  `npa` varchar(3) NOT NULL,
  `nxx` varchar(3) NOT NULL,
  `timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
```

If you need a quick table creation script, save you a few minutes.

I feel stupid.

```
db.connect()
```

added this and it works.

Thanks and sorry for wasting your time. It's odd that updates and other inserts would work without that.

That is odd, it probably should have failed in a different way, though - so as to make clear that the connection needs to be set-up before the transaction is opened.

I'll fix.