Issue: Memory leak when using insert_many
Author: rammie
State: closed
Created: 2018-06-04 17:28:56

We are ingesting a large feed of data and had issues with ingestion jobs running out of memory. After adding some instrumentation, we were able to narrow down the issue to the peewee insertion code. We replaced the following lines:

```python
with db.database.atomic():
    SomeModel.insert_many(items).execute()
```

with

```python
with db.database.atomic():
    for item in items:
        SomeModel.insert(**item).execute()
```

This fixed our memory leak. We are using Postgres 9.6 as the database engine. Unfortunately, I did not have the time to put together an isolated example, but will do so if you think that will help. I hope this information will suffice and that this is indeed a peewee bug. Sorry for the sloppy bug report.

Edit:
Also, we routinely call gc.collect() during ingestion, so hopefully our leak was not gc related. My best guess is that a reference to the inserted rows is left behind somewhere in insert_many.

Very likely this did cause a memory issue, because peewee has to literally go through all the items and build up a sql string in-memory as well as a parameter list in-memory. Doing it one-at-a-time in a loop like your second example allows python gc to keep memory down. It's not a leak, just the nature of how it works. Another way you could do this would be to batch-insert them X at-a-time:

```python
with db.database.atomic():
    # 100 at a time.
    for i in range(0, len(items), 100):
        item_slice = items[i:i + 100]
        SomeModel.insert_many(item_slice).execute()
```

That makes sense. Though, here is what's picking at me. We call `gc.collect()` after each call to `insert_many` and then log process memory usage. It wasn't just that the process took more memory, it was that it kept growing after each subsequent call to `insert_many`. It seemed like objects were being generated that were not being collected (though I did not profile enough to confirm this).

How about this - I'll try to put together an isolated example that displays the same behavior and test it out against sqlite and postgres to see if it's a db driver issue, a peewee one, or a user error.

I put together a test script and it replicates the behavior in sqlite as well. Your suggestion helped drive down memory usage significantly, so I think you are absolutely right, and generating very large strings seems to be the issue. I've included the script for completeness, but am closing this issue. Thanks for the prompt response.

Run in cpython 2.7.10
```python
import gc
import os
import psutil
from peewee import *

# From: https://stackoverflow.com/a/1094933
def sizeof_fmt(num, suffix='B'):
    for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:
        if abs(num) < 1024.0:
            return "%3.1f%s%s" % (num, unit, suffix)
        num /= 1024.0
    return "%.1f%s%s" % (num, 'Yi', suffix)


sqlite_db = SqliteDatabase('test.db')

class SomeModel(Model):

    class Meta:
        database = sqlite_db

    a = IntegerField()
    b = TextField()

SomeModel.create_table()
SomeModel.delete().execute()

for i in xrange(50):
    rows = ({"a": 5, "b": "Hello world"} for i in xrange(5000))
    with sqlite_db.atomic():
        SomeModel.insert_many(rows).execute()
        # Uncomment and notice that memory consumption is flat
        # for r in rows:
        #    SomeModel.insert(**r).execute()

    gc.collect()
    process = psutil.Process(os.getpid())
    print("Memory: %s" % sizeof_fmt(process.memory_info().rss))
```

You tha man, glad you got it working.