Issue: Free memory from cache?
Author: SamNPowers
State: closed
Created: 2021-05-03 21:17:27

Hi, thanks for the great repo.

I have a db Model with one very large field on it that I don't need to access very often. I would like to be able to load it in, then remove it from cache after I'm done with it, to free the RAM for other things. I saw that iterator() exists, but it doesn't quite fit my use case. Could there be a more granular mechanism? (Or if there already is, apologies! I couldn't find it in the documentation.)

Thanks!

Unfortunately, besides `.iterator()` the only other thing you might do would be to explicitly `.select()` all columns except for the large one. You could then implement a property/accessor to load the large column on-demand.

```python

class Post(Model):
    title = TextField()
    _attachment = BlobField(column_name='attachment')  # may have large binary data, e.g.

    @property
    def attachment(self):
        if self._attachment is None:
            post = Post.select(Post._attachment).get(Post.id == self.id)
            self._attachment = post._attachment
        return self_attachment
```

I do have a property/accessor to load the large column on demand. The problem is unloading it after I'm done with it. 

Would deleting the object and then re-loading all but the large column be the best way forward?

>The problem is unloading it after I'm done with it.

You can use property deleters to free the memory:

```python

class Post(Model):
    @property
    def attachment(self):
        # same as above...

    @attachment.deleter
    def attachment(self):
        self.__data__.pop('_attachment', None)

p = Post...
p.attachment  # load from memory or pull from database
del p.attachment  # frees cached data in model instance
```

Is there an easy way to do this for an entry in a ManyToManyField?

```
from peewee import SqliteDatabase, Model, TextField, BlobField, ManyToManyField


class Post(Model):
    title = TextField()
    _attachment = BlobField(column_name='attachment')  # may have large binary data, e.g.

    @property
    def attachment(self):
        if self._attachment is None:
            post = Post.select(Post._attachment).get(Post.id == self.id)
            self._attachment = post._attachment
        return self._attachment


    @attachment.deleter
    def attachment(self):
        self.__data__.pop('_attachment', None)


class PostCollector(Model):
    post_collection = ManyToManyField(Post)


if __name__ == "__main__":
    db = SqliteDatabase("test_db")
    PostCollectionRelationship = PostCollector.post_collection.get_through_model()
    db.bind([Post, PostCollector, PostCollectionRelationship])
    if not Post.table_exists():
        db.create_tables([Post, PostCollector, PostCollectionRelationship])
    db.connect(reuse_if_open=True)

    post = Post.create(title="foo", _attachment="bar")
    fetched_posts = Post.select().limit(1).execute()  # Get an entry (saves us from needing to specify id above)
    collector = PostCollector.create()
    collector.post_collection.add(fetched_posts[0])

    attachment = collector.post_collection[0].attachment
    del collector.post_collection[0].attachment
    print(collector.post_collection[0]._attachment)
```

It seems to work if you're manipulating the Post directly, but if it was accessed via a ManyToManyField, it doesn't seem to hold. (b'bar' prints after execution of the above code, not "attribute does not exist" or something like it.)

Because you're accessing it through the property, which re-fetches it from the db. Your code:

1. gets the attachment
2. frees the cached attachment data
3. re-fetches it from the db

Er, I'm sorry, the code has a typo. The last "attachment" is supposed to be "_attachment". Fixed it. But it still outputs b"bar".

In this case, the field shouldn't be getting refetched, since it's not going through the accessor. But nevertheless it is still there.

Oh, I see - every time you index into that `collector.post_collection` (`collector.post_collection[0]`) you're executing a query and getting a new instance from the db.

If you pull it into a variable then it will work:

```python
    post = collector.post_collection[0]
    print(post.attachment)
    del post.attachment
    print(post._attachment)
```

I do need to be able to manipulate (my version of) post_collection in some ways. My goal was to be able to ignore this specific field on posts except when it's necessary. Is it the case that every action (such as fetching a set of random posts) will re-fetch all attachments? Is there a convenient way to tell the ManyToManyField not to re-fetch its entries in this way? (E.g. if I want to use collector.post_collection.order_by(fn.Rand()).limit(5), will this cause all attachments to be re-fetched? Just the set returned? Either is undesirable, for my use case.)

Relatedly: is there a recommended way to view the amount of RAM a given model is consuming? I've in general been using `from pympler import asizeof`, then using `asizeof.asizeof(obj)`, but neither adding elements to post_collection nor removing attachment from elements are causing changes in size of post_collection or collector (while a reduction is seen for individual posts). As I can't access elements on post_collection directly to check whether attachment is or is not on them again without ensuring the answer is "yes", I'm not sure how to debug this.

My essential problem is that even with del'ing these large fields off of my objects, my memory consumption has not been reduced as I would expect.

Also, it would seem a ManyToManyField can't be pop'd off `__data__` the way other fields can (as `__data__` doesn't seem to contain it). What would the analogy be for a ManyToManyField?