Issue: What's the proper way of serializing a dynamically extended model for use in another process?
Author: Jessime
State: closed
Created: 2022-11-04 00:32:18

I can hack my way out of this situation, but it seems a common enough pattern that there might be a conical way of doing it that I'm not able to Google.

I want to be able to:
1. Dynamically add fields to some basic model, then save a bunch of data to a db.
2. In a second python process, I want to be able to query the data in the db I just created, using the dynamically extended class

However, if I try serializing the class with `pickle`, the dynamically added fields are missing from the model. Here's some toy code to demonstrate:

```python
import peewee
import pickle

DB = peewee.SqliteDatabase("test1.db")

class BasicModel(peewee.Model):
    class Meta:
        database = DB
    
    header = peewee.CharField()

user_defined_field_name = "my_field"
BasicModel._meta.add_field(user_defined_field_name, peewee.CharField())

row1 = BasicModel(header="header", my_field="data")
row1.save()

# This part runs, but doesn't work as intended
pickle.dump(BasicModel, open("BasicModel.pkl", "wb"))
```

So great, I have a db that has the data I want in it. Now, in a second process, I'd like to be able to query that data:

```python
import pickle

BasicModel = pickle.load(open("BasicModel.pkl", "rb"))
example = BasicModel.select().limit(1)[0]
print(example.my_field)
```

Unfortunately, this results in:

```python
AttributeError: 'VariantRecord' object has no attribute 'my_field'
```

A couple of sanity checks:
1. In the first process, I'm able to do `print(dir(BasicModel))` and see `my_field`
2. In the second process, I'm able to do `print(example.header)`

It seems pretty clear that `pickle` isn't serializing anything from `add_field`, but I'm not sure what to do about it. I don't think it would be that hard to pickle enough information to re-execute the `add_field` code at runtime of the second process, but it's still a hassle I'd rather avoid if possible.

Any suggestions? Thanks!

So you're wanting to store a pickle of the model class itself? My own interpretation without knowing any of the context is that you've got an accretion of hacks and are looking for more hacks to add on top, without considering that maybe the whole approach is flawed. When pickle looks up the class in the 2nd process, it doesn't serialize the model class and all its attributes -- it's just storing basically the path to the class (`__main__.BasicModel`). So you would have to apply your changes there, too.

That said if you want to treat your model classes themselves as just more data, I'd suggest an intermediate format like `{"BasicModel": [["field1", "int"], ["field2", "text"]]}` and create them dynamically.

Hi! Thanks for the quick response.

> you've got an accretion of hacks and are looking for more hacks to add on top

Thankfully, I'm just starting out on this project. So, no hacks yet, just some early code smells. Hopefully you don't mind if I give this question one more attempt. I suspect that 1) a little more context might make things clearer 2) I have a gut feeling there's an interesting point here.

First, a little more context. I'm building a CLI to deal with fairly messy bioinformatics user data, which is coming in as multiple flat text files. Each file contains a handful of required columns and likely multiple optional/custom fields as well. The whole spec for a `.vcf` file is a whopping 29 pages long, if you're morbidly curious. https://samtools.github.io/hts-specs/VCFv4.2.pdf

Because all of the files (despite having different columns) all contain information about genomic variants, there's a lot of value to be gained from loading these files into a database. But it's impossible to know a priori what the columns are going to be. 

So as I mentioned previously, step 1 is to build the db. That's not too bad. We've got the `add_field` API, so no hacks needed there. 

Now, how do I help users visualize/query/analyze/etc. their new db in future processes, given that I don't know their db schema? Pickling the class created during the load was one idea, dumping a custom intermediate format is another, but I'm by no means tied to either of them. A third option I thought of in writing up this response is that maybe [pwiz](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#pwiz-a-model-generator) is a reasonable approach?

Finally, I have a philosophical question. Given that `add_field` exists, dynamic generation of models seems a supported, non-hack pattern. However, since there's no standard way of saving the generated/modified model, does that imply that the modified model is only valid for the lifetime of the originating process? Is there a specific reason why one should avoid using a modified data model in a downstream process?

I'm by no means a db expert, so genuinely curious to hear thoughts!

>Now, how do I help users visualize/query/analyze/etc. their new db in future processes, given that I don't know their db schema?

You'll have to use introspection at some point or other -- take a look at `dataset`, it might be just what you need: http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#dataset -- it can handle messy data and makes heavy use of the introspection stuff that pwiz also uses.

>does that imply that the modified model is only valid for the lifetime of the originating process?

You're conflating a pickled class with the actual class itself. The class itself is a perfectly valid python construct, however you created it. The issue is that pickle does not serialize all the attributes of the class. It serializes the import path. That's it.

```python
In [1]: from peewee import *

In [2]: db = SqliteDatabase(':memory:')

In [3]: class Reg(db.Model):
   ...:     key = TextField()
   ...: 

In [4]: import pickle

In [5]: pickle.dumps(Reg)
Out[5]: b'\x80\x03c__main__\nReg\nq\x00.'
```

See how "key" appears nowhere in there? Neither does "id" (which is also a field on the model).

The models are perfectly valid wherever they're created, however they're created. You just need to find a serialization transport or other method of reading and writing data that does not rely on a broken/flawed assumption.

Ah, I should have used a more descriptive word than just "valid". I meant valid not so much in the technical sense as in a "this is a normal pattern we support in peewee" sense. In other words, maybe there could/should be a canonical serialization method, perhaps wrapped in `Model.serialize()`?

 Regardless of whether that's a reasonable suggestion:
 1. I played around with `pwiz` and it get's me where I need to go
 2. I've actually used `dataset` a ton and love its simplicity and ease-of-use
 3. Thanks for all the time and effort you've put into this library!

The database schema is always the ultimate source of truth, regardless of the model class -- that's why introspection / `playhouse.reflection` is probably the best bet.