Issue: Detect (and lazy load) columns that were not loaded from DB
Author: tuukkamustonen
State: closed
Created: 2016-09-28 10:25:26

With:

``` python
class Entity(Model):
    small_field = ...
    large_field = ...
    huge_field = ...
```

Large and huge field may contain big chunks of data and are not always needed.

Consider:

``` python
data = [
    Entity.get(id=1),
    Entity.select(Entity.id, Entity.small_field).where(Entity.id == 2).get(),
    Entity.select(Entity.id, Entity.small_field, Entity.large_field).where(Entity.id == 3).get(),
]

for obj in data:
    print(len(obj.large_field) + len(obj.huge_field))  # how to lazy load the missing fields here?
```

Is there a property to check or a method to call to check if a field has been populated from the DB? `_data` also contains fields that have `default` defined so I can't use that...

Even better, would be nice if peewee could assist in loading these missing fields. However, if the fields were simply lazily loaded, the above example would generate 1 extra SELECT per missing column, which is not very nice.

So maybe something like this:

``` python
for obj in data:
    obj.load_missing()  # would refresh missing fields, if any, 1 SELECT
    print(len(obj.large_field) + len(obj.huge_field))
```

Or maybe:

``` python
for obj in data:
    obj.ensure_fields(Entity.large_field, Entity.huge_field)  # would only load if they are really missing, 1 SELECT in total
    print(len(obj.large_field) + len(obj.huge_field))
```


Peewee does not, to the extent that makes sense to me, do things that violate expectations or issue queries. If you think of the model instances as containers for row data, instead of stateful representations of query results, then I think you might be closer to what I intended the library to be. The general idea of issuing queries to get individual column values "behind-the-scenes" seems like the kind of magic I would rather avoid, because it can lead to very inefficient applications that appear to work. If you are explicitly selecting a sub-set of columns, presumably your application knows which columns are un-fetched and you could write a helper method to load those from the database yourself. That's the approach I'd take.


@coleifer I understand the skepticism against additional magic. However, doesn't peewee already do behind-the-scenes magic with lazy-loading `ForeignKeys`? Having similar behavior for other fields is not very different from that.

Also, I wouldn't call it "magic" if we required user to call a method (`load_missing ()` and `ensure_fields (...)` in my example) to load up the remaining fields. To clarify, I don't really want lazy loading either, because it would generate unneeded SELECTs to the db. It's better when you have to explicitly call something.

As this is probably not very popular need, I'm fine with peewee not supporting lazy loading fields. However, would it be possible to allow detecting which fields were not populated from the DB? Yeah, I could attach this info myself during load-time, but it would be cleaner (and better for debugging) if I could just check this from the model instance directly. Is there a specific need why defaults get populated into `_data` while non-loaded fields without default value dont?


Comparing this to the lazy-loading of foreign-keys isn't quite apples-to-apples. You can always add something like this to your base model class in your project:

``` python
class BaseModel(Model):
    def refresh(self):
        return type(self).get(self._meta.primary_key == self._get_pk_value())
```

> would it be possible to allow detecting which fields were not populated from the DB

As I mentioned, since your application is controlling this, theoretically it should be known to your application which fields need to be reloaded.