Issue: Unreliable hybrid_fields order in model_to_dict(model, extra_attrs=hybrid_fields) bug?
Author: neutralvibes
State: closed
Created: 2022-01-13 14:38:27

# Hybrid fields output order from model_to_dict - bug?

## Problem
 - Python Version: 3.7.6 
 
I have a case where a gui table requires row data in tuple format. This must be ordered correctly in relation to the provided headers. The data is supplied by a model with a couple of hybrid fields.

I am finding that irrespective of the order provided to the `extra_attrs` parameter of `model_to_dict`, while the order of the data directly from the database is predictable, the order of the resultant hybrid fields attached is not in my case. 

## Example

Here is an example missing definitions but, I hope gives enough to demonstate with a fictious Person table where `status` and `age` are hybrids getting their data from other Person fields. 

~~~python
# Fields to list, * denotes hybrid field
display_list = [ 'id', 'name', '*status', '*age' ] 

# Get hybrids list
hybrid_fields = [ f for f in display_list if f.startswith('*') ]

# Printing shows supplied order consistently
print(f'hybrids: {hybrid_fields}')

# Output: hybrids: ['status', 'age']

# Grab list of non hybrid fields to extract 
only_prep = [ f for f in display_list if not f.startswith('*') ]

# Get attributes for `only_list` from table class
only_list = ( attrgetter(*only_prep)(Person) ) if len(only_prep) else ()

print(only_list)
# Output: (<AutoField: Person.id>, <CharField: Person.name>) 

table_data = [] # to hold return from encapsulating function

for rec in Person.select():
    row = model_to_dict( rec, only=only_list, extra_attrs=hybrid_fields )
    
    print( row.keys() )

    # Expected: dict_keys(['id', 'name', 'status','age']) ....and so on
    # Sometimes on python restart: dict_keys(['id', 'name', 'age', 'status']) .... notice the change in the last two. ​
    
    table_data.append( tuple(row.values()) )

~~~

## Tales of the unexpected

What is bizzare is that while python is running the order returned seems fixed. Run the program from scratch a few times after a period then it can alternate.

## Advice appreciated

Not sure if I'm doing something wrong. Tried making `hybrid_fields `  a tuple with no joy. Any suggestions would be appreciated as I am not sure if this is a bug or an interface with the keyboard problem.


~~Despite spending a few hours on it, a slap from myself helped me realise that perhaps OrderedDict isn't being used, despite database fields seemingly reproduced in reliable in order.~~ Nope, dict keys became reliably ordered from python 3.7.

Dict keys are not sorted -- exactly. This is not an issue in my opinion, this is just the nature of dictionaries. A dictionary is an inherently unordered data-structure, and while later Python preserves the insert order, it's not something to rely upon.

You already have your list of attrs - so you probably want to do something like: `table.append([getattr(model, attr) for attr in list_of_attrs])`. Or just pull the keys out of the resulting dict in the order you need them.

First, many thanks for reviewing.

Dicts have guaranteed order as of python 3.7 so can be relied upon. 
 I have looked at the underlyng code and the reason is you convert the input to a `Set`. The reason for the lack of stability is because of a shortcut you use to convert the input.

~~~python
_clone_set = lambda s: set(s) if s else set()
~~~

Sets still do not have this assurance, and I wonder if using a List would be better saving on converting many rows of data for the sake of it. I bow to your superior knowledge so shall leave it there.

Thanks again.

We don't bow to anyone in Peewee!

I think my recommendation to you is the same:

* either do not use `model_to_dict()` and just grab the attrs you want explicitly
* or, once you have the dict back, pull the fields out of it explicitly using a list comprehension or the like