Repository: evennia/evennia

Issue:
[BUG] cannot serialize dict subclasses
#### Describe the bug
Dear evennia developers, I've found that I'm not able to persist a dict subclass and after tracking it down a bit I think there's a generic problem when working w/ base types subclasses.

#### To Reproduce
A simple PoC using a dict:
1. put this in `/typeclasses/foo.py`:
```python
from evennia import CmdSet, Command, DefaultObject


class MaskedDict(dict):
    pass


class CmdFoo(Command):

    key = "cf"

    def func(self):
        self.obj.db.foo = MaskedDict({"foo": 1, "bar": 2})


class CmdSetFoo(CmdSet):
    def at_cmdset_creation(self):
        self.add(CmdFoo())


class FooObject(DefaultObject):
    def at_object_creation(self):
        self.cmdset.add_default(CmdSetFoo)
```
2. reload and create a foo object
```
reload
create foo :foo.FooObject
```
3. try to persist a dict subclass in foo obj:
```
cf
Traceback (most recent call last):
  File "/home/oggei/dev/evennia/evennia/commands/cmdhandler.py", line 621, in _run_command
    ret = cmd.func()
  File "/home/oggei/dev/evennia/dict-bug/typeclasses/foo.py", line 13, in func
    self.obj.db.foo = MaskedDict({"foo": 1, "bar": 2})
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1414, in __setattr__
    _GA(self, _GA(self, "name")).add(attrname, value)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1251, in add
    self.backend.create_attribute(keystr, category, lockstring, value, strattr)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 697, in create_attribute
    attr = self.do_create_attribute(key, category, lockstring, value, strvalue)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1033, in do_create_attribute
    kwargs["db_value"] = to_pickle(value)
  File "/home/oggei/dev/evennia/evennia/utils/dbserialize.py", line 663, in to_pickle
    return process_item(data)
  File "/home/oggei/dev/evennia/evennia/utils/dbserialize.py", line 650, in process_item
    return item.__class__([process_item(val) for val in item])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

An untrapped error occurred.
(Traceback was logged 22-07-19 15:38:57).
```
4. confirm that nothing has indeed ben set
```
examine foo
--------------------------------------------------------------------------------
Name/key: foo (#4)
Typeclass: FooObject (typeclasses.foo.FooObject)
Location: oggei (#1)
Home: oggei (#1)
Locks:   call:true(); control:id(1) or perm(Admin); delete:id(1) or perm(Admin);
  drop:holds(); edit:perm(Admin); examine:perm(Builder); get:all();
  puppet:pperm(Developer); tell:perm(Admin); view:all()
Stored Cmdset(s):
  typeclasses.foo.CmdSetFoo [Unnamed CmdSet] (Union, prio 0)
Merged Cmdset(s):
  typeclasses.foo.CmdSetFoo [Unnamed CmdSet] (Union, prio 0)
Commands available to foo (result of Merged Cmdset(s)):
  cf
Persistent Attributes:
  desc=You see nothing special.
--------------------------------------------------------------------------------
```

#### Expected behavior
I'd like to be able to persist those subclasses :)

#### Environment, Evennia version, OS etc
```
    Evennia 1.0-dev (rev ae5b353ef) (rev ae5b353ef)
    OS: posix
    Python: 3.9.13
    Twisted: 21.7.0
    Django: 4.0.6
```

the problem occurs in 0.9.5 too

#### Moving toward a solution

At a glance, it's clear that an eg. `dict`subclass will never match the correct [condition here](https://github.com/evennia/evennia/blob/ae5b353ef245450a05fba08e863b36d407600031/evennia/utils/dbserialize.py#L620).  
The first solution that come to mind is to check `isinstance` calls rather than `type` ones, by reordering the checks following inheritance chains bottom-up, which does not seems to pose particular diamond problems:

```mermaid
classDiagram

int <|-- bool


str <|-- django_utils_safestring_SafeString

django_utils_safestring_SafeData <|-- django_utils_safestring_SafeString



evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverList

collections_abc_MutableSequence <|-- evennia_utils_dbserialize__SaverList
collections_abc_Sequence <|-- collections_abc_MutableSequence
collections_abc_Reversible <|-- collections_abc_Sequence
collections_abc_Iterable <|-- collections_abc_Reversible

collections_abc_Collection <|-- collections_abc_Sequence
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection

dict <|-- collections_defaultdict

evennia_utils_dbserialize__SaverDict <|-- evennia_utils_dbserialize__SaverDefaultDict
evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverSet

collections_abc_MutableSet <|-- evennia_utils_dbserialize__SaverSet
collections_abc_Set <|-- collections_abc_MutableSet
collections_abc_Collection <|-- collections_abc_Set
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection

dict <|-- collections_OrderedDict

evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverOrderedDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverOrderedDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDeque

```

I could work on that but maybe I'm missing an obvious, already existing way of doing things so what do you think? 

Many thanks, regards
[BUG] cannot serialize dict subclasses
#### Describe the bug
Dear evennia developers, I've found that I'm not able to persist a dict subclass and after tracking it down a bit I think there's a generic problem when working w/ base types subclasses.

#### To Reproduce
A simple PoC using a dict:
1. put this in `/typeclasses/foo.py`:
```python
from evennia import CmdSet, Command, DefaultObject


class MaskedDict(dict):
    pass


class CmdFoo(Command):

    key = "cf"

    def func(self):
        self.obj.db.foo = MaskedDict({"foo": 1, "bar": 2})


class CmdSetFoo(CmdSet):
    def at_cmdset_creation(self):
        self.add(CmdFoo())


class FooObject(DefaultObject):
    def at_object_creation(self):
        self.cmdset.add_default(CmdSetFoo)
```
2. reload and create a foo object
```
reload
create foo :foo.FooObject
```
3. try to persist a dict subclass in foo obj:
```
cf
Traceback (most recent call last):
  File "/home/oggei/dev/evennia/evennia/commands/cmdhandler.py", line 621, in _run_command
    ret = cmd.func()
  File "/home/oggei/dev/evennia/dict-bug/typeclasses/foo.py", line 13, in func
    self.obj.db.foo = MaskedDict({"foo": 1, "bar": 2})
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1414, in __setattr__
    _GA(self, _GA(self, "name")).add(attrname, value)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1251, in add
    self.backend.create_attribute(keystr, category, lockstring, value, strattr)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 697, in create_attribute
    attr = self.do_create_attribute(key, category, lockstring, value, strvalue)
  File "/home/oggei/dev/evennia/evennia/typeclasses/attributes.py", line 1033, in do_create_attribute
    kwargs["db_value"] = to_pickle(value)
  File "/home/oggei/dev/evennia/evennia/utils/dbserialize.py", line 663, in to_pickle
    return process_item(data)
  File "/home/oggei/dev/evennia/evennia/utils/dbserialize.py", line 650, in process_item
    return item.__class__([process_item(val) for val in item])
ValueError: dictionary update sequence element #0 has length 3; 2 is required

An untrapped error occurred.
(Traceback was logged 22-07-19 15:38:57).
```
4. confirm that nothing has indeed ben set
```
examine foo
--------------------------------------------------------------------------------
Name/key: foo (#4)
Typeclass: FooObject (typeclasses.foo.FooObject)
Location: oggei (#1)
Home: oggei (#1)
Locks:   call:true(); control:id(1) or perm(Admin); delete:id(1) or perm(Admin);
  drop:holds(); edit:perm(Admin); examine:perm(Builder); get:all();
  puppet:pperm(Developer); tell:perm(Admin); view:all()
Stored Cmdset(s):
  typeclasses.foo.CmdSetFoo [Unnamed CmdSet] (Union, prio 0)
Merged Cmdset(s):
  typeclasses.foo.CmdSetFoo [Unnamed CmdSet] (Union, prio 0)
Commands available to foo (result of Merged Cmdset(s)):
  cf
Persistent Attributes:
  desc=You see nothing special.
--------------------------------------------------------------------------------
```

#### Expected behavior
I'd like to be able to persist those subclasses :)

#### Environment, Evennia version, OS etc
```
    Evennia 1.0-dev (rev ae5b353ef) (rev ae5b353ef)
    OS: posix
    Python: 3.9.13
    Twisted: 21.7.0
    Django: 4.0.6
```

the problem occurs in 0.9.5 too

#### Moving toward a solution

At a glance, it's clear that an eg. `dict`subclass will never match the correct [condition here](https://github.com/evennia/evennia/blob/ae5b353ef245450a05fba08e863b36d407600031/evennia/utils/dbserialize.py#L620).  
The first solution that come to mind is to check `isinstance` calls rather than `type` ones, by reordering the checks following inheritance chains bottom-up, which does not seems to pose particular diamond problems:

```mermaid
classDiagram

int <|-- bool


str <|-- django_utils_safestring_SafeString

django_utils_safestring_SafeData <|-- django_utils_safestring_SafeString



evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverList

collections_abc_MutableSequence <|-- evennia_utils_dbserialize__SaverList
collections_abc_Sequence <|-- collections_abc_MutableSequence
collections_abc_Reversible <|-- collections_abc_Sequence
collections_abc_Iterable <|-- collections_abc_Reversible

collections_abc_Collection <|-- collections_abc_Sequence
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection

dict <|-- collections_defaultdict

evennia_utils_dbserialize__SaverDict <|-- evennia_utils_dbserialize__SaverDefaultDict
evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverSet

collections_abc_MutableSet <|-- evennia_utils_dbserialize__SaverSet
collections_abc_Set <|-- collections_abc_MutableSet
collections_abc_Collection <|-- collections_abc_Set
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection

dict <|-- collections_OrderedDict

evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverOrderedDict

collections_abc_MutableMapping <|-- evennia_utils_dbserialize__SaverOrderedDict
collections_abc_Mapping <|-- collections_abc_MutableMapping
collections_abc_Collection <|-- collections_abc_Mapping
collections_abc_Sized <|-- collections_abc_Collection

collections_abc_Iterable <|-- collections_abc_Collection

collections_abc_Container <|-- collections_abc_Collection


evennia_utils_dbserialize__SaverMutable <|-- evennia_utils_dbserialize__SaverDeque

```

I could work on that but maybe I'm missing an obvious, already existing way of doing things so what do you think? 

Many thanks, regards


Discussion:
Oh, ha, I've been working with a dict subclass for a while but I guess I haven't tried to save it to an attribute directly - I've never run into this. Just tried it with my own subclass to see though and as expected, it hits the same issue.

I've been mulling on this a bit since you posted - why it happens and how to minimize other similar holes popping up later. The root problem isn't the elif chain, I think, but the fact that dictionaries are weird iterables. It has a nice generic iterable class processing block there which works great except for when you're using a dictionary subclass, specifically.

So my counterpoint suggestion is, instead of changing the existing type-checking logic, add an extra "generic" block specifically for dictionary subclasses before the `hasattr("__iter__")` one

e.g.
```py
if isinstance(item, dict):
    try:
        return item.__class__({process_item(key), process_item(val) for key, val in item.items()})
    except (AttributeError, TypeError):
        return {process_item(key), process_item(val) for key, val in item.items()}
```

That'd both acount for the dict iteration issues and also keep any unusual logic the subclass might have.
I'm not sure what exactly you are trying to do by adding this dictionary as a direct attribute. Evennia's Attribute system has some special handling of dictionaries and lists and nested data so that it can handle a good deal of use-cases, such as saving model references or lists. This might be what's causing an issue with subclasses of dict - it's trying to coerce what it thinks is a dict into a SaverDict.

Have you considered the Handler approach for your code, whatever it is? It's a good way of wrapping up complex logic attached to a Typeclass while using Attributes for serializing/deserializing data. Example:

```python
from evennia.utils.utils import lazy_property

class FooHandler:
    def __init__(self, owner):
        self.owner = owner
        self.data = None
        self.load()

    def load(self):
        self.data = self.owner.attributes.get("mydata", dict())


class GameObject(DefaultObject):

    @lazy_property
    def foo(self):
        return FooHandler(self)
```

This pattern allows you to do obj.foo.whatever(), creating methods and data members to FooHandler which manipulate your data and extend the typeclass's API.

My personal recommendation is to NOT store custom types at all within Attributes and instead just store primitives, lists, dicts, sets, and references to Django models, which is what Attributes are practically guaranteed to handle well without ever having problems. If you need logic in there, use Handlers as an interface to that data and have them load up your data-handling API classes.
@InspectorCaracal I never said the problem reside in elif chain, but that couple checks should flip order in an hypotetical type->isinstance switch, see the useless PoC [I've just pushed](https://github.com/aogier/evennia/commit/cb03fe660fbd421a52cd3d753d66ac6ffecc7784). Seems to be a dead-end tho, as I would want back my implementation when deserializing so this will not easily work.

@volundmush the problem here is the dict/saver dict is completely missed during serialization thus the code fail in the line which I highlighted and you can find in the backtrace. But that's ok, as this analysis has already been done and this is what happen :)  
Now, what I'm attempting to do is to store some secrets I don't want non-developers to see by any mean, but I don't want to override a lot of stuff/commands eg. `examine`. So a quick way was overriding repr/str dict's methods by removing what I needed.  
I choose this way because I'm studying everything as I'm returning after many years on that and I am sure that I will find a welcoming community, today as yesterday, "whatever my code is" :grinning: - do you have an idea on how I could keep some secret away from the eyes of non-`py` users ?
> @InspectorCaracal I never said the problem reside in elif chain, but that couple checks should flip order in an hypotetical type->isinstance switch, see the useless PoC [I've just pushed](https://github.com/aogier/evennia/commit/cb03fe660fbd421a52cd3d753d66ac6ffecc7784). Seems to be a dead-end tho, as I would want back my implementation when deserializing so this will not easily work.

Sorry, didn't mean to sound like I was correcting you; I was thinking through different approaches to solving it, one of which was rewriting the elif chain. And yeah... deserializing it is a whole different kettle of fish. There's no good way to do that besides implementing your own `_Saver` type or knowing which attributes you want to run through your subclass when accessed.
> Now, what I'm attempting to do is to store some secrets I don't want non-developers to see by any mean

Well... then don't give them access to @py or `examine`....

> but I don't want to override a lot of stuff/commands eg

I hear ya, but you'll probably end up doing this sooner or later, fair warning.

> I'm returning after many years

wibs!

> do you have an idea on how I could keep some secret away from the eyes of non-py users ?

`@examine`'s default locks are "cmd:perm(examine) or perm(Builder)"

If you don't have any Builders or Admin then only Developer/superuser is gonna be able to use it!

But, Builders really should have access to `@examine` obviously.

What Builders don't have, though, is `@py`.

You could create new Django models and store data in those. Those won't show up in `@examine`.

Attributes also can have an attrread lock which could hide them...
@InspectorCaracal yeah it looks like it could introduce a lot of complexity/coupling and I don't think it's a good idea, mine was just an unsuccessful exploration. If you want to iterate further on that count me in though :)

@volundmush 
> Well... then don't give them access to @py or examine....

a bit overshooting and non specs-friendly: my specs are I need people to be builder, so they ultimately need to `examine` stuff, but in very, super-specific case, I don't want them to see specific information bits (there are bearer tokens that other users activate on my system so as to authorize certain operations, I can't allow third parties to get  them but I would like the builders to be other of their peers who are more experienced in the moo but still their peers, and this is enough to disclose for now :P)  

To me, a less obtrusive way could seem to incapsulate those secret in another object then lock `examine` on it and then I'll consume this information via python only and that's it, I'm not completely happy w/ that because a rogue/inexperienced developer could then leak this info via other means but I guess some limitations exists anyway in realizing a real least privilege model. 


But instead getting back on topic: at the moment we have something which fails badly against some (unsupported?) behaviour which is easy to spot. In light of everything we've said to each other on the topic do you two think it might make sense to at least implement a more understandable error message?
Hrm. I think I still would go for having a dict-subclass-specific version of the `__iter__` block as probably the best solution for this particular traceback, with an amendment to the documentation that dict subclasses will be converted to dictionaries like how it mentions that non-builtin iterables are converted to lists.

p.s. I think the `attread` lock Volund mentioned at the end up there is what you need for your use-case. `obj.attributes.add("myattr", hidden_data, lockstring="attrread:perm(Developer);attredit:perm(Developer)")` should do it
ok tomorrow I'll put some brain on that as now I'm flatlined :) 

for `attread`: unless I'm missing something important, this way is feasible but it require to override everything, notably examine, for having it respected so it's a lot of work in respect of encapsulating stuff in an object w/ a locked `examine` lock and call it a day


Oh hey, you're right, there are no attribute lock checks in Examine. That makes the lock a bit useless doesn't it, it must only be used in the Set command. I might file an issue re: that later.
if examine isn't respecting Attribute locks, it should be made to do so. Otherwise, there's no point.
to me, it seemed a deliberate design choice not to implement such a check in such an important basic command, which is something I love in Evennia and I can understand in https://www.evennia.com/docs/latest/Attributes.html#locking-and-checking-attributes:

> A lock is no good if nothing checks it – and by default Evennia does not check locks on Attributes. You have to add a check to your commands/code wherever it fits (such as before setting an Attribute).

So I'd go w/ other solutions and leave the sane design choices alone, but I'm also nobody so apart my :-1: there's not much I can do :) 
> So I'd go w/ other solutions and leave the sane design choices alone, but I'm also nobody so apart my 👎 there's not much I can do :)

An argument you don't make is one lost by default. If you feel a change needs to be made to Evennia, speak up or make a PR.
Dear @volundmush , I absolutely made an argument, let me try to better phrase it:

> if examine isn't respecting Attribute locks, it should be made to do so. Otherwise, there's no point.

this seems to negate specific design choices which I like so I hope it does not happen. Plus, it's offtopic in respect to either the initial issue and the thread as have evolved, sticking to suggestions nobody has asked you for even if you decided not to use this annoying rudeness which happens to be your registry of choice for today.

I sincerely hope I did it better, this time.  



Now, back to the task: today I'll try to spend a bit of time and energy on what @InspectorCaracal proposed especially on docs' updates so a future newb like me could understand things faster. I'd be great if @volundmush would like to contribute too w/ something relevant I'm more than sure he could spare on that :)


To focus on the actual issue reported here :grin:  - I can't test it right now (on my way out the door), but I would have expected a subclassed dict to to work - but not work if you update it directly on the Attribute. That is, I would have expected you to be able to do 

    obj.db.mydict = MysubclassedDict({1:2, 3:4}) 

and then 

    mydict = obj.db.mydict 
    mydict[1] = 5
    obj.db.mydict = mydict 

but not 
  
    obj.db.mydict[1] = 5 

The latter should not work because the system does not understand that this is a dict and will not convert it to a `_SaverDict` that understands the second type (and besides, serializing it back from `_SaverDict` later would turn it into a normal `dict` in that case,  and not a `MysubclassedDict` anyway). 

So in short, if the first two steps works, this is not a bug - it's a use-case that is not supported (but we may have a feature request to figure out to support it). If the first two don't work either, then this is really a bug that needs to be fixed. Could someone verify which is the case? 

As for the other points raised later: 

The examine command does not check attribute locks, this is deliberately done, so as to make this command as useful as possible (you can't edit Attributes via `examine`). That said, if you explicitly want to hide data from your builders in Attributes, you can of course override `examine` to do just that. 

(Overall though, if your secrets are static, you probably want to put them in your `settings.py` or `secret_settings.py` instead than store them in an Attribute.)

Hello, super quick recap

It is not currently possible to persist dict subclasses because how [`to_pickle` works](https://github.com/evennia/evennia/blob/ae5b353ef245450a05fba08e863b36d407600031/evennia/utils/dbserialize.py#L609,L620).  
It has been suggested a documentation enhancement could be made to better clarify that at the moment this don't indeed work, I was about to start doing that when I got an idea on how to try to implement a tweak and ATM I've got a proper obj->db serialization with the other half of the trip maybe on its way :)

the secrets I want to hide aren't static as I'm implementing a cyberdeck which users activate in-game by completing a device flow oauth login, which at the end spits back a bearer token with high personal privileges I don't want to share by default w/ every builder in town.  
ATM I'm considering to encapsulate this specific bit of information in another object whose life cycle is managed by activation machinery and a locked down `examine` lock.
@Griatch 
> The examine command does not check attribute locks, this is deliberately done, so as to make this command as useful as possible (you can't edit Attributes via examine). That said, if you explicitly want to hide data from your builders in Attributes, you can of course override examine to do just that.

Methinks we should modify `@examine` so that there's a settings.py boolean that tells it to respect it.
> Methinks we should modify `@examine` so that there's a settings.py boolean that tells it to respect it.

looks like a great idea to me, as it's unobtrusive and adds a feature. Count me in, should you need help implementing/test covering this proposal!

due edit: could this live in another issue? It is not very pertinent to what we discuss here
Oh, ha, I've been working with a dict subclass for a while but I guess I haven't tried to save it to an attribute directly - I've never run into this. Just tried it with my own subclass to see though and as expected, it hits the same issue.

I've been mulling on this a bit since you posted - why it happens and how to minimize other similar holes popping up later. The root problem isn't the elif chain, I think, but the fact that dictionaries are weird iterables. It has a nice generic iterable class processing block there which works great except for when you're using a dictionary subclass, specifically.

So my counterpoint suggestion is, instead of changing the existing type-checking logic, add an extra "generic" block specifically for dictionary subclasses before the `hasattr("__iter__")` one

e.g.
```py
if isinstance(item, dict):
    try:
        return item.__class__({process_item(key), process_item(val) for key, val in item.items()})
    except (AttributeError, TypeError):
        return {process_item(key), process_item(val) for key, val in item.items()}
```

That'd both acount for the dict iteration issues and also keep any unusual logic the subclass might have.
I'm not sure what exactly you are trying to do by adding this dictionary as a direct attribute. Evennia's Attribute system has some special handling of dictionaries and lists and nested data so that it can handle a good deal of use-cases, such as saving model references or lists. This might be what's causing an issue with subclasses of dict - it's trying to coerce what it thinks is a dict into a SaverDict.

Have you considered the Handler approach for your code, whatever it is? It's a good way of wrapping up complex logic attached to a Typeclass while using Attributes for serializing/deserializing data. Example:

```python
from evennia.utils.utils import lazy_property

class FooHandler:
    def __init__(self, owner):
        self.owner = owner
        self.data = None
        self.load()

    def load(self):
        self.data = self.owner.attributes.get("mydata", dict())


class GameObject(DefaultObject):

    @lazy_property
    def foo(self):
        return FooHandler(self)
```

This pattern allows you to do obj.foo.whatever(), creating methods and data members to FooHandler which manipulate your data and extend the typeclass's API.

My personal recommendation is to NOT store custom types at all within Attributes and instead just store primitives, lists, dicts, sets, and references to Django models, which is what Attributes are practically guaranteed to handle well without ever having problems. If you need logic in there, use Handlers as an interface to that data and have them load up your data-handling API classes.
@InspectorCaracal I never said the problem reside in elif chain, but that couple checks should flip order in an hypotetical type->isinstance switch, see the useless PoC [I've just pushed](https://github.com/aogier/evennia/commit/cb03fe660fbd421a52cd3d753d66ac6ffecc7784). Seems to be a dead-end tho, as I would want back my implementation when deserializing so this will not easily work.

@volundmush the problem here is the dict/saver dict is completely missed during serialization thus the code fail in the line which I highlighted and you can find in the backtrace. But that's ok, as this analysis has already been done and this is what happen :)  
Now, what I'm attempting to do is to store some secrets I don't want non-developers to see by any mean, but I don't want to override a lot of stuff/commands eg. `examine`. So a quick way was overriding repr/str dict's methods by removing what I needed.  
I choose this way because I'm studying everything as I'm returning after many years on that and I am sure that I will find a welcoming community, today as yesterday, "whatever my code is" :grinning: - do you have an idea on how I could keep some secret away from the eyes of non-`py` users ?
> @InspectorCaracal I never said the problem reside in elif chain, but that couple checks should flip order in an hypotetical type->isinstance switch, see the useless PoC [I've just pushed](https://github.com/aogier/evennia/commit/cb03fe660fbd421a52cd3d753d66ac6ffecc7784). Seems to be a dead-end tho, as I would want back my implementation when deserializing so this will not easily work.

Sorry, didn't mean to sound like I was correcting you; I was thinking through different approaches to solving it, one of which was rewriting the elif chain. And yeah... deserializing it is a whole different kettle of fish. There's no good way to do that besides implementing your own `_Saver` type or knowing which attributes you want to run through your subclass when accessed.
> Now, what I'm attempting to do is to store some secrets I don't want non-developers to see by any mean

Well... then don't give them access to @py or `examine`....

> but I don't want to override a lot of stuff/commands eg

I hear ya, but you'll probably end up doing this sooner or later, fair warning.

> I'm returning after many years

wibs!

> do you have an idea on how I could keep some secret away from the eyes of non-py users ?

`@examine`'s default locks are "cmd:perm(examine) or perm(Builder)"

If you don't have any Builders or Admin then only Developer/superuser is gonna be able to use it!

But, Builders really should have access to `@examine` obviously.

What Builders don't have, though, is `@py`.

You could create new Django models and store data in those. Those won't show up in `@examine`.

Attributes also can have an attrread lock which could hide them...
@InspectorCaracal yeah it looks like it could introduce a lot of complexity/coupling and I don't think it's a good idea, mine was just an unsuccessful exploration. If you want to iterate further on that count me in though :)

@volundmush 
> Well... then don't give them access to @py or examine....

a bit overshooting and non specs-friendly: my specs are I need people to be builder, so they ultimately need to `examine` stuff, but in very, super-specific case, I don't want them to see specific information bits (there are bearer tokens that other users activate on my system so as to authorize certain operations, I can't allow third parties to get  them but I would like the builders to be other of their peers who are more experienced in the moo but still their peers, and this is enough to disclose for now :P)  

To me, a less obtrusive way could seem to incapsulate those secret in another object then lock `examine` on it and then I'll consume this information via python only and that's it, I'm not completely happy w/ that because a rogue/inexperienced developer could then leak this info via other means but I guess some limitations exists anyway in realizing a real least privilege model. 


But instead getting back on topic: at the moment we have something which fails badly against some (unsupported?) behaviour which is easy to spot. In light of everything we've said to each other on the topic do you two think it might make sense to at least implement a more understandable error message?
Hrm. I think I still would go for having a dict-subclass-specific version of the `__iter__` block as probably the best solution for this particular traceback, with an amendment to the documentation that dict subclasses will be converted to dictionaries like how it mentions that non-builtin iterables are converted to lists.

p.s. I think the `attread` lock Volund mentioned at the end up there is what you need for your use-case. `obj.attributes.add("myattr", hidden_data, lockstring="attrread:perm(Developer);attredit:perm(Developer)")` should do it
ok tomorrow I'll put some brain on that as now I'm flatlined :) 

for `attread`: unless I'm missing something important, this way is feasible but it require to override everything, notably examine, for having it respected so it's a lot of work in respect of encapsulating stuff in an object w/ a locked `examine` lock and call it a day


Oh hey, you're right, there are no attribute lock checks in Examine. That makes the lock a bit useless doesn't it, it must only be used in the Set command. I might file an issue re: that later.
if examine isn't respecting Attribute locks, it should be made to do so. Otherwise, there's no point.
to me, it seemed a deliberate design choice not to implement such a check in such an important basic command, which is something I love in Evennia and I can understand in https://www.evennia.com/docs/latest/Attributes.html#locking-and-checking-attributes:

> A lock is no good if nothing checks it – and by default Evennia does not check locks on Attributes. You have to add a check to your commands/code wherever it fits (such as before setting an Attribute).

So I'd go w/ other solutions and leave the sane design choices alone, but I'm also nobody so apart my :-1: there's not much I can do :) 
> So I'd go w/ other solutions and leave the sane design choices alone, but I'm also nobody so apart my 👎 there's not much I can do :)

An argument you don't make is one lost by default. If you feel a change needs to be made to Evennia, speak up or make a PR.
Dear @volundmush , I absolutely made an argument, let me try to better phrase it:

> if examine isn't respecting Attribute locks, it should be made to do so. Otherwise, there's no point.

this seems to negate specific design choices which I like so I hope it does not happen. Plus, it's offtopic in respect to either the initial issue and the thread as have evolved, sticking to suggestions nobody has asked you for even if you decided not to use this annoying rudeness which happens to be your registry of choice for today.

I sincerely hope I did it better, this time.  



Now, back to the task: today I'll try to spend a bit of time and energy on what @InspectorCaracal proposed especially on docs' updates so a future newb like me could understand things faster. I'd be great if @volundmush would like to contribute too w/ something relevant I'm more than sure he could spare on that :)


To focus on the actual issue reported here :grin:  - I can't test it right now (on my way out the door), but I would have expected a subclassed dict to to work - but not work if you update it directly on the Attribute. That is, I would have expected you to be able to do 

    obj.db.mydict = MysubclassedDict({1:2, 3:4}) 

and then 

    mydict = obj.db.mydict 
    mydict[1] = 5
    obj.db.mydict = mydict 

but not 
  
    obj.db.mydict[1] = 5 

The latter should not work because the system does not understand that this is a dict and will not convert it to a `_SaverDict` that understands the second type (and besides, serializing it back from `_SaverDict` later would turn it into a normal `dict` in that case,  and not a `MysubclassedDict` anyway). 

So in short, if the first two steps works, this is not a bug - it's a use-case that is not supported (but we may have a feature request to figure out to support it). If the first two don't work either, then this is really a bug that needs to be fixed. Could someone verify which is the case? 

As for the other points raised later: 

The examine command does not check attribute locks, this is deliberately done, so as to make this command as useful as possible (you can't edit Attributes via `examine`). That said, if you explicitly want to hide data from your builders in Attributes, you can of course override `examine` to do just that. 

(Overall though, if your secrets are static, you probably want to put them in your `settings.py` or `secret_settings.py` instead than store them in an Attribute.)

Hello, super quick recap

It is not currently possible to persist dict subclasses because how [`to_pickle` works](https://github.com/evennia/evennia/blob/ae5b353ef245450a05fba08e863b36d407600031/evennia/utils/dbserialize.py#L609,L620).  
It has been suggested a documentation enhancement could be made to better clarify that at the moment this don't indeed work, I was about to start doing that when I got an idea on how to try to implement a tweak and ATM I've got a proper obj->db serialization with the other half of the trip maybe on its way :)

the secrets I want to hide aren't static as I'm implementing a cyberdeck which users activate in-game by completing a device flow oauth login, which at the end spits back a bearer token with high personal privileges I don't want to share by default w/ every builder in town.  
ATM I'm considering to encapsulate this specific bit of information in another object whose life cycle is managed by activation machinery and a locked down `examine` lock.
@Griatch 
> The examine command does not check attribute locks, this is deliberately done, so as to make this command as useful as possible (you can't edit Attributes via examine). That said, if you explicitly want to hide data from your builders in Attributes, you can of course override examine to do just that.

Methinks we should modify `@examine` so that there's a settings.py boolean that tells it to respect it.
> Methinks we should modify `@examine` so that there's a settings.py boolean that tells it to respect it.

looks like a great idea to me, as it's unobtrusive and adds a feature. Count me in, should you need help implementing/test covering this proposal!

due edit: could this live in another issue? It is not very pertinent to what we discuss here

Resolving patch:
diff --git a/evennia/utils/dbserialize.py b/evennia/utils/dbserialize.py
--- a/evennia/utils/dbserialize.py
+++ b/evennia/utils/dbserialize.py
@@ -263,7 +263,7 @@ class _SaverList(_SaverMutable, MutableSequence):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        self._data = list()
+        self._data = kwargs.pop("_class", list)()
 
     @_save
     def __iadd__(self, otherlist):
@@ -307,7 +307,7 @@ class _SaverDict(_SaverMutable, MutableMapping):
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
-        self._data = dict()
+        self._data = kwargs.pop("_class", dict)()
 
     def has_key(self, key):
         return key in self._data
@@ -645,11 +645,20 @@ def process_item(item):
                 pass
 
         if hasattr(item, "__iter__"):
-            # we try to conserve the iterable class, if not convert to list
             try:
-                return item.__class__([process_item(val) for val in item])
-            except (AttributeError, TypeError):
-                return [process_item(val) for val in item]
+                # we try to conserve the iterable class, if not convert to dict
+                try:
+                    return item.__class__(
+                        (process_item(key), process_item(val)) for key, val in item.items()
+                    )
+                except (AttributeError, TypeError):
+                    return {process_item(key): process_item(val) for key, val in item.items()}
+            except:
+                # we try to conserve the iterable class, if not convert to list
+                try:
+                    return item.__class__([process_item(val) for val in item])
+                except (AttributeError, TypeError):
+                    return [process_item(val) for val in item]
         elif hasattr(item, "sessid") and hasattr(item, "conn_time"):
             return pack_session(item)
         try:
@@ -714,11 +723,20 @@ def process_item(item):
             return deque(process_item(val) for val in item)
         elif hasattr(item, "__iter__"):
             try:
-                # we try to conserve the iterable class if
-                # it accepts an iterator
-                return item.__class__(process_item(val) for val in item)
-            except (AttributeError, TypeError):
-                return [process_item(val) for val in item]
+                # we try to conserve the iterable class, if not convert to dict
+                try:
+                    return item.__class__(
+                        (process_item(key), process_item(val)) for key, val in item.items()
+                    )
+                except (AttributeError, TypeError):
+                    return {process_item(key): process_item(val) for key, val in item.items()}
+            except:
+                try:
+                    # we try to conserve the iterable class if
+                    # it accepts an iterator
+                    return item.__class__(process_item(val) for val in item)
+                except (AttributeError, TypeError):
+                    return [process_item(val) for val in item]
 
         if hasattr(item, "__deserialize_dbobjs__"):
             # this allows the object to custom-deserialize any embedded dbobjs
@@ -780,13 +798,30 @@ def process_tree(item, parent):
             return dat
         elif hasattr(item, "__iter__"):
             try:
-                # we try to conserve the iterable class if it
-                # accepts an iterator
-                return item.__class__(process_tree(val, parent) for val in item)
-            except (AttributeError, TypeError):
-                dat = _SaverList(_parent=parent)
-                dat._data.extend(process_tree(val, dat) for val in item)
-                return dat
+                # we try to conserve the iterable class, if not convert to dict
+                try:
+                    dat = _SaverDict(_parent=parent, _class=item.__class__)
+                    dat._data.update(
+                        (process_item(key), process_tree(val, dat)) for key, val in item.items()
+                    )
+                    return dat
+                except (AttributeError, TypeError):
+                    dat = _SaverDict(_parent=parent)
+                    dat._data.update(
+                        (process_item(key), process_tree(val, dat)) for key, val in item.items()
+                    )
+                    return dat
+            except:
+                try:
+                    # we try to conserve the iterable class if it
+                    # accepts an iterator
+                    dat = _SaverList(_parent=parent, _class=item.__class__)
+                    dat._data.extend(process_tree(val, dat) for val in item)
+                    return dat
+                except (AttributeError, TypeError):
+                    dat = _SaverList(_parent=parent)
+                    dat._data.extend(process_tree(val, dat) for val in item)
+                    return dat
 
         if hasattr(item, "__deserialize_dbobjs__"):
             try:
@@ -800,7 +835,9 @@ def process_tree(item, parent):
         # convert lists, dicts and sets to their Saved* counterparts. It
         # is only relevant if the "root" is an iterable of the right type.
         dtype = type(data)
-        if dtype == list:
+        if dtype in (str, int, float, bool, bytes, SafeString, tuple):
+            return process_item(data)
+        elif dtype == list:
             dat = _SaverList(_db_obj=db_obj)
             dat._data.extend(process_tree(val, dat) for val in data)
             return dat
@@ -830,6 +867,34 @@ def process_tree(item, parent):
             dat = _SaverDeque(_db_obj=db_obj)
             dat._data.extend(process_item(val) for val in data)
             return dat
+        elif hasattr(data, "__iter__"):
+            try:
+                # we try to conserve the iterable class, if not convert to dict
+                try:
+                    dat = _SaverDict(_db_obj=db_obj, _class=data.__class__)
+                    dat._data.update(
+                        (process_item(key), process_tree(val, dat)) for key, val in data.items()
+                    )
+                    return dat
+                except (AttributeError, TypeError):
+                    dat = _SaverDict(_db_obj=db_obj)
+                    dat._data.update(
+                        (process_item(key), process_tree(val, dat)) for key, val in data.items()
+                    )
+                    return dat
+            except:
+                try:
+                    # we try to conserve the iterable class if it
+                    # accepts an iterator
+                    dat = _SaverList(_db_obj=db_obj, _class=data.__class__)
+                    dat._data.extend(process_tree(val, dat) for val in data)
+                    return dat
+
+                except (AttributeError, TypeError):
+                    dat = _SaverList(_db_obj=db_obj)
+                    dat._data.extend(process_tree(val, dat) for val in data)
+                    return dat
+
     return process_item(data)
 
 


Test patch:
diff --git a/evennia/typeclasses/tests.py b/evennia/typeclasses/tests.py
--- a/evennia/typeclasses/tests.py
+++ b/evennia/typeclasses/tests.py
@@ -2,9 +2,10 @@
 Unit tests for typeclass base system
 
 """
+
 from django.test import override_settings
-from evennia.utils.test_resources import BaseEvenniaTest, EvenniaTestCase
 from evennia.typeclasses import attributes
+from evennia.utils.test_resources import BaseEvenniaTest, EvenniaTestCase
 from mock import patch
 from parameterized import parameterized
 
@@ -13,6 +14,10 @@
 # ------------------------------------------------------------
 
 
+class DictSubclass(dict):
+    pass
+
+
 class TestAttributes(BaseEvenniaTest):
     def test_attrhandler(self):
         key = "testattr"
@@ -22,6 +27,25 @@ def test_attrhandler(self):
         self.obj1.db.testattr = value
         self.assertEqual(self.obj1.db.testattr, value)
 
+        # "plain" subclasses
+        value = DictSubclass({"fo": "foo", "bar": "bar"})
+        self.obj1.db.testattr = value
+        self.assertEqual(self.obj1.db.testattr, value)
+
+        self.obj1.db.testattr["fo"] = "foo2"
+        value.update({"fo": "foo2"})
+        self.assertEqual(self.obj1.db.testattr, value)
+        self.assertEqual(self.obj1.attributes.get("testattr"), value)
+
+        # nested subclasses
+        value = DictSubclass({"nested": True, "deep": DictSubclass({"fo": "foo", "bar": "bar"})})
+        self.obj1.db.testattr = value
+
+        self.obj1.db.testattr["deep"]["fo"] = "nemo"
+        value["deep"].update({"fo": "nemo"})
+        self.assertEqual(self.obj1.db.testattr, value)
+        self.assertEqual(self.obj1.attributes.get("testattr"), value)
+
     @override_settings(TYPECLASS_AGGRESSIVE_CACHE=False)
     @patch("evennia.typeclasses.attributes._TYPECLASS_AGGRESSIVE_CACHE", False)
     def test_attrhandler_nocache(self):
@@ -35,6 +59,27 @@ def test_attrhandler_nocache(self):
         self.assertEqual(self.obj1.db.testattr, value)
         self.assertFalse(self.obj1.attributes.backend._cache)
 
+        # "plain" subclasses
+        value = DictSubclass({"fo": "foo", "bar": "bar"})
+        self.obj1.db.testattr = value
+        self.assertEqual(self.obj1.db.testattr, value)
+
+        self.obj1.db.testattr["fo"] = "foo2"
+        value.update({"fo": "foo2"})
+        self.assertEqual(self.obj1.db.testattr, value)
+        self.assertEqual(self.obj1.attributes.get("testattr"), value)
+        self.assertFalse(self.obj1.attributes.backend._cache)
+
+        # nested subclasses
+        value = DictSubclass({"nested": True, "deep": DictSubclass({"fo": "foo", "bar": "bar"})})
+        self.obj1.db.testattr = value
+
+        self.obj1.db.testattr["deep"]["fo"] = "nemo"
+        value["deep"].update({"fo": "nemo"})
+        self.assertEqual(self.obj1.db.testattr, value)
+        self.assertEqual(self.obj1.attributes.get("testattr"), value)
+        self.assertFalse(self.obj1.attributes.backend._cache)
+
     def test_weird_text_save(self):
         "test 'weird' text type (different in py2 vs py3)"
         from django.utils.safestring import SafeText