Repository: evennia/evennia

Issue:
Nested attribute acccess
#### Brief summary of issue / Description of requested feature:

When using complex data types on object attrs, accessing nested keys/indexes would be awesome.

#### Steps to reproduce the issue / Reasons for adding feature:

1. `set here/test_list = ['this', 'that', 'the other']`
2. `set here/test_list/0`
3. 

#### Error output / Expected result of feature

Expected: `Attribute Location Name/test_list/0 =  this`
Actual: `Attribute Location Name/test_list = ['This', 'that', 'the other'] Location Name has no attribute '0'.`

#### Extra information, such as Evennia revision/repo/branch, operating system and ideas for how to solve / implement:

Looks like `search_for_obj` in `evennia/commands/default/building.py` would be the place to implement this chained attr lookup. Shouldn't be too hard to implement, but would break folks expecting the multiple attr lookup, so it might need some new semantics/command.

I'm happy to attempt a PR if there is consensus on the interface.


Discussion:
@aarcro This would make the set command a lot more powerful and would be a useful addition. I have some comments on the syntax though, should you feel like trying your hand at an implementation. 

When coming to these kind of operations you are accessing Python structures stored in the Attribute. So I think it may be better and more intuitive to use Python syntax for it and implement this kind of operation just the way as you would expect in Python - that is, a syntax like

    set here/test_list[0] = 'foo' 
    set here/test_dict['bar'] = 32342
    set here/test_dict_list['bar'][4] = 'test'

One key thing to remember with any such implementation is that Python `eval` or `exec` must _never_ be used to perform the operation - this would be a major security issue (making `set` possible to make work as the `py` command). Rather, the `[ ]` argument must be manually parsed and the content used to query the value in the Attribute, relaying proper, sanitized errors to the user. 
Perfect. Yes, would have to parse and sanitize ourself, not eval. We'd
probably want to check for the exact name before attempting nested access,
just in case there's an existing attribute with `[]` in the name.

On Sun, Sep 15, 2019, 8:02 AM Griatch <notifications@github.com> wrote:

> @aarcro <https://github.com/aarcro> This would make the set command a lot
> more powerful and would be a useful addition. I have some comments on the
> syntax though, should you feel like trying your hand at an implementation.
>
> When coming to these kind of operations you are accessing Python
> structures stored in the Attribute. So I think it may be better and more
> intuitive to use Python syntax for it and implement this kind of operation
> just the way as you would expect in Python - that is, a syntax like
>
> set here/test_list[0] = 'foo'
> set here/test_dict['bar'] = 32342
> set here/test_dict_list['bar'][4] = 'test'
>
> One key thing to remember with any such implementation is that Python eval
> or exec must *never* be used to perform the operation - this would be a
> major security issue (making set possible to make work as the py
> command). Rather, the [ ] argument must be manually parsed and the
> content used to query the value in the Attribute, relaying proper,
> sanitized errors to the user.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/evennia/evennia/issues/1928?email_source=notifications&email_token=AACEJLNWKZBIBFY2MJ5IVH3QJYP6JA5CNFSM4IWY6DCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XPLGQ#issuecomment-531559834>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AACEJLNXORYH2YBSAZHRFTTQJYP6JANCNFSM4IWY6DCA>
> .
>


Resolving patch:
diff --git a/evennia/commands/default/building.py b/evennia/commands/default/building.py
--- a/evennia/commands/default/building.py
+++ b/evennia/commands/default/building.py
@@ -53,6 +53,7 @@
 
 # used by set
 from ast import literal_eval as _LITERAL_EVAL
+LIST_APPEND_CHAR = '+'
 
 # used by find
 CHAR_TYPECLASS = settings.BASE_CHARACTER_TYPECLASS
@@ -1569,7 +1570,7 @@ class CmdSetAttribute(ObjManipCommand):
       set <obj>/<attr> = <value>
       set <obj>/<attr> =
       set <obj>/<attr>
-      set *<account>/attr = <value>
+      set *<account>/<attr> = <value>
 
     Switch:
         edit: Open the line editor (string values only)
@@ -1584,7 +1585,7 @@ class CmdSetAttribute(ObjManipCommand):
     Sets attributes on objects. The second example form above clears a
     previously set attribute while the third form inspects the current value of
     the attribute (if any). The last one (with the star) is a shortcut for
-    operatin on a player Account rather than an Object.
+    operating on a player Account rather than an Object.
 
     The most common data to save with this command are strings and
     numbers. You can however also set Python primitives such as lists,
@@ -1592,8 +1593,10 @@ class CmdSetAttribute(ObjManipCommand):
     the functionality of certain custom objects).  This is indicated
     by you starting your value with one of |c'|n, |c"|n, |c(|n, |c[|n
     or |c{ |n.
-    Note that you should leave a space after starting a dictionary ('{ ')
-    so as to not confuse the dictionary start with a colour code like \{g.
+
+    Once you have stored a Python primative as noted above, you can include
+    |c[<key>]|n in <attr> to reference nested values.
+
     Remember that if you use Python primitives like this, you must
     write proper Python syntax too - notably you must include quotes
     around your strings or you will get an error.
@@ -1603,6 +1606,8 @@ class CmdSetAttribute(ObjManipCommand):
     key = "set"
     locks = "cmd:perm(set) or perm(Builder)"
     help_category = "Building"
+    nested_re = re.compile(r'\[.*?\]')
+    not_found = object()
 
     def check_obj(self, obj):
         """
@@ -1627,30 +1632,139 @@ def check_attr(self, obj, attr_name):
         """
         return attr_name
 
+    def split_nested_attr(self, attr):
+        """
+        Yields tuples of (possible attr name, nested keys on that attr).
+        For performance, this is biased to the deepest match, but allows compatability
+        with older attrs that might have been named with `[]`'s.
+
+        > list(split_nested_attr("nested['asdf'][0]"))
+        [
+            ('nested', ['asdf', 0]),
+            ("nested['asdf']", [0]),
+            ("nested['asdf'][0]", []),
+        ]
+        """
+        quotes = '"\''
+
+        def clean_key(val):
+            val = val.strip('[]')
+            if val[0] in quotes:
+                return val.strip(quotes)
+            if val[0] == LIST_APPEND_CHAR:
+                # List insert/append syntax
+                return val
+            try:
+                return int(val)
+            except ValueError:
+                return val
+
+        parts = self.nested_re.findall(attr)
+
+        base_attr = ''
+        if parts:
+            base_attr = attr[:attr.find(parts[0])]
+        for index, part in enumerate(parts):
+            yield (base_attr, [clean_key(p) for p in parts[index:]])
+            base_attr += part
+        yield (attr, [])
+
+    def do_nested_lookup(self, value, *keys):
+        result = value
+        for key in keys:
+            try:
+                result = result.__getitem__(key)
+            except (IndexError, KeyError, TypeError):
+                return self.not_found
+        return result
+
     def view_attr(self, obj, attr):
         """
         Look up the value of an attribute and return a string displaying it.
         """
-        if obj.attributes.has(attr):
-            return "\nAttribute %s/%s = %s" % (obj.name, attr, obj.attributes.get(attr))
-        else:
-            return "\n%s has no attribute '%s'." % (obj.name, attr)
+        nested = False
+        for key, nested_keys in self.split_nested_attr(attr):
+            nested = True
+            if obj.attributes.has(key):
+                val = obj.attributes.get(key)
+                val = self.do_nested_lookup(val, *nested_keys)
+                if val is not self.not_found:
+                    return "\nAttribute %s/%s = %s" % (obj.name, attr, val)
+        error = "\n%s has no attribute '%s'." % (obj.name, attr)
+        if nested:
+            error += ' (Nested lookups attempted)'
+        return error
 
     def rm_attr(self, obj, attr):
         """
-        Remove an attribute from the object, and report back.
+        Remove an attribute from the object, or a nested data structure, and report back.
         """
-        if obj.attributes.has(attr):
-            val = obj.attributes.has(attr)
-            obj.attributes.remove(attr)
-            return "\nDeleted attribute '%s' (= %s) from %s." % (attr, val, obj.name)
-        else:
-            return "\n%s has no attribute '%s'." % (obj.name, attr)
+        nested = False
+        for key, nested_keys in self.split_nested_attr(attr):
+            nested = True
+            if obj.attributes.has(key):
+                if nested_keys:
+                    del_key = nested_keys[-1]
+                    val = obj.attributes.get(key)
+                    deep = self.do_nested_lookup(val, *nested_keys[:-1])
+                    if deep is not self.not_found:
+                        try:
+                            del deep[del_key]
+                        except (IndexError, KeyError, TypeError):
+                            continue
+                    return "\nDeleted attribute '%s' (= nested) from %s." % (attr, obj.name)
+                else:
+                    exists = obj.attributes.has(key)
+                    obj.attributes.remove(attr)
+                    return "\nDeleted attribute '%s' (= %s) from %s." % (attr, exists, obj.name)
+        error = "\n%s has no attribute '%s'." % (obj.name, attr)
+        if nested:
+            error += ' (Nested lookups attempted)'
+        return error
 
     def set_attr(self, obj, attr, value):
+        done = False
+        for key, nested_keys in self.split_nested_attr(attr):
+            if obj.attributes.has(key) and nested_keys:
+                acc_key = nested_keys[-1]
+                lookup_value = obj.attributes.get(key)
+                deep = self.do_nested_lookup(lookup_value, *nested_keys[:-1])
+                if deep is not self.not_found:
+                    # To support appending and inserting to lists
+                    # a key that starts with LIST_APPEND_CHAR will insert a new item at that
+                    # location, and move the other elements down.
+                    # Using LIST_APPEND_CHAR alone will append to the list
+                    if isinstance(acc_key, str) and acc_key[0] == LIST_APPEND_CHAR:
+                        try:
+                            if len(acc_key) > 1:
+                                where = int(acc_key[1:])
+                                deep.insert(where, value)
+                            else:
+                                deep.append(value)
+                        except (ValueError, AttributeError):
+                            pass
+                        else:
+                            value = lookup_value
+                            attr = key
+                            done = True
+                            break
+
+                    # List magic failed, just use like a key/index
+                    try:
+                        deep[acc_key] = value
+                    except TypeError as err:
+                        # Tuples can't be modified
+                        return "\n%s - %s" % (err, deep)
+
+                    value = lookup_value
+                    attr = key
+                    done = True
+                    break
+
+        verb = "Modified" if obj.attributes.has(attr) else "Created"
         try:
-            verb = "Modified" if obj.attributes.has(attr) else "Created"
-            obj.attributes.add(attr, value)
+            if not done:
+                obj.attributes.add(attr, value)
             return "\n%s attribute %s/%s = %s" % (verb, obj.name, attr, repr(value))
         except SyntaxError:
             # this means literal_eval tried to parse a faulty string


Test patch:
diff --git a/evennia/commands/default/tests.py b/evennia/commands/default/tests.py
--- a/evennia/commands/default/tests.py
+++ b/evennia/commands/default/tests.py
@@ -535,6 +535,169 @@ def test_attribute_commands(self):
         self.call(building.CmdWipe(), "Obj2/test2/test3", "Wiped attributes test2,test3 on Obj2.")
         self.call(building.CmdWipe(), "Obj2", "Wiped all attributes on Obj2.")
 
+    def test_nested_attribute_commands(self):
+        # list - adding white space proves real parsing
+        self.call(building.CmdSetAttribute(), "Obj/test1=[1,2]", "Created attribute Obj/test1 = [1, 2]")
+        self.call(building.CmdSetAttribute(), "Obj/test1", "Attribute Obj/test1 = [1, 2]")
+        self.call(building.CmdSetAttribute(), "Obj/test1[0]", "Attribute Obj/test1[0] = 1")
+        self.call(building.CmdSetAttribute(), "Obj/test1[1]", "Attribute Obj/test1[1] = 2")
+        self.call(building.CmdSetAttribute(), "Obj/test1[0] = 99", "Modified attribute Obj/test1 = [99, 2]")
+        self.call(building.CmdSetAttribute(), "Obj/test1[0]", "Attribute Obj/test1[0] = 99")
+        # list delete
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test1[0] =",
+                  "Deleted attribute 'test1[0]' (= nested) from Obj.")
+        self.call(building.CmdSetAttribute(), "Obj/test1[0]", "Attribute Obj/test1[0] = 2")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test1[1]",
+                  "Obj has no attribute 'test1[1]'. (Nested lookups attempted)")
+        # Delete non-existent
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test1[5] =",
+                  "Obj has no attribute 'test1[5]'. (Nested lookups attempted)")
+        # Append
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test1[+] = 42",
+                  "Modified attribute Obj/test1 = [2, 42]")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test1[+0] = -1",
+                  "Modified attribute Obj/test1 = [-1, 2, 42]")
+
+        # dict - removing white space proves real parsing
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2={ 'one': 1, 'two': 2 }",
+                  "Created attribute Obj/test2 = {'one': 1, 'two': 2}")
+        self.call(building.CmdSetAttribute(), "Obj/test2", "Attribute Obj/test2 = {'one': 1, 'two': 2}")
+        self.call(building.CmdSetAttribute(), "Obj/test2['one']", "Attribute Obj/test2['one'] = 1")
+        self.call(building.CmdSetAttribute(), "Obj/test2['one]", "Attribute Obj/test2['one] = 1")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2['one']=99",
+                  "Modified attribute Obj/test2 = {'one': 99, 'two': 2}")
+        self.call(building.CmdSetAttribute(), "Obj/test2['one']", "Attribute Obj/test2['one'] = 99")
+        self.call(building.CmdSetAttribute(), "Obj/test2['two']", "Attribute Obj/test2['two'] = 2")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[+'three']",
+                  "Obj has no attribute 'test2[+'three']'. (Nested lookups attempted)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[+'three'] = 3",
+                  "Modified attribute Obj/test2 = {'one': 99, 'two': 2, \"+'three'\": 3}")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[+'three'] =",
+                  "Deleted attribute 'test2[+'three']' (= nested) from Obj.")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2['three']=3",
+                  "Modified attribute Obj/test2 = {'one': 99, 'two': 2, 'three': 3}")
+        # Dict delete
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2['two'] =",
+                  "Deleted attribute 'test2['two']' (= nested) from Obj.")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2['two']",
+                  "Obj has no attribute 'test2['two']'. (Nested lookups attempted)")
+        self.call(building.CmdSetAttribute(), "Obj/test2", "Attribute Obj/test2 = {'one': 99, 'three': 3}")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[0]",
+                  "Obj has no attribute 'test2[0]'. (Nested lookups attempted)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2['five'] =",
+                  "Obj has no attribute 'test2['five']'. (Nested lookups attempted)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[+]=42",
+                  "Modified attribute Obj/test2 = {'one': 99, 'three': 3, '+': 42}")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test2[+1]=33",
+                  "Modified attribute Obj/test2 = {'one': 99, 'three': 3, '+': 42, '+1': 33}")
+
+        # tuple
+        self.call(building.CmdSetAttribute(), "Obj/tup = (1,2)", "Created attribute Obj/tup = (1, 2)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/tup[1] = 99",
+                  "'tuple' object does not support item assignment - (1, 2)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/tup[+] = 99",
+                  "'tuple' object does not support item assignment - (1, 2)")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/tup[+1] = 99",
+                  "'tuple' object does not support item assignment - (1, 2)")
+        self.call(building.CmdSetAttribute(),
+                  # Special case for tuple, could have a better message
+                  "Obj/tup[1] = ",
+                  "Obj has no attribute 'tup[1]'. (Nested lookups attempted)")
+
+        # Deaper nesting
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test3=[{'one': 1}]",
+                  "Created attribute Obj/test3 = [{'one': 1}]")
+        self.call(building.CmdSetAttribute(), "Obj/test3[0]['one']", "Attribute Obj/test3[0]['one'] = 1")
+        self.call(building.CmdSetAttribute(), "Obj/test3[0]", "Attribute Obj/test3[0] = {'one': 1}")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test3[0]['one'] =",
+                  "Deleted attribute 'test3[0]['one']' (= nested) from Obj.")
+        self.call(building.CmdSetAttribute(), "Obj/test3[0]", "Attribute Obj/test3[0] = {}")
+        self.call(building.CmdSetAttribute(), "Obj/test3", "Attribute Obj/test3 = [{}]")
+
+        # Naughty keys
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test4[0]='foo'",
+                  "Created attribute Obj/test4[0] = 'foo'")
+        self.call(building.CmdSetAttribute(), "Obj/test4[0]", "Attribute Obj/test4[0] = foo")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test4=[{'one': 1}]",
+                  "Created attribute Obj/test4 = [{'one': 1}]")
+        self.call(building.CmdSetAttribute(), "Obj/test4[0]['one']", "Attribute Obj/test4[0]['one'] = 1")
+        # Prefer nested items
+        self.call(building.CmdSetAttribute(), "Obj/test4[0]", "Attribute Obj/test4[0] = {'one': 1}")
+        self.call(building.CmdSetAttribute(), "Obj/test4[0]['one']", "Attribute Obj/test4[0]['one'] = 1")
+        # Restored access
+        self.call(building.CmdWipe(), "Obj/test4", "Wiped attributes test4 on Obj.")
+        self.call(building.CmdSetAttribute(), "Obj/test4[0]", "Attribute Obj/test4[0] = foo")
+        self.call(building.CmdSetAttribute(),
+                  "Obj/test4[0]['one']",
+                  "Obj has no attribute 'test4[0]['one']'.")
+
+    def test_split_nested_attr(self):
+        split_nested_attr = building.CmdSetAttribute().split_nested_attr
+        test_cases = {
+            'test1': [('test1', [])],
+            'test2["dict"]': [('test2', ['dict']), ('test2["dict"]', [])],
+            # Quotes not actually required
+            'test3[dict]': [('test3', ['dict']), ('test3[dict]', [])],
+            'test4["dict]': [('test4', ['dict']), ('test4["dict]', [])],
+            # duplicate keys don't cause issues
+            'test5[0][0]': [('test5', [0, 0]), ('test5[0]', [0]), ('test5[0][0]', [])],
+            # String ints preserved
+            'test6["0"][0]': [('test6', ['0', 0]), ('test6["0"]', [0]), ('test6["0"][0]', [])],
+            # Unmatched []
+            'test7[dict': [('test7[dict', [])],
+        }
+
+        for attr, result in test_cases.items():
+            self.assertEqual(list(split_nested_attr(attr)), result)
+
+    def test_do_nested_lookup(self):
+        do_nested_lookup = building.CmdSetAttribute().do_nested_lookup
+        not_found = building.CmdSetAttribute.not_found
+
+        def do_test_single(value, key, result):
+            self.assertEqual(do_nested_lookup(value, key), result)
+
+        def do_test_multi(value, keys, result):
+            self.assertEqual(do_nested_lookup(value, *keys), result)
+
+        do_test_single([], 'test1', not_found)
+        do_test_single([1], 'test2', not_found)
+        do_test_single([], 0, not_found)
+        do_test_single([], '0', not_found)
+        do_test_single([1], 2, not_found)
+        do_test_single([1], 0, 1)
+        do_test_single([1], '0', not_found)  # str key is str not int
+        do_test_single({}, 'test3', not_found)
+        do_test_single({}, 0, not_found)
+        do_test_single({'foo': 'bar'}, 'foo', 'bar')
+
+        do_test_multi({'one': [1, 2, 3]}, ('one', 0), 1)
+        do_test_multi([{}, {'two': 2}, 3], (1, 'two'), 2)
+
     def test_name(self):
         self.call(building.CmdName(), "", "Usage: ")
         self.call(building.CmdName(), "Obj2=Obj3", "Object's name changed to 'Obj3'.")