Repository: Pyomo/pyomo

Issue:
`Bunch.__delattr__` does not (necessarily) remove attribute from `Bunch` object
## Summary

<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `pyomo solve` to optimize a model and ..." -->
When attempting attribute deletion on a `Bunch` object (from `pyomo.common.collections`), such as through `delattr` or `del`, the attribute does not seem to have been removed from the underlying `dict` structure. Complete removal of the attribute seems to require calls to both `Bunch.__delattr__` and `Bunch.__delitem__` successively.

### Steps to reproduce the issue

<!-- Please include the command run and/or provide an example script that demonstrates the problem. -->

```python
# example.py
from pyomo.common.collections import Bunch


def display_bunch_attr_val(bunch_obj, attr):
    # attribute value remains unchanged after delattr
    print("-" * 30)
    print("Attribute value check:")
    print("Getattr:", getattr(bunch_obj, attr, None))
    print("Getitem:", bunch_obj[attr])


def display_bunch_attr_in_keys(bunch_obj):
    print("-" * 30)
    print("Bunch keys check:")
    print(bunch_obj.keys())  # remains unchanged after delattr
    print(f"Attribute in Bunch keys: {attr_name in bunch_obj.keys()}")
    print(
        "Attribute in Bunch.__dict__ keys: "
        f"{attr_name in bunch_obj.__dict__}"
    )


bunch = Bunch()
attr_name = "example"
val = 300

print(f"Setting attribute name {attr_name!r} to value {val}")
setattr(bunch, attr_name, val)
print(f"Bunch keys: {bunch.keys()}")

print("-" * 30)
print("Invoking delattr")
delattr(bunch, attr_name)

# attribute value remains unchanged after delattr
display_bunch_attr_val(bunch, attr_name)

# bunch keys check
display_bunch_attr_in_keys(bunch)

print("-" * 30)
print("Invoking delitem")
del bunch[attr_name]

# keys and the attribute value
display_bunch_attr_val(bunch, attr_name)

# bunch keys check
display_bunch_attr_in_keys(bunch)
```

### Error Message

<!-- If Pyomo reported an error, provide the error message. PLEASE INCLUDE THE ENTIRE STACK TRACE. -->

<!-- If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->

```console
$ python example.py
Setting attribute name 'example' to value 300
Bunch keys: dict_keys(['example'])
------------------------------
Invoking delattr
------------------------------
Attribute value check:
Getattr: 300
Getitem: 300
------------------------------
Bunch keys check:
dict_keys(['example'])
Attribute in bunch keys: True
Attribute in bunch.__dict__ keys: False
------------------------------
Invoking delitem
------------------------------
Attribute value check:
Getattr: None
Getitem: None
------------------------------
Bunch keys check:
dict_keys([])
Attribute in bunch keys: False
Attribute in bunch.__dict__ keys: False
```

### Information on your system

Pyomo version: 6.4.5dev0
Python version: 3.9.13
Operating system: Ubuntu 20.04
How Pyomo was installed (PyPI, conda, source): source
Solver (if applicable): N/A


### Additional information

<!-- If you have any additional information, please list it here. -->


<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question (how to use Pyomo, what it can do, etc.), please refer to our [online documentation](https://github.com/Pyomo/pyomo#getting-help) on ways to get help. -->




Discussion:

Resolving patch:
diff --git a/pyomo/common/collections/bunch.py b/pyomo/common/collections/bunch.py
--- a/pyomo/common/collections/bunch.py
+++ b/pyomo/common/collections/bunch.py
@@ -4,7 +4,7 @@
 #  Copyright (c) 2008-2022
 #  National Technology and Engineering Solutions of Sandia, LLC
 #  Under the terms of Contract DE-NA0003525 with National Technology and
-#  Engineering Solutions of Sandia, LLC, the U.S. Government retains certain 
+#  Engineering Solutions of Sandia, LLC, the U.S. Government retains certain
 #  rights in this software.
 #  This software is distributed under the 3-clause BSD License.
 #  ___________________________________________________________________________
@@ -17,106 +17,147 @@
 #  ___________________________________________________________________________
 
 import shlex
-
+from collections.abc import Mapping
 
 class Bunch(dict):
-    """
-    A class that can be used to store a bunch of data dynamically.
-    This class allows all other attributes to have a default value of None. 
-    This borrows the output formatting ideas from the
+    """A class that can be used to store a bunch of data dynamically.
+
+    This class allows for unspecified attributes to have a default value
+    of None.  This borrows the output formatting ideas from the
     ActiveState Code Container (recipe 496697).
+
+    For historical reasons, attributes / keys are stored in the
+    underlying dict unless they begin with an underscore, in which case
+    they are stored as object attributes.
+
     """
 
     def __init__(self, *args, **kw):
+        self._name_ = self.__class__.__name__
         for arg in args:
+            if not isinstance(arg, str):
+                raise TypeError("Bunch() positional arguments must be strings")
             for item in shlex.split(arg):
-                r = item.find('=')
-                if r != -1:
-                    try:
-                        val = eval(item[r + 1:])
-                    except:
-                        val = item[r + 1:]
-                    kw[item[:r]] = val
-        dict.__init__(self, kw)
-        self.__dict__.update(kw)
-        if not '_name_' in kw:
-            self._name_ = self.__class__.__name__
+                item = item.split('=', 1)
+                if len(item) != 2:
+                    raise ValueError(
+                        "Bunch() positional arguments must be space separated "
+                        f"strings of form 'key=value', got '{item[0]}'")
+
+                # Historically, this used 'exec'.  That is unsafe in
+                # this context (because anyone can pass arguments to a
+                # Bunch).  While not strictly backwards compatible,
+                # Pyomo was not using this for anything past parsing
+                # None/float/int values.  We will explicitly parse those
+                # values
+                try:
+                    val = float(item[1])
+                    if int(val) == val:
+                        val = int(val)
+                    item[1] = val
+                except:
+                    if item[1].strip() == 'None':
+                        item[1] = None
+                self[item[0]] = item[1]
+        for k, v in kw.items():
+            self[k] = v
 
     def update(self, d):
         """
         The update is specialized for JSON-like data.  This
-        recursively replaces dictionaries with Container objects.
+        recursively replaces dictionaries with Bunch objects.
         """
-        for k in d:
-            if type(d[k]) is dict:
-                tmp = Bunch()
-                tmp.update(d[k])
-                self.__setattr__(k, tmp)
-            elif type(d[k]) is list:
-                val = []
-                for i in d[k]:
-                    if type(i) is dict:
-                        tmp = Bunch()
-                        tmp.update(i)
-                        val.append(tmp)
-                    else:
-                        val.append(i)
-                self.__setattr__(k, val)
+        def _replace_dict_in_list(lst):
+            ans = []
+            for v in lst:
+                if type(v) is dict:
+                    ans.append(Bunch())
+                    ans[-1].update(v)
+                elif type(v) is list:
+                    ans.append(_replace_dict_in_list(v))
+                else:
+                    ans.append(v)
+            return ans
+
+        if isinstance(d, Mapping):
+            item_iter = d.items()
+        else:
+            item_iter = d
+        for k, v in item_iter:
+            if type(v) is dict:
+                self[k] = Bunch()
+                self[k].update(v)
+            elif type(v) is list:
+                self[k] = _replace_dict_in_list(v)
             else:
-                self.__setattr__(k, d[k])
+                self[k] = v
 
     def set_name(self, name):
         self._name_ = name
 
+    def __getitem__(self, name):
+        if not isinstance(name, str):
+            raise ValueError(
+                f'Bunch keys must be str (got {type(name).__name__})')
+        # Map through Python's standard getattr functionality (which
+        # will resolve known attributes without hitting __getattr__)
+        return getattr(self, name)
+
     def __setitem__(self, name, val):
-        self.__setattr__(name, val)
+        if not isinstance(name, str):
+            raise ValueError(
+                f'Bunch keys must be str (got {type(name).__name__})')
+        setattr(self, name, val)
 
-    def __getitem__(self, name):
-        return self.__getattr__(name)
+    def __delitem__(self, name):
+        if not isinstance(name, str):
+            raise ValueError(
+                f'Bunch keys must be str (got {type(name).__name__})')
+        delattr(self, name)
+
+    def __getattr__(self, name):
+        if name[0] == '_':
+            raise AttributeError(f"Unknown attribute '{name}'")
+        return self.get(name, None)
 
     def __setattr__(self, name, val):
-        if name[0] != '_':
-            dict.__setitem__(self, name, val)
-        self.__dict__[name] = val
+        if name[0] == '_':
+            super().__setattr__(name, val)
+        else:
+            super().__setitem__(name, val)
 
-    def __getattr__(self, name):
-        try:
-            return dict.__getitem__(self, name)
-        except:
-            if name[0] == '_':
-                raise AttributeError("Unknown attribute %s" % name)
-        return None
+    def __delattr__(self, name):
+        if name[0] == '_':
+            super().__delattr__(name)
+        else:
+            super().__delitem__(name)
 
     def __repr__(self):
-        attrs = sorted("%s = %r" % (k, v) for k, v in self.__dict__.items()
+        attrs = sorted("%s = %r" % (k, v) for k, v in self.items()
                        if not k.startswith("_"))
         return "%s(%s)" % (self.__class__.__name__, ", ".join(attrs))
 
-    def __str__(self):
-        return self.as_string()
-
     def __str__(self, nesting=0, indent=''):
         attrs = []
         indentation = indent + "    " * nesting
-        for k, v in self.__dict__.items():
-            if not k.startswith("_"):
-                text = [indentation, k, ":"]
-                if isinstance(v, Bunch):
-                    if len(v) > 0:
-                        text.append('\n')
-                    text.append(v.__str__(nesting + 1))
-                elif isinstance(v, list):
-                    if len(v) == 0:
-                        text.append(' []')
-                    else:
-                        for v_ in v:
-                            text.append('\n' + indentation + "-")
-                            if isinstance(v_, Bunch):
-                                text.append('\n' + v_.__str__(nesting + 1))
-                            else:
-                                text.append(" " + repr(v_))
+        for k, v in self.items():
+            text = [indentation, k, ":"]
+            if isinstance(v, Bunch):
+                if len(v) > 0:
+                    text.append('\n')
+                text.append(v.__str__(nesting + 1))
+            elif isinstance(v, list):
+                if len(v) == 0:
+                    text.append(' []')
                 else:
-                    text.append(' ' + repr(v))
-                attrs.append("".join(text))
+                    for v_ in v:
+                        text.append('\n' + indentation + "-")
+                        if isinstance(v_, Bunch):
+                            text.append('\n' + v_.__str__(nesting + 1))
+                        else:
+                            text.append(" " + repr(v_))
+            else:
+                text.append(' ' + repr(v))
+            attrs.append("".join(text))
         attrs.sort()
         return "\n".join(attrs)
diff --git a/pyomo/scripting/util.py b/pyomo/scripting/util.py
--- a/pyomo/scripting/util.py
+++ b/pyomo/scripting/util.py
@@ -277,8 +277,8 @@ def create_model(data):
         else:
             model_options = data.options.model.options.value()
             tick = time.time()
-            model = ep.service().apply( options = Bunch(*data.options),
-                                       model_options=Bunch(*model_options) )
+            model = ep.service().apply( options = Bunch(**data.options),
+                                       model_options=Bunch(**model_options) )
             if data.options.runtime.report_timing is True:
                 print("      %6.2f seconds required to construct instance" % (time.time() - tick))
                 data.local.time_initial_import = None


Test patch:
diff --git a/pyomo/common/tests/test_bunch.py b/pyomo/common/tests/test_bunch.py
--- a/pyomo/common/tests/test_bunch.py
+++ b/pyomo/common/tests/test_bunch.py
@@ -4,7 +4,7 @@
 #  Copyright (c) 2008-2022
 #  National Technology and Engineering Solutions of Sandia, LLC
 #  Under the terms of Contract DE-NA0003525 with National Technology and
-#  Engineering Solutions of Sandia, LLC, the U.S. Government retains certain 
+#  Engineering Solutions of Sandia, LLC, the U.S. Government retains certain
 #  rights in this software.
 #  This software is distributed under the 3-clause BSD License.
 #  ___________________________________________________________________________
@@ -22,30 +22,36 @@
 
 class Test(unittest.TestCase):
     def test_Bunch1(self):
-        opt = Bunch('a=None c=d e="1 2 3"', foo=1, bar='x')
+        opt = Bunch('a=None c=d e="1 2 3" f=" 5 "', foo=1, bar='x')
         self.assertEqual(opt.ll, None)
         self.assertEqual(opt.a, None)
         self.assertEqual(opt.c, 'd')
         self.assertEqual(opt.e, '1 2 3')
+        self.assertEqual(opt.f, 5)
         self.assertEqual(opt.foo, 1)
         self.assertEqual(opt.bar, 'x')
         self.assertEqual(opt['bar'], 'x')
         opt.xx = 1
         opt['yy'] = 2
         self.assertEqual(
-            set(opt.keys()), set(['a', 'bar', 'c', 'foo', 'e', 'xx', 'yy']))
+            set(opt.keys()),
+            set(['a', 'bar', 'c', 'f', 'foo', 'e', 'xx', 'yy'])
+        )
         opt.x = Bunch(a=1, b=2)
         self.assertEqual(
-            set(opt.keys()), set(
-                ['a', 'bar', 'c', 'foo', 'e', 'xx', 'yy', 'x']))
+            set(opt.keys()),
+            set(['a', 'bar', 'c', 'f', 'foo', 'e', 'xx', 'yy', 'x'])
+        )
         self.assertEqual(
             repr(opt),
-            "Bunch(a = None, bar = 'x', c = 'd', e = '1 2 3', foo = 1, x = Bunch(a = 1, b = 2), xx = 1, yy = 2)")
+            "Bunch(a = None, bar = 'x', c = 'd', e = '1 2 3', f = 5, "
+            "foo = 1, x = Bunch(a = 1, b = 2), xx = 1, yy = 2)")
         self.assertEqual(
             str(opt), """a: None
 bar: 'x'
 c: 'd'
 e: '1 2 3'
+f: 5
 foo: 1
 x:
     a: 1
@@ -54,16 +60,19 @@ def test_Bunch1(self):
 yy: 2""")
         opt._name_ = 'BUNCH'
         self.assertEqual(
-            set(opt.keys()), set(
-                ['a', 'bar', 'c', 'foo', 'e', 'xx', 'yy', 'x']))
+            set(opt.keys()),
+            set(['a', 'bar', 'c', 'f', 'foo', 'e', 'xx', 'yy', 'x'])
+        )
         self.assertEqual(
             repr(opt),
-            "Bunch(a = None, bar = 'x', c = 'd', e = '1 2 3', foo = 1, x = Bunch(a = 1, b = 2), xx = 1, yy = 2)")
+            "Bunch(a = None, bar = 'x', c = 'd', e = '1 2 3', f = 5, "
+            "foo = 1, x = Bunch(a = 1, b = 2), xx = 1, yy = 2)")
         self.assertEqual(
             str(opt), """a: None
 bar: 'x'
 c: 'd'
 e: '1 2 3'
+f: 5
 foo: 1
 x:
     a: 1
@@ -71,8 +80,147 @@ def test_Bunch1(self):
 xx: 1
 yy: 2""")
 
-    def test_Container2(self):
+        with self.assertRaisesRegex(
+                TypeError, r"Bunch\(\) positional arguments must be strings"):
+            Bunch(5)
+
+        with self.assertRaisesRegex(
+                ValueError, r"Bunch\(\) positional arguments must be space "
+                "separated strings of form 'key=value', got 'foo'"):
+            Bunch('a=5 foo = 6')
+
+    def test_pickle(self):
         o1 = Bunch('a=None c=d e="1 2 3"', foo=1, bar='x')
         s = pickle.dumps(o1)
         o2 = pickle.loads(s)
         self.assertEqual(o1, o2)
+
+    def test_attr_methods(self):
+        b = Bunch()
+        b.foo = 5
+        self.assertEqual(list(b.keys()), ['foo'])
+        self.assertEqual(b.foo, 5)
+        b._foo = 50
+        self.assertEqual(list(b.keys()), ['foo'])
+        self.assertEqual(b.foo, 5)
+        self.assertEqual(b._foo, 50)
+
+        del b.foo
+        self.assertEqual(list(b.keys()), [])
+        self.assertEqual(b.foo, None)
+        self.assertEqual(b._foo, 50)
+
+        del b._foo
+        self.assertEqual(list(b.keys()), [])
+        self.assertEqual(b.foo, None)
+        with self.assertRaisesRegex(
+                AttributeError, "Unknown attribute '_foo'"):
+            b._foo
+
+    def test_item_methods(self):
+        b = Bunch()
+        b['foo'] = 5
+        self.assertEqual(list(b.keys()), ['foo'])
+        self.assertEqual(b['foo'], 5)
+        b['_foo'] = 50
+        self.assertEqual(list(b.keys()), ['foo'])
+        self.assertEqual(b['foo'], 5)
+        self.assertEqual(b['_foo'], 50)
+
+        del b['foo']
+        self.assertEqual(list(b.keys()), [])
+        self.assertEqual(b['foo'], None)
+        self.assertEqual(b['_foo'], 50)
+
+        del b['_foo']
+        self.assertEqual(list(b.keys()), [])
+        self.assertEqual(b['foo'], None)
+        with self.assertRaisesRegex(
+                AttributeError, "Unknown attribute '_foo'"):
+            b['_foo']
+
+        with self.assertRaisesRegex(
+                ValueError, r"Bunch keys must be str \(got int\)"):
+            b[5]
+
+        with self.assertRaisesRegex(
+                ValueError, r"Bunch keys must be str \(got int\)"):
+            b[5] = 5
+
+        with self.assertRaisesRegex(
+                ValueError, r"Bunch keys must be str \(got int\)"):
+            del b[5]
+
+    def test_update(self):
+        data = {
+            'a': 1,
+            'b': [2, {'bb':3}, [4, {'bbb':5}]],
+            'c': {'cc': 6, 'ccc': {'e': 7}},
+            'd': []
+        }
+
+        # Test passing a dict
+        b = Bunch()
+        b.update(data)
+        self.assertEqual(
+            repr(b),
+            'Bunch(a = 1, '
+            'b = [2, Bunch(bb = 3), [4, Bunch(bbb = 5)]], '
+            'c = Bunch(cc = 6, ccc = Bunch(e = 7)), '
+            'd = [])'
+        )
+
+        # Test passing a generator
+        b = Bunch()
+        b.update(data.items())
+        self.assertEqual(
+            repr(b),
+            'Bunch(a = 1, '
+            'b = [2, Bunch(bb = 3), [4, Bunch(bbb = 5)]], '
+            'c = Bunch(cc = 6, ccc = Bunch(e = 7)), '
+            'd = [])'
+        )
+
+        # Test passing a list
+        b = Bunch()
+        b.update(list(data.items()))
+        self.assertEqual(
+            repr(b),
+            'Bunch(a = 1, '
+            'b = [2, Bunch(bb = 3), [4, Bunch(bbb = 5)]], '
+            'c = Bunch(cc = 6, ccc = Bunch(e = 7)), '
+            'd = [])'
+        )
+
+    def test_str(self):
+        data = {
+            'a': 1,
+            'b': [2, {'bb':3}, [4, {'bbb':5}]],
+            'c': {'cc': 6, 'ccc': {'e': 7}},
+            'd': []
+        }
+
+        b = Bunch()
+        b.update(data)
+        self.assertEqual(
+            str(b),
+'''
+a: 1
+b:
+- 2
+-
+    bb: 3
+- [4, Bunch(bbb = 5)]
+c:
+    cc: 6
+    ccc:
+        e: 7
+d: []
+'''.strip()
+        )
+
+    def test_set_name(self):
+        b = Bunch()
+        self.assertEqual(b._name_, 'Bunch')
+        b.set_name('TEST')
+        self.assertEqual(b._name_, 'TEST')
diff --git a/pyomo/core/tests/examples/test_pyomo.py b/pyomo/core/tests/examples/test_pyomo.py
--- a/pyomo/core/tests/examples/test_pyomo.py
+++ b/pyomo/core/tests/examples/test_pyomo.py
@@ -127,10 +127,15 @@ def compare_files(self, file1, file2):
                 f1_filtered = []
                 f2_filtered = []
                 for item1, item2 in zip_longest(f1_contents, f2_contents):
-                    if not item1.startswith('['):
+                    if not item1:
+                        f1_filtered.append(item1)
+                    elif not item1.startswith('['):
                         items1 = item1.strip().split()
-                        items2 = item2.strip().split()
                         f1_filtered.append(self.filter_items(items1))
+                    if not item2:
+                        f2_filtered.append(item2)
+                    elif not item2.startswith('['):
+                        items2 = item2.strip().split()
                         f2_filtered.append(self.filter_items(items2))
                 self.assertStructuredAlmostEqual(f2_filtered, f1_filtered,
                                                  abstol=1e-6,