Repository: Pyomo/pyomo

Issue:
Block set_value can't assign to index of an IndexedBlock
Calling set_value on a Block fails when assigning either a Block or a BlockData to an index in an IndexedBlock.

If you assign a Block:
```
m = ConcreteModel()
m.block_list = Block([1, 2])

movingBlock = m.some_other_block = Block()
m.del_component(movingBlock)

m.block_list[1] = movingBlock
```
```
Traceback (most recent call last):
  File "blockSetValue.py", line 11, in <module>
    m.block_list[1] = movingBlock
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 412, in __setitem__
    return self._setitem_when_not_present(index, val)
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 675, in _setitem_when_not_present
    obj.set_value(value)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 711, in set_value
    self.add_component(k,val[k])
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 923, in add_component
    super(_BlockData, self).__setattr__(name, val)
TypeError: attribute name must be string, not 'NoneType'
```
If you assign a BlockData:
```
m = ConcreteModel()
m.block_list = Block(Any)

m.some_other_block_list = Block([3,4])
movingBlock = m.some_other_block_list[4]
m.del_component(movingBlock)

m.block_list[len(m.block_list)] = movingBlock
```
```
Traceback (most recent call last):
  File "blockSetValue.py", line 21, in <module>
    m.block_list[len(m.block_list)] = movingBlock
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 412, in __setitem__
    return self._setitem_when_not_present(index, val)
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 675, in _setitem_when_not_present
    obj.set_value(value)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 710, in set_value
    for k in sorted(iterkeys(val)):
  File "/projects/sems/install/rhel6-x86_64/pyomo/python/six/1.11.0/lib/python2.7/site-packages/six.py", line 599, in iterkeys
    return d.iterkeys(**kw)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 549, in __getattr__
    % (self.__class__.__name__, val))
AttributeError: '_BlockData' object has no attribute 'iterkeys'
```
Block set_value can't assign to index of an IndexedBlock
Calling set_value on a Block fails when assigning either a Block or a BlockData to an index in an IndexedBlock.

If you assign a Block:
```
m = ConcreteModel()
m.block_list = Block([1, 2])

movingBlock = m.some_other_block = Block()
m.del_component(movingBlock)

m.block_list[1] = movingBlock
```
```
Traceback (most recent call last):
  File "blockSetValue.py", line 11, in <module>
    m.block_list[1] = movingBlock
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 412, in __setitem__
    return self._setitem_when_not_present(index, val)
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 675, in _setitem_when_not_present
    obj.set_value(value)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 711, in set_value
    self.add_component(k,val[k])
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 923, in add_component
    super(_BlockData, self).__setattr__(name, val)
TypeError: attribute name must be string, not 'NoneType'
```
If you assign a BlockData:
```
m = ConcreteModel()
m.block_list = Block(Any)

m.some_other_block_list = Block([3,4])
movingBlock = m.some_other_block_list[4]
m.del_component(movingBlock)

m.block_list[len(m.block_list)] = movingBlock
```
```
Traceback (most recent call last):
  File "blockSetValue.py", line 21, in <module>
    m.block_list[len(m.block_list)] = movingBlock
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 412, in __setitem__
    return self._setitem_when_not_present(index, val)
  File "/home/esjohn/src/pyomo/pyomo/core/base/indexed_component.py", line 675, in _setitem_when_not_present
    obj.set_value(value)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 710, in set_value
    for k in sorted(iterkeys(val)):
  File "/projects/sems/install/rhel6-x86_64/pyomo/python/six/1.11.0/lib/python2.7/site-packages/six.py", line 599, in iterkeys
    return d.iterkeys(**kw)
  File "/home/esjohn/src/pyomo/pyomo/core/base/block.py", line 549, in __getattr__
    % (self.__class__.__name__, val))
AttributeError: '_BlockData' object has no attribute 'iterkeys'
```


Discussion:
Interesting.  Neither of these examples are producing exceptions for me.  For your second example, there is something weird going on, as the `del_component` is removing the *entire* `some_other_block_list` and not just the individual _BlockData.  That is probably a bug.  That said, for the second example, I think you want to:

```
del movingBlock.parent_component()[movingBlock.index()]
```
---
**EDIT**: Scratch that.  I can reproduce both errors on clean versions of master.
Interesting.  Neither of these examples are producing exceptions for me.  For your second example, there is something weird going on, as the `del_component` is removing the *entire* `some_other_block_list` and not just the individual _BlockData.  That is probably a bug.  That said, for the second example, I think you want to:

```
del movingBlock.parent_component()[movingBlock.index()]
```
---
**EDIT**: Scratch that.  I can reproduce both errors on clean versions of master.

Resolving patch:
diff --git a/pyomo/core/base/block.py b/pyomo/core/base/block.py
--- a/pyomo/core/base/block.py
+++ b/pyomo/core/base/block.py
@@ -12,15 +12,23 @@
            'active_components', 'components', 'active_components_data',
            'components_data', 'SimpleBlock']
 
+import collections
 import copy
+import logging
 import sys
 import weakref
-import logging
+import textwrap
+
 from inspect import isclass
 from operator import itemgetter, attrgetter
 from six import iteritems, iterkeys, itervalues, StringIO, string_types, \
     advance_iterator, PY3
 
+if PY3:
+    from collections.abc import Mapping as collections_Mapping
+else:
+    from collections import Mapping as collections_Mapping
+
 from pyutilib.misc.indent_io import StreamIndenter
 
 from pyomo.common.timing import ConstructionTimer
@@ -33,7 +41,6 @@
 from pyomo.core.base.suffix import ComponentMap
 from pyomo.core.base.indexed_component import IndexedComponent, \
     ActiveIndexedComponent, UnindexedComponent_set
-import collections
 
 from pyomo.opt.base import ProblemFormat, guess_format
 from pyomo.opt import WriterFactory
@@ -702,15 +709,116 @@ def __delattr__(self, name):
             #
             super(_BlockData, self).__delattr__(name)
 
+    def _compact_decl_storage(self):
+        idxMap = {}
+        _new_decl_order = []
+        j = 0
+        # Squeeze out the None entries
+        for i, entry in enumerate(self._decl_order):
+            if entry[0] is not None:
+                idxMap[i] = j
+                j += 1
+                _new_decl_order.append(entry)
+        # Update the _decl map
+        self._decl = {k:idxMap[idx] for k,idx in iteritems(self._decl)}
+        # Update the ctypes, _decl_order linked lists
+        for ctype, info in iteritems(self._ctypes):
+            idx = info[0]
+            entry = self._decl_order[idx]
+            while entry[0] is None:
+                idx = entry[1]
+                entry = self._decl_order[idx]
+            info[0] = last = idxMap[idx]
+            while entry[1] is not None:
+                idx = entry[1]
+                entry = self._decl_order[idx]
+                if entry[0] is not None:
+                    this = idxMap[idx]
+                    _new_decl_order[last] = (_new_decl_order[last][0], this)
+                    last = this
+            info[1] = last
+            _new_decl_order[last] = (_new_decl_order[last][0], None)
+        self._decl_order = _new_decl_order
+
     def set_value(self, val):
-        for k in list(getattr(self, '_decl', {})):
-            self.del_component(k)
-        self._ctypes = {}
-        self._decl = {}
-        self._decl_order = []
-        if val:
-            for k in sorted(iterkeys(val)):
-                self.add_component(k,val[k])
+        raise RuntimeError(textwrap.dedent(
+            """\
+            Block components do not support assignment or set_value().
+            Use the transfer_attributes_from() method to transfer the
+            components and public attributes from one block to another:
+                model.b[1].transfer_attributes_from(other_block)
+            """))
+
+    def clear(self):
+        for name in iterkeys(self.component_map()):
+            if name not in self._Block_reserved_words:
+                self.del_component(name)
+        for attr in tuple(self.__dict__):
+            if attr not in self._Block_reserved_words:
+                delattr(self, attr)
+        self._compact_decl_storage()
+
+    def transfer_attributes_from(self, src):
+        """Transfer user-defined attributes from src to this block
+
+        This transfers all components and user-defined attributes from
+        the block or dictionary `src` and places them on this Block.
+        Components are transferred in declaration order.
+
+        If a Component on `src` is also declared on this block as either
+        a Component or attribute, the local Component or attribute is
+        replaced by the incoming component.  If an attribute name on
+        `src` matches a Component declared on this block, then the
+        incoming attribute is passed to the local Component's
+        `set_value()` method.  Attribute names appearing in this block's
+        `_Block_reserved_words` set will not be transferred (although
+        Components will be).
+
+        Parameters
+        ----------
+        src: _BlockData or dict
+            The Block or mapping that contains the new attributes to
+            assign to this block.
+        """
+        if isinstance(src, _BlockData):
+            # There is a special case where assigning a parent block to
+            # this block creates a circular hierarchy
+            if src is self:
+                return
+            p_block = self.parent_block()
+            while p_block is not None:
+                if p_block is src:
+                    raise ValueError(
+                        "_BlockData.transfer_attributes_from(): Cannot set a "
+                        "sub-block (%s) to a parent block (%s): creates a "
+                        "circular hierarchy" % (self, src))
+                p_block = p_block.parent_block()
+            # record the components and the non-component objects added
+            # to the block
+            src_comp_map = src.component_map()
+            src_raw_dict = {k:v for k,v in iteritems(src.__dict__)
+                            if k not in src_comp_map}
+        elif isinstance(src, collections_Mapping):
+            src_comp_map = {}
+            src_raw_dict = src
+        else:
+            raise ValueError(
+                "_BlockData.transfer_attributes_from(): expected a "
+                "Block or dict; received %s" % (type(src).__name__,))
+
+        # Use component_map for the components to preserve decl_order
+        for k,v in iteritems(src_comp_map):
+            if k in self._decl:
+                self.del_component(k)
+            src.del_component(k)
+            self.add_component(k,v)
+        # Because Blocks are not slotized and we allow the
+        # assignment of arbitrary data to Blocks, we will move over
+        # any other unrecognized entries in the object's __dict__:
+        for k in sorted(iterkeys(src_raw_dict)):
+            if k not in self._Block_reserved_words or not hasattr(self, k) \
+               or k in self._decl:
+                setattr(self, k, src_raw_dict[k])
 
     def _add_temporary_set(self, val):
         """TODO: This method has known issues (see tickets) and needs to be
@@ -853,7 +961,7 @@ def add_component(self, name, val):
         if not val.valid_model_component():
             raise RuntimeError(
                 "Cannot add '%s' as a component to a block" % str(type(val)))
-        if name in self._Block_reserved_words:
+        if name in self._Block_reserved_words and hasattr(self, name):
             raise ValueError("Attempting to declare a block component using "
                              "the name of a reserved attribute:\n\t%s"
                              % (name,))
@@ -892,6 +1000,18 @@ def add_component(self, name, val):
 component, use the block del_component() and add_component() methods.
 """ % (msg.strip(),))
         #
+        # If the new component is a Block, then there is the chance that
+        # it is the model(), and assigning it would create a circular
+        # hierarchy.  Note that we only have to check the model as the
+        # check immediately above would catch any "internal" blocks in
+        # the block hierarchy
+        #
+        if isinstance(val, Block) and val is self.model():
+            raise ValueError(
+                "Cannot assign the top-level block as a subblock of one of "
+                "its children (%s): creates a circular hierarchy"
+                % (self,))
+        #
         # Set the name and parent pointer of this component.
         #
         val._name = name
@@ -1765,7 +1885,7 @@ def __init__(self, *args, **kwargs):
             self.construct()
 
     def _getitem_when_not_present(self, idx):
-        return self._setitem_when_not_present(idx, None)
+        return self._setitem_when_not_present(idx)
 
     def find_component(self, label_or_component):
         """
@@ -1825,16 +1945,10 @@ def construct(self, data=None):
             if id(_block) in _BlockConstruction.data:
                 del _BlockConstruction.data[id(_block)]
 
-            if isinstance(obj, _BlockData) and obj is not _block:
-                # If the user returns a block, use their block instead
-                # of the empty one we just created.
-                for c in list(obj.component_objects(descend_into=False)):
-                    obj.del_component(c)
-                    _block.add_component(c.local_name, c)
-                # transfer over any other attributes that are not components
-                for name, val in iteritems(obj.__dict__):
-                    if not hasattr(_block, name) and not hasattr(self, name):
-                        super(_BlockData, _block).__setattr__(name, val)
+            if obj is not _block and isinstance(obj, _BlockData):
+                # If the user returns a block, transfer over everything
+                # they defined into the empty one we created.
+                _block.transfer_attributes_from(obj)
 
             # TBD: Should we allow skipping Blocks???
             # if obj is Block.Skip and idx is not None:
diff --git a/pyomo/core/base/indexed_component.py b/pyomo/core/base/indexed_component.py
--- a/pyomo/core/base/indexed_component.py
+++ b/pyomo/core/base/indexed_component.py
@@ -95,6 +95,8 @@ def normalize_index(x):
 
 class _NotFound(object):
     pass
+class _NotSpecified(object):
+    pass
 
 #
 # Get the fully-qualified name for this index.  If there isn't anything
@@ -712,7 +714,7 @@ def _setitem_impl(self, index, obj, value):
         obj.set_value(value)
         return obj
 
-    def _setitem_when_not_present(self, index, value):
+    def _setitem_when_not_present(self, index, value=_NotSpecified):
         """Perform the fundamental component item creation and storage.
 
         Components that want to implement a nonstandard storage mechanism
@@ -729,11 +731,12 @@ def _setitem_when_not_present(self, index, value):
         else:
             obj = self._data[index] = self._ComponentDataClass(component=self)
         try:
-            obj.set_value(value)
-            return obj
+            if value is not _NotSpecified:
+                obj.set_value(value)
         except:
             del self._data[index]
             raise
+        return obj
 
     def set_value(self, value):
         """Set the value of a scalar component."""
diff --git a/pyomo/gdp/disjunct.py b/pyomo/gdp/disjunct.py
--- a/pyomo/gdp/disjunct.py
+++ b/pyomo/gdp/disjunct.py
@@ -74,29 +74,12 @@ def process(arg):
 
 class _DisjunctData(_BlockData):
 
+    _Block_reserved_words = set()
+
     def __init__(self, component):
         _BlockData.__init__(self, component)
         self.indicator_var = Var(within=Binary)
 
-    def set_value(self, val):
-        _indicator_var = self.indicator_var
-        # Remove everything
-        for k in list(getattr(self, '_decl', {})):
-            self.del_component(k)
-        self._ctypes = {}
-        self._decl = {}
-        self._decl_order = []
-        # Now copy over everything from the other block.  If the other
-        # block has an indicator_var, it should override this block's.
-        # Otherwise restore this block's indicator_var.
-        if val:
-            if 'indicator_var' not in val:
-                self.add_component('indicator_var', _indicator_var)
-            for k in sorted(iterkeys(val)):
-                self.add_component(k,val[k])
-        else:
-            self.add_component('indicator_var', _indicator_var)
-
     def activate(self):
         super(_DisjunctData, self).activate()
         self.indicator_var.unfix()
@@ -177,6 +160,7 @@ def __init__(self, *args, **kwds):
 class IndexedDisjunct(Disjunct):
     pass
 
+_DisjunctData._Block_reserved_words = set(dir(Disjunct()))
 
 
 class _DisjunctionData(ActiveComponentData):


Test patch:
diff --git a/pyomo/core/tests/unit/test_block.py b/pyomo/core/tests/unit/test_block.py
--- a/pyomo/core/tests/unit/test_block.py
+++ b/pyomo/core/tests/unit/test_block.py
@@ -647,6 +647,154 @@ def test_set_attr(self):
         self.block.x = None
         self.assertEqual(self.block.x._value, None)
 
+        ### creation of a circular reference
+        b = Block(concrete=True)
+        b.c = Block()
+        with self.assertRaisesRegexp(
+                ValueError, "Cannot assign the top-level block as a subblock "
+                "of one of its children \(c\): creates a circular hierarchy"):
+            b.c.d = b
+
+    def test_set_value(self):
+        b = Block(concrete=True)
+        with self.assertRaisesRegexp(
+                RuntimeError, "Block components do not support assignment "
+                "or set_value"):
+            b.set_value(None)
+
+        b.b = Block()
+        with self.assertRaisesRegexp(
+                RuntimeError, "Block components do not support assignment "
+                "or set_value"):
+            b.b = 5
+
+    def test_clear(self):
+        class DerivedBlock(SimpleBlock):
+            _Block_reserved_words = None
+
+        DerivedBlock._Block_reserved_words \
+            = set(['a','b','c']) | _BlockData._Block_reserved_words
+
+        m = ConcreteModel()
+        m.clear()
+        self.assertEqual(m._ctypes, {})
+        self.assertEqual(m._decl, {})
+        self.assertEqual(m._decl_order, [])
+
+        m.w = 5
+        m.x = Var()
+        m.y = Param()
+        m.z = Var()
+        m.clear()
+        self.assertFalse(hasattr(m, 'w'))
+        self.assertEqual(m._ctypes, {})
+        self.assertEqual(m._decl, {})
+        self.assertEqual(m._decl_order, [])
+
+        m.b = DerivedBlock()
+        m.b.a = a = Param()
+        m.b.x = Var()
+        m.b.b = b = Var()
+        m.b.y = Var()
+        m.b.z = Param()
+        m.b.c = c = Param()
+        m.b.clear()
+        self.assertEqual(m.b._ctypes, {Var: [1, 1, 1], Param:[0,2,2]})
+        self.assertEqual(m.b._decl, {'a':0, 'b':1, 'c':2})
+        self.assertEqual(len(m.b._decl_order), 3)
+        self.assertIs(m.b._decl_order[0][0], a)
+        self.assertIs(m.b._decl_order[1][0], b)
+        self.assertIs(m.b._decl_order[2][0], c)
+        self.assertEqual(m.b._decl_order[0][1], 2)
+        self.assertEqual(m.b._decl_order[1][1], None)
+        self.assertEqual(m.b._decl_order[2][1], None)
+
+    def test_transfer_attributes_from(self):
+        b = Block(concrete=True)
+        b.x = Var()
+        b.y = Var()
+        c = Block(concrete=True)
+        c.z = Param(initialize=5)
+        c.x = c_x = Param(initialize=5)
+        c.y = c_y = 5
+
+        b.clear()
+        b.transfer_attributes_from(c)
+        self.assertEqual(list(b.component_map()), ['z','x'])
+        self.assertEqual(list(c.component_map()), [])
+        self.assertIs(b.x, c_x)
+        self.assertIs(b.y, c_y)
+
+        class DerivedBlock(SimpleBlock):
+            _Block_reserved_words = set()
+            def __init__(self, *args, **kwds):
+                super(DerivedBlock, self).__init__(*args, **kwds)
+                self.x = Var()
+                self.y = Var()
+        DerivedBlock._Block_reserved_words = set(dir(DerivedBlock()))
+
+        b = DerivedBlock(concrete=True)
+        b_x = b.x
+        b_y = b.y
+        c = Block(concrete=True)
+        c.z = Param(initialize=5)
+        c.x = c_x = Param(initialize=5)
+        c.y = c_y = 5
+
+        b.clear()
+        b.transfer_attributes_from(c)
+        self.assertEqual(list(b.component_map()), ['y','z','x'])
+        self.assertEqual(list(c.component_map()), [])
+        self.assertIs(b.x, c_x)
+        self.assertIsNot(b.y, c_y)
+        self.assertIs(b.y, b_y)
+        self.assertEqual(value(b.y), value(c_y))
+
+        ### assignment of dict
+        b = DerivedBlock(concrete=True)
+        b_x = b.x
+        b_y = b.y
+        c = { 'z': Param(initialize=5),
+              'x': Param(initialize=5),
+              'y': 5 }
+
+        b.clear()
+        b.transfer_attributes_from(c)
+        self.assertEqual(list(b.component_map()), ['y','x','z'])
+        self.assertEqual(sorted(list(iterkeys(c))), ['x','y','z'])
+        self.assertIs(b.x, c['x'])
+        self.assertIsNot(b.y, c['y'])
+        self.assertIs(b.y, b_y)
+        self.assertEqual(value(b.y), value(c_y))
+
+        ### assignment of self
+        b = Block(concrete=True)
+        b.x = b_x = Var()
+        b.y = b_y = Var()
+        b.transfer_attributes_from(b)
+
+        self.assertEqual(list(b.component_map()), ['x','y'])
+        self.assertIs(b.x, b_x)
+        self.assertIs(b.y, b_y)
+
+        ### creation of a circular reference
+        b = Block(concrete=True)
+        b.c = Block()
+        b.c.d = Block()
+        b.c.d.e = Block()
+        with self.assertRaisesRegexp(
+                ValueError, '_BlockData.transfer_attributes_from\(\): '
+                'Cannot set a sub-block \(c.d.e\) to a parent block \(c\):'):
+            b.c.d.e.transfer_attributes_from(b.c)
+
+        ### bad data type
+        b = Block(concrete=True)
+        with self.assertRaisesRegexp(
+                ValueError,
+                '_BlockData.transfer_attributes_from\(\): expected a Block '
+                'or dict; received str'):
+            b.transfer_attributes_from('foo')
+
     def test_iterate_hierarchy_defaults(self):
         self.assertIs( TraversalStrategy.BFS,
                        TraversalStrategy.BreadthFirstSearch )