Repository: Pyomo/pyomo

Issue:
Non-deterministic behavior with Set (shouldn't the default be ordered)
I thought that pull request https://github.com/Pyomo/pyomo/pull/568 was going to make Sets be ordered by default. However, the following behavior illustrates that this is not the case.

Please see the following code:
```
from pyomo.environ import *
a = ["a", "x", "c", "b"]

model = ConcreteModel()
model.S = Set(initialize=a)
model.OS = Set(initialize=a, ordered=True)

model.S.pprint()
model.OS.pprint()

print('set not ordered: {}'.format([i for i in model.S]))
print('set ordered: {}'.format([i for i in model.OS]))
```

The first time I run this, I get:
```
S : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=False, Bounds=None
    ['a', 'b', 'c', 'x']
OS : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=Insertion, Bounds=None
    ['a', 'x', 'c', 'b']
set not ordered: ['c', 'a', 'x', 'b']
set ordered: ['a', 'x', 'c', 'b']
```

and the second time, I get:
```
S : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=False, Bounds=None
    ['a', 'b', 'c', 'x']
OS : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=Insertion, Bounds=None
    ['a', 'x', 'c', 'b']
set not ordered: ['c', 'a', 'b', 'x']
set ordered: ['a', 'x', 'c', 'b']
```

To me, this implies the following:
1) ordered does not appear to be the default (see the Set pprint)
2) If you mark a set as ordered, then the order in the set appears to be the same as the initialize list and the order is maintained during iteration.
3) If you do not mark a set as ordered, then the order in the set is not the same as the initialize list (although it appears to be consistent), but the order during iteration is not consistent (and does not match the order in the Set).



Non-deterministic behavior with Set (shouldn't the default be ordered)
I thought that pull request https://github.com/Pyomo/pyomo/pull/568 was going to make Sets be ordered by default. However, the following behavior illustrates that this is not the case.

Please see the following code:
```
from pyomo.environ import *
a = ["a", "x", "c", "b"]

model = ConcreteModel()
model.S = Set(initialize=a)
model.OS = Set(initialize=a, ordered=True)

model.S.pprint()
model.OS.pprint()

print('set not ordered: {}'.format([i for i in model.S]))
print('set ordered: {}'.format([i for i in model.OS]))
```

The first time I run this, I get:
```
S : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=False, Bounds=None
    ['a', 'b', 'c', 'x']
OS : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=Insertion, Bounds=None
    ['a', 'x', 'c', 'b']
set not ordered: ['c', 'a', 'x', 'b']
set ordered: ['a', 'x', 'c', 'b']
```

and the second time, I get:
```
S : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=False, Bounds=None
    ['a', 'b', 'c', 'x']
OS : Dim=0, Dimen=1, Size=4, Domain=None, Ordered=Insertion, Bounds=None
    ['a', 'x', 'c', 'b']
set not ordered: ['c', 'a', 'b', 'x']
set ordered: ['a', 'x', 'c', 'b']
```

To me, this implies the following:
1) ordered does not appear to be the default (see the Set pprint)
2) If you mark a set as ordered, then the order in the set appears to be the same as the initialize list and the order is maintained during iteration.
3) If you do not mark a set as ordered, then the order in the set is not the same as the initialize list (although it appears to be consistent), but the order during iteration is not consistent (and does not match the order in the Set).





Discussion:
@jsiirola indicated that there is sorting done in the Set's pprint method, so that explains why the order appears consistent in the set even if ordered is not set to True. The concern is still present that ordered does not appear to be the default behavior.
Quoting a comment in the conversation from the original pull request,

>After thinking about this and the response to my thoughts about 'ordered=True', I realized that we shouldn't change the default value of 'ordered'.  Instead, we should change the storage mechanism when 'ordered=False'.  I've commit these changes, which should be reviewed.
>
>The basic idea is that even when the user doesn't specify 'ordered=True', the insertion order is preserved and used when iterating over the set.  When specifying 'ordered=True', not much more happens;  there's an API change allowing methods like next() and prev(), since the user has indicated that these are valid operations on their set.

_Originally posted by @whart222 in https://github.com/Pyomo/pyomo/pull/568#issuecomment-397881930_

I think that "the insertion order is preserved and **used when iterating over the set**." is not working. I think that this behavior is the desired behavior, but not what is being observed.
@carldlaird Should we talk about this next week when we are at Sandia?
@jsiirola indicated that there is sorting done in the Set's pprint method, so that explains why the order appears consistent in the set even if ordered is not set to True. The concern is still present that ordered does not appear to be the default behavior.
Quoting a comment in the conversation from the original pull request,

>After thinking about this and the response to my thoughts about 'ordered=True', I realized that we shouldn't change the default value of 'ordered'.  Instead, we should change the storage mechanism when 'ordered=False'.  I've commit these changes, which should be reviewed.
>
>The basic idea is that even when the user doesn't specify 'ordered=True', the insertion order is preserved and used when iterating over the set.  When specifying 'ordered=True', not much more happens;  there's an API change allowing methods like next() and prev(), since the user has indicated that these are valid operations on their set.

_Originally posted by @whart222 in https://github.com/Pyomo/pyomo/pull/568#issuecomment-397881930_

I think that "the insertion order is preserved and **used when iterating over the set**." is not working. I think that this behavior is the desired behavior, but not what is being observed.
@carldlaird Should we talk about this next week when we are at Sandia?

Resolving patch:
diff --git a/pyomo/core/base/sets.py b/pyomo/core/base/sets.py
--- a/pyomo/core/base/sets.py
+++ b/pyomo/core/base/sets.py
@@ -121,7 +121,7 @@ def _value_sorter(self, obj):
 
     """
     if self.ordered:
-        return obj.value
+        return obj.value_list
     else:
         return sorted_robust(obj)
 
@@ -279,7 +279,7 @@ class _OrderedSetData(_SetDataBase):
                                 if the discard method is used.
     """
 
-    __slots__ = ('value', 'order_dict', '_bounds', '_is_sorted')
+    __slots__ = ('value', 'value_list', 'order_dict', '_bounds', '_is_sorted')
 
     def __init__(self, owner, bounds):
         #
@@ -323,7 +323,7 @@ def data(self):
         'value' attribute.  The underlying set values are not be stored
         as a Python set() object.
         """
-        return set(self.value)
+        return self.value
 
     def _sort(self):
         """
@@ -332,15 +332,19 @@ def _sort(self):
         that the set is sorted.
         """
         _sorter = self.parent_component().ordered
-        self.value = sorted(self.value, key=None if _sorter is Set.SortedOrder else _sorter)
-        self.order_dict = dict((j,i) for i,j in enumerate(self.value))
+        self.value_list = sorted(
+            self.value_list,
+            key=None if _sorter is Set.SortedOrder else _sorter
+        )
+        self.order_dict = dict((j,i) for i,j in enumerate(self.value_list))
         self._is_sorted = 1
 
     def _clear(self):
         """
         Reset the set data
         """
-        self.value = []
+        self.value = set()
+        self.value_list = []
         self.order_dict = {}
         if self._is_sorted:
             self._is_sorted = 1
@@ -353,8 +357,9 @@ def _add(self, val, verify=True):
         """
         if verify:
             self._component()._verify(val)
-        self.order_dict[val] = len(self.value)
-        self.value.append(val)
+        self.order_dict[val] = len(self.value_list)
+        self.value_list.append(val)
+        self.value.add(val)
         if self._is_sorted:
             self._is_sorted = 2
 
@@ -367,19 +372,20 @@ def _discard(self, val):
             _id = self.order_dict.pop(val)
         except KeyError:
             return
-        del self.value[_id]
+        del self.value_list[_id]
+        self.value.remove(val)
         #
         # Update the order_dict: this assumes the user-specified sorter
         # (if one was used) is stable.
         #
-        for i in xrange(_id,len(self.value)):
-            self.order_dict[self.value[i]] = i
+        for i in xrange(_id,len(self.value_list)):
+            self.order_dict[self.value_list[i]] = i
 
     def __len__(self):
         """
         Return the number of elements in the set.
         """
-        return len(self.value)
+        return len(self.value_list)
 
     def __iter__(self):
         """
@@ -387,7 +393,7 @@ def __iter__(self):
         """
         if self._is_sorted == 2:
             self._sort()
-        return self.value.__iter__()
+        return self.value_list.__iter__()
 
     def __contains__(self, val):
         """
@@ -423,11 +429,11 @@ def __getitem__(self, idx):
         if idx >= 1:
             if idx > len(self):
                 raise IndexError("Cannot index a RangeSet past the last element")
-            return self.value[idx-1]
+            return self.value_list[idx-1]
         elif idx < 0:
             if len(self)+idx < 0:
                 raise IndexError("Cannot index a RangeSet past the first element")
-            return self.value[idx]
+            return self.value_list[idx]
         else:
             raise IndexError("Valid index values for sets are 1 .. len(set) or -1 .. -len(set)")
 
@@ -480,7 +486,7 @@ def nextw(self, match_element, k=1):
         except KeyError:
             raise KeyError("Cannot obtain nextw() member of set="+self.name+"; input element="+str(match_element)+" is not a member of the set!")
         #
-        return self[(element_position+k-1) % len(self.value) + 1]
+        return self[(element_position+k-1) % len(self.value_list) + 1]
 
     def prev(self, match_element, k=1):
         """
@@ -522,7 +528,7 @@ def __call__(self):
         """
         Return the underlying set data.
         """
-        return self.value
+        return self.data()
 
     def clear(self):
         """
@@ -557,7 +563,7 @@ def __call__(self):
         """
         Return the underlying set data.
         """
-        return self.value
+        return self.data()
 
     def clear(self):
         """
@@ -871,7 +877,7 @@ def _pprint(self):
             None, #("Members",),
             lambda k, v: [
                 "Virtual" if not self.concrete or v.virtual \
-                    else v.value if v.ordered \
+                    else v.value_list if v.ordered \
                     else sorted(v), ] )
 
     def _set_repn(self, other):
@@ -890,7 +896,7 @@ def __len__(self):
         """
         if not self.concrete:
             raise ValueError("The size of a non-concrete set is unknown")
-        return len(self.value)
+        return len(self.value_list)
 
     def __iter__(self):
         """
@@ -902,7 +908,8 @@ def __iter__(self):
                 "been constructed (initialized)." % (self.name,) )
         if not self.concrete:
             raise TypeError("Cannot iterate over a non-concrete set '%s'" % self.name)
-        return self.value.__iter__()
+        return self.value_list.__iter__()
+        #return super(SimpleSetBase, self).__iter__()
 
     def __reversed__(self):
         """


Test patch: