Repository: Pylons/pyramid

Issue:
Cannot add_route in PHASE0_CONFIG if using view derivers
From what I can tell, if you attempt to use view derivers you cannot call `Configurator().add_route()` inside of a action where `order=PHASE0_CONFIG`. Using the following code to reproduce:

``` python
from pyramid.config import PHASE0_CONFIG, Configurator
from pyramid.response import Response


def no_op_view(view, info):
    return view

no_op_view.options = {"noop"}


def add_a_thing(config, thing, *args, **kwargs):
    def register():
        config.add_route(thing, *args, **kwargs)
    config.action(("thing", thing), register, order=PHASE0_CONFIG)


def hello_world(request):
    return Response('Hello %(name)s!' % request.matchdict)


config = Configurator()
config.add_view_deriver(no_op_view)
config.add_directive("add_a_thing", add_a_thing)
config.add_a_thing("my-thing", "/")
config.add_route('hello', '/hello/{name}')
config.add_view(hello_world, route_name="hello", noop=True)
config.commit()
```

If you execute that is, you'll get an error like:

```
Traceback (most recent call last):
  File "demo2.py", line 27, in <module>
    config.commit()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 654, in commit
    self.action_state.execute_actions(introspector=self.introspector)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1158, in execute_actions
    list(pending_actions) +
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1121, in resume
    for a, b in zip_longest(actions, executed_actions):
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1255, in resolveConflicts
    discriminator = undefer(action['discriminator'])
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 271, in undefer
    v = v.resolve()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 263, in resolve
    return self.value
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/decorator.py", line 45, in __get__
    val = self.wrapped(inst)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 260, in value
    return self.func()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/views.py", line 810, in discrim_func
    self._check_view_options(**dvals)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/views.py", line 1070, in _check_view_options
    raise ConfigurationError('Unknown view options: %s' % (kw,))
pyramid.exceptions.ConfigurationError: Unknown view options: {'noop': True}
```

However, if you do one of:
- Remove the call to `config.add_route` inside of the action.
- Remove the `noop=True` parameter to `config.add_view()`.
- Change the `order=PHASE0_CONFIG` to `order=PHASE1_CONFIG`.

Then the code will execute successfully, although I assume the third option will break if you attempt to call something that is executed in Phase 1 instead of add_route which is executed in Phase 2.



Discussion:
After a little sleuthing I've determined that the deferred discriminators are being undeferred at the start of the conflict resolution sort in `resolveConflicts`. This is an issue specific to the `add_view` discriminator which touches predicates and view options. It's the only action using a deferred discriminator.

I'm looking into how to fix this, but wanted to leave some notes here before going to bed.

Ah, it occurs on a re-entrant sort (triggered by the `add_route` inside of `register`). `resolveConflicts` sorts all the actions it receives, without regard for where we're currently at in the order. It needs to stop sorting until the execution has caught up in order.

`resolveConflicts` is currently written with two important, and not very clear, assumptions. 1) It can test discriminators whenever it wants (this is breaking in the case of the deferred discriminator) and 2) conflict resolution only occurs between actions in the same order (you cannot use a discriminator in order A to affect an action/discriminator in order B).

I do not see a way to preserve both of these assumptions and since assumption 2 has been in place longer, I'll err toward keeping that one. It's not possible to do conflict resolution across orders AND leave some of the discriminators in a deferred state. It just doesn't work.

If we break assumption 1 then it means we should only undefer / sort actions in the order we are executing and never undefer / sort actions that come in a later order. This should be possible but will require somewhat of a rewrite of the `execute_actions` and `resolveConflicts` logic.

Crap, now I've dug too far into this. There were some traces of code in `resolveConflicts` that didn't really make sense regarding the order and so I looked further. The conflict resolution algorithm actually changed in undocumented ways when the deferred discriminator was introduced. Prior to pyramid 1.4, discriminators worked across orders.

https://github.com/Pylons/pyramid/commit/9c8ec5c7b7f12abb741f9d4467bc85f15b893420

Spent a little while longer thinking about this - it's something I ran into while implementing the re-entrant stuff originally but missed this aspect. There are basically three options.

1) Figure out how to get rid of the deferred discriminator.
2) Document and live with the fact that discriminators can only affect actions of the same order.
3) Figure out how to resolve conflicts on a deferred discriminator.

Unfortunately, I think option 2 is the only option as I cannot see a way to get rid of the deferred discriminator. It's solving a real problem where the discriminator itself cannot be computed until actions from an earlier order have executed.

The only solution to option 3 that I can think of is to execute action A and then later when we undefer the discriminator for action B, if it ends up conflicting with the one from action A such that it has a higher priority then we have no choice but to error.

The only solution to option 3 that I can think of is to execute action A and then later when we undefer the discriminator for action B, if it ends up conflicting with the one from action A such that it has a higher priority then we have no choice but to error. However if we stick with an approach like option 2 then we can just allow the author to define 2 actions with the same discriminator on different orders to kill action A(order=0) and allow action B(order=1) to execute. The only downside is that the author needs to know, in advance, for each action he wants to override, what the order is of the action.


Resolving patch:
diff --git a/pyramid/config/__init__.py b/pyramid/config/__init__.py
--- a/pyramid/config/__init__.py
+++ b/pyramid/config/__init__.py
@@ -27,7 +27,6 @@
     text_,
     reraise,
     string_types,
-    zip_longest,
     )
 
 from pyramid.events import ApplicationCreated
@@ -1110,29 +1109,8 @@ def execute_actions(self, clear=True, introspector=None):
         try:
             all_actions = []
             executed_actions = []
-            pending_actions = iter([])
-
-            # resolve the new action list against what we have already
-            # executed -- if a new action appears intertwined in the list
-            # of already-executed actions then someone wrote a broken
-            # re-entrant action because it scheduled the action *after* it
-            # should have been executed (as defined by the action order)
-            def resume(actions):
-                for a, b in zip_longest(actions, executed_actions):
-                    if b is None and a is not None:
-                        # common case is that we are executing every action
-                        yield a
-                    elif b is not None and a != b:
-                        raise ConfigurationError(
-                            'During execution a re-entrant action was added '
-                            'that modified the planned execution order in a '
-                            'way that is incompatible with what has already '
-                            'been executed.')
-                    else:
-                        # resolved action is in the same location as before,
-                        # so we are in good shape, but the action is already
-                        # executed so we skip it
-                        assert b is not None and a == b
+            action_iter = iter([])
+            conflict_state = ConflictResolverState()
 
             while True:
                 # We clear the actions list prior to execution so if there
@@ -1141,26 +1119,14 @@ def resume(actions):
                 # ensures that the previously executed actions have no new
                 # conflicts.
                 if self.actions:
-                    # Only resolve the new actions against executed_actions
-                    # and pending_actions instead of everything to avoid
-                    # redundant checks.
-                    # Assume ``actions = resolveConflicts([A, B, C])`` which
-                    # after conflict checks, resulted in ``actions == [A]``
-                    # then we know action A won out or a conflict would have
-                    # been raised. Thus, when action D is added later, we only
-                    # need to check the new action against A.
-                    # ``actions = resolveConflicts([A, D]) should drop the
-                    # number of redundant checks down from O(n^2) closer to
-                    # O(n lg n).
                     all_actions.extend(self.actions)
-                    pending_actions = resume(resolveConflicts(
-                        executed_actions +
-                        list(pending_actions) +
-                        self.actions
-                    ))
+                    action_iter = resolveConflicts(
+                        self.actions,
+                        state=conflict_state,
+                    )
                     self.actions = []
 
-                action = next(pending_actions, None)
+                action = next(action_iter, None)
                 if action is None:
                     # we are done!
                     break
@@ -1176,9 +1142,7 @@ def resume(actions):
                 try:
                     if callable is not None:
                         callable(*args, **kw)
-                except (KeyboardInterrupt, SystemExit): # pragma: no cover
-                    raise
-                except:
+                except Exception:
                     t, v, tb = sys.exc_info()
                     try:
                         reraise(ConfigurationExecutionError,
@@ -1193,65 +1157,102 @@ def resume(actions):
 
                 executed_actions.append(action)
 
+            self.actions = all_actions
+            return executed_actions
+
         finally:
             if clear:
-                del self.actions[:]
-            else:
-                self.actions = all_actions
+                self.actions = []
+
+
+class ConflictResolverState(object):
+    def __init__(self):
+        # keep a set of resolved discriminators to test against to ensure
+        # that a new action does not conflict with something already executed
+        self.resolved_ainfos = {}
+
+        # actions left over from a previous iteration
+        self.remaining_actions = []
+
+        # after executing an action we memoize its order to avoid any new
+        # actions sending us backward
+        self.min_order = None
+
+        # unique tracks the index of the action so we need it to increase
+        # monotonically across invocations to resolveConflicts
+        self.start = 0
+
 
 # this function is licensed under the ZPL (stolen from Zope)
-def resolveConflicts(actions):
+def resolveConflicts(actions, state=None):
     """Resolve conflicting actions
 
     Given an actions list, identify and try to resolve conflicting actions.
     Actions conflict if they have the same non-None discriminator.
+
     Conflicting actions can be resolved if the include path of one of
     the actions is a prefix of the includepaths of the other
     conflicting actions and is unequal to the include paths in the
     other conflicting actions.
+
+    Actions are resolved on a per-order basis because some discriminators
+    cannot be computed until earlier actions have executed. An action in an
+    earlier order may execute successfully only to find out later that it was
+    overridden by another action with a smaller include path. This will result
+    in a conflict as there is no way to revert the original action.
+
+    ``state`` may be an instance of ``ConflictResolverState`` that
+    can be used to resume execution and resolve the new actions against the
+    list of executed actions from a previous call.
+
     """
+    if state is None:
+        state = ConflictResolverState()
+
+    # pick up where we left off last time, but track the new actions as well
+    state.remaining_actions.extend(normalize_actions(actions))
+    actions = state.remaining_actions
 
     def orderandpos(v):
         n, v = v
-        if not isinstance(v, dict):
-            # old-style tuple action
-            v = expand_action(*v)
         return (v['order'] or 0, n)
 
-    sactions = sorted(enumerate(actions), key=orderandpos)
-
     def orderonly(v):
         n, v = v
-        if not isinstance(v, dict):
-            # old-style tuple action
-            v = expand_action(*v)
         return v['order'] or 0
 
+    sactions = sorted(enumerate(actions, start=state.start), key=orderandpos)
     for order, actiongroup in itertools.groupby(sactions, orderonly):
         # "order" is an integer grouping. Actions in a lower order will be
         # executed before actions in a higher order.  All of the actions in
         # one grouping will be executed (its callable, if any will be called)
         # before any of the actions in the next.
-        
-        unique = {}
         output = []
+        unique = {}
+
+        # error out if we went backward in order
+        if state.min_order is not None and order < state.min_order:
+            r = ['Actions were added to order={0} after execution had moved '
+                 'on to order={1}. Conflicting actions: '
+                 .format(order, state.min_order)]
+            for i, action in actiongroup:
+                for line in str(action['info']).rstrip().split('\n'):
+                    r.append("  " + line)
+            raise ConfigurationError('\n'.join(r))
 
         for i, action in actiongroup:
             # Within an order, actions are executed sequentially based on
             # original action ordering ("i").
 
-            if not isinstance(action, dict):
-                # old-style tuple action
-                action = expand_action(*action)
-
-            # "ainfo" is a tuple of (order, i, action) where "order" is a
-            # user-supplied grouping, "i" is an integer expressing the relative
-            # position of this action in the action list being resolved, and
-            # "action" is an action dictionary.  The purpose of an ainfo is to
-            # associate an "order" and an "i" with a particular action; "order"
-            # and "i" exist for sorting purposes after conflict resolution.
-            ainfo = (order, i, action)
+            # "ainfo" is a tuple of (i, action) where "i" is an integer
+            # expressing the relative position of this action in the action
+            # list being resolved, and "action" is an action dictionary.  The
+            # purpose of an ainfo is to associate an "i" with a particular
+            # action; "i" exists for sorting after conflict resolution.
+            ainfo = (i, action)
 
+            # wait to defer discriminators until we are on their order because
+            # the discriminator may depend on state from a previous order
             discriminator = undefer(action['discriminator'])
             action['discriminator'] = discriminator
 
@@ -1266,28 +1267,39 @@ def orderonly(v):
 
         # Check for conflicts
         conflicts = {}
-
         for discriminator, ainfos in unique.items():
-            # We use (includepath, order, i) as a sort key because we need to
+            # We use (includepath, i) as a sort key because we need to
             # sort the actions by the paths so that the shortest path with a
             # given prefix comes first.  The "first" action is the one with the
-            # shortest include path.  We break sorting ties using "order", then
-            # "i".
+            # shortest include path.  We break sorting ties using "i".
             def bypath(ainfo):
-                path, order, i = ainfo[2]['includepath'], ainfo[0], ainfo[1]
+                path, i = ainfo[1]['includepath'], ainfo[0]
                 return path, order, i
 
             ainfos.sort(key=bypath)
             ainfo, rest = ainfos[0], ainfos[1:]
-            output.append(ainfo)
-            _, _, action = ainfo
-            basepath, baseinfo, discriminator = (
-                action['includepath'],
-                action['info'],
-                action['discriminator'],
-                )
+            _, action = ainfo
+
+            # ensure this new action does not conflict with a previously
+            # resolved action from an earlier order / invocation
+            prev_ainfo = state.resolved_ainfos.get(discriminator)
+            if prev_ainfo is not None:
+                _, paction = prev_ainfo
+                basepath, baseinfo = paction['includepath'], paction['info']
+                includepath = action['includepath']
+                # if the new action conflicts with the resolved action then
+                # note the conflict, otherwise drop the action as it's
+                # effectively overriden by the previous action
+                if (includepath[:len(basepath)] != basepath or
+                        includepath == basepath):
+                    L = conflicts.setdefault(discriminator, [baseinfo])
+                    L.append(action['info'])
+
+            else:
+                output.append(ainfo)
 
-            for _, _, action in rest:
+            basepath, baseinfo = action['includepath'], action['info']
+            for _, action in rest:
                 includepath = action['includepath']
                 # Test whether path is a prefix of opath
                 if (includepath[:len(basepath)] != basepath or  # not a prefix
@@ -1298,14 +1310,30 @@ def bypath(ainfo):
         if conflicts:
             raise ConfigurationConflictError(conflicts)
 
-        # sort conflict-resolved actions by (order, i) and yield them one
-        # by one
-        for a in [x[2] for x in sorted(output, key=operator.itemgetter(0, 1))]:
-            yield a
+        # sort resolved actions by "i" and yield them one by one
+        for i, action in sorted(output, key=operator.itemgetter(0)):
+            # do not memoize the order until we resolve an action inside it
+            state.min_order = action['order']
+            state.start = i + 1
+            state.remaining_actions.remove(action)
+            state.resolved_ainfos[action['discriminator']] = (i, action)
+            yield action
 
 
-def expand_action(discriminator, callable=None, args=(), kw=None,
-                  includepath=(), info=None, order=0, introspectables=()):
+def normalize_actions(actions):
+    """Convert old-style tuple actions to new-style dicts."""
+    result = []
+    for v in actions:
+        if not isinstance(v, dict):
+            v = expand_action_tuple(*v)
+        result.append(v)
+    return result
+
+
+def expand_action_tuple(
+    discriminator, callable=None, args=(), kw=None, includepath=(),
+    info=None, order=0, introspectables=(),
+):
     if kw is None:
         kw = {}
     return dict(
@@ -1319,4 +1347,5 @@ def expand_action(discriminator, callable=None, args=(), kw=None,
         introspectables=introspectables,
         )
 
+
 global_registries = WeakOrderedSet()


Test patch:
diff --git a/pyramid/tests/test_config/test_init.py b/pyramid/tests/test_config/test_init.py
--- a/pyramid/tests/test_config/test_init.py
+++ b/pyramid/tests/test_config/test_init.py
@@ -1545,6 +1545,31 @@ def g(*a, **k):
         c.execute_actions()
         self.assertEqual(output, [('f', (1,), {}), ('g', (8,), {})])
 
+    def test_reentrant_action_with_deferred_discriminator(self):
+        # see https://github.com/Pylons/pyramid/issues/2697
+        from pyramid.registry import Deferred
+        output = []
+        c = self._makeOne()
+        def f(*a, **k):
+            output.append(('f', a, k))
+            c.actions.append((4, g, (4,), {}, (), None, 2))
+        def g(*a, **k):
+            output.append(('g', a, k))
+        def h(*a, **k):
+            output.append(('h', a, k))
+        def discrim():
+            self.assertEqual(output, [('f', (1,), {}), ('g', (2,), {})])
+            return 3
+        d = Deferred(discrim)
+        c.actions = [
+            (d, h, (3,), {}, (), None, 1), # order 1
+            (1, f, (1,)), # order 0
+            (2, g, (2,)), # order 0
+        ]
+        c.execute_actions()
+        self.assertEqual(output, [
+            ('f', (1,), {}), ('g', (2,), {}), ('h', (3,), {}), ('g', (4,), {})])
+
     def test_reentrant_action_error(self):
         from pyramid.exceptions import ConfigurationError
         c = self._makeOne()
@@ -1570,6 +1595,28 @@ def g(*a, **k): pass
             (3, g, (8,)),
         ])
 
+    def test_executing_conflicting_action_across_orders(self):
+        from pyramid.exceptions import ConfigurationConflictError
+        c = self._makeOne()
+        def f(*a, **k): pass
+        def g(*a, **k): pass
+        c.actions = [
+            (1, f, (1,), {}, (), None, -1),
+            (1, g, (2,)),
+        ]
+        self.assertRaises(ConfigurationConflictError, c.execute_actions)
+
+    def test_executing_conflicting_action_across_reentrant_orders(self):
+        from pyramid.exceptions import ConfigurationConflictError
+        c = self._makeOne()
+        def f(*a, **k):
+            c.actions.append((1, g, (8,)))
+        def g(*a, **k): pass
+        c.actions = [
+            (1, f, (1,), {}, (), None, -1),
+        ]
+        self.assertRaises(ConfigurationConflictError, c.execute_actions)
+
 class Test_reentrant_action_functional(unittest.TestCase):
     def _makeConfigurator(self, *arg, **kw):
         from pyramid.config import Configurator
@@ -1597,6 +1644,21 @@ def my_view(request): return request.response
         self.assertEqual(route.name, 'foo')
         self.assertEqual(route.path, '/foo')
 
+    def test_deferred_discriminator(self):
+        # see https://github.com/Pylons/pyramid/issues/2697
+        from pyramid.config import PHASE0_CONFIG
+        config = self._makeConfigurator()
+        def deriver(view, info): return view
+        deriver.options = ('foo',)
+        config.add_view_deriver(deriver, 'foo_view')
+        # add_view uses a deferred discriminator and will fail if executed
+        # prior to add_view_deriver executing its action
+        config.add_view(lambda r: r.response, name='', foo=1)
+        def dummy_action():
+            # trigger a re-entrant action
+            config.action(None, lambda: None)
+        config.action(None, dummy_action, order=PHASE0_CONFIG)
+        config.commit()
 
 class Test_resolveConflicts(unittest.TestCase):
     def _callFUT(self, actions):
@@ -1666,15 +1728,14 @@ def test_it_success_tuples(self):
 
     def test_it_success_dicts(self):
         from pyramid.tests.test_config import dummyfactory as f
-        from pyramid.config import expand_action
         result = self._callFUT([
-            expand_action(None, f),
-            expand_action(1, f, (1,), {}, (), 'first'),
-            expand_action(1, f, (2,), {}, ('x',), 'second'),
-            expand_action(1, f, (3,), {}, ('y',), 'third'),
-            expand_action(4, f, (4,), {}, ('y',), 'should be last', 99999),
-            expand_action(3, f, (3,), {}, ('y',)),
-            expand_action(None, f, (5,), {}, ('y',)),
+            (None, f),
+            (1, f, (1,), {}, (), 'first'),
+            (1, f, (2,), {}, ('x',), 'second'),
+            (1, f, (3,), {}, ('y',), 'third'),
+            (4, f, (4,), {}, ('y',), 'should be last', 99999),
+            (3, f, (3,), {}, ('y',)),
+            (None, f, (5,), {}, ('y',)),
             ])
         result = list(result)
         self.assertEqual(
@@ -1740,17 +1801,16 @@ def test_it_conflict(self):
 
     def test_it_with_actions_grouped_by_order(self):
         from pyramid.tests.test_config import dummyfactory as f
-        from pyramid.config import expand_action
         result = self._callFUT([
-            expand_action(None, f),                                 # X
-            expand_action(1, f, (1,), {}, (), 'third', 10),         # X
-            expand_action(1, f, (2,), {}, ('x',), 'fourth', 10),
-            expand_action(1, f, (3,), {}, ('y',), 'fifth', 10),
-            expand_action(2, f, (1,), {}, (), 'sixth', 10),         # X
-            expand_action(3, f, (1,), {}, (), 'seventh', 10),       # X
-            expand_action(5, f, (4,), {}, ('y',), 'eighth', 99999), # X
-            expand_action(4, f, (3,), {}, (), 'first', 5),          # X
-            expand_action(4, f, (5,), {}, ('y',), 'second', 5),
+            (None, f),                                 # X
+            (1, f, (1,), {}, (), 'third', 10),         # X
+            (1, f, (2,), {}, ('x',), 'fourth', 10),
+            (1, f, (3,), {}, ('y',), 'fifth', 10),
+            (2, f, (1,), {}, (), 'sixth', 10),         # X
+            (3, f, (1,), {}, (), 'seventh', 10),       # X
+            (5, f, (4,), {}, ('y',), 'eighth', 99999), # X
+            (4, f, (3,), {}, (), 'first', 5),          # X
+            (4, f, (5,), {}, ('y',), 'second', 5),
             ])
         result = list(result)
         self.assertEqual(len(result), 6)
@@ -1812,7 +1872,32 @@ def test_it_with_actions_grouped_by_order(self):
                   'order': 99999}
                   ]
                   )
-        
+
+    def test_override_success_across_orders(self):
+        from pyramid.tests.test_config import dummyfactory as f
+        result = self._callFUT([
+            (1, f, (2,), {}, ('x',), 'eek', 0),
+            (1, f, (3,), {}, ('x', 'y'), 'ack', 10),
+            ])
+        result = list(result)
+        self.assertEqual(result, [
+            {'info': 'eek',
+            'args': (2,),
+            'callable': f,
+            'introspectables': (),
+            'kw': {},
+            'discriminator': 1,
+            'includepath': ('x',),
+            'order': 0},
+        ])
+
+    def test_conflicts_across_orders(self):
+        from pyramid.tests.test_config import dummyfactory as f
+        result = self._callFUT([
+            (1, f, (2,), {}, ('x', 'y'), 'eek', 0),
+            (1, f, (3,), {}, ('x'), 'ack', 10),
+            ])
+        self.assertRaises(ConfigurationConflictError, list, result)
 
 class TestGlobalRegistriesIntegration(unittest.TestCase):
     def setUp(self):