Repository: Pylons/pyramid

Issue:
Document interaction of .override_asset() with .add_translation_dirs() and others
It looks like `.override_asset()` is not effective in the subsequent `.add_translation_dirs()` unless `.commit()` is called between these two calls. However I can't find this stated neither at http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/assets.html#assets-chapter nor at http://docs.pylonsproject.org/projects/pyramid/en/latest/api/config.html#pyramid.config.Configurator.override_asset
I think interaction of `.override_asset()` with other directives should be described in documentation.

Document interaction of .override_asset() with .add_translation_dirs() and others
It looks like `.override_asset()` is not effective in the subsequent `.add_translation_dirs()` unless `.commit()` is called between these two calls. However I can't find this stated neither at http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/assets.html#assets-chapter nor at http://docs.pylonsproject.org/projects/pyramid/en/latest/api/config.html#pyramid.config.Configurator.override_asset
I think interaction of `.override_asset()` with other directives should be described in documentation.



Discussion:
@piotr-dobrogost please check against the [master branch](http://docs.pylonsproject.org/projects/pyramid/en/master/narr/assets.html#overriding-assets). After doing so, please suggest an improvement either on `master` (same as `1.6-branch`) or `1.5-branch` (which RTD renders as "latest". (The mapping of RTD URLs to branches on GitHub is not obvious at first.)

@stevepiercy 
If you mean I should check docs in master branch then there's no difference with regard to this issue. Before suggesting any changes to docs I would like to know if current behavior is by design or by accident. If the latter then I would raise another issue suggesting to fix it and then we could document proper behavior in docs which would close this issue.

I think there's a bug in the current implementation of `add_translation_dirs`.

Consider this code:

```
config.override_asset('some.package', 'other.package')
config.commit()
config.add_translation_dirs('some.package:locale')
```

If `some.package` contains a `locale` directory but `other.package` does not, a `pyramid.exceptions.ConfigurationError: "<path-to-other.package>/locale" is not a directory` will be raised.  This is caused by first splitting the asset spec, then concatenating the package path and the file name (https://github.com/Pylons/pyramid/blob/master/pyramid/config/i18n.py#L83).  In the example above `package_path(package)` will be the absolute path of `other.package`, which doesn't contain a `locale` directory and hence raises the exception.

If https://github.com/Pylons/pyramid/blob/master/pyramid/config/i18n.py#L81-L83 would be `directory = pkg_resources.resource_filename(package_name, filename)` instead, everything should work as expected, also with asset overrides in place. 

Also see https://github.com/Kotti/Kotti/issues/475.

@disko Could you please submit a fix with a failing unit test? In Pyramid 1.6 we also support overriding absolute paths so there may be some issues there that it would be helpful if you could investigate. I think your fix makes sense though.

I agree that `add_translation_dirs` should be using an `AssetResolver` to discover the locations. We should also adjust the PHASE of `override_asset` such that it occurs in an earlier cycle than `add_translation_dirs` which would ensure that the order no longer matters between the two.

Another subtle issue that I'm trying to understand is why asset overrides work at all. It looks like it only works if the package itself is overridden, but not the folder name inside the package. For example, I expect `override_asset('some.package:locale/' 'other.package:not_locale/')` to fail because it isn't passing the folder name through any asset resolution path... only the package name itself.

Finally, this only works because kotti is calling `config.begin()` to push the threadlocale asset resolver into `pkg_resources`. This won't work in other pyramid apps without calling `config.begin()` (or proposed #2872). This should also be documented better.

2 bugs and bad docs! Lots to fix!
@piotr-dobrogost please check against the [master branch](http://docs.pylonsproject.org/projects/pyramid/en/master/narr/assets.html#overriding-assets). After doing so, please suggest an improvement either on `master` (same as `1.6-branch`) or `1.5-branch` (which RTD renders as "latest". (The mapping of RTD URLs to branches on GitHub is not obvious at first.)

@stevepiercy 
If you mean I should check docs in master branch then there's no difference with regard to this issue. Before suggesting any changes to docs I would like to know if current behavior is by design or by accident. If the latter then I would raise another issue suggesting to fix it and then we could document proper behavior in docs which would close this issue.

I think there's a bug in the current implementation of `add_translation_dirs`.

Consider this code:

```
config.override_asset('some.package', 'other.package')
config.commit()
config.add_translation_dirs('some.package:locale')
```

If `some.package` contains a `locale` directory but `other.package` does not, a `pyramid.exceptions.ConfigurationError: "<path-to-other.package>/locale" is not a directory` will be raised.  This is caused by first splitting the asset spec, then concatenating the package path and the file name (https://github.com/Pylons/pyramid/blob/master/pyramid/config/i18n.py#L83).  In the example above `package_path(package)` will be the absolute path of `other.package`, which doesn't contain a `locale` directory and hence raises the exception.

If https://github.com/Pylons/pyramid/blob/master/pyramid/config/i18n.py#L81-L83 would be `directory = pkg_resources.resource_filename(package_name, filename)` instead, everything should work as expected, also with asset overrides in place. 

Also see https://github.com/Kotti/Kotti/issues/475.

@disko Could you please submit a fix with a failing unit test? In Pyramid 1.6 we also support overriding absolute paths so there may be some issues there that it would be helpful if you could investigate. I think your fix makes sense though.

I agree that `add_translation_dirs` should be using an `AssetResolver` to discover the locations. We should also adjust the PHASE of `override_asset` such that it occurs in an earlier cycle than `add_translation_dirs` which would ensure that the order no longer matters between the two.

Another subtle issue that I'm trying to understand is why asset overrides work at all. It looks like it only works if the package itself is overridden, but not the folder name inside the package. For example, I expect `override_asset('some.package:locale/' 'other.package:not_locale/')` to fail because it isn't passing the folder name through any asset resolution path... only the package name itself.

Finally, this only works because kotti is calling `config.begin()` to push the threadlocale asset resolver into `pkg_resources`. This won't work in other pyramid apps without calling `config.begin()` (or proposed #2872). This should also be documented better.

2 bugs and bad docs! Lots to fix!

Resolving patch:
diff --git a/pyramid/config/__init__.py b/pyramid/config/__init__.py
--- a/pyramid/config/__init__.py
+++ b/pyramid/config/__init__.py
@@ -451,9 +451,6 @@ def _make_spec(self, path_or_spec):
             return filename # absolute filename
         return '%s:%s' % (package, filename)
 
-    def _split_spec(self, path_or_spec):
-        return resolve_asset_spec(path_or_spec, self.package_name)
-
     def _fix_registry(self):
         """ Fix up a ZCA component registry that is not a
         pyramid.registry.Registry by adding analogues of ``has_listeners``,
@@ -606,11 +603,15 @@ def action(self, discriminator, callable=None, args=(), kw=None, order=0,
         if autocommit:
             # callables can depend on the side effects of resolving a
             # deferred discriminator
-            undefer(discriminator)
-            if callable is not None:
-                callable(*args, **kw)
-            for introspectable in introspectables:
-                introspectable.register(self.introspector, action_info)
+            self.begin()
+            try:
+                undefer(discriminator)
+                if callable is not None:
+                    callable(*args, **kw)
+                for introspectable in introspectables:
+                    introspectable.register(self.introspector, action_info)
+            finally:
+                self.end()
 
         else:
             action = extra
@@ -651,7 +652,11 @@ def commit(self):
         of this error will be information about the source of the conflict,
         usually including file names and line numbers of the cause of the
         configuration conflicts."""
-        self.action_state.execute_actions(introspector=self.introspector)
+        self.begin()
+        try:
+            self.action_state.execute_actions(introspector=self.introspector)
+        finally:
+            self.end()
         self.action_state = ActionState() # old actions have been processed
 
     def include(self, callable, route_prefix=None):
@@ -885,14 +890,30 @@ def absolute_asset_spec(self, relative_spec):
 
     absolute_resource_spec = absolute_asset_spec # b/w compat forever
 
-    def begin(self, request=None):
+    def begin(self, request=_marker):
         """ Indicate that application or test configuration has begun.
         This pushes a dictionary containing the :term:`application
         registry` implied by ``registry`` attribute of this
         configurator and the :term:`request` implied by the
         ``request`` argument onto the :term:`thread local` stack
         consulted by various :mod:`pyramid.threadlocal` API
-        functions."""
+        functions.
+
+        If ``request`` is not specified and the registry owned by the
+        configurator is already pushed as the current threadlocal registry
+        then this method will keep the current threadlocal request unchanged.
+
+        .. versionchanged:: 1.8
+           The current threadlocal request is propagated if the current
+           threadlocal registry remains unchanged.
+
+        """
+        if request is _marker:
+            current = self.manager.get()
+            if current['registry'] == self.registry:
+                request = current['request']
+            else:
+                request = None
         self.manager.push({'registry':self.registry, 'request':request})
 
     def end(self):
@@ -992,11 +1013,11 @@ def make_wsgi_app(self):
         # Push the registry onto the stack in case any code that depends on
         # the registry threadlocal APIs used in listeners subscribed to the
         # IApplicationCreated event.
-        self.manager.push({'registry': self.registry, 'request': None})
+        self.begin()
         try:
             self.registry.notify(ApplicationCreated(app))
         finally:
-            self.manager.pop()
+            self.end()
 
         return app
 
diff --git a/pyramid/config/assets.py b/pyramid/config/assets.py
--- a/pyramid/config/assets.py
+++ b/pyramid/config/assets.py
@@ -4,7 +4,10 @@
 
 from zope.interface import implementer
 
-from pyramid.interfaces import IPackageOverrides
+from pyramid.interfaces import (
+    IPackageOverrides,
+    PHASE1_CONFIG,
+)
 
 from pyramid.exceptions import ConfigurationError
 from pyramid.threadlocal import get_current_registry
@@ -387,6 +390,7 @@ def register():
             )
         intr['to_override'] = to_override
         intr['override_with'] = override_with
-        self.action(None, register, introspectables=(intr,))
+        self.action(None, register, introspectables=(intr,),
+                    order=PHASE1_CONFIG)
 
     override_resource = override_asset # bw compat
diff --git a/pyramid/config/i18n.py b/pyramid/config/i18n.py
--- a/pyramid/config/i18n.py
+++ b/pyramid/config/i18n.py
@@ -1,13 +1,10 @@
-import os
-import sys
-
 from pyramid.interfaces import (
     ILocaleNegotiator,
     ITranslationDirectories,
     )
 
 from pyramid.exceptions import ConfigurationError
-from pyramid.path import package_path
+from pyramid.path import AssetResolver
 from pyramid.util import action_method
 
 class I18NConfiguratorMixin(object):
@@ -69,32 +66,32 @@ def add_translation_dirs(self, *specs):
         directories will be inserted into the beginning of the directory list
         in the order they're provided in the ``*specs`` list argument (items
         earlier in the list trump ones later in the list).
+
         """
         directories = []
         introspectables = []
-
-        for spec in specs[::-1]: # reversed
-            package_name, filename = self._split_spec(spec)
-            if package_name is None: # absolute filename
-                directory = filename
-            else:
-                __import__(package_name)
-                package = sys.modules[package_name]
-                directory = os.path.join(package_path(package), filename)
-
-            if not os.path.isdir(os.path.realpath(directory)):
-                raise ConfigurationError('"%s" is not a directory' %
-                                         directory)
-            intr = self.introspectable('translation directories', directory,
-                                       spec, 'translation directory')
-            intr['directory'] = directory
-            intr['spec'] = spec
-            introspectables.append(intr)
-            directories.append(directory)
+        resolver = AssetResolver(self.package_name)
 
         def register():
-            for directory in directories:
+            # defer spec resolution until register to allow for asset
+            # overrides to take place in an earlier config phase
+            for spec in specs[::-1]: # reversed
+                # the trailing slash helps match asset overrides for folders
+                if not spec.endswith('/'):
+                    spec += '/'
+                asset = resolver.resolve(spec)
+                directory = asset.abspath()
+                if not asset.isdir():
+                    raise ConfigurationError('"%s" is not a directory' %
+                                            directory)
+                intr = self.introspectable('translation directories', directory,
+                                        spec, 'translation directory')
+                intr['directory'] = directory
+                intr['spec'] = spec
+                introspectables.append(intr)
+                directories.append(directory)
 
+            for directory in directories:
                 tdirs = self.registry.queryUtility(ITranslationDirectories)
                 if tdirs is None:
                     tdirs = []


Test patch:
diff --git a/pyramid/tests/test_config/test_i18n.py b/pyramid/tests/test_config/test_i18n.py
--- a/pyramid/tests/test_config/test_i18n.py
+++ b/pyramid/tests/test_config/test_i18n.py
@@ -36,9 +36,8 @@ def test_set_locale_negotiator_dottedname(self):
     def test_add_translation_dirs_missing_dir(self):
         from pyramid.exceptions import ConfigurationError
         config = self._makeOne()
-        self.assertRaises(ConfigurationError,
-                          config.add_translation_dirs,
-                          '/wont/exist/on/my/system')
+        config.add_translation_dirs('/wont/exist/on/my/system')
+        self.assertRaises(ConfigurationError, config.commit)
 
     def test_add_translation_dirs_no_specs(self):
         from pyramid.interfaces import ITranslationDirectories
@@ -87,3 +86,30 @@ def test_add_translation_dirs_abspath(self):
         self.assertEqual(config.registry.getUtility(ITranslationDirectories),
                          [locale])
 
+    def test_add_translation_dirs_uses_override_out_of_order(self):
+        from pyramid.interfaces import ITranslationDirectories
+        config = self._makeOne()
+        config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale')
+        config.override_asset('pyramid.tests.pkgs.localeapp:locale/',
+                              'pyramid.tests.pkgs.localeapp:locale2/')
+        config.commit()
+        self.assertEqual(config.registry.getUtility(ITranslationDirectories),
+                         [locale2])
+
+    def test_add_translation_dirs_doesnt_use_override_w_autocommit(self):
+        from pyramid.interfaces import ITranslationDirectories
+        config = self._makeOne(autocommit=True)
+        config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale')
+        config.override_asset('pyramid.tests.pkgs.localeapp:locale/',
+                              'pyramid.tests.pkgs.localeapp:locale2/')
+        self.assertEqual(config.registry.getUtility(ITranslationDirectories),
+                         [locale])
+
+    def test_add_translation_dirs_uses_override_w_autocommit(self):
+        from pyramid.interfaces import ITranslationDirectories
+        config = self._makeOne(autocommit=True)
+        config.override_asset('pyramid.tests.pkgs.localeapp:locale/',
+                              'pyramid.tests.pkgs.localeapp:locale2/')
+        config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale')
+        self.assertEqual(config.registry.getUtility(ITranslationDirectories),
+                         [locale2])
diff --git a/pyramid/tests/test_config/test_init.py b/pyramid/tests/test_config/test_init.py
--- a/pyramid/tests/test_config/test_init.py
+++ b/pyramid/tests/test_config/test_init.py
@@ -91,13 +91,54 @@ def test_begin_with_request(self):
                          {'registry':config.registry, 'request':request})
         self.assertEqual(manager.popped, False)
 
+    def test_begin_overrides_request(self):
+        from pyramid.config import Configurator
+        config = Configurator()
+        manager = DummyThreadLocalManager()
+        req = object()
+        # set it up for auto-propagation
+        pushed = {'registry': config.registry, 'request': None}
+        manager.pushed = pushed
+        config.manager = manager
+        config.begin(req)
+        self.assertTrue(manager.pushed is not pushed)
+        self.assertEqual(manager.pushed['request'], req)
+        self.assertEqual(manager.pushed['registry'], config.registry)
+
+    def test_begin_propagates_request_for_same_registry(self):
+        from pyramid.config import Configurator
+        config = Configurator()
+        manager = DummyThreadLocalManager()
+        req = object()
+        pushed = {'registry': config.registry, 'request': req}
+        manager.pushed = pushed
+        config.manager = manager
+        config.begin()
+        self.assertTrue(manager.pushed is not pushed)
+        self.assertEqual(manager.pushed['request'], req)
+        self.assertEqual(manager.pushed['registry'], config.registry)
+
+    def test_begin_does_not_propagate_request_for_diff_registry(self):
+        from pyramid.config import Configurator
+        config = Configurator()
+        manager = DummyThreadLocalManager()
+        req = object()
+        pushed = {'registry': object(), 'request': req}
+        manager.pushed = pushed
+        config.manager = manager
+        config.begin()
+        self.assertTrue(manager.pushed is not pushed)
+        self.assertEqual(manager.pushed['request'], None)
+        self.assertEqual(manager.pushed['registry'], config.registry)
+
     def test_end(self):
         from pyramid.config import Configurator
         config = Configurator()
         manager = DummyThreadLocalManager()
+        pushed = manager.pushed
         config.manager = manager
         config.end()
-        self.assertEqual(manager.pushed, None)
+        self.assertEqual(manager.pushed, pushed)
         self.assertEqual(manager.popped, True)
 
     def test_ctor_with_package_registry(self):
@@ -1940,10 +1981,13 @@ def __init__(self, environ=None):
         self.cookies = {}
 
 class DummyThreadLocalManager(object):
-    pushed = None
-    popped = False
+    def __init__(self):
+        self.pushed = {'registry': None, 'request': None}
+        self.popped = False
     def push(self, d):
         self.pushed = d
+    def get(self):
+        return self.pushed
     def pop(self):
         self.popped = True