Repository: Pylons/pyramid

Issue:
add_translation_dirs behaves the wrong way around
This is a follow-up to #1473. See the reproducer and notes there.

I would propose an API change that changes behaviour of add_translation_dirs to append specs at the end of existing specs instead of prepending it. That way, API users simply need to understand "last spec wins". This matches much closer to the mental model of "add".

It matches the current documented behaviour of how the method is to _behave_ in case of multiple calls.



Discussion:
https://github.com/Kotti/Kotti which is based on Pyramid and meant as a framework calls its `includeme()` and then calls `includeme()` of all add-ons. Current behavior thus makes it impossible to override Kotti's own translations in add-on. Is the order of calling `includeme()` the same in other apps based on Pyramid and meant to be extensible? If so then I agree with @rbu's  suggestion to change implementation so that _last spec wins_. Could we have acceptance for this API change from maintainers? `includeme()` is clearly meant to add or override declarations from earlier calls to other `includeme()`s Current behavior with regard to translations is directly opposite to this idea thus counter-intuitive and should be changed so that [`add_translation_dirs`](http://docs.pylonsproject.org/projects/pyramid/en/latest/api/config.html?highlight=add_translation_dirs#pyramid.config.Configurator.add_translation_dirs) directive be in line with all other [_configuration directives_](http://docs.pylonsproject.org/projects/pyramid/en/latest/glossary.html#term-configuration-directive).

It seems docstring with statements not in line with behavior was introduced in https://github.com/Pylons/pyramid/commit/5bf23fad3e0c9a949e8b01ab68991633af2d6df7 by @mcdonc 

-1 on anything that changes existing behaviour. There are far too many existing applications relying on the current behaviour, all of which would be broken by such a change.

FWIW I agree there is a problem here that should be solved, I just don't think we can make change existing behaviour. The simplest solution I can see is to add a parameter that to make `add_translation_dirs` add the new path at start of the search path. That should be a simple change to make.

I'm hesitant to change the existing behavior but on the other hand I think the current behavior is wrong and it would be weird for any apps to be relying on it.

@wichert can you say anything about that? Would it break any of your apps? I'm talking specifically about 2 consecutive calls to `config.add_translation_dirs()`.

It seems to me this is simply about fixing the dirs to append instead of prepend as the last folder wins in an i18n search instead of the first folder.

Is something like the following remotely correct or am I misunderstanding this entirely?

``` diff
diff --git a/pyramid/config/i18n.py b/pyramid/config/i18n.py
index 69af0f9..f8e5be9 100644
--- a/pyramid/config/i18n.py
+++ b/pyramid/config/i18n.py
@@ -73,7 +73,7 @@ class I18NConfiguratorMixin(object):
         directories = []
         introspectables = []

-        for spec in specs[::-1]: # reversed
+        for spec in specs:
             package_name, filename = self._split_spec(spec)
             if package_name is None: # absolute filename
                 directory = filename
@@ -93,15 +93,12 @@ class I18NConfiguratorMixin(object):
             directories.append(directory)

         def register():
-            for directory in directories:
-
-                tdirs = self.registry.queryUtility(ITranslationDirectories)
-                if tdirs is None:
-                    tdirs = []
-                    self.registry.registerUtility(tdirs,
-                                                  ITranslationDirectories)
-
-                tdirs.insert(0, directory)
+            tdirs = self.registry.queryUtility(ITranslationDirectories)
+            if tdirs is None:
+                tdirs = []
+                self.registry.registerUtility(tdirs,
+                                                ITranslationDirectories)
+            tdirs.extend(directories)

         self.action(None, register, introspectables=introspectables)
```

@mmerickel I have several apps in production which use a chain of PO files so a package can override (parts of) the translation from another package. Your proposed change would break those: the system would suddenly return the original translation instead of the app-specific override. It would mean that as part of migrating to a newer Pyramid version I will have to reverse the order of all add_translation_dirs() calls (some of which are indirect via config.include() to other packages).

I think there are two factors here: 1) in which order the list of directories passed to add_translation_dirs() is scanned, and 2) how successive calls to add_translation_dirs() are added to the search path. Changing 1) would break at least one of my apps, and not solve the real problem here. I can't easily tell if your proposed change does this or not, not knowing how spec is used. Fixing 2 by making a later add_translation_dirs() call add paths at the start of the lookup should address the problem, but has a risk of breaking apps (like mine) that currently use a specific order of config.include() and config.add_translation_dirs() to get the right translation search order.

@wichert 

> I think there are two factors here: 1) in which order the list of directories passed to add_translation_dirs() is scanned, and 2) how successive calls to add_translation_dirs() are added to the search path.

I think this is described in issue #1473, no?

@wichert I'm explicitly uninterested in breaking a single call to `add_translation_dirs`. I want to fix the issue with multiple calls.

At a high level pyramid's goal is to not enforce strict ordering on includes, but for certain things like "add to a search path" it's pretty much impossible without some sort of weighting.

@wichert 
> Fixing 2 by making a later add_translation_dirs() call add paths at the start of the lookup should address the problem

Actually fixing means putting them at the end not at the start. Michael is right here (emphasis mine):
> It seems to me this is simply about fixing the dirs to append instead of prepend **as the last folder wins in an i18n search instead of the first folder**.

As to
> (...) but has a risk of breaking apps (like mine) that currently use a specific order of config.include() and config.add_translation_dirs() to get the right translation search order.

here too I agree with Michael:

> (...) I think the current behavior is wrong and it would be weird for any apps to be relying on it.


I would accept a patch that reverses the order such that later `add_translation_dirs` calls cause their folders to be searched before earlier calls, especially if a new parameter is added to the signature to preserve bw-compat such as a default of `prepend=False`. This would mirror the `config.add_jinja2_search_path` API [1] as well, which seems sane.

[1] http://docs.pylonsproject.org/projects/pyramid-jinja2/en/latest/api.html#pyramid_jinja2.add_jinja2_search_path
> I would accept a patch that reverses the order such that later add_translation_dirs calls cause their folders to be searched before earlier calls, especially if a new parameter is added to the signature to preserve bw-compat such as a default of `prepend=False`.

This is the wrong way around. You had it right here (emphasis mine):
> It seems to me this is simply about fixing the dirs to append instead of prepend as **the last folder wins in an i18n search instead of the first folder**.

It's the opposite of the common search pattern where some list is being iterated and the first item found wins (like in PATH env var or jinja2's search path I guess). This is just how gettext works.
So this would be `append=False` not `prepend=False`. However I would suggest biting the bullet and making it `append=True`. Yes, it would break some apps but rightfully so.

Resolving patch:
diff --git a/pyramid/config/i18n.py b/pyramid/config/i18n.py
--- a/pyramid/config/i18n.py
+++ b/pyramid/config/i18n.py
@@ -42,7 +42,7 @@ def _set_locale_negotiator(self, negotiator):
         self.registry.registerUtility(locale_negotiator, ILocaleNegotiator)
 
     @action_method
-    def add_translation_dirs(self, *specs):
+    def add_translation_dirs(self, *specs, **kw):
         """ Add one or more :term:`translation directory` paths to the
         current configuration state.  The ``specs`` argument is a
         sequence that may contain absolute directory paths
@@ -61,18 +61,27 @@ def add_translation_dirs(self, *specs):
         translations defined later have precedence over translations defined
         earlier.
 
+        By default, consecutive calls to ``add_translation_dirs`` will add
+        directories to the start of the list. This means later calls to
+        ``add_translation_dirs`` will have their translations trumped by
+        earlier calls. If you explicitly need this call to trump an earlier
+        call then you may set ``override`` to ``True``.
+
         If multiple specs are provided in a single call to
         ``add_translation_dirs``, the directories will be inserted in the
         order they're provided (earlier items are trumped by later items).
 
-        .. warning::
+        .. versionchanged:: 1.8
 
-           Consecutive calls to ``add_translation_dirs`` will sort the
-           directories such that the later calls will add folders with
-           lower precedence than earlier calls.
+           The ``override`` parameter was added to allow a later call
+           to ``add_translation_dirs`` to override an earlier call, inserting
+           folders at the beginning of the translation directory list.
 
         """
         introspectables = []
+        override = kw.pop('override', False)
+        if kw:
+            raise TypeError('invalid keyword arguments: %s', sorted(kw.keys()))
 
         def register():
             directories = []
@@ -80,7 +89,7 @@ def register():
 
             # defer spec resolution until register to allow for asset
             # overrides to take place in an earlier config phase
-            for spec in specs[::-1]:  # reversed
+            for spec in specs:
                 # the trailing slash helps match asset overrides for folders
                 if not spec.endswith('/'):
                     spec += '/'
@@ -100,8 +109,11 @@ def register():
             if tdirs is None:
                 tdirs = []
                 self.registry.registerUtility(tdirs, ITranslationDirectories)
-            for directory in directories:
-                tdirs.insert(0, directory)
+            if override:
+                tdirs.extend(directories)
+            else:
+                for directory in reversed(directories):
+                    tdirs.insert(0, directory)
 
         self.action(None, register, introspectables=introspectables)
 


Test patch:
diff --git a/pyramid/tests/test_config/test_i18n.py b/pyramid/tests/test_config/test_i18n.py
--- a/pyramid/tests/test_config/test_i18n.py
+++ b/pyramid/tests/test_config/test_i18n.py
@@ -79,6 +79,23 @@ def test_add_translation_dirs_multiple_specs_multiple_calls(self):
         self.assertEqual(config.registry.getUtility(ITranslationDirectories),
                          [locale3, locale, locale2])
 
+    def test_add_translation_dirs_override_multiple_specs_multiple_calls(self):
+        from pyramid.interfaces import ITranslationDirectories
+        config = self._makeOne(autocommit=True)
+        config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale',
+                                    'pyramid.tests.pkgs.localeapp:locale2')
+        config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale3',
+                                    override=True)
+        self.assertEqual(config.registry.getUtility(ITranslationDirectories),
+                         [locale, locale2, locale3])
+
+    def test_add_translation_dirs_invalid_kwargs(self):
+        from pyramid.interfaces import ITranslationDirectories
+        config = self._makeOne(autocommit=True)
+        with self.assertRaises(TypeError):
+            config.add_translation_dirs('pyramid.tests.pkgs.localeapp:locale',
+                                        foo=1)
+
     def test_add_translation_dirs_abspath(self):
         from pyramid.interfaces import ITranslationDirectories
         config = self._makeOne(autocommit=True)