Repository: getnikola/nikola

Issue:
Problems when using parallel processing with threads and multiple languages
There's a problem with Nikola if your site has more than one language and if you use `nikola run -n4 -P thread` (run four tasks in parallel using threads instead of forking, see here: http://pydoit.org/cmd_run.html#parallel-execution). Namely, when rendering a page/..., `LocaleBorg` is storing the current language in global storage which is then used by `messages("...")` and `formatted_date()` (among others). When two tasks run in parallel for two different languages, one will overwrite the `LocaleBorg` setup of the other, and with high probability one of the output files ends up with wrong strings or wrongly formatted dates.
(When using forking, this should not be a problem, but that doesn't always work -- it might be problematic under Windows, and totally breaks for the `earlytask_impl` branch.)

I don't know how easy/complicated fixing this would be. There are two aspects:
1. If we store the current language not in a global state, but a thread-local state, at least `messages("...")` will always use the correct language.
2. This might not help for `formatted_date` and `get_month_name`, since they have to set the correct locale and then call some Python system function -- Python might not store the locale thread-local, but global. (Or does it store it thread-local? If that is guaranteed for both Python 2 and 3, using thread-local storage would be enough.) But this could be solved by using a global lock which avoids that `formatted_date` and `get_month_name` run in parallel.

What do yout think? While standard Nikola not run in parallel mode is totally not affected by this (and after all we don't advertise that you can run several tasks in parallel), I think that making Nikola more thread-safe is always a good idea.



Discussion:
3. drop parallelization

Since it seems to be only localborg-related we could drop parallelization if there's more than 1 language.

Automatically dropping parallelization or at least printing a strong warning would be good.

Hmm, how about adding a optional argument `lang` to the `formatted_date`s, which if set changes the locale and later resets it to the previous value?
This would also allow `formatted_date` handlers to get the correct language when it is specified, and not having to rely that `LocaleBorg().current_lang` is correct. I think I could modify my theme to be on the thread-safe side with this change (since all locale-dependent date rendering is done via handlers anyway, in my case). And this is a rather small and simple change, as opposed to adding thread-local storage etc.

@felixfontein there are a lot of things where lang is optional and guessed from LocaleBorg (most templates, for one thing)

@felixfontein just curious, you see this problem both with threads and multiprocessing? 

I now looked at it in more detail. For `tags`, `prev_post`, `next_post` (of `Post`) and `formatted_date` (of `Post` and `LocaleBorg`) it is not possible to provide the current language; for all other calls, one can provide `lang`.
I can modify my templates to provide `lang` whenever needed (I already did so for all `messages()` calls in them), but of course I can only do that if I _can_ specify `lang`.

Do you mind if I start a WIP branch where I try to allow specifying `lang` everywhere (mostly for the four above properties/functions) in a backwards-compatible way? Or possibly even modify the default themes to provide `lang` as often as possible, so that the dependency on `LocaleBorg().current_lang` is minimized? Or does this has no chance of being included in Nikola?

(Rebuilding all `.html` files of my blog single-threaded currently takes almost 12 minutes. I'd be really happy to have some parallelization back, even if I have to invest quite some work to make my theme thread-safe.)

@felixfontein are you using -n N ? In that case those are not threads.

The `earlytask_impl` branch has to use threads (instead of forking) as pickling the task creation functions fails.

If you try that using master and processes, does it work?

What do you mean with master and processes?

Using the master branch, not earlytasks, and parallelizing using processes, not threads.

Hmm, I have to try that another time. It'll take some preparations to be able to efficiently check for failures. (I don't want to check all > 5000 HTML files manually for wrong strings etc.) Maybe I'll manage to do that tomorrow...

Thanks, if you can do that it would really help me a lot :-)

Ok, I did two runs with master branch and processes instead threads, and it seemed to have worked fine. (Output was binary identical to single threaded.) Takes a bit less than 6 minutes. (I used `-n 4` and `-n 8`, but this laptop only has two cores anyway.) When using the master branch with threads instead of processes, it took a bit longer (roughtly 8 1/2 minutes) and produced incorrect output (461 lines of diff output).

@felixfontein so, it seems to me it's just a matter of making LocaleBorg thread-local. It's a fix in a single place, so it should not be horribly hard to do?

Yes, using thread-local storage is a good way. But we also need to make sure that the locale is always set before calling date-related functions, and that no other date-related function in executed at the same time. But since they are all in `LocaleBorg` as well (I hope?), this shouldn't be a problem either.

Exactly. Locale is always set, and within a thread everything is sequential, so each task sets the locale, does its thing and dies. So if it's thread-local it _should_ work.

Except that `formatted_date` currently doesn't ensure that the locale is set (again). But that's trivial to change ;-)

Should I start working on that?

By all means! Thanks!


Resolving patch:
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -1074,21 +1074,35 @@ def initialize(cls, locales, initial_lang):
             encodings[lang] = encoding
 
         cls.encodings = encodings
-        cls.__shared_state['current_lang'] = initial_lang
+        cls.__initial_lang = initial_lang
         cls.initialized = True
 
+    def __get_shared_state(self):
+        if not self.initialized:
+            raise LocaleBorgUninitializedException()
+        shared_state = getattr(self.__thread_local, 'shared_state', None)
+        if shared_state is None:
+            shared_state = {'current_lang': self.__initial_lang}
+            self.__thread_local.shared_state = shared_state
+        return shared_state
+
     @classmethod
     def reset(cls):
         """Reset LocaleBorg.
 
         Used in testing to prevent leaking state between tests.
         """
+        import threading
+        cls.__thread_local = threading.local()
+        cls.__thread_lock = threading.Lock()
+
         cls.locales = {}
         cls.encodings = {}
-        cls.__shared_state = {'current_lang': None}
         cls.initialized = False
         cls.month_name_handlers = []
         cls.formatted_date_handlers = []
+        cls.thread_local = None
+        cls.thread_lock = None
 
     @classmethod
     def add_handler(cls, month_name_handler=None, formatted_date_handler=None):
@@ -1115,7 +1129,16 @@ def __init__(self):
         """Initialize."""
         if not self.initialized:
             raise LocaleBorgUninitializedException()
-        self.__dict__ = self.__shared_state
+
+    @property
+    def current_lang(self):
+        """Return the current language."""
+        return self.__get_shared_state()['current_lang']
+
+    def __set_locale(self, lang):
+        """Set the locale for language lang without updating current_lang."""
+        locale_n = self.locales[lang]
+        locale.setlocale(locale.LC_ALL, locale_n)
 
     def set_locale(self, lang):
         """Set the locale for language lang, returns an empty string.
@@ -1124,58 +1147,64 @@ def set_locale(self, lang):
         in windows that cannot be guaranted.
         In either case, the locale encoding is available in cls.encodings[lang]
         """
-        # intentional non try-except: templates must ask locales with a lang,
-        # let the code explode here and not hide the point of failure
-        # Also, not guarded with an if lang==current_lang because calendar may
-        # put that out of sync
-        locale_n = self.locales[lang]
-        self.__shared_state['current_lang'] = lang
-        locale.setlocale(locale.LC_ALL, locale_n)
-        return ''
+        with self.__thread_lock:
+            # intentional non try-except: templates must ask locales with a lang,
+            # let the code explode here and not hide the point of failure
+            # Also, not guarded with an if lang==current_lang because calendar may
+            # put that out of sync
+            self.__set_locale(lang)
+            self.__get_shared_state()['current_lang'] = lang
+            return ''
 
     def get_month_name(self, month_no, lang):
         """Return localized month name in an unicode string."""
-        for handler in self.month_name_handlers:
-            res = handler(month_no, lang)
-            if res is not None:
-                return res
-        if sys.version_info[0] == 3:  # Python 3
-            with calendar.different_locale(self.locales[lang]):
-                s = calendar.month_name[month_no]
-            # for py3 s is unicode
-        else:  # Python 2
-            with calendar.TimeEncoding(self.locales[lang]):
-                s = calendar.month_name[month_no]
-            enc = self.encodings[lang]
-            if not enc:
-                enc = 'UTF-8'
-
-            s = s.decode(enc)
-        # paranoid about calendar ending in the wrong locale (windows)
-        self.set_locale(self.current_lang)
-        return s
+        # For thread-safety
+        with self.__thread_lock:
+            for handler in self.month_name_handlers:
+                res = handler(month_no, lang)
+                if res is not None:
+                    return res
+            if sys.version_info[0] == 3:  # Python 3
+                with calendar.different_locale(self.locales[lang]):
+                    s = calendar.month_name[month_no]
+                # for py3 s is unicode
+            else:  # Python 2
+                with calendar.TimeEncoding(self.locales[lang]):
+                    s = calendar.month_name[month_no]
+                enc = self.encodings[lang]
+                if not enc:
+                    enc = 'UTF-8'
+
+                s = s.decode(enc)
+            # paranoid about calendar ending in the wrong locale (windows)
+            self.__set_locale(self.current_lang)
+            return s
 
     def formatted_date(self, date_format, date):
         """Return the formatted date as unicode."""
-        fmt_date = None
-        # First check handlers
-        for handler in self.formatted_date_handlers:
-            fmt_date = handler(date_format, date, self.__shared_state['current_lang'])
-            if fmt_date is not None:
-                break
-        # If no handler was able to format the date, ask Python
-        if fmt_date is None:
-            if date_format == 'webiso':
-                # Formatted after RFC 3339 (web ISO 8501 profile) with Zulu
-                # zone desgignator for times in UTC and no microsecond precision.
-                fmt_date = date.replace(microsecond=0).isoformat().replace('+00:00', 'Z')
-            else:
-                fmt_date = date.strftime(date_format)
+        with self.__thread_lock:
+            current_lang = self.current_lang
+            # For thread-safety
+            self.__set_locale(current_lang)
+            fmt_date = None
+            # First check handlers
+            for handler in self.formatted_date_handlers:
+                fmt_date = handler(date_format, date, current_lang)
+                if fmt_date is not None:
+                    break
+            # If no handler was able to format the date, ask Python
+            if fmt_date is None:
+                if date_format == 'webiso':
+                    # Formatted after RFC 3339 (web ISO 8501 profile) with Zulu
+                    # zone desgignator for times in UTC and no microsecond precision.
+                    fmt_date = date.replace(microsecond=0).isoformat().replace('+00:00', 'Z')
+                else:
+                    fmt_date = date.strftime(date_format)
 
-        # Issue #383, this changes from py2 to py3
-        if isinstance(fmt_date, bytes_str):
-            fmt_date = fmt_date.decode('utf8')
-        return fmt_date
+            # Issue #383, this changes from py2 to py3
+            if isinstance(fmt_date, bytes_str):
+                fmt_date = fmt_date.decode('utf8')
+            return fmt_date
 
 
 class ExtendedRSS2(rss.RSS2):


Test patch:
diff --git a/tests/test_locale.py b/tests/test_locale.py
--- a/tests/test_locale.py
+++ b/tests/test_locale.py
@@ -214,8 +214,8 @@ def test_services_ensure_initialization(self):
     def test_services_reject_dumb_wrong_call(self):
         lang_11, loc_11 = LocaleSupportInTesting.langlocales['default']
         nikola.utils.LocaleBorg.reset()
+        self.assertRaises(Exception, nikola.utils.LocaleBorg)
         self.assertRaises(Exception, nikola.utils.LocaleBorg.set_locale, lang_11)
-        self.assertRaises(Exception, getattr, nikola.utils.LocaleBorg, 'current_lang')
 
     def test_set_locale_raises_on_invalid_lang(self):
         lang_11, loc_11 = LocaleSupportInTesting.langlocales['default']