Repository: getnikola/nikola

Issue:
Incremental build seems not to be working
### Environment

**Python Version:**
Python 3.5.2

**Nikola Version:**
Nikola v8.0.1

**Operating System:**
Ubuntu 16.04

### Description:

If I run `nikola build` twice in a row, according to [this page](https://getnikola.com/features/fast.html) I would expect it to do nothing; and yet:
```
$ nikola build 2>&1 | wc -l
1910
```

It is less than a full rebuild:
```
$ nikola build -a 2>&1 | wc -l
4077
```
If I compare the actual output, it seems that the incremental rebuild doesn't do the `render_posts:cache/posts/...` tasks and the `render_taxonomies:output/...` for the `atom` and `xml` feeds.

Do I have some configuration or template in my site that makes Nikola misbehave? Are there some ways to debug this?

For the record, the source of my site is [here](https://gitlab.com/mardy/mardy.it).
Incremental build seems not to be working
### Environment

**Python Version:**
Python 3.5.2

**Nikola Version:**
Nikola v8.0.1

**Operating System:**
Ubuntu 16.04

### Description:

If I run `nikola build` twice in a row, according to [this page](https://getnikola.com/features/fast.html) I would expect it to do nothing; and yet:
```
$ nikola build 2>&1 | wc -l
1910
```

It is less than a full rebuild:
```
$ nikola build -a 2>&1 | wc -l
4077
```
If I compare the actual output, it seems that the incremental rebuild doesn't do the `render_posts:cache/posts/...` tasks and the `render_taxonomies:output/...` for the `atom` and `xml` feeds.

Do I have some configuration or template in my site that makes Nikola misbehave? Are there some ways to debug this?

For the record, the source of my site is [here](https://gitlab.com/mardy/mardy.it).


Discussion:
On my side, there seem to be some random rebuilds, but far less than 1910. (about 150 on first rebuild, 1 on second, 4 on third).
Sigh, these are always tricky.
Is there some way to have Nikola dump a dependency graph, or anything else that could help in understanding where the issue is?
`nikola dumpdb` can help (run multiple times, before and after building, compare outputs)
Yes, basically the "solution" is:

* build
* dumpdb (save)
* build (see what spurious changes are there)
* dumpdb
* see *why* each spurious change happened
* track down why nikola thinks each thing needed rebuilding
* some of the rebuilds will be because of other expected rebuilds
* There will be one or more "root" spurious rebuilds
* see why those happened
* figure out whether it's a config problem or a stupid bug Roberto left there 3 years ago
* surely it's roberto's fault
* tell roberto
I'm taking a look now :-)
Can't build it yet because of:

```
babel.core.UnknownLocaleError: unknown locale 'ia'
```

(workaround: use "ia": "en_GB" in locales)
Right, I have a locally patched babel; but if you remove `ia` from `conf.py` and also
```
rm posts/2019/01/looking-forward-to-your-comments.ia.md posts/2018/04/what-you-dont-hear-about-ukraine.ia.html pages/about.ia.md
```
you should be able to build the site and I've verified that the problem is still there.
One interesting (maybe) thing: if I rebuild it immediately, I get a small spurious rebuild. If I wait, the rebuild gets larger and larger.
And I think I know what it is.

<img width="1644" alt="screen shot 2019-02-05 at 12 53 51" src="https://user-images.githubusercontent.com/1579/52285516-3d556680-2945-11e9-9dde-1e7cdc32014b.png">

Templates in dependencies in jinja are not sorted.
Grrr that is *a* thing but is not *the* thing.
Ok, so the order of the file lists in deps doesn't matter (although it would be lovely if doit printed those sorted, I will take a look later). The problem is in the config_changed tagged as "generic_renderer". If I remove that one, the spurious rebuilds disappear. Further analysis ongoing.
The guilty line is this:

```
deps_dict['global'] = self.GLOBAL_CONTEXT
```

which is ... extremely suprising.
narrowed it down to self.GLOBAL_CONTEXT['momentjs_locales'] of all things
And I got it :-)

This is an **obscure** bug dudes!

* Because interlingua is not supported in momentjs
* And momentjs_locales is a defaultdict
* Depending on "random things" sometimes that dict would have an extra element for interlingua
* So, "sometimes" this would trigger a corner case in our CustomEncoder for JSON

One-liner fix coming up. 
On my side, there seem to be some random rebuilds, but far less than 1910. (about 150 on first rebuild, 1 on second, 4 on third).
Sigh, these are always tricky.
Is there some way to have Nikola dump a dependency graph, or anything else that could help in understanding where the issue is?
`nikola dumpdb` can help (run multiple times, before and after building, compare outputs)
Yes, basically the "solution" is:

* build
* dumpdb (save)
* build (see what spurious changes are there)
* dumpdb
* see *why* each spurious change happened
* track down why nikola thinks each thing needed rebuilding
* some of the rebuilds will be because of other expected rebuilds
* There will be one or more "root" spurious rebuilds
* see why those happened
* figure out whether it's a config problem or a stupid bug Roberto left there 3 years ago
* surely it's roberto's fault
* tell roberto
I'm taking a look now :-)
Can't build it yet because of:

```
babel.core.UnknownLocaleError: unknown locale 'ia'
```

(workaround: use "ia": "en_GB" in locales)
Right, I have a locally patched babel; but if you remove `ia` from `conf.py` and also
```
rm posts/2019/01/looking-forward-to-your-comments.ia.md posts/2018/04/what-you-dont-hear-about-ukraine.ia.html pages/about.ia.md
```
you should be able to build the site and I've verified that the problem is still there.
One interesting (maybe) thing: if I rebuild it immediately, I get a small spurious rebuild. If I wait, the rebuild gets larger and larger.
And I think I know what it is.

<img width="1644" alt="screen shot 2019-02-05 at 12 53 51" src="https://user-images.githubusercontent.com/1579/52285516-3d556680-2945-11e9-9dde-1e7cdc32014b.png">

Templates in dependencies in jinja are not sorted.
Grrr that is *a* thing but is not *the* thing.
Ok, so the order of the file lists in deps doesn't matter (although it would be lovely if doit printed those sorted, I will take a look later). The problem is in the config_changed tagged as "generic_renderer". If I remove that one, the spurious rebuilds disappear. Further analysis ongoing.
The guilty line is this:

```
deps_dict['global'] = self.GLOBAL_CONTEXT
```

which is ... extremely suprising.
narrowed it down to self.GLOBAL_CONTEXT['momentjs_locales'] of all things
And I got it :-)

This is an **obscure** bug dudes!

* Because interlingua is not supported in momentjs
* And momentjs_locales is a defaultdict
* Depending on "random things" sometimes that dict would have an extra element for interlingua
* So, "sometimes" this would trigger a corner case in our CustomEncoder for JSON

One-liner fix coming up. 

Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -171,61 +171,60 @@
         'sr_latin': 'sr_Latn',
     },
     'RTL_LANGUAGES': ('ar', 'fa', 'he', 'ur'),
-    'MOMENTJS_LOCALES': defaultdict(
-        str,
-        af='af',
-        ar='ar',
-        az='az',
-        bg='bg',
-        bn='bn',
-        bs='bs',
-        ca='ca',
-        cs='cs',
-        cz='cs',
-        da='da',
-        de='de',
-        el='el',
-        en='en',
-        eo='eo',
-        es='es',
-        et='et',
-        eu='eu',
-        fa='fa',
-        fi='fi',
-        fr='fr',
-        gl='gl',
-        hi='hi',
-        he='he',
-        hr='hr',
-        hu='hu',
-        id='id',
-        it='it',
-        ja='ja',
-        ko='ko',
-        lt='lt',
-        ml='ml',
-        nb='nb',
-        nl='nl',
-        pa='pa-in',
-        pl='pl',
-        pt='pt',
-        pt_br='pt-br',
-        ru='ru',
-        sk='sk',
-        sl='sl',
-        sq='sq',
-        sr='sr-cyrl',
-        sr_latin='sr',
-        sv='sv',
-        te='te',
-        tr='tr',
-        th='th',
-        uk='uk',
-        ur='ur',
-        vi='vi',
-        zh_cn='zh-cn',
-        zh_tw='zh-tw'
-    ),
+    'MOMENTJS_LOCALES': {
+        'af': 'af',
+        'ar': 'ar',
+        'az': 'az',
+        'bg': 'bg',
+        'bn': 'bn',
+        'bs': 'bs',
+        'ca': 'ca',
+        'cs': 'cs',
+        'cz': 'cs',
+        'da': 'da',
+        'de': 'de',
+        'el': 'el',
+        'en': 'en',
+        'eo': 'eo',
+        'es': 'es',
+        'et': 'et',
+        'eu': 'eu',
+        'fa': 'fa',
+        'fi': 'fi',
+        'fr': 'fr',
+        'gl': 'gl',
+        'hi': 'hi',
+        'he': 'he',
+        'hr': 'hr',
+        'hu': 'hu',
+        'id': 'id',
+        'it': 'it',
+        'ja': 'ja',
+        'ko': 'ko',
+        'lt': 'lt',
+        'ml': 'ml',
+        'nb': 'nb',
+        'nl': 'nl',
+        'pa': 'pa-in',
+        'pl': 'pl',
+        'pt': 'pt',
+        'pt_br': 'pt-br',
+        'ru': 'ru',
+        'sk': 'sk',
+        'sl': 'sl',
+        'sq': 'sq',
+        'sr': 'sr-cyrl',
+        'sr_latin': 'sr',
+        'sv': 'sv',
+        'te': 'te',
+        'tr': 'tr',
+        'th': 'th',
+        'uk': 'uk',
+        'ur': 'ur',
+        'vi': 'vi',
+        'zh_cn': 'zh-cn',
+        'zh_tw': 'zh-tw'
+    },
     'PYPHEN_LOCALES': {
         'af': 'af',
         'bg': 'bg',
@@ -1169,6 +1168,10 @@ def _set_global_context_from_config(self):
         self._GLOBAL_CONTEXT['date_fanciness'] = self.config.get('DATE_FANCINESS')
         self._GLOBAL_CONTEXT['js_date_format'] = self.config.get('JS_DATE_FORMAT')
         self._GLOBAL_CONTEXT['momentjs_locales'] = LEGAL_VALUES['MOMENTJS_LOCALES']
+        # Patch missing locales into momentjs defaulting to English (Issue #3216)
+        for l in self._GLOBAL_CONTEXT['translations']:
+            if l not in self._GLOBAL_CONTEXT['momentjs_locales']:
+                self._GLOBAL_CONTEXT['momentjs_locales'][l] = ""
         self._GLOBAL_CONTEXT['hidden_tags'] = self.config.get('HIDDEN_TAGS')
         self._GLOBAL_CONTEXT['hidden_categories'] = self.config.get('HIDDEN_CATEGORIES')
         self._GLOBAL_CONTEXT['hidden_authors'] = self.config.get('HIDDEN_AUTHORS')


Test patch: