Repository: getnikola/nikola

Issue:
Incremental build seems not to be working
### Environment

**Python Version:**
Python 3.5.2

**Nikola Version:**
Nikola v8.0.1

**Operating System:**
Ubuntu 16.04

### Description:

If I run `nikola build` twice in a row, according to [this page](https://getnikola.com/features/fast.html) I would expect it to do nothing; and yet:
```
$ nikola build 2>&1 | wc -l
1910
```

It is less than a full rebuild:
```
$ nikola build -a 2>&1 | wc -l
4077
```
If I compare the actual output, it seems that the incremental rebuild doesn't do the `render_posts:cache/posts/...` tasks and the `render_taxonomies:output/...` for the `atom` and `xml` feeds.

Do I have some configuration or template in my site that makes Nikola misbehave? Are there some ways to debug this?

For the record, the source of my site is [here](https://gitlab.com/mardy/mardy.it).


Discussion:
On my side, there seem to be some random rebuilds, but far less than 1910. (about 150 on first rebuild, 1 on second, 4 on third).
Sigh, these are always tricky.
Is there some way to have Nikola dump a dependency graph, or anything else that could help in understanding where the issue is?
`nikola dumpdb` can help (run multiple times, before and after building, compare outputs)
Yes, basically the "solution" is:

* build
* dumpdb (save)
* build (see what spurious changes are there)
* dumpdb
* see *why* each spurious change happened
* track down why nikola thinks each thing needed rebuilding
* some of the rebuilds will be because of other expected rebuilds
* There will be one or more "root" spurious rebuilds
* see why those happened
* figure out whether it's a config problem or a stupid bug Roberto left there 3 years ago
* surely it's roberto's fault
* tell roberto
I'm taking a look now :-)
Can't build it yet because of:

```
babel.core.UnknownLocaleError: unknown locale 'ia'
```

(workaround: use "ia": "en_GB" in locales)
Right, I have a locally patched babel; but if you remove `ia` from `conf.py` and also
```
rm posts/2019/01/looking-forward-to-your-comments.ia.md posts/2018/04/what-you-dont-hear-about-ukraine.ia.html pages/about.ia.md
```
you should be able to build the site and I've verified that the problem is still there.
One interesting (maybe) thing: if I rebuild it immediately, I get a small spurious rebuild. If I wait, the rebuild gets larger and larger.
And I think I know what it is.

<img width="1644" alt="screen shot 2019-02-05 at 12 53 51" src="https://user-images.githubusercontent.com/1579/52285516-3d556680-2945-11e9-9dde-1e7cdc32014b.png">

Templates in dependencies in jinja are not sorted.
Grrr that is *a* thing but is not *the* thing.
Ok, so the order of the file lists in deps doesn't matter (although it would be lovely if doit printed those sorted, I will take a look later). The problem is in the config_changed tagged as "generic_renderer". If I remove that one, the spurious rebuilds disappear. Further analysis ongoing.
The guilty line is this:

```
deps_dict['global'] = self.GLOBAL_CONTEXT
```

which is ... extremely suprising.
narrowed it down to self.GLOBAL_CONTEXT['momentjs_locales'] of all things
And I got it :-)

This is an **obscure** bug dudes!

* Because interlingua is not supported in momentjs
* And momentjs_locales is a defaultdict
* Depending on "random things" sometimes that dict would have an extra element for interlingua
* So, "sometimes" this would trigger a corner case in our CustomEncoder for JSON

One-liner fix coming up. 
BTW @mardy nice to see a fellow former-canonicaler around :-)
Ok, the fix is wrong. Sigh.
> BTW @mardy nice to see a fellow former-canonicaler around :-)

Yes, I noticed that your name was familiar! Then linked-in did the rest :-)

Note that this bug, as I wrote in [a previous comment](https://github.com/getnikola/nikola/issues/3216#issuecomment-460685025), this happens even if I completely remove interlingua. Though, maybe, it might still be referenced in some cached file, dunno.
@mardy Could you check if the proposed branch fixes it for you?
> @mardy Could you check if the proposed branch fixes it for you?

It doesn't seem to fix it, no :-(
@mardy grmbl, will look harder today
@mardy I can't reproduce it anymore with that PR.
Mmm... I'll try again tonight, maybe I did something wrong in applying it.
No, @ralsina, it looks like it's not working here. I deleted the `.doit.db` file, and the `cache` and `__pycache__` directories, but nothing changes.
The `default()` function is called many times, but that line you added (`self.encode(dict(obj))`) is never called (I added a print statement right before it)
@mardy I think you are testing old code, that is no longer there in the last version of the PR, see https://github.com/getnikola/nikola/pull/3218
Ah! Indeed, I was on the wrong commit. Now I've checked out e8916c85773d9fde2b23798fd9ea7ccfa970d984, but I still don't see an improvement:

    $ nikola build 2>&1 | wc -l
    1926

Can you tell me what is the expected output?
I also tried adding `ia: 'ia'` to the `MOMENTJS_LOCALES` dictionary, but it didn't help.
By the way, the latest `python-babel` from git supports the interlingua locale. You can follow the instruction from [here](https://github.com/python-babel/babel/blob/master/docs/installation.rst) (in the "Living on the edge" section) to install it in your virtualenv.
With Python 3.7, macOS, the branch, and latest Babel, I cannot reproduce. An important thing to notice is that Python 3.5 has random dict ordering, so there might still be something there…
Confirmed random rebuilds still happen with Python 3.5.
And I installed python3.7 in my home (I'm using Ubuntu 16.04), and I can confirm that there are no random rebuilds in 3.7. Thanks @ralsina for the fix, and @Kwpolska for the intuition of using 3.7! :-)

Resolving patch:
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -544,6 +544,8 @@ def default(self, obj):
         except TypeError:
             if isinstance(obj, (set, frozenset)):
                 return self.encode(sorted(list(obj)))
+            elif isinstance(obj, TranslatableSetting):
+                s = json.dumps(obj._inp, cls=CustomEncoder, sort_keys=True)
             else:
                 s = repr(obj).split('0x', 1)[0]
             return s
@@ -559,6 +561,29 @@ def __init__(self, config, identifier=None):
         if identifier is not None:
             self.identifier += ':' + identifier
 
+    # DEBUG (for unexpected rebuilds)
+    @classmethod
+    def _write_into_debug_db(cls, digest: str, data: str) -> None:  # pragma: no cover
+        """Write full values of config_changed into a sqlite3 database."""
+        import sqlite3
+        try:
+            cls.debug_db_cursor
+        except AttributeError:
+            cls.debug_db_conn = sqlite3.connect("cc_debug.sqlite3")
+            cls.debug_db_id = datetime.datetime.now().isoformat()
+            cls.debug_db_cursor = cls.debug_db_conn.cursor()
+            cls.debug_db_cursor.execute("""
+            CREATE TABLE IF NOT EXISTS hashes (hash CHARACTER(32) PRIMARY KEY, json_data TEXT);
+            """)
+            cls.debug_db_conn.commit()
+
+        try:
+            cls.debug_db_cursor.execute("INSERT INTO hashes (hash, json_data) VALUES (?, ?);", (digest, data))
+            cls.debug_db_conn.commit()
+        except sqlite3.IntegrityError:
+            # ON CONFLICT DO NOTHING, except Ubuntu 16.04’s sqlite3 is too ancient for this
+            cls.debug_db_conn.rollback()
+
     def _calc_digest(self):
         """Calculate a config_changed digest."""
         if isinstance(self.config, str):
@@ -570,9 +595,14 @@ def _calc_digest(self):
             else:
                 byte_data = data
             digest = hashlib.md5(byte_data).hexdigest()
+
+            # DEBUG (for unexpected rebuilds)
+            # self._write_into_debug_db(digest, data)
+            # Alternative (without database):
             # LOGGER.debug('{{"{0}": {1}}}'.format(digest, byte_data))
             # Humanized format:
             # LOGGER.debug('[Digest {0} for {2}]\n{1}\n[Digest {0} for {2}]'.format(digest, byte_data, self.identifier))
+
             return digest
         else:
             raise Exception('Invalid type of config_changed parameter -- got '
diff --git a/scripts/debug_rebuilds/step1_build_and_dumpdb.py b/scripts/debug_rebuilds/step1_build_and_dumpdb.py
new file mode 100755
--- /dev/null
+++ b/scripts/debug_rebuilds/step1_build_and_dumpdb.py
@@ -0,0 +1,50 @@
+import dbm
+import json
+import subprocess
+import sys
+
+
+def dbm_iter(db):
+    # try dictionary interface - ok in python2 and dumbdb
+    try:
+        return db.items()
+    except Exception:
+        # try firstkey/nextkey - ok for py3 dbm.gnu
+        def iter_gdbm(db):
+            k = db.firstkey()
+            while k is not None:
+                yield k, db[k]
+                k = db.nextkey(k)
+        return iter_gdbm(db)
+
+
+def dumpdb():
+    with dbm.open('.doit.db') as data:
+        return {key: json.loads(value_str.decode('utf-8'))
+                for key, value_str in dbm_iter(data)}
+
+
+print_ = print
+
+
+def print(*args, **kwargs):
+    print_(*args, file=sys.stdout)
+    sys.stdout.flush()
+
+
+print("==> Removing stuff...")
+subprocess.call(['rm', '-rf', '.doit.db', 'output', 'cache', 'cc_debug.sqlite3'])
+print("==> Running first build...")
+subprocess.call(['nikola', 'build'])
+print("==> Fetching database...")
+first = dumpdb()
+print("==> Running second build...")
+subprocess.call(['nikola', 'build'])
+print("==> Fetching database...")
+second = dumpdb()
+print("==> Saving dumps...")
+with open('first_dump.py', 'w', encoding='utf-8') as fh:
+    fh.write(repr(first))
+
+with open('second_dump.py', 'w', encoding='utf-8') as fh:
+    fh.write(repr(second))
diff --git a/scripts/debug_rebuilds/step2_analyze_py_files.py b/scripts/debug_rebuilds/step2_analyze_py_files.py
new file mode 100755
--- /dev/null
+++ b/scripts/debug_rebuilds/step2_analyze_py_files.py
@@ -0,0 +1,58 @@
+import sqlite3
+import sys
+print_ = print
+
+
+def print(*args, **kwargs):
+    print_(*args, file=sys.stdout)
+    sys.stdout.flush()
+
+
+with open('first_dump.py', 'r', encoding='utf-8') as fh:
+    first = eval(fh.read())
+
+with open('second_dump.py', 'r', encoding='utf-8') as fh:
+    second = eval(fh.read())
+
+if len(first) != len(second):
+    print(" [!] Databases differ in size.")
+    for k in first:
+        if k not in second:
+            print("    Item", k, "not found in second database.")
+    for k in second:
+        if k not in first:
+            print("    Item", k, "not found in first database.")
+
+conn = sqlite3.connect("cc_debug.sqlite3")
+
+
+def get_from_db(value):
+    cursor = conn.cursor()
+    try:
+        cursor.execute("SELECT json_data FROM hashes WHERE hash = ?", (value,))
+        return cursor.fetchone()[0]
+    except Exception:
+        print(" [!] Cannot find", value, "in database.")
+        return None
+
+
+if first == second:
+    print("==> Both files are identical.")
+    exit(0)
+
+VAL_KEY = '_values_:'  # yes, ends with a colon
+for k in first:
+    fk, sk = first[k], second[k]
+    try:
+        first_values, second_values = fk[VAL_KEY], sk[VAL_KEY]
+    except KeyError:
+        print(" [!] Values not found for,", k)
+        continue
+
+    if first_values != second_values:
+        print(" -> Difference:", k)
+        for vk in first_values:
+            fv, sv = first_values[vk], second_values[vk]
+            if fv != sv:
+                print("    first :", fv, get_from_db(fv))
+                print("    second:", sv, get_from_db(sv))


Test patch: