Repository: getnikola/nikola

Issue:
Iterative build does not work when using SHOW_UNTRANSLATED_POSTS
<!--
Before creating an issue:
* make sure you are using an up-to-date version of Nikola
* search for existing issues that might be related

If you are reporting a bug:
* provide information about your environment (below)
* include all the output you get, and any other information related to your problem

Nikola v7.6.4, as provided by Ubuntu, is NOT SUPPORTED.
If you are using this version, you should upgrade: https://getnikola.com/getting-started.html
-->

### Environment (if reporting a bug)

**Python Version:** Python 3.6.5

**Nikola Version:** Nikola v8.0.0b2

**Operating System:** Linux 4.17.3-1-ARCH #1 SMP PREEMPT Tue Jun 26 04:42:36 UTC 2018 x86_64 GNU/Linux


### Description:
`nikola build` rebuild the output when a markdown post has changed, but the change does not appear in the output. The stdout shows that the corresponding html is being rebuild, but the output content does not change (verified by having the same md5sum on the input and on the output).

`nikola clean; nikola build` works as expected.

### Further analysis

It should be configuration dependant, because I failed to reproduce it when creating a new test site from scratch.

More testing shows that `SHOW_UNTRANSLATED_POSTS` may be the culprit, as shown by this little script reproducing the problem:

```shell
nikola init -q test$$
cd test$$
sed -i -e 's/# "es": ".\/es",/"fr": ".\/fr",/' \
   -e 's/# \(SHOW_UNTRANSLATED_POSTS\)/\1/' conf.py
nikola new_post -f markdown -t "Test post"
mv posts/test-post.md posts/test-post.fr.md
nikola build
md5sum posts/test-post.fr.md
md5sum output/posts/test-post/index.html
echo "OOOO" >> posts/test-post.fr.md
nikola build
md5sum posts/test-post.fr.md
md5sum output/posts/test-post/index.html
```
The output mds5sum does not differ where it should have.




Discussion:
For completeness, here follows the test script output:
```
[2018-07-05T08:37:19Z] INFO: init: Created empty site at test13465.
Scanning posts........done!
[2018-07-05T08:37:20Z] INFO: new_post: Your post's text is at: posts/test-post.md
Creating New Post
-----------------

Title: Test post
Scanning posts........done!
.  render_taxonomies:output/archive.html
.  render_taxonomies:output/categories/index.html
.  render_taxonomies:output/fr/archive.html
.  render_taxonomies:output/fr/categories/index.html
.  render_posts:timeline_changes
.  render_posts:cache/posts/test-post.html
.  render_posts:cache/posts/test-post.fr.html
.  render_listings:output/listings/index.html
.  render_galleries:output/fr/galleries
.  render_galleries:output/galleries/index.html
.  render_galleries:output/galleries/rss.xml
.  render_galleries:output/fr/galleries/index.html
.  render_galleries:output/fr/galleries/rss.xml
.  copy_assets:output/assets/css/bootblog.css
.  copy_assets:output/assets/js/bootstrap.min.js
.  copy_assets:output/assets/js/popper.min.js
.  copy_assets:output/assets/js/jquery.min.js
.  copy_assets:output/assets/css/bootstrap.min.css
.  copy_assets:output/assets/css/theme.css
.  copy_assets:output/assets/js/gallery.min.js
.  copy_assets:output/assets/js/moment-with-locales.min.js
.  copy_assets:output/assets/js/baguetteBox.min.js
.  copy_assets:output/assets/js/gallery.js
.  copy_assets:output/assets/js/justified-layout.min.js
.  copy_assets:output/assets/js/html5shiv-printshiv.min.js
.  copy_assets:output/assets/js/fancydates.js
.  copy_assets:output/assets/js/html5.js
.  copy_assets:output/assets/js/fancydates.min.js
.  copy_assets:output/assets/xml/rss.xsl
.  copy_assets:output/assets/xml/atom.xsl
.  copy_assets:output/assets/css/nikola_ipython.css
.  copy_assets:output/assets/css/html4css1.css
.  copy_assets:output/assets/css/rst.css
.  copy_assets:output/assets/css/rst_base.css
.  copy_assets:output/assets/css/baguetteBox.min.css
.  copy_assets:output/assets/css/ipython.min.css
.  copy_assets:output/assets/css/nikola_rst.css
.  copy_assets:output/assets/css/code.css
.  render_sources:output/posts/test-post/index.md
.  render_sources:output/fr/posts/test-post/index.md
.  render_taxonomies:output/index.html
.  render_taxonomies:output/2018/index.html
.  render_taxonomies:output/fr/index.html
.  render_taxonomies:output/fr/2018/index.html
.  render_taxonomies:output/fr/rss.xml
.  render_pages:output/fr/posts/test-post/index.html
.  render_taxonomies:output/rss.xml
.  render_pages:output/posts/test-post/index.html
.  create_bundles:output/assets/css/all-nocdn.css
.  create_bundles:output/assets/css/all.css
.  create_bundles:output/assets/js/all-nocdn.js
.  create_bundles:output/assets/js/all.js
.  sitemap:output/sitemap.xml
.  sitemap:output/sitemapindex.xml
.  robots_file:output/robots.txt
6752ffb95e5becfc79798354bb3a2ec9  posts/test-post.fr.md
bfd8fb2d9256e28e436adb5c9beb0f65  output/posts/test-post/index.html
Scanning posts........done!
.  render_galleries:output/galleries
.  render_sources:output/posts/test-post/index.md
.  render_sources:output/fr/posts/test-post/index.md
.  render_posts:timeline_changes
.  render_taxonomies:output/index.html
.  render_taxonomies:output/fr/index.html
.  render_taxonomies:output/2018/index.html
.  render_taxonomies:output/fr/2018/index.html
.  render_taxonomies:output/rss.xml
.  render_pages:output/fr/posts/test-post/index.html
.  render_pages:output/posts/test-post/index.html
.  render_taxonomies:output/fr/rss.xml
.  sitemap:output/sitemap.xml
41436a785a3a24c815c340ebf134ace0  posts/test-post.fr.md
bfd8fb2d9256e28e436adb5c9beb0f65  output/posts/test-post/index.html
```

Can you check what the actual difference is between those two files?

On Thu, Jul 5, 2018 at 5:38 AM matclab <notifications@github.com> wrote:

> For completeness, here follows the test script output:
>
> [2018-07-05T08:37:19Z] INFO: init: Created empty site at test13465.
> Scanning posts........done!
> [2018-07-05T08:37:20Z] INFO: new_post: Your post's text is at: posts/test-post.md
> Creating New Post
> -----------------
>
> Title: Test post
> Scanning posts........done!
> .  render_taxonomies:output/archive.html
> .  render_taxonomies:output/categories/index.html
> .  render_taxonomies:output/fr/archive.html
> .  render_taxonomies:output/fr/categories/index.html
> .  render_posts:timeline_changes
> .  render_posts:cache/posts/test-post.html
> .  render_posts:cache/posts/test-post.fr.html
> .  render_listings:output/listings/index.html
> .  render_galleries:output/fr/galleries
> .  render_galleries:output/galleries/index.html
> .  render_galleries:output/galleries/rss.xml
> .  render_galleries:output/fr/galleries/index.html
> .  render_galleries:output/fr/galleries/rss.xml
> .  copy_assets:output/assets/css/bootblog.css
> .  copy_assets:output/assets/js/bootstrap.min.js
> .  copy_assets:output/assets/js/popper.min.js
> .  copy_assets:output/assets/js/jquery.min.js
> .  copy_assets:output/assets/css/bootstrap.min.css
> .  copy_assets:output/assets/css/theme.css
> .  copy_assets:output/assets/js/gallery.min.js
> .  copy_assets:output/assets/js/moment-with-locales.min.js
> .  copy_assets:output/assets/js/baguetteBox.min.js
> .  copy_assets:output/assets/js/gallery.js
> .  copy_assets:output/assets/js/justified-layout.min.js
> .  copy_assets:output/assets/js/html5shiv-printshiv.min.js
> .  copy_assets:output/assets/js/fancydates.js
> .  copy_assets:output/assets/js/html5.js
> .  copy_assets:output/assets/js/fancydates.min.js
> .  copy_assets:output/assets/xml/rss.xsl
> .  copy_assets:output/assets/xml/atom.xsl
> .  copy_assets:output/assets/css/nikola_ipython.css
> .  copy_assets:output/assets/css/html4css1.css
> .  copy_assets:output/assets/css/rst.css
> .  copy_assets:output/assets/css/rst_base.css
> .  copy_assets:output/assets/css/baguetteBox.min.css
> .  copy_assets:output/assets/css/ipython.min.css
> .  copy_assets:output/assets/css/nikola_rst.css
> .  copy_assets:output/assets/css/code.css
> .  render_sources:output/posts/test-post/index.md
> .  render_sources:output/fr/posts/test-post/index.md
> .  render_taxonomies:output/index.html
> .  render_taxonomies:output/2018/index.html
> .  render_taxonomies:output/fr/index.html
> .  render_taxonomies:output/fr/2018/index.html
> .  render_taxonomies:output/fr/rss.xml
> .  render_pages:output/fr/posts/test-post/index.html
> .  render_taxonomies:output/rss.xml
> .  render_pages:output/posts/test-post/index.html
> .  create_bundles:output/assets/css/all-nocdn.css
> .  create_bundles:output/assets/css/all.css
> .  create_bundles:output/assets/js/all-nocdn.js
> .  create_bundles:output/assets/js/all.js
> .  sitemap:output/sitemap.xml
> .  sitemap:output/sitemapindex.xml
> .  robots_file:output/robots.txt
> 6752ffb95e5becfc79798354bb3a2ec9  posts/test-post.fr.md
> bfd8fb2d9256e28e436adb5c9beb0f65  output/posts/test-post/index.html
> Scanning posts........done!
> .  render_galleries:output/galleries
> .  render_sources:output/posts/test-post/index.md
> .  render_sources:output/fr/posts/test-post/index.md
> .  render_posts:timeline_changes
> .  render_taxonomies:output/index.html
> .  render_taxonomies:output/fr/index.html
> .  render_taxonomies:output/2018/index.html
> .  render_taxonomies:output/fr/2018/index.html
> .  render_taxonomies:output/rss.xml
> .  render_pages:output/fr/posts/test-post/index.html
> .  render_pages:output/posts/test-post/index.html
> .  render_taxonomies:output/fr/rss.xml
> .  sitemap:output/sitemap.xml
> 41436a785a3a24c815c340ebf134ace0  posts/test-post.fr.md
> bfd8fb2d9256e28e436adb5c9beb0f65  output/posts/test-post/index.html
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/getnikola/nikola/issues/3112#issuecomment-402648436>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAAGK4Aj25DC0H8EQABz4ADscaCuyL_qks5uDdCBgaJpZM4VDdlT>
> .
>

Hmmm not sure what you expected, but:
```diff
--- /tmp/test-post.fr.md	2018-07-05 14:44:05.560164417 +0200
+++ posts/test-post.fr.md	2018-07-05 10:28:28.222556100 +0200
@@ -10,3 +10,4 @@
 -->
 
 Write your post here.
+OOOO
```
Either I am missing something, or this is not a bug.

Here is what your test script does:

* Create a new empty site
* Enable a french translation
* Set SHOW_UNTRANSLATED_POSTS to True
* Creates an empty post in french
* Build the site

At that point, build the site. The output in output/posts/test-post/index.html shows the empty post.

Then:

* Add new content in that post ("OOOOOO").
* Rebuild the site

At this point, yes, the output is different, because it now contains the new content.

* The build was incremental, it builds a lot less than the first one.
* The content shown is the current one, and is correct.

What exactly do you expect to be different?
Ah, I see, the problem is that the changed output is stale.

Sorry!
I guess the file dependencies for indexes are wrong and don't take the "real" source of the file into account.
So, this is handled by the taxonomy plugin, which seems to get it from ``post.deps(lang)`` ... looking...
Weird.

This seems correct.

```
In [3]: site.timeline[0].deps('en')
Out[3]: ['cache/posts/foo.es.html', 'posts/foo.es.rst']
```

This is just lazy

```
In [4]: site.timeline[0].deps('es')
Out[4]:
['cache/posts/foo.es.html',
 'cache/posts/foo.es.html',
 'posts/foo.es.rst',
 'posts/foo.es.rst']
```
Ok, the problem is not actually with the indexes but with the HTML fragments.

* The english index and so on ARE rebuilt correctly. BUT
* They use cache/posts/foo.html which is not marked as dirty when you modify posts/foo.es.rst

So, that's it, this has probably been there since nikola 2 or so :-)
post.fragment_deps seems to be deeply broken:

```
In [6]: site.timeline[0].fragment_deps('en')
Out[6]: []

In [7]: site.timeline[0].fragment_deps('es')
Out[7]: []
```

Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -623,7 +623,7 @@ def deps(self, lang):
             deps.append(self.meta('data', lang))
         deps += self._get_dependencies(self._dependency_file_page[lang])
         deps += self._get_dependencies(self._dependency_file_page[None])
-        return sorted(deps)
+        return sorted(set(deps))
 
     def deps_uptodate(self, lang):
         """Return a list of uptodate dependencies to build this post's page.
@@ -664,14 +664,8 @@ def compile(self, lang):
                 self.source_path, self.date))
 
     def fragment_deps(self, lang):
-        """Return a list of uptodate dependencies to build this post's fragment.
-
-        These dependencies should be included in ``uptodate`` for the task
-        which generates the fragment.
-        """
-        deps = []
-        if self.default_lang in self.translated_to:
-            deps.append(self.source_path)
+        """Return a list of dependencies to build this post's fragment."""
+        deps = [self.source_path]
         if os.path.isfile(self.metadata_path):
             deps.append(self.metadata_path)
         lang_deps = []


Test patch: