Repository: getnikola/nikola

Issue:
Make life easier and stop requiring building twice to get the full site
In other words, prevent this:

``` console
$ nikola init --demo n
$ cd n
$ nikola build
…lots of things get built…
$ nikola build
Scanning posts....done!
.  render_posts:cache/stories/listings-demo.html
.  render_pages:output/stories/listings-demo.html
.  sitemap:output/sitemap.xml
.  create_bundles:output/assets/css/all.css
.  create_bundles:output/assets/js/all-nocdn.js
.  create_bundles:output/assets/css/all-nocdn.css
.  create_bundles:output/assets/js/all.js
```

`sitemap` is our main culprit.  Notice that `create_bundles` not executed can break everything!



Discussion:
Looks like the bundles are only rebuilt the first time ever you build a site. After that, even deleting cache/ and output/ means you only get sitemap.xml on the second build.

Also, the bundles are identical in both builds, so that's rather harmless (but yes, needs to be avoided). Looking into sitemap.xml now, and again, the file is identical before and after building it.

So, this is the problem:

The 1st run: we calculate the dependencies for sitemap with the site still empty, so it knows no files.
On running the task, the site is now full, so we add all the LOCs to the map.

The 2nd run: we calculate the dependencies for sitemap with the site full, and exactly the same LOCs we used in effect before. So it's a do-nothing build and it outputs the exact same file.

Have not come up with a solution yet, I'll check doit's docs, calculated dependencies (deps calculated by a task) look good.

The unneeded rebuild of bundles (and listings-demo) seems to be caused by the code.css generation being out of order, I'll take a look at it later, probably.

On 1st run, create_bundles has no files as dependencies:

```
create_bundles:output/assets/css/all-nocdn.css -> {   u'_values_:': {   u'_config_changed': u'091946018db3f5830f7762160cc6f773'}}
```

On the second run, it does:

```
create_bundles:output/assets/css/all-nocdn.css -> {   u'_values_:': {   u'_config_changed': u'091946018db3f5830f7762160cc6f773'},
    u'output/assets/css/bootstrap.css': [   1391642462.05212,
                                            122998,
                                            u'b9db9e5b28c7a55e1de8f9ebfbd2c28d'],
    u'output/assets/css/code.css': [1392211009.8548136, 3856, u'cae9ecc680bd5333183e4880d97502aa'],
(and so on)
```

The file it produces is identical anyway. So, checking.

After #1048 lands, there is still a problem with listing-demo (probably caused by code.css)

`code.css` shouldn’t be depended on in any case, as it’s always included, in one form or another.

``` diff
diff --git a/db1 b/db2
index 8404fbf..d29e4a1 100644
--- a/db1
+++ b/db2
@@ -824,6 +824,7 @@ render_galleries:output/galleries -> {   u'_values_:': {   u'_config_changed': u
 render_posts:cache/stories/1.html -> {   u'_values_:': {   u'_config_changed': u'6fef980d3ed8920ebaf6f73a2f366b1e'},
     u'stories/1.rst': [1388762352.484822, 257, u'5c103e28532d50ca47655029412012ab']}
 render_posts:cache/stories/listings-demo.html -> {   u'_values_:': {   u'_config_changed': u'6fef980d3ed8920ebaf6f73a2f366b1e'},
+    u'listings/hello.py': [1388762352.484822, 161, u'a5c3ed2687e09e17861e71051594ba9b'],
     u'stories/listings-demo.rst': [1388762352.487822, 201, u'51a10b82708995acee28c65fa75bc585']}
 render_sources:output/stories/internals.txt -> {   u'_values_:': {   u'_config_changed': u'3fdf2f9fa08b24359a6cc875b27d3686'},
     u'stories/internals.txt': [1392123591.473427, 5973, u'426e1b520d8cde7ed231f3f78692a569']}
@@ -1335,9 +1336,10 @@ render_pages:output/stories/listings-demo.html -> {   u'/home/kwpolska/git/nikol
                                                                                       10,
                                                                                       u'85814f1e9c8b96a7199d1caf5531d2a0'],
     u'_values_:': {   u'_config_changed': u'4b7b12c2b9193a4e66522f27be97c929'},
-    u'cache/stories/listings-demo.html': [   1392214782.4761975,
+    u'cache/stories/listings-demo.html': [   1392214824.9585125,
                                              1136,
                                              u'b22dec0fad2cda61b119fbbef056254c'],
+    u'listings/hello.py': [1388762352.484822, 161, u'a5c3ed2687e09e17861e71051594ba9b'],
     u'stories/listings-demo.rst': [1388762352.487822, 201, u'51a10b82708995acee28c65fa75bc585']}
 render_pages:output/stories/quickstart.html -> {   u'/home/kwpolska/git/nikola/nikola/data/themes/base/templates/annotation_helper.tmpl': [   1388762352.5068216,
```

Don't worry about it, I'll figure it out when I have a few minutes :-)

On first run, listings-demo.html doesn't depend on hello.py, on second run it does. Should be a simple fix.

And of course it's not. Detecting the dependency on hello.py requires parsing the text. And of course, parsing the text is what this task is, so the task modifies its own dependencies.

Marking as won't fix that part :-(

I have an idea about how to fix the listings problem. Reopening.

The idea is:

split parsing and rendering reSt into 2 steps.
- 1st step parses the file, calculates dependencies.
- 2nd step depends on the result of 1st step then renders HTML

That way a single build will detect the listing dep on the source file, then trigger a single write.
It's tricky tho, because it involves a whole mess of docutils internals :-( 


Resolving patch:
diff --git a/nikola/plugins/task/posts.py b/nikola/plugins/task/posts.py
--- a/nikola/plugins/task/posts.py
+++ b/nikola/plugins/task/posts.py
@@ -31,6 +31,15 @@
 from nikola import utils
 
 
+def rest_deps(post, task):
+    """Add extra_deps from ReST into task.
+
+    The .dep file is created by ReST so not available before the task starts
+    to execute.
+    """
+    task.file_dep.update(post.extra_deps())
+
+
 class RenderPosts(Task):
     """Build HTML fragments from metadata and text."""
 
@@ -59,7 +68,9 @@ def gen_tasks(self):
                     'name': dest,
                     'file_dep': post.fragment_deps(lang),
                     'targets': [dest],
-                    'actions': [(post.compile, (lang, ))],
+                    'actions': [(post.compile, (lang, )),
+                                (rest_deps, (post,)),
+                                ],
                     'clean': True,
                     'uptodate': [utils.config_changed(deps_dict)],
                 }
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -319,6 +319,16 @@ def wrap_encrypt(path, password):
             LOGGER.notice('{0} is scheduled to be published in the future ({1})'.format(
                 self.source_path, self.date))
 
+    def extra_deps(self):
+        """get extra depepencies from .dep files
+        This file is created by ReST
+        """
+        dep_path = self.base_path + '.dep'
+        if os.path.isfile(dep_path):
+            with codecs.open(dep_path, 'rb+', 'utf8') as depf:
+                return [l.strip() for l in depf.readlines()]
+        return []
+
     def fragment_deps(self, lang):
         """Return a list of dependencies to build this post's fragment."""
         deps = []
@@ -326,10 +336,7 @@ def fragment_deps(self, lang):
             deps.append(self.source_path)
         if os.path.isfile(self.metadata_path):
             deps.append(self.metadata_path)
-        dep_path = self.base_path + '.dep'
-        if os.path.isfile(dep_path):
-            with codecs.open(dep_path, 'rb+', 'utf8') as depf:
-                deps.extend([l.strip() for l in depf.readlines()])
+        deps.extend(self.extra_deps())
         lang_deps = []
         if lang != self.default_lang:
             lang_deps = [get_translation_candidate(self.config, d, lang) for d in deps]


Test patch: