Repository: getnikola/nikola

Issue:
[BUG] Date in metadata can't be parsed using HTML post
### Environment (if reporting a bug)

**Python Version:** 3.6

**Nikola Version:** v7.8.14

**Operating System:** OS X

### Description:

It was 7.2 since I create last blog using Nikola website. I've been using it since 3 years now. But when I rebuild all of my posts, suddenly the HTML metadata can't parse date. I post using HTML, and always give prefix like meta example below for a particular html:

```
<!--
                    .. title: Aadhaar Data and Relational Databases
                    .. slug: Aadhaar-Data-and-Relational-Databases
                    .. date: 2014-08-28 07:01:28 UTC+07:00
                    .. tags: Data Science,Data Wrangling
                    .. link: 
                    .. description: 
                    .. type: text
                    -->
```

All the metadata parsed correctly, except date, which points today at 00:00. I checked with `conf.py` and still comment out. Is there any changes? I checked with all the issues and seems nothing similar. 


[BUG] Date in metadata can't be parsed using HTML post
### Environment (if reporting a bug)

**Python Version:** 3.6

**Nikola Version:** v7.8.14

**Operating System:** OS X

### Description:

It was 7.2 since I create last blog using Nikola website. I've been using it since 3 years now. But when I rebuild all of my posts, suddenly the HTML metadata can't parse date. I post using HTML, and always give prefix like meta example below for a particular html:

```
<!--
                    .. title: Aadhaar Data and Relational Databases
                    .. slug: Aadhaar-Data-and-Relational-Databases
                    .. date: 2014-08-28 07:01:28 UTC+07:00
                    .. tags: Data Science,Data Wrangling
                    .. link: 
                    .. description: 
                    .. type: text
                    -->
```

All the metadata parsed correctly, except date, which points today at 00:00. I checked with `conf.py` and still comment out. Is there any changes? I checked with all the issues and seems nothing similar. 


Error obtaining title from document in reSt in v8
reSt pages with title in the document but no title in the metadata have the slug as title.

This used to work correctly in v7 (document title was used), but it doesn't in v8 (slug or filename is used).


Discussion:
I can’t reproduce after copy-pasting your header. (macOS 10.13/Python 3.6/Nikola v7.8.14)

![image](https://user-images.githubusercontent.com/327323/38374815-faf490e0-38f4-11e8-8dbb-c595264894bb.png)

So when nikola build the `posts`, it produce 2 html for each post right? Actual HTML and src HTML. 

for src HTML, first few lines that I get, 

```HTML
<!--
                    .. title: Aadhaar Data and Relational Databases
                    .. slug: Aadhaar-Data-and-Relational-Databases
                    .. date: 2014-08-28 07:01:28 UTC+07:00
                    .. tags: Data Science,Data Wrangling
                    .. link: 
                    .. description: 
                    .. type: text
                    -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
 <head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <meta content="Evernote Mac 5.7.2 (451040)" name="exporter-version"/>
  <meta content="Data Science, Data Wrangling" name="keywords"/>
  <meta content="harijh2" name="author"/>
  <meta content="2014-08-28 07:01:28 +0000" name="created"/>
  <meta content="2014-08-28 09:50:55 +0000" name="updated"/>
  <title>
   Aadhaar Data and Relational Databases
  </title>
 </head>
```

Whereas the actual output HTML,


```HTML

<!DOCTYPE html>
<html prefix="
og: http://ogp.me/ns#
article: http://ogp.me/ns/article#
" lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width">
<title>
   Aadhaar Data and Relational Databases
   | Data Science, Python, Games</title>
<link href="../assets/css/all-nocdn.css" rel="stylesheet" type="text/css">
<link rel="alternate" type="application/rss+xml" title="RSS" href="../rss.xml">
<link rel="canonical" href="http://napitupulu-jon.appspot.com/posts/aadhaar-data-and-relational-databases.html">
<!--[if lt IE 9]><script src="/assets/js/html5.js"></script><![endif]--><meta name="author" content="harijh2">
<meta property="og:site_name" content="Data Science, Python, Games">
<meta property="og:title" content="
   Aadhaar Data and Relational Databases
  ">
<meta property="og:url" content="http://napitupulu-jon.appspot.com/posts/aadhaar-data-and-relational-databases.html">
<meta property="og:description" content="Aadhaar Data and Relational Databases
   
    Relational databases is set of database that related with each other
   
   
    We see that the table in the left (table = sprite">
<meta property="og:type" content="article">
<meta property="article:published_time" content="2018-04-11T00:00:00+07:00">
<meta property="article:tag" content="Data Science">
<meta property="article:tag" content="Data Wrangling">
</head>
```

Does this helpful? I'm currently also using ipython zen theme. 


Can we see the full `conf.py`, source `.html` file, and the paths you use?
Reproduced. Happens due to HTML metadata reading. The `<title>` triggers the html metadata reader, but since you don’t have a date there, it fails.

```html
<!--
.. title: WTF
.. slug: wtf
.. date: 2014-08-28 07:01:28 UTC+07:00
-->
<title>Foo a Bar</title>
```

This should be fixed by making the HTML meta reader a best-effort thing and still look for comments. Needs a minor refactoring, and perhaps an API change (which we can do for v8).
I can’t reproduce after copy-pasting your header. (macOS 10.13/Python 3.6/Nikola v7.8.14)

![image](https://user-images.githubusercontent.com/327323/38374815-faf490e0-38f4-11e8-8dbb-c595264894bb.png)

So when nikola build the `posts`, it produce 2 html for each post right? Actual HTML and src HTML. 

for src HTML, first few lines that I get, 

```HTML
<!--
                    .. title: Aadhaar Data and Relational Databases
                    .. slug: Aadhaar-Data-and-Relational-Databases
                    .. date: 2014-08-28 07:01:28 UTC+07:00
                    .. tags: Data Science,Data Wrangling
                    .. link: 
                    .. description: 
                    .. type: text
                    -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
 <head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <meta content="Evernote Mac 5.7.2 (451040)" name="exporter-version"/>
  <meta content="Data Science, Data Wrangling" name="keywords"/>
  <meta content="harijh2" name="author"/>
  <meta content="2014-08-28 07:01:28 +0000" name="created"/>
  <meta content="2014-08-28 09:50:55 +0000" name="updated"/>
  <title>
   Aadhaar Data and Relational Databases
  </title>
 </head>
```

Whereas the actual output HTML,


```HTML

<!DOCTYPE html>
<html prefix="
og: http://ogp.me/ns#
article: http://ogp.me/ns/article#
" lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width">
<title>
   Aadhaar Data and Relational Databases
   | Data Science, Python, Games</title>
<link href="../assets/css/all-nocdn.css" rel="stylesheet" type="text/css">
<link rel="alternate" type="application/rss+xml" title="RSS" href="../rss.xml">
<link rel="canonical" href="http://napitupulu-jon.appspot.com/posts/aadhaar-data-and-relational-databases.html">
<!--[if lt IE 9]><script src="/assets/js/html5.js"></script><![endif]--><meta name="author" content="harijh2">
<meta property="og:site_name" content="Data Science, Python, Games">
<meta property="og:title" content="
   Aadhaar Data and Relational Databases
  ">
<meta property="og:url" content="http://napitupulu-jon.appspot.com/posts/aadhaar-data-and-relational-databases.html">
<meta property="og:description" content="Aadhaar Data and Relational Databases
   
    Relational databases is set of database that related with each other
   
   
    We see that the table in the left (table = sprite">
<meta property="og:type" content="article">
<meta property="article:published_time" content="2018-04-11T00:00:00+07:00">
<meta property="article:tag" content="Data Science">
<meta property="article:tag" content="Data Wrangling">
</head>
```

Does this helpful? I'm currently also using ipython zen theme. 


Can we see the full `conf.py`, source `.html` file, and the paths you use?
Reproduced. Happens due to HTML metadata reading. The `<title>` triggers the html metadata reader, but since you don’t have a date there, it fails.

```html
<!--
.. title: WTF
.. slug: wtf
.. date: 2014-08-28 07:01:28 UTC+07:00
-->
<title>Foo a Bar</title>
```

This should be fixed by making the HTML meta reader a best-effort thing and still look for comments. Needs a minor refactoring, and perhaps an API change (which we can do for v8).
How is it different from https://getnikola.com/handbook.html which seems to work correctly?
The Handbook has a `.. title: The Nikola Handbook` line as well.
So does creating-a-theme.txt:

```
.. title: Creating a Theme
.. slug: creating-a-theme
.. date: 2015-05-28 18:46:48 UTC
.. tags:
.. category:
.. link:
.. description:
.. type: text

Creating a Theme
================

```

```
.. title: The Nikola Handbook
.. slug: handbook
.. date: 2012-03-30 23:00:00 UTC-03:00
.. link:
.. description:
.. tags: mathjax
.. author: The Nikola Team

The Nikola Handbook
===================

```
Honestly I have no idea why they look different.
It looks different [on the site](https://getnikola.com/creating-a-theme.txt).
Annnnnd that explains it. I suppose this will go away once we redo the site for v8 one of these years. I will do a quick rebuild fixing that.
Fixed on the site, I am pushing it in the v7 branch as well
Still need to fix the root cause.
So, it seems we either don't have or it's not using, a metadata extractor for reSt data.
This one is supposed to be used: https://github.com/getnikola/nikola/blob/master/nikola/plugins/compile/rest/__init__.py#L65

Perhaps there’s some issue with priorities, or with merging output of different extractors?
Don't know, am trying to see :-)
Looks like this is False and should be True

```
metadata_extractors.check_conditions(post, post.source_path, post.compiler.metadata_conditions, config, None))
```

That is because it needs USE_REST_DOCINFO_METADATA to be set to True.

OTOH, if that is True, it crashes with 

```
Scanning posts.[2018-03-17T16:28:25Z] ERROR: Nikola: Missing date in file posts/foo.rst
[2018-03-17T16:28:25Z] ERROR: scan_posts: Error reading post posts/foo.rst
[2018-03-17T16:28:25Z] ERROR: Nikola: Error reading timeline
[2018-03-17T16:28:25Z] ERROR: Nikola: Error loading tasks. An unhandled exception occurred.
[2018-03-17T16:28:25Z] ERROR: Nikola: ValueError: Missing date in file posts/foo.rst
```

And that's because we have this:

```
# Moreover, we should not read the file if we have compiler meta.
```

So, it seems compiler meta is all-or-nothing, and in v7 it was not.

We need to decide if this is the behaviour we want, or not. @kwpolska opinions?

Resolving patch:
diff --git a/nikola/plugins/compile/rest/__init__.py b/nikola/plugins/compile/rest/__init__.py
--- a/nikola/plugins/compile/rest/__init__.py
+++ b/nikola/plugins/compile/rest/__init__.py
@@ -28,6 +28,7 @@
 
 import io
 import os
+import logbook
 import logbook.base
 
 import docutils.core
@@ -68,9 +69,13 @@ def read_metadata(self, post, lang=None):
             lang = LocaleBorg().current_lang
         source_path = post.translated_source_path(lang)
 
+        # Silence reST errors, some of which are due to a different
+        # environment. Real issues will be reported while compiling.
+        null_logger = logbook.Logger('NULL')
+        null_logger.handlers = [logbook.NullHandler()]
         with io.open(source_path, 'r', encoding='utf-8') as inf:
             data = inf.read()
-            _, _, _, document = rst2html(data, logger=self.logger, source_path=source_path, transforms=self.site.rst_transforms, no_title_transform=False)
+            _, _, _, document = rst2html(data, logger=null_logger, source_path=source_path, transforms=self.site.rst_transforms, no_title_transform=False)
         meta = {}
         if 'title' in document:
             meta['title'] = document['title']
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -1012,9 +1012,9 @@ def get_meta(post, lang):
         metadata_extractors_by = metadata_extractors.default_metadata_extractors_by()
 
     # If meta file exists, use it
-    meta.update(get_metadata_from_meta_file(post.metadata_path, post, config, lang, metadata_extractors_by))
+    metafile_meta = get_metadata_from_meta_file(post.metadata_path, post, config, lang, metadata_extractors_by)
 
-    if not meta:
+    if not metafile_meta:
         post.is_two_file = False
 
     # Fetch compiler metadata.
@@ -1026,11 +1026,12 @@ def get_meta(post, lang):
         used_extractor = post.compiler
         meta.update(compiler_meta)
 
-    if not post.is_two_file and not compiler_meta:
-        # Meta file has precedence over file, which can contain garbage.
-        # Moreover, we should not read the file if we have compiler meta.
+    # Meta files and inter-file metadata override compiler metadata
+    if not post.is_two_file:
         new_meta, used_extractor = get_metadata_from_file(post.source_path, post, config, lang, metadata_extractors_by)
         meta.update(new_meta)
+    else:
+        meta.update(metafile_meta)
 
     # Filename-based metadata extractors (fallback only)
     if not meta:


Test patch: