Repository: getnikola/nikola

Issue:
Index pages do not respect body-content div after first post
<!--
Before creating an issue:
* make sure you are using an up-to-date version of Nikola
* search for existing issues that might be related

Make sure to:
* provide information about your environment (below)
* include all the output you get, and any other information related to your problem

Nikola v7.6.4, as provided by Ubuntu, is NOT SUPPORTED.
If you are using this version, you should upgrade: https://getnikola.com/getting-started.html
-->

### Environment

**Python Version:** 3.9.6

**Nikola Version:** 8.1.3

**Operating System:** Windows 10 

### Description:

When INDEX_DISPLAY_POST_COUNT = 2 (or higher) the first post on the index page is correctly inside the div class="body-content"

All following posts are outside that div, and therefore do not inherit the formatting of "body-content"

The most obvious visual clue is that the width of the posts following the first post is the full page width, not the width of the body-content div.

I have changed to the unmodified base theme and the problem is the same - and there is nothing obvious in the index template to indicate a problem. 

One reason this may have slipped through is that the demo sites for all current Nikola Themes only has index pages with one item :-)
Generated index.html closes surrounding html tags
<!--
Before creating an issue:
* make sure you are using an up-to-date version of Nikola
* search for existing issues that might be related

Make sure to:
* provide information about your environment (below)
* include all the output you get, and any other information related to your problem

Nikola v7.6.4, as provided by Ubuntu, is NOT SUPPORTED.
If you are using this version, you should upgrade: https://getnikola.com/getting-started.html
-->

### Environment

**Python Version:** 3.9.6
**Nikola Version:** 8.1.3
**Operating System:** Linux

### Description:
The actual issue is:
The main `index.html` landing page doesn't center the listed posts:
![image](https://user-images.githubusercontent.com/46302524/128340835-a9360b27-3c00-484a-a841-02b6a91a16e8.png)
I would assume that the second post would also be directly under the first one.

To reproduce
```sh
nikola init --demo
nikola new_post -f markdown
nikola auto --browser
```

Looking at the output for the `index.html` one can see that too many tags are closed.
Specifically, referring to the source [tags.tmpl](https://github.com/getnikola/nikola/blob/master/nikola/data/themes/base/templates/index.tmpl#L27) file:
There is a surrounding div class that loops over the posts [`<div class="postindex">`](https://github.com/getnikola/nikola/blob/67fe01f445ed0aac86fe7a10e721e1fcad1f665a/nikola/data/themes/base/templates/index.tmpl#L27)
If we add an HTML comment before/after loading the text, we see that the div and everything before that is closed and the next line of the loop starts with a new HTML tag.

```sh
nikola theme --copy-template index.tmpl
```
Add some comments in tmpl file, note that the endfor does not contain any `</html>` tag or similar

```html
            <!-- From here to next comment post.text is inserted and article closes the main html tag, even if it shouldn't -->
            %if index_teasers:
                <div class="p-summary entry-summary">
                    ${post.text(teaser_only=True)}
                </div>
            %else:
                <div class="e-content entry-content">
                    ${post.text(teaser_only=False)}
                </div>
            %endif
            </article>
            <!-- According to index.tmpl should start with new article now! -->
        % endfor
```

The index.html in output becomes after build:
```html
                    <!-- From here to next comment post.text is inserted and article closes the main html tag, even if it shouldn't --> <div class="e-content entry-content">
                        <p>Write your post here.</p>
                    </div>
                </article>
            </div>
        </div>
    </div>
</body>

</html><!-- According to index.tmpl should start with new article now! -->
```


Discussion:
We had a report about this in #3564 — it was caused by a broken version of lxml shipped by Conda. Can you try again with a `lxml` version from PyPI?
I've followed that #3564 thread and have a little more information to add - sorry for not mentioning it up front.
I am running on Windows 10, but I am using the git for windows SDK - so that's MYSY2 Mingw64.
The version of libxml2 is libxml2-2.9.12-2 and MSYS2 prefers to NOT downgrade packages but ...
https://www.msys2.org/docs/package-management/
After downgrading to http://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-libxml2-2.9.10-8-any.pkg.tar.zst the multi-post index pages look fine again :-)
Thanks for the quick response - closing this issue but I cannot help but think that a significant issue with libxml 2.9.11 and 2.9.12 would have shown up in other use cases.
Is it possible that there is still an issue in nikola due to incorrect assumptions in how libxml2 is used?
There is a possibility that the HTML fed to lxml contains unbalanced tags, or some other combination libxml2 dislikes since 2.9.11. Or there’s a regression in libxml2. That could be verified by removing `lxml` from the Nikola pipeline and checking the produced code.
Sorry to keep this thread going after the issue is closed, but could you point me at where to remove lxml from the Nikola pipeline? I've poked around in nikola.py and have scanned the internals document but nothing obvious is jumping out at me :-)

There seems to be a LOT of lxml usage scattered in the Nikola source and I'm pretty sure that from your comment there is one RIGHT place to catch the output.
I’d say that commenting out the lxml-related parts in `render_template` should do it, and that would be [nikola.py, lines 1497-1509](
https://github.com/getnikola/nikola/blob/db97e3da73d1ce6fcf4138e938967db8a4efc0c6/nikola/nikola.py#L1497-L1509).

https://github.com/getnikola/nikola/blob/db97e3da73d1ce6fcf4138e938967db8a4efc0c6/nikola/nikola.py#L1497-L1509
Thanks for the hint - I did try to comment that section out previously but forgot to set my libxml2 back to the current version.
Now that I have that straight, I can confirm the issue sits here:
https://github.com/getnikola/nikola/blob/db97e3da73d1ce6fcf4138e938967db8a4efc0c6/nikola/nikola.py#L1501
and
https://github.com/getnikola/nikola/blob/db97e3da73d1ce6fcf4138e938967db8a4efc0c6/nikola/nikola.py#L1509

If you comment out those lines (and the preceeding else) and add:
data = data.encode('utf-8')
Then the HTML is rendered correctly - there seems to be a problem when going from HTML data to XML doc and pretty printing back to HTML data ... 
Let me know if there's anything I can do to test/verify - I have a workaround for now so it's not critical
I tried investigating a bit further and found this in the output:

```

                <div class="e-content entry-content">
                    <body><p>Write your post here.</p></body>
</html>


                </div>
```

There might be something wrong with `Post.text()` that causes it to insert unexpected tags into its output, which seemingly confuse new versions of libxml2.
This looks like a duplicate of #3560. Please `rm -rf cache` and make sure all Nikola dependencies (in particular `lxml`, `docutils`, `mako` / `jinja2` are up-to-date).
Interesting.
Yes, as in #3560 I am also using conda as a package manager (actually mamba with conda-forge).
I can reproduce the error by running:
```sh
mamba create --name nikola
conda activate nikola
mamba install -c conda-forge nikola
pip install Nikola[extra]
```
Then the output becomes ill-formatted.
Here is the exact dependency yml file, to reproduce:
```yml
name: nikola2
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - babel=2.9.1=pyh44b312d_0
  - blinker=1.4=py_1
  - brotlipy=0.7.0=py39h3811e60_1001
  - ca-certificates=2021.7.5=h06a4308_1
  - certifi=2021.5.30=py39hf3d152e_0
  - cffi=1.14.6=py39he32792d_0
  - chardet=4.0.0=py39hf3d152e_1
  - charset-normalizer=2.0.0=pyhd8ed1ab_0
  - cloudpickle=1.6.0=py_0
  - cryptography=3.4.7=py39hbca0aa6_0
  - docutils=0.17.1=py39hf3d152e_0
  - doit=0.33.1=py39hde42818_1
  - freetype=2.10.4=h0708190_1
  - icu=68.1=h58526e2_0
  - idna=3.2=pyhd3eb1b0_0
  - importlib-metadata=4.6.3=py39hf3d152e_0
  - jbig=2.1=h7f98852_2003
  - jpeg=9d=h516909a_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=2.2.1=h9c3ff4c_0
  - libdeflate=1.7=h7f98852_5
  - libffi=3.3=h58526e2_2
  - libgcc-ng=11.1.0=hc902ee8_8
  - libgomp=11.1.0=hc902ee8_8
  - libiconv=1.16=h516909a_0
  - libpng=1.6.37=hed695b0_2
  - libstdcxx-ng=11.1.0=h56837e0_8
  - libtiff=4.3.0=hf544144_1
  - libwebp-base=1.2.0=h7f98852_2
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - lxml=4.6.3=py39h107f48f_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - mako=1.1.4=pyh44b312d_0
  - markdown=3.3.4=pyhd8ed1ab_0
  - markupsafe=2.0.1=py39h3811e60_0
  - natsort=7.1.1=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - nikola=8.1.3=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1k=h7f98852_0
  - piexif=1.1.3=py_2
  - pillow=8.3.1=py39ha612740_0
  - pip=21.2.2=pyhd8ed1ab_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.9.0=pyhd8ed1ab_0
  - pyinotify=0.9.6=py39hf3d152e_1003
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyrss2gen=1.1=pyhd3deb0d_0
  - pysocks=1.7.1=py39hf3d152e_3
  - python=3.9.6=h49503c6_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - readline=8.1=h46c0cb4_0
  - requests=2.26.0=pyhd8ed1ab_0
  - setuptools=52.0.0=py39h06a4308_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.36.0=h9cd32fc_0
  - tk=8.6.10=hed695b0_1
  - tzdata=2021a=he74cb21_1
  - unidecode=1.2.0=pyhd8ed1ab_0
  - urllib3=1.26.6=pyhd8ed1ab_0
  - wheel=0.36.2=pyhd3deb0d_0
  - xz=5.2.5=h516909a_1
  - yapsy=1.12.2=py_0
  - zipp=3.5.0=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.5.0=ha95c52a_0
  - pip:
    - aiohttp==3.7.4.post0
    - argon2-cffi==20.1.0
    - async-generator==1.10
    - async-timeout==3.0.1
    - attrs==21.2.0
    - backcall==0.2.0
    - bleach==4.0.0
    - debugpy==1.4.1
    - decorator==5.0.9
    - defusedxml==0.7.1
    - entrypoints==0.3
    - ghp-import==2.0.1
    - hsluv==5.0.2
    - html5lib==1.1
    - ipykernel==6.0.3
    - ipython==7.26.0
    - ipython-genutils==0.2.0
    - jedi==0.18.0
    - jinja2==3.0.1
    - jsonschema==3.2.0
    - jupyter-client==6.1.12
    - jupyter-core==4.7.1
    - jupyterlab-pygments==0.1.2
    - matplotlib-inline==0.1.2
    - micawber==0.5.4
    - mistune==0.8.4
    - multidict==5.1.0
    - nbclient==0.5.3
    - nbconvert==6.1.0
    - nbformat==5.1.3
    - nest-asyncio==1.5.1
    - notebook==6.4.1
    - packaging==21.0
    - pandocfilters==1.4.3
    - parso==0.8.2
    - pexpect==4.8.0
    - phpserialize==1.3
    - pickleshare==0.7.5
    - prometheus-client==0.11.0
    - prompt-toolkit==3.0.19
    - ptyprocess==0.7.0
    - pygal==2.4.0
    - pyparsing==2.4.7
    - pyphen==0.11.0
    - pyrsistent==0.18.0
    - pyzmq==22.2.1
    - ruamel-yaml==0.17.10
    - ruamel-yaml-clib==0.2.6
    - send2trash==1.7.1
    - smartypants==2.0.1
    - terminado==0.10.1
    - testpath==0.5.0
    - toml==0.10.2
    - tornado==6.1
    - traitlets==5.0.5
    - typing-extensions==3.10.0.0
    - typogrify==2.0.7
    - watchdog==2.1.3
    - wcwidth==0.2.5
    - webencodings==0.5.1
    - yarl==1.6.3
```
If I create a new environment and install everything via pip, I do not get the error. The output of `pip freeze` is:
```yml
aiohttp==3.7.4.post0
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==21.2.0
Babel==2.9.1
backcall==0.2.0
bleach==4.0.0
blinker==1.4
certifi==2021.5.30
cffi==1.14.6
chardet==4.0.0
charset-normalizer==2.0.4
cloudpickle==1.6.0
debugpy==1.4.1
decorator==5.0.9
defusedxml==0.7.1
docutils==0.17.1
doit==0.33.1
entrypoints==0.3
ghp-import==2.0.1
hsluv==5.0.2
html5lib==1.1
idna==3.2
ipykernel==6.0.3
ipython==7.26.0
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==3.0.1
jsonschema==3.2.0
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
lxml==4.6.3
Mako==1.1.4
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib-inline==0.1.2
micawber==0.5.4
mistune==0.8.4
multidict==5.1.0
natsort==7.1.1
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nest-asyncio==1.5.1
Nikola==8.1.3
notebook==6.4.1
packaging==21.0
pandocfilters==1.4.3
parso==0.8.2
pexpect==4.8.0
phpserialize==1.3
pickleshare==0.7.5
piexif==1.1.3
Pillow==8.3.1
prometheus-client==0.11.0
prompt-toolkit==3.0.19
ptyprocess==0.7.0
pycparser==2.20
pygal==2.4.0
Pygments==2.9.0
pyinotify==0.9.6
pyparsing==2.4.7
pyphen==0.11.0
pyrsistent==0.18.0
PyRSS2Gen==1.1
python-dateutil==2.8.2
pytz==2021.1
pyzmq==22.2.1
requests==2.26.0
ruamel.yaml==0.17.10
ruamel.yaml.clib==0.2.6
Send2Trash==1.7.1
six==1.16.0
smartypants==2.0.1
terminado==0.10.1
testpath==0.5.0
toml==0.10.2
tornado==6.1
traitlets==5.0.5
typing-extensions==3.10.0.0
typogrify==2.0.7
Unidecode==1.2.0
urllib3==1.26.6
watchdog==2.1.3
wcwidth==0.2.5
webencodings==0.5.1
Yapsy==1.12.2
yarl==1.6.3
```
I will do some extra research tomorrow and maybe I am able to find the underlying dependency issue.
So please let the issue open for now. :) 
As expected, re-installing in the "broken" conda environment with:
`pip install -U --force-reinstall Nikola[extras]` fixes the issue.
At least from the version number, there seems to be no difference between
```
lxml (both 4.6.3)
mako (both 1.1.4
jinja2 (both 3.0.1)
docutils (both 0.17.3)
```
Bug is also reproducible after only installing from conda-forge feedstock
So only installing `mamba install -c conda-forge nikola`
```yml
name: nikola2
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - babel=2.9.1=pyh44b312d_0
  - blinker=1.4=py_1
  - brotlipy=0.7.0=py39h3811e60_1001
  - ca-certificates=2021.7.5=h06a4308_1
  - certifi=2021.5.30=py39hf3d152e_0
  - cffi=1.14.6=py39he32792d_0
  - chardet=4.0.0=py39hf3d152e_1
  - charset-normalizer=2.0.0=pyhd8ed1ab_0
  - cloudpickle=1.6.0=py_0
  - cryptography=3.4.7=py39hbca0aa6_0
  - docutils=0.17.1=py39hf3d152e_0
  - doit=0.33.1=py39hde42818_1
  - freetype=2.10.4=h0708190_1
  - icu=68.1=h58526e2_0
  - idna=3.2=pyhd3eb1b0_0
  - importlib-metadata=4.6.3=py39hf3d152e_0
  - jbig=2.1=h7f98852_2003
  - jpeg=9d=h516909a_0
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=2.2.1=h9c3ff4c_0
  - libdeflate=1.7=h7f98852_5
  - libffi=3.3=h58526e2_2
  - libgcc-ng=11.1.0=hc902ee8_8
  - libgomp=11.1.0=hc902ee8_8
  - libiconv=1.16=h516909a_0
  - libpng=1.6.37=hed695b0_2
  - libstdcxx-ng=11.1.0=h56837e0_8
  - libtiff=4.3.0=hf544144_1
  - libwebp-base=1.2.0=h7f98852_2
  - libxml2=2.9.12=h72842e0_0
  - libxslt=1.1.33=h15afd5d_2
  - lxml=4.6.3=py39h107f48f_0
  - lz4-c=1.9.3=h9c3ff4c_1
  - mako=1.1.4=pyh44b312d_0
  - markdown=3.3.4=pyhd8ed1ab_0
  - markupsafe=2.0.1=py39h3811e60_0
  - natsort=7.1.1=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - nikola=8.1.3=pyhd8ed1ab_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1k=h7f98852_0
  - piexif=1.1.3=py_2
  - pillow=8.3.1=py39ha612740_0
  - pip=21.2.2=pyhd8ed1ab_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pygments=2.9.0=pyhd8ed1ab_0
  - pyinotify=0.9.6=py39hf3d152e_1003
  - pyopenssl=20.0.1=pyhd8ed1ab_0
  - pyrss2gen=1.1=pyhd3deb0d_0
  - pysocks=1.7.1=py39hf3d152e_3
  - python=3.9.6=h49503c6_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytz=2021.1=pyhd8ed1ab_0
  - readline=8.1=h46c0cb4_0
  - requests=2.26.0=pyhd8ed1ab_0
  - setuptools=52.0.0=py39h06a4308_0
  - six=1.16.0=pyh6c4a22f_0
  - sqlite=3.36.0=h9cd32fc_0
  - tk=8.6.10=hed695b0_1
  - tzdata=2021a=he74cb21_1
  - unidecode=1.2.0=pyhd8ed1ab_0
  - urllib3=1.26.6=pyhd8ed1ab_0
  - wheel=0.36.2=pyhd3deb0d_0
  - xz=5.2.5=h516909a_1
  - yapsy=1.12.2=py_0
  - zipp=3.5.0=pyhd8ed1ab_0
  - zlib=1.2.11=h516909a_1010
  - zstd=1.5.0=ha95c52a_0
```
It might be some other package causing that, or it might be caused by bad builds of one of those in conda repos.
For future reference:
The issue seems to be caused by the `lxml` package from conda-forge.
After installing via conda and then running:
`pip install -U --force-reinstall lxml` the issue seems resolved
@kai-tub Could you report this as a bug to conda-forge?
Done :+1: 
I've also spent some more time to find the issue: It is caused by a **newer** libxml2 package from conda over my local version.
The bug appears starting with libxml2==2.9.11 and is fixed when installing libxml2==2.9.10 (at least in the conda environment)
So, I think this bug might appear more often in the future when/if the Linux distros move away from 2.9.10 to 2.9.11
But I did not check if the issue really appears when installing pip and linking to the newer libxml2 version.
Still, maybe something to keep in mind for the future :)
Hopefully, we can update the conda-recipe to ensure libxml2<=2.9.10
If true, this is an upstream bug that should not be reported to the conda-forge package, but to libxml2. 2.9.x should be critical bug fixing that is non-breaking.
Possibly you should report to https://github.com/conda-forge/libxml2-feedstock too so that they can block the problematic releases. If there exist such a bug report on libxml2 already, reference that. The maintainer(s) of libxml2-feedstock should decide(s) what they want to do about it.

Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -910,10 +910,7 @@ def text(self, lang=None, teaser_only=False, strip_html=False, show_read_more_li
         if self.hyphenate:
             hyphenate(document, real_lang)
 
-        try:
-            data = lxml.html.tostring(document.body, encoding='unicode')
-        except Exception:
-            data = lxml.html.tostring(document, encoding='unicode')
+        data = utils.html_tostring_fragment(document)
 
         if teaser_only:
             teaser_regexp = self.config.get('TEASER_REGEXP', TEASER_REGEXP)
@@ -936,10 +933,7 @@ def text(self, lang=None, teaser_only=False, strip_html=False, show_read_more_li
                         post_title=self.title(lang))
                 # This closes all open tags and sanitizes the broken HTML
                 document = lxml.html.fromstring(teaser)
-                try:
-                    data = lxml.html.tostring(document.body, encoding='unicode')
-                except IndexError:
-                    data = lxml.html.tostring(document, encoding='unicode')
+                data = utils.html_tostring_fragment(document)
 
         if data and strip_html:
             try:
@@ -952,11 +946,11 @@ def text(self, lang=None, teaser_only=False, strip_html=False, show_read_more_li
             if self.demote_headers:
                 # see above
                 try:
-                    document = lxml.html.fromstring(data)
+                    document = lxml.html.fragment_fromstring(data, "body")
                     demote_headers(document, self.demote_headers)
-                    data = lxml.html.tostring(document.body, encoding='unicode')
+                    data = utils.html_tostring_fragment(document)
                 except (lxml.etree.ParserError, IndexError):
-                    data = lxml.html.tostring(document, encoding='unicode')
+                    pass
 
         return data
 
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -30,6 +30,7 @@
 import datetime
 import hashlib
 import io
+import lxml.html
 import operator
 import os
 import re
@@ -656,6 +657,25 @@ def get_theme_chain(theme, themes_dirs):
     return themes
 
 
+def html_tostring_fragment(document):
+    """Convert a HTML snippet to a fragment, ready for insertion elsewhere."""
+    try:
+        doc = lxml.html.tostring(document.body, encoding='unicode').strip()
+    except Exception:
+        doc = lxml.html.tostring(document, encoding='unicode').strip()
+    start_fragments = ["<html>", "<body>"]
+    end_fragments = ["</body>", "</html>"]
+    for start in start_fragments:
+        if doc.startswith(start):
+            doc = doc[len(start):].strip()
+            print(repr(doc))
+    for end in end_fragments:
+        if doc.endswith(end):
+            doc = doc[:-len(end)].strip()
+            print(repr(doc))
+    return doc
+
+
 INCOMPLETE_LANGUAGES_WARNED = set()
 
 


Test patch: