Repository: getnikola/nikola

Issue:
import_wordpress don't convert to markdown
import without errors

text from XML `<content:encoded>` is copied with a newline after each line.

export: https://davidak.basecom.de/paste/?dc58ba722a84925a#fTZSvcajZOpwJv5uhFRSe7vLARmWCqhtL7497BObAto=

![bildschirmfoto 2016-02-23 um 17 09 25](https://cloud.githubusercontent.com/assets/91113/13257836/b6594eda-da50-11e5-8bc1-06b24e75ac81.png)



Discussion:
also reproducible with fresh installed wordpress and pages.

[demosite.wordpress.2016-02-23.xml.zip](https://github.com/getnikola/nikola/files/143073/demosite.wordpress.2016-02-23.xml.zip)

The reason why this happens is here: https://github.com/getnikola/nikola/blob/master/nikola/plugins/command/import_wordpress.py#L385 as `line` already contains an `'\n'`. Replacing this line with `return b''.join(xml)` should fix this.

@Kwpolska, since you wrote that part: is there a reason why the extra newlines are needed?

They are probably not needed and just a bug. Feel free to remove that.

Ok, I created a PR for that part.

I don't understand the "don't convert to markdown", though. @davidak, did you only mean that additional newlines are introduced, or did you mean something else as well?

@felixfontein thanks!

take a look at my screenshot. the upper editor window shows the current result. it's plain HTML, not markdown.

Sorry, I overlooked the screenshot (had no RequestPolicy role for cloud.githubusercontent.com yet).

Well, the default "conversion" of WordPress posts to MarkDown was always quite strange (from my point of view); it does some little things and then runs the content through a HTML parser (lxml) and lets it output its state. That's why the output is so "strange". Since MarkDown allows HTML, this is probably kind of ok. It looks really ugly, though.

(That was also one of the main reasons why I started hacking around and finally created the WordPress compiler plugin, since I didn't want such a conversion which screws up the post sources in this way.)

So it's not really a bug, but a "feature". Fixing this requires a lot of effort.

(Your blog post also shows why it is so hard to work with WordPress posts: they can contain any random usage of HTML tags, like `<li>...</li>` without `<ul>...</ul>` and `<ol>...</ol>`. Converting such a mess correctly to another markup language is really, really hard. That's why I simply took the original WordPress conversion code and converted it to Python for the compiler plugin, so that it wouldn't try to convert anything but just use the same code as WordPress to render the result as HTML, so that the resulting HTML included in the blog output should at least be "as expected".)

what is wrong with the link or the more-tag?

i would call this a serious problem.

i found a solution with Exitwp and the Nikola Jekyll import. there will be a documentation on my new site.
but this should get fixed if you want to promote the wordpress import.

@davidak plain HTML _is_ valid markdown. The conversion basically takes some of the not-HTML things WP does and converts them into markdown, and leaves the rest as-is.

The Documentation of my Workaround is here: https://github.com/davidak/davidak.de/blob/master/pages/wordpress-artikel-nach-markdown-und-restructuredtext-konvertieren.rst (maybe it helps someone)

Hmm, we could of course modify the WordPress importer to use html2text (either the version from https://github.com/thomasf/exitwp or the original one from http://www.aaronsw.com/2002/html2text/). Unfortunately, it is licensed under a less liberal license than Nikola core, so it cannot simply be included. One possibility would be to allow the WordPress importer to use this utility, though.

Ah, there's a PyPi package, so we can just use it this way: https://pypi.python.org/pypi/html2text

@felixfontein want to take a crack at this one?


Resolving patch:
diff --git a/nikola/plugins/command/import_wordpress.py b/nikola/plugins/command/import_wordpress.py
--- a/nikola/plugins/command/import_wordpress.py
+++ b/nikola/plugins/command/import_wordpress.py
@@ -37,6 +37,11 @@
 from lxml import etree
 from collections import defaultdict
 
+try:
+    import html2text
+except:
+    html2text = None
+
 try:
     from urlparse import urlparse
     from urllib import unquote
@@ -169,6 +174,20 @@ class CommandImportWordpress(Command, ImportMixin):
             'type': bool,
             'help': "Export comments as .wpcomment files",
         },
+        {
+            'name': 'html2text',
+            'long': 'html2text',
+            'default': False,
+            'type': bool,
+            'help': "Uses html2text (needs to be installed with pip) to transform WordPress posts to MarkDown during import",
+        },
+        {
+            'name': 'transform_to_markdown',
+            'long': 'transform-to-markdown',
+            'default': False,
+            'type': bool,
+            'help': "Uses WordPress page compiler to transform WordPress posts to HTML and then use html2text to transform them to MarkDown during import",
+        },
         {
             'name': 'transform_to_html',
             'long': 'transform-to-html',
@@ -262,6 +281,9 @@ def _read_options(self, options, args):
         self.export_categories_as_categories = options.get('export_categories_as_categories', False)
         self.export_comments = options.get('export_comments', False)
 
+        self.html2text = options.get('html2text', False)
+        self.transform_to_markdown = options.get('transform_to_markdown', False)
+
         self.transform_to_html = options.get('transform_to_html', False)
         self.use_wordpress_compiler = options.get('use_wordpress_compiler', False)
         self.install_wordpress_compiler = options.get('install_wordpress_compiler', False)
@@ -280,10 +302,18 @@ def _read_options(self, options, args):
         self.separate_qtranslate_content = options.get('separate_qtranslate_content')
         self.translations_pattern = options.get('translations_pattern')
 
-        if self.transform_to_html and self.use_wordpress_compiler:
-            LOGGER.warn("It does not make sense to combine --transform-to-html with --use-wordpress-compiler, as the first converts all posts to HTML and the latter option affects zero posts.")
+        count = (1 if self.html2text else 0) + (1 if self.transform_to_html else 0) + (1 if self.transform_to_markdown else 0)
+        if count > 1:
+            LOGGER.error("You can use at most one of the options --html2text, --transform-to-html and --transform-to-markdown.")
+            return False
+        if (self.html2text or self.transform_to_html or self.transform_to_markdown) and self.use_wordpress_compiler:
+            LOGGER.warn("It does not make sense to combine --use-wordpress-compiler with any of --html2text, --transform-to-html and --transform-to-markdown, as the latter convert all posts to HTML and the first option then affects zero posts.")
+
+        if (self.html2text or self.transform_to_markdown) and not html2text:
+            LOGGER.error("You need to install html2text via 'pip install html2text' before you can use the --html2text and --transform-to-markdown options.")
+            return False
 
-        if self.transform_to_html:
+        if self.transform_to_html or self.transform_to_markdown:
             self._find_wordpress_compiler()
             if not self.wordpress_page_compiler and self.install_wordpress_compiler:
                 if not install_plugin(self.site, 'wordpress_compiler', output_dir='plugins'):  # local install
@@ -667,10 +697,10 @@ def replacement(m, c=content):
         return content
 
     @staticmethod
-    def transform_caption(content):
+    def transform_caption(content, use_html=False):
         """Transform captions."""
-        new_caption = re.sub(r'\[/caption\]', '', content)
-        new_caption = re.sub(r'\[caption.*\]', '', new_caption)
+        new_caption = re.sub(r'\[/caption\]', '</h1>' if use_html else '', content)
+        new_caption = re.sub(r'\[caption.*\]', '<h1>' if use_html else '', new_caption)
 
         return new_caption
 
@@ -693,6 +723,26 @@ def transform_content(self, content, post_format, attachments):
                 except TypeError:  # old versions of the plugin don't support the additional argument
                     content = self.wordpress_page_compiler.compile_to_string(content)
                 return content, 'html', True
+            elif self.transform_to_markdown:
+                # First convert to HTML with WordPress plugin
+                additional_data = {}
+                if attachments is not None:
+                    additional_data['attachments'] = attachments
+                try:
+                    content = self.wordpress_page_compiler.compile_to_string(content, additional_data=additional_data)
+                except TypeError:  # old versions of the plugin don't support the additional argument
+                    content = self.wordpress_page_compiler.compile_to_string(content)
+                # Now convert to MarkDown with html2text
+                h = html2text.HTML2Text()
+                content = h.handle(content)
+                return content, 'md', False
+            elif self.html2text:
+                # TODO: what to do with [code] blocks?
+                # content = self.transform_code(content)
+                content = self.transform_caption(content, use_html=True)
+                h = html2text.HTML2Text()
+                content = h.handle(content)
+                return content, 'md', False
             elif self.use_wordpress_compiler:
                 return content, 'wp', False
             else:


Test patch:
diff --git a/tests/test_command_import_wordpress.py b/tests/test_command_import_wordpress.py
--- a/tests/test_command_import_wordpress.py
+++ b/tests/test_command_import_wordpress.py
@@ -171,6 +171,8 @@ def test_create_import_work_without_argument(self):
     def test_populate_context(self):
         channel = self.import_command.get_channel_from_file(
             self.import_filename)
+        self.import_command.html2text = False
+        self.import_command.transform_to_markdown = False
         self.import_command.transform_to_html = False
         self.import_command.use_wordpress_compiler = False
         context = self.import_command.populate_context(channel)
@@ -195,6 +197,8 @@ def test_importing_posts_and_attachments(self):
         self.import_command.no_downloads = False
         self.import_command.export_categories_as_categories = False
         self.import_command.export_comments = False
+        self.import_command.html2text = False
+        self.import_command.transform_to_markdown = False
         self.import_command.transform_to_html = False
         self.import_command.use_wordpress_compiler = False
         self.import_command.tag_saniziting_strategy = 'first'
@@ -315,6 +319,8 @@ def test_transforming_content(self):
         transform_caption = mock.MagicMock()
         transform_newlines = mock.MagicMock()
 
+        self.import_command.html2text = False
+        self.import_command.transform_to_markdown = False
         self.import_command.transform_to_html = False
         self.import_command.use_wordpress_compiler = False