Repository: getpelican/pelican

Issue:
pelican-import corrupts posts imported from WordPress
Hello,

I tried to move my blog ( https://eax.me/ ) from WordPress to Pelican using pelican-import. It works for the most part. There were a few difficulties but I managed to find workarounds for most of them. Particularly pelican-import doesn't handle images well and doesn't support CodeColorer plugin so I had to use some regular expressions (see below).

Also I discovered that pelican-import doesn't work with pandoc 2.0.2 properly so I had to patch the script around line 729:

```
            parse_raw = ''
            cmd = ('pandoc {0} --from=html'
                   ' --to={1}+raw_html -o "{2}" "{3}"')
            cmd = cmd.format(parse_raw, out_markup,
                             out_filename, html_filename)
```

These are minor issues though. The most serious issues are the following.

1. After migration characters like `'`, `"` and `$` were replaced to `\'`, `\"` and `\$`. I can't just replace these sequences back to `'`, `"` and `$` since these sequences sometimes are used in code snippets. There is a similar issue with the dash symbol that after migrations turn from `---` to `\-\--`.

2. Also in some code snippets code like `#include <something.h>` just turns into `#include`.

3. Last but not least one of the posts created by pelican-import (see below) hangs `make devserver` with 100% CPU usage.

You can reproduce all these issues using the XML file that I exported from WordPress. Please leave your email and I will send it to you.

Exact steps to reproduce:

```
perl -pi -e 's/\[cci.*?\]/<code>/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/\[\/cci.*?\]/<\/code>/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/\[cc_(\w+).*?\]/<pre><code>:::$1/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/\[cc.*?lang="(\w+)".*?\]/<pre><code>:::$1/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/\[cc.*?\]/<pre><code>/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/\[\/cc.*?\]/<\/code><\/pre>/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/<img .*?((src=".*?" alt=".*?")|(alt=".*?" src=".*?)).*?>/<img $1 \/>/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's/<span style="white\-space: nowrap;">(.*?)<\/span>/$1/g' ~/temp/del-me/wordpress.2017-11-29.xml
perl -pi -e 's!"https?:\/\/eax\.me\/([a-z0-9\-_]+)\/"!"https:\/\/eax.me\/$1.html"!g' ~/temp/del-me/wordpress.2017-11-29.xml
pelican-import -o content/ -m markdown --wpfile ~/temp/del-me/wordpress.2017-11-29.xml
```
Run `make devserver` to make sure that content/mojolicious.md hangs the server (issue 3). Temporary replace it somewhere and run `make devserver` again.

Then see:

```
http://localhost:8000/diy-presentation-remote.html
http://localhost:8000/elliptic-curves-crypto.html
```
^ symbols `$`, `'` and `"` turned into `\$`, `\'` and `\"` (issue 1).

```
http://localhost:8000/cpp-gtest.html
```
^ `#include <something.h>` was replaced to `#include` (issue 2).

Pelican version is 3.7.1.


Discussion:
I maybe have the same trouble as you had.
My environment is as follows.
pelican 3.7.1
pandoc 2.0.5

When I run the following command, an error occurred.
```
$ pelican-import --wpfile -o ./output ./wp.xml
...
--normalize has been removed.  Normalization is now automatic.
--parse-raw/-R has been removed. Use +raw_html or +raw_tex extension.

Try pandoc --help for more information.
Please, check your Pandoc installation.
```
It seems that Pandoc's arguments have changed. 
In order to avoid these error, you patched the script around line 729, didn't you?

Right.
Just ran into the very same error messages as @waura 
pelican=3.7.1
Pandoc 2.0.6

```
$ pelican-import --wpfile -o output2/ osp-blog.wordpress.2017-10-10.xml 
output2/-.rst
--normalize has been removed.  Normalization is now automatic.
--parse-raw/-R has been removed. Use +raw_html or +raw_tex extension.

Try pandoc --help for more information.
Please, check your Pandoc installation.
```


No news on this issue? Seems simply like Pandoc arguments have changed, any idea what version of pandoc we could downgrate do to get this working again?

Thanks!


I was personally able to get my xml export to convert to markdown, although both @afiskon and @waura seem to desire .rst

This is what I did to lines 728 - 732

```
            parse_raw = '' if not strip_raw else ''
            cmd = ('pandoc {0} --from=html'
                   ' --to=gfm+raw_html -o "{2}" "{3}"')
            cmd = cmd.format(parse_raw, out_markup,
                             out_filename, html_filename)
```
And this let me import, convert and works ok with the other pelican import args like `--dir-cat` & ` --dir-page`

Hope this helps in solving the issue for .rst convertions.
Cheers
I just hit this issue too.
@davidwilemski: It seems you might have [implemented a workaround](https://github.com/davidwilemski/pelican/commit/77580dbbfbf8fb7f003bf132c1e7ee9a75b3521d) for this issue. Would you consider submitting a pull request so that others don't run into this problem?
@justinmayer, sure, I've opened up #2289 to get the ball rolling.
I hit this issue as well. The solution that @afiskon showed helped.

Resolving patch:
diff --git a/pelican/tools/pelican_import.py b/pelican/tools/pelican_import.py
--- a/pelican/tools/pelican_import.py
+++ b/pelican/tools/pelican_import.py
@@ -674,6 +674,22 @@ def download_attachments(output_path, urls):
     return locations
 
 
+def is_pandoc_needed(fields):
+    in_markup_idx = 9
+    return filter(lambda f: f[in_markup_idx] in ('html', 'wp-html'), fields)
+
+
+def get_pandoc_version():
+    cmd = ['pandoc', '--version']
+    try:
+        output = subprocess.check_output(cmd, universal_newlines=True)
+    except (subprocess.CalledProcessError, OSError) as e:
+        logger.warning("Pandoc version unknown: %s", e)
+        return ''
+
+    return output.split()[1]
+
+
 def update_links_to_attached_files(content, attachments):
     for old_url, new_path in attachments.items():
         # url may occur both with http:// and https://
@@ -689,6 +705,14 @@ def fields2pelican(
         dircat=False, strip_raw=False, disable_slugs=False,
         dirpage=False, filename_template=None, filter_author=None,
         wp_custpost=False, wp_attach=False, attachments=None):
+
+    pandoc_version = get_pandoc_version()
+
+    if is_pandoc_needed(fields) and not pandoc_version:
+        error = ('Pandoc must be installed to complete the '
+                 'requested import action.')
+        exit(error)
+
     for (title, content, filename, date, author, categories, tags, status,
             kind, in_markup) in fields:
         if filter_author and filter_author != author:
@@ -735,11 +759,17 @@ def fields2pelican(
 
                 fp.write(new_content)
 
-            parse_raw = '--parse-raw' if not strip_raw else ''
-            cmd = ('pandoc --normalize {0} --from=html'
-                   ' --to={1} -o "{2}" "{3}"')
-            cmd = cmd.format(parse_raw, out_markup,
-                             out_filename, html_filename)
+            if pandoc_version[0] == '1':
+                parse_raw = '--parse-raw' if not strip_raw else ''
+                cmd = ('pandoc --normalize {0} --from=html'
+                       ' --to={1} -o "{2}" "{3}"')
+                cmd = cmd.format(parse_raw, out_markup,
+                                 out_filename, html_filename)
+            else:
+                from_arg = '-f html+raw_html' if not strip_raw else '-f html'
+                cmd = ('pandoc {0} --to={1}-smart -o "{2}" "{3}"')
+                cmd = cmd.format(from_arg, out_markup,
+                                 out_filename, html_filename)
 
             try:
                 rc = subprocess.call(cmd, shell=True)


Test patch:
diff --git a/pelican/tests/content/wordpressexport.xml b/pelican/tests/content/wordpressexport.xml
--- a/pelican/tests/content/wordpressexport.xml
+++ b/pelican/tests/content/wordpressexport.xml
@@ -554,7 +554,11 @@ Pelicans are supposed to eat fish, damn it!
 
 <iframe width="420" height="315" src="http://www.youtube.com/embed/QNNl_uWmQXE" frameborder="0" allowfullscreen></iframe>
 
-Bottom line: don't mess up with birds]]></content:encoded>
+Bottom line: don't mess up with birds
+
+"That's a 'wonderful' shoe."
+
+“That’s a ‘magic’ sock.”]]></content:encoded>
         <excerpt:encoded><![CDATA[]]></excerpt:encoded>
         <wp:post_id>173</wp:post_id>
         <wp:post_date>2012-02-16 15:52:55</wp:post_date>
diff --git a/pelican/tests/test_importer.py b/pelican/tests/test_importer.py
--- a/pelican/tests/test_importer.py
+++ b/pelican/tests/test_importer.py
@@ -268,6 +268,19 @@ def r(f):
             code_line = re.search(r'\s+a = \[1, 2, 3\]', md).group(0)
             self.assertTrue(sample_line.rindex('This') < code_line.rindex('a'))
 
+    def test_dont_use_smart_quotes(self):
+        def r(f):
+            with open(f, encoding='utf-8') as infile:
+                return infile.read()
+        silent_f2p = mute(True)(fields2pelican)
+        test_post = filter(
+            lambda p: p[0].startswith("Post with raw data"),
+            self.posts)
+        with temporary_folder() as temp:
+            md = [r(f) for f in silent_f2p(test_post, 'markdown', temp)][0]
+            escaped_quotes = re.search(r'\\[\'"“”‘’]', md)
+            self.assertFalse(escaped_quotes)
+
 
 class TestBuildHeader(unittest.TestCase):
     def test_build_header(self):