Repository: getnikola/nikola

Issue:
Wordpress importer doesn't translate [code]
In my Wordpress block, I made good use of their code blocks, which are delimited like 

```
[code]
code here
[/code]
```

or, to use a language

```
[code language="py"]
python_code_here
[/code]
```

(I think `language` can also be spelled just `lang`). 

If the wordpress importer uses Markdown, it could just translate `[code]` to ```, `[code lang='x']` to ```x, and enable fenced code blocks.



Discussion:
This is a wordpress.com-only thing, documented here: http://en.support.wordpress.com/code/posting-source-code/

I just did a simple regex replace of `\[/?code( lang(uage)?="(.*)")?\]` with ```\3. It would have been nice if the importer did this for me, though. 

One issue with it was that wordpress replaced things like `>` with `&gt;`. So I guess it maybe needs to be smarter than this, using some `<pre>` environment that calls directly to the syntax highlighter. 

We’d need a smarter parser, because:
- `lang(uage)` is not required, it can be just `[code]`
- there are other meta fields we do not support
- `&lt; &gt; &amp;` replacements would break it

This requires some work and time, though.  Someone might take care of this, one day…

> lang(uage) is not required, it can be just [code]

That is handled by my regular expression

The other problem I found is that there are blank lines between each line in `[code]` blocks. 

There is already this https://github.com/getnikola/nikola/blob/master/nikola/plugins/command/import_wordpress.py#L352, but I don't understand what it is doing. What does the `~~~~~~~~~~~` do?

probably some other form of fenced code blocks in python-markdown.  (it’s also seemingly for _another_ code-block extension.  wordpress is a fucking mess.)

I am willing to take a shot at fixing this, since I have to do it anyway for my imported blog. 

Understood.  Go on, and have fun.  (make sure to read the documentation for the wordpress feature, linked above.)

I'll probably end up leaving the configuration parameters unimplemented, as I didn't use them.

s/unimplemented/ignored/

s/I didn't/Nikola doesn’t/

The extra newlines are only appearing in my oldest blog posts. So probably that was some Wordpress thing that was fixed. There's no reliable way to detect the issue programmatically (unless we have some date that we know it changed). 

OK, so I guess the only question I have is, what is the best way to convert things like `&lt;` back to `<`?

> s/unimplemented/ignored/

I was actually planning on unimplemented, but I guess I can do ignored too. I just need to make my regex a little more general. 

Just happily ignore the newline issue, as users are meant to review the output anyways.

For converting, you’d need to know that you are in a `[code]` block and do replacements there.  But that is not easy, not without some hardcore parsing.  (though I can think of a regexp cheat that goes like `[code(.*?)](.*?)&lt;(.*?)[/code]` → `[code\1]\2<\3[/code]` — though this might very well fail, especially in corner cases, like `]<[` or `]aa<[` or `]<aa[`.

I mean, is there a `&whatever;` to the appropriate actual character mapping somewhere (standard library, external library, already in Nikola, ...)? 

There are three you need to take care of:

```
&amp; = &
&lt;  = <
&gt;  = >
```

Ah, the rest are just literal? I found http://www.ascii.cl/htmlcodes.htm. I think I also need `&quot;`.


Resolving patch:
diff --git a/nikola/plugins/command/import_wordpress.py b/nikola/plugins/command/import_wordpress.py
--- a/nikola/plugins/command/import_wordpress.py
+++ b/nikola/plugins/command/import_wordpress.py
@@ -335,13 +335,30 @@ def download_additional_image_sizes(self, item, wordpress_namespace, source_path
                     links[url] = '/' + dst_url
                     links[url] = '/' + dst_url
 
-    @staticmethod
-    def transform_sourcecode(content):
-        new_content = re.sub('\[sourcecode language="([^"]+)"\]',
-                             "\n~~~~~~~~~~~~{.\\1}\n", content)
-        new_content = new_content.replace('[/sourcecode]',
-                                          "\n~~~~~~~~~~~~\n")
-        return new_content
+    code_re1 = re.compile(r'\[code.* lang.*?="(.*?)?".*\](.*?)\[/code\]', re.DOTALL | re.MULTILINE)
+    code_re2 = re.compile(r'\[sourcecode.* lang.*?="(.*?)?".*\](.*?)\[/sourcecode\]', re.DOTALL | re.MULTILINE)
+    code_re3 = re.compile(r'\[code.*?\](.*?)\[/code\]', re.DOTALL | re.MULTILINE)
+    code_re4 = re.compile(r'\[sourcecode.*?\](.*?)\[/sourcecode\]', re.DOTALL | re.MULTILINE)
+
+    def transform_code(self, content):
+        # http://en.support.wordpress.com/code/posting-source-code/. There are
+        # a ton of things not supported here. We only do a basic [code
+        # lang="x"] -> ```x translation, and remove quoted html entities (<,
+        # >, &, and ").
+        def replacement(m):
+            language = m.group(1) or ''
+            code = m.group(2)
+            code = code.replace('&amp;', '&')
+            code = code.replace('&gt;', '>')
+            code = code.replace('&lt;', '<')
+            code = code.replace('&quot;', '"')
+            return '```{language}\n{code}\n```'.format(language=language, code=code)
+
+        content = self.code_re1.sub(replacement, content)
+        content = self.code_re2.sub(replacement, content)
+        content = self.code_re3.sub(replacement, content)
+        content = self.code_re4.sub(replacement, content)
+        return content
 
     @staticmethod
     def transform_caption(content):
@@ -358,10 +375,10 @@ def transform_multiple_newlines(self, content):
             return content
 
     def transform_content(self, content):
-        new_content = self.transform_sourcecode(content)
-        new_content = self.transform_caption(new_content)
-        new_content = self.transform_multiple_newlines(new_content)
-        return new_content
+        content = self.transform_code(content)
+        content = self.transform_caption(content)
+        content = self.transform_multiple_newlines(content)
+        return content
 
     def import_item(self, item, wordpress_namespace, out_folder=None):
         """Takes an item from the feed and creates a post file."""


Test patch:
diff --git a/tests/test_command_import_wordpress.py b/tests/test_command_import_wordpress.py
--- a/tests/test_command_import_wordpress.py
+++ b/tests/test_command_import_wordpress.py
@@ -228,13 +228,13 @@ def test_importing_posts_and_attachments(self):
 
 Some source code.
 
-~~~~~~~~~~~~{.Python}
+```Python
 
 import sys
 
 print sys.version
 
-~~~~~~~~~~~~
+```
 
 The end.
 
@@ -302,16 +302,16 @@ def test_importing_posts_and_attachments(self):
 
     def test_transforming_content(self):
         """Applying markup conversions to content."""
-        transform_sourcecode = mock.MagicMock()
+        transform_code = mock.MagicMock()
         transform_caption = mock.MagicMock()
         transform_newlines = mock.MagicMock()
 
-        with mock.patch('nikola.plugins.command.import_wordpress.CommandImportWordpress.transform_sourcecode', transform_sourcecode):
+        with mock.patch('nikola.plugins.command.import_wordpress.CommandImportWordpress.transform_code', transform_code):
             with mock.patch('nikola.plugins.command.import_wordpress.CommandImportWordpress.transform_caption', transform_caption):
                 with mock.patch('nikola.plugins.command.import_wordpress.CommandImportWordpress.transform_multiple_newlines', transform_newlines):
                     self.import_command.transform_content("random content")
 
-        self.assertTrue(transform_sourcecode.called)
+        self.assertTrue(transform_code.called)
         self.assertTrue(transform_caption.called)
         self.assertTrue(transform_newlines.called)
 
@@ -325,21 +325,18 @@ def test_transforming_source_code(self):
 print sys.version
 [/sourcecode]"""
 
-        content = self.import_command.transform_sourcecode(content)
+        content = self.import_command.transform_code(content)
 
         self.assertFalse('[/sourcecode]' in content)
         self.assertFalse('[sourcecode language=' in content)
 
         replaced_content = """Hello World.
-
-~~~~~~~~~~~~{.Python}
+```Python
 
 import sys
 print sys.version
 
-~~~~~~~~~~~~
-"""
-
+```"""
         self.assertEqual(content, replaced_content)
 
     def test_transform_caption(self):