Repository: getnikola/nikola

Issue:
import_wordpress write_attachments_info failed
i have one article with an attachment site. import of this don't work. (the last `<wp:postmeta>` is copied from the actual data into this test installation)

```
$ rm -r new_site/ ; nikola import_wordpress --no-downloads /Users/davidak/sync-arbeit/demosite.wordpress.2016-02-23-attachment-page.xml
[2016-02-23T19:01:34Z] INFO: init: Created empty site at new_site.
[2016-02-23T19:01:34Z] INFO: import_wordpress: Downloading http://127.0.0.1:8081/wp-content/uploads/2016/02/first-test.png => new_site/files/wp-content/uploads/2016/02/first-test.png
[2016-02-23T19:01:34Z] INFO: import_wordpress: Downloading http://127.0.0.1:8081/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png => new_site/files/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png
[2016-02-23T19:01:34Z] INFO: import_wordpress: Downloading http://127.0.0.1:8081/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png => new_site/files/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png
[2016-02-23T19:01:34Z] INFO: import_wordpress: Downloading http://127.0.0.1:8081/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png => new_site/files/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png
{5: {'date_utc': '2016-02-23 18:09:53', 'wordpress_user_name': 'admin', 'files_meta': [{'height': 800, 'width': 1280}, {'size': 'medium', 'width': b'300', 'height': b'187'}, {'size': 'large', 'width': b'1024', 'height': b'640'}, {'size': 'thumbnail', 'width': b'150', 'height': b'150'}], 'files': ['/wp-content/uploads/2016/02/first-test.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png'], 'title': 'first-test'}}
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/doit/doit_cmd.py", line 168, in run
    return command.parse_execute(args)
  File "/usr/local/lib/python3.5/site-packages/doit/cmd_base.py", line 122, in parse_execute
    return self.execute(params, args)
  File "/usr/local/lib/python3.5/site-packages/nikola/plugin_categories.py", line 124, in execute
    return self._execute(options, args)
  File "/usr/local/lib/python3.5/site-packages/nikola/plugins/command/import_wordpress.py", line 332, in _execute
    self.import_posts(channel)
  File "/usr/local/lib/python3.5/site-packages/nikola/plugins/command/import_wordpress.py", line 1016, in import_posts
    self.process_item_if_post_or_page(item)
  File "/usr/local/lib/python3.5/site-packages/nikola/plugins/command/import_wordpress.py", line 1006, in process_item_if_post_or_page
    self.write_attachments_info(destination, attachments)
  File "/usr/local/lib/python3.5/site-packages/nikola/plugins/command/import_wordpress.py", line 986, in write_attachments_info
    file.write(json.dumps(attachments).encode('utf-8'))
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/encoder.py", line 180, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'300' is not JSON serializable
```

i add some debug output:

```
    def write_attachments_info(self, path, attachments):
        """Write attachments info file."""
        with io.open(path, "wb") as file:
            print(attachments)
            file.write(json.dumps(attachments).encode('utf-8'))
```

and then manually tested the json without the problem:

```
$ python3
Python 3.5.1 (default, Jan 22 2016, 08:52:08)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> a="{5: {'date_utc': '2016-02-23 18:09:53', 'wordpress_user_name': 'admin', 'files_meta': [{'height': 800, 'width': 1280}, {'size': 'medium', 'width': b'300', 'height': b'187'}, {'size': 'large', 'width': b'1024', 'height': b'640'}, {'size': 'thumbnail', 'width': b'150', 'height': b'150'}], 'files': ['/wp-content/uploads/2016/02/first-test.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png'], 'title': 'first-test'}}"
>>> json.dumps(a).encode('utf-8')
b'"{5: {\'date_utc\': \'2016-02-23 18:09:53\', \'wordpress_user_name\': \'admin\', \'files_meta\': [{\'height\': 800, \'width\': 1280}, {\'size\': \'medium\', \'width\': b\'300\', \'height\': b\'187\'}, {\'size\': \'large\', \'width\': b\'1024\', \'height\': b\'640\'}, {\'size\': \'thumbnail\', \'width\': b\'150\', \'height\': b\'150\'}], \'files\': [\'/wp-content/uploads/2016/02/first-test.png\', \'/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png\', \'/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png\', \'/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png\'], \'title\': \'first-test\'}}"'
```



Discussion:
this json is ok:
`{5: {'files_meta': [{'height': 540, 'width': 960}, {'size': 'medium_large', 'height': 432, 'width': 768}, {'size': 'medium', 'height': 169, 'width': 300}, {'size': 'thumbnail', 'height': 150, 'width': 150}], 'title': 'first-test', 'wordpress_user_name': 'admin', 'files': ['/wp-content/uploads/2016/02/first-test.png', '/wp-content/uploads/2016/02/first-test-768x432.png', '/wp-content/uploads/2016/02/first-test-300x169.png', '/wp-content/uploads/2016/02/first-test-150x150.png'], 'date_utc': '2016-02-23 18:09:53'}}`

source:
`a:5:{s:5:"width";i:960;s:6:"height";i:540;s:4:"file";s:22:"2016/02/first-test.png";s:5:"sizes";a:3:{s:9:"thumbnail";a:4:{s:4:"file";s:22:"first-test-150x150.png";s:5:"width";i:150;s:6:"height";i:150;s:9:"mime-type";s:9:"image/png";}s:6:"medium";a:4:{s:4:"file";s:22:"first-test-300x169.png";s:5:"width";i:300;s:6:"height";i:169;s:9:"mime-type";s:9:"image/png";}s:12:"medium_large";a:4:{s:4:"file";s:22:"first-test-768x432.png";s:5:"width";i:768;s:6:"height";i:432;s:9:"mime-type";s:9:"image/png";}}s:10:"image_meta";a:12:{s:8:"aperture";s:1:"0";s:6:"credit";s:0:"";s:6:"camera";s:0:"";s:7:"caption";s:0:"";s:17:"created_timestamp";s:1:"0";s:9:"copyright";s:0:"";s:12:"focal_length";s:1:"0";s:3:"iso";s:1:"0";s:13:"shutter_speed";s:1:"0";s:5:"title";s:0:"";s:11:"orientation";s:1:"0";s:8:"keywords";a:0:{}}}`

---

this is not ok:
`{5: {'files_meta': [{'height': 800, 'width': 1280}, {'height': b'150', 'size': 'thumbnail', 'width': b'150'}, {'height': b'640', 'size': 'large', 'width': b'1024'}, {'height': b'187', 'size': 'medium', 'width': b'300'}], 'wordpress_user_name': 'admin', 'date_utc': '2016-02-23 18:09:53', 'files': ['/wp-content/uploads/2016/02/first-test.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png', '/wp-content/uploads/2016/02/Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png'], 'title': 'first-test'}}`

source:
`a:6:{s:5:"width";s:4:"1280";s:6:"height";s:3:"800";s:14:"hwstring_small";s:23:"height='80' width='128'";s:4:"file";s:49:"2010/12/Bildschirmfoto-2010-12-14-um-19.47.29.png";s:5:"sizes";a:3:{s:9:"thumbnail";a:3:{s:4:"file";s:49:"Bildschirmfoto-2010-12-14-um-19.47.29-150x150.png";s:5:"width";s:3:"150";s:6:"height";s:3:"150";}s:6:"medium";a:3:{s:4:"file";s:49:"Bildschirmfoto-2010-12-14-um-19.47.29-300x187.png";s:5:"width";s:3:"300";s:6:"height";s:3:"187";}s:5:"large";a:3:{s:4:"file";s:50:"Bildschirmfoto-2010-12-14-um-19.47.29-1024x640.png";s:5:"width";s:4:"1024";s:6:"height";s:3:"640";}}s:10:"image_meta";a:10:{s:8:"aperture";s:1:"0";s:6:"credit";s:0:"";s:6:"camera";s:0:"";s:7:"caption";s:0:"";s:17:"created_timestamp";s:1:"0";s:9:"copyright";s:0:"";s:12:"focal_length";s:1:"0";s:3:"iso";s:1:"0";s:13:"shutter_speed";s:1:"0";s:5:"title";s:0:"";}}`

difference:
`i:150`
`s:3:"150"`

the problem are the `"` in the source.

the post is from Dec 2010, maybe the wordpress version produced such php arrays...

how could we make this more robust?

A filter for height and width, or some decoding, seems to be needed here: https://github.com/getnikola/nikola/blob/master/nikola/plugins/command/import_wordpress.py#L518

wow, i just realized how php serialization works:

`s:6:"height"` is a string with 6 chars ans `i:150` is an integer.

so the data is incorrect, not the importer.
no problems after changing manually.

i created a bugreport for wordpress https://core.trac.wordpress.org/ticket/35927#ticket

strange is, that the unittest works.
i successfully run it locally and Travis has also no errors.

but when i try to import your included export, it also fails.

https://github.com/getnikola/nikola/blob/master/tests/wordpress_export_example.xml#L56
`s:5:"width";s:3:"300"`

i am on OS X, tested with 10.10 and 10.9.5.
can you reproduce it? and what system do you have?

I also get this exception when I run `nikola import_wordpress wordpress_export_example.xml` in `nikola/tests`. I guess the unit tests disable downloading stuff, because if I add `--no-downloads` it works fine.


Resolving patch:
diff --git a/nikola/plugins/command/import_wordpress.py b/nikola/plugins/command/import_wordpress.py
--- a/nikola/plugins/command/import_wordpress.py
+++ b/nikola/plugins/command/import_wordpress.py
@@ -562,8 +562,8 @@ def add(our_key, wp_key, is_int=False, ignore_zero=False, is_float=False):
                         meta = {}
                         meta['size'] = size.decode('utf-8')
                         if width_key in metadata[size_key][size] and height_key in metadata[size_key][size]:
-                            meta['width'] = metadata[size_key][size][width_key]
-                            meta['height'] = metadata[size_key][size][height_key]
+                            meta['width'] = int(metadata[size_key][size][width_key])
+                            meta['height'] = int(metadata[size_key][size][height_key])
 
                         path = urlparse(url).path
                         dst_path = os.path.join(*([self.output_folder, 'files'] + list(path.split('/'))))


Test patch: