Repository: getnikola/nikola

Issue:
Wordpress allows case insensitive tags. nikola breaks when it tries to build them
I had `books` as well as `Books` in my tags and nikola frequently kept crashing while building the site.



Discussion:
What version of Nikola?

Was the crash when importing the wordpress data, or when building the site after it was imported?

7.7.0 and on master.
This was after the import and when building the site.

Can you show me the error? Usually it should not crash but tell you that you have two tags that produce the same slug and ask you to change them :)

@orsenthil this is what I get with two posts where tags differ only in case:

```
$ nikola build
Scanning posts....[2015-09-04T14:40:56Z] ERROR: Nikola: You have tags that are too similar: bar and Bar
[2015-09-04T14:40:56Z] ERROR: Nikola: Tag bar is used in: posts/foo.rst
[2015-09-04T14:40:56Z] ERROR: Nikola: Tag Bar is used in: posts/foo2.rst
done!
```

Yes, this is the error.

```
(nikola)[localhost new_site (wordpress_import)]$ nikola build
Scanning posts........[2015-09-05T06:03:47Z] ERROR: Nikola: You have tags that are too similar: Books and books
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag Books is used in: posts/2013/03/10/book-review-the-startup-of-you.md
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag books is used in: posts/2007/02/22/recent-books.md, posts/2007/12/31/books-i-could-read-this-year.md, posts/2008/12/13/the-agony-and-the-ecstasy.md, posts/2010/10/09/the-pleasures-and-sorrows-of-work-amazon-book-link.md
[2015-09-05T06:03:47Z] ERROR: Nikola: You have tags that are too similar: incredibles and Incredibles
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag incredibles is used in: posts/2014/06/20/the-incredibles-bill-and-melida-gates-on-humanity.md
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag Incredibles is used in: posts/2013/06/09/the-incredibles-1.md, posts/2014/02/11/codex-leicester.md
[2015-09-05T06:03:47Z] ERROR: Nikola: You have tags that are too similar: Minecraft and minecraft
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag Minecraft is used in: posts/2014/06/23/madurai-meenakshi-temple-in-minecraft-next-step-north-tower.md
[2015-09-05T06:03:47Z] ERROR: Nikola: Tag minecraft is used in: posts/2013/07/08/my-three-wishes-for-minecraft.md, posts/2014/06/23/madurai-meenakshi-temple-in-minecraft-next-step-north-tower.md
done!
(nikola)[localhost new_site (wordpress_import)]$ ls output
assets/
```

Then I’m afraid we can’t really help. This is an information for you, the site author, that you should manually clean up your posts to use only one way to capitalize your tags.  We can’t really automate the cleanups, and a wizard would not be of much help.

I acknowledge your point. Given the friendliness of Nikola,  can't this be handled transparently by nikola? why not go a step beyond and lower case all tags and build? As I stated at the start of my bug, this is from activate wordpress export, which allowed this kind of thing.

Lower casing all tags sounds like a terrible default choice. I'd rather consider it a bug of WordPress to allow different spellings of the same tags. We could add an option to the WordPress importer to apply such a conversion (everthing to lower case, or ask for a normal form per tag if two different forms are detected), though.

Asking for normal form if you different cases are detected sounds like a helpful thing to do. I tried to normalize in wordpress and it was not possible for me.  I did a sed and went ahead with the import. Asking a normal form seems like a good idea.

I wonder if the tags plugin would help here:

https://plugins.getnikola.com/#tags

Specifically:

```
--merge=ARG               Merges a list of comma-separated tags, replacing them with the last tag
                           Requires a list of file names to be passed as arguments.
                               $ nikola tags --merge "foo,bar,baz,useless" posts/*.rst
                           The above command will replace foo, bar, and baz with 'useless'
                           in all rst posts.
```

I have not tested it and it's perfectly likely that it will refuse to start because of the tag collision but maybe worth trying and see if we can adapt it to a fix.

If it doesn't work, one can also do this by hand. Assuming you only use `.meta` files, the following Python code crawls through all posts and stories, reads the `.meta` files, allows to modify the tags, and writes them back:

``` .py
#!/usr/bin/python
import io
import os
import nikola.post
import nikola.utils

folders = ['posts', 'stories']

for folder in folders:
    for root, dirs, files in os.walk(folder):
        for file in files:
            if file.endswith('.meta'):
                fn = os.path.join(root, file)
                data, ok = nikola.post.get_metadata_from_meta_file(fn)
                if ok:
                    tags = [f.strip() for f in data.get('tags', '').split(',') if f.strip()]
                    ...
                    data['tags'] = ','.join(tags)
                    data_str = nikola.utils.write_metadata(data)
                    with io.open(fn, "w+", encoding="utf8") as fd:
                        fd.write(data_str)
                else:
                    print("Loading {0} failed!".format(fn))
```

(Wrote that recently for some other quick hack.)


Resolving patch:
diff --git a/nikola/plugins/command/import_wordpress.py b/nikola/plugins/command/import_wordpress.py
--- a/nikola/plugins/command/import_wordpress.py
+++ b/nikola/plugins/command/import_wordpress.py
@@ -191,6 +191,12 @@ class CommandImportWordpress(Command, ImportMixin):
             'type': bool,
             'help': "Automatically installs the WordPress page compiler (either locally or in the new site) if required by other options.\nWarning: the compiler is GPL software!",
         },
+        {
+            'name': 'tag_saniziting_strategy',
+            'long': 'tag-saniziting-strategy',
+            'default': 'first',
+            'help': 'lower: Convert all tag and category names to lower case\nfirst: Keep first spelling of tag or category name',
+        },
     ]
     all_tags = set([])
 
@@ -239,6 +245,8 @@ def _read_options(self, options, args):
         self.install_wordpress_compiler = options.get('install_wordpress_compiler', False)
         self.wordpress_page_compiler = None
 
+        self.tag_saniziting_strategy = options.get('tag_saniziting_strategy', 'first')
+
         self.auth = None
         if options.get('download_auth') is not None:
             username_password = options.get('download_auth')
@@ -750,6 +758,24 @@ def _create_metadata(self, status, excerpt, tags, categories, post_name=None):
             tags_cats = tags + categories
         return tags_cats, other_meta
 
+    _tag_sanitize_map = {True: {}, False: {}}
+
+    def _sanitize(self, tag, is_category):
+        if self.tag_saniziting_strategy == 'lower':
+            return tag.lower()
+        if tag.lower() not in self._tag_sanitize_map[is_category]:
+            self._tag_sanitize_map[is_category][tag.lower()] = [tag]
+            return tag
+        previous = self._tag_sanitize_map[is_category][tag.lower()]
+        if self.tag_saniziting_strategy == 'first':
+            if tag != previous[0]:
+                LOGGER.warn("Changing spelling of {0} name '{1}' to {2}.".format('category' if is_category else 'tag', tag, previous[0]))
+            return previous[0]
+        else:
+            LOGGER.error("Unknown tag sanitizing strategy '{0}'!".format(self.tag_saniziting_strategy))
+            sys.exit(1)
+        return tag
+
     def import_postpage_item(self, item, wordpress_namespace, out_folder=None, attachments=None):
         """Take an item from the feed and creates a post file."""
         if out_folder is None:
@@ -837,7 +863,6 @@ def import_postpage_item(self, item, wordpress_namespace, out_folder=None, attac
                 type = tag.attrib['domain']
             if text == 'Uncategorized' and type == 'category':
                 continue
-            self.all_tags.add(text)
             if type == 'category':
                 categories.append(text)
             else:
@@ -846,6 +871,16 @@ def import_postpage_item(self, item, wordpress_namespace, out_folder=None, attac
         if '$latex' in content:
             tags.append('mathjax')
 
+        for i, cat in enumerate(categories[:]):
+            cat = self._sanitize(cat, True)
+            categories[i] = cat
+            self.all_tags.add(cat)
+
+        for i, tag in enumerate(tags[:]):
+            tag = self._sanitize(tag, False)
+            tags[i] = tag
+            self.all_tags.add(tag)
+
         # Find post format if it's there
         post_format = 'wp'
         format_tag = [x for x in item.findall('*//{%s}meta_key' % wordpress_namespace) if x.text == '_tc_post_format']


Test patch:
diff --git a/tests/test_command_import_wordpress.py b/tests/test_command_import_wordpress.py
--- a/tests/test_command_import_wordpress.py
+++ b/tests/test_command_import_wordpress.py
@@ -196,6 +196,7 @@ def test_importing_posts_and_attachments(self):
         self.import_command.export_comments = False
         self.import_command.transform_to_html = False
         self.import_command.use_wordpress_compiler = False
+        self.import_command.tag_saniziting_strategy = 'first'
         self.import_command.context = self.import_command.populate_context(
             channel)