Repository: getnikola/nikola

Issue:
Disable asciination of titles/slugs
Hello,

First of all, of course thanks for the great application!

Currently, Nikola converts every Non-ASCII title to ASCII only slugs. I actually have a hard time seeing the benefit of that. Most languages suffer quite a bit if they are converted to ASCII. And is this still really necessary? In 2014, many web servers have no problem displaying non-ascii filenames and for webbrowsers, it is absolutely no problem at all. Take a look at Wikipedia, there are non-ascii file names all the way.

The other problem is that many languages suffer quite severely during the conversation. Languages with the latin alphabet may only look weird, but for example Japanese is converted horribly (it is converted in Chinese style, which is completely uncomprehensable to a Japanese). The current Japanese conversation in effect gives the same result as if there were only random characters in the file name.

For example, if I have a post with the Japanese title 初めての日本語の記事 (_hajimete no nihongo no kiji_), I get the following slug: `posts/chu-metenori-ben-yu-noji-shi.rst`

I don’t know, this is so wrong it even hurts :D

It would be really good if this feature could be disabled. Readable titles are, in my opinion, really a must-have feature :)

Thank you :)



Discussion:
Just change the slug yourself.  We would have to do magic for all this, because some people (and one of them might be named _Windows_) might struggle here or there.  I am all in favor of _it’s 2014_ in most cases, but Windows — and some misconfigured Unices — might not let us here.

Yes, I am doing that right now, but well... :D

Would it maybe be possible to include a optional configuration option in conf.py, if it is not too much work? I am not so versed with the Nikola source code, but isn't it possible to check like this:

``` python
if convert_titles:
    slug = convert_slug(title)
else:
    slug = title
```

?

Of course, no problem if this is not done, it was just a proposal from my side :)

Certainly doable.  And might be done, soon.  (it’s about 5 minutes’ work, and that’s quite generous).

Wow, great, thank you so much :) I know that ascii is sometimes necessary, but if the environment completely supports unicode, I guess it should not be such a great problem.

Thanks!


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -330,6 +330,7 @@ def __init__(self, **config):
             'USE_CDN': False,
             'USE_FILENAME_AS_TITLE': True,
             'USE_OPEN_GRAPH': True,
+            'USE_SLUGIFY': True,
             'TIMEZONE': 'UTC',
             'DEPLOY_DRAFTS': True,
             'DEPLOY_FUTURE': False,
@@ -400,6 +401,9 @@ def __init__(self, **config):
         # We provide the arguments to format in CONTENT_FOOTER_FORMATS.
         self.config['CONTENT_FOOTER'].langformat(self.config['CONTENT_FOOTER_FORMATS'])
 
+        # propagate USE_SLUGIFY
+        utils.USE_SLUGIFY = self.config['USE_SLUGIFY']
+
         # Make sure we have pyphen installed if we are using it
         if self.config.get('HYPHENATE') and pyphen is None:
             utils.LOGGER.warn('To use the hyphenation, you have to install '
@@ -1220,7 +1224,7 @@ def scan_posts(self, really=False):
                                     utils.LOGGER.error('Tag {0} is used in: {1}'.format(other_tag, ', '.join([p.source_path for p in self.posts_per_tag[other_tag]])))
                                     quit = True
                             else:
-                                slugged_tags.add(utils.slugify(tag))
+                                slugged_tags.add(utils.slugify(tag, force=True))
                             self.posts_per_tag[tag].append(post)
                         self.posts_per_category[post.meta('category')].append(post)
                     else:
diff --git a/nikola/plugins/task/listings.py b/nikola/plugins/task/listings.py
--- a/nikola/plugins/task/listings.py
+++ b/nikola/plugins/task/listings.py
@@ -73,7 +73,7 @@ def render_listing(in_name, out_name, folders=[], files=[]):
                     code = highlight(fd.read(), lexer,
                                      HtmlFormatter(cssclass='code',
                                                    linenos="table", nowrap=False,
-                                                   lineanchors=utils.slugify(in_name),
+                                                   lineanchors=utils.slugify(in_name, force=True),
                                                    anchorlinenos=True))
                 # the pygments highlighter uses <div class="codehilite"><pre>
                 # for code.  We switch it to reST's <pre class="code">.
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -85,6 +85,8 @@ def get_logger(name, handlers):
 LOGGER = get_logger('Nikola', STDERR_HANDLER)
 STRICT_HANDLER = ExceptionHandler(ApplicationWarning, level='WARNING')
 
+USE_SLUGIFY = True
+
 # This will block out the default handler and will hide all unwanted
 # messages, properly.
 logbook.NullHandler().push_application()
@@ -672,7 +674,7 @@ def remove_file(source):
 _slugify_hyphenate_re = re.compile(r'[-\s]+')
 
 
-def slugify(value):
+def slugify(value, force=False):
     """
     Normalizes string, converts to lowercase, removes non-alpha characters,
     and converts spaces to hyphens.
@@ -691,11 +693,26 @@ def slugify(value):
     """
     if not isinstance(value, unicode_str):
         raise ValueError("Not a unicode object: {0}".format(value))
-    value = unidecode(value)
-    # WARNING: this may not be python2/3 equivalent
-    # value = unicode(_slugify_strip_re.sub('', value).strip().lower())
-    value = str(_slugify_strip_re.sub('', value).strip().lower())
-    return _slugify_hyphenate_re.sub('-', value)
+    if USE_SLUGIFY or force:
+        # This is the standard state of slugify, which actually does some work.
+        # It is the preferred style, especially for Western languages.
+        value = unidecode(value)
+        value = str(_slugify_strip_re.sub('', value).strip().lower())
+        return _slugify_hyphenate_re.sub('-', value)
+    else:
+        # This is the “disarmed” state of slugify, which lets the user
+        # have any character they please (be it regular ASCII with spaces,
+        # or another alphabet entirely).  This might be bad in some
+        # environments, and as such, USE_SLUGIFY is better off being True!
+
+        # We still replace some characters, though.  In particular, we need
+        # to replace ? and #, which should not appear in URLs, and some
+        # Windows-unsafe characters.  This list might be even longer.
+        rc = '/\\?#"\'\r\n\t*:<>|"'
+
+        for c in rc:
+            value = value.replace(c, '-')
+        return value
 
 
 def unslugify(value, discard_numbers=True):


Test patch: