Repository: getnikola/nikola

Issue:
IDN SITE_URL is not converted to Punycode
The SITE_URL is not converted correctly to Punycode. For example, when initialising a new Blog and writing:

Site URL [http://getnikola.com/]: http://exämple.com/täst/
, this will result in conf.py to:
SITE_URL = "http://ex\u00e4mple.com/t\u00e4st/"

Correct should be that the domain name is converted to Punycode: 
SITE_URL = "http://xn--exmple-cua.com/t\u00e4st/"

The result is that for example Firefox throws an error when clicking on the logo.

I guess that (only) the domain part needs to be isolated from SITE_URL and then converted with "exämple.com".encode("idna") to xn--exmple-cua.com.

Nikola should also keep in mind that the user may edit the SITE_URL in conf.py directly and write the IDN without punycode directly, so for example:
SITE_URL = "http://exämple.com/täst"
Therefore the Punycode convert should best be applied while building, not in the blog init.



Discussion:
Interesting. It looks easy-ish :-)

> SITE_URL = "http://exämple.com/täst"

This is equivalent to:

> SITE_URL = "http://ex\u00e4mple.com/t\u00e4st/"

To solve this, we could just `urlsplit()`, encode the domain part and `urljoin()` it back.

PS. the issue is caused by a dumb algorithm (in lxml?) that is handling links like it’s 1999:

```
<a href="http://%D0%BF%D1%80%D0%B5%D0%B7%D0%B8%D0%B4%D0%B5%D0%BD%D1%82.%D1%80%D1%84/">broken</a>
```

This is the problem.  Firefox wouldn’t mind the punycode form, and it also wouldn’t mind the real unicode:

```
<a href="http://xn--d1abbgf6aiiy.xn--p1ai/">works</a>
<a href="http://президент.рф/">works too</a>
```

On a side note, Chrome supports the percent-escaped link, IE and Safari also fail.

test: https://dl.dropboxusercontent.com/u/1933476/IDN.html

_This is **not** a statement of support for the Russian Federation_

As a side node, maybe it would be good to write the variables in conf.py direclty in UTF-8. Escaping everything reduces readability to 0 and is not necessary, because conf.py's encoding is given as utf8 either way.

Done in fb3a7db.  Requires UTF-8 input for this to work.

Found that the bug has a slightly larger impact when using Isso as a comment system. "script src" in the output html file will be incorrect and then the comment file is not loaded, and comments don't work.

Solution/workaround: Same as above, write the Domain in Punycode in COMMENT_SYSTEM_ID.

It looks like the best solution would be to fix things in `nikola init` and require Punycode in the config file (don’t allow to build if Unicode is found in the domain).

Well, in my (humble) opinion, it is best if Nikola would not convert the UTF-8 to anything: No Punycoding, no escaping. All current web browsers should understand URLs like http://президент.рф/президент.html. Therefore I don't see the necessity to convert this to http://xn--d1abbgf6aiiy.xn--p1ai/%D0%BF%D1%80%D0%B5%D0%B7%D0%B8%D0%B4%D0%B5%D0%BD%D1%82.html.

It would greatly improve readability if no conversions would be made. First, in conf.py: If you actually want to read what you have entered in SITE_URL, non-Punycode is better. The HTML sourcecode is, of course, not a problem, but this would be easier to read as well (and save some bytes :D)

... but of course I understand that this may need extreme overhaul. 

We can’t fix this on our own.  lxml or one of _their_ upstreams can — talk to the appropriate vendor if you want this fixed nicely.

Ah ok, sorry for the misunderstanding :) I'll try it.

@Kwpolska so, if I understand this correctly there's nothing more we can do? Close it?

@ralsina Possible solutions include:

(a) trying to get the link replacer to fix this (which will probably not fix everything);  
(b) fixing things in nikola init and warning users/failing to build if Unicode characters are encountered in SITE_URL.

Which one do we choose?

I'd say b) which looks much easier.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -37,9 +37,9 @@
 import sys
 import mimetypes
 try:
-    from urlparse import urlparse, urlsplit, urljoin
+    from urlparse import urlparse, urlsplit, urlunsplit, urljoin, unquote
 except ImportError:
-    from urllib.parse import urlparse, urlsplit, urljoin  # NOQA
+    from urllib.parse import urlparse, urlsplit, urlunsplit, urljoin, unquote  # NOQA
 
 try:
     import pyphen
@@ -625,6 +625,15 @@ def __init__(self, **config):
             utils.LOGGER.warn("Your BASE_URL doesn't end in / -- adding it, but please fix it in your config file!")
             self.config['BASE_URL'] += '/'
 
+        try:
+            _bnl = urlsplit(self.config['BASE_URL']).netloc
+            _bnl.encode('ascii')
+            urlsplit(self.config['SITE_URL']).netloc.encode('ascii')
+        except (UnicodeEncodeError, UnicodeDecodeError):
+            utils.LOGGER.error("Your BASE_URL or SITE_URL contains an IDN expressed in Unicode.  Please convert it to Punycode.")
+            utils.LOGGER.error("Punycode of {}: {}".format(_bnl, _bnl.encode('idna')))
+            sys.exit(1)
+
         # todo: remove in v8
         if not isinstance(self.config['DEPLOY_COMMANDS'], dict):
             utils.LOGGER.warn("A single list as DEPLOY_COMMANDS is deprecated.  DEPLOY_COMMANDS should be a dict, with deploy preset names as keys and lists of commands as values.")
@@ -984,6 +993,24 @@ def url_replacer(self, src, dst, lang=None, url_type=None):
             if dst_url.scheme == 'link':  # Magic link
                 dst = self.link(dst_url.netloc, dst_url.path.lstrip('/'), lang)
             else:
+                print(dst)
+                if '%' in dst_url.netloc:
+                    # convert lxml percent-encoded garbage to punycode
+                    nl = unquote(dst_url.netloc)
+                    try:
+                        nl = nl.decode('utf-8')
+                    except AttributeError:
+                        # python 3: already unicode
+                        pass
+
+                    nl = nl.encode('idna')
+
+                    dst = urlunsplit((dst_url.scheme,
+                                      nl,
+                                      dst_url.path,
+                                      dst_url.query,
+                                      dst_url.fragment))
+                    print(dst)
                 return dst
         elif dst_url.scheme == 'link':  # Magic absolute path link:
             dst = dst_url.path
diff --git a/nikola/plugins/command/init.py b/nikola/plugins/command/init.py
--- a/nikola/plugins/command/init.py
+++ b/nikola/plugins/command/init.py
@@ -39,7 +39,7 @@
 import tarfile
 
 import nikola
-from nikola.nikola import DEFAULT_TRANSLATIONS_PATTERN, DEFAULT_INDEX_READ_MORE_LINK, DEFAULT_RSS_READ_MORE_LINK, LEGAL_VALUES
+from nikola.nikola import DEFAULT_TRANSLATIONS_PATTERN, DEFAULT_INDEX_READ_MORE_LINK, DEFAULT_RSS_READ_MORE_LINK, LEGAL_VALUES, urlsplit, urlunsplit
 from nikola.plugin_categories import Command
 from nikola.utils import ask, ask_yesno, get_logger, makedirs, STDERR_HANDLER, load_messages
 from nikola.packages.tzlocal import get_localzone
@@ -261,6 +261,36 @@ def create_empty_site(cls, target):
     @staticmethod
     def ask_questions(target):
         """Ask some questions about Nikola."""
+        def urlhandler(default, toconf):
+            answer = ask('Site URL', 'http://getnikola.com/')
+            try:
+                answer = answer.decode('utf-8')
+            except (AttributeError, UnicodeDecodeError):
+                pass
+            if not answer.startswith(u'http'):
+                print("    ERROR: You must specify a protocol (http or https).")
+                urlhandler(default, toconf)
+                return
+            if not answer.endswith('/'):
+                print("    The URL does not end in '/' -- adding it.")
+                answer += '/'
+
+            dst_url = urlsplit(answer)
+            try:
+                dst_url.netloc.encode('ascii')
+            except (UnicodeEncodeError, UnicodeDecodeError):
+                # The IDN contains characters beyond ASCII.  We must convert it
+                # to Punycode. (Issue #1644)
+                nl = dst_url.netloc.encode('idna')
+                answer = urlunsplit((dst_url.scheme,
+                                     nl,
+                                     dst_url.path,
+                                     dst_url.query,
+                                     dst_url.fragment))
+                print("    Converting to Punycode:", answer)
+
+            SAMPLE_CONF['SITE_URL'] = answer
+
         def lhandler(default, toconf, show_header=True):
             if show_header:
                 print("We will now ask you to provide the list of languages you want to use.")
@@ -375,7 +405,7 @@ def chandler(default, toconf):
             ('Site author', 'Nikola Tesla', True, 'BLOG_AUTHOR'),
             ('Site author\'s e-mail', 'n.tesla@example.com', True, 'BLOG_EMAIL'),
             ('Site description', 'This is a demo site for Nikola.', True, 'BLOG_DESCRIPTION'),
-            ('Site URL', 'http://getnikola.com/', True, 'SITE_URL'),
+            (urlhandler, None, True, True),
             ('Questions about languages and locales', None, None, None),
             (lhandler, None, True, True),
             (tzhandler, None, True, True),


Test patch: