Repository: mkdocs/mkdocs

Issue:
Remove unicode_literals import in the CLI code
Looks like this can cause some issues with Click, I've not seen any but we should probably remove it anyway or we will start to get warnings from Click 5.0.

https://github.com/mitsuhiko/click/commit/5f337705f68bdfa66d7c7a9fe7fc5d6bfd48db94



Discussion:
Groan. Everything passed to MkDocs from the CLI should (must?) be in Unicode. However, without `unicode_literals` how can we get Unicode in all supported Python versions? I know that more recent Python 3 versions have restored support for `u""` syntax (which Click is recommending be used instead of `unicode_literals`), but MkDocs still supports Python versions which do not support  the `u""` syntax. Therefore, we can't just go through the `cli.py` file and add `u` prefixes to all strings. 

I would suppose the following options are available:
1. Use `unicode_literals` and turn off warnings (see below)
2. Drop support for all Python versions which do not support the `u""` syntax and preface all strings with `u` in cli.py. 
3. Drop support for all Python 2 versions
4. Wrap each string in a call to `util.text_type`. 
5. Only support `click<5.0`

I see that the diff to Click includes the ability to turn off the warning (option 1 above):

``` python
import click
click.disable_unicode_literals_warning = False
...
```

Having read Clicks reasoning for the change and considering how MkDocs is (currently) using Click, I don't expect that any of the alleged bugs will be a concern for MkDocs. However, MkDocs could start using additional features which will raise issues in the future and the option to disable the warning could be removed in a future version of Click. Although, perhaps by then MkDocs will have dropped support for any Python 2 versions and all strings will be all Unicode all the time anyway.

Perhaps the way forward is to remove the use of `unicode_literals` in `cli.py` and wrap every single string in `util.text_type` (option 4 above). It seems ridiculous to me that it has to be done that way, but the whole Python2/Python3 Unicode thing has been pretty ridiculous, so I suppose I shouldn't be surprised.

I'm assuming that with this change, Click is probably dropping support for all Python versions which do not support the `u""` syntax (option 2 above). So maybe it makes more sense for MkDocs to do the same.

Interestingly, option 5 is the easiest. Require that Click versions less that 5.0 be used. That could always be reevaluated in the future when MkDocs is ready to drop support for additional Python versions.

Seems like there is an argument for every option except 3. Actually, that would be the easiest, but given the hesitation to drop support for 2.,6, I expect dropping support for all Python 2 versions is a non-starter. Although, doing so would certainly make the entire problem go away and there would be consistent Unicode handling across all support Python versions.

I recommend either option 4 or 5 and could live with any of the others. Note that I can live with options 2 or 3 because I always have multiple versions available to me and can build in whatever version of Python which might be supported (in a virtualenv created specifically for that purpose). However, I realize that that is not likely to be the situation for most users. 

> but MkDocs still supports Python versions which do not support the u"" syntax. Therefore, we can't just go through the cli.py file and add u prefixes to all strings.

All versions we support have the `u''` syntax. 2.6, 2.7, 3.3 and 3.4. I am not sure we have explicitly stated that we don't support 3.0-3.2 (other than the classifiers). They are not tested and I have no idea if they work.

Okay, I couldn't remember offhand if MkDocs supports 3.2 or whether 3.3 supports the `u""` syntax. But if all supported Python versions have support for the feature, then I suppose that is the way to go. I can say with some certainty that 3.0-3.2 do not support it, so if users out there are running MkDocs in those versions, this change will definitely be a breaking change for them (if something else hasn't already broken it for them).

Is there some technical reason they won't/don't/can't support `unicode_literals` in click?

I don't know if it is already done, but if you have to drop `unicode_literals` in the click file, I would just isolate the click/command line parsing in a file by itself; keep all the other code with `unicode_literals`.

Edit: Maybe because they support old python versions?

@facelessuser the command line stuff (using Click) is already all in one file (`cli.py`). So, yes, the assumption is that the changes would be made to that file only. 

As far as why Click is making that change, you need to follow the links provided above to read about it. As I understand it, `unicode_literals` can be buggy in certain situations involving reading files on some systems. However, I don't believe that MkDocs currently uses those features of Click (MkDocs uses Click `Path` objects and MkDocs opens the file itself rather than using Click `File` objects in which Click opens and reads the file). This is basically an issue of how MkDocs is going to avoid having Click raise the warning while not changing MkDocs behavior.

>  I am not sure we have explicitly stated that we don't support 3.0-3.2 (other than the classifiers).

The frontpage of mkdocs.org states:

```
MkDocs supports Python 2.6, 2.7, 3.3 and 3.4.
```

Just turn the warning off.

> @facelessuser the command line stuff (using Click) is already all in one file (cli.py). So, yes, the assumption is that the changes would be made to that file only.

Yeah, I looked it up, just didn't get around to updating the post.

> As far as why Click is making that change, you need to follow the links provided above to read about it. 

Yeah, I did; it wasn't very obvious, but thanks for your summary.

> This is basically an issue of how MkDocs is going to avoid having Click raise the warning while not changing MkDocs behavior.

Totally understand, I was just curious about the technical limitations.  Seems like a rather annoying flaw in click.

@tomchristie If I may ask, why are you against updating to support explicit `u''` ?

For anyone who wants more details about why Click is making this change, see this page: http://click.pocoo.org/4/python3/. 

> For anyone who wants more details about why Click is making this change, see this page: http://click.pocoo.org/4/python3/.

Thanks, this explains the problem quite well.  Very informative.

> Thanks, this explains the problem quite well. Very informative.

I'm still unclear on how that page addresses the particular issue of unicode literals by default in the source code interacting with Click. Commented here to see if its worth clarifying or not, but not a big deal either way:

https://github.com/mitsuhiko/click/commit/5f337705f68bdfa66d7c7a9fe7fc5d6bfd48db94#commitcomment-12526928

> I'm still unclear on how that page addresses the particular issue of unicode literals by default in the source code interacting with Click. 

I think it explains Unicode issues in general.  It seems there is a complaint about Python3 handling of the stdin and such.  And I assumed `unicode_literals` gives a similar behavior in Python2 which is undesirable.   Now I haven't taken much time to fact check any of this, but that was my initial impression.  But I was more satisfied that there was **some** kind of explanation opposed to the vague blurb I read before.

This is what y'all are looking for :) http://click.pocoo.org/dev/python3/#unicode-literals

It isn't in the Click 4 docs, only the dev version.

> This is what y'all are looking for :) http://click.pocoo.org/dev/python3/#unicode-literals

So apparently the primary issue is that when a user accesses your API, they might be surprised when your API does not return native strings (presumably in Python 2). This seems week to me. I've been supporting an all-Unicode-all-the-time API in Python-Markdown since before Python 3 was a real thing (it may have been in the early stages of a PEP at the time). As long as a lib documents that it only ever returns Unicode, the problems are limited to people who don't read the docs.

Given that MkDocs had no public facing APIs, I don't see much cause for concern. The config settings are all explicitly forced to Unicode, ensuring that MkDocs is all-Unicode-all-the-time. The few places were Python standard lib calls can have different behaviors, MkDocs is calling the Python 3 API (using `io.File` and friends), never relies in the system default encoding, and always passes in Unicode strings (some path handling stuff returns Unicode only when Unicode is passed in). Therefore, MkDocs ensures consistent results across Python versions.

There might be a few potential causes for concern:
1. Users can override some of the settings from the command line. I'm not certain if those are being explicitly converted to Unicode (or if they already are Unicode). This should probably be looked into. However, on problematic systems, passing in non-ASCII strings from the commandline will not work anyway, so I expect this is mostly a non-issue.
2. Some systems may not properly support paths and filenames with non ASCII characters in them. However, those users will have problems outside of MkDocs anyway (like when they clone their repo to a non-supporting system). However, if those users keep to supported systems, and ensure all their files use the proper encoding, everything should work fine for them. So I see this as a non-issue.
3. If and when a Plugin API is created, MkDocs will have a public facing API. However, as long as it is documented upfront that it is an all-Unicode-all-the-time API, I don't see a cause for concern. I couldn't help but note that the strongest examples cited in argument against such a thing were the pain points in migrating the massive codebase of Django to be all-Unicode-all-the-time. MkDocs is many times smaller (with much fewer potential edge cases), and, more importantly, a Plugin API will be new. Users will not be migrating their code from an old non-Unicode API to a new Unicode one (which is were most of the problems seemed to have been encountered). That being the case, this is a moot point IMO.
4. The Unicode text in a docstring issue is only a concern on Python<2.7, so that is a non-issue (even for Click which doesn't support Python<2.7).

Given the above, I think disabling the warning makes the most sense.

> I think disabling the warning makes the most sense.

Agreed - not worth wasting any more time on it.

The linked page in the issues explains things a bit better.  There seems to be arguments both ways.  I think the gist is handling Unicode is hard and sometimes when people port from 2 to 3, it isn't intuitive to understand the diverse ways in which you need to handle Unicode changes, especially when it comes to other modules you're importing; therefore, bugs get introduced that may take time to discover.  I still dream if the day that we can drop PY2 for good.

I still get the sense that well maintained projects should aim for future.  Anyways, it was a good read as I've run into issues before dealing with byte strings and Unicode when trying to support 2 and 3, but I usually just push through the issues and still use future.

Obviously, we can't change click's stance, but it is nice to understand the reasoning.


Resolving patch:
diff --git a/mkdocs/__main__.py b/mkdocs/__main__.py
--- a/mkdocs/__main__.py
+++ b/mkdocs/__main__.py
@@ -14,6 +14,11 @@
 
 log = logging.getLogger(__name__)
 
+# Disable the warning that Click displays (as of Click version 5.0) when users
+# use unicode_literals in Python 2.
+# See http://click.pocoo.org/dev/python3/#unicode-literals for more details.
+click.disable_unicode_literals_warning = True
+
 
 class State(object):
     ''' Maintain logging level.'''


Test patch: