Repository: ckan/ckan

Issue:
ur'strings' are a Syntax Errors in Python 3
In Python 3, you can have __u'strings'__ and you can have __r'strings'__ but you can not have __ur'strings'__.
* __python3 -c "print(ur'string')"__  # --> Syntax Error

ur'strings' currently appear in three files...

$ __python3 -m flake8 . --count --select=E901,E999 --show-source --statistics__
```
./ckan/tests/test_coding_standards.py:127:55: E999 SyntaxError: invalid syntax
    pattern = re.compile(ur'#.*?coding[:=][ \t]*utf-?8')
                                                      ^
./ckan/lib/munge.py:159:42: E999 SyntaxError: invalid syntax
    filename = re.sub(ur'[^a-zA-Z0-9_. -]', '', filename).replace(u' ', u'-')
                                         ^
./doc/conf.py:341:49: E999 SyntaxError: invalid syntax
  ('contents', 'CKAN.tex', ur'CKAN documentation',
                                                ^
3     E999 SyntaxError: invalid syntax
```

The last one is governed by:
* http://www.sphinx-doc.org/en/stable/config.html#confval-latex_documents

Based on the [example __conf.py__ file](http://www.sphinx-doc.org/en/stable/config.html#example-of-configuration-file), I believe that we should use u'strings' in this context.  #4049


Discussion:
Related to https://github.com/ckan/ckan/issues/3309.
Discussed in https://github.com/ckan/ckan/pull/4035#issuecomment-368993517
@torfsen @amercader What about converting __ur'string'__ --> __six.u(r'string')__?

```
python2 -c "print(ur'你好 Lüsai' == '你好 Lüsai')"    # False
python2 -c "print(ur'你好 Lüsai' == r'你好 Lüsai')"   # False
python2 -c "print(ur'你好 Lüsai' == u'你好 Lüsai')"   # True

python2 -c "import six ; print(ur'你好 Lüsai' == six.u('你好 Lüsai'))"     # True
python2 -c "import six ; print(ur'你好 Lüsai' == six.u(r'你好 Lüsai'))"    # True
# python2 -c "import six ; print(ur'你好 Lüsai' == six.u(u'你好 Lüsai'))"  # TypeError

python2 -c "print(type('你好 Lüsai'))"                       # <type 'str'>
python2 -c "print(type(r'你好 Lüsai'))"                      # <type 'str'>
python2 -c "print(type(ur'你好 Lüsai'))"                     # <type 'unicode'>
python2 -c "import six ; print(type(six.u(r'你好 Lüsai')))"  # <type 'unicode'>
```
In general, we don't have many `ur` strings in CKAN:

```
$ grep -rI "\<ur[\"']" --include "*.py"
ckan/lib/munge.py:    filename = re.sub(ur'[^a-zA-Z0-9_. -]', '', filename).replace(u' ', u'-')
ckan/lib/munge.py:    filename = re.sub(ur'-+', u'-', filename)
ckan/tests/test_coding_standards.py:    pattern = re.compile(ur'#.*?coding[:=][ \t]*utf-?8')
ckan/tests/test_coding_standards.py:                        if (i > 1) and (line[i - 2:i].lower() == u'ur'):
doc/conf.py:  ('contents', 'CKAN.tex', ur'CKAN documentation',
doc/conf.py:   ur'CKAN contributors', 'manual'),
```

All of these are ASCII-only strings, so from an encoding point of view, nothing is lost when they are simply converted to `r` strings.

Judging from my experiments, the return type of Python 2's regular expression functions depends on the type of the input string, but not on the type of the pattern string:

```
>>> re.match(r'\w*', 'aä').group()
'a'
>>> re.match(r'\w*', u'aä').group()
u'a'
>>> re.match(ur'\w*', 'aä').group()
'a'
>>> re.match(ur'\w*', u'aä').group()
u'a'
```

Also, the type of the pattern string doesn't influence whether character classes like `\w` capture non-ASCII characters (as seen above, the `ä` is never matched, even when both the pattern and the input are `unicode`) -- that is decided by the presence of the `re.UNICODE` (or `re.U`) flag (in Python 2 at least).

In conclusion, it should be OK to convert all the `ur` instances of the current code base to `r` strings.

As mentioned before, we also should update [our Unicode handling documentation](https://github.com/ckan/ckan/blob/master/doc/contributing/unicode.rst) which currently says

> In CKAN, every string literal must carry either a u or a b prefix. While the latter is redundant in Python 2, it makes the developer's intention explicit and eases a future migration to Python 3.
>
> This rule also holds for raw strings, which are created using an r prefix. Simply use ur instead:
> 
>     m = re.match(ur'A\s+Unicode\s+pattern')


Resolving patch:
diff --git a/ckan/lib/munge.py b/ckan/lib/munge.py
--- a/ckan/lib/munge.py
+++ b/ckan/lib/munge.py
@@ -158,8 +158,8 @@ def munge_filename(filename):
     # Clean up
     filename = filename.lower().strip()
     filename = substitute_ascii_equivalents(filename)
-    filename = re.sub(ur'[^a-zA-Z0-9_. -]', '', filename).replace(u' ', u'-')
-    filename = re.sub(ur'-+', u'-', filename)
+    filename = re.sub(u'[^a-zA-Z0-9_. -]', '', filename).replace(u' ', u'-')
+    filename = re.sub(u'-+', u'-', filename)
 
     # Enforce length constraints
     name, ext = os.path.splitext(filename)


Test patch:
diff --git a/ckan/tests/test_coding_standards.py b/ckan/tests/test_coding_standards.py
--- a/ckan/tests/test_coding_standards.py
+++ b/ckan/tests/test_coding_standards.py
@@ -18,6 +18,7 @@
 import sys
 
 from six import text_type
+from six.moves import xrange
 
 FILESYSTEM_ENCODING = text_type(
     sys.getfilesystemencoding() or sys.getdefaultencoding()
@@ -125,7 +126,7 @@ def test_source_files_specify_encoding():
 
     Empty files and files that only contain comments are ignored.
     '''
-    pattern = re.compile(ur'#.*?coding[:=][ \t]*utf-?8')
+    pattern = re.compile(u'#.*?coding[:=][ \\t]*utf-?8')
     decode_errors = []
     no_specification = []
     for abs_path, rel_path in walk_python_files():