Repository: readthedocs/readthedocs.org

Issue:
UTF8 characters on version slugging -- or slugging in general
This ticket came up as part of #1407. We should make sure version slugging is handling UTF8 characters in a sane way.



Discussion:
Is there any algorithm you can suggest? Currently if a all-non-ascii name is provided the algorithm will return `unknown`. Partly non-ascii words like `Straße` will look "chunked" like `stra-e`.

This might be a good solution: https://pypi.python.org/pypi/unicode-slugify
-- It convert utf8 chars to their "equivalent" in ascii. We could also just
generate utf-8 slugs. I don't know a lot about this, but in general I think
we support UTF-8, but we might run into issues with nginx/etc downstream,
but perhaps that is the best solution.

On Mon, Jul 6, 2015 at 11:30 AM, Gregor Müllegger notifications@github.com
wrote:

> Is there any algorithm you can suggest? Currently if a all-non-ascii name
> is provided the algorithm will return unknown. Partly non-ascii words
> like Straße will look "chunked" like stra-e.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/rtfd/readthedocs.org/issues/1410#issuecomment-118949432
> .

## 

Eric Holscher
Maker of the internet residing in Portland, Oregon
http://ericholscher.com

I think using UTF-8 in the URL is no good as there are just too many tools that do not properly handle UTF-8. However Wikipedia is doing it. The problem I see is with subdomains that contain project slugs. They definitely will make problems when they contain non-ascii characters.

We've been using the std library to handle this (convert all the unicode to its ascii representation):

```
>>> import unicodedata
>>> unicodedata.normalize('NFKD', u'camión').encode('ascii', 'ignore')
'camion'
>>> 
```
> think using UTF-8 in the URL is no good as there are just too many tools that do not properly handle UTF-8. However Wikipedia is doing it. 

I think it's not a problem using unicode chars on the URL.

Actually, we do support this for filenames: https://test-builds.readthedocs.io/en/unicode-filename/

but we do replace the unicode chars when they are in the version's name/identifier/slug: https://test-builds.readthedocs.io/en/d--branch/

> The problem I see is with subdomains that contain project slugs. They definitely will make problems when they contain non-ascii characters.

We should probably keep the project's slug as ASCII.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I want to reopen it and apply the solution proposed by Eric: use `unicode-slugify`.

I also considered `django.utils.text.slugify` but it does not work as good as `unicode-slugify`:

`django.utils.text.slugify`:

```
In [1]: from django.utils.text import slugify                                                                                                                                                 

In [2]: slugify('北京 (capital of China)')                                                                                                                                                    
Out[2]: 'capital-of-china'
```

`unicode-slugify`:

```
from slugify import slugify
slugify(u'北京 (capital of China)', only_ascii=True)
# u'bei-jing-capital-of-china'
```

(there are more examples on its docs)
@humitos 
Just a doubt.
I think to solve this, these lines
https://github.com/rtfd/readthedocs.org/blob/05cc76b89a72cc963e091854b38adf28ab19ae3e/readthedocs/builds/version_slug.py#L85-L87
are to be replaced by using `unicode-slugify`.

Something like:
```
from slugify import slugify
slugified = slugify(content)
```


Resolving patch:
diff --git a/readthedocs/builds/version_slug.py b/readthedocs/builds/version_slug.py
--- a/readthedocs/builds/version_slug.py
+++ b/readthedocs/builds/version_slug.py
@@ -26,6 +26,7 @@
 
 from django.db import models
 from django.utils.encoding import force_text
+from slugify import slugify as unicode_slugify
 
 
 def get_fields_with_model(cls):
@@ -53,13 +54,15 @@ def get_fields_with_model(cls):
 
 class VersionSlugField(models.CharField):
 
-    """Inspired by ``django_extensions.db.fields.AutoSlugField``."""
+    """
+    Inspired by ``django_extensions.db.fields.AutoSlugField``.
 
-    invalid_chars_re = re.compile('[^-._a-z0-9]')
-    leading_punctuation_re = re.compile('^[-._]+')
-    placeholder = '-'
-    fallback_slug = 'unknown'
+    Uses ``unicode-slugify`` to generate the slug.
+    """
+
+    ok_chars = '-._'  # dash, dot, underscore
     test_pattern = re.compile('^{pattern}$'.format(pattern=VERSION_SLUG_REGEX))
+    fallback_slug = 'unknown'
 
     def __init__(self, *args, **kwargs):
         kwargs.setdefault('db_index', True)
@@ -78,13 +81,42 @@ def get_queryset(self, model_cls, slug_field):
                 return model._default_manager.all()
         return model_cls._default_manager.all()
 
+    def _normalize(self, content):
+        """
+        Normalize some invalid characters (/, %, !, ?) to become a dash (``-``).
+
+        .. note::
+
+            We replace these characters to a dash to keep compatibility with the
+            old behavior and also because it makes this more readable.
+
+        For example, ``release/1.0`` will become ``release-1.0``.
+        """
+        return re.sub('[/%!?]', '-', content)
+
     def slugify(self, content):
+        """
+        Make ``content`` a valid slug.
+
+        It uses ``unicode-slugify`` behind the scenes which works properly with
+        Unicode characters.
+        """
         if not content:
             return ''
 
-        slugified = content.lower()
-        slugified = self.invalid_chars_re.sub(self.placeholder, slugified)
-        slugified = self.leading_punctuation_re.sub('', slugified)
+        normalized = self._normalize(content)
+        slugified = unicode_slugify(
+            normalized,
+            only_ascii=True,
+            spaces=False,
+            lower=True,
+            ok=self.ok_chars,
+            space_replacement='-',
+        )
+
+        # Remove first character wile it's an invalid character for the
+        # beginning of the slug
+        slugified = slugified.lstrip(self.ok_chars)
 
         if not slugified:
             return self.fallback_slug


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_version_slug.py b/readthedocs/rtd_tests/tests/test_version_slug.py
--- a/readthedocs/rtd_tests/tests/test_version_slug.py
+++ b/readthedocs/rtd_tests/tests/test_version_slug.py
@@ -105,3 +105,15 @@ def test_uniquifying_suffix(self):
         self.assertEqual(field.uniquifying_suffix(25), '_z')
         self.assertEqual(field.uniquifying_suffix(26), '_ba')
         self.assertEqual(field.uniquifying_suffix(52), '_ca')
+
+    def test_unicode(self):
+        version = Version.objects.create(
+            verbose_name='camión',
+            project=self.pip,
+        )
+        self.assertEqual(version.slug, 'camion')
+        version = Version.objects.create(
+            verbose_name='ŭñíč°də-branch',
+            project=self.pip,
+        )
+        self.assertEqual(version.slug, 'unicd-branch')