Repository: sphinx-doc/sphinx

Issue:
Invalid sort in index for characters with diacritic signs (french locale)
Sphinx 1.0.7. Theme sphinxdoc.

In French, A,a,à,À,Á are sorted at the same level (the same for c, C, ç, Ç and other letters with diacritic signs).
With Sphinx 1.0.7, theses letters are put at the end of the index list (for example, à and À are after all the a or A).
It seems to work when they are inside a word but not when this the very first letter:

Example of index entries (some words does not mean anything they are here for test):

```
      pair: Àfficher; çe marche
      pair: Àfficher; çb marche
      pair: Àfficher; ça larche
      pair: Àfficher; Ça marche
      single:ça fouette
      single:appalache
      single:appblache
      single:appàlache
```

The last three are correctly sorted.
The others are at the end of A and C are at the end.
In subentry of Àfficher, 'Ça marche' is before 'ça larche' which is incorrect (l is before m)

---
- Bitbucket: https://bitbucket.org/birkenfeld/sphinx/issue/736
- Originally reported by: Anonymous
- Originally created at: 2011-07-16T10:24:45.311


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=h1) Report
> Merging [#6540](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=desc) into [2.0](https://codecov.io/gh/sphinx-doc/sphinx/commit/cb7e2a9dea433e7f060093dbb713ee4689327694?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `88.23%`.

[![Impacted file tree graph](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540/graphs/tree.svg?width=650&token=VFlFSELbkw&height=150&src=pr)](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##              2.0    #6540      +/-   ##
==========================================
- Coverage   83.67%   83.67%   -0.01%     
==========================================
  Files         268      268              
  Lines       40648    40657       +9     
  Branches     5982     5984       +2     
==========================================
+ Hits        34014    34021       +7     
- Misses       5309     5310       +1     
- Partials     1325     1326       +1
```


| [Impacted Files](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tests/test\_environment\_indexentries.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9lbnZpcm9ubWVudF9pbmRleGVudHJpZXMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [sphinx/environment/adapters/indexentries.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540/diff?src=pr&el=tree#diff-c3BoaW54L2Vudmlyb25tZW50L2FkYXB0ZXJzL2luZGV4ZW50cmllcy5weQ==) | `91.08% <77.77%> (-1.47%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=footer). Last update [cb7e2a9...f601c52](https://codecov.io/gh/sphinx-doc/sphinx/pull/6540?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGES b/CHANGES
index b264164db..7c5c5379c 100644
--- a/CHANGES
+++ b/CHANGES
@@ -49,6 +49,7 @@ Bugs fixed
 * #6511: LaTeX: autonumbered list can not be customized in LaTeX
   since Sphinx 1.8.0 (refs: #6533)
 * #6531: Failed to load last environment object when extension added
+* #736: Invalid sort in pair index
 
 Testing
 --------
diff --git a/sphinx/builders/html.py b/sphinx/builders/html.py
index 1a7054bdc..beb6af2e2 100644
--- a/sphinx/builders/html.py
+++ b/sphinx/builders/html.py
@@ -741,66 +741,52 @@ def to_relpath(f: str) -> str:
                     logger.warning(__('cannot copy downloadable file %r: %s'),
                                    path.join(self.srcdir, src), err)
 
-    def create_pygments_style_file(self) -> None:
-        """create a style file for pygments."""
-        with open(path.join(self.outdir, '_static', 'pygments.css'), 'w') as f:
-            f.write(self.highlighter.get_stylesheet())
-
-    def copy_translation_js(self) -> None:
-        """Copy a JavaScript file for translations."""
-        if self.config.language is not None:
-            jsfile = self._get_translations_js()
-            if jsfile:
-                copyfile(jsfile, path.join(self.outdir, '_static', 'translations.js'))
-
-    def copy_stemmer_js(self) -> None:
-        """Copy a JavaScript file for stemmer."""
-        if self.indexer is not None:
-            jsfile = self.indexer.get_js_stemmer_rawcode()
-            if jsfile:
-                copyfile(jsfile, path.join(self.outdir, '_static', '_stemmer.js'))
-
-    def copy_theme_static_files(self, context: Dict) -> None:
-        if self.theme:
-            for entry in self.theme.get_theme_dirs()[::-1]:
-                copy_asset(path.join(entry, 'static'),
-                           path.join(self.outdir, '_static'),
-                           excluded=DOTFILES, context=context, renderer=self.templates)
-
-    def copy_html_static_files(self, context: Dict) -> None:
-        excluded = Matcher(self.config.exclude_patterns + ["**/.*"])
-        for entry in self.config.html_static_path:
-            copy_asset(path.join(self.confdir, entry),
-                       path.join(self.outdir, '_static'),
-                       excluded, context=context, renderer=self.templates)
-
-    def copy_html_logo(self) -> None:
-        if self.config.html_logo:
-            copy_asset(path.join(self.confdir, self.config.html_logo),
-                       path.join(self.outdir, '_static'))
-
-    def copy_html_favicon(self) -> None:
-        if self.config.html_favicon:
-            copy_asset(path.join(self.confdir, self.config.html_favicon),
-                       path.join(self.outdir, '_static'))
-
     def copy_static_files(self) -> None:
         try:
+            # copy static files
             with progress_message(__('copying static files... ')):
                 ensuredir(path.join(self.outdir, '_static'))
+                # first, create pygments style file
+                with open(path.join(self.outdir, '_static', 'pygments.css'), 'w') as f:
+                    f.write(self.highlighter.get_stylesheet())
+                # then, copy translations JavaScript file
+                if self.config.language is not None:
+                    jsfile = self._get_translations_js()
+                    if jsfile:
+                        copyfile(jsfile, path.join(self.outdir, '_static',
+                                                   'translations.js'))
+
+                # copy non-minified stemmer JavaScript file
+                if self.indexer is not None:
+                    jsfile = self.indexer.get_js_stemmer_rawcode()
+                    if jsfile:
+                        copyfile(jsfile, path.join(self.outdir, '_static', '_stemmer.js'))
+
+                ctx = self.globalcontext.copy()
 
-                # prepare context for templates
-                context = self.globalcontext.copy()
+                # add context items for search function used in searchtools.js_t
                 if self.indexer is not None:
-                    context.update(self.indexer.context_for_searchtool())
-
-                self.create_pygments_style_file()
-                self.copy_translation_js()
-                self.copy_stemmer_js()
-                self.copy_theme_static_files(context)
-                self.copy_html_static_files(context)
-                self.copy_html_logo()
-                self.copy_html_favicon()
+                    ctx.update(self.indexer.context_for_searchtool())
+
+                # then, copy over theme-supplied static files
+                if self.theme:
+                    for theme_path in self.theme.get_theme_dirs()[::-1]:
+                        entry = path.join(theme_path, 'static')
+                        copy_asset(entry, path.join(self.outdir, '_static'), excluded=DOTFILES,
+                                   context=ctx, renderer=self.templates)
+                # then, copy over all user-supplied static files
+                excluded = Matcher(self.config.exclude_patterns + ["**/.*"])
+                for static_path in self.config.html_static_path:
+                    entry = path.join(self.confdir, static_path)
+                    copy_asset(entry, path.join(self.outdir, '_static'), excluded,
+                               context=ctx, renderer=self.templates)
+                # copy logo and favicon files if not already in static path
+                if self.config.html_logo:
+                    entry = path.join(self.confdir, self.config.html_logo)
+                    copy_asset(entry, path.join(self.outdir, '_static'))
+                if self.config.html_favicon:
+                    entry = path.join(self.confdir, self.config.html_favicon)
+                    copy_asset(entry, path.join(self.outdir, '_static'))
         except OSError as err:
             logger.warning(__('cannot copy static file %r'), err)
 
diff --git a/sphinx/environment/adapters/indexentries.py b/sphinx/environment/adapters/indexentries.py
index 430c3dce2..68198040d 100644
--- a/sphinx/environment/adapters/indexentries.py
+++ b/sphinx/environment/adapters/indexentries.py
@@ -133,11 +133,21 @@ def keyfunc(entry: Tuple[str, List]) -> Tuple[str, str]:
                 oldsubitems = subitems
                 i += 1
 
+        # sort the sub-index entries
+        def keyfunc2(entry: Tuple[str, List]) -> str:
+            key = unicodedata.normalize('NFD', entry[0].lower())
+            if key.startswith('\N{RIGHT-TO-LEFT MARK}'):
+                key = key[1:]
+            if key[0:1].isalpha() or key.startswith('_'):
+                key = chr(127) + key
+            return key
+
         # group the entries by letter
-        def keyfunc2(item: Tuple[str, List]) -> str:
+        def keyfunc3(item: Tuple[str, List]) -> str:
             # hack: mutating the subitems dicts to a list in the keyfunc
             k, v = item
-            v[1] = sorted((si, se) for (si, (se, void, void)) in v[1].items())
+            v[1] = sorted(((si, se) for (si, (se, void, void)) in v[1].items()),
+                          key=keyfunc2)
             if v[2] is None:
                 # now calculate the key
                 if k.startswith('\N{RIGHT-TO-LEFT MARK}'):
@@ -151,4 +161,4 @@ def keyfunc2(item: Tuple[str, List]) -> str:
             else:
                 return v[2]
         return [(key_, list(group))
-                for (key_, group) in groupby(newlist, keyfunc2)]
+                for (key_, group) in groupby(newlist, keyfunc3)]


Test patch:
diff --git a/tests/test_environment_indexentries.py b/tests/test_environment_indexentries.py
index ec76acdc0..4083eae04 100644
--- a/tests/test_environment_indexentries.py
+++ b/tests/test_environment_indexentries.py
@@ -47,19 +47,30 @@ def test_create_pair_index(app):
     app.env.indexentries.clear()
     text = (".. index:: pair: docutils; reStructuredText\n"
             ".. index:: pair: Python; interpreter\n"
-            ".. index:: pair: Sphinx; documentation tool\n")
+            ".. index:: pair: Sphinx; documentation tool\n"
+            ".. index:: pair: Sphinx; :+1:\n"
+            ".. index:: pair: Sphinx; Ель\n"
+            ".. index:: pair: Sphinx; ёлка\n")
     restructuredtext.parse(app, text)
     index = IndexEntries(app.env).create_index(app.builder)
-    assert len(index) == 5
-    assert index[0] == ('D',
+    assert len(index) == 7
+    assert index[0] == ('Symbols', [(':+1:', [[], [('Sphinx', [('', '#index-3')])], None])])
+    assert index[1] == ('D',
                         [('documentation tool', [[], [('Sphinx', [('', '#index-2')])], None]),
                          ('docutils', [[], [('reStructuredText', [('', '#index-0')])], None])])
-    assert index[1] == ('I', [('interpreter', [[], [('Python', [('', '#index-1')])], None])])
-    assert index[2] == ('P', [('Python', [[], [('interpreter', [('', '#index-1')])], None])])
-    assert index[3] == ('R',
+    assert index[2] == ('I', [('interpreter', [[], [('Python', [('', '#index-1')])], None])])
+    assert index[3] == ('P', [('Python', [[], [('interpreter', [('', '#index-1')])], None])])
+    assert index[4] == ('R',
                         [('reStructuredText', [[], [('docutils', [('', '#index-0')])], None])])
-    assert index[4] == ('S',
-                        [('Sphinx', [[], [('documentation tool', [('', '#index-2')])], None])])
+    assert index[5] == ('S',
+                        [('Sphinx', [[],
+                                     [(':+1:', [('', '#index-3')]),
+                                      ('documentation tool', [('', '#index-2')]),
+                                      ('ёлка', [('', '#index-5')]),
+                                      ('Ель', [('', '#index-4')])],
+                                     None])])
+    assert index[6] == ('Е', [('ёлка', [[], [('Sphinx', [('', '#index-5')])], None]),
+                               ('Ель', [[], [('Sphinx', [('', '#index-4')])], None])])
 
 
 @pytest.mark.sphinx('dummy')