Repository: sphinx-doc/sphinx

Issue:
Chinese language doesn't trigger Chinese search automatically
Subject: Chinese language doesn't trigger Chinese search automatically

### Problem
According to [doc](http://www.sphinx-doc.org/en/stable/config.html#confval-html_search_language),

> `html_search_language`
> ...  This defaults to the global language selected with `language`.

However, while I set the `language` to (Simplified) Chinese:

```
language = 'zh_CN'
```
The search function still uses the `en` version:

```
$ make html
... (some output)
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in _build\html.
```

For those who doesn't know Chinese:

sphinx supports two versions of Chinese: `zh_CN` – Simplified Chinese, `zh_TW` – Traditional Chinese, **under different names**.

Also sphinx supports the search, under the name `zh`.

I reasonably guess sphinx just don't know that `zh_CN` and `zh_TW` `language` should both trigger `zh` `html_search_language`.

#### Procedure to reproduce the problem

In `conf.py`, set `language` and **not set** `html_search_language`
```
language = 'zh_CN'
# html_search_language # <- do NOT set it and let sphinx guess it
```

Then

```bash
make html
```

#### Error logs / results

```
$ make html
... (some output)
dumping search index in English (code: en) ... done # <- notice this
...
```

#### Expected results

```
$ make html
... (some output)
dumping search index in Chinese (code: zh) ... done # <- should automatically call Chinese search index
...
```

### Reproducible project / your project

None. Just use any sphinx example project and change the `conf.py` as above.

### Environment info
- OS: Win
- Python version: 3.6
- Sphinx version: 1.6 (1.7dev20171020 has not fixed this issue but has slightly different output)


Discussion:

Patch:
diff --git a/CHANGES b/CHANGES
index 6ebe2c50a..a136c7452 100644
--- a/CHANGES
+++ b/CHANGES
@@ -32,6 +32,7 @@ Bugs fixed
 * #4152: HTML writer crashes if a field list is placed on top of the document
 * #4063: Sphinx crashes when labeling directive ``.. todolist::``
 * #4134: [doc] :file:`docutils.conf` is not documented explicitly
+* #4169: Chinese language doesn't trigger Chinese search automatically
 
 Testing
 --------
diff --git a/sphinx/search/__init__.py b/sphinx/search/__init__.py
index ef74d3b71..17560213d 100644
--- a/sphinx/search/__init__.py
+++ b/sphinx/search/__init__.py
@@ -265,6 +265,11 @@ def __init__(self, env, lang, options, scoring):
                                     # objtype index -> (domain, type, objname (localized))
         lang_class = languages.get(lang)    # type: Type[SearchLanguage]
                                             # add language-specific SearchLanguage instance
+
+        # fallback; try again with language-code
+        if lang_class is None and '_' in lang:
+            lang_class = languages.get(lang.split('_')[0])
+
         if lang_class is None:
             self.lang = SearchEnglish(options)  # type: SearchLanguage
         elif isinstance(lang_class, str):


Test patch:
diff --git a/tests/test_search.py b/tests/test_search.py
index 21c0badb7..f1825dfa4 100644
--- a/tests/test_search.py
+++ b/tests/test_search.py
@@ -228,3 +228,15 @@ def test_IndexBuilder():
     }
     assert index._objtypes == {('dummy', 'objtype'): 0}
     assert index._objnames == {0: ('dummy', 'objtype', 'objtype')}
+
+
+def test_IndexBuilder_lookup():
+    env = DummyEnvironment('1.0', {})
+
+    # zh
+    index = IndexBuilder(env, 'zh', {}, None)
+    assert index.lang.lang == 'zh'
+
+    # zh_CN
+    index = IndexBuilder(env, 'zh_CN', {}, None)
+    assert index.lang.lang == 'zh'