Repository: sphinx-doc/sphinx

Issue:
English words cannot be searched in Chinese word segmentation mode
## Problem:
I can't search for English words in Chinese search mode.

![image](https://user-images.githubusercontent.com/29136507/48240907-9d547380-e40f-11e8-81ba-68f6cd46736e.png)

## Procedure to reproduce the problem

1. When I set `html_search_language = 'zh'` to use Chinese search, .
2. And add the following string `可以查看 FAQ 模块中 test 部分`.
3. search keyword `test`.

## Error results
When searching keyword `test`, there will be no result display.

## Expected results
There will be test string in search results.

## Sugestions
Why do space characters appear at the end of the word segmentation result?

I look up the source code and send the following regular expression, which will identify the space.
[sphinx/search/zh.py](https://github.com/sphinx-doc/sphinx/blob/16f47cded956bfba55380034ff30988d2ab5ee7c/sphinx/search/zh.py#L236)

Now test as follows：
```
In [3]: re.compile(u'(?u)\\w+[\u0000-\u00ff]').findall("可以查看 FAQ 模块中 test 部分")
Out[3]: ['可以查看 ', 'FAQ ', '模块中 ', 'test ']

```
The result clearly contains the space character, here is the source of the problem.
Can we solve this problem by adjusting regular expressions?

## Environment info

- OS: MacOS 10.14.1 with Python 2.7 in a sphinx specific virtualenv
- Python version: Python 2.7.10
- Sphinx version: 1.8.1
- Browser Chrome


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=h1) Report
> Merging [#5611](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=desc) into [master](https://codecov.io/gh/sphinx-doc/sphinx/commit/4c96c6efce9975919f0bdd675eff4f862c0beeae?src=pr&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611/graphs/tree.svg?width=650&token=VFlFSELbkw&height=150&src=pr)](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5611      +/-   ##
==========================================
+ Coverage   83.17%   83.21%   +0.04%     
==========================================
  Files         294      292       -2     
  Lines       39245    39213      -32     
  Branches     5876     5874       -2     
==========================================
- Hits        32642    32633       -9     
+ Misses       5236     5215      -21     
+ Partials     1367     1365       -2
```


| [Impacted Files](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [tests/test\_search.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9zZWFyY2gucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [sphinx/search/zh.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611/diff?src=pr&el=tree#diff-c3BoaW54L3NlYXJjaC96aC5weQ==) | `79.41% <100%> (+14.7%)` | :arrow_up: |
| [sphinx/\_\_init\_\_.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611/diff?src=pr&el=tree#diff-c3BoaW54L19faW5pdF9fLnB5) | | |
| [sphinx/make\_mode.py](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611/diff?src=pr&el=tree#diff-c3BoaW54L21ha2VfbW9kZS5weQ==) | | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=footer). Last update [4c96c6e...c9deb3f](https://codecov.io/gh/sphinx-doc/sphinx/pull/5611?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

TimKam: @DaGeger can you test once again? What do you think, @tk0miya?
tk0miya: Is this still WIP?
TimKam: Sorry, yes. I wrote some tests locally and found out it needs more work. Will get back to it at the weekend.
tk0miya: Okay. Please let me know if you finished. I'll review this again :-)
TimKam: @tk0miya Could you have another look? I needed to add a list that contains all Latin terms and change the regexp that splits Chinese from Latin characters. @DaGeger comments from you regarding the initial problem are also welcome.

Patch:
diff --git a/CHANGES b/CHANGES
index 867cdc950..d3f8b0a0e 100644
--- a/CHANGES
+++ b/CHANGES
@@ -15,6 +15,8 @@ Features added
 
 Bugs fixed
 ----------
+* #5605 If the documentation language is set to Chinese, English words could not
+  be searched.
 
 Testing
 --------
diff --git a/sphinx/search/zh.py b/sphinx/search/zh.py
index 6c5b65d6b..3753bc990 100644
--- a/sphinx/search/zh.py
+++ b/sphinx/search/zh.py
@@ -233,7 +233,8 @@ class SearchChinese(SearchLanguage):
     language_name = 'Chinese'
     js_stemmer_code = js_porter_stemmer
     stopwords = english_stopwords
-    latin1_letters = re.compile(u'(?u)\\w+[\u0000-\u00ff]')
+    latin1_letters = re.compile(r'[a-zA-Z0-9_]+')
+    latin_terms = []  # type: List[unicode]
 
     def init(self, options):
         # type: (Dict) -> None
@@ -250,7 +251,9 @@ def split(self, input):
         if JIEBA:
             chinese = list(jieba.cut_for_search(input))
 
-        latin1 = self.latin1_letters.findall(input)
+        latin1 = \
+            [term.strip() for term in self.latin1_letters.findall(input)]
+        self.latin_terms.extend(latin1)
         return chinese + latin1
 
     def word_filter(self, stemmed_word):
@@ -259,4 +262,15 @@ def word_filter(self, stemmed_word):
 
     def stem(self, word):
         # type: (unicode) -> unicode
+
+        # Don't stem Latin words that are long enough to be relevant for search
+        # if not stemmed, but would be too short after being stemmed
+        # avoids some issues with acronyms
+        should_not_be_stemmed = (
+            word in self.latin_terms and
+            len(word) >= 3 and
+            len(self.stemmer.stem(word.lower())) < 3
+        )
+        if should_not_be_stemmed:
+            return word.lower()
         return self.stemmer.stem(word.lower())


Test patch:
diff --git a/tests/roots/test-search/tocitem.rst b/tests/roots/test-search/tocitem.rst
index f082abf65..5d99f0a66 100644
--- a/tests/roots/test-search/tocitem.rst
+++ b/tests/roots/test-search/tocitem.rst
@@ -8,3 +8,9 @@ textinheading
 =============
 
 lorem ipsum
+
+可以查看 FAQ 模块中 Chinesetest 部分
+
+模块中 CAS service部分
+
+可以Chinesetesttwo查看
diff --git a/tests/test_search.py b/tests/test_search.py
index 886151831..4c7eb8b21 100644
--- a/tests/test_search.py
+++ b/tests/test_search.py
@@ -240,3 +240,18 @@ def test_IndexBuilder_lookup():
     # zh_CN
     index = IndexBuilder(env, 'zh_CN', {}, None)
     assert index.lang.lang == 'zh'
+
+
+@pytest.mark.sphinx(
+    testroot='search',
+    confoverrides={'html_search_language': 'zh'},
+    srcdir='search_zh'
+)
+def test_search_index_gen_zh(app, status, warning):
+    app.builder.build_all()
+    # jsdump fails if search language is 'zh'; hence we just get the text:
+    searchindex = (app.outdir / 'searchindex.js').text()
+    assert 'chinesetest ' not in searchindex
+    assert 'chinesetest' in searchindex
+    assert 'chinesetesttwo' in searchindex
+    assert 'cas' in searchindex