Repository: searxng/searxng

Issue:
Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**

The latest commit as of this report, 75587d03a6766c87f2f1964c82080393559ed623

**How did you install SearXNG?**

```
make run
```

**What happened?**

As the title says. Current code redirects simplified search to the Hong Kong site, which actually uses traditional Chinese, but marks simplified pages with a `<span class="fz-s fc-obsidian ml-4">簡</span>` after the title text node, at the end:

```html
<h3 style="display:block;margin-top:24px;margin-bottom:2px;" class="title tc"><a style="line-height:26px" class=" d-ib fz-20 lh-26 td-hu tc va-bot mxw-100p" href="https://zh.wikipedia.org/wiki/%E7%B9%81%E4%BD%93%E5%AD%97" referrerpolicy="origin" target="_blank" data-439="6515fd318fbac" aria-label="繁体字 - 维基百科，自由的百科全书"><span class=" s-url d-ib p-abs t-0 l-0 fz-14 lh-20 fc-obsidian wr-bw ls-n pb-4">zh.wikipedia.org<span class=" fc-pewter"> › wiki › <b>繁体字</b></span></span>繁体字 - 维基百科，自由的百科全书<span class="fz-s fc-obsidian ml-4">簡</span></a></h3>
```

This character is extracted by `.xpath('span')` in the first line, which, without knowledge of this situation, actually extracted text from two `span`s, making the second line substring off by one:

https://github.com/searxng/searxng/blob/75587d03a6766c87f2f1964c82080393559ed623/searx/engines/yahoo.py#L139-L140

**How To Reproduce**

Search for some simplified Chinese keyword and look for a Yahoo result with 簡 being the last character of its title.

**Expected behavior**

That mark should be ignored, eliminating the off by one.

**Screenshots & Logs**

<img width="664" alt="problematic search result" src="https://github.com/searxng/searxng/assets/4474501/f8ea1959-0487-4fed-b93f-c3e9a061eaea">

You can see that the last breadcrumb is 繁体字, but the title says 体字, cutting off 繁; while there is an extraneous 簡 at the end.

**Additional context**

```diff
diff --git a/searx/engines/yahoo.py b/searx/engines/yahoo.py
index 0fdeacec2..4858e7e3f 100644
--- a/searx/engines/yahoo.py
+++ b/searx/engines/yahoo.py
@@ -136,8 +136,9 @@ def response(resp):
         title = eval_xpath_getindex(result, './/h3/a', 0, default=None)
         if title is None:
             continue
-        offset = len(extract_text(title.xpath('span')))
-        title = extract_text(title)[offset:]
+        offset = len(extract_text(title.xpath('span[1]')))
+        offset_right = -len(extract_text(title.xpath('span[2]'))) or None
+        title = extract_text(title)[offset:offset_right]

         content = eval_xpath_getindex(result, './/div[contains(@class, "compText")]', 0, default='')
         content = extract_text(content, allow_none=True)
```

This is an apparent fix, but I'm not sure it's worth it.


Discussion:
Successfully reproduce using keyword `!yahoo 繁体字`.

I shall review your fix later.
However I think it's more likely to be a engine api problem. Because when I use Yahoo to search in simplified Chinese, it redirect me to `search.yahoo.com` instead of `hk.search.yahoo.com`.

And when I switch the search language in searxng to english, the problem seems to disappear.

![Screenshot_20230929_104805_com.microsoft.emmx.jpg](https://github.com/searxng/searxng/assets/88757735/3112062d-997f-4d56-9962-4ce82e54afc9)

Maybe it's a engine api problem?

I'm using the default language setting, auto-detect. If by engine API you mean `searx/engines/yahoo.py`, it might be advisable that `zh_hans` be mapped to `search.yahoo.com` in `lang2domain`.
@bnoctis i've open a PR #2874 , please check it out.

Resolving patch:
diff --git a/searx/engines/yahoo.py b/searx/engines/yahoo.py
--- a/searx/engines/yahoo.py
+++ b/searx/engines/yahoo.py
@@ -133,12 +133,7 @@ def response(resp):
             continue
         url = parse_url(url)
 
-        title = eval_xpath_getindex(result, './/h3/a', 0, default=None)
-        if title is None:
-            continue
-        offset = len(extract_text(title.xpath('span')))
-        title = extract_text(title)[offset:]
-
+        title = extract_text(result.xpath('.//h3//a/@aria-label'))
         content = eval_xpath_getindex(result, './/div[contains(@class, "compText")]', 0, default='')
         content = extract_text(content, allow_none=True)
 
@@ -164,7 +159,7 @@ def fetch_traits(engine_traits: EngineTraits):
 
     resp = network.get('https://search.yahoo.com/preferences/languages')
     if not resp.ok:
-        print("ERROR: response from peertube is not OK.")
+        print("ERROR: response from yahoo is not OK.")
 
     dom = html.fromstring(resp.text)
     offset = len('lang_')


Test patch: