Repository: searxng/searxng

Issue:
Bug: bing engine crashes sometimes
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2023.01.08-4e735b28

**How did you install SearXNG?**
Docker
**What happened?**
Bing searches sometimes don't work.

**How To Reproduce**
`!bi certbot` (does not work at all for me)  
`!bi current events` (works ~50% of the time)

**Expected behavior**
Search works

**Screenshots & Logs**
```
2023-01-08 13:07:27,885 WARNING:searx.engines.bing: ErrorContext('searx/engines/bing.py', 89, "link = eval_xpath(result, './/h2/a')[0]", 'IndexError', None, ()) False
2023-01-08 13:07:27,885 ERROR:searx.engines.bing: exception : list index out of range
Traceback (most recent call last):
  File "/usr/local/searxng/searx/search/processors/online.py", line 160, in search
    search_results = self._search_basic(query, params)
  File "/usr/local/searxng/searx/search/processors/online.py", line 148, in _search_basic
    return self.engine.response(response)
  File "/usr/local/searxng/searx/engines/bing.py", line 89, in response
    link = eval_xpath(result, './/h2/a')[0]
IndexError: list index out of range
```

**Additional context**
Many searches *do* work.

Reproducible on some*, but not most, public instances running `2023.01.08-4e735b28`. At least using the specific repro searches above.

\* https://search.rhscz.eu , https://xo.wtf , out of the 7 running the current version


Discussion:
Thanks for pointing out, I assume its a follow up issue introduced by https://github.com/searxng/searxng/commit/9ee99423fe22550ef566245ef23e2a9e8ee76c27

@ahmad-alkadri by selecting *contains* ..

https://github.com/searxng/searxng/blob/4e735b289b2a71583664e214e45aafccd0b85bfc/searx/engines/bing.py#L87

we select more than we want .. and some of this selections might not have a link .. can you have a look / thanks!
Hello @return42. Firstly thanks for tagging me.

> Thanks for pointing out, I assume its a follow up issue introduced by [9ee9942](https://github.com/searxng/searxng/commit/9ee99423fe22550ef566245ef23e2a9e8ee76c27)
> 
> @ahmad-alkadri by selecting _contains_ ..
> 
> https://github.com/searxng/searxng/blob/4e735b289b2a71583664e214e45aafccd0b85bfc/searx/engines/bing.py#L87
> 
> we select more than we want .. and some of this selections might not have a link .. can you have a look / thanks!

A bit surprising; I thought that that class should've contained all the results with the links (including the Wikipedia). But probably can be resolved by putting some error catching. I'll work on this tonight, do some tests and make a PR.
Bing is full of surprises and its result structure is like everything from Microsoft: full of quirks and exceptions.

>  But probably can be resolved by putting some error catching

That is true; we need to check every intermediate result, assumptions about the intermediate results are always risky with so many quirks.

By example before we assume there is an item in a list ..

https://github.com/searxng/searxng/blob/4e735b289b2a71583664e214e45aafccd0b85bfc/searx/engines/bing.py#L89

.. we have to check the list is not empty.


Currently taking a look into this and I found that I can't reproduce the Issue. I'm on the same commit as mentioned above, `4e735b28` (full hash: `4e735b289b2a71583664e214e45aafccd0b85bfc`). Here are the search results from my instance for `!bi certbot`:
![screenshot-127 0 0 1_8888-2023 01 08-17_36_20](https://user-images.githubusercontent.com/22837764/211208374-3a26daaf-ef7a-44f4-9ad5-c71677dbcc36.png)

and `!bi current events`:
![screenshot-127 0 0 1_8888-2023 01 08-17_36_40](https://user-images.githubusercontent.com/22837764/211208394-65da1e49-4951-4ce3-ac89-5ddec7153262.png)

I'm using Linux Ubuntu 20.04.5 LTS (GNU/Linux 5.15.0-1026-aws x86_64). Maybe the error is system-specific? Need your help @return42; could you please check/test this Issue on your side also, please? Thanks.

PS: I'll still add the error catching and make the draft PR later.
Seems to take place when using 'all' language (default language), try [!bi certbot :all](https://search.rhscz.eu/search?q=%21bi%20certbot%20%3Aall)
@return42 I checked that query search on the instance's link that you gave and yes it returned the error:

<img width="1483" alt="Capture d’écran 2023-01-08 à 18 09 33" src="https://user-images.githubusercontent.com/22837764/211209486-51784eb0-4735-4359-856d-344b47baf1ff.png">

But in my local instance, it worked:

<img width="1483" alt="Capture d’écran 2023-01-08 à 18 09 52" src="https://user-images.githubusercontent.com/22837764/211209496-6d2276b6-472e-4da3-af61-4bef91c8a5c3.png">

Both of us are running the same commit version. I wonder what happened. Have you tested it locally on your machine?

EDIT: in all case, I've put the error-catching and pushed it to my branch; I'll make the PR draft ASAP.
> Both of us are running the same commit version. I wonder what happened. Have you tested it locally on your machine?

Seems we both can't reproduce the issue but the issue exists: I assume that it depends on the IP where the request comes from. My IPs are in DE, rhscz.eu is in netherland (seems bing looks from what country the request comes from and the response from bing depends on this).

When we can't reproduce the issue, the only thing we can do is to hardening the implementation.

Hardening is what I meant in my comment above ..

> By example before we assume there is an item in a list ..
>
 https://github.com/searxng/searxng/blob/4e735b289b2a71583664e214e45aafccd0b85bfc/searx/engines/bing.py#L89
>
> .. we have to check the list is not empty.


Resolving patch:
diff --git a/searx/engines/bing.py b/searx/engines/bing.py
--- a/searx/engines/bing.py
+++ b/searx/engines/bing.py
@@ -9,7 +9,7 @@
 import re
 from urllib.parse import urlencode, urlparse, parse_qs
 from lxml import html
-from searx.utils import eval_xpath, extract_text, eval_xpath_list, match_language
+from searx.utils import eval_xpath, extract_text, eval_xpath_list, match_language, eval_xpath_getindex
 from searx.network import multi_requests, Request
 
 about = {
@@ -84,9 +84,12 @@ def response(resp):
 
     url_to_resolve = []
     url_to_resolve_index = []
-    for i, result in enumerate(eval_xpath_list(dom, '//li[contains(@class, "b_algo")]')):
+    i = 0
+    for result in eval_xpath_list(dom, '//ol[@id="b_results"]/li[contains(@class, "b_algo")]'):
 
-        link = eval_xpath(result, './/h2/a')[0]
+        link = eval_xpath_getindex(result, './/h2/a', 0, None)
+        if link is None:
+            continue
         url = link.attrib.get('href')
         title = extract_text(link)
 
@@ -119,6 +122,8 @@ def response(resp):
 
         # append result
         results.append({'url': url, 'title': title, 'content': content})
+        # increment result pointer for the next iteration in this loop
+        i += 1
 
     # resolve all Bing redirections in parallel
     request_list = [


Test patch: