internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searxng/searxng

Issue:
Switch versioning format since SearXNG is rolling release
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Is your feature request related to a problem? Please describe.**
Right now we use the shortened git commit SHA as version. This can be hard for people to know which version an instance is running on.

**Describe the solution you'd like**
Instad of displaying version 1.0.0-commit_sha we should do a version like this for example: `2022-02-20-1` This way its more straight forward what version and instance is running and how old this version is. The `1.0.0` is not really needed in rolling release IMO.

**Describe alternatives you've considered**
Use the shortened commit SHA as version but still drop the `1.0.0`.

**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->



Discussion:
That's a very good idea, in fact at invidious we have the date + the commit hash for the version.
See example: https://api.invidious.io/

Resolving patch:
diff --git a/searx/version.py b/searx/version.py
--- a/searx/version.py
+++ b/searx/version.py
@@ -2,7 +2,6 @@
 # lint: pylint
 # pylint: disable=,missing-module-docstring,missing-class-docstring
 
-import re
 import os
 import shlex
 import subprocess
@@ -59,25 +58,8 @@ def get_git_url_and_branch():
 
 
 def get_git_version():
-    try:
-        tag = subprocess_run("git describe HEAD")
-        # a. HEAD is on tag name, example: tag = "v1.0.1"
-        # b. HEAD is not a tag name, example "<tag>-<distance>-g<commit>"
-        tag_version, tag_distance, tag_commit = (tag.split("-") + ["", ""])[:3]
-        if re.match(r"v[0-9]+\.[0-9]+\.[0-9]+", tag_version):
-            # tag_version "v1.0.0" becomes "1.0.0" (without the v)
-            # other patterns are kept untouched
-            tag_version = tag_version[1:]
-        # remove "g" prefix from tag_commit
-        if tag_commit and tag_commit[0] == "g":
-            tag_commit = tag_commit[1:]
-        # set git_version to "1.0.0-590-0686e274" or '1.0.0'
-        git_version = "-".join(filter(bool, [tag_version, tag_distance, tag_commit]))
-    except subprocess.CalledProcessError:
-        # fall back to "YYYY.MM.DD.Hash" if there is no tag at all
-        git_version = subprocess_run(r"git show -s --format='%as-%h'")
-        # PEP 440: replace - with .
-        tag_version = git_version = git_version.replace("-", ".")
+    git_commit_date_hash = subprocess_run(r"git show -s --format='%cs-%h'").replace("-", ".", 2)
+    tag_version = git_version = git_commit_date_hash
 
     # add "-dirty" suffix if there are uncommited changes except searx/settings.yml
     try:


Test patch:
Repository: searxng/searxng

Issue:
Internal wikidata entities should be filtered
**What happened?**

With searxng master I searched for `test` and got this fun wikidata box:

![image](https://user-images.githubusercontent.com/73739153/147099734-4a1b79f7-ff50-497a-b0ac-639db89fcec5.png)

**Expected behavior**

https://www.wikidata.org/wiki/Q4115189 has `instance of` [wikidata internal entity](https://www.wikidata.org/wiki/Q21281405) ... we should probably filter those in `wikidata.py`.


Discussion:

Resolving patch:
diff --git a/searx/engines/wikidata.py b/searx/engines/wikidata.py
--- a/searx/engines/wikidata.py
+++ b/searx/engines/wikidata.py
@@ -65,6 +65,7 @@
         mwapi:language "%LANGUAGE%".
         ?item wikibase:apiOutputItem mwapi:item.
   }
+  hint:Prior hint:runFirst "true".
 
   %WHERE%
 
@@ -93,6 +94,12 @@
 }
 """
 
+# see the property "dummy value" of https://www.wikidata.org/wiki/Q2013 (Wikidata)
+# hard coded here to avoid to an additional SPARQL request when the server starts
+DUMMY_ENTITY_URLS = set(
+    "http://www.wikidata.org/entity/" + wid for wid in ("Q4115189", "Q13406268", "Q15397819", "Q17339402")
+)
+
 
 # https://www.w3.org/TR/sparql11-query/#rSTRING_LITERAL1
 # https://lists.w3.org/Archives/Public/public-rdf-dawg/2011OctDec/0175.html
@@ -177,7 +184,7 @@ def response(resp):
     for result in jsonresponse.get('results', {}).get('bindings', []):
         attribute_result = {key: value['value'] for key, value in result.items()}
         entity_url = attribute_result['item']
-        if entity_url not in seen_entities:
+        if entity_url not in seen_entities and entity_url not in DUMMY_ENTITY_URLS:
             seen_entities.add(entity_url)
             results += get_results(attribute_result, attributes, language)
         else:


Test patch:
Repository: searxng/searxng

Issue:
move donation page to docs.searxng.org and link to it from instances
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
```
3a75d3c1ccbf979a483b4e5b209a59db9876ba33; searxng master
``` 

**How did you install SearXNG?**

with docker

**What happened?**
There is a donation page on the instance.

**Expected behavior**
The donation page should be in the official documentation and linked to by instances instead of having it on the instance. Also: There should be more information about who is receiving the money and what it is used for.

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
Suggestion in matrix, since donation page on instances could mean legal trouble.



Discussion:

Resolving patch:
diff --git a/searx/infopage/__init__.py b/searx/infopage/__init__.py
--- a/searx/infopage/__init__.py
+++ b/searx/infopage/__init__.py
@@ -157,10 +157,9 @@ def get_page(self, pagename: str, locale: typing.Optional[str] = None):
             return None
 
         cache_key = (pagename, locale)
-        page = self.CACHE.get(cache_key)
 
-        if page is not None:
-            return page
+        if cache_key in self.CACHE:
+            return self.CACHE[cache_key]
 
         # not yet instantiated
 
@@ -183,4 +182,6 @@ def iter_pages(self, locale: typing.Optional[str] = None, fallback_to_default=Fa
             if fallback_to_default and page is None:
                 page_locale = self.locale_default
                 page = self.get_page(page_name, self.locale_default)
-            yield page_name, page_locale, page
+            if page is not None:
+                # page is None if the page was deleted by the administrator
+                yield page_name, page_locale, page


Test patch:
Repository: searxng/searxng

Issue:
Bug: bing videos engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2022.08.14-649d1a14
<!-- Check if these values are correct -->

**How did you install SearXNG?**
searxng-docker

**What happened?**
bing video search crashed

**How To Reproduce**
search for anything

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
<!-- Add any other context about the problem here. -->

**Technical report**

Error
  * Error: IndexError
  * Percentage: 100
  * Parameters: `()`
  * File name: `searx/engines/bing_videos.py:84`
  * Function: `response`
  * Code: `metadata = loads(result.xpath('.//div[@class="vrhdata"]/@vrhm')[0])`




Discussion:

Resolving patch:
diff --git a/searx/engines/bing_videos.py b/searx/engines/bing_videos.py
--- a/searx/engines/bing_videos.py
+++ b/searx/engines/bing_videos.py
@@ -80,7 +80,7 @@ def response(resp):
 
     dom = html.fromstring(resp.text)
 
-    for result in dom.xpath('//div[@class="dg_u"]'):
+    for result in dom.xpath('//div[@class="dg_u"]/div[contains(@class, "mc_vtvc")]'):
         metadata = loads(result.xpath('.//div[@class="vrhdata"]/@vrhm')[0])
         info = ' - '.join(result.xpath('.//div[@class="mc_vtvc_meta_block"]//span/text()')).strip()
         content = '{0} - {1}'.format(metadata['du'], info)


Test patch:
Repository: searxng/searxng

Issue:
Duckduckgo weather formatting error
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2022.10.01-901143f0

**How did you install SearXNG?**
Script
**What happened?**
Humidity formatting has floating point error, 14+ numbers after decimal. Stuff like `56.00000000000001%` and `56.99999999999999%`
**How To Reproduce**
`!ddw 12345` (or some other zip)

Scroll through results until you see error, it's fairly common

**Expected behavior**
Humidity will be formatted such as 56%, 57%, etc

**Screenshots & Logs**
![humid](https://user-images.githubusercontent.com/1284317/193945463-bebf6a7c-8f92-43b3-861c-3460d7bf15ab.png)

Duckduckgo weather formatting error
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2022.10.01-901143f0

**How did you install SearXNG?**
Script
**What happened?**
Humidity formatting has floating point error, 14+ numbers after decimal. Stuff like `56.00000000000001%` and `56.99999999999999%`
**How To Reproduce**
`!ddw 12345` (or some other zip)

Scroll through results until you see error, it's fairly common

**Expected behavior**
Humidity will be formatted such as 56%, 57%, etc

**Screenshots & Logs**
![humid](https://user-images.githubusercontent.com/1284317/193945463-bebf6a7c-8f92-43b3-861c-3460d7bf15ab.png)



Discussion:

Resolving patch:
diff --git a/searx/engines/duckduckgo_weather.py b/searx/engines/duckduckgo_weather.py
--- a/searx/engines/duckduckgo_weather.py
+++ b/searx/engines/duckduckgo_weather.py
@@ -39,12 +39,12 @@ def generate_condition_table(condition):
 
     res += (
         f"<tr><td>{gettext('Wind')}</td><td>{condition['windBearing']}° — "
-        f"{'%.2f' % (condition['windSpeed'] * 1.6093440006147)} km/h / {condition['windSpeed']} mph</td></tr>"
+        f"{(condition['windSpeed'] * 1.6093440006147):.2f} km/h / {condition['windSpeed']} mph</td></tr>"
     )
 
     res += f"<tr><td>{gettext('Visibility')}</td><td>{condition['visibility']} km</td>"
 
-    res += f"<tr><td>{gettext('Humidity')}</td><td>{condition['humidity'] * 100}%</td></tr>"
+    res += f"<tr><td>{gettext('Humidity')}</td><td>{(condition['humidity'] * 100):.1f}%</td></tr>"
 
     return res
 


Test patch:
Repository: searxng/searxng

Issue:
SearXNG Firefox user-agent doesn't match real Firefox
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2022.10.29-fc9986de
**How did you install SearXNG?**
Script
**What happened?**
SearXNG will send a user-agent such as:  
`Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0.2) Gecko/20100101 Firefox/106.0.2`
However, real Firefox looks like this:  
`Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0`

Firefox does not send servers the patch version the user is running.

**How To Reproduce**
See https://github.com/searxng/searxng/blob/fc9986de0a21b01c1c8e342a9c4c8c6b96cc7ea2/searx/utils.py#L69-L74 and https://github.com/searxng/searxng/blob/fc9986de0a21b01c1c8e342a9c4c8c6b96cc7ea2/searx/data/useragents.json

To confirm Firefox's behavior, use a Firefox version with non-zero patch version and check its user-agent.

**Expected behavior**
SearXNG creates a User-Agent header that matches the format of the browser it is pretending to be.

**Additional context**
Firefox seems to have stopped sending the patch version with the release of Firefox 16: https://web.archive.org/web/20181123134532/https://www.fxsitecompat.com/en-CA/docs/2012/ua-string-no-longer-contains-patch-level-version-number/


Discussion:

Resolving patch:
diff --git a/searxng_extra/update/update_firefox_version.py b/searxng_extra/update/update_firefox_version.py
--- a/searxng_extra/update/update_firefox_version.py
+++ b/searxng_extra/update/update_firefox_version.py
@@ -64,8 +64,11 @@ def fetch_firefox_last_versions():
     major_list = (major_last, major_last - 1)
     for version in versions:
         major_current = version.version[0]
+        minor_current = version.version[1]
         if major_current in major_list:
-            result.append(version.vstring)
+            user_agent_version = f'{major_current}.{minor_current}'
+            if user_agent_version not in result:
+                result.append(user_agent_version)
 
     return result
 


Test patch:
Repository: searxng/searxng

Issue:
[Bug] "Google" is missing a lot of results.
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG.**

2022.09.29-a7d69323

**What happened?**
<!-- A clear and concise description of what the bug is. -->

the Google engine is missing a lot of results.

this is **not** a duplicate of #1794. it's an entirely different issue that started to happen way before that.

**How To Reproduce:**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

go to any [SearXNG](https://searx.space/) public instance, and search things like: `!go hl2 ost`, `!go tf2 ost`, `!go mgs ost`

you will get only 1 result.

but, if you go to any [Whoogle](https://github.com/benbusby/whoogle-search#public-instances) instance, and do the following searches: `hl2 ost`, `tf2 ost`, `mgs ost`

... you will get a lot more of results compared to SearXNG.

**Expected behavior.**
<!-- A clear and concise description of what you expected to happen. -->

you should get more results by using `!go` in SearXNG.

**Screenshots & Logs.**
<!-- If applicable, add screenshots, logs to help explain your problem. -->
| ![screenshot_20221004-084731](https://user-images.githubusercontent.com/85243826/193813135-262aa82d-f119-47c3-aa51-fb14d4be0e78.png) | ![screenshot_20221004-084741](https://user-images.githubusercontent.com/85243826/193813190-0f1ca9b6-6cf3-4374-9b16-aa95b8af5b67.png) | ![screenshot_20221004-084749](https://user-images.githubusercontent.com/85243826/193813218-7e080c10-c676-4010-b530-425fd5a40865.png) |
| --- | --- | --- |


Discussion:
I have also found that the longer your search query is, the less results will be pulled from Google. This doesn't happen when searching directly on Google.
Compared to Whoogle, SearXNG selectively find the results based on [XPath](https://en.wikipedia.org/wiki/XPath) selectors which may exclude some results if Google creates new kind of results that may not be covered by our [XPath](https://en.wikipedia.org/wiki/XPath) selectors.

https://github.com/searxng/searxng/pull/1749 should fix this issue but it's blocked due to https://github.com/searxng/searxng/pull/1749#issuecomment-1261334691

Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -112,9 +112,9 @@
 # specific xpath variables
 # ------------------------
 
-results_xpath = '//div[contains(@class, "MjjYud")]'
-title_xpath = './/h3[1]'
-href_xpath = './/a/@href'
+results_xpath = './/div[@data-sokoban-container]'
+title_xpath = './/a/h3[1]'
+href_xpath = './/a[h3]/@href'
 content_xpath = './/div[@data-content-feature=1]'
 
 # google *sections* are no usual *results*, we ignore them


Test patch:
Repository: searxng/searxng

Issue:
Setting autocomplete to “-“ (none) does not save the preference in preference URL
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
``` 
If you are using searxng-docker then look at the bottom of the SearXNG page
and check for the version after "Powered by SearXNG"

Please also stipulate if you are using a forked version of SearXNG and
include a link to the fork source code.
-->

2022.07.26

**How did you install SearXNG?**
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
Using a public instance 
**What happened?**
<!-- A clear and concise description of what the bug is. -->
When setting the autocomplete to “-“ (none/disable), the setting is not saved in the preferences URL but resets to the default autocomplete engine when pasting the preferences URL in a clean browser window. I use the custom preferences URL function and would like to disable autocomplete.

**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
1. Go to Searx Preferences
2. Set autocomplete to “-“ (none)
3. Go to cookies tab
4. Copy preferences URL (the bottom one)
5. Paste into new browser/private window
6. Go to Searx preferences in new browser/private window
7. Autocomplete shows default setting (in my case “google”) instead of “-“

**Expected behavior**
It should be possible to deactivate autocomplete completely when using the preferences URL.

<!--  **Screenshots & Logs**
If applicable, add screenshots, logs to help explain your problem.

**Additional context**
Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/searx/preferences.py b/searx/preferences.py
--- a/searx/preferences.py
+++ b/searx/preferences.py
@@ -441,7 +441,7 @@ def parse_encoded_data(self, input_data: str):
         """parse (base64) preferences from request (``flask.request.form['preferences']``)"""
         bin_data = decompress(urlsafe_b64decode(input_data))
         dict_data = {}
-        for x, y in parse_qs(bin_data.decode('ascii')).items():
+        for x, y in parse_qs(bin_data.decode('ascii'), keep_blank_values=True).items():
             dict_data[x] = y[0]
         self.parse_dict(dict_data)
 


Test patch:
Repository: searxng/searxng

Issue:
DuckDuckGo returning "access denied" errors
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2023.01.06-b241015e

**How did you install SearXNG?**
searxng-docker

**What happened?**
DuckDuckGo started returning "access denied" error messages. Very similar to previous issue #1854 

**How To Reproduce**
Enable DuckDuckGo and try to search anything.

**Expected behavior**
DDG results should return and no "Access Denied" error message should be displayed.

**Screenshots & Logs**
Error message in question:
![image](https://user-images.githubusercontent.com/4787751/211107269-bc163fce-162a-4a6f-b43b-7dcd1dea3138.png)

- Exception: searx.exceptions.SearxEngineAccessDeniedException
- Parameter: HTTP error 403
- Filename: searx/search/processors/online.py:113
- Function: _send_http_request
- Code: response = req(params['url'], **request_args)

**Additional context**

It looks like it can be fixed by adding a HTTP `Referer` header to the request.
DuckDuckGo returning "access denied" errors
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
2023.01.06-b241015e

**How did you install SearXNG?**
searxng-docker

**What happened?**
DuckDuckGo started returning "access denied" error messages. Very similar to previous issue #1854 

**How To Reproduce**
Enable DuckDuckGo and try to search anything.

**Expected behavior**
DDG results should return and no "Access Denied" error message should be displayed.

**Screenshots & Logs**
Error message in question:
![image](https://user-images.githubusercontent.com/4787751/211107269-bc163fce-162a-4a6f-b43b-7dcd1dea3138.png)

- Exception: searx.exceptions.SearxEngineAccessDeniedException
- Parameter: HTTP error 403
- Filename: searx/search/processors/online.py:113
- Function: _send_http_request
- Code: response = req(params['url'], **request_args)

**Additional context**

It looks like it can be fixed by adding a HTTP `Referer` header to the request.


Discussion:

Resolving patch:
diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py
--- a/searx/engines/duckduckgo.py
+++ b/searx/engines/duckduckgo.py
@@ -73,6 +73,7 @@ def request(query, params):
     # link again and again ..
 
     params['headers']['Content-Type'] = 'application/x-www-form-urlencoded'
+    params['headers']['Referer'] = 'https://lite.duckduckgo.com/'
 
     # initial page does not have an offset
     if params['pageno'] == 2:


Test patch:
Repository: searxng/searxng

Issue:
[Feature request] Add description in Wikidata results
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Is your feature request related to a problem? Please describe.**
The results from Wikidata always have empty description
![image](https://user-images.githubusercontent.com/10577978/225126751-f65c1739-45b1-4643-90f7-5ed1d13059d8.png)

**Describe the solution you'd like**
Get the description from the Wikidata page, make sure you get the description from the right language (english and spanish)
https://www.wikidata.org/wiki/Q251

![image](https://user-images.githubusercontent.com/10577978/225126124-a960ecec-9239-448f-bbf7-dc1cc38bea26.png)




Discussion:

Resolving patch:
diff --git a/searx/engines/wikidata.py b/searx/engines/wikidata.py
--- a/searx/engines/wikidata.py
+++ b/searx/engines/wikidata.py
@@ -260,7 +260,7 @@ def get_results(attribute_result, attributes, language):
                     infobox_urls.append({'title': attribute.get_label(language), 'url': url, **attribute.kwargs})
                     # "normal" results (not infobox) include official website and Wikipedia links.
                     if attribute.kwargs.get('official') or attribute_type == WDArticle:
-                        results.append({'title': infobox_title, 'url': url})
+                        results.append({'title': infobox_title, 'url': url, "content": infobox_content})
                     # update the infobox_id with the wikipedia URL
                     # first the local wikipedia URL, and as fallback the english wikipedia URL
                     if attribute_type == WDArticle and (


Test patch:
Repository: searxng/searxng

Issue:
Double bangs without query redirect to search URL instead of main page
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
`2023.4.21+98387e29`

**How did you install SearXNG?**
`searxng-docker`

**What happened?**
Double bangs without query redirect to search URL instead of main page

**How To Reproduce**
Use external bang without any search items e.g. `!!gh` and we land on `https://github.com/search?utf8=%E2%9C%93&q=` 

**Expected behavior**
`!!gh` should redirect to `https://github.com` if no search items


Discussion:

Resolving patch:
diff --git a/searx/external_bang.py b/searx/external_bang.py
--- a/searx/external_bang.py
+++ b/searx/external_bang.py
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-from urllib.parse import quote_plus
+from urllib.parse import quote_plus, urlparse
 from searx.data import EXTERNAL_BANGS
 
 LEAF_KEY = chr(16)
@@ -40,9 +40,15 @@ def get_bang_definition_and_ac(external_bangs_db, bang):
 
 def resolve_bang_definition(bang_definition, query):
     url, rank = bang_definition.split(chr(1))
-    url = url.replace(chr(2), quote_plus(query))
     if url.startswith('//'):
         url = 'https:' + url
+    if query:
+        url = url.replace(chr(2), quote_plus(query))
+    else:
+        # go to main instead of search page
+        o = urlparse(url)
+        url = o.scheme + '://' + o.netloc
+
     rank = int(rank) if len(rank) > 0 else 0
     return (url, rank)
 


Test patch: