internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searx/searx

Issue:
[fix] Sphinx 3.2.1 build: add missing "engines" variable in the jinja context
## What does this PR do?

Fix PR #2210 which upgrade Sphinx version to 3.2.1.

The admin/engines.html page is empty.

## Why is this change important?

Fix the admin/engines.html page.

## How to test this PR locally?

Check the admin/engines.html page.

## Author's checklist

<!-- additional notes for reviewiers -->

## Related issues

#2210



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -28,10 +28,11 @@
 exclude_patterns = ['build-templates/*.rst']
 
 from searx import webapp
-from searx.engines import engines
+import searx.engines
+searx.engines.initialize_engines(searx.settings['engines'])
 jinja_contexts = {
     'webapp': dict(**webapp.__dict__),
-    'engines': engines
+    'engines': searx.engines.engines
 }
 
 # usage::   lorem :patch:`f373169` ipsum


Test patch:
Repository: searx/searx

Issue:
Make secret_key default one that will fail if not set to a custom value
Currently, the `secret_key` default value is `ultrasecretkey` which is a valid value. Would it not be better to let the default value of this setting be one that will make searx fail to start? This will force the user to conciously change this setting to a secure value instead of accidentally forgetting to set this to something random and secure.


Discussion:

Resolving patch:
diff --git a/searx/__init__.py b/searx/__init__.py
--- a/searx/__init__.py
+++ b/searx/__init__.py
@@ -85,3 +85,7 @@ def check_settings_yml(file_name):
     settings['server']['secret_key'] = environ['SEARX_SECRET']
 if 'SEARX_BIND_ADDRESS' in environ:
     settings['server']['bind_address'] = environ['SEARX_BIND_ADDRESS']
+
+if not searx_debug and settings['server']['secret_key'] == 'ultrasecretkey':
+    logger.error('server.secret_key is not changed. Please use something else instead of ultrasecretkey.')
+    exit(1)


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -0,0 +1,2 @@
+import os
+os.environ['SEARX_DEBUG'] = '1'
Repository: searx/searx

Issue:
Open Semantic Search Engine
This new engine will be using Solr search server used by an Open Semantic Search setup.

This should close #1318 .


Discussion:

Resolving patch:
diff --git a/searx/engines/opensemantic.py b/searx/engines/opensemantic.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/opensemantic.py
@@ -0,0 +1,42 @@
+"""
+Open Semantic Search
+
+ @website    https://www.opensemanticsearch.org/
+ @provide-api yes (https://www.opensemanticsearch.org/dev)
+
+ @using-api  yes
+ @results    JSON
+ @stable     yes
+ @parse      url, title, content, publishedDate
+"""
+from dateutil import parser
+from json import loads
+from urllib.parse import quote
+
+base_url = 'http://localhost:8983/solr/opensemanticsearch/'
+search_string = 'query?q={query}'
+
+
+def request(query, params):
+    search_path = search_string.format(
+        query=quote(query),
+    )
+    params['url'] = base_url + search_path
+    return params
+
+
+def response(resp):
+    results = []
+    data = loads(resp.text)
+    docs = data.get('response', {}).get('docs', [])
+
+    for current in docs:
+        item = {}
+        item['url'] = current['id']
+        item['title'] = current['title_txt_txt_en']
+        if current.get('content_txt'):
+            item['content'] = current['content_txt'][0]
+        item['publishedDate'] = parser.parse(current['file_modified_dt'])
+        results.append(item)
+
+    return results


Test patch:
Repository: searx/searx

Issue:
RuntimeError: Set changed size during iteration
Sometimes I will make a search and I will get 500 Internal Server Error. I can't reproduce it, but looking at the traceback it looks like something happening with timeouts and such:

```
ERROR:searx.webapp:Exception on / [POST]
Traceback (most recent call last):
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/searx/searx-pyenv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/searx/searx-src/searx/webapp.py", line 704, in index
    unresponsive_engines=__get_translated_errors(result_container.unresponsive_engines),
  File "/usr/local/searx/searx-src/searx/webapp.py", line 717, in __get_translated_errors
    for unresponsive_engine in unresponsive_engines:
RuntimeError: Set changed size during iteration
```

Is it my fault or is it a bug?

I have v0.17.0 installed on a Raspberry Pi 4. Thanks!


Discussion:

Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -735,12 +735,12 @@ def search():
 
 
 def __get_translated_errors(unresponsive_engines):
-    translated_errors = []
+    translated_errors = set()
     for unresponsive_engine in unresponsive_engines:
         error_msg = gettext(unresponsive_engine[1])
         if unresponsive_engine[2]:
             error_msg = "{} {}".format(error_msg, unresponsive_engine[2])
-        translated_errors.append((unresponsive_engine[0], error_msg))
+        translated_errors.add((unresponsive_engine[0], error_msg))
     return translated_errors
 
 


Test patch:
Repository: searx/searx

Issue:
Bug in external command engine, resulting in engine crash.
**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
```
commit a0ddc27766271428d6f1f906c774cf6f5ccbf3fa (HEAD -> master)
Merge: 8c887382 cdceec1c
Author: Searx Service Account <searx@leandra.virtadpt.net>
Date:   Sat Dec 5 17:21:41 2020 -0800

    Merge branch 'master' of https://github.com/asciimoo/searx
```
**How did you install Searx?**
Installed using instructions from the official wiki, by hand.

**What happened?**
I went to the Searx page on my server and ran the query `!locate art bell mp3` after enabling the Locate search engine in `/opt/searx/searx/searx/settings.yml`.

**How To Reproduce**
Enable the Locate search engine in settings.yml thusly:

```
  - name: locate
    engine: command
    command: ['locate', '--existing', '--ignore-case', '{{QUERY}}']
    shortcut: locate
    tokens: []
    disabled: False
    delimiter:
        chars: ' '
        keys: ['line']
```
Restart Searx.

Execute a `!locate` search while watching Searx's output, either by running it by hand or using `journalctl -xf` to tail the systemd journal.

**Expected behavior**
Searx runs the `locate` command on the server and returns the results.

**Screenshots & Logs**
Logs (datestamp, hostname, and PID elided to prevent having to scroll back and forth repeatedly):
```
: Traceback (most recent call last):
:   File "/opt/searx/searx/searx/search.py", line 281, in search_one_offline_request_safe
:     search_results = search_one_offline_request(engine, query, request_params)
:   File "/opt/searx/searx/searx/search.py", line 274, in search_one_offline_request
:     return engine.search(query, request_params)
:   File "/opt/searx/searx/searx/engines/command.py", line 70, in search
:     cmd = _get_command_to_run(query)
:   File "/opt/searx/searx/searx/engines/command.py", line 83, in _get_command_to_run
:     params = shlex_split(query.decode('utf-8'))
: AttributeError: 'str' object has no attribute 'decode'
```

**Additional context**
Searx is being run as a system service, through systemd, with a searx.sh shell script:

```
#!/usr/bin/env bash

SEARX=/opt/searx/searx

# Change to the Searx installation directory.
cd $SEARX

# Initialize the Python virtual environment.
. env/bin/activate

# Start up Searx.
#python searx/webapp.py
uwsgi --ini searx.ini
```

Searx is being run with uwsgi to improve responsiveness.

Other searches on this instance are not impacted in this manner.


Discussion:

Resolving patch:
diff --git a/searx/engines/command.py b/searx/engines/command.py
--- a/searx/engines/command.py
+++ b/searx/engines/command.py
@@ -80,7 +80,7 @@ def search(query, params):
 
 
 def _get_command_to_run(query):
-    params = shlex_split(query.decode('utf-8'))
+    params = shlex_split(query)
     __check_query_params(params)
 
     cmd = []


Test patch:
Repository: searx/searx

Issue:
Startpage not returning any results
**Version:**
I tried it with `v0.17.0` and `820b468bfe96f693d60ce06f1e78af51f00deefc`

**Installation-Method:** 
Manually according to wiki (with uwsgi)

**What happened?**
The startpage engine is not returning any results

**How To Reproduce**
Execute a query with `!sp test`

**Expected behavior**
Results from startpage

**Additional context**
I added some log lines to the startpage engine file and it seems like it gets some
response back from startpage. 

Maybe startpage changed their formatting? 
I didn't have a closer look on the way results are parsed yet. 


Discussion:

Resolving patch:
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -36,8 +36,8 @@
 # specific xpath variables
 # ads xpath //div[@id="results"]/div[@id="sponsored"]//div[@class="result"]
 # not ads: div[@class="result"] are the direct childs of div[@id="results"]
-results_xpath = '//div[@class="w-gl__result"]'
-link_xpath = './/a[@class="w-gl__result-title"]'
+results_xpath = '//div[@class="w-gl__result__main"]'
+link_xpath = './/a[@class="w-gl__result-url result-link"]'
 content_xpath = './/p[@class="w-gl__description"]'
 
 


Test patch:
Repository: searx/searx

Issue:
SUGGESTION: Contacting the instance's maintainer(s)
Hello, so I use searx, but I personally think that there should be any way to contact the maintainer(s) of a public instance (email for example). It is harder to trust this awesome service if there is no way to contact the maintainer(s). 



Discussion:
> It is harder to trust this awesome service if there is no way to contact the maintainer(s).

I see it the same way, but unfortunately we cannot force our users to do so.

The public instances are managed in the https://github.com/dalf/searx-instances project, one option would only be to register instances that also give an indication of the operator.

May you like to move this issue to https://github.com/dalf/searx-instances ? / Thanks!
Maybe an optional config that allow the maintainer to give an email address or a place for contacting them ? And when this config is set, a "contact" link is added in the footer.
> > It is harder to trust this awesome service if there is no way to contact the maintainer(s).
> 
> I see it the same way, but unfortunately we cannot force our users to do so.
> 
> The public instances are managed in the https://github.com/dalf/searx-instances project, one option would only be to register instances that also give an indication of the operator.
> 
> May you like to move this issue to https://github.com/dalf/searx-instances ? / Thanks!

Just did, thank you
> Maybe an optional config that allow the maintainer to give an email address or a place for contacting them ? And when this config is set, a "contact" link is added in the footer.

yeah, I like this idea, but I would personally prefer it to be obligated so there is *at least* one way to contact them
That's a good idea.
The ability to add an address email to the `settings.yml` so that it gets displayed on Searx.
I'm glad you guys liked it
FYI: I left this issue open as ["enhancement"](https://github.com/asciimoo/searx/issues/1972#issuecomment-634718073)
> > It is harder to trust this awesome service if there is no way to contact the maintainer(s).
> 
> I see it the same way, but unfortunately we cannot force our users to do so.
> 
> The public instances are managed in the https://github.com/dalf/searx-instances project, one option would only be to register instances that also give an indication of the operator.
> 
> May you like to move this issue to https://github.com/dalf/searx-instances ? / Thanks!

btw, in case you weren't aware, it is possible to do it yourself: https://help.github.com/en/github/managing-your-work-on-github/transferring-an-issue-to-another-repository

Resolving patch:
diff --git a/searx/brand.py b/searx/brand.py
--- a/searx/brand.py
+++ b/searx/brand.py
@@ -4,3 +4,4 @@
 SEARX_URL = 'https://searx.me'
 DOCS_URL = 'https://searx.github.io/searx'
 PUBLIC_INSTANCES = 'https://searx.space'
+CONTACT_URL = 'mailto:contact@example.com'


Test patch:
Repository: searx/searx

Issue:
Startpage: The title result is showing the url
![CleanShot 2020-12-16 at 12 00 49](https://user-images.githubusercontent.com/7293811/102400176-652ca100-3f96-11eb-86be-0aafa9c424e8.jpg)
Just updated to the newest version. 0.18.0. The title in StartPage result is showing the url rather than the page title. Same happens to other public instance.

_Originally posted by @lucky13820 in https://github.com/searx/searx/pull/2385#issuecomment-746927618_


Discussion:

Resolving patch:
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -37,7 +37,7 @@
 # ads xpath //div[@id="results"]/div[@id="sponsored"]//div[@class="result"]
 # not ads: div[@class="result"] are the direct childs of div[@id="results"]
 results_xpath = '//div[@class="w-gl__result__main"]'
-link_xpath = './/a[@class="w-gl__result-url result-link"]'
+link_xpath = './/a[@class="w-gl__result-title result-link"]'
 content_xpath = './/p[@class="w-gl__description"]'
 
 


Test patch:
Repository: searx/searx

Issue:
Input turns language to Chinese
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->

**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
0.17.0-17b48ff6e858b0c74116068cf6444bd578bbb747
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
``` 
If you are using searx-docker then look at the bottom of the Searx page
and check for the version after "Powered by searx"

Please also stipulate if you are using a forked version of Searx and
include a link to the fork source code.
-->
**How did you install Searx?**
Manual install
<!-- Did you install Searx using the official wiki or using searx-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
If I search the phrase `parser error : invalid character in attribute value`, the search language changes to `zh`.
<!-- A clear and concise description of what the bug is. -->

**How To Reproduce**
This works on every searx instance I can find. Just search the phrase `parser error : invalid character in attribute value`.
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

**Expected behavior**
Results in the language chosen.
<!-- A clear and concise description of what you expected to happen. -->



Discussion:

Resolving patch:
diff --git a/searx/query.py b/searx/query.py
--- a/searx/query.py
+++ b/searx/query.py
@@ -77,7 +77,7 @@ def _parse_query(self):
                     pass
 
             # this force a language
-            if query_part[0] == ':':
+            if query_part[0] == ':' and len(query_part) > 1:
                 lang = query_part[1:].lower().replace('_', '-')
 
                 # check if any language-code is equal with


Test patch:
diff --git a/tests/unit/test_query.py b/tests/unit/test_query.py
--- a/tests/unit/test_query.py
+++ b/tests/unit/test_query.py
@@ -59,6 +59,15 @@ def test_invalid_language_code(self):
         self.assertEqual(len(query.languages), 0)
         self.assertFalse(query.specific)
 
+    def test_empty_colon_in_query(self):
+        query_text = 'the : query'
+        query = RawTextQuery(query_text, [])
+
+        self.assertEqual(query.getFullQuery(), query_text)
+        self.assertEqual(len(query.query_parts), 0)
+        self.assertEqual(len(query.languages), 0)
+        self.assertFalse(query.specific)
+
     def test_timeout_below100(self):
         query_text = '<3 the query'
         query = RawTextQuery(query_text, [])
Repository: searx/searx

Issue:
Feature suggestion: Add Wordnik
Hi there, would it be possible to add Wordnik? This is where DuckDuckGo gets its definitions from, and they're much better than Wikipedia/Wiktionary. It would be nice to have this for instant answers on the sidebar, with top priority above other services that offer definitions, such as Wikipedia and Wolfram|Alpha. Thanks!


Discussion:
This! I am missing those handy definitions so much. 
Just type `define: feature` and get back:
```
 feature
    n.    Any of the distinct parts of the face, as the eyes, nose, or mouth.
    n.    The overall appearance of the face or its parts.
```
Preferably in the little frame on the right had side.
@plague-doctor Feel free to contribute to the Searx code if you want a faster implementation. Currently, unfortunately we don't have the amount of developers needed for working on non-important features.

Resolving patch:
diff --git a/searx/engines/wordnik.py b/searx/engines/wordnik.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/wordnik.py
@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+"""Wordnik (general)
+
+"""
+
+from lxml.html import fromstring
+from searx import logger
+from searx.utils import extract_text
+from searx.raise_for_httperror import raise_for_httperror
+
+logger = logger.getChild('Wordnik engine')
+
+# about
+about = {
+    "website": 'https://www.wordnik.com',
+    "wikidata_id": 'Q8034401',
+    "official_api_documentation": None,
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+categories = ['general']
+paging = False
+
+URL = 'https://www.wordnik.com'
+SEARCH_URL = URL + '/words/{query}'
+
+
+def request(query, params):
+    params['url'] = SEARCH_URL.format(query=query)
+    logger.debug(f"query_url --> {params['url']}")
+    return params
+
+
+def response(resp):
+    results = []
+
+    raise_for_httperror(resp)
+    dom = fromstring(resp.text)
+    word = extract_text(dom.xpath('//*[@id="headword"]/text()'))
+
+    definitions = []
+    for src in dom.xpath('//*[@id="define"]//h3[@class="source"]'):
+        src_text = extract_text(src).strip()
+        if src_text.startswith('from '):
+            src_text = src_text[5:]
+
+        src_defs = []
+        for def_item in src.xpath('following-sibling::ul[1]/li'):
+            def_abbr = extract_text(def_item.xpath('.//abbr')).strip()
+            def_text = extract_text(def_item).strip()
+            if def_abbr:
+                def_text = def_text[len(def_abbr):].strip()
+            src_defs.append((def_abbr, def_text))
+
+        definitions.append((src_text, src_defs))
+
+    if not definitions:
+        return results
+
+    infobox = ''
+    for src_text, src_defs in definitions:
+        infobox += f"<small>{src_text}</small>"
+        infobox += "<ul>"
+        for def_abbr, def_text in src_defs:
+            if def_abbr:
+                def_abbr += ": "
+            infobox += f"<li><i>{def_abbr}</i> {def_text}</li>"
+        infobox += "</ul>"
+
+    results.append({
+        'infobox': word,
+        'content': infobox,
+    })
+
+    return results


Test patch:
Repository: searx/searx

Issue:
Updating languages of qwant is broken
The URL used for retrieving available languages has gone:

```
    engines_languages = fetch_supported_languages()
  File "/home/runner/work/searx/searx/./searx_extra/update/update_languages.py", line 30, in fetch_supported_languages
    engines_languages[engine_name] = engines[engine_name].fetch_supported_languages()
  File "/home/runner/work/searx/searx/searx/engines/__init__.py", line 141, in <lambda>
    lambda: engine._fetch_supported_languages(get(engine.supported_languages_url, headers=headers)))
  File "/home/runner/work/searx/searx/searx/poolrequests.py", line 209, in get
    return request('get', url, **kwargs)
  File "/home/runner/work/searx/searx/searx/poolrequests.py", line 202, in request
    raise_for_httperror(response)
  File "/home/runner/work/searx/searx/searx/raise_for_httperror.py", line 66, in raise_for_httperror
    resp.raise_for_status()
  File "/home/runner/work/searx/searx/local/py3/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.qwant.com/region
```

Full run: https://github.com/searx/searx/runs/2487497774?check_suite_focus=true

@MarcAbonce If you have time could you please look into it? Or at least give some pointers how it can be fixed?


Discussion:

Resolving patch:
diff --git a/searx/engines/qwant.py b/searx/engines/qwant.py
--- a/searx/engines/qwant.py
+++ b/searx/engines/qwant.py
@@ -23,7 +23,7 @@
 # engine dependent config
 categories = []
 paging = True
-supported_languages_url = 'https://qwant.com/region'
+supported_languages_url = about['website']
 
 category_to_keyword = {'general': 'web',
                        'images': 'images',
@@ -125,15 +125,15 @@ def response(resp):
 def _fetch_supported_languages(resp):
     # list of regions is embedded in page as a js object
     response_text = resp.text
-    response_text = response_text[response_text.find('regionalisation'):]
-    response_text = response_text[response_text.find('{'):response_text.find(');')]
+    response_text = response_text[response_text.find('INITIAL_PROPS'):]
+    response_text = response_text[response_text.find('{'):response_text.find('</script>')]
 
     regions_json = loads(response_text)
 
-    supported_languages = {}
-    for lang in regions_json['languages'].values():
-        for country in lang['countries']:
-            lang_code = "{lang}-{country}".format(lang=lang['code'], country=country)
-            supported_languages[lang_code] = {'name': lang['name']}
+    supported_languages = []
+    for country, langs in regions_json['locales'].items():
+        for lang in langs['langs']:
+            lang_code = "{lang}-{country}".format(lang=lang, country=country)
+            supported_languages.append(lang_code)
 
     return supported_languages


Test patch: