internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searxng/searxng

Issue:
MWMBL
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Working URL to the engine**
<!-- Please check if the engine is responding correctly before submitting it. -->
https://mwmbl.org/
**Why do you want to add this engine?**
<!-- What's special about this engine? Is it open source or libre? -->
Open source, has a distributed crawler.
**Features of this engine**
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->
it lets users contribute to the index by installing an extension.

Also, it crawls pages from hackernews.
**How can SearXNG fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank. -->
doesn't look like it has an API and it requires JS, you can probably reverse engineer the scripts, I don't know enough JS to do it myself.
**Applicable category of this engine**
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:
general, files, images, it, map, music, news, science, social media and videos.
You can add multiple categories at the same time. -->
general
**Additional context**
<!-- Add any other context about this engine here. -->



Discussion:

Resolving patch:
diff --git a/searx/engines/mwmbl.py b/searx/engines/mwmbl.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/mwmbl.py
@@ -0,0 +1,41 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""mwmbl (general)
+"""
+
+from urllib.parse import urlencode
+
+about = {
+    "website": 'https://github.com/mwmbl/mwmbl',
+    "official_api_documentation": None,
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+paging = False
+categories = ['general']
+
+api_url = "https://api.mwmbl.org"
+
+
+def request(query, params):
+    params['url'] = f"{api_url}/search?{urlencode({'s': query})}"
+    return params
+
+
+def response(resp):
+    results = []
+
+    json_results = resp.json()
+
+    for result in json_results:
+        title_parts = [title['value'] for title in result['title']]
+        results.append(
+            {
+                'url': result['url'],
+                'title': ''.join(title_parts),
+                'content': result['extract'][0]['value'],
+            }
+        )
+
+    return results


Test patch:
Repository: searxng/searxng

Issue:
[Engine Request] scholar.archive.org
Hi, would it be possible to add this engine to the science category?

**Working URL to the engine**
https://scholar.archive.org/

**Why do you want to add this engine?**
It provides lots of resources in full text, including very old papers.

**Features of this engine**
Doesn’t track its users, fast.

**How can SearXNG fetch the information from this engine?**
I don’t know. The service is still in beta.

**Applicable category of this engine**
Science.

Thanks,


Discussion:

Resolving patch:
diff --git a/searx/engines/internet_archive_scholar.py b/searx/engines/internet_archive_scholar.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/internet_archive_scholar.py
@@ -0,0 +1,72 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Internet Archive scholar(science)
+"""
+
+from datetime import datetime
+from urllib.parse import urlencode
+from searx.utils import html_to_text
+
+about = {
+    "website": "https://scholar.archive.org/",
+    "wikidata_id": "Q115667709",
+    "official_api_documentation": "https://scholar.archive.org/api/redoc",
+    "use_official_api": True,
+    "require_api_key": False,
+    "results": "JSON",
+}
+categories = ['science', 'scientific publications']
+paging = True
+
+base_url = "https://scholar.archive.org"
+results_per_page = 15
+
+
+def request(query, params):
+    args = {
+        "q": query,
+        "limit": results_per_page,
+        "offset": (params["pageno"] - 1) * results_per_page,
+    }
+    params["url"] = f"{base_url}/search?{urlencode(args)}"
+    params["headers"]["Accept"] = "application/json"
+    return params
+
+
+def response(resp):
+    results = []
+
+    json = resp.json()
+
+    for result in json["results"]:
+        publishedDate, content, doi = None, '', None
+
+        if result['biblio'].get('release_date'):
+            publishedDate = datetime.strptime(result['biblio']['release_date'], "%Y-%m-%d")
+
+        if len(result['abstracts']) > 0:
+            content = result['abstracts'][0].get('body')
+        elif len(result['_highlights']) > 0:
+            content = result['_highlights'][0]
+
+        if len(result['releases']) > 0:
+            doi = result['releases'][0].get('doi')
+
+        results.append(
+            {
+                'template': 'paper.html',
+                'url': result['fulltext']['access_url'],
+                'title': result['biblio']['title'],
+                'content': html_to_text(content),
+                'publisher': result['biblio'].get('publisher'),
+                'doi': doi,
+                'journal': result['biblio'].get('container_name'),
+                'authors': result['biblio'].get('contrib_names'),
+                'tags': result['tags'],
+                'publishedDate': publishedDate,
+                'issns': result['biblio'].get('issns'),
+                'pdf_url': result['fulltext'].get('access_url'),
+            }
+        )
+
+    return results


Test patch:
Repository: searxng/searxng

Issue:
Bug: internetarchivescholar engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2023.9.19+3ac7c40b6
<!-- Check if these values are correct -->

**How did you install SearXNG?**
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->

**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
<!-- Add any other context about the problem here. -->

**Technical report**

Error
  * Error: KeyError
  * Percentage: 25
  * Parameters: `()`
  * File name: `searx/engines/internet_archive_scholar.py:59`
  * Function: `response`
  * Code: `'title': result['biblio']['title'],`




Discussion:
Please share a search query to reproduce this with.

Resolving patch:
diff --git a/searx/engines/internet_archive_scholar.py b/searx/engines/internet_archive_scholar.py
--- a/searx/engines/internet_archive_scholar.py
+++ b/searx/engines/internet_archive_scholar.py
@@ -56,7 +56,7 @@ def response(resp):
             {
                 'template': 'paper.html',
                 'url': result['fulltext']['access_url'],
-                'title': result['biblio']['title'],
+                'title': result['biblio'].get('title') or result['biblio'].get('container_name'),
                 'content': html_to_text(content),
                 'publisher': result['biblio'].get('publisher'),
                 'doi': doi,


Test patch:
Repository: searxng/searxng

Issue:
Pinterest
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Working URL to the engine**
<!-- Please check if the engine is responding correctly before submitting it. -->

https://www.pinterest.com/search

**Why do you want to add this engine?**
<!-- What's special about this engine? Is it open source or libre? -->

Pinterest has great image search

**Features of this engine**
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->

Better image results

**How can SearXNG fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank.>

**Applicable category of this engine**
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:
general, files, images, it, map, music, news, science, social media and videos.
You can add multiple categories at the same time. -->

Images

**Additional context**
<!-- Add any other context about this engine here. -->


Discussion:

Resolving patch:
diff --git a/searx/engines/pinterest.py b/searx/engines/pinterest.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/pinterest.py
@@ -0,0 +1,63 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+"""Pinterest (images)
+"""
+
+from json import dumps
+
+about = {
+    "website": 'https://www.pinterest.com/',
+    "wikidata_id": 'Q255381',
+    "official_api_documentation": 'https://developers.pinterest.com/docs/api/v5/',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'JSON',
+}
+
+categories = ['images']
+paging = True
+
+base_url = 'https://www.pinterest.com'
+
+
+def request(query, params):
+    args = {
+        'options': {
+            'query': query,
+            'bookmarks': [params['engine_data'].get('bookmark', '')],
+        },
+        'context': {},
+    }
+    params['url'] = f"{base_url}/resource/BaseSearchResource/get/?data={dumps(args)}"
+
+    return params
+
+
+def response(resp):
+    results = []
+
+    json_resp = resp.json()
+
+    results.append(
+        {
+            'engine_data': json_resp['resource_response']['bookmark'],
+            # it's called bookmark by pinterest, but it's rather a nextpage
+            # parameter to get the next results
+            'key': 'bookmark',
+        }
+    )
+
+    for result in json_resp['resource_response']['data']['results']:
+        results.append(
+            {
+                'template': 'images.html',
+                'url': result['link'] or f"{base_url}/pin/{result['id']}/",
+                'title': result.get('title') or result.get('grid_title'),
+                'content': (result.get('rich_summary') or {}).get('display_description') or "",
+                'img_src': result['images']['orig']['url'],
+                'thumbnail_src': result['images']['236x']['url'],
+                'source': (result.get('rich_summary') or {}).get('site_name'),
+            }
+        )
+
+    return results


Test patch:
Repository: searxng/searxng

Issue:
Bug: bilibili engine is broken
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

Something has changed, and now some fixes are needed to use the api successfully.

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2023.9.27+1a66d7467+dirty
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
``` 
If you are using searxng-docker then look at the bottom of the SearXNG page
and check for the version after "Powered by SearXNG"

Please also stipulate if you are using a forked version of SearXNG and
include a link to the fork source code.
-->
**How did you install SearXNG?**
make run
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->

**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/searx/engines/bilibili.py b/searx/engines/bilibili.py
--- a/searx/engines/bilibili.py
+++ b/searx/engines/bilibili.py
@@ -26,7 +26,7 @@
 categories = ["videos"]
 
 # Search URL
-base_url = "https://api.bilibili.com/x/web-interface/wbi/search/type"
+base_url = "https://api.bilibili.com/x/web-interface/search/type"
 
 cookie = {
     "innersign": "0",


Test patch:
Repository: searxng/searxng

Issue:
Time duration for piped videos
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Is your feature request related to a problem? Please describe.**
Like piped music feature, it il possible to add time video duration please ?

**Additional context**
With music whe can see Length : 
![final_music](https://github.com/searxng/searxng/assets/55273439/0f0aebe9-281b-49cf-87f5-50dfdb0eceee)

With video we can not see Length : 
![final_video](https://github.com/searxng/searxng/assets/55273439/2c0f73b9-d5dd-40e3-9c8a-59c824699604)

Could you add this feature please ?
Thanks :)



Discussion:

Resolving patch:
diff --git a/searx/engines/piped.py b/searx/engines/piped.py
--- a/searx/engines/piped.py
+++ b/searx/engines/piped.py
@@ -140,6 +140,9 @@ def response(resp):
             "publishedDate": parser.parse(time.ctime(uploaded / 1000)) if uploaded != -1 else None,
             "iframe_src": _frontend_url() + '/embed' + result.get("url", ""),
         }
+        length = result.get("duration")
+        if length:
+            item["length"] = datetime.timedelta(seconds=length)
 
         if piped_filter == 'videos':
             item["template"] = "videos.html"
@@ -151,9 +154,6 @@ def response(resp):
             item["template"] = "default.html"
             item["img_src"] = result.get("thumbnail", "")
             item["content"] = result.get("uploaderName", "") or ""
-            length = result.get("duration")
-            if length:
-                item["length"] = datetime.timedelta(seconds=length)
 
         results.append(item)
 


Test patch:
Repository: searxng/searxng

Issue:
Owner request: Remove matrixrooms.info instance
Hi there,
I'm one of the etke.cc developers, and we operate MRS demo instance on matrixrooms.info / apicdn.matrixrooms.info

Please remove that **instance** from your code base - nobody contacted or notified us before its inclusion, and now we have to pay for the traffic your project generates.

Moreover, the integration metadata is completely invalid:

* [WikiData ID is wrong](https://www.wikidata.org/wiki/Q107565255)
* [Official API documentation is not provided](https://gitlab.com/etke.cc/mrs/api/-/blob/main/openapi.yml?ref_type=heads)
* It uses the official API
* Category should be instant messages

---

To avoid any misunderstanding, let me clarify some things.

1. You may want to integrate Matrix Rooms Search - self-hosted search engine over matrix federation: https://gitlab.com/etke.cc/mrs/api
2. You're supposed to host your own instance of MRS
3. The matrixrooms.info is a demo instance, and it is not intended for such integration
4. We pay for matrixrooms.info / apicdn.matrixrooms.info traffic from our own pockets and if it becomes unbearable, we can consider shutting it down.


Discussion:
@Bnyro @return42 

Resolving patch:
diff --git a/searx/engines/matrixrooms.py b/searx/engines/mrs.py
similarity index 54%
rename from searx/engines/matrixrooms.py
rename to searx/engines/mrs.py
--- a/searx/engines/matrixrooms.py
+++ b/searx/engines/mrs.py
@@ -1,26 +1,52 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""Matrixrooms.info (social media)
+"""Matrix Rooms Search - a fully-featured, standalone, matrix rooms search service.
+
+Configuration
+=============
+
+The engine has the following mandatory settings:
+
+- :py:obj:`base_url`
+
+.. code:: yaml
+
+  - name: MRS
+    engine: mrs
+    base_url: https://mrs-host
+    ...
+
+Implementation
+==============
 """
 
 from urllib.parse import quote_plus
 
 about = {
     "website": 'https://matrixrooms.info',
-    "wikidata_id": 'Q107565255',
-    "official_api_documentation": None,
-    "use_official_api": False,
+    "wikidata_id": None,
+    "official_api_documentation": 'https://gitlab.com/etke.cc/mrs/api/-/blob/main/openapi.yml?ref_type=heads',
+    "use_official_api": True,
     "require_api_key": False,
     "results": 'JSON',
 }
 paging = True
 categories = ['social media']
 
-base_url = "https://apicdn.matrixrooms.info"
+base_url = ""
 matrix_url = "https://matrix.to"
 page_size = 20
 
 
+def init(engine_settings):  # pylint: disable=unused-argument
+    """The ``base_url`` must be set in the configuration, if ``base_url`` is not
+    set, a :py:obj:`ValueError` is raised during initialization.
+
+    """
+    if not base_url:
+        raise ValueError('engine MRS, base_url is unset')
+
+
 def request(query, params):
     params['url'] = f"{base_url}/search/{quote_plus(query)}/{page_size}/{(params['pageno']-1)*page_size}"
     return params


Test patch:
Repository: searxng/searxng

Issue:
Keep the number of pages display when no results are found when page number is superior to 1
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Is your feature request related to a problem? Please describe.**
If you go to the next page but no engines return results, you lose the ability to easily come back to the previous page.

Last working page:
![image](https://github.com/searxng/searxng/assets/4016501/fe9f0deb-bed0-48cc-88ad-a0e40aca051f)

When you click on the next page (that do not return any results):
![image](https://github.com/searxng/searxng/assets/4016501/8610240d-4600-4431-99bd-f69097a78968)

**Describe the solution you'd like**
For a better user experience, it would be ideal to still display the number of pages but only keep the button "previous page".

Something like this:
![image](https://github.com/searxng/searxng/assets/4016501/2e9052ce-c960-4e83-bb91-2548c8c07ff3)

This display would only be available when the user is on another page than the first page because it wouldn't make sense on the first page.

**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->



Discussion:

Resolving patch:
diff --git a/searx/results.py b/searx/results.py
--- a/searx/results.py
+++ b/searx/results.py
@@ -232,7 +232,7 @@ def extend(self, engine_name, results):
         if engine_name in engines:
             histogram_observe(standard_result_count, 'engine', engine_name, 'result', 'count')
 
-        if not self.paging and standard_result_count > 0 and engine_name in engines and engines[engine_name].paging:
+        if not self.paging and engine_name in engines and engines[engine_name].paging:
             self.paging = True
 
     def _merge_infobox(self, infobox):


Test patch:
Repository: searxng/searxng

Issue:
Custom Links in the Footer
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Is your feature request related to a problem? Please describe.**

No.
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

**Describe the solution you'd like**

Support for custom footer links. Currently, all that can be set are links to git repos, project-related pages, and instance owner mailto:. I would like to link back to a status page for the things that I host, and any other arbitrary thing. Preferably this would be a key(link name) -> value(url) map somewhere in the config.
<!-- A clear and concise description of what you want to happen. -->

**Describe alternatives you've considered**

Editing the templates manually.
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

**Additional context**

<!-- Add any other context or screenshots about the feature request here. -->



Discussion:

Resolving patch:
diff --git a/searx/settings_defaults.py b/searx/settings_defaults.py
--- a/searx/settings_defaults.py
+++ b/searx/settings_defaults.py
@@ -151,6 +151,7 @@ def apply_schema(settings, schema, path_list):
         'docs_url': SettingsValue(str, 'https://docs.searxng.org'),
         'public_instances': SettingsValue((False, str), 'https://searx.space'),
         'wiki_url': SettingsValue(str, 'https://github.com/searxng/searxng/wiki'),
+        'custom': SettingsValue(dict, {'links': {}}),
     },
     'search': {
         'safe_search': SettingsValue((0, 1, 2), 0),


Test patch:
Repository: searxng/searxng

Issue:
Bug: lingva engine / redirects & Key-Errors
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**

Repository: https://github.com/return42/searxng
Branch: darmarit.org
Version: 2024.2.3+a6f5d690

**How did you install SearXNG?**

(unmodified fork/brand) from master branch

**What happened?**

With the default config / the "official instance" we have the errors reported below:

https://github.com/searxng/searxng/blob/df1a774003c285866a96b149bf92412037b4932d/searx/settings.yml#L1037-L1041

**How To Reproduce**

```
!lingva en-de convenient
```

**Technical report**

```
Error
  * Error: httpx.ReadTimeout
  * Percentage: 50
  * Parameters: `(None, None, 'lingva.thedaviddelta.com')`
  * File name: `searx/search/processors/online.py:118`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`
```

```
Error
  * Error: 1 redirects, maximum: 0
  * Percentage: 50
  * Parameters: `('200', 'OK', 'lingva.thedaviddelta.com')`
  * File name: `searx/search/processors/online.py:127`
  * Function: `_send_http_request`
  * Code: `count_error(`
```

```
Error
  * Error: KeyError
  * Percentage: 50
  * Parameters: `()`
  * File name: `searx/engines/lingva.py:51`
  * Function: `response`
  * Code: `infobox += f"<b>{translation['type']}</b>"`
```


Discussion:

Resolving patch:
diff --git a/searx/engines/lingva.py b/searx/engines/lingva.py
--- a/searx/engines/lingva.py
+++ b/searx/engines/lingva.py
@@ -16,7 +16,7 @@
 engine_type = 'online_dictionary'
 categories = ['general']
 
-url = "https://lingva.thedaviddelta.com/"
+url = "https://lingva.thedaviddelta.com"
 search_url = "{url}/api/v1/{from_lang}/{to_lang}/{query}"
 
 
@@ -48,8 +48,6 @@ def response(resp):
     infobox = ""
 
     for translation in info["extraTranslations"]:
-        infobox += f"<b>{translation['type']}</b>"
-
         for word in translation["list"]:
             infobox += f"<dl><dt>{word['word']}</dt>"
 


Test patch:
Repository: searxng/searxng

Issue:
remove ask engine
Reasons:

1. User Tracking via query parameters.

    Example Query: [!ask best museums in Köln](https://searx.be/search?q=%21ask+best+museums+in+K%C3%B6ln&category_none=&language=auto&time_range=&safesearch=0&theme=simple)

    Notice the ```euid=some_id``` at the end for most urls.

2. Very bad results - Something must have changed.

   Example Query: [!ask "yt-dlp"](https://searx.be/search?q=%21ask+%22yt-dlp%22&category_none=&language=auto&time_range=&safesearch=0&theme=simple)




Discussion:

Resolving patch:
diff --git a/searx/engines/ask.py b/searx/engines/ask.py
--- a/searx/engines/ask.py
+++ b/searx/engines/ask.py
@@ -19,6 +19,7 @@
 # Engine Configuration
 categories = ['general']
 paging = True
+max_page = 5
 
 # Base URL
 base_url = "https://www.ask.com/web"
@@ -61,7 +62,7 @@ def response(resp):
 
         results.append(
             {
-                "url": item['url'],
+                "url": item['url'].split('&ueid')[0],
                 "title": item['title'],
                 "content": item['abstract'],
                 "publishedDate": pubdate_original,


Test patch: