internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searx/searx

Issue:
AttributeError: 'module' object has no attribute 'old_where'
I updated my searx instance today, and got the following error:

```
Traceback (most recent call last):
  File "/usr/local/searx/searx/__init__.py", line 55, in <module>
    environ['REQUESTS_CA_BUNDLE'] = certifi.old_where()
AttributeError: 'module' object has no attribute 'old_where'
```

I updated the dependencies with `pip install --upgrade -r requirements.txt` before running searx.



Discussion:

Resolving patch:
diff --git a/searx/__init__.py b/searx/__init__.py
--- a/searx/__init__.py
+++ b/searx/__init__.py
@@ -52,7 +52,8 @@
 # Workaround for openssl versions <1.0.2
 # https://github.com/certifi/python-certifi/issues/26
 if OPENSSL_VERSION_INFO[0:3] < (1, 0, 2):
-    environ['REQUESTS_CA_BUNDLE'] = certifi.old_where()
+    if hasattr(certifi, 'old_where'):
+        environ['REQUESTS_CA_BUNDLE'] = certifi.old_where()
     logger.warning('You are using an old openssl version({0}), please upgrade above 1.0.2!'.format(OPENSSL_VERSION))
 
 logger.info('Initialisation done')


Test patch:
Repository: searx/searx

Issue:
Wish: API to programmatically query capabilities
I have a practical need to be able to know beforehand from my program which engines and plugins a given Searx server supports and/or has enabled. Right now, the only way I know of that would allow me to do that would be to scrape the Preferences HTML. That is not ideal.

Searx would probably benefit from wider API in general, but this is probably the most immediately obvious missing thing to tackle after having an interface to obtain search results themselves (which we have).

My use case, in case this is of interest, involves having multiple Searx servers at my disposal (most of the publicly available ones), and being able to choose the ideal one not just randomly but according to what it offers. This general problem may be related to things proposed in other issues, involving the ability to use Searx itself as an engine, and automatically picking a Searx server.



Discussion:

Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -49,6 +49,7 @@
     redirect, send_from_directory
 )
 from flask.ext.babel import Babel, gettext, format_date
+from flask.json import jsonify
 from searx import settings, searx_dir
 from searx.engines import (
     categories, engines, get_engines_stats, engine_shortcuts
@@ -685,6 +686,24 @@ def clear_cookies():
     return resp
 
 
+@app.route('/config')
+def config():
+    return jsonify({'categories': categories.keys(),
+                    'engines': [{'name': engine_name,
+                                 'categories': engine.categories,
+                                 'enabled': not engine.disabled}
+                                for engine_name, engine in engines.items()],
+                    'plugins': [{'name': plugin.name,
+                                 'enabled': plugin.default_on}
+                                for plugin in plugins],
+                    'instance_name': settings['general']['instance_name'],
+                    'locales': settings['locales'],
+                    'default_locale': settings['ui']['default_locale'],
+                    'autocomplete': settings['search']['autocomplete'],
+                    'safe_search': settings['search']['safe_search'],
+                    'default_theme': settings['ui']['default_theme']})
+
+
 def run():
     app.run(
         debug=settings['general']['debug'],


Test patch:
Repository: searx/searx

Issue:
Idea] An endless research
Hi,

Is it possible to add an endless research , like here for google search : https://openuserjs.org/scripts/tumpio/Endless_Google

?



Discussion:
Here's a quick snippet that does this for the oscar theme:

``` js
$(document).ready(function() {
    var win = $(window);
    win.scroll(function() {
        if ($(document).height() - win.height() == win.scrollTop()) {
            var formData = $('#pagination form:last').serialize();
            $('#pagination').html('Loading next page...');
            $.post('/', formData, function (data) {
                var body = $(data);
                $('#pagination').remove();
                $('#main_results').append(body.find('.result'));
                $('#main_results').append(body.find('#pagination'));
            });
        }
    });
});
```

However, I'm not sure what the best way to generalise it for all the themes would be.

@davidar currently only oscar theme supports client side plugins.

@asciimoo ah, that makes it easier :)

thanks a lot ! :)


Resolving patch:
diff --git a/searx/plugins/__init__.py b/searx/plugins/__init__.py
--- a/searx/plugins/__init__.py
+++ b/searx/plugins/__init__.py
@@ -21,6 +21,7 @@
 
 from searx.plugins import (doai_rewrite,
                            https_rewrite,
+                           infinite_scroll,
                            open_results_on_new_tab,
                            self_info,
                            search_on_category_select,
@@ -76,6 +77,7 @@ def call(self, plugin_type, request, *args, **kwargs):
 plugins = PluginStore()
 plugins.register(doai_rewrite)
 plugins.register(https_rewrite)
+plugins.register(infinite_scroll)
 plugins.register(open_results_on_new_tab)
 plugins.register(self_info)
 plugins.register(search_on_category_select)
diff --git a/searx/plugins/infinite_scroll.py b/searx/plugins/infinite_scroll.py
new file mode 100644
--- /dev/null
+++ b/searx/plugins/infinite_scroll.py
@@ -0,0 +1,8 @@
+from flask_babel import gettext
+
+name = gettext('Infinite scroll')
+description = gettext('Automatically load next page when scrolling to bottom of current page')
+default_on = False
+
+js_dependencies = ('plugins/js/infinite_scroll.js',)
+css_dependencies = ('plugins/css/infinite_scroll.css',)


Test patch:
Repository: searx/searx

Issue:
custom error-sites
We could support custom error sites inside the templates, to show for example 404-Errors in the design of the template instead of some naked text. (continuous design).

But we have to be cautious with 5xx-Errors caused by template problems (fallback-solution, if the error-template is not working like expected)



Discussion:
404 error page can be the search page with a clear message.
- it's avoid one click to go elsewhere most of the time ( the purpose of searx is to search something )
- it's simple to implement

Just an idea.

If we display one dedicated error page, we should provide links to elsewhere. 
It's more or less "free" with the oscar theme, but not with the others themes.


Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -715,6 +715,11 @@ def config():
                     'default_theme': settings['ui']['default_theme']})
 
 
+@app.errorhandler(404)
+def page_not_found(e):
+    return render('404.html')
+
+
 def run():
     app.run(
         debug=settings['general']['debug'],


Test patch:
diff --git a/tests/robot/test_basic.robot b/tests/robot/test_basic.robot
--- a/tests/robot/test_basic.robot
+++ b/tests/robot/test_basic.robot
@@ -9,6 +9,11 @@ Front page
     Page Should Contain  about
     Page Should Contain  preferences
 
+404 page
+    Go To  http://localhost:11111/no-such-page
+    Page Should Contain  Page not found
+    Page Should Contain  Go to search page
+
 About page
     Click Element  link=about
     Page Should Contain  Why use Searx?
Repository: searx/searx

Issue:
Fixed two typos in index.html, fixed one typo in dev/plugins.html, an…
…d fixed grammar in dev/engine_overview.html



Discussion:
Hey! Thanks for the docs contrib! However, could you please follow the steps in it (https://asciimoo.github.io/searx/dev/contribution_guide.html#documentation) and commit the other required files along with the html versions?

Sorry about that, I must have missed that. I'll fix it.


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -57,9 +57,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.9.0'
+version = '0.10.0'
 # The full version, including alpha/beta/rc tags.
-release = '0.9.0'
+release = '0.10.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.


Test patch:
Repository: searx/searx

Issue:
Tags <xml> are hidden from result titles



Discussion:
Example : https://searx.me/?q=%21google+xml+header+syntax&category_general=on&time_range=

The title to the link 
http://stackoverflow.com/questions/13743250/meaning-of-xml-version-1-0-encoding-utf-8
should be
`xml declaration - Meaning of - <?xml version="1.0" encoding="utf-8`
it is
`xml declaration - Meaning of -`
in searx

(the `<?xml version="1.0" encoding="utf-8` is not in the HTML source).

Technical details : 
* [```extract_text```](https://github.com/asciimoo/searx/blob/master/searx/engines/google.py#L223) is used in google.py
* Inside this function, [```xpath_results.text_content()```](https://github.com/asciimoo/searx/blob/master/searx/engines/xpath.py#L31) returns ```... <?xml ...``` instead the original content ```... &lt;?xml ...```
* Then [```html_to_text```](https://github.com/asciimoo/searx/blob/7eed8a5dd909ca70d45fdff5706bb207e6aa8bdf/searx/utils.py#L127) sanitizes the content, and removes the thing that look like an xml tag

About ```xpath_results.text_content()``` it seems there are an option not to decode entities for the XML parser but not for the HTML parser : http://lxml.de/parsing.html#parsers

Perhaps extract_text should uses [```etree.tostring(html, method="text")```](http://lxml.de/tutorial.html)


Resolving patch:
diff --git a/searx/engines/xpath.py b/searx/engines/xpath.py
--- a/searx/engines/xpath.py
+++ b/searx/engines/xpath.py
@@ -42,7 +42,9 @@ def extract_text(xpath_results):
         return ''.join(xpath_results)
     else:
         # it's a element
-        return html_to_text(xpath_results.text_content()).strip()
+        text = html.tostring(xpath_results, encoding='unicode', method='text', with_tail=False)
+        text = text.strip().replace('\n', ' ')
+        return ' '.join(text.split())
 
 
 def extract_url(xpath_results, search_url):


Test patch:
Repository: searx/searx

Issue:
Add a software categorie and add the Free software directory search engine
Shame on me I forgot to ask this.
I am a volunteer on the [FSD](https://directory.fsf.org/wiki/Main_Page) (Free software directory)
It would be nice if people could look for free/libre software in the searx engine.
When possible could someone please add the free software directory so that people can easily find free software.


Discussion:

Resolving patch:
diff --git a/searx/engines/mediawiki.py b/searx/engines/mediawiki.py
--- a/searx/engines/mediawiki.py
+++ b/searx/engines/mediawiki.py
@@ -21,6 +21,7 @@
 language_support = True
 paging = True
 number_of_results = 1
+search_type = 'nearmatch' # possible values: title, text, nearmatch
 
 # search-url
 base_url = 'https://{language}.wikipedia.org/'
@@ -30,7 +31,7 @@
     '&format=json'\
     '&sroffset={offset}'\
     '&srlimit={limit}'\
-    '&srwhat=nearmatch'  # search for a near match in the title
+    '&srwhat={searchtype}'
 
 
 # do search-request
@@ -39,7 +40,8 @@ def request(query, params):
 
     string_args = dict(query=urlencode({'srsearch': query}),
                        offset=offset,
-                       limit=number_of_results)
+                       limit=number_of_results,
+                       searchtype=search_type)
 
     format_strings = list(Formatter().parse(base_url))
 


Test patch:
Repository: searx/searx

Issue:
[bug] Python 3.6 Autocomplete not work
Use searx with archlinux and python 3.6 [https://aur.archlinux.org/packages/searx-py3](https://aur.archlinux.org/packages/searx-py3)
Autocomplete not working, log :
```
     rv = self.dispatch_request()
   File "/usr/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "/usr/lib/python3.6/site-packages/searx/webapp.py", line 609, in autocompleter
     raw_text_query.parse_query()
   File "/usr/lib/python3.6/site-packages/searx/query.py", line 55, in parse_query
     raw_query_parts = re.split(r'(\s+)', self.query)
   File "/usr/lib/python3.6/re.py", line 212, in split
     return _compile(pattern, flags).split(string, maxsplit)
 TypeError: cannot use a string pattern on a bytes-like object
```


Discussion:
on my instance i fix this bug in file
/searx/query.py line 55
```python
raw_query_parts = re.split(r'(\s+)' if isinstance(self.query,str) else b'(\s+)', self.query)
```
no test on pyhton 2.6  but I think it's more scalable ..
Good solution indeed! Don't you want to create a PR with these changes?

Resolving patch:
diff --git a/searx/query.py b/searx/query.py
--- a/searx/query.py
+++ b/searx/query.py
@@ -51,7 +51,7 @@ def parse_query(self):
         self.query_parts = []
 
         # split query, including whitespaces
-        raw_query_parts = re.split(r'(\s+)', self.query)
+        raw_query_parts = re.split(r'(\s+)' if isinstance(self.query, str) else b'(\s+)', self.query)
 
         parse_next = True
 


Test patch:
Repository: searx/searx

Issue:
Yacy results crash
Getting:
Engines cannot retrieve results:
yacy (unexpected crash)

> ERROR:searx.search:engine yacy : exception : 'url'
> Traceback (most recent call last):
>   File "/home/leo/searx/searx/search.py", line 118, in search_one_request_safe
>     search_results = search_one_request(engine, query, request_params, start_time, timeout_limit)
>   File "/home/leo/searx/searx/search.py", line 110, in search_one_request
>     return engine.response(response)
>   File "/home/leo/searx/searx/engines/yacy.py", line 80, in response
>     results.append({'url': result['url'],
> KeyError: 'url'


Discussion:
I tried reproducing your issue with a local YaCy instance. But everything works as expected. Which instance are you using?
I have the same issue for more than a month now. I also use a local YaCy instance. You can reach it from the outside at https://yacy.schefczyk.net/

Resolving patch:
diff --git a/searx/engines/yacy.py b/searx/engines/yacy.py
--- a/searx/engines/yacy.py
+++ b/searx/engines/yacy.py
@@ -74,8 +74,17 @@ def response(resp):
     for result in search_results[0].get('items', []):
         # parse image results
         if result.get('image'):
+
+            result_url = ''
+            if 'url' in result:
+                result_url = result['url']
+            elif 'link' in result:
+                result_url = result['link']
+            else:
+                continue
+
             # append result
-            results.append({'url': result['url'],
+            results.append({'url': result_url,
                             'title': result['title'],
                             'content': '',
                             'img_src': result['image'],


Test patch:
Repository: searx/searx

Issue:
eBay Search 
eBay search would be greatly appreciated! 


Discussion:

Resolving patch:
diff --git a/searx/engines/ebay.py b/searx/engines/ebay.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/ebay.py
@@ -0,0 +1,68 @@
+#  Ebay (Videos, Music, Files)
+#
+# @website     https://www.ebay.com
+# @provide-api no (nothing found)
+#
+# @using-api   no
+# @results     HTML (using search portal)
+# @stable      yes (HTML can change)
+# @parse       url, title, content, price, shipping, source
+
+from lxml import html
+from searx.engines.xpath import extract_text
+from urllib.parse import quote
+
+categories = ['shopping']
+paging = True
+
+url = 'https://www.ebay.com'
+search_url = url + '/sch/i.html?_nkw={query}&_sacat={pageno}'
+
+results_xpath = '//li[contains(@class, "s-item")]'
+url_xpath = './/a[@class="s-item__link"]/@href'
+title_xpath = './/h3[@class="s-item__title"]'
+content_xpath = './/div[@span="SECONDARY_INFO"]'
+price_xpath = './/div[contains(@class, "s-item__detail")]/span[@class="s-item__price"][1]/text()'
+shipping_xpath = './/span[contains(@class, "s-item__shipping")]/text()'
+source_country_xpath = './/span[contains(@class, "s-item__location")]/text()'
+thumbnail_xpath = './/img[@class="s-item__image-img"]/@src'
+
+
+def request(query, params):
+    params['url'] = search_url.format(query=quote(query), pageno=params['pageno'])
+    return params
+
+
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+    results_dom = dom.xpath(results_xpath)
+    if not results_dom:
+        return []
+
+    for result_dom in results_dom:
+        url = extract_text(result_dom.xpath(url_xpath))
+        title = extract_text(result_dom.xpath(title_xpath))
+        content = extract_text(result_dom.xpath(content_xpath))
+        price = extract_text(result_dom.xpath(price_xpath))
+        shipping = extract_text(result_dom.xpath(shipping_xpath))
+        source_country = extract_text(result_dom.xpath(source_country_xpath))
+        thumbnail = extract_text(result_dom.xpath(thumbnail_xpath))
+
+        if title == "":
+            continue
+
+        results.append({
+            'url': url,
+            'title': title,
+            'content': content,
+            'price': price,
+            'shipping': shipping,
+            'source_country': source_country,
+            'thumbnail': thumbnail,
+            'template': 'products.html',
+
+        })
+
+    return results


Test patch:
Repository: searx/searx

Issue:
findx crashes
... with message on web page: findx (unexpected crash: string indices must be integers) 


Discussion:

Resolving patch:
diff --git a/searx/engines/findx.py b/searx/engines/findx.py
--- a/searx/engines/findx.py
+++ b/searx/engines/findx.py
@@ -49,7 +49,7 @@ def response(resp):
     results_json = loads(extract_text(results_raw_json))
 
     if len(results_json['web']['results']) > 0:
-        return _general_results(results_json['web']['results'])
+        return _general_results(results_json['web']['results']['webSearch']['results'])
 
     if len(results_json['images']['results']) > 0:
         return _images_results(results_json['images']['results'])


Test patch:
Repository: searx/searx

Issue:
Gigablast crash
Greetings,

I have been experimenting with SearX lately and have been seeing this message:

"
Engines cannot retrieve results:

gigablast (unexpected crash: No JSON object could be decoded)
"

Seems like something is wrong with the Gigablast driver but I am not sure how to fix it.

I'm using:  searx - 0.14.0 

Thanks


Discussion:

Resolving patch:
diff --git a/searx/engines/gigablast.py b/searx/engines/gigablast.py
--- a/searx/engines/gigablast.py
+++ b/searx/engines/gigablast.py
@@ -34,6 +34,7 @@
     '&qlang={lang}'\
     '&ff={safesearch}'\
     '&rxiec={rxieu}'\
+    '&ulse={ulse}'\
     '&rand={rxikd}'  # current unix timestamp
 
 # specific xpath variables
@@ -64,6 +65,7 @@ def request(query, params):
                                        number_of_results=number_of_results,
                                        rxikd=int(time() * 1000),
                                        rxieu=random.randint(1000000000, 9999999999),
+                                       ulse=random.randint(100000000, 999999999),
                                        lang=language,
                                        safesearch=safesearch)
 


Test patch:
Repository: searx/searx

Issue:
Is Bing feeding garbage results to searx?
I've noticed most results from bing are irrelevant. Not even based on any of the words I use on my queries.

For example, the query "youtube dash stream full hd" returns as a frist results a tesco link to buy a tv. I guess the "full hd" triggered that maybe, but if you scroll down the 7th result is a link to the FBI podcast (???). For that particular query out of 7 bing results only 1 is maybe relevant.

Here are other query examples that yield garbage resuts.

Query: easy privacy list . 1st result is a page called muscchoice, completely irrelevant. 3rd bing result is a page called growweedeasy, again nothing to do with my query. out of 10 bing results for this query once again maybe just one is partially relevant.

Query: adblock plus rules div tag. This time the first result is correct from both bing and google, but the 4th one (3rd bing result) is for a page called globolister. with the title "vote for icefilmsdotinfo" which is unrelated to my query

Query: "reply all" podcasts. For this one I quoted the term "reply all" which is the name of the podcast. and the very first result from bing point to buzzfeed podcast. I feel like this is just plain advertisement


Query: can vitamin c audio. For a more confusing term, since the the query contains words used in other contexts, the first bing result links to amazonand what  I'm actually looking for is a song. google gets it right .

Query: bert jansch "needle of death" lyrics. Another quoted query, the first result is from a page called popmatters talking about another song, and it doesn't even contain the lyrics. the 3rd link is from a newspaper that makes a small reference to the singer,nothing else. 4th result is for a page called michaelchapman, irrelevant. Out of 6 bing results, again, maybe a couple are partially relevant, none are what I'm looking for.


I could keep adding queries but I think I made my point very clear. In my opinion Bing is feeding garbage to searx. Or maybe it is just that bing is that bad. So I would like to request the removal of Bing as a default search engine


Discussion:

Resolving patch:
diff --git a/searx/engines/bing.py b/searx/engines/bing.py
--- a/searx/engines/bing.py
+++ b/searx/engines/bing.py
@@ -16,7 +16,7 @@
 from lxml import html
 from searx.engines.xpath import extract_text
 from searx.url_utils import urlencode
-from searx.utils import match_language
+from searx.utils import match_language, gen_useragent
 
 # engine dependent config
 categories = ['general']
@@ -43,6 +43,9 @@ def request(query, params):
         offset=offset)
 
     params['url'] = base_url + search_path
+
+    params['headers']['User-Agent'] = gen_useragent('Windows NT 6.3; WOW64')
+
     return params
 
 
diff --git a/searx/utils.py b/searx/utils.py
--- a/searx/utils.py
+++ b/searx/utils.py
@@ -57,9 +57,9 @@
                 'style')
 
 
-def gen_useragent():
+def gen_useragent(os=None):
     # TODO
-    return ua.format(os=choice(ua_os), version=choice(ua_versions))
+    return ua.format(os=os or choice(ua_os), version=choice(ua_versions))
 
 
 def searx_useragent():


Test patch: