internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: searx/searx

Issue:
New release?
Hi,

I'm the Debian maintainer of searx and we are still shipping version 0.14.0 of searx because there has not been a more recent release since February.

Unfortunately, we see a number of services that do not work anymore with 0.14.0 but which are fixed in git. Would it be possible to make a release soon?

The last commit to the repository was back in August. Is the project still alive? Debian plans to release its next stable version soon and we should not include searx if it's not maintained anymore.

Thanks!
cheers, josch


Discussion:
> Is the project still alive?

Good question, because I'm not aware about any alternatives...
https://github.com/asciimoo/searx/issues/1409 :-)
Maybe he could appoint/ask for co-maintainers to the repo who can merge, test and fix the many pull requests. This project should be kept alive, many people uses it and its everyone's best interest to protect our privacy. 
Yes, I agree with this, he needs co-maintainers if he busy.
yeah i have a dockerhub image on ARM64 too, i hope we can have a new tag asap with new features and/or bugfixes.
He should find a good co maintainer, it's not easy
I hate to say this, and I mean, I REALLY HATE to say this... but searx may need to be forked if he doesn't find a co-maintainer.  It would be desperate though... 
I really think most would prefer to have this not happen.  but sooner or later, someone has to do something. Of course, you could make an unofficial searx fork until he gets back on his feet so to speak. Ask him if that's okay, same name, but different.
@asciimoo would an unofficial searx fork with the same name be okay for you, Adam? We/some of the guys here or at #1409 would test and merge many of the existing PRs, and then create one _huge_ PR for you to merge as 15.0? How is that sound to ease up the maintaining process for you? I'm/we're asking it with all due respect for your work so far and in the name of keeping this project alive, useful and prosperous. 
I will try to pick up the missing PRs and create a new release this weekend. Sorry for keeping you waiting.

Resolving patch:
diff --git a/searx/version.py b/searx/version.py
--- a/searx/version.py
+++ b/searx/version.py
@@ -18,7 +18,7 @@
 
 # version of searx
 VERSION_MAJOR = 0
-VERSION_MINOR = 14
+VERSION_MINOR = 15
 VERSION_BUILD = 0
 
 VERSION_STRING = "{0}.{1}.{2}".format(VERSION_MAJOR,


Test patch:
Repository: searx/searx

Issue:
Asksteem is gone
The API has been discontinued so it should probably be removed as an option entirely.


Discussion:

Resolving patch:
diff --git a/searx/engines/asksteem.py b/searx/engines/asksteem.py
deleted file mode 100644
--- a/searx/engines/asksteem.py
+++ /dev/null
@@ -1,44 +0,0 @@
-"""
- Asksteem (general)
-
- @website     https://asksteem.com/
- @provide-api yes
-
- @using-api   yes
- @results     JSON (https://github.com/Hoxly/asksteem-docs/wiki)
- @stable      yes
- @parse       url, title, content
-"""
-
-from json import loads
-from searx.url_utils import urlencode
-
-# engine dependent config
-categories = ['general']
-paging = True
-language_support = False
-disabled = True
-
-# search-url
-search_url = 'https://api.asksteem.com/search?{params}'
-result_url = 'https://steemit.com/@{author}/{title}'
-
-
-# do search-request
-def request(query, params):
-    url = search_url.format(params=urlencode({'q': query, 'pg': params['pageno']}))
-    params['url'] = url
-    return params
-
-
-# get response from search-request
-def response(resp):
-    json = loads(resp.text)
-
-    results = []
-
-    for result in json.get('results', []):
-        results.append({'url': result_url.format(author=result['author'], title=result['permlink']),
-                        'title': result['title'],
-                        'content': result['summary']})
-    return results


Test patch:
Repository: searx/searx

Issue:
Timeout limit breaks moving between pages
Timeout limits introduced on https://github.com/asciimoo/searx/commit/72029d27ded8d93ab891c616d6bffbe8d3a67dd2 cause pressing "next page" button to cause in a search error.

Returning back to commit 2179079a9173b33b81e1084fc1e8e181c19ef8e9 fixes this problem.

![](https://elixi.re/i/7q0l9w8t.png)

```
ERROR:searx.webapp:search error
Traceback (most recent call last):
  File "searx/webapp.py", line 510, in index
    search_query, raw_text_query = get_search_query_from_webapp(request.preferences, request.form)
  File "/home/ave/Projects/searx/searx/search.py", line 285, in get_search_query_from_webapp
    raise SearxParameterException('timeout_limit', raw_time_limit)
SearxParameterException: Invalid value "None" for parameter timeout_limit
```

edit; for some reason, it seems like it doesn't happen now as I'm testing, but I will wait until I figure out why it happened in the first place to close it.


Discussion:
It seems like timeout_limit is returned as None in some cases, which is what causes this issue:

![](https://elixi.re/i/3nhez50p.png)

No clue yet why it happens though.
Setting a default limit in

https://github.com/asciimoo/searx/blob/72029d27ded8d93ab891c616d6bffbe8d3a67dd2/searx/webapp.py#L632

in place of `None` seems to fix the problem, though there's likely a more elegant fix for this.

Resolving patch:
diff --git a/searx/search.py b/searx/search.py
--- a/searx/search.py
+++ b/searx/search.py
@@ -279,10 +279,13 @@ def get_search_query_from_webapp(preferences, form):
     query_timeout = raw_text_query.timeout_limit
     if query_timeout is None and 'timeout_limit' in form:
         raw_time_limit = form.get('timeout_limit')
-        try:
-            query_timeout = float(raw_time_limit)
-        except ValueError:
-            raise SearxParameterException('timeout_limit', raw_time_limit)
+        if raw_time_limit in ['None', '']:
+            raw_time_limit = None
+        else:
+            try:
+                query_timeout = float(raw_time_limit)
+            except ValueError:
+                raise SearxParameterException('timeout_limit', raw_time_limit)
 
     # query_categories
     query_categories = []


Test patch:
Repository: searx/searx

Issue:
Google Images & DeviantArt don't work anymore
From one day to another, Google Images and DeviantArt stopped to show me anything, even with simple searches.
![Screenshot_1](https://user-images.githubusercontent.com/48600424/63694426-f73ef200-c805-11e9-93a9-10124fb32b85.png)
![Screenshot_2](https://user-images.githubusercontent.com/48600424/63694427-f7d78880-c805-11e9-9e37-87042fba85a6.png)
They are of course activated in the engines. It has been a few days, with a restart every day (local instance using Docker), without modifying engines nor any other setting. Using searx 0.15.0


Discussion:

Resolving patch:
diff --git a/searx/engines/google_images.py b/searx/engines/google_images.py
--- a/searx/engines/google_images.py
+++ b/searx/engines/google_images.py
@@ -70,11 +70,21 @@ def response(resp):
 
         try:
             metadata = loads(result)
-            img_format = "{0} {1}x{2}".format(metadata['ity'], str(metadata['ow']), str(metadata['oh']))
-            source = "{0} ({1})".format(metadata['st'], metadata['isu'])
+
+            img_format = metadata.get('ity', '')
+            img_width = metadata.get('ow', '')
+            img_height = metadata.get('oh', '')
+            if img_width and img_height:
+                img_format += " {0}x{1}".format(img_width, img_height)
+
+            source = metadata.get('st', '')
+            source_url = metadata.get('isu', '')
+            if source_url:
+                source += " ({0})".format(source_url)
+
             results.append({'url': metadata['ru'],
                             'title': metadata['pt'],
-                            'content': metadata['s'],
+                            'content': metadata.get('s', ''),
                             'source': source,
                             'img_format': img_format,
                             'thumbnail_src': metadata['tu'],


Test patch:
Repository: searx/searx

Issue:
No results with google search
Hello,

I can't search anything if only google is selected. No matter what i search, i get the message «Sorry! we didn't find any results. Please use another query or search in more categories. »


Discussion:
I'm pretty sure Google is pushing hard again their new UI like discuted here: #1596 and the pull request #1597 was a temporary fix.

Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -199,8 +199,9 @@ def request(query, params):
     params['headers']['Accept-Language'] = language + ',' + language + '-' + country
     params['headers']['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
 
-    # Force Internet Explorer 12 user agent to avoid loading the new UI that Searx can't parse
-    params['headers']['User-Agent'] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
+    # Force Safari 3.1 on Mac OS X (Leopard) user agent to avoid loading the new UI that Searx can't parse
+    params['headers']['User-Agent'] = ("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4)"
+                                       "AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1")
 
     params['google_hostname'] = google_hostname
 


Test patch:
Repository: searx/searx

Issue:
Startpage parser breaks on Spanish date format
To reproduce, follow these steps:

1. open searx.info (or another instance)

2. search for a spanish term, eg "madrid tiempo ahora"

This results in an error like the following:
```
Los motores no pueden obtener resultados: (english: engines cant obtain results)

startpage (unexpected crash: ('Unknown string format:', '22 Ene 2020'))
```


Discussion:

Resolving patch:
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -99,11 +99,14 @@ def response(resp):
         if re.match(r"^([1-9]|[1-2][0-9]|3[0-1]) [A-Z][a-z]{2} [0-9]{4} \.\.\. ", content):
             date_pos = content.find('...') + 4
             date_string = content[0:date_pos - 5]
-            published_date = parser.parse(date_string, dayfirst=True)
-
             # fix content string
             content = content[date_pos:]
 
+            try:
+                published_date = parser.parse(date_string, dayfirst=True)
+            except ValueError:
+                pass
+
         # check if search result starts with something like: "5 days ago ... "
         elif re.match(r"^[0-9]+ days? ago \.\.\. ", content):
             date_pos = content.find('...') + 4


Test patch:
Repository: searx/searx

Issue:
Hiding timeout error for one engine
Is it possible to hide/disable timeout error for one engine?


Discussion:
Just set the default timeout to a large number?
> Just set the default timeout to a large number?

The thing is I want to decrease the timeout as much as possible to make my searching even faster. There are some engines that are irrelevant to me and I'd be ok with "hit or miss", but the whole timeout thingy is annoying.
It might make more sense to disable the search engines you don't care about in the searx/searx/settings.yml file.
> It might make more sense to disable the search engines you don't care about in the searx/searx/settings.yml file.

Yeah, I disabled some but there are some that I'd rather keep with "hit or miss" :)
Okay, that makes sense.

Resolving patch:
diff --git a/searx/engines/__init__.py b/searx/engines/__init__.py
--- a/searx/engines/__init__.py
+++ b/searx/engines/__init__.py
@@ -55,6 +55,7 @@
                        'continuous_errors': 0,
                        'time_range_support': False,
                        'offline': False,
+                       'display_error_messages': True,
                        'tokens': []}
 
 
diff --git a/searx/results.py b/searx/results.py
--- a/searx/results.py
+++ b/searx/results.py
@@ -346,7 +346,8 @@ def results_number(self):
         return resultnum_sum / len(self._number_of_results)
 
     def add_unresponsive_engine(self, engine_name, error_type, error_message=None):
-        self.unresponsive_engines.add((engine_name, error_type, error_message))
+        if engines[engine_name].display_error_messages:
+            self.unresponsive_engines.add((engine_name, error_type, error_message))
 
     def add_timing(self, engine_name, engine_time, page_load_time):
         self.timings.append({


Test patch:
Repository: searx/searx

Issue:
Category Order
What determines the category order? Is there a way to change it? Any setting in the settings.yml?



Discussion:
Categories are sorted by alphabetic order except for the general which is the first one.
This order can't be modified.
See : https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L268

Related to https://github.com/asciimoo/searx/issues/258
It should be configured by settings.yml IHMO.

> On 7 Jun 2016, at 7:52 PM, Alexandre Flament notifications@github.com wrote:
> 
> Categories are sorted by alphabetic order except for the general which is the first one.
> This order can't be modified.
> See : https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L268 https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L268
> Related to #258 https://github.com/asciimoo/searx/issues/258
> It should be configured by settings.yml IHMO.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub https://github.com/asciimoo/searx/issues/573#issuecomment-224233338, or mute the thread https://github.com/notifications/unsubscribe/AHdxhqWIF2tTYahXQWIJ9WqTvt-5uop6ks5qJT9YgaJpZM4IpOJb.

i agree, settings.yml OR by users for their own settings.


Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -355,17 +355,12 @@ def render(template_name, override_theme=None, **kwargs):
                              if (engine_name, category) not in disabled_engines)
 
     if 'categories' not in kwargs:
-        kwargs['categories'] = ['general']
-        kwargs['categories'].extend(x for x in
-                                    sorted(categories.keys())
-                                    if x != 'general'
-                                    and x in enabled_categories)
+        kwargs['categories'] = [x for x in
+                                _get_ordered_categories()
+                                if x in enabled_categories]
 
     if 'all_categories' not in kwargs:
-        kwargs['all_categories'] = ['general']
-        kwargs['all_categories'].extend(x for x in
-                                        sorted(categories.keys())
-                                        if x != 'general')
+        kwargs['all_categories'] = _get_ordered_categories()
 
     if 'selected_categories' not in kwargs:
         kwargs['selected_categories'] = []
@@ -443,6 +438,17 @@ def render(template_name, override_theme=None, **kwargs):
         '{}/{}'.format(kwargs['theme'], template_name), **kwargs)
 
 
+def _get_ordered_categories():
+    ordered_categories = []
+    if 'categories_order' not in settings['ui']:
+        ordered_categories = ['general']
+        ordered_categories.extend(x for x in sorted(categories.keys()) if x != 'general')
+        return ordered_categories
+    ordered_categories = settings['ui']['categories_order']
+    ordered_categories.extend(x for x in sorted(categories.keys()) if x not in ordered_categories)
+    return ordered_categories
+
+
 @app.before_request
 def pre_request():
     request.start_time = time()


Test patch:
Repository: searx/searx

Issue:
mymemory_translated engine: unexpected crash 'str' object has no attribute 'decode' 
mymemory engine does not work.
You can see it in the search engine statistics: https://searx.space/#.

Either: "unexpected crash 'str' object has no attribute 'decode'"
Or: "no result"

My instance is https://searx.hlfh.space (I use antibot-proxy) and I have the first issue.
I am using mymemory with the API key I got from the service.


Discussion:

Resolving patch:
diff --git a/searx/engines/translated.py b/searx/engines/translated.py
--- a/searx/engines/translated.py
+++ b/searx/engines/translated.py
@@ -9,23 +9,19 @@
  @parse       url, title, content
 """
 import re
-from sys import version_info
 from searx.utils import is_valid_lang
 
-if version_info[0] == 3:
-    unicode = str
-
 categories = ['general']
-url = u'http://api.mymemory.translated.net/get?q={query}&langpair={from_lang}|{to_lang}{key}'
-web_url = u'http://mymemory.translated.net/en/{from_lang}/{to_lang}/{query}'
+url = u'https://api.mymemory.translated.net/get?q={query}&langpair={from_lang}|{to_lang}{key}'
+web_url = u'https://mymemory.translated.net/en/{from_lang}/{to_lang}/{query}'
 weight = 100
 
-parser_re = re.compile(u'.*?([a-z]+)-([a-z]+) (.{2,})$', re.I)
+parser_re = re.compile(b'.*?([a-z]+)-([a-z]+) (.{2,})$', re.I)
 api_key = ''
 
 
 def request(query, params):
-    m = parser_re.match(unicode(query, 'utf8'))
+    m = parser_re.match(query)
     if not m:
         return params
 
@@ -43,9 +39,9 @@ def request(query, params):
         key_form = ''
     params['url'] = url.format(from_lang=from_lang[1],
                                to_lang=to_lang[1],
-                               query=query,
+                               query=query.decode('utf-8'),
                                key=key_form)
-    params['query'] = query
+    params['query'] = query.decode('utf-8')
     params['from_lang'] = from_lang
     params['to_lang'] = to_lang
 


Test patch:
Repository: searx/searx

Issue:
Allow server admins to choose default search method
Currently, the default search method used by Searx is `POST` which breaks compatibility with Firefox containers. Since FF's query to `/opensearch.xml` does not include the cookies, the user's preferred method is not reflected and they're forced create a custom search engine with the correct URL formatting and method.

A good solution is to make the method parameter configurable in `settings.yml`. This was already mentioned in #703 and looks fairly easy to implement. Let me know if you want me to open a PR.


Discussion:
It makes sense for me.


Resolving patch:
diff --git a/searx/preferences.py b/searx/preferences.py
--- a/searx/preferences.py
+++ b/searx/preferences.py
@@ -348,7 +348,7 @@ def __init__(self, themes, categories, engines, plugins):
                 }
             ),
             'method': EnumStringSetting(
-                'POST',
+                settings['server'].get('method', 'POST'),
                 choices=('GET', 'POST')
             ),
             'safesearch': MapSetting(


Test patch:
Repository: searx/searx

Issue:
Results links open in the same tab
How to set searx to open links in a new tab?



Discussion:
You just need to do a little modification in the html code of the template you're using. By adding a target attribute to a tag with the value "_blank", it makes the click open a link.

Assuming you're using the default template, you just have to replace : 

``` html
<a href="{{ result.url }}" rel="noreferrer">{{ result.title|safe }}</a>
```

by : 

``` html
<a href="{{ result.url }}" rel="noreferrer" target="_blank">{{ result.title|safe }}</a>
```

in templates/default/result_templates/default.html

PS : You can also add this attribute to every link of your template that you want to be opened in a new tab.

Thanks
But I guess this is possible only with my own Searx? 
But I use only the standard Searx: searx.me

It seems you're right :)

It's however (perhaps) not hard to imagine to let the user choose what he wants in his preferences. I'll see that when I find some time for it... but as I've never work on searx it will certainly take me some time !

OK, thanks!

@nettlebay No, keep this bug open, it's not resolved yet :)
While it's open, it's on our todo list, if it's closed, we'll probably forget it...

OK! Thanks

I like to use a direct click with an open of a new tab and to keep the request page with its result in the original tab (for example like https://www.ixquick.com/)
It is possible by add inside the <A tag the javascript code onclick="window.open(this.href); return false;"
For exemple 
<a href="http://www.w3schools.com/html/default.asp" onclick="window.open(this.href); return false;">
Is it posible to add an option to choose if you want add this option?
So the people like me that prefer to keep the original request tab and to open a new tab can choose this option
And the people which prefer to not open a new Tab can not use this option

The user can choose to open links in new tab by going to
preferences -> General -> Results on new tabs

But it would be useful to be able to set the default value in settings.yml

Resolving patch:
diff --git a/searx/preferences.py b/searx/preferences.py
--- a/searx/preferences.py
+++ b/searx/preferences.py
@@ -364,7 +364,7 @@ def __init__(self, themes, categories, engines, plugins):
                 choices=themes
             ),
             'results_on_new_tab': MapSetting(
-                False,
+                settings['ui'].get('results_on_new_tab', False),
                 map={
                     '0': False,
                     '1': True,


Test patch:
Repository: searx/searx

Issue:
500 internal error for json data from some searches
For some searches with searx I receive an 500 internal error when trying to access the json data (or csv or rss). To reproduce this try the following steps:

1. Search for `Maya_sculpture-71.893.4` e.g. https://searx.xyz/?q=Maya_sculpture-71.893.4
2. Click on the `json` button. to download the results as json data.

This leads to the 500 error. However, most other searches work well and also the json data can be downloaded. Only with this very specific search and few others I have experienced this problem.

Does anyone see why this causes a problem? How can this been solved?




Discussion:
On a local instance, I get the same error, related to https://github.com/searx/searx/blob/master/searx/utils.py#L112

The exception is not caught.

Resolving patch:
diff --git a/searx/utils.py b/searx/utils.py
--- a/searx/utils.py
+++ b/searx/utils.py
@@ -77,6 +77,10 @@ def highlight_content(content, query):
     return content
 
 
+class HTMLTextExtractorException(Exception):
+    pass
+
+
 class HTMLTextExtractor(HTMLParser):
 
     def __init__(self):
@@ -92,7 +96,7 @@ def handle_endtag(self, tag):
             return
 
         if tag != self.tags[-1]:
-            raise Exception("invalid html")
+            raise HTMLTextExtractorException()
 
         self.tags.pop()
 
@@ -128,7 +132,10 @@ def html_to_text(html):
     html = html.replace('\n', ' ')
     html = ' '.join(html.split())
     s = HTMLTextExtractor()
-    s.feed(html)
+    try:
+        s.feed(html)
+    except HTMLTextExtractorException:
+        logger.debug("HTMLTextExtractor: invalid HTML\n%s", html)
     return s.get_text()
 
 


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -52,6 +52,10 @@ def test_html_to_text(self):
         self.assertIsNotNone(utils.html_to_text(html))
         self.assertEqual(utils.html_to_text(html), "Test text")
 
+    def test_html_to_text_invalid(self):
+        html = '<p><b>Lorem ipsum</i>dolor sit amet</p>'
+        self.assertEqual(utils.html_to_text(html), "Lorem ipsum")
+
     def test_prettify_url(self):
         data = (('https://searx.me/', 'https://searx.me/'),
                 ('https://searx.me/ű', 'https://searx.me/ű'),
@@ -116,6 +120,11 @@ def test_handle_entityref(self):
         self.html_text_extractor.handle_entityref(entity)
         self.assertIn(entity, self.html_text_extractor.result)
 
+    def test_invalid_html(self):
+        text = '<p><b>Lorem ipsum</i>dolor sit amet</p>'
+        with self.assertRaises(utils.HTMLTextExtractorException):
+            self.html_text_extractor.feed(text)
+
 
 class TestUnicodeWriter(SearxTestCase):