internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: flairNLP/flair

Issue:
Why checkpoint related methods are named inconsistently
For the `ModelTrainer` it's named `load_checkpoint` and for the `LanguageModelTrainer` it is `load_from_checkpoint`?


Discussion:
I totally agree with you, @dchaplinsky. I also came across the same issue... I think the API really needs some "(re)polishing". In addition, a lot of functionality is "hidden", i.e. not documented in code or not mentioned in the tutorials (which in my eyes act as API documentation since there is no other way besides code deep diving!).

So I call on everyone able to revisit the tutorials and/or add more expressive code documentation - really appreciated.

PS: I have found a similar issue along those lines, see https://github.com/flairNLP/flair/issues/2374

Resolving patch:
diff --git a/flair/trainers/language_model_trainer.py b/flair/trainers/language_model_trainer.py
--- a/flair/trainers/language_model_trainer.py
+++ b/flair/trainers/language_model_trainer.py
@@ -477,7 +477,7 @@ def _repackage_hidden(h):
         return tuple(v.clone().detach() for v in h)
 
     @staticmethod
-    def load_from_checkpoint(
+    def load_checkpoint(
         checkpoint_file: Union[str, Path], corpus: TextCorpus, optimizer: Optimizer = SGD
     ):
         if type(checkpoint_file) is str:


Test patch:
diff --git a/tests/test_language_model.py b/tests/test_language_model.py
--- a/tests/test_language_model.py
+++ b/tests/test_language_model.py
@@ -81,7 +81,7 @@ def test_train_resume_language_model(
     )
     del trainer, language_model
 
-    trainer = LanguageModelTrainer.load_from_checkpoint(
+    trainer = LanguageModelTrainer.load_checkpoint(
         results_base_path / "checkpoint.pt", corpus
     )
     trainer.train(
Repository: flairNLP/flair

Issue:
SciSpacyTokenizer.tokenize() function is broken
**Describe the bug**
Two minor bugs were introduced to Flair's `SciSpacyTokenizer` as part of #2645
The bugs prevent usage of `SciSpacyTokenizer` and tokenizers that leverage it.

**To Reproduce**
Any use of classes that leverage `SciSpacyTokenizer` will raise errors. For example:

```python
from flair.tokenization import SciSpacySentenceSplitter

splitter = SciSpacySentenceSplitter()

sentences = splitter.split("abcd")
```

Causes
```
[/usr/local/lib/python3.7/dist-packages/flair/tokenization.py](https://localhost:8080/#) in tokenize(self, text)
    257         words: List[str] = []
    258         for word in sentence:
--> 259             word.append(word)
    260         return words
    261 

AttributeError: 'spacy.tokens.token.Token' object has no attribute 'append'
```

**To Fix**

***First issue***
The first problem is that there is a typo here: https://github.com/flairNLP/flair/blob/480d2c9afd66ab8d3bf40a676917e84dba3c4cee/flair/tokenization.py#L259
It should be `words.append`, not `word.append`.

***Second issue***
the `SciSpacyTokenizer.tokenize()` is supposed to return a list of `str` , but instad it returns a list of Spacy `Token` objects.


Happy to open a PR shortly


Discussion:

Resolving patch:
diff --git a/flair/tokenization.py b/flair/tokenization.py
--- a/flair/tokenization.py
+++ b/flair/tokenization.py
@@ -256,7 +256,7 @@ def tokenize(self, text: str) -> List[str]:
         sentence = self.model(text)
         words: List[str] = []
         for word in sentence:
-            word.append(word)
+            words.append(word.text)
         return words
 
     @property


Test patch:
Repository: flairNLP/flair

Issue:
[Feature]: Fixed versions of black and ruff in CI
### Problem statement

Hi everyone,

I was using latest `black` and `ruff` locally and it changed some files that I definitely not touched/changes.

Then I could also see test failures of my two recent PRs.

Maybe we should pin a specific version of both `black` and `ruff` to avoid this in future.

Before a new release of Flair we could just use the latest versions of them, perform tree-wide reformating and then update the pinned versions again.

What do you think?


Discussion:
Yes, pinning the version makes sense!

Resolving patch:
diff --git a/flair/embeddings/base.py b/flair/embeddings/base.py
--- a/flair/embeddings/base.py
+++ b/flair/embeddings/base.py
@@ -168,9 +168,7 @@ def forward(self, tensors: List[torch.Tensor]) -> torch.Tensor:
         """
         if len(tensors) != self.mixture_size:
             log.error(
-                "{} tensors were passed, but the module was initialized to mix {} tensors.".format(
-                    len(tensors), self.mixture_size
-                )
+                f"{len(tensors)} tensors were passed, but the module was initialized to mix {self.mixture_size} tensors."
             )
 
         normed_weights = torch.nn.functional.softmax(torch.cat(list(self.scalar_parameters)), dim=0)
diff --git a/flair/models/pairwise_regression_model.py b/flair/models/pairwise_regression_model.py
--- a/flair/models/pairwise_regression_model.py
+++ b/flair/models/pairwise_regression_model.py
@@ -318,9 +318,7 @@ def evaluate(
 
                     if out_path is not None:
                         for pair, prediction, true_value in zip(batch, results, true_values):
-                            eval_line = "{}\t{}\t{}\t{}\n".format(
-                                pair.first.to_original_text(), pair.second.to_original_text(), true_value, prediction
-                            )
+                            eval_line = f"{pair.first.to_original_text()}\t{pair.second.to_original_text()}\t{true_value}\t{prediction}\n"
                             out_file.write(eval_line)
 
                     store_embeddings(batch, embedding_storage_mode)
diff --git a/flair/training_utils.py b/flair/training_utils.py
--- a/flair/training_utils.py
+++ b/flair/training_utils.py
@@ -80,12 +80,7 @@ def to_empty_tsv():
         return "\t_\t_\t_\t_"
 
     def __str__(self) -> str:
-        line = "mean squared error: {:.4f} - mean absolute error: {:.4f} - pearson: {:.4f} - spearman: {:.4f}".format(
-            self.mean_squared_error(),
-            self.mean_absolute_error(),
-            self.pearsonr(),
-            self.spearmanr(),
-        )
+        line = f"mean squared error: {self.mean_squared_error():.4f} - mean absolute error: {self.mean_absolute_error():.4f} - pearson: {self.pearsonr():.4f} - spearman: {self.spearmanr():.4f}"
         return line
 
 


Test patch:
Repository: searx/searx

Issue:
Startpage engine doesn't support russian queries




Discussion:

Resolving patch:
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -10,7 +10,6 @@
 #
 # @todo        paging
 
-from urllib import urlencode
 from lxml import html
 from cgi import escape
 import re
@@ -38,7 +37,6 @@
 # do search-request
 def request(query, params):
     offset = (params['pageno'] - 1) * 10
-    query = urlencode({'q': query})[2:]
 
     params['url'] = search_url
     params['method'] = 'POST'


Test patch:
Repository: searx/searx

Issue:
Google engine : latency caused by redirection
The current implementation of the google engine has a little issue : there is a redirection for each request, but not for everyone.
It's depend of the geoip : https://support.google.com/websearch/answer/873?hl=en

If you look at the response time of the different searx instance, some time the google time is slow compare to the other engine, I guess this is this issue.

One way to avoid this is to set the PREF cookie (set but http://www.google.com/ncr ). Of course there is an unique ID. On solution is to initialize the cookie at start of the searx instance : 

``` python
pref_cookie = ''

def get_google_pref_cookie():
    global pref_cookie
    if pref_cookie == '':
        resp = get('https://www.google.com/ncr', allow_redirects=False)
        pref_cookie = resp.cookies["PREF"]
    return pref_cookie
```

and set the cookie in the request method : 

``` python
   params['cookies']['PREF'] = get_google_pref_cookie()
```

The cookie is global to the searx instance, so in my opinion there is no privacy issue.

Of course, if there searx instance is behind a router and share a public IP with other machine, the request are more "tagged", but isn't already the case since there is not all the normal header that a browser send ?

Another solution is to revert to the previous solution that api.google.com, but the results are different from the "browser" version (see #92)



Discussion:
Sounds reasonable. I like the first solution if it's enough to fetch the `PREF` cookie value only once, in the initialization process.


Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -11,6 +11,7 @@
 from urllib import urlencode
 from urlparse import urlparse, parse_qsl
 from lxml import html
+from searx.poolrequests import get
 from searx.engines.xpath import extract_text, extract_url
 
 # engine dependent config
@@ -39,6 +40,17 @@
 image_url_xpath = './@href'
 image_img_src_xpath = './img/@src'
 
+pref_cookie = ''
+
+
+# see https://support.google.com/websearch/answer/873?hl=en
+def get_google_pref_cookie():
+    global pref_cookie
+    if pref_cookie == '':
+        resp = get('https://www.google.com/ncr', allow_redirects=False)
+        pref_cookie = resp.cookies["PREF"]
+    return pref_cookie
+
 
 # remove google-specific tracking-url
 def parse_url(url_string):
@@ -64,6 +76,7 @@ def request(query, params):
                                       query=urlencode({'q': query}))
 
     params['headers']['Accept-Language'] = language
+    params['cookies']['PREF'] = get_google_pref_cookie()
 
     return params
 


Test patch:
Repository: searx/searx

Issue:
bing_news can't parse other languages date
When searching for french article, the time is noted as "Il y a 5 minutes", and so, doesn't match for the regex `"^[0-9]+ minute(s|) ago$"`.

Do you see a way to internationalize this detection ?



Discussion:
probably `"^.*[0-9]+\s*minute(s|).*$"`?

It won't work in russian or every other languages...

I think there is only the possibility of adding a dictionary, because otherwise we couldn't differ between minutes and hours, if the number is to low (<24)

I agree with you.
I just hoped there was a python librairy I didn't know about that could do that ;)

one other (stupid) idea:

we know that hours can only be displayed between 0-23, so numbers above that are automatic minutes. If we parse all dateobjects, from a site, it should be possible to find the exact phrase, in which minutes/hours are displayed, as long one value is above 23. :)

I like that !
But wait, there are days too...

But it may be not so stupid.
Or maybe, there is a way to have the bing interface in english while the results are in any other language...


Resolving patch:
diff --git a/searx/engines/bing_news.py b/searx/engines/bing_news.py
--- a/searx/engines/bing_news.py
+++ b/searx/engines/bing_news.py
@@ -39,8 +39,7 @@ def request(query, params):
         query=urlencode({'q': query, 'setmkt': language}),
         offset=offset)
 
-    params['cookies']['SRCHHPGUSR'] = \
-        'NEWWND=0&NRSLT=-1&SRCHLANG=' + language.split('-')[0]
+    params['cookies']['_FP'] = "ui=en-US"
 
     params['url'] = base_url + search_path
     return params


Test patch:
Repository: searx/searx

Issue:
Requests does not follow on an HTTP 301
We had already an issue when kickass redirected from https to http I seem to remember. Or it was on another engine. And the request wasn't followed so we had an empty response with status_code 301 in the engine.

Anyway, that is still the case in kickass if we try to perform a search when the query contains capitalized letters.

So why isn't 301 followed ? Is there a way to force it ?



Discussion:
Seems strange since it was the case for the google engine : #193
There is an allow_redirects parameter for the Requests

Yes, I know, and it is set at True (it is set in poolrequest). But in some cases, it doesn't follow.

@Cqoicebordel could you show a reproducable example?

Works : http://searx.me/?q=%21ka+22+jump+street&category_general=on
Doesn't work : http://searx.me/?q=%21ka+22+Jump+street&category_general=on

The only difference is the capitalized J.
Kickass tolower() it to put it in URL. And so, does a redirection in the second case, returning a 301 that is not followed.

seems, it's fixed, thanks @pointhi =)

Uh ? Where ?

ah, right, it isn't..

This issue wasn't Kickass specific, so I don't think it should be closed until there is actually the wrapper.

(Unless you plan to do the wrapper shortly. I just don't want us to forget about this issue)


Resolving patch:
diff --git a/searx/engines/kickass.py b/searx/engines/kickass.py
--- a/searx/engines/kickass.py
+++ b/searx/engines/kickass.py
@@ -45,11 +45,6 @@ def request(query, params):
 def response(resp):
     results = []
 
-    # check if redirect comparing to the True value,
-    # because resp can be a Mock object, and any attribut name returns something.
-    if resp.is_redirect is True:
-        return results
-
     dom = html.fromstring(resp.text)
 
     search_res = dom.xpath('//table[@class="data"]//tr')
diff --git a/searx/search.py b/searx/search.py
--- a/searx/search.py
+++ b/searx/search.py
@@ -87,6 +87,11 @@ def make_callback(engine_name, results_queue, callback, params):
 
     # creating a callback wrapper for the search engine results
     def process_callback(response, **kwargs):
+        # check if redirect comparing to the True value,
+        # because resp can be a Mock object, and any attribut name returns something.
+        if response.is_redirect is True:
+            return
+
         response.search_params = params
 
         timeout_overhead = 0.2  # seconds


Test patch:
Repository: searx/searx

Issue:
Use the method setting for opensearch.xml
In the preferences, there is a setting for setting the method (POST or GET) you want to use, but when you add searx to your web browser's search engines, you can only search with POST (except Chrome that can only handle GET method).

It seems logical to use the method the user choose for the web interface, don't you think? I could do a PR if you tell me it's OK.



Discussion:
For the record, GET request leaks a lot more information than POST. Not just on server and client side, but the referrer also passes the search query to the target site. Thats why searx doesn't support GET by default.

Yes, I know, but I like to get an history of my searches since I often re-search the same thing.

Besides, it will stay on POST by default so it's up to the user to choose an non-private method.

What do you prefer?
- [ ] use the existing setting for method
- [ ] add a new setting just for the browser's integration

Imho the existing is okay. Thanks!

Note : that's point of the rel="noreferrer" : hide the referrer to the target site.
Of course, I would have been too simple if all browser support this feature.


Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -687,6 +687,10 @@ def robots():
 @app.route('/opensearch.xml', methods=['GET'])
 def opensearch():
     method = 'post'
+
+    if request.cookies.get('method', 'POST') == 'GET':
+        method = 'get'
+
     # chrome/chromium only supports HTTP GET....
     if request.headers.get('User-Agent', '').lower().find('webkit') >= 0:
         method = 'get'


Test patch:
Repository: searx/searx

Issue:
Flick engine is broken
The flick engine seems to be broken : no result 

In the console :

```
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): secure.flickr.com
DEBUG:requests.packages.urllib3.connectionpool:"GET /search/?text=test&page=1 HTTP/1.1" 302 20
DEBUG:searx.search:flickr redirect on: <Response [302]>
DEBUG:requests.packages.urllib3.connectionpool:"GET /search/?rb=1&text=test&page=1 HTTP/1.1" 302 91
DEBUG:searx.search:flickr redirect on: <Response [302]>
DEBUG:requests.packages.urllib3.connectionpool:"GET /browser/upgrade/?continue=/search/?rb=1&text=test&page=1 HTTP/1.1" 200 None
```



Discussion:

Resolving patch:
diff --git a/searx/engines/flickr_noapi.py b/searx/engines/flickr_noapi.py
--- a/searx/engines/flickr_noapi.py
+++ b/searx/engines/flickr_noapi.py
@@ -20,8 +20,8 @@
 
 categories = ['images']
 
-url = 'https://secure.flickr.com/'
-search_url = url + 'search/?{query}&page={page}'
+url = 'https://www.flickr.com/'
+search_url = url + 'search?{query}&page={page}'
 photo_url = 'https://www.flickr.com/photos/{userid}/{photoid}'
 regex = re.compile(r"\"search-photos-models\",\"photos\":(.*}),\"totalItems\":", re.DOTALL)
 image_sizes = ('o', 'k', 'h', 'b', 'c', 'z', 'n', 'm', 't', 'q', 's')
diff --git a/searx/utils.py b/searx/utils.py
--- a/searx/utils.py
+++ b/searx/utils.py
@@ -17,17 +17,16 @@
 
 logger = logger.getChild('utils')
 
-ua_versions = ('31.0',
-               '32.0',
-               '33.0',
+ua_versions = ('33.0',
                '34.0',
-               '35.0')
+               '35.0',
+               '36.0',
+               '37.0')
 
 ua_os = ('Windows NT 6.3; WOW64',
          'X11; Linux x86_64',
          'X11; Linux x86')
-
-ua = "Mozilla/5.0 ({os}) Gecko/20100101 Firefox/{version}"
+ua = "Mozilla/5.0 ({os}; rv:{version}) Gecko/20100101 Firefox/{version}"
 
 blocked_tags = ('script',
                 'style')


Test patch:
Repository: searx/searx

Issue:
Piratebay engine : update the URL
The current URL is https://thepiratebay.se which doesn't respond.
This one seems to work https://thepiratebay.mn  (see https://en.wikipedia.org/wiki/The_Pirate_Bay )

Another solution : disable / remove the engine in settings.yml



Discussion:

Resolving patch:
diff --git a/searx/engines/piratebay.py b/searx/engines/piratebay.py
--- a/searx/engines/piratebay.py
+++ b/searx/engines/piratebay.py
@@ -20,7 +20,7 @@
 paging = True
 
 # search-url
-url = 'https://thepiratebay.se/'
+url = 'https://thepiratebay.am/'
 search_url = url + 'search/{search_term}/{pageno}/99/{search_type}'
 
 # piratebay specific type-definitions
@@ -42,10 +42,6 @@ def request(query, params):
                                       search_type=search_type,
                                       pageno=params['pageno'] - 1)
 
-    # FIX: SSLError: hostname 'kthepiratebay.se'
-    # doesn't match either of 'ssl2000.cloudflare.com', 'cloudflare.com', '*.cloudflare.com'
-    params['verify'] = False
-
     return params
 
 


Test patch:
diff --git a/searx/tests/engines/test_piratebay.py b/searx/tests/engines/test_piratebay.py
--- a/searx/tests/engines/test_piratebay.py
+++ b/searx/tests/engines/test_piratebay.py
@@ -15,7 +15,7 @@ def test_request(self):
         params = piratebay.request(query, dicto)
         self.assertIn('url', params)
         self.assertIn(query, params['url'])
-        self.assertIn('piratebay.se', params['url'])
+        self.assertIn('piratebay.am', params['url'])
         self.assertIn('0', params['url'])
 
         dicto['category'] = 'music'
@@ -99,7 +99,7 @@ def test_response(self):
         self.assertEqual(type(results), list)
         self.assertEqual(len(results), 2)
         self.assertEqual(results[0]['title'], 'This is the title')
-        self.assertEqual(results[0]['url'], 'https://thepiratebay.se/this.is.the.link')
+        self.assertEqual(results[0]['url'], 'https://thepiratebay.am/this.is.the.link')
         self.assertEqual(results[0]['content'], 'This is the content and should be OK')
         self.assertEqual(results[0]['seed'], 13)
         self.assertEqual(results[0]['leech'], 334)
@@ -149,7 +149,7 @@ def test_response(self):
         self.assertEqual(type(results), list)
         self.assertEqual(len(results), 1)
         self.assertEqual(results[0]['title'], 'This is the title')
-        self.assertEqual(results[0]['url'], 'https://thepiratebay.se/this.is.the.link')
+        self.assertEqual(results[0]['url'], 'https://thepiratebay.am/this.is.the.link')
         self.assertEqual(results[0]['content'], 'This is the content and should be OK')
         self.assertEqual(results[0]['seed'], 0)
         self.assertEqual(results[0]['leech'], 0)
Repository: searx/searx

Issue:
Rewriting results with  "www.***" to "http://www.***
Today, I got a result only with `www.***`, and my browser tried to open it as subpath from my search-engine: `https://searx.oe5tpo.com/www.***`. We should add a http to those strings to prevent such behaviour.

Rewriting results with  "www.***" to "http://www.***
Today, I got a result only with `www.***`, and my browser tried to open it as subpath from my search-engine: `https://searx.oe5tpo.com/www.***`. We should add a http to those strings to prevent such behaviour.



Discussion:
I can confirm this on several occasions.

I've been seeing this also, a page reload seems to "fix" it.

I can confirm this on several occasions.

I've been seeing this also, a page reload seems to "fix" it.


Resolving patch:
diff --git a/searx/__init__.py b/searx/__init__.py
--- a/searx/__init__.py
+++ b/searx/__init__.py
@@ -40,7 +40,7 @@
 with open(settings_path) as settings_yaml:
     settings = load(settings_yaml)
 
-if settings.get('server', {}).get('debug'):
+if settings.get('general', {}).get('debug'):
     logging.basicConfig(level=logging.DEBUG)
 else:
     logging.basicConfig(level=logging.WARNING)
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -66,7 +66,15 @@ def response(resp):
         url = link.attrib.get('href')
 
         # block google-ad url's
-        if re.match("^http(s|)://www.google.[a-z]+/aclk.*$", url):
+        if re.match("^http(s|)://(www\.)?google\.[a-z]+/aclk.*$", url):
+            continue
+
+        # block startpage search url's
+        if re.match("^http(s|)://(www\.)?startpage\.com/do/search\?.*$", url):
+            continue
+
+        # block ixquick search url's
+        if re.match("^http(s|)://(www\.)?ixquick\.com/do/search\?.*$", url):
             continue
 
         title = escape(extract_text(link))
diff --git a/searx/search.py b/searx/search.py
--- a/searx/search.py
+++ b/searx/search.py
@@ -206,6 +206,10 @@ def score_results(results):
         # if there is no duplicate found, append result
         else:
             res['score'] = score
+            # if the result has no scheme, use http as default
+            if res['parsed_url'].scheme == '':
+                res['parsed_url'] = res['parsed_url']._replace(scheme="http")
+
             results.append(res)
 
     results = sorted(results, key=itemgetter('score'), reverse=True)


Test patch:
Repository: searx/searx

Issue:
Error page not valid with format param provided
hello, 

when you call the the search api with a bad parameter, the landing page html is return without error message.
If you provide format like json or rss, you receive an html page, not great to provided revelant error message.

By example : https://searx.me/search?q=searx&format=json&pageno=error



Discussion:

Resolving patch:
diff --git a/searx/search.py b/searx/search.py
--- a/searx/search.py
+++ b/searx/search.py
@@ -358,7 +358,7 @@ def __init__(self, request):
         # set pagenumber
         pageno_param = self.request_data.get('pageno', '1')
         if not pageno_param.isdigit() or int(pageno_param) < 1:
-            raise Exception('wrong pagenumber')
+            pageno_param = 1
 
         self.pageno = int(pageno_param)
 


Test patch: