Repository: searx/searx

Issue:
Startpage is blocking searx - startpage engine doesn't work
**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
COMMIT: 117dbd462f39fb4cc2f12e9f7b9acf8d3fff419b

**How did you install Searx?**
git clone
pip install -r requirements.txt

**What happened?**
starpage doesn't return any results

**How To Reproduce**
1. Select startpage as the only search engine
2. Make a query

**Expected behavior**
A list with results

**Screenshots & Logs**
```
{'category': 'general', 'pageno': 1, 'safesearch': 0, 'time_range': None, 'engine_data': {}, 'language': 'en-US', 'method': 'GET', 'headers': {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0'}, 'data': {}, 'url': 'https://startpage.com/sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english', 'cookies': {}, 'verify': True, 'auth': None}
DEBUG:urllib3.connectionpool:https://startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english HTTP/1.1" 307 180
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english HTTP/1.1" 302 277
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /en/errors HTTP/1.1" 200 None
DEBUG:searx:startpage: ErrorContext('/home/luis/work/searx/searx/search/processors/online.py', 106, 'record_error(self.engine_name,', None, '2 redirects, maximum: 0', ('200', 'OK', 'www.startpage.com'))

```
**Additional context**
Commenting [this](https://github.com/searx/searx/blob/master/searx/engines/startpage.py#L125) line solved the issue for me. It appears this parameter is no longer needed/recognized.
The problem now is the CAPTCHA... I could only do one query before being blocked.
```
{'category': 'general', 'pageno': 1, 'safesearch': 0, 'time_range': None, 'engine_data': {}, 'language': 'en-US', 'method': 'GET', 'headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:106.0) Gecko/20100101 Firefox/106.0'}, 'data': {}, 'url': 'https://startpage.com/sp/search?query=query&page=1&cat=web&language=english&lui=english', 'cookies': {}, 'verify': True, 'auth': None}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): startpage.com:443
DEBUG:urllib3.connectionpool:https://startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&language=english&lui=english HTTP/1.1" 307 180
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.startpage.com:443
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&language=english&lui=english HTTP/1.1" 307 829
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/captcha HTTP/1.1" 200 None
DEBUG:searx:startpage: ErrorContext('/home/luis/work/searx/searx/search/processors/online.py', 106, 'record_error(self.engine_name,', None, '2 redirects, maximum: 0', ('200', 'OK', 'www.startpage.com'))
```

Startpage is blocking searx - startpage engine doesn't work
**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
COMMIT: 117dbd462f39fb4cc2f12e9f7b9acf8d3fff419b

**How did you install Searx?**
git clone
pip install -r requirements.txt

**What happened?**
starpage doesn't return any results

**How To Reproduce**
1. Select startpage as the only search engine
2. Make a query

**Expected behavior**
A list with results

**Screenshots & Logs**
```
{'category': 'general', 'pageno': 1, 'safesearch': 0, 'time_range': None, 'engine_data': {}, 'language': 'en-US', 'method': 'GET', 'headers': {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0'}, 'data': {}, 'url': 'https://startpage.com/sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english', 'cookies': {}, 'verify': True, 'auth': None}
DEBUG:urllib3.connectionpool:https://startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english HTTP/1.1" 307 180
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&sc=ymIIkkO00&language=english&lui=english HTTP/1.1" 302 277
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /en/errors HTTP/1.1" 200 None
DEBUG:searx:startpage: ErrorContext('/home/luis/work/searx/searx/search/processors/online.py', 106, 'record_error(self.engine_name,', None, '2 redirects, maximum: 0', ('200', 'OK', 'www.startpage.com'))

```
**Additional context**
Commenting [this](https://github.com/searx/searx/blob/master/searx/engines/startpage.py#L125) line solved the issue for me. It appears this parameter is no longer needed/recognized.
The problem now is the CAPTCHA... I could only do one query before being blocked.
```
{'category': 'general', 'pageno': 1, 'safesearch': 0, 'time_range': None, 'engine_data': {}, 'language': 'en-US', 'method': 'GET', 'headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:106.0) Gecko/20100101 Firefox/106.0'}, 'data': {}, 'url': 'https://startpage.com/sp/search?query=query&page=1&cat=web&language=english&lui=english', 'cookies': {}, 'verify': True, 'auth': None}
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): startpage.com:443
DEBUG:urllib3.connectionpool:https://startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&language=english&lui=english HTTP/1.1" 307 180
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.startpage.com:443
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/search?query=query&page=1&cat=web&language=english&lui=english HTTP/1.1" 307 829
DEBUG:urllib3.connectionpool:https://www.startpage.com:443 "GET /sp/captcha HTTP/1.1" 200 None
DEBUG:searx:startpage: ErrorContext('/home/luis/work/searx/searx/search/processors/online.py', 106, 'record_error(self.engine_name,', None, '2 redirects, maximum: 0', ('200', 'OK', 'www.startpage.com'))
```



Discussion:
Have you tried changing the timeout on [this line](https://github.com/searx/searx/blob/c647b55eb06920234ed9ca5fc8aa23920f75cc71/searx/engines/startpage.py#L86)?  I'm curious if that signature is now rotating more frequently and if we don't scrape it the bot detector goes off.
Nope, it's just an incredibly dumb bug that I introduced by accident.

[This line](https://github.com/searx/searx/blob/c647b55eb06920234ed9ca5fc8aa23920f75cc71/searx/engines/startpage.py#L101) is broken.  It's trimming the first five characters off the sc code.  This was left behind from the old codepath.

Changing that line to just

   sc_code = href

Will fix it, but the cleaner patch (which is what I should've done in the first place) is to change that block to:

```python
        try:
            sc_code = eval_xpath(dom, '//input[@name="sc"]')[0].get('value')
        except IndexError as exc:
            # suspend startpage API --> https://github.com/searxng/searxng/pull/695
            raise SearxEngineResponseException(
                suspended_time=7 * 24 * 3600, message="PR-695: query new sc time-stamp failed!"
            ) from exc

        sc_code_ts = time()
        logger.debug("new value is: %s", sc_code)
```

I'll probably push a PR later tonight to fix this, but meanwhile that simple change should get you going again.
Have you tried changing the timeout on [this line](https://github.com/searx/searx/blob/c647b55eb06920234ed9ca5fc8aa23920f75cc71/searx/engines/startpage.py#L86)?  I'm curious if that signature is now rotating more frequently and if we don't scrape it the bot detector goes off.
Nope, it's just an incredibly dumb bug that I introduced by accident.

[This line](https://github.com/searx/searx/blob/c647b55eb06920234ed9ca5fc8aa23920f75cc71/searx/engines/startpage.py#L101) is broken.  It's trimming the first five characters off the sc code.  This was left behind from the old codepath.

Changing that line to just

   sc_code = href

Will fix it, but the cleaner patch (which is what I should've done in the first place) is to change that block to:

```python
        try:
            sc_code = eval_xpath(dom, '//input[@name="sc"]')[0].get('value')
        except IndexError as exc:
            # suspend startpage API --> https://github.com/searxng/searxng/pull/695
            raise SearxEngineResponseException(
                suspended_time=7 * 24 * 3600, message="PR-695: query new sc time-stamp failed!"
            ) from exc

        sc_code_ts = time()
        logger.debug("new value is: %s", sc_code)
```

I'll probably push a PR later tonight to fix this, but meanwhile that simple change should get you going again.

Resolving patch:
diff --git a/searx/engines/startpage.py b/searx/engines/startpage.py
--- a/searx/engines/startpage.py
+++ b/searx/engines/startpage.py
@@ -91,14 +91,13 @@ def get_sc_code(headers):
         dom = html.fromstring(resp.text)
 
         try:
-            href = eval_xpath(dom, '//input[@name="sc"]')[0].get('value')
+            sc_code = eval_xpath(dom, '//input[@name="sc"]')[0].get('value')
         except IndexError as exc:
             # suspend startpage API --> https://github.com/searxng/searxng/pull/695
             raise SearxEngineResponseException(
                 suspended_time=7 * 24 * 3600, message="PR-695: query new sc time-stamp failed!"
             ) from exc
 
-        sc_code = href[5:]
         sc_code_ts = time()
         logger.debug("new value is: %s", sc_code)
 


Test patch: