Repository: searxng/searxng

Issue:
Bug: duckduckgo engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2022.09.09-eb3d185e
<!-- Check if these values are correct -->

**How did you install SearXNG?**
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->
installed via docker / when running a search, duck duck go is sending an access denied error. this did not happen before
**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
<!-- Add any other context about the problem here. -->

**Technical report**

Error
  * Error: httpx.ReadTimeout
  * Percentage: 0
  * Parameters: `(None, None, 'lite.duckduckgo.com')`
  * File name: `searx/search/processors/online.py:107`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`

Error
  * Error: searx.exceptions.SearxEngineAccessDeniedException
  * Percentage: 0
  * Parameters: `('HTTP error 403',)`
  * File name: `searx/search/processors/online.py:107`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`


Bug: duckduckgo engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2022.09.09-eb3d185e
<!-- Check if these values are correct -->

**How did you install SearXNG?**
<!-- Did you install SearXNG using the official wiki or using searxng-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
<!-- A clear and concise description of what the bug is. -->
installed via docker / when running a search, duck duck go is sending an access denied error. this did not happen before
**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->

**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->

**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->

**Additional context**
<!-- Add any other context about the problem here. -->

**Technical report**

Error
  * Error: httpx.ReadTimeout
  * Percentage: 0
  * Parameters: `(None, None, 'lite.duckduckgo.com')`
  * File name: `searx/search/processors/online.py:107`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`

Error
  * Error: searx.exceptions.SearxEngineAccessDeniedException
  * Percentage: 0
  * Parameters: `('HTTP error 403',)`
  * File name: `searx/search/processors/online.py:107`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`




Discussion:
same problem
![image](https://user-images.githubusercontent.com/9739913/195157240-4e6c9fbb-1ef2-4342-9681-59a8a47e98c2.png)
version: 2022.10.01-901143f0 (I will update soon)

I am getting this same issue on a public instance at https://searx.ebnar.xyz on version 2022.10.07-666cd1f6. I'm receiving this issue in another public instance at https://search.sapti.me/ with completely default settings on version 2022.10.11-a7337612. All the other instances I tried did not use DuckDuckGo by default.

Issue appears to be version, setting, and provider agnostic. Seems to me that there was a likely change in the DuckDuckGo API that is now causing an error. Here are some screenshots from my end on the https://searx.ebnar.xyz instance:

![Screenshot_20221012-072818_Mull](https://user-images.githubusercontent.com/73729354/195370484-8130fa0d-23c3-4033-8a37-a64d7f860094.png)
![Screenshot_20221012-072844_Mull](https://user-images.githubusercontent.com/73729354/195370520-bea1cf51-e043-4bd3-a8cc-57bcc6a1ea4c.png)

I also have the same situation
Looks like it's a general issue. 
None of the instances listed on https://searx.space was able to deliver results from DuckDuckGo
`use_mobile_ui` is only supported by the google engine.
> Fixed the Suspended: access denied issue by adding
use_mobile_ui: true
timeout: 5.0

To `settings.yml`

For me after stopping the instance and restarting it the error goes away for several hours, sometimes an entire day.  It's possible that the changes you made didn't actually fix it, but the restart to put the changes into effect did.  
Not sure if that's relevant for debugging _(haven't checked the code yet)_, but the error does also occure in **line 113**
![image](https://user-images.githubusercontent.com/38140317/196401576-3bf7d87c-3af2-48b8-bdb6-b574f19a210e.png)

@azvampyre what the hell are you on about? I've seen some weird conspiracy theory stuff but this is absurd even for conspiracy theories. What does not partnering with Yandex and not serving certain search results have to do with SearxNG having some issues accessing DuckDuckGo? Also, search engines not indexing certain things isn't new.
> Also, search engines not indexing certain things isn't new.

This is where  meta search engines like SearXNG can help. By combining results from many search engines in one result list, they ensure a much more unbiased search compared to single sources.

But I would agree with @daemonspudguy --> this discussion is OT and not related to the bug reported here.

Same problem on [my instance](https://search.sapti.me):

![image](https://user-images.githubusercontent.com/26819407/206531633-6dfc2e28-e4cb-40c3-b8ad-f1daf03feb6d.png)

Also getting this. Since yesterday.
Having this now too since a few days.
Same on my instance. It is a personal one so I don't think I'd blacklisted for the few queries.
similar situation here; I had to temporarily disable DuckDuckGo in order to remedy the error message.
Same here from today


Exception | searx.exceptions.SearxEngineAccessDeniedException | Percentage | 100
HTTP error 403
searx/search/processors/online.py:113
_send_http_request
response = req(params['url'], **request_args)


Also seeing this on my personal instance. I'm guessing DDG probably found a way to identify requests coming from Searx and block them. I've found that changing line 67 in `searx/engines/duckduckgo.py` from:
```
params['method'] = 'POST'
```
to:
```
params['method'] = 'GET'
```
so that requests are sent as a `GET` makes DDG start working again (at least until they improve their detection method?). Still messing with it to see what might be making Searx requests distinguishable from natural browser requests.
> Also seeing this on my personal instance. I'm guessing DDG probably found a way to identify requests coming from Searx and block them. I've found that changing line 67 in `searx/engines/duckduckgo.py` from:
> 
> ```
> params['method'] = 'POST'
> ```
> 
> to:
> 
> ```
> params['method'] = 'GET'
> ```
> 
> so that requests are sent as a `GET` makes DDG start working again (at least until they improve their detection method?). Still messing with it to see what might be making Searx requests distinguishable from natural browser requests.

Can confirm that this works 👍🏻 
The SearXNG engine used for DDG uses https://lite.duckduckgo.com/lite/ .. and this form sends POST requests including vqd value (at least when you press 'next page') ..

![grafik](https://user-images.githubusercontent.com/554536/209146575-beca732f-d290-4285-86fa-809bade857ce.png)

> so that requests are sent as a GET makes DDG start working again (at least until they improve their detection method?). 

try to use 'next page' in SearXNG's result page .. I assume it will not work?
Another change that fixes it (and lets you leave the requests as `POST`) is add a slash to the end of the url on line 49, change:
```
url = 'https://lite.duckduckgo.com/lite'
```
to:
```
url = 'https://lite.duckduckgo.com/lite/'
```
@rudism seems you hit the nail .. do you like to send a PR  / thanks a lot!
> Another change that fixes it (and lets you leave the requests as `POST`) is add a slash to the end of the url on line 49, change:
> 
> ```
> url = 'https://lite.duckduckgo.com/lite'
> ```
> 
> to:
> 
> ```
> url = 'https://lite.duckduckgo.com/lite/'
> ```

Can confirm. You sir, hit the nail on the head. Great job!
same problem
![image](https://user-images.githubusercontent.com/9739913/195157240-4e6c9fbb-1ef2-4342-9681-59a8a47e98c2.png)
version: 2022.10.01-901143f0 (I will update soon)

I am getting this same issue on a public instance at https://searx.ebnar.xyz on version 2022.10.07-666cd1f6. I'm receiving this issue in another public instance at https://search.sapti.me/ with completely default settings on version 2022.10.11-a7337612. All the other instances I tried did not use DuckDuckGo by default.

Issue appears to be version, setting, and provider agnostic. Seems to me that there was a likely change in the DuckDuckGo API that is now causing an error. Here are some screenshots from my end on the https://searx.ebnar.xyz instance:

![Screenshot_20221012-072818_Mull](https://user-images.githubusercontent.com/73729354/195370484-8130fa0d-23c3-4033-8a37-a64d7f860094.png)
![Screenshot_20221012-072844_Mull](https://user-images.githubusercontent.com/73729354/195370520-bea1cf51-e043-4bd3-a8cc-57bcc6a1ea4c.png)

I also have the same situation
Looks like it's a general issue. 
None of the instances listed on https://searx.space was able to deliver results from DuckDuckGo
`use_mobile_ui` is only supported by the google engine.
> Fixed the Suspended: access denied issue by adding
use_mobile_ui: true
timeout: 5.0

To `settings.yml`

For me after stopping the instance and restarting it the error goes away for several hours, sometimes an entire day.  It's possible that the changes you made didn't actually fix it, but the restart to put the changes into effect did.  
Not sure if that's relevant for debugging _(haven't checked the code yet)_, but the error does also occure in **line 113**
![image](https://user-images.githubusercontent.com/38140317/196401576-3bf7d87c-3af2-48b8-bdb6-b574f19a210e.png)

@azvampyre what the hell are you on about? I've seen some weird conspiracy theory stuff but this is absurd even for conspiracy theories. What does not partnering with Yandex and not serving certain search results have to do with SearxNG having some issues accessing DuckDuckGo? Also, search engines not indexing certain things isn't new.
> Also, search engines not indexing certain things isn't new.

This is where  meta search engines like SearXNG can help. By combining results from many search engines in one result list, they ensure a much more unbiased search compared to single sources.

But I would agree with @daemonspudguy --> this discussion is OT and not related to the bug reported here.

Same problem on [my instance](https://search.sapti.me):

![image](https://user-images.githubusercontent.com/26819407/206531633-6dfc2e28-e4cb-40c3-b8ad-f1daf03feb6d.png)

Also getting this. Since yesterday.
Having this now too since a few days.
Same on my instance. It is a personal one so I don't think I'd blacklisted for the few queries.
similar situation here; I had to temporarily disable DuckDuckGo in order to remedy the error message.
Same here from today


Exception | searx.exceptions.SearxEngineAccessDeniedException | Percentage | 100
HTTP error 403
searx/search/processors/online.py:113
_send_http_request
response = req(params['url'], **request_args)


Also seeing this on my personal instance. I'm guessing DDG probably found a way to identify requests coming from Searx and block them. I've found that changing line 67 in `searx/engines/duckduckgo.py` from:
```
params['method'] = 'POST'
```
to:
```
params['method'] = 'GET'
```
so that requests are sent as a `GET` makes DDG start working again (at least until they improve their detection method?). Still messing with it to see what might be making Searx requests distinguishable from natural browser requests.
> Also seeing this on my personal instance. I'm guessing DDG probably found a way to identify requests coming from Searx and block them. I've found that changing line 67 in `searx/engines/duckduckgo.py` from:
> 
> ```
> params['method'] = 'POST'
> ```
> 
> to:
> 
> ```
> params['method'] = 'GET'
> ```
> 
> so that requests are sent as a `GET` makes DDG start working again (at least until they improve their detection method?). Still messing with it to see what might be making Searx requests distinguishable from natural browser requests.

Can confirm that this works 👍🏻 
The SearXNG engine used for DDG uses https://lite.duckduckgo.com/lite/ .. and this form sends POST requests including vqd value (at least when you press 'next page') ..

![grafik](https://user-images.githubusercontent.com/554536/209146575-beca732f-d290-4285-86fa-809bade857ce.png)

> so that requests are sent as a GET makes DDG start working again (at least until they improve their detection method?). 

try to use 'next page' in SearXNG's result page .. I assume it will not work?
Another change that fixes it (and lets you leave the requests as `POST`) is add a slash to the end of the url on line 49, change:
```
url = 'https://lite.duckduckgo.com/lite'
```
to:
```
url = 'https://lite.duckduckgo.com/lite/'
```
@rudism seems you hit the nail .. do you like to send a PR  / thanks a lot!
> Another change that fixes it (and lets you leave the requests as `POST`) is add a slash to the end of the url on line 49, change:
> 
> ```
> url = 'https://lite.duckduckgo.com/lite'
> ```
> 
> to:
> 
> ```
> url = 'https://lite.duckduckgo.com/lite/'
> ```

Can confirm. You sir, hit the nail on the head. Great job!

Resolving patch:
diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py
--- a/searx/engines/duckduckgo.py
+++ b/searx/engines/duckduckgo.py
@@ -18,7 +18,7 @@
 
 # about
 about = {
-    "website": 'https://lite.duckduckgo.com/lite',
+    "website": 'https://lite.duckduckgo.com/lite/',
     "wikidata_id": 'Q12805',
     "official_api_documentation": 'https://duckduckgo.com/api',
     "use_official_api": False,
@@ -46,7 +46,7 @@
 time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
 
 # search-url
-url = 'https://lite.duckduckgo.com/lite'
+url = 'https://lite.duckduckgo.com/lite/'
 url_ping = 'https://duckduckgo.com/t/sl_l'
 
 # match query's language to a region code that duckduckgo will accept


Test patch: