Repository: searxng/searxng

Issue:
Bug: google engine - too many requests
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 2022.11.19-b5371b7a
<!-- Check if these values are correct -->

**How did you install SearXNG?**
I installed SearXNG using the official wiki (no docker).
**What happened?**
After launching a search, I receive the error below.

**How To Reproduce**
See before.

**Expected behavior**
No errors.

**Screenshots & Logs**
![sshot_2022-11-21_16 24 10](https://user-images.githubusercontent.com/24752671/203093020-ff79747c-9610-46cc-9430-7a0e315cc6cf.png)

**Additional context**
None

**Technical report**

Error
  * Error: searx.exceptions.SearxEngineAccessDeniedException
  * Percentage: 0
  * Parameters: `('HTTP error 403',)`
  * File name: `searx/search/processors/online.py:113`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`




Discussion:
Check https://github.com/tiekoetter/searxng/commit/464fc76a0df620b0af0bbb78afdfb0eab7250993

This fixed it for me (especially the change in raise_for_httperror.py).
> Check [tiekoetter@464fc76](https://github.com/tiekoetter/searxng/commit/464fc76a0df620b0af0bbb78afdfb0eab7250993)
> 
> This fixed it for me (especially the change in raise_for_httperror.py).

Thank you.
I did as you suggested, and I restarted SearXNG.
However, I don't see the Google error but the DDG one, as you can see from the attached screenshot.

![sshot_2022-11-21_16 44 02](https://user-images.githubusercontent.com/24752671/203097506-25319772-5a21-4932-861b-7e5bacf9b98b.png)

However, my question is: what's happened once I run an update? Does the update overwrite those files?

You mean the changes to the engine suspended_time? Yes they will be overwritten if you don't "secure" (e.g. through git) them.

DDG really blocks you (same for me) although they only seem to block for a short period of time. You can't really do anything without doing load balancing to prevent this.

Please keep in mind that this issue is about the google engine. I think your question is answered in this regard.
> but the DDG one, as you can see from the attached screenshot.

This is a [different issue](https://github.com/searxng/searxng/issues/1854) and not related to this google issue.

> I think your question is answered in this regard.

@tiekoetter do you like to send a PR with your fix?



Resolving patch:
diff --git a/searx/exceptions.py b/searx/exceptions.py
--- a/searx/exceptions.py
+++ b/searx/exceptions.py
@@ -69,11 +69,19 @@ class SearxEngineAPIException(SearxEngineResponseException):
 class SearxEngineAccessDeniedException(SearxEngineResponseException):
     """The website is blocking the access"""
 
-    def __init__(self, suspended_time=24 * 3600, message='Access denied'):
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineAccessDenied"
+
+    def __init__(self, suspended_time=None, message='Access denied'):
+        suspended_time = suspended_time or self._get_default_suspended_time()
         super().__init__(message + ', suspended_time=' + str(suspended_time))
         self.suspended_time = suspended_time
         self.message = message
 
+    def _get_default_suspended_time(self):
+        from searx import get_setting
+
+        return get_setting(self.SUSPEND_TIME_SETTING)
+
 
 class SearxEngineCaptchaException(SearxEngineAccessDeniedException):
     """The website has returned a CAPTCHA
@@ -81,7 +89,9 @@ class SearxEngineCaptchaException(SearxEngineAccessDeniedException):
     By default, searx stops sending requests to this engine for 1 day.
     """
 
-    def __init__(self, suspended_time=24 * 3600, message='CAPTCHA'):
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineCaptcha"
+
+    def __init__(self, suspended_time=None, message='CAPTCHA'):
         super().__init__(message=message, suspended_time=suspended_time)
 
 
@@ -91,7 +101,9 @@ class SearxEngineTooManyRequestsException(SearxEngineAccessDeniedException):
     By default, searx stops sending requests to this engine for 1 hour.
     """
 
-    def __init__(self, suspended_time=3600, message='Too many request'):
+    SUSPEND_TIME_SETTING = "search.suspended_times.SearxEngineTooManyRequests"
+
+    def __init__(self, suspended_time=None, message='Too many request'):
         super().__init__(message=message, suspended_time=suspended_time)
 
 
diff --git a/searx/network/raise_for_httperror.py b/searx/network/raise_for_httperror.py
--- a/searx/network/raise_for_httperror.py
+++ b/searx/network/raise_for_httperror.py
@@ -9,6 +9,7 @@
     SearxEngineTooManyRequestsException,
     SearxEngineAccessDeniedException,
 )
+from searx import get_setting
 
 
 def is_cloudflare_challenge(resp):
@@ -33,15 +34,22 @@ def raise_for_cloudflare_captcha(resp):
         if is_cloudflare_challenge(resp):
             # https://support.cloudflare.com/hc/en-us/articles/200170136-Understanding-Cloudflare-Challenge-Passage-Captcha-
             # suspend for 2 weeks
-            raise SearxEngineCaptchaException(message='Cloudflare CAPTCHA', suspended_time=3600 * 24 * 15)
+            raise SearxEngineCaptchaException(
+                message='Cloudflare CAPTCHA', suspended_time=get_setting('search.suspended_times.cf_SearxEngineCaptcha')
+            )
 
         if is_cloudflare_firewall(resp):
-            raise SearxEngineAccessDeniedException(message='Cloudflare Firewall', suspended_time=3600 * 24)
+            raise SearxEngineAccessDeniedException(
+                message='Cloudflare Firewall',
+                suspended_time=get_setting('search.suspended_times.cf_SearxEngineAccessDenied'),
+            )
 
 
 def raise_for_recaptcha(resp):
     if resp.status_code == 503 and '"https://www.google.com/recaptcha/' in resp.text:
-        raise SearxEngineCaptchaException(message='ReCAPTCHA', suspended_time=3600 * 24 * 7)
+        raise SearxEngineCaptchaException(
+            message='ReCAPTCHA', suspended_time=get_setting('search.suspended_times.recaptcha_SearxEngineCaptcha')
+        )
 
 
 def raise_for_captcha(resp):
diff --git a/searx/settings_defaults.py b/searx/settings_defaults.py
--- a/searx/settings_defaults.py
+++ b/searx/settings_defaults.py
@@ -160,6 +160,14 @@ def apply_schema(settings, schema, path_list):
         'languages': SettingSublistValue(LANGUAGE_CODES, LANGUAGE_CODES),
         'ban_time_on_fail': SettingsValue(numbers.Real, 5),
         'max_ban_time_on_fail': SettingsValue(numbers.Real, 120),
+        'suspended_times': {
+            'SearxEngineAccessDenied': SettingsValue(numbers.Real, 86400),
+            'SearxEngineCaptcha': SettingsValue(numbers.Real, 86400),
+            'SearxEngineTooManyRequests': SettingsValue(numbers.Real, 3600),
+            'cf_SearxEngineCaptcha': SettingsValue(numbers.Real, 1296000),
+            'cf_SearxEngineAccessDenied': SettingsValue(numbers.Real, 86400),
+            'recaptcha_SearxEngineCaptcha': SettingsValue(numbers.Real, 604800),
+        },
         'formats': SettingsValue(list, OUTPUT_FORMATS),
     },
     'server': {


Test patch:
diff --git a/tests/unit/test_exceptions.py b/tests/unit/test_exceptions.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_exceptions.py
@@ -0,0 +1,41 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+from tests import SearxTestCase
+import searx.exceptions
+from searx import get_setting
+
+
+class TestExceptions(SearxTestCase):
+    def test_default_suspend_time(self):
+        with self.assertRaises(searx.exceptions.SearxEngineAccessDeniedException) as e:
+            raise searx.exceptions.SearxEngineAccessDeniedException()
+        self.assertEqual(
+            e.exception.suspended_time,
+            get_setting(searx.exceptions.SearxEngineAccessDeniedException.SUSPEND_TIME_SETTING),
+        )
+
+        with self.assertRaises(searx.exceptions.SearxEngineCaptchaException) as e:
+            raise searx.exceptions.SearxEngineCaptchaException()
+        self.assertEqual(
+            e.exception.suspended_time, get_setting(searx.exceptions.SearxEngineCaptchaException.SUSPEND_TIME_SETTING)
+        )
+
+        with self.assertRaises(searx.exceptions.SearxEngineTooManyRequestsException) as e:
+            raise searx.exceptions.SearxEngineTooManyRequestsException()
+        self.assertEqual(
+            e.exception.suspended_time,
+            get_setting(searx.exceptions.SearxEngineTooManyRequestsException.SUSPEND_TIME_SETTING),
+        )
+
+    def test_custom_suspend_time(self):
+        with self.assertRaises(searx.exceptions.SearxEngineAccessDeniedException) as e:
+            raise searx.exceptions.SearxEngineAccessDeniedException(suspended_time=1337)
+        self.assertEqual(e.exception.suspended_time, 1337)
+
+        with self.assertRaises(searx.exceptions.SearxEngineCaptchaException) as e:
+            raise searx.exceptions.SearxEngineCaptchaException(suspended_time=1409)
+        self.assertEqual(e.exception.suspended_time, 1409)
+
+        with self.assertRaises(searx.exceptions.SearxEngineTooManyRequestsException) as e:
+            raise searx.exceptions.SearxEngineTooManyRequestsException(suspended_time=1543)
+        self.assertEqual(e.exception.suspended_time, 1543)