Repository: searxng/searxng

Issue:
 RuntimeError: Set changed size during iteration in webutils.py
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
searxng-docker: 2024.5.7+58320222e
Not forked

**How did you install SearXNG?**
searxng-docker
`image: searxng/searxng:latest`

**What happened?**
When search engines timeout, I'm seeing the error below in the logs.

**How To Reproduce**
Search with several engines timing out.

**Screenshots & Logs**

```
searxng  | 2024-05-09 12:00:02,806 ERROR:searx.engines.brave: engine timeout
searxng  | 2024-05-09 12:00:02,853 ERROR:searx.webapp: Exception on /search [POST]
searxng  | Traceback (most recent call last):
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
searxng  |     response = self.full_dispatch_request()
searxng  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
searxng  |     rv = self.handle_user_exception(e)
searxng  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
searxng  |     rv = self.dispatch_request()
searxng  |          ^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
searxng  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
searxng  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/local/searxng/searx/webapp.py", line 782, in search
searxng  |     unresponsive_engines = webutils.get_translated_errors(
searxng  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/local/searxng/searx/webutils.py", line 73, in get_translated_errors
searxng  |     for unresponsive_engine in unresponsive_engines:
searxng  | RuntimeError: Set changed size during iteration
```

**Additional context**
<!-- Add any other context about the problem here. -->

 RuntimeError: Set changed size during iteration in webutils.py
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
searxng-docker: 2024.5.7+58320222e
Not forked

**How did you install SearXNG?**
searxng-docker
`image: searxng/searxng:latest`

**What happened?**
When search engines timeout, I'm seeing the error below in the logs.

**How To Reproduce**
Search with several engines timing out.

**Screenshots & Logs**

```
searxng  | 2024-05-09 12:00:02,806 ERROR:searx.engines.brave: engine timeout
searxng  | 2024-05-09 12:00:02,853 ERROR:searx.webapp: Exception on /search [POST]
searxng  | Traceback (most recent call last):
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1473, in wsgi_app
searxng  |     response = self.full_dispatch_request()
searxng  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 882, in full_dispatch_request
searxng  |     rv = self.handle_user_exception(e)
searxng  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 880, in full_dispatch_request
searxng  |     rv = self.dispatch_request()
searxng  |          ^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 865, in dispatch_request
searxng  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
searxng  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/local/searxng/searx/webapp.py", line 782, in search
searxng  |     unresponsive_engines = webutils.get_translated_errors(
searxng  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng  |   File "/usr/local/searxng/searx/webutils.py", line 73, in get_translated_errors
searxng  |     for unresponsive_engine in unresponsive_engines:
searxng  | RuntimeError: Set changed size during iteration
```

**Additional context**
<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/searx/results.py b/searx/results.py
--- a/searx/results.py
+++ b/searx/results.py
@@ -430,21 +430,38 @@ def number_of_results(self) -> int:
         """Returns the average of results number, returns zero if the average
         result number is smaller than the actual result count."""
 
-        resultnum_sum = sum(self._number_of_results)
-        if not resultnum_sum or not self._number_of_results:
-            return 0
+        with self._lock:
+            if not self._closed:
+                logger.error("call to ResultContainer.number_of_results before ResultContainer.close")
+                return 0
+
+            resultnum_sum = sum(self._number_of_results)
+            if not resultnum_sum or not self._number_of_results:
+                return 0
 
-        average = int(resultnum_sum / len(self._number_of_results))
-        if average < self.results_length():
-            average = 0
-        return average
+            average = int(resultnum_sum / len(self._number_of_results))
+            if average < self.results_length():
+                average = 0
+            return average
 
     def add_unresponsive_engine(self, engine_name: str, error_type: str, suspended: bool = False):
-        if engines[engine_name].display_error_messages:
-            self.unresponsive_engines.add(UnresponsiveEngine(engine_name, error_type, suspended))
+        with self._lock:
+            if self._closed:
+                logger.error("call to ResultContainer.add_unresponsive_engine after ResultContainer.close")
+                return
+            if engines[engine_name].display_error_messages:
+                self.unresponsive_engines.add(UnresponsiveEngine(engine_name, error_type, suspended))
 
     def add_timing(self, engine_name: str, engine_time: float, page_load_time: float):
-        self.timings.append(Timing(engine_name, total=engine_time, load=page_load_time))
+        with self._lock:
+            if self._closed:
+                logger.error("call to ResultContainer.add_timing after ResultContainer.close")
+                return
+            self.timings.append(Timing(engine_name, total=engine_time, load=page_load_time))
 
     def get_timings(self):
-        return self.timings
+        with self._lock:
+            if not self._closed:
+                logger.error("call to ResultContainer.get_timings before ResultContainer.close")
+                return []
+            return self.timings


Test patch: