Repository: searxng/searxng

Issue:
Google engine is broken
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version**
```
Docker: 1.0.0-1516-96ab5e57
``` 
**What happened?**
No results for query `rclone docker plugin`

**How To Reproduce**
Set google to be the only search engine, then search.

**Screenshots & Logs**
![image](https://user-images.githubusercontent.com/309092/149750182-7f0d757a-df81-420e-8183-ab8dcf07c6e1.png)

This works fine if I search directly in google. I'm confused. There is nothing in the stdout/stderr logs.

My `settings.yaml`:
```
use_default_settings: true

general:
  debug: false

search:
  safe_search: 0
  autocomplete: ''
  default_lang: 'en-GB'
  formats:
    - html
  languages:
   - pt-BR
   - en-GB

```
Google engine is broken
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Version**
```
Docker: 1.0.0-1516-96ab5e57
``` 
**What happened?**
No results for query `rclone docker plugin`

**How To Reproduce**
Set google to be the only search engine, then search.

**Screenshots & Logs**
![image](https://user-images.githubusercontent.com/309092/149750182-7f0d757a-df81-420e-8183-ab8dcf07c6e1.png)

This works fine if I search directly in google. I'm confused. There is nothing in the stdout/stderr logs.

My `settings.yaml`:
```
use_default_settings: true

general:
  debug: false

search:
  safe_search: 0
  autocomplete: ''
  default_lang: 'en-GB'
  formats:
    - html
  languages:
   - pt-BR
   - en-GB

```


Discussion:
did you tried `use_mobile_ui` ... see https://docs.searxng.org/src/searx.engines.google.html#module-searx.engines.google

```yaml
- name: google
  ...
  use_mobile_ui: true
```
OK now I'm not getting results on anything. I guess I am being blocked by google?

Are there logs somewhere?
I found  this:

```
Error
  * Error: some results are invalids: invalid content
  * Percentage: 35
  * Parameters: `()`
  * File name: `searx/search/processors/abstract.py:111`
  * Function: `_extend_container_basic`
  * Code: `result_container.extend(self.engine_name, search_results)`
```
>  I guess I am being blocked by google?

I assume.

> Are there logs somewhere?

You can use `make run` (on local instances) and will get a DEBUG log on the console.  On a server the location of log and the log level depend on how you are installed SearXNG.  Most often you should find some logs in the systemd log system of your server.

In the UI you can open the `stats/` URL and klick on the engine e.g. on the google engine.



OK now I see your https://github.com/searxng/searxng/issues/771#issuecomment-1014380198

Seems there are also other problems with the response from google (when yo did not activate us_mobile_ui?).
`tail -f /var/log/uwsgi/uwsgi.log`

```
2022-01-17 10:52:52,468 WARNING:searx.engines.google: ErrorContext('searx/search/processors/abstract.py', 111, 'result_container.extend(self.engine_name, search_results)', None, 'some results are invalids: invalid content', ()) True
```

This is all I get, on every search.

With `use_mobile_ui` true or false, same thing.
There is an ongoing issue with the Google engine, it is broken. With mobile_ui enabled and disabled. And I'm not blocked by Google.

I saved the response HTML result here:
[google.html.zip](https://github.com/searxng/searxng/files/7881748/google.html.zip)

And here is a saved mitmproxy file (to be imported with mitmproxy) in order to have more details about the response: 
[flows.zip](https://github.com/searxng/searxng/files/7881264/flows.zip)

EDIT: I think there is some A/B testing going on because I just got a new HTML file result:
[googlenew.html.zip](https://github.com/searxng/searxng/files/7881747/googlenew.html.zip)

----
@return42 By the way, we are talking about the issue on matrix: https://matrix.to/#/#searxng:matrix.org


ah, I thought Google blocked me for some reason. Hopefully, we can figure out how to solve this. 
did you tried `use_mobile_ui` ... see https://docs.searxng.org/src/searx.engines.google.html#module-searx.engines.google

```yaml
- name: google
  ...
  use_mobile_ui: true
```
OK now I'm not getting results on anything. I guess I am being blocked by google?

Are there logs somewhere?
I found  this:

```
Error
  * Error: some results are invalids: invalid content
  * Percentage: 35
  * Parameters: `()`
  * File name: `searx/search/processors/abstract.py:111`
  * Function: `_extend_container_basic`
  * Code: `result_container.extend(self.engine_name, search_results)`
```
>  I guess I am being blocked by google?

I assume.

> Are there logs somewhere?

You can use `make run` (on local instances) and will get a DEBUG log on the console.  On a server the location of log and the log level depend on how you are installed SearXNG.  Most often you should find some logs in the systemd log system of your server.

In the UI you can open the `stats/` URL and klick on the engine e.g. on the google engine.



OK now I see your https://github.com/searxng/searxng/issues/771#issuecomment-1014380198

Seems there are also other problems with the response from google (when yo did not activate us_mobile_ui?).
`tail -f /var/log/uwsgi/uwsgi.log`

```
2022-01-17 10:52:52,468 WARNING:searx.engines.google: ErrorContext('searx/search/processors/abstract.py', 111, 'result_container.extend(self.engine_name, search_results)', None, 'some results are invalids: invalid content', ()) True
```

This is all I get, on every search.

With `use_mobile_ui` true or false, same thing.
There is an ongoing issue with the Google engine, it is broken. With mobile_ui enabled and disabled. And I'm not blocked by Google.

I saved the response HTML result here:
[google.html.zip](https://github.com/searxng/searxng/files/7881748/google.html.zip)

And here is a saved mitmproxy file (to be imported with mitmproxy) in order to have more details about the response: 
[flows.zip](https://github.com/searxng/searxng/files/7881264/flows.zip)

EDIT: I think there is some A/B testing going on because I just got a new HTML file result:
[googlenew.html.zip](https://github.com/searxng/searxng/files/7881747/googlenew.html.zip)

----
@return42 By the way, we are talking about the issue on matrix: https://matrix.to/#/#searxng:matrix.org


ah, I thought Google blocked me for some reason. Hopefully, we can figure out how to solve this. 

Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -111,8 +111,8 @@
 # specific xpath variables
 # ------------------------
 
-# google results are grouped into <div class="g" ../>
-results_xpath = '//div[@class="g"]'
+# google results are grouped into <div class="g ..." ../>
+results_xpath = '//div[contains(@class, "g")]'
 
 # google *sections* are no usual *results*, we ignore them
 g_section_with_header = './g-section-with-header'
@@ -124,8 +124,8 @@
 # href=...>
 href_xpath = './/div[@class="yuRUbf"]//a/@href'
 
-# in the result group there is <div class="IsZvec" ../> containing he *content*
-content_xpath = './/div[@class="IsZvec"]'
+# in the result group there is <div class="VwiC3b ..." ../> containing the *content*
+content_xpath = './/div[contains(@class, "VwiC3b")]'
 
 # Suggestions are links placed in a *card-section*, we extract only the text
 # from the links not the links itself.


Test patch: