Repository: searx/searx

Issue:
Handle parsing of the new Google UI
In a previous issue #1596 we talked about Google forcing the new UI to every  Searx instance thus breaking the current parser and having no results when doing a search. This has been temporally fixed by changing the user agent to Internet Explorer on this PR #1597.

But this might not work/break in the future if Google decide for example to drop the support for Internet Explorer or if they found a way to make their new UI compatible with Internet Explorer.

So we need to rework the `google.py` by introducing a new parser that can handle the new Google UI.
A PR #1603 has been opened but it's still in work in progress (see [this review](https://github.com/asciimoo/searx/pull/1603/files/481b0de121bc4519530143633d299e52060836f1)).


Discussion:
hi, i am the one who made #1603 pr. this case is not often happen to me so i have difficulties to test it

if anyone want to help me, you can put these code on that pr

<original code removed>

```python
def response(resp):
    ...
    from datetime import datetime
    dts = datetime.now().strftime('%Y%m%d_%H%M%S')
    is_new_parser = False
    if not results:
        is_new_parser = True
        with open('google_{}.html'.format(dts), 'w') as f:
            f.write(resp.text)
        ...
    ...
    # return results
    if is_new_parser:
        import yaml
        with open('google_{}.yaml'.format(dts), 'w') as f:
            yaml.dump(results, f)
    return results
```

what this will do is to create html and yaml file from the engine. this can be used on #1606 to compare it with different result set

e: ~~yaml part is optional. you can skip it and generate yaml later. or you can change the code above so it will only write yaml if no result given from first parser.~~

e3: remove original code and fix the second one


Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -107,13 +107,12 @@
 supported_languages_url = 'https://www.google.com/preferences?#languages'
 
 # specific xpath variables
-results_xpath = '//div[@class="g"]'
-url_xpath = './/h3/a/@href'
-title_xpath = './/h3'
-content_xpath = './/span[@class="st"]'
-content_misc_xpath = './/div[@class="f slp"]'
-suggestion_xpath = '//p[@class="_Bmc"]'
-spelling_suggestion_xpath = '//a[@class="spell"]'
+results_xpath = '//div[contains(@class, "ZINbbc")]'
+url_xpath = './/div[@class="kCrYT"][1]/a/@href'
+title_xpath = './/div[@class="kCrYT"][1]/a/div[1]'
+content_xpath = './/div[@class="kCrYT"][2]//div[contains(@class, "BNeawe")]//div[contains(@class, "BNeawe")]'
+suggestion_xpath = '//div[contains(@class, "ZINbbc")][last()]//div[@class="rVLSBd"]/a//div[contains(@class, "BNeawe")]'
+spelling_suggestion_xpath = '//div[@id="scc"]//a'
 
 # map : detail location
 map_address_xpath = './/div[@class="s"]//table//td[2]/span/text()'
@@ -199,10 +198,6 @@ def request(query, params):
     params['headers']['Accept-Language'] = language + ',' + language + '-' + country
     params['headers']['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
 
-    # Force Safari 3.1 on Mac OS X (Leopard) user agent to avoid loading the new UI that Searx can't parse
-    params['headers']['User-Agent'] = ("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4)"
-                                       "AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1")
-
     params['google_hostname'] = google_hostname
 
     return params
@@ -274,9 +269,7 @@ def response(resp):
                 content = extract_text_from_dom(result, content_xpath)
                 if content is None:
                     continue
-                content_misc = extract_text_from_dom(result, content_misc_xpath)
-                if content_misc is not None:
-                    content = content_misc + "<br />" + content
+
                 # append result
                 results.append({'url': url,
                                 'title': title,


Test patch:
diff --git a/tests/unit/engines/test_google.py b/tests/unit/engines/test_google.py
--- a/tests/unit/engines/test_google.py
+++ b/tests/unit/engines/test_google.py
@@ -58,93 +58,50 @@ def test_response(self):
         self.assertEqual(google.response(response), [])
 
         html = """
-        <div class="g">
-            <h3 class="r">
-                <a href="http://this.should.be.the.link/">
-                    <b>This</b> is <b>the</b> title
-                </a>
-            </h3>
-            <div class="s">
-                <div class="kv" style="margin-bottom:2px">
-                    <cite>
-                        <b>test</b>.psychologies.com/
-                    </cite>
-                    <div class="_nBb">‎
-                        <div style="display:inline" onclick="google.sham(this);" aria-expanded="false"
-                            aria-haspopup="true" tabindex="0" data-ved="0CBUQ7B0wAA">
-                            <span class="_O0">
-                            </span>
+        <div class="ZINbbc xpd O9g5cc uUPGi">
+            <div>
+                <div class="kCrYT">
+                    <a href="/url?q=http://this.should.be.the.link/">
+                        <div class="BNeawe">
+                            <b>This</b> is <b>the</b> title
                         </div>
-                        <div style="display:none" class="am-dropdown-menu" role="menu" tabindex="-1">
-                            <ul>
-                                <li class="_Ykb">
-                                    <a class="_Zkb" href="http://www.google.fr/url?url=http://webcache.googleusercontent
-                                        .com/search%3Fcache:R1Z_4pGXjuIJ:http://test.psychologies.com/">
-                                        En cache
-                                    </a>
-                                </li>
-                                <li class="_Ykb">
-                                    <a class="_Zkb" href="/search?safe=off&amp;q=related:test.psy.com/">
-                                        Pages similaires
-                                    </a>
-                                </li>
-                            </ul>
+                        <div class="BNeawe">
+                            http://website
+                        </div>
+                    </a>
+                </div>
+                <div class="kCrYT">
+                    <div>
+                        <div class="BNeawe">
+                            <div>
+                                <div class="BNeawe">
+                                    This should be the content.
+                                </div>
+                            </div>
                         </div>
                     </div>
                 </div>
-                <span class="st">
-                    This should be the content.
-                </span>
-                <br>
-                <div class="osl">‎
-                    <a href="http://www.google.fr/url?url=http://test.psychologies.com/tests/">
-                        Test Personnalité
-                    </a> - ‎
-                    <a href="http://www.google.fr/url?url=http://test.psychologies.com/test/">
-                        Tests - Moi
-                    </a> - ‎
-                    <a href="http://www.google.fr/url?url=http://test.psychologies.com/test/tests-couple">
-                        Test Couple
-                    </a>
-                    - ‎
-                    <a href="http://www.google.fr/url?url=http://test.psychologies.com/tests/tests-amour">
-                        Test Amour
+            </div>
+        </p>
+        <div class="ZINbbc xpd O9g5cc uUPGi">
+            <div>
+                <div class="kCrYT">
+                    <span>
+                        <div class="BNeawe">
+                            Related searches
+                        </div>
+                    </span>
+                </div>
+                <div class="rVLSBd">
+                    <a>
+                        <div>
+                            <div class="BNeawe">
+                                suggestion title
+                            </div>
+                        </div>
                     </a>
                 </div>
             </div>
-        </div>
-        <div class="g">
-            <h3 class="r">
-                <a href="http://www.google.com/images?q=toto">
-                    <b>This</b>
-                </a>
-            </h3>
-        </div>
-        <div class="g">
-            <h3 class="r">
-                <a href="http://www.google.com/search?q=toto">
-                    <b>This</b> is
-                </a>
-            </h3>
-        </div>
-        <div class="g">
-            <h3 class="r">
-                <a href="€">
-                    <b>This</b> is <b>the</b>
-                </a>
-            </h3>
-        </div>
-        <div class="g">
-            <h3 class="r">
-                <a href="/url?q=url">
-                    <b>This</b> is <b>the</b>
-                </a>
-            </h3>
-        </div>
-        <p class="_Bmc" style="margin:3px 8px">
-            <a href="/search?num=20&amp;safe=off&amp;q=t&amp;revid=1754833769&amp;sa=X&amp;ei=-&amp;ved=">
-                suggestion <b>title</b>
-            </a>
         </p>
         """
         response = self.mock_response(html)