Repository: searxng/searxng

Issue:
Pagination is broken for Google Images engine
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->

**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
Commit number: `056550d523ddf4e06821957e13623fc322cf4675`

**How did you install Searx?**
Manually with `python -m searx.webapp`

**What happened?**
The Google Images engine doesn't have multiple pages, it only has one page. In contrary to Bing images which have multiple pages and the button "next page" is available.

**How To Reproduce**
1. `!goi test`
2. No "next page" button.

**Expected behavior**
Have multiple pages with "next page" button available.

**Screenshots & Logs**

<table>
<tr>
	<td>Google images
	<td>Bing images
<tr>
	<td><img src="https://user-images.githubusercontent.com/4016501/123101827-f425e480-d423-11eb-8735-462ef566fdb0.png" width="400">
	<td><img src="https://user-images.githubusercontent.com/4016501/123102259-64cd0100-d424-11eb-876f-780e9a0e383d.png" width="400">
</table>

**Additional context**
<!-- Add any other context about the problem here. -->



Discussion:
@unixfox sorry for the delay .. I think we have to fix #183 first / see my https://github.com/searxng/searxng/issues/183#issuecomment-873611982 .. not so easy .. :-(
Sorry I could not find a solution for paging support in google images.  I leave this open, may be others have a clever idea how we can solve the *infinite scrolling page problem*.

Google images is an infinite scrolling page and I do not have an idea [how to scrap infinite scrolling page](https://darmarit.org/searx/search?q=how%20to%20scrap%20infinite%20scrolling%20page%20%3Aen&categories=general&language=en).

I tried option [start](https://developers.google.com/custom-search/docs/xml_results#startsp) and [num](https://developers.google.com/custom-search/docs/xml_results#numsp) .. see diff below .. but google images seems to ignore both options 😞 

```diff
@@ -51,7 +51,7 @@ about = {
 
 # engine dependent config
 categories = ['images']
-paging = False
+paging = True
 use_locale_domain = True
 time_range_support = True
 safesearch = True
@@ -94,6 +94,8 @@ def scrap_img_by_id(script, data_id):
 def request(query, params):
     """Google-Video search request"""
 
+    offset = (params['pageno'] - 1) * 10
+
     lang_info = get_lang_info(
         # pylint: disable=undefined-variable
         params, supported_languages, language_aliases, False
@@ -107,7 +109,10 @@ def request(query, params):
         **lang_info['params'],
         'ie': "utf8",
         'oe': "utf8",
-        'num': 30,
+        # https://developers.google.com/custom-search/docs/xml_results#numsp
+        'num': 10,
+        # https://developers.google.com/custom-search/docs/xml_results#startsp
+        'start': offset,
     })
 
     if params['time_range'] in time_range_dict:
```
Something to note that I have stumbled upon in tinkering with this issue myself; Passing certain User-Agent headers to google image results restores pagination support. I have been able to demonstrate this locally. Passing the User-Agent for a Kindle Fire HDX:

```bash
curl -X GET -H 'User-Agent: Mozilla/5.0 (Linux; U; en-us; KFAPWI Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Silk/3.13 Safari/535.19 Silk-Accelerated=true' 'https://www.google.com/search?q=monkey&biw=320&bih=480&tbm=isch&ei=fs_1YMb7C5PXgQb525VA&start=20&sa=N'
```
Will then return a different set of images when the `start=` parameter is changed from 20 to 40:

```bash
curl -X GET -H 'User-Agent: Mozilla/5.0 (Linux; U; en-us; KFAPWI Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Silk/3.13 Safari/535.19 Silk-Accelerated=true' 'https://www.google.com/search?q=monkey&biw=320&bih=480&tbm=isch&ei=fs_1YMb7C5PXgQb525VA&start=40&sa=N'
```

Neglecting to specify this User-Agent header seems to make google ignore the `start=` parameter.
@kennedn thanks for sharing your experience ..

I can confirm; with the User-Agent from your examples, google-images returns to a paginated view.

I gave this solution a try and analysed google's response on this User-Agent ..

The paginated view (the HTML) differs completely from the HTML  _normal_ browsers will get.  If we want to parse this HTML response we have to reimplement the parser part of the gooogle-engine completely new.

I compared the HTML response with the parser we had before 16f8ec89 but the response for the User-Agent is not comparable with the old google-images parser we had before 16f8ec89 nor with the implementation we have today.

I still hesitate to (re-) implement the parser of the google-images because I do not know how long this "paginated view" will be returned by google ..

My experience is that _special responds_ depending on the User-Agent will disappear _sooner or later_ or are different in the different domains (google.com, google.fr, google.de ..). ATM I do not have much time to implement the new parser and run various tests to see how stable such a solution might be.  

If someone other has time to re-implement the parser part in the google-images engine, we are wiling to merge ... :-)





 




Thanks @return42. I confirmed the same thing myself the other day in regards to the parser needing a refactor to make the user-agent method work. Something else that appears to be a deal breaker is that there's no way to retrieve full sized image urls from the response when using a malformed User-Agent. 

I have done a bit of a dig into the infinite scrolling problem itself though and have found that google now uses a "batchexecute" endpoint to get a batch of images periodically after a certain scroll threshold is hit. 

More images than are visible on screen are returned in any given batch and it appears some pixel values are sent along with this request (screen width and position from bottom). This hints that some server side calculations are going on to figure out how many images to send back in a given batch. The returned images can be controlled via a sequence number inserted into a couple of places in the request data. I've distilled this call down to the minimum required parameters and have had some success in manipulating the request data to get different sets of images in the response:

```bash
curl 'https://www.google.com/_/VisualFrontendUi/data/batchexecute' --data 'f.req=[[["HoAMBc","[null,null,[1,null,450,1,1440,[],[],[],null,null,null,545,100,[]],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[\"monkey\"],null,null,null,null,null,null,null,null,[\"\",\"\",\"\"],null,false]",null,"generic"]]]'

# HoAMBc - Magic value, always the same
# 1 - Sequence number
# 450 - ???
# 1 - ???
# 1440 - Screen width px
# 545 - distance from end of loaded images?
# 100 - Sequence number * 100
# monkey - Search term
```
You can vary the returned set of data by incrementing the sequence number and presumably could also tweak this by modifying the pixel values. It appears that all the required information could be plucked from the response if it were cleaned up and indeed may be a requirement going forward since additional images are added dynamically to the page from the data returned in this call, which breaks the current method of scraping the initial html for values.

I don't have a lot of free time right now to pursue this further so I thought I would share my findings as they are. I will however continue to tinker and am hoping to be able to get some sort of readable data out of a batchexecute response as a PoC.
@kennedn thanks for pointing out _batchexecute_ / I already digging a into this API and I want to link here an article I found along others:

- [Deciphering Google’s mysterious ‘batchexecute’ system](https://kovatch.medium.com/deciphering-google-batchexecute-74991e4e446c)

**update**:  In a [branch](https://github.com/return42/searxng/tree/google-images-batchexecute) I am working on an implementation which uses _batchexecute_  / does not work yet / more investigation is needed.
@return42 - I noticed that this was still open and decided to take another crack at deciphering the batchexecute protocol for image searches. I have managed to figure it out finally and have created a [reference script + step by step documentation](https://github.com/kennedn/google_batchexecute).
> @return42 - I noticed that this was still open and decided to take another crack at deciphering the batchexecute protocol for image searches. I have managed to figure it out finally and have created a [reference script + step by step documentation](https://github.com/kennedn/google_batchexecute).

Can anybody implement this solution in searxng? Is it possible?

Resolving patch:
diff --git a/searx/engines/google_images.py b/searx/engines/google_images.py
--- a/searx/engines/google_images.py
+++ b/searx/engines/google_images.py
@@ -1,28 +1,20 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # lint: pylint
-"""This is the implementation of the google images engine.
+"""This is the implementation of the google images engine using the google
+internal API used the Google Go Android app.
 
-.. admonition:: Content-Security-Policy (CSP)
+This internal API offer results in
 
-   This engine needs to allow images from the `data URLs`_ (prefixed with the
-   ``data:`` scheme)::
+- JSON (_fmt:json)
+- Protobuf (_fmt:pb)
+- Protobuf compressed? (_fmt:pc)
+- HTML (_fmt:html)
+- Protobuf encoded in JSON (_fmt:jspb).
 
-       Header set Content-Security-Policy "img-src 'self' data: ;"
-
-.. _data URLs:
-   https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs
 """
 
-import re
-from urllib.parse import urlencode, unquote
-from lxml import html
-
-from searx.utils import (
-    eval_xpath,
-    eval_xpath_list,
-    eval_xpath_getindex,
-    extract_text,
-)
+from urllib.parse import urlencode
+from json import loads
 
 from searx.engines.google import (
     get_lang_info,
@@ -42,12 +34,12 @@
     "official_api_documentation": 'https://developers.google.com/custom-search',
     "use_official_api": False,
     "require_api_key": False,
-    "results": 'HTML',
+    "results": 'JSON',
 }
 
 # engine dependent config
 categories = ['images', 'web']
-paging = False
+paging = True
 use_locale_domain = True
 time_range_support = True
 safesearch = True
@@ -56,74 +48,8 @@
 filter_mapping = {0: 'images', 1: 'active', 2: 'active'}
 
 
-def scrap_out_thumbs(dom):
-    """Scrap out thumbnail data from <script> tags."""
-    ret_val = {}
-    for script in eval_xpath(dom, '//script[contains(., "_setImgSrc(")]'):
-        _script = script.text
-        # _setImgSrc('0','data:image\/jpeg;base64,\/9j\/4AAQSkZJR ....');
-        _thumb_no, _img_data = _script[len("_setImgSrc(") : -2].split(",", 1)
-        _thumb_no = _thumb_no.replace("'", "")
-        _img_data = _img_data.replace("'", "")
-        _img_data = _img_data.replace(r"\/", r"/")
-        ret_val[_thumb_no] = _img_data.replace(r"\x3d", "=")
-    return ret_val
-
-
-# [0, "-H96xjSoW5DsgM", ["https://encrypted-tbn0.gstatic.com/images?q...", 155, 324]
-# , ["https://assets.cdn.moviepilot.de/files/d3bf..", 576, 1200],
-_RE_JS_IMAGE_URL = re.compile(
-    r'"'
-    r'([^"]*)'  # -H96xjSoW5DsgM
-    r'",\s*\["'
-    r'https://[^\.]*\.gstatic.com/images[^"]*'  # https://encrypted-tbn0.gstatic.com/images?q...
-    r'[^\[]*\["'
-    r'(https?://[^"]*)'  # https://assets.cdn.moviepilot.de/files/d3bf...
-)
-
-
-def parse_urls_img_from_js(dom):
-
-    # There are two HTML script tags starting with a JS function
-    # 'AF_initDataCallback(...)'
-    #
-    # <script nonce="zscm+Ab/JzBk1Qd4GY6wGQ">
-    #   AF_initDataCallback({key: 'ds:0', hash: '1', data:[], sideChannel: {}});
-    # </script>
-    # <script nonce="zscm+Ab/JzBk1Qd4GY6wGQ">
-    #   AF_initDataCallback({key: 'ds:1', hash: '2', data:[null,[[["online_chips",[["the big",
-    #     ["https://encrypted-tbn0.gstatic.com/images?q...",null,null,true,[null,0],f
-    #   ...
-    # </script>
-    #
-    # The second script contains the URLs of the images.
-
-    # The AF_initDataCallback(..) is called with very large dictionary, that
-    # looks like JSON but it is not JSON since it contains JS variables and
-    # constants like 'null' (we can't use a JSON parser for).
-    #
-    # The alternative is to parse the entire <script> and find all image URLs by
-    # a regular expression.
-
-    img_src_script = eval_xpath_getindex(dom, '//script[contains(., "AF_initDataCallback({key: ")]', 1).text
-    data_id_to_img_url = {}
-    for data_id, url in _RE_JS_IMAGE_URL.findall(img_src_script):
-        data_id_to_img_url[data_id] = url
-    return data_id_to_img_url
-
-
-def get_img_url_by_data_id(data_id_to_img_url, img_node):
-    """Get full image URL by @data-id from parent element."""
-
-    data_id = eval_xpath_getindex(img_node, '../../../@data-id', 0)
-    img_url = data_id_to_img_url.get(data_id, '')
-    img_url = unquote(img_url.replace(r'\u00', r'%'))
-
-    return img_url
-
-
 def request(query, params):
-    """Google-Video search request"""
+    """Google-Image search request"""
 
     lang_info = get_lang_info(params, supported_languages, language_aliases, False)
 
@@ -132,7 +58,17 @@ def request(query, params):
         + lang_info['subdomain']
         + '/search'
         + "?"
-        + urlencode({'q': query, 'tbm': "isch", **lang_info['params'], 'ie': "utf8", 'oe': "utf8", 'num': 30})
+        + urlencode(
+            {
+                'q': query,
+                'tbm': "isch",
+                **lang_info['params'],
+                'ie': "utf8",
+                'oe': "utf8",
+                'asearch': 'isch',
+                'async': '_fmt:json,p:1,ijn:' + str(params['pageno']),
+            }
+        )
     )
 
     if params['time_range'] in time_range_dict:
@@ -141,9 +77,9 @@ def request(query, params):
         query_url += '&' + urlencode({'safe': filter_mapping[params['safesearch']]})
     params['url'] = query_url
 
-    params['cookies']['CONSENT'] = "YES+"
     params['headers'].update(lang_info['headers'])
-    params['headers']['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
+    params['headers']['User-Agent'] = 'NSTN/3.60.474802233.release Dalvik/2.1.0 (Linux; U; Android 12; US) gzip'
+    params['headers']['Accept'] = '*/*'
     return params
 
 
@@ -153,78 +89,34 @@ def response(resp):
 
     detect_google_sorry(resp)
 
-    # convert the text to dom
-    dom = html.fromstring(resp.text)
-    img_bas64_map = scrap_out_thumbs(dom)
-    data_id_to_img_url = parse_urls_img_from_js(dom)
-
-    # parse results
-    #
-    # root element::
-    #     <div id="islmp" ..>
-    # result div per image::
-    #     <div jsmodel="tTXmib"> / <div jsaction="..." data-id="..."
-    #     The data-id matches to a item in a json-data structure in::
-    #         <script nonce="I+vqelcy/01CKiBJi5Z1Ow">AF_initDataCallback({key: 'ds:1', ... data:function(){return [ ...
-    #     In this structure the link to the origin PNG, JPG or whatever is given
-    # first link per image-div contains a <img> with the data-iid for bas64 encoded image data::
-    #      <img class="rg_i Q4LuWd" data-iid="0"
-    # second link per image-div is the target link::
-    #      <a class="VFACy kGQAp" href="https://en.wikipedia.org/wiki/The_Sacrament_of_the_Last_Supper">
-    # the second link also contains two div tags with the *description* and *publisher*::
-    #      <div class="WGvvNb">The Sacrament of the Last Supper ...</div>
-    #      <div class="fxgdke">en.wikipedia.org</div>
-
-    root = eval_xpath(dom, '//div[@id="islmp"]')
-    if not root:
-        logger.error("did not find root element id='islmp'")
-        return results
-
-    root = root[0]
-    for img_node in eval_xpath_list(root, './/img[contains(@class, "rg_i")]'):
-
-        img_alt = eval_xpath_getindex(img_node, '@alt', 0)
-
-        img_base64_id = eval_xpath(img_node, '@data-iid')
-        if img_base64_id:
-            img_base64_id = img_base64_id[0]
-            thumbnail_src = img_bas64_map[img_base64_id]
-        else:
-            thumbnail_src = eval_xpath(img_node, '@src')
-            if not thumbnail_src:
-                thumbnail_src = eval_xpath(img_node, '@data-src')
-            if thumbnail_src:
-                thumbnail_src = thumbnail_src[0]
-            else:
-                thumbnail_src = ''
-
-        link_node = eval_xpath_getindex(img_node, '../../../a[2]', 0)
-        url = eval_xpath_getindex(link_node, '@href', 0, None)
-        if url is None:
-            logger.error("missing @href in node: %s", html.tostring(link_node))
-            continue
-
-        pub_nodes = eval_xpath(link_node, './div/div')
-        pub_descr = img_alt
-        pub_source = ''
-        if pub_nodes:
-            pub_descr = extract_text(pub_nodes[0])
-            pub_source = extract_text(pub_nodes[1])
-
-        src_url = get_img_url_by_data_id(data_id_to_img_url, img_node)
-        if not src_url:
-            src_url = thumbnail_src
-
-        results.append(
-            {
-                'url': url,
-                'title': img_alt,
-                'content': pub_descr,
-                'source': pub_source,
-                'img_src': src_url,
-                'thumbnail_src': thumbnail_src,
-                'template': 'images.html',
-            }
-        )
+    json_start = resp.text.find('{"ischj":')
+    json_data = loads(resp.text[json_start:])
+
+    for item in json_data["ischj"]["metadata"]:
+
+        result_item = {
+            'url': item["result"]["referrer_url"],
+            'title': item["result"]["page_title"],
+            'content': item["text_in_grid"]["snippet"],
+            'source': item["result"]["site_title"],
+            'img_format': f'{item["original_image"]["width"]} x {item["original_image"]["height"]}',
+            'img_src': item["original_image"]["url"],
+            'thumbnail_src': item["thumbnail"]["url"],
+            'template': 'images.html',
+        }
+
+        author = item["result"].get('iptc', {}).get('creator')
+        if author:
+            result_item['author'] = ', '.join(author)
+
+        copyright_notice = item["result"].get('iptc', {}).get('copyright_notice')
+        if copyright_notice:
+            result_item['source'] += ' / ' + copyright_notice
+
+        file_size = item.get('gsa', {}).get('file_size')
+        if file_size:
+            result_item['source'] += ' (%s)' % file_size
+
+        results.append(result_item)
 
     return results


Test patch: