Repository: searx/searx

Issue:
autocomplete in the browser URL doesn't work
Scenario:
* I pick one instance
* in the searx preferences, I select one autocomplete provider, then save
* in Firefox, I add the new search engine
* in the Firefox preferences, I select this new search engine as the default one
* in the Search bar, I type a query
* there is no autocomplete

I've replaced
https://github.com/asciimoo/searx/blob/fc5d1c69cc01ec1c44e5d2d9644269b6b737fa5a/searx/templates/__common__/opensearch.xml#L16
by
```html
<Url rel="suggestions" type="application/x-suggestions+json" template="{{ host }}autocompleter?q={searchTerms}"/>
```

but I'm not sure it works on http://localhost:8888

Split autocompleter into (browser)suggestions and API
## What does this PR do?
Since current browers do not seem to either POST requests at all, or
supplying a format-parameter as the current assumption is, the easiest
reasonably correct way seems to be to split the autocompleter endpoint
into:
- /autocompleter as used currently, continuing to work for the API-usage
  on the web-frondend
- /autocompleter_suggestions which is included in opensearch.xml defined
  for browser-suggestios-requests

While it is sad no get rid of POST for browser-autocompleter, it seems
neither chromium nor firefox currently support it — and opensearch
cannot reasonably specify it, as far as I understand. Thus removing it
seems fair for the time being.

Fixes: #2105

## Why is this change important?
For users, suggestions can be vitally important. It is a integrated part of todays searching workflow.

## How to test this PR locally?
- Start up searx using this branch
- Add it as search-engine via your browsers opensearch UI
- Enable it as a suggestion engine
- See it work or fail

## Author's checklist
- [ ] Do I understand opensearch.xml correct that POST suggestions do not work?
- [ ] Does this break any browsers I can’t try at the moment?
- [ ] Is having GET instead of POST a good idea?

## Related issues
Closes #2105


Discussion:
Cannot make it work on localhost. Using Google Chrome (want to switch to Firefox though).
I haven't change anything and it works on my instance: https://searx.be
I'm using Firefox desktop
https://searx.be doesn't work for me (FF 79).

Hi,

I have the very same problem, also on FF 79. I can confirm autocomplete was enabled and working on-site before adding the search engine. Firefox gets and correctly parses the `opensearch.xml`. In the end, firefox does send a autocomplete request, which leads to a searx internal server error

```
DEBUG:searx.webapp:default locale from browser info is `en`
DEBUG:searx.webapp:selected locale is `en`
!! query parts is [b'']
ERROR:searx.webapp:Exception on /autocompleter [GET]
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/searx/searx/webapp.py", line 758, in autocompleter
    raw_text_query.parse_query()
  File "/usr/local/searx/searx/query.py", line 76, in parse_query
    if query_part[0] == '<':
IndexError: index out of range
[pid: 14|app: 0|req: 3/5] 192.168.96.2 () {44 vars in 662 bytes} [Fri Aug  7 12:21:29 2020] GET /autocompleter => generated 290 bytes in 4 msecs (HTTP/1.1 500) 4 headers in 150 bytes (1 switches on core 0)
```

Where the query was a `GET /autocompleter` … GET without any query-string or anything else.

looking at the opensearch spec, maybe we either somehow (not sure how) specify we want to `POST` with a `q`, or append something like  `?q={searchTerms}` to the `suggestion` entry in the XML? I tried the latter, which didn’t work out of the box, but I need to investigate a bit further …
Hi,

I tinkered around a bit more, and i think i found the cause. First, I could not make firefox issue any `POST` suggestions queries whatever i did. So, switching `opensearch.xml` to `method="get"` and appending a `?q={searchTerms}` was my startingpoint.

Furthermore, here:
https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L793
the original query string is only prepended, as it seems Firefox requires it, if there is a `format=x-suggestions` query parameter. However, no matter how i formatted the `content-type` or `rel` attributes in the `opensearch.xml`, i could not get a `format` parameter in the query from Firefox.

I then tried a dirty hack, and formatted the result in L797 in the way that i assumed Firefox wanted:
```
    return Response(json.dumps([raw_text_query.query, results]),
                    mimetype='application/json')
```

And tadah, suggestions worked!

… I’m not firm with either firefox, nor searx. Thus, I need some feedback on what the best path forward is here.

Thanks for this great project ^_^
@Nebukadneza Thank you for research! Can you create pull request for that please (if it's not difficult for you of course)?
I’d love to, but the problem is two-fold:
- Uses GET instead of POST, may not be up to the strict privacy standards regarding logging of Searx(?)
- Breaks autocompletion on the web-page, since there, the additional wrapping with the search-term is not expected

In the quick time I couldn’t figure out how to fiddle the frontend to either supply a parameter which might be used as a differentiator, or in the request handler find out if it’s the browser making a suggestions-call … :( 
I found a way that’s … a bit hackish and unclean, but gets the job done. Would be glad about feedback and reviews~
Oh, travis checks failed because of PEP8 (problems with code formatting and style, a common problem). A log was given:
```
TEST      pep8
. ./local/py3/bin/activate; pep8 --exclude='searx/static, searx/engines/gigablast.py' --max-line-length=120 --ignore "E402,W503" searx tests
searx/webapp.py:800:67: W291 trailing whitespace
searx/webapp.py:804:4: E111 indentation is not a multiple of four
searx/webapp.py:806:1: E302 expected 2 blank lines, found 1
searx/webapp.py:808:4: E111 indentation is not a multiple of four
searx/webapp.py:810:1: E302 expected 2 blank lines, found 1
Makefile:246: recipe for target 'test.pep8' failed
make: *** [test.pep8] Error 1
```
Thanks for the ping, and sorry for not running pep8 locally first. Fixed …

Build look green now ^_^
Thank you for your PR.

If only [this line](https://github.com/asciimoo/searx/pull/2125/files#diff-1a3127efb52515b63627f49aad3596c1R16) is changed does it solve the issue :
```
<Url rel="suggestions" type="application/json" template="{{ host }}autocompleter?format=x-suggestions&amp;q={searchTerms}" method="get" />
```
?
Unfortunately not, as far as my testing went. I couldn’t figure out why, but when i add the format query-parameter, Firefox rejects the `opensearch.xml`, only stating it’s `invalid` with no further details (in the browser-console). :( 
> Unfortunately not, as far as my testing went. 

:(

Out of topic: how do you make sure that opensearch.xml is reloaded ? If I remove the engine from FF, add it again it seems the method settings stick to the previous value.

For me, explicitly navigating to the `opensearch.xml` and force-reloading (ctrl+shift+r) did the trick …. It’s not a great workflow, though :| 
I was debugging this issue on my personal instance before seeing this PR. I ended up avoiding splitting routes by checking for `X-Requested-With: XMLHttpRequest`. Changing `opensearch.xml` to explicitly include the query param is still necessary though (currently using an FF extension to do that).

```
    # return autocompleter results
    if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
        return Response(json.dumps(results),
                        mimetype='application/json')

    return Response(json.dumps([raw_text_query.query, results]),
                    mimetype='application/x-suggestions+json')

```
This was tested on FF79. Not sure if all browsers include that header when making a request.
jquery adds the ```X-Requested-With: XMLHttpRequest``` header: https://api.jquery.com/jquery.ajax/ (see "headers" parameter), so it will work whatever the browser.


Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -790,12 +790,12 @@ def autocompleter():
         results.append(raw_text_query.getFullQuery())
 
     # return autocompleter results
-    if request.form.get('format') == 'x-suggestions':
-        return Response(json.dumps([raw_text_query.query, results]),
+    if request.headers.get('X-Requested-With') == 'XMLHttpRequest':
+        return Response(json.dumps(results),
                         mimetype='application/json')
 
-    return Response(json.dumps(results),
-                    mimetype='application/json')
+    return Response(json.dumps([raw_text_query.query, results]),
+                    mimetype='application/x-suggestions+json')
 
 
 @app.route('/preferences', methods=['GET', 'POST'])


Test patch: