Repository: searx/searx

Issue:
Chromium-based browsers (Android) don't detect searx



Discussion:
It requires more information : 
* Chromium-based browsers: which one ? is Chrome or another browser ?
* what do you mean by "detect" ?
1) I tested in Chrome and Bromite
2) I meant by "detect" being able to set as the default engine: https://coinsh.red/u/Screenshot_20190803-171609_Bromite.png
Should appear in the "recently visited" 
Did you do a search on the Searx instance? According to [this comment](https://bugs.chromium.org/p/chromium/issues/detail?id=348360#c53) on the chromium bug tracker you need to do a search on the search engine before being able to add it as a default search engine.
Yes, I did a search in the searx instance. 
I've tested it with the latest chrome with multiple opensearch setups - even with 1.0 and 1.1 opensearch specifications and it doesn't work. I also tried to research on the internet, but I couldn't find any useful info about mobile chromes' opensearch support. Do you have any information about this topic?
It seems to me that mobile chrome doesn't handle opensearch standards.
Well, bugs.chromium.org have told me that there are "old internal bugs mentioned in the code" but they "WontFix".

The OpenSearch method with ANDROID chromium-based browsers doesn't work for searx.be and startpage.com sites.

But it works OK for metager.org and qwant.com sites.
So they must have figured out how to work around those "old internal bugs" in the chromium code.

@bat999 can you provide the references to these bugs ?
Hi **dalf**
Look in this post...
https://bugs.chromium.org/p/chromium/issues/detail?id=1097707&q=os%3DAndroid
> I've tested it with the latest chrome with multiple opensearch setups 

@asciimoo I can't speak for android, but chrome(ium) has this limitation ...

> Chromium only adds sites that the user navigated to without a path.

You can find this under the link @bat999 and we have it also named in our 'about' page / added with commit e7788980 (https://github.com/asciimoo/searx/commit/e7788980ad1bb82e9ae084428d05a79b235c17ff#diff-a670502bb265cc5b6631bec680aaabccR70). 

The about page links to https://www.chromium.org/tab-to-search where you find a description of chrome's heuristics that automatically add a site to the list of searchable sites.  For my Instance I use the solution described under (1.).  By example ..

My searx instance is locate under https://darmarit.org/searx .. to support chrome also I added a

      <link title="searx" type="application/opensearchdescription+xml" rel="search" href="/searx/opensearch.xml">

to the intro site at https://darmarit.org (see HTML source line 8)

Was this what you are looking for (I thought it was known) / or did you ask explicit about android chrome?




> 
> But it works OK for metager.org and qwant.com sites.
> So they must have figured out how to work around those "old internal bugs" in the chromium code.

This is screenshot with Chromium 83.0.4103.119 and ANDROID 8.1.0.

![metager_and_qwant](https://user-images.githubusercontent.com/640761/87084463-2eec3800-c226-11ea-9349-35a655249efc.jpg)



This is strange.. As I see we have an identical opensearch.xml with ddg. Probably I miss something..
> This is strange.. As I see we have an identical opensearch.xml with ddg. Probably I miss something..

On the screenshot of @bat999, ddg is a default search engine so that's why it's listed even if its opensearch is working or not.
The interesting part is "Recently Visited" where the newly discovered search engines are added by Chrome on Android.

-------------------
The opensearch.xml from [qwant lite](https://lite.qwant.com/opensearch.xml) is quite different from the opensearch.xml of Searx ([an example from searx.info](https://searx.info/opensearch.xml)).
The noticeable differences are:
- the image that is encoded in base64
- qwant lite doesn't have:
```xml
<Param name="format" value="x-suggestions" />
<Param name="q" value="{searchTerms}" />
```
- And qwant lite added this line:
```xml
<moz:SearchForm>https://lite.qwant.com/</moz:SearchForm>
```
with this other one which is related to the previous one:
```xml
xmlns:moz="http://www.mozilla.org/2006/browser/search/">
```
> 
> On the screenshot of @bat999, ddg is a default search engine so that's why it's listed even if its opensearch is working or not.
> The interesting part is "Recently Visited" where the newly discovered search engines are added by Chrome on Android.
> 
Yes, that's correct.
DuckDuckGo is already one of the default search engines.
But DuckDuckGo **LITE** can **not** be added with OpenSearch method.
PS
Both Qwant and Qwant LITE will add OK with OpenSearch method.


Interesting.. Could you help to produce an opensearch.xml which is acceptable for mobile chrome?
> Interesting.. Could you help to produce an opensearch.xml which is acceptable for mobile chrome?


Hi
There's info how to make the opensearch.xml file.
https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#OpenSearch_1.1_parameters

I've compared the **Searx** file to **Metager** and **Qwant**.

There are four lines in Searx that I don't understand...
```
<Url type="application/x-suggestions+json" method="get" template="https://searx.info/autocompleter">
<Param name="format" value="x-suggestions"/>
<Param name="q" value="{searchTerms}"/>
</Url>
```


Try a new xml file without those four lines and add the mozilla reference to the first line (maybe it's used by Firefox, I dunno).

Like this...

```
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/">
<ShortName>searx</ShortName>
<LongName>searx metasearch</LongName>
<Description>a privacy-respecting, hackable metasearch engine</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image>https://searx.info/static/themes/oscar/img/favicon.png</Image>
<Url type="text/html" method="get" template="https://searx.info/search?q={searchTerms}"/>
</OpenSearchDescription>
```


Also it said on stackoverflow about the xml file...
Note, that unlike Firefox, Chrome will discover your Open Search Description only if you **put it on the root of your website**.



This is the xml file from MetaGer for comparison (those dudes at Hannover Uni are probably very smart).

```
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/">
<ShortName> MetaGer </ShortName>
<Description>MetaGer: secure search &amp; find, protecting privacy</Description>
<Contact>office@suma-ev.de</Contact>
<Image width="16" height="16" type="image/x-icon">https://metager.org/favicon.ico</Image>
<Url type="text/html" template="https://metager.org/meta/meta.ger3?eingabe={searchTerms}" method="GET" />
<InputEncoding>UTF-8</InputEncoding>
</OpenSearchDescription>
```

EDIT
Those four lines that I didn't understand are for Firefox search plugins.
To support search suggestions, a search plugin needs to define an extra <Url> element with its type attribute set to "application/x-suggestions+json". (This means that a suggestion-supporting engine plugin will have two <Url> elements, the other one being the main text/html search URL.)
https://developer.mozilla.org/en-US/docs/Archive/Add-ons/Supporting_search_suggestions_in_search_plugins
> Note, that unlike Firefox, Chrome will discover your Open Search Description only if you put it on the root of your website.

Please please take note of my https://github.com/asciimoo/searx/issues/1666#issuecomment-656140487

> Please please take note of my [#1666 (comment)](https://github.com/asciimoo/searx/issues/1666#issuecomment-656140487)
**...to support chrome also I added... to the intro site...** 

Yes, that works OK with Chromium on my Linux Desktop PC.
It doesn't work if I go to https://darmarit.org/searx
It only appears if I go to the intro site https://darmarit.org/

But none of this works with Chromium on my ANDROID phone.
---------------------------------------------------------

Also with Desktop Chromium metager.org and qwant.com appear straight away without the need to make a search
or visit a root directory.

But that's Desktop Chromium.
OpenSearch works as documented "...without a path. For example http://mysite.com, but not http://mysite.com/foo".

But this is not how it works with Chromium on my ANDROID phone.
For metager.org and qwant.com they appear only if I do a search first.
That will be equivalent to "http://mysite.com/foo".
For darmarit.org/ and darmarit.org/searx and searx.be they don't appear, nothing nada zilch.

Screenshot with Chromium 81.0.4044.138 and Linux Mint 16.04.

![YScreenshot](https://user-images.githubusercontent.com/640761/87221036-edac7300-c360-11ea-8eeb-ad8070ca997d.jpg)

After doing some research I got my [Searx instance](https://searx.be) added under "Recently visited" without any modification to the Searx code:
![Screenshot_20200711-112514_Chromium_1](https://user-images.githubusercontent.com/4016501/87221055-b934b900-c358-11ea-96ba-9bee09146f39.png)

## Explanation
Basically Google Chrome on Android doesn't want to add the website if it hasn't visited the path. In the default opensearch.xml of Searx the path is `/search?q={searchTerms}`.

So in order to make it work you have to visit the `/search?q={searchTerms}` path with a GET method. Google Chrome on Android is very pesky so if you visit the path with a POST request it won't work.

## Working example

Here is a working example with my own Searx instance:
1. Do a normal search on the instance: https://searx.be
2. Visit this URL: https://searx.be/search?q=ok
3. The Searx instance is now added in "Recently visited" in the settings.

## How would we fix that in Searx?
1. First thing first is to remove the `search` path from the opensearch.xml because by default Searx doesn't/never use this path for its requests from the home page search form:
```patch
9c9
<     <Url rel="results" type="text/html" method="get" template="{{ host }}search?q={searchTerms}"/>
---
>     <Url rel="results" type="text/html" method="get" template="{{ host }}?q={searchTerms}"/>
```
2. Set GET method by default for Google Chrome Android users because like I said earlier Google Chrome on Android **won't** add the website under "Recently visited" if it hasn't visited the path "/?q={searchTerms}" **with a GET method**. I know it's bad for privacy but unfortunately that's the only way to have it working but if someone has a better idea then I'm open for suggestions.
![image](https://user-images.githubusercontent.com/4016501/87221370-277a7b00-c35b-11ea-8ba9-0616ed32d4dc.png)
OK
![searx_be](https://user-images.githubusercontent.com/640761/87221692-dfad2100-c365-11ea-9b50-ef4da5b7bf8f.jpg)

> Set GET method by default for Google Chrome Android users because like I said earlier Google Chrome on Android won't add the website under "Recently visited" if it hasn't visited the path "/?q={searchTerms}" with a GET method. I know it's bad for privacy but unfortunately that's the only way to have it working but if someone has a better idea then I'm open for suggestions.

I just want to leave the hint, that we have exceptions implemented for Chrome clients:https://github.com/asciimoo/searx/blob/c41db5f1ba9984a916261535a9227fadb897f4bf/searx/webapp.py#L946-L955

https://github.com/asciimoo/searx/blob/c41db5f1ba9984a916261535a9227fadb897f4bf/searx/templates/__common__/opensearch.xml#L8-L14

> I just want to leave the hint, that we have exceptions implemented for Chrome clients:

I'm talking about a search request not a request to the opensearch path/route.

Please understand that Google Chrome on Android doesn't behave the same way as Google Chrome on Desktop.
> Please understand that Google Chrome on Android doesn't behave the same way as Google Chrome on Desktop.

Since https://github.com/asciimoo/searx/issues/1666#issuecomment-657025284 it is clear.

> How would we fix that in Searx?
>    First thing first is to remove the `search` path from the opensearch.xml because by default Searx doesn't/never use this path for its requests from the home page search form:

The the `search` path comes from the template and the template has implemented a condition for the 'get' method.  The 'get or post' method in the context is set in the webapp.y line 947, in line 953 we have a special treatment ... when the string `webkit` is in the header (User-Agent) this implementation asumes the client is a chrome based browser and sets the 'get' method.

I thought that was relevant / sorry if it was noise.

> ## Working example
Hi
The 1,2,3 method from unixfox works OK with other instances too.

![success](https://user-images.githubusercontent.com/640761/87249797-2a5b9580-c459-11ea-925d-5ad4d4758ee6.jpg)



That's a bit funny. We introduced the `/search` path just because of chrome.. If you make searx your default search engine with search url `/`, then chrome/chromium doesn't allow to send POST requests anymore to `/` - so as a workaround we added `/search`, so we can do GET requests at `/search` and then continue to use POST on `/`. But seems, this causes our current issue with the mobile chrome.. Hurray chrome..
Perhaps we could switch the GET search url to `/` and use `/search` for doing POST requests. In this case the opensearch should work and we could still do POST requests if the search isn't performed from the URL bar.
> That's a bit funny. 

Hi
I've tested it again with the latest public instance that I can find.
0.17.0-5-7b19e6c5

curl shows method="post".
curl with ANDROID Chromium user-agent shows method="get".

But it did not appear in "Recently Visited" until I followed 1,2,3 method from unixfox.

```
@mint ~ $ curl https://suche.uferwerk.org/opensearch.xml
<?xml version="1.0" encoding="utf-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
  <ShortName>searx</ShortName>
  <Description>a privacy-respecting, hackable metasearch engine</Description>
  <InputEncoding>UTF-8</InputEncoding>
  <Image>https://suche.uferwerk.org/static/themes/oscar/img/favicon.png</Image>
  <LongName>searx metasearch</LongName>
    <Url rel="results" type="text/html" method="post" template="https://suche.uferwerk.org/">
        <Param name="q" value="{searchTerms}" />
    </Url>
</OpenSearchDescription>

@mint ~ $ curl -A "Mozilla/5.0 (Linux;Android 9; Unspecified Device) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.0.0 Mobile Safari/537.36" https://suche.uferwerk.org/opensearch.xml
<?xml version="1.0" encoding="utf-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
  <ShortName>searx</ShortName>
  <Description>a privacy-respecting, hackable metasearch engine</Description>
  <InputEncoding>UTF-8</InputEncoding>
  <Image>https://suche.uferwerk.org/static/themes/oscar/img/favicon.png</Image>
  <LongName>searx metasearch</LongName>
    <Url rel="results" type="text/html" method="get" template="https://suche.uferwerk.org/search?q={searchTerms}"/>
</OpenSearchDescription>
```



Thanks for testing.

> But it did not appear in "Recently Visited" until I followed 1,2,3 method from unixfox.

Yeah, that's why I suggested to swap the search urls in this case and use `/` for GET requests and opensearch and `/search` for post.
I don't have an Android to test this right now, but wouldn't something like this work?
In [webapp.py, inside the index function between lines 561 and 562](https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L561):
```
user_agent = request.headers.get('User-Agent', '').lower()
if request.path == '/' and 'webkit' in user_agent and 'Android' in user_agent:
    return redirect('/search', 308)
```
I know this is a horrible hack, but it looks like it could work. Besides, we already handle special case for Chrome [somewhere else in webapp.py](https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L953), so it wouldn't even be the first time we do something like this.

Could this have a bad side-effect? I tried this with desktop Chromium, both with and without the `and 'Android' in user_agent` part and I didn't find any obvious failure.
@MarcAbonce this could also work.
> I don't have an Android to test this right now,...
> ```
@**MarcAbonce** 
Hi
If you try this "horrible hack" on your instance I will be pleased to test it with my ANDROID phone.
@MarcAbonce 

> I don't have an Android to test this right now, but wouldn't something like this work?
> In [webapp.py, inside the index function between lines 561 and 562](https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L561):
> 
> ```
> user_agent = request.headers.get('User-Agent', '').lower()
> if request.path == '/' and 'webkit' in user_agent and 'Android' in user_agent:
>     return redirect('/search', 308)
> ```
> 
> I know this is a horrible hack, but it looks like it could work. Besides, we already handle special case for Chrome [somewhere else in webapp.py](https://github.com/asciimoo/searx/blob/master/searx/webapp.py#L953), so it wouldn't even be the first time we do something like this.
> 
> Could this have a bad side-effect? I tried this with desktop Chromium, both with and without the `and 'Android' in user_agent` part and I didn't find any obvious failure.

This won't work because Google Chrome on Android wants you to visit `/search?q=yourkeyword` with a GET method and not just `/search`. When I said that Google Chrome on Android is very annoying, I really mean it.

I just tested your patch anyway (I modified `Android` to `android` in the 2nd line) and indeed it doesn't work. Also, it breaks the search if I change `method` to `GET` in the preferences.

Here is the patch in case I didn't modify the correct line:
````patch
559a560,563
>     user_agent = request.headers.get('User-Agent', '').lower()
>     if request.path == '/' and 'webkit' in user_agent and 'android' in user_agent:
>         return redirect('/search', 308)
>
````
> When I said that Google Chrome on Android is very annoying, I really mean it.

This is hilarious. @unixfox is there a documentation which describes this behavior?
> This is hilarious. @unixfox is there a documentation which describes this behavior?

I did some search and I haven't found anything documenting this behavior. In fact there is hardly anything documenting the opensearch behavior on Google Chrome for Android. Just some comments on Chromium's bug tracker explaining some kind of behavior but that's it.

The best idea is to look at the code but it's not an easy task with the amount of code that Google Chrome on Android is made of.
@unixfox respect for the research.

I think the redirect could still work with adding the query as GET parameter.

Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -463,6 +463,9 @@ def pre_request():
     request.errors = []
 
     preferences = Preferences(themes, list(categories.keys()), engines, plugins)
+    user_agent = request.headers.get('User-Agent', '').lower()
+    if 'webkit' in user_agent and 'android' in user_agent:
+        preferences.key_value_settings['method'].value = 'GET'
     request.preferences = preferences
     try:
         preferences.parse_dict(request.cookies)


Test patch: