Repository: searx/searx

Issue:
Using single apostrophe causes a "search error"
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->

**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
1.1.0-27-bf0a583f
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
``` 
If you are using searx-docker then look at the bottom of the Searx page
and check for the version after "Powered by searx"

Please also stipulate if you are using a forked version of Searx and
include a link to the fork source code.
-->
**How did you install Searx?**
searx-docker. Specifically using docker-compose and running ``docker-compose up``.
Compose file:
```yaml
version: '3.3'
services:
    searx:
        volumes:
            - /mnt/storage1/configs/searx:/etc/searx
        ports:
            - 9999:8080
        environment:
            - BASE_URL=http://localhost:9999/
        image: searx/searx
        restart: unless-stopped
```
<!-- Did you install Searx using the official wiki or using searx-docker
or manually by executing the searx/webapp.py file? -->
**What happened?**
Using a single apostrophe in a word (for example, "What's") causes a search error
<!-- A clear and concise description of what the bug is. -->
**How To Reproduce**
Use any single apostrophe in a search query
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
**Expected behavior**
Should return results, instead of an error.
<!-- A clear and concise description of what you expected to happen. -->
**Screenshots & Logs**
https://imgur.com/a/E2b4mNX
https://paste.ee/p/e5gS8

**Additional context**
It's also somewhat infuriating that it also returns from a result page to the home page when this happens, meaning I have to retype my query (though that's another issue entirely).


Discussion:
Can't reproduce on a local install of searx. Please test on a vanilla browser without any extension installed.

Please also explain your setup and how you installed searx.
I've updated the original post and added a bit more info about the installation.

Using a fresh Firefox profile it worked. However disabling all extensions on my current profile didn't.

Here's the network request from Firefox it makes:
![image](https://user-images.githubusercontent.com/7394119/201664509-1d92d4c6-5e7b-4d3c-b337-fc7f42f54cea.png)
Docker logs doesn't seem to give anything useful, though for completeness I'll post in the original post.

(I'm not familiar with how searx is build, but how could the browser config cause an error to a GET request? Maybe something I can look into)
Have you tried using version 1.1.0-37 by chance?
> Have you tried using version 1.1.0-37 by chance?

Updated to latest (that being 1.1.0-37), still gives the same error
Here is the URL from your picture that I tried testing on my instance with the same version:

`/search?q=what's&language=en-US&time_range=&category_general=on&safesearch=0&theme=simple`

I am also running 1.1.0-37 with no issue. See below for my request headers running on Windows + Firefox 107.0.

```
GET /search?q=what%27s&category_general=on&time_range=&language=en-US HTTP/2
Host: localhost:8080
User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:107.0) Gecko/20100101 Firefox/107.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Cookie: categories=general; language=en-US; locale=en; autocomplete=; autofocus=1; archive_today=0; image_proxy=; method=GET; safesearch=0; theme=oscar; results_on_new_tab=0; doi_resolver=oadoi.org; oscar-style=logicodev; advanced_search=0; disabled_engines=; enabled_engines=; disabled_plugins=; enabled_plugins=; tokens=
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Sec-GPC: 1
TE: trailers
```

Something tells me its an issue with your FireFox profile based on this comment:
`Using a fresh Firefox profile it worked. However disabling all extensions on my current profile didn't.` 

What is also interesting is that the error is on submitting

Maybe try this on both the fresh profile and your current profile:
1. Open your searx instance homepage
2. Developer Tools -> Network -> Settings -> Persist Logs
3. Search "what's"
4. After results load: Developer Tools -> Network -> Settings -> Save All as HAR

Upload the HAR results and we can compare the two.

Looking at the HAR file I noticed there were a lot of cookies. Deleting all the cookies however fixed it. So it seems some cookie(s) was/were causing it to act up. I've included the file with cookies and the one without for reference.

~With cookies:
https://paste.ee/p/VDdo7
Without cookies:
https://paste.ee/p/lBcZq~

edit: Just noticed how to upload files, uploading them here instead.
[without cookies.txt](https://github.com/searx/searx/files/10037183/without.cookies.txt)
[cookies.txt](https://github.com/searx/searx/files/10037184/cookies.txt)

Amazing, thank you! I'll try running it locally outside of docker later to see if I can replicate it. For quick reference to anyone else, here is the list of cookies:

```json
"cookies": [
	{
	  "name": "jwt",
	  "value": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwiZW1haWwiOiJ0ZXN0QHRlc3QuY29tIiwibmFtZSI6IkFkbWluaXN0cmF0b3IiLCJhdiI6bnVsbCwidHoiOiJBbWVyaWNhL05ld19Zb3JrIiwibGMiOiJlbiIsImRmIjoiIiwiYXAiOiIiLCJwZXJtaXNzaW9ucyI6WyJtYW5hZ2U6c3lzdGVtIl0sImdyb3VwcyI6WzFdLCJpYXQiOjE2NTk1MzgyMzIsImV4cCI6MTY1OTU0MDAzMiwiYXVkIjoidXJuOndpa2kuanMiLCJpc3MiOiJ1cm46d2lraS5qcyJ9.YbhAMASmZnjaxUn9XrzYW6p-bVWKLO-74YqisWz_WHT-vpEMoo4SAR472ntqZmLX6edLrdLgiHfLQcGYrTWbgdjuCmwwDbI9HwxmC14GmYYSspJSO4FghKrSnrgeLBRBuHef9KnWwEADuSojbcUD87CzFKHm2h-fpwjPNuBa4l0aE58iWztrdcJR83YTR3d-FY6cfQACo7Y4f_VbLtpI-e5hxZD3s_QCztq3vwREiJj4PB3fIeaW23gnRexSTKNX0LlfUl5Wt03yWTZlxRnYtI4JqZWFnE3HczFFvU8iIWGPAhfr7eBohQ-FyLwtKP2zdfFNdWLVcgEal3sT0EmE0Q"
	},
	{
	  "name": "compact_display_state",
	  "value": "true"
	},
	{
	  "name": "categories",
	  "value": "general"
	},
	{
	  "name": "language",
	  "value": "en-US"
	},
	{
	  "name": "locale",
	  "value": "en"
	},
	{
	  "name": "autocomplete",
	  "value": ""
	},
	{
	  "name": "autofocus",
	  "value": "1"
	},
	{
	  "name": "archive_today",
	  "value": "0"
	},
	{
	  "name": "image_proxy",
	  "value": ""
	},
	{
	  "name": "method",
	  "value": "GET"
	},
	{
	  "name": "safesearch",
	  "value": "0"
	},
	{
	  "name": "theme",
	  "value": "simple"
	},
	{
	  "name": "results_on_new_tab",
	  "value": "0"
	},
	{
	  "name": "doi_resolver",
	  "value": "oadoi.org"
	},
	{
	  "name": "oscar-style",
	  "value": "logicodev"
	},
	{
	  "name": "advanced_search",
	  "value": "0"
	},
	{
	  "name": "disabled_engines",
	  "value": "brave__general"
	},
	{
	  "name": "enabled_engines",
	  "value": ""
	},
	{
	  "name": "disabled_plugins",
	  "value": ""
	},
	{
	  "name": "enabled_plugins",
	  "value": "\"Open_Access_DOI_rewrite\\054Search_operators\\054Vim-like_hotkeys\""
	},
	{
	  "name": "tokens",
	  "value": ""
	},
	{
	  "name": "maintab",
	  "value": "on"
	},
	{
	  "name": "enginetab",
	  "value": "on"
	},
	{
	  "name": "session",
	  "value": "1b226b8b-3cfe-400f-bb1f-8184c91d1992"
	}
]
```
Any update on this? I have this problem also. Browsing with Firefox 108 under Linux. Searx built from source, at 117dbd462f39fb4cc2f12 .  The server logging output is:

```
ERROR:searx.search.processor.online:engine brave : requests exception(search duration : 0.274029016494751 s, timeout: 6[23/1905]
'Received response with content-encoding: br, but failed to decode it.', error('BrotliDecoderDecompressStream failed while processing the stream'))                                                                                                             
Traceback (most recent call last):                                                                                              
  File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 407, in _decode
    data = self._decoder.decompress(data)                                                                                       
brotli.error: BrotliDecoderDecompressStream failed while processing the stream
                                                                
During handling of the above exception, another exception occurred:
                                                                
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 832, in read_chunked
    decoded = self._decode(                                                                                                     
  File "/usr/lib/python3.10/site-packages/urllib3/response.py", line 410, in _decode                                  
    raise DecodeError(                                                                                                          
urllib3.exceptions.DecodeError: ('Received response with content-encoding: br, but failed to decode it.', error('BrotliDecoderDecompressStream failed while processing the stream'))                                                                            
                                                                                                                                
During handling of the above exception, another exception occurred:                                                        
                                                                                                                                
Traceback (most recent call last):
  File "/home/searx/searx/searx/search/processors/online.py", line 144, in search
    search_results = self._search_basic(query, params)
  File "/home/searx/searx/searx/search/processors/online.py", line 124, in _search_basic
    response = self._send_http_request(params)
  File "/home/searx/searx/searx/search/processors/online.py", line 96, in _send_http_request
    response = req(params['url'], **request_args)
  File "/home/searx/searx/searx/poolrequests.py", line 209, in get
    return request('get', url, **kwargs)
  File "/home/searx/searx/searx/poolrequests.py", line 181, in request
    response = session.request(method=method, url=url, **kwargs) 
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 745, in send
    r.content
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 820, in generate
    raise ContentDecodingError(e)
requests.exceptions.ContentDecodingError: ('Received response with content-encoding: br, but failed to decode it.', error('BrotliDecoderDecompressStream failed while processing the stream'))
ERROR:searx.webapp:search error
Traceback (most recent call last):
  File "/home/searx/searx/searx/webapp.py", line 615, in search
    result_container = search.search()
  File "/home/searx/searx/searx/search/__init__.py", line 211, in search
    plugins.call(self.ordered_plugin_list, 'on_result', self.request, self, result)
  File "/home/searx/searx/searx/plugins/__init__.py", line 82, in call
    ret = getattr(plugin, plugin_type)(request, *args, **kwargs) 
  File "/home/searx/searx/searx/plugins/search_operators.py", line 14, in on_result
    qs = shlex.split(q)
  File "/usr/lib/python3.10/shlex.py", line 315, in split
    return list(lex)
  File "/usr/lib/python3.10/shlex.py", line 300, in __next__
    token = self.get_token()
  File "/usr/lib/python3.10/shlex.py", line 109, in get_token
    raw = self.read_token()
  File "/usr/lib/python3.10/shlex.py", line 191, in read_token
    raise ValueError("No closing quotation")
ValueError: No closing quotation 
```


It seems to be a problem with the input given to `shlex.split`? https://docs.python.org/3/library/shlex.html

https://github.com/searx/searx/blob/117dbd462f39fb4cc2f12e9f7b9acf8d3fff419b/searx/plugins/search_operators.py#L12-L14
Recently the error returned, so I tried building it from source. I got the same exception as @Pistos. I'll try removing cookies one at a time to see if it's a specific one that causes it
These 2 cookies seem to cause the problem:
```
localhost	FALSE	/	FALSE	1673349962	disabled_plugins	
localhost	FALSE	/	FALSE	1673350178	enabled_plugins	Search_operators
```
Deleting either one fixes it.
I'll try get to more logging details later
Have the same issue myself, I put Firefox in troubleshoot mode to disable all extensions and loaded the public instance I use. I got the error with an apostrophe every time, Then I deleted the disabled_plugins cookie as mentioned above and ran the search again and the search went through with no error.

Resolving patch:
diff --git a/searx/plugins/search_operators.py b/searx/plugins/search_operators.py
--- a/searx/plugins/search_operators.py
+++ b/searx/plugins/search_operators.py
@@ -11,7 +11,11 @@
 
 def on_result(request, search, result):
     q = search.search_query.query
-    qs = shlex.split(q)
+    # WARN: shlex.quote is designed only for Unix shells and may be vulnerable
+    # to command injection on non-POSIX compliant shells (Windows)
+    # https://docs.python.org/3/library/shlex.html#shlex.quote
+    squote = shlex.quote(q)
+    qs = shlex.split(squote)
     spitems = [x.lower() for x in qs if ' ' in x]
     mitems = [x.lower() for x in qs if x.startswith('-')]
     siteitems = [x.lower() for x in qs if x.startswith('site:')]


Test patch: