Repository: searx/searx

Issue:
Searchengine Wikipedia causes Internal Error 500 when searching for placenames
When the searchengine Wikipedia is enabled; I get Internal server errors ("TypeError: encoding without a string argument") when searching for placenames (tested with searchterms 'Amsterdam', 'Singapore', 'Eindhoven' and others).

Disabling Wikipedia completely avoids these errors.


Discussion:
hi @djpbessems, unfortunately i can't reproduce it

can you post error traceback?

~~based on the error it seem it come from this line~~

```python
    wikipedia_link = base_url.format(language=url_lang(resp.search_params['language'])) \
        + 'wiki/' + quote(title.replace(' ', '_').encode('utf8'))
```
Sure: https://pastebin.com/2y56HCbd

Thanks for looking into it along with me :)
can you add this to new_hmac function on utils module

```python
    import logging
    logging.debug('secret_key: {!r}\nurl: {!r}'.format(secret_key, url))
```
on my machine it output following for `Singapore` search
```
DEBUG:root:secret_key: 'ultrasecretkey'
url: b'https://upload.wikimedia.org/wikipedia/commons/thumb/4/48/Flag_of_Singapore.svg/300px-Flag_of_Singapore.svg.png'
```
This is my output:
```
DEBUG:root:secret_key: b'D?0k\xe0%BI\xd6H\xe8\xf8h>\x82\x1d\xb6?)A;\x05\xa4\x94\xce\xa2\xa4\x866\xf5\x7f7'
url: b'https://upload.wikimedia.org/wikipedia/commons/thumb/4/48/Flag_of_Singapore.svg/300px-Flag_of_Singapore.svg.png'
```
```python
return hmac.new(bytes(secret_key, 'utf-8'), url, hashlib.sha256).hexdigest()
TypeError: encoding without a string argument
```
looking at error once again and your debug output, it is strange that your secret keys is in bytes instead of string

i will create pr for to handle this case

Resolving patch:
diff --git a/searx/utils.py b/searx/utils.py
--- a/searx/utils.py
+++ b/searx/utils.py
@@ -384,10 +384,17 @@ def load_module(filename, module_dir):
 
 
 def new_hmac(secret_key, url):
+    try:
+        secret_key_bytes = bytes(secret_key, 'utf-8')
+    except TypeError as err:
+        if isinstance(secret_key, bytes):
+            secret_key_bytes = secret_key
+        else:
+            raise err
     if sys.version_info[0] == 2:
         return hmac.new(bytes(secret_key), url, hashlib.sha256).hexdigest()
     else:
-        return hmac.new(bytes(secret_key, 'utf-8'), url, hashlib.sha256).hexdigest()
+        return hmac.new(secret_key_bytes, url, hashlib.sha256).hexdigest()
 
 
 def to_string(obj):


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -128,3 +128,17 @@ def test_write_rows(self):
         rows = [1, 2, 3]
         self.unicode_writer.writerows(rows)
         self.assertEqual(self.unicode_writer.writerow.call_count, len(rows))
+
+
+class TestNewHmac(SearxTestCase):
+
+    def test_bytes(self):
+        for secret_key in ['secret', b'secret', 1]:
+            if secret_key == 1:
+                with self.assertRaises(TypeError):
+                    utils.new_hmac(secret_key, b'http://example.com')
+                continue
+            res = utils.new_hmac(secret_key, b'http://example.com')
+            self.assertEqual(
+                res,
+                '23e2baa2404012a5cc8e4a18b4aabf0dde4cb9b56f679ddc0fd6d7c24339d819')