Repository: searx/searx

Issue:
Fix YaCy engine
This fixes a bug were the favicons of the general YaCy search results got interpreted as images.


Discussion:
How is it decided if and when this pull request gets merged?
And should I update my pull request to the latest state of `master`?
I also added a small enhancement to the yacy engine: It now supports http authentication.
While your at it 0xhtml, are you going to add support for "some search words /language/en" ? Yacy supports that. And yes, I know it's a stupid way to support languages but that's what yacy supports.
It would be great to merge this PR @asciimoo @dalf.

It fixes the following bug (got this error with searx-checker): **AttributeError: 'NoneType' object has no attribute 'startswith'**

> juin 23 17:04:41 arch-server.example.com python3[2040757]: yacy.Traceback (most recent call last):
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 228, in <module>
juin 23 17:04:41 arch-server.example.com python3[2040757]:     main(args.url, args.o)
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 215, in main
juin 23 17:04:41 arch-server.example.com python3[2040757]:     state = check_engines_state(instance_url, engines)
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 168, in check_engines_state
juin 23 17:04:41 arch-server.example.com python3[2040757]:     provides_results = _request_results(instance_url, engine)
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 156, in _request_results
juin 23 17:04:41 arch-server.example.com python3[2040757]:     engine_result = _query_engine_result(instance_url, engine, query)
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 146, in _query_engine_result
juin 23 17:04:41 arch-server.example.com python3[2040757]:     return _check_response(resp)
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 132, in _check_response
juin 23 17:04:41 arch-server.example.com python3[2040757]:     return _check_results(resp_json['results'])
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 86, in _check_results
juin 23 17:04:41 arch-server.example.com python3[2040757]:     if not _check_result(result):
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 78, in _check_result
juin 23 17:04:41 arch-server.example.com python3[2040757]:     return _is_url_image(result.get('thumbnail_src'))
juin 23 17:04:41 arch-server.example.com python3[2040757]:   File "checker/checker.py", line 42, in _is_url_image
juin 23 17:04:41 arch-server.example.com python3[2040757]:     if image_url.startswith('//'):
juin 23 17:04:41 arch-server.example.com python3[2040757]: AttributeError: 'NoneType' object has no attribute 'startswith'

Resolving patch:
diff --git a/searx/engines/yacy.py b/searx/engines/yacy.py
--- a/searx/engines/yacy.py
+++ b/searx/engines/yacy.py
@@ -16,6 +16,8 @@
 from dateutil import parser
 from urllib.parse import urlencode
 
+from requests.auth import HTTPDigestAuth
+
 from searx.utils import html_to_text
 
 # engine dependent config
@@ -23,6 +25,8 @@
 paging = True
 language_support = True
 number_of_results = 5
+http_digest_auth_user = ""
+http_digest_auth_pass = ""
 
 # search-url
 base_url = 'http://localhost:8090'
@@ -51,6 +55,9 @@ def request(query, params):
                           limit=number_of_results,
                           search_type=search_type)
 
+    if http_digest_auth_user and http_digest_auth_pass:
+        params['auth'] = HTTPDigestAuth(http_digest_auth_user, http_digest_auth_pass)
+
     # add language tag if specified
     if params['language'] != 'all':
         params['url'] += '&lr=lang_' + params['language'].split('-')[0]
@@ -75,7 +82,7 @@ def response(resp):
 
     for result in search_results[0].get('items', []):
         # parse image results
-        if result.get('image') and result.get('width') and result.get('height'):
+        if resp.search_params.get('category') == 'images':
 
             result_url = ''
             if 'url' in result:
@@ -104,5 +111,4 @@ def response(resp):
 
         # TODO parse video, audio and file results
 
-    # return results
     return results
diff --git a/searx/search.py b/searx/search.py
--- a/searx/search.py
+++ b/searx/search.py
@@ -97,7 +97,8 @@ def send_http_request(engine, request_params):
     request_args = dict(
         headers=request_params['headers'],
         cookies=request_params['cookies'],
-        verify=request_params['verify']
+        verify=request_params['verify'],
+        auth=request_params['auth']
     )
 
     # setting engine based proxies
@@ -281,7 +282,8 @@ def default_request_params():
         'data': {},
         'url': '',
         'cookies': {},
-        'verify': True
+        'verify': True,
+        'auth': None
     }
 
 


Test patch: