Repository: searx/searx

Issue:
recoll engine
Recoll is a local search engine based on Xapian:

http://www.lesbonscomptes.com/recoll/

Although Recoll seems to be mostly aimed at desktop users the engine can be run without any graphical interface or interaction, I never use the GUI tools while I've been running it for many years over a large (currently ~2TB) document collection.

By itself recoll does not offer web or API access, this can be achieved using recoll-webui:

https://github.com/koniu/recoll-webui

As recoll-webui by default does not support paged JSON results it is advisable to use a patched version which does:

https://github.com/Yetangitu/recoll-webui/tree/jsonpage

   (A pull request was sent upstream, if this is merged the patched version is no longer needed)

This engine uses a custom `files` result template included in this PR (only for the _oscar_ theme using the _logicdev_ style, I can make versions for other themes/styles if this PR goes through)

Use:
===

set `base_url` to the location where recoll-webui can be reached
set `dl_prefix` to a location where the file hierarchy as indexed by recoll can be reached
set `search_dir` to the part of the indexed file hierarchy to be searched, use an empty string to search the entire search domain

Example from settings.yml:

```yml
    # this entry (with search_dir set to an empty string) covers the entire recoll search domain
  - name : library
    engine : recoll
    shortcut : lib
    base_url: 'https://recoll.example.org/'
    search_dir : ''
    dl_prefix : 'https://download.example.org'
    timeout : 30.0
    categories : files

    # this entry only searches the 'reference' directory
  - name : library reference
    engine : recoll
    base_url: 'https://recoll.example.org/'
    search_dir : reference
    dl_prefix : 'https://download.example.org'
    shortcut : libr
    timeout : 30.0
    categories : files
    disabled : True
```

Example output:

![image](https://user-images.githubusercontent.com/2754838/38278112-07ac4266-379b-11e8-9947-c75e2aa48b0c.png)

BTW, I'm using Searx with a custom theme so I adapted the _oscar_ theme for this PR. I did not test the adaptations so if something doesn't work this ight be the cause. The result should more or less look like the image above.




Discussion:

Resolving patch:
diff --git a/searx/engines/recoll.py b/searx/engines/recoll.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/recoll.py
@@ -0,0 +1,104 @@
+"""
+ Recoll (local search engine)
+
+ @using-api   yes
+ @results     JSON
+ @stable      yes
+ @parse       url, content, size, abstract, author, mtype, subtype, time, \
+              filename, label, type, embedded
+"""
+
+from datetime import date, timedelta
+from json import loads
+from urllib.parse import urlencode, quote
+
+# engine dependent config
+time_range_support = True
+
+# parameters from settings.yml
+base_url = None
+search_dir = ''
+mount_prefix = None
+dl_prefix = None
+
+# embedded
+embedded_url = '<{ttype} controls height="166px" ' +\
+    'src="{url}" type="{mtype}"></{ttype}>'
+
+
+# helper functions
+def get_time_range(time_range):
+    sw = {
+        'day': 1,
+        'week': 7,
+        'month': 30,
+        'year': 365
+    }
+
+    offset = sw.get(time_range, 0)
+    if not offset:
+        return ''
+
+    return (date.today() - timedelta(days=offset)).isoformat()
+
+
+# do search-request
+def request(query, params):
+    search_after = get_time_range(params['time_range'])
+    search_url = base_url + 'json?{query}&highlight=0'
+    params['url'] = search_url.format(query=urlencode({
+        'query': query,
+        'after': search_after,
+        'dir': search_dir}))
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    response_json = loads(resp.text)
+
+    if not response_json:
+        return []
+
+    for result in response_json.get('results', []):
+        title = result['label']
+        url = result['url'].replace('file://' + mount_prefix, dl_prefix)
+        content = '{}'.format(result['snippet'])
+
+        # append result
+        item = {'url': url,
+                'title': title,
+                'content': content,
+                'template': 'files.html'}
+
+        if result['size']:
+            item['size'] = int(result['size'])
+
+        for parameter in ['filename', 'abstract', 'author', 'mtype', 'time']:
+            if result[parameter]:
+                item[parameter] = result[parameter]
+
+        # facilitate preview support for known mime types
+        if 'mtype' in result and '/' in result['mtype']:
+            (mtype, subtype) = result['mtype'].split('/')
+            item['mtype'] = mtype
+            item['subtype'] = subtype
+
+            if mtype in ['audio', 'video']:
+                item['embedded'] = embedded_url.format(
+                    ttype=mtype,
+                    url=quote(url.encode('utf8'), '/:'),
+                    mtype=result['mtype'])
+
+            if mtype in ['image'] and subtype in ['bmp', 'gif', 'jpeg', 'png']:
+                item['img_src'] = url
+
+        results.append(item)
+
+    if 'nres' in response_json:
+        results.append({'number_of_results': response_json['nres']})
+
+    return results


Test patch: