Repository: searx/searx

Issue:
Brainstorming: on_result plugin / https_rewrite plugin
Different plugin ideas require to filter / modify the results according to the host of the URL. 

Usage examples:
* the https_rewrite plugin, and the current implementation fails to do so: https://github.com/asciimoo/searx/tree/master/searx/plugins/https_rules are outdated, and the number of rules is a fraction of the actual list to avoid an overhead. Here there is one lookup per .xml files: https://github.com/asciimoo/searx/blob/6e37255e59693d7a3768491b2b8d06441051131f/searx/plugins/https_rewrite.py#L140
* [Add support for Youtube and Twitter links rewrite PR #1726](https://github.com/asciimoo/searx/pull/1726) rewrite some URL too.
* [feature request: private user blocklists/blacklists #2001](https://github.com/asciimoo/searx/issues/2001) asks to remove some domain names from the results.

Currently, ```on_result``` plugin scan for each URLs and process them. 
What if we provide provide each plugin register for a list of domain.

Some technical solutions to create a static list of domains:
* a [trie](https://en.wikipedia.org/wiki/Trie). (This issue superseed this one https://github.com/asciimoo/searx/issues/936 )
* one hash table per label in the domain name: https://github.com/sethmlarson/hstspreload

According to https://www.eff.org/https-everywhere/rulesets#wildcard-targets the wildcard in HTTPS everywhere can be either:
* *.eff.org
* www.eff.*

A solution: two lookups in two different structures, one in reversed order (org -> eff -> *), one in normal order (www -> eff -> *).

Prototype: https://gist.github.com/dalf/26462c1627c8b00fdfacdd55525b08b6



The idea is provide: 
* a format to store the rules (either a binary or plain text format), with a module to create this output format.
* a plugin can register this structure: the searx core calls the plugin where there is a match.
* it is difficult to merge all trie without a lot of repetition, some most probably one "double trie" per plugin; but in simple cases I guess it can be done.

Note about the issue #2001:
* the choice of domains to remove is a preference, then the issue is not solved.
* if we can make some maintained list, then the issue is solved.



Discussion:
Related to https://github.com/asciimoo/searx/issues/211

More or less to https://github.com/asciimoo/searx/issues/260


Resolving patch:
diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py
--- a/searx/engines/duckduckgo.py
+++ b/searx/engines/duckduckgo.py
@@ -18,7 +18,7 @@
 
 # about
 about = {
-    "website": 'https://lite.duckduckgo.com/lite',
+    "website": 'https://lite.duckduckgo.com/lite/',
     "wikidata_id": 'Q12805',
     "official_api_documentation": 'https://duckduckgo.com/api',
     "use_official_api": False,
@@ -45,7 +45,7 @@
 time_range_dict = {'day': 'd', 'week': 'w', 'month': 'm', 'year': 'y'}
 
 # search-url
-url = 'https://lite.duckduckgo.com/lite'
+url = 'https://lite.duckduckgo.com/lite/'
 url_ping = 'https://duckduckgo.com/t/sl_l'
 
 
@@ -73,6 +73,7 @@ def request(query, params):
     # link again and again ..
 
     params['headers']['Content-Type'] = 'application/x-www-form-urlencoded'
+    params['headers']['Referer'] = 'https://lite.duckduckgo.com/'
 
     # initial page does not have an offset
     if params['pageno'] == 2:


Test patch: