Repository: searx/searx

Issue:
Capability to rewrite hits url 
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->

**Is your feature request related to a problem? Please describe.**
Not related to a problem with `searx`, per se. I am looking to have `searx` rewrite the hits urls to another domain. 



<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->
For example, if my results have a few urls from `www.reddit.com`, I'd like to have those rewitten to `old.reddit.com`.  If you could point me to where to start looking at, I'd be glad to send a PR if I am successful. My apologies if this already exists - if so, please guide how to accomplish this. 

In `settings.yml`, a section to include plain domain rewrites
<pre>
www.reddit.com: old.reddit.com
www1.domain.com: www3.domain.com
old.foo.org: new.foo.org
</pre>
**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
Currently, I go to the address bar and perform manual rewrites. It would be great if `searx` can do the rewrites for me instead.  
**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->
Sometimes, you'd want to access a specific endpoint for a results URI. 




Discussion:
You can implement this functionality as a plugin. We have a documentation on plugins: https://searx.github.io/searx/dev/plugins.html

We already have a plugin that rewrite URLs here: https://github.com/searx/searx/blob/master/searx/plugins/https_rewrite.py. It rewrites URLs from HTTP to HTTPS. You could use that plugin as a base or example.

I'm not sure how you would allow customization in a plugin though, for example to add URLs like www.reddit.com that should be rewritten to old.reddit.com. Maybe @dalf could give you a few hints about that.

As a personal note, your feature request really interest me and I would also really like to see it implemented.
Related to
* https://github.com/searx/searx/issues/2080
* https://github.com/searx/searx/blob/master/searx/data/external_urls.json 
      * see `["urls"]["default"]` 
      * the purpose is to fetch indivious, nitter, ... instances and automatically update this file. See https://github.com/searx/searx/pull/2500
      * if it is implemented, then the user can choose which instance to use.
      * it conflicts with the idea that everything is inside `settings.yml` (but [sci-hub URL](https://github.com/searx/searx/blob/e2fb50089219207f7e11bcfc049b48f2f4b9a6c8/searx/settings.yml#L1399) is not updated)

Some untested code:

-----

`plugins/generic_url_rewrite.py` :

```python
import re
import yaml
from urllib.parse import urlparse, urlunparse
from lxml import etree
from os import listdir, environ
from os.path import isfile, isdir, join
from searx.plugins import logger
from flask_babel import gettext
from searx import searx_dir


name = "generic URL rewrite"
description = gettext('generic URL rewrite')
default_on = True
preference_section = 'privacy'


config_path = join(searx_dir, 'plugins/generic_url_rewrite/config.yml')
config = None

logger = logger.getChild("https_rewrite")


def load_yaml(file_name):
    try:
        with open(file_name, 'r', encoding='utf-8') as settings_yaml:
            return yaml.safe_load(settings_yaml)
    except IOError as e:
        raise SearxSettingsException(e, file_name) from e
    except yaml.YAMLError as e:
        raise SearxSettingsException(e, file_name) from e


def get_config():
	"""Alternative: from searx import settings
        and then return settings['plugins']['generic_rewrite']
	"""
	global config
	if config is not None:
		config = load_yaml(config_path)
	return config


def rewrite_url(result, new_hostname):
	parsed_url = result['parsed_url']
	new_parsed_url = parsed_url._replace(netloc="{}:{}".format(new_hostname, parsed_url.port))
	result['parsed_url'] = new_parsed_url
	result['url'] = urlunparse(new_parsed_url)


def on_result(request, search, result):
    if 'parsed_url' not in result:
        return True

	config = get_config()
	new_hostname = config.get(result['parsed_url'].hostname)
	if config_for_hostname:
		rewrite_url(result, new_hostname)
    return True

```

-----

`plugins/generic_url_rewrite/config.yml` :

```yaml
www.reddit.com: old.reddit.com
www1.domain.com: www3.domain.com
old.foo.org: new.foo.org
```

Thank you @dalf, @unixfox. This was a good opportunity to know `searx` a little closer! 

1. In your example, I think you may have meant `new_hostname` for `config_for_hostname`. 
2. So I know: is it required that we rewrite `result[parsed_url]` too? My below attempt (based on your above example) to change just `result[url]` seems to work. 

<pre>
import yaml
from searx import searx_dir
from searx.plugins import logger
from os.path import join 
from flask_babel import gettext

name = gettext("generic URL rewrite")
description = gettext("generic URL rewrite")
default_on = True
preference_section = "general"

configPath = join(searx_dir, "plugins/generic_url_rewrite/config.yml")
config = None

logger = logger.getChild("generic_url_rewrite")

def loadYaml(fileName):
	try:
		with open(fileName) as fobj:
			return yaml.safe_load(fobj)
	except Exception as e:
		raise SearxSettingsException(e, fileName) from e

def get_config():
	global config
	if config is None:
		config = loadYaml(configPath)
	else:
		return config 

def on_result(request, search, result):
	if "parsed_url" not in result:
		return True
	config = get_config()

	newHostname = config.get(result["parsed_url"].hostname)
	if newHostname:
		splat = result["url"].split("/")
		splat[2] = newHostname
		mod = "/".join(splat)
		result["url"] = mod
	return True
</pre>


> 1. In your example, I think you may have meant new_hostname for config_for_hostname.

Yes 

> 2. So I know: is it required that we rewrite result[parsed_url] too? My below attempt (based on your above example) to change just result[url] seems to work.

Searx excepts `result["parsed_url"]` to match `result["url"]`:
* Some other plugins relies on `result[parsed_url]`
* The webapp relies on it (okay... just the CSV output format)

Thanks!

Considering (1) how wonderfully easy it was to achieve my original goal and also (2) 
since it added a requirement of `python3-yaml` to be installed,
 I am not certain this feature request warrants a PR as another `sears` plugin. Please let me know if I am mistaken, I can submit a PR.  (I do lack the `searx` culture/practice.)

Either way, these are the steps I took on my system:

1. Setup `plugins/generic_url_rewrite/config.yml `:
<pre>
www.reddit.com: old.reddit.com
www1.domain.com: www3.domain.com
old.foo.org: new.foo.org
 </pre>
2. Contents of  `plugins/generic_url_rewrite.py` :
<pre>
import yaml
from searx import searx_dir
from searx.plugins import logger
from os.path import join as pathJoin
from flask_babel import gettext
import urllib.parse


name = gettext("generic URL rewrite")
description = gettext("generic URL rewrite")
default_on = True
preference_section = "general"

configPath = pathJoin(searx_dir, "plugins/generic_url_rewrite/config.yml")
config = None

logger = logger.getChild("generic_url_rewrite")

def loadYaml(fileName):
	try:
		with open(fileName) as fobj:
			return yaml.safe_load(fobj)
	except Exception as e:
		raise SearxSettingsException(e, fileName) from e

def get_config():
	global config
	if config is None:
		config = loadYaml(configPath)
	else:
		return config

def rewriteUrl(result, newHostname):
	splat = result["url"].split("/")
	splat[2] = newHostname
	newUrl = "/".join(splat)
	result["url"] = newUrl

	parsed_url = result["parsed_url"]
	new_parsed = parsed_url._replace(netloc = "{dest}:{port}".format(dest=newUrl, port=parsed_url.port))
	result["parsed_url"] = new_parsed 

def on_result(request, search, result):
	if "parsed_url" not in result:
		return True
	config = get_config()

	newHostname = config.get(result["parsed_url"].hostname)
	if newHostname:
		rewriteUrl(result, newHostname)

	return True
</pre>
Why using `python3-yaml` instead of the already included `pyyaml` library?
We surely welcome any new contribution, especially new plugins.
> Why using `python3-yaml` instead of the already included `pyyaml` library?

@unixfox, I was not able to import `yaml` on my repl  terminal unless `python3-yaml` was installed - Did not know this was not an added dependency for `searx`. 
>  We surely welcome any new contribution, especially new plugins.

https://github.com/searx/searx/pull/2637 - checks seem to pass - 
pls see https://github.com/searx/searx/pull/2638 

Resolving patch:
diff --git a/searx/plugins/__init__.py b/searx/plugins/__init__.py
--- a/searx/plugins/__init__.py
+++ b/searx/plugins/__init__.py
@@ -32,6 +32,7 @@
                            https_rewrite,
                            infinite_scroll,
                            self_info,
+                           hostname_replace,
                            search_on_category_select,
                            tracker_url_remover,
                            vim_hotkeys)
@@ -168,6 +169,7 @@ def sha_sum(filename):
 plugins.register(https_rewrite)
 plugins.register(infinite_scroll)
 plugins.register(self_info)
+plugins.register(hostname_replace)
 plugins.register(search_on_category_select)
 plugins.register(tracker_url_remover)
 plugins.register(vim_hotkeys)
diff --git a/searx/plugins/hostname_replace.py b/searx/plugins/hostname_replace.py
new file mode 100644
--- /dev/null
+++ b/searx/plugins/hostname_replace.py
@@ -0,0 +1,28 @@
+import re
+from urllib.parse import urlunparse
+from searx import settings
+from searx.plugins import logger
+from flask_babel import gettext
+
+name = gettext('Hostname replace')
+description = gettext('Rewrite result hostnames')
+default_on = False
+preference_section = 'general'
+
+plugin_id = 'hostname_replace'
+parsed = 'parsed_url'
+
+replacements = {re.compile(p): r for (p, r) in settings[plugin_id].items()} if plugin_id in settings else {}
+
+logger = logger.getChild(plugin_id)
+
+
+def on_result(request, search, result):
+    if parsed not in result:
+        return True
+    for (pattern, replacement) in replacements.items():
+        if pattern.search(result[parsed].netloc):
+            result[parsed] = result[parsed]._replace(netloc=pattern.sub(replacement, result[parsed].netloc))
+            result['url'] = urlunparse(result[parsed])
+
+    return True


Test patch: