Repository: searxng/searxng

Issue:
DuckDuckGo Videos/News/Maps
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SearXNG -->

**Working URL to the engine**
<!-- Please check if the engine is responding correctly before submitting it. -->
https://duckduckgo.com/?q=news&t=h_&iar=news&ia=news -- a search for "news"
https://duckduckgo.com/?q=videos&t=h_&iax=videos&ia=videos -- a search for "videos"
https://duckduckgo.com/?q=maps&t=h_&ia=web&iaxm=maps -- a search for "maps"

**Why do you want to add this engine?**
<!-- What's special about this engine? Is it open source or libre? -->
It's DuckDuckGo and this specific news category of it is not yet implemented in SearX.

**Features of this engine**
<!-- Features of this engine: Doesn't track its users, fast, easy to integrate, ... -->
DuckDuckGo supposedly doesn't track its users and the news category is fairly reliable in my experience with another library (linked below).

**How can SearXNG fetch the information from this engine?**
<!-- List API URL, example code (using the correct markdown) and more
that could be useful for the developers in order to implement this engine.
If you don't know what to write, let this part blank.>
This package does it fairly well by scraping the `vqd` value from duckduckgo's main results for videos, images, and maps: https://github.com/deedy5/duckduckgo_search/blob/main/duckduckgo_search.py (is this kind of scraping supported by SearX? I'm not too familiar with the codebase)

**Applicable category of this engine**
<!-- Where should this new engine fit in SearXNG? Current categories in SearXNG:
general, files, images, it, map, music, news, science, social media and videos.
You can add multiple categories at the same time. -->
news, map, videos

**Additional context**
<!-- Add any other context about this engine here. -->
It would be great to be able to expand the number of news search engines SearX supports. At the very least, it would be nice to have news. I would love to submit a PR myself, but I wouldn't know how to implement or test it. Could someone point me in the right direction for something like a template for creating new engines?



Discussion:
Whoops, forgot to link to the library that does the searching:
https://pypi.org/project/duckduckgo-search/ 

Lines 323-625 in the [source](https://github.com/deedy5/duckduckgo_search/blob/main/duckduckgo_search.py) cover the scraping process. Is this breakage-prone? If so, does SearX allow simply importing the lib and using the results from there?

This would be nice.
Can something similar be done for Startpage Images/Videos/News?
> Can something similar be done for Startpage Images/Videos/News?

Its on my ToDo list since a long time .. but before we need to fix startpage (the fix is on the way https://github.com/searxng/searxng/pull/1667/commits/964d24134d8f589a278584f6d175905232a18486)
> > Can something similar be done for Startpage Images/Videos/News?
> 
> Its on my ToDo list since a long time .. but before we need to fix startpage (the fix is on the way [964d241](https://github.com/searxng/searxng/commit/964d24134d8f589a278584f6d175905232a18486))

That's great! I know it doesn't really make a difference when using SearXNG, but I still want to stick to private engines. But if I do that I only have DDG and Qwant in images and only Qwant in both videos and news (excluding Wikinews). So I'm looking forward to this!

PS: I don't have any issues with Startpage, but Brave on the other hand...

I actually got it to work yesterday, maybe once. It's reliability is consistently low, and it usually just returns `Error! Engines cannot retrieve results: brave ( [Suspended: too many requests](https://searx.namejeff.xyz/searx/stats?engine=brave) )`. This is on a public instance though.
FYI the API works very well, the only downside is that there is no paging option.
https://github.com/deedy5/duckduckgo_search/blob/main/duckduckgo_search/ddg_images.py#L15

Supports up to 1000 results https://github.com/deedy5/duckduckgo_search/blob/main/duckduckgo_search/ddg_images.py#L48
Has there been any progress on this? DuckDuckGo Videos/News/Maps and Startpage Images/Videos/News would be very welcome
> Has there been any progress on this? 

No, sorry, I haven't had time yet for .. but pull requests are welcome :-) 

> Can something similar be done for Startpage Images/Videos/News?

Its the same with startpage .. first we need to solve the CAPTCHA issue https://github.com/searxng/searxng/issues/1081 .. I have tried a lot but I have not found a solution for it yet.
> > Has there been any progress on this?
> 
> No, sorry, I haven't had time yet for .. but pull requests are welcome :-)

I might look into that at some point. If it's something I can manage.
> 
> > Can something similar be done for Startpage Images/Videos/News?
> 
> Its the same with startpage .. first we need to solve the CAPTCHA issue #1081 .. I have tried a lot but I have not found a solution for it yet.

Oh, that's unfortunate.

What about Brave? They also have images/videos/news
> What about Brave? They also have images/videos/news

so many opportunities, so little time .. we are a small team, SearXNG thrives on the support of the community .. contributions are welcome :+1:
@julianfairfax For Brave, you can test https://github.com/searxng/searxng/pull/2599

Resolving patch:
diff --git a/searx/engines/duckduckgo.py b/searx/engines/duckduckgo.py
--- a/searx/engines/duckduckgo.py
+++ b/searx/engines/duckduckgo.py
@@ -66,8 +66,10 @@ def cache_vqd(query, value):
     The vqd value depends on the query string and is needed for the follow up
     pages or the images loaded by a XMLHttpRequest:
 
-    - DuckDuckGo Web: `https://links.duckduckgo.com/d.js?q=...&vqd=...`
-    - DuckDuckGo Images: `https://duckduckgo.com/i.js??q=...&vqd=...`
+    - DuckDuckGo Web: ``https://links.duckduckgo.com/d.js?q=...&vqd=...``
+    - DuckDuckGo Images: ``https://duckduckgo.com/i.js??q=...&vqd=...``
+    - DuckDuckGo Videos: ``https://duckduckgo.com/v.js??q=...&vqd=...``
+    - DuckDuckGo News: ``https://duckduckgo.com/news.js??q=...&vqd=...``
 
     """
     c = redisdb.client()
diff --git a/searx/engines/duckduckgo_images.py b/searx/engines/duckduckgo_extra.py
similarity index 51%
rename from searx/engines/duckduckgo_images.py
rename to searx/engines/duckduckgo_extra.py
--- a/searx/engines/duckduckgo_images.py
+++ b/searx/engines/duckduckgo_extra.py
@@ -1,9 +1,10 @@
 # SPDX-License-Identifier: AGPL-3.0-or-later
 """
-DuckDuckGo Images
-~~~~~~~~~~~~~~~~~
+DuckDuckGo Extra (images, videos, news)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 """
 
+from datetime import datetime
 from typing import TYPE_CHECKING
 from urllib.parse import urlencode
 
@@ -32,6 +33,9 @@
 
 # engine dependent config
 categories = ['images', 'web']
+ddg_category = 'images'
+"""The category must be any of ``images``, ``videos`` and ``news``
+"""
 paging = True
 safesearch = True
 send_accept_language_header = True
@@ -39,6 +43,8 @@
 safesearch_cookies = {0: '-2', 1: None, 2: '1'}
 safesearch_args = {0: '1', 1: None, 2: '1'}
 
+search_path_map = {'images': 'i', 'videos': 'v', 'news': 'news'}
+
 
 def request(query, params):
 
@@ -69,28 +75,61 @@ def request(query, params):
         args['p'] = safe_search  # "-1", "1"
 
     logger.debug("cookies: %s", params['cookies'])
-    args = urlencode(args)
-    params['url'] = 'https://duckduckgo.com/i.js?{args}'.format(args=args)
+
+    params['url'] = f'https://duckduckgo.com/{search_path_map[ddg_category]}.js?{urlencode(args)}'
 
     return params
 
 
+def _image_result(result):
+    return {
+        'template': 'images.html',
+        'url': result['url'],
+        'title': result['title'],
+        'content': '',
+        'thumbnail_src': result['thumbnail'],
+        'img_src': result['image'],
+        'img_format': '%s x %s' % (result['width'], result['height']),
+        'source': result['source'],
+    }
+
+
+def _video_result(result):
+    return {
+        'template': 'videos.html',
+        'url': result['content'],
+        'title': result['title'],
+        'content': result['description'],
+        'thumbnail': result['images'].get('small') or result['images'].get('medium'),
+        'iframe_src': result['embed_url'],
+        'source': result['provider'],
+        'length': result['duration'],
+        'metadata': result.get('uploader'),
+    }
+
+
+def _news_result(result):
+    return {
+        'url': result['url'],
+        'title': result['title'],
+        'content': result['excerpt'],
+        'source': result['source'],
+        'publishedDate': datetime.utcfromtimestamp(result['date']),
+    }
+
+
 def response(resp):
     results = []
     res_json = resp.json()
 
     for result in res_json['results']:
-        results.append(
-            {
-                'template': 'images.html',
-                'title': result['title'],
-                'content': '',
-                'thumbnail_src': result['thumbnail'],
-                'img_src': result['image'],
-                'url': result['url'],
-                'img_format': '%s x %s' % (result['width'], result['height']),
-                'source': result['source'],
-            }
-        )
+        if ddg_category == 'images':
+            results.append(_image_result(result))
+        elif ddg_category == 'videos':
+            results.append(_video_result(result))
+        elif ddg_category == 'news':
+            results.append(_news_result(result))
+        else:
+            raise ValueError(f"Invalid duckduckgo category: {ddg_category}")
 
     return results


Test patch: