internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: yt-dlp/yt-dlp

Issue:
Disable thumbnails conversion
### Checklist

- [X] I'm reporting a feature request
- [X] I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- [X] I've verified that I'm running yt-dlp version **2022.03.08.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Description

When embedding thumbnails using `--embed-thumbnail`, they get automatically converted to `.png`
People like me would like to keep them in the original format. (i.e. `webp`), which is compatible with the `.mkv` container when inserted as attachment.

actual output
```
[ThumbnailsConvertor] Converting thumbnail "/path/to/thumbnail.webp" to png
[EmbedThumbnail] ffmpeg: Adding thumbnail to "/path/to/video.mkv"
```

expected output
```
[EmbedThumbnail] ffmpeg: Adding thumbnail to "/path/to/thumbnail.mkv"
```


Discussion:
I would rather not have them as `webp`, because I always save videos as `mp4`.
So I think this should be an option instead of the default.
@nicolaasjan We can detect the file format and use webp only if it is mkv
* `--thumbnail-format=webp` ??
* `--(no-)convert-thumbnail` ??

No option is needed

> We can detect the file format and use webp only if it is mkv
Fair enough.

Resolving patch:
diff --git a/yt_dlp/postprocessor/embedthumbnail.py b/yt_dlp/postprocessor/embedthumbnail.py
--- a/yt_dlp/postprocessor/embedthumbnail.py
+++ b/yt_dlp/postprocessor/embedthumbnail.py
@@ -79,12 +79,10 @@ def run(self, info):
 
         original_thumbnail = thumbnail_filename = info['thumbnails'][idx]['filepath']
 
-        # Convert unsupported thumbnail formats to PNG (see #25687, #25717)
-        # Original behavior was to convert to JPG, but since JPG is a lossy
-        # format, there will be some additional data loss.
-        # PNG, on the other hand, is lossless.
         thumbnail_ext = os.path.splitext(thumbnail_filename)[1][1:]
-        if thumbnail_ext not in ('jpg', 'jpeg', 'png'):
+        # Convert unsupported thumbnail formats (see #25687, #25717)
+        # PNG is preferred since JPEG is lossy
+        if info['ext'] not in ('mkv', 'mka') and thumbnail_ext not in ('jpg', 'jpeg', 'png'):
             thumbnail_filename = convertor.convert_thumbnail(thumbnail_filename, 'png')
             thumbnail_ext = 'png'
 
@@ -102,7 +100,7 @@ def run(self, info):
         elif info['ext'] in ['mkv', 'mka']:
             options = list(self.stream_copy_opts())
 
-            mimetype = 'image/%s' % ('png' if thumbnail_ext == 'png' else 'jpeg')
+            mimetype = 'image/%s' % ('jpeg' if thumbnail_ext in ('jpg', 'jpeg') else thumbnail_ext)
             old_stream, new_stream = self.get_stream_number(
                 filename, ('tags', 'mimetype'), mimetype)
             if old_stream is not None:


Test patch:
Repository: yt-dlp/yt-dlp

Issue:
[TF1] Extracting subtitles
### Checklist

- [X] I'm reporting a site feature request
- [X] I've verified that I'm running yt-dlp version **2021.11.10.1**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

France

### Example URLs

https://www.tf1.fr/tf1/les-douze-coups-de-midi/videos/les-12-coups-de-midi-du-15-novembre-2021-95575074.html


### Description

yt-dlp can't fetch subtitles from TF1 at all.
I've checked the links in devs tools and (for the video quoted above) the segmented subs can be found found at:
https://das-q1-ssl.tf1.fr/2/USP-0x0/63/40/13826340/ssm/b1bcdeeab79b5130e4dc3ebc672cf9ce62d317429216edc2bce655df36daaabb.ism/dash/b1bcdeeab79b5130e4dc3ebc672cf9ce62d317429216edc2bce655df36daaabb-textstream_fra=1000.dash

You can get the vtt subs by simply replacing .dash at the end with .vtt.


So, would it be possible to update TF1 extractor so it can download subs (from. dash? .ism?)?

TIA


 



Discussion:

Resolving patch:
diff --git a/yt_dlp/extractor/wat.py b/yt_dlp/extractor/wat.py
--- a/yt_dlp/extractor/wat.py
+++ b/yt_dlp/extractor/wat.py
@@ -54,7 +54,7 @@ def _real_extract(self, url):
         #     'http://www.wat.tv/interface/contentv4s/' + video_id, video_id)
         video_data = self._download_json(
             'https://mediainfo.tf1.fr/mediainfocombo/' + video_id,
-            video_id, query={'context': 'MYTF1'})
+            video_id, query={'context': 'MYTF1', 'pver': '4020003'})
         video_info = video_data['media']
 
         error_desc = video_info.get('error_desc')


Test patch:
Repository: yt-dlp/yt-dlp

Issue:
use Python's native XATTR support rather than xattr/pyxattr
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm requesting a feature unrelated to a specific site
- [X] I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- [X] I've verified that I'm running yt-dlp version **2023.09.24** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

Hey.

AFAICS from current `master`:
https://github.com/yt-dlp/yt-dlp/blob/c54ddfba0f7d68034339426223d75373c5fc86df/yt_dlp/dependencies/__init__.py#L63
yt-dlp would always use the `xattr`-module (by one of the two implementations) fr XATTR support.

However, since version 3.3, Python itself has XATTR support:
https://docs.python.org/3/library/os.html#linux-extended-attributes

I think it would be nice if those function would be used and the dependencies could be dropped.

Cheers,
Chris.

PS: I guess the stuff from the issue template that I didn't check or fill out is not really relevant for this issue.

### Provide verbose output that clearly demonstrates the problem

- [ ] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [ ] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

_No response_


Discussion:

Resolving patch:
diff --git a/yt_dlp/utils/_utils.py b/yt_dlp/utils/_utils.py
--- a/yt_dlp/utils/_utils.py
+++ b/yt_dlp/utils/_utils.py
@@ -4441,10 +4441,12 @@ def write_xattr(path, key, value):
             raise XAttrMetadataError(e.errno, e.strerror)
         return
 
-    # UNIX Method 1. Use xattrs/pyxattrs modules
+    # UNIX Method 1. Use os.setxattr/xattrs/pyxattrs modules
 
     setxattr = None
-    if getattr(xattr, '_yt_dlp__identifier', None) == 'pyxattr':
+    if callable(getattr(os, 'setxattr', None)):
+        setxattr = os.setxattr
+    elif getattr(xattr, '_yt_dlp__identifier', None) == 'pyxattr':
         # Unicode arguments are not supported in pyxattr until version 0.5.0
         # See https://github.com/ytdl-org/youtube-dl/issues/5498
         if version_tuple(xattr.__version__) >= (0, 5, 0):


Test patch:
Repository: searxng/searxng

Issue:
Bug: engine "archive is" reports HTTP 404 / Not found
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**

1970d28a

**Technical report**

Error
  * Error: httpx.HTTPStatusError
  * Percentage: 100
  * Parameters: `('404', 'Not Found', 'archive.is')`
  * File name: `searx/search/processors/online.py:99`
  * Function: `_send_http_request`
  * Code: `response = req(params['url'], **request_args)`




Discussion:

Resolving patch:
diff --git a/searx/engines/xpath.py b/searx/engines/xpath.py
--- a/searx/engines/xpath.py
+++ b/searx/engines/xpath.py
@@ -14,6 +14,7 @@
 results_xpath = ''
 cached_xpath = ''
 cached_url = ''
+soft_max_redirects = 0
 
 # parameters for engines with paging support
 #
@@ -33,6 +34,7 @@ def request(query, params):
 
     params['url'] = search_url.format(**fp)
     params['query'] = query
+    params['soft_max_redirects'] = soft_max_redirects
 
     return params
 


Test patch:
Repository: searxng/searxng

Issue:
searx_extra/update/update_languages.py : crash on peertube engine
The URL https://peer.tube/api/v1/videos/languages doesn't exist anymore:
https://github.com/searxng/searxng/blob/ee83c99d2b40efaaf4f3e6635e7a43f7731cb128/searx/engines/peertube.py#L24-L25

See also https://framagit.org/framasoft/peertube/search-index/-/tree/master/server/controllers/api

----

See https://framagit.org/framasoft/peertube/search-index/-/blob/master/client/src/views/Search.vue#L644-667


Discussion:

Resolving patch:
diff --git a/searx/engines/peertube.py b/searx/engines/peertube.py
--- a/searx/engines/peertube.py
+++ b/searx/engines/peertube.py
@@ -22,7 +22,9 @@
 categories = ["videos"]
 paging = True
 base_url = "https://peer.tube"
-supported_languages_url = base_url + "/api/v1/videos/languages"
+supported_languages_url = (
+    'https://framagit.org/framasoft/peertube/search-index/-/raw/master/client/src/views/Search.vue'
+)
 
 
 # do search-request
@@ -92,5 +94,9 @@ def response(resp):
 
 
 def _fetch_supported_languages(resp):
-    peertube_languages = list(loads(resp.text).keys())
+    import re
+
+    # https://docs.python.org/3/howto/regex.html#greedy-versus-non-greedy
+    videolanguages = re.search(r"videoLanguages \(\) \{(.*?)\]", resp.text, re.DOTALL)
+    peertube_languages = [m.group(1) for m in re.finditer(r"\{ id: '([a-z]+)', label:", videolanguages.group(1))]
     return peertube_languages


Test patch:
Repository: searxng/searxng

Issue:
Bug: google news engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**

7ef33c91e057defde55cc90d4123f284910723d9

**How did you install SearXNG?**
docker

**How To Reproduce**
`!google_news time` crashes whatever the search language is

**Technical report**

Error
  * Error: KeyError
  * Percentage: 100
  * Parameters: `()`
  * File name: `searx/engines/google_news.py:97`
  * Function: `request`
  * Code: `if lang_info['hl'] == 'en':`

Checker
  * simple: `KeyError `,
  * lang_fr: `KeyError `,
  * lang_en: `KeyError `,


Discussion:
I will have a look ...

Resolving patch:
diff --git a/searx/engines/google_news.py b/searx/engines/google_news.py
--- a/searx/engines/google_news.py
+++ b/searx/engines/google_news.py
@@ -19,6 +19,7 @@
 # pylint: disable=invalid-name, missing-function-docstring
 
 import binascii
+from datetime import datetime
 import re
 from urllib.parse import urlencode
 from base64 import b64decode
@@ -94,8 +95,8 @@ def request(query, params):
     ceid = "%s:%s" % (lang_info['country'], lang_info['language'])
 
     # google news redirects en to en-US
-    if lang_info['hl'] == 'en':
-        lang_info['hl'] = 'en-US'
+    if lang_info['params']['hl'] == 'en':
+        lang_info['params']['hl'] = 'en-US'
 
     # Very special to google-news compared to other google engines, the time
     # range is included in the search term.
@@ -115,6 +116,7 @@ def request(query, params):
     params['headers']['Accept'] = (
         'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
         )
+    params['headers']['Cookie'] = "CONSENT=YES+cb.%s-14-p0.en+F+941;" % datetime.now().strftime("%Y%m%d")
 
     return params
 


Test patch:
Repository: searxng/searxng

Issue:
Bug: google images engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
30e4a2a2 

**How did you install SearXNG?**
docker, it happens on searx.be too.

**What happened?**
The engine crashes.

**How To Reproduce**
Search for anything using the engine, for example: `!goi anything`

**Expected behavior**
To get the results

**Screenshots & Logs**
![image](https://user-images.githubusercontent.com/1594191/123794822-968e0e00-d8e3-11eb-81cf-9836c1f76d98.png)

**Additional context**
Perhaps related to #175

**Technical report**

Error
  * Error: IndexError
  * Percentage: 5
  * Parameters: `()`
  * File name: `searx/engines/google_images.py:93`
  * Function: `scrap_img_by_id`
  * Code: `url_line = _script[i + 1]`




Discussion:
FYI: today I had a look at this issue .. the result from google-images has changed and the scrapper we have is not robust to handle this change. I think I have an idea how to harden the scrapper, but this needs more investigation and I do not have more time .. may be next weekend I have time to have a look.


related: https://github.com/searx/searx/pull/2115#issuecomment-876716010

Resolving patch:
diff --git a/searx/engines/google_images.py b/searx/engines/google_images.py
--- a/searx/engines/google_images.py
+++ b/searx/engines/google_images.py
@@ -84,7 +84,7 @@ def scrap_img_by_id(script, data_id):
     img_url = ''
     _script = script.split('\n')
     for i, line in enumerate(_script):
-        if 'gstatic.com/images' in line and data_id in line:
+        if 'gstatic.com/images' in line and data_id in line and i + 1 < len(_script):
             url_line = _script[i + 1]
             img_url = url_line.split('"')[1]
             img_url = unquote(img_url.replace(r'\u00', r'%'))


Test patch:
Repository: searxng/searxng

Issue:
Bug: openstreetmap engine
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
Repository: https://github.com/searxng/searxng
Branch: master
Version: 1.0.0-728-b57d776e

**How did you install SearXNG?**
`git clone`

**What happened?**
osm engine crash

**How To Reproduce**
any query

**Technical report**

Error
  * Error: NameError
  * Percentage: 100
  * Parameters: `()`
  * File name: `searx/engines/wikidata.py:137`
  * Function: `send_wikidata_query`
  * Code: `logger.debug('request time %s', str(http_response.elapsed))`

---

* https://github.com/searxng/searxng/pull/296 initialize an logger per engine.
* https://github.com/searxng/searxng/blob/b57d776edb65665cb514d5f8197c6f1960c80412/searx/engines/openstreetmap.py#L18
* the wikidata module imported by openstreetmap doesn't contains `logger`
* this line raises an exception:
  https://github.com/searxng/searxng/blob/b57d776edb65665cb514d5f8197c6f1960c80412/searx/engines/wikidata.py#L137



Discussion:

Resolving patch:
diff --git a/searx/engines/openstreetmap.py b/searx/engines/openstreetmap.py
--- a/searx/engines/openstreetmap.py
+++ b/searx/engines/openstreetmap.py
@@ -439,3 +439,8 @@ def get_key_label(key_name, lang):
         if labels is None:
             return None
     return get_label(labels, lang)
+
+
+def init(_):
+    import searx.engines.wikidata  # pylint: disable=import-outside-toplevel
+    searx.engines.wikidata.logger = logger


Test patch:
Repository: searxng/searxng

Issue:
Search suggestions are lumped together if Yahoo is enabled
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**
1.0.0-940-32fb2bdf, master branch, not forked

**How did you install SearXNG?**
searxng-docker, fresh install from yesterday.

**What happened?**
Search keyword suggestions are lumped together in one.

**How To Reproduce**
Enable the Yahoo engine.
You can also reproduce this issue with the Yahoo bang (!yh). 

**Expected behavior**
Normally, you would have separate keyword suggestions instead of what's happening right now. 

**Screenshots & Logs**
![image](https://user-images.githubusercontent.com/52939984/138177579-a1a93608-e5d3-4617-9b35-73e858fd6e98.png)

**Additional context**
I have Google, Qwant, Duckduckgo, Startpage, Brave and Yahoo engines enabled by default for all users.



Discussion:

Resolving patch:
diff --git a/searx/engines/yahoo.py b/searx/engines/yahoo.py
--- a/searx/engines/yahoo.py
+++ b/searx/engines/yahoo.py
@@ -153,7 +153,7 @@ def response(resp):
             'content': content
         })
 
-    for suggestion in eval_xpath_list(dom, '//div[contains(@class, "AlsoTry")]'):
+    for suggestion in eval_xpath_list(dom, '//div[contains(@class, "AlsoTry")]//table//a'):
         # append suggestion
         results.append({'suggestion': extract_text(suggestion)})
 


Test patch:
Repository: searxng/searxng

Issue:
Disabling all engines in a category makes the bang search in general
**How To Reproduce**
1. Disable all engines in one category in the user preferences (e.g. in the files category).
2. Search in the category using the bang syntax (e.g. `!files test`).

**Expected behavior**
No results (and maybe a message that all engines in this category are disabled).

**Observed behavior**
The search is performed in the general category.


Discussion:

Resolving patch:
diff --git a/searx/webadapter.py b/searx/webadapter.py
--- a/searx/webadapter.py
+++ b/searx/webadapter.py
@@ -236,7 +236,7 @@ def get_search_query_from_webapp(
     external_bang = raw_text_query.external_bang
     engine_data = parse_engine_data(form)
 
-    if not is_locked('categories') and raw_text_query.enginerefs and raw_text_query.specific:
+    if not is_locked('categories') and raw_text_query.specific:
         # if engines are calculated from query,
         # set categories by using that informations
         query_engineref_list = raw_text_query.enginerefs


Test patch:
Repository: searxng/searxng

Issue:
Engine without an about property raises an exception
**Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG**

current master branch

**How did you install SearXNG?**

Add an engine without an about property to your `settings.yml` / e.g.:

```yaml
  - name: locate
    engine: command
    command: ['locate', '{{QUERY}}']
    shortcut: locate
    tokens: [ '123', '456' ]
    disabled: true
    delimiter:
        chars: ' '
        keys: ['line']
```

**What happened?**

    $ make run

Open preferences ..

```
Traceback (most recent call last):
  ...
  File "searxng/searx/webutils.py", line 162, in engine_sort_key
    return (engine.about.get('language', ''), engine.name)
AttributeError: module 'command' has no attribute 'about'
```

**Additional context**

@not-my-profile if you have time, could you please have a look about / thanks!


Discussion:

Resolving patch:
diff --git a/searx/engines/__init__.py b/searx/engines/__init__.py
--- a/searx/engines/__init__.py
+++ b/searx/engines/__init__.py
@@ -44,6 +44,7 @@
     "enable_http": False,
     "display_error_messages": True,
     "tokens": [],
+    "about": {},
 }
 # set automatically when an engine does not have any tab category
 OTHER_CATEGORY = 'other'


Test patch:
Repository: searxng/searxng

Issue:
Search results pulled from google inaccurate
Search results pulled from google inaccurate searching for  some pulls horrible results 

![image](https://user-images.githubusercontent.com/39034611/153094845-47568f41-346e-44b7-a758-1e3cf3d9c118.png)

 

![image](https://user-images.githubusercontent.com/39034611/153095194-e73dba33-85ca-41d7-93df-f421aca627c0.png)

The first results should be pulled for such sites not outdated or odd links



Discussion:
> Search results pulled from google inaccurate searching for some pulls horrible results
> The first results should be pulled for such sites not outdated or odd links

Sorry I do not understand what you asking for and the screenshot do not look bad to me. Can you please explain a little more precise what you expect and if you use your own instance or an public one (which?) / thanks.

FYI: we do filter out some adds from google resulta, this might confuse when you are looking for product advertising.
I'm the one that asked @asianmusicguy to create an issue here because I can reproduce the issue too.

Basically, I don't really know why, but it seems like since the previous modifications of the XPath of Google engine the results are not great. I can reproduce the same issue with "English" as a default language.

Here is an obvious issue in the results given by Google, when I search for "telegraf centos" I don't find the official documentation on how to install the latest version of telegraf on centos.

On Google:
![image](https://user-images.githubusercontent.com/4016501/153153669-b487a36a-abe3-4c93-ada2-4d6acf390597.png)

On Searx.be:
![image](https://user-images.githubusercontent.com/4016501/153153737-4f1740be-e6cc-4a96-b3b6-1c6fd79a1e8c.png)

The URL I should have on the first results: https://docs.influxdata.com/telegraf/v1.21/introduction/installation/

I get the same bad results with these keywords too:
- `videojs Range header`: I don't get this link: github.com /videojs /video.js/issues/1968
- `royal tsx mosh`: I don't get this link: https://support.royalapps.com/support/discussions/topics/17000018892

Resolving patch:
diff --git a/searx/engines/google.py b/searx/engines/google.py
--- a/searx/engines/google.py
+++ b/searx/engines/google.py
@@ -111,9 +111,8 @@
 # specific xpath variables
 # ------------------------
 
-# google results are grouped into <div class="g ..." ../>
-results_xpath = '//div[@id="search"]//div[contains(@class, "g ")]'
-results_xpath_mobile_ui = '//div[contains(@class, "g ")]'
+# google results are grouped into <div class="jtfYYd ..." ../>
+results_xpath = '//div[@class="jtfYYd"]'
 
 # google *sections* are no usual *results*, we ignore them
 g_section_with_header = './g-section-with-header'
@@ -338,11 +337,7 @@ def response(resp):
 
     # parse results
 
-    _results_xpath = results_xpath
-    if use_mobile_ui:
-        _results_xpath = results_xpath_mobile_ui
-
-    for result in eval_xpath_list(dom, _results_xpath):
+    for result in eval_xpath_list(dom, results_xpath):
 
         # google *sections*
         if extract_text(eval_xpath(result, g_section_with_header)):


Test patch: