Repository: yt-dlp/yt-dlp

Issue:
[Redgifs] Stopped working, gives HTTP Error 403: Forbidden 
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.08.19** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Earth

### Provide a description that is worded well enough to be understood

Redgifs is giving `HTTP Error 403: Forbidden`, started recently. 

Playing in the browser, I see it uses thumbs4.redgifs.com (+ some query strings)

https://thumbs4.redgifs.com/IllFatDarwinsfox.mp4?expires=1661967000&signature=38ad19e81038ae4cdae01999001cbee43061ccced3dc2bdfd62b97fb1ea74050&for=[REDACTED-IP]#t=5.483452

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.redgifs.com/watch/illfatdarwinsfox']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.19 [48c88e0] (zip)
[debug] Python 3.8.13 (CPython 64bit) - FreeBSD-13.0-RELEASE-p11-amd64-64bit-ELF (libc 7)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.12.0, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.19, Current version: 2022.08.19
yt-dlp is up to date (2022.08.19)
[debug] [RedGifs] Extracting URL: https://www.redgifs.com/watch/illfatdarwinsfox
[RedGifs] illfatdarwinsfox: Downloading video info
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] illfatdarwinsfox: Downloading 1 format(s): hd
[debug] Invoking http downloader on "https://thumbs2.redgifs.com/IllFatDarwinsfox.mp4"
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "/home/weird/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3170, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "/home/weird/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2894, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/weird/bin/yt-dlp/yt_dlp/downloader/common.py", line 442, in download
    ret = self.real_download(filename, info_dict)
  File "/home/weird/bin/yt-dlp/yt_dlp/downloader/http.py", line 371, in real_download
    establish_connection()
  File "/home/weird/bin/yt-dlp/yt_dlp/downloader/http.py", line 129, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/weird/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3648, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/local/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/local/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
```



Discussion:
I saw that today. Sadly RedGifs did some changes and yt-dlp doesn't work.
@chio0hai / @Deer-Spangle  may be interested
We use their [API](https://api.redgifs.com/docs/index.html) for URLs of the videos, which currently returns URLs in thumbs2.redgifs.com that are broken at the moment.
e.g. The api request, https://api.redgifs.com/v2/gifs/illfatdarwinsfox, returns the following URLs for HD and SD videos:
https://thumbs2.redgifs.com/IllFatDarwinsfox.mp4
https://thumbs2.redgifs.com/IllFatDarwinsfox-mobile.mp4

The html page itself returns URLs in the og:video meta values in thumbs4.redgifs.com, and contain extra parameters.
https://thumbs4.redgifs.com/PrivateVirtualAcornweevil.mp4?expires=xxx&signature=yyy&for=1.2.3.4
https://thumbs4.redgifs.com/PrivateVirtualAcornweevil-mobile.mp4?expires=xxx&signature=yyy&for=1.2.3.4

Interestingly, they have another server, thumbs3.redgifs.com, which appears to work without the extra parameters.

Maybe we could replace "thumbs2.redgifs.com" with "thumbs3.redgifs.com" as a workaround for now?

```diff
--- a/yt_dlp/extractor/redgifs.py
+++ b/yt_dlp/extractor/redgifs.py
@@ -30,6 +30,7 @@ def _parse_gif_data(self, gif_data):
             video_url = gif_data['urls'].get(format_id)
             if not video_url:
                 continue
+            video_url = video_url.replace('thumbs2.redgifs.com', 'thumbs3.redgifs.com')
             height = min(orig_height, height or orig_height)
             formats.append({
                 'url': video_url,
```
P. S. This [post](https://old.reddit.com/r/redgifs/comments/x29szf/reddit_3rd_party_apps_changes_to_our_api/) talks about "some changes to our API" and "necessary keys in order to use the new API". However, I couldn't find the relevant information.

I think the options we have at the moment are:
- The workaround above.
- Change our code to take URLs from the html page instead of the json data from the api.

P. S. Guess they are starting to enforce [client tokens](https://github.com/Redgifs/api/wiki/API-access) for the API.

I have found a way to make their API work. Forgive me for being cryptic, but I worry they might "fix" it once again.

After some reading and analysis, it turns out their main website uses an OAuth bearer token on the API, i.e. one which is "deemed good" by simply having it. How long it lasts or where it comes from is not clear, so I don't want to paste it in here lest they change it. But it appears to have a fairly long life.

Given this, it is possible for anyone to download videos again by doing the following:

1. View a video on their website with developer tools open.
2. Watch for the request to api.redgifs.com, and notice the `authorizaton` header. This contains the bearer token.
3. Add the following patch, including the token where I put `<...>`
4. Convince Cloudflare you are not trying to imitate a browser (caused by the default user agent I believe)

```diff
diff --git a/yt_dlp/extractor/redgifs.py b/yt_dlp/extractor/redgifs.py
index e3712a1d6..b5bb7ffc6 100644
--- a/yt_dlp/extractor/redgifs.py
+++ b/yt_dlp/extractor/redgifs.py
@@ -59,6 +59,11 @@ def _parse_gif_data(self, gif_data):
 
     def _call_api(self, ep, video_id, *args, **kwargs):
+        headers = {'authorization': 'Bearer <...>',
+                   'x-customheader': f'https://www.redgifs.com/watch/{video_id}',
+                   'referer': 'https://www.redgifs.com/',
+                   'origin': 'https://www.redgifs.com',
+                   'content-type': 'application/json'}
         data = self._download_json(
-            f'https://api.redgifs.com/v2/{ep}', video_id, *args, **kwargs)
+            f'https://api.redgifs.com/v2/{ep}', video_id, headers=headers, *args, **kwargs)
         if 'error' in data:
             raise ExtractorError(f'RedGifs said: {data["error"]}', expected=True, video_id=video_id)
```

Good luck!

Resolving patch:
diff --git a/yt_dlp/extractor/redgifs.py b/yt_dlp/extractor/redgifs.py
--- a/yt_dlp/extractor/redgifs.py
+++ b/yt_dlp/extractor/redgifs.py
@@ -18,6 +18,12 @@ class RedGifsBaseInfoExtractor(InfoExtractor):
         'hd': None,
     }
 
+    _API_HEADERS = {
+        'referer': 'https://www.redgifs.com/',
+        'origin': 'https://www.redgifs.com',
+        'content-type': 'application/json',
+    }
+
     def _parse_gif_data(self, gif_data):
         video_id = gif_data.get('id')
         quality = qualities(tuple(self._FORMATS.keys()))
@@ -43,7 +49,7 @@ def _parse_gif_data(self, gif_data):
         return {
             'id': video_id,
             'webpage_url': f'https://redgifs.com/watch/{video_id}',
-            'ie_key': RedGifsIE.ie_key(),
+            'extractor_key': RedGifsIE.ie_key(),
             'extractor': 'RedGifs',
             'title': ' '.join(gif_data.get('tags') or []) or 'RedGifs',
             'timestamp': int_or_none(gif_data.get('createDate')),
@@ -57,9 +63,29 @@ def _parse_gif_data(self, gif_data):
             'formats': formats,
         }
 
+    def _fetch_oauth_token(self, video_id):
+        # These pages contain the OAuth token that is necessary to make API calls.
+        index_page = self._download_webpage(f'https://www.redgifs.com/watch/{video_id}', video_id)
+        index_js_uri = self._html_search_regex(
+            r'href="?(/assets/js/index[.a-z0-9]*.js)"?\W', index_page, 'index_js_uri')
+        index_js = self._download_webpage(f'https://www.redgifs.com/{index_js_uri}', video_id)
+        # It turns out that a { followed by any valid JSON punctuation will always result in the
+        # first two characters of the base64 encoding being "ey".
+        # Use this fact to find any such string constant of a reasonable length with the correct
+        # punctuation for an oauth token
+        oauth_token = self._html_search_regex(
+            r'\w+\s*[=:]\s*"(ey[^"]+\.[^"]*\.[^"]{43,45})"', index_js, 'oauth token')
+        self._API_HEADERS['authorization'] = f'Bearer {oauth_token}'
+
     def _call_api(self, ep, video_id, *args, **kwargs):
+        if 'authorization' not in self._API_HEADERS:
+            self._fetch_oauth_token(video_id)
+        assert 'authorization' in self._API_HEADERS
+
+        headers = dict(self._API_HEADERS)
+        headers['x-customheader'] = f'https://www.redgifs.com/watch/{video_id}'
         data = self._download_json(
-            f'https://api.redgifs.com/v2/{ep}', video_id, *args, **kwargs)
+            f'https://api.redgifs.com/v2/{ep}', video_id, headers=headers, *args, **kwargs)
         if 'error' in data:
             raise ExtractorError(f'RedGifs said: {data["error"]}', expected=True, video_id=video_id)
         return data
@@ -102,6 +128,7 @@ class RedGifsIE(RedGifsBaseInfoExtractor):
             'like_count': int,
             'categories': list,
             'age_limit': 18,
+            'tags': list,
         }
     }, {
         'url': 'https://thumbs2.redgifs.com/SqueakyHelplessWisent-mobile.mp4#t=0',
@@ -117,13 +144,14 @@ class RedGifsIE(RedGifsBaseInfoExtractor):
             'like_count': int,
             'categories': list,
             'age_limit': 18,
+            'tags': list,
         }
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url).lower()
         video_info = self._call_api(
-            f'gifs/{video_id}', video_id, note='Downloading video info')
+            f'gifs/{video_id}?views=yes', video_id, note='Downloading video info')
         return self._parse_gif_data(video_info['gif'])
 
 


Test patch: