Repository: yt-dlp/yt-dlp
Pull request #12107 (2025-01-16 17:25:14)

[ie/twitter] Fix syndication token generation
Porting this over from [bellingcat/auto-archiver](https://github.com/bellingcat/auto-archiver).

Rewrites Vercel's [javascript token generator](https://github.com/vercel/react-tweet/blob/main/packages/react-tweet/src/api/fetch-tweet.ts#L27) into python, and fixes TODO

I also have a unit test, but didn't know where it should go in the test folder. These tokens are generated against Vercels' javascript generator:

```python
@pytest.mark.parametrize("tweet_id, expected_token", [
    ("1874097816571961839", "4jjngwkifa"),
    ("1674700676612386816", "42586mwa3uv"),
    ("1877747914073620506", "4jv4aahw36n"),
    ("1876710769913450647", "4jruzjz5lux"),
    ("1346554693649113090", "39ibqxei7mo")
    ])
def test_reverse_engineer_token(self, tweet_id, expected_token):
    # see Vercel's implementation here: https://github.com/vercel/react-tweet/blob/main/packages/react-tweet/src/api/fetch-tweet.ts#L27C1-L31C2
    # and the discussion here: https://github.com/JustAnotherArchivist/snscrape/issues/996#issuecomment-2211358215

    generated_token = self.archiver.generate_token(tweet_id)
    assert expected_token == generated_token
```


<details open><summary>Template</summary> <!-- OPEN is intentional -->

<!--

# PLEASE FOLLOW THE GUIDE BELOW

- You will be asked some questions, please read them **carefully** and answer honestly
- Put an `x` into all the boxes `[ ]` relevant to your *pull request* (like [x])
- Use *Preview* tab to see how your *pull request* will actually look like

-->

### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check all of the following options that apply:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)

### What is the purpose of your *pull request*?
- [x] Fix or improvement to an extractor (Make sure to add/update tests)

</details>


Grub4K: `js_number_to_string` from #12110 could potentially be used here. It should be more accurate in its results
pjrobertson: > `js_number_to_string` from #12110 could potentially be used here. It should be more accurate in its results

Should be possible. One thing to note here is that the length is important. In my tests, sometimes a longer string is returned, but the twitter token seems to be capped at 10 chars. Probably something for the `generate_token` function though and not the `js_number_to_string` function
Grub4K: Thats because of properties from `Number.prototype.toString(radix)`, and shouldn't be needed in any of these. the code would just be
```py
# ((Number(twid) / 1e15) * Math.PI).toString(36).replace(/(0+|\.)/g, '')
translation = str.maketrans(dict.fromkeys("0."))
js_number_to_string((int(tweet_id) / 1e15) * math.PI, 36).translate(translation)
```
bashonly: In my testing, a random 10-char string still works as the token. Is the token actually required under certain circumstances?
Grub4K: The pr got merged so the suggested code can now be used after merging master

Patch:
diff --git a/yt_dlp/extractor/francetv.py b/yt_dlp/extractor/francetv.py
index 01b18d8da..c6036b306 100644
--- a/yt_dlp/extractor/francetv.py
+++ b/yt_dlp/extractor/francetv.py
@@ -358,7 +358,8 @@ def _real_extract(self, url):
             # For livestreams we need the id of the stream instead of the currently airing episode id
             video_id = traverse_obj(nextjs_data, (
                 ..., ..., 'children', ..., 'children', ..., 'children', ..., 'children', ..., ...,
-                'children', ..., ..., 'children', ..., ..., 'children', ..., 'options', 'id', {str}, any))
+                'children', ..., ..., 'children', ..., ..., 'children', (..., (..., ...)),
+                'options', 'id', {str}, any))
         else:
             video_id = traverse_obj(nextjs_data, (
                 ..., ..., ..., 'children',
diff --git a/yt_dlp/extractor/twitter.py b/yt_dlp/extractor/twitter.py
index c05b5bf9c..d32ae3f18 100644
--- a/yt_dlp/extractor/twitter.py
+++ b/yt_dlp/extractor/twitter.py
@@ -1,11 +1,12 @@
 import functools
 import json
-import random
+import math
 import re
 import urllib.parse
 
 from .common import InfoExtractor
 from .periscope import PeriscopeBaseIE, PeriscopeIE
+from ..jsinterp import js_number_to_string
 from ..networking.exceptions import HTTPError
 from ..utils import (
     ExtractorError,
@@ -1330,6 +1331,11 @@ def _build_graphql_query(self, media_id):
             },
         }
 
+    def _generate_syndication_token(self, twid):
+        # ((Number(twid) / 1e15) * Math.PI).toString(36).replace(/(0+|\.)/g, '')
+        translation = str.maketrans(dict.fromkeys('0.'))
+        return js_number_to_string((int(twid) / 1e15) * math.PI, 36).translate(translation)
+
     def _call_syndication_api(self, twid):
         self.report_warning(
             'Not all metadata or media is available via syndication endpoint', twid, only_once=True)
@@ -1337,8 +1343,7 @@ def _call_syndication_api(self, twid):
             'https://cdn.syndication.twimg.com/tweet-result', twid, 'Downloading syndication JSON',
             headers={'User-Agent': 'Googlebot'}, query={
                 'id': twid,
-                # TODO: token = ((Number(twid) / 1e15) * Math.PI).toString(36).replace(/(0+|\.)/g, '')
-                'token': ''.join(random.choices('123456789abcdefghijklmnopqrstuvwxyz', k=10)),
+                'token': self._generate_syndication_token(twid),
             })
         if not status:
             raise ExtractorError('Syndication endpoint returned empty JSON response')
diff --git a/yt_dlp/extractor/zdf.py b/yt_dlp/extractor/zdf.py
index 703766cd7..b64a88f6c 100644
--- a/yt_dlp/extractor/zdf.py
+++ b/yt_dlp/extractor/zdf.py
@@ -187,12 +187,20 @@ class ZDFIE(ZDFBaseIE):
         'info_dict': {
             'id': '151025_magie_farben2_tex',
             'ext': 'mp4',
+            'duration': 2615.0,
             'title': 'Die Magie der Farben (2/2)',
             'description': 'md5:a89da10c928c6235401066b60a6d5c1a',
-            'duration': 2615,
             'timestamp': 1465021200,
-            'upload_date': '20160604',
             'thumbnail': 'https://www.zdf.de/assets/mauve-im-labor-100~768x432?cb=1464909117806',
+            'upload_date': '20160604',
+            'episode': 'Die Magie der Farben (2/2)',
+            'episode_id': 'POS_954f4170-36a5-4a41-a6cf-78f1f3b1f127',
+            'season': 'Staffel 1',
+            'series': 'Die Magie der Farben',
+            'season_number': 1,
+            'series_id': 'a39900dd-cdbd-4a6a-a413-44e8c6ae18bc',
+            'season_id': '5a92e619-8a0f-4410-a3d5-19c76fbebb37',
+            'episode_number': 2,
         },
     }, {
         'url': 'https://www.zdf.de/funk/druck-11790/funk-alles-ist-verzaubert-102.html',
@@ -200,12 +208,13 @@ class ZDFIE(ZDFBaseIE):
         'info_dict': {
             'ext': 'mp4',
             'id': 'video_funk_1770473',
-            'duration': 1278,
-            'description': 'Die Neue an der Schule verdreht Ismail den Kopf.',
+            'duration': 1278.0,
             'title': 'Alles ist verzaubert',
+            'description': 'Die Neue an der Schule verdreht Ismail den Kopf.',
             'timestamp': 1635520560,
-            'upload_date': '20211029',
             'thumbnail': 'https://www.zdf.de/assets/teaser-funk-alles-ist-verzaubert-102~1920x1080?cb=1663848412907',
+            'upload_date': '20211029',
+            'episode': 'Alles ist verzaubert',
         },
     }, {
         # Same as https://www.phoenix.de/sendungen/dokumentationen/gesten-der-maechtigen-i-a-89468.html?ref=suche
@@ -248,22 +257,52 @@ class ZDFIE(ZDFBaseIE):
             'title': 'Das Geld anderer Leute',
             'description': 'md5:cb6f660850dc5eb7d1ab776ea094959d',
             'duration': 2581.0,
-            'timestamp': 1675160100,
-            'upload_date': '20230131',
+            'timestamp': 1728983700,
+            'upload_date': '20241015',
             'thumbnail': 'https://epg-image.zdf.de/fotobase-webdelivery/images/e2d7e55a-09f0-424e-ac73-6cac4dd65f35?layout=2400x1350',
+            'series': 'SOKO Stuttgart',
+            'series_id': 'f862ce9a-6dd1-4388-a698-22b36ac4c9e9',
+            'season': 'Staffel 11',
+            'season_number': 11,
+            'season_id': 'ae1b4990-6d87-4970-a571-caccf1ba2879',
+            'episode': 'Das Geld anderer Leute',
+            'episode_number': 10,
+            'episode_id': 'POS_7f367934-f2f0-45cb-9081-736781ff2d23',
         },
     }, {
         'url': 'https://www.zdf.de/dokumentation/terra-x/unser-gruener-planet-wuesten-doku-100.html',
         'info_dict': {
-            'id': '220605_dk_gruener_planet_wuesten_tex',
+            'id': '220525_green_planet_makingof_1_tropen_tex',
+            'ext': 'mp4',
+            'title': 'Making-of Unser grüner Planet - Tropen',
+            'description': 'md5:d7c6949dc7c75c73c4ad51c785fb0b79',
+            'duration': 435.0,
+            'timestamp': 1653811200,
+            'upload_date': '20220529',
+            'format_note': 'hd, main',
+            'thumbnail': 'https://www.zdf.de/assets/unser-gruener-planet-making-of-1-tropen-100~3840x2160?cb=1653493335577',
+            'episode': 'Making-of Unser grüner Planet - Tropen',
+        },
+        'skip': 'No longer available: "Leider kein Video verfügbar"',
+    }, {
+        'url': 'https://www.zdf.de/serien/northern-lights/begegnung-auf-der-bruecke-100.html',
+        'info_dict': {
+            'id': '240319_2310_sendung_not',
             'ext': 'mp4',
-            'title': 'Unser grüner Planet - Wüsten',
-            'description': 'md5:4fc647b6f9c3796eea66f4a0baea2862',
-            'duration': 2613.0,
-            'timestamp': 1654450200,
-            'upload_date': '20220605',
-            'format_note': 'uhd, main',
-            'thumbnail': 'https://www.zdf.de/assets/saguaro-kakteen-102~3840x2160?cb=1655910690796',
+            'title': 'Begegnung auf der Brücke',
+            'description': 'md5:e53a555da87447f7f1207f10353f8e45',
+            'thumbnail': 'https://epg-image.zdf.de/fotobase-webdelivery/images/c5ff1d1f-f5c8-4468-86ac-1b2f1dbecc76?layout=2400x1350',
+            'upload_date': '20250203',
+            'duration': 3083.0,
+            'timestamp': 1738546500,
+            'series_id': '1d7a1879-01ee-4468-8237-c6b4ecd633c7',
+            'series': 'Northern Lights',
+            'season': 'Staffel 1',
+            'season_number': 1,
+            'season_id': '22ac26a2-4ea2-4055-ac0b-98b755cdf718',
+            'episode': 'Begegnung auf der Brücke',
+            'episode_number': 1,
+            'episode_id': 'POS_71049438-024b-471f-b472-4fe2e490d1fb',
         },
     }]
 
@@ -316,12 +355,31 @@ def _extract_entry(self, url, player, content, video_id):
             'timestamp': unified_timestamp(content.get('editorialDate')),
             'thumbnails': thumbnails,
             'chapters': chapters or None,
+            'episode': title,
+            **traverse_obj(content, ('programmeItem', 0, 'http://zdf.de/rels/target', {
+                'series_id': ('http://zdf.de/rels/cmdm/series', 'seriesUuid', {str}),
+                'series': ('http://zdf.de/rels/cmdm/series', 'seriesTitle', {str}),
+                'season': ('http://zdf.de/rels/cmdm/season', 'seasonTitle', {str}),
+                'season_number': ('http://zdf.de/rels/cmdm/season', 'seasonNumber', {int_or_none}),
+                'season_id': ('http://zdf.de/rels/cmdm/season', 'seasonUuid', {str}),
+                'episode_number': ('episodeNumber', {int_or_none}),
+                'episode_id': ('contentId', {str}),
+            })),
         })
 
     def _extract_regular(self, url, player, video_id):
-        content = self._call_api(
-            player['content'], video_id, 'content', player['apiToken'], url)
-        return self._extract_entry(player['content'], player, content, video_id)
+        player_url = player['content']
+
+        try:
+            content = self._call_api(
+                update_url_query(player_url, {'profile': 'player-3'}),
+                video_id, 'content', player['apiToken'], url)
+        except ExtractorError as e:
+            self.report_warning(f'{video_id}: {e.orig_msg}; retrying with v2 profile')
+            content = self._call_api(
+                player_url, video_id, 'content', player['apiToken'], url)
+
+        return self._extract_entry(player_url, player, content, video_id)
 
     def _extract_mobile(self, video_id):
         video = self._download_v2_doc(video_id)
diff --git a/yt_dlp/jsinterp.py b/yt_dlp/jsinterp.py
index bd6a47004..ac0629715 100644
--- a/yt_dlp/jsinterp.py
+++ b/yt_dlp/jsinterp.py
@@ -25,7 +25,7 @@ def zeroise(x):
         with contextlib.suppress(TypeError):
             if math.isnan(x):  # NB: NaN cannot be checked by membership
                 return 0
-        return x
+        return int(float(x))
 
     def wrapped(a, b):
         return op(zeroise(a), zeroise(b)) & 0xffffffff


Test patch:
diff --git a/test/test_jsinterp.py b/test/test_jsinterp.py
index c1464f2cd..ed3ca61c4 100644
--- a/test/test_jsinterp.py
+++ b/test/test_jsinterp.py
@@ -93,6 +93,16 @@ def test_operators(self):
         self._test('function f(){return 0 ?? 42;}', 0)
         self._test('function f(){return "life, the universe and everything" < 42;}', False)
         self._test('function f(){return 0  - 7 * - 6;}', 42)
+        self._test('function f(){return true << "5";}', 32)
+        self._test('function f(){return true << true;}', 2)
+        self._test('function f(){return "19" & "21.9";}', 17)
+        self._test('function f(){return "19" & false;}', 0)
+        self._test('function f(){return "11.0" >> "2.1";}', 2)
+        self._test('function f(){return 5 ^ 9;}', 12)
+        self._test('function f(){return 0.0 << NaN}', 0)
+        self._test('function f(){return null << undefined}', 0)
+        # TODO: Does not work due to number too large
+        # self._test('function f(){return 21 << 4294967297}', 42)
 
     def test_array_access(self):
         self._test('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2.0] = 7; return x;}', [5, 2, 7])
diff --git a/test/test_youtube_signature.py b/test/test_youtube_signature.py
index 13436f088..7ae627f2c 100644
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@@ -201,6 +201,10 @@
         'https://www.youtube.com/s/player/2f1832d2/player_ias.vflset/en_US/base.js',
         'YWt1qdbe8SAfkoPHW5d', 'RrRjWQOJmBiP',
     ),
+    (
+        'https://www.youtube.com/s/player/9c6dfc4a/player_ias.vflset/en_US/base.js',
+        'jbu7ylIosQHyJyJV', 'uwI0ESiynAmhNg',
+    ),
 ]