Repository: yt-dlp/yt-dlp

Issue:
Bad Guest Token after extended hours of a Discord bot session
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Malaysia

### Provide a description that is worded well enough to be understood

Currently using the ``yt-dlp`` package in a Discord bot, and the bot is being hosted on PythonAnywhere.

Using this video for the testing: https://twitter.com/jongheon_iu0516/status/1592310896399122434?s=46&t=3tmbRIfdrDrLg0MlN48dEg.

The video would download just fine in the first few hours, after that (usually around the 4 hours marks) it will be permanently returning ``yt_dlp.utils.DownloadError: ERROR: [twitter] 1592310896399122434: Error(s) while querying api: Bad guest token.``

Back when I was using an older version of the ``yt-dlp`` package (can't remember which one exactly), the problem was never there. It was only after I updated to the latest for the multi-video support that it started occurring. Manually restarting the bot will fix it for another few hours until the same issue shows up again.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
Nov 15 10:54:47 [twitter] 1592310896399122434: Downloading JSON metadata
Nov 15 10:54:47 ERROR: [twitter] 1592310896399122434: Error(s) while querying api: Bad guest token.
Nov 15 10:54:47 E 2022-11-15 10:54:47,892 hikari.event_manager: an exception occurred handling an event (InteractionCreateEvent)
Nov 15 10:54:47 Traceback (most recent call last):
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 674, in extract
Nov 15 10:54:47     ie_result = self._real_extract(url)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/extractor/twitter.py", line 829, in _real_extract
Nov 15 10:54:47     status = self._call_api(f'statuses/show/{twid}.json', twid, {
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/extractor/twitter.py", line 146, in _call_api
Nov 15 10:54:47     raise ExtractorError(f'Error(s) while querying api: {error_message}', expected=True)
Nov 15 10:54:47 yt_dlp.utils.ExtractorError: Error(s) while querying api: Bad guest token.
Nov 15 10:54:47 During handling of the above exception, another exception occurred:
Nov 15 10:54:47 Traceback (most recent call last):
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1485, in wrapper
Nov 15 10:54:47     return func(self, *args, **kwargs)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1561, in __extract_info
Nov 15 10:54:47     ie_result = ie.extract(url)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/extractor/common.py", line 700, in extract
Nov 15 10:54:47     raise type(e)(e.orig_msg, **kwargs)
Nov 15 10:54:47 yt_dlp.utils.ExtractorError: [twitter] 1592310896399122434: Error(s) while querying api: Bad guest token.
Nov 15 10:54:47 During handling of the above exception, another exception occurred:
Nov 15 10:54:47 Traceback (most recent call last):
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/app.py", line 1093, in invoke_application_command
Nov 15 10:54:47     await context.invoke()
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/context/base.py", line 276, in invoke
Nov 15 10:54:47     await self.command.invoke(self)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/commands/base.py", line 419, in invoke
Nov 15 10:54:47     await self(context)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/commands/base.py", line 357, in __call__
Nov 15 10:54:47     return await self.callback(context)
Nov 15 10:54:47   File "/home/kaeeeeede/TwitterMediaBot/main.py", line 61, in linkMedia
Nov 15 10:54:47     raise e
Nov 15 10:54:47   File "/home/kaeeeeede/TwitterMediaBot/main.py", line 35, in linkMedia
Nov 15 10:54:47     paths = download_manager.downloadMedia(ctx.options.address)
Nov 15 10:54:47   File "/home/kaeeeeede/TwitterMediaBot/download_manager.py", line 31, in downloadMedia
Nov 15 10:54:47     info = ydl.extract_info(url, download=False)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1474, in extract_info
Nov 15 10:54:47     return self.__extract_info(url, self.get_info_extractor(key), download, extra_info, process)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1503, in wrapper
Nov 15 10:54:47     self.report_error(str(e), e.format_traceback())
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 1004, in report_error
Nov 15 10:54:47     self.trouble(f'{self._format_err("ERROR:", self.Styles.ERROR)} {message}', *args, **kwargs)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/yt_dlp/YoutubeDL.py", line 944, in trouble
Nov 15 10:54:47     raise DownloadError(message, exc_info)
Nov 15 10:54:47 yt_dlp.utils.DownloadError: ERROR: [twitter] 1592310896399122434: Error(s) while querying api: Bad guest token.
Nov 15 10:54:47 The above exception was the direct cause of the following exception:
Nov 15 10:54:47 Traceback (most recent call last):
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/app.py", line 1134, in handle_interaction_create_for_application_commands
Nov 15 10:54:47     await self.invoke_application_command(context)
Nov 15 10:54:47   File "/home/kaeeeeede/.local/lib/python3.8/site-packages/lightbulb/app.py", line 1111, in invoke_application_command
Nov 15 10:54:47     raise new_exc
Nov 15 10:54:47 lightbulb.errors.CommandInvocationError: An error occurred during command 'linkmedia' invocation
```



Discussion:

Resolving patch:
diff --git a/yt_dlp/extractor/twitter.py b/yt_dlp/extractor/twitter.py
--- a/yt_dlp/extractor/twitter.py
+++ b/yt_dlp/extractor/twitter.py
@@ -107,46 +107,54 @@ def _call_api(self, path, video_id, query={}, graphql=False):
                 'x-twitter-active-user': 'yes',
             })
 
-        result, last_error = None, None
+        last_error = None
         for bearer_token in self._TOKENS:
-            headers['Authorization'] = f'Bearer {bearer_token}'
+            for first_attempt in (True, False):
+                headers['Authorization'] = f'Bearer {bearer_token}'
 
-            if not self.is_logged_in:
-                if not self._TOKENS[bearer_token]:
-                    headers.pop('x-guest-token', None)
-                    guest_token_response = self._download_json(
-                        self._API_BASE + 'guest/activate.json', video_id,
-                        'Downloading guest token', data=b'', headers=headers)
-
-                    self._TOKENS[bearer_token] = guest_token_response.get('guest_token')
+                if not self.is_logged_in:
                     if not self._TOKENS[bearer_token]:
-                        raise ExtractorError('Could not retrieve guest token')
-                headers['x-guest-token'] = self._TOKENS[bearer_token]
-
-            try:
-                allowed_status = {400, 403, 404} if graphql else {403}
-                result = self._download_json(
-                    (self._GRAPHQL_API_BASE if graphql else self._API_BASE) + path,
-                    video_id, headers=headers, query=query, expected_status=allowed_status)
-                break
-
-            except ExtractorError as e:
-                if last_error:
-                    raise last_error
-                elif not isinstance(e.cause, urllib.error.HTTPError) or e.cause.code != 404:
-                    raise
-                last_error = e
-                self.report_warning(
-                    'Twitter API gave 404 response, retrying with deprecated token. '
-                    'Only one media item can be extracted')
-
-        if result.get('errors'):
-            error_message = ', '.join(set(traverse_obj(
-                result, ('errors', ..., 'message'), expected_type=str))) or 'Unknown error'
-            raise ExtractorError(f'Error(s) while querying api: {error_message}', expected=True)
-
-        assert result is not None
-        return result
+                        headers.pop('x-guest-token', None)
+                        guest_token_response = self._download_json(
+                            self._API_BASE + 'guest/activate.json', video_id,
+                            'Downloading guest token', data=b'', headers=headers)
+
+                        self._TOKENS[bearer_token] = guest_token_response.get('guest_token')
+                        if not self._TOKENS[bearer_token]:
+                            raise ExtractorError('Could not retrieve guest token')
+
+                    headers['x-guest-token'] = self._TOKENS[bearer_token]
+
+                try:
+                    allowed_status = {400, 403, 404} if graphql else {403}
+                    result = self._download_json(
+                        (self._GRAPHQL_API_BASE if graphql else self._API_BASE) + path,
+                        video_id, headers=headers, query=query, expected_status=allowed_status)
+
+                except ExtractorError as e:
+                    if last_error:
+                        raise last_error
+
+                    if not isinstance(e.cause, urllib.error.HTTPError) or e.cause.code != 404:
+                        raise
+
+                    last_error = e
+                    self.report_warning(
+                        'Twitter API gave 404 response, retrying with deprecated auth token. '
+                        'Only one media item can be extracted')
+                    break  # continue outer loop with next bearer_token
+
+                if result.get('errors'):
+                    errors = traverse_obj(result, ('errors', ..., 'message'), expected_type=str)
+                    if first_attempt and any('bad guest token' in error.lower() for error in errors):
+                        self.to_screen('Guest token has expired. Refreshing guest token')
+                        self._TOKENS[bearer_token] = None
+                        continue
+
+                    error_message = ', '.join(set(errors)) or 'Unknown error'
+                    raise ExtractorError(f'Error(s) while querying API: {error_message}', expected=True)
+
+                return result
 
     def _build_graphql_query(self, media_id):
         raise NotImplementedError('Method must be implemented to support GraphQL')
@@ -328,7 +336,7 @@ class TwitterIE(TwitterBaseIE):
             'id': '665052190608723968',
             'display_id': '665052190608723968',
             'ext': 'mp4',
-            'title': 'md5:3f57ab5d35116537a2ae7345cd0060d8',
+            'title': 'md5:55fef1d5b811944f1550e91b44abb82e',
             'description': 'A new beginning is coming December 18. Watch the official 60 second #TV spot for #StarWars: #TheForceAwakens. https://t.co/OkSqT2fjWJ',
             'uploader_id': 'starwars',
             'uploader': r're:Star Wars.*',
@@ -364,6 +372,7 @@ class TwitterIE(TwitterBaseIE):
             # Test case of TwitterCardIE
             'skip_download': True,
         },
+        'skip': 'Dead external link',
     }, {
         'url': 'https://twitter.com/jaydingeer/status/700207533655363584',
         'info_dict': {
@@ -568,10 +577,10 @@ class TwitterIE(TwitterBaseIE):
             'id': '1577855447914409984',
             'display_id': '1577855540407197696',
             'ext': 'mp4',
-            'title': 'oshtru \U0001faac\U0001f47d - gm \u2728\ufe0f now I can post image and video. nice update.',
-            'description': 'gm \u2728\ufe0f now I can post image and video. nice update. https://t.co/cG7XgiINOm',
+            'title': 'md5:9d198efb93557b8f8d5b78c480407214',
+            'description': 'md5:b9c3699335447391d11753ab21c70a74',
             'upload_date': '20221006',
-            'uploader': 'oshtru \U0001faac\U0001f47d',
+            'uploader': 'oshtru',
             'uploader_id': 'oshtru',
             'uploader_url': 'https://twitter.com/oshtru',
             'thumbnail': r're:^https?://.*\.jpg',
@@ -1096,7 +1105,6 @@ def _real_extract(self, url):
 class TwitterSpacesIE(TwitterBaseIE):
     IE_NAME = 'twitter:spaces'
     _VALID_URL = TwitterBaseIE._BASE_REGEX + r'i/spaces/(?P<id>[0-9a-zA-Z]{13})'
-    _TWITTER_GRAPHQL = 'https://twitter.com/i/api/graphql/HPEisOmj1epUNLCWTYhUWw/'
 
     _TESTS = [{
         'url': 'https://twitter.com/i/spaces/1RDxlgyvNXzJL',


Test patch: