Repository: yt-dlp/yt-dlp

Issue:
Twitter login support
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm requesting a feature unrelated to a specific site
- [X] I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

twitter currently does not have specific support for credential authentication.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://twitter.com/ShitpostGate/status/1651904099749699590']
[debug] User config "C:\Users\japri\AppData\Roaming\yt-dlp\config.txt": ['--downloader', 'aria2c']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2023.04.29.181950 [4d9280c9c] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg n6.0-7-gadb4688bfb-20230316 (setts), ffprobe n6.0-7-gadb4688bfb-20230316
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.2
[debug] Proxy map: {}
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Available version: nightly@2023.04.29.181950, Current version: nightly@2023.04.29.181950
Current Build Hash: 5f6363a11a63e1ed08553785fa7704f5207bfb1be4639a80f05f67cc4f4e4976
yt-dlp is up to date (nightly@2023.04.29.181950)
WARNING: [twitter] Login with password is not supported for this website. Use --cookies-from-browser or --cookies for the authentication. See  https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp  for how to manually pass cookies
[twitter] Extracting URL: https://twitter.com/ShitpostGate/status/1651904099749699590
[twitter] 1651904099749699590: Downloading guest token
[twitter] 1651904099749699590: Downloading JSON metadata
[debug] [twitter] Extracting from video info: 1651747074675703809
[twitter] 1651904099749699590: Downloading m3u8 information
[debug] Sort order given by extractor: res, br, size, proto
[debug] Formats sorted by: hasvid, ie_pref, res, tbr, vbr, abr, filesize, fs_approx, proto, lang, quality, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, asr, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 1651747074675703809: Downloading 1 format(s): http-950
[debug] Invoking aria2c downloader on "https://video.twimg.com/ext_tw_video/1651747074675703809/pu/vid/480x600/-yNzlUc4KXyR40kE.mp4?tag=12"
[download] ShitpostGateway - https：⧸⧸t.co⧸uvHY9r5noW [1651747074675703809].mp4 has already been downloaded
[download] 100% of  226.24KiB
```



Discussion:
This one's a lot harder to implement than Reddit's login. The website uses a multi-step login flow, and the 2nd step requires POST JSON data with multiple parameters (I am assuming are generated by the javascript) looking like this:
```json
"response": {
  "rf": {
    "e5ba13d8c9a2c1de100f11e8dc7b3d08a5a77ac8f485b1a3fdbb9b3e969b2d5e": -62,
    "ca7b4cc404b09126894cd257f10ca0be610a769d23c568a3de19dad7163836c3": -10,
    "a1a33537de11bf80c8e2a77cb08e71de90e694ac9d14e2681182527025d88f56": -11,
    "ad66f8a765857badba52d26ac4f38c637fd170b1e1d29e6af022697ab265790d": 8
  },
  "s": "MlsKpre5bmN65UMb5tB078IAZ6PN8eOp_MgnTy4mnQUOZWwMw-4qQkus0NGgPmLj8PHxasopFX2_2ndeqdPliz3eODvKdZXaqUanwFHOwIdoZKus6WtzoN9fPY5tP7rGkU7M-xuUmc9Wieof0a_b1UyYrvnGkPJbv56PtjZHlV8gtnqonkbw68gvRdH3DkJaMtTo4FZL4ydpVAPk1cSQQC1PoNDrpPJDjGq3JVhfWKZe7yukyGwmXBcFo-XT_WtdfafeRoIdjOH15YOjkp9We9Sq9f0aBQ2Xl0LJepJAvg9lXsjNT7tfEPztwVFnoNXkPJCw3ADCRT6hgfM1ij6fNi9ntQawUt5K"
}
```
Since this would require some experimentation and a lot could go wrong, I don't want to risk my personal acct. If I had a burner/throwaway account to play around with, I might be up for digging deeper into it. But Twitter requires phone # for new accounts now
I have an old account that i don't longer use, can i pass you the credentials?
Sure, DM on discord: bashonly#4457

Resolving patch:
diff --git a/yt_dlp/extractor/twitter.py b/yt_dlp/extractor/twitter.py
--- a/yt_dlp/extractor/twitter.py
+++ b/yt_dlp/extractor/twitter.py
@@ -3,7 +3,6 @@
 
 from .common import InfoExtractor
 from .periscope import PeriscopeBaseIE, PeriscopeIE
-from ..compat import functools  # isort: split
 from ..compat import (
     compat_parse_qs,
     compat_urllib_parse_unquote,
@@ -30,11 +29,67 @@
 
 
 class TwitterBaseIE(InfoExtractor):
+    _NETRC_MACHINE = 'twitter'
     _API_BASE = 'https://api.twitter.com/1.1/'
     _GRAPHQL_API_BASE = 'https://twitter.com/i/api/graphql/'
     _BASE_REGEX = r'https?://(?:(?:www|m(?:obile)?)\.)?(?:twitter\.com|twitter3e4tixl4xyajtrzo62zg5vztmjuricljdp2c5kshju4avyoid\.onion)/'
     _AUTH = {'Authorization': 'Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA'}
     _guest_token = None
+    _flow_token = None
+
+    _LOGIN_INIT_DATA = json.dumps({
+        'input_flow_data': {
+            'flow_context': {
+                'debug_overrides': {},
+                'start_location': {
+                    'location': 'unknown'
+                }
+            }
+        },
+        'subtask_versions': {
+            'action_list': 2,
+            'alert_dialog': 1,
+            'app_download_cta': 1,
+            'check_logged_in_account': 1,
+            'choice_selection': 3,
+            'contacts_live_sync_permission_prompt': 0,
+            'cta': 7,
+            'email_verification': 2,
+            'end_flow': 1,
+            'enter_date': 1,
+            'enter_email': 2,
+            'enter_password': 5,
+            'enter_phone': 2,
+            'enter_recaptcha': 1,
+            'enter_text': 5,
+            'enter_username': 2,
+            'generic_urt': 3,
+            'in_app_notification': 1,
+            'interest_picker': 3,
+            'js_instrumentation': 1,
+            'menu_dialog': 1,
+            'notifications_permission_prompt': 2,
+            'open_account': 2,
+            'open_home_timeline': 1,
+            'open_link': 1,
+            'phone_verification': 4,
+            'privacy_options': 1,
+            'security_key': 3,
+            'select_avatar': 4,
+            'select_banner': 2,
+            'settings_list': 7,
+            'show_code': 1,
+            'sign_up': 2,
+            'sign_up_review': 4,
+            'tweet_selection_urt': 1,
+            'update_users': 1,
+            'upload_media': 1,
+            'user_recommendations_list': 4,
+            'user_recommendations_urt': 1,
+            'wait_spinner': 3,
+            'web_modal': 1
+        }
+    }, separators=(',', ':')).encode()
 
     def _extract_variant_formats(self, variant, video_id):
         variant_url = variant.get('url')
@@ -86,18 +141,151 @@ def _search_dimensions_in_video_url(a_format, video_url):
                 'height': int(m.group('height')),
             })
 
-    @functools.cached_property
+    @property
     def is_logged_in(self):
         return bool(self._get_cookies(self._API_BASE).get('auth_token'))
 
-    def _call_api(self, path, video_id, query={}, graphql=False):
-        cookies = self._get_cookies(self._API_BASE)
+    def _fetch_guest_token(self, headers, display_id):
+        headers.pop('x-guest-token', None)
+        self._guest_token = traverse_obj(self._download_json(
+            f'{self._API_BASE}guest/activate.json', display_id,
+            'Downloading guest token', data=b'', headers=headers), 'guest_token')
+        if not self._guest_token:
+            raise ExtractorError('Could not retrieve guest token')
+
+    def _set_base_headers(self):
         headers = self._AUTH.copy()
+        csrf_token = try_call(lambda: self._get_cookies(self._API_BASE)['ct0'].value)
+        if csrf_token:
+            headers['x-csrf-token'] = csrf_token
+        return headers
+
+    def _call_login_api(self, note, headers, query={}, data=None):
+        response = self._download_json(
+            f'{self._API_BASE}onboarding/task.json', None, note,
+            headers=headers, query=query, data=data, expected_status=400)
+        error = traverse_obj(response, ('errors', 0, 'message', {str}))
+        if error:
+            raise ExtractorError(f'Login failed, Twitter API says: {error}', expected=True)
+        elif traverse_obj(response, 'status') != 'success':
+            raise ExtractorError('Login was unsuccessful')
+
+        subtask = traverse_obj(
+            response, ('subtasks', ..., 'subtask_id', {str}), get_all=False)
+        if not subtask:
+            raise ExtractorError('Twitter API did not return next login subtask')
+
+        self._flow_token = response['flow_token']
+
+        return subtask
+
+    def _perform_login(self, username, password):
+        if self.is_logged_in:
+            return
+
+        self._request_webpage('https://twitter.com/', None, 'Requesting cookies')
+        headers = self._set_base_headers()
+        self._fetch_guest_token(headers, None)
+        headers.update({
+            'content-type': 'application/json',
+            'x-guest-token': self._guest_token,
+            'x-twitter-client-language': 'en',
+            'x-twitter-active-user': 'yes',
+            'Referer': 'https://twitter.com/',
+            'Origin': 'https://twitter.com',
+        })
 
-        csrf_cookie = cookies.get('ct0')
-        if csrf_cookie:
-            headers['x-csrf-token'] = csrf_cookie.value
+        def build_login_json(*subtask_inputs):
+            return json.dumps({
+                'flow_token': self._flow_token,
+                'subtask_inputs': subtask_inputs
+            }, separators=(',', ':')).encode()
 
+        def input_dict(subtask_id, text):
+            return {
+                'subtask_id': subtask_id,
+                'enter_text': {
+                    'text': text,
+                    'link': 'next_link'
+                }
+            }
+
+        next_subtask = self._call_login_api(
+            'Downloading flow token', headers, query={'flow_name': 'login'}, data=self._LOGIN_INIT_DATA)
+
+        while not self.is_logged_in:
+            if next_subtask == 'LoginJsInstrumentationSubtask':
+                next_subtask = self._call_login_api(
+                    'Submitting JS instrumentation response', headers, data=build_login_json({
+                        'subtask_id': next_subtask,
+                        'js_instrumentation': {
+                            'response': '{}',
+                            'link': 'next_link'
+                        }
+                    }))
+
+            elif next_subtask == 'LoginEnterUserIdentifierSSO':
+                next_subtask = self._call_login_api(
+                    'Submitting username', headers, data=build_login_json({
+                        'subtask_id': next_subtask,
+                        'settings_list': {
+                            'setting_responses': [{
+                                'key': 'user_identifier',
+                                'response_data': {
+                                    'text_data': {
+                                        'result': username
+                                    }
+                                }
+                            }],
+                            'link': 'next_link'
+                        }
+                    }))
+
+            elif next_subtask == 'LoginEnterAlternateIdentifierSubtask':
+                next_subtask = self._call_login_api(
+                    'Submitting alternate identifier', headers,
+                    data=build_login_json(input_dict(next_subtask, self._get_tfa_info(
+                        'one of username, phone number or email that was not used as --username'))))
+
+            elif next_subtask == 'LoginEnterPassword':
+                next_subtask = self._call_login_api(
+                    'Submitting password', headers, data=build_login_json({
+                        'subtask_id': next_subtask,
+                        'enter_password': {
+                            'password': password,
+                            'link': 'next_link'
+                        }
+                    }))
+
+            elif next_subtask == 'AccountDuplicationCheck':
+                next_subtask = self._call_login_api(
+                    'Submitting account duplication check', headers, data=build_login_json({
+                        'subtask_id': next_subtask,
+                        'check_logged_in_account': {
+                            'link': 'AccountDuplicationCheck_false'
+                        }
+                    }))
+
+            elif next_subtask == 'LoginTwoFactorAuthChallenge':
+                next_subtask = self._call_login_api(
+                    'Submitting 2FA token', headers, data=build_login_json(input_dict(
+                        next_subtask, self._get_tfa_info('two-factor authentication token'))))
+
+            elif next_subtask == 'LoginAcid':
+                next_subtask = self._call_login_api(
+                    'Submitting confirmation code', headers, data=build_login_json(input_dict(
+                        next_subtask, self._get_tfa_info('confirmation code sent to your email or phone'))))
+
+            elif next_subtask == 'LoginSuccessSubtask':
+                raise ExtractorError('Twitter API did not grant auth token cookie')
+
+            else:
+                raise ExtractorError(f'Unrecognized subtask ID "{next_subtask}"')
+
+        self.report_login()
+
+    def _call_api(self, path, video_id, query={}, graphql=False):
+        headers = self._set_base_headers()
         if self.is_logged_in:
             headers.update({
                 'x-twitter-auth-type': 'OAuth2Session',
@@ -106,15 +294,10 @@ def _call_api(self, path, video_id, query={}, graphql=False):
             })
 
         for first_attempt in (True, False):
-            if not self.is_logged_in and not self._guest_token:
-                headers.pop('x-guest-token', None)
-                self._guest_token = traverse_obj(self._download_json(
-                    f'{self._API_BASE}guest/activate.json', video_id,
-                    'Downloading guest token', data=b'', headers=headers), 'guest_token')
-            if self._guest_token:
+            if not self.is_logged_in:
+                if not self._guest_token:
+                    self._fetch_guest_token(headers, video_id)
                 headers['x-guest-token'] = self._guest_token
-            elif not self.is_logged_in:
-                raise ExtractorError('Could not retrieve guest token')
 
             allowed_status = {400, 401, 403, 404} if graphql else {403}
             result = self._download_json(


Test patch: