Repository: ytdl-org/youtube-dl

Issue:
Cannot download private YT videos on brand new accounts or accounts with 2FA
Logging in does not allow you to download private Youtube videos as an account with 2-Factor Auth enabled.

Attempt 1. Created a new google account, shared the video with it.

```
kane@newlaptop:~/Videos$ youtube-dl -u riking.ytdl.issue.report@gmail.com -p 2U4XTF24E4 http://youtu.be/HUTXjZtt4Io
[youtube] Setting language
[youtube] Downloading login page
[youtube] Logging in
[youtube] Confirming age
[youtube] HUTXjZtt4Io: Downloading webpage
[youtube] HUTXjZtt4Io: Downloading video info webpage
ERROR: HUTXjZtt4Io: YouTube said: This video is private.
```

Note that this by itself is problematic - it says that we successfully signed in, but the video is private?! A better error message would be nice.

Attempt 2. [Enabled "access for less secure apps"](https://www.google.com/settings/u/0/security/lesssecureapps).

```
kane@newlaptop:~/Videos$ youtube-dl -u riking.ytdl.issue.report@gmail.com -p 2U4XTF24E4 http://youtu.be/HUTXjZtt4Io
[youtube] Setting language
[youtube] Downloading login page
[youtube] Logging in
[youtube] Confirming age
[youtube] HUTXjZtt4Io: Downloading webpage
[youtube] HUTXjZtt4Io: Downloading video info webpage
[youtube] HUTXjZtt4Io: Extracting video information
[download] Destination: youtube dl test video-HUTXjZtt4Io.mp4
[download] 100% of 46.16KiB in 00:00
```

This seems to work.

Attempt 3. Enabled 2FA.
Google Authenticator shared secret: `xha7 ryxa d6ah dk6t wxw5 5mq3 cxxj 2f25`

```
kane@newlaptop:~/Videos$ youtube-dl -u riking.ytdl.issue.report@gmail.com -p 2U4XTF24E4 http://youtu.be/HUTXjZtt4Io
[youtube] Setting language
[youtube] Downloading login page
[youtube] Logging in
[youtube] Confirming age
[youtube] HUTXjZtt4Io: Downloading webpage
[youtube] HUTXjZtt4Io: Downloading video info webpage
ERROR: HUTXjZtt4Io: YouTube said: This video is private.
```

Attempt 4. Using application-specific password, generated as "Youtube on my Windows Computer"
`plwwtjwpaedjxurz`

```
kane@newlaptop:~/Videos$ youtube-dl -u riking.ytdl.issue.report@gmail.com -p plwwtjwpaedjxurz http://youtu.be/HUTXjZtt4Io
[youtube] Setting language
[youtube] Downloading login page
[youtube] Logging in
WARNING: unable to log in: bad username or password
[youtube] HUTXjZtt4Io: Downloading webpage
[youtube] HUTXjZtt4Io: Downloading video info webpage
ERROR: HUTXjZtt4Io: YouTube said: This video is private.
```

At this point, "Access for less secure apps" is not available.

---

Acceptable solutions include (not limited to) having youtube-dl ask for a 2FA code or an application-specific password. However, it seems they clearly expect a different login path for the app-specific passwords.

Also an issue is that it doesn't work for brand-new accounts, either.

---

I'm not quite sure what Google is expecting the app to do, but it's clearly something different.

Please refrain from using this test account for anything other than trying to download the private video named above. Also please refrain from calling my phone number.

I can add more accounts to the permitted list on request. The video is just a short clip of a text editor containing the account's password.



Discussion:
Was browsing around developers.google.com and found the deprecated ClientLogin interface - I think this might be what takes the app passwords. https://developers.google.com/accounts/docs/AuthForInstalledApps


Resolving patch:
diff --git a/youtube_dl/__init__.py b/youtube_dl/__init__.py
--- a/youtube_dl/__init__.py
+++ b/youtube_dl/__init__.py
@@ -314,6 +314,8 @@ def _hide_login_info(opts):
             dest='username', metavar='USERNAME', help='account username')
     authentication.add_option('-p', '--password',
             dest='password', metavar='PASSWORD', help='account password')
+    authentication.add_option('-2', '--twofactor',
+            dest='twofactor', metavar='TWOFACTOR', help='two-factor auth code')
     authentication.add_option('-n', '--netrc',
             action='store_true', dest='usenetrc', help='use .netrc authentication data', default=False)
     authentication.add_option('--video-password',
@@ -748,6 +750,7 @@ def _real_main(argv=None):
         'usenetrc': opts.usenetrc,
         'username': opts.username,
         'password': opts.password,
+        'twofactor': opts.twofactor,
         'videopassword': opts.videopassword,
         'quiet': (opts.quiet or any_printing),
         'no_warnings': opts.no_warnings,
diff --git a/youtube_dl/extractor/common.py b/youtube_dl/extractor/common.py
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -434,6 +434,22 @@ def _get_login_info(self):
         
         return (username, password)
 
+    def _get_tfa_info(self):
+        """
+        Get the two-factor authentication info
+        TODO - asking the user will be required for sms/phone verify
+        currently just uses the command line option
+        If there's no info available, return None
+        """
+        if self._downloader is None:
+            return None
+        downloader_params = self._downloader.params
+
+        if downloader_params.get('twofactor', None) is not None:
+            return downloader_params['twofactor']
+
+        return None
+
     # Helper functions for extracting OpenGraph info
     @staticmethod
     def _og_regexes(prop):
diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -37,6 +37,7 @@
 class YoutubeBaseInfoExtractor(InfoExtractor):
     """Provide base functions for Youtube extractors"""
     _LOGIN_URL = 'https://accounts.google.com/ServiceLogin'
+    _TWOFACTOR_URL = 'https://accounts.google.com/SecondFactor'
     _LANG_URL = r'https://www.youtube.com/?hl=en&persist_hl=1&gl=US&persist_gl=1&opt_out_ackd=1'
     _AGE_URL = 'https://www.youtube.com/verify_age?next_url=/&gl=US&hl=en'
     _NETRC_MACHINE = 'youtube'
@@ -50,12 +51,19 @@ def _set_language(self):
             fatal=False))
 
     def _login(self):
+        """
+        Attempt to log in to YouTube.
+        True is returned if successful or skipped.
+        False is returned if login failed.
+
+        If _LOGIN_REQUIRED is set and no authentication was provided, an error is raised.
+        """
         (username, password) = self._get_login_info()
         # No authentication to be performed
         if username is None:
             if self._LOGIN_REQUIRED:
                 raise ExtractorError(u'No login info available, needed for using %s.' % self.IE_NAME, expected=True)
-            return False
+            return True
 
         login_page = self._download_webpage(
             self._LOGIN_URL, None,
@@ -73,6 +81,7 @@ def _login(self):
                 u'Email': username,
                 u'GALX': galx,
                 u'Passwd': password,
+
                 u'PersistentCookie': u'yes',
                 u'_utf8': u'霱',
                 u'bgresponse': u'js_disabled',
@@ -88,6 +97,7 @@ def _login(self):
                 u'uilel': u'3',
                 u'hl': u'en_US',
         }
+
         # Convert to UTF-8 *before* urlencode because Python 2.x's urlencode
         # chokes on unicode
         login_form = dict((k.encode('utf-8'), v.encode('utf-8')) for k,v in login_form_strs.items())
@@ -99,6 +109,68 @@ def _login(self):
             note=u'Logging in', errnote=u'unable to log in', fatal=False)
         if login_results is False:
             return False
+
+        if re.search(r'id="errormsg_0_Passwd"', login_results) is not None:
+            raise ExtractorError(u'Please use your account password and a two-factor code instead of an application-specific password.', expected=True)
+
+        # Two-Factor
+        # TODO add SMS and phone call support - these require making a request and then prompting the user
+
+        if re.search(r'(?i)<form[^>]* id="gaia_secondfactorform"', login_results) is not None:
+            tfa_code = self._get_tfa_info()
+
+            if tfa_code is None:
+                self._downloader.report_warning(u'Two-factor authentication required. Provide it with --twofactor <code>')
+                self._downloader.report_warning(u'(Note that only TOTP (Google Authenticator App) codes work at this time.)')
+                return False
+
+            # Unlike the first login form, secTok and timeStmp are both required for the TFA form
+
+            match = re.search(r'id="secTok"\n\s+value=\'(.+)\'/>', login_results, re.M | re.U)
+            if match is None:
+                self._downloader.report_warning(u'Failed to get secTok - did the page structure change?')
+            secTok = match.group(1)
+            match = re.search(r'id="timeStmp"\n\s+value=\'(.+)\'/>', login_results, re.M | re.U)
+            if match is None:
+                self._downloader.report_warning(u'Failed to get timeStmp - did the page structure change?')
+            timeStmp = match.group(1)
+
+            tfa_form_strs = {
+                u'continue': u'https://www.youtube.com/signin?action_handle_signin=true&feature=sign_in_button&hl=en_US&nomobiletemp=1',
+                u'smsToken': u'',
+                u'smsUserPin': tfa_code,
+                u'smsVerifyPin': u'Verify',
+
+                u'PersistentCookie': u'yes',
+                u'checkConnection': u'',
+                u'checkedDomains': u'youtube',
+                u'pstMsg': u'1',
+                u'secTok': secTok,
+                u'timeStmp': timeStmp,
+                u'service': u'youtube',
+                u'hl': u'en_US',
+            }
+            tfa_form = dict((k.encode('utf-8'), v.encode('utf-8')) for k,v in tfa_form_strs.items())
+            tfa_data = compat_urllib_parse.urlencode(tfa_form).encode('ascii')
+
+            tfa_req = compat_urllib_request.Request(self._TWOFACTOR_URL, tfa_data)
+            tfa_results = self._download_webpage(
+                tfa_req, None,
+                note=u'Submitting TFA code', errnote=u'unable to submit tfa', fatal=False)
+
+            if tfa_results is False:
+                return False
+
+            if re.search(r'(?i)<form[^>]* id="gaia_secondfactorform"', tfa_results) is not None:
+                self._downloader.report_warning(u'Two-factor code expired. Please try again, or use a one-use backup code instead.')
+                return False
+            if re.search(r'(?i)<form[^>]* id="gaia_loginform"', tfa_results) is not None:
+                self._downloader.report_warning(u'unable to log in - did the page structure change?')
+                return False
+            if re.search(r'smsauth-interstitial-reviewsettings', tfa_results) is not None:
+                self._downloader.report_warning(u'Your Google account has a security notice. Please log in on your web browser, resolve the notice, and try again.')
+                return False
+
         if re.search(r'(?i)<form[^>]* id="gaia_loginform"', login_results) is not None:
             self._downloader.report_warning(u'unable to log in: bad username or password')
             return False


Test patch: