Repository: yt-dlp/yt-dlp

Issue:
Add support for Google Drive folders
### Checklist

- [X] I'm reporting a site feature request
- [X] I've verified that I'm running yt-dlp version **2022.04.08** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

United States

### Example URLs

https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf
https://drive.google.com/drive/folders/0Bx8RePQCi0zsdTJHekFzMVB5Zzg?resourcekey=0-6JmqCQiqGowKBuLOMg1reQ

### Description

For some odd reason, yt-dlp doesn't support Google Drive folders. I did some snooping around and I think I've figured out how one would add functionality. Before I get into that, because a playlist--in this case a Google Drive folder--can contain other playlists, would yt-dlp recursively retrieve all videos, or would it stop at the first layer?

Visiting https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf:

* It looks like only the first 50 items in a folder are hydrated in the HTML. We'll need to use Drive's REST API to get the rest (heh) of them.
* If we do want to go down the HTML route though, there's a few things we can gather:
  * Every item is a `div[data-id]`, with the aforementioned attribute allowing one to retrieve the item with `https://drive.google.com/file/d/$data-id/view`
  * Video thumbnails can be retrieved with `https://lh3.googleusercontent.com/u/0/d/$data-id=w200-h190-p-k-nu-iv1`. That last stuff other than the `=` and at least one `-` at the end seems to be unnecessary; it changes the resolution of the thumbnail.
  * The title of the video can also be grabbed with `div[data-id] div[role=gridcell]` or a variety of other ways.
  * Videos can be filtered for with `div[data-id] img[alt=Video]` and audio with `div[data-id] img[alt=Audio]`.
* The REST API, as most Google ones are, is quite strange.
  * In order to request stuff from the API, you need a key, which can be found in one of Google's trademark JSON array blobs in a `body script[nonce]` tag.
  ```html
  <script nonce="wnAizpurtlm58kq5DMBh/g">__initData = [[null, [null, 0, null, 0], /* data expunged */ "AIzaSyC1qbk75NzWBvSaDh6KnsjjA9pIrP4lYIE", /* <- the key | data expunged */</script>
  ```
  * Once we do that, we can then make a POST request to `https://clients6.google.com/batch/drive/v2beta?$ct=multipart/mixed; boundary="=====7dp34d9nia8h====="&key=$key` to get information about the folder. Why is Google using [mixed requests](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html) in such a weird way? I dunno. Also you'll want your `Content-Type` header to be `text/plain`, which makes even less sense. Your body is going to look like this:
  ```
  --=====7dp34d9nia8h=====
  content-type: application/http
  content-transfer-encoding: binary

  GET /drive/v2beta/files/$folder-id?openDrive=true&reason=1001&syncType=0&errorRecovery=false&fields=kind%2CmodifiedDate%2CmodifiedByMeDate%2ClastViewedByMeDate%2CfileSize%2Cowners(kind%2CpermissionId%2Cid)%2ClastModifyingUser(kind%2CpermissionId%2Cid)%2ChasThumbnail%2CthumbnailVersion%2Ctitle%2Cid%2CresourceKey%2Cshared%2CsharedWithMeDate%2CuserPermission(role)%2CexplicitlyTrashed%2CmimeType%2CquotaBytesUsed%2Ccopyable%2CfileExtension%2CsharingUser(kind%2CpermissionId%2Cid)%2Cspaces%2Cversion%2CteamDriveId%2ChasAugmentedPermissions%2CcreatedDate%2CtrashingUser(kind%2CpermissionId%2Cid)%2CtrashedDate%2Cparents(id)%2CshortcutDetails(targetId%2CtargetMimeType%2CtargetLookupStatus)%2Ccapabilities(canCopy%2CcanDownload%2CcanEdit%2CcanAddChildren%2CcanDelete%2CcanRemoveChildren%2CcanShare%2CcanTrash%2CcanRename%2CcanReadTeamDrive%2CcanMoveTeamDriveItem)%2Clabels(starred%2Ctrashed%2Crestricted%2Cviewed)&supportsTeamDrives=true&retryCount=0&key=$key HTTP/1.1
  X-Goog-Drive-Client-Version: drive.web-frontend_20220330.00_p1

  --=====7dp34d9nia8h=====--
  ```
  I'm just as flabbergasted as you.
  * Now, I hope that urllib can parse `multipart/mixed` requests, or getting the JSON from the above response is going to be really annoying:
  ```json
  {
  	"kind": "drive#file",
  	"id": "1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf",
  	"thumbnailVersion": "0",
  	"title": "Naruto Kai Dub",
  	"mimeType": "application/vnd.google-apps.folder",
  	"labels": {
  	 "starred": false,
  	 "trashed": false,
  	 "restricted": false,
  	 "viewed": false
  	},
  	"createdDate": "2021-10-18T03:42:39.327Z",
  	"modifiedDate": "2021-10-21T16:43:40.157Z",
  	"version": "20025",
  	"parents": [],
  	"userPermission": {
  	 "role": "reader"
  	},
  	"quotaBytesUsed": "0",
  	"owners": [
  	 {
  	  "kind": "drive#user",
  	  "permissionId": "14091304371559574575",
  	  "id": "113368024712280017234"
  	 }
  	],
  	"lastModifyingUser": {
  	 "kind": "drive#user",
  	 "permissionId": "14091304371559574575",
  	 "id": "113368024712280017234"
  	},
  	"capabilities": {
  	 "canAddChildren": false,
  	 "canCopy": false,
  	 "canDelete": false,
  	 "canDownload": true,
  	 "canEdit": false,
  	 "canRemoveChildren": false,
  	 "canRename": false,
  	 "canShare": false,
  	 "canTrash": false
  	},
  	"copyable": false,
  	"shared": true,
  	"explicitlyTrashed": false,
  	"spaces": [
  	 "drive"
  	],
  	"hasThumbnail": false
  }
  ```
  * Now, let's start paginating over files. We'll want to do the same POST request as before, but our body's going to look a bit different:
  ```
  --=====dd73l7rsvti0=====
  content-type: application/http
  content-transfer-encoding: binary

  GET /drive/v2beta/files?openDrive=false&reason=102&syncType=0&errorRecovery=false&q=trashed%20%3D%20false%20and%20'$folder-id'%20in%20parents&fields=kind%2CnextPageToken%2Citems(kind%2CmodifiedDate%2CmodifiedByMeDate%2ClastViewedByMeDate%2CfileSize%2Cowners(kind%2CpermissionId%2Cid)%2ClastModifyingUser(kind%2CpermissionId%2Cid)%2ChasThumbnail%2CthumbnailVersion%2Ctitle%2Cid%2CresourceKey%2Cshared%2CsharedWithMeDate%2CuserPermission(role)%2CexplicitlyTrashed%2CmimeType%2CquotaBytesUsed%2Ccopyable%2CfileExtension%2CsharingUser(kind%2CpermissionId%2Cid)%2Cspaces%2Cversion%2CteamDriveId%2ChasAugmentedPermissions%2CcreatedDate%2CtrashingUser(kind%2CpermissionId%2Cid)%2CtrashedDate%2Cparents(id)%2CshortcutDetails(targetId%2CtargetMimeType%2CtargetLookupStatus)%2Ccapabilities(canCopy%2CcanDownload%2CcanEdit%2CcanAddChildren%2CcanDelete%2CcanRemoveChildren%2CcanShare%2CcanTrash%2CcanRename%2CcanReadTeamDrive%2CcanMoveTeamDriveItem)%2Clabels(starred%2Ctrashed%2Crestricted%2Cviewed))%2CincompleteSearch&appDataFilter=NO_APP_DATA&spaces=drive&maxResults=50&supportsTeamDrives=true&includeItemsFromAllDrives=true&corpora=default&orderBy=folder%2Ctitle_natural%20asc&retryCount=0&key=$key HTTP/1.1
  X-Goog-Drive-Client-Version: drive.web-frontend_20220330.00_p1

  --=====dd73l7rsvti0=====--
  ```
  * The JSON is going to look like this:
  ```json
  {
   "kind": "drive#fileList",
   "nextPageToken": "~!!~AI9FV7TV3nrJoLhJ0_x84rZR4BshUnZoPrt__8LCNIqRXb5NV0Q8KMeToqsUSkyFQzLvWjm5YjIfLPpxQ8_5fjm54MYbhsXSm8-5pm25AGRpbkpS1u20IFnm3sbdpzcc-iJBzBNG4JPu1H5-4ddl5Oo_jMYsZAZXViSV96dCRZhzaeYK9iOR599s747i14f_6qObMdGTV_NyHEuSlOwzTeg2CpjLMqxWG6tMRx6Z_NeMMiFWo31y3qh-Z1g-HFMUatpSZ2MUviQubhjG5bJ78HC6K7zt6LFmAMhf5N5ouA4xfHXvY1HNFsmW8se5w8-Dp1duxqCktDmo",
   "incompleteSearch": false,
   "items": [
    {
     "kind": "drive#file",
     "id": "1jD9H1n9UdjdAEm4c7gt9VI7VvHOOeh32",
     "thumbnailVersion": "1",
     "title": "Naruto Kai Dub Ep 1.mp4",
     "mimeType": "video/mp4",
     "labels": {
      "starred": false,
      "trashed": false,
      "restricted": false,
      "viewed": false
     },
     "createdDate": "2021-10-18T04:09:42.824Z",
     "modifiedDate": "2021-07-13T04:56:45.000Z",
     "version": "3711",
     "parents": [
      {
       "id": "1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf"
      }
     ],
     "userPermission": {
      "role": "reader"
     },
     "fileExtension": "mp4",
     "fileSize": "1919950685",
     "quotaBytesUsed": "1919950685",
     "owners": [
      {
       "kind": "drive#user",
       "permissionId": "14091304371559574575",
       "id": "113368024712280017234"
      }
     ],
     "lastModifyingUser": {
      "kind": "drive#user",
      "permissionId": "14091304371559574575",
      "id": "113368024712280017234"
     },
     "capabilities": {
      "canAddChildren": false,
      "canCopy": false,
      "canDelete": false,
      "canDownload": true,
      "canEdit": false,
      "canRemoveChildren": false,
      "canRename": false,
      "canShare": false,
      "canTrash": false
     },
     "copyable": false,
     "shared": true,
     "explicitlyTrashed": false,
     "spaces": [
      "drive"
     ],
     "hasThumbnail": true
    },
    /* up to 49 more items */
   ]
  }
  ```
  * We can then paginate using `nextPageToken`:
  ```
  --=====lq1nk53gsoei=====
  content-type: application/http
  content-transfer-encoding: binary
  
  GET /drive/v2beta/files?openDrive=true&reason=102&syncType=0&errorRecovery=false&q=trashed%20%3D%20false%20and%20'$folder-id'%20in%20parents&fields=kind%2CnextPageToken%2Citems(kind%2CmodifiedDate%2CmodifiedByMeDate%2ClastViewedByMeDate%2CfileSize%2Cowners(kind%2CpermissionId%2Cid)%2ClastModifyingUser(kind%2CpermissionId%2Cid)%2ChasThumbnail%2CthumbnailVersion%2Ctitle%2Cid%2CresourceKey%2Cshared%2CsharedWithMeDate%2CuserPermission(role)%2CexplicitlyTrashed%2CmimeType%2CquotaBytesUsed%2Ccopyable%2CfileExtension%2CsharingUser(kind%2CpermissionId%2Cid)%2Cspaces%2Cversion%2CteamDriveId%2ChasAugmentedPermissions%2CcreatedDate%2CtrashingUser(kind%2CpermissionId%2Cid)%2CtrashedDate%2Cparents(id)%2CshortcutDetails(targetId%2CtargetMimeType%2CtargetLookupStatus)%2Ccapabilities(canCopy%2CcanDownload%2CcanEdit%2CcanAddChildren%2CcanDelete%2CcanRemoveChildren%2CcanShare%2CcanTrash%2CcanRename%2CcanReadTeamDrive%2CcanMoveTeamDriveItem)%2Clabels(starred%2Ctrashed%2Crestricted%2Cviewed))%2CincompleteSearch&appDataFilter=NO_APP_DATA&spaces=drive&pageToken=$nextPageToken&maxResults=50&supportsTeamDrives=true&includeItemsFromAllDrives=true&corpora=default&orderBy=folder%2Ctitle_natural%20asc&retryCount=0&key=$key HTTP/1.1
  X-Goog-Drive-Client-Version: drive.web-frontend_20220330.00_p1
  
  --=====lq1nk53gsoei=====--
  ```

This, as with any Google service, is really annoying, but not impossible.

### Verbose log

```shell
[debug] Command-line config: ['-vU', 'https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf']
[debug] User config "/home/kwilliams/.config/yt-dlp/config": ['--netrc', '--sub-lang', 'en,en-US,eng', '--format', '(bestvideo+bestaudio/best)[format_id*=en]/(bestvideo+bestaudio/best)']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.04.08 [7884ade65]
[debug] Python version 3.10.0 (CPython 64bit) - Linux-5.4.0-107-generic-x86_64-with-glibc2.31
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Optional libraries: brotli, certifi, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.04.08, Current version: 2022.04.08
yt-dlp is up to date (2022.04.08)
[debug] [generic] Extracting URL: https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf
[generic] 1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] 1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf: Downloading webpage
[generic] 1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf: Extracting information
[debug] Looking for video embeds
ERROR: Unsupported URL: https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf
Traceback (most recent call last):
  File "/home/kwilliams/.asdf/installs/yt-dlp/2022.4.8/venv/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1408, in wrapper
    return func(self, *args, **kwargs)
  File "/home/kwilliams/.asdf/installs/yt-dlp/2022.4.8/venv/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1478, in __extract_info
    ie_result = ie.extract(url)
  File "/home/kwilliams/.asdf/installs/yt-dlp/2022.4.8/venv/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 641, in extract
    ie_result = self._real_extract(url)
  File "/home/kwilliams/.asdf/installs/yt-dlp/2022.4.8/venv/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 4039, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://drive.google.com/drive/folders/1INm-oPuCTjuFVOzKLQ1TJjYLztjXQVBf
```



Discussion:
> Before I get into that, because a playlist--in this case a Google Drive folder--can contain other playlists, would yt-dlp recursively retrieve all videos, or would it stop at the first layer?

Assuming the inner playlists have unique URLs (which I think is the case here), it will recurse. Otherwise, the extractor will have to do the recursion and put eveything in a single playlist
> Assuming the inner playlists have unique URLs (which I think is the case here), it will recurse.

Inner playlists do have unique URLs.

Resolving patch:
diff --git a/yt_dlp/extractor/common.py b/yt_dlp/extractor/common.py
--- a/yt_dlp/extractor/common.py
+++ b/yt_dlp/extractor/common.py
@@ -1188,7 +1188,7 @@ def _search_regex(self, pattern, string, name, default=NO_DEFAULT, fatal=True, f
             self.report_warning('unable to extract %s' % _name + bug_reports_message())
             return None
 
-    def _search_json(self, start_pattern, string, name, video_id, *, end_pattern='', contains_pattern='.+', fatal=True, **kwargs):
+    def _search_json(self, start_pattern, string, name, video_id, *, end_pattern='', contains_pattern='(?s:.+)', fatal=True, **kwargs):
         """Searches string for the JSON object specified by start_pattern"""
         # NB: end_pattern is only used to reduce the size of the initial match
         return self._parse_json(
diff --git a/yt_dlp/extractor/extractors.py b/yt_dlp/extractor/extractors.py
--- a/yt_dlp/extractor/extractors.py
+++ b/yt_dlp/extractor/extractors.py
@@ -597,7 +597,10 @@
 from .gofile import GofileIE
 from .golem import GolemIE
 from .goodgame import GoodGameIE
-from .googledrive import GoogleDriveIE
+from .googledrive import (
+    GoogleDriveIE,
+    GoogleDriveFolderIE,
+)
 from .googlepodcasts import (
     GooglePodcastsIE,
     GooglePodcastsFeedIE,
diff --git a/yt_dlp/extractor/googledrive.py b/yt_dlp/extractor/googledrive.py
--- a/yt_dlp/extractor/googledrive.py
+++ b/yt_dlp/extractor/googledrive.py
@@ -276,3 +276,59 @@ def add_source_format(urlh):
             'automatic_captions': self.extract_automatic_captions(
                 video_id, subtitles_id, hl),
         }
+
+
+class GoogleDriveFolderIE(InfoExtractor):
+    IE_NAME = 'GoogleDrive:Folder'
+    _VALID_URL = r'https?://(?:docs|drive)\.google\.com/drive/folders/(?P<id>[\w-]{28,})'
+    _TESTS = [{
+        'url': 'https://drive.google.com/drive/folders/1dQ4sx0-__Nvg65rxTSgQrl7VyW_FZ9QI',
+        'info_dict': {
+            'id': '1dQ4sx0-__Nvg65rxTSgQrl7VyW_FZ9QI',
+            'title': 'Forrest'
+        },
+        'playlist_count': 3,
+    }]
+    _BOUNDARY = '=====vc17a3rwnndj====='
+    _REQUEST = "/drive/v2beta/files?openDrive=true&reason=102&syncType=0&errorRecovery=false&q=trashed%20%3D%20false%20and%20'{folder_id}'%20in%20parents&fields=kind%2CnextPageToken%2Citems(kind%2CmodifiedDate%2CmodifiedByMeDate%2ClastViewedByMeDate%2CfileSize%2Cowners(kind%2CpermissionId%2Cid)%2ClastModifyingUser(kind%2CpermissionId%2Cid)%2ChasThumbnail%2CthumbnailVersion%2Ctitle%2Cid%2CresourceKey%2Cshared%2CsharedWithMeDate%2CuserPermission(role)%2CexplicitlyTrashed%2CmimeType%2CquotaBytesUsed%2Ccopyable%2CfileExtension%2CsharingUser(kind%2CpermissionId%2Cid)%2Cspaces%2Cversion%2CteamDriveId%2ChasAugmentedPermissions%2CcreatedDate%2CtrashingUser(kind%2CpermissionId%2Cid)%2CtrashedDate%2Cparents(id)%2CshortcutDetails(targetId%2CtargetMimeType%2CtargetLookupStatus)%2Ccapabilities(canCopy%2CcanDownload%2CcanEdit%2CcanAddChildren%2CcanDelete%2CcanRemoveChildren%2CcanShare%2CcanTrash%2CcanRename%2CcanReadTeamDrive%2CcanMoveTeamDriveItem)%2Clabels(starred%2Ctrashed%2Crestricted%2Cviewed))%2CincompleteSearch&appDataFilter=NO_APP_DATA&spaces=drive&pageToken={page_token}&maxResults=50&supportsTeamDrives=true&includeItemsFromAllDrives=true&corpora=default&orderBy=folder%2Ctitle_natural%20asc&retryCount=0&key={key} HTTP/1.1"
+    _DATA = f'''--{_BOUNDARY}
+content-type: application/http
+content-transfer-encoding: binary
+
+GET %s
+
+--{_BOUNDARY}
+'''
+
+    def _call_api(self, folder_id, key, data, **kwargs):
+        response = self._download_webpage(
+            'https://clients6.google.com/batch/drive/v2beta',
+            folder_id, data=data.encode('utf-8'),
+            headers={
+                'Content-Type': 'text/plain;charset=UTF-8;',
+                'Origin': 'https://drive.google.com',
+            }, query={
+                '$ct': f'multipart/mixed; boundary="{self._BOUNDARY}"',
+                'key': key
+            }, **kwargs)
+        return self._search_json('', response, 'api response', folder_id, **kwargs) or {}
+
+    def _get_folder_items(self, folder_id, key):
+        page_token = ''
+        while page_token is not None:
+            request = self._REQUEST.format(folder_id=folder_id, page_token=page_token, key=key)
+            page = self._call_api(folder_id, key, self._DATA % request)
+            yield from page['items']
+            page_token = page.get('nextPageToken')
+
+    def _real_extract(self, url):
+        folder_id = self._match_id(url)
+
+        webpage = self._download_webpage(url, folder_id)
+        key = self._search_regex(r'"(\w{39})"', webpage, 'key')
+
+        folder_info = self._call_api(folder_id, key, self._DATA % f'/drive/v2beta/files/{folder_id} HTTP/1.1', fatal=False)
+
+        return self.playlist_from_matches(
+            self._get_folder_items(folder_id, key), folder_id, folder_info.get('title'),
+            ie=GoogleDriveIE, getter=lambda item: f'https://drive.google.com/file/d/{item["id"]}')


Test patch: