Repository: yt-dlp/yt-dlp

Issue:
Can't download long Twitter Spaces
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

UK, US, WorldWide

### Provide a description that is worded well enough to be understood

Unable to download longer Twitter Spaces recordings. 

I was very pleased to see this added recently, and it has worked well on Spaces of up to 1h 30m,  but today I tried to get a couple of longer spaces (7 - 9 hours) and it crashed as per verbose log below.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://twitter.com/i/spaces/1BdGYyLYqrBGX?s=20']                                                                                              [6/1107]
[debug] User config "/home/tek/.config/yt-dlp/config": ['--no-mtime', '--restrict-filenames', '-o', '~/Downloads/yt/%(title)s_%(upload_date)s_f%(format_id)s_%(id)s.%(ext)s', '--write-auto-s
ub', '--write-sub', '--sub-lang', 'en,EN,en-gb,en-GB,EN-GB,en-ca,en-CA,EN-CA,en-ie,en-IE,EN-IE,en-us,en-US,EN-US,eng,ENG,english,English,ENGLISH,subtitles,en*,EN*', '--sub-format', 'best', 
'--embed-sub', '--add-metadata'] 
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b64402] (zip)
[debug] Python 3.9.2 (CPython x86_64 64bit) - Linux-5.10.0-19-amd64-x86_64-with-glibc2.31 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.31)
[debug] exe versions: ffmpeg 4.3.5-0, ffprobe 4.3.5-0, rtmpdump 2.4                                                                                                                          
[debug] Optional libraries: Cryptodome-3.9.7, certifi-2020.06.20, pyxattr-0.7.2, sqlite3-2.6.0, websockets-9.1
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] [twitter:spaces] Extracting URL: https://twitter.com/i/spaces/1BdGYyLYqrBGX?s=20
[twitter:spaces] 1BdGYyLYqrBGX: Downloading guest token
[twitter:spaces] 1BdGYyLYqrBGX: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading m3u8 information
ERROR: [twitter:spaces] 1BdGYyLYqrBGX: Failed to download m3u8 information: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.c
om/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 674, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/twitter.py", line 1168, in _real_extract
    formats = self._extract_m3u8_formats(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 2168, in _extract_m3u8_formats 
    fmts, subs = self._extract_m3u8_formats_and_subtitles(*args, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 2179, in _extract_m3u8_formats_and_subtitles
    res = self._download_webpage_handle(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 868, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 825, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 807, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3692, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
```

Can't download long Twitter Spaces
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.11.11** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

UK, US, WorldWide

### Provide a description that is worded well enough to be understood

Unable to download longer Twitter Spaces recordings. 

I was very pleased to see this added recently, and it has worked well on Spaces of up to 1h 30m,  but today I tried to get a couple of longer spaces (7 - 9 hours) and it crashed as per verbose log below.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://twitter.com/i/spaces/1BdGYyLYqrBGX?s=20']                                                                                              [6/1107]
[debug] User config "/home/tek/.config/yt-dlp/config": ['--no-mtime', '--restrict-filenames', '-o', '~/Downloads/yt/%(title)s_%(upload_date)s_f%(format_id)s_%(id)s.%(ext)s', '--write-auto-s
ub', '--write-sub', '--sub-lang', 'en,EN,en-gb,en-GB,EN-GB,en-ca,en-CA,EN-CA,en-ie,en-IE,EN-IE,en-us,en-US,EN-US,eng,ENG,english,English,ENGLISH,subtitles,en*,EN*', '--sub-format', 'best', 
'--embed-sub', '--add-metadata'] 
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.11.11 [8b64402] (zip)
[debug] Python 3.9.2 (CPython x86_64 64bit) - Linux-5.10.0-19-amd64-x86_64-with-glibc2.31 (OpenSSL 1.1.1n  15 Mar 2022, glibc 2.31)
[debug] exe versions: ffmpeg 4.3.5-0, ffprobe 4.3.5-0, rtmpdump 2.4                                                                                                                          
[debug] Optional libraries: Cryptodome-3.9.7, certifi-2020.06.20, pyxattr-0.7.2, sqlite3-2.6.0, websockets-9.1
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] [twitter:spaces] Extracting URL: https://twitter.com/i/spaces/1BdGYyLYqrBGX?s=20
[twitter:spaces] 1BdGYyLYqrBGX: Downloading guest token
[twitter:spaces] 1BdGYyLYqrBGX: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading m3u8 information
ERROR: [twitter:spaces] 1BdGYyLYqrBGX: Failed to download m3u8 information: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.c
om/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 674, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/twitter.py", line 1168, in _real_extract
    formats = self._extract_m3u8_formats(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 2168, in _extract_m3u8_formats 
    fmts, subs = self._extract_m3u8_formats_and_subtitles(*args, **kwargs)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 2179, in _extract_m3u8_formats_and_subtitles
    res = self._download_webpage_handle(
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 868, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 825, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/usr/local/bin/yt-dlp/yt_dlp/extractor/common.py", line 807, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3692, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
```



Discussion:
Does that spaces link play in your browser when you're not logged in? (use a private browsing tab)

It won't play for me in my browser whether I'm logged in or logged out; dev tools shows a 403 response for the m3u8
It plays in a private browser (Firefox 102.5.0esr (64-bit) on Linux)  whilst not logged in (takes a couple of mins for the audio to actually start) , even if ignoring all the cookies & login prompts on the page.

Thanks for your help.
Hi, I'm the author of the Twitter Space extractor. I'm glad someone is using the new extractor.

Back to the issue you reported:

The extractor correctly extracts the url of m3u8 manifest of the audiostream, but then fails to process it. Even opening the manifest url in the browser returns 403 but... opening it in httpie is ok:
```
❯ http "https://prod-fastly-ap-south-1.video.pscp.tv/Transcoding/v1/hls/bHEuB21FGdT3OttjkJujOpqywRjBDTMonbdlvCv29Y3w0Z6QvDjEXTnbjW70f1kob_ZSJi76A9yRpQ0WlDIeGw/non_transcode/ap-southeast-1/periscope-replay-direct-prod-ap-southeast-1-public/audio-space/playlist_16778111767038456159.m3u8?type=replay" --download
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 4461
Connection: keep-alive
Content-Length: 900049
Content-Type: application/x-mpegURL
Date: Thu, 17 Nov 2022 22:14:04 GMT
Last-Modified: Wed, 16 Nov 2022 20:58:31 GMT
Vary: Origin
Via: 1.1 varnish, 1.1 varnish
X-Amzn-Trace-Id: Root=1-63754ef7-649a5c17a3e954540ff3509d
X-Cache: HIT, HIT
X-Cache-Hits: 45, 1
X-Served-By: cache-fjr990025-FJR, cache-mxp6927-MXP
X-Timer: S1668723244.292364,VS0,VE1

Downloading to playlist_16778111767038456159.m3u8-2
Done. 900.0 kB in 00:0.79500 (1.1 MB/s)
```
downloading with aria2 or ffmpeg is fine as well.
But downloading the m3u8 manifest directly with yt-dlp it does not work.

It probably has to do with headers sent with the http request. I'll investigate and figure out how to fix it.


EDIT: Fixed it! Added custom headers to the http request and it works. I'll test it with other links to see if it's ok.
```
❯ ydpm https://twitter.com/i/spaces/1BdGYyLYqrBGX -F
[twitter:spaces] 1BdGYyLYqrBGX: Downloading guest token
[twitter:spaces] 1BdGYyLYqrBGX: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading m3u8 information
[info] Available formats for 1BdGYyLYqrBGX:
ID EXT RESOLUTION | PROTO | VCODEC     ACODEC
---------------------------------------------
0  m4a audio only | m3u8F | audio only aac
```
Does that spaces link play in your browser when you're not logged in? (use a private browsing tab)

It won't play for me in my browser whether I'm logged in or logged out; dev tools shows a 403 response for the m3u8
It plays in a private browser (Firefox 102.5.0esr (64-bit) on Linux)  whilst not logged in (takes a couple of mins for the audio to actually start) , even if ignoring all the cookies & login prompts on the page.

Thanks for your help.
Hi, I'm the author of the Twitter Space extractor. I'm glad someone is using the new extractor.

Back to the issue you reported:

The extractor correctly extracts the url of m3u8 manifest of the audiostream, but then fails to process it. Even opening the manifest url in the browser returns 403 but... opening it in httpie is ok:
```
❯ http "https://prod-fastly-ap-south-1.video.pscp.tv/Transcoding/v1/hls/bHEuB21FGdT3OttjkJujOpqywRjBDTMonbdlvCv29Y3w0Z6QvDjEXTnbjW70f1kob_ZSJi76A9yRpQ0WlDIeGw/non_transcode/ap-southeast-1/periscope-replay-direct-prod-ap-southeast-1-public/audio-space/playlist_16778111767038456159.m3u8?type=replay" --download
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 4461
Connection: keep-alive
Content-Length: 900049
Content-Type: application/x-mpegURL
Date: Thu, 17 Nov 2022 22:14:04 GMT
Last-Modified: Wed, 16 Nov 2022 20:58:31 GMT
Vary: Origin
Via: 1.1 varnish, 1.1 varnish
X-Amzn-Trace-Id: Root=1-63754ef7-649a5c17a3e954540ff3509d
X-Cache: HIT, HIT
X-Cache-Hits: 45, 1
X-Served-By: cache-fjr990025-FJR, cache-mxp6927-MXP
X-Timer: S1668723244.292364,VS0,VE1

Downloading to playlist_16778111767038456159.m3u8-2
Done. 900.0 kB in 00:0.79500 (1.1 MB/s)
```
downloading with aria2 or ffmpeg is fine as well.
But downloading the m3u8 manifest directly with yt-dlp it does not work.

It probably has to do with headers sent with the http request. I'll investigate and figure out how to fix it.


EDIT: Fixed it! Added custom headers to the http request and it works. I'll test it with other links to see if it's ok.
```
❯ ydpm https://twitter.com/i/spaces/1BdGYyLYqrBGX -F
[twitter:spaces] 1BdGYyLYqrBGX: Downloading guest token
[twitter:spaces] 1BdGYyLYqrBGX: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading JSON metadata
[twitter:spaces] 28_1592642205298352128: Downloading m3u8 information
[info] Available formats for 1BdGYyLYqrBGX:
ID EXT RESOLUTION | PROTO | VCODEC     ACODEC
---------------------------------------------
0  m4a audio only | m3u8F | audio only aac
```

Resolving patch:
diff --git a/yt_dlp/extractor/twitter.py b/yt_dlp/extractor/twitter.py
--- a/yt_dlp/extractor/twitter.py
+++ b/yt_dlp/extractor/twitter.py
@@ -1167,7 +1167,8 @@ def _real_extract(self, url):
             # XXX: Native downloader does not work
             formats = self._extract_m3u8_formats(
                 traverse_obj(source, 'noRedirectPlaybackUrl', 'location'),
-                metadata['media_key'], 'm4a', 'm3u8', live=live_status == 'is_live')
+                metadata['media_key'], 'm4a', 'm3u8', live=live_status == 'is_live',
+                headers={'Referer': 'https://twitter.com/'})
             for fmt in formats:
                 fmt.update({'vcodec': 'none', 'acodec': 'aac'})
 


Test patch: