Repository: yt-dlp/yt-dlp
Issue category: Bug Report

Issue:
Substack: follow redirects when it's necessary
### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.08.08** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

worldwide

### Provide a description that is worded well enough to be understood

For some Substack sites, `yt-dlp` detects the audio URL with `'unknown'`  as fields value:

```
$ yt-dlp --dump-single-json --no-check-certificates --no-warnings --prefer-free-formats 'https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un' | jq
{
  "id": "61841178",
  "formats": [
    {
      "url": "https://api.substack.com/api/v1/audio/upload/fed204e2-c499-47f2-bbd3-f54a375b8f72/src",
      "protocol": "https",
      "ext": "unknown_video",
      "video_ext": "unknown_video",
      "audio_ext": "none",
      "format_id": "0",
      "format": "0 - unknown",
      "resolution": null,
      "dynamic_range": "SDR",
      "http_headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-us,en;q=0.5",
        "Sec-Fetch-Mode": "navigate",
        "Cookie": "__cf_bm=q3K.eGd_u9GL01IGnhI7j5ho5_lKGQMd8hzxnX.lF.E-1659998386-0-Afc0FzC9Mm5JjdnyYmDzgghLomXN0XclAJJATx+eWWifelyMY/XFdf/ASf6o9ydl0yCDuNDe0Gna6/cbZ6Gmxu8="
      }
    }
  ]
}
```

This is happening because the URL detected is actually a redirect:

```
$ headers https://api.substack.com/api/v1/audio/upload/fed204e2-c499-47f2-bbd3-f54a375b8f72/src
HTTP/2 307
location: https://substackcdn.com/video_upload/post/61841178/fed204e2-c499-47f2-bbd3-f54a375b8f72/transcoded.mp3?post_id=61841178&relation=podcast&Expires=1660084845&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=H3ibpkrnP12zFGfTkdBxvXtUXQ-o8uj-pIJLYE-hG5nQnzTk6A8RIzWI1de25PRmWTPDgRpNrjda-2FhTFjYu0BU98D1bWe9~3IcIf8ZZ8doEpVF4h0NB~w8iltN5PRnI4Jsuz9YUGKz0EVs-Lsx76dqzNHv6yRZcdSz5O3NjsOpCwOAYcAyMT36tKH-iYKUY5VZPEA5q1xIpXz85~wXzNhSpltkNS7g0KRQylKF8WkGr7XsWRiUUIrj2BAD3qdxkLlbPmYuP2C03S3x1oGCv9hPukbJ0G8Ix-XpjbkAz6om3ndv5~WnwichOWay0L6Z0gKt1g2CBJvChsoxrR5Ugg__
```

If `yt-dlp` is run over the final location URL, then fields are resolved properly:

```
./bin/yt-dlp --dump-single-json --no-check-certificates --no-warnings --prefer-free-formats 'https://substackcdn.com/video_upload/post/61841178/fed204e2-c499-47f2-bbd3-f54a375b8f72/transcoded.mp3?post_id=61841178&relation=podcast&Expires=1660084845&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=H3ibpkrnP12zFGfTkdBxvXtUXQ-o8uj-pIJLYE-hG5nQnzTk6A8RIzWI1de25PRmWTPDgRpNrjda-2FhTFjYu0BU98D1bWe9~3IcIf8ZZ8doEpVF4h0NB~w8iltN5PRnI4Jsuz9YUGKz0EVs-Lsx76dqzNHv6yRZcdSz5O3NjsOpCwOAYcAyMT36tKH-iYKUY5VZPEA5q1xIpXz85~wXzNhSpltkNS7g0KRQylKF8WkGr7XsWRiUUIrj2BAD3qdxkLlbPmYuP2C03S3x1oGCv9hPukbJ0G8Ix-XpjbkAz6om3ndv5~WnwichOWay0L6Z0gKt1g2CBJvChsoxrR5Ugg__' | jq
{
  "id": "transcoded",
  "title": "transcoded",
  "timestamp": 1656571004,
  "direct": true,
  "formats": [
    {
      "format_id": "mpeg",
      "url": "https://substackcdn.com/video_upload/post/61841178/fed204e2-c499-47f2-bbd3-f54a375b8f72/transcoded.mp3?post_id=61841178&relation=podcast&Expires=1660084845&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=H3ibpkrnP12zFGfTkdBxvXtUXQ-o8uj-pIJLYE-hG5nQnzTk6A8RIzWI1de25PRmWTPDgRpNrjda-2FhTFjYu0BU98D1bWe9~3IcIf8ZZ8doEpVF4h0NB~w8iltN5PRnI4Jsuz9YUGKz0EVs-Lsx76dqzNHv6yRZcdSz5O3NjsOpCwOAYcAyMT36tKH-iYKUY5VZPEA5q1xIpXz85~wXzNhSpltkNS7g0KRQylKF8WkGr7XsWRiUUIrj2BAD3qdxkLlbPmYuP2C03S3x1oGCv9hPukbJ0G8Ix-XpjbkAz6om3ndv5~WnwichOWay0L6Z0gKt1g2CBJvChsoxrR5Ugg__",
      "vcodec": "none",
      "protocol": "https",
      "ext": "mp3",
      "audio_ext": "mp3",
      "video_ext": "none",
      "format": "mpeg - audio only",
      "resolution": "audio only",
      "http_headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-us,en;q=0.5",
        "Sec-Fetch-Mode": "navigate"
      }
    }
  ]
}
```

so ideally, `yt-dlp` should to follow the URL in case it's a redirect. The HTTP status or pattern matching can be used to make the conditional logic.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
yt-dlp -vU --dump-single-json 'https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un'
[debug] Command-line config: ['-vU', '--dump-single-json', 'https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.08.08 [3157158] (zip)
[debug] Python 3.9.13 (CPython 64bit) - macOS-12.5-arm64-arm-64bit
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 5.1 (setts), ffprobe 5.1, rtmpdump 2.4
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.08.08, Current version: 2022.08.08
yt-dlp is up to date (2022.08.08)
[debug] [Substack] Extracting URL: https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un
[Substack] la-guerre-de-troie-pour-ecrire-un: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), channels, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 61841178: Downloading 1 format(s): 0
{"id": "61841178", "formats": [{"url": "https://api.substack.com/api/v1/audio/upload/fed204e2-c499-47f2-bbd3-f54a375b8f72/src", "protocol": "https", "ext": "unknown_video", "video_ext": "unknown_video", "audio_ext": "none", "format_id": "0", "format": "0 - unknown", "resolution": null, "dynamic_range": "SDR", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate", "Cookie": "__cf_bm=2K134opp_fYn5W8J1jvgWRTuo7F0bwvTmpLucUVAWHM-1659998889-0-AWsqrqJmF3c+z7gCqSBWCr13TJp092NGH+4zIt1ShF4i0JJF4SjWQF7hCrBm8ny6dj2cukhtNJd6Uggzdm4vbXA="}}], "subtitles": {}, "title": "La guerre de Troie pour e\u0301crire un script", "description": "Listen now (16 min) | Comment \u00e9crire rapidement un script en 3 \u00e9tapes Maintenable, efficace m\u00eame quand on ne sait pas scripter Si \u00e9crire un script est une Odyss\u00e9e pour toi Si tu te sens comme M\u00e9n\u00e9las ayant perdu H\u00e9l\u00e8ne quand on te dis de le faire Je te propose de d\u00e9couvrir ma m\u00e9thode", "thumbnail": "https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/b8547545-d828-4330-b24d-b6ce1c70d3b9_1400x1400.png", "uploader": "Le DevOPS accessible aux Dev Junior", "uploader_id": "584595", "webpage_url": "https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un", "original_url": "https://imrane.substack.com/p/la-guerre-de-troie-pour-ecrire-un", "webpage_url_basename": "la-guerre-de-troie-pour-ecrire-un", "webpage_url_domain": "imrane.substack.com", "extractor": "Substack", "extractor_key": "Substack", "playlist": null, "playlist_index": null, "thumbnails": [{"url": "https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/b8547545-d828-4330-b24d-b6ce1c70d3b9_1400x1400.png", "id": "0"}], "display_id": "61841178", "fulltitle": "La guerre de Troie pour e\u0301crire un script", "requested_subtitles": null, "_has_drm": null, "requested_downloads": [{"url": "https://api.substack.com/api/v1/audio/upload/fed204e2-c499-47f2-bbd3-f54a375b8f72/src", "protocol": "https", "ext": "unknown_video", "video_ext": "unknown_video", "audio_ext": "none", "format_id": "0", "format": "0 - unknown", "dynamic_range": "SDR", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate", "Cookie": "__cf_bm=2K134opp_fYn5W8J1jvgWRTuo7F0bwvTmpLucUVAWHM-1659998889-0-AWsqrqJmF3c+z7gCqSBWCr13TJp092NGH+4zIt1ShF4i0JJF4SjWQF7hCrBm8ny6dj2cukhtNJd6Uggzdm4vbXA="}, "epoch": 1659998889, "_filename": "La guerre de Troie pour e\u0301crire un script [61841178].unknown_video", "__write_download_archive": false}], "url": "https://api.substack.com/api/v1/audio/upload/fed204e2-c499-47f2-bbd3-f54a375b8f72/src", "protocol": "https", "ext": "unknown_video", "video_ext": "unknown_video", "audio_ext": "none", "format_id": "0", "format": "0 - unknown", "resolution": null, "dynamic_range": "SDR", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate", "Cookie": "__cf_bm=2K134opp_fYn5W8J1jvgWRTuo7F0bwvTmpLucUVAWHM-1659998889-0-AWsqrqJmF3c+z7gCqSBWCr13TJp092NGH+4zIt1ShF4i0JJF4SjWQF7hCrBm8ny6dj2cukhtNJd6Uggzdm4vbXA="}, "epoch": 1659998889, "_type": "video"}
```


Hints:
As of 2 years later: Partially confirmed. To be clear, the file is indeed downloaded and the redirect is followed. But some filetype data is missing, yes, and the resulting download has extension `.unknown_video`.

I think the stated reasoning about the HTTP 307 redirect is irrelevant. **Metadata (incl. filetype) is identical whether you pass the *extracted* URL or you pass the *redirected* URL.** Significant metadata (title, description, id) is correct **only** if you pass the normal page URL and not when passing the more-direct URLs.

The given sample URL is now a 404, so I used a different one: https://www.persuasion.community/p/summers

- Page URL: https://www.persuasion.community/p/summers
- Extracted URL: https://api.substack.com/api/v1/audio/upload/a98d2387-4a2b-47fe-9437-f6f7056e13c7/src
- Redirect (http 307) from extracted URL (varies each time, probably will be expired after a short time):
  ```
  https://substackcdn.com/video_upload/post/141970405/a98d2387-4a2b-47fe-9437-f6f7056e13c7/transcoded.mp3?post_id=141970405&relation=podcast&Expires=1729271509&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=RjL3VPk38SGNBE3QmSEIOUBHEUgnGOSY5SXWsuoT3GVYvpYszUX7kvbQR1SP9oXOmjwaL5DcW2EQow8rK9sV56DI4CgY1HS9ruurU9ROCYu6mnaK8jpFMzVrjx9ZKWQ0U-GqS9Sbr~HNkojvxboGsEG344bW9h7qYwVzidYTveA1vWL9wIhEPfSbgEqGv6TWig6Be9vKnefv81tWwM--lF7HuN1DkBPPiyo75bdOhLNy03j6N-liIol9pmh4ESRJfbAnd9ALkqcIwOjwZGo7W9Kn-kTazQCltTfxEJSMAfVCcwtTeNaeQcXqRhrKeLX4jY8aNZx6B~xT334nO8QyxQ__
  ```


Patch:
diff --git a/yt_dlp/extractor/substack.py b/yt_dlp/extractor/substack.py
index 30cb322dc250..b70d40f2cad9 100644
--- a/yt_dlp/extractor/substack.py
+++ b/yt_dlp/extractor/substack.py
@@ -2,7 +2,13 @@
 import urllib.parse
 
 from .common import InfoExtractor
-from ..utils import js_to_json, str_or_none, traverse_obj
+from ..networking import HEADRequest
+from ..utils import (
+    determine_ext,
+    js_to_json,
+    str_or_none,
+)
+from ..utils.traversal import traverse_obj
 
 
 class SubstackIE(InfoExtractor):
@@ -43,6 +49,19 @@ class SubstackIE(InfoExtractor):
             'uploader': "Andrew Zimmern's Spilled Milk ",
             'uploader_id': '577659',
         },
+    }, {
+        # Podcast that needs its file extension resolved to mp3
+        'url': 'https://persuasion1.substack.com/p/summers',
+        'md5': '1456a755d46084744facdfac9edf900f',
+        'info_dict': {
+            'id': '141970405',
+            'ext': 'mp3',
+            'title': 'Larry Summers on What Went Wrong on Campus',
+            'description': 'Yascha Mounk and Larry Summers also discuss the promise and perils of artificial intelligence.',
+            'thumbnail': r're:https://substackcdn\.com/image/.+\.jpeg',
+            'uploader': 'Persuasion',
+            'uploader_id': '61579',
+        },
     }]
 
     @classmethod
@@ -89,7 +108,15 @@ def _real_extract(self, url):
         post_type = webpage_info['post']['type']
         formats, subtitles = [], {}
         if post_type == 'podcast':
-            formats, subtitles = [{'url': webpage_info['post']['podcast_url']}], {}
+            fmt = {'url': webpage_info['post']['podcast_url']}
+            if not determine_ext(fmt['url'], default_ext=None):
+                # The redirected format URL expires but the original URL doesn't,
+                # so we only want to extract the extension from this request
+                fmt['ext'] = determine_ext(self._request_webpage(
+                    HEADRequest(fmt['url']), display_id,
+                    'Resolving podcast file extension',
+                    'Podcast URL is invalid').url)
+            formats.append(fmt)
         elif post_type == 'video':
             formats, subtitles = self._extract_video_formats(webpage_info['post']['videoUpload']['id'], canonical_url)
         else:


Test patch: