Repository: yt-dlp/yt-dlp
Pull request #2012 (2021-12-16 02:44:09)

Add replacement output template via `&`
### Before submitting a *pull request* make sure you have:
- [x] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
- [x] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
- [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)

### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
- [x] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
- [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)

### What is the purpose of your *pull request*?
- [ ] Bug fix
- [ ] Improvement
- [ ] New extractor
- [x] New feature

---

### Description of your *pull request* and other information

This pull requests adds a new internal operator to output templates: `&`. This causes the value of the field to be replaced by a given static string.

The purpose of this is to build output templates that are highly flexible, e.g. for handling non-playlist and playlist videos with the same configuration. Suppose you want to prefix the playlist index of a video in front of the filename. Using the following template works fine: ```%(playlist_index)s - %(title)s.%(ext)s```

But it has a flaw: if downloading non-playlist video (i.e. `playlist_index` is None), the filename is prefixed by `NA - `; we don't want that. A first improvement would be to use an empty default string like so:
```%(playlist_index|)s - %(title)s.%(ext)s```
This at least removes the `NA`, but still leaves the useless ` - `.

With this pull request, it is possible to create the following template, which removes both the `NA` and ` - ` if `playlist_index` is None (or more precisely, only adds the ` - ` if `playlist_index` is *not* None):
```%(playlist_index|)s%(playlist_index& - )s%(title)s.%(ext)s```

An alternative I have considered is to add an *append* operator, which appends a given string to the value if the field is *not* None:
```%(playlist_index& - |)s%(title)s.%(ext)s```
This makes the output template more readable, but is also less flexible (e.g. prefixes are not possible). By using the "replacement" interpretation we have the highest flexibility in creating custom output templates.

The resulting output template quickly become complex, but they are clearly a power-user feature. The idea is to write an output template once, that covers all use-cases and save it in the config file. This removes the necessity to switch config files (or pass `-o` parameter by other means) when downloading playlists vs. non-playlist videos.

pukkandan: This can already be done using `--parse-metadata`, but that is much more complicated that your approach
```
--parse-metadata "%(playlist_index|)s - :(?P<prefix>\d.+)"
-o "%(prefix)s%(title)s"
```
So this is a nice addition.

---

Should falsey values trigger this? On one hand, this removes the ability to distinguish b/w False and None. On other hand, we can do `%(is_live&LIVE|VOD)s`
PilzAdam: > This can already be done using `--parse-metadata`, but that is much more complicated that your approach
> 
> ```
> --parse-metadata "%(playlist_index|)s - :(?P<prefix>\d.+)"
> -o "%(prefix)s%(title)s"
> ```

I did not know this. But as you say, the method from this PR is probably easier to discover and use.

> Should falsey values trigger this? On one hand, this removes the ability to distinguish b/w False and None. On other hand, we can do `%(is_live&LIVE|VOD)s`

That's an interesting question. I don't feel strongly either way. How does the current default operator `|` work exactly? I think `&` and `|` should have the same semantic, otherwise things become confusing.

Another option would be to add an `=` operator: use the value of the field *only if* it is equal the given string. So your example would become:
```
%(is_live=true&LIVE|VOD)s
```
This is, again, the most flexible solution. However, that's out of scope for this particular PR.
pukkandan: You are right. It was unnecessary confusion in my part. `|` only looks at `None`. So `&` should do the same
PilzAdam: I integrated all your suggestions; thanks for your feedback.
pukkandan: >  I also rebased onto current master (I hope that's ok)

generally, don't do this. But this PR is small enough that it doesn't actually matter

Patch:
diff --git a/README.md b/README.md
index da0d9be9f..452ad9b22 100644
--- a/README.md
+++ b/README.md
@@ -1076,6 +1076,8 @@ # OUTPUT TEMPLATE
 
 1. **Alternatives**: Alternate fields can be specified seperated with a `,`. Eg: `%(release_date>%Y,upload_date>%Y|Unknown)s`
 
+1. **Replacement**: A replacement value can specified using a `&` separator. If the field is *not* empty, this replacement value will be used instead of the actual field content. This is done after alternate fields are considered; thus the replacement is used if *any* of the alternative fields is *not* empty.
+
 1. **Default**: A literal default value can be specified for when the field is empty using a `|` seperator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s`
 
 1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `B`, `j`, `l`, `q` can be used for converting to **B**ytes, **j**son (flag `#` for pretty-printing), a comma seperated **l**ist (flag `#` for `\n` newline-seperated) and a string **q**uoted for the terminal (flag `#` to split a list into different arguments), respectively
@@ -1084,7 +1086,7 @@ # OUTPUT TEMPLATE
 
 To summarize, the general syntax for a field is:
 ```
-%(name[.keys][addition][>strf][,alternate][|default])[flags][width][.precision][length]type
+%(name[.keys][addition][>strf][,alternate][&replacement][|default])[flags][width][.precision][length]type
 ```
 
 Additionally, you can set different output templates for the various metadata files separately from the general output template by specifying the type of file followed by the template separated by a colon `:`. The different file types supported are `subtitle`, `thumbnail`, `description`, `annotation` (deprecated), `infojson`, `link`, `pl_thumbnail`, `pl_description`, `pl_infojson`, `chapter`. For example, `-o '%(title)s.%(ext)s' -o 'thumbnail:%(title)s\%(title)s.%(ext)s'`  will put the thumbnails in a folder with the same name as the video. If any of the templates (except default) is empty, that type of file will not be written. Eg: `--write-thumbnail -o "thumbnail:"` will write thumbnails only for playlists and not for video.
@@ -1252,6 +1254,9 @@ # Download YouTube playlist videos in separate directory indexed by video order
 # Download YouTube playlist videos in separate directories according to their uploaded year
 $ yt-dlp -o '%(upload_date>%Y)s/%(title)s.%(ext)s' https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re
 
+# Prefix playlist index with " - " separator, but only if it is available
+$ yt-dlp -o '%(playlist_index|)s%(playlist_index& - |)s%(title)s.%(ext)s' BaW_jenozKc https://www.youtube.com/user/TheLinuxFoundation/playlists
+
 # Download all playlists of YouTube channel/user keeping each playlist in separate directory:
 $ yt-dlp -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' https://www.youtube.com/user/TheLinuxFoundation/playlists
 
diff --git a/yt_dlp/YoutubeDL.py b/yt_dlp/YoutubeDL.py
index 317526d10..ec69151d7 100644
--- a/yt_dlp/YoutubeDL.py
+++ b/yt_dlp/YoutubeDL.py
@@ -1055,7 +1055,8 @@ def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
             (?P<fields>{field})
             (?P<maths>(?:{math_op}{math_field})*)
             (?:>(?P<strf_format>.+?))?
-            (?P<alternate>(?<!\\),[^|)]+)?
+            (?P<alternate>(?<!\\),[^|&)]+)?
+            (?:&(?P<replacement>.*?))?
             (?:\|(?P<default>.*?))?
             $'''.format(field=FIELD_RE, math_op=MATH_OPERATORS_RE, math_field=MATH_FIELD_RE))
 
@@ -1114,11 +1115,12 @@ def create_key(outer_mobj):
             key = outer_mobj.group('key')
             mobj = re.match(INTERNAL_FORMAT_RE, key)
             initial_field = mobj.group('fields').split('.')[-1] if mobj else ''
-            value, default = None, na
+            value, replacement, default = None, None, na
             while mobj:
                 mobj = mobj.groupdict()
                 default = mobj['default'] if mobj['default'] is not None else default
                 value = get_value(mobj)
+                replacement = mobj['replacement']
                 if value is None and mobj['alternate']:
                     mobj = re.match(INTERNAL_FORMAT_RE, mobj['alternate'][1:])
                 else:
@@ -1128,7 +1130,7 @@ def create_key(outer_mobj):
             if fmt == 's' and value is not None and key in field_size_compat_map.keys():
                 fmt = '0{:d}d'.format(field_size_compat_map[key])
 
-            value = default if value is None else value
+            value = default if value is None else value if replacement is None else replacement
 
             flags = outer_mobj.group('conversion') or ''
             str_fmt = f'{fmt[:-1]}s'


Test patch:
diff --git a/test/test_YoutubeDL.py b/test/test_YoutubeDL.py
index 63ef50e1a..6c2530046 100644
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@@ -836,6 +836,11 @@ def gen():
         test('%(title3)s', ('foo/bar\\test', 'foo_bar_test'))
         test('folder/%(title3)s', ('folder/foo/bar\\test', 'folder%sfoo_bar_test' % os.path.sep))
 
+        # Replacement
+        test('%(id&foo)s.bar', 'foo.bar')
+        test('%(title&foo)s.bar', 'NA.bar')
+        test('%(title&foo|baz)s.bar', 'baz.bar')
+
     def test_format_note(self):
         ydl = YoutubeDL()
         self.assertEqual(ydl._format_note({}), '')