Repository: ManimCommunity/manim
Pull request: Added basic support for adding subcaptions via :meth:`.Scene.add_subcaption`
<!-- Thank you for contributing to Manim! Learn more about the process in our contributing guidelines: https://docs.manim.community/en/latest/contributing.html -->

## Overview: What does this pull request change?
<!-- If there is more information than the PR title that should be added to our release changelog, add it in the following changelog section. This is optional, but recommended for larger pull requests. -->
<!--changelog-start-->
- New method :meth:`.Scene.add_subcaption`
- New keyword arguments ``subcaption``, ``subcaption_duration``, ``subcaption_offset`` for :meth:`.Scene.play`
<!--changelog-end-->



## Motivation and Explanation: Why and how do your changes improve the library?
<!-- Optional for bugfixes, small enhancements, and documentation-related PRs. Otherwise, please give a short reasoning for your changes. -->

If a user writes any subcaptions in a Scene via the newly introduced methods (or kwargs), an `.srt` file is written containing these subcaptions.

Closes #910.

## Links to added or changed documentation pages
<!-- Please add links to the affected documentation pages (edit the description after opening the PR). The link to the documentation for your PR is https://manimce--####.org.readthedocs.build/en/####/, where #### represents the PR number. -->

Documentation of Scene: https://manimce--2314.org.readthedocs.build/en/2314/reference/manim.scene.scene.Scene.html#manim.scene.scene.Scene


<!-- Thank you again for contributing! Do not modify the lines below, they are for reviewers. -->
## Reviewer Checklist
- [ ] The PR title is descriptive enough for the changelog, and the PR is labeled correctly
- [ ] If applicable: newly added non-private functions and classes have a docstring including a short summary and a PARAMETERS section
- [ ] If applicable: newly added functions and classes are tested


Problem statement:
Feature request: Subtitle timings
## Description of proposed feature
<!-- Add a clear and concise description of the new feature, including a motivation: why do you think this will be useful? -->

It would be fairly useful to have animation options that aren't for actually displaying anything, but for marking the timestamps in an animation with text that can be used to export subtitles.

Since a lot of videos made in manim are intended to include additional voiceover, this would make timing subtitles, or at least adding subtitles for the intended script to videos, a lot easier. Even if the actual voiceover doesn't 100% match the script, it's a lot easier to tweak the words and the timing than to make them from scratch.

Alternatively, folks who don't do voiceover but want to add subtitles as an alternative can still do so.

## How can the new feature be used?
<!-- If possible, illustrate how this new feature could be used. -->

I'm imagining an implementation similar to:

1. Animations can set the subtitle text or clear it (e.g. end the latest subtitle), which would happen as soon as the animation starts.
2. There is a dedicated animation `Subtitle(text)` which sets the subtitle to the given string, or clears it if passed `None`.
3. Alternatively, there could be dedicated `set_subtitle` and `clear_subtitle` methods since setting a subtitle takes zero time.

This way, you can still use the existing animation system to leverage adding subtitles, e.g. `wait` still works as expected and adding a subtitle animation alongside other animations would set it as soon as those animations are started.  

## Additional comments
<!-- Add further context that you think might be relevant. -->

The output would probably make the most sense as an srt file since the videos are output as MP4. This is a relatively simple format where each line includes a start/end timestamp for a subtitle and the text. The lines could be written out whenever a subtitle is cleared or changed, or the video ends.

Hints from issue discussion:
Hello and thanks for the suggestion!

I must say that my position is a "soft no". I am not an expert, but I believe there are ways of adding subtitles that are external to how the video is rendered. For example, in most playback software that accepts subtitles, you have the option to turn subtitles off and on. However, in what you propose, there is no way of turning subtitles off and on once the video is rendered with subtitles. So if anyone is really serious about adding subtitles, I would encourage them to look for other ways of adding them after the video is rendered.

However, if others think this would be useful, I could see this being a manim plugin.
To clarify, this would specifically be for closed subtitles (i.e. not directly rendered) rather than open subtitles (rendered onto the video itself). Since it'd be storing the subtitles in a separate file the main benefit is to export timing information from the rendered animation.
Oh okay, that makes more sense. 

If I'm not mistaken, this would depend on the underlying renderer. For example, I would assume there is a way of telling ffmpeg to add closed captions and/or telling ffmpeg about the timing information for the closed captions. If that's the case, then this would probably be more than a plugin as it would have to change how the rendering step happens... 


(I also suggest in the post that it could just output as a .srt file, which is just a text file and super easy to do by hand. Most video players nowadays will automatically read this if it's in the same folder.)

I believe that since the way manim generates animations is each animation is a separate video, then all we'd have to do is get the times for the renders to get the timing for the subtitles. But I could be mistaken.
Is that all that needs to be done? Or does the actual mp4 file need to include some information as well? (Sorry I didn't see that part first!)
No worries! I am also guilty of commenting without reading everything first. :p 

And… yeah, pretty much. There are more complicated video containers like mkv that can include the subtitles with the video, but in a lot of cases it's sufficient to just output an SRT file next to the video with the same name and the extension replaced with .srt.

I wasn't 100% right on the timings just being individual lines, but it's still extremely straightforward. Since it's just text you don't need any fancy code to generate it, just a few print statements. See this page on wikipedia: https://en.wikipedia.org/wiki/SubRip#File_format
In that case, then I do think we can make this part of the core library. 

I agree with you that the interface should look like an Animation.

The part that needs a bit of thinking would be the actual writing of the file. I suggest we create a new class SRTWriter or CCWriter or something like that. And all of the functionality can be imported as `from manim import captions` and then do `captions.clear()`, or something similar.

Commit message: fix typo

Patch to review:
diff --git a/manim/scene/scene.py b/manim/scene/scene.py
index afbfa73cfc..b716162f40 100644
--- a/manim/scene/scene.py
+++ b/manim/scene/scene.py
@@ -3,6 +3,7 @@
 __all__ = ["Scene"]
 
 import copy
+import datetime
 import inspect
 import platform
 import random
@@ -12,6 +13,8 @@
 from queue import Queue
 from typing import List, Optional
 
+import srt
+
 from manim.scene.section import DefaultSectionType
 
 try:
@@ -899,8 +902,46 @@ def get_run_time(self, animations):
         else:
             return np.max([animation.run_time for animation in animations])
 
-    def play(self, *args, **kwargs):
+    def play(
+        self,
+        *args,
+        subcaption=None,
+        subcaption_duration=None,
+        subcaption_offset=0,
+        **kwargs,
+    ):
+        r"""Plays an animation in this scene.
+
+        Parameters
+        ----------
+
+        args
+            Animations to be played.
+        subcaption
+            The content of the external subcaption that should
+            be added during the animation.
+        subcaption_duration
+            The duration for which the specified subcaption is
+            added. If ``None`` (the default), the run time of the
+            animation is taken.
+        subcaption_offset
+            An offset (in seconds) for the start time of the
+            added subcaption.
+        kwargs
+            All other keywords are passed to the renderer.
+
+        """
+        start_time = self.renderer.time
         self.renderer.play(self, *args, **kwargs)
+        run_time = self.renderer.time - start_time
+        if subcaption:
+            if subcaption_duration is None:
+                subcaption_duration = run_time
+            self.add_subcaption(
+                content=subcaption,
+                duration=subcaption_duration,
+                offset=-run_time + subcaption_offset,
+            )
 
     def wait(self, duration=DEFAULT_WAIT_TIME, stop_condition=None):
         self.play(Wait(run_time=duration, stop_condition=stop_condition))
@@ -1242,6 +1283,52 @@ def update_to_time(self, t):
         self.update_meshes(dt)
         self.update_self(dt)
 
+    def add_subcaption(
+        self, content: str, duration: float = 1, offset: float = 0
+    ) -> None:
+        r"""Adds an entry in the corresponding subcaption file.
+
+        Parameters
+        ----------
+
+        content
+            The subcaption content.
+        duration
+            The duration (in seconds) for which the subcaption is shown.
+        offset
+            This offset (in seconds) is added to the starting time stamp
+            of the subcaption.
+
+        Examples
+        --------
+
+        This example illustrates both possibilities for adding
+        subcaptions to Manimations::
+
+            class SubcaptionExample(Scene):
+                def construct(self):
+                    square = Square()
+                    circle = Circle()
+
+                    # first option: via the add_subcaption method
+                    self.add_subcaption("Hello square!", duration=1)
+                    self.play(Create(square))
+
+                    # second option: within the call to Scene.play
+                    self.play(
+                        Transform(square, circle),
+                        subcaption="The square transforms."
+                    )
+
+        """
+        subtitle = srt.Subtitle(
+            index=len(self.renderer.file_writer.subcaptions),
+            content=content,
+            start=datetime.timedelta(seconds=self.renderer.time + offset),
+            end=datetime.timedelta(seconds=self.renderer.time + offset + duration),
+        )
+        self.renderer.file_writer.subcaptions.append(subtitle)
+
     def add_sound(self, sound_file, time_offset=0, gain=None, **kwargs):
         """
         This method is used to add a sound to the animation.
diff --git a/manim/scene/scene_file_writer.py b/manim/scene/scene_file_writer.py
index 2b04e22928..0a30c6c646 100644
--- a/manim/scene/scene_file_writer.py
+++ b/manim/scene/scene_file_writer.py
@@ -12,6 +12,7 @@
 from typing import Any, Dict, List, Optional
 
 import numpy as np
+import srt
 from PIL import Image
 from pydub import AudioSegment
 
@@ -71,6 +72,7 @@ def __init__(self, renderer, scene_name, **kwargs):
         self.init_audio()
         self.frame_count = 0
         self.partial_movie_files: List[str] = []
+        self.subcaptions: List[srt.Subtitle] = []
         self.sections: List[Section] = []
         # first section gets automatically created for convenience
         # if you need the first section to be skipped, add a first section by hand, it will replace this one
@@ -449,6 +451,8 @@ def finish(self):
         elif is_png_format() and not config["dry_run"]:
             target_dir, _ = os.path.splitext(self.image_file_path)
             logger.info("\n%i images ready at %s\n", self.frame_count, target_dir)
+        if self.subcaptions:
+            self.write_subcaption_file()
 
     def open_movie_pipe(self, file_path=None):
         """
@@ -712,6 +716,13 @@ def flush_cache_directory(self):
             {"par_dir": self.partial_movie_directory},
         )
 
+    def write_subcaption_file(self):
+        """Writes the subcaption file."""
+        subcaption_file = Path(config.output_file).with_suffix(".srt")
+        with open(subcaption_file, "w") as f:
+            f.write(srt.compose(self.subcaptions))
+        logger.info(f"Subcaption file has been written as {subcaption_file}")
+
     def print_file_ready_message(self, file_path):
         """Prints the "File Ready" message to STDOUT."""
         config["output_file"] = file_path
diff --git a/poetry.lock b/poetry.lock
index 23efeb5f96..554956804e 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -1977,6 +1977,14 @@ python-versions = ">=3.6"
 [package.dependencies]
 sphinx = ">=2.0"
 
+[[package]]
+name = "srt"
+version = "3.5.0"
+description = "A tiny library for parsing, modifying, and composing SRT files."
+category = "main"
+optional = false
+python-versions = ">=2.7"
+
 [[package]]
 name = "terminado"
 version = "0.12.1"
@@ -2163,7 +2171,7 @@ webgl_renderer = ["grpcio", "grpcio-tools"]
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.7"
-content-hash = "9edc33b7502b40d0b2bcd06d48e40ec743d468d45d4f38178bd009a4aa596d10"
+content-hash = "4c12c12b7be3fb2cb13d5b511fe613a92b53b458ad58ed3505766019be3d93ef"
 
 [metadata.files]
 alabaster = [
@@ -3589,6 +3597,9 @@ sphinxext-opengraph = [
     {file = "sphinxext-opengraph-0.4.2.tar.gz", hash = "sha256:5da1a6eaab25e377b841f506ef7d7c44b539428f3d6094ccc16d1dfd3ef65501"},
     {file = "sphinxext_opengraph-0.4.2-py3-none-any.whl", hash = "sha256:a51f2604f9a5b6c0d25d3a88e694d5c02e20812dc0e482adf96c8628f9109357"},
 ]
+srt = [
+    {file = "srt-3.5.0.tar.gz", hash = "sha256:9378e67fcc7c110eca7ec12bdb17a174085f67e3b371f7e586a4c68a0dda245b"},
+]
 terminado = [
     {file = "terminado-0.12.1-py3-none-any.whl", hash = "sha256:09fdde344324a1c9c6e610ee4ca165c4bb7f5bbf982fceeeb38998a988ef8452"},
     {file = "terminado-0.12.1.tar.gz", hash = "sha256:b20fd93cc57c1678c799799d117874367cc07a3d2d55be95205b1a88fa08393f"},
diff --git a/pyproject.toml b/pyproject.toml
index a36e4a9bd5..fe3a3b4f10 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -55,6 +55,7 @@ dearpygui = { version = "^0.8", optional = true }
 screeninfo = "^0.6.7"
 skia-pathops = "^0.7.0"
 isosurfaces = "0.1.0"
+srt = "^3.5.0"
 
 [tool.poetry.extras]
 webgl_renderer = ["grpcio","grpcio-tools"]
diff --git a/tests/test_scene.py b/tests/test_scene.py
index f842864dfb..be9f3223c5 100644
--- a/tests/test_scene.py
+++ b/tests/test_scene.py
@@ -1,6 +1,9 @@
+import datetime
+
 import pytest
 
 from manim import Circle, FadeIn, Mobject, Scene, Square, config, tempconfig
+from manim.animation.animation import Wait
 
 
 def test_scene_add_remove():
@@ -46,3 +49,25 @@ def test_scene_time():
         scene.renderer._original_skipping_status = True
         scene.play(FadeIn(Square()), run_time=5)  # this animation gets skipped.
         assert pytest.approx(scene.renderer.time) == 7.5
+
+
+def test_subcaption():
+    with tempconfig({"dry_run": True}):
+        scene = Scene()
+        scene.add_subcaption("Testing add_subcaption", duration=1, offset=0)
+        scene.wait()
+        scene.play(
+            Wait(),
+            run_time=2,
+            subcaption="Testing Scene.play subcaption interface",
+            subcaption_duration=1.5,
+            subcaption_offset=0.5,
+        )
+        subcaptions = scene.renderer.file_writer.subcaptions
+        assert len(subcaptions) == 2
+        assert subcaptions[0].start == datetime.timedelta(seconds=0)
+        assert subcaptions[0].end == datetime.timedelta(seconds=1)
+        assert subcaptions[0].content == "Testing add_subcaption"
+        assert subcaptions[1].start == datetime.timedelta(seconds=1.5)
+        assert subcaptions[1].end == datetime.timedelta(seconds=3)
+        assert subcaptions[1].content == "Testing Scene.play subcaption interface"


Code review comments:
diff_hunk: @@ -899,8 +902,46 @@ def get_run_time(self, animations):
         else:
             return np.max([animation.run_time for animation in animations])
 
-    def play(self, *args, **kwargs):
+    def play(
+        self,
+        *args,
+        subcaption=None,
+        subcaption_duration=None,
+        subcaption_offset=0,
+        **kwargs,
+    ):
+        r"""Plays an animation in this scene.
+
+        Parameters
+        ----------
+
+        args
+            Animations to be played.
+        subcaption
+            The content of the external subcaption that should
+            be added during the animation.
+        subcaption_duration
+            The duration for which the specified subcaption is
+            added. If ``None`` (the default), the run time of the
+            animation is taken.
+        subcaption_offset
+            An offset (in seconds) for the start time of the
+            added subcaption.
+        kwargs
+            All other keywords are passed to the renderer.
+
+        """
+        start_time = self.renderer.time
         self.renderer.play(self, *args, **kwargs)
+        run_time = self.renderer.time - start_time
+        if subcaption:
+            if subcaption_duration is None:
+                subcaption_duration = run_time
+            self.add_subcaption(
+                content=subcaption,
+                duration=subcaption_duration,
+                offset=-run_time + subcaption_offset,
path: manim/scene/scene.py
text: @user1:
Why are you subtracting the `run_time` here? And if this is intended, we have two different `offset`s, which is rather confusing, isn't it?

@author:
It is intended: the subcaption is added after the animation has been played, which means that the time stamp (`self.renderer.time`) has already been updated to the time after the animation has been played. To get the subcaption to be displayed when the animation starts playing, its starting time needs to be shifted by the run time of the animation.

What do you mean with two different `offset`s?

@user1:
> What do you mean with two different `offsets`?

We have both a `subcaption_offset` and an `offset` here. I know this is quite picky but it still confused me.

@author:
Ah – I see! I think it makes sense for the keyword argument for `Scene.play` to be called `subcaption_offset`, to explicitly make clear that we are talking about the offset of subcaptions there -- but when using `Scene.add_subcaption`, naming the corresponding parameter `subcaption_offset` is not really necessary I believe.

I could live with refactoring it, if you'd strongly prefer that. :-)