Repository: ManimCommunity/manim

Issue:
Feature request: Subtitle timings
## Description of proposed feature
<!-- Add a clear and concise description of the new feature, including a motivation: why do you think this will be useful? -->

It would be fairly useful to have animation options that aren't for actually displaying anything, but for marking the timestamps in an animation with text that can be used to export subtitles.

Since a lot of videos made in manim are intended to include additional voiceover, this would make timing subtitles, or at least adding subtitles for the intended script to videos, a lot easier. Even if the actual voiceover doesn't 100% match the script, it's a lot easier to tweak the words and the timing than to make them from scratch.

Alternatively, folks who don't do voiceover but want to add subtitles as an alternative can still do so.

## How can the new feature be used?
<!-- If possible, illustrate how this new feature could be used. -->

I'm imagining an implementation similar to:

1. Animations can set the subtitle text or clear it (e.g. end the latest subtitle), which would happen as soon as the animation starts.
2. There is a dedicated animation `Subtitle(text)` which sets the subtitle to the given string, or clears it if passed `None`.
3. Alternatively, there could be dedicated `set_subtitle` and `clear_subtitle` methods since setting a subtitle takes zero time.

This way, you can still use the existing animation system to leverage adding subtitles, e.g. `wait` still works as expected and adding a subtitle animation alongside other animations would set it as soon as those animations are started.  

## Additional comments
<!-- Add further context that you think might be relevant. -->

The output would probably make the most sense as an srt file since the videos are output as MP4. This is a relatively simple format where each line includes a start/end timestamp for a subtitle and the text. The lines could be written out whenever a subtitle is cleared or changed, or the video ends.


Discussion:
Hello and thanks for the suggestion!

I must say that my position is a "soft no". I am not an expert, but I believe there are ways of adding subtitles that are external to how the video is rendered. For example, in most playback software that accepts subtitles, you have the option to turn subtitles off and on. However, in what you propose, there is no way of turning subtitles off and on once the video is rendered with subtitles. So if anyone is really serious about adding subtitles, I would encourage them to look for other ways of adding them after the video is rendered.

However, if others think this would be useful, I could see this being a manim plugin.
To clarify, this would specifically be for closed subtitles (i.e. not directly rendered) rather than open subtitles (rendered onto the video itself). Since it'd be storing the subtitles in a separate file the main benefit is to export timing information from the rendered animation.
Oh okay, that makes more sense. 

If I'm not mistaken, this would depend on the underlying renderer. For example, I would assume there is a way of telling ffmpeg to add closed captions and/or telling ffmpeg about the timing information for the closed captions. If that's the case, then this would probably be more than a plugin as it would have to change how the rendering step happens... 


(I also suggest in the post that it could just output as a .srt file, which is just a text file and super easy to do by hand. Most video players nowadays will automatically read this if it's in the same folder.)

I believe that since the way manim generates animations is each animation is a separate video, then all we'd have to do is get the times for the renders to get the timing for the subtitles. But I could be mistaken.
Is that all that needs to be done? Or does the actual mp4 file need to include some information as well? (Sorry I didn't see that part first!)
No worries! I am also guilty of commenting without reading everything first. :p 

And… yeah, pretty much. There are more complicated video containers like mkv that can include the subtitles with the video, but in a lot of cases it's sufficient to just output an SRT file next to the video with the same name and the extension replaced with .srt.

I wasn't 100% right on the timings just being individual lines, but it's still extremely straightforward. Since it's just text you don't need any fancy code to generate it, just a few print statements. See this page on wikipedia: https://en.wikipedia.org/wiki/SubRip#File_format
In that case, then I do think we can make this part of the core library. 

I agree with you that the interface should look like an Animation.

The part that needs a bit of thinking would be the actual writing of the file. I suggest we create a new class SRTWriter or CCWriter or something like that. And all of the functionality can be imported as `from manim import captions` and then do `captions.clear()`, or something similar.

Resolving patch:
diff --git a/manim/scene/scene.py b/manim/scene/scene.py
--- a/manim/scene/scene.py
+++ b/manim/scene/scene.py
@@ -3,6 +3,7 @@
 __all__ = ["Scene"]
 
 import copy
+import datetime
 import inspect
 import platform
 import random
@@ -12,6 +13,8 @@
 from queue import Queue
 from typing import List, Optional
 
+import srt
+
 from manim.scene.section import DefaultSectionType
 
 try:
@@ -899,8 +902,50 @@ def get_run_time(self, animations):
         else:
             return np.max([animation.run_time for animation in animations])
 
-    def play(self, *args, **kwargs):
+    def play(
+        self,
+        *args,
+        subcaption=None,
+        subcaption_duration=None,
+        subcaption_offset=0,
+        **kwargs,
+    ):
+        r"""Plays an animation in this scene.
+
+        Parameters
+        ----------
+
+        args
+            Animations to be played.
+        subcaption
+            The content of the external subcaption that should
+            be added during the animation.
+        subcaption_duration
+            The duration for which the specified subcaption is
+            added. If ``None`` (the default), the run time of the
+            animation is taken.
+        subcaption_offset
+            An offset (in seconds) for the start time of the
+            added subcaption.
+        kwargs
+            All other keywords are passed to the renderer.
+
+        """
+        start_time = self.renderer.time
         self.renderer.play(self, *args, **kwargs)
+        run_time = self.renderer.time - start_time
+        if subcaption:
+            if subcaption_duration is None:
+                subcaption_duration = run_time
+            # The start of the subcaption needs to be offset by the
+            # run_time of the animation because it is added after
+            # the animation has already been played (and Scene.renderer.time
+            # has already been updated).
+            self.add_subcaption(
+                content=subcaption,
+                duration=subcaption_duration,
+                offset=-run_time + subcaption_offset,
+            )
 
     def wait(self, duration=DEFAULT_WAIT_TIME, stop_condition=None):
         self.play(Wait(run_time=duration, stop_condition=stop_condition))
@@ -1242,6 +1287,55 @@ def update_to_time(self, t):
         self.update_meshes(dt)
         self.update_self(dt)
 
+    def add_subcaption(
+        self, content: str, duration: float = 1, offset: float = 0
+    ) -> None:
+        r"""Adds an entry in the corresponding subcaption file
+        at the current time stamp.
+
+        The current time stamp is obtained from ``Scene.renderer.time``.
+
+        Parameters
+        ----------
+
+        content
+            The subcaption content.
+        duration
+            The duration (in seconds) for which the subcaption is shown.
+        offset
+            This offset (in seconds) is added to the starting time stamp
+            of the subcaption.
+
+        Examples
+        --------
+
+        This example illustrates both possibilities for adding
+        subcaptions to Manimations::
+
+            class SubcaptionExample(Scene):
+                def construct(self):
+                    square = Square()
+                    circle = Circle()
+
+                    # first option: via the add_subcaption method
+                    self.add_subcaption("Hello square!", duration=1)
+                    self.play(Create(square))
+
+                    # second option: within the call to Scene.play
+                    self.play(
+                        Transform(square, circle),
+                        subcaption="The square transforms."
+                    )
+
+        """
+        subtitle = srt.Subtitle(
+            index=len(self.renderer.file_writer.subcaptions),
+            content=content,
+            start=datetime.timedelta(seconds=self.renderer.time + offset),
+            end=datetime.timedelta(seconds=self.renderer.time + offset + duration),
+        )
+        self.renderer.file_writer.subcaptions.append(subtitle)
+
     def add_sound(self, sound_file, time_offset=0, gain=None, **kwargs):
         """
         This method is used to add a sound to the animation.
diff --git a/manim/scene/scene_file_writer.py b/manim/scene/scene_file_writer.py
--- a/manim/scene/scene_file_writer.py
+++ b/manim/scene/scene_file_writer.py
@@ -12,6 +12,7 @@
 from typing import Any, Dict, List, Optional
 
 import numpy as np
+import srt
 from PIL import Image
 from pydub import AudioSegment
 
@@ -71,6 +72,7 @@ def __init__(self, renderer, scene_name, **kwargs):
         self.init_audio()
         self.frame_count = 0
         self.partial_movie_files: List[str] = []
+        self.subcaptions: List[srt.Subtitle] = []
         self.sections: List[Section] = []
         # first section gets automatically created for convenience
         # if you need the first section to be skipped, add a first section by hand, it will replace this one
@@ -449,6 +451,8 @@ def finish(self):
         elif is_png_format() and not config["dry_run"]:
             target_dir, _ = os.path.splitext(self.image_file_path)
             logger.info("\n%i images ready at %s\n", self.frame_count, target_dir)
+        if self.subcaptions:
+            self.write_subcaption_file()
 
     def open_movie_pipe(self, file_path=None):
         """
@@ -712,6 +716,13 @@ def flush_cache_directory(self):
             {"par_dir": self.partial_movie_directory},
         )
 
+    def write_subcaption_file(self):
+        """Writes the subcaption file."""
+        subcaption_file = Path(config.output_file).with_suffix(".srt")
+        with open(subcaption_file, "w") as f:
+            f.write(srt.compose(self.subcaptions))
+        logger.info(f"Subcaption file has been written as {subcaption_file}")
+
     def print_file_ready_message(self, file_path):
         """Prints the "File Ready" message to STDOUT."""
         config["output_file"] = file_path


Test patch:
diff --git a/tests/test_scene.py b/tests/test_scene.py
--- a/tests/test_scene.py
+++ b/tests/test_scene.py
@@ -1,6 +1,9 @@
+import datetime
+
 import pytest
 
 from manim import Circle, FadeIn, Mobject, Scene, Square, config, tempconfig
+from manim.animation.animation import Wait
 
 
 def test_scene_add_remove():
@@ -46,3 +49,25 @@ def test_scene_time():
         scene.renderer._original_skipping_status = True
         scene.play(FadeIn(Square()), run_time=5)  # this animation gets skipped.
         assert pytest.approx(scene.renderer.time) == 7.5
+
+
+def test_subcaption():
+    with tempconfig({"dry_run": True}):
+        scene = Scene()
+        scene.add_subcaption("Testing add_subcaption", duration=1, offset=0)
+        scene.wait()
+        scene.play(
+            Wait(),
+            run_time=2,
+            subcaption="Testing Scene.play subcaption interface",
+            subcaption_duration=1.5,
+            subcaption_offset=0.5,
+        )
+        subcaptions = scene.renderer.file_writer.subcaptions
+        assert len(subcaptions) == 2
+        assert subcaptions[0].start == datetime.timedelta(seconds=0)
+        assert subcaptions[0].end == datetime.timedelta(seconds=1)
+        assert subcaptions[0].content == "Testing add_subcaption"
+        assert subcaptions[1].start == datetime.timedelta(seconds=1.5)
+        assert subcaptions[1].end == datetime.timedelta(seconds=3)
+        assert subcaptions[1].content == "Testing Scene.play subcaption interface"