gaspard7/wiki_gpac huggingface dataset:
How to stream 360 video content

Once the content is generated and stored on the server side, you can play it from the client side. The only required argument is the URL of the MPD file. In the case of 360/VR videos, it is possible to render only the viewport using `#VR` right after the URL. For example:

```
gpac -gui http://server:port/test.mpd
gpac -gui http://server:port/test.mpd#VR
```

Note that 360 rendering requires usage of the compositor filter (so use `gpac -gui` or `gpac -mp4c`).
Main Rendering Loop in Compositor Filter

The main loop to render the 360 video content (in `#VR` mode) is roughly as follows.

```
gf_sc_draw_frame
   gf_sc_render_frame
      gf_sc_draw_scene
         visual_draw_frame
            visual_3d_draw_frame
               visual_3d_setup
               visual_3d_setup_traversing_state
               visual_3d_setup_clipper
               visual_3d_init_shaders
               for each view
                  visual_3d_draw_node // for root
                  for each scenegraph
                     gf_sc_traverse_subscene
```

Since HEVC supports tiling, putting the tiles back to form the whole frame is done inside the decoder. Therefore, the output of the decoder is the projected 2D video (e.g. equirectangular). The decoder puts each decoded frame as a composition unit (CU) inside the composition buffer (CB). The compositor takes out CUs one at a time (by calling `gf_sc_texture_update_frame()` function) and tries to push the texture to the graphics device (by calling `gf_sc_texture_push_image()` function).


The conversion from YUV (YU12) to RGB is done with the help of a fragment shader. `visual_3d_shader_with_flags()` function is used to load the shaders.


The video output interface (DirectX, SDL, or Raw) is loaded from a module in `gf_sc_create()` function and signaled to setup in `gf_sc_reconfig_task()` function.
Scene Graph in #VR Mode

```
TAG_MPEG4_OrderedGroup
   |
   |--- TAG_MPEG4_Background2D
   |--- TAG_MPEG4_Viewpoint
   |--- TAG_MPEG4_NavigationInfo
   |--- TAG_MPEG4_Sound2D
   |       |
   |       |--- Source: TAG_MPEG4_AudioClip
   |
   |--- TAG_MPEG4_Transform2D
           |
           |--- TAG_MPEG4_TouchSensor
           |--- TAG_MPEG4_Transform2D
                   |
                   |--- TAG_MPEG4_Shape
                           |
                           |--- Appearance: TAG_MPEG4_Appearance
                           |                   |
                           |                   |--- Texture: TAG_MPEG4_MovieTexture
                           |
                           |--- Geometry: TAG_MPEG4_Sphere
```
DASH Basics: MPD and Segments

Let's quickly summarize how a DASH content is made of:

*   MPD: an XML document describing where the various media resources present in the content are located. The media resources can be single-media (for example, a video-only MP4 file) or a multiplexed set of streams (for example an AV MPEG-2 Transport Stream). Streams can be scalable (such as SVC) but we won't go into such details as GPAC doesn't support advanced description of scalable streams in DASH. Some media resources may exist in different versions, for example different bitrate or language or resolutions. In DASH, such a "version" of the stream is called a representation, and all representations are grouped together in an AdaptationSet.
*   segment: a continuous part of a media resource. The segment is the smallest part of the media that can be located in the MPD. What a segment exactly contains depends on the underlying media format of the content.
*   subsegment: a continuous part of a segment, or of a subsegment.
*   sidx: short name for SegmentIndexBox, this is an ISOBMF (MP4/3GP container) structure describing a segment by giving its earliest presentation time, how the segment is further divided into subsegments, random access points locations (byte offset) and timing in the segment payload. The goal of the SIDX is to build an index of the segment at a given granularity to simplify trick modes (seeking, fast-forward, fast-rewind, ...).

There are several ways to refer to a segment in an MPD. If the file is made of a single segment (-single-segment option for ISOBMF), one will likely use SegmentBase element. If a file is made of several segments, each segment will be identified by the SegmentList syntax in the MPD, using byte ranges. For other cases, we need to instruct MP4Box how to refer to segments (and how to store them as well). The following switches are defined:

*   \-segment-ext EXT: tells MP4Box to generate segments with EXT extension (by default **m4s** for ISOBMF and or **ts** for MPEG-2)
*   \-segment-name NAME: tells MP4Box to generate each segment in a dedicated file, called NAME%d.EXT. NAME can also have **%s** in it, in which case %s will be replaced by the name of the file being dashed without folder indication and extension. By default, such segments will be stored using the **SegmentList** syntax in the MPD.
*   \-url-template: if set when generating segments in different files, the segments will be referred to using the **SegmentTemplate** syntax in the MPD.
ISO Base Media File Format

For content based on ISOBMF (ISO/IEC 14496-12), MP4Box can be used to cut files into DASH segments. Before going any further, some definitions will be needed:

*   segment: for ISOBMF, a segment is a consecutive set of movie fragments. Each movie fragment is composed of a moof box followed by mdat box(es), and all data addressing in the mdat(s) are done using relative offsets in the moof.
*   subsegment: a part of a segment, made of a consecutive set of movie fragments. A subsegment can be further divided in subsegments, until only a single movie fragment per subsegment is present.

With that in mind, we can generate DASH content by playing with the following MP4Box parameters:

*   \-dash X: produce segments of roughly X milliseconds.
*   \-frag Y: use movie fragments of roughly Y milliseconds. By default, fragments duration is 500 milliseconds.
*   \-rap:  attempts to cut segments so that they all start with an access point (IDR, I-frame or beginning of a gradual decoding refresh for example).
*   \-subsegs-per-sidx N: specifies how many subsegments per sidx we would like. This only covers the first level of segment splitting (MP4Box doesn't handle subsegments subdivision into subsegments). Important values are:

> *   <0: disable: `sidx will not be produced`
> *   0: `a single sidx box is used for the entire segment, and each subsegment is made of a single movie fragment (i.e., there will be X/Y subsegments in sidx). This is the default value.`
> *   \>0: `produces` X/Y/N `subsegments` `referenced in the first sidx.`

*   \-daisy-chain: this is only used when producing multiple subsegments per segment (-subsegs-per-sidx). If specified, subsegments will be described in SIDX in which the last entry (subsegment) points to the next SIDX. Otherwise, multiple SIDXs will be stored in a hierarchical way, with the first SIDX pointing to each SIDX of the subsegments.
*    -single-segment: special mode indicating that the file should be segmented as one single segment. In that case, the dash duration X becomes the subsegment duration, and a single sidx is produced before any movie fragment.

Now let's see an example. Dashing a file with 10 seconds, rap-aligned segments with a fragment duration (i.e. subsegment duration since we don't subdivide the SIDX)  of 1 sec:

```
MP4Box -dash 10000 -frag 1000 -rap test.mp4
```

The same with a separated segment using template addressing, and 5 subsegments per segments:

```
MP4Box -dash 10000 -frag 1000 -rap -segment-name myDash
          -subsegs-per-sidx 5 -url-template test.mp4
```

Generating an onDemand profile DASH file (single segment) is just as simple:

```
MP4Box -dash 10000 -frag 1000 -rap -single-segment test.mp4
```
MPEG-2 TS

MP4Box can also be used to segment MPEG-2 TS files. The same options as the ISOBMF case are used, with the following restrictions:

*   \-single-segment, -frag, -subsegs-per-sidx and -daisy-chain are ignored
*   \-rap splits at the PAT preceding the RAP found, but does not re-multiplex the TS to make sure it begins with the RAP

For example, splitting a TS in 10 seconds segments can be done with

```
 MP4Box -dash 10000 -url-template -segment-name segments test.ts
```

Also note that it is possible to use MP4Box to translate an existing m3u8 (Apple HLS) to a compliant MPD, using the -mpd switch:

```
MP4Box -mpd test.mpd [-url-template] [http://...]myfile.m3u8
```
Multiple Representations

You now know how to create a compliant DASH content from a given file, but what about the 'A for Adaptive' in DASH ? At first thought it would just be enough to let you with a bunch of MPD and a good XSLT, to produce your final MPD (which you will have to do anyway I believe). However, there are some tricks in the segment generation itself that cannot be easily done. The most problematic thing is that, when building ISOBMF files designed for bitstream switching, the initial bootstrap of the DASH session (i.e. moov and co) must contain all sample descriptions used in all representations of the media. Therefore, we will need MP4Box here to generate files with correct sample descriptions, and segments with correct sampleDescriptionIndex Although this might look a bit complex, the process is itself quite simple; assuming you have encoded file1, .., fileN version of your movie, you can generate a nice adaptive MPD as follows: MP4Box -dash 10000 \[other options as seen above\] -out final.mpd file1.mp4 ... fileN.mp4 This works for both ISOBMF and TS files. Additionally, you don't want segments to be overridden when generating segments in dedicated files for each representation. MP4Box gives you the possibility to format the name of the segment using -segment-name:

```
MP4Box -dash 10000 -segment-name mysegs_%s -url-template
         -out final.mpd file1.mp4 ... fileN.mp
```
Playback

You can test DASH playback using [gpac player](player). The [DASH/HLS reader](dashin) supports a good part of the technology:

*   playback from an HTTP server or from local storage for test purposes.
*   Media segments based on TS and ISOBMF
*   independent component download (one adaptation set for audio, one for video)
*   most MPD syntax
*   bitstream switching
*   manual quality switching by using ctrl+h and ctrl+l
*   multiple algorithms for automatic quality switching when playing HTTP urls
Encoding your video

Thanks to our friends at [TUT](http://ultravideo.cs.tut.fi/#main), recent updates to the open-source [Kvazaar](https://github.com/ultravideo/kvazaar) encoder allow anyone to encode HEVC with motion constrained tiling. In order to perform tile-based adaptation in DASH, the tiles must be constrained in motion prediction, and each tile has to be encapsulated in one slice (typical configuration). 

Putting all your tiles in the same slice will not work, since the slice NALU(s) will likely not be split in different tracks, thereby forbidding individual downloads of tiles in DASH. In terms of Kvazaar encoding options, this means that you have to add to your usual command lines the following;

```
--tiles 3x3 --slices tiles --mv-constraint frametilemargin
```

This will instruct the decoder to use 3x3 uniform tiling for the encoding, with a single tile per slice in motion constrained mode. There are options in the Kvazaar encoder allowing you to specify a non-uniform tile grid, check the encoder usage/help.
Packaging your video

You will need to package your raw HEVC bitstream with MP4Box, instructing it to split the tiles of each frame in different tracks, using the `:split_tiles` import switch:

```
MP4Box -add video_tiled.hvc:split_tiles -new video_tiled.mp4
```
Or using gpac [tilesplit](tilesplit) filter:
```
gpac -i video_tiled.hvc tilesplit -o video_tiled.mp4
```

For a NxM tiling, the resulting file will contain 1+NxM video tracks, a "base tile track" of type hvc2/hev2 containing parameter sets and/or SEI messages, if any, and NxM tile track of type hvt1 containing tile data.

You can check the motion constrained is well respected by removing a tile track from the file:

```
MP4Box -rem 3 video_tiled.mp4 -out test_tile_lost.mp4
```

You should then see a video with a clean cut, green area in place of the removed tile:

[![](https://gpac.io/files/2017/02/LostHEVCTile-300x177.png)](https://gpac.io/files/2017/02/LostHEVCTile.png)

You can also do this in one pass using gpac by combining  [tilesplit](tilesplit)  and  [tileagg](tileagg) filters:
```
gpac -i video_tiled.hvc tilesplit:tiledrop=2 tileagg -o test_tile_lost.mp4
```

__Note__
It is not recommended to discard the first tile track, as playback may be broken with most decoders (except OpenHEVC decoder).
DASHing your video

Your video can be DASHed as any other video with MP4Box, for example

```
MP4Box -dash 1000 -profile live -out dash_tiled.mpd video_tiled.mp4
```

or with gpac:
```
gpac -i video_tiled.mp4 -o dash_tiled.mpd
#alternative version splitting tiles and tiling from raw bitstream in one pass
gpac -i video_tiled.hvc tilesplit -o dash_tiled.mpd
```

The MPD will contain as many adaptation sets as there are tile tracks and tile base tracks in the source file. 

_NOTE:_ If you want to perform tile-based adaptation, you will need to encode several motion constrained tiled videos at various qualities, and DASH them as usual. Note however that GPAC support for HEVC tiles adaptation is for now limited to videos encoded with the same configuration (same SPS/PPS/VPS), so do not change your encoder settings too much except for the bitrate (especially, do not change the QP) and always work with the same source video resolution/bit depth and tiling grid.

The resulting MPD will contain as many adaptation sets are there are tile tracks and tile base tracks in one of the input video, and each tile adaptation set will contain representations for each quality specified. One such example is [available here](http://download.tsi.telecom-paristech.fr/gpac/SRD/srd_hevc/multi_rate/hevc_srd.mpd).

You can now playback your MPD using GPAC, and have fun with the different adaptation modes for tiling (through the GUI or through the configuration file).
Content Playback

In HEVC tile-based adaptation, the files corresponding to the different tiles after the DASH reader are automatically reassembled using the [tileagg](tileagg) filter into a single HEVC bitstream fed to the decoder. This means that a single video object is exposed to the player. The current DASH implementation for tiles creates a single download instance for the complete tile set, queuing segment requests one after the other (no parallel download of tiles, hence no low latency). This might change in the future.


In regular DASH+SRD mode (i.e., each video is an independently decodable stream), each video is exposed at the player and can be inspected independently in the GUI.
360 Video with HEVC tiling

If you want to experiment with DASH adaptation for 360 video in GPAC, you will need to follow the above guide using an equirectangular projection video, and play the resulting MPD as follows:

```
gpac -gui DASH_URL#VR
gpac -mp4c DASH_URL#VR
```

If you open a URL through the GUI, add `#VR` at the end of the URL. If you open a file through the GUI, long-click on the file and then select "View as 360".

In 360 navigation when the DASH [tile_mode](dashin#tile_mode) option is set to `none`, the DASH engine will select the tile quality based on the visible part of the 360 sphere.

_NOTE: projection signalling in MP4 files or DASH MPD is still a hot debate at MPEG, this will likely be updated in the near future. We may also introduce other projection formats, such as cube maps._
Gaze adaptation

The compositor can use gaze information to automatically decrease the quality of the tiles not below the gaze. 
The gaze information can be:

- emulated via mouse using [--sgaze](compositor#sgaze) option.
- signaled through filter updates on the [gazer_enabled](compositor#gazer_enabled) [gaze_x](compositor#gaze_x) [gaze_y](compositor#gaze_y) 

Gaze adaptation works for both 360 (ray hit to partial sphere) and regular 2D modes (mouse over rectangle).
Debugging the tile selection

The default behaviour of GPAC's adaptation logic is to select the lowest quality for tiles that are not completely visible. In order to debug the content (especially when qualities are visually too close), you can skip fetching and decoding the hidden tiles by using the option [--skip_lqt](dashin#skip_lqt).
In HEVC tile mode, this will result in having a video where only the selected tiles are updated, the rest of the image showing the content of the associated reference frames at last decode (this may vary depending on the decoder implementation, i.e. whether the reference frames memory are reset to 0 at each new GOP or not).

For regular (not HEVC tiles) tiling with a full coverage video and tiles, you can also use the compositor [tvtd](compositor#tvtd) to debug the tiling.
CMAF Generation

GPAC can be used to generate DASH or HLS following the CMAF specification.


CMAF defines two structural brands `cmfc`and `cmf2`  for ISOBMFF-segmented content. 
The `cmfc` brand constraints:

- some default values in ISOBMFF boxes
- a single media per file
- a single track fragment per movie fragment (`moof`)
- a single track run per track fragment
- presence of ColorInformationBox and PixelAspectRatioBox in video tracks
- presence of a `tfdt` box in each track fragment
- no mix of clear and encrypted samples in a CMAF fragment (<=> DASH segment or HLS chunk)
- Restricted edit lists consisting of media skip only


The `cmf2`brand further restrict the `cmfc` brand for video tracks:

- no edit list shall be used
-  negative composition offset (`trun` version 1) shall be used
- sample default values shall be repeated in each track fragment
 
Activating CMAF is simply done by specifying a global filter option:
```
MP4Box -dash 1000 -profile live -out live/manifest.mpd --cmaf=cmfc source.mp4
MP4Box -dash 1000 -profile live -out live/manifest.mpd --cmaf=cmf2 source.mp4
```
or using option inheritance:
```
MP4Box -dash 1000 -profile live -out live/manifest.mpd:cmaf=cmfc
```

You can set the option per MPD if you need to generate both CMFC and CMF2 profiles:
```
gpac -i source.mp4 -o live/manifest.mpd:cmaf=cmfc @1 -o live2/manifest.mpd:cmaf=cmf2 
```

Note that usage of CMAF packaging is independent of the DASH profile chosen.
CMAF Multiplexing Validation status

_Note: Currently GPAC only ensures conformance of the container, not the codec level._

Color information, when not present in input file, is extracted from VUI if present or defaults to BT709 otherwise.

Constraints on SPS (VUI, cropping) and PPS  in AVC or HEVC are not checked for the moment.

Constraints on CENC (AVC slice header clear, bytesOfProtectedData multiple of 16 in `cenc` )  are not checked, it is assumed the encryption configuration is CENC compatible. Be careful when setting up your encryption - see [encryption XML format](Common-Encryption) for more details.


TTML packaging should be aligned with CMAF specification (injection of a `mime` box with properly formatted mime type in the sample description).
Packaging your video

You will need to package your raw HEVC bitstream, rewriting each tile as a stand-alone HEVC bitstream using the [hevcsplit](hevcsplit) filter:


```
MP4Box -add video_tiled.hvc:@hevcsplit -new video_tiled.mp4
```
Or using gpac:
```
gpac -i video_tiled.hvc hevcsplit -o video_tiled.mp4
```

For a NxM tiling, the resulting file will contain NxM independent HEVC video tracks.

You can view the result using:
```
gpac -i video_tiled.mp4 hevcmerge vout
```


You can check the motion constrained is well respected by removing a tile track from the file:

```
MP4Box -rem 4 video_tiled.mp4 -out test_tile_lost.mp4
gpac -i test_tile_lost.mp4 hevcmerge vout
```
DASHing your video

Your video can be DASHed as any other video with MP4Box, for example

```
MP4Box -dash 1000 -profile live -out dash_tiled.mpd video_tiled.mp4
```

or with gpac:
```
gpac -i video_tiled.mp4 -o dash_tiled.mpd
#alternative version splitting tiles and tiling from raw bitstream in one pass
gpac -i video_tiled.hvc hevcsplit -o dash_tiled.mpd
```

The resulting MPD will contain as many adaptation sets are there are tile tracks in the input video(s), and each tile adaptation set will contain representations for each quality specified.
Content Playback

The logic of content playback is as follows:

- the MPD indicates SRD information and a GPAC extension for mergeable bitstream
- when the compositor is used, the [hevcmerge](hevcmerge) filter is automatically created to reassemble the streams
- otherwise (using vout), each PID is declared as an alternative to the other

A quick way to look at the output of the hevc merger filter is to run:
```
gpac -i dash.mpd hevcmerge vout
```

You will see the reconstructed tiles, very likely not in the order you would expect. You can comment out one of the qualities and check the result.
You can also set `--auto_switch=-1`, which will change qualities of tiles one after the other.

The reordering of the tiles into a proper texture cannot be done by the merger, due to the different tile sizes.
This reordering is done by the compositor filter (3D mode required), remapping texture coordinates to restore the proper layout:

```
#2D playabck without GUI
gpac -mp4c dash.mpd  

#2D playabck with GUI
gpac -gui dash.mpd

#VR playabck without GUI
gpac -mp4c dash.mpd#VR  

#VR playabck with GUI
gpac -gui dash.mpd#VR
```

The GUI will only indicate a single visual object, as all input streams are merged back to a single HEVC bitstream.

Tile adaptation logic is similar to the one using HEVC tile tracks.

The option [--skip_lqt](dashin#skip_lqt) is not supported in this mode.
HLS with one file per segment

Generating an HLS with one file per media segment is equivalent to dashing under the live profile.

```
MP4Box -dash 1000 -profile live -out res/live.m3u8 source1 source2 
gpac -i source1 -i source2 -o res/live.m3u8:profile=live
```


This will by default generate ISOBMFF / fmp4 files.  To generate MPEG-2 TS files, use:  
```gpac -i source1 -i source2 -o res/live.m3u8:profile=live:muxtype=ts```

Note that gpac will by default generate non-multiplexed outputs. To generate a multiplexed file, you need to flag inputs as belonging to the same quality , or `Representation` in DASH terminology:  
```gpac -i video:#Representation=main -i audio:#Representation=main -o res/live.m3u8:profile=live:muxtype=ts```
Specifying manifest names

The child playlist names are by default the manifest (master playlist) radical suffixed with `_1`,  `_2`  ...
This can be changed by using the PID property `HLSPL` :

```gpac -i source1:#HLSPL=video.m3u8 -i source2:#HLSPL=audio.m3u8 -o res/live.m3u8```

This will generate `live.m3u8`, `video.m3u8` and `audio.m3u8`