gaspard7/wiki_gpac huggingface dataset:
Advanced reception of data

Assume you want gpac to grab from a TCP socket some  AVC|H264 content which could be sent sequentially by sender(s). typically, an encoder processing a playlist and sending each result to a socket. Having the encoder create the socket will result in closing the socket at each file, hence closing gpac after the first file, which is not the goal. We need to
- create the socket by asking the [socket](sockin) input to create it for us using [listen](sockin) option
- tell the socket input to keep the session alive even when socket connection close messages are received, using [ka](sockin) option


```gpac -i tcp://127.0.0.1:1234/:ext=avc:listen:ka -o test.mp4```

This will open the socket, creating it if not found, and we can now start sending data on the socket. 

__Discussion__
The above command will run forever, since socket connection close messages are ignored. You can abort the session using `ctr+c` and ask to flush pending data to disk. If you have enabled prompt interactivity in gpac [-k](gpac_general), simply press `q`.
There is currently no way to signal from the sender that the session should be closed, we might add this feature in the near future.
Output socket

Assume you have an application which consumes AVC|H264 in Annex B format from a UDP or TCP socket  `localhost:1234`. You can direct GPAC output to this socket:
 
```
gpac -i source.mp4 -o udp://127.0.0.1:1234/:ext=avc
gpac -i source.mp4 -o tcp://127.0.0.1:1234/:ext=avc
```

The above command will open the file `source.mp4`, open/connect the socket  `127.0.0.1:1234` and send ([potentially transcoding](encoding)) the AVC|H264 data to this socket. Obviously you can use whatever input and processing:

```gpac -i source.yuv ffsws:osize=640x360 -o tcp://127.0.0.1:1234/:ext=avc```

This will grab the source, resize it, encode it and write it to the socket.


__Discussion__  
The  `ext` option is used to indicate the output format of the data carried on the socket. If SHALL be set. Any mux format used by GPAC can be used on output [sockets](sockout). 

You should however be very careful using containers (e.g. ISOBMFF) as a format, since some containers format will require the entire file to be produced to make it valid. For example, using ISOBMFF (`mp4`) as a data format will fail, since we need to patch either the `moov` or `mdat` box at the end of the multiplexing. However, if using fragmented ISOBMFF, it will be possible to push the mux result to a socket.
This problematic of patching previous blocks of data in a file is signaled by the property `DisableProgressive` on the `file` PID. At the current time, only the [file output](fout) filter can support PIDs with this property set.

Contrary to [output pipes](pout), output socket can work in server mode and use a keep-alive mode to keep the connection open until new clients come in - see the [socket output](sockout) filter help.
MP4Box vs gpac {: data-level="all" }

Following the introduction of the filter architecture and the gpac application, you may have a hard time choosing between MP4Box and gpac.
 
Before going any further, we assume:

-  you are familiar with [MP4Box](MP4Box)
-  you understand the principles of GPAC [filters](filters_general) and are somehow familiar with using the [gpac](gpac_general) application

We recommend that you quickly read the article on GPAC [re-architecture](Rearchitecture).
MP4Box, not gpac

There are many features of libgpac available in MP4Box only, and most of them will probably never be ported to the general filter architecture.

The things you can do with both MP4Box and gpac are:

- adding media tracks or image items to a __new__ ISOBMFF file
- extracting media track to raw formats
- fragmenting and DASHing a set of sources (ISOBMFF or not)
- encrypting or decrypting an ISOBMFF file
- splitting an ISOBMF file
- some XML dump operations (-dnal option of MP4Box)

File concatenation can also be done using MP4Box as well as with gpac, but they do not use the same code base:

- MP4Box only concatenates ISOBMFF files, potentially requiring temporary ISOBMFF import
- gpac can concatenate any source  (live or not) using the [flist](flist) filter.


All the rest is MP4Box-specific and unaware of the filter architecture.
Many operations in MP4Box, such as tagging, track info and timing modifications, are moreover optimized to reduce rewrite time, whereas gpac always completely rewrites the output.
MP4Box and filters

The track import syntax and the dashing syntax may be combined with filter declarations, as discussed [here](mp4box-filters).
They are however restricted as follows:

- for importing, the destination format is always ISOBMFF
- the filter chain described is fairly simple, going from source to destination ([mp4mx](mp4mx) or [dasher](dasher) filters) without any possible branch in-between.

For example:
```
source_vid -> rescale -> encode1 \
                      -> encode2 -> dasher
                      -> encode3 /
```

This cannot be described using MP4Box, this must be converted into something like:
```

source_vid -> rescale -> encode1 -> dasher
source_vid -> rescale -> encode2 /
source_vid -> rescale -> encode3 /
```

The syntax will be:
```
MP4Box -dash 2000 video.264:@ffsws:osize=1280x720:@enc:c=avc:fintra=2:b=4m:@@ffsws:osize=1280x720:@enc:c=avc:fintra=2:b=2m:@@ffsws:osize=1280x720:@enc:c=avc:fintra=2:b=1m -out result.mp4
```

This will work as expected, but with three rescaler filters doing all the same thing (and a heavy syntax).

Using gpac, this is much simpler
```
gpac -i video.264 ffsws:osize=1280x720 --fintra=2 enc:c=avc:b=4m @@1 enc:c=avc:b=2m @@1 enc:c=avc:b=1m -o result.mp4
```
Subtle differences in gpac

There are some subtle differences between MP4Box and gpac that may result in different behavior. Some of these are:

- Default interleaving in MP4Box is 500ms, while it is 1000ms in gpac
- `deps` option is set by default on mp4mx, you must set in on sources in gpac
- Handler name is not set by `-for-test` in gpac, for that you need to set `#HandlerName=...` property on the source
gpac, not MP4Box

You should use gpac rather than MP4Box in the following cases:

- sources are either live or simulate live (running forever)
- outputs are not local files: HTTP output, RTSP server, ROUTE output, etc...
- there are many filters manually specified in the pipeline
- monitoring of the filter session (`gpac -r`)
- the MP4Box pipeline combines several filter-session related aspects (`-dash`, `-split`, `-crypt`, `-add`, `-frag`)
- need for complex filter graph connections, reusing non-source filters in the graph
- the pipeline has multiple outputs (e.g. dash/hls and live TS multicast)
- the pipeline processes an unknown number of streams that must be redirected to different outputs, requiring URL templating.
- pipeline involves playback or composition, using [vout](vout), [aout](aout) or [compositor](compositor) filters
- input (live or not) inspection before ISOBMFF packaging
- remux input(s) without ISOBMFF conversion (e.g. audio + video to MPEG2 TS)
- distributed processing (using GSF or other formats through pipes, sockets or other means)
- any other complex use case we never thought of :)


Another good reason to use gpac to test a pipeline is when you plan on using GPAC bindings in [Python](python) or [NodeJS](nodejs) scripts.
- these bindings are indeed very close to gpac logic, and moving from a gpac command line to the bindings is trivial
- rewriting an MP4Box pipeline to the bindings will first require rewriting the MP4Box command line as a gpac one.


Check the [gpac one-liner](filters-oneliners) page for more examples !
Overview {:data-level="all"}

We discuss here how to use the AVMix filter to do offline or live video editing.
This filter is a [JS filter](jsf) and can be modified fairly simply.
The [AVMix scripts](https://github.com/gpac/testsuite/tree/master/media/avmix) in the gpac test suite are also a good source of examples.

Before reading any further, read the filter [documentation](avmix) !

The filter is playlist based, using a JSON format. We assume the name of your playlist is `avmix.json`, which is the default name expected by the filter.


The filter outputs raw (uncompressed) audio and video, which can be later used in a filter chain. 

To play the result:
```
#audio only
gpac avmix aout
#video only
gpac avmix vout
#audio-video
gpac -play avmix
gpac avmix aout vout
```

To encode the result:

`gpac avmix enc:c=avc:b=1m enc:c=aac:b=128k record.mp4`

By default, the filter will run in live mode, and will never stop. You can run in offline using [live=0](avmix#live) option, or using a `config` object in the playlist:
```
{"type": "config", "live": false}
```
Declaring media

To declare media for the mixer, you need to use a sequence object along with your media source. Typically:
```
[
{"seq":
 [
  { "src": [{"in": "media.mp4"}]}
 ]
}
]
```

Your media can be any URL supported by GPAC, not simply local files.

The default behavior is to play once the media, from its beginning to its end.  To play a given time range:

```
[
{ "id": "seq1", "seq":
 [
  { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5}
 ]
}
]
```

_Note: Media cannot play in loop, only sequences can._

This basic playlist can now be played, it will result in a possible resizing of the video and/or resampling of the audio.

In case your media comes from several sources (e.g., video in one file, audio in the other), you will have to specify as many inputs as needed:

```
[
{ "id": "seq1", "seq":
 [
  { "src": [
	{"in": "media_video.mp4"}
	{"in": "media_audio.mp3"}
  ], "start": 1, "stop": 5}
 ]
}
]
```

The formats and protocols used by the different inputs do not have to be the same.
The synchronization is done by assuming that the first media frame received corresponds to the desired start time, this might be further improved in the future.
Declaring a scene

In the previous example, a default scene was created when loading the playlist. We will now create our own scene to draw the video in a rounded rectangle, over a blue background:


```
[
{ "id": "seq1", "seq":
 [
  { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5}
 ]
},
{"id": "scene0", "js": "clear", "color": "cyan"}, 
{"id": "scene1", "sources": ["seq1"], "x": 0, "y": 0, "width": 60, "height": 60, "rx":10, "ry":"x"} 
]
```

Dimensions and coordinates are given in percent of the output size, the origin `{x=0,y=0}` of the scene being the center of the output frame. 

_Note_
A sequence not attached with a scene will not be visible nor played, even if active.

Now let's add :

- a logo
- a bottom rectangle with a gradient
- some text

```
[
{ "id": "seq1", "seq":
 [
  { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5}
 ]
},
{"js": "clear", "color": "cyan"}, 
{"sources": ["seq1"], "x": -5, "y": 5, "width": 90, "height": 90, "rx":10, "ry":"x"},
{"x": 45, "y": 40, "width": 10, "height": "width", "img":"logo.png", "alpha":0.5},
{"x": 0, "y": -45, "width": 100, "height": 10, "fill":"linear", "grad_p": [0, 0.5, 1], "grad_c": ["none", "white", "red"]},
{"x": 0, "y": -45, "width": 100, "height": 10, "text": ["Super Text"], "size": 50, "align": "center", "fill": "black"}
]
```

For more information on the default `shape` scene and its multiple capabilities, see [gpac -h avmix:shape](avmix#scene-shape).
 
In the following examples, we always use [relative coordinates system](avmix#coordinate-system), but you can also specify coordinates and dimensions in pixels using [`units` property](avmix#common-properties-for-group-and-scene-objects).
Animating a scene

Scenes can be animated through timer objects providing value interpolation instructions. A timer provides:

- a start time, stop time and a loop count
- a duration for the interpolation period
- a set of animation values and their targets

In order to use a timer with a scene, we need a scene ID.

Let's move our logo and blink our text !

```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ]},
{"js": "clear", "color": "cyan"},
{"sources": ["seq1"], "x": -5, "y": 5, "width": 90, "height": 90, "rx":10, "ry":"x"},
{"id": "logo", "x": 45, "y": 40, "width": 10, "height": "width", "img":"logo.png", "alpha":0.5},
{"x": 0, "y": -45, "width": 100, "height": 10, "fill":"linear", "grad_p": [0, 0.5, 1], "grad_c": ["none", "white", "red"]},
{"id": "text", "x": 0, "y": -45, "width": 100, "height": 10, "text": ["Super Text"], "size": 50, "align": "center", "fill": "black"},

{"start": 0, "loop": true, "dur": 2, "keys": [0, 0.5, 1], "anims": [
    {"values": [true, false, true], "targets": ["text@active"], "mode": "discrete"},
    {"values": [40, -30, 40], "targets": ["logo@y"], "mode": "interp = interp*interp*interp;"}
 ]
}
]
```

We blink the text by simply activating or deactivating the associated scene, using discrete interpolation. This avoids invalidating the underlying vector graphics data.
We also use here a pre-interpolation custom function to modify the interpolation factor rather than the simpler `linear` mode. 
If you need to animate multiple targets with the same value, simply add more targets, e.g. `"targets": ["logo@x", "logo@y"]`.
Media Sequences

### Transitions

You will at some point need to chain some videos. AVMix handles this through `sequence` objects describing how sources are to be chained.
Sequences are designed to:

- take care of media prefetching to reduce loading times
- perform transitions between sources, activating / prefetching based on the desired transition duration

For example, to create a sequence repeated twice with two source and a cross-fade transition:

```
[
{ "id": "seq1", "loop": 1, "seq":
 [
  { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5},
  { "src": [{"in": "media2.mp4"}], "start": 600, "stop": 660}
 ],
 "transition": { "type": "mix", "dur": 2}
},
{"id": "scene0", "js": "clear", "color": "cyan"}, 
{"id": "scene1", "sources": ["seq1"], "width": 80, "height": 80, "rx":10, "ry":"x"} 
]
```

### Mixing

There are many cases where one needs to mix videos running in parallel with a given amount of mix effect.
AVMix handles this by allowing scenes to use more than one sequence as input, and mix the videos from these sequences.

_Note: Currently, defined scenes only support 0, 1 or 2 input sequences_

This is done at scene declaration through:

- a `mix` object, describing a transition
- a `mix_ratio` property, describing the transition ratio

The `mix_ratio` property can of course be animated:  
```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "seq2", "seq": [ { "src": [{"in": "s2.mp4"}], "start": 220, "stop": 230} ] },

{"id": "scene0", "js": "clear", "color": "cyan"}, 
{"id": "scene1", "sources": ["seq1", "seq2"], "width": 80, "height": 80, "rx":10, "ry":"x",
	"mix": { "type": "mix", "dur": 2}, 
	"mix_ratio": 0.5
},

{"start": 0, "loop": true, "dur": 2, "keys": [0, 0.5, 1], "anims":
 [
  {"values": [0, 1, 0], "targets": ["scene1@mix_ratio"]}
 ]
}
]
```
Playlist updates

The playlist file will be updated whenever modified. This is mostly used for live mode, but can be used in offline mode.

When updating a playlist, you must assign IDs to root elements of the playlist in order to identify them across reloads.
For example, load:

```
[
{"seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] }
]
```

then load (updating `loop` parameter):
```
[
{"seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] , "loop": "x"}
]
```

Upon reloading, the initial sequence won't be find in the new playlist, so associated sources will be stopped and new sequence will be evaluated.
Specifying an identifier on the sequence avoids that.
Live mode

Live mode works like offline mode, with the following additions:

- detection and display of signal lost or no input sequences
- `sequence` and `timer` start and stop time can be expressed as UTC dates (absolute) or current UTC offset

Let's load the following

```
[
{ "id": "seq1", "start": -1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "scene1", "sources": ["seq1"]}
]
```

You should now see "no input" message when playing. Without closing the player, reload playlist by modifying `start` to `"now"`:

```
[
{ "id": "seq1", "start": "now", "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "scene1", "sources": ["seq1"]}
]
```

And the video sequence will start ! You can use for start and stop time values:

- "now": will resolve to current UTC time
- integer: will resolve to current UTC time plus the number of seconds specified by the integer
- date: will use the date as the start/stop time
Transparent Logo insertion

Simplified version of above example, with a single source and logo

```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4"}] } ] },
{"sources": ["seq1"]},
{"x": 45, "y": 40, "width": 10, "height": "width", "img":"logo.png", "alpha":0.5}
]
```
Text insertion

Here we insert a text at the top of the video, with black fill color and white outline 
```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4"}] } ] },
{"sources": ["seq1"]},
{"x": 0, "y": 45, "width": 100, "height": 10, "text": ["Some nice text !"], "size": 60, "fill": "black", "line_width": -2, "line_color": "white"}
]
```
Dynamic text insertion

Here we insert a text read from file "input.txt". The text file can be modified while the filter runs to update the text. We only use the last 2 lines of the text file
```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4"}] } ] },
{"sources": ["seq1"], "x": 0, "y": 0, "width": 100, "height": 100},
{"x": 0, "y": 45, "width": 100, "height": 10, "text": ["input.txt", 2], "size": 60, "fill": "black", "line_width": -2, "line_color": "white"}
]
```
Dynamic path

Here we use a dynamic path construction in JS, creating an ellipse with size varying from {0,0} to {2*width,2*height} based on the source timing, reloaded at each frame (assuming 25 fps).
```
[
{ "id": "seq1", "seq": [ { "id": "s1", "src": [{"in": "media.mp4"}] } ] },
{"id": "scene0", "js": "clear", "color": "cyan"}, 
{"sources": ["seq1"], "shape":"let time_sec = 2*((100*get_media_time('s1')) % 100) / 100; this.path.ellipse(0, 0, this.width*time_sec, this.height*time_sec); this.reload=40;"}
]
```
Mosaic

Here we construct a mosaic of 4 videos, all muted

```
[
{ "id": "seq1", "seq": [ {"src": [{"in": "media1.mp4"}] } ] },
{ "id": "seq2", "seq": [ {"src": [{"in": "media2.mp4"}] } ] },
{ "id": "seq3", "seq": [ {"src": [{"in": "media3.mp4"}] } ] },
{ "id": "seq4", "seq": [ {"src": [{"in": "media4.mp4"}] } ] },

{"sources": ["seq1"], "x": -25, "y": 25, "width": 50, "height": 50, "volume":0},
{"sources": ["seq2"], "x": 25, "y": 25, "width": 50, "height": 50, "volume":0},
{"sources": ["seq3"], "x": -25, "y": -25, "width": 50, "height": 50, "volume":0},
{"sources": ["seq4"], "x": 25, "y": -25, "width": 50, "height": 50, "volume":0}
]
```
Source monitoring

If your sources are unreliable and lead to crashes or failures in the demultiplexing or decoding pipeline, this usually result to gpac exiting with failure.
This is problematic if you use AVMix to generate a live feed supposed to be up 24/7.

To prevent this, the filter allows launching the sources as dedicated child processes. When the child process exits unexpectedly, or when source data is no longer received, the filter can then kill and relaunch the child process.

There are three supported methods for this:

- running a gpac instance over a pipe
- running a gpac instance over TCP
- running any other process capable of communicating with gpac

The declaration is done at the `sourceURL` level through the port option.

For each of these mode, the `keep_alive` option is used to decide if the child process shall be restarted:

- if no more data is received after `rtimeout`.
- stream is in end of stream but child process exited with an error code greater than 2.

The default behavior for pipe and TCP modes is to use raw (uncompressed) data exchange, as usually unexpected issues come from a mix of demultiplexing/decoding.

### Pipe example

```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4", "port": "pipe"} ] } ] }
]
```

### TCP example
We use keep-alive and force the child gpac process to crash after 1s. We specify `seek` here to try to restart the source at the estimated elapsed time since our source is on demand.

```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "media.mp4", "port": "tcp", "opts": "-runfors=1000"} ], "keep_alive":true, "seek":true } ] }
]
```

### Generic process example
In this example, the generic process is gpac, delivering over a pipe an MPEG-2 TS stream.

```
[
{ "id": "seq1", "seq": [ { "src": [{"in": "gpac -i media.mp4 -o pipe://gpipe:ext=ts", "port": "pipe://gpipe:mkp"} ] } ] }
]
```
Playlist updates

A playlist can be modified through updates, in order to avoid reloading a complete playlist for simple modifications.

To simplify the workflow, the playlist update is done through a dedicated file, so that you only update the playlist for consequent changes such as adding/removing media, scene and groups.

Updates file is specified by the `updates` option of the filter, and reloaded whenever modified.

To update a scene position:  
```
[
{ "replace": "scene1@x", "with": 100},
{ "replace": "scene1@y", "with": 100}
]
```

To start an inactive sequence and a timer 4s after
```
[
{ "replace": "seq1@start", "with": "now"},
{ "replace": "timer1@start", "with": 4}
]
```

To update a transition on a sequence:  
```
[
{ "replace": "seq1@transition", "with": {"type": "fade", "color": "white"}}
]
```