gaspard7/wiki_gpac huggingface dataset:
Offscreen groups

Groups can be defined to be offscreen and reused later on. 
Offscreen groups are only redrawn whenever a change in the child scene happens.

There are two common places where an offscreen group can be used, as detailed below.
In both cases, the offscreen group is handled as a regular sequence for texturing and will follows parent coordinate system transformations.

### As input source to a scene
By listing the offscreen group ID in the `sources` array of a scene, you instruct the scene to use the offscreen group as if it was a sequence.
This allows applying transition effects between a video source and an offscreen source.

 

The following shows a cross-fade between the two:

```
[
{ "id": "seq1", "loop": 1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },

{"id": "gr1", "offscreen":"color", "width": 20, "height": 20, "reference": true, "scenes": [
{"width": 100, "height": 100, "fill":"white"},
{"y": -40, "width": 90, "height": 20, "fill":"red"},
{"width": 100, "height": 40, "text": ["Super Text"], "size": 50, "align": "center", "fill": "black"}
]},

{"id":"v1", "sources": ["seq1", "gr1"], "mix": {"type":"mix"}, "mix_ratio":0.5 },


{ "start": 0, "loop": true, "dur":2, "keys": [0, 0.5, 1], "anims": [
	{"values": [1, 0, 1], "targets": ["v1@mix_ratio"] }
] }

]
```

Note that we change the reference coordinate system in this example, so that scenes in the offscreen group are given in percent of the group offscreen size and not in percent of the output frame size. 

In this mode, the texturing parameters used by the offscreen group are the same as the ones applying to a regular sequence.

### As additional input to scene modules

The [shape](avmix#scene-shape) module allows specifying a replacement image  `img`  along with some replacement operations  `replace`. The image can be a sequence or an offscreen group. 

 
The following shows a cross-fade of two inputs using an offscreen group in alphagrey color mode as the source for alpha blending value:

```
[
{ "id": "seq1", "loop": 1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "seq2", "loop": 1, "seq": [ { "src": [{"in": "media2.mp4"}], "start": 0, "stop": 10} ] },

{"id": "gr1", "offscreen":"mask", "width": 20, "height": 20, "reference": true, "scenes": [
	{"id": "b1", "width": 100, "height": 100, "fill":"white"},
	{"y": -40, "width": 90, "height": 20, "fill":"red"},
	{"width": 100, "height": 40, "text": ["Super Text"], "size": 50, "align": "center", "fill": "black"}
]},

{"id":"v1", "sources": ["seq1", "seq2"], "img": "gr1", "replace":"xa"},


{ "start": 0, "loop": true, "dur":2, "keys": [0, 0.5, 1], "anims": [
	{"values": [1, 0, 1], "targets": ["b1@alpha"] }
] }
]
```

In this mode, the texturing parameters used by the offscreen group can be modified using the properties `*_rep`of the shape object.
Masking

AVMix can use a global alpha mask (covering the entire output frame) for draw operations, through the [mask](avmix#scene-mask) scene module.

This differs from using an offscreen group as an alpha operand input to [shape](avmix#scene-shape) as discussed above as follows:

- the mask is global and not subject to any transformation
- the mask is always cleared at the beginning of a frame
- the mask is only one alpha channel
- the mask operations can be accumulated between draws

The following example shows using a mask in regular mode:

- enable and clear mask
- draw a circle with alpha 0.4
- use mask and draw video, which will be blended only where the circle was drawn using alpha= 0.4
- enable mask without clearing
- draw the same circle at the same place with alpha 0.4, resulting in composite alpha of 0.8
- draw the same circle next to first one with alpha 0.4, resulting in composite alpha of 0.4
- use mask and draw second, which will be blended only where the circles were drawn with different alpha


```
[
{ "id": "seq1", "loop": 1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "seq2", "loop": 1, "seq": [ { "src": [{"in": "media2.mp4"}], "start": 0, "stop": 10} ] },

{"js":"mask", "mode":"on"},
{"id": "b1", "x": -25, "width": 50, "height": "width", "shape":"ellipse", "fill": "white", "alpha": 0.4},

{"js":"mask", "mode":"use"},
{"sources": ["seq1"]},

{"js":"mask", "mode":"onkeep"},
{"use": "b1"},
{"x": 50, "use": "b1"},

{"js":"mask", "mode":"use"},
{"sources": ["seq2"]}
]
```


The mask can also be updated while drawing using a record mode. In this mode, the mask acts as a binary filter, any pixel drawn to the mask will no longer get drawn.

The following draws:

- an ellipse with first video at half opacity, appearing blended on the background
- the entire second video at full opacity, which will only appear where mask was not set

```
[
{ "id": "seq1", "loop": 1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "seq2", "loop": 1, "seq": [ { "src": [{"in": "media2.mp4"}], "start": 0, "stop": 10} ] },

{"js":"clear", "color":"cyan"},

{"js":"mask", "mode":"rec"},
{"sources": ["seq1"], "x": -25, "width": 50, "height": 50, "shape":"ellipse", "alpha": 0.5},

{"js":"mask", "mode":"use"},
{"sources": ["seq2"]}
]
```
Monitoring changes

AVMix allows for simple monitoring of changes in the scene tree. Any property that can be animated or updated can be monitored through object called `watchers`.

The simplest usage of a watcher is to forward a property to another object property, typically for translations and such

```
[
{ "id": "seq1", "loop": 1, "seq": [ { "src": [{"in": "media.mp4"}], "start": 1, "stop": 5} ] },
{ "id": "seq2", "loop": 1, "seq": [ { "src": [{"in": "media2.mp4"}], "start": 0, "stop": 10} ] },

{"js":"clear", "color":"cyan"},

{"id":"s1", "sources": ["seq1"], "x": -25, "width": 100, "height": 10, "shape":"ellipse", "alpha": 0.5},
{"id":"s2","sources": ["seq2"]},

{"watch": "s1@x", "target": "s2@x"},
{"watch": "s1@y", "target": "s2@y"}
]
```

In this example, any modification to `s1.x` (resp `s1.y`) through timers, playlist update or other JS code will automatically copy the values to  `s2.x` (resp `s2.y`) .

If you want to modify the value, simply use a script instead of a builtin target:

```
{"watch": "s1@x", "target": "get_scene('s2').set('x', value/2);"},
```

If you need to modify something other than group or scene:
```
{"watch": "s1@x", "target": "update_element('timer', 'loop',  (x<0) ? false : true);"},
```

This will pause the timer `timer` whenever the x coordinate of `s1` is greater than 0, and resume the timer otherwise.


As indicated in the scripting section, you can also add watcher redirecting to module functions, using `target=moduleID.function` syntax:

```
{"id": "mod", "js": "mymod.js"},
{"watch": "s1@x", "target": "mod.on_x"},
```

And in `mymod.js`:

```
export function load() { return {

update: function()
{
return 0;
},

//the function is called with uses 3 arguments: the first one is the value, the second is the ID of the watched object, the third the property name
on_x: function(value)
{
print('x changed to ' + x);
return 0;
}

}; }

```
Overview {: data-level="all" }

We discuss here how to use network captures with GPAC 2.3-DEV or above.



GPAC can:

- write packets to a custom file format called GPC (GPAC Packet Capture), and
- read packets from pcap, pcapng and gpc files.

GPAC uses a concept of network capture rules identified by ID. This allows having a set of filters in GPAC use one capture file while other filters will use either no capture file or another capture file.

Network capture rules are defined using the [-netcap](core_options#netcap) option.
Creating network captures

When creating a network capture with GPAC, output packets are NOT sent over the network, they are only written to file. 
Use Wireshark if you need to both send and record packets.


To record UDP output in TS format, use:
 
```
gpac -i source.mp4 reframer:rt=on -o udp://234.0.0.1:1234/:ext=ts -netcap=dst=cap_ts.gpc
```

To record RTP output, use:
 
```
gpac -i source.mp4 reframer:rt=on -o session.sdp -netcap=dst=cap_rtp.gpc
```

To record ROUTE output, use:
 
```
gpac -i source.mp4 reframer:rt=on -o route://234.1.1.1:1234/live.mpd:dynamic -netcap=dst=cap_route.gpc
```

You can also use Wireshark to capture packets.
Playing network captures

To replay a capture file (from the above examples), use: 

```
gpac -i udp://234.0.0.1:1234/ inspect:deep -netcap=src=cap_ts.gpc

gpac -i session.sdp inspect:deep -netcap=src=cap_rtp.gpc

gpac -i route://234.1.1.1:1234/live.mpd inspect:deep -netcap=src=cap_route.gpc
```

By default, the packets from a capture file are made available to the app in real-time. You can disable this using `nrt` flag:

```
gpac -i session.sdp inspect:deep -netcap=src=cap_rtp.gpc,nrt

```

_NOTE_  Using `nrt` will likely require much more memory when consuming the data in real-time (e.g. playback).
Assigning network captures to filters

If you need one filter to read from a capture file and others to not use a capture file, you will need to set an ID to the network capture rules and tag the filters to use this ID.

For example, playing a route capture and exposing the resulting files to a real HTTP server filter:

```
gpac -i route://234.1.1.1:1234/live.mpd:gcache=false:NCID=RTE -netcap=id=RTE,src=route.gpc httpout:port=8080:rdirs=./dash

```
Stress-testing

You can also add rules for packet dropping or modifications, in order to simulate errors in the transmission chain or to test GPAC demux reliability.

This can be done using a capture file or using the device network interface(s). The assignment to filters using `NCID` also works when not using a capture file:


```
gpac -i session.sdp inspect:deep -netcap=src=cap_rtp.gpc,nrt[s=10]

```

This will drop the 10th UDP packet of the capture file, read in non-realtime mode:

```
gpac -i session.sdp inspect:deep -netcap=[f=100]

```

This will drop the first packet every 100 packets coming from the network interface.
Overview {:data-level="all"}

We discuss here how to customize ISOBMFF files using box patches.  Please first check the XML [Box Patch](BoxPatch) syntax before reading.

__Warning__: the box patch does not allow modifications of fields of a box, it can only be used to insert new boxes or  remove existing boxes. Box modifications must be done through available options of [MP4Box](MP4Box-Introduction) or the [ISOBMF multiplexer](mp4mx).
Injecting a box

Let's take the following patch:
```xml
<?xml version="1.0" encoding="UTF-8" />
<GPACBOXES>

<Box path="trak.tkhd+">
<BS fcc="GPAC"/>
<BS value="2" bits="32"/>
<BS value="1" bits="32"/>
</Box>

</GPACBOXES>
```

This patch describes insertion of a box after a track header box, with a 4CC value of `GPAC` and a payload of `0x0000000200000001`.

```
MP4Box -patch box.xml source.mp4
```

This will inject the new box after the track header of the first track in the file.

```
MP4Box -patch 4=box.xml source.mp4
```

This will inject the new box after the track header of the track with ID 4.
Removing a box

Let's take the following patch:
```xml
<?xml version="1.0" encoding="UTF-8" />
<GPACBOXES>

<Box path="trak.GPAC">
</Box>

</GPACBOXES>
```

This patch describes removal of a box with a 4CC value `GPAC`located in the `trak` box.
```
MP4Box -patch box.xml source.mp4
```

This will remove the new box if present in the track header of the first track in the file.

```
MP4Box -patch 4=box.xml source.mp4
```

This will remove the new box if present in the track header of the track with ID 4.
Injecting a box patch in your workflow {: data-level="beginner"}

You may also want to customize your ISOBMFF files while they are being produced in a filter chain. 

This can be done by specifying the [-boxpatch](mp4mx#boxpatch) option to the ISOBMFF multiplexer:

```
gpac -i source.264 -o test.mp4:boxpatch=patch.xml
```

This can  also be done on a per-PID basis; this is typically used to inject different boxes for different media types:

```
gpac -i source.264#boxpatch=patch.xml -i source.aac -o test.mp4
```

This will obviously also work in DASH mode:

```
gpac -i source.264#boxpatch=patch.xml -i source.aac -o test.mpd
```

You can also set globally the [-boxpatch](mp4mx#boxpatch) option to use with MP4Box dashing:
```
MP4Box -dash 1000 source.mp4 -out test.mpd --boxpatch=patch.xml
```

or when fragmenting a file with MP4Box:

```
MP4Box -frag 1000 source.mp4 --boxpatch=patch.xml
```
Foreword {:data-level="all"}

We discuss here how to use the [JavaScript Filter](jsf) WebGL along with Three.js in GPAC.

[Three.js](https://threejs.org/) is a powerful JS library for 3D rendering, potentially using a software renderer as a fallback. Software rendering is not supported, as Three.js requires Canvas2D/SVG/CSS3D for this, which are not available in GPAC.

_Note: you may try to write a Canvas2D polyfill based on GPAC [EVG](evg)._

We recommend reading the [WebGL HowTo](webgl) before anything else.

GPAC does not allow loading JS filters using remote scripts, so you will need to download the latest release of Three.js (this howto was tested with r130). We assume:

- your JS filter script is called `ex3D.js`
- your Three.js distribution is unzipped as "three" in the same directory as  `ex3D.js`

Launching the result is done using:
```
gpac ex3d.js vout
```

__WARNING The code examples below do not check for errors :)__
Using textures

In WebGL, textures are passed using `img`, `video` or `canvas` tags, which we don't have in GPAC. Three.js uses the DOM to load these elements with the desired source.

We'll need to:

- trick Three.js again, by creating a `document` object intercepting calls to element creation. 
- use EVG textures to pass the data to WebGL

Import EVG:
```
import * as evg from 'evg'
```

Create our texture loader:

```
let pending_images = [];

let document = {};
document.createElementNS = function(ns, tag)
{
  if (tag != 'img') return null;
  //create a texture, empty since we don't yet have the `src` attribute !
  let img = new evg.Texture();
  img.on_load = function() {};
  img.on_error = function() {};

  //setup addEventListener for this image
  img.addEventListener = function(evt_name, callback, use_capture) { 
      if (evt_name == "load") {
        img.on_load = callback;
      } 
      else if (evt_name == "error") {
        img.on_error = callback;
      } 
  };
  //setup removeEventListener for this image - we don't use it in this example
  img.removeEventListener = function(evt_name, callback, use_capture) {};

  //remember the image still to be loaded
  pending_images.push(img);
  return img;

}
globalThis.document=document;
```

So far, what we did is create an EVG texture instead of an `img` DOM element, so that any call to `gl.texImage2D`  made for that image is indeed passing an EVG texture.


We need to load the image once the `src` attribute is set. In this example this is "brute force", you will likely find more elegant ways for this.

```

function process_pending_images()
{

  for (let i=0; i<pending_images.length; i++) {
    let img = pending_images[i];
    if (typeof img.src !== 'string') continue;
    pending_images.splice(i, 1);
    i--;

    print('Image URL is ' + img.src);
    //load image into EVG texture - this assumes JPG or PNG only.
    img.load(img.src, true);
    if (img.on_load != null) {
	  //pass a dummy event
      img.on_load({});
    }
  }
}

```

_Note: we pass `true` to `img.load` to indicate that the given path is relative to our script and not to the current working directory._

We can now setup texture loader as usual in Three.js:

```
const loader = new THREE.TextureLoader();

let cube = new THREE.Mesh(
            new THREE.BoxGeometry(1, 1, 1),
            new THREE.MeshBasicMaterial({
                map: loader.load('logo.jpg')
            }))

scene.add(cube);

```

The complete code for this example is [here](examples/three/ex2.js).

You can expand this to create textures containing the result of an EVG canvas draw, as long as what you return is an object that is accepted by GPAC as a WebGL texture, as explained [here](https://doxygen.gpac.io/group__webgl__grp.html).
Using Three.js loaders

Three.js model loaders rely on XmlHttpRequest to fetch data. You will need to import this:

```
import { XMLHttpRequest } from 'xhr'
globalThis.XMLHttpRequest = XMLHttpRequest;
```

Again, note that we need to export  to `globalThis` for Three.js to see the object.


You will also need to import the loader; in this example, we use the GLTF loader:

 
```
import {GLTFLoader} from 'three/examples/jsm/loaders/GLTFLoader.js';
```

Also not that some loaders will use the `self` object without checking for its presence, so let's prevent any type error here:

```
globalThis.self={};
```

Then we only need to load the model !

```
const gltfLoader = new GLTFLoader();

let scene_root=null;
gltfLoader.load('Cube.gltf', (gltf) => {
  scene_root = gltf.scene;
  scene.add(scene_root);
  print('GLTF loaded');
});
```

The complete code for this example is [here](examples/three/ex3.js).
Overview {:data-level="all"}

We discuss here how to use the [JavaScript Filter](jsf) to write JavaScript-based custom filters in GPAC.  
The [JS scripts](https://github.com/gpac/testsuite/tree/filters/media/jsf) in the gpac test suite are also a good source of examples.

The JS filter provides JS bindings to the GPAC filter architecture. JavaScript support in GPAC is powered by QuickJS. Check the documentation of the [JS APIs](https://doxygen.gpac.io/group__jsf__grp.html) for more details.
Principles {:data-level="beginner"}

In order to load, a JS filter requires a source JS, specified in the filter  [js](jsf#js) option. A short-cut syntax is to directly specify the js script. In other words, the following syntaxes are equivalent:
```
gpac -h jsf:js=script.js
gpac -h script.js
```



A JS filter behaves like any other filters in GPAC, and can be a source, sink, filter or server.

Your script will be called with the JS API already loaded with a global object called  `filter`  implementing the JSFilter API.
We will assume in the rest of this article that the script file is called `script.js` 


JS filter life cycle can be described as follows:

* creation of JS context
* loading of the JS file (load or setup phase)
* filter initializing, executed in the callback function `filter.initialize`
* if the filter accepts inputs, configuration of input PIDs in the callback function `filter.configure_pid`
* filter process task, executed in the callback function `filter.process`. This function should not block and typically consumes/produces one packet at a time.
* filter termination, executed in the callback function `filter.finalize`
* final destruction of JS context

While a filter is active, it can get the following notifications:

 * if the filter accepts inputs, (re)configuration of input PIDs in the callback function `filter.configure_pid`
 * if the filter accepts inputs, removal of input PIDs through the callback function `filter.remove_pid`
 * events sent by the pipeline through the callback function `filter.process_event`
 * update of arguments values through the callback function `filter.update_arg`

Callback functions defined for `filter` object do not use exception error handling, they shall either return nothing (success) or a GF_Err code. 
 
It is possible to use several JS filters in a given chain, but each JS filter will create its own JavaScript context, and JS objects cannot be shared between JS filters.
If you need to pass JS data across filters, you will have to serialize to JSON your data and either:

- send it as PID information on a PID of your choice
- send it as JSON-only packets through a dedicated JS PID
- send it as associated property on existing packets
Declaring a filter (optional) {:data-level="beginner"}

The first thing to do when creating a filter is to setup a few things about your filter. 

Give a name to your filter (optional): 
 ```
 filter.set_name("MyFilter");
```
This name will be used when logging messages and inspecting the filter graph. If not set, the filter name will be the script name.

Give a description to your filter  (optional): 
 ```
 filter.set_desc("A demonstration JS filter");
 ```

 This description should provide a quick hint as to what the purpose of the filter is.  It will be shown by the command  `gpac -h script.js`.


You can also setup version and author information  (optional):
 ```
 filter.set_version("0.1beta");
 filter.set_author("GPAC team");
 ```

You should finally set some help for your filter (optional):
 ```
 filter.set_help("This filter provides a very simple javascript filter");
```

This will help other users understand what your filter does and how to use it. It will be shown by the command  `gpac -h script.js`.

You can specify arguments to your filter  (optional):
```
filter.set_arg({ name: "raw", desc: "if set, accept non-demultiplexed input PIDs", type: GF_PROP_BOOL, def: "false"} );
...
filter.set_arg({name: "str", desc: "string to send", type: GF_PROP_STRING, def: "GPAC JS Filter Packet"} );
```

Arguments simplify script configuration and usage, and will be shown by the command  `gpac -h script.js`.

Defined arguments  will be parsed from command line. Each defined argument results in a JS property in the `filter` object with the given value. 

For the above command:

`gpac script.js` will result in the `filter` object having `raw: false` and  `str: "GPAC JS Filter Packet"`  properties defined in the filter object.

`gpac script.js:raw=true` will result in the `filter` object having `raw: true` and  `str: "GPAC JS Filter Packet"`  properties defined in the filter object.


Filter arguments are parsed once the script is loaded. If you need to test the arguments during the initialization phase, do this in the `filter.initialize` callback: 
 
 ```
 filter.initialize = function() {
	if (this.raw) {
		//do something
	} else {
		//do something else
	}
 }
```
Setting up filter capabilities

Each filter in GPAC uses a set of input and/or output capabilities in order to solve connections between filters, and JS filters are no exception. 

Capabilities shall be set during load or initialization phase, it is not possible to modify capabilities after initialization.

The following will add an output capability indicating the property `StreamType` with a value `Visual` is requested on input PIDs.
```
filter.set_cap({id: "StreamType", value: "Visual"});
```

The following will add an input capability indicating the property `StreamType` with a value `File` is excluded on input PIDs, forcing a demultiplexing of the source.
```
filter.set_cap({id: "StreamType", value: "File", excluded: true} );
```

The following will add an output capability indicating the property `StreamType` with a value `Visual` is always present on output PIDs.
```
filter.set_cap({id: "StreamType", value: "Visual", output: true});
```

The following will add an input and output capability indicating the property `CodecID` with a value  `raw` is required on input PIDs and present on output PIDs (typically, raw audio or video processing filter).
```
filter.set_cap({id: "CodecID", value: "raw", inout: true} );
```

__Discussion__  
GPAC uses a concept of capabilities bundles for complex filters, allowing to describe characteristics of different classes of input or output PIDs. This is also possible using JS filters, by adding an empty cap to your filter:

```

filter.set_cap({id: "StreamType", value: "Visual", inout: true});
filter.set_cap({id: "CodecID", value: "raw", inout: true} );
//video specific PID characteristics for input and output

...
//start a new capability bundle
filter.set_cap();

filter.set_cap({id: "StreamType", value: "Audio", inout: true});
filter.set_cap({id: "CodecID", value: "raw", inout: true} );
//audio specific PID characteristics for input and output
...
```

You can check your JS filter sources and sinks links by using `gpac -h links script.js`.