gaspard7/wiki_gpac huggingface dataset:
Overview {:data-level="all"}

We discuss here how to use the [JavaScript Filter](jsf) to generate 2D vector graphics in GPAC.  
The [JS scripts](https://github.com/gpac/testsuite/tree/filters/media/jsf) in the gpac test suite are also a good source of examples.

GPAC provides a media composition engine through its [Compositor Filter](compositor), capable of rendering BIFS, SVG, VRML and X3D. It requires however a scene description, and is limited to what the scene description syntax allows.
For a simpler and/or more customized support of 2D graphics, GPAC includes JavaScript bindings to its 2D software rasterizer, allowing 2D graphics generation using the JS filter.
 
 Check the documentation of the [EVG APIs](https://doxygen.gpac.io/group__jsevg__grp.html) for more details. 


Please note that the EVG API is an immediate mode API and does not track objects drawn. If you want to partially redraw the surface, you will need to track changes and use clippers to redraw only changed areas.
Creating a canvas {:data-level="beginner"}

The simplest way to create a canvas is to allocate your own memory and setup the canvas:

```
let width=400;
let height=200;
let buffer = new ArrayBuffer(width*height*3);
let canvas = new evg.Canvas(width, height, 'rgb', buffer);

```

If your canvas is the output of your filter, simply allocate an output packet and create the canvas from this:

```
let width=400;
let height=200;
filter.set_cap({id: "StreamType", value: "Video", output: true} );
filter.set_cap({id: "CodecID", value: "raw", output: true} );
let opid = filter.new_pid();
opid.set_prop('StreamType', 'Visual');
opid.set_prop('Width', width);
opid.set_prop('Height', height);
opid.set_prop('PixelFormat', 'rgb');

filter.process = function
{
	let pck = opid.new_packet(width*height*3);
	let canvas = new evg.Canvas(width, height, 'rgb', pck.data);
	
	...
	pck.send();
}
```

If your PID properties do not change between frames (as is usually the case), you don't need to recreate a canvas at each frame, simply reassigning the buffer is enough:

```
if (canvas == null) {
	canvas = new evg.Canvas(width, height, 'rgb', pck.data);
} else {
	canvas.reassign(pck.data);
}

```

Once your canvas is setup, you need to indicate if the coordinate system is centered (0,0 at the center of the canvas, Y-axis going up) or not (0,0 at the top-left of the canvas, Y-axis going down). Non-centered coordinates are also referred to as "pixel coordinates".
```
canvas.centered=true;
```

You can then erase your canvas to the desired color:

```
canvas.clear('yellow');
canvas.clearf(1.0, 0.0., 0.0, 1.0);
```

All draw operations on the canvas, including clear, can be clipped by an axis-aligned rectangle whose coordinate are always in pixel coordinates (non-centered coordinates).

```
let clip = {x: 10, y:10, w: 40, h: 20};
canvas.clear(clip, 'yellow');
canvas.clearf(clip, 1.0, 0.0., 0.0, 1.0);
```
Drawing a circle

The canvas works by filling with a given brush (solid color, linear or radial gradient, textures) a path. A path is a sequence of move, line and curves (cubic and quadratic bezier) in 2D.

The first step is therefore to create a graphical path containing a circle:

```
let circle = new Path();
//adds a centered ellipse with horizontal axis equal to vertical axis (i.e., circle with given diameter)
let circle.add_ellipse(0, 0, 100, 100);

```

Then create a simple brush:
```
let brush = new SolidBrush();
brush.set_color('cyan');
```

Then assign path to surface and draw path
```
canvas.path = circle;
canvas.fill(brush);
```
 
If we put all this together:

```

import * as evg from 'evg'

let width=400;
let height=200;
filter.set_cap({id: "StreamType", value: "Video", output: true} );
filter.set_cap({id: "CodecID", value: "raw", output: true} );
let opid = filter.new_pid();
opid.set_prop('StreamType', 'Visual');
opid.set_prop('Width', width);
opid.set_prop('Height', height);
opid.set_prop('PixelFormat', 'rgb');
//generate at 25 frames per second
opid.set_prop('Timescale', '25');
let canvas = null;
let cts = 0;

//create a static solid brush
let brush = new SolidBrush();
brush.set_color('cyan');

//create a static circle
let circle = new Path();
let circle.add_ellipse(0, 0, 100, 100);

filter.process = function()
{
	let pck = opid.new_packet(width*height*3);
	if (!pck) return GF_OUT_OF_MEM;

	if (canvas == null) {
		canvas = new evg.Canvas(width, height, 'rgb', pck.data);
		canvas.centered=true;
	} else {
		canvas.reassign(pck.data);
	}
	//animate clear color at each frame
	let red = (cts % 100) /100;
	canvas.clearf(red, 0.0, 0.0, 1.0);
	
	canvas.path = circle;	
	canvas.fill(brush);
	pck.cts = cts;
	cts++;
	pck.send();
}
```

You can also use a radial gradient:
```
let brush = new evg.RadialGradient();
brush.set_points(0.0, 0.0, 10.0, 10.0, 15.0, 15.0);
brush.set_stop(0.0, 'red');
brush.set_stopf(1.0, 0.0, 0.0, 1.0, 0.5);
brush.mode = GF_GRADIENT_MODE_SPREAD;
```

or a linear gradient:

```
let brush = new evg.LinearGradient();
brush.set_points(0.0, 0.0, 1.0, 0.0);
brush.set_stop(0.0, 'red');
brush.set_stopf(1.0, 0.0, 0.0, 1.0, 0.5);
brush.mode = GF_GRADIENT_MODE_PAD;
```
Moving a Path

In GPAC EVG, a path is constructed in its local coordinate system; for the example above, the center of the circle was 0,0. Without any transformation, the (0,0) of the local coordinate system is positioned at the origin of the canvas (depending on the centered mode of the canvas).

In order to change the positioning of an object on the canvas, a Matrix2D object must be used. This matrix can specify any affine transformation (scale, rotate, translate, skew, ...).

```
let mx = new EVG.Matrix2D();
mx.scale(2.0, 0.5);

//our circle drawn with this matrix will now appear as an ellipse
canvas.matrix = mx;
canvas.path = circle;	
canvas.fill(brush);

```

You can also use a 3D matrix to draw a path, usually to apply perspective transform. In this case, all points in the path are considered to have 0 as Z coordinate.

 
 ```
 let mx = new EVG.Matrix();
 mx.perspective(Math.PI / 4, width/height, 0.1, 100);
 //apply other transformation to the matrix
 
 //our circle drawn with this matrix will now appear as an ellipse
 canvas.matrix3d = mx;
 canvas.path = circle;	
 canvas.fill(brush);
 ```
Outlining a Path

GPAC EVG handles path outlines ("striking") as a regular path. If you want to outline a path, you first need to create the corresponding outline path using pen properties, then draw it as usual.

```

let outline = circle.outline({width: 5.0, align: GF_PATH_LINE_OUTSIDE, join: GF_LINE_JOIN_BEVEL, dash: GF_DASH_STYLE_DASH_DASH_DOT});


canvas.path = circle;	
canvas.fill(brush);

canvas.path = outline;	
canvas.fill(brush_green);

```
Drawing text

GPAC EVG handles text by converting a text string with a given font as a set of Path objects. This conversion is done automatically and is transparent to the user.

```
/*create a text*/

let text = new evg.Text();
text.font = 'Times';
text.fontsize = 20;
text.align=GF_TEXT_ALIGN_CENTER;
text.set_text(['My awesome text', 'Powered by GPAC']);

canvas.path = text;	
canvas.fill(brush);
```

If you want to draw the outline of a text, you will need to first construct the text and then get its associated path:

```
let txtpath = text.get_path();
let outline = text.outline({width: 5.0, align: GF_PATH_LINE_OUTSIDE, join: GF_LINE_JOIN_BEVEL, dash: GF_DASH_STYLE_DASH_DASH_DOT});

canvas.path = outline;	
canvas.fill(brush);
```
Using textures

GPAC EVG can use textures to fill path. There are several ways of creating a texture:

- create texture from your script data

```
let ab = new ArrayBuffer(2*2*4); //RGBA 2x2 data buffer
//fill the buffer
...
//create the texture
let tx = new EVG.Texture(2, 2, 'rgba', ab);
```

- create texture from a local PNG or JPEG file

```
let tx = new EVG.Texture('myimage.jpg', true);
```

Note that the image loader can resolve the image path as relative to the source JS or to the current working directory. In this example, we use source JS relative path. Only local files are supported by this API.

- create texture from a remote PNG or JPEG file

```
//get remote file using xhr with a response type set to arrayBuffer
xhr = xhr_fetch_file(file_url);
let tx = new EVG.Texture(xhr.response);
```

Check GPAC [XHR API](https://doxygen.gpac.io/group__xhr__grp.html) for more details.



- create texture from a GPAC packet

```
let pck = ipid.get_packet();

let tx = new EVG.Texture(pck);
```

In this case, the texture properties are derived from the packet's parent PID properties.


- create texture from a Canvas object

```
let tx = new EVG.Texture(canvas);
```

In this case, the texture properties are derived from the canvas properties. This is typically used to draw an offscreen canvas and use the result as a texture, similar to MPEG-4 CompositeTexture2D.



Once the texture data is setup, you will then need to setup the base transformation of your texture. This implies scaling the texture to the desired size in object local coordinate system, and translating it at the right position (e.g. center). If we take the example of our circle with 100 pixel, diameter, the texture must be scaled to occupy the full width and height (e.g. 100, 100) and translated to the center of the local coordinate system (e.g. 50, 50).

```
let mmx = new evg.Matrix2D();
mmx.scale(100/tx.width, 100/tx.height);
mmx.translate(50, 50);
//and apply the same transformation as the object
mmx.add(mx);
tx.mx = mmx;

canvas.path = circle;	
canvas.fill(tx);
```

Note that the canvas matrix is always applied to the texture matrix, so that you don't need to adjust texture matrix when moving the objects on the canvas.
Overlays and canvas

As seen previously, it is possible to create a canvas from an output  packet. Now recall how packets can be created from JS: it is possible to clone a source packet as an output, thereby transferring the source data into the destination packet (or creating a copy of the source into the destination otherwise).
This means that you can create a canvas that operate on the pixels of your input video, thereby providing fast 2D graphics overlay:

```
let pfmt = ipid.get_prop('PixelFormat');
let width = ipid.get_prop('Width');
let height = ipid.get_prop('Height');

let ipck = ipid.get_packet();
let opck = opid.new_packet(ipck);
let canvas = new EVG.Canvas(width, height, pfmt, opck.data);

//draw on canvas, this will draw directly on the video

ipid.drop_packet();
opck.send();
```

_Note: this will only work if the input video data is not a set of GPU textures unavailable for read access._

In overlay mode, solid brush and gradients colors are internally converted to destination pixel format if needed (typically when drawing over YUV surfaces). This is however not the case for textures, you must do this conversion manually. For example when drawing an RGBA PNG image over a YUV canvas:

```
let canvas = ...
let image = new EVG.texture('myimage.png');
//if canvas is YUV, convert image to YUV
image = image.rgb2yuv(canvas);

```
 
This will avoid doing the RGB to YUV conversion each time the texture is used.
Standalone script {: data-level ="beginner"}

The standalone mode works by specifying a JS file to the httpin filter using its [-js](httpout#js) option:

```
gpac httpout:port=8080:js=myserver.js ...
```

The JS script is loaded with a global object called `httpout` with one callback function, called at each new request.


The following is a basic JS example performing header injection :

```
import { Sys as sys } from 'gpaccore'

httpout.on_request = function (request)
{
	print("Got request " + JSON.stringify(request));
	request.headers_out.push({name: "x-gpac", value: "foo"});
	//let gpac handle the request
	request.reply=0;
	//throttle the connection, always delaying by 100 us
	request.throttle = function(done, total) {
		return 100;
	}
	//check end of request
	request.close = function(error) {
		print('closed with code ' + error_string(error));
	}
	request.send();
}

```
Attaching from a JS session {: data-level ="beginner"}

The first step in your JS is to create an object implementing the callback previously indicated:


```
//custom rate adaptation object
let req_handler = {
	on_request: function(request) {
		//same code as above
	}
};
```


You will then need to setup a JS session monitoring filter creation process:

```
session.set_new_filter_fun( (f) => {
		print("new filter " + f.name);

		//bind our custom rate adaptation logic
		if (f.name == "httpout") {
			f.bind(req_handler);
		}
} ); 

```

And you're good to go !
Handling requests in JS

You can handle the GET request in your own code rather han using GPAC httpout logic. You will need to use the read callback for GET and the write callback for PUT/POST:


```
import { Sys as sys } from 'gpaccore'

httpout.on_request = function (request)
{
	print("Got request " + JSON.stringify(request));
	//handle the request ourselves
	request.reply=200;
	//decide what to to - here we always open the same file
	request.src = new File('av1.mp4', 'rb');

	//our reader callback for GET
	request.read = function(buf) {
		if (request.src.eof) return 0;
		let nb = request.src.read(ab);
		return nb;
	}

	//our write callback for PUT/POST
	request.read = function(buf) {
		print('Got new bytes: ' + ab.byteLength);
		return 0;
	}

	//check end of request
	request.close = function(error) {
		print('closed with code ' + error_string(error));
		request.src.close();
	}
	//send the request - this can also be done later on, e.g. in a callback task
	request.send();
}

```
Overview

We discuss here how to use the [JavaScript Filter](jsf) to generate 3D vector graphics in GPAC.  
The [JS scripts](https://github.com/gpac/testsuite/tree/filters/media/jsf) in the GPAC test suite are also a good source of examples.

GPAC provides a media composition engine through its [Compositor Filter](compositor), capable of rendering BIFS, SVG, VRML and X3D. It requires however a scene description, and is limited to what the scene description syntax allows.
For a simpler and/or more customized support of 3D graphics, GPAC includes WebGL JavaScript bindings, allowing 3D graphics generation using the JS filter.
 
 Check the documentation of the [WebGL APIs](https://doxygen.gpac.io/group__webgl__grp.html) for more details.
Setting up WebGL

You need to import WebGL JavaScript bindings in your script:

```
import {WebGLContext} from 'webgl'
```

You will also very likely need to use matrices and textures. Helper tools from [EVG](evg) might help here:

```
import {Texture, Matrix} from 'evg'
```

As explained in the API documentation, WebGL in GPAC is not using DOM and does not use a canvas element to access the context. Rather, the context is explicitly created through a constructor:

```
let width=1280;
let height=720;
let gl = new WebGLContext(width, height);
```

You can also pass a WebGLContextAttributes object to the constructor:

```
let gl = new WebGLContext(width, height, {depth: false});
```

The second important thing regarding WebGL in GPAC is that a filter is responsible for deciding when to issue GL calls, and this is not always in the process callback. Your therefore must call `activate` on the context before issuing calls:

 ```
gl.activate(true);

//draw scene
...
//done
gl.activate(false);
 ```
 
Similarly, the context being not tied to a window or a particular PID, it is the filter's responsibility to resize the underlying framebuffer of the WebGL context:

```
gl.resize(640, 480);
```

You now have a working WebGL context to draw your content. There is no such thing as `requestAnimationFrame` in GPAC, so you either need to draw your content upon `filter.process` callback or post a given for your filter (using `filter.post_task`).
GPU texture output

In this mode, the dispatched packet will only contain reference to the GL texture associated with the frame buffer used, typically color. This avoids unnecessary GPU->systems memory transfer if the consuming filter is capable of processing GL textures.

This approach is simply performed by creating a new packet from your canvas:

```
//draw scene
...

let output_pck = output_pid.new_packet(gl);
//setup output packet, set DTS/CTS/etc..
...
//send packet
output_pck.send();
```

Note that a single framebuffer is created for each WebGL context. One problem with the above approach is that you will not know when the output frame is consumed, hence you will probably erase the associated framebuffer before it is processed. It is therefore recommended to use a callback to track the packet release:

```
if (filter.frame_pending) return;
//draw scene
...

//output canvas
let output_pck = output_pid.new_packet(gl, () => { filter.frame_pending=false; } );
//setup output packet, set DTS/CTS/etc..
...
//send packet
filter.frame_pending=true;
output_pck.send();
```
Framebuffer copy output

In this mode, the dispatched packet will contain a copy of the framebuffer directly in system memory. The simplest way to do this is configuring the output PID to `rgb` or 'rgba` , allocating the packet and reading back the framebuffer throw gl.ReadPixels:

 ```
 let output_pck = output_pid.new_packet(width*height*3);
 
 gl.ReadPixels(0, 0, width, height, gl.RGB, gl.UNSIGNED_BYTE, output_pck.data);
 gl.activate(false);

 //setup output packet, set DTS/CTS/etc..
 ...
 //send packet
 output_pck.send();
 ```

Note that in this case you don't need to track when the output packet is consumed since it contains a full copy of the framebuffer.  
 
 # Using textures
 
 GPAC uses regular WebGL textures as well as what we call `NamedTexture` .
 
 Please [read the WebGL GPAC doc](https://doxygen.gpac.io/group__webgl__grp.html) for more details on textures.
Regular textures

[EVG](evg) textures can be used to quickly load JPG or PNG images:
```
let texture = gl.createTexture();
let tx = new evg.Texture('source.jpg');
gl.bindtexture(gl.TEXTURE_2D, texture);
gl.texImage2D(target, level, internalformat, format, type, tx);
//at this point the data is uploaded on GPU, the EVG texture is no longer needed and can be GC'ed 
```

EVG texture combined with EVG Canvas can be used to draw text and 2D shapes:

```
let canvas = new evg.Canvas(200, 200, 'rgba');
 /* draw stuff on 2D canvas
 ...
 */
 let texture = gl.createTexture();
 let tx = new evg.Texture(canvas);
 gl.bindtexture(gl.TEXTURE_2D, texture);
 gl.texImage2D(target, level, internalformat, format, type, tx);
 //at this point the data is uploaded on GPU, the EVG texture and canvas are no longer needed and can be GC'ed 
```
NamedTextures

Named textures provide a quick way of setting up texturing in various formats supported by GPAC for later use in GLSL shaders, hiding all the complexity of pixel conversion.

A named texture is a texture created with a name:
```
let tx = gl.createTexture('myVidTex');
```

The texture data is then associated using upload():
```

//source data is in system memory or already in OpenGL textures
let pck = input_pid.get_packet();
tx.upload(pck);
//or
//source data is only in system memory
tx.upload(some_evg_texture);
```

Regular bindTexture and texImage2D can also be used if you don't like changing your code too much:
```
let pck = input_pid.get_packet();
gl.bindTexture(gl.TEXTURE_2D, tx);
//source data is in system memory or already in OpenGL textures
gl.texImage2D(target, level, internalformat, format, type, pck);
//or
gl.bindTexture(gl.TEXTURE_2D, tx);
//source data is only in system memory
gl.texImage2D(target, level, internalformat, format, type, some_evg_texture);
```

The magic comes in when creating your shaders: any call to texture2D on a sampler2D using the same name as the NamedTexture is rewritten before compilation and replaced with GLSL code handling the pixel format conversion for you !
```
varying vec2 vTextureCoord;
uniform sampler2D myVidTex; //this will get replaced before compilation
uniform sampler2D imageSampler; //this will NOT get replaced
void main(void) {
  vec2 tx = vTextureCoord;
  vid = texture2D(myVidTex, tx); //this will get replaced before compilation
  img = texture2D(imageSampler, tx); //this will NOT get replaced
  vid.alpha = img.alpha;
  gl_FragColor = vid;
}
```

The resulting fragment shader may contain one or more sampler2D and a few additional uniforms, but they are managed for you by GPAC!

The named texture is then used as usual:
```
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, tx);
//this one is ignored for named textures (the uniformlocation object exists but is deactivated) but you can just keep your code as usual
gl.uniform1i(myVidTexUniformLocation, 0);
gl.activeTexture(gl.TEXTURE0 + tx.nb_textures);
gl.bindTexture(gl.TEXTURE_2D, imageTexture);
gl.uniform1i(imageSamplerUniformLocation, 0);
```

In the above code, note the usage of `tx.nb_textures` : this allows fetching the underlying number of texture units used by the named texture, and properly setting up multi-texturing.

The core concept for dealing with NamedTexture is that the fragment shader sources must be set AFTER the texture is being setup (upload / texImage2D). Doing it before will result in an unmodified fragment shader and missing uniforms.

To summarize, NamedTexture allows you to use existing glsl fragment shaders sources with any pixel format for your source, provided that:

- you tag the texture with the name of the sampler2D you want to replace
- you upload data to your texture before creating the program using it
Primary framebuffer

You can setup your WebGL context to work on the main (primary) framebuffer. This can be done by specifying the `primary` attribute in the WebGLContextAttributes object:

```
let gl = new WebGLContext(width, height, {primary: true});
```

You can then dispatch your output packets as usual:
```
if (filter.frame_pending) return;
//draw scene
...

//output canvas
let output_pck = output_pid.new_packet(gl, () => { filter.frame_pending=false; } );
//setup output packet, set DTS/CTS/etc..
...
//send packet
filter.frame_pending=true;
output_pck.send();
```

In this mode, the `vout` filter consuming the output packets will only perform a backbuffer swap.
Dispatching the depth buffer

You can setup your WebGL context to allow dispatching of the depth buffer as GPU textures by specifying the `depth` attribute in the WebGLContextAttributes object:

```
let gl = new WebGLContext(width, height, {depth: texture});
```

The depth buffer can then be dispatched by creating an output packet:

```
let output_pck = output_pid.new_packet(gl, () => { filter.frame_pending=false; }, true );
```

This mode still lets you output the color buffer.  This mode is however not compatible with usage of the primary framebuffer.

You can also dispatch the depth buffer as a regular system memory packet by reading back the depth buffer using `gl.ReadPixels`.
Apple and QuickTime specifics

You need to change the QT handler name to ```stbl```, using:
```
MP4Box -i SRC:hdlr=sbtl ... 
```

The alternate group shall be set if multiple subs are present e.g.:
```
gpac -i video.264 \
   -i audio.aac:#udta_tagc="public.accessibility.describes-video":#udta_name="English (describes video)":#Language=en \
   -i sub-fr.srt:#Language=fr:#AltGroup=2:#StreamSubtype=sbtl:#Disable \
   -i sub-en.srt:#Language=en:#AltGroup=2:#StreamSubtype=sbtl \
  -o result.mp4
```
Rendering support

GPAC support rendering of text:tx3g, text:wvtt and subt:stpp. The former is natively supported while the latter are supported through JavaScript by converting WebVTT or TTML samples to SVG.

Rendering is automatically activated when using the compositor filter (e.g. `gpac -gui` or `gpac -mp4c`).
It is also possible to render subtitles to image or video sequences:
```
gpac -i source.vtt -o dump_$num$.png
gpac -i source.vtt -o dump.264
```