Package: django-peertube-runner-connector
django-peertube-runner-connector: A django application to connect to a peertube runner and transcode videosOverviewdjango-peertube-runner-connector is designed to usePeertubetranscoding runners outside of Peertube App. It implements a set of endpoints withDjango Rest Frameworkand aSocketIOserver that allow runners to request jobs, updated job status, download media files and upload the transcoded media files. It provides a function that can be used by your app that will launch the transcoding process.To make use of the SocketIO server, this app only work in ASGI.ArchitectureRunner APIThis part will interact with Peertube runners. It is not designed to be used by a user as it reproduces what the Peertube App is doing in order to manage runners / jobs.Runner BehaviorJobs are stored in a Database, and runners hit the/requestendpoint to get the available jobs to transcode.The transcode video functionThe function receives a video file and a name, and then creates transcoding jobs for it.We use functionprobeofpython-ffmpeglibrary, to get a thumbnail and all the necessary metadata to create transcoding jobs. Once the jobs are created, the WebSocket server emits an event to inform runners of a new pending jobs.Job implementationCurrently we didn't implement all the transcoding jobs the runner can do. We are planning to implement more jobs in the future. For now, the API only implements the following jobs:HLS transcodingVOD web video transcodingLive transcodingVOD audio merge transcodingVideo Studio transcodingTheses jobs are created and handled through their respective classes inapi.transcoding.utils.job_handlersdirectory. Some of them are already almost implemented but are not used, so they are commented.SocketIO serverThe SocketIO server is used to communicate with runners. It is only used to inform runners of new jobs, thus, make this part very simple. It implements only one function that emits the eventavailable-jobsto runners when a new job is created. Once a runner receives this event, it will hit the/requestendpoint in the Runner API to get the new job.InstallationOnce you have installed the library, you will need to setup your project to use it (seeconfigurationpart). You can find a demo application in thetestsdirectory.PyPiTo install the library with pip, enter the following command:pipinstalldjango-peertube-runner-connectorLocalTo install the library locally, enter the following commands at the root of the project:build the library:pythonsetup.pysdistbdist_wheelThis should create adistdirectory with the library files.Then you can go in your application, source your virtual environment and install the library with pip:pipinstall/path/to/django-peertube-runner-connector/dist/django_peertube_runner_connector-1.tarSetupConfiguration# settings.pyINSTALLED_APPS=[# ..."rest_framework","django_peertube_runner_connector.apps.DjangoPeertubeRunnerConnectorConfig","storages",# optional django-storages library]# Transcoding resolution settingsTRANSCODING_ALWAYS_TRANSCODE_ORIGINAL_RESOLUTION=FalseTRANSCODING_RESOLUTIONS_144P=FalseTRANSCODING_RESOLUTIONS_240P=FalseTRANSCODING_RESOLUTIONS_360P=TrueTRANSCODING_RESOLUTIONS_480P=TrueTRANSCODING_RESOLUTIONS_720P=TrueTRANSCODING_RESOLUTIONS_1080P=FalseTRANSCODING_RESOLUTIONS_1440P=FalseTRANSCODING_RESOLUTIONS_2160P=False# Transcoding fps settingsTRANSCODING_FPS_MIN=1TRANSCODING_FPS_STANDARD=[24,25,30]TRANSCODING_FPS_HD_STANDARD=[50,60]TRANSCODING_FPS_AUDIO_MERGE=25TRANSCODING_FPS_AVERAGE=30TRANSCODING_FPS_MAX=60TRANSCODING_FPS_KEEP_ORIGIN_FPS_RESOLUTION_MIN=720# Max number of times a job can fail before being marked as failedTRANSCODING_RUNNER_MAX_FAILURE=5# The callback path to a function that will be called when a video transcoding endedTRANSCODING_ENDED_CALLBACK_PATH=""# The django-peertube-runner-connector app uses the django storage system to store the transcoded videos.# It uses the "videos" storage where you can configure the storage backend you want to use.STORAGES={"default":{"BACKEND":"django.core.files.storage.FileSystemStorage",},"videos":{# This is the storage used to store the transcoded videos"BACKEND":"app.storage.MyCustomFileSystemVideoStorage",# You can use the storage backend you want},"staticfiles":{"BACKEND":"django.contrib.staticfiles.storage.StaticFilesStorage",},}StorageDjango-peertube-runner-connector uses the django storage system to store the transcoded videos. It uses the "videos" storage where you can configure the storage backend you want to use. To use an S3 like storage, you can use thedjango-storageslibrary. Here is an example of a custom storage backend that uses the S3 storage:# app/storage.pyfromstorages.backends.s3boto3importS3Boto3StorageclassMyS3VideoStorage(S3Boto3Storage):"""Custom S3 storage class."""bucket_name="my-bucket"then you can use it in your settings:# settings.py# ... S3 settingsSTORAGES={"default":{"BACKEND":"django.core.files.storage.FileSystemStorage",},"videos":{"BACKEND":"app.storage.MyS3VideoStorage",# Your custom storage backend},"staticfiles":{"BACKEND":"django.contrib.staticfiles.storage.StaticFilesStorage",},}ServerTo make use of the SocketIO server, you need to have ASGi server likeuvicorn.Here is an example on how to configure your asgi server to use the SocketIO server:fromconfigurations.asgiimportget_asgi_applicationdjango_asgi_app=get_asgi_application()# its important to make all other imports below this commentimportsocketiofromdjango_peertube_runner_connector.socketimportsioapplication=socketio.ASGIApp(sio,django_asgi_app)Add the runners api views to your urls:# urls.pyfromdjango_peertube_runner_connector.urlsimport(urlpatternsasdjango_peertube_runner_connector_urls,)urlpatterns+=django_peertube_runner_connector_urlsIf your application is distributed on multiple servers, you will probably need
to use a message queue. We manage redis and redis sentinel manager. For this,
you have to define this settingsRedis sentinelDJANGO_PEERTUBE_RUNNER_CONNECTOR_SENTINELS: A list of sentinel nodes.
Each node is represented by a pair (hostname, port). Example: [('localhost', 26379)]DJANGO_PEERTUBE_RUNNER_CONNECTOR_SENTINELS_MASTER: The master sentinel name. Example: mymasterRedisDJANGO_PEERTUBE_RUNNER_CONNECTOR_REDIS: The redis url. Example:redis://localhost:6379Voilà! Your server should be ready!Demo applicationFor testing purpose, you can find a basic django app using the django-peertube-runner-connector library in thetestsdirectory. You use run it with the following commands:Create your virtual environment:python-mvenvenvsourceenv/bin/activateInstall the dependencies:pipinstall-e."[dev]"Go to the tests directory:cdtestsCreate the database and run the migrations:pythonmanage.pymigrateCollect the static files:pythonmanage.pycollectstaticCreate a super user:pythonmanage.pycreatesuperuserLaunch the server with an asgi server like uvicorn:python-muvicornapp.asgi:application--reloadOnce the server is running, you can register your server to a peertube runner.Registering a Peertube runnerFirst you will to generate a registration token. To do so, use the following command and keep the registrationToken for late:pythontests/manage.pycreate_runner_registration_tokenFirst you will need a Peertube runner. To launch one, follow the instructions (theses instructions are made by me and for development purpose only)Clone and go to thepeertube repositorygitclonehttps://github.com/Chocobozzz/PeerTubecdPeerTubeInstall the dependenciescdapps/peertube-runner
npminstallcd../../Build the runnernpmrunbuild:peertube-runnerLaunch the runner./apps/peertube-runner/dist/peertube-runner.jsserverOpen a new terminal in the same directory and register your runner to your django app./apps/peertube-runner/dist/peertube-runner.jsregister--urlhttp://localhost:8000--registration-token$MY_TOKEN--runner-nametranscode-apiCreated a transcoding jobs and receive the transcoded videoYou can now launch a transcoding job with using thehttp://127.0.0.1:8000/videos/uploadend point of the django app by sending a multipart/form-data request with your file as the value of thevideoFilekey. This video view is given by the test app not by the django-peertube-runner-connector app. This should population a directory namedvideo-[uuid]in the root of the project with the result of the transcoding job.Launch testTo launch the tests, enter the following commands at the root of the project:Create your virtual environment:python-mvenvenvsourceenv/bin/activateInstall the dependencies:pipinstall-e."[dev]"Launch the tests:maketestLicenseThis work is released under the MIT License (seeLICENSE).
MIT LicenseCopyright (c) 2023 France Université NumériquePermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.