Package: video_sampler
video-samplerVideo sampler allows you to efficiently sample video frames.
Currently, it uses keyframe decoding, frame interval gating and perceptual hashing to reduce duplicated samples.Use case:for sampling videos for later annotations used in machine learning.Table of Contentsvideo-samplerTable of ContentsFeaturesInstallation and UsageBasic usageYT-DLP integration pluginExtra YT-DLP optionsAPI examplesAdvanced usageGatingCLIP-based gating comparisonBlur gatingBenchmarksBenchmark videosFlit commandsBuildInstallPublish🛡 License📃 CitationFeaturesDirect sampling methods:hash- uses perceptual hashing to reduce duplicated samplesentropy- uses entropy to reduce duplicated samples (work in progress)gzip- uses gzip compressed size to reduce duplicated samples (work in progress)buffer- uses sliding buffer to reduce duplicated samplesgrid- uses grid sampling to reduce duplicated samplesGating methods (modifications on top of direct sampling methods):clip- uses CLIP to filter out frames that do not contain the specified objectsblur- uses blur detection to filter out frames that are too blurryIntegrationsYTDLP integration -- streams directly fromyt-dlpqueries,
playlists or single videosInstallation and Usagepipinstall-Uvideo_samplerthen you can runpython3-mvideo_sampler--helpor simplyvideo_sampler--helpBasic usagepython3-mvideo_samplerhashFatCat.mp4./dataset-frames/--hash-size3--buffer-size20YT-DLP integration pluginBefore using please consult the ToS of the website you are scraping from -- use responsibly and for research purposes.
To use the YT-DLP integration, you need to installyt-dlpfirst (seeyt-dlp).
Then, you simply add--yt-dlpto the command, and it changes the meaning of thevideo_pathargument.to searchvideo_samplerhash"ytsearch:cute cats"./folder-frames/\--hash-size3--buffer-size20--yt-dlpto sample a single videovideo_samplerhash"https://www.youtube.com/watch?v=W86cTIoMv2U"./folder-frames/\--hash-size3--buffer-size20--yt-dlpto sample a playlistvideo_samplerhash"https://www.youtube.com/watch?v=GbpP3Sxp-1U&list=PLFezMcAw96RGvTTTbdKrqew9seO2ZGRmk"./folder-frames/\--hash-size3--buffer-size20--yt-dlpThe videos are never directly downloaded, only streamed, so you can use it to sample videos from the internet without downloading them first.Extra YT-DLP optionsYou can pass extra options to yt-dlp by using the-yt-extra-argsflag. For example:this will only sample videos uploaded before 2019-01-01:...--ytdlp--yt-extra-args'--datebefore 20190101'or this will only sample videos uploaded after 2019-01-01:...--ytdlp--yt-extra-args'--dateafter 20190101'or this will skip all shorts:...--ytdlp--yt-extra-args'--match-filter"original_url!*=/shorts/ & url!*=/shorts/"API examplesSee examples in./scripts.Advanced usageThere are 3 sampling methods available:hash- uses perceptual hashing to reduce duplicated samplesentropy- uses entropy to reduce duplicated samples (work in progress)gzip- uses gzip compressed size to reduce duplicated samples (work in progress)To launch any of them you can run and substitutemethod-namewith one of the above:video_samplerbuffer`method-name`...otheroptionse.g.video_samplerbufferentropy--buffer-size20...wherebuffer-sizeforentropyandgzipmean the top-k sliding buffer size. Sliding buffer also uses hashing to reduce duplicated samples.GatingAside from basic sampling rules, you can also apply gating rules to the sampled frames, further reducing the number of frames.
There are 3 gating methods available:pass- pass all framesclip- use CLIP to filter out frames that do not contain the specified objectsblur- use blur detection to filter out frames that are too blurryHere's a quick example of how to use clip:python3-mvideo_samplerclip./videos./scratch/clip--pos-samples"a cat"--neg-samples"empty background, a lemur"--hash-size4CLIP-based gating comparisonHere's a brief comparison of the frames sampled with and without CLIP-based gating with the following config:gate_def=dict(type="clip",pos_samples=["a cat"],neg_samples=["an empty background","text on screen","a forest with no animals",],model_name="ViT-B-32",batch_size=32,pos_margin=0.2,neg_margin=0.3,)Evidently, CLIP-based gating is able to filter out frames that do not contain a cat and in consequence, reduce the number of frames with plain background. It also thinks that a lemur is a cat, which is not entirely wrong as fluffy creatures go.Pass gate (no gating)CLIP gateGridThe effects of gating in numbers, for this particular set of examples (seeproducedvsgatedcolumns).producedrepresents the number of frames sampled without gating, here after the perceptual hashing, whilegatedrepresents the number of frames sampled after gating.videobuffergatedecodedproducedgatedFatCat.mp4gridpass1793131SmolCat.mp4gridpass1182424HighLemurs.mp4gridpass1613535FatCat.mp4hashpass179101101SmolCat.mp4hashpass1186161HighLemurs.mp4hashpass161126126FatCat.mp4hashclip17910173SmolCat.mp4hashclip1186131HighLemurs.mp4hashclip16112666Blur gatingHelps a little with blurry videos. Adjust threshold and method (laplacianorfft) for best results.
Some results fromfftatthreshold=20:videobuffergatedecodedproducedgatedMadLad.mp4gridpass1203131MadLad.mp4hashpass120110110MadLad.mp4hashblur12011085BenchmarksConfiguration for this benchmark:SamplerConfig(min_frame_interval_sec=1.0,keyframes_only=True,buffer_size=30,hash_size=X,queue_wait=0.1,debug=True)VideoTotal framesHash sizeDecodedSavedSmolCat29368118106SmolCat-4-61Fat Cat44628179163Fat Cat-4-101HighLemurs40208161154HighLemurs-4-126SamplerConfig(min_frame_interval_sec=1.0,keyframes_only=True,queue_wait=0.1,debug=False,print_stats=True,buffer_config={'type':'entropy'/'gzip','size':30,'debug':False,'hash_size':8,'expiry':50})VideoTotal framesTypeDecodedSavedSmolCat2936entropy11839SmolCat-gzip-39Fat Cat4462entropy17964Fat Cat-gzip-73HighLemurs4020entropy16159HighLemurs-gzip-63Benchmark videosSmolCatFat CatHighLemursMadLadFlit commandsBuildflit buildInstallflit installPublishRemember to bump the version inpyproject.tomlbefore publishing.flit publish🛡 LicenseThis project is licensed under the terms of theMITlicense. SeeLICENSEfor more details.📃 Citation@misc{video-sampler,author={video-sampler},title={Video sampler allows you to efficiently sample video frames},year={2023},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/LemurPwned/video-sampler}}}