Package: ffmpegcv
FFMPEGCV is an alternative to OPENCV for video read and write.English Version |中文版本The ffmpegcv provide Video Reader and Video Witer with ffmpeg backbone, which are faster and powerful than cv2.The ffmpegcv is apicompatibleto open-cv.The ffmpegcv can useGPU accelerateencoding and decoding*.The ffmpegcv support much more videocodecsv.s. open-cv.The ffmpegcv supportRGB& BGR & GRAY format as you like.The ffmpegcv supportStream reading(IP Camera) in low latency.The ffmpegcv can support ROI operations.You cancrop,resizeandpadthe ROI.In all, ffmpegcv is just similar to opencv api. But is has more codecs and does't require opencv installed.Functions:VideoWriter: Write a video file.VideoCapture: Read a video file.VideoCaptureNV: Read a video file by NVIDIA GPU.VideoCaptureQSV: Read a video file by Intel QuickSync Video.VideoCaptureCAM: Read a camera.VideoCaptureStream: Read a RTP/RTSP/RTMP/HTTP stream.VideoCaptureStreamRT: Read a RTSP stream (IP Camera) in real time low latency as possible.noblock: Read/Write a video file in background using mulitprocssing.InstallYou need to download ffmpeg before you can use ffmpegcv.#1A. LINUX: sudo apt install ffmpeg
 #1B. MAC (No NVIDIA GPU): brew install ffmpeg
 #1C. WINDOWS: download ffmpeg and add to the path
 #1D. CONDA: conda install ffmpeg=6.0.0     #don't use the default 4.x.x version
 
 #2. python
 pip install ffmpegcvWhen should chooseffmpegcvother thanopencv:Theopencvis hard to install. The ffmpegcv only requiresnumpyandFFmpeg, works across Mac/Windows/Linux platforms.Theopencvpackages too much image processing toolbox. You just want a simple video/camero IO with GPU accessible.Theopencvdidn't supporth264/h265and other video writers.You want tocrop,resizeandpadthe video/camero ROI.Basic exampleRead a video by CPU, and rewrite it by GPU.vidin=ffmpegcv.VideoCapture(vfile_in)vidout=ffmpegcv.VideoWriterNV(vfile_out,'h264',vidin.fps)#NVIDIA-GPUwithvidin,vidout:forframeinvidin:cv2.imshow('image',frame)vidout.write(frame)Read the camera.# by device IDcap=ffmpegcv.VideoCaptureCAM(0)# by device namecap=ffmpegcv.VideoCaptureCAM("Integrated Camera")Cross platformThe ffmpegcv is based on Python+FFmpeg, it can cross platform amongWindows, Linux, Mac, X86, Armsystems.GPU AccelerationSupportNVIDIAcard only, test in x86_64 only.Works inWindows,LinuxandAnaconda.Works in theGoogle Colabnotebook.Infeasible in theMacOS. That ffmpeg didn't supports NVIDIA at all.* The ffmegcv GPU reader is a bit slower than CPU reader, but much faster when use ROI operations (crop, resize, pad).CodecsCodecsOpenCV-readerffmpegcv-cpu-rgpu-rOpenCV-writerffmpegcv-cpu-wgpu-wh264√√√×√√h265 (hevc)not sure√√×√√mjpeg√√×√√×mpeg√√×√√×othersnot sureffmpeg -decoders×not sureffmpeg -encoders×BenchmarkOn the way...（maybe never）Video ReaderThe ffmpegcv is just similar to opencv in api.# open cvimportcv2cap=cv2.VideoCapture(file)whileTrue:ret,frame=cap.read()ifnotret:breakpass# ffmpegcvimportffmpegcvcap=ffmpegcv.VideoCapture(file)whileTrue:ret,frame=cap.read()ifnotret:breakpasscap.release()# alternativecap=ffmpegcv.VideoCapture(file)nframe=len(cap)forframeincap:passcap.release()# more pythonic, recommandwithffmpegcv.VideoCapture(file)ascap:nframe=len(cap)foriframe,frameinenumerate(cap):ififrame>100:breakpassUse GPU to accelerate decoding. It depends on the video codes.
h264_nvcuvid, hevc_nvcuvid ....cap_cpu=ffmpegcv.VideoCapture(file)cap_gpu=ffmpegcv.VideoCapture(file,codec='h264_cuvid')#NVIDIA GPU0cap_gpu0=ffmpegcv.VideoCaptureNV(file)#NVIDIA GPU0cap_gpu1=ffmpegcv.VideoCaptureNV(file,gpu=1)#NVIDIA GPU1cap_qsv=ffmpegcv.VideoCaptureQSV(file)#Intel QSV, experimentalUsergb24instead ofbgr24. Thegrayversion would be more efficient.cap=ffmpegcv.VideoCapture(file,pix_fmt='rgb24')#rgb24, bgr24, grayret,frame=cap.read()plt.imshow(frame)ROI OperationsYou can crop, resize and pad the video. These ROI operation isffmpegcv-GPU>ffmpegcv-CPU>>opencv.Cropvideo, which will be much faster than read the whole canvas.cap=ffmpegcv.VideoCapture(file,crop_xywh=(0,0,640,480))Resizethe video to the given size.cap=ffmpegcv.VideoCapture(file,resize=(640,480))Resizeand keep the aspect ratio with black borderpadding.cap=ffmpegcv.VideoCapture(file,resize=(640,480),resize_keepratio=True)Cropand thenresizethe video.cap=ffmpegcv.VideoCapture(file,crop_xywh=(0,0,640,480),resize=(512,512))Video Writer# cv2out=cv2.VideoWriter('outpy.avi',cv2.VideoWriter_fourcc('M','J','P','G'),10,(w,h))out.write(frame1)out.write(frame2)out.release()# ffmpegcv, default codec is 'h264' in cpu 'h265' in gpu.# frameSize is decided by the size of the first frameout=ffmpegcv.VideoWriter('outpy.mp4',None,10)out.write(frame1)out.write(frame2)out.release()# more pythonicwithffmpegcv.VideoWriter('outpy.mp4',None,10)asout:out.write(frame1)out.write(frame2)Use GPU to accelerate encoding. Such as h264_nvenc, hevc_nvenc.out_cpu=ffmpegcv.VideoWriter('outpy.mp4',None,10)out_gpu0=ffmpegcv.VideoWriterNV('outpy.mp4','h264',10)#NVIDIA GPU0out_gpu1=ffmpegcv.VideoWriterNV('outpy.mp4','hevc',10,gpu=1)#NVIDIA GPU1out_qsv=ffmpegcv.VideoWriterQSV('outpy.mp4','h264',10)#Intel QSV, experimentalInput image is rgb24 instead of bgr24out=ffmpegcv.VideoWriter('outpy.mp4',None,10,pix_fmt='rgb24')out.write(cv2.cvtColor(frame,cv2.COLOR_BGR2RGB))Video Reader and Writerimportffmpegcvvfile_in='A.mp4'vfile_out='A_h264.mp4'vidin=ffmpegcv.VideoCapture(vfile_in)vidout=ffmpegcv.VideoWriter(vfile_out,None,vidin.fps)withvidin,vidout:forframeinvidin:vidout.write(frame)Camera ReaderExperimental feature. The ffmpegcv offers Camera reader. Which is consistent with VideoFiler reader.TheVideoCaptureCAMaims to support ROI operations. The Opencv will be general fascinating than ffmpegcv in camera read.I recommand the opencv in most camera reading case.The ffmpegcv can use name to retrieve the camera device. Useffmpegcv.VideoCaptureCAM("Integrated Camera")is readable thancv2.VideoCaptureCAM(0).TheVideoCaptureCAMwill be laggy and dropping frames if your post-process takes long time. The VideoCaptureCAM will buffer the recent frames.TheVideoCaptureCAMis continously working on background even if you didn't read it.Please release it in time.Works perfect in Windows, not-perfect in Linux and macOS.importcv2cap=cv2.VideoCapture(0)whileTrue:ret,frame=cap.read()cv2.imshow('frame',frame)ifcv2.waitKey(1)&0xFF==ord('q'):breakcap.release()# ffmpegcv, in Windows&Linuximportffmpegcvcap=ffmpegcv.VideoCaptureCAM(0)whileTrue:ret,frame=cap.read()cv2.imshow('frame',frame)ifcv2.waitKey(1)&0xFF==ord('q'):breakcap.release()# ffmpegcv use by camera name, in Windows&Linuxcap=ffmpegcv.VideoCaptureCAM("Integrated Camera")# ffmpegcv use camera path if multiple cameras conflictcap=ffmpegcv.VideoCaptureCAM('@device_pnp_\\\\?\\usb#vid_2304&''pid_oot#media#0001#{65e8773d-8f56-11d0-a3b9-00a0c9223196}''\\global')# ffmpegcv use camera with ROI operationscap=ffmpegcv.VideoCaptureCAM("Integrated Camera",crop_xywh=(0,0,640,480),resize=(512,512),resize_keepratio=True)List all camera devicesfromffmpegcv.ffmpeg_reader_cameraimportquery_camera_devicesdevices=query_camera_devices()print(devices){0: ('Integrated Camera', '@device_pnp_\\?\usb#vid_2304&pid_oot#media#0001#{65e8773d-8f56-11d0-a3b9-00a0c9223196}\global'),1: ('OBS Virtual Camera', '@device_sw_{860BB310-5D01-11D0-BD3B-00A0C911CE86}\{A3FCE0F5-3493-419F-958A-ABA1250EC20B}')}Set the camera resolution, fps, vcodec/pixel-formatfromffmpegcv.ffmpeg_reader_cameraimportquery_camera_optionsoptions=query_camera_options(0)# or query_camera_options("Integrated Camera")print(options)cap=ffmpegcv.VideoCaptureCAM(0,**options[-1])[{'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (1280, 720), 'camfps': 60.0002}, {'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (640, 480), 'camfps': 60.0002}, {'camcodec': 'mjpeg', 'campix_fmt': None, 'camsize_wh': (1920, 1080), 'camfps': 60.0002}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (1280, 720), 'camfps': 10}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (640, 480), 'camfps': 30}, {'camcodec': None, 'campix_fmt': 'yuyv422', 'camsize_wh': (1920, 1080), 'camfps': 5}]Known issuesThe VideoCaptureCAM didn't give a smooth experience in macOS. You must specify all the camera parameters. And the query_camera_options woun't give any suggestion. That's because theffmpegcannot list device options using mac nativeavfoundation.# The macOS requires full argument.cap=ffmpegcv.VideoCaptureCAM('FaceTime HD Camera',camsize_wh=(1280,720),camfps=30,campix_fmt='nv12')The VideoCaptureCAM cann't list the FPS in linux. Because theffmpegcound't query the device's FPS using linux nativev4l2module. However, it's just OK to let the FPS empty.Stream Reader (Live streaming, RTSP IP cameras)Experimental feature. The ffmpegcv offers Stream reader. Which is consistent with VideoFiler reader, and more similiar to the camera.
Becareful when using it.SupportRTSP,RTP,RTMP,HTTP,HTTPSstreams.TheVideoCaptureStreamwill be laggy and dropping frames if your post-process takes long time. The VideoCaptureCAM will buffer the recent frames.TheVideoCaptureStreamis continously working on background even if you didn't read it.Please release it in time.Low latencyRTSP IP camera reader. Batter than opencv.It's still experimental. Recommand you to use opencv.# opencvimportcv2stream_url='http://devimages.apple.com.edgekey.net/streaming/examples/bipbop_4x3/gear2/prog_index.m3u8'cap=cv2.VideoCapture(stream_url,cv2.CAP_FFMPEG)ifnotcap.isOpened():print('Cannot open the stream')exit(-1)whileTrue:ret,frame=cap.read()ifnotret:breakpass# ffmpegcvimportffmpegcvcap=ffmpegcv.VideoCaptureStream(stream_url)whileTrue:ret,frame=cap.read()ifnotret:breakpass# ffmpegcv, IP Camera Low-latency# e.g. HIK Vision IP Camera, `101` Main camera stream, `102` the secondstream_url='rtsp://admin:PASSWD@192.168.1.xxx:8554/Streaming/Channels/102'cap=ffmpegcv.VideoCaptureStreamRT(stream_url)# Low latency & recent bufferedcap=ffmpegcv.ReadLiveLast(ffmpegcv.VideoCaptureStreamRT,stream_url)#no bufferwhileTrue:ret,frame=cap.read()ifnotret:breakpassNoblockA proxy to automatic prepare frames in backgroud, which does not block when reading&writing current frame (multiprocessing). This make your python program more efficient in CPU usage. Up to 2x boost.ffmpegcv.VideoCapture(*args) -> ffmpegcv.noblock(ffmpegcv.VideoCapture, *args)ffmpegcv.VideoWriter(*args) -> ffmpegcv.noblock(ffmpegcv.VideoWriter, *args)#Proxy any VideoCapture&VideoWriter args and kargsvid_noblock=ffmpegcv.noblock(ffmpegcv.VideoCapture,vfile,pix_fmt='rbg24')# this is fastdefcpu_tense():time.sleep(0.01)for_intqdm.trange(1000):ret,img=vid_noblock.read()#current img is already buffered, take no timecpu_tense()#meanwhile, the next img is buffering in background# this is slowvid=ffmpegcv.VideoCapture(vfile,pix_fmt='rbg24')for_intqdm.trange(1000):ret,img=vid.read()#this read will block cpu, take timecpu_tense()