How to merge multiple uncompressed digital video streams? FPGA? GPU? - Page 1
Forum section: electronics / projects

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 03, 2015, 03:17:50 am
content: Hi,Happy New Year to everyone.In 2015, I'm hoping to get to work on a project that will take uncompressed, digital video streams from a number of Omnivision cameras and stitch them horizontally prior to doing some DSP on the combined stream before outputting the result to an LCD screen. Its early days and I'm just thinking through the concept at the moment.I would greatly appreciate any suggestions on how best to do the merging of the video streams (all of which should have the same resolution, frame rate, etc.) so that I can generate a "wide angle" video effect.My initial thoughts are to use a FPGA to merge the multiple streams - between 3 and 5 streams seems realistic at the moment. The resultant stream could then be scaled (if necessary) and used as an input to a DSP or GPU to do some fun video processing.Is an FPGA the most reasonable approach to handle the multiple input streams? Are there any other options out there that I should consider?Many thanks,Kieran

author: Marco
author_rank: Super Contributor
timestamp: January 03, 2015, 04:01:25 am
content: Unless your aim is to learn a DSP architecture or GPU programming I'd avoid them. Make up your mind on what kind of processing you want to do and see if it will fit in a Zynq (2 GHz cores with NEON can do quite a bit).

author: Marco
author_rank: Super Contributor
timestamp: January 03, 2015, 04:14:08 am
content: PCs are cheap, methods of getting large datastreams into them are ridiculously expensive ... having to go analog->digital->analog->digital is just sad.

author: Rasz
author_rank: Super Contributor
timestamp: January 03, 2015, 08:47:41 am
content: depends on what you want to spend your time on - hardware or software?if you want results fast get USB 3.0/1Gbit IP camera modules (plenty for scientific/surveillance market)

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: January 03, 2015, 09:00:53 am
content: The merging should be pretty straightforward with an FPGA. More so if you can run all the cams off a common clock and phase-lock the line/frame rates (latter can probably be done with I2C commands twiddling registers)

author: nctnico
author_rank: Super Contributor
timestamp: January 03, 2015, 11:54:11 am
content: It depends greatly on what type of image sensor is used. A couple of years ago I did a project with an Omnivision camera module. To achieve the maximum framerate it needed a MIPI interface (kind of high speed LVDS with a proprietary protocol). I got round by overclocking it a bit and use the parallel interface anyway. Another caveat may be that the camera outputs YUV instead of RGB. Handling YUV may not be a bad thing. Some color spaces are easier to work with than RGB for certain operations.In addition to Mike: the camera probably has an internal PLL so even though the camera's may run at the same speed you may have some jitter. I don't think it is very easy to make the camera's run frame-locked. That would require creating a PLL in software which alters the output framerate slightly by varying the length of a line/frame. It certainly would take time for the PLL to converge.

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: January 03, 2015, 05:06:15 pm
content: Quote from: nctnico on January 03, 2015, 11:54:11 amMost of the cams I've played with cand do YUV or RGB. RGB tends to be either 16 bit 5:6:5 or raw 10bpp, the latter bypasses some of the onboard processing.YUV is probably the easiest to work with for merging etc. if it suits the final output you want to generate.

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 06, 2015, 02:39:04 am
content: Quote from: Marco on January 03, 2015, 04:01:25 amThanks Marco. I'm familiar with the Zynq from Xilinx and I'm certainly considering it as the SoC for the system I have in mind.Out of interest, would you mind expanding on why you think a DSP or GPU should be avoided please? Merging parallel video streams would seem to fit more naturally in hardware, i.e. in FPGA fabric, than in software but other than that, are there other reasons that you are thinking of?Thanks,Kieran

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 06, 2015, 02:47:20 am
content: Quote from: mikeselectricstuff on January 03, 2015, 09:00:53 amThanks Mike. This is what I would have thought. Running the cameras off a common clock from the FPGA makes sense. I'd like to have the cameras a few metres apart from each other and a few metres from the FPGA. Transmitting multiple synchronous high-quality, high-speed clocks will require some consideration but should be perfectly doable with some thought.The transmission of the video data from the cameras to the FPGA may require a SERDES for each camera. I'm thinking that through at the moment.

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 06, 2015, 02:56:33 am
content: Quote from: Rasz on January 03, 2015, 08:47:41 amI'm more familiar with video hardware and FPGAs than the associated software layers so my tendency is to err towards hardware dominated solutions. That is a personal bias though so I'm open to suggestions.The application that I have in mind requires as low latency as possible. The desire to merge multiple camera streams to form a single output stream also means that deterministic timing from each camera to the FPGA is necessary (or at least, extremely desirable). I would like the output stream to be a snapshot in time from multiple cameras. That requires a strong degree of timing synchronisation between the cameras.The need for extremely low latency and timing determinism is the reason that I am inclined to transmit uncompressed video rather than compressed video and focus on a hardware solution rather than a software based one for the input stream merging.

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 06, 2015, 03:02:09 am
content: Quote from: nctnico on January 03, 2015, 11:54:11 amThanks for this. To get the application I have in mind to work well, getting the clocking right will be key. I'll keep these details in mind.

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 06, 2015, 03:38:01 am
content: Quote from: evb149 on January 03, 2015, 04:05:04 amThanks for the detailed reply. Its given me a lot to consider. I just tired to post a detailed response but got a forum error and lost my replyThe application that I have in mind is a panoramic video stream, of sorts. I would really like to develop it as an embedded system without the need for a traditional PC. I am considering a FPGA + DSP/GPU combination or a FPGA operating as a SoC with an embedded processor running Linux might do nicely also.A single board computer of some sort in conjunction with a FPGA board would also do the trick perhaps but I'd like to achieve a higher level of integration than that, if possible.Quote from: evb149 on January 03, 2015, 04:05:04 amThe application that I have in mind requires very low latency between the cameras and LCD. Synchronisation between the cameras is also important to ensure that the final merged image doesn't look obviously stitched together. You alluded to this in some of your frame buffer comments also. For this reason, my instinct is to avoid any compression until after the streams have been merged. This simplifies some things but the trade-off is a massive amount of data requiring potentially significant bandwidth and processing power. The associated data transmission also becomes more difficult from a signal integrity stand-point (though still doable of course).Quote from: evb149 on January 03, 2015, 04:05:04 amThat's an interesting thought and one that I had not really considered. Placing a video decoder (ADC) for each camera on the FPGA board would simplify things in some senses ... and complicate them in other sensesI'll look into this some more.

author: Rasz
author_rank: Super Contributor
timestamp: January 06, 2015, 07:56:08 am
content: Quote from: MixedSignalSystems on January 06, 2015, 02:56:33 amis the end product merged stream still to be low latency in relation to source being filmed? or do you simply figured its a requirement to get a proper synced merge?if its the first one you can exclude GPU and computing on PCs, only DSPs and FPGAs will provide you with deterministic low latencyOpenCV should be called copy_that_buffer_one_more_time, no chance of low latencyin second case you dont need low latency, just timestamps in individual streams, plenty of ways of imprinting/extracting those, for example adding something (8 segmeng LED display) to filmed scene, syncing on sound in individual streams, syncing on visual cues.so at this point im guessing VR?not stitched:http://www.gizmag.com/360-degree-f1-car/24858/I remember seeing similar video from a stitched setup, but cant find it nowstitched and synced in post:http://punkoryan.com/2011/02/08/shooting-360-degree-video-with-four-gopro-hd-hero-cameras

author: nctnico
author_rank: Super Contributor
timestamp: January 06, 2015, 02:41:21 pm
content: IMHO stitching the camera streams together may be more troublesome than it looks. It will need extremely accurate mechanical linkage. Each camera chip will be different and therefore needing individual gamma, brightness and contrast correction. These adjustment can be a nuisance without a proper color calibration card (Xrite has these). Also auto white balance, auto exposure control won't work so you have to do these in software as well. Maybe the easiest way to do this is to blend the edges of the images so they merge together. Dunno if that will be invisible though.What kind of framerate and resolution do you expect to achieve?I found an interesting posting about Freescale's iMX6 SoC. Appearantly you can have up to 4 cameras on a single MIPI interface:https://community.freescale.com/thread/324174

author: Marco
author_rank: Super Contributor
timestamp: January 06, 2015, 04:46:50 pm
content: Quote from: MixedSignalSystems on January 06, 2015, 02:39:04 amDSP development tools are expensive, GPU programming is a pain in the ass.What kind of image processing do you want to do? If it's just about the affine transforms and colour correction to stitch the images, I'm pretty sure the Arm cores in the Zynq can handle it. For 16 bit precision you have 16 GMACs to work with (using Neon).

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 07, 2015, 02:57:33 am
content: Quote from: Rasz on January 06, 2015, 07:56:08 amThe merged stream needs to be low latency relative to the stream being captured by the camera. I don't see it as a fundamental requirement for the synchronisation.Quote from: Rasz on January 06, 2015, 07:56:08 amHeading the right directionNot quite VR but I'm hoping to experiment a bit in the assisted living field so the applications and requirements are somewhat comparable. Hence the need for the low latency.Quote from: Rasz on January 06, 2015, 07:56:08 amCool!

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 07, 2015, 03:04:24 am
content: Quote from: Marco on January 06, 2015, 04:46:50 pmUnderstood. FPGA tools aren't particularly cheap either if you find yourself needing a beefy FPGA but I see your point. Webpack can be used with about half the Zynq devices if I remember correctly. That should give me plenty scope to get started with no software licensing costs.Quote from: Marco on January 06, 2015, 04:46:50 pmThis is a bit of a journey of discovery for me so, to begin with at least, stitching the streams, scaling the output and carrying out some basic to medium complexity signal processing, e.g. basic motion detection is all that's required.

author: MixedSignalSystems
author_rank: Contributor
timestamp: January 07, 2015, 03:15:12 am
content: Quote from: nctnico on January 06, 2015, 02:41:21 pmI don't have any particular specs in mind at the moment. Right now, I'm just investigating the feasibility of the project to see what might be possible for one guy in a reasonable timeframe with a small budget. I haven't crunched the numbers on the bandwidth requirements for any given combination of frame rate, resolution and number of channels.Its a fair questionTime for me to get crunching the numbers I think

author: nctnico
author_rank: Super Contributor
timestamp: January 07, 2015, 06:08:43 pm
content: I think so. There is a huge difference in development time between hacking something together with Gstreamer + SoC or going pedal to the metal with an FPGA.