Stereo Vision on Raspberry Pi using CSI MUXing - Page 1
Forum section: electronics / projects

author: nufflee
author_rank: Newbie
timestamp: May 03, 2020, 02:15:58 pm
content: Hi everyone, I've been researching this topic for a while and have finally come up with something but I'm looking for advice and I have a couple of questions. The end goal is to put this on a drone and receive data from both cameras at up to 480p @ 60 FPS and run the stereo matching algorithm on that data.Note that I am fully aware of commercial solutions for this like StereoPi and Arducam Stereo HAT but I want to see if I can come up with something myself, that is the point of this whole project.To set the scene, Raspberry Pi boards have only a single CSI interface that I want to leverage to use 2 cameras for stereo vision. This can be done using an FPGA that merges two camera streams together into one that is sent to the Pi, but I want to see whether this can be done using a MUX and by interleaving the frames. I have talked to some knowledgeable people on the Raspberry Pi forums (https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=270939- anyone know how to hyperlink a word?) and they have said this should be possible in theory, so why not try it?My plan is the following: I want to use a NX3DV642 (https://www.nxp.com/docs/en/data-sheet/NX3DV642.pdf) MIPI MUXes for the CSI stream and a general purpose 74LV4053 (https://assets.nexperia.com/documents/data-sheet/74LV4053.pdf) MUXes for the SCCB (I2C). The Raspberry Pi provides a frame start and frame end interrupts which I'm planning to use to cycle through the cameras/MUXes using some discrete digital logic to minimize latency. These interrupts are triggered when the Pi starts and ends receiving data from the CSI stream respectively, they are not generated by the camera itself.I am aware that clock drift becomes a problem when trying to synchronize two cameras with different clock sources/oscillators and the Arducam Stereo HAT solution suffers from this too. They have released a guide on how to modify the official Raspberry Pi camera modules to synchronize the two clocks but I reckon it wouldn't be too complicated on third party camera modules either (especially since they probably don't have any crypto protection like the Raspberry Pi camera module V2).I currently only have rolling shutter cameras on hand but my actual project would use global shutter cameras. This is the diagram of what I'm planning to do with the rolling shutter cameras (and hoping would work):In short, I want to interleave the frames during the blanking period (idle time) of the two cameras. The main issue with doing this is having to figure out exactly how long after enabling a camera it will start producing data so that it can be interleaved properly (`Ts` on the diagram). I do not have a Raspberry Pi on hand currently to measure the actual blanking period using the aforementioned interrupts but I will do so when it arrives. I understand that increasing the blanking period means decreasing the FPS but the FPS I'll be able to get may very well be usable.I also have this diagram that I made when I didn't completely understand how rolling shutter cameras work but is this how global shutter cameras work on the high level?I'm looking for any and all feedback I can get on all of this and here are my specific questions:1. The actual CSI transfer time is exactly the same for any FPS in a specific camera mode/resolution and is calculated as 1/(max FPS in that mode). The CSI transfer could be far shorter since CSI is a very fast interface but it's limited by the fact that sensor has to transfer the whole image line by line. Is this understanding correct?2. Since global shutter cameras do not transfer data line by line and instead buffer it before transferring, would the transfer time be shorter on global shutter cameras? And is there a way to calculate it and/or what do I look for in the datasheet of the camera that may tell me this?3. Will I be able to get predictable and repeatable enough timing both in terms of how long it takes for the camera to start producing data after being turned on and also the blanking period (when their clocks are synchronized)?4. This approach would certainly introduce latency and images from both cameras wouldn't be taken at the same time but there would be a 1 frame time worth of delay. Is that a big deal when it comes to computer vision and stereo matching?5. Are there any other differences in global shutter camera behavior, especially regarding the things I talked about here?6. Lastly, a general MIPI CSI question. Why are there two clocks? There's a low frequency oscillator on the camera module itself but there's an additional differential clock coming in through the CSI interface.Thank you for reading this, I know it's a big post but I've been working on this for a while and would love to make sure my understanding is mostly correct.