Best way to get stereo camera depth data from FPGA -> RPi ?? - Page 1
Forum section: electronics / general

author: David1234
author_rank: Contributor
timestamp: December 31, 2018, 10:03:51 pm
content: I have developed a sub-$150 FPGA system that generates real-time stereo camera depth/disparity data at 30+fps, at VGA resolution, with no lag. I plan to sell this as a super cheap alternative to $20k+ outdoor, 3d LIDAR systems. The last step is to figure out the best protocol for outputting this data into a raspberry pi or beaglebone. What protocol can support 307,200, 8-bit values, 30 times per second (9,216,000, 8-bit values per second)? SPI? (This is assuming that I'm not sending an address with each depth value. If an address is needed then the math works out to 9,216,000, 19 addr bits + 8 data bits per second. Can SPI on the Pi support that?

author: ataradov
author_rank: Super Contributor
timestamp: December 31, 2018, 10:27:38 pm
content: USB or Ethernet are the only realistic choices. There is no way you can make SPI work at 75 MHz reliably.And a standard USB video would be the perfect choice. There are plenty of libraries that can take the video and do further processing on it.

author: David1234
author_rank: Contributor
timestamp: December 31, 2018, 10:49:46 pm
content: I would love to output USB video, but remember, the master device is the FPGA. I heard that it was totally infeasible for an FPGA to somehow output USB video by itself. Do you know of a break-out board for some ASIC that will generate the required USB signals?

author: ataradov
author_rank: Super Contributor
timestamp: December 31, 2018, 10:54:16 pm
content: Cypress makes generic parallel port to USB bridges. That is probably the easiest way. You just need to research what they need to stream video.In theory it is pretty easy. You need to supply correct descriptors and then just stream the data over isochronous endpoints. Cypress chips have embedded MCU that can handle all the slow speed stuff like control endpoints.There must also be chips specifically for video streaming. But I have not used any, and I don't know how easy they are to get.

author: NorthGuy
author_rank: Super Contributor
timestamp: January 01, 2019, 12:14:20 am
content: I don't know if RPi can support parallel port, but if it does, 10 MHz parallel port is the easiest option.If you go USB route - I would try FT232H. It can convert USB CDC to what they call "parallel FIFO interface" which lets you achieve 40 MBytes/s communication with FPGA, according to FTDI specs. It is much cheaper than Cypress, and also has built-in drivers for all major OSes in case you decide to drop RPi. It requires some external parts - oscillator, EEPROM etc., but nothing major. Here's the link:https://www.ftdichip.com/Products/ICs/FT232H.htm100BaseT Ethernet may be ok, but may be too slow, depending on how you use it. Gigabit Ethernet is definitely enough, but is more expensive. Both require PHY, magnetics, and connector, and also will consume some sizeable amount of FPGA logic.

author: David1234
author_rank: Contributor
timestamp: January 01, 2019, 12:32:00 am
content: The FPGA board I'm using has a FTDI FT2232HQ chip on it that I currently use. Would that be fast enough to get 30fps of 8 bit data out? If so, I then need to see if the Pi can support that level of serial input.

author: ataradov
author_rank: Super Contributor
timestamp: January 01, 2019, 12:33:55 am
content: But having that video data just available though a serial port will likely to be useless to potential customers. You are simplifying life for yourself, but potentially making a product nobody would want.

author: David1234
author_rank: Contributor
timestamp: January 01, 2019, 12:40:19 am
content: Right. USB would be best. There needs to be a breakout board that takes in parallel camera data, in whatever form, and outputs USB video stream. It also has to cost $50 or less. If it is several hundred dollars then that defeats the purpose.

author: ataradov
author_rank: Super Contributor
timestamp: January 01, 2019, 12:42:44 am
content: Well, 2 minute search resulted in finding FTDI FT602, which sounds like what you need exactly.It is $10 on DigiKey.I bet there are cheaper alternatives from China.

author: David1234
author_rank: Contributor
timestamp: January 01, 2019, 12:43:50 am
content: Quote from: NorthGuy on January 01, 2019, 12:14:20 amThe board I'm using apparently only reliably outputs about 1Mbaudhttps://forum.digilentinc.com/topic/2682-uart-communication-control-with-cmod-a7/

author: NorthGuy
author_rank: Super Contributor
timestamp: January 01, 2019, 01:23:38 am
content: Quote from: David1234 on January 01, 2019, 12:32:00 amThey have different connection modes to FPGA. Plain UART, which you have on your board, is very slow. I don't think your board has wiring for faster options. You would need to connect an external FT232H breakout board or make your own FPGA board to test. What you need is "synchronous FIFO". Read the datasheet for details.For RPi (or any other host) it'll look like USB CDC. Linux has support for it. You just open the device as a file and read the data.

author: ogden
author_rank: Super Contributor
timestamp: January 01, 2019, 01:47:43 am
content: Best way to get video into RPi is camera (MIPI CSI2) interface whereRPi camera modulesusually are connected to.You would want to read following thread:https://www.raspberrypi.org/forums/viewtopic.php?t=151840

author: ataradov
author_rank: Super Contributor
timestamp: January 01, 2019, 01:49:17 am
content: Quote from: ogden on January 01, 2019, 01:47:43 amYes, I totally forgot that RPi has MIPI interface. This is definitely a way to go if you are targeting RPi.

author: Rerouter
author_rank: Super Contributor
timestamp: January 01, 2019, 02:07:31 am
content: If you approached it like a block transfer, rather than individually addressed you can keep your transmissions closer to your Ideal of about 9MB/sE.g. both Ethernet and USB shine for large block transfers, so instead of addressing every data point, you just address the block for reassembly.If you did want to go with SPI, so long as its on a board you control and the lengths are low, then you can definitely send that fast, but i can make no claim for how fast a raspberry pi can interpret it.Seeing as it seems you are encoding it as a depth map image, could you not encode for the camera interface on a raspberry pi?

author: David1234
author_rank: Contributor
timestamp: January 01, 2019, 03:11:18 am
content: Quote from: Rerouter on January 01, 2019, 02:07:31 amMaybe. I so far haven't found the details needed for *sending* camera data over the CSI port. If I could find that data, I would then need to write a CSI entity. I'd rather do USB if that is feasible since it is more widely applicable.

author: ataradov
author_rank: Super Contributor
timestamp: January 01, 2019, 03:13:35 am
content: Quote from: David1234 on January 01, 2019, 03:11:18 amThen FTDI FT602 or equivalent is the best option. Your device will essentially appear as a web camera and will be automatically usable by all software that can work with web cams.

author: David1234
author_rank: Contributor
timestamp: January 01, 2019, 03:29:50 am
content: Quote from: ataradov on January 01, 2019, 03:13:35 amYea, that would be ideal. I need to find a breakout board for the FTDI FT602 and an example project where someone has actually used one, or alternatively, really good documentation.edit:Found a breakout boardhttps://brtchip.com/product/umft602a-b/

author: asmi
author_rank: Super Contributor
timestamp: January 02, 2019, 03:27:09 pm
content: I made a board with FT601, which is identical to FT602 as far as FPGA side goes, it's not complicated at all. I tested the board in a 1IN-1OUT configuration while only sending data to the PC from FPGA, and I got about 323 MBytes/s (~2.5 GBit/s) of real bandwidth (meaning driver overhead has been factored in, as I was measuring the speed at the application side). I believe FT602 runs only in a single channel mode, so you can expect somewhat higher bandwidth. If you curious to see the schematic, you can lookhere, on the page 4.

author: Morgan127
author_rank: Contributor
timestamp: January 02, 2019, 06:00:52 pm
content: I would definitely use Ethernet. Then you get a stable and reliable product.You will also have no problem to switch from BB to RPI or whatever board you choose. No drivers etc.These board (linux) handles Ethernet very good.On the FPGA side you use UDP with fix ip. I guess you have Ethernet on your FPGA board.There exist Ethernet cores (at least for Xilinx) with all necessary functionality without use of embedded processor. Using one of them and and you will be up and running in minutes.I have been using cores fromFPGA Ethernet Coresin similar projects. Free download

author: asmi
author_rank: Super Contributor
timestamp: January 02, 2019, 06:55:25 pm
content: Quote from: Morgan127 on January 02, 2019, 06:00:52 pm"Ethernet" and "reliable" do not belong to a single sentence. There are way too many corner cases.Quote from: Morgan127 on January 02, 2019, 06:00:52 pmIt's free only for non-commercial use, while the OP clearly started he wants to sell the product

author: David1234
author_rank: Contributor
timestamp: January 03, 2019, 05:33:29 am
content: Quote from: asmi on January 02, 2019, 03:27:09 pmI think I found the most inexpensive, simplest thing that will result in the FPGA stereo system being recognized as a USB video device, the cypress FX2LPhttps://community.cypress.com/docs/DOC-9329The FX3 is newer, but more expensive and not needed as I only plan to run VGA and USB 2.0 is good enough. The cypress products seem to have better documentation and examples vs the FTDI stuff.

author: xyrtek
author_rank: Regular Contributor
timestamp: January 03, 2019, 07:00:58 am
content: I know you have your own hardware and I dont like to suggest something that doesn't address your question or that points you in a direction you probably considered already but....Did you consider using a SoC?Xilinx Zynq 7000 series comes to mind.What functions are you trying to implementd on the raspberry pi? (SBC side)If only to "display" the video then why not implement the VGA port on the fpga? Quiet simple actually.If price is your concern check the good folks at krtkl, they managed to produce a sub $150 board based using the z7000 series.https://www.crowdsupply.com/krtkl/snickerdoodle#products-topIt may seems hard to change course while you are well on your way with you project but the interconnect provided by a SoC can save you lots of time and bring true flexibility to you project.Hope this help.

author: ataradov
author_rank: Super Contributor
timestamp: January 03, 2019, 07:02:36 am
content: Quote from: xyrtek on January 03, 2019, 07:00:58 amRPi is just an example. The device and its depth map output is the actual product. So that video output would be the "API".

author: xyrtek
author_rank: Regular Contributor
timestamp: January 03, 2019, 07:34:07 am
content: Quote from: ataradov on January 03, 2019, 07:02:36 amI understood that part and thats why I included SBC side on my question.Anyways without hijacking the thread, what the OP describes (what I sort of understood) real time, 3D lidar, price etc...sounds like a job for a SOC to me.Not going to explain the advantages, that's an easy google search away and common sense.

author: MarkR42
author_rank: Regular Contributor
timestamp: January 03, 2019, 11:11:56 am
content: Maybe there is a way to hack it into the Raspberry PI SDIO bus?The SDIO bus is used on the Pi 3 to connect to the wifi, which avoids the bottleneck of the usb controller which affected wifi dongles on earlier Pis. This is how it gets better wifi performance.I read somewhere that it is possible to hack other devices to the sdio, but it is mostly used for storage devices.As you have a fpga, presumably it is possible to make it talk whatever prototcol you want, provided it is electrically compatible.

author: technix
author_rank: Super Contributor
timestamp: January 04, 2019, 05:37:16 pm
content: If you can emulate the signal given by the few MIPI CSI2 devices that the Pi supports, you can use that interface to send an uncompressed video feed to the Pi, and try to pack your depth map data into the pixels.This is one place where Pi has its shortcomings: it isn’t easy to add support for CSI and DSI devices. If it were Allwinner V3s (using the recently mainlined 8-bit camera interface) things can be a bit easier.