Advice on an imaging project - Page 1
Forum section: electronics / general

author: makedestroyrepeat
author_rank: Newbie
timestamp: September 13, 2019, 11:41:17 am
content: Hey everyone,I am in the early stages of two projects involving image acquisition. I have never had any significant experience with image sensors before, therefore I would be really grateful for any ideas on how to realize these ideas in the most efficient and least painful wayThe first one is a simple imaging platform, where I would read an image sensor and output the data to a display (HD quality and better). As I have read through several OmniVision sensor datasheets, the procedure to run up such a sensor is not trivial, however it is possible to find examples/drivers for some of the sensors. As a plus, one can also adjust some properties of the image, such as auto-white balance, contrast, etc.While this could be achieved with a cheap camera module from AliExpress, I would like to add some more features in the future, such as communication with a PC, saving pictures or adding on-screen menus.The second one involves reading the data from two sensors and fusing them together. For example, I would use a sensor that was used in the first project (just for the sake of convenience having already set it up), and a thermal camera core (e.g. FLIR Boson) which has a USB interface and can also output analog data through the BT.656 standard.I have come to a conclusion that for both of these cases an SoC containing an FPGA would make most sense. I currently own a Digilent Zybo z7-10 board and am trying to learn how to work with it. As I had no previous experience with FPGAs, I have started with simple examples such as HDMI output through plain VHDL and IP integrator in Vivado. While producing the video output was not that difficult, reading a sensor seems to be the opposite. I am now trying to run up the OV5640 image sensor, a.k.a. Digilent PCAM as one can at least find for it some half-working examples online.I have decided to use the Zybo as it contains lots of peripherals that could be handy, such as MIPI CSI-2, HDMI, USB and so on.Are there perhaps some simpler solutions that did not come to my mind? Do I need to use some digitizer circuitry in order to convert the analog signal/retrieve the data from the sensor, or are there already pre-made solutions that I could buy?In the case of the second project, would it be OK to use embedded linux and then read one of the sensors through the programmable logic, while the other (say, thermal camera) reads frames through the USB, or is it not possible? Would the performance suffer?Any ideas are welcome

author: Whales
author_rank: Super Contributor
timestamp: September 13, 2019, 12:22:18 pm
content: Any particular reason you want to implement all of the hardware yourself? Or would you prefer to stick the software side of things? ie do you have any goals related to the experience, or just the outcomes?If you don't want to be working the hard-side of video and high-speed acquisition hardware: I'd recommend skipping all of the hardware step. Use USB webcams and Linux.Getting the video off a camera and into your program is easy. Here's how I did it for one of my hobby people-tracking projects:Code:ffmpeg -loglevel quiet -i /dev/video0 -s 32x24 -video_size 32x24 -c:v rawvideo -f rawvideo -pix_fmt rgb24 - | ./yourprogramFfmpeg will output raw RGB bytes, in order, forever, from your webcam (/dev/video0). No metadata or file headers will be sent and it's really easy to read from stdin in your program. Note that I'mheavilyresizing things down too (32x24!), you probably don't want to do thisIn terms of displaying things back on-screen: I wrote my own custom big-square-pixel video viewer using SDL, this allowed me to draw boxes around certain regions and have other fun. If you don't care about any of this: ffmpeg works in reverse too (raw -> common video), allowing you to pipe the result into a video player (like mplayer or mpv).

author: colorado.rob
author_rank: Frequent Contributor
timestamp: September 13, 2019, 04:20:59 pm
content: I recommend that you start here:http://www.pynq.io/exampleshttps://github.com/Xilinx/PYNQ-ComputerVisionShould at least give you a few ideas. The Zybo should work OK for this. The Pynq distro is essentially PetaLinux with Python and Xilinx Zynq Python support libraries. You can probably make a Pynq distro for the board you have. Or it may make sense to get a Pynq-Z2 board for $120.Even if you don't want to use the Pynq system, the HLS code for image processing is pretty neat. They have accelerated a good deal of OpenCV.

author: makedestroyrepeat
author_rank: Newbie
timestamp: September 16, 2019, 07:03:23 am
content: Quote from: Whales on September 13, 2019, 12:22:18 pmThanks for the suggestion - there is no particular reason I chose to do everything myself, I just could not find any other way to make it fit my needs, one of which is portability. In this regard a USB camera is not ideal, since you can not really minimize the design and a powerful SoC, running some OS like linux might be needed, which again can become bulky and consume a lot of power. Perhaps I am wrong as I do not know everything on the market but that's the way I see it at the momentQuote from: colorado.rob on September 13, 2019, 04:20:59 pmThat is probably the way I will try next - while the Zybo is not supported by Pynq, Vivado has it's own High-Level Synthesis, which I could try to play around with

author: laugensalm
author_rank: Regular Contributor
timestamp: September 16, 2019, 10:03:21 am
content: Quote from: makedestroyrepeat on September 13, 2019, 11:41:17 amJust my advice: if you don't want to dive into endless fun with semi-working image acquisition, you might want to stay away from MIPI for a while and choose a simple camera sensor with parallel output. You'll just learn the most, and you'll still be able to scope on signals. Not much so with MIPI.If you happen to have a direct connector to a COTS cam module and a working reference design, then you're lucky and you can focus on the processing fun. I don't know about the Zybo refdesign stuff, I went for a Lattice VIP kit with two MIPI sensors. Even though this has a reference design for stereo acquisition, it can get quite tedious, not very fun to begin with. The most versatile board for that purpose I can think of is the HDR60 kit, but I think it's no longer sold.Not sure what your desired architecture is, but you could probably have quite some fun with the linux v4l2 video layer already, once you're deviating from the plug'n'play consumer stuff or reference development kits.To learn image processing, you could also start totally virtually and try this:- Implement Bayer conversion in software- Implement it again in hardware (go through the fun of designing pixel synchronous pipelines)- compare/verify HW/SW solution against each otherThe next thing then is to get a real sensor properly configured. That might require you to integrate some i2c controller, if you don't have a ready-made System on Chip (SoC). Then you really want a publicly well documented sensor (which most MIPI-related HW is not) which in this case can be configured for Bayer pattern, if not monochrome or built-in DSP for YCbCr, etc. . If you got this all set in the board supply package, then you can cross off this one from the sensor control side and either choose to use a sensor driver in the kernel or configure the sensor from user space through /dev/i2c-something.Then Linux video acquisition (or the v4l2 driver) adds another layer of complexity which you avoid when using a USB cam, but not when down to bare metal sensor control. Then you might better be off with a sensor that's already supported (check kernel source) so that you don't have to touch much of this part.So, you've got to decide where you'd want to start hacking. I'd personally start with bare metal and avoid any higher OS complexities. For example, I'm using a simple soft CPU core to drive the i2c and configure a DMA descriptor chain for UDP packet streaming, the rest is all HDL that can be simulated and debugged down to the cycles. Of course you can also go 'bare metal' with the Zynq hard-ARM cores, but there will be lacking simulation models.

author: Whales
author_rank: Super Contributor
timestamp: September 18, 2019, 04:37:16 am
content: Quote from: makedestroyrepeat on September 16, 2019, 07:03:23 amAh, you have a focus on power draw.Power consumption and battery life are all relative terms. What are you aiming for? How portable? What battery life?

author: Berni
author_rank: Super Contributor
timestamp: September 18, 2019, 06:14:03 am
content: It also depends on what sort of camera you want to run.If you want to run a 640x480 ish CMOS image sensor you can do it pretty easily with almost any FPGA or MCU/MPU that has a parallel video input bus. On a raspberry pi there are MIPI camera modules that are simply plug and play with it. Or you can always plug a USB webcam into a linux machine and get it working as long as it uses fairly standard drivers.If you step up to modern USB3.0 solutions you can get 1080p 60fps or 4K 30fps streaming in. These things are available as off the shelf solutions as a CMOS sensor on a PCB with a super speed USB port coming out, or with the thing assembled into a nice aluminum case and sold as industrial cameras.If what you are trying to build is a clone of the FLIR thermal cameras with the thermal image fusion feature, then all you need is a reasonably grunty ARM MCU with a parallel camera interface and LCD controller, letting you easily reach a power consumption of 1W. Linux is not needed at all, but might be very handy if you want fancy features like recording video to mp4 or streaming video over wifi and all that jazz.