high speed spi master - Page 1
Forum section: electronics / projects

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 01:11:26 am
content: I have a sensor that acquires data at a rate of about 20 Mbps that needs to be delivered to a CPU. The sensor uses a CPLD to packetize and shift out the data, so I can make it a master or slave on whatever protocol I want. Currently it's acting as an McBSP master running at 30 MHz to push the data out to the CPU. For reasons that don't matter to this post, I'd like to switch to a new CPU, but unfortunately the one I've chosen does not have McBSP - it does have a SPI that can run up to 50 MHz, but it's master-only.So my question is, how can I get this data that's being autonomously generated on the sensor into the CPU, when the CPU has to function as the master, without losing any data, and with very limited buffering available on the sensor?My first thought was to have the sensor send a flag whenever data is available (data comes out in 32-bit chunks, so it'd throw this flag roughly every 1.4us. I could then use this flag to trigger an interrupt on the CPU to pull in the data. Unfortunately it looks like the interrupt latency of a kernel module on a typical embedded Linux system is on the order of 5-10us, during which the CPLD would blow through any buffering it's capable of and throw away a bunch of data.My second thought was to have the CPU just run the SPI as fast as it can, constantly pulling in data, and use a header bit/byte on the CPLD to indicate whether the data it's sending is valid or empty because the buffer is empty. Unfortunately this would waste a lot of clock cycles on the CPU pulling in and sorting through garbage. I could reduce the clock to minimize the garbage, but then I risk a brief interruption causing it to miss data.My last thought was to let the CPLD run as a master, and stick a 3rd device with a bunch of RAM in between the sensor and the CPU to act as a big fifo. The sensor can push its data into this device at its standard rate, the device can throw a flag whenever a chunk is available, and the CPU can pull it in when it gets around to it. Then the question is, what kind of device would be a good fit to act as this fifo? Maybe a Cypress PSoC? I've never used them before.Any other thoughts/suggestions?I should add that price is of little concern if a 3rd device needs to be added into the mix. The final product will be on the order of $xxxx, so an extra $10 here or there doesn't matter much. Power consumption is a priority though, <10mW is in the noise and wouldn't matter, but much more than that and it'll start to be a problem.

author: MarkF
author_rank: Super Contributor
timestamp: December 11, 2015, 02:05:17 am
content: What is the sensor?My thought is that just because the sensor acquires the data at a high sample rate doesn't mean you have to read the data at that rate. It just means that it is the fastest data you can receive. I would expect you could request the data from the sensor at the rate you require while the sensor would return the latest sample at that time. Other samples would be discarded by the sensor.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 02:07:39 am
content: Quote from: MarkF on December 11, 2015, 02:05:17 amIt's for an SDR application, and the sensor is already producing data at the minimum rate for the bandwidth required, it can't be decimated any further. ALL samples need to be pulled in by the CPU. I've already verified the CPU is capable of processing the data in real time, now the only task is to get the data in.

author: MarkF
author_rank: Super Contributor
timestamp: December 11, 2015, 02:15:38 am
content: Can the sensor interrupt your processor when a sample is available? That would be the best use of processor time if the processor is using master mode.I have not done slave mode on the processor side, but that sounds like what you will need to do.

author: AndreasF
author_rank: Frequent Contributor
timestamp: December 11, 2015, 09:50:28 am
content: Quote from: suicidaleggroll on December 11, 2015, 01:11:26 amThis sounds like a prime example for the use of DMA, because then you don't need to deal with interrupts for every single incoming byte. That way you should be able to do buffering on the CPU side with very little involvement of the CPU. Only once the buffer is full would you trigger an interrupt (and probably use double buffering to not miss data while the CPU is processing the full buffer). This would probably be easier if the SPI peripheral on the CPU side is a slave, but should also be possible if run as a master.

author: Marco
author_rank: Super Contributor
timestamp: December 11, 2015, 02:24:06 pm
content: If he's using something like a phone oriented SOC there won't be a lot he can do with DMA, the camera MIPI interface might be abusable in that case though.Why not just add a SRAM to the CPLD?PS. don't be coy, just tell us what processor and CPLD you are using.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 04:20:01 pm
content: Quote from: MarkF on December 11, 2015, 02:15:38 amSure, but the CPU can't respond to the interrupt in time to grab the data before the next sample (and the one after that, and the one after that, ...) is available and the original one is lost.Quote from: MarkF on December 11, 2015, 02:15:38 amThe CPU can't operate in slave mode.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 04:23:23 pm
content: Quote from: AndreasF on December 11, 2015, 09:50:28 amI'm sure I could set up a big DMA'd read from the CPU, but what does the sensor respond with if there's no data? I could just throw in fill values, but then the CPU has to waste time sorting through a bunch of garbage fill values to get to the real samples. I can't shake the feeling that there's a more elegant way of handling it.Unfortunately the PSoC is limited to 5 Mbps in SPI slave, so that's out as a FIFO middle-man.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 04:30:30 pm
content: Quote from: Marco on December 11, 2015, 02:24:06 pmThe processor hasn't been decided, but almost any of the new quad core A53s should be able to hit our power/performance numbers. I haven't been able to find any with a SPI slave though.The CPLD is just a Xilinx Coolrunner II, nothing special. I was hoping I wouldn't have to modify the sensor board, but maybe the SRAM could be placed on the CPU board.

author: MarkF
author_rank: Super Contributor
timestamp: December 11, 2015, 04:39:05 pm
content: If you are building hardware with the sensor and cpu, why not use some dual port memory or FIFO memory to buffer the samples between the sensor and the cpu. The cpu just needs to be able to read and process the samples faster than the sensor is writing.

author: Howardlong
author_rank: Super Contributor
timestamp: December 11, 2015, 04:50:17 pm
content: Some DMAs have compare/pattern match facilities, so you could send out a flag (eg 0xFFFFFFFF) to a pattern match DMA which continually polls with no CPU interaction and then chains to a regular ping-pong DMA directly following the pattern match.

author: nctnico
author_rank: Super Contributor
timestamp: December 11, 2015, 04:50:59 pm
content: Use USB instead of SPI. USB is much easier to interface with a SoC and solves much of the bandwidth and flow control issues.

author: Howardlong
author_rank: Super Contributor
timestamp: December 11, 2015, 04:55:36 pm
content: Quote from: suicidaleggroll on December 11, 2015, 01:11:26 amQuote from: suicidaleggroll on December 11, 2015, 04:30:30 pmnon comprende <scratches head>

author: Fungus
author_rank: Super Contributor
timestamp: December 11, 2015, 05:06:20 pm
content: Quote from: suicidaleggroll on December 11, 2015, 04:20:01 pmIt can if it's a ARM running at 1GHz...(or even much less than that)

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 05:19:04 pm
content: Quote from: Howardlong on December 11, 2015, 04:55:36 pmI should have said the CPU architecture/class/another-name...as none of the ones I've been able to find that meet the power/performance specs needed for the application have a SPI slave. They all have the same ~50 MHz SPI master, presumably because they're developed for phones/tablets that have no need for a SPI slave on the host CPU.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 05:27:03 pm
content: Quote from: Fungus on December 11, 2015, 05:06:20 pmDo you have any evidence of that? Everything I can find points to >10us latency on the interrupts in Linux regardless of the CPU clock. It's unfortunate, but it makes sense. It's not running bare metal and it's not an RTOS. I've found posts/articles for TI and Broadcom that both come to similar conclusions.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 05:44:32 pm
content: Quote from: nctnico on December 11, 2015, 04:50:59 pmThanks.I'm looking at some USB HS to parallel FIFO interface ICs, it looks like that might be a decent option.

author: Ice-Tea
author_rank: Super Contributor
timestamp: December 11, 2015, 05:55:31 pm
content: You could get something like an IceStickhttp://www.latticestore.com/products/tabid/417/categoryid/59/productid/258/searchid/1/searchvalue/ice40hx1k-stick-evn/default.aspx. Has 64k of memory, which should be plenty for buffering. Cheap, but not gonna work for 10mW

author: Karel
author_rank: Super Contributor
timestamp: December 11, 2015, 06:46:09 pm
content: If you are running Linux, keep the following in mind.Linux does not support nested interrupts and/or interrupt priorities. Not even on multi-core processors.When different (hardware) interrupts occur at the same time, only one will be served, the other has to wait.Because you wil never know when other interrupts occur and how much time they will take to service,you can not make any assumption about the worst case delay to service your interrupt handler when your interrupt occurs.I wrote our own kernel driver to aquire data from a high speed multi-channel adc chip (via SPI). At first we tried to interface the chipdirectly with the cpu (NVidia Tegra). This worked more or less at low samplerates when you driver doesnotuse the kernel'sSPI-framework but interact with the registers of the SPI peripheral directly. Using the SPI-framework is way too slow (because oflatency, the framework is cueue based).In the end I ended up with an MCU based SPI-buffer between the adc and the cpu. The SPI-buffer uses DMA and can buffer forapprox. 30 milli-seconds. It's the only way to guarantee no samples will be missed. Also, thanks to this SPI-buffer, the kernel drivercan use the SPI-framework which makes the writing of the driver much easier e.g. all the setup for DMA etc. will be done for you.In theory there could be another option by using FIQ (Fast Interrupt) but the support for it in Linux is minimal. You can only usea limited range of cpu registers and you can't use any kernel functions. This makes it practically useless.

author: Karel
author_rank: Super Contributor
timestamp: December 11, 2015, 07:26:53 pm
content: Quote from: suicidaleggroll on December 11, 2015, 05:44:32 pmPay attention, at high data rates you still need to buffer. The RX/TX buffer of those USB fifo chips are pretty small.The USB host controller (in the soc) needs to generate an interrupt in order to deliver the data.The same problem as I wrote above occurs here. Usually, those USB fifo chips have a "tx-enable" indicator outputthat will tell when you can write new data. Be prepared that it will indicate "do not transmit" a considerable amount of time.

author: bson
author_rank: Unknown
timestamp: December 11, 2015, 08:01:57 pm
content: DMA is going to be the answer here, but you need to be somewhat careful with the software design and hardware selection. You specifically need a DMA controller that can either interrupt on a high-water mark (<= N bytes left to transfer, or >= M transferred) rather than on the last transfer, or that has chaining support. The former gives your interrupt routine time to respond and reload the DMA destination controls to point to the next buffer; if you don't get this early interrupt you will still only have 1.4?s to reload and will get killed by latencies. ARM7T used to have a FIR interrupt that was perfect for this sort of uses, but I think it's gone in Cortex-M. (Not sure, haven't checked, but now that I think of it I can't recall seeing it mentioned in the docs.) FIR was a fast interrupt that had its own limited register set that was switched in and could nest anything. But not so great in a Linux context, and ARM7T with its lack of cache was pathetically slow running Linux since it's too big to fit into on-chip resources (which are generally single cycle, which at 70-150MHz is blindingly fast; external SDRAM and NAND flash, not so much). The chaining lets you reload in advance and set the DMA controller to the next buffer before the current one has been filled. When the buffer is full the new destination is automatically is copied to the actual controller from the CSRs and you get an interrupt indicating a filled buffer for the previous one. The advantage over the former scheme (high-water mark) is you always get full buffers of a fixed size. The Linux kernel is used to reserve DMA channels, but you still need to make sure the right hardware capabilities are there. For something like a disk controller that transfers a fixed size that's known in advance and doesn't continuously stream any old dumb controller is fine.Another option is to use a Zynq 7000 and just put all this stuff on the FPGA fabric as a bus peripheral with SPI inputs, its own soft DMA controller and/or FIFO.

author: Karel
author_rank: Super Contributor
timestamp: December 11, 2015, 08:07:45 pm
content: I have to make a small correction. On a multi-core cpu, Linux can handledifferentinterrupts concurrently.In theory it's possible to instruct the kernel to move all interrupts to cpu0 and your interrupt to cpu1 (on a dual-core cpu).However, depending on the architecture/soc/cpu some interrupts can not be moved.

author: Marco
author_rank: Super Contributor
timestamp: December 11, 2015, 08:17:33 pm
content: Quote from: Karel on December 11, 2015, 06:46:09 pmPreempt-rt fixed that, it's been getting a little more love and care recently too.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: December 11, 2015, 11:39:34 pm
content: Thanks for all of the input. I was looking at the FTDI FT2232H to go from parallel FIFO to USB. It could keep up with the data rate, but the 4kB buffer would run out in about a millisecond, and I'd be worried about other activity on USB hanging the bus longer than that.Right now I'm thinking a second CPLD/FPGA with a big SRAM to act as a buffer. The sensor could push data in at its standard rate, the CPLD could throw a flag when X bytes are available to read, and the CPU could pull it off when it gets around to it. 2MB of SRAM isn't too expensive and would last nearly a second. That's probably overkill, but I could scale it back once I got a feel for the performance.

author: Howardlong
author_rank: Super Contributor
timestamp: December 12, 2015, 12:40:50 am
content: FTDI is expensive glue if you don't mind me saying.

author: Karel
author_rank: Super Contributor
timestamp: December 12, 2015, 08:45:22 am
content: Quote from: suicidaleggroll on December 11, 2015, 05:19:04 pmAlso, the (vanilla) Linux kernel's SPI-framework does not support SPI-slaves.Some soc manufactures however, provide patched kernel sources that do support slaves.

author: Karel
author_rank: Super Contributor
timestamp: December 12, 2015, 08:51:27 am
content: Quote from: suicidaleggroll on December 11, 2015, 11:39:34 pm+1.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: January 10, 2016, 01:10:07 am
content: Took about a day for the part selection and schematic, a day for the layout, a day to write the VHDL, a few hours to populate it, and a day for debugging, and the board is up and running.It's more of a general purpose FIFO buffer right now on its own dedicated board, with its own power input, on-board switching DC/DC converters, voltage selection jumper, LEDs showing the current RAM usage, I2C interface for querying the RAM usage, etc. Dedicated 12 pins for input and 12 pins for output, that can be used to implement whatever protocol is necessary. Right now I have it running SPI in and SPI out and it's humming along happily at 50 MHz with a Raspberry Pi as a test fixture.I only populated a 512 kB RAM chip for now, but there are alternatives up to 2 MB in the same package and pinout. It's using a CoolRunner II XC2C256. It should be able to run up to about 10 MB/s either serial or parallel. With some work on the SRAM interface it could probably get up to 20 MB/s, but that's not necessary for this application.I'm calling it the "bufferizer":

author: Karel
author_rank: Super Contributor
timestamp: January 10, 2016, 09:16:21 am
content: Well done.What kerneldriver have you been using? Did you write it yourself?Do you have an order number for those patchcables on the headers?Thanks.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: January 10, 2016, 04:45:44 pm
content: Quote from: Karel on January 10, 2016, 09:16:21 amI'm using spidev on the RPi for now, it's just running in bursts for testing. Once I switch over to the final SOC I'll see what kind of speed it can sustain and go from there. I'm hoping I don't have to write my own kernel module, but we'll see.I think those jumpers are the sparkfun ones:https://www.sparkfun.com/products/10898They're a bit expensive, but sparkfun is right around the corner from my office, so it's convenient.

author: Kalvin
author_rank: Super Contributor
timestamp: January 10, 2016, 04:54:48 pm
content: Upgrade the CPLD to FPGA. Buffer the samples in the FPGA until you read them in bursts/blocks.