ARM External Bus Controller Timing - Page 1
Forum section: electronics / general

author: SimonR
author_rank: Regular Contributor
timestamp: May 06, 2016, 09:36:12 pm
content: This question is a bit specialised but I wondered if anyone else has seen the same problem with the external bus controller on any ARM based microcontroller.I have a current design that uses an Atmel SAM9 device, the at91Sam9XE to be exact, with three external peripherals connected to it.One of the operations I do is to burst a block of data from one peripheral to the other one.From memory this should take about 130nS, 80nS read and 70nS Write more or less.Interestingly the time the code takes to run internally is completely hidden from this process so the read and write operations are truly back to back.However the actual read write time is about 150 - 200nS longer and the reason for this, that the data sheet does not make obvious, is that whenever the bus controller changes direction it takes an extra 150-200nS. I cant remember the exact times but if I find the logic analyser traces I'll post them.To confirm this I did 2 reads in a row and it takes 160nS, 4 reads takes 320nS. The same happens for writes, the timing is exactly what you would expect.OK I thought there is another option. Lets read all the data into internal SRAM first as its got a 5nS access time and avoids changing bus direction more than once.But that didn't work either. Even writing to internal RAM incurs the cost of a direction change.I've reduced the impact a bit by doing 8 reads to 8 cpu registers and then 8 writes so there is now only get one direction change every 8 cycles, but its not really good enough.I wondered if anyone else had noticed this problem on the processor they are using.The reason I asked is that the next version of the board will get a new processor and I want to avoid this problem if I can.I spoke to ST and they seemed to think their devices would do the same thing, although they weren't completely certain.Reading the data sheet may not help either.

author: jnz
author_rank: Frequent Contributor
timestamp: May 07, 2016, 07:30:29 pm
content: I'm not following the topic really... But are you running into the "speed" settings of the IOs? I don't remeber if it's STM32 or CMSIS that specs that each output has a "speed" of 2, 10, 50Mhz... I can't remeber if they were talking about switching speed or a programmable slew.

author: exmadscientist
author_rank: Frequent Contributor
timestamp: May 07, 2016, 09:10:18 pm
content: It's hard to say without more detail, but I suspect you are running in to the details of the AHB bus matrix implementation inside the CPU. Atmel, of course, does not provide a bus matrix diagram for that part (that I could quickly find).What is the data source, sink, size, and relevant clock rates (core, peripheral, and external if applicable)? That should shine some more light on what is reasonable.I also have to mention that expecting hard realtime response from an MCU is sometimes not feasible. An FPGA might be a better fit for your needs. Newer Cortex-M parts also typically have enhanced internal buses but changing architectures has its own challenges, as always.

author: rstofer
author_rank: Super Contributor
timestamp: May 08, 2016, 05:57:44 am
content: Take a look at the block diagram and see where the GPIO pins are located. If they are on the other side of an AHB contoller from the CPU, they won't run at CPU rate. In the early years, the LPC2106 was a dandy ARM7 but people complained that they couldn't toggle the pins fast enough. The LPC2148 came along and there was a Fast I/O bus that didn't cross through an AHB controller. Instead the pins showed up in two places in the block diagram. Same pins, just two different ways to get there.You can download the relevant documents and see how this was done.

author: Kilrah
author_rank: Unknown
timestamp: May 08, 2016, 09:09:37 am
content: Quote from: exmadscientist on May 07, 2016, 09:10:18 pm+1.ARM MCUs aren't like your good old 8-bitter where peripherals are directly in memory space and access is instant, they're complex beasts with multiple shared buses and interconnects with boundaries to cross - while your peripherals are still memory-mapped, they may reside on a different bus that has a different clock speed than your CPU requiring going through a bus controller, the bus you need to use to talk to them might be held busy for a while by other peripherals (e.g. during a DMA transfer) etc, so precise timing is hard to predict and actually quite variable.A few months ago I came across a very interesting article from a guy who implemented complex 3D video routines on an STM32 and made a demo, and he went into great detail explaining the bus matrix and how to choose paths to minimize bus contention, but unfortunately I can't seem to find it anymore

author: andersm
author_rank: Super Contributor
timestamp: May 08, 2016, 10:33:21 am
content: If it is the bus matrix, according to the documentation it can be reprogrammed from the default round-robin configuration to a fixed-priority scheme.

author: SimonR
author_rank: Regular Contributor
timestamp: May 08, 2016, 07:29:20 pm
content: Thanks for the replies so far.Some more information.The CPU speed is I think 198Mhz, its what was set in the project I based this on, which just shy of 200Mhz.My peripherals are NOT connected to GPIO lines they are properly memory mapped onto the external Address and Data busses.This is an intermetiate microcontroller half way between SOC and pure microprocessor so it has all the usual onboard peripherals, 512K of Flash and 32K of RAM but it also has a proper external bus.There are 8 decoded chip selects each accessing 256 Meg of address space.The bus controller is fully programable and in theory you could programe it with a 5nS access time but in practice only 20nS or faster is practical.Kilrah you are correct there are several clock domains to cross between the bus and the CPU.The ARM core is conneted to a 6 layer Matrix (possibly 6 AHB busses, it doesn't say) and that is connected to the EBI (external bus interface).The EBI contains an SDRAM controller, am SRAM controller and a NAND flash controller. My peripherals are connected to the SRAM controller.Havinig said that if I set the timing to be 80nS access time then an ARM instruction should see 80nS. and it does see exactly that for sucessive reads or sucessive writes.The external peripheral are8Mbit SRAM 16 bits wide (10nS but not currently used )Compact Flash 16 bits wide memory maped (uses the SRAM controller in CF mode)(fastest card is 80 ns)An external comunications device 16 bits wide (80nS writes 70nS writes)What I am trying to achieve is 10 Mbytes/S in a burst between peripheral and CF card, which is 5Mwords/S which is pushing the two devices close to the limit bus is entirely acheivable without the turn around issue.I would consider using compact flash UDMA mode but a design cockup prevents this, so I am stuck with the basic bus timings.I am considering an FPGA next time but other consideratins may get in the way of that idea.

author: nctnico
author_rank: Super Contributor
timestamp: May 08, 2016, 07:57:34 pm
content: If the ARM controller has a DMA engine: have you tried using it to see if it works better/differently?

author: SimonR
author_rank: Regular Contributor
timestamp: May 08, 2016, 08:34:16 pm
content: Andersm thanks for the round-robin tip I hadn't spotted that, although I still think the EBI is the cause, its worthy of further investigation.nctnico unfortunately DMA is a whole other problem that I had intended to post seperately.This part doesn't have DMA that is of any use to me as there is no DMA available on the external bus, it doesn't even have memory to memory DMA internally or externally.The only DMA is between onboard peripheals, and that is true of almost every modern device on the market.I couldn't use memory to memory anyway because the comms peripheral uses good old fashioned REQ/ACK signals connected to an 18 word FIFO. Memory to Memory requires that you know in advance how much data to transfer, and I don't.I had to implement DMA in software and that led to this problem. The software implementation is unbelievably quick considering it uses the Atmel GPIO library, easily as fast of faster than the 1980s hardware it replaced. The proble is that each REQ/ACK results in a READ/WRITE as described above.Rumour has it that Freescale and I think NXP do actually have proper external DMA pins available but I have not yet invesigated.

author: Kilrah
author_rank: Unknown
timestamp: May 08, 2016, 08:41:57 pm
content: Quote from: SimonR on May 08, 2016, 08:34:16 pmI might be talking crap, but couldn't you investigate "cheating" and duplicating some of the connections to GPIOs, and handle the REQ/ACK transations separately there in order to avoid changing direction on the main bus?

author: SimonR
author_rank: Regular Contributor
timestamp: May 08, 2016, 09:53:05 pm
content: I don't think you can cheatIf your thinking of handling REQ?ACK in a seperate thread (so to speak) from the READ/WRITE, then no because every REQ?ACK must have a read.Then you have to store the value somewhere.If you are thinking of doing a number of ACKs and only then doing a number of reads then no, the peripheral chip gets confused. I effectively tried this when debugging the DMA code.If you are interested in how the DMA code actually works then here it is almost entirely. I had to merge the c file and the header but its pretty much complete.If anyone else finds it useful the please feel free to use itCode:/*** \file DMA.c* \brief functions for the DMA engine*//** Includes*//* Standard headers */#include <stdint.h>#include <assert.h>/* Library headers */#include <board.h>#include <pio/pio.h>/* Dependencies */#include "DMA.h"/** Constants*/#define PIN_DMA_REQ {1 << 13, AT91C_BASE_PIOC, AT91C_ID_PIOC, PIO_INPUT, PIO_DEFAULT}#define PIN_DMA_ACK {1 << 1, AT91C_BASE_PIOC, AT91C_ID_PIOC, PIO_OUTPUT_1, PIO_DEFAULT}#define DMA_MODE_IO2MEM (0x20) /**< univerasl(I/O to memory) */#define DMA_MODE_MEM2IO (0x40) /**< univerasl(memory to I/O) *//** Type definitions*/typedef struct DMA_s {uint8_t Mode;uint16_t *Memory;volatile uint16_t *Peripheral;volatile uint8_t *Peripheral8;} DMA_t;/** Private prototypes*/static void DMA_Request_ISR(void);static void DMA_Process_8Bit(void);static void DMA_Process_16Bit(void);/** Public data*/DMA_t DMA;/** Private data*/const Pin pinDmaAcknowledge = PIN_DMA_ACK;const Pin pinDmaRequest = PIN_DMA_REQ;/**********************************************************************\* Public functions\**********************************************************************/void DMA_Initialise(void){DMA.Mode = 0;DMA.Memory = 0;PIO_Configure(&pinDmaRequest, 1);PIO_Configure(&pinDmaAcknowledge, 1);PIO_ConfigureIt(&pinDmaRequest, (void (*)(const Pin *))DMA_Request_ISR);}void DMA_EnableInterrupt(void){PIO_EnableIt(&pinDmaRequest);}void DMA_Set_Mode(uint8_t mode){DMA.Mode = mode;}uint8_t DMA_Get_Mode(void){return DMA.Mode;}void DMA_Set_Memory(uint16_t *value){DMA.Memory = value;}void DMA_Set_Peripheral0(uint16_t volatile *value){DMA.Peripheral = value;DMA.Peripheral8 = (uint8_t *)value;}/**********************************************************************\* Private functions\**********************************************************************/#define PIO_PDSR (AT91_CAST(AT91_REG *)0x0000003C) // (PIO_PDSR) Pin Data Status Register#define PIO_SODR (AT91_CAST(AT91_REG *)0x00000030) // (PIO_SODR) Set Output Data Register#define PIO_CODR (AT91_CAST(AT91_REG *)0x00000034) // (PIO_CODR) Clear Output Data Register#define PORTC_READ*((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_PDSR)))#define PIN_DMA_REQEST_MSK (1 << 13)#define PORTC_SET*((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_SODR)))#define PORTC_CLEAR*((volatile unsigned long*)(((int)AT91C_BASE_PIOC + (int)PIO_CODR)))#define PIN_DMA_ACK_MSK (1 << 1)/*** \brief DMA Interrupt handler* \details optimised to use port registers directly* Peripheral addresses don't change so pointer stays fixed* Memory addresses do chage so pointer increments*/static void DMA_Request_ISR(void){if(DMA.Mode & DMA_W_WORD)DMA_Process_16Bit();elseDMA_Process_8Bit(); // this is almost identical to 16Bit}static void DMA_Process_16Bit(void){uint16_t *ptr = DMA.Memory;uint16_t volatile * periph = (uint16_t *)DMA.Peripheral;uint32_t volatile * Request = (uint32_t *)&PORTC_READ;uint32_t volatile * SetAck = (uint32_t *)&PORTC_SET;uint32_t volatile * ClearAck = (uint32_t *)&PORTC_CLEAR;if((DMA.Mode & 0x60) == DMA_MODE_IO2MEM){while(*Request & PIN_DMA_REQEST_MSK){*ClearAck = PIN_DMA_ACK_MSK;*ptr++ = *periph;*SetAck = PIN_DMA_ACK_MSK;}}else // must be mem to IO then{while(*Request & PIN_DMA_REQEST_MSK){*ClearAck = PIN_DMA_ACK_MSK;*periph = *ptr++;*SetAck = PIN_DMA_ACK_MSK;}}DMA.Memory = ptr;}/**********************************************************************\End of file Dma.c\**********************************************************************/It looks like I did have to optimize it afterall. It uses the Atmel Library to set up and then the registers directly

author: Jeroen3
author_rank: Super Contributor
timestamp: May 09, 2016, 06:04:00 am
content: NXP has dual core parts where you can use the extra M0 as dma. Some are flashless, so I'd expect the external bus behavior to be well documented.

author: westfw
author_rank: Super Contributor
timestamp: May 09, 2016, 07:21:35 am
content: I doesn't surprise me that any modern external memory controller optimizes for consecutive reads/writes at the expense of "turnaround" time. After all, cache-fill, fast page mode, etc, etc. It does surprise me that this isn't documented carefully (although I guess it falls between the pieces that ARM would document (bus) and the parts that the individual vendor would document (memory controller. Maybe? Is that one of the pieces you can buy from ARM as well?)Did you see this bit in the datasheet:Quote

author: SimonR
author_rank: Regular Contributor
timestamp: May 09, 2016, 09:44:26 am
content: Thanks westfwYes I did see that bit of the datasheet, unfortunately not before the board was designed. It was not the highest priority to fix and luckily it still just about meets the spec.The actual question in this thread is to ask if there is another chip I can use that doesn't have this problem. The next version needs to be faster but I am still stuck with the demand driven DMA architecture.Further down the line there will be an FPGA solution but not this time around as the requirements are not yet set.

author: Jeroen3
author_rank: Super Contributor
timestamp: May 09, 2016, 11:11:25 am
content: Is that single wait state that critical? If so, isn't it better to get an application processor that is running faster than 180 MHz?Such as the category used on a raspberry pi.

author: SimonR
author_rank: Regular Contributor
timestamp: May 10, 2016, 09:31:54 am
content: Jeroen I fear you may be right.That single wait state is unbelievably critical.The irony here is that the ARM replaced a 1980s 16 bit microcontroller running at 20MHz and it only had a fixed 100nS bus cycle itself.It seems total madness that I might have to go to 1GHz processor just to get a fast enough bus interface.I don't want to as I need on board flash as I don't have the board space.

author: Jeroen3
author_rank: Super Contributor
timestamp: May 10, 2016, 10:11:31 am
content: You know you can use quad spi flash? That is relatively fast. And if you copy the right parts of code to ram, even faster.Meanwhile, can you overclock the EBI?

author: nctnico
author_rank: Super Contributor
timestamp: May 10, 2016, 10:33:30 am
content: An alternative is using SD cards since many microcontrollers have a dedicated SD card controller which (most) probably can use DMA.

author: SimonR
author_rank: Regular Contributor
timestamp: May 10, 2016, 03:44:46 pm
content: Its not flash speed thats the problem its board space. quad flatpaks take quite a bit of space. Going to BGA is not a desirable option in this case because it changes the production process and adds 2 more layers to the PCB.Overclocking the EBI? I hadn't thought of that, I'll investigate.SD cards might be an option later, its already on the list but we need to be backward compatible this time so we are stuck with CF cards.

author: knielsen
author_rank: Newbie
timestamp: May 11, 2016, 10:22:57 am
content: A number of the STM32 MCUs (STM32F469 for example) have a core-coupled SRAM memory. This is not on the AHB or bus matrix, but connected directly to the Cortex M4 core.So if the DMA code is put in the core-coupled SRAM, along with the intermediate buffer, I'm guessing that arbitrary amounts of words could be buffered between switching direction (well, up to 64 kByte).However, I'm a bit confused - you write that you can do back-to-back reads at full speed (and similar writes) and buffer in registers. But your code also does a read/write switch per word to handle REQ/ACK - doesn't that prevent the speedup from buffering? Or is it only when the external ram controller switches direction that you get the penalty?I any case, it must be the external memory controller that has the penalty from switching direction, right? At least I am pretty sure that the high-end STM32's (that I am most familiar with) can easily read/write access internal busses much faster than that.If you are interested, I have a custom board with an STM32F405 and an external SRAM on the FSMC (ST's external memory controller). I could do some testing to check the speed of reading/writing when switching direction?Also, the FSMC has some extra signals available, like an external WAIT signal and such. Maybe your MCU has something similar, perhaps with a bit of trickery those can be used to handle the REQ/ACK protocol internally in the controller?

author: SimonR
author_rank: Regular Contributor
timestamp: May 11, 2016, 10:46:07 am
content: Hi knielsenThanks for the offer, theres nothing like having proof that something works before buying it.Sorry for the confusion.Yes my actual production code does READ WRITE READ WRITE because of the demand driven DMA problem.I wrote some seperate test code that did 4 reads in a row just to prove the point. This was from the CF card as I can't set up a test from the DMA peripheral.I did consider writing to internal SRAM and then to external CF (and the otherway round ) but my collegue reminded me that the wait state still applied to internal RAM as well.Its about a year since the test so I need to run this test agian to confirm it. It may be true because I remember swearing heavily at the time.I'll see if I can find my logic analyser traces, as I think they may be helpful.

author: SimonR
author_rank: Regular Contributor
timestamp: May 11, 2016, 11:49:57 am
content: I found the email trail to ATMEL which I'll sumarize later but here are the actual timings.Basically Atmel confirmed that its a bus matrix problem, when the master in controll of it changes.The exact code running here is:-Code:*periph; //0x204024 0xe1d230b0 LDRH R3, ; this always reads because its volatile before any C programmers complain*buffer++ = *periph; //0x204028 0xe1d230b0 LDRH R3,//0x20402C 0xe0c030b2 STRH R3,, #0x02And its running out of cache so has zero impact on bus timing.The timing for the *periph; section is the first traceand *buffer++ = *periph; is the second.

author: Jeroen3
author_rank: Super Contributor
timestamp: May 11, 2016, 11:58:39 am
content: Quote from: knielsen on May 11, 2016, 10:22:57 amOnly the D-bus of the cortex m4 has access to this. No instructions and no dma access to there.

author: SimonR
author_rank: Regular Contributor
timestamp: May 11, 2016, 12:17:23 pm
content: the ST32 may still solve the problem if its not connected to any bus matrix type architecture.Also if anyones wondering why this is such a problem, its because we sometimes access the whole card in a single transaction.That means that for 64Gig card it adds an extra 82 minutes. 144 nS * 32Gig Reads = 4947 seconds.Its only 5 minutes on a 4Gig card which is why its not a problem now. But when the card sizes increase it will become a problem.(it is a problem but we can live with it)

author: knielsen
author_rank: Newbie
timestamp: May 11, 2016, 01:46:02 pm
content: Ok, so this was quite interesting. I made a quick ('n dirty) test programusing an STM32F405 MCU connected to an external 16-bit SRAM IC. The MCU isrunning at 168 MHz (6 ns CPU cycle). The SRAM has 55 ns access time.The FSMC (memory controller) is configured like this:1 BUSTURN delay after read6 DATAST data-phase duration0 ADDHLD address-hold phase duration3 ADDSET address setup phase durationThat is something like 10 CPU cycles per FSMC transaction (60 ns to matchthe 55 ns access time), plus any overhead. The exact timing would need to bedetermined for the requirements of your specific peripherals, of course.I tested single and burst read, single and burst write, read followed bywrite, and GPIO read + SRAM read + SRAM write + GPIO write (this last one toresemble your DMA code). For example:do{(void)*gpio_read_ptr;(void)*sram_ptr;*sram_ptr = some_value;*gpio_set_ptr = gpio_set_val;-- i;} while (i != 0);The results show that an external SRAM read or write takes 12 CPU cycles, or72 ns, in all the cases basically. I see no penalty for changing directionon the bus. In fact, the overhead of adding the GPIO read/write is exactlyzero cycles! I believe this is because the FSMC has a CPU cycle or two ofdelay between transactions, which can be overlapped with the GPIO accesses.Put another way, I am getting 14 MByte/second bandwidth copying external ramto external ram on the STM32F405, with this particular SRAM IC.So basically, the ST's Cortex M4 in my tests seem to be able to communicateon the external bus at the full configured speed. Just note that the M4usually takes 2 CPU cycles (12 ns) to do a store or a load (plus any waitstates), which sometimes can go to a single cycle if load and stores arescheduled optimally in the code.But I noticed that the at91Sam9XE you are using seems to be a full ARMapplication processor with MMU and such? The STM32F405 is a Cortex M4microcontroller, it does not have MMU or caches or other such things...These ARM chips are designed specifically with a mind to get fast andpredictable timing for things like peripheral access and interupts and such,which might explain the difference...If you are interested, you can find my test code on my github (note thenon-default branch 'eevblog', I hacked an old test program I had):https://github.com/knielsen/stm32f4_sram_memtest/tree/eevblogHope this helps,- Kristian.

author: SimonR
author_rank: Regular Contributor
timestamp: May 11, 2016, 10:53:09 pm
content: Thats brilliant, thanky so much for doing that. I might get an ST32 dev board and do some testing.Amazing an application processor out gunned by an embedded one.That processor was picked for its CF card interface, external bus and on boarf FLASH/SRAM. Thats all we needed, although we do use the ethernet now as well.You would have thougkt that a 5 nS instruction cycle would be more than enough, and actually it is if you have a good bus controller.

author: knielsen
author_rank: Newbie
timestamp: May 12, 2016, 02:52:05 pm
content: As another option, have you considered bit-banging the external bus? Since you already need to bit-bang the REQ/ACK, maybe doing the whole bus protocol yourself would be a reasonable option. This is assuming that your CPU can do GPIO in few tens of nanoseconds, and that your hardware pinout makes it possible, of course...- Kristian.

author: SimonR
author_rank: Regular Contributor
timestamp: May 12, 2016, 03:18:59 pm
content: I did consider that that, I think the GPIO lines are more than fast enough, because I already use them for the DMA signals. The trouble is the Address and Data busses are completely seperate from the GPIO lines so it would be a board re-spin.

author: krho
author_rank: Regular Contributor
timestamp: May 12, 2016, 03:39:29 pm
content: According to the manual STM32F4xxx famil with external FSMC also has a support for CF cards.Quote

author: SimonR
author_rank: Regular Contributor
timestamp: May 12, 2016, 06:09:05 pm
content: Quote from: krho on May 12, 2016, 03:39:29 pmThat makes it even more interesting