Running parallel RAM at a different clock than CPU? - Page 1
Forum section: electronics / general

author: Artlav
author_rank: Frequent Contributor
timestamp: November 14, 2016, 12:52:59 am
content: Let's say i have a parallel static RAM that is running at one frequency (8 MHz), and a CPU (in the FPGA) that is running at a higher frequency (16MHz).The CPU asserts a read or write signal, the RAM gets accessed, sets a done flag for one clock, the CPU sees the flag and deassert the request signal.It all works fine when they both use the same clock, but if i lower the RAM clock (either from dividing the CPU one or async from a PLL), things start to lock up and act weird.I suspect this needs some sort of synchronization. I tried to sync the done flag (running theory was that the flag stays up as the CPU carries on and messes something up on next reads) and read and write flags (just in case), as described here:http://fpga4fun.com/CrossClockDomain2.htmlHowever, that had no effect (maybe even made it worse, since with the sync it locks up almost reliably).What can i be doing wrong?How is this normally done?The code for the RAM controller:Code://----------------------------------------------------------------------------//module ffram_controller(input rst,input clk,output reg [14:0] fram_addr,inout [15:0] fram_io,output reg fram_oe,output reg fram_we,output reg fram_ce,input [31:0] addr,input [31:0] data2mem,output reg [31:0] data2cpu,input re,input we,output reg done);//----------------------------------------------------------------------------//reg [2:0] state;reg [15:0] xlate_data;wire [14:0] xlate_addr;//----------------------------------------------------------------------------//parameter [2:0] idle=3'd0;parameter [2:0] re_a=3'd1;parameter [2:0] re_b=3'd2;parameter [2:0] we_a=3'd3;parameter [2:0] we_b=3'd4;parameter [2:0] we_c=3'd5;parameter [2:0] skip=3'd6;//----------------------------------------------------------------------------//assign xlate_addr=addr[15:1];//----------------------------------------------------------------------------//always @(posedge clk)beginif(rst)begindata2cpu<=0;state<=idle;done<=0;fram_addr<=15'h7FFF;xlate_data<=16'hFFFF;fram_oe<=1;fram_we<=1;fram_ce<=1;end else begincase(state)idle:begindone<=0;fram_oe<=1;fram_we<=1;fram_ce<=1;if(re)beginstate<=re_a;fram_addr<=xlate_addr;fram_oe<=0;fram_we<=1;fram_ce<=0;endif(we)beginstate<=we_a;fram_addr<=xlate_addr;xlate_data<=data2mem[15:0];fram_oe<=1;fram_we<=0;fram_ce<=0;endendre_a:begindata2cpu[15:0]<=fram_io;fram_addr<=xlate_addr+1'd1;state<=re_b;endre_b:begindata2cpu[31:16]<=fram_io;done<=1;state<=skip;fram_oe<=1;fram_we<=1;fram_ce<=1;endwe_a:beginfram_we<=1;state<=we_b;endwe_b:beginxlate_data<=data2mem[31:16];fram_addr<=xlate_addr+1'd1;fram_we<=0;state<=we_c;endwe_c:begindone<=1;state<=skip;fram_oe<=1;fram_we<=1;fram_ce<=1;endskip:begindone<=0;state<=idle;endendcaseendend//----------------------------------------------------------------------------//assign fram_io=we?xlate_data:16'bzzzzzzzzzzzzzzzz;//----------------------------------------------------------------------------//endmodule//----------------------------------------------------------------------------//

author: hamster_nz
author_rank: Super Contributor
timestamp: November 14, 2016, 03:59:31 am
content: Quote from: Artlav on November 14, 2016, 12:52:59 amTry synchronizing the 're' and 'we' signals into the 'clk' clock domain, and see how you get on. It will add some latency, but such is life.Also, are you sure that the CPU checks that 'done' has been dropped before issuing the next transaction?Depending on protocol, the sequence of events for handshaking between unsynchronized clock domains might be:- address and possibly data is presented- 're' or 'we' is asserted, ensuring address and data are stable when this is asserted.- controller sees 're' and 'we' and performs transaction- data from memory to CPU is presented- 'done' is asserted, when data is stable.- CPU sees 'done' asserted, and can consume any data presented.- 're' or 'we' is removed- 'done' is removed- CPU sees 'done' is removed, and is free to issue next transaction.'re' and 'we' need to synced into the target memory controller domain, and 'done' needs to be synchronized into the CPU clock domain.Because the synchronizer inserts a delay. this can be shortened to fewer states:- address and possibly data is presented; 're' or 'we' is asserted at the same time- controller sees synchronized 're' and 'we' and performs transaction- data from memory to CPU is presented; 'done' is asserted at the same time- CPU sees synchronized 'done' asserted, and can consume any data presented; 're' or 'we' is removed- controller sees synchronized 'we' or 're' drop, and in response 'done' is removed- CPU sees synchronized 'done' is removed, and is free to issue next transaction.(short answer - keep in one clock domain if at all possible and have wait states)

author: Artlav
author_rank: Frequent Contributor
timestamp: November 14, 2016, 09:06:10 am
content: Quote from: hamster_nz on November 14, 2016, 03:59:31 amTried that as mentioned in the post, with no improvement.Even worse, this makes lockup happen rather reliably (almost each access instead of only happening on reading in quick succession).The exact instantiation looks like this:Code:wire fram_done;wire fram_rd;wire fram_wr;clock_cross fram_x_done(.clk_a(fram_clk),.flag_in_clk_a(fram_done),.clk_b(cpu_clk), .flag_out_clk_b(mem_done));clock_cross fram_x_rd (.clk_a(cpu_clk), .flag_in_clk_a(mem_rdreq),.clk_b(fram_clk),.flag_out_clk_b(fram_rd));clock_cross fram_x_wr (.clk_a(cpu_clk), .flag_in_clk_a(mem_wrreq),.clk_b(fram_clk),.flag_out_clk_b(fram_wr));ffram_controller fframc(.rst(all_reset),.clk(fram_clk),.addr(mem_addr),.data2mem(mem_wrdata),.data2cpu(mem_rddata),.re(fram_rd),.we(fram_wr),.done(fram_done),.fram_addr(FRAM_ADDR),.fram_io(FRAM_IO),.fram_oe(FRAM_OE),.fram_we(FRAM_WE),.fram_ce(FRAM_CE));//...module clock_cross(input clk_a,input flag_in_clk_a,input clk_b,output flag_out_clk_b);reg flag_toggle_clk_a;always @(posedge clk_a) flag_toggle_clk_a <= flag_toggle_clk_a ^ flag_in_clk_a;reg [2:0] sync_a_clk_b;always @(posedge clk_b) sync_a_clk_b <= {sync_a_clk_b[1:0], flag_toggle_clk_a};assign flag_out_clk_b = (sync_a_clk_b ^ sync_a_clk_b);endmodule

author: hamster_nz
author_rank: Super Contributor
timestamp: November 14, 2016, 09:49:57 am
content: Quote from: Artlav on November 14, 2016, 09:06:10 amI can't see what clock_cross is doing - is it suppose to pass single-cycle pulses through regardless of relative clock speeds?It doesn't scan as right to me - If clk_b is very much slower than clk_a, and flag_in_clk_a is asserted for two cycles of clk_a then odds are nothing will change in sync_a_clk_b and the pulse will be lost.

author: Artlav
author_rank: Frequent Contributor
timestamp: November 14, 2016, 10:05:19 am
content: Quote from: hamster_nz on November 14, 2016, 09:49:57 amYes, it's supposed to translate a single-clock pulse.Quote from: hamster_nz on November 14, 2016, 09:49:57 amInitially i only tried to sync the done signal, from the slow clock to the fast.This had exactly zero effect - RAM accesses kept hanging randomly.The fast rd/wr signals syncing into the slow RAM clock causes reliable lockups, but does not feel like necessary - they would go up for as long as the done is not set, so plenty of time to be seen, and they would be deasserted once done is seen, so the slow clock would have plenty of time to not see them again.

author: ale500
author_rank: Frequent Contributor
timestamp: November 14, 2016, 12:17:31 pm
content: And what does the simulation show ? You can write an Async RAM module and simulate the access.Can you use ChipScope or Reveal Analyzer or whatever your FPGA supports to see what happens ?

author: hamster_nz
author_rank: Super Contributor
timestamp: November 14, 2016, 05:42:13 pm
content: Quote from: Artlav on November 14, 2016, 10:05:19 amI can see two scenarios that will cause a lockup - the memory controller not seeing the 're' or 'we' pulse that signals the start of the transaction (i.e. the memory controller stays idle), or the CPU misses the 'done' pulse, and is waiting forever for a reply that has already been presented.The handshaking in the controller itself looks pretty bulletproof - so the problem is most likely in the CPU interface.

author: Scrts
author_rank: Frequent Contributor
timestamp: November 14, 2016, 09:17:22 pm
content: I would highly suggest using a dual-clock FIFO with logic behind both ports to handle clock crossings. On the FPGA side you use 16MHz clock, on the memory side - 8MHz.It will be significantly easier to transit to higher frequencies this way.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: November 14, 2016, 09:25:09 pm
content: A dual clock FIFO will make this very simple, as Scrts said. I didn't notice what FPGA you're using, but I know Xilinx has a FIFO IP core built into ISE and Vivado that uses minimal resources and allows dual-clocks and different read/write data widths if you so desire.

author: hamster_nz
author_rank: Super Contributor
timestamp: November 14, 2016, 10:16:55 pm
content: Quote from: Scrts on November 14, 2016, 09:17:22 pmYeah, but the added latency will be a performance killer - it many cycles for a write to update flags on the far side of a FIFO, and then another 5 to get the response back again - so nearly a microsecond. For Xilinx, the latency for the EMPTY signals are inhttps://www.xilinx.com/support/documentation/ip_documentation/fifo_generator/v12_0/pg057-fifo-generator.pdfon page 140:"1 wr_clk + N + 2 rd_clk (+1 rd_clk uncertainty)" (where N is the stages in the synchronizer).So if you use a 2 stage synchronizers you get up to 6 cycles in each direction.I think a much better idea for static RAM is to have the memory controller run at the same frequency as the CPU, and just add extra states to meet the external memory's timings.It should only complicate the memory FSM a tiny bit, but will allow more precise timings due to the higher clock rate.

author: David Hess
author_rank: Super Contributor
timestamp: November 15, 2016, 08:55:37 am
content: I am confused about how the static RAM is running at any frequency. Isn't it an asynchronous SRAM?

author: Artlav
author_rank: Frequent Contributor
timestamp: November 15, 2016, 09:11:22 am
content: TL;DR: I've abandoned the whole idea of using two clocks, and solved the problem by using a delay/wait states to ensure the right timings at high frequency.Apparently that's the way to do it properly, and it works just fine with CPU at 50MHz.Code://----------------------------------------------------------------------------//module ffram_controller(input rst,input clk,output reg [14:0] fram_addr,inout [15:0] fram_io,output reg fram_oe,output reg fram_we,output reg fram_ce,input [31:0] addr,input [31:0] data2mem,output reg [31:0] data2cpu,input re,input we,output reg done);//----------------------------------------------------------------------------//(* syn_encoding = "user" *) reg [3:0] state;(* syn_encoding = "user" *) reg [3:0] next_state;reg [15:0] xlate_data;wire [14:0] xlate_addr;reg [1:0] skip_cnt;//----------------------------------------------------------------------------//parameter [3:0] idle =4'd0;parameter [3:0] re_a =4'd1;parameter [3:0] re_b =4'd2;parameter [3:0] re_c =4'd3;parameter [3:0] re_d =4'd4;parameter [3:0] we_a =4'd5;parameter [3:0] we_b =4'd6;parameter [3:0] we_c =4'd7;parameter [3:0] we_0 =4'd9;parameter [3:0] we_1 =4'd10;parameter [3:0] skip =4'd11;parameter [3:0] wait_time=4'd12;//----------------------------------------------------------------------------//assign xlate_addr=addr[15:1];//----------------------------------------------------------------------------//task do_idle(input [3:0] next);beginskip_cnt<=2'd2;state<=wait_time;next_state<=next;endendtask//----------------------------------------------------------------------------//always @(posedge clk)beginif(rst)begindata2cpu<=0;state<=idle;done<=0;fram_addr<=15'h7FFF;xlate_data<=16'hFFFF;fram_oe<=1;fram_we<=1;fram_ce<=1;end else begincase(state)idle:begindone<=0;fram_oe<=1;fram_we<=1;fram_ce<=1;if(re)beginstate<=re_a;fram_addr<=xlate_addr;fram_oe<=0;fram_we<=1;fram_ce<=1;endif(we)beginstate<=we_0;fram_addr<=xlate_addr;xlate_data<=data2mem[15:0];fram_oe<=1;fram_we<=0;fram_ce<=1;endendre_a:beginfram_ce<=0;do_idle(re_b);endre_b:begindata2cpu[15:0]<=fram_io;fram_ce<=1;fram_oe<=1;fram_addr<=xlate_addr+1'd1;do_idle(re_c);endre_c:beginfram_ce<=0;fram_oe<=0;do_idle(re_d);endre_d:begindata2cpu[31:16]<=fram_io;fram_oe<=1;fram_we<=1;fram_ce<=1;done<=1;state<=skip;endwe_0:beginfram_ce<=0;do_idle(we_a);endwe_a:beginfram_ce<=1;fram_we<=1;do_idle(we_b);endwe_b:beginxlate_data<=data2mem[31:16];fram_addr<=xlate_addr+1'd1;fram_we<=0;state<=we_1;endwe_1:beginfram_ce<=0;do_idle(we_c);endwe_c:begindone<=1;state<=skip;fram_oe<=1;fram_we<=1;fram_ce<=1;endskip:begindone<=0;state<=idle;endwait_time:beginskip_cnt<=skip_cnt-2'd1;if(skip_cnt==2'd0)state<=next_state;endendcaseendend//----------------------------------------------------------------------------//assign fram_io=we?xlate_data:16'bzzzzzzzzzzzzzzzz;//----------------------------------------------------------------------------//endmodule//----------------------------------------------------------------------------//Quote from: ale500 on November 14, 2016, 12:17:31 pmI rarely use simulation. It's the sort of a nuclear option that takes a while to set up, find or make the right models for all the peripherals, run, decode the results and so on, that never really gets finished since before the end of which you usually paid enough attention to enough details to spot the issue.Quote from: David Hess on November 15, 2016, 08:55:37 amYes, i might have minced words a bit.It's a regular async SRAM, the kind that got access time limits (70ns) but no specific frequency requirements (should work down to near-DC or so).The problem was that the CPU can and should run at much higher frequency than the simple one-state-per-step controller can run at while ensuring the access time limits implicitly.