Understanding FPGA timing optimisation - Page 1
Forum section: electronics / general

author: tom66
author_rank: Super Contributor
timestamp: February 14, 2019, 07:31:23 pm
content: I am working on a small, currently personal project, implementing a MIPI CSI-2 transmitter on an FPGA using a Xilinx Zynq 7010. I am fairly new to the world of FPGA design. It's taken a while for me to get to the point of being able to implement this project, probably about 6 months on and off to get a working, reliable transmitter, starting at very low clock rates*. This took a while, and a lot of reverse engineering!Now, after all this work, I have successfully got my FPGA to send data to a Raspberry Pi at 200MHz. Fantastic!However, the performance of my system degrades much beyond 250MHz. Random bit errors occur and frames are dropped periodically. I would like to reach 500MHz minimum, ideally 600MHz, so I need to improve the performance of the bus. I'm reasonably confident this is due to FPGA-sourced timing. If I heat the FPGA up, I can get bus performance to further degrade to the point at which it stops working entirely.I suspect I have issues with the setup and hold of my data transmission, so when the Pi attempts to read certain bit patterns, an error occurs. This tends to occur at specific word values (some are OK; primarily ones that involve continuous strings of 1's or 0's), which supports this hypothesis.For those who are not aware, MIPI CSI-2 receives data on the clock edge transition and data is asserted during the period between clock edges. Data is transferred on both edges of the clock (it is DDR).I am using an OSERDESE2 module connected to an OBUFTDS to generate the differential data. Two SERDESes are used, one for each data lane, and data is clocked in to each SERDES at 8MHz. To keep clock rates inside the FPGA vaguely sensible, I deal with 16-bit wide data words at relatively low clock rates (50-80MHz).I would need to set up my timing constraints to keep my data centred between the two clock edges. So I have a few questions:Is there a good guide on how to set up bus timing constraints, for external buses, with FPGAs?Is my goal of 500MHz clock (~1Gbit/s per lane) realistic using regular FPGA logic (rather than specific transceivers - to reduce the cost of implementing this design, I'd rather stick to a cheaper Spartan or Artix chip, not a Kintex or similar with high-speed transceiver modules)Using my relatively basic 100MHz oscilloscope, is there any way I can analyse bus timing (e.g. if I measure some error at 50MHz, is this error likely to be proportional to higher clock rates, or do I need to borrow a faster scope...this is an option, but it's a pain)Thanks for any input from an FPGA greenie.*It was exceedingly helpful to determine that a Pi camera can be configured to clock data at under 800kHz (and the Pi will accept such data) allowing detailed bit pattern analysis.

author: legacy
author_rank: Super Contributor
timestamp: February 14, 2019, 07:36:19 pm
content: Quote from: tom66 on February 14, 2019, 07:31:23 pmbah ...

author: tom66
author_rank: Super Contributor
timestamp: February 14, 2019, 08:09:27 pm
content: Quote from: legacy on February 14, 2019, 07:36:19 pmbah?500MHz is unrealistic for a SERDES peripheral? The actual whole FPGA will not be clocked at that frequency, only the SERDES block.

author: AndyC_772
author_rank: Super Contributor
timestamp: February 14, 2019, 08:41:13 pm
content: Your FPGA design tools should give you a report showing the maximum speed at which your design is expected to work, as well as information about the critical paths and where the delays come from. In Quartus (Altera) it's TimeQuest; isn't there a Xilinx equivalent?Timing specifications are an absolutely integral part of FPGA design, never an afterthought or nice-to-have option. You'll need to specify clock speeds and the timing relationships that need to be maintained between signals, and the synthesis tool should either meet those requirements or tell you which paths fail to run fast enough.What do you know about SDC files?

author: DC1MC
author_rank: Super Contributor
timestamp: February 14, 2019, 08:44:12 pm
content: I'm also relatively green in FPGA issues, but here are my answers, I see that there is yet another sensor connected do FPGA camera/gadget thingie:Is there a good guide on how to set up bus timing constraints, for external buses, with FPGAs?It is but it's its a pain to do it properly, one have to take maaany factors into consideration:https://www.xilinx.com/support/documentation/user_guides/ug381.pdfIs my goal of 500MHz clock (~1Gbit/s per lane) realistic using regular FPGA logic (rather than specific transceivers - to reduce the cost of implementing this design, I'd rather stickto a cheaper Spartan or Artix chip, not a Kintex or similar with high-speed transceiver modules)NO without external transceivers (best) or resistive components(meh), at least for Spartan 6/7https://www.xilinx.com/support/documentation/application_notes/xapp894-d-phy-solutions.pdf.Same with Artix:https://forums.xilinx.com/t5/7-Series-FPGAs/MIPI-CSI-Interface-with-Artix-7/td-p/777470Xilinx do sell a MIPI core for Artix.Here's a list of what's available for different families, THEORETICALLY one may reach this speeds:https://www.xilinx.com/products/technology/high-speed-serial.html, your issue may stem from not using correct transceivers or resistive networks, at high speed it matters for the signal integrity, and here you really need a proper scope with the proper probe or even a specialized MIPI analyzer.Using my relatively basic 100MHz oscilloscope, is there any way I can analyse bus timing (e.g. if I measure some error at 50MHz, is this error likely to be proportional to higher clock rates, or do I need to borrow a faster scope...this is an option, but it's a pain).HELL NO.Thanks for any input from an FPGA greenie.*It was exceedingly helpful to determine that a Pi camera can be configured to clock data at under 800kHz (and the Pi will accept such data) allowing detailed bit pattern analysis.Cheers,DC1MC

author: tom66
author_rank: Super Contributor
timestamp: February 14, 2019, 08:58:30 pm
content: @AndyC_772So far I have reached 200MHz without any custom timing, just using (presumably) the default FPGA planner settings. However, I am well aware this is not sustainable -- and I am surprised I was able to get this far! 400MHz just about works if the FPGA is warm, so it looks like I may be on the edge of getting this to work as-is.I know enough about XDC files to be dangerous and have started writing some basic constraint files. Obviously I already have IO constraints set up. I initially tried hacking a simple setup/hold limit together, setting both at 100ps. I suspect this is (a) incorrect, and (b) wildly optimistic. Ideally, the data paths would be length-matched -- but I am not sure how this works on an FPGA -- hence the ask for some advice on timing (or where to find said advice)Code:create_generated_clock -name mipi_clkn_out -source [get_ports csi_clk_n] -multiply_by 1 -add -master_clock clk_out1_270_clk_wiz_0 [get_pins clk_wiz_0_x/clk_out1_270]create_generated_clock -name mipi_clkp_out -source [get_ports csi_clk_p] -multiply_by 1 -add -master_clock clk_out1_90_clk_wiz_0 [get_pins clk_wiz_0_x/clk_out1_90]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d0_p]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d0_p]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d0_n]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d0_n]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d0_p]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d0_p]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d0_n]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d0_n]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d1_p]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d1_p]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d1_n]set_output_delay -clock [get_clocks mipi_clkp_out] -max 0.100 [get_ports csi_d1_n]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d1_p]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d1_p]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d1_n]set_output_delay -clock [get_clocks mipi_clkp_out] -min 0.100 [get_ports csi_d1_n]This may be completely wrong, so please do tell me if it is! I know it doesn't seem to make the timing analyser happy.@DC1MCI am using the XAPP894 reference design on a custom 4-layer PCB with a high-speed connector directly hooked onto my MYiR FPGA dev board. I believe the '894 solution should be capable of operation at 800Mbit/s per lane, which would set an upper clock limit of ~400MHz. I may be able to push this a little higher in the final design if I keep the distance between FPGA & Pi short. The implementation will be using a Compute Module 3, so this should be easier. I can live with 800Mbit/s per lane, but the faster, the better, in this type of application (which I want to keep under wraps for now, but watch this space, I guess). Picture of my layout below.The MIPI CSI-2 controller, I have written myself, without using any commercial Xilinx IP. It is about 1,000 lines of Verilog, but it is restricted, by design to operate in a two-lane configuration only. There are several other simplifications, for instance the CRC is always 0xffff as this saves some logic (& a lot of RE), and the Pi does not appear to care.Regarding the 100 meg scope - I figured - but a man can hope, eh? I'm fairly impressed I was able to get this far with JUST a Rigol DS1074Z, admittedly with the B/W unlocked.My probing solution, BTW, is perhaps not ideal, but is certainly not just a couple of scope probes... I have created a break out board with u.FL connectors on it which lets me probe each signal with a u.FL to BNC adapter cable. The biggest issue with this probing solution is it creates a lovely stub on the transmission line, which causes pixel-level glitches on the real Pi Camera. However, it does not appear to create enough signal degradation to stop the camera working altogether, so has proven sufficient for signal analysis.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 14, 2019, 09:03:26 pm
content: On the FPGA side, the best you can do is to provide a good clocking. Use MMCM to generate a data clock (CLK), as well as a 90 degree shifted clock (CLK90) for the external clock.Then do for data:CLK->BUFIO->OSERDES(clk) to clock dataCLK->BUFR->OSERDES(clkdiv)Use OUT_FIFO to cross between your FPGA clock domain and the BUFR clock domain for your dataFor external clock:CLK90->BUFIO->OSERDES(clock) to generate the external clock (transmit 10101010 ... in DDR mode)CLK90->BUFR->OSERDES(clkdiv)You can also use ODDR instead of OSEDES for the external clock. Then you don't need CLK90->BUFR.Make sure that all the pins are in the same bank and MMCM is in the same block. Also, use MMCM outputs which have direct connection to BUFIO/BUFR (through so called HPC lines).This produces the least skew between data and the clock you can master, which is around 200 ps. You can go up to the maximum clock rate (depends on the speed grade of the chip), but the faster you go, the harder it gets.Keep in mind that you also may have SI problems between the FPGA and the receiver as well as problems on the receiver side.

author: tom66
author_rank: Super Contributor
timestamp: February 14, 2019, 09:26:20 pm
content: Quote from: NorthGuy on February 14, 2019, 09:03:26 pmThanks for the suggestion. Interesting suggestion on using an OSERDES module to generate the clock.I am currently using an MMCM to generate a 0, 90, and 270 degree clock. The 90 and 270 degree clocks are used to produce the CLKp and CLKn output signals, and the 0 degree clock is used for data. Is it better to generate just a 90 degree clock and use the differential block to create the P/N clocks? Now I think about it, my solution does sound needlessly complex, and prone to any phasing error or jitter as a result of the MMCM clock output performance.My data path is similar to your suggestion though I will look into adding a BUFIO in the clock path to the OSERDES and an OUT_FIFO.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 14, 2019, 10:26:02 pm
content: Quote from: tom66 on February 14, 2019, 09:26:20 pmAlthough you can simulate a differential signal with two single-ended, it is always better to use real differential output buffer.

author: hamster_nz
author_rank: Super Contributor
timestamp: February 15, 2019, 08:02:29 am
content: As you understand the problem pretty well, the best advice would be to read the User Guide for the SERDES blocks with a highlighter in hand, taking note of anything that is of interest.On the FPGA you can achieve the speeds i the datasheet sheet - e.g. for Spartan 6 950Mb/s, 1050Mb/s or 1250Mb/s depending on speed grade. When experimenting you can go higher than this, but will see timing errors (e.g. I have got 1,485Gb/s out of Spartan 6 for DVI-D output).To achieve these speeds you will need to pull out quite a few stops, using the highest serialization factors and keep your high speed logic simple. You will most likely need a different clock domain to your main design. E.g. the main design runs at 120MHz, handing off processed data to a 240MHz domain for final formatting, and then off to the 4:! SERDES to get 960Mb/s on the wire. Making a lot of complex logic run at 240MHz is possible, but is in the "you really have to work at it" territory for most low-end FPGA chips.To skew the clock signal, if you have no luck with the 90 degree clock then investigate using the ODELAYE2 primitive to shift the edge of the clock a little. It is designed for this sort of fine tuning but is finicky to get working. Make sure you use the same output structures for the clock signal that you do for the data signals. So rather than forwarding out a clock signal. So if you user SERDES, then SERDES a pattern of alternating ones and zeros for the clock, or if you use DDR then DDR out ones and zeros for the clock.Off the chip signal integrity depends on the quality of the PCB design and connectors. About the best that the FPGA can do to help a dodgy PCB is compensate for track length by adding output delays (using the ODELAYE2 primitive). If it looks rough then it will be rough.Use differential output pairs, don't try to make a differential output using two single ended ones and NOT logic.I have some sort of feeling that the quality of phase shift is different from MMCMs to the PLLs, with the MMCM being the lower quality of the too It would pay to check datasheets and documentation.

author: TimCambridge
author_rank: Regular Contributor
timestamp: February 15, 2019, 10:57:13 am
content: Two small points:- use Xilinx ChipScope to see what is really happening.- read through the Lattice MIPI Bridge stuff. They have done this for a while with regular FPGAs and reference designs. Also, they now have some specialized FPGAs

author: NorthGuy
author_rank: Super Contributor
timestamp: February 15, 2019, 01:32:26 pm
content: Quote from: hamster_nz on February 15, 2019, 08:02:29 amHis chip doesn't have ODELAY, but a similar effect may be achieved by changing the phase shift for the MMCM output. This can be done at run-time.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 15, 2019, 03:13:36 pm
content: Quote from: hamster_nz on February 15, 2019, 08:02:29 amYeah, from my experience, on Spartan 6 FPGAs, making anything moderately complex run at above 200MHz (even with the highest speed grade) gets pretty challenging. You usually end up having to pipeline like crazy.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 15, 2019, 04:48:22 pm
content: Quote from: SiliconWizard on February 15, 2019, 03:13:36 pmI don't think the OP's task is time challenging. The speed is important for output only. Then, the SERDES can be driven at 8:1, which, even for 680 MHz output (max for BUFIO and OSERDES at -3 speed grade), this requires BUFR speed of only 680/8 = 85 MHz. Thus, OP can run the main logic at 100 MHz.You cannot do much to improve output stages - they don't depend on what the logic does - the only dependence is clocking. Once you optimize the clocking, nothing further can be improved. You also need to select the correct IO standard. From this point on, it's all SI.

author: tom66
author_rank: Super Contributor
timestamp: February 15, 2019, 06:30:58 pm
content: Yes, in my case, I work with 16 bit word sizes, everything is done in 16 bit to keep the clock rate down.The CSI block I have written works at 1/4 the output clock so a 500MHz output clock (1Gbit/s per lane, 2Gbit/s total, 250MB/s total) requires a 125MHz clock.My first test implementation of CSI - to better understand the protocol - was written "bit-bang" style. That had an Fmax of around 100MHz in the best case, probably because the timing was non optimal.I am fairly hopeful I can keep most of the logic at a "reasonably sane" 100-125MHz which shouldn't prove too difficult. I am currently using the lowest speed grade FPGA but upgrading this is an option. One of the things I will need to look at is the thermal performance of the system as a hot FPGA may lead to drift in timing.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 15, 2019, 06:49:24 pm
content: Quote from: tom66 on February 15, 2019, 06:30:58 pmThe speed ratings in FPGA should hold over all variations of temperature and voltage. At lower temperatures, or if you overvoltage a bit, things should work faster than specified.The -1 grade has BUFIO rated at 600 MHz, so 500 MHz should be Ok. However, if you use BUFG instead, it is only rated at 464 MHz - it still should work most of the time, but in theory you may have temperature problems.Of course, these ratings say nothing about the data lines outside of the FPGA.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 15, 2019, 09:03:03 pm
content: Yes, nothing tells us the problem lies in the FPGA itself here. Could be a signal integrity issue, or a problem with the RPi.

author: tom66
author_rank: Super Contributor
timestamp: February 16, 2019, 08:51:42 am
content: I have moved to using an OSERDESE2 to generate the clock - this works well. However, same limit of ~300MHz exists. It may indeed be a SI problem, which I need to investigate with a higher speed oscilloscope sometime. In the meantime, I need to brush up more on optimising the timing elsewhere.Is there a preferred and simple way of generating a 1/4 clock for the lower-speed logic? I'm currently using a BUFGCE with the CE driven by a counter. This isn't ideal, I suspect, because it will generate a very short ~12.5% duty clock pulse...requiring any connected logic to be able to handle the setup/hold requirements of that clock. I'm going to look into moving this to a BUFH with a divider - but I recall trying this before and it did not work so I need to investigate this again.

author: NorthGuy
author_rank: Super Contributor
timestamp: February 16, 2019, 04:20:16 pm
content: Quote from: tom66 on February 16, 2019, 08:51:42 amYou may try to calibrate the phase shift between clock and data in MMCM. Now you have clock shifted by 90 degrees. Try other values and find the one which gives you the best speed.Otherwise - it's most likely SI. You should impedance match the lines, have proper terminations, match the length (at least within differential pairs), have good bypass capacitors etc. I have never worked with CSI-2, but any protocol should describe the physical requirements to the signals they use. Also, the companies who sell devices supporting the protocol will often have routing guidelines in their datasheets or app notes. These may help.Quote from: tom66 on February 16, 2019, 08:51:42 amThe simplest way is to create a separate output from MMCM of desired frequency, which is independent of the clocks you use for IO. There are IN_FIFO and OUT_FIFO elements (which are shallow and wide). You can use them for clock-domain crossing. One side is clocked by IO clock (usually BUFR), the other side is clocked by your main FPGA clock. This way you can speed up the main clock or slow it down as necessary, but it is independent of the IO. Similarly, if you want to change something in the IO clocking, such as adding a delay. Such arrangement lets you have a main FPGA clock (driven by BUFG), which easily interfaces to all the different IOs you may have. You do need to follow common sense though - if you write something to FIFO and want uninterrupted read at the other end, you want the write clock to be the same speed or faster than the read clock.Nothing you can do with BUFH - it is just an intermediary buffer between other clocks and the logic. It's there any way whether you use it in your design or not. You can instantiate BUFHCE if you want to disable it, that's all.

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 16, 2019, 06:13:28 pm
content: Quote from: tom66 on February 16, 2019, 08:51:42 amOn Spartan 6, I use the DCM_SP primitive for that (if I understand your need correctly though).

author: asmi
author_rank: Super Contributor
timestamp: February 18, 2019, 05:32:11 pm
content: Quote from: tom66 on February 16, 2019, 08:51:42 amIf phase relationship between original and 1/4 clock is important, you will have to use the MCMM to generate that. Alternatively if you only need 1/4 clock inside a single region, you can use BUFR as it's got built-in clock divider. It's typically used for receiving source-synchronous data together with ISERDES.

author: tom66
author_rank: Super Contributor
timestamp: February 20, 2019, 09:26:20 pm
content: Thanks for the suggestions all. I'm going to revisit this as my focus has moved to getting another part of the project working for an upcoming PoC.