SerDes headache on Xilinx FPGA  - Page 1
Forum section: electronics / general

author: Alex
author_rank: Regular Contributor
timestamp: June 21, 2017, 08:14:09 pm
content: Do you have any experience debugging SerDes inputs? Apologies for the lengthy post, this is proving a particularly difficult bug.This concerns a high speed parallel DAQ. Right now we are getting erroneous output data when sampling a sinewave (attached image).We have 14 bits coming in from the ADC, serial, DDR at 580 Mb/s. Xilinx SerDes IP cores implement 1:7 decoding, we then implement 7:14 demultiplexing. The data from this is often stable, outputting correct portions of the input sinusoid (see attached).The system 'bitslips', until the framing signal is correctly locked, i.e. '01111111000000' is bitslipped by one to obtain '11111110000000'. This works and remains locked for greater than 10 mins.The user demultiplexing appears to work fine, as we get portions of the sinusoid and the boundary between the top nibble (7 bits) and bottom nibble (7 bits) seems to also work fine.In the output sinusoid, we get periodic amplitude jumps that happen to be exactly 2^10, i.e. 1024 from an erroneous bit toggle on the 10th bit. Rather than being a single sample glitch, this lasts for approx 50 samples. Meanwhile the other bits correctly switch, allowing the shape of the sinusoid to be correct within that 50 sample period. The output sinusoid therefore looks to have 2^10 amplitude jumps within it (see attached).Now, the 10th bit is in the middle of the top 7bit nibble. As the other bits of that nibble are correct, it is clearly not an issue with 7 bit to 14 bit time demultiplexing. Likewise if it was PCB trace routing delays, it would be more random rather than periodic and would probably effect other bits. It would also change with different physical constraints and firmware re-builds as logic is moved around on the FPGA, which it doesn't. I've checked the bitslip functionality and it appears to be fine. The ADCs are definitely configured, and their clock (160 MHZ) is stable.As many of the bits are correct, and indeed bit 10 is often correct for 100s of samples, I don't think it is SerDes singaling (timing or voltages). I'm using a test widget to pretend it is the synchronisation module, allowing the RAW FIFO to be setup and to grab some data. With test data this is perfect. The test widget starts RAW acquisition 800-ish clock cycles after the widget is enabled from the embedded software, and crucially grabs data into the FIFO just once, hence no problem with data writing when we are reading the FIFO etc.Now, there are other things going on. For example the input sinusoid has an amplitude of 200mVac and frequency 200kHz with signal generator DC bias set to the same as the ADC's natural DC biasing point. However with no change in hardware, if I increase amplitude to 400mVac, I get almost a flat-line with +/-200 DN noise, no sinusoid!The issue doesn't seem to go away if we reduce the ADC speed, however I suspect it is not LVDS SerDes signalling anyway, but somewhere in the 1:7:14 deserialisation process. I've re-coded the 7:17 user demultiplexers for more robust timing, demultiplexing in time rather than two 7bit registers with asynchronous multiplexers. I.e. increasing the demultiplexing action from tclk/2 for the case of asynchronous multiplexer CLBs to the full tclk for synchronous 7:14 demultiplexing.Now, there are other issues, one being single sample glitches, the other being another periodic bit issue with the 14bit signing bit (MSB), hence the offsets that go from positive to negative etc. But I expect those are symptoms of the same issue, i.e. fix the issue with bit 10 and bit 13 is also likely to be fixed.Any pointers would be much appreciated. This is truly doing my head in...

author: hamster_nz
author_rank: Super Contributor
timestamp: June 21, 2017, 09:29:51 pm
content: Any chance of seeing the raw data?External causes:- Signal integrity on the serial data- Intersymbol interference - Too many '1's or zeros in a row cause the value to bleed into the following bit.Maybe map out the raw data, as it is on the wire, and then 'corrected' raw data, and see what the differences look like.Internal causes:- Bitslip - does it really work as you are expecting. I never trust/understood bitslip, and have sometimes implemented my own logic to find the correct framing.- Sampling phase - if not already doing so, try adding an IDELAY2 and some option to set the delay. That will allow you to fine-tune when the bits are sampled. This can look a lot like a signal integrity issue.In 7-series parts you can sort of automatically detect and adjust sampling phase, but you can't use 'wide' Serdes. You use two 4-bit Serdes, one sampling on the inverted clock, and then tune the IDELAY so that for one of the serdes blocks the transitions are picked up 50% of the time.At 580Mb/s You could also consider sampling twice as fast at 1:7, combining that into a oversampled 1:14 frame, and then tuning the delays so that you are sure that the transition occurs on close to the odd bits, then use the even bits as good data.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 21, 2017, 10:28:45 pm
content: There also seems to be some sort of pattern of errors at a finer scale too. maybe at the 256 or 128 counts.

author: langwadt
author_rank: Super Contributor
timestamp: June 21, 2017, 10:50:13 pm
content: I'm thinking that the timing or signal integrity is marginal so occasionally an odd and even bit gets swapped/duplicatedcan you setup the ADC to output a test pattern?

author: BrianHG
author_rank: Super Contributor
timestamp: June 21, 2017, 11:11:34 pm
content: Quote from: hamster_nz on June 21, 2017, 10:28:45 pmIf the serial bits are being shifted to the left or right by 1, the MSB may be mapped in the previous or next LSB bit, and you could see the type of waveform you are getting since you see only the 7 LSBs on bits 7 through 1, with what should have been the MSB bit 7 on the next or previous LSB bit 0. This may create that type of chop in the waveform when your signal crosses the level which triggers the true MSB of the ADC.Otherwise the output of your ADC is 2's complement AC and you are reading a straight linear signal, or, visa-versa. This means your serdes is good, you just need to invert the 7 LSBs when the MSB is set, or change your graphing from an UNSIGNED BYTE to a SIGNED BYTE, or visa-versa.

author: jefflieu
author_rank: Contributor
timestamp: June 21, 2017, 11:21:29 pm
content: Quote from: Alex on June 21, 2017, 08:14:09 pmI don't see how you sample the middle of the data eye. Bitslip alone only give you the word boundary.In case you're not aware, there's a block called IDELAY that inserts delay into either the clock or data. And you have to scan it to get the whole eye. Then pick the middle value.

author: NorthGuy
author_rank: Super Contributor
timestamp: June 22, 2017, 12:08:18 am
content: To me, the picture looks like the 10-th bit (or whatever the bit is) isn't outputted at all - always zero.Imagine, for example, the signal is 1023, then it needs to go to 1030, but the 10-th bit is always zero, so you get 1030 - 1024 = 6 instead. Then the curve continues at this new level until the time where the 10-th bit is supposed to be cleared, at which points the curbe jumps back to normal.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 22, 2017, 12:35:58 am
content: I was more thinking that the 10th bit was always the same as the 11th bit..... e.g. 1024 to 2047 ends up as 0 to 1023, and -1024 to -2047 ends up as -1 to -1023.This could be caused by the long run of 0s or 1s (e.g. 11110xx:xxxxxxx is reading as 11111xx:xxxxxxx and 00001xx:xxxxxxx is reading 00000xx:xxxxxxx due to the run of 1s or 0s causing it to miss-read).It would be interesting to see if you have patterns like 0000000:1xxxxxx where 0000001:xxxxxxx actually fits the expected curve better.It could also explain the odd glitches, were something like 111101x:xxxxxxx gets read as 111110x:xxxxxxx, causing spikes.

author: KE5FX
author_rank: Super Contributor
timestamp: June 22, 2017, 12:48:53 am
content: What do your .ucf or .xdc timing constraints look like?

author: hamster_nz
author_rank: Super Contributor
timestamp: June 22, 2017, 12:49:49 am
content: Oh, I don't know if it is applicable at these speeds or your setup, but at Gb/s speeds over longer cables, transmit preemphasis can solve problems like this.https://www.maximintegrated.com/en/app-notes/index.mvp/id/5045

author: exmadscientist
author_rank: Frequent Contributor
timestamp: June 22, 2017, 03:48:06 am
content: Quote from: langwadt on June 21, 2017, 10:50:13 pm+1, this is EXACTLY what test patterns are for -- you will be amazed what they help you find!Can you post the part numbers or datasheets for the things you're trying to interface? Knowing exactly what you are working with may reveal a few gotchas that aren't obvious to the untrained eye. Having spent far too much of my life working with Spartan-6 BITSLIPs (OK, it wasn't that long, but it was still too long!) I can say there's a ton of things that can go wrong. In particular your clocks have to be perfect. Even a single missing FIFO can cause huge troubles, and if the crossing clocks are only a few PPM apart, they can beraretroubles, which is always fun.One thing that stands out is you say you're using "SerDes IP cores". Last time I did this I found it was less trouble to readUG381and instantiate ISERDES2 blocks directly rather than try and figure out what the magic settings were to get the wizards to do what I wanted. If you haven't tried it this is something to consider. I might also be able to post known-working sample code for interfacing to an AD9637 (12-bit DDR LVDS) if that would be helpful to you.

author: Alex
author_rank: Regular Contributor
timestamp: June 22, 2017, 08:52:03 pm
content: Thank you all for your insightful responses. In summary, my immediate plan of attack is to setup an output pattern. Now for the other points,First things first, I am using:- FPGA - Xilinx Spartan 6XC6SLX45in CSG324- ADC - Analog DevicesAD925740MHz- PLL - Analog DevicesAD9577outputting 160MHz for ADCs and 140MHz for FPGAI have attached:- A diagram showing the SerDes clocking- A diagram showing inside a single SerDes- A picture of the 'User Demultiplexer' 7 to 14 bits. Let me know if this is not clear.Quote from: hamster_nz on June 21, 2017, 09:29:51 pmSure. I have attached a .txt dump from a UART. This is imported and parsed by the two matlab files also attached marked 'MATLAB' and extension changed from .m to .txt. Note that:- The 14 bit data is bit extended, in this case using the MSB- The data is concatonated by action of the 16bit input 32bit output FIFO- The data is interleaved in time into two FIFOs B0 and B1- The data is output onto UART in hex, but is actually 14 bit 2's complementQuote from: hamster_nz on June 21, 2017, 10:28:45 pmYou are right, so solving the issue with bit 10 might solve the other errors.Quote from: exmadscientist on June 22, 2017, 03:48:06 amSure, if you are able. No harm done in scanning through it.Quote from: exmadscientist on June 22, 2017, 03:48:06 amYes the SerDes is manually instantiated. Granted though I cannot guarantee error-free.Quote from: KE5FX on June 22, 2017, 12:48:53 amI can send the .ucf but it only really contains the pin and clocking resource placements, none of which can be changed as the PCB is custom and now fixed.

author: KE5FX
author_rank: Super Contributor
timestamp: June 22, 2017, 09:15:14 pm
content: Quote from: Alex on June 22, 2017, 08:52:03 pmSo there are no timing constraints at all...?

author: nctnico
author_rank: Super Contributor
timestamp: June 22, 2017, 09:21:16 pm
content: No constraints is asking for trouble! For these kind of designs you need to setup constraints which specify the setup and hold timing (before & after) so the right delay can be set in the IOB. Every input and output needs a timing constraint (input to flipflops and flipflops to output)!

author: hamster_nz
author_rank: Super Contributor
timestamp: June 22, 2017, 09:55:20 pm
content: Quote from: nctnico on June 22, 2017, 09:21:16 pmI see you use AlteraMy suggestion would be to at least add a manually instantiated IDELAY2 primitive and allow the Microblaze to set the input delay and then experiment....As long as your external clocks have a period set you should be fine - there is only one path from the pin, through the IDELAY2 to the SERDES block, and all the other clocks should be derived as needed by tools.You could also just try playing with the phase settings on the DMC/PLL block to shift the sampling phase a little relative to the ADC clock.

author: BrianHG
author_rank: Super Contributor
timestamp: June 22, 2017, 10:29:31 pm
content: Quote from: hamster_nz on June 22, 2017, 09:55:20 pmDon't you need to do both?Without specifying timing constraints, how will the chip fitter know if your source signal will fit in the timing window unless you just trust everything to sit on the clock edge perfectly? (Though I did a pretty big non time constraint, other than FMAX, design once with Altera, it just happened to work great since all my IOs were forced to use dedicated clocked marcocells right at each IO pin before the data was fed to the rest of the FPGA. Xilinx probably has a similar option to set for each IO pin. The timing of each IO pin was right on the PLL with the tightest possible clocking.)

author: Someone
author_rank: Super Contributor
timestamp: June 23, 2017, 01:04:22 am
content: Quote from: BrianHG on June 22, 2017, 10:29:31 pmWith dynamic alignment of the sampling in a SERDES RX you don't need any constraint on the data timing only the clock rate. In a well designed high speed serial system (source synchronous or embedded clock) the RX handles all the complexity in realtime and aligns the data output transparently.

author: BrianHG
author_rank: Super Contributor
timestamp: June 23, 2017, 07:27:19 am
content: Quote from: Someone on June 23, 2017, 01:04:22 amOOOPS, I didn't realize the OP was using a SER RX with embedded clock. In that case, it's up to the FPGA's capabilities for clock reconstruction and everything should be self correcting unless your source has noise, isn't the right voltage level, has a noisy clock reference, isn't terminated correctly or the FPGA PLL clock's analog supply isn't filtered properly, or, you signal traces aren't balanced properly.In this case, we are no longer dealing with a SERDES problem in the FPGA firmware, but, a hardware problem on the PCB.

author: BrianHG
author_rank: Super Contributor
timestamp: June 23, 2017, 07:40:01 am
content: Just out of curiosity, have you tried configuring these bits in the AD9257 to see what happens:0x15 bits 0, 4&5 - Output drive strength and termination values for the LVDS port.0x16 bits 0,1,2,3, Serial output phase adjust.These controls might fix your bit problem without changing any of your FPGA code. I know very well I had problems with Analog devices in the past with significantly larger more sophisticated IC with mega huge I2C control maps (like over 1000 controls with their high end HDMI interface ICs), but your issue may boil down to these few registers for such a simple chip.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 23, 2017, 07:44:21 am
content: SERDES on Xilinx can not use an embedded clock - I assume that there is a second pair for the clock, or it is supplying the clock.Only the Transciever blocks (GTP primatives) can do CDR, and IIRC they need much higher data rates (and also run length limited too, so the PLL can keep lock).

author: BrianHG
author_rank: Super Contributor
timestamp: June 23, 2017, 07:47:32 am
content: Quote from: hamster_nz on June 23, 2017, 07:44:21 amI know, 'Someone' didn't read that the OP was using an AD9257 and didn't check how the IC interfaced with the FPGA.I still say that the configuration controls I mentioned 1 post earlier has a good chance of completely solving the OP's issue without changing his FPGA firmware.

author: BrianHG
author_rank: Super Contributor
timestamp: June 23, 2017, 08:02:38 am
content: Out of curiosity, have you tried freeze-it on the FPGA or ADC after the 10 minutes of operation to see if it will quickly un-do the problem?This wont tell you much more than you have a timing issue right on the edge. Adjusting the ADC's drive strength and PLL phase may help this.If you are using the DCO phase of 0011 (default), I would try adjusting the DCO phase to:00100100Just for clarification, are you clocking your SERDES from the CLK+/- input from the ACD or are you using the DCO+/- as your SERDES clock. How do you capture the FCO+/-?

author: Someone
author_rank: Super Contributor
timestamp: June 23, 2017, 08:22:00 am
content: Quote from: BrianHG on June 23, 2017, 07:47:32 amRead the original quote, I'll add some more emphasis.Quote from: Someone on June 23, 2017, 01:04:22 amDynamic alignment can be done on the IO SERDES (or just abstract this to any arbitrary high speed RX) and you don't specify the data relative to the clock since it could now be anywhere and still work..... other clause .....The receiver design (wrapper, module, function, whatever your terminology) should deal with all of this internally no matter if its source synchronous or embedded clock.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 23, 2017, 08:24:44 am
content: Quote from: BrianHG on June 23, 2017, 07:40:01 am^^^^^^ yes this! ^^^^^

author: Alex
author_rank: Regular Contributor
timestamp: June 23, 2017, 05:42:23 pm
content: Quote from: KE5FX on June 22, 2017, 09:15:14 pmApologies, all timing constraints are in the .ucf file. I have attached it in .txt.Quote from: BrianHG on June 23, 2017, 08:02:38 amI have tried freezing the fpga back when the logic was considerably different, I will try it again.SerDes is clocked using the DCO. The FCO is captured in the same way as the DCO. See attachments in post #12.Quote from: BrianHG on June 23, 2017, 07:40:01 amNo, another thing to try!Quote from: BrianHG on June 23, 2017, 07:27:19 amLet's also have a look at the PCB and routing. There are two ADC, Left and Right., each feeding 8 channels to the FPGA for a total of 16 differential pairs. The PPL sits in the middle, providing two completely separate 160MHz clocks one for each ADC. The signals from the ADCs are Data 1-8, DCO and FCO. All LVDS. All signals from the Left ADC are routed to Bank 0. All signals from the Left ADC are routed to Bank 2. So that is 10 LVDS pairs from each ADC, on different layers, as you can see in the PCB images attached.The traces have been matched to within 2.5 mm worst case, which would be in the order of 20ps. The DDR clock is 580MHz so tclk = 1.72ns or 1720ps. 20ps as a proportion of 1720 is less than 1.2%. However, there are other features on these lines like vias and test pads. One via per trace was necessary due to the packages used, but the PCB design was constrained to 1 via per trace and all traces have one via. So the exact propagation difference is unknown, but it is controlled to some extent. In any case, this is a 16 channel DAQ and every channel will have a slightly different delay. From this perspective, any corrective measure (e.g. iDELAY) would have to be on a per-channel basis.Now going inside the FPGA.Quote from: jefflieu on June 21, 2017, 11:21:29 pmNot robustly enough. There is a fixed delay between data and sample clock using the ADC's timing specs *at the ADC's pads*.There is also an issue with the 7:14 User Demux block I posted in post #12. I reuploaded the internal of the block here (the hand drawn image). This block is meant to split the input stream into a higher and lower nibbles. The clocking of the Post Mux has issues. Although the two DIV_CLK inputs are time constrained, the output of the toggle flip-flop is not. Overall this might cause a bit from one nibble in the serial stream to be muxed into the other. Even if this doesnt break the sampling timing, it certainly eats away from any headroom. So this needs to be fixed as a priority.Pressing on...

author: nctnico
author_rank: Super Contributor
timestamp: June 23, 2017, 06:17:19 pm
content: I glanced over the UCF file but I could not spot constraints which specify the relation between the ADC clocks and data lines. I'd expect something with before/after. I think you need this because you are clocking the serdes with an internal (derived) clock.

author: BrianHG
author_rank: Super Contributor
timestamp: June 23, 2017, 10:28:09 pm
content: Quote from: Alex on June 23, 2017, 05:42:23 pmWarning: The FCO is in sync with the AD9257's data outputs, not the DCO. In Altera, I made this mistake with a HSYNC signal from a DDR (a wide parallel 320Mhz buss with balanced clock) out HDMI receiver. At some sensitive frequencies, video has a huge spread on clock speeds, I would occasionally randomly slip 2 pixels left and right. What was going on is I was receiving my HDMI clock on a dedicated clock input as well as the HSYNC. The timing of the sync matched the data buss, which was on DDR inputs, not the clock phase & I had to fix that timing. Believe it or not, this occasionally was so stable, but would hiccup not due to the temperature of the FPGA & HDMI RXic, but the temperature of my tiny analog supply linear regulator for the PLL/DLL.I got lucky finding a set clock phase and current IO drive strength in the HDMI decoder's I2C settings.Since you are working with 1 clock frequency, adjusting the clock phase of the AD9257 should do the trick. If you are using a variable frequency source clock, you may want to increase the drive strength and lower the impedance settings on the AD9257 to make things stick under all environments and different PCBs.Remember, if you are already accessing the AD9257's control registers, you may accidentally be fiddling with the clock phase without knowing at 10 minutes due to a software bug, not a firmware problem in the FPGA's SERDES.