Using IEEE 1588 timestamps to measure packet arrival times? - Page 1
Forum section: electronics / projects

author: daloev
author_rank: Contributor
timestamp: May 30, 2024, 04:28:53 pm
content: I need to measure the arrival time (relative to a PPS signal from White Rabbit equipment) of a subset of UDP packets arriving over a 10G Ethernet interface. We have used Endace cards for packet capture, but they have long been unsupported and they require a PC running an obsolete Linux to work.I already have a system that uses a Xilinx UltraScale+ ZU4EV doing 10G Ethernet, and it would be wonderful if I could use the plentiful excess resources I have on the chip to capture timestamps and some other information (like MAC addresses) on network traffic.One approach could be to build something around the SERDES, looking for the packet preamble and getting timestamps and other information. However, it seems to me that IEEE 1588 hardware timestamping is doing that already, and it would be much easier to use that if it suitable than build something from scratch.Does anybody has experience using Ethernet hardware timestamping for this purpose? How accurate are the hardware timestamps? I would like to have accuracy to 5 ns or less.

author: bson
author_rank: Unknown
timestamp: May 30, 2024, 05:08:19 pm
content: Isn't packet timestamping typically based on a frame detection pulse on a PHY pin? This is then used to perform a timer capture and interrupt. 5ns seems awfully optimistic though.

author: nctnico
author_rank: Super Contributor
timestamp: May 30, 2024, 07:09:37 pm
content: Microchip (formerly Microsemi, formerly Vitesse) offers switch chips which can reach these kind of timestamp accuracies according to the specs. But these aren't easy to integrate. And without knowing when the UDP packets are send and the network component delays, the 5ns target will be swamped by other unknowns.

author: bingo600
author_rank: Super Contributor
timestamp: May 30, 2024, 07:23:42 pm
content: Quote from: daloev on May 30, 2024, 04:28:53 pmMost of those "stamp" against a 1PPS pulse.If using GPS, just getting that to 5ns or less could be a challenge.

author: ejeffrey
author_rank: Super Contributor
timestamp: May 30, 2024, 11:54:42 pm
content: Yes you can do this. AMD has some app notes on running PTP on their FPGAs and getting generating accurate time stamps and their Ethernet MAC / PHY IP support iee1588 timing signals. I have looked into this and talked to an AE about it but not actually done it yet.Cisco Nexus SmartNICs supposedly just do this out of the box with built in support for PTP, PPS signals, and high precision time-stamping. They are basically an US+ FPGA on a PCIe card, so it should be possible to do on your own. I think they claim 3.4 ns uncertainty, which is just based on the internal clock frequency,. since the true SOP signal is asynchronous from the FPGA timer.

author: ejeffrey
author_rank: Super Contributor
timestamp: May 30, 2024, 11:56:22 pm
content: You will need a naturally stable time base on your FPGA since you need very long integration times to get 5 ns accuracy. Also unless you use SyncE switches, the uncertainty compounds for each switch you go through.

author: ch_scr
author_rank: Frequent Contributor
timestamp: May 31, 2024, 06:31:42 am
content: To get 5ns resolution you can timestamp incoming packets with a (wide) counter running at 200MHz. Even a Tang Nano 9k 20€ board can do that nowadays. No need for integration at all in that case.Still needs a reasonable clock feeding the FPGA PLL, a decent 10MHz TCXO comes to mind.

author: bson
author_rank: Unknown
timestamp: June 02, 2024, 12:24:57 am
content: 5ns precision and 5ns accuracy are two very different things. The accuracy is going to be dependent on the jitter of the frame leader, and at some point more precision just allows more precise measurement of this jitter. It takes very little noise on top of a multi-level signal to create that much jitter.

author: bson
author_rank: Unknown
timestamp: June 02, 2024, 12:27:30 am
content: Integration doesn't reduce jitter. The absolute jitter of N+1 cycles of a signal is exactly the same as N cycles (assuming it's white/unmodulated).Edit: and also, the jitter of N cycles is the same as that of 1 cycle.

author: daloev
author_rank: Contributor
timestamp: June 06, 2024, 12:01:51 pm
content: We are measuring relative to the PPS from a White Rabbit slave whose master relies upon a GPSDO. Our measurements have shown that the variation between PTP edges is far below 1 ns, and, thankfully, averages out to zero within the limits of our measurements. I would take the FPGA clock from a slave clock output synchronized to the PPS. In other situations, we use a Metamako (now Arista) switch to timestamp packets, and that is basically a Xilinx FPGA in a box with other components that is doing what I want, but we need something embeddable. I should do as ejeffrey has done and talk to an AMD AE.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 06, 2024, 05:30:16 pm
content: Quote from: daloev on June 06, 2024, 12:01:51 pmYou also have the advantage that the white rabbit systems use synchronous ethernet as well. I haven't used it, but my understanding is that it means that the ethernet transmit/receive clocks are synchronized to the internal FPGA clock and the PPS. In normal ethernet, the start-of-packet can come at any time relative to the internal clock and the deserialization and clock domain crossing are major sources of jitter. So 5 ns should be pretty achievable.

author: nctnico
author_rank: Super Contributor
timestamp: June 06, 2024, 05:55:24 pm
content: Quote from: ejeffrey on June 06, 2024, 05:30:16 pmYes, WR uses SyncE but one of the things you need to account for is phase shift due to the receiver needing to shift (bit slide) a number of UI (link frequency periods) to sync the recovered clock to the data stream. When taking 16 bit words from the link's datastream (recovered clock = link frequency / 20) and each UI beying 800ps, this results in a variable delay between 0ns and 20x800ps = 16ns.Another route is to go to CERN's OHWR website and use one of the reference designs to implement a WR node with multiple ethernet ports where one port is used to synchronise to the WR switch. The traffic on the other ports can be timestamped using the internal TOD counter + bit slide.

author: daloev
author_rank: Contributor
timestamp: June 07, 2024, 02:38:15 am
content: Quote from: nctnico on June 06, 2024, 05:55:24 pmI have considered the idea of implementing a WR node because that would let me eliminate the add-on WR card I currently use and using it to timestamp packets would be a bonus. I just noticed BabyWR (https://ohwr.org/project/babywr) and that looks very interesting for its simplicity compared to a CUTE-WR-A7. However, it is still under development.

author: tggzzz
author_rank: Super Contributor
timestamp: June 07, 2024, 09:48:19 am
content: Quote from: daloev on May 30, 2024, 04:28:53 pmLight travels ~2m in 5ns. So, which end of thecableare you planning on timestamping? If timestamping both ends, which is "correct"?Then add protocol stacks, interrupts, cache misses, TLB misses....Then add multiple devices connected to a network switch. Then add network bridges...You need to understand the consequences noted by Leslie Lamport 45 years ago. Start withhttps://en.wikipedia.org/wiki/Lamport_timestamphttps://www.cs.cornell.edu/courses/cs6410/2018fa/slides/14-lamport-clocks.pdfhttp://lamport.azurewebsites.net/pubs/time-clocks.pdf

author: aeg
author_rank: Regular Contributor
timestamp: June 08, 2024, 05:35:41 am
content: Quote from: tggzzz on June 07, 2024, 09:48:19 amBoth are correct. If you're an ordinary clock and you want to know what time it is, then you adjust the timestamp according to the path delay, as described in IEEE 1588.QuoteNone of these come into play. Timestamping is by the network hardware.QuoteResidence time correction is performed using the correctionField value inserted in the packet by IEEE 1588 compliant switches.The devil is in the details, but notthosedetails.

author: tggzzz
author_rank: Super Contributor
timestamp: June 08, 2024, 07:45:41 am
content: Quote from: aeg on June 08, 2024, 05:35:41 amSo IEEE1588 has managed to circumvent the limitations defined by Lamport? I'm impressed;veryimpressedThat would be equivalent to solving the Byzantine General's problem.QuoteOK to some extent, but when exactly? With 5ns being the target, is that the beginning of packet, end of packet?Quote

author: ejeffrey
author_rank: Super Contributor
timestamp: June 08, 2024, 08:52:57 pm
content: Quote from: tggzzz on June 08, 2024, 07:45:41 amWhat? No. That would only apply if you allowed the network to be an arbitrary adversary. PTP is only used when you have control over the network. The goal is to bypass areas of understood non-determinism, primarily buffering and switch fabrics, and measure only deterministic links, the transceivers and network cables themselves. So switches measure the ingress and egress time and that gets applied as a correction.QuoteThe normal reference point is when the first byte of the Ethernet header after the preamble as it passes the PHY. The main thing is to measure input and output packets consistently, and to either eliminate or separately measure path asymmetry due to e.g. dispersion in fibers.Fwiw, I just measured jitter between a normal Intel 10G/1000base-T NIC operating at 1 gigabit and an FPGA and got less than 100 ns jitter even passing through a switch that doesn't do residency correction. This isn't too surprising since there was no other traffic at the time. You wouldn't expect a lot of buffer based jitter, but it's still surprisingly good with little effort. Getting to 5ns is challenging but possible.

author: tggzzz
author_rank: Super Contributor
timestamp: June 08, 2024, 10:43:34 pm
content: Quote from: ejeffrey on June 08, 2024, 08:52:57 pmAny network with the possibility of contentionisan an adversary.But on reflection, I think Byzantine generalsisa distraction in this context.Where there is a finite time-of-flight, partial ordering is the best you will ever be able to achieve.It will be interesting to see what the standards bodies define - and don't define - w.r.t. defining time on the moon relative to time on the earth. And Mars. And Alpha Centauri.QuoteJitter is relevant, but isn't the principal conceptual problem with the idea of a single unique time on all interconnected processors.

author: nctnico
author_rank: Super Contributor
timestamp: June 08, 2024, 10:53:04 pm
content: Quote from: tggzzz on June 08, 2024, 10:43:34 pmThat is why PTP traffic bypasses packet switching and goes through a seperate pipe allowing a switch which supports being an IEEE 1588 transparent clock, to correct the timestamps for any internal delays. This results in the switch being (almost) invisible where it comes to time transfer (= time synchronisation) between two points no matter the amount of traffic.

author: nctnico
author_rank: Super Contributor
timestamp: June 08, 2024, 11:19:35 pm
content: Quote from: daloev on June 07, 2024, 02:38:15 amYou could send the creator of the project an email to see what is possible where it comes to joining forces. I've met the creator a few times and he is a reasonable guy (employed by a research institute to do this kind of work).

author: ejeffrey
author_rank: Super Contributor
timestamp: June 08, 2024, 11:30:35 pm
content: Quote from: tggzzz on June 08, 2024, 10:43:34 pmThere is no contention on an individual ethernet link, at least in a switched network. At any given time, either a specific packet is being transmitted or it is not. Even in a hub, either the packet is transmitted or not, a collision cancels both packets, it doesn't delay them non-deterministically.QuoteDepends on what you mean. There is always a limit to synchronization and resolution. So there will be events that are so close you can't distinguish which is "actually first". But you can still assign a total order -- every event has a timestamp, timestamps *trivially* have a total order. That isn't the problem -- the problem is preserving causal ordering -- no node should be able to receive a message about the future. But that's conceptually easy, it only requires synchronization to be better than the shortest hop delay.QuotePTP itself doesn't require a specific reference frame, or even a specific rate. It's almost universal to use something derived from GPS, which means that the second is relative to TAI, aka the SI referenced to MSL. But you can run it with your own time reference, for instance if you have your own atomic clock and want to run it with your own local SI second rather than the MSL reference TAI. Or you can use a different or arbitrary time offset if you want.The primary goal of PTP is to ensure that every clock in the system is running at the same rate. It's basically a way to distribute PPS signals without a dedicated network. The secondary goal is to make sure that their offsets are equal, which requires a description of the reference plane where the time will be measured, typically the PHY.QuoteThere is no conceptual problem with having a unique time defined everywhere, and I don't know why you think there is. Because time is different in different reference frames due to GR and SR, you *do* have to pick a reference frame, but once you do it's perfectly well defined how to synchronize clocks everywhere.

author: aeg
author_rank: Regular Contributor
timestamp: June 09, 2024, 02:52:43 am
content: I have to admit that I kind of missed the point in my previous reply. You get path delay correction and residence time correction on PTP packets themselves, but OP was asking about timestamping random UDP packets, presumably not from the PTP peer.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 09, 2024, 04:07:01 am
content: That's true. You can have an accurate clock and get a precise timestamp of the packet arrival. Whether that means anything depends on how the packet got there.

author: tggzzz
author_rank: Super Contributor
timestamp: June 09, 2024, 05:55:48 pm
content: Quote from: ejeffrey on June 08, 2024, 11:30:35 pmSo Leslie Lamport got it wrong?Nowadays even short hops are "long".QuoteGPS on the moon? Mars?What happens to the second in a very different gravity well?What happens when the path length is continually changing? Obviously that implies the relative velocity can be positive and negative.QuoteThe concept of a single reference frame for Earth, Mars, and the Moon is not trivial.

author: ejeffrey
author_rank: Super Contributor
timestamp: June 09, 2024, 10:34:14 pm
content: Quote from: tggzzz on June 09, 2024, 05:55:48 pmNo, you have just misunderstood his work. The opening lines of the abstract for "Time, Clocks, and the Ordering of Events in a Distributed System" read:QuoteOnce you assign timestamps you have a total order. As opposed to the partial order, which only declares event A to precede B if there is a message going from A to B (possibly indirectly). Without the a clock, there is a ambiguous range of "concurrent" events limited by the communication latency, but defining clocks defines an order for the concurrent events.QuoteWell, you can either define an independent timekeeping reference on a different celestial body or you can define that it should stay in sync with earth standard time. This is much the way that GPS clocks experience a vastly different gravity and velocity compared to NIST atomic clocks in Boulder which in turn have very different gravitational potential to an observatory in Paris. Very likely the latter would be chosen, that's how NASA currently does time synchronization for spacecraft.

author: artag
author_rank: Super Contributor
timestamp: June 15, 2024, 10:29:36 am
content: Quote from: tggzzz on June 07, 2024, 09:48:19 amWhilst I approve of the SI units, I find it convenient to remember that light (etc.) travels about a foot per ns in a vacuum.However , I'm old enough to be forgiven for remembering feet as a unit. I long ago ditched 186,000 mph for 3*10^8 m/s.