FPGA high speed UDP stream and packet drops on PC - Page 1
Forum section: electronics / general

author: radiolistener
author_rank: Super Contributor
timestamp: June 10, 2022, 08:27:13 am
content: I need to receive and process high speed UDP stream from FPGA, the speed is about 400-1000 Mbps. But I catch some issue with packet drops on PC side.When CPU load is zero, I can capture stream with test app almost with no packet drops up to 600-700 Mbps. But if CPU load grows up to 20-30%, I got a lot of packet drops. Also I got packet drops even at 85 Mbps if CPU load grows up to 40-50%.I'm using QuadCore Intel Core i5-760 which has 4 cores running at 3200 MHz (overclocked) and Win7 x64 OS. Ethernet card is onboard 1G Ethernet, it is displayed as "Realtek PCIe GBE Family Controller" on the device list. The chip is probably Realtek 8112L, as said for my motherboard on some review site, but I'm not sure.Packet drops happening at OS kernel level, so dropped packets never received from a socket. I tried to use multiple threads to call recvfrom(), but it leads to broken packets order, so I needs to analyze packets id and reconstruct original packets sequence before processing. It add even more CPU load and memory usage which is very critical, because I'm working with gigabit realtime stream with no pauses for processing.I found that enter into realtime kernel mode (change OS kernel time slice to 1 ms with timeBeginPeriod) helps a lot to get rid of packet drops in a single threaded receiver app. But it also leads to 2-3 times higher CPU load, probably due to excessive overhead for thread context switch.So, I wonder what is the best strategy to receive gigabit UDP stream with no packet drops on a usual PC?

author: BradC
author_rank: Super Contributor
timestamp: June 10, 2022, 08:36:22 am
content: If you're going to use it on commodity hardware you might need a different operating system that offers some better packet handling. Failing that you could try a better network card that has a decent on-board receive buffer and a driver that knows how to use it.

author: radiolistener
author_rank: Super Contributor
timestamp: June 10, 2022, 09:05:30 am
content: yes, it seems that there is definitely some issue with network card and OS performance, because when I push gigabit UDP stream from FPGA into PC and there is no app listening for that stream, the PC shows about 40-50% CPU load. This is CPU load with no running applications...Is it possible to reduce that load with network card which has some hardware acceleration/buffers? What network card can you suggest?Forgot to note, I'm using a small 1032 bytes UDP packets. I know it's better to switch for a large jumbo-packets in order to reduce streaming overhead and improve network performance. But it also requires additional OS/router configuration. So, I'm trying to use classic packet size.And the question about receiving high speed stream is still active, what is the best algorithm to receive high speed UDP stream from a gigabit interface?What is better - to use multi-threaded or single-threaded code to read the socket?How much receive buffer memory should be assigned to the socket?Any tricks to improve performance?Any suggestions for driver precise tuning? (there are a lot of settings for ethernet driver in the device properties)

author: BradC
author_rank: Super Contributor
timestamp: June 10, 2022, 09:19:12 am
content: Quote from: radiolistener on June 10, 2022, 09:05:30 amAn enterprise grade Intel card would be my go-to. They tend to have on-board buffers, packet and interrupt coalescing and well supported DMA with drivers that are reasonably well tested. I'd avoid anything by Realtek. Even if you do get one that doesn't have massive errata to work around, the drivers are usually a steaming pile.QuoteWhatever works best for your scenario. Honestly it's very dependent on what you are doing with the data and how your OS manages packets. I know nothing about the Windows architecture because I don't use it for anything remotely arduous or critical. I did once, many years ago and learned a valuable lesson.QuoteSee above. Also depends on your software architecture.QuoteDepends. If what you are doing is not latency sensitive, lots and lots. If it's latency sensitive then as much as you can reasonably tolerate.QuoteObservation, instrumentation and profiling. Find out where the bottlenecks are by measuring them and hammering them out.QuoteNope. Entirely driver and device specific. I'd leave another disparaging comment about Windows, but you get the idea.

author: dmills
author_rank: Super Contributor
timestamp: June 10, 2022, 12:19:48 pm
content: For really screaming network performance you are often better off forgoing interrupts entirely and just pinning a core to the task of polling the NIC for new data.Needs an OS that lets you play those games of course, but the fact is an interrupt on an X64 is EXPENSIVE, in my case with a 10G interface running UDP I could not afford them so I just spun on the rx status register, 100% CPU load on that core, but so what?Even at 1G you would benefit from a card that can do interrupt coalescing and maybe zero copy networking.

author: bd139
author_rank: Super Contributor
timestamp: June 10, 2022, 12:56:31 pm
content: Going to keep this simple: don't use UDP if you don't want packet drops. There is no guarantee any packets will reach the destination and no way to know if they did or not or why they didn't. Even if it looks like it works today, a tiny kernel or environmental change somewhere may break every assumption you have.Accept the packet loss in your design philosophy or use a different protocol.

author: agehall
author_rank: Frequent Contributor
timestamp: June 10, 2022, 01:56:19 pm
content: While UDP isn’t reliable 100%, pushing 1Gb/s on a LAN using UDP shouldn’t be an issue if you have proper hardware. And by “proper” I mean pretty much any server grade NICs that have real offloading and interrupt coalescing.

author: FenTiger
author_rank: Regular Contributor
timestamp: June 10, 2022, 02:25:37 pm
content: Packet loss across the open Internet is always possible and is something you need to design for.Packet loss across a single point-to-point Ethernet link is a symptom that something is wrong - either a hardware problem, or (as in this case) the receiver isn't keeping up.Quote from: radiolistener on June 10, 2022, 08:27:13 amSome people would regard that as a hardware problem.

author: bd139
author_rank: Super Contributor
timestamp: June 10, 2022, 02:38:39 pm
content: Oh Realtek. Yes I'd start there too

author: SiliconWizard
author_rank: Super Contributor
timestamp: June 10, 2022, 06:13:09 pm
content: Quote from: agehall on June 10, 2022, 01:56:19 pmYou're right, but you still won't have any hard guarantee. If packet loss is unacceptable for the given context, then UDP is never going to work.If you can accept some low rate of packet loss (TBD), then it's doable with proper hardware AND software.As a workaround, if you manage to keep packet loss rate to a very low level, you could implement (if at all possible with your protocol) a simple packet resend scheme.

author: BrianHG
author_rank: Super Contributor
timestamp: June 10, 2022, 10:14:47 pm
content: Quote from: SiliconWizard on June 10, 2022, 06:13:09 pmEven in a project with a MCU in the past, using UDP required us to use multi-level checksum and packet retransmission on a regular basis. Having different routers in our network environment could even make data loss and errors worse. You better test whichever communication code you generate with an entire bunch of network hardware from great to garbage especially including WiFi as that networking hardware will put your com code through hell, specially at large distances or in a noisy environment.

author: spaceballs
author_rank: Newbie
timestamp: June 10, 2022, 10:36:19 pm
content: Like others say, use NIC card with offboard logic, enterprise grade.

author: radar_macgyver
author_rank: Frequent Contributor
timestamp: June 10, 2022, 11:09:41 pm
content: If you have the opportunity to dual-boot the Windows box in question, try booting a live-USB Linux distribution and run iperf in UDP mode. That will test your existing NIC. I haven't used a Realtek NIC, but getting zero packet loss for 6-700 Mbit/s over a 1G ethernet link is not difficult, unless there is a cabling issue.

author: FenTiger
author_rank: Regular Contributor
timestamp: June 11, 2022, 07:38:24 am
content: Nobody has asked the OP what kind of network configuration he is using, or what other traffic is present at the same time.I'll hazard a guess that it doesn't involve any WiFi, though.

author: agehall
author_rank: Frequent Contributor
timestamp: June 11, 2022, 02:25:30 pm
content: Quote from: SiliconWizard on June 10, 2022, 06:13:09 pmOf course - if you can’t accept any packet loss, UDP is never the right choice but on a LAN with modern hardware, 1Gb/s is not exactly a data rate where you would expect any issues at all, at least not in my network. But since OP choose UDP to begin with, I’m going to make the assumption that the inherent limitations of UDP are acceptable and in that case, some packet loss is not an issue. However, when using Realtek NICs, it doesn’t take much to make things fall over and I’m fairly certain that is the actual problem here.

author: radiolistener
author_rank: Super Contributor
timestamp: June 11, 2022, 03:42:49 pm
content: Quote from: FenTiger on June 11, 2022, 07:38:24 amCurrently I'm using at 100-300 Mbps through home gigabit router, which includes Wi-Fi, but when I using it at higher speed, I'm using dedicated direct cable connection from FPGA board to PC. Because router is unable to handle such traffic. The router starts to choke at about 500-700 Mbps speed, it just stops to respond for all ports and starts dropping packets.Regarding to resend the packet, there is no memory in FPGA to store packets. And since this is realtime stream, it will be impossible to resend the packet, because it will require extended network bandwidth, which limit is already reached. Also it requires complicated logic to store all packets and write it at about 125 Megabytes per second to the memory.As I see, UDP transfer works pretty good on 100M ethernet link at max speed and there is almost no dropped packet at all. So I think it can work the same on 1G ethernet link. But there is an issue on PC side, for some unknown reason it just cannot handle so many packets. I have no idea why. Of course I was used direct cable connection with no intermediate devices for testing it.Unfortunately I don't have other network adapters than realtek for testing it.

author: BradC
author_rank: Super Contributor
timestamp: June 11, 2022, 04:00:30 pm
content: Quote from: radiolistener on June 11, 2022, 03:42:49 pmAs politely as possible and paraphrasing the replies above : "Because you are using a crap NIC with a crap driver on a crap operating system".If you really need the hardware to work at 100% then you need decent hardware with decent software backing it up.

author: radiolistener
author_rank: Super Contributor
timestamp: June 11, 2022, 04:09:19 pm
content: Quote from: BradC on June 11, 2022, 04:00:30 pmI'm not sure about that, because when I'm using windows file transfer between two PC it shows about 1G transfer speed. This is why I think that there is possible way to use it at 1G speed, but I just don't know how. Usual code examples for UDP listening have a lot of packet drops at 1G speed.I set 256 Megabytes for socket receive buffer, but it doesn't help much. The only thing that I found which affects it is a time slice switch in OS thread scheduler with timeBeginPeriod call. When I call timeBeginPeriod(1) packet drops almost disappeared, but CPU load grows up very significantly...

author: agehall
author_rank: Frequent Contributor
timestamp: June 11, 2022, 07:19:03 pm
content: Quote from: radiolistener on June 11, 2022, 03:42:49 pmYou seriously need to get at least half decent gear.A router that chokes at 700Mb/s is either utter crap to begin with or seriously broken. (Might also be misconfigured and reject the UDP stream as some sort of attack) With a server class NIC and a proper router, or better yet, a good switch, this should not be an issue at all.

author: SiliconWizard
author_rank: Super Contributor
timestamp: June 11, 2022, 08:08:53 pm
content: Many "low-end" routers have pretty limited CPU power and handle much of the routing in software, so you're never going to get the full throughput with them unless you have like zero routing rule and use them just as a switch.The better ones have hardware routing capabilities which helps performance tremendously.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 11, 2022, 09:12:51 pm
content: What do your network statistics say? ("netstat -e" and "netstat -s") Any errors or checksum issues?Are you using jumbo frames, rather than the "safe minimum" of 548 bytes of user data? This will reduce the OS overhead a lot.Are you maintaining the required interpacket gaps? 12 idle bytes between packets?What are you doing when you get a packet? Could it be distracting the OS long enough to cause the NIC ring buffer to overrun?For a similar data acquisition issue I had to allocate a large memory buffer, write junk into each block so it was actually present, before receiving the data into the buffer.

author: FenTiger
author_rank: Regular Contributor
timestamp: June 11, 2022, 11:40:57 pm
content: Quote from: radiolistener on June 11, 2022, 04:09:19 pmThis way is called TCP. It is very good at dealing with flaky networks. It is also very complicated. You don't want to try to implement it in an FPGA. Trust me on this.

author: BrianHG
author_rank: Super Contributor
timestamp: June 12, 2022, 03:42:09 am
content: Quote from: FenTiger on June 11, 2022, 11:40:57 pmTo get this performance, in TCP, they also use 'Packet Scheduling'. Do not believe you can achieve this with an FPGA running simple UDP. I'm not sure packet scheduling exists for UDP, or what layer of the network protocol where this is implemented.

author: nightfire
author_rank: Frequent Contributor
timestamp: June 12, 2022, 04:39:55 am
content: First, I would make sure that the hardware is up to the task.Usually in this kind of load scenario, there are lots of interrupts to deal with, and this leads to latency and congestion.A server grade Intel Card would also be my tip, as most of the other manufacturers are whether special and hard to get or difficult with drivers (Broadcom etc.)Realtek cards have gotten better since the 10 Mbit Koax systems, but consumer Realteks are not designed for a real stressful environment.Network layer: To really make sure everything is ok, I would go for a good quality connection, using CAT6 or better cables from a brand manufacturer, and use a dedicated "real" switch, not the stuff embedded in some small router. With a managed switch it ís also possible to watch for possible issues with the connection.Also important: Depending on the type of switch/manufacturer, latency times can vary significantly for packet forwarding.At work I use HP Procurve/Aruba 2530 for critical servers, something similar to that should be available used for decent money-so it is possible to rule out other reasons during transmission.

author: radiolistener
author_rank: Super Contributor
timestamp: June 12, 2022, 05:16:37 am
content: Quote from: hamster_nz on June 11, 2022, 09:12:51 pmNo, there is no errors. But when packet drops starting, it shows that "Discards" value starts to grow very significantly. When there is no packet drop it shows "Discards" = 0. And "Errors" value still remains zero.I'm not sure what "Discards" value means exactly. As I readhereit shows "The number of packets rejected by the NIC, perhaps because they were damaged".But that packet drops issue appears when CPU load is above some threshold. When there is no CPU load, there is no packet drops. So this is definitely not damaged or malformed packets.By the way, most of my speed tests with UDP I performed withAlex Forencich verilog ethernet stack implementation, which seems pretty stable and tested for up to 10G and 25G speed.Quote from: hamster_nz on June 11, 2022, 09:12:51 pmHm... I didn't know about that "safe minimum" 548 bytes... Why 548 bytes?I don't use jumbo frames, because it needs to enable it in network card properties. By default it is disabled on my driver. And as I remember when I enable it it affects network performance for a usual packets. So I'm using 1024+8 bytes packet, 1024 bytes for a data payload and 8 bytes for id and streaming control.Quote from: hamster_nz on June 11, 2022, 09:12:51 pmyes, 12 bytes gap. When I tried to reduce it, the network card just stops to accept the packets and as I remember I even don't see any packet in the Wireshark.Quote from: hamster_nz on June 11, 2022, 09:12:51 pmBasically I send it to HDSDR program which performs FFT and further DSP processing. But for a cleaner test of network performance, I tried to receive data, check packet id and just drop it with no further processing. In both cases I have packet drops. So, it seems that data processing in HDSDR doesn't affect it much, the only note here is that data processing add some CPU load.I tried two methods to receive the data:1) using fixed buffer for socket recvfrom call, next I extract payload to another fixed buffer and send it to HDSDR. All this is done from the same thread with highest priority dedicated to the socket receive task.2) using fixed buffer for socket recvfrom call, next I copy the packet content to another fixed buffer and enqueue it for processing in another thread which extracts data and sends it to HDSDR.The first method has a smaller CPU load and less packet drops. Probably due to additional cost for a thread synchronization and queue processing.Interesting thing is that change receive thread priority to normal sometimes can reduce packet drops. But when I added timeBeginPeriod(1) call it turns into opposite - highest priority helps to reduce packet drops.Also I found that for some unknown reason, the packet drops are significantly reduced when I run receive process under debugger (MSVS), I have no idea why... When I thinking why it may happens I got the idea to try to change OS time slice with timeBeginPeriod. And it really helps to reduce packet drops very significantly. But as a side effect I got higher CPU load.It seems that there is possible another some internal OS thread which is involved in a network card processing. So, playing with time slice interval and thread priority affects that hidden thread.

author: hamster_nz
author_rank: Super Contributor
timestamp: June 12, 2022, 06:02:44 am
content: Quote from: radiolistener on June 12, 2022, 05:16:37 amThe minimum MTU anywhere on the internet is 576, so 548 used to be the "should not be fragmented by any router" size, This limit doesn't apply in this case, so you can use bigger packets. If you can enable and use jumbo frames I am pretty sure it will be a big help in reducing OS CPU usage and latency.Have you tested to see if all packets are received if you call recvfrom() in a tight loop doing nothing but asking for packets?

author: radiolistener
author_rank: Super Contributor
timestamp: June 12, 2022, 06:31:23 am
content: Quote from: hamster_nz on June 12, 2022, 06:02:44 amyes, I tried a tight loop which just call recvfrom and just check packet id from first bytes and nothing more. Since packet id is incremented for every packet, it just check if packet id is incremented by one for each next packet and if there is difference put a warning. And it also had packet drops on a high speed.

author: pcprogrammer
author_rank: Super Contributor
timestamp: June 12, 2022, 08:41:17 am
content: Keep in mind that TCP is a protocol that uses handshaking to make it reliable. Each packet needs to be acknowledged by the receiver, and is resent if not. UDP is a streaming protocol without all of this. The data within needs support for error recovery if that is important.VOIP used UDP to transfer the voice data. Noticeable when the connection has a lot of dropouts.

author: nightfire
author_rank: Frequent Contributor
timestamp: June 12, 2022, 11:15:19 am
content: Quote from: hamster_nz on June 12, 2022, 06:02:44 amJumbo frames are a mixed bag of whatever you want to do with it. In Gigabit networks it has been introduced mainly to allow for big chunks of data to be moved most sequentially, like storage systems, and to help in keeping the interrupt per Megabytes transferred ratio low.The downside is, that every network card and switchport had to be configured for this, and hell could break loose when due to whichever problem or issue some port was not doing jumbo frames right.Actually, with the more modern 10GbE cards, jumbo is on the decline, as those have enough mechanisms to cope with the interrupts created.Also jumbo only helps when you have enough data to coagluate together in the application.But, for testing a certain behaviour, and to make sure that the interrupts are not killing the machine, ist is a quick test that should give some answers.On windows, the build-in application "perfmon" would have the answers to that:- IPv4 System for packets received- network adapter for the physical adapter in the machine- network interface to watch the queue- objects to watch for mutexes- processor for interrupts/s- UDPv4Creates some nice graphs to look into- also can be run without GUI in the background for data akquisition and later watching

author: agehall
author_rank: Frequent Contributor
timestamp: June 12, 2022, 02:50:39 pm
content: Btw, OP, why are you using a router? Don’t you have a switch? And what kind of router are you using?Switches use dedicated hardware switch chips to forward packets at wire speed while in routers, it is common for packets to always pass via a CPU and this could be a severe bottle neck (which you already mentioned). It is possible that a few of the ports on your router are connected as a switch allowing packets to be forwarded at wire speed while some ports are only connected to the CPU. It is important to know these things if you care about maximizing performance.If you are only sending packets on your own LAN, I would highly suggest to ensure that you only pass packets via switches and not via any router that forces packets to go via a CPU. That way you should not have any packetloss in the network itself but you will probably still have issues with your NIC.

author: pcprogrammer
author_rank: Super Contributor
timestamp: June 12, 2022, 03:34:30 pm
content: How about using a cross cable and no switch or router at all. If one of the interface is auto MDIX it can even be a straight cable. That way there is no delay or packet loss due to data passing through a CPU. Just setup the PC as the gateway for the FPGA and it should work.

author: RoGeorge
author_rank: Super Contributor
timestamp: June 12, 2022, 04:31:17 pm
content: I didn't read all the posts here, but if you think the speed hiccups are caused by the i5 CPU, check the power profiles in BIOS. Some motherboards have explicit menu for that, others don't.The idea is i3/i5/i7 have very aggressivecore parkingalgorithm (like a standby). This is so in order to lower the power consumption and the proc temperature. Try disabling all power savings, disable clock throttling and thermal throttling. For windows, I remember I had to do set the most power hungry profile in both the BIOS settings and in Windows.If those settings were done already, just ignore this post.

author: agehall
author_rank: Frequent Contributor
timestamp: June 12, 2022, 04:57:47 pm
content: Quote from: pcprogrammer on June 12, 2022, 03:34:30 pmNo need to set the PC as the gateway - gateway is only used if the destination IP isn’t on the same subnet.But again, the without a half decent NIC in the PC, this won’t solve anything.

author: radiolistener
author_rank: Super Contributor
timestamp: June 12, 2022, 07:48:42 pm
content: Quote from: agehall on June 12, 2022, 02:50:39 pmFor 1G transfer I'm using direct cable connection from FPGA LAN connector to PC LAN connector. The issue which I'm talking about happens when there is no any kind of device between PC and FPGA module, just a direct cable connection.I'm using router when I don't need a lot of network bandwidth, for example when connection uses just 100-200 Mbps I put it through router to share connection with internet.Quote from: RoGeorge on June 12, 2022, 04:31:17 pmI disable it in the bios, all cores working at max speed even in idle state. Because with that "energy saving technology" I got a lot of performance issues. So, I just disable it and forgot about issues.And there is no CPU throttling. I'm usingScythe Ninja 3cooler which can keep my CPU very cold even at max load with fully powered off fan. But of course it has a fan and it working. CPU temperature is about 45°C according to sensors

author: FenTiger
author_rank: Regular Contributor
timestamp: June 12, 2022, 10:20:50 pm
content: Quote from: radiolistener on June 12, 2022, 05:16:37 amIn my experience, a "discard" occurs when a NIC receives a packet which it cannot transfer to the host because it has not been given a buffer to put it in.In other words, the host is not keeping up.

author: KE5FX
author_rank: Super Contributor
timestamp: June 13, 2022, 01:22:30 am
content: Calling recvfrom() in a loop is OK at much lower rates but I wouldn't expect great results in the case you're talking about. At these data rates, you need to use multithreading to gain access to asynchronous I/O features. This will be true regardless of OS (and no, there's absolutely nothing wrong with Windows in this regard.) I'd suggest looking atI/O completion ports.Also as others have mentioned, you need to be prepared for packets not only to be lost, but duplicated, corrupted, or received out of order. In networking, it seems that everybody eventually feels the need to implement their own TCP/IP-like model over UDP. Admittedly that's sometimes appropriate, because TCP/IP sucks. But it's also true that everybody underestimates just how much it does under the hood.

author: radar_macgyver
author_rank: Frequent Contributor
timestamp: June 13, 2022, 05:24:13 am
content: Quote from: KE5FX on June 13, 2022, 01:22:30 amHaha, too true. Just as a point of reference for the OP, I had just this experience a few years ago. We had a few radar data acq. systems using an FPGA that output fixed UDP packets (about 500 Mbit/s), sized to be under the 1500 byte MTU. There was no embedded CPU, so no chance of switching to TCP. Data was received on an x86 server running Linux through a direct connection (no switches, routers in between), but over a long cable. Worked fine for a while, then multiple systems started to show data loss. The kludge that held together for a while was to add another PC in between that implemented a 'retransmission protocol' which was a grad-student's version of TCP. After the rate of data drops became too high to bear, we finally took a look at the hardware. Discovered that the RJ45 magjack on the FPGA board included an EMI filter using ferrites and chip capacitors held in place with phosphor-bronze springs that failed under vibration, leaving a small pile of 0603 chip components inside the metal shell. Replacing the connectors solved all the issues, and we could take out the whole retransmission BS.We went round in circles for a long time believing that the code that called recvfrom() was at fault. We tried all sorts of 'improvements', including multiple threads, boosting priorities, customized NIC drivers, waving dead chickens etc.By contrast, I had also built another data acq. system using a PCIe FPGA board on a Linux host which streamed the data out on TCP (instead of UDP) with similar bandwidths, never had any issues with dropped data. Both server and client network code were single-threaded.

author: radiolistener
author_rank: Super Contributor
timestamp: June 13, 2022, 06:17:42 am
content: Quote from: FenTiger on June 12, 2022, 10:20:50 pmI also think something like no memory for packet issue, but I allocate 128 megabytes for socket buffer in the following way:Code:hr = setsockopt(udpSocket, SOL_SOCKET, SO_RCVBUF, (const char*)&bsize, sizeof(bsize));I tried also 512 M and even 1 GB, but it doesn't help. Such a buffer should be enough to keep at least 1-4 seconds of packets with no polling. So it seems that socket buffer is not a memory which is required for NIC.

author: ve7xen
author_rank: Super Contributor
timestamp: June 13, 2022, 07:41:40 pm
content: OP what does your traffic pattern actually look like? It's all well and good to talk about mbps, but in the context of servicing receive traffic, what often matters much more is packet rate and 'burstiness'. If your average bandwidth is only 1Mbps, but that is a burst of a 15,000 64b packets back to back and then silence, it's probably still going to cause severe problems in the typical receive pipeline. Most commodity equipment and software can't handle wire rate with small packets for more than a short burst. I know you are using 1k packets here, but the point is there is a difference between evenly paced traffic and bursty traffic.It sounds like either the OS isn't servicing the driver often enough, and the on-chip buffers are filling up and tail-dropping, or your application isn't servicing the receive queue often enough and the OS is starting to tail-drop arriving packets. If you are seeing discards in the driver counters, it's probably the first one, but I don't know much about the Windows counter semantics, and this in itself might just be a ripple effect if the OS doesn't have anywhere to put the packet. Process priority and so on might come into it as well, or you may have some other driver that is taking a long time in interrupt context and starving the NIC of interrupt time.As has been reiterated many times here, the best performance you'll get out of a normal stack is to use a high-quality NIC (Intel server NICs are best IME) with the largest possible buffers and interrupt coalescing enabled. Jumbos could also help, but there are a lot of caveats with using them, so if it's not a direct point-to-point connection, I would avoid it. Commodity hardware should be pretty capable of handling a steady pace of 900mbps or so of 1k packets without much trouble. It might also help to make sure you are draining the receive queue as often/quickly as possible, even if it's just to throw it in your own in-application queue.

author: laugensalm
author_rank: Regular Contributor
timestamp: June 14, 2022, 01:05:32 pm
content: I haven't read all postings, so I might post redundant info.First, you'll have to make sure to use a network card that has proper scatter-gather DMA support without requiring massive CPU intervention, for example Intel chipsets as 82567 and older families. You might gather some information from legacy Pleora iPort hardware, as they have designed their UDP streaming backend for GigE Vision exactly for that (I've done some major testing and RE many years ago).That kinda assures that all pcap based tools (Wireshark, ...) can be trusted, so you can effectively monitor incoming peer to peer packets to ensure your transmitter is sane.One remedy for less efficient NICs is to use a Linux system. Even if not an option for deployment, it's way easier to track down problems and do long term stats/tests.The Linux stack may also decide to discard packets if they come in too bursty, e.g. it is better to stream like this:Code:. . . . . . . . .instead ofCode:... . .. ...Jumbo packets (what the Pleora solution used to appply) can be problematic to some interfaces, in particular I've found hubs that don't just block them but forward flaky packets. The performance gain isn't massive, compression may have way more impact.

author: joeqsmith
author_rank: Super Contributor
timestamp: June 15, 2022, 10:23:19 am
content: Quote from: radiolistener on June 13, 2022, 06:17:42 amI would have banked on setsockopt, buffer size.Outside of stack implementation problems, at one time I ran into problems with what ever the IT group was using for a virus scanner. There was also a time I was wanting to write data to a hard drive as part of my testing. The delays would cause drops. It also damaged a new hard drive in short order. Loaded up the PC with RAM and created a virtual drive to solve it (short term).We normally are testing at about half the data rate, sustained. Outside of that, I can't think of any other odd problems.I use the old microsoft load test from the 90s to stress my apps. You can easily drag the PC to a stop with it.

author: radiolistener
author_rank: Super Contributor
timestamp: July 06, 2022, 11:18:44 pm
content: some update. I've found that call timeBeginPeriod helps a lot on windows OS. I'm done some minor optimizations and now with timeBeginPeriod call I didn't seen packet loss for a long time. But I'm using 100-400 Mbps, I didn't tried it for higher speed.It leads to higher CPU load, but I found that CPU load unexpectedly drops down for 1/3 when I run some heavy app in the background and I still have no packet loss. It looks a bit strange, because additional CPU load in background process leads to lower total CPU loadMy tests for higher speed was performed without timeBeginPeriod call. At the moment I don't need higher speed, so I'm happy that timeBeginPeriod helps to solve packet loss issues.Currently I'm using the following algorithm to receive data:1) call timeBeginPeriod(1)2) set socket receive buffer 128 megabytes3) run single dedicated thread with highest priority, which calls recvfrom in a loop and push received data for processing.

author: Foxxz
author_rank: Regular Contributor
timestamp: July 07, 2022, 12:19:35 am
content: Quote from: radiolistener on July 06, 2022, 11:18:44 pmI'm guessing you are seeing the OS change your CPU clock speed (CPU scaling). IE - you are running a CPU intensive app in the background causing the OS to reclock your CPU cores at a higher speed which makes it look like your application is using less CPU power.Look at disabling CPU power optimizations or setting them to performance mode.