Why I Use Gigabit Equipment & Save $$$ While Fully Utilizing Xfinity Gigabit Pro 3gbps Internet
Tags: Xfinity, Gigabit Pro, Netgear, Ubiquiti, Networking

Why I Use Gigabit Equipment & Save $$$ While Fully Utilizing Xfinity Gigabit Pro 3gbps Internet #hope Follow Dec 12 · 6 min read

I did it because in my office we have gigabit fiber ISP and gigabit switches. But we had many free ports because lots of empty spaces (in the beginning). I needed extra bandwidth as I work on the cloud, so I got a two port gigabit card for my workstation (Dell 5820 10 core, 20 thread, 4+ Ghz Xeon CPU) and ended up with three gigabit ports.

It is a testament to how bad Microsoft writes code that on this “Aircraft Carrier” Workstation, Visual Studio could still use 95% CPU for no reason!

There was in instant improvement in performance. As I connect to different servers in different locations multi-threaded, I noticed improvements as soon as the new card was installed and connected to my PC. These thoughts are based on my observations when I tested the new setup.

This was an experiment fair and square. I had no expectations at all. It was just a matter of trying out: “what if?” with the limitations of the hardware in the office.

I was very surprised to see an improvement in performance

However, having worked with software interfacing with hardware before & being an Engineering Graduate in Applied Electronics, seeing the three network interfaces split the data transfer between themselves and the faster performance as a result — made sense. Do note that the ports on the PC were connected to three different ports on the main office networking switch (so not my own switch in between — that would not work)

Once we all started WFH, I improved this technique further by ethernet teaming all three ports into one windows network interface. It is an ugly way to do this — but I can confirm it works well.

This was a technique I had heard about (in the server world) to implement multi-gigabit networking with existing gigabit ports in data centers. To me, it makes sense this would work because now all data is definitely split more efficiently between three “internal” network interface physical hardware.

So, if I had a switch with a single 10 gigabit backbone port and multiple 1 gigabit ports, this is a well known technique to avoid having to get a “ full” multi-gigabit switch. Essentially, using existing PC ports to get multi-gigabit speeds without spending significant $$$.

All along I had been searching for a cheap multi gigabit switch (frantically) and only yesterday did I realize that I really don’t need multi gigabit in every port. I noticed that each multi gigabit port in a device really costs you $$.

A fully optical switch is relatively cheap, but add the backwards compatible ethernet module at $65 each and the cost goes up pretty fast. That forced me to rethink a better buying strategy.

This is why it is kind of sad, Ubiquiti does not have a device like this.

Not everyone wants to change the Ethernet cards or cables “everywhere”

I do understand though that needing everything in multi-gigabit is a more common use case for users with $$$ than me because of editing huge files and NAS and I can see why people change “everything” to multi-gigabit.

However, I still don’t think it is uncommon for one household to have two very powerful workstations, need to communicate between both in multi gigabit, and it already has more than one gigabit port on each, and cables support more than 1 gbps and they don’t want to spend on multi gigabit everywhere.

I don’t transfer single large files, I have some multithreaded programs specifically designed to be able to take advantage of this as well as different servers in physically different locations all over the world.

In most cases, there is no single destination on the Internet which will accept more than ~200 Mbps incoming, but split across 20 or 40 locations each accepting anywhere from 10–200 mbps, you can see how multiple single gigabit cards would work better than a single one. As a note, I know Azure tops out at 200 mbps incoming because I have tested that out from my office.

I am explaining all this to indicate why it makes sense because it may not be that self evident as this use case is definitely not typical.

I don’t have too much expertise in networking, but I have done extensive Wireshark based network analysis in a windows server environment a few years ago (WCF communication to be exact). WCF is not a very great remote communication technology and this was confirmed after Microsoft did not migrate it to .NET Core and also .NET 5.0 (bummer & screams from millions who use it everywhere).

I can say that I have observed multiple calls being made even as part of a single WCF call some of which could be done in parallel, and three ports are waiting on three calls to the destination — pretty sure many of the calls would be like that. A single remoting call in C#, translates to multiple calls taking a bit of time between each call to the remote server.

With my limited networking knowledge “why this works for me” seems to be a matter of:

Windows on my client machine being able to split the work between three network interfaces which can then individually talk to each remote server. Note my program is not single threaded — there is one thread per destination.

From an electronics perspective, having three physical devices splitting the work, even though the total pipe is only 1 gbps does make sense. This is because Windows is not coded very well and has lots of legacy crud slowing it down.

Hence, splitting the work at the source or the location to give you what may seem only like a slight advantage, ends up speeding it up much more than expected because of other un-optimized code.

I have so many examples of this… I just wish I could share more examples, but that is all confidential information.

Even in cases where Windows is splitting a single threaded, single package file transfer workload between three network cards, each of them communicate individually teaming together to talk to the same destination in different streams. This may seem unintuitive, but what my client sees as one destination is often behind a load balancer (as in Azure), and I know the final location is also multiple servers (like Azure Blob Storage).

I think in Linux it would not make much difference in some specific use cases within my set. But for sure, Windows has lots of baggage from the past.

Unless you use NAS based file transfer for huge files, you are right that you really do not need multi-gigabit. I have been observing my network throughout since I got Xfinity Gigabit Pro. I don’t see large throughput during Zoom, 4K video or calls. Well under 200 mbps so far.

If anything I am surprised at how less bandwidth I am using. Although ping times are super low, so everything is faster even if I am not using anywhere near full capacity.

This is one reason why — other than my single workstation PC, everything else is gigabit and I deliberately made the momentous decision to save $$$ to stay at single gigabit networking.

If you can get consistent 200 mbps up and down ISP fiber — that would be good for more use cases. To make it future proof further, 500 mbps up and down should be more than adequate.

Note, it is all a matter of scale though. If there are more people in a household, and all of them are watching 4K video, high-res audio, HD voice calls, it can all add up. Then, you need to figure out what is the reasonably priced equipment to meet the needs best.

If you think $$$ is not a limitation, go fiber instead of ethernet. You can spend a lot of $$$ to do almost anything possible. I believe in staying within “reasonable cost” however.

My last note is that, WiFi has scaling issues with the number of devices. Wifi cards die after a certain period of usage especially with heavy usage. The inherent problems with Wifi 5 and lower is a bigger problem than wired bandwidth limits.

I solve that by increasing the Wifi Access Points so it is spread out better (I like Nest/ Google Wifi but Apple was better when they had their own routers because Nest does not support bridge mode with an upstream router).

Anything before WiFi 6 has hard limits beyond 100–200 devices which are busy, and even WiFi 6 is not fully mature yet, and WiFi 6 does not matter if your clients are on WiFi 5 or below.

Nobody will say what I said about WiFi out loud. But everyone who has used WiFi at scale knows this.

Here is what happened next:

To be continued…