by imadbuljic | 2026-02-01 11:34:59+00:00
1. The public entry point is the part you should not DIY most of the time
Most real systems converge to a single public entrance
DNS
load balancer
your stack behind it

The lowkey hard part is not “routing a request”
It is doing it globally, under failure, at high connection volume.
Hosted LBs handle a bunch of sharp edges for you
What a global LB is doing under the hood (for example GCP LB style)

anycast VIP- users hit a nearby edge POP via BGP, not a single region
edge proxies terminate connections- TCP, TLS, HTTP parsing, header rules, WAF hooks
request steering* forwarding rule -> target proxy -> URL map -> backend service
load balancing decision- consistent hashing or similar policies, plus health checks
a real private backbone- you spend less time on the public internet, more time on the provider core

GCP specifically is pretty open about the building blocks
GFEs at the edge
Maglev-style load balancing
Envoy-based proxies in the data plane
Andromeda SDN underneath(Fascinating topic btw)

Other hosted options map to the same idea,
AWS often pairs ALB/NLB with Global Accelerator, static anycast IPs, TCP/UDP termination at edge, then ride AWS backbone
Cloudflare load balancing- anycast edge, health checks, steering, fast failover

Takeaway from this... Your “one entrance” should be boring, global, and managed, until you have a real reason to own that complexity.

2. Tail latency is rarely an endpoint problem
It is usually a queueing problem you forgot you built
p95 and p99 get created by invisible lines forming inside your tiers
Examples that bite in real systems
thread pools- you saturate workers, requests sit in a queue, p99 explodes
connection pools- DB pool at 200, 2k requests pile up waiting for a slot
async executors- one slow dependency turns the task queue into a latency amplifier
GC and memory pressure- one replica pauses, LB keeps sending, everyone waits behind it
retries- timeouts trigger retries, retries increase load, load causes more timeouts

If you want better p99, look for queues first then add backpressure and load shedding before you add more code.

3. Request coalescing is dope
If 1,000 users ask for the same thing at once, you can
do 1,000 downstream calls and die
collapse duplicates and do far less work

This shows up as
request collapsing
singleflight
cache stampede protection
batching windows

Netflix had request collapsing in Hystrix land.
Discord has also talked about coalescing in hot-path data services
Different stacks, same idea...One in-flight computation per key... Everyone else waits and shares the result

4. Horizontal scaling is THE move
Splitting the same compute budget into more smaller replicas can help
less contention per process- locks, pools, GC pressure become less spiky
smaller blast radius- one pod goes weird, you lose a slice, not the whole service
better rollout story- rolling updates, quick rollback, canaries, autoscaling hooks

Kubernetes does not magically fix bottlenecks but it makes replica-based design the default and that alone improves survivability

Takeaway...
One big box is simple, until it becomes one big bottleneck

Terminology note...
distributed means multiple nodes over a network, partial failures included
big systems is scale and complexity
*cloud is the platform you run it on