MurphyA/hacker-news huggingface dataset:
by derrickrburns | 2025-10-19 22:43:23+00:00
# HackerNews Announcement

## Title

Generalized K-Means Clustering for Apache Spark with Bregman Divergences

## Body (3,982 characters)

I've built a production-ready K-Means library for Apache Spark that supports multiple distance functions beyond Euclidean.

*Why use this instead of Spark MLlib?*

MLlib's KMeans is hard-coded to Euclidean distance, which is mathematically wrong for many data types:

- *Probability distributions* (topic models, histograms): KL divergence is the natural metric. Euclidean treats [0.5, 0.3, 0.2] and [0.49, 0.31, 0.2] as similar even though they represent different distributions. - *Audio/spectral data*: Itakura-Saito respects multiplicative power spectra. Euclidean incorrectly treats -20dB and -10dB as closer than -10dB and 0dB. - *Count data* (traffic, sales): Generalized-I divergence for Poisson-distributed data. - *Outlier robustness*: L1/Manhattan gives median-based clustering vs mean-based (L2).

Using the wrong divergence yields mathematically valid but semantically meaningless clusters.

*Available divergences:* KL, Itakura-Saito, L1/Manhattan, Generalized-I, Logistic Loss, Squared Euclidean

*What's included:* - 6 algorithms: GeneralizedKMeans, BisectingKMeans, XMeans (auto k), SoftKMeans (fuzzy), StreamingKMeans, KMedoids - Drop-in MLlib replacement (same DataFrame API) - 740 tests, deterministic behavior, cross-version persistence (Spark 3.4↔3.5, Scala 2.12↔2.13) - Automatic optimization (broadcast vs crossJoin based on k×dim to avoid OOM) - Python and Scala APIs

*Example:*

```scala // Clustering topic distributions from LDA val topics: DataFrame = // probability vectors

// WRONG: MLlib with Euclidean new org.apache.spark.ml.clustering.KMeans() .setK(10).fit(topics)

// CORRECT: KL divergence for probabilities new GeneralizedKMeans() .setK(10) .setDivergence("kl") .fit(topics)

// For standard data, drop-in replacement: new GeneralizedKMeans() .setDivergence("squaredEuclidean") .fit(numericData) ```

*Quick comparison:*

| Use Case | MLlib | This Library | |----------|-------|--------------| | General numeric | L2 | L2 (compatible) | | Probability distributions | Wrong | KL divergence | | Outlier-robust | | L1 or KMedoids | | Auto k selection | | XMeans (BIC/AIC) | | Fuzzy clustering | | SoftKMeans |

*Performance:* ~870 pts/sec (SE), ~3,400 pts/sec (KL) on modest hardware. Scales to billions of points with automatic strategy selection.

*Production-ready:* - Cross-version model persistence - Scalability guardrails (chunked assignment) - Determinism tests (same seed → identical results) - Performance regression detection - Executable documentation

GitHub: https://github.com/derrickburns/generalized-kmeans-clusterin...

This started as an experiment to understand Bregman divergences. Surprisingly, KL divergence is often faster than Euclidean for probability data. Open to feedback!
by TheTaytay | 2025-10-19 22:43:59+00:00
If you see this as the innocent equivalent of a “content warning,” then I would expect more apps to be flagged. The commenters in that thread point out numerous apps or games that are obviously built around content that is not appropriate for children.

Perhaps this is the only “think of the children” content warning they have, and therefore it seems odd when applied to religious texts. It’s like a movie rating system where there are only G and X ratings. If it’s not G, it gets lumped in with other stuff, including X-rated porn, and the only way to find it in our App Store is to allow for X-rated content.

Seems like a bug at best, but I think you’d have to be pretty naive to think this is an “aww shucks, rules are rules” application of some policy.
by baobun | 2025-10-19 22:44:41+00:00
We should really compare it to Windows here, since that's the target. But if we do compare it to a classic Linux dist like xubuntu as baseline:

Using Qubes would limit the blast radius for a scenario like this. In QubesOS, you would use disposable VMs (with no access to your crypto wallets or other user files) to download and flash an ISO. So even if this malware was targeting Linux, it wouldn't get zit and disappear when you finish flashing and shut down that VM (as long as there isn't an unpatched exploit breaking the VM isolation involved).

Of course, if the ISO is bad then this won't save you from compromise once you boot it. But that's not what happened here.
by tavavex | 2025-10-19 22:44:53+00:00
The whole story seems utterly insane to me. The fact that Novo Nordisk went back to ask for a refund of $250 they paid for the patent in 2017 shows that there was active intent behind this, this wasn't just some internal disconnect between payment systems that prevented the money from being sent to the government. Who at NN thought that saving, what, ~$1000 total was justified for this? I thought companies of this scale didn't even think on the scale of thousands of dollars anymore.

Oh well, at least we in Canada get more generic drug variants. Thanks!
by autoexec | 2025-10-19 22:45:38+00:00
Not a great day to try to argue how well maintained and "rock solid" windows is considering the issues it's having (see https://www.techpowerup.com/342032/windows-11-25h2-october-u...) not to mention all the other updates that've caused data loss or broken things and that's just windows 11! Just paying a bunch of people to push out updates isn't enough for a well maintained OS.
by sarchertech | 2025-10-19 22:45:42+00:00
Even if the brain showed a real deviation from the Standard Model, we’d still need colliders. The brain is a messy, low energy environment, and observation isn’t identification.

Colliders provide clean, high energy tests to pin down masses, spins, and couplings, and we already need them for open problems. A bio anomaly would set targets, not replace colliders.
by Telaneo | 2025-10-19 22:46:33+00:00
CRTs were only ever made sense to manufacture on a really big scale, so that costs could be reduced. Early tubes which weren't manufactured on such a scale were accordingly stupid expensive.

I doubt anyone is going to spin up another factory to satisfy the potential demand, since the demand isn't that great to begin with (OLED satisfies most use-cases that CRTs do), and very few people are going to pay $5000+ for a new CRT, and I doubt they're going to be any cheaper than that.
by userbinator | 2025-10-19 22:47:06+00:00
The newer compressors are smaller so they cost less, but are run faster to pump at the same rate. Many old hermetic compressors used a 4-pole 1500/1800RPM motor, then they became a 2-pole 3000/3600RPM, and the newest VFD/inverter motors can go even faster.
by JumpCrisscross | 2025-10-19 22:48:02+00:00
> With de minimis for US-bound packages suspended I suspect way more packages are inspected than used to be

But a smaller fraction.

If you’re paranoid, route it via the UAE. All my European and Indian shippers are doing that for tariff-free pricing. (Personal stuff. I’ll pay a customs duty if I get it, of course.)
by epcoa | 2025-10-19 22:49:05+00:00
Nah, dishwashers are pretty light too. With a muscle mass of 1% I usually just flip it over to work on it. This is just peak HN, PhDs still phased by something requiring an 8th grade level of education. In the US, the supply is usually a screw on, the drain a clamp and if the wiring isn’t already a quick connect just throw some Wagos on.
by Telaneo | 2025-10-19 22:49:18+00:00
CRTs wear out with use, so they're only getting rarer by the day. The electronics can mostly be fixed, but the tubes can't. You can extent their lives a bit, but you're only delaying the inevitable. When it's gone (too low brightness, burn-in, bad focus), there's nothing that can be done about it to get it back to the way it was when it was new.
by kenjackson | 2025-10-19 22:49:54+00:00
“Mathematician Thomas Bloom, who runs erdosproblems.com, pushed back right away. He called the statements "a dramatic misinterpretation," clarifying that "open" on his site just means he personally doesn't know the solution - not that the problem is actually unsolved.”

What mathematician uses this as the definition for “open”? I don’t go around saying that most problems in this textbook are open questions, just because I don’t know how to do them.
by kace91 | 2025-10-19 22:50:57+00:00
>I'm not sure why orders/purchases/subscriptions would be seen as mutable. At some point in time, customer X decided to exchange money amount M for a subscription Y.

Think for example about orders where a costumer bought three items and later cancelled one, the order value mutates as it is updated and at most we have a copy of the previous order state before price was updated (for some cases not even that).

If you think that’s not a good modeling for financial processes, well so do I, it’s the legacy we’re supposed to manage; moving out from that type of non ideal system to something more solid is what I’m researching.
by mihaaly | 2025-10-19 22:50:59+00:00
Keyboard troubles?!

I wonder if this is related to what I experienced. After the update (update and shoutdown reliably updates and restarts again, does not shut down) a parctice of mine switching Control and Fn keys on my Mac so Control key function gets into the same physical position as Ctrl on the PC keyboard, so using Windwos through Microsoft Remote Desktop and at the keyboard of the Windows computer is a smoother switch, is not working anymore. Windows, through the Remote Desktop does not register the Fn key as Ctrl anymore. The whole thing does not make sense to me. The Remote Desktop software on Mac did not change, MacOS should send Remote Desktop the signal of Control key pressed when pressing Fn, the Windows update shall have no effect, yet the sole change here was the Windows update when this annoying thing emerged. I simply had no time to dive into diagnostics and find the underlying reason, it is less resource intensive and less painfull - but one more annoyance on top of the many concerning Windows use - learning to use different Ctrl button location on Mac and PC keyboard again (done before, before learning the Control <> Fn switch trick).
by analog31 | 2025-10-19 22:52:42+00:00
I'm not a Matlab user, but from what I can tell, even if the language can be cloned, there's a lot more to Matlab: It's a GUI driven software suite, with a lot of pre-written apps that eliminate the need for coding in many cases.

It comes with vendor support and "official-ness" for lack of a better word.

Things are changing rapidly in this area but it wasn't very long ago that most people reacted to open-source software as something weird that shouldn't be trusted.
by Nux | 2025-10-19 22:53:48+00:00
Microsoft gaining the most I reckon.

Kind of sad seeing businesses getting screwed by closed source proprietary software, then making the same choices all over again.

Nutanix also seeing huge demand.

Not everyone is repeating their mistakes, with Proxmox and Xcp-ng seeing huge new level of business, as well, which is nice.

I'm part of the Apache CloudStack project and that too is seeing unparalleled levels of demand. The KVM hypervisor has sort of become the de facto choice, thanks to virt-v2v tool which can help migrate VMware guests.
by skopje | 2025-10-19 22:56:46+00:00
My first reaction to this was, "this is dumb and annoying", and then I played with it for a few minutes and found a set of levels that were causing my symptoms to briefly go away. It still isn't perfect, but a combination of Steve Seq and Pulses High creates "gaps" in my tinnitus, where my brain seems to "forget" about it briefly and I don't hear it. But then it comes back a millisecond or so later. I don't think I could get work done with this on all day, but it has more effect on my symptoms than other websites I've tried. It's really interesting, like there are brief (millisecond) periods of no tinnitus. [I've had it in my left ear for probably 35 years (I played in a punk band in the early 80's with no hearing protection but it didn't show up until a decade later).]

Keep at it, I think you're on to something!
by weitendorf | 2025-10-19 22:57:28+00:00
How willing are you to farm for your own food, or let someone else decide what's worth eating? How willing are you to order other people what to do without taking those orders yourself? Do you think that you're going to be the one in charge in a world where someone gets to decide we have enough different kinds of cereal? There are places where you can live like that right now, most people don't like them.

And if you think I'm whatever cultural archetype you hate that keeps things this way, just know that my life sucks too and I'm giving up a lot to just try to make it better for other people instead of moseying along hating what I do and how powerless I am. But pretty quickly you realize people hate that even more than they hate their own suffering, nothing you do will ever be good enough for them because they just want to be miserable and make it someone else's fault - if they take any responsibility for it, they have to admit that it's not going to just change for them.

Nothing is actually stopping you from making your own music, nobody has to listen to it though, unless you make them. But it's your only choice if you don't like the music you have already. Just get ready for people to tell you they don't like it.
by tuatoru | 2025-10-19 22:57:36+00:00
This time actually is different, though.

If everything that a human can do, a robot can do better and cheaper, then humans are completely shut out of the production function. Humans have a minimum level of consumption that they need to stay alive whether or not they earn a wage; robots do not.

Since most humans live off wages which they get from work, they are then shut out of life. The only humans left alive are those who fund their consumption from capital rents.
by lgats | 2025-10-19 22:57:42+00:00
update: thanks for all the suggestions

I decided to do some testing with redirecting to a small vps that just keeps the connections open and sends a byte every 10-30 seconds. This worked and the traffic substantially dropped off. After doing some more digging though, I got concerned this may be in itself an abuse of my VPS providers ToS. The risk did not outweigh the benefit. Gzip bombs fell under a similar category of concern.
by the_gipsy | 2025-10-19 22:58:56+00:00
Let me be clear: yes, I know I might be wrong. I hope I'm not dumb and wrong, or at least not dumb. I am also not writing here as some kind of debate exercise. I do because I find this topic extremely interesting and insightful. What if language is the intelligence? What if "guessing the next word" really was all that was there, to peak human intelligence, knowledge, and understanding of our world? I am not hyped by AI, it's rather that I find this possibility somewhat sad.

I've made up a model, an idea, and I don't think the data opposing it is trustworthy. My first problem is that there are many people that claim that they have NO internal monologue, which means NEVER constructing a sentence from theirselves or others in their head (except directly as verbal speech), and this seems outright impossible. When pressed, these people usually either admit that they do have some monologue, just "much less". Or they misunderstood it for something similar to schizophrenia, actual hallucinations. If they don't admit to actually, sometimes, having them, then they fail to explain where exactly the line between "thinking of someone or themselves saying something" and the internal monologue/dialogue is. As if they had been caught lying by the detective, they end the conversation. Or at least that's how I feel, I really don't know how to ask more questions here before making them feel too interrogated, or like someone that has self-diagnosed being told that they are imagining things.

With "absolutely none" group out of the way, it leaves us with people who claim to perceive the internal monologue very scarcely, and claim that they do not need to "think" or "do". How can we possibly test this scientifically? The data is all self-reported. Or at least I don't know if this can or has been neurologically researched.

Consider also that all self-reported data about internal monologue is "poisoned": we are trying to get objective data with the data itself as a vehicle. We are not asking if someone feels pain, or if they can solve a puzzle in a timeframe. We cannot measure electric activity with some instruments, nor evaluate yes-or-no questions.

What if it is true that some people do not perceive their internal monologue? I certainly don't remember it "popping" into my head at a certain age, and I think nobody does. When we learn language, we become conscious with it, because it allows to model the world, beyond putting things in our mouth and screaming. So it could be that not everybody perceives it equally, a spectrum like you said, and that some people rationalize it retroactively as not being there - just "thoughts", ideas, feelings. We reconstruct past events via a narration, filling in details by guessing, so why wouldn some people guess that they are not narrating in their head? It is not something that is taught in school or from or parents, you either perceive it as "internal monologue", or as "just thinking", because, well, it's the thinking doing it's thing.
by Pannoniae | 2025-10-19 22:59:16+00:00
Amazing presentation, the real gold is buried somewhere in the middle:

"To paraphrase Kent Beck, professional software development is about...

Communication and collaboration between large numbers of people with different perspectives."

This is why large organisations suffer, communication;)

People say it a lot that the code isn't most of the value but the auxiliary stuff is, which is true - but the conclusion from that shouldn't be that the work itself isn't that important, it's just that you'll see way less of it in a large organisation.

This is also why a well-aimed startup can easily blow the socks off companies with thousands of employees - the communication overhead is high, accountability is very diffused and everything is codified.

In an ideal world, you'd spend 100% of your time with either the work or something adjacent (like research), but of course, that's completely unrealistic. But if you aren't doing work but something else, it's probably a good idea to ask yourself, does this matter? Does it help my goals?
by zamadatix | 2025-10-19 23:00:11+00:00
Hard to say much given we weren't given much info on how it'd be used.

E.g. Parallel's is only useful for people looking to run VMs locally on their Mac, but Hyper-V can be anything from that for a Windows PC to a full-blown headless hypervisor cluster with HA, shared volumes, replication, etc.

For several of the common categories, these are my takes:

- Traditional Enterprise: Nutanix [paid] if money is available, otherwise Hyper-V [paid] if a large Microsoft contract is already in place. If neither fit: fall through to acting like an SMB.

- SMB/Modern Mid-Sized Enterprise: Cloud [paid] only and/or Proxmox [free/paid]

- Tech Company: Doesn't matter, they'll do whatever sounds cool that year and make it work well enough

- Home Lab: Proxmox [free/paid]

- Windows PC: Hyper-V [free w/ Windows] (it's meh, but it's integrated - doubly so if you plan on using WSL on the side).

- Mac PC: Parallels [paid] if you need a GPU accelerated Windows guest, UTM [free] otherwise.

- Linux PC: QEMU+KVM [free], the choice of (optional) GUI client is up to preference.

Some extra notes by solution:

- Nutanix: Enterprises were staring to use this more and more even prior to the VMware sale. It's definitely the spiritual successor of traditional VMware usage in the data center. A bit less full of themselves, for now at least, than VMware ever managed to keep themselves (IMO).

- Proxmox: Has a bit of a habit of feeling like it always ends up a little broken by the time you've used an install/cluster for 6 months, but is by far the best option for the homelabber type use case (even ignoring that it's free as a reason). It's basically like someone configured KVM with what you want to be able to just (try to) use it without thinking about what's underneath, while still having access to the underneath to un-stick it in certain situations. Also does host-native containers! I never did have the guts to pitch my company try to run anything production on a cluster, but they do have reasonably priced support plans and advanced feature tiers for that.

- Parallels: Kind of sucks for the price, but there isn't anything else on macOS with the same GPU acceleration for Windows.

- Hyper-V: I think this is mostly still around because it helps Microsoft stay sticky at companies when the yearly renewal comes up. That said, it's alright - and it's also integrated into Windows in some pretty nifty ways for local use these days.

- UTM: Fantastic QEMU client for macOS, worth giving a few bucks for even though it's free.
by Neywiny | 2025-10-19 23:00:27+00:00
I'm sure there are exceptions to these but: SBCs have micro*processors* not microcontrollers. What's the implication? SBCs run Linux. What's the implication? You can develop a set of programs and stacks on a big desktop, and kinda just copy-paste boom it's on an embedded device. Often including networking, USB, HMI, cameras, etc. All can be done on a microcontroller (though you'd want something with more power than a pico), but not copy paste.
by ekr____ | 2025-10-19 23:00:31+00:00
This is only a tiny part of the trainwreck that is IETF and IETF-adjacent document nomenclature.

IETF documents start as Internet-Drafts, which officially are just draft documents that can be changed at any time. As a practical matter, some I-Ds really have no status and some have been accepted by WGs to work on. You can tell which are which because (mostly) the latter are named draft-ietf-something. As WG documents progress, it's not uncommon to see very wide deployment based on an I-D (this was the situation for QUIC and TLS 1.3), whereas other drafts are just totally half-baked and nobody is deploying. There is no good way to know which is which without paying attention.

Inside the IETF, RFCs can be published be any of Standards Track, BCP, Informational, or Experimental. Nominally, the first group are really standards (see asterisk below), whereas the BCPs are normative but not really standards (e.g., they might describe how the IETF is supposed to run), whereas the latter two are not standards. Except sometimes they are de facto standards, like RFC 6962, specifying Certificate Transparency (note that there is another RFC for CT, RFC 9162, which nominally supercedes RFC 6962, but is not widely implemented). There are two standards levels, Proposed Standard, and Standard (there used to be a middle one, Draft Standard), but as a practical matter lots of specifications stay at PS forever because the WG or authors can't be bothered to get them promoted to full standard. QUIC, TLS, and HTTP are all Proposed Standards.

Moreover, there are other IETF-adjacent organizations that publish RFCs, such as the Internet Architecture Board (IAB) and the Internet Research Task Force (IRTF). These documents aren't standards at all. Finally, there is an Independent Submissions Editor (ISE), appointed by the IAB, who basically can publish whatever he/she wants on the Independent Stream. Note that this is different from an Individual Submission, which is a document processed by the IETF without going through a WG.