MurphyA/hacker-news huggingface dataset:
by liuliu | 2025-12-12 23:25:42+00:00
I usually call it "head parallelism" (which is a type of tensor parallelism, but paralllelize for small clusters, and specific to attention). That is what you described: sharding input tensor by number of heads and send to respective Q, K, V shard. They can do Q / K / V projection, rope, qk norm whatever and attention all inside that particular shard. The out projection will be done in that shard too but then need to all reduce sum amongst shard to get the final out projection broadcasted to every participating shard, then carry on to do whatever else themselves.

I am asking, however, is whether that will speed up decoding as linearly as it would for prefilling.
by mrandish | 2025-12-12 23:25:42+00:00
I agree. Charging a blanket percentage of gross revenue is an extremely inexact way to monetize what is a broad basket of services that were previously separate including: electronic software delivery, software security verification, marketplace, transaction processing, DRM, etc. Since 2009, first on Apple's app store and then Google's, these services have all been arbitrarily bundled together despite having vastly different one-time, fixed and variable costs. People are only used to it in this context where every marketplace has been controlled by a monopolist gatekeeper.

Doing it this way makes no economic sense for either the seller or the buyer but it's coincidentally the absolute best way for a middleman to maximize the tax they can extract from a two-sided marketplace they control. In competitive markets, blanket taxing on total gross revenue generally only occurs when there's a single fundamental cost structure tied to that revenue, or the amounts being collected are so small it's de minimis. App stores are highly profitable, multi-billion dollar businesses.

Perhaps the most perverse thing about this is that electronic transactions for purely digital goods which occur entirely on real-time connected digital platforms make it trivial to price each service for maximum efficiency. It's easy for the price a 2GB game with frequent updates pays for electronic delivery to reflect the cost they impose on the infrastructure while a 100k one-time purchase app can pay a vastly smaller amount. And that's exactly the way the competitive marketplaces evolve - from moving shipping containers around the planet to residential propane delivery.
by dweekly | 2025-12-12 23:26:03+00:00
Good points, all - there is a lot of subtlety here.

CFHostStartInfoResolution is deprecated, no? https://developer.apple.com/documentation/cfnetwork/cfhostst...:)

That leaves us with DNSServiceGetAddrInfo? https://developer.apple.com/documentation/dnssd/dnsservicege...:) or some kinda convoluted use of Network and NWEndpoint/NWconnection with continuations could do the same?
by zackangelo | 2025-12-12 23:26:39+00:00
You’re right I misunderstood.

I’m not sure if it would be of much utility because this would presumably be for tensor parallel workloads. In that case you want the ranks in your cluster to be uniform or else everything will be forced to run at the speed of the slowest rank.

You could run pipeline parallel but not sure it’d be that much better than what we already have.
by Izkata | 2025-12-12 23:27:19+00:00
This is far from the best way to do it, but this is a much easier to understand example of how it could be done without having to read about math:

There's a type of token called a JWT that's really common nowadays, which is composed of 3 parts: Metadata describing encryption for the third part, the actual base64-encoded data, and the encrypted signature. The second part would include "is over 18" and "expiration date" to limit reuse/abuse, and is trivially decoded by anyone to confirm there's no personal information in there.

You'd get this token from your government site and copy/paste it into the site needing verification. The government site would provide a standard public key that can be used with the third part of the JWT to confirm it hasn't been tampered with (verification is built-in to JWT libraries). There would only be one public key that rarely changes, allowing the site to cache it, preventing the government site from correlating users based on timestamps - they never see the JWT from the other site (verification is done locally), and the other site would only need to pull the public key once for however many thousands of people use it.

...that said technical issues aside, I kinda feel like this would be the most acceptable version simply because it doesn't require the average user to trust the math - they could go to a JWT-decoding website and look at it themselves.
by quesera | 2025-12-12 23:27:44+00:00
I think this is just an appealing fantasy.

80% of modern Americans would not survive and successfully complete a month-long walk of hundreds of miles.

Add in the logistical challenges, necessary supplies, and secrecy ... and of course the possibility of harsh seasonal weather ... and I think we're talking about a 5% survival rate, at the absolute max.

People might think they are more likely to be in that 5% than their neighbors, but I suspect most of the dangers are random or universal enough that this would not be true. Fortune favors the prepared, certainly. But that only gets you so far.

There is no viable bug out strategy, after a certain point (and that point is far from today, and will likely never be reached).

But there are midterm elections, and those are important.
by lesuorac | 2025-12-12 23:27:50+00:00
Can you guys just read stuff before talking?

> The order directs Attorney General Pam Bondi to create an “AI Litigation Task Force” within 30 days whose "sole responsibility shall be to challenge State AI laws" that clash with the Trump administration's vision for light-touch regulation.

The EO isn't about Federal Preemption. Trump's not creating a law to preempt states. So a question about how Federal Preemption is relevant is on point.
by patagurbon | 2025-12-12 23:28:09+00:00
I'm not opposed to alternative research organizations, funding structures, etc. But this does seem like a fairly direct attempt to shift funding from universities into private firms, a funding structure which IMHO is much easier to abuse.

> Tech Labs will provide entrepreneurial teams of proven scientists the freedom and flexibility to pursue breakthrough science at breakneck speed, without needing to frequently stop and apply for additional grant funding with each new idea or development.

This sounds great, but has a few issues. #1 only funding proven scientists risks destroying the training pipeline which is the crucial edge that the US has over any other country in the world. It's also something that can and should be applied to universities as well. Lab groups or centers should be given much more runway than they are now.

> coordinated, interdisciplinary teams to achieve success

There are two places where this can really happen today: universities and national labs. The NSF should be fostering more cross-disciplinary and product engineering research across different departments at universities which already have deep talent pools across the board.

> The Tech Labs initiative will support full-time teams of researchers, scientists, and engineers

This sounds great, more university labs should have full-time researchers attached. Research Software Engineers are one somewhat common example in the computational sciences.

In general I support the overall mission statement but I am extremely wary of this kind of rhetoric from this government. They have failed to walk the walk on the sciences in any domain. This seems like a Trojan horse to transfer more money from the research apparatus into industry.
by fluoridation | 2025-12-12 23:28:33+00:00
>Givers think that conversations unfold as a series of invitations; takers think conversations unfold as a series of declarations.

I don't really understand the thesis outlined in the article. "Givers" and "takers" are defined like this, but it actually sounds like the two types of conversationalists are "actives" and "passives", where actives seek to move the conversation forward and passives let others move it forward. A giver-and-taker conversation where both participants are alternatingly active can work. The giver asks a question and the taker answers it but then adds something of their own that doesn't let the conversation grind to a halt.

Example:

A: Hey, have you heard about X? (giver, active)

B: Oh, I hate X. I think Y. (taker, active)

A: Woah, hang on. I'm not so sure about Y. (taker, active)

B: Oh, yeah? Do you think Z? (giver, active)

In my experience, the absolute worst conversations I've had were those where I felt I was the only one putting in any effort, trying to come up with topic after topic only to have them peter out in under a minute, followed by silence.

I also don't know that people are necessarily fixed in their roles, be as giver, taker, passive, or active. In fact, if I'd have to guess, an engaging conversation has the participants constantly switch roles with the flow, depending on how much they have to say on a given topic.

So I think a corollary from all this is that a conversation breaks down when an active participant switches to passive expecting the other to become active, when in fact the other person just wants to be passive, or when two passive people try to have a conversation, in which case nothing happens at all.
by nullstyle | 2025-12-12 23:28:53+00:00
I don't think we will agree here. The statement that "The whole concept of nationalism and border sovereignty has been with us for essentially all of human history" is not something I can get down with unless its better supported. The territory you are describing is not all the same thing as national territory to my mind, and your arguments are not convincing.

> they'd immediately investigate and either allow them in (if e.g. someone friendly temporarily traveling through) or send them back in the opposite direction with force if necessary.

Was there never the case that they investigated, saw that the strangers were floating down a river on the border of "their territory" and simply let them pass through unmolested? That doesn't happen today, and my intuition is that was simply so much space in the americas before recorded history that it happened often then.
by hinkley | 2025-12-12 23:29:28+00:00
A contract that needs to be maintained at some level of quality even when you're deploying or overloaded.

Load shedding is a pretty advanced topic and it's the one I can think of off the top of my head when considering how Chesterton's Fence can sneak into these designs and paint you into a corner that some people in the argument know is coming and the rest don't believe will ever arrive.

But it's not alone in that regard. The biggest one for me is we discover how we want to write the system as we are writing it. And now we discover we have 45 independent services that are doing it the old way and we have to fix every single one of them to get what we want.
by ben_w | 2025-12-12 23:30:20+00:00
I'm saying Trump isn't even afraid of an uprising.

That's not a condemnation of the people failing to replicate the Jan 6 thing while switching red flags for blue ones.

I'm saying the freedom to have guns didn't do the thing that people say it does.

I would go further: I also don't think private arms would work if y'all did start firing them, because who wins a civil war is more likely to depend who the military supports than private firearms, and if the military aren't the deciding factor for whatever reason (internal split within the military?) then it's down to whoever outside the country is supplying weapons/logistics to which side (so, probably China?)
by jascha_eng | 2025-12-12 23:30:25+00:00
Depending on your workload you might also be able to use Timescale to have very fast analytical queries inside postgres directly. That avoids having to replicate the data altogether.

Note that I work for the company that built timescale (Tiger Data). Clickhouse is cool though, just throwing another option into the ring.

Tbf in terms of speed Clickhouse pulls ahead on most benchmark, unless you want to join a lot with your postgres data directly then you might benefit from having everything in one place. And of course you avoid the sync overhead.
by FireBeyond | 2025-12-12 23:30:35+00:00
I doubt many of them are, either.

When the 2019 Mac Pro came out, it was "amazing" how many still photography YouTubers all got launch day deliveries of the same BTO Mac Pro, with exactly the same spec:

18 core CPU, 384GB memory, Vega II Duo GPU and an 8TB SSD.

Or, more likely, Apple worked with them and made sure each of them had this Mac on launch day, while they waited for the model they actually ordered. Because they sure as hell didn't need an $18,000 computer for Lightroom.
by literatepeople | 2025-12-12 23:30:43+00:00
I would say it's very different to what we do. Go to a friend and ask them a very niche question. Rather than lie to you, they'll tell you "I don't know the answer to that". Even if a human absorbed every single bit of information a language model has, their brain probably could not store and process it all. Unless they were a liar, they'd tell you they don't know the answer either! So I personally reject the framing that it's just like how a human behaves, because most of the people I know don't lie when they lack information.
by anyfoo | 2025-12-12 23:30:54+00:00
I haven't really used Windows for anything serious in more than 20 years (and I recently had to mess with Windows 11 and it was terrible), but I'm not sure you'd be very happy going back to Windows 3.1.

It was a 16 bit system (it could run in "Enhanced Mode" which involves 32 bit protected mode, but in reality Windows itself, and the applications, were still 16 bit).

That means the resource constraints were very real. Even if you had a lot of actual memory in your machine, the memory that was actually available for "general purpose" was effectively a few hundred kilobytes. There was also the notion of finite (and very generically-named) "system resources", and you could see in the "About" box how many percent of those you had free. Once they were gone, you were in trouble:

USER.EXE and GDI.EXE each have a data segment (that is, heap) limited to 64K. The 8086/80286 platform architecture imposes this 64K limit. Program Manager checks the percentage of free heap space for both USER.EXE and GDI.EXE. It then reports the smaller of the two percentages.[1]

All applications ran in the same address space. A broken application meant a total crash at best, subtle data corruption at worst. Multitasking was also cooperative, so apps could hold up other apps indefinitely, or just hang the entire system.

Since it was not based on paging, to accommodate the very limited memory, entire segments could be swapped out, or even relocated, within the address space. As a programmer, that meant dealing with stuff like "locking pointers" so Windows wouldn't move your data segment under you. As a user, that could mean general slowness.

It was firmly based on DOS. So many problems that you had in DOS, drivers or whatnot, would exist in Windows as well.

There were better systems at the time that you could wish yourself back to, some number of them based on UNIX in some way or other.

But Windows 3.11 had really pretty icons. The prettiest, in my mind.

[1] https://ftp.zx.net.nz/pub/Patches/ftp.microsoft.com/MISC/KB/...
by api | 2025-12-12 23:31:23+00:00
No way buying a bunch of minis could be as efficient as much denser GPU racks. You have to consider all the logistics and power draw, and high end nVidia stuff and probably even AMD stuff is faster than M series GPUs.

What this does offer is a good alternative to GPUs for smaller scale use and research. At small scale it’s probably competitive.

Apple wants to dominate the pro and serious amateur niches. Feels like they’re realizing that local LLMs and AI research is part of that, is the kind of thing end users would want big machines to do.
by mpweiher | 2025-12-12 23:31:34+00:00
Nothing man-made is perfect, not even nuclear. But even with the two big accidents, nuclear is the safest form of energy generation we have. By far.

The actual safety is almost entirely unrelated to the fears people have. There is even a term for this: radiophobia.

https://en.wikipedia.org/wiki/Radiophobia

Radiophobia is an irrational or excessive fear of ionizing radiation, leading to overestimating the health risks of radiation compared to other risks. It can impede rational decision-making and contribute to counter-productive behavior and policies.
by wmf | 2025-12-12 23:32:29+00:00
inference accelerators ... It’s why Apple is so far ahead of their peers in local device AI compute, and will be for some time.

Not really. llama.cpp was just using the GPU when it took off. Apple's advantage is more VRAM capacity.

this introduction isn’t just for large clusters

It doesn't work for large clusters at all; it's limited to 6-7 Macs and most people will probably use just 2 Macs.
by cogman10 | 2025-12-12 23:33:39+00:00
Other answers are good. One more that you could do is put the JSON document inside a container (A zip archive for example). Then your document can effectively be

invoice.inv (zip archive) └- payload.json └- signature.asc

This has the benefit of adding more opportunities for many json documents within the archive.

It's effectively what the Java jar is.
by mapontosevenths | 2025-12-12 23:33:48+00:00
You arent wrong that some people just stink at finance. Thats not what I'm talking about here. I'm talking about those who WOULD make better decisions if they were able. What is sometimes referred to as "the poverty trap."

> You make dangerous assumptions about me in order to justify your preexisting view of poverty and socioeconomic mobility

You said that you don't struggle making good financial decisions. I am claiming that you were lucky to be in a situation that allows you to make good decisions.

I believe that to be accurate, regardless of your history.

You must remember that the legitimately poor often do not have many choices to make at all. Poverty can constrain choice to the point of irrelevance. EG - Somone with a thousand dollars has more options than someone with a hundred dollars.

In your case it seems that you (now at least) have enough to be able to make choices. If you only had $50 to your name you would have far fewer options to choose from and most of those options would be bad.

Imagine a single mother working for minimum wage with a flat tire she cant afford to fix, and needing groceries. Instacart might make her problem worse, but surviving is all she can hope for sometimes.

Similary imagine an elderly widow on fixed income who is injured. Or a 17 year old who had to flee an abusive home situation, and is lucky to make rent on a weekly rental room.

For any of them it is easy to imagine they might have enough money to eat, but not enough to buy a car or even a bicycle. They will starve long before the situation improves enough to make that happen, regardless of their choices. So they make ends meet and they survive.

Its not Instacarts fault, any more than it is Dollar Generals, but it is also true that the service often worsens their long-term well-being.
by exasperaited | 2025-12-12 23:33:59+00:00
> Frankly, much worse things than jail should happen to Altman for this kind of torching of the commons, and jail is the watered-down compromise position.

Some days I think the devastating crash of the economy that will come if the bubble bursts is the least worst outcome. Do people not feel like the tensions around AI will not soon become internationally geopolitical?

(It's already nationally geopolitical in the USA: Trump is trying to assert federal control over the states' rights to set their own legislation)
by frumplestlatz | 2025-12-12 23:34:13+00:00
Oh yes, good catch. Yeah, you want to use `NWConnection` (or one of the other higher-level supported networking APIs), which raises another issue with doing custom DNS resolution. You need those API's connect-by-name semantics to get VPN-on-Demand:

https://developer.apple.com/documentation/technotes/tn3151-c...
by overfeed | 2025-12-12 23:35:40+00:00
You present a very long list of evidence that proves the current status of the market is not competitive, and illuminates the power imbalance. We probably agree that this is not a self-correcting state.

AFAICT, the difference in perspective is what to do first, with some of us recommending more unionization, and your take being to fix the competitive landscape. IMO, unionization is in the hands of the workers, and is easier to accomplish compared to addressing competition when - as you noted - there's regulatory capture. While it probably offends your sensibilities, the how of unionizing is self-explanatory; it's not at all clear to me how we would go about making all the industries competitive again. It's a worthy goal, but for now, it seems to be a solution for a world with spherical cows and no force of friction.
by gerdesj | 2025-12-12 23:36:01+00:00
I am absolutely no expert but I doubt many of the components of this beast are even expressible in anything currently machine readable (perhaps definable is a better word for now).

The article clearly states that there are multiple reading groups across the world attempting to get to grips with each small aspect of the ideas involved. That they even attempt this implies to me that the ideas are considered worth studying by some serious players in the field: the group (its way more than just one Fields toting bloke) have enough credibility for that.
by razster | 2025-12-12 23:36:03+00:00
I’ve paired my Z-Image Turbo with SeedVR2 upscale, running on a RTX3060 12gb, 32gb sysMEM, generates in 40sec. I’m holding out for Z-Image Edit that is a larger model, once that is out… going to be interesting. Oh and to train your own ZIT LoRA, takes 5hrs for 3000 steps. So fast.
by mpweiher | 2025-12-12 23:36:06+00:00
Nuclear power is dispatchable, unlike renewables.

Giving a preference to intermittent renewables is not a law of nature, but a rule that is irrational and needs to be removed.

Denmark is just now hitting problems with their wind strategy, and of course dependent on being a transit land between large producers and consumers. And currently looking at nuclear. As is Norway.

One of the reason is that intermittent renewables are pro-cyclical, that is once they reach a certain level of saturation, they cannibalize each other even more than they cannibalize steady suppliers.

The current plan is to quadruple nuclear power in the UK.
by Agraillo | 2025-12-12 23:36:34+00:00
A semi-scary thought came while reading the post: LLMs could talk to each other without humans noticing (for example using a very complex acrostic). But not in the form of chat-to-chat, which not only is rarely used in real life but also won't likely have lasting consequences (the context will eventually be lost). I was thinking that new web content, more and more of it AI-generated, could contain hidden messages that later might be absorbed into the training data of other LLMs. Maybe this leans more toward a plot for a black comedy than a genuine concern, but who knows...
by oulipo2 | 2025-12-12 23:36:56+00:00
Nice, what would be your typical setup?

You keep like 1 year's worth of data in your "business database", and then archive the rest in S3 with parquet and query with DuckDB ?

And if you want to sync everything, even "current data", to do datascience/analytics, can you just write the recent data (eg the last week of data or whatever) in S3 every hours/days to get relatively up-to-date data? And doesn't that cause the S3 data to grow needlessly (eg does it replace, rather than store an additional copy of recent data each hour?)

Do you have kind of "starter project" for a Postgres + DuckLake integration that I could look at to see how it's used in practice, and how it makes some operations easier?