MurphyA/hacker-news huggingface dataset:
Show HN: Cobalt – Unit tests for AI agents, like Jest but for LLMs
by fdefitte | 2026-02-20 17:43:05+00:00
Hey HN, I built Cobalt, an open-source testing framework for AI agents and LLM apps.

Most eval tools (Braintrust, Arize, LangSmith) want you to live in their UI. Dashboards, manual reviews, clicking through results. That's fine for exploration, but it doesn't catch regressions. We needed something that runs in CI like any other test suite, lives in code, and fails the build when quality drops.

npm install @basalt-ai/cobalt
npx cobalt init
npx cobalt run

Write experiments as code:

import { experiment, Dataset, Evaluator } from '@basalt-ai/cobalt'

const dataset = Dataset.fromLangfuse('support-tickets')

experiment('support-agent', dataset, async ({ item }) => {
const result = await myAgent(item.input)
return { output: result }
}, {
evaluators: [
new Evaluator({ name: 'Helpful', type: 'llm-judge', prompt: 'Is this response helpful and accurate? {{output}}' }),
new Evaluator({ name: 'No hallucination', type: 'llm-judge', prompt: 'Does this contain fabricated info? {{output}}' }),
]
})

`npx cobalt run --ci` exits with code 1 if thresholds are violated. The GitHub Action posts score tables on PRs and auto-compares against base branch.

The part I'm most excited about: Cobalt ships with a built-in MCP server, so you can drive it entirely from Claude Code. Just tell it "compare GPT 5.2 with 5.1 on my support agent" or "run my experiments, find the failing cases, and fix the prompt." It runs the experiments, diffs the results, and iterates on your code without you leaving the terminal. Turns eval from a chore into a conversation.

Pull datasets from Langfuse, LangSmith, Braintrust, or plain JSON/JSONL/CSV. Results stored locally in SQLite. No accounts, no dashboards, no vendor lock-in.
by vunderba | 2026-02-20 17:43:26+00:00
> Taste is subjective.

If I spend twenty years subsisting solely on a high sodium cup-of-noodle diet, get severely impaired under the influence of everclear while trying to use a straight edge razor for the first time, hang up a white canvas, and spin around like a whirling dervish yard sprinkler and then display this finished piece next to Jan van Eyck’s The Last Judgement - we’ve long since left the realm of pure subjectivity.

I'm being silly but I've always thought that the "taste is subjective" argument is not very compelling. Taste, if not entirely objective, at least can be measured in demographic thermoclines.
by jovial_cavalier | 2026-02-20 17:43:26+00:00
So the chefs are preparing food that has the same macros as ultraprocessed meals (I assume like tv dinners or something?) Why do they keep referring to the freshly-prepared food as "ultraprocessed"?

“Is this processed or unprocessed?” I asked.

Kozlosky smiled. “Ultra-processed,” she said. “Lots of participants can’t tell the difference.”

If the term has any meaning, you could tell very easily. Go look at a freshly fried tortilla chip, and compare it to a tostito. You know which one is which instinctively.

I thought I understood the study but now I'm not sure. I thought the idea was to take the exact same thing you'd get in a tv dinner and make it fresh, so no freeze drying, no preservatives, etc. Then if that food on its own causes the same pattern of health issues, we know it's simply a diet problem. It sounds like they replicated that effect. So they got evidence that ultraprocessing doesn't actually matter all that much?
by refulgentis | 2026-02-20 17:43:38+00:00
> If not, I think autoregressive models will still be preferred. Diffusion models become fixed very fast, they can't actually refine their outputs, so we're not talking about some kind of refinement along the lines of: initial idea -> better idea -> something actually sound.

I'm really curious about this, I'm but a simple client developer, so I don't actually grok some of the differences.

For lack of a better word, there's a "normie" position that "omg diffusion means it can edit!!111! big unlock!" -- I think that's cute but I also don't see it as intuitively correct. And I guess I don't even know why I don't see it that way. But regardless, it sounds like I'm correct there.

> If not, I think autoregressive models will still be preferred.

But here I get lost, at least so far, diffusion models seem strictly significantly faster, and on par with models with the same parameter count.

If that is the case, why would autoregressive models still be preferred?

Asking this also makes me realize I am treating "diffusion models are better" as a premise, if I'm asserting they're always faster and ~same quality...
by zzrrt | 2026-02-20 17:43:42+00:00
His felony convictions came from crimes committed in the 2016 campaign. The judge “subsequently ruled that Trump's conviction related "entirely to unofficial conduct" and "poses no danger of intrusion on the authority and function of the Executive Branch."”
(https://abcnews.com/US/judge-trumps-hush-money-case-expected...) so I don’t think it relates to SCOTUS’s immunity ruling.
by cursuve | 2026-02-20 17:43:48+00:00
Wait... So because I chose a life of not struggling to raise a child and I love my dog; I'm pathetic? What a warped-ass world-view. Maybe I just have no desire to have kids and am enjoying the life not having children warrants me - including pathetically loving my dog.
Eff off with your judgements of what I chose the path of my life to be. And, hey, godspeed on your struggle-bus. Apparently it warrants you some entitlements and status, or something...
by advisedwang | 2026-02-20 17:44:15+00:00
> I also avoid it because I'm concerned about being over-reliant on google (what if they close my account?)

Most if the "sign-in with google" accounts I have seen treat it as a shortcut to creating and logging in with an account with the primary email address of the Google account. So you can hit "reset password" and get a conventional password log-in to an account you previously made with the Google auth. If you get locked out of google, it's NBD.

Of course, this is probably not universally the case.
by righthand | 2026-02-20 17:44:21+00:00
I tried to sign up for paypal to send money collected from coworkers for a pregnancy gift. I had to sign up, enter my bank info, then verify deposits went through to use the bank account. Once I did that my account was instantly locked, then I still couldn’t use my account until I called customer support and scanned in my ID. I called them to delete the account and just bought a digital gift card online.
by C6JEsQeQa5fCjE | 2026-02-20 17:44:31+00:00
> Godwin's law? It seems dismissive of the holocaust to call ICE facilities concentration camps.

Concentration camps have a long history (you can start with the Wikipedia article on the topic). Nazi concentration camps in 30s and 40s, and the holocaust that they are linked to, happened over a relatively short time period in that history that continued even after. So Godwin's law indeed, brought about by yourself.
by sonofhans | 2026-02-20 17:44:46+00:00
Avoiding “ultra-processed foods” is like trying to stay healthy by avoiding ultra-processed uranium — excellent idea, but not enough.

Michael Pollan offers [1] guidelines: “Eat food, not too much, mostly plants.” This doesn’t talk about avoiding anything, instead chasing after known good things — food. Carrots. Steak. Wheat. Things you see in a children’s book labelled “food.” If it grows in the sun and the rain then it’s food.

Thinking in terms of “ultra-processed” still leaves you captive to industry. Buy some rice and beans and forget about it.

1. https://michaelpollan.com/books/in-defense-of-food/
by Aerroon | 2026-02-20 17:44:47+00:00
I've done the same thing with a todo app.

I find that a convenient UI becomes the most important aspect of some applications (to-do list, alarm clocks etc). Getting it to be exactly the way I like it is a benefit by itself.

I've been thinking of making a note taking app for my phone as well. The 10 or so that I've used all have had issues that made me not like them for one reason or another. Eg 16k char limit per note, no searching inside a note, broken bullet lists, long startup time etc.
by AnimalMuppet | 2026-02-20 17:44:51+00:00
Yeah... I seem to remember seeing this before, only it was by the end of 2027. So the schedule is slipping by one year per year. (In fairness, it might not have been Altman who made the previous prediction.)

But one year slip per year is not the pattern of a successful project - it's the pattern of a floundering one. You see this sometimes in projects where they still haven't figured out what the spec is for what they're trying to build. (So, do they know the spec for building a superintelligence? I'm pretty sure that no, they don't.)

What they have is evidence that they're making progress, and a completely-without-evidence idea of how much further ahead superintelligence might be, and an extrapolation based on progress continuing at the same rate. Well, the part that is the most suspect is the guess as to how far away superintelligence is. If that's wrong, the whole estimate is worthless.
by mechette | 2026-02-20 17:44:52+00:00
I learned CAD (formally, in a classroom) on Solidworks/AutoCAD, and once I understood the difference between part and part design, I didn't have major issues with FreeCAD-the concepts of good modeling are the same. I had some "okay, what does this software call this function" moments, but it wasn't nearly as bad as when I learned NX and couldn't find the constraints menu. I personally find the shortcomings in techdraw & assembly to be a much bigger problem than the UI-frankly, if I compare it to every engineering software I've used, it is solidly middle-of-the-pack in UI. The worst ones were commercial, expensive software. At least with FreeCAD if something is unbearably stupid I can go in and _fix_ it.
by AnthonyMouse | 2026-02-20 17:45:30+00:00
> As far as I can tell the US system is designed for gridlock.

At the federal level the US system was designed for gridlock on purpose, with the premise that something shouldn't be federal policy without widespread consensus, and without that consensus it should be left to the states.

The problem is really that many of the gridlock-inducing measures have been thwarted, e.g. delegation of rulemaking power from Congress to the executive and direct election of Senators to prevent state-representing Senators from voting down federal overreach. But those things weren't just there to induce gridlock, they were also the accountability measures, so without them you put corruption on rails and here we are.
by burningChrome | 2026-02-20 17:45:44+00:00
>> My main problem is defining "ultra-processed," such that I could take action on this in my life.

I asked this question Claude, ChatGPT and Gemini. None of them could give me a good explanation either. One of the themes that did come through was the agents seemed to land on the idea that if something is made purposefully to make you want more of it (in a sense crave it), then its "ultra processed" which is interesting.

I then asked them what the difference between Chipolte and Taco Bell was then. It said some of the ingredients that Chipolte uses are still designed with specific flavorings and salts which would then be considered ultra processed because the point is not to make it healthier. Its to make you want it over other things that would be considered healthy.

It was an interesting conversation, and in the end, I came to the same conclusion, its impossible to tell these days where the threshold is for something to go from processed to ultra processed is.
by ceejayoz | 2026-02-20 17:46:29+00:00
Yes, I'm aware of that approach.

It's likely tougher than it seems; the big important bits that the news will care about have to match up when checked, and anyone with high-level access to this stuff likely has a significantly sized staff who also has access to it. Paraphrasing reduces the chance of some minute detail tweak being included in the reporting at all.

You also have to actively expect and plan to do it in advance, which takes a lot of labor, time, and chances of people comparing notes and saying "what the fuck, we're being tested". You can't canary trap after the leak.
by troupo | 2026-02-20 17:46:35+00:00
> If you really can't find a use case for web components then you're living in a bubble.

There's a very tiny use-case for web components. And even there it's riddled with a huge amount of potential (and actual) footguns that "in the bubble" devs have been talking about for a decade at this point, and some which were finally acknowledged: https://w3c.github.io/webcomponents-cg/2022.html (no updates since)
by gus_massa | 2026-02-20 17:46:37+00:00
> I am looking for community feedback on how to turn this from a private methodology into a standard that many can benefit from.

Have you convinced someone else to use it?

You can copy some ideas about how to present it from a few old posts by Joel Spolsky, like:

https://www.joelonsoftware.com/2001/12/25/getting-things-don...

https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

Neither is a standard, but at least he grow a nice community around his blog.
by PaulHoule | 2026-02-20 17:46:53+00:00
I get exhausted very quickly reading stuff about AI by people who think there is some secret language of prompts or some better model or better framework which will make them successful at developing things.

I'm left with the same feeling I have when I read blogs by celebrity managers and developers like DHH or Spolsky or Graham or Atwood or Yegge, they talk as if you could learn something transferable from their experiences except... you can't. Their opinions about spaces or tabs or whether you should use static or dynamic languages are as good as anybody else's but not better!

The difference is that those guys actually made something and sold it, whereas the vibe coder almost made something.

People who make something significant with AI are going to do it because of all the others skills and attributes they have: good taste, domain knowledge, modeling, knowing what good code looks like, knowing what good user interfaces feel like, etc.

That's why I am not doomscrolling X to see what celebrity vibe coders say they are doing right now.
by mannyv | 2026-02-20 17:46:55+00:00
If you know what you're doing you don't need AWS support.

We add support when we want to do something new, like MediaTailor + SSAI. At that point we're exploring and trying to get our heads around how things work. Once it works there's no real point in support.

That said, you need to ask your account manager about (1) discounts in exchange for spend commitments, and (2) technical assistance. In general we have a talk with our AM when we're doing something new, and they rope in SMEs from the various products for us.

We're not that big, and I haven't worked for large companies, and it's always been a mystery to me why people have problems dealing with AWS. I've always found them to be super responsive and easy to get ahold of. OTOH we actually know what we're doing technically.

Google Cloud, OTOH, is super fucked up. I mean seriously, I doubt anyone there has any idea WTF is happening or how anything works anymore. There's no real cohesion, or at least there wasn't the last time I was abused by GCP.
by burnt_toast | 2026-02-20 17:46:59+00:00
I'm a big fan of FreeCAD and have been using it to design and sell 3d printed parts. The learning curve was kinda steep because FreeCAD really wants you to things the "FreeCAD way" but once I got past that I started to enjoy using it.

My only gripe with FreeCAD is that the program runs on a single CPU core as far as I can tell and it's easy to lock up the program for multiple seconds if you do too complex of an operation. This isn't usually an issue for me though.
by evanjrowley | 2026-02-20 17:47:10+00:00
Context:

https://www.npr.org/2025/06/29/nx-s1-5428227/readers-flock-t...

https://www.elle.com/uk/life-and-culture/culture/a61623350/s...

https://www.statsignificant.com/p/how-romance-romantasy-and-...

E.g., not child appropriate.
by takinola | 2026-02-20 17:47:12+00:00
A couple of suggestions:

1. Do some introspection to learn what you actually want (vs what the society around you tells you to want). Free yourself of judgement of your true ambitions and accept what you care about.

2. Follow your curiosity. Do things that interest you, not because there is any end goal or destination but just for the sheer love of it.

3. Help someone else. Focusing on yourself (which we all do) tends to depress. Focusing on other people tends to uplift. Every time you interact with someone else, ask yourself how you can make their day better and take at least one action (could be as simple as a smile, a thoughtful compliment, a gift, whatever)
by fainpul | 2026-02-20 17:47:30+00:00
> So a more productive specific thing would perhaps be indeed to strike up a "It seems to work nicely now, why isn't it default?"

That may be. It's just not what I'm interested in. I don't have the energy to fight lots of little battles to improve some minor things, when – in my perception – the end result would still *suck. I think a major rework is needed. For example, I don't think simply enabling "Make Internals" would be the best thing to do. IMO it should be always on AND the setting should be removed AND the toggle from the data view should be removed AND that would imply that it must always work bugfree so that nobody has a need to disable it. I don't think this is ever gonna happen if I start a discussion or even make a pull request.

* I'm not a hater, I'm a FreeCAD user. Out of all the offerings that exist, it's my preferred CAD tool for my private hobby use. I just wish it was better, because I see untapped potential.
by ruszki | 2026-02-20 17:47:34+00:00
You need to be cautious with the notion of “his votes go either way”. In Hungary, where I’m from, and a Trump kinda guy rules for 16 years, judges vote either way… but they vote against the government only when it doesn’t really matter for the ruling party. Either the government wants a scapegoat anyway why they cannot do something, or just simply nobody cares or even see the consequences. Like the propaganda newspapers are struck down routinely… but they don’t care because nobody, who they really care about, see the consequences of those. So judges can say happily that they are independent, yet they are not at all.

This fake independence works so well, that most Hungarians lie themselves that judiciary is free.
by dgroshev | 2026-02-20 17:47:37+00:00
It's probably impossible for FreeCAD to catch up with the industry-standard CAD systems (SOLIDWORKS, NX, Fusion) unless they somehow pour a stupendous amount of money into their geometry kernel [1].

All major CAD systems use mature geometry kernels like Parasolid [2]. Parasolid was developed for 40 years and is still in active development. This is the piece of code that enables CAD systems to do things like computing an intersection of a G3 smooth fillet with embossed text, handling all corner cases.

FreeCAD runs on OpenCASCADE [3], which is both less sophisticated today and is slower to gain new features than Parasolid, being seemingly maintained by one person [4]. FreeCAD's geometry is hard limited by what OpenCASCADE can do.

This is the main difference from Blender. Blender ultimately operates on vertices, which doesn't require nearly the same level of inherent complexity. Blender isn't bottlenecked in what it can do like FreeCAD is.

[1]: https://en.wikipedia.org/wiki/Geometric_modeling_kernel

[2]: https://en.wikipedia.org/wiki/Parasolid

[3]: https://en.wikipedia.org/wiki/Open_Cascade_Technology

[4]: https://github.com/Open-Cascade-SAS/OCCT/commits/master/
by travisd | 2026-02-20 17:47:53+00:00
I found two separate bugs in GCP products. One with gVisor where it would sometimes null-truncate large network packets (this was very hard to diagnose – why is my JSON full of null bytes?) and one where Cloud Run broke sudo sporadically (sudo in a FaaS is definitely niche, I had essentially containerized a very old application written by undergraduates).

Both times they were serious production bugs that took at least a week to resolve, though I only had the lowest tier of support package.
by SoftTalker | 2026-02-20 17:47:55+00:00
The defense for the civil engineer is that his design was in accordance with usual and customary engineering standards. If he did something unusual or new, he might be liable if that was the root cause of the failure. If he signed off on a sound design, he's probably OK.

Should work the same with software. The problem is that nobody learns that, schools don't teach it (school isn't even required to be a software developer), and there's no licencing bodies that enforce it. And, ultimately, most software failures don't cause death or injury.
by paulddraper | 2026-02-20 17:48:04+00:00
It’s not timeframe per se, but rather holding indefinitely without actual charges.

Jails are not concentration camps. Detention centers are not concentration camps. (Unless they hold indefinitely without charges.)

Seamus Culleton is welcome to leave…to his country of citizenship. He illegally overstayed his visa, and wants to stay anyway.

ICE doesn’t want people in detention centers. Generally speaking they are all welcome to leave.

This is not the same as eg FDR’s Japanese internment camps, let alone Hilter’s.
by maxwell | 2026-02-20 17:48:05+00:00
Please, they're ngmi with no fat. The unhealthy frat boy office sounds like a throwback to the early '10s. What woman would work there? They seem poised to crash and burn out.

Historical aristocracy were defined by eating meat, while their subjects ate grain. "Beef" for the Normans, "cows" raised and slaughtered by the Anglo-Saxons.
by overfeed | 2026-02-20 17:48:07+00:00
> on the contrary many would-be competitors seems to have almost given up

maybe the smaller ones; Blue Origin succeeded, and French and Chinese nu-space companies will continue to get funding for decades - national governments are capable of footing the bill of large CapEx projects. SpaceX competition is irreversibly tied to US foreign policy, and only scientific amd commercial launches are price-sensitive
by epistasis | 2026-02-20 17:48:12+00:00
Getting enough fiber has actually been my primary dietary goal for the past year. It's insanely hard! Hitting high amounts of protein with low carbs is a very very easy goal in comparison. At least when preparing your own food.

When eating out, it's practically impossible to hit goals on macronutrient categories, much less fiber goals! The best one can do is try to count calories.
by s_gourichon | 2026-02-20 17:48:25+00:00
Time to watch again this montage from the 1974 movie "Dark Star" by John Carpenter, parody of 2001 a space Odyssey.

Topic: "talking to the bomb"

https://www.youtube.com/watch?v=h73PsFKtIck (warning this is considered to spoil the movie).
by toomuchtodo | 2026-02-20 17:48:31+00:00
"Vote better next time I suppose" is the message to the electorate, because it would be impossible to return the funds to them due to diffusion.

The best you could do is perhaps model the additional per household cost (which has been done) and issue them checks from the Treasury (stimulus check style), but who is going to pay for it? The taxpayer! There is no way to incur this economic cost on the people who incurred the harm (this administration). You could potentially get the funds back from companies through higher corp taxes. Is Congress going to pass that? Certainly not. Them the breaks of electing Tariff Man. Does exactly what it says on the tin.

> ....I am a Tariff Man. When people or countries come in to raid the great wealth of our Nation, I want them to pay for the privilege of doing so. It will always be the best way to max out our economic power. We are right now taking in $billions in Tariffs. MAKE AMERICA RICH AGAIN
9:03 AM · Dec 4, 2018

https://x.com/realDonaldTrump/status/1069970500535902208 | https://archive.today/BBEmH

Historical lesson in governance failure. Can't change history, can only try to do better in the future. Onward. Let the lesson not be for naught.