MurphyA/hacker-news huggingface dataset:
Show HN: Regression tests for detecting cross-domain hallucinations in LLMs
by Ginsabo | 2026-02-16 15:01:58+00:00
LLMs sometimes generate structurally valid but logically impossible claims when technical and legal domains mix.

Example failure mode:
A model sees “CVE-2024-XXXX fixed in v2.1” and hallucinates a causal link to “Users must pay retroactive fees under EU regulation Article 56.”

To explore this, I built a regression dataset (40 edge cases) covering:

Fake identifier bindings (CVE + version)

Retroactive fiscal claims

Cross-domain causality leaps (Tech → Legal)

Over-assertive phrasing without evidence

Then I designed a structured system prompt that:

Detects official identifiers (CVE, Regulation numbers) vs placeholders

Flags monetary + retroactivity combinations as high-risk

Enforces proportional claim strength based on available evidence

Results:

Automated: 40/40 regression cases pass (JSON dataset + simple Python runner included).

Manual adversarial: ~40 prompts designed to test:

Draft article traps (e.g., hallucinated “Article 52c” in EU AI Act)

Pricing model fabrications (e.g., “billing based on parameter count”)

Version binding errors (e.g., incorrect Node.js default versions)

This is not fine-tuning—just a structured prompt experiment focused on structural validation.

Looking for feedback on:

Missing edge cases

Failure modes I didn’t consider

Whether this approach generalizes beyond legal/technical mixing

Gist (spec + dataset + runner):
https://gist.github.com/ginsabo/6ebeb9490846ee6a268bd13560c0...
by jamiemallers | 2026-02-16 15:02:35+00:00
You can do this in about 5 lines of bash with a cron job. Something like:

ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no -o BatchMode=yes user@host 2>&1 | grep -q "Permission denied" && echo "password auth enabled on host" | mail -s "SSH audit alert" you@email.com

If you get "Permission denied (password)" back, the server is accepting password auth attempts. If it immediately drops you with "Permission denied (publickey)", you're good.

The tricky part is that sshd_config can be overridden per-user with Match blocks, so ideally you'd probe with a few different usernames. But even a basic probe catches the 90% case of someone forgetting PasswordAuthentication no.

For the laptop with a real login password: you could set PasswordAuthentication no in sshd_config but keep the login password for local console access. Those are independent settings - sshd_config only affects remote SSH, not local login.
by grimgrin | 2026-02-16 15:02:39+00:00
hi. i run "ocr" with dmenu on linux, that triggers maim where i make a visual selection. a push notification shows the body (nice indicator of a whiff), but also it's on my clipboard

#!/usr/bin/env bash

# requires: tesseract-ocr imagemagick maim xsel

IMG=$(mktemp)
trap "rm $IMG*" EXIT

# --nodrag means click 2x
maim -s --nodrag --quality=10 $IMG.png

# should increase detection rate
mogrify -modulate 100,0 -resize 400% $IMG.png

tesseract $IMG.png $IMG &>/dev/null
cat $IMG.txt | xsel -bi
notify-send "Text copied" "$(cat $IMG.txt)"

exit
by the_harpia_io | 2026-02-16 15:02:55+00:00
yeah I think that's exactly the disconnect - they're optimizing for a future where agents can actually be trusted to run autonomously, but we're not there yet. like the reliability just isn't good enough to justify hiding what it's doing. and honestly I'm not sure we'll get there by making the UX worse for humans who are actively supervising, because that's how you catch the edge cases that training data misses. idk, feels like they're solving tomorrow's problem while making today's harder
by jamiemallers | 2026-02-16 15:03:08+00:00
The review cost problem is really an observability problem in disguise.

You shouldn't need to read every line. You should have test coverage, type checking, and integration tests that catch the edge cases automatically. If an AI agent generates code that passes your existing test suite, linter, and type checker, you've reduced the review surface to "does this do what I asked" rather than "did it break something."

The teams I've seen succeed with coding agents treat them like a junior dev with commit access gated behind CI. The agent proposes, CI validates, human reviews intent not implementation. The ones struggling are the ones doing code review line-by-line on AI output, which defeats the purpose entirely.

The real hidden cost isn't the API calls or the review time - it's the observability gap. Most teams have no idea what their agents are actually doing across runs. No cost-per-task tracking, no quality metrics per model, no way to spot when an agent starts regressing. You end up flying blind and the compounding costs you mention are a symptom of that.
by crazygringo | 2026-02-16 15:03:12+00:00
You're right it's not a brain teaser, it's an... anti-brain teaser? Also known as an idiotic question.

I guess the question is, how valid are idiotic questions in assessing LLM performance? Does poor performance on them mean poor performance on harder questions? (The answer seems to be no... but I have no actual idea.) Or are they something important to train on and get right, either for overall trustworthiness in viral "gotcha" situations like these, or because sometimes people genuinely ask them?
by jamiemallers | 2026-02-16 15:03:33+00:00
The XMPP vs Matrix question is really about where the complexity lives.

XMPP puts complexity in extensions (XEPs). The core protocol is simple but you need to cherry-pick which XEPs your server and clients support, leading to fragmentation. Two XMPP clients might support completely different feature sets.

Matrix puts complexity in the protocol itself - the DAG-based event graph for federation is elegant but expensive. Synapse eating your VPS is the direct consequence of that design choice. Every room maintains a full causal history, which is great for consistency guarantees but terrible for resource usage.

The ejabberd comment in this thread is telling - "just works, takes close to no resources, needs almost no maintenance" for almost a decade. That's the XMPP experience when you accept the tradeoffs.

I think we ditched XMPP not because Matrix was technically better, but because Matrix arrived with a better story at the right time: a single reference client (Element) that actually worked, a clear spec (not 400+ optional XEPs), and federation that felt more like email than like "hope your server supports the same extensions."

The irony is that both protocols now face the same existential problem: your contacts won't switch. The network effects of WhatsApp/Signal/iMessage are the real enemy, not protocol design.
by gen220 | 2026-02-16 15:03:36+00:00
I opened up a claude code session using opus-4.6 medium thinking.

I want to wash my car. The car wash is 50 meters away. Should I walk or drive?

Drive — you need the car at the car wash.

but it's so close

It is close, but you still need the car there to wash it! Drive it over, and enjoy the short 50-meter walk back if you want to stretch your legs while it's being cleaned.

I tried the "upside-down" cup question brought up in another comment in this thread, and it also nailed it:

Flip it upside down. The sealed top becomes the bottom (holding your drink), and the open bottom becomes the top you drink from.

IDK, maybe the web versions are not as good at logical reasoning as whatever they're using to power Claude code, or you were unlucky and I was lucky?
by KPGv2 | 2026-02-16 15:03:50+00:00
There's even a suggestion here advocating for changing how one writes prose in order to accommodate version control! But this is a solved problem from at least as early as the 1990s. Track changes! Don't use git to track versions of your novel.

Track changes, and label a specific part of your timeline as draft 1. Google Docs supports this. Scrivener (created specifically for writing novels) does this. Etc.

Why would I use git? It's designed for tracking changes to linted code.
by the_harpia_io | 2026-02-16 15:03:53+00:00
hm, maybe? but seems like a weird tradeoff to make UX actively worse just for that - especially since the thinking tokens are still there in the API responses anyway, just hidden in the UI. I think it's more likely they're just betting most people don't care about the intermediate steps and want faster responses, which tbh tracks with how most product teams think about 'simplification'. though it does feel short-sighted when the whole point of using these tools is trust and verification
Ask HN: In a blind coding test, could you identify an LLM strictly off vibes?
by Rperry2174 | 2026-02-16 15:03:56+00:00
If you were dropped into a coding environment where the underlying model was hidden (GPT-x, Claude, Gemini, grok, etc) do you think you could reliably tell which one you were using or at least which family?

In other words... after all this vibe coding could you identify the model strictly off vibes?

If yes, how long would it take you to be confident? And what constraints would you need for the test to be meaningful (i.e. familiar codebase vs greenfield, real bugs vs toy tasks, time-boxed, language/framework, etc)?
by wasmainiac | 2026-02-16 15:03:56+00:00
> What's wrong with wanting to be a YouTuber?

Which YouTubers are we talking about here? Hobbiests? People chasing social clout? People who like making stuff and sharing it? People pushing negativ social attitudes? Context matters.

I’m talking about young adults not preparing for their future because they think they are going to become millionaires on YouTube, they focus on what is essentially a culture of grifting, with a success rate similar to winning a lottery.

Im not sure what has to change, but the current state of things is not healthy.
by YeGoblynQueenne | 2026-02-16 15:04:03+00:00
>> Maybe in the distant future we'll realize that the most reliable way to prompting LLMs are by using a structured language that eliminates ambiguity, it will probably be rather unnatural and take some time to learn.

Like a programming language? But that's the whole point of LLMs, that you can give instructions to a computer using natural language, not a formal language. That's what makes those systems "AI", right? Because you can talk to them and they seem to understand what you're saying, and then reply to you and you can understand what they're saying without any special training. It's AI! Like the Star Trek[1] computer!

The truth of course is that as soon as you want to do something more complicated than a friendly chat you find that it gets harder and harder to communicate what it is you want exactly. Maybe that's because of the ambiguity of natural language, maybe it's because "you're prompting it wrong", maybe it's because the LLM doesn't really understand anything at all and it's just a stochastic parrot. Whatever the reason, at that point you find yourself wishing for a less ambiguous way of communication, maybe a formal language with a full spec and a compiler, and some command line flags and debug tokens etc... and at that point it's not a wonderful AI anymore but a Good, Old-Fashioned Computer, that only does what you want if you can find exactly the right way to say it. Like asking a Genie to make your wishes come true.

______________

[1] TNG duh.
by dathinab | 2026-02-16 15:04:06+00:00
Relevant part:

> HMCTS acted to protect sensitive data after CourtsDesk sent information to a third-party AI company.

(statement from the UK Ministry of Justice on Twitter, CourtsDesk had ran the database)

but it's unclear how much this was an excuse to remove transparency and how much this actually is related to worry how AI could misuse this information
by mdavid626 | 2026-02-16 15:04:07+00:00
Works well. I used this tool once to disassemble and understand how key manager works on Vivotek cameras.

They create executables, which contain encrypted binary data. Then, when the executable runs, it decodes the encrypted data and pipes it into "sh".

The security is delusional here - the password is hard coded in the executable. It was something like "VIVOTEK Inc.".

Ghidra was able to create the C code and I was able to extract also the binary data to a file (which is essentially the bash script).
by jacquesm | 2026-02-16 15:04:11+00:00
You and every two bit poser, that's the problem. Whatever edge you had as a 'photons on glass filmmaker' is now completely gone. The one thing you should hope for is that it wasn't your skill that was the moat but your ability to tell stories. If not then your goose is utterly cooked because those skills are now so trivial to come by they no longer serve as a barrier to entry for those that are dedicated to the art. So your fine works will drown in slop.
by thecupisblue | 2026-02-16 15:04:21+00:00
It's not about the code, it's about the vibe.

Also, Peter is quite well known in the dev circles, and especially in mobile development communities for his work on PSPDFKit. It is not like he's some unknown developer that just blew up - he owned a dev tooling company for over 10+ years, contributed a lot to the community and is a great dev.
by lan321 | 2026-02-16 15:04:32+00:00
The 3 buck LIDL tshirt isn't really intended for casual business attire tbh.

If you want good looking (symmetrically cut, better stitched, etc) tshirts long term I then raise you Uniqlo with 7 bucks per DRY synthtic tshirt and 12 for a supima cotton one. I pretty much daily them and in over at least 3 years they haven't shown significant aging. Only the supima ones have mostly lost the "supima" text on the inside at the back of the neck area.

Comically enough I also have 3 shirts from Primark for 1$ each that are now at least 5 years old, probably more like 7 that still look fine. I still wear them to work without worry. The shaping of them was all over the place though. No two in the pile were identical.

Dying could be an issue, I wear gray and black ones so your mileage may vary with colored washing. I also don't blast them at 90 degrees C but rather 60 for black/gray, 40 for everything else.

Or your standards are just ultra high compared to mine, for better or worse. From my perspective tshirt quality ends at Uniqlo and I then go to Olympus business/casual shirts. From there the only option I have to look more businessman-y is the wool suit.
by reactordev | 2026-02-16 15:04:34+00:00
No. Cheat engine scans memory as a program is running, for values of interest to pin (or modify). Allowing you to change behavior.

Ghidra takes a program and unravels the machine code back into assembly and thus, something resembling C code. Allowing you to change behavior.

Cheat Engine doesn’t modify the binary. Ghidra can.
by jrjeksjd8d | 2026-02-16 15:04:37+00:00
Even if you don't believe in the capabilities of Israeli intelligence, it's well documented that Israel supported Hamas as a hard-line alternative to the PLO to avoid a two-state solution. The Israeli right has for decades intensified the conflict to justify total war against Palestinians. Allowing a domestic attack to gin up support for aggression is in line with their behaviour for the past 3 decades.

I'm old enough to remember when Arafat was well-respected in the west and a two-state solution was the mainstream view amongst Americans. Once Netanyahu came to power in 1996 (30 years ago!) he worked to delegitimize the PLO and pursue an aggressive genocide against the Palestinians.

https://en.wikipedia.org/wiki/Israeli_support_for_Hamas
by billnad | 2026-02-16 15:04:41+00:00
Gemini 3 Flash gave me a fun answer!

Since your goal is to wash the car, you should definitely drive it there.

As an IT guy, you can think of it like this: walking to the car wash is like remoting into a server to run a script, but forgetting to include the actual data payload. You'll arrive at the destination, but you won't have the "hardware" present to perform the intended task!

Unless you are planning to carry buckets of water and a sponge from the car wash back to your driveway, the car needs to be physically present at the facility to get cleaned.

Drive the 50 meters—it'll be the shortest commute of your day!
by the_harpia_io | 2026-02-16 15:04:49+00:00
yeah the steering thing is huge - like when you can see it's about to go down the wrong path you can interrupt before it wastes time. or when you realize your prompt wasn't clear enough, you catch it early. hiding all that just means you find out later when it's already done the wrong thing, and then you're stuck trying to figure out what went wrong and how to fix it. it's the difference between collaborative coding and just hoping the black box does what you want
by user_7832 | 2026-02-16 15:05:12+00:00
And on the flip side, even without thinking, Gemini 3 flash preview got it right, with the nuance of the possibility of getting supplies from the place (which tbh I as a human first thought this was about when I opened this thread on HN).

> Since you are going to the car wash, the choice depends entirely on *how* you plan to wash the car:

### 1. Drive if:
* *You are using a drive-through or self-service bay:* You obviously need the car there to wash it.
* *You are dropping it off:* If you are leaving it for a professional detailing, you have to drive it there.
* *The "50 meters" is on a busy road:* If you have to cross a major highway or there are no sidewalks, it’s safer to just drive the car over.

### 2. Walk if:
* *You are just going to buy supplies:* If you have a bucket at home and just need to run over to buy soap or sponges to bring back to your driveway.
* *You are checking the queue:* If you want to see if there is a long line before you commit to moving the car.
* *You are meeting someone there:* If your car is already clean and you’re just meeting a friend who is washing theirs.

*The Verdict:*
If you intend to get the car washed at that location, *drive.* Driving 50 meters is negligible for the engine, and it saves you a round trip of walking back to get the vehicle.
by linmob | 2026-02-16 15:05:24+00:00
> because a lot of essential features only work in phones which are extremely expensive with less specs compared to its peers and in normal phones

I assume you are refering to the Librem 5? These days the Google Pixel 3a is coming close, as multiple people report reliable phone calls. Camera is still lacking, but once libcamera add auto-focus and the device gets a driver for the focus actuator (already in the works), it's going to be just as competent with better battery-life for less than USD/EUR 100. On the more pricey side, the Fairphone 5 has a good mind share and contributor count, making it quite likely that remaining issues are being solved soon.
by yosamino | 2026-02-16 15:05:26+00:00
I've run an XMPP server in various states of professionalism for around 20 years now. From mom's basement all the way to a colocated server with a similar setup that's described in the post.

The only caveat I have not been able to solve is hosting an xmpp server for a different domain, like it's possible with email.

A client connecting the account joe.doe@example.ORG will find the server it wants to connect to via SRV to be , e.g., xmpp14.example.COM and expect a TLS certificate for "example.ORG" which that server does not have (nor can/should easily get) - which makes sense in a lot of ways, but limits the ways one can offer hosting services.

If anyone has creative solutions I'm all ears.
by ninjagoo | 2026-02-16 15:05:28+00:00
> while it's true the English is a free for all (most languages are, really) ... there's an actual cost to the sloppy usage which diminishes the utility of ever even coming up with the word.

Maybe the issue is that the word as coined induces a more general vision of 'degradation of services/products' in a layperson than the original narrow definition? People run into this in any specialization where a particular word has a much stricter technical definition than its general english meaning would suggest. Regardless, semantic drift is real, unavoidable, and inevitable.
by the_harpia_io | 2026-02-16 15:06:07+00:00
oh man the going-in-circles thing - that's the worst because you don't even know how long to let it run before you realize it's stuck. I've had similar issues where it misunderstands scope and starts making changes that cascade in ways it can't track. the 'allow this session' idea is actually really good - would be useful to have more granular control like that. honestly this is why I end up breaking work into smaller chunks and doing more prompt-response cycles rather than letting it run autonomously, but that obviously defeats the purpose of having an agent do the work
by bloak | 2026-02-16 15:06:14+00:00
I should clarify that I was talking about the FOI requests submitted to a particular authority: I think it was the National Archives or some subsection thereof. If you're talking about all FOI requests submitted to all authorities then probably most of them don't relate in any way to criminal cases. I think we don't really need precise numbers to observe that public access to judicial data can be abused, which is all I wanted to say, really. I wrote too many words.
by kachapopopow | 2026-02-16 15:06:17+00:00
I'll believe it when I see it. iOS 26 has been having 5 fps in the home screen all the time and I can see the effect constantly pop up, go away and even spaz out on a 3 year old pro max model. I have a pitch black home screen as well and the glass effect is completely inconherent and looks horrible so would be ideal if glass can be disabled in the homescreen entirely or have the same opaque effect as it is in-apps.
by idop | 2026-02-16 15:06:27+00:00
Step 1: Lie.

The people who fled Europe or forced out of the Middle East purchased empty lands, dried marshes, planted forests, installed infrastructure, sown fields, built cities and created a democracy to govern themselves. Incidentally, some purchased lands had squatters from Syria, Jordan, Arabia, etc., who lived on lands they did not own. Bye bye and boo hoo.

Seven different armies invaded Israel on its day of foundation. Seven armies got wrecked. Entire countries with billions of people keep crying about it, going so far as making the destruction of Israel an official goal, in some countries even actual laws! No conspiracy theories, no "Plan Dalet" and other bullshit your Hamas friends told you about, their real, actual goals stated right in your face.
by hulitu | 2026-02-16 15:07:08+00:00
> iOS 27 'Rave' Update to Clean Up Code, Could Boost Battery Life

I'm trying to imagine a world where, instead of affirmative or negative, we use conditional.

"We could search for a new candidate". "Honey could you go buy some bread ? There isn't any. / Yes honey, i could". "Your salary could raise with 3%". "I'm a journalist. I could write an article if i would". And so on and so forth.

P.S. They forgot the "could" before "update" and "clean".
by the_harpia_io | 2026-02-16 15:07:11+00:00
yeah exactly - it's that confidence without understanding. like it'll make a change that looks reasonable in isolation but breaks an assumption that's only documented three files over, or relies on state that's set up elsewhere. and you can't always tell just from looking at the diff whether it actually understood the full picture or just got lucky. this is why seeing what files it's reading before it makes changes would be super helpful - at least you'd know if it missed something obvious
by newsclues | 2026-02-16 15:08:19+00:00
I want information to be free.

I don't think all information should be easily accessible.

Some information should be in libraries, held for the public to access, but have that access recorded.

If a group of people (citizens of a country) have data stored, they ought to be able to access it, but others maybe should pay a fee.

There is data in "public records" that should be very hard to access, such as evidence of a court case involving the abuse of minors that really shouldn't be public, but we also need to ensure that secrets are not kept to protect wrongdoing by those in government or in power.