Hallucinated reference citations - Page 1
Forum section: electronics / general

author: TimFox
author_rank: Super Contributor
timestamp: August 04, 2025, 09:36:17 pm
content: By now, we have all heard of lazy lawyers using AI to search for legal citations and precedents for their briefs, only to get in trouble when the AI hallucinates a fictional citation in proper format, and the legal staff didn't bother to check.If I were writing a script for a Perry Mason episode, that would be good enough, but judges in both UK and US are not amused.Similarly, recent articles about NIH and FDA expediting review of medical research and applications found that the AI "confidently hallucinated" non-existent technical papers in the scientific literature.https://gizmodo.com/fdas-new-drug-approval-ai-is-generating-fake-studies-report-2000633153How difficult can it be to program an AI thingy to check if the formal citation it spews out actually exists in the open literature?

author: radiolistener
author_rank: Super Contributor
timestamp: August 04, 2025, 10:00:07 pm
content: I’ve asked myself that same question more than once. On the surface, it seems almost trivial for an AI system to verify whether a citation actually exists - cross-checking against open-access databases, DOI registries, or PubMed APIs, for example.And yet, many AI tools still "confidently hallucinate" references that are plausible in format and context, but entirely fictional. It makes one wonder whether this omission is merely a matter of technical oversight or if it’s a conscious design goal.But now that these systems are being used in legal and biomedical contexts, where false information can have serious consequences, it's no longer a minor flaw - it’s a systemic risk. Just like we wouldn’t accept a calculator that "sometimes makes up numbers", we shouldn’t accept language models that fabricate references while sounding authoritative.Perhaps it’s time citation validation becomes a standard requirement, not an optional post-processing step.

author: TimFox
author_rank: Super Contributor
timestamp: August 04, 2025, 10:19:07 pm
content: Reporting a judicial threat in UK of criminal charges to lawyers who cite hallucinated documents:https://www.ibanet.org/Technology-UK-judge-warns-lawyers-about-risks-of-AI-use-in-courtOf course, there are legitimate database search engines for lawyers to search the legal literature:Free ones (US), seehttps://guides.temple.edu/c.php?g=1046977&p=7597201The standard US search engine is Lexis.Seehttps://guides.temple.edu/c.php?g=1046977&p=7597200

author: Whales
author_rank: Super Contributor
timestamp: August 04, 2025, 10:49:17 pm
content: Quote from: TimFox on August 04, 2025, 09:36:17 pmHard problem. You cite something because you think it agrees, disagrees or is relevant to a point you are discussing; this requires interpretation skills and critical thinking. I am not aware of any algorithms that can do that reliably.Eg imagine you have a citation that correctly names a paper and date, but claims it said something it that it does not.

author: TimFox
author_rank: Super Contributor
timestamp: August 04, 2025, 10:56:02 pm
content: Quote from: Whales on August 04, 2025, 10:49:17 pmYour example is a related problem that has also been reported: citing an existing technical paper, but the paper does not say what the AI declares that it says.I don't think an AI is capable of that kind of analysis of the text, just that it contains keywords and phrases.My question is about fictional references, thatdo not exist at all, that the AI hallucinates because the grammatical structure and format agree with normal citation practices.Surely the AI could be programmed to check the relevant databases to see if the reference exists?

author: Whales
author_rank: Super Contributor
timestamp: August 04, 2025, 11:30:59 pm
content: Quote from: TimFox on August 04, 2025, 10:56:02 pmThat would be traditional programming, not an LLM, doing such a jobFWIW I consider "referenced paper actually exists and metadata is correct but its contents have been misrepresented" to be functionally no different to "referenced paper does not exist and its contents have been made up", other than the former being harder to spot than the latter. Using either type of reference is a way of making your own writing falsely appear more authoritative. The distinction is illusionary.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 04, 2025, 11:39:06 pm
content: Quote from: Whales on August 04, 2025, 11:30:59 pmOf course both outcomes are bad.I think Tim's point is that while actually verifying the output of an LLM as to its grasp of meaning of a writing is difficult or impossible, it should be possible to ascertainwhether or not a certain cited paper actually exists, using standard search techniques without resorting to "AI". That's a much simpler problem to solve, one would think.

author: BrianHG
author_rank: Super Contributor
timestamp: August 05, 2025, 01:13:42 am
content: Quote from: Analog Kid on August 04, 2025, 11:39:06 pmUnfortunately, when ChatGPT once fed me reference to a questionable imaginary reference study doc, I googled the doc. Guess what, in the first listing, Google's AI created an imaginary an different imaginary doc replicating the study's results as well as 2 other questionable web links to weird website name with their own confirming docs. Not just docs, but an entire imaginary magazine publication.The entire system has already been poisoned.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 05, 2025, 01:25:22 am
content: Quote from: BrianHG on August 05, 2025, 01:13:42 amOK, trying to disentangle your reply here, which was a little mangled (I think you typed "an" instead of "and"?)So Google's "AI" showed a bogus ("imaginary") doc replicating "the study's results", which I assume means an actual, existing document, correct? So two documents there.If one doesnotuse "AI" in this situation but instead uses standard search techniques to see what actually exists, the first doc in the example above would not be found while the second one would, validating its existence.Yes, the entire system may well be poisoned by now. But we still should be able to somehow identify those documents which actually exist as opposed to those where "AI" is just making shit up.

author: BrianHG
author_rank: Super Contributor
timestamp: August 05, 2025, 02:00:10 am
content: See attached image: (Yes, on some searches, google now uses AI to give you an answer, see red arrow.)3 months ago, this was labeled as a fact and there was a fake magazine site with an artificially generated story containing the facts.I just tried this search again tonight and it appears the AI has been corrected and it at least says it was wrong in the past. Or it at least claims there was inaccurate info when it was the original source at the time.At the time, the 'TikTok' listing was never found, but, a few complete articles seem to have now disappeared.

author: Zucca
author_rank: Unknown
timestamp: August 05, 2025, 04:09:44 am
content: Somehow related,*waste of time alert*the AI will spam our future and the one of our next generations.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 05, 2025, 04:29:46 am
content: Quote from: Zucca on August 05, 2025, 04:09:44 amWell, hell's bells, apart from needing a banana-to-Molex adapter, what's the problem?

author: radiolistener
author_rank: Super Contributor
timestamp: August 05, 2025, 06:10:14 am
content: Quote from: BrianHG on August 05, 2025, 01:13:42 amYes, that’s a real and growing problem. Recently someone asked whether power supplies like the DP832 and SPD3303X are any good, mentioning noise issues. I googled both models and right at the top, Google's AI confidently stated that users complain about SMPS noise in both units... It seemed believable... I fell for it, because for years I trusted Google to show real, reliable links. Not anymore.It’s worse than hallucinated studies - we’re entering an era where even confirming links, supporting articles, and "independent" reviews can all be AI-generated fiction. The whole digital ecosystem is now polluted. At this point, I’m starting to think that unless it’s ink on paper with a real stamp and a human signature, it can’t be trusted as proof anymore...

author: magic
author_rank: Super Contributor
timestamp: August 05, 2025, 06:34:57 am
content: Quote from: Whales on August 04, 2025, 11:30:59 pmThat's the problem, they would need to know the relevant databases for each domain that their LLM might hallucinate about and write software to query them, keep the software up to date with changing APIs, and so on.It's easier to say that one day "GAI" and "AI singularity" will solve it all and keep tweaking the LLM forever. That's the business they are inAlso, where is ROI in that? Should they advertise that their legal citations are fact-checked now? What if it ends up being an instance of "the case exists, but is different than the LLM wrote it is"? Who takes responsibility for that? Who will pay for the extra fact-checking if the LLM vendor doesn't take this responsibility? Are they technologically capable of taking it?

author: Siwastaja
author_rank: Super Contributor
timestamp: August 05, 2025, 06:56:33 am
content: Quote from: Whales on August 04, 2025, 11:30:59 pmExactly, and that's what they already do, I believe. They probably already spawn some kind of math module to do simple calculations; just some two years ago ChatGPT, while its text generation capabilities and "knowledge" was on pretty similar level than today, was unable to do simple calculations like 123 * 456 correctly, instead hallucinating random numbers. Today raw number crunching usually just works, and it only breaks down when doing the math is distributed over separate steps during which it can lose the track.Similarly, it is now perfectly clear that an LLM totally sucks in playing Chess and loses to any beginner-level 1980's chess algorithm.But an LLM is well capable of figuring out what are popular and well-regarded open source chess algorithm implementations, and how to build and run them. So why won't it do exactly that, and then act as a middleman feeding data to the algorithm? Checking theexistenceof citations using traditional algorithms should be even simpler.I believe exactlythis, combining "traditional programming" with LLMs, is going to be a much more significant improvement than just throwing even more training data and even more neurons at the problem.

author: tggzzz
author_rank: Super Contributor
timestamp: August 05, 2025, 07:41:44 am
content: Quote from: BrianHG on August 05, 2025, 01:13:42 amNow consider the consequences of slopsquatting:https://www.csoonline.com/article/3961304/ai-hallucinations-lead-to-new-cyber-threat-slopsquatting.htmlhttps://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_chain/Then add vibe coding, either deliberate or accidental.

author: TimFox
author_rank: Super Contributor
timestamp: August 05, 2025, 02:15:29 pm
content: Quote from: magic on August 05, 2025, 06:34:57 amI would think that the AI should be able to "know" if the citation it generated came from somewhere external, or if it produced the citation itself.The contents of the former reference might be interpreted incorrectly, and that's probably not possible to be determined by the AI itself.However, the process by which the AI specified that reference should be somehow "known" to the AI and capable of being flagged.Besides legal and scientific, another recent hallucination scandal involved a free-lance author tasked to produce a list of books for summer reading.He used AI, which came up with several real books, and a host of hallucinations with real authors but non-existent titles.The authors' names came from the usual process, but the AI had to generate the fake titles itself.https://www.npr.org/2025/05/20/nx-s1-5405022/fake-summer-reading-list-ai

author: tggzzz
author_rank: Super Contributor
timestamp: August 05, 2025, 02:48:12 pm
content: Quote from: TimFox on August 05, 2025, 02:15:29 pmLLMs are the weights in and connections between their neurons. As such they are akin to a hash of their training material.Understanding (let alone predicting) how an LLM reaches its output is beyond current understanding, and is, ahem, an active research topic.So no, unlike a human or rule-based AI system, you can't get an LLM to indicate how/why it generated the output.

author: magic
author_rank: Super Contributor
timestamp: August 05, 2025, 03:01:53 pm
content: Quote from: Siwastaja on August 05, 2025, 06:56:33 am1. upload a piece of malware to GitHub2. add README which says "this algorithm plays the game of FooBangBiz"3. wait a month4. nobody cares about this repository5. "OK Google, let's play a game of FooBangBiz"

author: Siwastaja
author_rank: Super Contributor
timestamp: August 05, 2025, 03:03:00 pm
content: Quote from: tggzzz on August 05, 2025, 02:48:12 pmHumans have similar behavior. For example, I have mentioned numerous times about a research paper which shows how commercial off-the-shelf lithium iron phosphate (LFP) type cells were more prone to catastrophic destruction in thermal runaway than their lithium cobalt oxide counterparts. But: 1) I can't remember authors / title so I can't refer to it, 2) I probably remember some details incorrectly. So I am just giving my "best effort" expertise, prone to hallucinating something, and too lazy to try to do the hard work of finding the references again.The last point, laziness, quite interestingly should not affect AIs. So clearly they need better googling capabilities to "reconstruct" the sources their claims originally are based on, and at the same time, validate they "remember" correctly.

author: TimFox
author_rank: Super Contributor
timestamp: August 05, 2025, 03:59:29 pm
content: Quote from: Siwastaja on August 05, 2025, 03:03:00 pm45 years ago, when writing my physics thesis, I could go to the university library, find citations the hard way, and chase them down and Xerox them.Hardcopy serial sources includedPhysics AbstractsandScience Citation Index.For the latter, I found that if could locate an early seminal paper on the topic, searching for later articles that cited it was an efficient strategy.At the end of my quest, early computerized search engines (analogous to Google for scientific journals) appeared, but required trained personnel to run them.Tedious work, that modern automated systems should have superseded, but in a reliable manner.

author: tggzzz
author_rank: Super Contributor
timestamp: August 05, 2025, 04:10:05 pm
content: Gurgle's original page rank algorithm was directly analogous to the SCI, and was one reason I instantly started using google.edu in preference to other engines.Page rank is now recognizably different

author: tggzzz
author_rank: Super Contributor
timestamp: August 05, 2025, 04:15:32 pm
content: Quote from: Siwastaja on August 05, 2025, 03:03:00 pmIndeed.Why are people satisfied because LLMs imitatebadaspects of human behaviour?Cory Doctorow has some interesting concepts, noting that there are undiscussed and subtleeconomicreasons LLMs are being pushed down our throats too hard.

author: BrianHG
author_rank: Super Contributor
timestamp: August 05, 2025, 07:22:05 pm
content: Thanks to this stuff, from what I've been seeing in the current VC investment space, the 'AI' is beginning to disappear from the top 'need to have'.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 05, 2025, 07:25:45 pm
content: Quote from: BrianHG on August 05, 2025, 07:22:05 pmIf that's true, it's great news.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 06, 2025, 07:46:23 am
content: Quote from: tggzzz on August 05, 2025, 04:15:32 pmBecause people would be terrified to see an artificial intelligence surpassing them, and behaving in a super-human way, really; being better than them."True AI" is clearly something people do not want; they are scared about it, and for obvious and very good reasons.We clearly still want just funny chatbots we enjoy playing with. And because nothing beats having fun and playing games on employer's time, "professional use of AI" is a big thing.And like all companies who want to make profit, AI companies deliver what people want.

author: BrianHG
author_rank: Super Contributor
timestamp: August 06, 2025, 08:48:44 pm
content: And now I just heard they are planning to use LLMs as therapists...Early tests show that the LLM recommends suicide and to commit crimes or kill others.

author: Whales
author_rank: Super Contributor
timestamp: August 06, 2025, 09:11:19 pm
content: The big LLM-as-a-service companies are billions of dollars in debt and their earnings are abysmally small.https://www.wheresyoured.at/the-haters-gui/QuoteWhen you're trying to hide a failing business model you drop all ethics and try to sell your product to do everything & anything.Medical advice. Girlfriends. Essay writing. Employee replacements. Legal advice. Legal/medical notetaking and summarisation.It can't do any of that without causing notable harms. It's not better than what we do already. But you have to keep the illusion up or people will realise you're no longer an investment, instead you're a disaster. Maybe, somehow, you'll 10x the income without incurring more costs before the bubble pops

author: bdunham7
author_rank: Super Contributor
timestamp: August 06, 2025, 09:39:10 pm
content: Quote from: Whales on August 04, 2025, 11:30:59 pmIn a legal context, the former has been a common practice for a long time and rarely gets the offending lawyer in any trouble unless the misrepresentation is obvious and egregious. Sometimes they'll win their case or their reasoning will show up in a judicial finding. The latter practice, the entirely fabricated citation, also has happened pre-AI and is typically a shortcut by the incompetent and/or extremely lazy (often writing the brief the night before the deadline) in low-level cases where they don't expect to be found out. Those cases get referred to bar disciplinary boards, as they should. Any attorney further up the food chain has absolutely no excuse for letting stuff like that slide into their work.What's new is that the latter is happening because of mere carelessness, not deliberate and knowing charlatanry.

author: TimFox
author_rank: Super Contributor
timestamp: August 06, 2025, 09:43:35 pm
content: I agree that both instances suck (real paper misinterpreted or non-existent reference confidently hallucinated).My original point is that it should be straightforward to program the gadget to recognize when the reference was made up by the AI, although one should not expect an AI to understand the contents of a real paper.The process by which the reference was generated should be "known" to the AI system., where it was done.

author: tggzzz
author_rank: Super Contributor
timestamp: August 06, 2025, 10:58:03 pm
content: Quote from: TimFox on August 06, 2025, 09:43:35 pmThey are notprogrammed; they aretrained. It is vital that is understood.Unfortunately nobody canpredictwhat training they have internalised, and the LLMs cannotexplainit afterwards.Hence medical diagnosis are made on the basis of thefonton x-ray images (last year), tanks/cars are distinguished based on whether the sun was shining (early 1980s), and road signs can be changed from "STOP" to "45mph" by changing one pixel.And if you train an LLM (cough Grok, cough) on Twatter/X material, it starts spouting far-right lies and propaganda.

author: sleemanj
author_rank: Super Contributor
timestamp: August 06, 2025, 11:13:10 pm
content: AI development continues at pace.As much as certain people like to harp on about how LLMs are just statistical machines piecing words togethor, the reality is that current RLM (Reasoning Language Models) are much "smarter" than that, further just recently a team from Singapore has developed what they are calling an HRM - Hierarchical Reasoning Model which on (certain) reasoning tasks appears to pretty much handily beat the others with less demand on resources and far fewer parameters.Hallucination is a big problem in the current lot of AI chatbots, I don't think it will be too far away before it can be surmounted however.

author: TimFox
author_rank: Super Contributor
timestamp: August 06, 2025, 11:14:04 pm
content: Quote from: tggzzz on August 06, 2025, 10:58:03 pmYes, the information that they can work with is due to training.However, the software itself has been programmed: I'm referring to the process.

author: magic
author_rank: Super Contributor
timestamp: August 07, 2025, 05:19:19 am
content: If they could, they would.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 07:16:30 am
content: Quote from: TimFox on August 06, 2025, 11:14:04 pmBy "software" are you referring to the LLM or something else?LLMs are absolutely nothing like the AI expert systems of the 1970s and 1980s, which were based on hand-crafted rules derived from knowledge elicited from experts in the field.Roughly speaking, the LLM programming consists of populating a vast lookup table with magic numbers during training, and using the magic numbers to traverse the table during operation. The magic numbers are the training.Nobody understands the relationship between the training, the magic numbers, and predict the endpoint of the traversal. That is, to use the code phrase, "an area of active research".

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 02:20:33 pm
content: I'm referring to the LLMs that confidently hallucinate.Way back at the top there is software that runs the complicated contraption underneath.Naively, I would assume that the operating software is capable of following the process.Since hallucinations are widely publicized, and have led many to doubt the systems, I would assume that the owners of the LLMs would be interested in taming this problem.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 03:59:16 pm
content: Quote from: TimFox on August 07, 2025, 02:20:33 pmThe implication of that surmise is that the "software at the top" is more intelligent/knowledgeable/understanding the intent behind the query than the LLM underneath. I'm sure you can see the recursive difficulty there!Analogy: sometimes people think that inside our head there is a little homunculus looking at the pictures received by our eyes. Obvious question is what's inside the homunculus?

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 04:04:43 pm
content: Quote from: tggzzz on August 07, 2025, 03:59:16 pmMy statement has nothing to do with the "intent behind the query": if the system produces something in the format of a standard reference, e.g. "Plessy v Ferguson" or "Phys Rev78124" (the former is real and the latter was totally made up by me), the software system should have a clue where that came from during the process that produced it. I'm not referring to incorrect surmises or even truth, but to non-existent fictional references suitable for submission in court and analogous places.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 04:11:41 pm
content: Quote from: TimFox on August 07, 2025, 04:04:43 pmThat software systemisthe LLM; there is nothing above it.The LLM cannot retain individual source information because that was discarded during as part of the training process. All that remains is the probabilities of words in contexts, not how the many many sources contributed to the probabilities.Fundamentally the training process isrequiredto lose information, and munge all the sources together.

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 04:15:24 pm
content: Quote from: tggzzz on August 07, 2025, 04:11:41 pmI don't pretend to understand the functional details. If your statement is literally true, and the LLM operation leaves no tracks behind about its process steps, then why would anyone trust the results? I wait for an actual criminal prosecution (as threatened already in UK) for anyone who uses these results in any consequential case (e.g., court filing).

author: Siwastaja
author_rank: Super Contributor
timestamp: August 07, 2025, 04:16:36 pm
content: Quote from: TimFox on August 07, 2025, 04:04:43 pmOf course there is a clue, it's just that with neural networks the clue is extremely wide, and as such the answer is something along the lines of "probably as a combination from some hundreds of millions of these billions of coefficients". Getting a number wrong, e.g. "issue 57" instead of "issue 42" would be very typical for a neural network; the problem is that number 57 occurred in training material for possiblymillions of timesand somehow was associated close to other symbols related to the question. It is very hard to know much more than this.You can ask the LLM of course, but then again you are at the mercy of those same neurons. There is very little connection to classic software algorithms where you can point out which lines of code calculated exactly what. That's both the strength and the weakness of neural networks.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 07, 2025, 04:20:29 pm
content: Quote from: TimFox on August 07, 2025, 04:15:24 pmBecause some people are idiots. You can also ask why would anyone <do stupid thing X>. Why would anyone smoke for example? Yet even many wise people do.But LLMscanleave quite decent tracks of how they did the "thinking". Try something like running Deepseek with the "deep think" button, or whatever it was called, enabled. But the process is still prone to the same hallucination, and same mistakes. You would need a true human expert to review both the steps and the result. But then you need to trust that human expert - humans are prone to exact same types of problems. It is not unheard of to pick the wrong experts and get utter hallucination instead of real information. In fact, if you read newspapers, significant % of their "experts say" stuff is total crap. AI is just making the thing even worse.Sothe problem of trustcannot be avoided.

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 04:32:38 pm
content: Quote from: Siwastaja on August 07, 2025, 04:20:29 pmIt's common for human "experts" to make mistakes or state their opinions as fact or misquote other authors or lie about the facts.It's rare for a human "expert" to make up a fictional reference citation in formats such as "Plessy v Ferguson" or "Phys Rev78124".That's my complaint.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 08:26:17 pm
content: Quote from: TimFox on August 07, 2025, 04:15:24 pmThese problems surfaced in the early 1980s, with Igor Aleksander's WISARD. They are not new, but youngsters don't know history and so remake old mistakes.WISARD was pure hardware, no software die to the lack of computation and memory. While there are significant differences, the fundamental principles are the same: show examples, let the machine compute weights, show something new and see what it makes of it.When trained to distinguish between tanks and cars, it performed well in the lab but dismally in the field. They couldn't figure out why. Eventually someone realised the picture of tanks were taken in North German grey skys, but the pictures of cars were from magazine - and hence with sunny skies. Yes, it had learned to distinguish sunny vs overcast days - but there was no way of knowing that.The modern LLM equivalent is differing fonts, and standing vs supine patients.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 08:31:52 pm
content: Quote from: Siwastaja on August 07, 2025, 04:20:29 pmLLMs can be useful iff their output is assessed by subject experts - who won't need LLMs in most (but not all) cases.It is common for people to think LLMs are good, except in their own area of expertise. Guess why.Companies are falling over themselves to replace humans with LLMs where those affected by the LLM output can't prove or disprove the output. Think customer service, probability of an offender absconding/reoffending, etc.

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 08:34:23 pm
content: Quote from: TimFox on August 07, 2025, 04:32:38 pmIt is a valid complaintEver come across the phrase "because the computer says so". S/computer/LLM/. Welcome to the idiocray.

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 09:06:30 pm
content: Quote from: tggzzz on August 07, 2025, 08:34:23 pmI was traveling years ago with a co-worker, and the airline movie was Superman III.We both ended up quoting a catch-phrase: "It's all done by computer, what could possibly go wrong?".

author: tggzzz
author_rank: Super Contributor
timestamp: August 07, 2025, 09:24:45 pm
content: Quote from: TimFox on August 07, 2025, 09:06:30 pmReplace "computer following explicitly programmed if-then-else statements and defined arithmetic" with "LLM incapable of arithmetic but imitating average/typical pre-existing behaviour".Feeling better now?

author: TimFox
author_rank: Super Contributor
timestamp: August 07, 2025, 09:27:08 pm
content: Repeating my complaint: "It's common for human "experts" to make mistakes or state their opinions as fact or misquote other authors or lie about the facts.It's rare for a human "expert" to make up a fictional reference citation in formats such as "Plessy v Ferguson" or "Phys Rev 78 124".I don't expect much of the LLM for purposes other than organization.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 08, 2025, 06:57:09 am
content: I'm finding "tggzzz"'s statements on this highly unsatisfactory, at least in addressing Tim Fox's quite reasonable concerns about determining whether references made by LLMs are imaginary or real.No doubt they (tggzzz) know quite a bit more about how LLMs work; I'm quite willing to concede that.But I don't see why there cannot be another software layer, another module, anothersomethingthat works in concert with a LLM to verify whether it's quoting real, bona fide, checkable references or whether it's just making shit up.Thissomethingwouldn't have to operate the same way a LLM does; it could use well-known, standard computer search techniques for the verification process.Why couldn't this be done? Assuming, of course, that there was the will to do it on the part of the LLM creators/vendors.They could make it an extra-cost accessory for those skeptics who have doubts about the LLM's output.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 08, 2025, 07:15:11 am
content: Quote from: Analog Kid on August 08, 2025, 06:57:09 amOf course it can be done, but remember, a "simple" verifiable fixed classic algorithm can only detect obvious hallucinations - like, the source material (book name, chapter number, author name, etc.) not existingat all. But these mistakes are just the tip of the iceberg. Much more problematic are misinterpretations of the materials, and for that you needintelligence. Of computer algorithms, LLMs are actuallyclosestto that, but still so far away, and progress seems to be slower than people are hoping (or fearing).Why it isn't being done? Maybe itis, maybe half a year from now on the usual AI engines have added some sort of existence checking algorithm on top. Like, I suspect, they added non-LLM arithmetic unit when "3 + 4 = -11" type mistakes were just too awkward for them. Fixes like this increase trust of AI, exactly what companies need, so I don't see why they would not be working on this. Yet, it does not increase the actual output qualityat all, just makes the answers more dangerously false.

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 12:15:51 pm
content: Quote from: Analog Kid on August 08, 2025, 06:57:09 amFirstly, do you understand the concept that a hash function computes a result based on the input, but the information in the input information cannot be found in the result: by design the information is thrown away. The same is true for LLMs, although the mechanisms are different.You are expecting there to be something that is better than LLMs. That is not possiblein general, since if it was it would simply be incorporated into LLMs.It is, of course, possible inconstrained limited cases- but not for the general ad-hoc queries responded to by LLMs.Such things wold not be LLMs, by definition.

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 12:19:32 pm
content: Quote from: Siwastaja on August 08, 2025, 07:15:11 amPrecisely. Emphases added.The result would be a better class of bullshit.

author: TimFox
author_rank: Super Contributor
timestamp: August 08, 2025, 02:18:29 pm
content: A recent press release about the next ChatGPT release stated that it would reduce hallucinations.It will be interesting to see how well that works.

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 04:08:08 pm
content: Quote from: TimFox on August 08, 2025, 02:18:29 pmDid they indicate how they will reduce hallucinations?In the absence of comprehendible explanation, the most I would expect is that a few special cases might be better. Whack-a-mole.In addition, tweaking a few weights might well mean new hallucinations arise. Squeeze a half-inflated balloon.Maybe that's why Grok recently started spouting neo-nazi alternative facts and conspiracy theories?

author: TimFox
author_rank: Super Contributor
timestamp: August 08, 2025, 04:11:21 pm
content: Quote from: tggzzz on August 08, 2025, 04:08:08 pmThe release I read had no details.Grok was ordered to avoid woke responses, so it naturally went to the opposite extreme, including ur-Nazi responses.

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 04:35:20 pm
content: Quote from: TimFox on August 08, 2025, 04:11:21 pmAll plausible, given the complete lack of predictability.As an engineer, I hate that. I want artefacts that are predictably better than people in some respect or other.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 08, 2025, 06:54:01 pm
content: Quote from: tggzzz on August 08, 2025, 04:35:20 pmYou need to realize that this unpredictability is exactly what makes LLMs appealing and interesting. It's exactly what makes it appear intelligent - something with its ownwill, so to speak. This is what peoplewantfrom "AI".Of course, asking it to do serious jobs and expecting correct and good results is a mistake.

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 07:23:14 pm
content: Quote from: Siwastaja on August 08, 2025, 06:54:01 pmInteresting perspective.You may well be on to something

author: Analog Kid
author_rank: Super Contributor
timestamp: August 08, 2025, 07:34:59 pm
content: Quote from: tggzzz on August 08, 2025, 12:15:51 pmCorrection: no, I am not expecting there to be something that's better than LLMs.I do understand (and completely agree with) the caveats and cautions that you and others have posted repeatedly on the subject. No disagreement there.What I'm proposing--naively, perhaps, but still--is the possibility of some kind of "overseer" function,outside of and apart from the LLM core, perhaps some kind of post-processor, which would act as a bullshit filter for citations issued by the LLM.Are you saying that not even this is possible?

author: tggzzz
author_rank: Super Contributor
timestamp: August 08, 2025, 09:18:39 pm
content: Quote from: Analog Kid on August 08, 2025, 07:34:59 pmFor reasons unknown you chose to snip my paragraph which answered that question before you asked it!Seehttps://www.eevblog.com/forum/chatgptai/hallucinated-reference-citations/msg6004509/#msg6004509

author: TimFox
author_rank: Super Contributor
timestamp: August 08, 2025, 09:22:59 pm
content: A recent (Aug 7, 2025) news article about ChatGPT5:https://www.theverge.com/news/756342/openai-chatgpt-gpt-5-update-biggest-changes?utm_source=firefox-newtab-en-usNothing in that article about hallucination.

author: DimitriP
author_rank: Super Contributor
timestamp: August 08, 2025, 09:48:08 pm
content: QuoteApproaching LLM output without scepticism is what us common folk call a mistake and can easily turn different LLMs into their own "religions", especially if they dissagree with each other. And we are back to square 1, faster!!!

author: Analog Kid
author_rank: Super Contributor
timestamp: August 08, 2025, 11:11:48 pm
content: Quote from: tggzzz on August 08, 2025, 09:18:39 pmMeaning this one, correct?QuoteYes, yes, but you're describing the production of the LLM's output.In case you missed it, my idea, valid or not, is to check that outputafter it's generated by the LLM, hence my suggestion of some kind of post-processing as a possibility.So the validity checking would have nothing to do with the opacity of the LLM's inner workings.Does that make sense to you?As for that subsequent reply pointing out the problem with "approaching LLM output without skepticism", well, yes, one should be skeptical. But my idea is, I guess, aimed at the great unwashed masses who'll tend to take LLM output at face value.Thinking about this a bit more, my idea of a post-LLM bullshit filter could probably easily be implemented as an online service. Kind of a dynamic Snopes, where you could submit LLM output and have it screened for veracity, at least for theexistenceof any citations referenced (but not the veracity of the LLM's conclusions based on them). In fact, I'd be surprised if someone doesn't create something much like this in the very near future.

author: tggzzz
author_rank: Super Contributor
timestamp: August 09, 2025, 06:14:53 am
content: Quote from: Analog Kid on August 08, 2025, 11:11:48 pmSigh.This is my paragraph that you snipped...Quote from: tggzzz on August 08, 2025, 12:15:51 pmThe "it" refers to your idea, as is evident if you look at myunsnippedpost.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 09, 2025, 06:45:10 am
content: Sigh, indeed.I think we're arguing at cross-purposes here.Could you just address this one thing:Would it be possible to take theoutputof an LLM--not interfere in any way with the mysterious inner workings of that diabolical mechanism--and run it through--something, some kind of post-processorthat would itself not be an LLM, and have that something verify just the one aspect that we've been discussing here lately, which is simply the existence or not of citations in the LLM's output?I'm assuming here that it would be possible to extract instances of citations (say in legal, technical or medical journals), for example in the forms given by Tim Fox above, so that their existence could be verified. Might not be trivial, but it should not require the services of "AI" to accomplish this, don't you agree?I'm not wedded to this idea, but I still have not gotten a decent answer to the question of whether this would be possible or not. I think it would. But I'm no expert in this field.

author: tggzzz
author_rank: Super Contributor
timestamp: August 09, 2025, 08:38:19 am
content: Quote from: Analog Kid on August 09, 2025, 06:45:10 amAs I wrote previously, "It is, of course, possible inconstrained limited cases- but not for the general ad-hoc queries responded to by LLMs". Which bit of that do you not understand?Alternatively... devise an algorithm and you will become extremely rich. Meta/Farcebook is reportedly offering people $100e6 (or more) to people from OpenAI, or $440e3 salaries.https://www.bbc.co.uk/news/articles/c8730088e5doStart by defining how your proposed system distinguishes "accurate" and "inaccurate" sources, and by defining the input your system accepts and parses to determine whatneedsto be checked. Make it general case, of course.

author: tggzzz
author_rank: Super Contributor
timestamp: August 09, 2025, 09:52:41 am
content: Quote from: TimFox on August 08, 2025, 04:11:21 pmThat rings a bell.Musk trains his general-purpose LLM to influence public opinion by not generating responses which conflict with his extreme political views.That's revolting, and damning of both the owner and their product. Expect similar to occur in the future, more and less overtly.N.B. there is one suitable general purpose algorithm: get a human skilled in the subject to carefully verify the LLM output.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 09, 2025, 06:05:05 pm
content: Quote from: tggzzz on August 09, 2025, 08:38:19 amDo you even bother reading what I fucking wrote?Has nothing to do with "accurate" or "inaccurate" sources.The only criteria iswhether or not the cited referenceactually exists or not.A simple Boolean criterion. Exists or was made up. Which after all is exactly what the OP was after here.Nothing about evaluating the use of the reference in the LLM's output.

author: tggzzz
author_rank: Super Contributor
timestamp: August 09, 2025, 08:38:51 pm
content: Quote from: Analog Kid on August 09, 2025, 06:05:05 pmYour post is internally inconsistent.If you manage to achieve your concepts in other than scripted special cases, then you will become as rich as Croesus. I won't hold my breath.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 09, 2025, 09:03:49 pm
content: Quote from: tggzzz on August 09, 2025, 08:38:51 pmHow so?QuoteIt's just an idea, certainly nothing I'd consider actually pursuing.And I'm not even sure it would be a good idea even if it were possible, as it might make "AI" more palatable when just the opposite is indicated.

author: Siwastaja
author_rank: Super Contributor
timestamp: August 10, 2025, 08:39:41 am
content: I guess tggzzz's point is similar to mine: trivial non-LLM existence check algorithm covers so little that it's not of any significant value. For each example of totally made up name and totally non-existing citation, we have 100 existing but misinterpreted citations, and 1000 cases of not being able to name a citation at all; much harder problems to solve.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 10, 2025, 09:41:14 pm
content: Quote from: Siwastaja on August 10, 2025, 08:39:41 amFair point, I guess.@TimFox, what do you think of that, since you're the one who brought all this up?

author: TimFox
author_rank: Super Contributor
timestamp: August 10, 2025, 09:55:02 pm
content: Yes, I was specifically referring to hallucinated citations in standard formats (e.g.Plessy v Fergusonin court filings orPhys Rev84142 for scientific papers), not the content thereof nor other false content made from combinations of stuff in the data scraped from the web or other sources.I naively thought that hot-stuff systems could be programmed (not necessarily trained) to recognize that specific result, but I do not claim to understand the back-office system that puts out the result.Un-referenced inaccurate or accurate results are perhaps less hazardous than hallucinated non-existent citations, since lawyers and students in a hurry are more likely to accept the citations without going to the library.

author: Analog Kid
author_rank: Super Contributor
timestamp: August 10, 2025, 10:36:21 pm
content: Quote from: TimFox on August 10, 2025, 09:55:02 pmWell, to me that sounds like kinda-sorta an endorsement of my idea (which is some kind of system that would simply identify non-existent citations made by an LLM after the fact).I rest my case. (Dewey v. Cheatham, 259 U.S. 47, 58 (1973) )

author: photon
author_rank: Frequent Contributor
timestamp: October 05, 2025, 11:25:17 pm
content: Quote from: tggzzz on August 05, 2025, 04:15:32 pmSince humans, until AI, were the only intelligent beings, why do you think bad behavior isn't a part of intelligence. Unless you know someone with zero bad behavior.

author: Howardlong
author_rank: Super Contributor
timestamp: October 10, 2025, 04:49:47 pm
content: Australia's Department of Employment and Workplace Relations (DEWR) outsourced their own employment to consultancy firm Deloitte, who relied on unchecked hallucinogenic AI slop to swindle Aussie taxpayers out of $440k.Deloitte claims "Human error". LOL!https://www.theregister.com/2025/10/06/deloitte_ai_report_australia/

author: ejeffrey
author_rank: Super Contributor
timestamp: October 11, 2025, 06:00:41 pm
content: Quote from: radiolistener on August 04, 2025, 10:00:07 pmI suspect it's a conscious design goal.In silicon valley, "scalability" is basically a cult. It's so ingrained into the thinking that many people don't even question the meaning.Putting in special purpose rules like "identify things that look like citations and run them through this other process optimized for citation verification" is probably viewed as non-scalable and thus not worth doing.It's probably considered non scalable on at least two axes. First and foremost, citations are only one specific form of information. Creating a correctness checker for one type of information kind of implies that you will have to do that for a neverending series of types of information. This is correct, but undesirable from the perspective of people.expecting to take one LLM, add a bit of prompt engineering and displace millions of human workers at negligible marginal cost.The second axis is the difficulty of citation checking itself. Citations come in a lot of different formats, and primary sources are extremely varied in how their data is accessible. Fixing your verification heuristic for each of those is again a long stream of engineering effort with decreasing marginal utility. Of course in the specific case of e.g., case citations in legal briefs this should be relatively easy, but I think the problem is that the mindset is wrong.So the solution silicon valley will come up with is "ignore the problem and hope it either goes away or people just accept a shitty product" which to be fair has been a pretty financially successful move.In the specific case of citations, I hope they don't implement targeted fixed simply because it's an excellent canary in the coal mine. Mis-stating conclusions of cited work or fabricating data can be very hard to catch. Fraudulent citations are relatively easy and once you find that you can know that the whole report is junk and should be thrown out and the person responsible sacked.

author: tggzzz
author_rank: Super Contributor
timestamp: October 11, 2025, 07:01:21 pm
content: There is a simpler explanation: adding a hallucination checker would be an admission that the product cannot be trusted.

author: photon
author_rank: Frequent Contributor
timestamp: October 12, 2025, 11:41:11 pm
content: Quote from: tggzzz on October 11, 2025, 07:01:21 pmJust as humans cannot be trusted.

author: Siwastaja
author_rank: Super Contributor
timestamp: October 13, 2025, 05:11:21 am
content: Quote from: photon on October 12, 2025, 11:41:11 pmYou know, 99% of people I know do not invent completely bogus random references. In the world of human beings, those who do are called "psychopaths" or "pathologic liars" or "nutjobs" etc. and are rare. Instead, with real humans, we see lighter variation: like the "law says so" idiots. They invent what "the law says" to support their argument - but do not actually go as far as inventing bogus references. When asked for a reference, most of these idiots just go silent. That's the difference. AI is like a real psychopath - it sounds convincing exactly because the trait is so rare. If it was more common to act like this, people would be more careful and always check carefully what others are saying.

author: tggzzz
author_rank: Super Contributor
timestamp: October 13, 2025, 10:16:03 am
content: Quote from: photon on October 12, 2025, 11:41:11 pmYes indeed.Unfortunately too many people fall back on the Nuremburg Defence, w.r.t. people ("I was only obeying orders") or computers "("because the computer says so").While people realise other people are fallible (even thought they still seek infallible "messiahs"), too many people trust "impartial" computers.LLMs bring new levels of non-impartiality, and people don't recognise that yet.

author: tggzzz
author_rank: Super Contributor
timestamp: October 13, 2025, 10:19:32 am
content: Quote from: Siwastaja on October 13, 2025, 05:11:21 amYes indeed.People spewing "lighter" rubbish are also called "bullshitters". Spotting them is part of life, trying to counter bullshit is an inevitable tax on life.A variant of that is the 37% of statistics that are made up on the spot.