AI parsing data sheets - Page 1
Forum section: electronics / general

author: AndyC_772
author_rank: Super Contributor
timestamp: September 16, 2024, 04:09:16 pm
content: Has anyone had any success getting any real help from AI to read, parse, and answer questions about parts from their data sheets?Given how relatively proficient ChatGPT seems to be in coding, I had reasonably high expectations that it might also be able to answer questions about components. In this instance, I found two ADCs from competing manufacturers, which have completely different part numbers but actually look as though one may have been designed as a pin compatible alternative to the other. They both come in the same package, have similar headline specs, and although many pins have different names (eg. "VCC" vs "AVDD"), their functions seem similar. Certainly there are too many similarities to be a coincidence.This seemed like a good opportunity to get some help from AI. Ask it what the functional differences are, and whether they are in fact likely to be compatible.Of course I can't trust the answers without checking, but if it notices that (say) one has a particular feature that the other lacks, that at least is something I can quickly look for in both data sheets. I often seem to find myself trying to compare and contrast similar parts, so any help in spotting differences is welcome.To say I was unimpressed would be an understatement. ChatGPT (-o1 preview) failed to correctly identify the number of inputs or even the number of ADC cores in each part. Features clearly common to both were missed in one device, and highlighted as a point of difference. Each time I said "no, try again" it would apologise, correct its mistake, then make a bunch of new mistakes next time around. Stupid, obvious errors, like "pin X is called NNNN", where it just isn't.In the end it was completely useless, and I gave up - but I do think this is an area where AI might one day help and save time.Has anyone found an AI tool that does a better job of "understanding" data sheets and answering questions about them?

author: tom66
author_rank: Super Contributor
timestamp: September 16, 2024, 04:56:13 pm
content: A while ago I attempted to use it to find pin-compatible alternative voltage regulators, but it just barfed some made up part numbers at me. Unfortunately, at the edge of its training data, it's quite likely a lot of the output is garbage. I can imagine an LLM would be quite good at this if trained specifically on electronics engineering datasets.

author: kripton2035
author_rank: Super Contributor
timestamp: September 16, 2024, 05:01:22 pm
content: this one :https://docs.flux.ai/claims to have been trained to do such a job of reading datasheets, but I've never been able to make it work correctly.

author: tggzzz
author_rank: Super Contributor
timestamp: September 16, 2024, 05:28:41 pm
content: Quote from: AndyC_772 on September 16, 2024, 04:09:16 pmRealise that LLMs are a mechanism for producing grammatically correct sentences of plausible words. They have zero understanding of the words an concepts.There's a name for humans that manage to do that: bullshitters.Is is any surprise that managers think LLMs work well?

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 16, 2024, 07:38:22 pm
content: But the marketing is apparently really good.

author: amyk
author_rank: Super Contributor
timestamp: September 17, 2024, 12:34:13 am
content: Given thathttps://www.eevblog.com/forum/chatgptai/ai-generated-lies-in-datasheet-search-results/has been all I've seen AI do with datasheets, I'm not surprised.

author: Berni
author_rank: Super Contributor
timestamp: September 17, 2024, 05:21:39 am
content: Quote from: kripton2035 on September 16, 2024, 05:01:22 pmThey are likely using OpenAIs GPT models via the API or something similar.Just blindly asking these general purpose LLM AIs about very specific things like chip partnumbers or specs is a bad idea because when they talk about a more niche topic they hallucinate like there is no tomorrow, making up stuff as they go, rather than saying that they don't know. To make things worse it puts the made up information into nice confident sentences that makes it look like it knows what it is talking about.Feeding it a PDF of a datasheet likely works fairly well since todays LLMs have enough context length to hold even a fairly sizable PDF in memory. Tho most of the time Ctrl+F will get your answer sooner in datasheets, while you also have to understand that the AI doesn't read a PDF like a human, so depending on the internal PDF structure it might not be able to see some diagrams or see them wrong.

author: ebastler
author_rank: Super Contributor
timestamp: September 17, 2024, 06:07:29 am
content: Quote from: tggzzz on September 16, 2024, 05:28:41 pmThat's a description on the same level as "Computers are really simple and really dumb. All they do is manipulate 0s and 1s." It's true on some level, but it leads you to totally underestimate the capabilities that can be obtained by piling on a few layers of abstraction and complexity.(That computer quote was actually pretty commonly heard when computers became more visible in the 1970s. Typically used by people who were slightly nervous about them and had a very limited idea what they were and how they worked.)

author: tggzzz
author_rank: Super Contributor
timestamp: September 17, 2024, 07:49:41 am
content: Quote from: ebastler on September 17, 2024, 06:07:29 amWith computers you could always work out why they produced the result they did, and the limits within the results were valid. With LLMs that is, ahem, "an active research topic"with no resolution in sight.If you read comp.risks (which everybody should), real world examples abound, from sentencing/custody decisions in the US, to things which appear to "work" for spurious "reasons"https://catless.ncl.ac.uk/Risks/32/80/#subj4.1"Some AIs were found to be picking up on the *text font* that certainhospitals used to label the scans. As a result, fonts from hospitals withmore serious caseloads became predictors of covid risk."That's a recent version of an old old problem. Igor Aleksander's 1983 WISARD, effectively the forerunner of today's LLMs, demonstrated a key property of modern LLMs: you didn't and couldn't predict/understand the result it would produce. WISARD correctly distinguished between cars and tanks in the lab, but failed dismally when taken to Luneberger Heath in north Germany.Eventuallythey worked out the training set was tanks under grey skys and car adverts under sunny skies.As for the "edge of the envelope" problem, consider there are documented examples where changing one pixel in a photo of a road "stop" sign, caused the ML system to change to interpreting it as a "40MPH" sign. The corollary is that if a bugfix is issued for that, there is no way of knowing whether it willcausea "slow" sign to be misinterpreted as a "50 mph". that's less true now because they have piled on so many layers of abstraction and complexity that not many people comprehend them all. Look at all the synchronous and asynchronous protocols layered on top of each other in a typical enterprise/telecom system. Too many practitioners think FSM== parsers, and believe the salesmen's (implicit) claims that their software avoids the Byzantine Generals problem. Except that either they don't know that problem or have forgotten it and don't relate it to their distributed systems.

author: ebastler
author_rank: Super Contributor
timestamp: September 17, 2024, 08:12:43 am
content: Quote from: tggzzz on September 17, 2024, 07:49:41 amThat is a very different concern from your prior over-simplification that "LLMs just produce grammatically correct sentences with plausible words." Which is all I had commented on.QuoteThat particular anecdote is a quote from a "quote" from aTechnology Reviewarticle. The Driggs paper which Technology Review refers to does not seem to deal with LLMs at all, and it certainly does not contain the word "font". Which confirms my bias that Technology Review is just a collection of grammatically correct sentences with plausible words...

author: AndyC_772
author_rank: Super Contributor
timestamp: September 17, 2024, 09:00:21 am
content: Quote from: tggzzz on September 16, 2024, 05:28:41 pmI'm going to go out on a limb here and suggest you've not spent any time with a recent paid-for ChatGPT interface.If you get the chance - and I strongly recommend subscribing for a month just so you can - ask it to explain a technical topic with which you're already somewhat familiar. Ask it to write some code in your preferred language to carry out a useful function, or upload code of your own and ask it to check for errors. I think you'll very soon realise it's far more capable than just a package that strings together plausible sounding words on a statistically likely basis.The latest -o1 preview build is specifically crafted to follow chains of logical reasoning. It even describes each step as it works toward composing a response.

author: tom66
author_rank: Super Contributor
timestamp: September 17, 2024, 09:07:55 am
content: Quote from: tggzzz on September 17, 2024, 07:49:41 am"AI was a bit shit in the 80s, so of course in 2024 it will still be shit."Yes, there are problems with LLMs & NN image processing, but that does not mean that they are useless. For your tank example it's a well known problem in model training where the model is overconstrained and the training data is too small. ChatGPT has been trained on some 40 terabytes of input data with man years of human filtering applied afterwards.And, anecdotally, I've found that over the past two years it's become increasingly difficult to get ChatGPT to output bullshit. It either says "I don't know the answer" or it answers.

author: Berni
author_rank: Super Contributor
timestamp: September 17, 2024, 10:51:14 am
content: Even the GPT4o you get from a free ChatGPT account is impressively smart.Yes deep down these kind of AIs are indeed glorified statistics. The actual technology here behind this AI is in figuring out a computationally more efficient way to calculate these statistics since there is no way you can store the probability of a word for all possible combinations of 10000+ words. So a giant neural network is trained to output something as close as possible to the real word probability.The intelligence part of LLMs is an emergent behavior (like the patterns in flocks of birds) where the simple actions come together to express a complex behavior. The neural net inside the LLM learns to encode various high level concepts inside the network because that is the only way it can squeeze them into the network, there are not enough neurons to store it brute force as a giant 'lookup table'.You can make the same point with a biological brain. It is just a pile of neurons that fire off signals depending on their inputs, there is no real intelligence inside a neuron. not even in groups of neurons. But when you put a huge number of them together and teach them on how to react to certain stimulus, the whole thing starts acting as something that appears to be intelligent. If the pile of biological neurons is trained badly it also performs badly.And yes an AI like this is only as good as its training data. For common things where they could scrape a lot of training data from the internet,books,etc.., there the AI performs remarkably well. For things where it did not have enough training data it will perform poorly. You need a LOT of data for a good training dataset. However even more importantly you needhigh qualitytraining data, so that there are enough properly varied examples to isolate the concept of interest and there are no large biases. There also has to be intentional examples of garbage in the training data so that it can learn to ignore the garbage.Fact of the matter is that LLM AIs have enabled computers to do tasks they could never do before. If you train an AI for a task properly and only use it for that task, they perform great. The problem is that this new AI stuff has picked up a massive amount of hype around it, so now everyone is forcefully shoehorning the new fangled AI tech into absolutely everything, regardless if it makes sense to do so.

author: daqq
author_rank: Super Contributor
timestamp: September 17, 2024, 04:51:59 pm
content: I tried it on a chinese datasheet for a LED driver - I was actually pleasantly surprised by the results. Wouldn't be very useful for an MCU probably, but it worked great for the SM16306SJ:https://chatgpt.com/share/66e9b300-d3e0-8002-bf35-3a4c456b2a20

author: tggzzz
author_rank: Super Contributor
timestamp: September 17, 2024, 07:31:37 pm
content: Quote from: tom66 on September 17, 2024, 09:07:55 amEr, no.AI in the 1980s was a set of rules, based on knowledge elicited from experts. It was then easy to see which rules caused the results, and why.WISARD was not like that; it was unique.In the 2020s ML is shit for different reasons, as foreshadowed by WISARD. In particular it is impossible to know the chain that caused the results, because there are no explicit rules.QuoteThat merely makes the problems more subtle.Never forget Tony Hoare's wise words from his 1980 Turing Award Lecture; Communications of the ACM 24 (2), (February 1981): pp. 75-83.There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.QuoteAre you one of those people that believes a piece of software is works because it passes the unit tests?

author: tggzzz
author_rank: Super Contributor
timestamp: September 17, 2024, 07:49:19 pm
content: Quote from: Berni on September 17, 2024, 10:51:14 amThe point about the similarity between wetware and LLMs is undeniable. After all, wetware was the inspiration for LLMs, starting with the earliest "perceptron" experiments in the 50s and 60s.The output of brains and LLMs is indeed a consequence of "emergent behaviour". That makes it impossible to determine what the response will be and why. As an engineer (not a psychologist, not a politician, etc) I regard that as a key failing.Bad training data is a key problem. People are already worrying about what happens when a lot of the content on the web is semi-accurate LLM output, and that is used as training data for the next round of LLMs. Experiments have demonstrated that is a realistic problem.Saying "it works if you have good training data" is merely opium that I would expect salesmen to spout. Similarly, fracking salesmen say that "fracking is safe when done properly", which is obviously a meaningless truism, and ignores the probability of it not being done properly.Some training data has, by definition, to be polluted. That doesn't stop users from claiming "the machine says so therefore it is right" (note the present tense). Classic example is the systems in the US that determine the risk associated by an accused person or a convicted felon and therefore whether they should be incarcerated before/after trial. Since black people have a higher probability of being incarcerated, the LLMs can only perpetuate that injustice.

author: SiliconWizard
author_rank: Super Contributor
timestamp: September 17, 2024, 09:17:05 pm
content: In other words, throwing stuff at the wall until it sticks, and calling that engineering.

author: Berni
author_rank: Super Contributor
timestamp: September 18, 2024, 06:59:59 am
content: Yep LLMs are most definitely not perfect and make mistakes (Pretty much any AI chat bot warns about that on the first page).But as imperfect as they are, they are still performing certain tasks better than any of the prior non AI based solutions. Great example of it is translation between languages. We had computer programs that can translate text to a different language for many decades, however if you ever used any of those you will know that they all produce sentences that sound broken or sometimes translate it outright wrong because a word is similar. But even the first ChatGPT that ran on GPT 3 was providing translations that seam to be as good as a human translator. Not only that, it provided good translations into niche languages (for example Slovenian is only spoken by 2 milion people and is a difficult language, yet it has no issues with it, despite it not even being an officially supported language). This is already from a general purpose chatbot AI that was not specifically trained to do translation, it just happened to have a mish mash of languages in its training data and it figured out the connections.What these AIs excel in is interpreting fuzzy input. Computers with classical programing work great for interpreting very rigid languages, this is what programing languages are. Here everything follows a strict set of well defined rules, however human languages that have organically evolved over thousands of years are very messy. Hence this is why translating human language text has always been so difficult to get right for computers.When it comes to things like summarizing large piles of human text or extracting a certain bit of information from it, we didn't really have any way of doing it using classical programing that follows strict rules. So even if it performs poorly that means it still performs way better than not performing at all. Similarly classical computer vision algorithms only really work in ideal conditions, so in some industrial machine they have the camera rigidly mounted to look exactly straight onto the scene, target is the same object every time and is perfectly illuminated..etc. However if you are going to be looking for traffic signs out of a car windshield you won't have those ideal conditions, so applying the classical machine vision approaches would perform even worse as it would either completely ignore most signs if set to be strict, or if set loose it would misidentify signs. This is a very difficult task to solve.I also don't like the unpredictable nature of large AI models just as much. There is no way to 100% guarantee it won't generate the wrong output. You can just put more 9s on the 99.9..% by more testing. If you can solve the problem with classical programming then you should NOT be involving AI in this at all. But for tasks that can't be solved this way having something that works 99.9% of the time is still way better than something that does not work at all. As long as you are aware that it will make a mistake at some point and have something that can handle the mistake.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 08:33:44 am
content: Quote from: Berni on September 18, 2024, 06:59:59 amThere's some validity in that, e.g. translating a restaurant menu is very convenient, and I expect LLMs would avoid the historic "out of sight, out of mind"->"invisible idiot" and "the spirit is weak but the flesh is willing"->"the vodka is strong but the meat is rotten". It is also possible to validate (to some extent) a translation by getting the machine (preferably a different LLM) to do the reverse translation.However, any theoretical benefits can be overwhelmed by the way LLM systemsareused in out in the wild...You have no way of knowing whether adding one new test input will cause the weights to be recalculated so that previously acceptable outputs become dangerous. That isinherentin the way LLMs work, and cannot be avoided. "You can't test quality into a product".LLMs aren't good at digesting large amounts of fuzzy inputhttps://hai.stanford.edu/news/hallucinating-law-legal-mistakes-large-language-models-are-pervasiveThe "99% is better than nothing" is valid in some circumstances - but not in others. The 99% falls into an "uncanny valley": the user has to be aware that you have to correct/ignore the output 1% of the time, but probably isn't ablejudge which 1%. Add to that the lazy/unscrupulous user (of which there are many) and it is a recipe for problems.Whether or not detecting speed limit signs is/isn't possible with a LLM or conventional system is a red herring. No unscrupulous manufacturers have claimed an non-LLM system could do that; LLM manufacturers on the other hand do. Ray of hope: US transportation authorities are finally catching onto that, and have started demanding manufacturers justify their claims; I wonder how that will pan out.LLMsarebeing used for life-changing applications, without understanding of how they reach safe/dangerous conclusions. In some cases the decision making is even hidden behind "commercial confidentiality" clauses, in others users actively want to not think about the conclusions, in others legal liability for their illegal and false outputs is denied.https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-knowThe medical problems I've previously mentioned.The "lock them up and let God sort them out" applications that I've previously mentioned.Any of the yoootoob vids showing Tesla cars trying to drive on the wrong side of the road, or drive down railway tracks, etc, etc.Cruise and Waymo in the US.Etc.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 08:47:48 am
content: Quote from: tom66 on September 17, 2024, 09:07:55 amThat, of course, has two major limitations...What's the test that ChatGPT uses to determe that it doesn't "know" the answer? Why should the output of that test be any more reliable than its other output?The problem, as with people, isn't when they don't "know" the answer, it is when they don't "know" that they "don't" know the answer. Human equivalents are bullshitters and those who cannot realise they suffer from the Dunning-Kruger syndrome.

author: 5U4GB
author_rank: Super Contributor
timestamp: September 18, 2024, 08:50:00 am
content: Quote from: tggzzz on September 16, 2024, 05:28:41 pmIn particular things like ChatGPT are an implementation of theChinese room:QuoteWhen you want to know whether an LLM can solve a particular type of problem, you need to consider whether a Chinese room could solve it. If not then the LLM can't either, although it will hallucinate a plausible-sounding solution if pressed hard enough.

author: tom66
author_rank: Super Contributor
timestamp: September 18, 2024, 09:09:07 am
content: Quote from: tggzzz on September 17, 2024, 07:31:37 pmYes, fundamentally, it is impossible to prove that a model is always accurate without testing every possible input (which is numerically infeasible for anything over a small set of input data). But I am not sure that is as much of a 'gotcha' as you think it is. In order to implement the level of object matching that a convolutional neural network performs, for instance, it is necessary to have a fuzzy statistical model, because the input data is not clean, and there are too many combinations to match against.Take the problem of recognising an apple in an image. How would you implement a purely algorithmic image matching process? You could test against hundreds or thousands of candidate apple images, perhaps using an image-distance algorithm, but that gives a similar fuzzy output, a statistical probability of the image looking like an apple. And really that's all a convolutional NN is doing, except it is far more effective with a far smaller input training data set. You could use something like Haar classification, but that requires high input image contrast and well-defined features, so it doesn't work well in low-light or when looking at objects off-axis for instance. There are a few other methods out there, but the majority of them have fallen out of favour compared to CNN/DNN models because those produce far better results.You don't need a neural network to be 100% accurate, you just need it to be sufficiently accurate to generate useful outputs. Take the oft-noted example of the Tesla Autopilot seeing themoon as a traffic light. The visual neural network does, indeed, perceive a yellow traffic signal. The rest of the image processing algorithm clearly realises this data is garbage, so no action is taken on the input. So the system is working correctly, although they might do better to filter the displayed output on the UI.Quote from: tggzzz on September 17, 2024, 07:31:37 pmIf you design a sufficiently complex testbench then any program can be proven to be correct, I fail to see why this is a controversial statement. Now whether such a testbench is feasible is another matter. Certainly in the world of FPGA and ASIC design it becomes essential to thoroughly testbench your hardware design because debugging a failure is far more difficult than in software. Unit-testing individual components and writing a well-designed interface specification between them helps reduce the problem space.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 09:14:13 am
content: Quote from: 5U4GB on September 18, 2024, 08:50:00 amAgreed.I wonder how many people (general public, not those with AI experience) have heard of that question, and whether they understand it. Or, being more cynical ("follow the money"),wantto understand it.Given the amount of unadulterated crap in the AI training material (i.e. the web) I have obvious doubts. That will become worse when LLMs ingest the LLM output already on the web. Meta/Farcebook has stated they will use Farcebook posts to train their LLM; no obvious problems in that, are there?

author: ebastler
author_rank: Super Contributor
timestamp: September 18, 2024, 09:18:53 am
content: Quote from: tggzzz on September 18, 2024, 08:47:48 amLike AndyC in one of the early replies in this thread, I keep wondering: Do you prefer to philosophize about LLMs or have you actually used one lately? What did you use it for? How was your experience with it? What impressed you, what disappointed you, what specific problems did you run into?

author: Berni
author_rank: Super Contributor
timestamp: September 18, 2024, 09:28:54 am
content: Yes exactly the problem is that Ai is being used in all sorts of applications without consideration if AI is an appropriate solution.The advent of these big powerful LLM AIs made it even worse because they are general purpose models that can do seemingly anything. Even worse is that they take in generic text or images and produce any kind of text, so this makes it easy for developers to just shove a bunch of data into it as text and just blindly hope the AI can make sense of it. This is where the shitty chatbot 'helpers' on websites come from, they are completely useless.AI can be used in safety critical or life changing situations, but only when there are extra checks and safeguards around it, this effectively adds more layers to the 'swiss cheese model' where it gets more and more improbable that a mistake makes it trough everything due to everything lining up just right.For example Mobileye developed a camera based vehicle safety system back in 2008 that manufacturers like BMW, Hyundai, Volvo..etc integrated into their cars as part of a automatic emergency braking feature. I got a Volvo from 2014 with this feature and it works well. They take in information from the camera, radar, ultrasonic etc.. letting it able to detect a dangerous situation, first warning the driver with sound and flashing (likely already enough to get the driver to look forward and react) then if no action is taken the brakes are slammed on full at the very last moment. The whole thing works using a machine learned model running on a FPGA. But the system is just one extra layer of cheese in the model to cover up any holes in the human drivers cheese slice (since humans make mistakes too). Be it because the driver is distracted by a billboard next to the road while someone slams on the brakes in front, or if you are going down the highway and can't see a stopped car trough the fog, but the radar can see it.Once you give AI the task of full autonomous driving things are different. That one is a difficult problem that i have no idea what the correct solution to it might be (apart from a dedicated driverless car road network, but that is too expensive to be viable)Quote from: 5U4GB on September 18, 2024, 08:50:00 amYep this is a good explanation of how a LLM works.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 09:32:19 am
content: Quote from: tom66 on September 18, 2024, 09:09:07 amYou are missing the point.The point is that it isimpossibleby designto understand what caused an LLM to generate an output.That wasn't true for "conventional AI" in the 80s, since it waseasytosee and understandwhy each rule fired, and why. Writing the rules wasn't easy, but that's a different problem.QuoteWhat is "sufficiently accurate" for life-changing decisions?How can you prove that an LLM is "sufficiently accurate" when the next training test could change internal weights to invalidate previously correct output?QuoteIn this contextOraclesare boring, because they don't and cannot exist.Alternatively, thermodynamics is a solved problem if you use Maxwell's Daemons.Alternatively, please show us your unit tests that proves an implementation has ACID properties. Publish it, and you will become not only famous but also as rich as Creosus.QuoteHaving first done that in the early 80s with 2µm semi-custom technology, I am well aware of that - and how difficult it is to get "stuck-at-0/1" coverage.

author: tom66
author_rank: Super Contributor
timestamp: September 18, 2024, 09:44:09 am
content: Quote from: tggzzz on September 18, 2024, 09:32:19 amAnd it's infeasible to design a system in the way you indicate so it's a pointless argument. So: Write me an apple-recognising program that can process any 640x640 image and tell me if, and where, there is an apple in it. Even if I gave you an allowance of 100k lines of code to do manual image filtering and object classification I doubt you'd beat a CNN from a decade ago running in software.Quote from: tggzzz on September 18, 2024, 09:32:19 amFor self-driving vehicles,sufficiently accurateis driving safer than, say, the top 10% of human drivers in real traffic scenarios, over many millions of miles of real world driving.Waymo already achieves this, having had no fatality accidents and 10% as many fender benders as equivalent human drivers would be expected to be involved in. Tesla hasn't yet reached this standard, but they are probably only a few years away based on recent progress. (Yes, that's about a decade later than Elon said, but he's a master bullshitter. That doesn't change my opinion of the technology his engineers have created, which is 'pretty impressive' right now.)

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 09:46:37 am
content: Quote from: Berni on September 18, 2024, 09:28:54 amI'd be satisfied with "provably acceptably accurate". Provably != our unit tests haven't found a problem. Provably implies a chain of reasoning and/or explanation.QuoteIf that works acceptably, the problem becomes the people. They will rely on it and physically or metaphorically fall asleep.Prediction is everything, unless you prefer to believe in faeries and similarQuoteThere are credible reports of ?Teslas? braking suddenly and unacceptably (Can't be bothered to search for the specific reports)Teslas (and other autonomous cars?) don't have radar, but rely solely on cameras (and GPS location data, which is another source of error!)QuoteAn unanswered question is when (not if) an autonomous vehicle causes an accident, who do you sue? The car manufacturer? The LLM manufacturer? The driver?That raises the question of whether the LLM will detect it can't cope in sufficient time to hand over control to a driver. It takes up to 20s for aircraft pilots to become sufficiently aware and take avoiding action, and that is a much simpler environment than the roads. Saying "you have control" 2s before impact is not acceptable!

author: AndyC_772
author_rank: Super Contributor
timestamp: September 18, 2024, 09:54:21 am
content: The question in this thread wasn't about an AI making life changing decisions, or needing to produce output that's knowably, objectively correct.In this case, the challenge was to take two self contained documents, each describing something that's finite in scope and presented in a common, structured way, and make inferences between them which can then be checked. That shouldn't be hard, and how it works behind the scenes is irrelevant.I'm just trying to save time by offloading a relatively straightforward problem to a computer. I could equally well give the task to a junior engineer, but I don't have one to hand right now.I do have a paid ChatGPT subscription, though - one which hasalreadypaid for itself many, many times over by teaching me some correct science that enabled a substantial cost saving in a product that's now shipping. Like it or not, believe it or not, it's amuchbetter teacher than asking similar questions on a forum. It never says "do a search" or "your approach is wrong" or "I wouldn't do it like that" or any answer that's just plain unhelpful - it just answers the questions asked of it to the best of its ability.Sometimes it produces complete garbage,just like people do. But on technical topics, it's surprisingly accurate.Really surprisinglyaccurate. And, of course, the beauty of correct science is it can be readily verified and proved to be so.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 10:05:52 am
content: Quote from: tom66 on September 18, 2024, 09:44:09 amSo what? That's irrelevant to the point.QuoteMusk's engineers technology is impressive (but insufficient), and Musk is an impressive bullshitter (dangerous, in manymanyways).Waymo? We shouldn't concentrate on one manufacturer and one application of LLMs. Howeverhttp://catless.ncl.ac.uk/Risks/search?query=waymoleads to...Are you sure Waymo isn't relying on remote drivers like some other self-driving cars? Alphabet/Google refused to answer that question, but does have a patent.Alphabet's spin-out company Waymo still has by far the biggest testing program—its 635,868 miles of testing accounted for over 95 percent of all miles driven by self-driving cars in California in 2016. Waymo's fleet of 60 self-driving cars reported a total of 124 disengagements, 51 of them due to software problems.Some of the biggest critics of Waymo and other self-driving cars are the Silicon Valley residents who know how they work. ... The problem isn't that she doesn't understand the technology. It's that she does, and she knows how flawed nascent technology can be. ... Silicon Valley types can be most skeptical of advanced technology because they know how it works and what its risks are.

author: tom66
author_rank: Super Contributor
timestamp: September 18, 2024, 10:32:57 am
content: Quote from: tggzzz on September 18, 2024, 10:05:52 amIt's entirely relevant. If the only option you have for data processing is the fuzzy model, you use the fuzzy model. Ok, you can't prove it's correct and when it sees every possible apple image it will return a positive detection, but if you need to detect apples in an image, it's what you've got to use. So you test it as much as you can to prove it is sufficiently correct for what you are trying to do.Quote from: tggzzz on September 18, 2024, 10:05:52 amWaymo uses remote driving in the event of a car getting stuck, but they don't use it for normal driving. The problem for SDCs is when you get really really infrequent events that have limited training data around them it becomes difficult to test the car's response. So the human intervenes. As more and more of those situations become part of the training corpus, then it may become feasible to remove the remote control altogether.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 12:28:22 pm
content: Quote from: tom66 on September 18, 2024, 10:32:57 amThat's just silly. Theethicalcourse of action would beavoidattempting (and failing) to detect apples in that way.QuoteThat problem is true for more than self-driving cars; it applies to LLMs in general.How long must a self-driving car wait before detecting "I can't do it", and handing control back to the driver? Hint: humans take up to 20s before becoming sufficiently aware of the situation.If 20s isn't achievable, then what is the ethical course of action? To say "1s and tough shit, no longer my problem, you deal with the consequences". Given the potential consequences, I think that would be unethical and immoral.

author: Berni
author_rank: Super Contributor
timestamp: September 18, 2024, 01:02:44 pm
content: Fully autonomous driving AI is a whole different can of worms. As i said, I don't know how to solve that, and if you do, i bet some company will be willing to pay you some serious money for the tech.All i am saying is that this automated emergency braking and warning system is AI technology from 2008 (Tesla only just made their first car back then that was just a Lotus with a electric motor strapped to it). This Mobileye tech was fitted to cars of many manufacturers and there was no notable issues with it. You can't blame the manufacturer for the accident because you as the driver was supposed to take action before the system does anything. There is no "here you go control, please avoid the crash in 2 seconds", you always had control. The system just augments the reliability of an existing human driver by turning what would otherwise be a 100kh/h rear ending accident on a highway into a no accident at all (or at least a low speed collision without any injury). The system reacts at the very last moment on purpose so that the user would not comfortably rely on it (really it does stop just inches from collision using maximum braking).It is an example of AI tech applied correctly. It works reliably and is applied in such a way that it does not cause an immediate safety problem if it fails. It just enhances the safety by stepping in when the human driver didn't (but should have). So even if the system was only 90% reliable that means 90% less car crashes of this type(failing to hit the brake pedal in time).Besides what software these days even is 100% reliable? I have seen just about every car infotainment system crash. The quality of software these days is horrendous.Besides what is the alternative to the functionality that ChatGPT (and sililar chatbots) provides? I can ask a question on this forum too. But i will have to wait to get an answer, then the answer i do get might not actually be correct as the human on the other side is not 100% reliable (they might just be a smartass that doesn't bother to actually look up anything, or they genuinely forgot and made a genuine mistake in giving the wrong answer). Sometimes i might need the answer to a dumb question (but just tricky enough to take more than 1 minute to google) and if i ask it on a forum i will get a response of "Are you to dumb to use google?". Then people start arguing about something slightly related to the topic but has nothing to do with the actual OPs question....etcSo sometimes ChatGPT will genuinely give me a better answer than this forum, and will give it to me in 5 seconds. Especially with how bad Google results have gotten in the past years. Not like you can treat things you find on Google as being 100% accurate. As always do your own research rather than just take the first thing you read, hold true for both ChatGPT or Google results.

author: tom66
author_rank: Super Contributor
timestamp: September 18, 2024, 01:05:48 pm
content: Quote from: tggzzz on September 18, 2024, 12:28:22 pmI don't agree that ethics has anything at all to do with the situation. There are ethical and non-ethical ways to use LLMs and neural networks. Just as if there are ethical and non-ethical ways to use conventional software.There are use cases for neural networks which are effectively harmless but genuinely useful. For instance, my phone categorises pictures of each of my cats (so I can search them by name and post pictures on here if I want), and I can search photos for things like "receipt <restaurant name>" and it will scan text in photos to look for that expenses receipt I forgot. I believe that runs on the cloud - but I could be wrong.QuoteYou're confusing "take overimmediately- I'm going to collide with something" with "take over please - I'm stuck and can't get out of this situation". The former is of course dangerous and I think early autopilot systems prove that it isn't a path to autonomous driving. The latter, on the other hand, is "fine", though not ideal. An example would be a police officer closing and road and providing hand signals - current generation SDCs generally can't handle this condition so would rely on the remote operator to provide guidance.

author: dobsonr741
author_rank: Frequent Contributor
timestamp: September 18, 2024, 01:49:16 pm
content: This is how it’s done:https://blog.adafruit.com/2023/09/02/writing-an-arduino-driver-with-openai-chatgpt-and-pdf-parsing-%F0%9F%A4%96%F0%9F%94%A7%F0%9F%93%84/I repeated with another dataset myself, and it works. The cat is out of the bag.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 04:38:33 pm
content: Quote from: AndyC_772 on September 18, 2024, 09:54:21 amJust do it, e.g. ask "which of these two opamps is more relevant for a voltage reference implementation. Please let us know the resultsQuoteNaive, e.g. the answer to the above question being decided by a coin throwing machine.The reasoning is at least as important as the result, and LLMs can't explain how/why they produced their text. That, as they say, is an "active research topic", and I haven't seen any such research that might lead to optimism with real-world LLMs. I would bedelightedto be presented withreasonsto change that opinion.QuoteRegrettably sometimes those responses areexactlythebestresponse to the question.QuoteSo: sometimes great, sometimes crap. Bullshitters are as "good" as that, and cold-reading clairvoyants are even better. Be careful to avoid the different types ofhttps://en.wikipedia.org/wiki/List_of_cognitive_biases#Confirmation_biasOf course, some people somedoask the same questiontoss coins repeatedly until they get the "right" answer. They also twiddle the coefficients in their decision matrices until they get the right answer. Completely unimpressive, IMNSHO.OTOH, the data going through your mind to produce unbiassed good answers, is entirely sensible. I wish more people did it.

author: tom66
author_rank: Super Contributor
timestamp: September 18, 2024, 05:05:33 pm
content: I guess you haven't seen ChatGPT o-1, which claims to be able to reason its way to the answer. It's in pre-release so us plebs without a Plus subscription don't get it, but there are some examples on the internet of what it can do. According to OpenAI, it's able to pass the US SAT exams now better than the average student.https://x.com/OpenAI/status/1834278219954225452I think you are assigning too much to what is a very nascent technology. It is not perfect, but if I look at what ChatGPT was doing ~2 years ago (arguing with me that Paraguay was in North America was a good one) to what it does now, it's so impressive. If the rate of improvement continues, then there will be some serious societal questions to answer about obsoleting certain jobs with AI. I have heard that there is already an impact on marketing, with lots of marketing copy now being written by AI instead of humans. Of course, a human has to review the work, but you can get a lot more productivity out of one reviewer than you can out of five writers.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 05:53:10 pm
content: Quote from: tom66 on September 18, 2024, 05:05:33 pmGiven my contentions, clearly I ought to believe that bullshitters should be worried about their jobs. Hence Idothink salesmen, political speech writers, managers (upper and middle for different reasons) will be take in by claims and ought to afraid for their jobs (and good riddance)Reviewing something is either surprisingly hard (if you do it conscientiously) or unsurprisingly easy (if you don't care a damn and/or are ignorant). Evidence: much of the modern "social media". Bullshitters emit apparently plausible sequences of words, especially technobabble, without understanding the concepts. I don't think there's any need to provide examples; we've all seen far too many. others disagree, but to me salesmen represent what the company has produced to customers, whereas marketing people represent what the customers wan/need to the factory. Too many people conflate "marketing professional" with "very successful salesman"

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 06:17:44 pm
content: Here's the considered opinion ofsomeone who has founded two AI companiesand testified to the US Senate. He's not an amateur.https://spectrum.ieee.org/gary-marcusSee the article to find his references; they are revealing.Many of the points below are directly relevant to assertions made in this thread. The article contains indicating why he makes the statements.QuoteQuoteQuoteQuoteA question that often produces illuminating answers. If nothing would cause someone to change their mind, that person can be dismissed as an idiot/zealot/similar.QuoteOne of the articles referenced gives some useful positive and negative points, approaches and predictionshttps://garymarcus.substack.com/p/alphaproof-alphageometry-chatgptQuoteQuoteQuoteQuote

author: AndyC_772
author_rank: Super Contributor
timestamp: September 18, 2024, 08:50:10 pm
content: Quote from: tggzzz on September 18, 2024, 04:38:33 pmHere is exactly the chain of thought presented by ChatGPT -o1 preview when I asked the original question.Quote from: ChatGPTIt then goes on to present the comparison between the two parts. It's objectively wrong, as per my original post.Note the entirely reasonable sequence of steps (which should be the hard part), but also the outright factual errors in the specs of the two devices (which should be the easy bit).

author: tooki
author_rank: Super Contributor
timestamp: September 18, 2024, 09:07:49 pm
content: One of the biggest problems we are likely to face with generative AI is the training data itself: as more and more AI-generated chum gets published, a lot of that is going to end up being ingested as training data, causing a vicious cycle of hallucinations becoming "real" to the AI.I think I'll call this "data incest", since the end result is severe inbreeding in the data itself.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 09:17:54 pm
content: In my ignorance I would have guessed that the steps detailed are bog standard data collection and collation, and are therefore relatively easy. Deciding which sections are to be compared, and the comparisons themselves require far moe.understanding and judgement, and are therefore more difficult.Can I suggest using the quotes I provided in my earlier post as a prompt to reading Marcus' articles. I suspect you will begin to see how that the failings you've noted aren't surprising.The enumeration of the steps should not, of course, be confused with an explanation of how/why it "deduced" incorrect "facts". Such explanations are currently no more than a pipedream.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 09:19:20 pm
content: Quote from: tooki on September 18, 2024, 09:07:49 pmYes. Apparently research indicates it is already a problem.

author: tggzzz
author_rank: Super Contributor
timestamp: September 18, 2024, 11:40:00 pm
content: I don't know why OpenAI bothers..."OpenAI has been sending out warning emails and threats of bans to any user who tries to probe how the model works""Others say the warning is triggered simply by asking ChatGPT about the model's "reasoning" at all."https://www.wired.com/story/openai-threatens-bans-as-users-probe-o1-model/Maybe they just want to (try to) avoid people making fun of the LLM's unintentionally humourous/frightening spiel?

author: thm_w
author_rank: Super Contributor
timestamp: September 19, 2024, 08:10:33 pm
content: Quote from: daqq on September 17, 2024, 04:51:59 pmLCSC needs to use this to automatically update their parametric search, it would be 100x better than what it is now.

author: tggzzz
author_rank: Super Contributor
timestamp: September 20, 2024, 08:31:56 am
content: A few posts back there was a short discussion of whose jobs might be done by LLMs. Here's today's Alex cartoon, one of a series on the subject.Background: Alex is the FinTech equivalent of Dilbert, and has been running since the 80s. Clearly written by people with deep knowledge of the area, and almost certainly supplied with insider tips, like the scriptwriters for "Yes, (Prime) Minister" and "The West Wing"The Cyrus character is the American head of MegaBank. All characters are highly status conscious and profit oriented, and don't miss a chance of making someone else look/feel stupid. Bastards,allof them.Use the "previous strip" button to see earlier posts on LLMs.https://www.alexcartoon.com/index.cfm?cartoon_num=8629