Tesla’s Autopilot disengages itself before crash? - Page 1
Forum section: electronics / general

author: madires
author_rank: Super Contributor
timestamp: March 19, 2025, 04:29:24 pm
content: Tesla fans expose Tesla’s own shadiness in attempt to defend Autopilot crash,https://electrek.co/2025/03/17/tesla-fans-exposes-shadiness-defend-autopilot-crash/

author: mtwieg
author_rank: Frequent Contributor
timestamp: March 24, 2025, 12:38:10 pm
content: Pretty sure there have been many confirmed examples of Autopilot/FSD disengaging just before incidents occurs over the years, not sure why anyone would be surprised at this point.The question of why this occurs is more speculative. There may be valid reasons to disable Autopilot in the case of an imminent crash. Tesla probably doesn't want to get caught in trolley problem scenarios (do I plow into the clearly visible fire engine 50m ahead of me, or swerve into the Subaru next to me?). But the fact that it apparently doesn't even attempt to brake at all is basically indefensible.Reminds me of these tests with child-size dummies from 2022:

author: jbb
author_rank: Super Contributor
timestamp: March 24, 2025, 08:33:26 pm
content: Personally, I wouldn’t use a self driving system that may request I take over without warning. I know that it would be hard for me to keep focus on the road when not actively driving; to then get control handed back to mewhen something is already wrongseems like A Bad Plan.On the videos: there’s a chance it’s ‘natural’ behaviour from the autopilot system; I imagine there must be some kind of ‘confidence’ score generated by the software, and if it goes low enough then human driving is required. So the car realises something isn’t quite right and requests manual control quite late in the scenario.On automatic emergency braking; I imagine that the Tesla engineers are trying really hard on this front. Ideally, they want emergency braking to be both sensitive - ie always brake in a real emergency - and selective - ie never brake when it’s not an emergency. It’s a trade off. Especially when you consider that some drivers tailgate.

author: tggzzz
author_rank: Super Contributor
timestamp: March 24, 2025, 09:01:05 pm
content: I would disengage such a system, too.So do many othershttps://www.which.co.uk/policy-and-insight/article/car-safety-tech-driving-motorists-round-the-bend-which-finds-avra32B7heXH

author: thm_w
author_rank: Super Contributor
timestamp: March 24, 2025, 11:58:18 pm
content: Discussed to death on other sites and videos but:- Autopilot and FSD can disengage before a crash, that is normal behavior. If the crash happens within 5s of the disengagement, its still recorded as a fault of the software.There is another guy that showed HW3 vehicle would have crashed into a fake image and HW4 did not. Maybe due to the extra detail and processing power, or maybe just just the angle of the light at the time of filming. Its not clear:https://www.youtube.com/watch?v=9KyIWpAevNsThe video would have been fine if Rober didn't misrepresent stuff up front, eg state specifically "I'm testing Autopilot on an older HW3 car for these examples", the implication was it was the latest and greatest which it is not. Autopilot is probably 5 years behind tech wise. They likely don't care about updating autopilot abilities, a free feature, because it will just eat into their FSD sales.Quote from: tggzzz on March 24, 2025, 09:01:05 pmNot really that related, you turn on autopilot as cruise control, its not on by default. Useful for long drives or slow moving traffic.

author: tggzzz
author_rank: Super Contributor
timestamp: March 25, 2025, 11:10:10 am
content: Quote from: thm_w on March 24, 2025, 11:58:18 pmDid you read the article? It doesn't mention cruise control, but mentions ADAS systems thatmodifywhat the car is doing, i.e. change direction dangerously, change speed unpredictably and for no reasondistract the driverat critical moments (my experience in a hire van), i.e. blind spot monitoring. Drowsiness monitoring and alerting may well have similar problems

author: Marco
author_rank: Super Contributor
timestamp: March 25, 2025, 11:33:00 am
content: Quote from: tggzzz on March 25, 2025, 11:10:10 amWhich describes level 2 in every car it is in.

author: thm_w
author_rank: Super Contributor
timestamp: March 25, 2025, 10:11:56 pm
content: Quote from: tggzzz on March 25, 2025, 11:10:10 amThis thread is more about autopilot/FSD specifically in a crash scenario, not ADAS in general.But as jbb says, its a tough balance with auto braking, to guarantee that you never get a false positive and yet always brake in emergency scenarios.They've seemingly almost eliminated it from the latest FSD versions, though, there were a few situations where the driver had to take over or the car would have crashed.I just don't see it being possible, at this time, without additional sensors.

author: tggzzz
author_rank: Super Contributor
timestamp: March 25, 2025, 11:24:46 pm
content: Quote from: thm_w on March 25, 2025, 10:11:56 pmIf companies can't get nominally simpler systems working reliably, they have no chance of.getting a more complex system working reliably, e.g. Tesla's FSD-beta.If you think Tesla FSD is remotely trustworthy, then you haven't been paying attention to the wealth of experiences to the contrary that have been well documented.Now that Musk is best pals with President Chamberlain, expect all the US Transport Department investigations to be terminated before they can produce a report. If they do produce a report, expect it to be removed from all web sites and other records. I know that is implausible and ridiculous, but that is what is happening in many spheres in the US this year.

author: thm_w
author_rank: Super Contributor
timestamp: March 25, 2025, 11:59:24 pm
content: Quote from: tggzzz on March 25, 2025, 11:24:46 pmNot sure where I said or implied that it was trustworthy, just that v13 is a large improvement over autopilot, and seems to no longer have phantom braking issues while also reducing number of interventions. You can find example videos of hundreds of miles of driving with no interventions.Of course it still does dangerous and dumb things regularly, which is why its level 2, human must be able to take over. Though, I'd take FSD as a pedestrian over most human drivers, it seems fairly cautious and always aware of where people are walking.

author: tggzzz
author_rank: Super Contributor
timestamp: March 26, 2025, 12:39:21 am
content: Quote from: thm_w on March 25, 2025, 11:59:24 pmThat is, of course, irrelevant. There are hundreds of photos of me not snapping my tendon, and of me remembering to lower the undercarriage before landing.QuoteIt reliably fails to recognise children - or at least objects that look like and move like children. The ethics of putting a child in a Tesla FSD test are highly suspectFFI see the videos athttps://dawnproject.com/The problem with LLM/NN/ML systems is that they cannot be designed. They can merely be trained and tested. Hence it is not possible topredictwhat decision they will make in a given circumstance. The examples of that problem are legion, going back to Igor Aleksander's WISARD in the early 1980s. Regrettably youngsters don't remember history, and so repeat mistakes.

author: m98
author_rank: Frequent Contributor
timestamp: March 26, 2025, 02:50:55 am
content: Dan O’Dowd, someone whose marketing claims are theoretically impossible, is not to be taken seriously.Autonomous cars and robots will necessarily use neural networks unless someone comes up with a fundamental revolution, as current rule-based algorithms are entirely unsuitable to work in diverse, real-world environments.

author: tggzzz
author_rank: Super Contributor
timestamp: March 26, 2025, 10:12:44 am
content: Quote from: m98 on March 26, 2025, 02:50:55 amNo, and no.The videos have to be take seriously, as do the reports that have lead to multiple investigations of the cars' misbehaviour. If you don't like those specific videos, have a look on yootoob; there are many examples there.As for marketing claims, Must is the one who makes excessive claims - and is being investigated for them.Dan O'Dowd has serious credibility w.r.t. software. His safety critical software products have made him a billionaire, which means Musk cannot use legal threats to silence him.Just because no other technique works doesn't mean that NN works, can ever work, should be used or must be used. Sometimes tasks are just not possible with the available technology.

author: Marco
author_rank: Super Contributor
timestamp: March 26, 2025, 12:03:49 pm
content: Quote from: m98 on March 26, 2025, 02:50:55 amAll of the current systems have an outer layer of rule based algorithms doing boundary checks, some of which with nearly raw sensor data.They are indeed unsuitable for diverse real-world environments. Which is why Waymo cars have remote operators which can push it through the safety checks. It is also why the only level 3 self driving cars can only follow the duckling in slow traffic on highways.This is unlikely to change until AI decides to make decisions for us.

author: m98
author_rank: Frequent Contributor
timestamp: March 26, 2025, 02:11:37 pm
content: Quote from: tggzzz on March 26, 2025, 10:12:44 amAre you living in some parallel reality where we aren't already massively utilizing NNs and the halting problem was solved by Dan O'Dowd?Demanding a system to be free of failure modes is simply something that is fundamentally impossible.

author: Marco
author_rank: Super Contributor
timestamp: March 26, 2025, 03:16:39 pm
content: It doesn't have to be free of failures, it has to present acceptable liability to beancounters. It won't and self driving car companies are unlikely to get handouts to wave the liability.Slow urban driving in LIDAR mapped areas with frequent stops and human handhold requests and follow the duckling driving in highway jams is the best we'll get.

author: tggzzz
author_rank: Super Contributor
timestamp: March 26, 2025, 05:33:30 pm
content: Quote from: m98 on March 26, 2025, 02:11:37 pmWrong, irrelevant and a strawman argument.Now please try to address the the other points. Here they are again, for ease of reference...Quote from: tggzzz on March 26, 2025, 10:12:44 am

author: Marco
author_rank: Super Contributor
timestamp: March 26, 2025, 10:24:28 pm
content: Quote from: tggzzz on March 26, 2025, 10:12:44 amHe's got some profitable defence contracts, but he does oversell himself a bit."Never Fails and Can't Be Hacked".

author: thm_w
author_rank: Super Contributor
timestamp: March 26, 2025, 10:55:34 pm
content: Quote from: tggzzz on March 26, 2025, 12:39:21 amOf course its relevant to the topic at hand, Robers test.Either try to stick to that main discussion point, or continue to irritate people by pushing discussion off topic.QuoteHere is an older v12.5 version where it stops and sees the children OK (current version is 13, pointing out flaws in older versions isn't too relevant):2:00 inIt doesn't "reliably fail to recognize children" here.Does that mean I would trust it to drive through a school zone unsupervised? no not even close.

author: tggzzz
author_rank: Super Contributor
timestamp: March 26, 2025, 10:57:52 pm
content: Quote from: Marco on March 26, 2025, 10:24:28 pmA buffer overflowpossibilitydoesn't mean somethinghasbeen hacked, and theremightbe other defense mechanisms to prevent that causing a problem. I suspect ithasn'tbeen hacked, but personally - without knowing more details - I'd be unhappy about claiming it can't be hacked. Nonetheless, O'Dowddoeshave serious credibility in the software field. Note that O'Dowd has Teslas and likes them (apart from the FSD-beta, of course). He has nothing (significant) to gain/lose by forcing Tesla to acknowledge the FSB-beta's misbehaviour.More importantly, Tesla's FSD-beta is demonstrably defective, despite Musk's wild claims. Only a fanbois, paid shill, or similar could attempt to claim otherwise.Personally I have serious doubts that it can be made reliable. The problems appear to stem from the NN/ML system classifying some groups of pixels as "immovable background that can be ignored". If that filter is made too loose then too many background objects will have to be actively avoided (e.g. braking for non-existent crossroads). If that filter is made too tight then relevant objects will be ignored (there are many examples of that happening, from children to parked lorries, school busses, emergency vehicles stopped at the roadside).AIUI, other manufacturers don't rely solely on cameras, but have other sensors (e.g. LIDAR) that have complementary advantages and disadvantages. Sensor fusion is, in general, a good technique.

author: tggzzz
author_rank: Super Contributor
timestamp: March 26, 2025, 11:10:04 pm
content: Quote from: thm_w on March 26, 2025, 10:55:34 pmAre youseriouslysuggesting that because it notices a child in one test that it doesn't ignore them in others?!Show us the design equations (or equivalent) that will now prevent it ignoring children. That's impossible, of course, since there are no design equations or principles - merely a few tests. (Few relative to the number of situations that will be encountered in the real world).What's to prevent the new system failing in ways that the old system didn't? Nothing, because there is no design, merely tests.What's to prevent the next update reverting to previous bad behaviour? Nothing.Remember the old engineering adage "You can't inspect quality into a product; it has to be designed in". Modern software weenies seriously believe that their product worksbecauseit passes the tests. They have difficulty comprehending that if the tests are inadequate then all bets are off. Idiocracy in action

author: vk6zgo
author_rank: Super Contributor
timestamp: March 26, 2025, 11:59:53 pm
content: Many years ago the Public Works Dept in Western Australia decided it would be a good idea to put a reflective warning on the rear of their "utes" so that people wouldn't run into the rear of them.To that end, they used a smaller version of the ones used to warn drivers on a terminating road that they were approaching a "T" junction, so that they didn't happily continue & sail off into the bush at night.Following a PWD "ute" one night, as we both slowed, suddenly, the 'ute' disappeared, as its warning pattern lined up perfectly with that on the other side of the "T" junction.The reflections were so strong that the tail lights didn't spoil the illusion, until he applied his brakes.I'm not sure what "FSD" would make of that, ---maybe not having the instinctive pattern recognition of a human brain, its "dumber" reaction may have not been fooled, but maybe it still would have.A human can see the illusion, but also know that cars don't just disappear, perhaps a machine may assume that the original "ute" was a false image.

author: Andy Chee
author_rank: Super Contributor
timestamp: March 27, 2025, 05:20:00 am
content: FWIW, Mark Rober's contribution to the issue

author: tggzzz
author_rank: Super Contributor
timestamp: March 27, 2025, 10:43:49 am
content: Quote from: vk6zgo on March 26, 2025, 11:59:53 pm- and concerning in some real-life situations.Ah yes, illusions.NN/ML vision systems are known to have significant issues, e.g. changing one pixel completely changing the interpretation. Having said that, human vision isfarfrom perfect; some of the "dots" and "motion induced" illusions here are astounding:https://michaelbach.de/ot/index.htmlI wonder how NN/ML vision copes with contrast reversal, where white road line appear black (after shower, sun reflecting off water film), and I'm sure there are others.I also wonder how students will "prank" NN/ML systems, in the vein of putting traffic cones to divert all traffic through the campus car park. I'd start with painting a pothole on the roadAny such susceptibility would be highly specific, so it might be possible to confuse just one car one brand.One day I'm going to go to a local Tesla saleroom, pretend to be interested, and have a test drive with the salesman in control using FSD-beta. I'd get him to go to my house, along a motorway and narrow roads with potholes, missing verges and blind bends.

author: tggzzz
author_rank: Super Contributor
timestamp: March 27, 2025, 11:14:02 am
content: Quote from: Andy Chee on March 27, 2025, 05:20:00 amI haven't seen that before; thanks. It is a good use of yootoob: short andrequiresmoving picturesThe only quibble I would have is that I don't think the "painted wall" illusion is the most damning. I'm sure similar things will exist, but they will be much rarer than the other everyday problems demonstrated.Real world example on a highway in TaiwanI've seen similar videos of a Teslas hitting an inflatable lorry parked on a runway. Why onearthdid the Tesla NN/ML decide to ignore it?Amusing Tesla accessory, which illustrates the company's attitudes:https://electrek.co/2023/06/30/tesla-gives-new-owners-uk-grabbing-stick-after-forcing-left-hand-drive/I suspect "Consumer Reports" is the US version of "Which?".https://www.consumerreports.org/cars/car-safety/tesla-autopilot-recall-fix-does-not-address-safety-problems-a5133751100/

author: Marco
author_rank: Super Contributor
timestamp: March 28, 2025, 04:17:57 pm
content: Quote from: tggzzz on March 26, 2025, 10:57:52 pmPlenty of level 2 cars without LIDAR and that's all Teslas are for the moment, not robotaxis nor level 3+.I don't think that LIDAR is that big a deal, it's slightly less prone to ambiguity than monocular depth from video ... but only slightly and it's still dependent on visible light. Stereo cameras + imaging 4D radar is what I would spend money on in a sensor package, LIDAR is of too little worth.

author: Marco
author_rank: Super Contributor
timestamp: March 28, 2025, 04:51:52 pm
content: Quote from: tggzzz on March 27, 2025, 11:14:02 amThe AEB test uses a small target. Something that huge will screw with monocular depth from video algorithms, traditional radar wouldn't help either.

author: mtwieg
author_rank: Frequent Contributor
timestamp: March 29, 2025, 12:47:50 pm
content: Quote from: Marco on March 28, 2025, 04:17:57 pmI've never heard of a LIDAR system using visible wavelengths. AFAIK everything is 900-1600nm.QuotePretty sure "4D" is just a marketing buzzword which doesn't coherently describe any aspect of the technology.Quote from: Marco on March 28, 2025, 04:51:52 pmAFAIK the vast majority of vehicles (basically everyone except Tesla) use LIDAR or radar for this sort of application, what in the world are you talking about? Do you actually think LIDAR/radar would have more trouble detecting a giant object than a camera?

author: Marco
author_rank: Super Contributor
timestamp: March 29, 2025, 04:47:12 pm
content: Quote from: mtwieg on March 29, 2025, 12:47:50 pmTraditional car radar ignores stationary objects, because the resolution is too poor to see anything without huge stationary objects in it (the road for one).Only the fancy double digit GHz imaging radars can "see" a stationary object, in marketing terms 4D radar. 4D means it creates a 2D image with depth and velocity, so 4 dimensions.Mobileye's solutions are mainly single camera based too for instance and without LIDAR. Lots of AEB systems will have similar failure modes.

author: thm_w
author_rank: Super Contributor
timestamp: March 31, 2025, 09:34:05 pm
content: Here is a recreation of Robers test showing HW4 fairly easily detects the fake wall:So as I noted, Rober video was misleading as it tested old tech (HW3 released in 2019) against new, without stating so clearly.This same guy is going to test the other scenarios and see if they fair better as well.

author: vk6zgo
author_rank: Super Contributor
timestamp: March 31, 2025, 10:39:20 pm
content: Quote from: thm_w on March 31, 2025, 09:34:05 pmAll very nice, but Tesla was touting the earlier system as "Fully Self Driving" back in 2019.Had it not been released until now, with the newest version, & spent the last 6 years testing the different iteration, nobody would have an argument."Move fast & break things" was never intended to include human bodies!

author: thm_w
author_rank: Super Contributor
timestamp: March 31, 2025, 11:45:08 pm
content: Quote from: vk6zgo on March 31, 2025, 10:39:20 pmNo one here would deny that. Their claims have always been BS, ignore that and look at the data, test and compare the latest tech.If its not clear who is responsible:Quotehttps://static.nhtsa.gov/odi/rcl/2023/RCLRPT-23V085-3451.PDF

author: tggzzz
author_rank: Super Contributor
timestamp: March 31, 2025, 11:58:02 pm
content: Quote from: thm_w on March 31, 2025, 11:45:08 pmMusk/Tesla/salesmen are responsible for making vague "aspirational" claims that they could not support - and which a lot of gullible people believed. Drivers died as a result. Worse: innocent third parties that happened to be in the vicinity were injured."Fool me once shame on you. Fool me twice shame on me".