The half a millimeter error that nearly cost 469 lives - Page 1
Forum section: electronics / general

author: NiHaoMike
author_rank: Super Contributor
timestamp: December 11, 2023, 11:59:32 pm
content: https://admiralcloudberg.medium.com/a-matter-of-millimeters-the-story-of-qantas-flight-32-bdaa62dc98e7I have heard of the near disaster before but the analysis of how the defect causing it happened was new to me. The most unbelievable part was how several loosening of tolerances were just accepted in the aerospace industry. I once worked at a place where any change to regression test code must be justified or it won't be accepted, none of which had to do with safety or security.

author: EPAIII
author_rank: Super Contributor
timestamp: December 15, 2023, 08:56:37 am
content: Fantastic read!A half mm is slightly less than 20 thousandths of an inch for those of you in the US or who otherwise aren't real familiar with metric measure. Or about the thickness of five sheets of ordinary copier paper. Not much to stand between life and death.In addition to working in electronics I have been a long time, amateur machinist. Even from my first forays in that realm and using amateur quality machines, i was able to easily make parts with a tolerance of a thousandth of an inch or 0.025mm. And can/could measure them to a ten thousandth of an inch or 0.0025mm. An error of half a mm may sound small but, in the reality of a modern machine shop, is actually enormous. And RR was using the very best available machines to make these assemblies and then to measure/inspect them. And that error was made by some of what are supposedly the best engineers and machinists in the business.I am not trying to sound like a pompous ass here. I too make mistakes, but I usually work alone with no back-up. It is just difficult to understand how so much can go wrong with so many experts involved. People who know far more about designing and building aircraft than I ever will. Perhaps when you work alone you know it is all on you. So you double and triple check everything. You ask questions and aren't afraid of the answers you may get either from yourself or from others. But when working as part of a large organization, there are so many who can be assumed to be backing you up that you get complacent. Is that it? Or are you afraid of admitting you are wrong? Or are afraid of being shouted down if you question things? I don't know.I can tell you one thing. The fact that no one was hurt or killed. And the fact that the plane survived and was restored to service tells you something about the process. It can't all be wrong. Perhaps it is a wonder that so little does go wrong with these monstrous projects. I am presently struggling with a project that has only about two dozen parts and a circuit as simple as a doorbell.

author: porter
author_rank: Regular Contributor
timestamp: December 15, 2023, 10:08:11 am
content: Here is an interesting article about the safety of the 737 max. Another look into design decisions and quality control.HOW THE BOEING 737 MAX DISASTER LOOKS TO A SOFTWARE DEVELOPERQuotehttps://spectrum.ieee.org/how-the-boeing-737-max-disaster-looks-to-a-software-developer

author: niconiconi
author_rank: Frequent Contributor
timestamp: December 15, 2023, 08:21:16 pm
content: Quote from: EPAIII on December 15, 2023, 08:56:37 amThe cheapest 2-layer circuit board process get you down to 10 mils traces, a reasonable one can make 6 mils, also that the dielectric layer between the power and ground in a modern 4-layer board is ~4 mils... So yeah, 20 mils is also a bad tolerance in the eyes of electronics workers - it's roughly the width of a power or ground trace to a logic chip on a circuit board.

author: johansen
author_rank: Super Contributor
timestamp: December 15, 2023, 09:42:35 pm
content: there are millions of other equally interesting failures waiting to happen.can you find them?

author: johansen
author_rank: Super Contributor
timestamp: December 15, 2023, 09:55:13 pm
content: Quote from: porter on December 15, 2023, 10:08:11 amsimple fix for this. hold the software developers liable for their criminal negligence.

author: SiliconWizard
author_rank: Super Contributor
timestamp: December 15, 2023, 10:03:08 pm
content: Quote from: johansen on December 15, 2023, 09:55:13 pmI don't disagree with that idea in theory, but in practice, as long as software developers will have subordinate relationships with their boss/employer (which is a majority of cases these days, especially in regulated fields), it's a Pandora's box. Any management screw-up will point fingers at some poor chap that may end up in jail, and the managers free and with a pay raise. Yes, I know it already happens occasionally, but without the "go to jail" part for the employee. So yeah, it would be much too easy to use this kind of law to twist liabilities.

author: coppice
author_rank: Super Contributor
timestamp: December 15, 2023, 10:14:39 pm
content: Quote from: SiliconWizard on December 15, 2023, 10:03:08 pmYep. The negligence is seldom attributable to one person. Some developers want to be thorough, and some don't, but very few are allowed the resources to be thorough. If management actually allows more resources, a lot of people relax and soak them up in an unproductive way. The more years you spend observing the dynamics of projects, the more amazing it is that anything bigger than a one person project ever gets finished and works well.

author: johansen
author_rank: Super Contributor
timestamp: December 15, 2023, 10:30:06 pm
content: Quote from: coppice on December 15, 2023, 10:14:39 pmthe negligence doesn't need to be one person.the epa iirc is now up to 12 million per person, that's what the gov thinks your life is worth. that's the number they use to determine if pollution is worth cleaning up. multiply by 469 deaths and you get a 2 billion dollar settlement. given that 12 mill per person is more than the cost to jail a person for the rest of their life... no need for the legal system to get involved. the company can pay the fine and keep working, its probably a drop in the bucket anyways.just make the companies pay for their negligence. if the cost of airplane flights go up.. so be it.oh and prosecute the FAA too. they signed off on it and allowed it to fly. oh my bad they are a gov agency. sorry, do not pass go do not collect 200$.

author: KE5FX
author_rank: Super Contributor
timestamp: December 15, 2023, 11:02:13 pm
content: Quote from: johansen on December 15, 2023, 10:30:06 pmIf the cost of airplane flights goes up, more people will drive, and you'll end up killing more people as a result.On a machine this complex, every knob you turn has interacting effects.

author: Black Phoenix
author_rank: Super Contributor
timestamp: December 16, 2023, 04:17:11 am
content: Quote from: KE5FX on December 15, 2023, 11:02:13 pmExactly. Whatever you do gravity will always win, what goes up must come down. That's the gist of the physics and we are still not that advance to rewrite physics as we known.Still I read a ton more of his articles and then cross referencing with the original investigations. It was a nice way to pass some boring evenings I had this week.

author: EPAIII
author_rank: Super Contributor
timestamp: December 16, 2023, 10:02:27 am
content: Quote from: johansen on December 15, 2023, 09:55:13 pmInteresting thought. But what about the managers who are providing the guidance and the PRESSURE to get it done yesterday?And no, I am not a software developer. But I have written some code, but nothing anywhere near the level that we are talking about here.

author: NiHaoMike
author_rank: Super Contributor
timestamp: December 16, 2023, 01:45:42 pm
content: Quote from: KE5FX on December 15, 2023, 11:02:13 pmThe US makes it too easy to get a driver's license, such that plenty who are driving shouldn't be driving. But that's for another topic.

author: porter
author_rank: Regular Contributor
timestamp: December 16, 2023, 01:50:39 pm
content: Some additional information regarding the 737 max software.QuoteBoeing’s Software Development Problemhttps://www.devteam.space/blog/boeings-software-development-problem/

author: tom66
author_rank: Super Contributor
timestamp: December 16, 2023, 03:18:55 pm
content: Talking about Swiss-cheese model, A330-300 flight CI202 operated by China Airways makes me a little nervous as to what bugs remain undiscovered.https://www.ttsb.gov.tw/media/4936/ci-202-final-report_english.pdfIn summary, a previously undiscovered timing bug between flight computers monitoring pilot rudder input led to a triplicate failure of flight systems (within a few seconds of each other) upon landing. However, it must be said, despite losing three FCs, the aircraft was still controllable and landed safely, though lost thrust reversers, the ground spoilers, and the autobraking meant the aircraft only had 10 metres of runway left to spare.

author: ejeffrey
author_rank: Super Contributor
timestamp: December 16, 2023, 07:17:02 pm
content: Quote from: EPAIII on December 15, 2023, 08:56:37 amBecause the issue wasn't with the accuracy of the machines per se, but the ability to locate a new machining pass relative to a previous reference surface which is buried inside a partly assembled subassembly. Ultimately it was a failure of communication between the design enginerrs the manufacturing engineers, the tool manufacturers, and the factory supervisors. It's exactly the type of problem that doesn't happen with single person operations.Presumably 99% of the original design was easily compatible with the available machining technology, but in a project this large 1% of hard to manufacture parts lead to either implementing extremely inefficient and potentially error prone manufacturing work arounds (as happened here) or seemingly endless redesign cycles with cascading impact.To phrase the issue mathematically, the native communications overhead in a project scales as O(N^2). Left to itself, a project with more than about 10-20 people will result in output decreasing, eventually to near zero. Bureaucracy and project management are the attempt to fix that and, at great cost, make communication overhead O(N). But it always has failures where the right person doesn't get the right information and have the authority to act on it.

author: coppice
author_rank: Super Contributor
timestamp: December 16, 2023, 08:47:55 pm
content: Quote from: johansen on December 15, 2023, 10:30:06 pmMaybe we should just hold the entire human race responsible for creating the culture in which these things happen. Come on. You know you're all to blame, wanting cheap transport, and putting convenience above everything else. Just cauterise the planet and be done with it.

author: harerod
author_rank: Frequent Contributor
timestamp: December 16, 2023, 10:39:44 pm
content: Quote from: NiHaoMike on December 11, 2023, 11:59:32 pmThanks for sharing this article. I may have flown on that very aircraft several times. The author is a bit too cheerful for my taste. Kudos to the crew and the overall aircraft design.While it is important to look at the whole chain of fuck-ups that led to this accident, it is important to note that we have a component manufacturer who altered design documents and failed to comply with mandatory procedure in well over a hundred different cases.Another wtf is the bit about a guy trying to figure out with a statistics tool (which he can't operate), how many defective items may be in service.My clients sometimes feel that I act too friendly towards manufacturers. However, any manufacturer that changes my design without my approval, will lose the contract. He may get away with one stunt like this, but only if he convinces me that he will make sure this will not repeat.To make that crystal clear: a manufacturer does not change designs. They give feedback to the designer, who may change the design.Edit: typo

author: watchmaker
author_rank: Unknown
timestamp: December 17, 2023, 01:32:04 pm
content: Eventually, the design comes off the computer and is executed. Most times this involves a fallible human being.Designs must be robust. The Ford Explorer/tire failure fiasco of 30 years ago made this crystal clear. Fatal rollovers because Firestone tires failed. Ford tried to blame it on the tires when the finding was that the vehicle was not designed to contend with a predictable (known) failure point. Tire failures.I had tires rotated at my dealership. Driving at 70 MPH there was a thump like a piston about to go through the block. Felt nothing in the car. Stopped, looked it over and found nothing. Continued on. Started going through a diagnosis with my wife. Engine mount? Nothing I could do about that.Speed related. Brake issue? No obvious change in braking. Gets worse on right hand turns? Bearing, axle? Get real loud!Pull over and start looking for nearby tows. Wife looks at wheels and notices two missing lugnuts on front left.The other three were hanging on by less than 25% thread. Tightened them down and went on my way.Robust design. In fact, it would hold with one lug nut.What scares me is I felt NOTHING thru the wheel. It is essentially steer by wire these days.There are several parables here. It took us about 15 minutes to identify the problem and while that was too long, we are reasonably intelligent. In my defense, we were in a remote area (White Mountain National Forest) and the weather was getting lousy. An hour from home. Nothing was obviously explainable so we decided to limp along.For completeness, in the US there is no liability for this unless we had actually rolled over. Courts have ruled that properly tightened lugnuts are not guaranteed. So I have a socket and wrench in the cars for checking them from now on.I calmly asked the dealer to replace the lugnuts and suggested he turn this into a teachable moment for the mechanic (young). How would he feel when a fatal accident was traced back to him? Very cheap life lesson for the young mechanic.Dewey

author: RJSV
author_rank: Super Contributor
timestamp: December 17, 2023, 09:18:16 pm
content: Ok but while reading this I see a logical disconnect:One post starts by presenting aircraft safety but from a software developer's perspective. That could maybe involve some (natural) degree of ignorance, but that could be often remedied...at least in the traditional engineering office setting, meaning that the software folks get daily exposure to aeroscience issues.That's fine, but then another post professes the remedy to be a 'get tough' blanket policy...that don't work, usually, you can't legislate or 'rule-make' to get rid of ignorance.

author: johansen
author_rank: Super Contributor
timestamp: December 17, 2023, 09:46:48 pm
content: Quote from: RJSV on December 17, 2023, 09:18:16 pmthe oversite on how the welding and drilling procedure works, i can understand that. attempting to fix the problem by.. not actually re-inspecting every engine?... hmmm.. sounds like the usual problems that end up screwing you over. no, you can't legislate your way out of that, but you can actually hold people accountable. every time i make a repair to a vehicle (and i work on a friends company's cars), i expect to get sued if the component fails, is found to be my fault, and someone gets hurt.designing software to rely on a single sensor, with controls strong enough to override the pilot? fuck that. fuck the FAA and their laziness at validating their delegation of authority to make decisions for them.there is very little actual ignorance left in most airplane crashes these days.one of the recently in the pnw was a lock ring that was forgotten. the elevator trim assembly unscrewed all the way (vibration will do that) and the plane dive bombed right into the ocean. no redundancy on the control cables. hmmm... now who actually forgot the lock ring? we probably don't know.