don't forget to reboot your boeing, or else - Page 1
Forum section: electronics / general

author: coppercone2
author_rank: Super Contributor
timestamp: April 02, 2020, 11:50:28 pm
content: https://tech.slashdot.org/story/20/04/02/206230/boeing-787s-must-be-turned-off-and-on-every-51-days-to-prevent-misleading-data-being-shown-to-pilotsSimilar run time to windows ME?its designed like the stations in the series Lost. Don't forget to press the button or else......

author: Halcyon
author_rank: Unknown
timestamp: April 03, 2020, 01:12:30 am
content: Reminds me more of Windows 10. Every week or so, I have to reboot my workstations at work because randomly, the USB ports stop working.

author: Ampera
author_rank: Super Contributor
timestamp: April 03, 2020, 02:32:52 am
content: ha ha yes kick the windows man in the gutter ha ha

author: floobydust
author_rank: Super Contributor
timestamp: April 03, 2020, 02:36:58 am
content: Is it the usual timer overflow somewhere, or maybe a memory leak? It doesn't happen during flight, right?Boeing outsourcing their flight control software saved so many important dollars. It also gives many new grads entry to aerospace, provides backdoor cash flow to important customers in developing countries. The MBA strategic management strategy is pure genius.

author: Monkeh
author_rank: Super Contributor
timestamp: April 03, 2020, 02:41:50 am
content: Quote from: floobydust on April 03, 2020, 02:36:58 amYou're not generally interested in airspeed, attitude, or altitude when not in flight, so.. yes, it happens in flight.Boeing, ladies and gentlemen..

author: JustMeHere
author_rank: Frequent Contributor
timestamp: April 03, 2020, 03:31:10 am
content: I was on an Airbus not too long ago. The crew was late. The co-pilot got there first, boarded and turned on the airplane so the air would circulate. It took most of an hour for the rest of the crew to get there. They went to push back and stopped.Pilot: We are sorry for the delay you already have been through, but we have a warning light and will have to be here longer.Pilot: Well the manual says to reboot the airplane. This will take about 5 minuets.

author: Monkeh
author_rank: Super Contributor
timestamp: April 03, 2020, 03:32:18 am
content: Quote from: JustMeHere on April 03, 2020, 03:31:10 amWell at least the Airbus tells you when it needs to reboot instead of trying to fly you into a mountain!

author: JustMeHere
author_rank: Frequent Contributor
timestamp: April 03, 2020, 03:36:19 am
content: Quote from: Monkeh on April 03, 2020, 02:41:50 amDon't worry, the do turn the airplanes off at night.

author: floobydust
author_rank: Super Contributor
timestamp: April 03, 2020, 05:43:37 am
content: "The FAA has received a report indicating that the stale-data monitoring function of CCS {common core system (CCS) – a Wind River VxWorks realtime OS product} may be lost when continuously powered on for 51 days. This could lead to undetected or unannunciated loss of CDN message age validation, combined with a CDN switch failure. The CDN handles all the flight-critical data (including airspeed, altitude, attitude, and engine operation), and several potentially catastrophic failure scenarios can result from this situation. Potential consequences include:• Display of misleading primary attitude data for both pilots.• Display of misleading altitude on both pilots' primary flight displays (PFDs).• Display of misleading airspeed data on both pilots' PFDs, without annunciation of failure, coupled with the loss of stall warning, or over-speed warning.• Display of misleading engine operating indications on both engines.The potential loss of the stale-data monitoring function of the CCS when continuously powered on for 51 days, if not addressed, could result in erroneous flight-critical data being routed and displayed as valid data, which could reduce the ability of the flightcrew to maintain the safe flight and landing of the airplane. "https://ad.easa.europa.eu/ad/US-2020-06-14"Persistent or unfiltered stale data is a known 787 problem. In 2014 a Japan Airlines 787 caught fire because of the (entirely separate, and since fixed) lithium-ion battery problem. Investigators realised the black boxes had been recording false information, hampering their task, because they were falsely accepting stale old data as up-to-the-second real inputs.""... another 787 stale data problem in years gone by saw superseded backup flight plans persisting in standby navigation computers, and activating occasionally. Activation caused the autopilot to wrongly decide it was halfway through flying a previous journey – and manoeuvre to regain the "correct" flight path."https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/No wonder Boeing took a charge of $410M for a re-do test of the Starliner."... Mr. Loverro said that the problems were not a consequence of the fixed-price structure of the SpaceX and Boeing contracts."I think it was the way we chose to manage it,"Mr. Loverro said."

author: Electro Detective
author_rank: Super Contributor
timestamp: April 03, 2020, 07:59:51 am
content: or they could just swap out the BIOS backup battery every six monthsinstead of waiting for the Black Box to FYI the investigatorsToo easy, right?

author: MK14
author_rank: Super Contributor
timestamp: April 03, 2020, 08:17:35 am
content: Quote from: blueskull on April 03, 2020, 03:41:47 amI heard that, from another source. But the rumour goes further.It uses the 1024/1000 (because some things do it that way, when converting) concept.So 49.7 x 1024/1000 = 50.9 = A lot closer to the 51 days.I.e. The 1 millisecond, may be a value (such as 1.024 ms per tick), so that rapid bit shifts can be used in calculations, rather than much slower full divides. Or some similar reason, why it is 1024/1000.N.B. I didn't write the software for them. So can only guess at why it is 1024/1000. Which it could easily NOT be, anyway.Anyway, I think it is completely crazy that they have to do that. So, if they miscount the 51 days (e.g. leap year), does the plane crash into a mountain and kill everyone on board ?Why can't they fix the software, properly ?(Yes, it would be expensive, but it seems a bit risky to leave a software bug like that, in the airplane).Such problems, shouldn't of been designed into the plane in the first place and also, should have been found, in their careful tests and analysis of the safety critical software, on the planes systems.This doesn't give me confidence, in their software.

author: Yansi
author_rank: Super Contributor
timestamp: April 03, 2020, 08:22:31 am
content: Quote from: Halcyon on April 03, 2020, 01:12:30 amWau! And I thought it was only my computer that does it! Insane crap...

author: Electro Detective
author_rank: Super Contributor
timestamp: April 03, 2020, 09:21:01 am
content: "Windows has deactivated the USB socketsto protect your computer due to user failure to update security on a regular basis.If you want functionality restored and not see this message again,YOU(expletive)KNOW WHAT TO(expletive)DO !or visitwww.obeymsknowsbest.comHave a nice day"

author: senso
author_rank: Frequent Contributor
timestamp: April 03, 2020, 09:31:02 am
content: Quote from: MK14 on April 03, 2020, 08:17:35 amDue to certification, they would have to re-test all the software if its in fact a tick timer that will be used almost everywhere in the code..And that is expensive, and the CxO can't go home without their 10+ million performance bonus..

author: MK14
author_rank: Super Contributor
timestamp: April 03, 2020, 09:50:09 am
content: Quote from: senso on April 03, 2020, 09:31:02 amBut the air standards authorities, should INSIST, that it is done properly.If there was a component, in the planes engine, which wore out every 51 days. The engine would explode and crash the plane, if they forgot to replace it.I suspect, they (air standards authorities), would insist that the planes are recalled, and the "faulty" short life parts are redesigned, to last the proper/expected life expectancies (e.g. life of the engines).Software, should be the same.E.g. The planes will likely be in storage or unused, for LONGER than 51 days, because of the virus. So, if it had been left on (e.g. some planes kept on, in standby (or whatever it is called, in plane jargon), for various reasons, so it can be ready quickly. I heard this a long time ago. It usually applies to military planes, especially), and needed to be rebooted. It could just take one plane, where they forget to reboot (or similar mishap), and it could spell disaster.In this case (extended coronavirus long term shutdown, many weeks or months), the aircraft would probably be shutdown/off.

author: SeanB
author_rank: Super Contributor
timestamp: April 03, 2020, 09:52:29 am
content: A lot of the flight subsystems are rerely turned off, and are running all the time between services, and often also during service pperiods, as the plane will be powered from an external power unit then. Reboot of some systems can be fast, because it is a warm reboot, but powering down, and doing a full restart can take hours, as you have to bring things back up in sequence, so you do things like first connect the main battery, then turn on the GPS and wait for it to acquire lock, so your main data bus systems will have a stable clock and accurate time data. Then you can start powering up the systems in sequence, such that each one communicates with the units it depends on to get valid inputs. finally you can power up the FMS and displays, which is the top layer the pilots see.That at least reboots fast, but only if the underlying systems are running, without having exceeded the abilities of the redundant subsystems. Things will fail all the time, and there are a whole lot of fault trees that you follow, to see if you can continue to destination without degradation, continue with degradation, divert to alternate airport soon, land as soon as possible, land immediately or " kiss your backside goodbye, because that is it folks", depending on the fault and just how many systems it affects. Then you get mechanical failures, with the same fault trees, but at least you can hope the electronics is going to give you a true indication of the system status.51 days of uptime then it breaks is a symptom of poor testing, just like the engine management turning off the alternators after 250 days of uptime. Both of which are common to have, as those systems run all the time, even with the plane on the ground in a standby mode. Likely to have been introduced as a consequence of another software bug being fixed some time ago, or to be a result of errors in interpretation of inputs, or validation of parameters, or just insufficient testing of all possible program paths.You might turn off the lighting on the ground, and also turn off the APU, and have a cockpit that is totally black, but there are still computer systems running to enable you to turn on a switch, and have the lights come on, and it to warm start in the morning, but the uptime on the computers will be that since the last time the battery was disconnected along with the external power connections.Only good thing is all the grounded 737MAX aircraft will have been parked with the main battery breakers turned off in the avionics bays, so as not to overdischarge the batteries, or they were removed so they could be put on maintenance charge schedule. They will be fine for the first 51 days after they fly again, at least for this bug, and perhaps even MCAS issues will be properly fixed as well.I will add that military aircraft rarely are powered for long times, as they have in general a much more frequent service schedule, due to parts being run much closer to the limit as compared to commercial aircraft, where they want parts to do long periods between services, and even longer between overhauls. Thus your jet engine might run 90 days between service intervals, or 500 running hours, while your military version of the same engine will be monthly, or 200 hours, due to the higher demands placed on it. Even in hot standby the same aircraft will rarely be run for more than a week continuously, before another is swapped in to replace it, while the first goes for service. In a situation of active military use though they might just be run till they break, but that is where you find that they are also going to be spares part sources quickly, to keep others running.Longest I saw was 2 weeks of continuous operation, in use 8 hours a day, with hot swapping pilots, because they were down to only having one aircraft available, the others being not available for various reasons, and the pilots needed the hours to keep flying. It did not break, though the engine came in right afterwards, well past the service hours count. I had no faults off it either, the avoinics i was responsible for worked correctly.

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: April 03, 2020, 08:46:48 pm
content: Quote from: MK14 on April 03, 2020, 08:17:35 amI bet it just counts microseconds:2^42 microseconds lines up with the time reportedMaybe a 10-bitsub-secondmicrosecondcounter and a 32-bitsecondmillisecondcounterThe sub-second counter could be shifted and masked from a hardware cycle counterLess likely, the system adds one tick's worth of microseconds each tickLike OP, I also immediately thought ofthe old Windows 49.7 day crash.Edit: In my defence,it made sense in my headIamlegally retarded.

author: MK14
author_rank: Super Contributor
timestamp: April 04, 2020, 12:29:05 am
content: Quote from: I wanted a rude username on April 03, 2020, 08:46:48 pmYour maths, doesn't seem to add up.Because a 10 bit divider, would be milliseconds, approximately, NOT microseconds. 10 bit = 1024.Quote from: I wanted a rude username on April 03, 2020, 08:46:48 pmI'm NOT quite understanding your maths, here.2^32 seconds = Over 136 Years!But, anyway. Further research seems to indicate that the software was actually fixed, later (maybe the 51 days, was ONLY until the software was fixed and recertified).The OS they seem to be reportedly using, VxWorks, seems to use 50 or 60Hz, as the Timer Tick.= 16.666ms (60Hz) approx. But it does seem to mention something about hardware timer resolution can be different.So, I'm not really sure how the 51 days comes about.

author: coppercone2
author_rank: Super Contributor
timestamp: April 04, 2020, 02:53:57 am
content: it just could be a sum of data samples that fills up a memory space from multiple sensors or something (does not make sense to sample some sensors at ultra high frequencies probably). Maybe it makes a SOA from different sensors data inputs.

author: MathWizard
author_rank: Super Contributor
timestamp: April 04, 2020, 03:06:12 am
content: Boeing, what an enigma. They make bombs to kill 3rd world people, to help make rich people richer, and they make nice planes, overall.

author: MK14
author_rank: Super Contributor
timestamp: April 04, 2020, 03:11:31 am
content: Quote from: coppercone2 on April 04, 2020, 02:53:57 am(To avoid confusion, I'm NOT attacking your post. Just describing the difficulty of trying to determine someone else's mistake/bug, when the full information is apparently not available).Yes, we could speculate, on the millions of ways, they could have messed up. But, it is like finding a needle in a haystack.Although I've not seen the precise details. There is some details on the specifics of the bug, floating around.Quotehttps://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/

author: duak
author_rank: Super Contributor
timestamp: April 04, 2020, 07:23:38 pm
content: One company I worked at had a number of aviation buffs. One of them said something interesting about passenger aircraft - I don't know if it's factually true or just truthy. He said that McDonnell-Douglas had two hydraulic systems, Lockheed had three and Boeing had four. This happened at a time when M-D had a number of spectacular failures in the DC-10. BTW, he worked for McDonnell before they were merged with Douglas. This probably contributed to me being more confident in Boeing.I've flown about 75 times domestically and overseas over the years. The most exciting mechanical failure was a hydraulic system crapout in a Fokker. The crew had to crank down the gear, and we circled until the emergency team got in place. It took a bit longer to come to a stop but if you didn't know something was up, you'd wonder what they were worried about. It was much more exciting to land in Jakarta, Indonesia during a thunderstorm in a DC-10 or the dogleg final approach at Hong Kong's old Kai Tak airport in a 747. A crosswind landing at SFO will also wake you up. I don't recall having been in a missed approach.In 1997, my wife and I flew from Seattle to LAX in a newish Alaskan Airlines 737. We were cleared to go, the engines spooled up then just spooled down. No bangs or funny noises. A few seconds later the engines spooled up then just spooled down again. Then a long pregnant pause before the captain came on and said we'd head back to the gate. About an hour later, the captain came on and said the tech had reset the computer a few times but still had a problem so the computer was changed out. I remember something a colleague said when asked why he didn't like to fly. "Planes have computers with firmware in them". BTW, both he and I designed hardware and wrote the firmware. I suppose I was thinking about the fallibility of computers and whether this was a warning to get off the aircraft. The flight was uneventful, but I wouldn't have been surprised if the engines decided to spool down and we had to glide in somewhere - assuming nothing else developed. Fly by wire indeed!

author: SiliconWizard
author_rank: Super Contributor
timestamp: April 05, 2020, 01:05:42 am
content: Whatever the technical reason at the implementation level, what is concerning IMO is that it never got caught during any test; either that, or it was actually specified this way, or not specified at all.So we can debate/guess internal counter widths or buffer sizes to no end, but the specification, verification and/or validation stage clearly failed somewhere.

author: JustMeHere
author_rank: Frequent Contributor
timestamp: April 06, 2020, 07:04:14 pm
content: Quote from: MathWizard on April 04, 2020, 03:06:12 amBoeing has never made a bomb.

author: Kilrah
author_rank: Unknown
timestamp: April 06, 2020, 07:18:16 pm
content: Again? They got another issue of the same type back in 2015...https://www.theregister.co.uk/2015/05/01/787_software_bug_can_shut_down_planes_generators/

author: peter-h
author_rank: Super Contributor
timestamp: April 07, 2020, 05:44:35 am
content: Airbus is not going to gloat too much becauseAll this shows is that those who write safety critical software are not supermen, and are probably no better than your nearest PHP hacker who has left SQL injection back doors all over the placeProbably the biggest thing that protects us is the fact that most of the systems are really old, perform very narrow functions (basically do the same thing every day; airline pilots tend to press the same buttons in the same order every time), and a lot of them (like jet engine ECUs) are carefully designed self contained boxes and then don’t get changed for many years.Defects discovered by crew are reported internally, covered up immediately because the Daily Trash sharks are always circling, while the mfg sh**s itself and issues a rapid fix under the guise of an innocent sounding “feature update” so the certification agency never finds outOne might think the reason Airbus are not making political capital out of the B737 saga is because they are upright and decent people, but in fact they are keeping quiet because they know their cupboard has just as many skeletons in it as all the others.Many years ago, 1983, I designed a multizone heating controller which had self-learning optimisation on the advance turn-on. It was all written in Z80 assembler. Many many hundreds were sold and installed (c. £500 each) and no bugs were ever found. Well, all of them will have stopped working in 2012. The reason is that it computed the day of the week from the date (which came out of a clock chip) and the algorithm I used was just a lookup table which took advantage of the fact (?) that a calendar repeats every 29 years, and I never provided for any subsequent 29 year period. I mean, it’s obvious, in 1983 I was 26, and when you are 26 it is unthinkable that you will ever be 55 because nobody who is of any interest to a 26 year old is that old. I left the company (my first business) in 1991, and it went bust in 1993. Go figure, as they say…I recently spoke to someone who worked on the Honeywell KFC225 autopilot, introduced into general aviation c. 2000 and dropped c. 2003 due to widespread servo burnout issues caused by defective autopilot computer software (believed to be integer maths underflow / dramatic loss of precision). The software had one file per function, because the FAA would push for recertification if more than x functions were changed. If you had 100 functions in one file then any edit of that file changed all 100 functions. The result was an almost unreadable source code…

author: 2N3055
author_rank: Super Contributor
timestamp: April 07, 2020, 06:39:59 am
content: Quote from: JustMeHere on April 06, 2020, 07:04:14 pmGBU-39 Small Diameter Bomb

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: April 07, 2020, 10:32:21 am
content: Yeah,BDS make all kinds of cool stuff.

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: April 07, 2020, 12:22:02 pm
content: Doesn't surprise me at all. Boeing have also screwed up with Starliner, but I see the same issues at other similar companies.Many projects I have ended up asking "what happens when this timer counter wraps? Have we tested it?". Frequent answers are "don't worry it will not matter". Even if that is true at the time, later coders don't realise, and use it for a purpose where it does matter. We had a bug where random "EEPROM timeouts" were reported, turned out to be a false error due to a timer wrap.Other frequent answer is "it's too difficult to test". Obviously testing a product for it's lifetime before release is impractical, so one has to be smart about identifying the timer limits and devising tests with special code if necessary.We were recently requested to add a so-called "safety layer" for an aviation product, it adds a timestamp to each message to identify stale data. Obviously I immediately asked "what happens when...". The answer in this case, "it's not our problem, we've been told to implement according to the spec".Y2K was a specific instance of a class of bugs, but those same bugs are everywhere in embedded software, even software supposed to be "safety critical". There is an assumption that "implementing to the spec" automagically means good code. Quality becomes a box ticking exercise, it's like a cargo cult. Coders and managers alike continue to operate in blissful ignorance, while the project slips further behind schedule.

author: Tepe
author_rank: Frequent Contributor
timestamp: April 07, 2020, 02:05:59 pm
content: Quote from: JustMeHere on April 06, 2020, 07:04:14 pmMaybe not the part that actually goes bang buthttps://www.boeing.com/defense/weapons/

author: eugenenine
author_rank: Frequent Contributor
timestamp: April 07, 2020, 02:35:40 pm
content: "The FAA has received a report indicating that the stale-data monitoring function of CCS {common core system (CCS) – a Wind River VxWorks realtime OS product} may be lost when continuously powered on for 51 days"Sounds like its technically not Boeing's fault here, it is the (VxWorks) OS they used. What else uses VxWorks that could also be affected? Airplanes, Rockets, medical devices?, nuclear launch controllers?

author: coppercone2
author_rank: Super Contributor
timestamp: April 07, 2020, 07:35:42 pm
content: you can't blame a subassembly for a failure unless it happens after it passes testing and is released, otherwise you did not integrate it correctly IMO

author: eugenenine
author_rank: Frequent Contributor
timestamp: April 07, 2020, 08:46:23 pm
content: My point was more that its not just a Boeing subassembly, its used in other places so other things could have issues as well.

author: coppercone2
author_rank: Super Contributor
timestamp: April 07, 2020, 09:06:57 pm
content: Quote from: eugenenine on April 07, 2020, 08:46:23 pmscary to think there is a missile silo or detection radar system some where that happens to do a generator test monthly and its not been found.. god forbid someone gets lazy. I wonder if the military is more prone to this (perhaps a UPS can be considered a reliability problem and they would rather tolerate a small power down for seconds on a generator, or budget cuts)? Also mobile things that setup temporarily but connect to a complex sensor grid might have this undetected, used for training frequently, etc (SAM stuff maybe?)

author: Psi
author_rank: Super Contributor
timestamp: April 07, 2020, 09:14:30 pm
content: Quote from: JustMeHere on April 06, 2020, 07:04:14 pmUnless Wikipedia is wrong, they make rockets and missiles.I'm not sure if they make the explosive payloads themselves, but even if they don't i think it's fair to say they make weapons of war.