Boeing Starliner: 2 SW bugs found, patched, uploaded in-flight to avoid disaster - Page 1
Forum section: electronics / general

author: splin
author_rank: Frequent Contributor
timestamp: February 11, 2020, 06:51:49 pm
content: It seems the software problems were worse than previously reported, which was a timer error which set the clock 11 hours off. But there was another serious bug:https://www.theregister.co.uk/2020/02/10/more_software_errors_beset_boeings_calamity_capsule/QuoteThe comments are well worth reading, specifically this:QuoteI'm not sure what the source for this claim is but if true, I'd struggle to come to a much different conclusion than this later comment:QuoteThe reviewer, if there was one, didn't verify the contents of the look-up tables which is just as important as the code itself - unforgivable.This wasn't one of those obscure and totally unexpected bugs such as a subtle compiler fault that very rarely causes a latent data error to be introduced into the software state which only reveals itself long after the initial source of the error and in a totally unrelated subsystem. Or a race condition which only occurs in very specific circumstances not exercised by even the most extensive test-suites. In such cases one can have a great deal of sympathy with the developers when these faults make it undetected to released product.The atmosphere must be very uncomfortable in Boeing's development teams right now. Hmm, my tablet auto-corrected Boeing to Boring - how wrong could it get.

author: SilverSolder
author_rank: Super Contributor
timestamp: February 11, 2020, 07:11:59 pm
content: If true, these are signs of near total decay at Boeing - needs a "back to basics" cultural change.

author: edy
author_rank: Super Contributor
timestamp: February 11, 2020, 07:24:28 pm
content: What ever happened to "measure 1,000,000 times, cut once". I would have expected there to be extensive simulator and systems testing, and not just on one but several different independently made simultation/testing platforms, just to hammer out every possible issue there could ever be. I guess the code is so huge now that it is becoming impossible to manage. That, or there could be several generations of code in there now that newer people in the development cycle are having a hard time understanding. Perhaps a "reboot" and recoding from basic fundamentals is needed every once in a while to build up the system from scratch again (although that may also introduce errors... isn't there a concept of "leave spaghetti code alone if it seems to work").Anyways, a lot of assuming still as to exactly what went wrong but it seems like one problem (timing error) was compounded by another (copying code without associated tables) when they tried to fix it mid-mission. Honestly I didn't even know Boeing and NASA were up to this... I think it is cool that they are using unmanned flights for supply missions.

author: Tomorokoshi
author_rank: Super Contributor
timestamp: February 11, 2020, 07:29:00 pm
content: What is it they say? "Test what you fly, fly what you test".https://appel.nasa.gov/2002/10/01/test-what-you-fly/

author: WastelandTek
author_rank: Frequent Contributor
timestamp: February 11, 2020, 07:30:00 pm
content: Quote from: SilverSolder on February 11, 2020, 07:11:59 pmIt's not just Boeing, it's culture wide, we can't even do the choo choo trains any more, and there never seem to be any real consequences for the perpetrators of failure.

author: chris_leyson
author_rank: Super Contributor
timestamp: February 11, 2020, 07:36:16 pm
content: QuoteThey must have been using the new StackOverflow keyboard. I've seen people write code just by cutting and pasting. It's not just software but hardware people do it as well. Was going to say engineers but that would be inapropriate. ESA did the same with the first Arian 5 launch.

author: splin
author_rank: Frequent Contributor
timestamp: February 11, 2020, 08:28:12 pm
content: Quote from: edy on February 11, 2020, 07:24:28 pmI think you misunderstood - they didn't create the second fault whilst fixing the first; it seems that while investigating the errant flight, they discovered two serious bugs - the clock problem that instigated the, presumably frantic and white knuckle real time code review, and the other cut and paste error which would have manifested itself disasterously later in the flight. They managed to fix both errors and upload them in time to avoid catastrophe.Maybe the first error was added deliberately by a conscientious engineer knowing it was the only way to ensure the rest of the steaming pile got a proper review.

author: rdl
author_rank: Super Contributor
timestamp: February 11, 2020, 08:42:37 pm
content: Because of this latest screw up Boeing will have to undergo a full "invasive" safety review, similar to what was done at SpaceX. They had basically bought their way out of that originally. Now SpaceX will probably beat them to launching astronauts to the ISS by months if not years.

author: DBecker
author_rank: Frequent Contributor
timestamp: February 11, 2020, 09:10:35 pm
content: Quote from: rdl on February 11, 2020, 08:42:37 pmThere were claims that the NASA safety review of SpaceX was a move by Boeing lobbyists in DC trying to slow SpaceX while justifying their own slow delivery.karma++

author: Gyro
author_rank: Super Contributor
timestamp: February 11, 2020, 09:32:20 pm
content: Quote from: DBecker on February 11, 2020, 09:10:35 pmWhat goes around comes around.

author: Red Squirrel
author_rank: Super Contributor
timestamp: February 12, 2020, 01:44:24 am
content: Seems like Boeing is taking the game approach to software development. It compiles! Ship it, we'll test and patch later.

author: wraper
author_rank: Unknown
timestamp: February 12, 2020, 02:15:27 am
content: Quote from: Red Squirrel on February 12, 2020, 01:44:24 amThere is even a name for it:Early access.https://en.wikipedia.org/wiki/Early_access

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: February 12, 2020, 01:23:25 pm
content: And if you are wondering "Is 1 million lines of code a lot?" thenhttps://informationisbeautiful.net/visualizations/million-lines-of-code/Some comparisons with similar projects (the "machine" category):Space shuttle 0.4 millionF22 Raptor fighter 1.7 millionHubble space telescope 2 millionBoeing 787 6.5 million (avionics and support)F35 fighter 24 millionSo on modern standards it is not that much. However, a comprehensive code review is still a lot of effort, let's say ballpark figure about 1 person-year. With 50 engineers should be done in less than ... 1 year.

author: Tomorokoshi
author_rank: Super Contributor
timestamp: February 12, 2020, 02:34:39 pm
content: Quote from: donotdespisethesnake on February 12, 2020, 01:23:25 pmOne year... that's a problem. Let's double that and add 50 more engineers:https://en.wikipedia.org/wiki/Brooks%27s_law

author: SiliconWizard
author_rank: Super Contributor
timestamp: February 12, 2020, 02:51:29 pm
content: This one is cute too:https://www.theregister.co.uk/2020/01/08/boeing_737_ng_cockpit_screen_blank_bug/

author: coppice
author_rank: Super Contributor
timestamp: February 12, 2020, 03:04:46 pm
content: Quote from: edy on February 11, 2020, 07:24:28 pmThe key problem with most of this kind of work is it takes a very specific kind of character to do it well. A character that largely ignores most of the obvious stuff that almost any simplistic testing will find, and seeks out the funky corner cases where most of the serious issues tend to be found too late, and often disasterously. These people need to be a part of every simulation team, and need to be the most highly regarded and rewarded. Instead they are too often suppressed, because they increase the test complexity to the point where its effective, while management just wants to ship stuff. People love talking about flushing out corner case issues, but it has a nasty tendency to go little further than talk.

author: splin
author_rank: Frequent Contributor
timestamp: March 04, 2020, 01:17:56 am
content: Boeing have finally fessed up to their failure to properly test the software:https://www.theregister.co.uk/2020/03/03/space_roundup/QuoteQuoteRemind me again how much Boeing were billing for this?

author: tom66
author_rank: Super Contributor
timestamp: March 04, 2020, 07:56:17 am
content: Boeing has some serious core engineering problems that need to be addressed.If this was truly a free market, rather than Boeing being in the pocket of the US Military and Gov't, then the 737MAX8 disasters would have been enough to take the company down. Think how it only took one accident to put people off the de Havilland Comet, or the accidents that brought down McDonnell Douglas (perhaps some of the engineering transplants from MD are responsible for the fuckups at Boeing?)

author: wraper
author_rank: Unknown
timestamp: March 04, 2020, 08:03:56 am
content: Quote from: tom66 on March 04, 2020, 07:56:17 amMore like management problems.QuoteManagement transplants. There was internal joke that McDonnell bought Boeing with Boeing's money.

author: wraper
author_rank: Unknown
timestamp: March 04, 2020, 08:07:54 am
content: Quotehttps://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/

author: tooki
author_rank: Super Contributor
timestamp: March 04, 2020, 08:46:26 am
content: Quote from: wraper on March 04, 2020, 08:03:56 amHeh, the same thing happened with Apple, where we say that NeXT bought Apple with negative money. The difference is that NeXT’s management replacing Apple’s (in 1997) was an indisputable win, whereas McDD’s management was a clear downgrade from Boeing’s.

author: Rerouter
author_rank: Super Contributor
timestamp: March 04, 2020, 09:36:19 am
content: When you skip testing, or just have horrible tools for testing, things start catching up with you in short order.About 1 year ago, I finally was able to pry enough information about how a bit of automotive electronics scripting was made, turns out its simple stack scripts, lets you read in data from the CANBUS, process it and react accordingly, including writing back to the CANThese scripts where mostly copy-pasta only 300-700 lines, the grey beard that retired 5 years ago wrote it, so we don't question it kind of crap.Well I went and wrote an emulator for there scripting language, just to check things out, as we kept seeing weird behavior in the data it was logging, and little hard to replicate glitches.Turns out the grey beard code was pushing an error value to stack any time it could not handle a packet, but never removed it. so the glitches where the stack hitting something it shouldn't, causing a very rapid reboot, however there debugger that they test scripts with, would remove that value, so they would never see the issue in testing, combined with only feeding "good" data to the scripts so they rarely handled packets that would trigger this behavior.I went and re-wrote the problematic scripts, explained what was going wrong, and how to fix it, even shared the emulator with the company, and don't think I have had a door slammed harder in my face before, with all future updates with the same bugs, that I would then patch.... company culture and ego plays a big part in these things.

author: SilverSolder
author_rank: Super Contributor
timestamp: March 04, 2020, 12:31:06 pm
content: Quote from: Rerouter on March 04, 2020, 09:36:19 amPerhaps echoes of what happened at Volkswagen, with the emissions scandal. Many people knew... but did not want to know.

author: tooki
author_rank: Super Contributor
timestamp: March 04, 2020, 04:25:26 pm
content: Quote from: Rerouter on March 04, 2020, 09:36:19 amOh man, I hear you.Very few people have the magical skills to successfully approach companies with a “hey, your product/website/menu/whatever sucks, and I’m the right person to fix it!”

author: MadTux
author_rank: Frequent Contributor
timestamp: March 04, 2020, 09:49:00 pm
content: So thats basically what happens if you put wings onto a M$ Windoze PC?The craziest part is that most airliners wouldn't even need FlybyWire. Only planes such as F16, F22, F35... which are designed to be aerodynamically unstable in some conditions, for max maneuverability, really need it. Not even a manual mode, where you control the hydraulics directly, is build into it.So, just like with M$ Windoze, you keep adding crap, because you can, until it crashes.

author: tom66
author_rank: Super Contributor
timestamp: March 04, 2020, 10:12:28 pm
content: Airbus aircraft are almost entirely fly by wire and they work pretty well, with a few documented incidents but primarily down to crew training. Nothing unsafe about a well-designed FBW aircraft, it just requires competent engineers and management, along with a strict and independent regulator to test it.

author: tszaboo
author_rank: Super Contributor
timestamp: March 04, 2020, 10:21:28 pm
content: I guess they needed......over the air update!ba dumm tss. Slowly awkwardly walks out.

author: eugenenine
author_rank: Frequent Contributor
timestamp: March 05, 2020, 12:37:32 am
content: Quote from: donotdespisethesnake on February 12, 2020, 01:23:25 pmwindows 10 50 millionhttps://answers.microsoft.com/en-us/windows/forum/all/windows-10-lines-of-code/a8f77f5c-0661-4895-9c77-2efd42429409Linux 27millionhttps://www.linux.com/news/linux-in-2020-27-8-million-lines-of-code-in-the-kernel-1-3-million-in-systemd/The F35 could be its own OS

author: rdl
author_rank: Super Contributor
timestamp: March 06, 2020, 10:51:04 pm
content: Don't confuse basic fly-by-wire with stability control systems. Manual fly-by-wire is how it all started.In other news:NASA declares Starliner mishap a “high visibility close call”(Ars Technica)Quote from: Jim Chilton, vice president and general manager of Space and Launch