Do you see the error in this code.  I sure as hell do not. Cost DM 1200 milllion - Page 1
Forum section: electronics / general

author: ez24
author_rank: Super Contributor
timestamp: July 29, 2016, 06:57:50 am
content: FYI this error cost DM 1200 million (I have no idea what this is in US $ )strict precondition 1:{Set."x"=FLPT and Set."y"=INT16 and -32768 <= x <= +32767}strict precondition 2:{Set."x"=FLPT and Set."y"=INT16 and int(x) in INT16}strict precondition 3:{Set."x"=FLPT and Set."y"=INT16 and int(x) in Set."y"}program code (convert floating point x to fixed point integer y):y := int(x)postcondition:{Set."x"=FLPT and Set."y"=INT16 and y=int(x)}for more details see:http://www.cas.mcmaster.ca/%7Ebaber/TechnicalReports/Ariane5/Ariane5.htm(I hope someone can explain in layman's words what this is)

author: PartialDischarge
author_rank: Super Contributor
timestamp: July 29, 2016, 08:54:39 am
content: It seems that the error was in the conversion from floating point to fixed point. One has to be very careful in doing this things , and actually it is a conversion that strictly speaking doesn't make sense and is a very bad programming practice.I've programmed some communications algorithms (a software PLL with I and Q phases for example) in assembly and used fixed point instead of floating point in C due to execution speed, and the thing is nowhere trivial.Most programmers lack the knowledge of the secrets behind these two types, especially the good old signed fixed point and its forgotten notation Qa.b

author: PartialDischarge
author_rank: Super Contributor
timestamp: July 29, 2016, 09:08:20 am
content: I'll give some examples* 0.1, float(0.1) and double(0.1) are NOT the same numbers* a fixed point signed 16 bit number (Q0.15) goes from -32768 to +32767, and it is not simmetrical, so you cannot represent with itan analog quantity of +1 to -1. It only goes from +0.999969 to -1. If this is not taken into account an overflow may occur. Reading the posted link it seems that this could have happened, ie if the routine thinks that the highest positive number (+1) is +32768, it will actually represent the largest negative number, -32768* floating point accuracy is limited!!! See attached code from matlab and the weird result.*Nevercompare for equality two floating point numbers. For example, there is a system defined constant for PI called M_PI with lots and lots of decimals, but this code:float x=M_PI;if (x-PI)==0) return 1else return 0;it returns 0.

author: Maxlor
author_rank: Frequent Contributor
timestamp: July 29, 2016, 09:50:24 am
content: The function looks correct (although I have never seen that syntax, no idea what it is.) It does exactly what it says it does, under given preconditions.The problem is with how this function was called, i.e. without checking preconditions, and handling the case where the preconditions aren't met. The default handler apparently was ending the program, which usually is a reasonable thing to do (and often, theonlything to do) but in this case, had some rather explody consequences.A precondition is a condition, or a check, that must hold (i.e., the check must pass) before a function is called. The function only guarantees its behaviour if the preconditions hold, and if they don't, it might behave in an undocumented way, or crash, or throw an exception.Similarly, a postcondition is a condition that holds after the function. If they break, it's usually because there's an error in the function implementation.Generally, when writing a function, it's quite possible that there are sets of input parameters for which the function doesn't make sense, and you don't have enough control over the compiler to prohibit those sets of parameters. An example is a square root function with a signature like this: float sqrt(float value); Now, with these types it's not possible in C to limitvalueto non-negative values, but that is a precondition of the function. (Note: it cannot return complex values, since the return type is specified as float as well.) When sqrt(-42) is called, the precondition is violated, which is clearly a programming error outside the sqrt() function. The reasonable thing to do here is to abort() or throw an exception so the error can be easily identified and fixed. Or maybe, if this behaviour is documented, return some special value like 0; although such behaviour is likely more trouble than its worth in the long run, since it hides errors, making them much harder to track down.With safety-conscious code, it's not uncommon to see preconditions and postconditions written down as code, and for the compiler to generate explicit checks for them, so that errors can be found as quickly as possible. But even if no code is generated to check them, all functions can be thought of as having pre- and postconditions. Sometimes they're documented, sometimes they're merely implied.

author: Dr. Frank
author_rank: Super Contributor
timestamp: July 29, 2016, 10:47:43 am
content: Quote from: ez24 on July 29, 2016, 06:57:50 amWell DM = Deutsche Mark (German currency before European Unions €uro)That is about 600 million Euro, or about 700 M US$ as of today.The description is very cryptic.The critical code simply isy:= int(x)which is not yet faulty itself, only in the context, that x is a real number, and y is integer.Such assignments are dangerous logically, as it has to be checked beforehand, whether the absolute value of x fits the maximum range for the integer function int(). In this case, it would have to be in the range of [-32768... +32767], determined by y (single integer).Otherwise, a runtime error (defined by the compiler) might occur, or y maybe filled with random numbers.That's a freshman's programming error, and bad SW testing.It reminds me of that Mars satellite crash, where two different SW teams mixed inch and meter calculations.Frank

author: Cervisia
author_rank: Regular Contributor
timestamp: July 29, 2016, 11:54:46 am
content: That web page does not make a good job of explaining the error.Seehttps://en.wikipedia.org/wiki/Cluster_(spacecraft).

author: bktemp
author_rank: Super Contributor
timestamp: July 29, 2016, 01:46:42 pm
content: What would be the best solution for such a problem? Clipping the value to the nearest possible value (-32768 or 32767)?Shutting down the whole computer sounds like the worst solution to me.

author: Maxlor
author_rank: Frequent Contributor
timestamp: July 29, 2016, 02:47:16 pm
content: Quote from: bktemp on July 29, 2016, 01:46:42 pmThe error is external to the function, so it should be handled externally. If it isn't, one might run into trouble no matter what the function does, be it crash, return 0, returned the clamped value, do modulo arithmetics, throw an exception... it's all wrong.If you find yourself writing code like this, throw an exception if your language supports that, return an error code if your signature allows for that (or a return value signifying an error, like NaN), otherwise just abort() (in other words, crash.)

author: suicidaleggroll
author_rank: Super Contributor
timestamp: July 29, 2016, 03:00:48 pm
content: Quote from: bktemp on July 29, 2016, 01:46:42 pmWhen the inputs to a function are out of bounds, there is a problem upstream from the function. It's not the function's fault, but how the function responds is very important. Far and away the worst thing you can do is proceed with the calculation anyway and return a value that seems correct, but is wrong. As long as you don't do that, the "correct" behavior depends on the situation. Usually returning an error code would be the right approach, but if the system doesn't support that, generally the best you can do is crash. Clipping the input to the rail might be the right approach, but it is VERY situation-specific, and should not be the de facto response, as that can make things 10x worse in the wrong situation.

author: rstofer
author_rank: Super Contributor
timestamp: July 29, 2016, 03:08:21 pm
content: Ada throws an exception if the integer overflows.

author: Kalvin
author_rank: Super Contributor
timestamp: July 29, 2016, 03:15:14 pm
content: Quote from: rstofer on July 29, 2016, 03:08:21 pmYes. But what happens if the exception is not handled properly? The software will probably crash and will be restarted, or it will hangup until the watchdog will restart it.

author: bktemp
author_rank: Super Contributor
timestamp: July 29, 2016, 03:40:13 pm
content: So the code shown here, was fine, the actual problem was either the missing input range checking or the wrong exception handling?

author: Kalvin
author_rank: Super Contributor
timestamp: July 29, 2016, 03:54:59 pm
content: Quote from: bktemp on July 29, 2016, 03:40:13 pmThe input range checking was there, and the resulting exception was handled so that the computer was shut down.Here is a copy of the report:https://www.ima.umn.edu/~arnold/disasters/ariane5rep.htmlSearching for a term "exception" will eventually lead to following paragraph:Quote

author: Len
author_rank: Frequent Contributor
timestamp: July 29, 2016, 03:55:13 pm
content: Quote from: bktemp on July 29, 2016, 03:40:13 pmIn short, yes, the cause was bad exception handling and unexpected input values.More details are in the report of the Ariane 501 Inquiry Board, available here:http://esamultimedia.esa.int/docs/esa-x-1819eng.pdf

author: rstofer
author_rank: Super Contributor
timestamp: July 29, 2016, 05:04:21 pm
content: Quote from: Kalvin on July 29, 2016, 03:15:14 pmAt least the Ada programmer would have had an opportunity to consider the exception. Maybe there is no solution, maybe there is.I don't know much about Ada but I do know that it is being used in critical systems where failure isn't an option.Furthermore, the programmer could have defined a data type for the 'sensor' output that limited it's range, in effect sanitizing the reading. Still, the reading must have been an exception so maybe sanitizing it is the wrong thing to do.In the version of Ada that I am playing with, I can't get within 64 of 2^31 before it throws an exception. The value 2^31-65 is the largest positive value that converts without an exception. I don't know why that is...In any event, Ada may not be capable of running on 16 bit machines so it might not have been an option.

author: Kilrah
author_rank: Unknown
timestamp: July 29, 2016, 05:46:26 pm
content: Quote from: rstofer on July 29, 2016, 05:04:21 pmI don't see why.There was error handling, and it did what people told it to becasue they didn't think it through correctly, the language is irrelevant. Obviously that case was never encountered at a time where "the programmer would have had an opportunity to consider the exception", becasue if the system had crashed during testing they'd for sure have investigated as well.

author: Kalvin
author_rank: Super Contributor
timestamp: July 29, 2016, 05:52:51 pm
content: Quote from: rstofer on July 29, 2016, 05:04:21 pmThe single precision IEEE floating point number has 24-bit mantissa and 8-bit exponent. I guess that the 2**31 - 65 is the maximum integer presentation of the 24-bit single precision IEEE floating point normalized number which will fit in the 31 bits.The Ada GNAT compiler is available even for the 8-bit Atmel Atmega328 etc. microcontrollers, so the 16-bits should not be a problem.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: July 29, 2016, 06:40:29 pm
content: Answers on this thread show a problem common to many programmers. The "solution" is to throw an exception, halt the code or something similar. Meanwhile, in the real world you are a few meters in the air, with hot flames and thousands of tons of fuel and oxidizer.The universal and first step in writing good code is understand what the system is trying to accomplish. Languages, data types, A/D converters, and other constructs are then tools to accomplish that. The entire chain must be remembered. The complexity of software often leads people to consider SW in isolation, thinking in terms of language purity, or ignoring hardware interfaces. I can't count the times that I have been in the lab integrating some complex system and received a dumb look from a programmer when asked "What format did you send to device X. Was it BCD, 2s complement, big endian or little endian and so on". The problem isn't so much that they didn't know the answer to the question, but that they didn't even know a question was there.

author: bktemp
author_rank: Super Contributor
timestamp: July 29, 2016, 07:10:40 pm
content: The report is an interesting read.It looks like the programmers of the individual sub-systems did exactly what was requested. Maybe they even got information about the ranges of the input data so they skipped the range checking. The same applies to the exception handler: It was only designed to handle random hardware errors, because the software was considered as correct.So the people writing the requirements obviously missed some facts, leading to a faulty software developement.

author: rstofer
author_rank: Super Contributor
timestamp: July 29, 2016, 07:26:40 pm
content: Quote from: Kalvin on July 29, 2016, 05:52:51 pmI hadn't even thought to try GNAT with an Atmega. Interesting! I am looking at it in the context of ARM processors but at the moment, I am just messing around in Linux. I'm just using gedit or nano and the command line. I have tried a couple of "hello world" kinds of things on one of my ARM boards and it seems to work fine.Exception handling for embedded processors is a little bizarre. What to do? It's not like there are people around to intercede. In the nuclear business, measurements are generally taken by 3 independent means. Actions are taken on best-out-of-three. But even there, the default solution may be to shut down the reaction. Not a satisfactory answer for something that flies.

author: rs20
author_rank: Super Contributor
timestamp: July 29, 2016, 07:43:25 pm
content: The exception handling strategy didnotcause this failure. The mistaken precondition, and the impossible attempt to squish a value into a int16 that couldn't fit, did.I don't understand why people are saying "the exception wasn't handled properly". The code was incapable of handling the large function parameter; the code was fatally flawed. What the hellareyou supposed to do if your function comes back to you and says "my inputs are out of bound, I cannot tell you where to point the rocket"? If all three computers are concurring on that conclusion, then you're just screwed. The handler can't say "oh, didn't need that number anyway, let's just soldier on then"*. Come on. Assuming that only one computer has had a hardware fault, and disabling the processor so that the remaining two can (hypothetically) continue to fly the rocket successfully may have been a incorrect assumption in this case, but it's the correct approach when there is a hardware failure, handling a failure of all three computers is almost impossible by definition unless someone can foresee the particular failure (in which case, why not just fix the software?), but there is no way to handle the exception properly either, so that assumption didn'tcausethe failure.Also, as a note of context, the software had flown successfully in the past, and the assumptions that the preconditions were true did hold true in the past flights. An upgraded rocket generated larger numbers (higher lateral accelerations, from memory?), and they didn't catch the fact that the precondition was no longer satisfied. To be fair, this is a bit of a subtle case -- hardware evolving to generate conditions that software, designed for earlier version of the hardware, could not have expected -- although I'd hope they had some sort of integration testing with virtual sensors and real rocket hardware to do a mock flight each time.* Now technically, they could have had a fallback navigation system to defer to, written by a totally different parallel team, etc, etc. But if that's what you're suggesting, be explicit about it because that is ahugeundertaking to prepare that defense across all software, even for a space agency.

author: Marco
author_rank: Super Contributor
timestamp: July 29, 2016, 08:14:02 pm
content: The primary problem I see is running with assertions on in production code on both the primary and backup.What's the point? Just try to keep going, the self destruct is autonomous any way ... the worst that can happen is self destruct, the best that can happen is that it manages to keep flying.Quote from: rs20 on July 29, 2016, 07:43:25 pmIt can and it should, maybe some code down the line has a sanity check to prevent bogus input, maybe after the next course correction the error will not trigger again, maybe a lot of things. May not self destruct abort is better than will self destruct abort.The bug is simply a bug, but this is a fundamental flaw.

author: Kjelt
author_rank: Super Contributor
timestamp: July 29, 2016, 08:23:21 pm
content: The one and only mistake was reusing software from an old product for a new product without decent testing , this is the result the first product(batch) fails.

author: suicidaleggroll
author_rank: Super Contributor
timestamp: July 29, 2016, 08:23:59 pm
content: Quote from: Marco on July 29, 2016, 08:14:02 pmBut it's not a bug, the software is doing exactly what it's supposed to do. The problem is person who decided to load software designed for a smaller rocket onto a bigger one without checking the acceptable input ranges first.

author: Marco
author_rank: Super Contributor
timestamp: July 29, 2016, 08:28:09 pm
content: Meh.Doesn't remove the fact that may not self destruct abort is better than will self destruct abort.

author: rs20
author_rank: Super Contributor
timestamp: July 29, 2016, 08:39:25 pm
content: Marco,My point is that if you ran the rocket with assertions turned off, the rocket still would have crashed in this case. Regardless of your opinion of whether assertions should be on or off, that doesn't change the fact that this crash would have occurred even if assertions were turned off, and thereforeblamingthe crash on the assertions is therefore simply ridiculous and wrong.As I side note, I respectfully disagree that running a secondary computer with assertions turned off is preferable to the popular triple-redundant system, but I'm not going to try to defend that point, other than to question if there has ever been a rocket failure where that would have helped, and to defer to the experts who designed the system, fallible as they evidently are. You seem to be focussed on the incredibly thin theoretical chance that "forging" ahead could lead to mission success, yet blind to all of the cost, complexity, and extra failure modes exposed by having multiple different computers with different stacks. But I digress, all I really want to claim and defend is what's in the paragraph above.

author: Marco
author_rank: Super Contributor
timestamp: July 29, 2016, 09:03:31 pm
content: Quote from: rs20 on July 29, 2016, 08:39:25 pm"This software module computes meaningful results only before lift off".No garbage value it would have produced would have been relevant.So as I said, bugs happen ... but not just soldiering on is a fundamental design flaw.