Computer math error that cost 28 lives - how would you have programmed it ? - Page 1
Forum section: electronics / general

author: ez24
author_rank: Super Contributor
timestamp: August 06, 2016, 06:41:36 am
content: On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks, killing 28 soldiers and injuring around 100 other people. A report of the General Accounting office, GAO/IMTEC-92-26, entitled Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia reported on the cause of the failure.It turns out that the cause was an inaccurate calculation of the time since boot due to computer arithmetic errors. Specifically, the time in tenths of second as measured by the system's internal clock was multiplied by 1/10 to produce the time in seconds. This calculation was performed using a 24 bit fixed point register. In particular, the value 1/10, which has a non-terminating binary expansion, was chopped at 24 bits after the radix point. The small chopping error, when multiplied by the large number giving the time in tenths of a second, led to a significant error. Indeed, the Patriot battery had been up around 100 hours, and an easy calculation shows that the resulting time error due to the magnified chopping error was about 0.34 seconds. (The number 1/10 equals 1/24+1/25+1/28+1/29+1/212+1/213+.... In other words, the binary expansion of 1/10 is 0.0001100110011001100110011001100.... Now the 24 bit register in the Patriot stored instead 0.00011001100110011001100 introducing an error of 0.0000000000000000000000011001100... binary, or about 0.000000095 decimal. Multiplying by the number of tenths of a second in 100 hours gives 0.000000095×100×60×60×10=0.34.) A Scud travels at about 1,676 meters per second, and so travels more than half a kilometer in this time. This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. Ironically, the fact that the bad time calculation had been improved in some parts of the code, but not all, contributed to the problem, since it meant that the inaccuracies did not cancel, as discussed here and here.Source:https://www.ima.umn.edu/~arnold/disasters/patriot.html

author: EEVblog
author_rank: Unknown
timestamp: August 06, 2016, 06:48:37 am
content: Should have been caught one of several test and review stages.

author: nfmax
author_rank: Super Contributor
timestamp: August 06, 2016, 07:28:44 am
content: Many years ago*, I was asked to test a system which, among other things, sent regular radio transmissions when in an alarm state. There were supposed to bexmessages inyseconds - I forget the details. I timed the messages, and found that the rate was slightly out, only a second or two in ten minutes or whatever it was. Meh, said the software developer, it's near enough. When the units entered service, they were found to drop off the network after a few days if no alarms were generated. It turned out that the underlying bug (an off-by-one error) caused the remote units clocks to drift relative to the base unit, so that the regular "I'm still here" messages the remote units sent were outside the time window when the base unit's (battery powered) receiver was switched on. The drift rate was outside the range of 'reasonable' clock errors that the software had been designed to cope with, so it didn't!The moral? When testing, always investigate any issues you find, no matter how apparently trivial. Also there should have been a tolerance limit on the message rate, based on the maximum allowable clock error. Blessed are those who go looking for trouble, for they shall surely find it...* it's depressing how often I find myself writing this, now

author: mikeselectricstuff
author_rank: Super Contributor
timestamp: August 06, 2016, 08:02:56 am
content: Quote from: ez24 on August 06, 2016, 06:41:36 amI'd have thought the issue was not so much an implementation error leading to a lack of accuracy, but a more fundamental design error that there was any dependence at all on time since boot. And of course using a floating point number for time is just plain stupid.

author: System Error Message
author_rank: Frequent Contributor
timestamp: August 06, 2016, 09:05:23 am
content: scuds are still a problem even today.a gattling gun like equipped on a destroyer is better because if computer control isnt reliable than perhaps someone could manually shoot it down with flak and gattling guns.

author: wraper
author_rank: Unknown
timestamp: August 06, 2016, 09:34:31 am
content: Quote from: System Error Message on August 06, 2016, 09:05:23 amThat someone will be already dead before he manages to aim at the incoming target.

author: Kalvin
author_rank: Super Contributor
timestamp: August 06, 2016, 10:24:31 am
content: Quote from: mikeselectricstuff on August 06, 2016, 08:02:56 amfixed point != floating point.

author: Ian.M
author_rank: Super Contributor
timestamp: August 06, 2016, 11:13:13 am
content: Chase down the links in the article! Raytheon FUBARed it by adding a new 48 bit maths ticks => fixed decimal time routine then failed to replace all calls to the 24 bit time conversion with the new one. The accumulated error was tolerable as long as everything was using 24 bit maths time conversions.

author: German_EE
author_rank: Super Contributor
timestamp: August 06, 2016, 01:28:10 pm
content: How would I have programmed it?Three CPUs with each CPU running a different version of the program written by a different person or organization. The three CPUs each come up with their answer of where the incoming target is and if one CPU is way out it is removed from service until the anomaly can be investigated. I strongly doubt that three different sets of code would contain the same error.Cost? How much are 28 lives worth?

author: Jr460
author_rank: Regular Contributor
timestamp: August 06, 2016, 01:52:36 pm
content: If I remember correctly the Patriot system was not originally designed for missiles. It could have ben the that original coders knew the time math could be off a bit, but with the speeds they cared about, everything would be fine.

author: Arjan Emm
author_rank: Regular Contributor
timestamp: August 06, 2016, 01:53:31 pm
content: Quote from: German_EE on August 06, 2016, 01:28:10 pmFor a compagny like Raytheon? The possible loss of revenue due to the the accident. No more, no less...

author: Kalvin
author_rank: Super Contributor
timestamp: August 06, 2016, 01:56:02 pm
content: Quote from: German_EE on August 06, 2016, 01:28:10 pmI would have changed the system ticks to be powers of two, for example 1/16th of a second.

author: wraper
author_rank: Unknown
timestamp: August 06, 2016, 02:09:54 pm
content: Quote from: German_EE on August 06, 2016, 01:28:10 pmAnd make unmanageable and unmaintainable mess.QuoteSo try your best to achieve that less systems are deployed (higher costs, longer development, inferior specs) so less people would be protected. If such Frankenstein would be delivered ever, before becoming obsolete. And software not nearly a single thing in the system which can cause such outcome.

author: amyk
author_rank: Super Contributor
timestamp: August 06, 2016, 03:30:48 pm
content: If you really want the time in seconds from a 10Hz source, increment a counter at 10Hz and every time it reaches 10, increment the time in seconds and reset the counter back to 0. Then the only source of accuracy errors will be that of the clock source. This, along with "don't use floating point unless you absolutely need to and care more about range than precision", is ridiculously basic stuff but I guess they thought they were really clever by replacing division with multiplication, and missed the even simpler and far more accurate solution.The Therac-25 was another example of massive overengineering cleverness leading to death.

author: donotdespisethesnake
author_rank: Super Contributor
timestamp: August 06, 2016, 04:08:10 pm
content: From one of the links :QuoteThis is typical of the software problems I see. Old software, written a while ago. Changing requirements means that original assumptions may no longer be valid. Management see the exercise as "a quick update" and don't invest sufficient resources. Throw in some bad design decisions. The result is pretty typical, though most bugs don't lead to people dying.There is no magic bullet with software (or engineering in general). It needs skilled people who are diligent in all aspects, and the proper support of management.In my experience there is not much difference between the practices used for ordinary (consumer) products and safety critical products. There tends to be more paperwork in safety critical, but unfortunately that does not lead to better design. e.g. one has to fill out lots of test reports instead of just saying "it was tested OK", and someone checks the reports have been filed, but no one checks the tests actually tested the right thing.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: August 06, 2016, 06:18:41 pm
content: Quote from: donotdespisethesnake on August 06, 2016, 04:08:10 pmLack of errors in large systems requires godlike people who know everything about everything and have time to at least set other people onto problems. Since those people are quite rare, and their rarity means they are always short on time, real large systems get improved by finding failures in real usage and fixing them. Sometimes at great cost in dollars, time or lives. Hindsight is always perfect in the review boards. There are always dozens of people saying how it should have been done. Sometimes the absence of these solutions before the failure is due to management skimping on resources, but more often it is just that even good people miss things sometimes.

author: David Hess
author_rank: Super Contributor
timestamp: August 06, 2016, 07:20:32 pm
content: Quote from: Kalvin on August 06, 2016, 01:56:02 pmThey could have also used BCD math to go along with their base-10 divisions but this change in general is what I would have done and have done in the past; when accumulating, only use divisions which result in no remainder. Note that floating point by itself would not have solved this problem.Sometimes you can get away with just using such high resolution that significant error can never accumulate between system resets but I consider this sloppy design. Periodically resetting the accumulated error through synchronization is not much better.I agree with Dave, it should have been caught during development.

author: SeanB
author_rank: Super Contributor
timestamp: August 06, 2016, 07:51:59 pm
content: Simpler solution, have 2 systems ( they tended to be deployed in pairs anyway) and restart them at staggered 24 hour intervals. That way you always have one with a good self test routine, and the active unit to provide coverage during the restart time. 12 hours later you restart the other unit, and still have the coverage. 100 hours uptime means they had a failed redundant unit, and spares were not available to fix the only active one.Similar problem exists with almost all CIWS systems and counterrmissile systems, in that they are not designed to be run continuously, and tend to fail when left running all the time. However some have long boot up times, so you need to either have multiple systems so you can cycle them sequentially to always have a single one running at all times to give you warning so you can bring up the others, or you keep them in standby and hope you can bring them up in time to counter. Some of these systems have a MTBF in the range of hours of operation between service, so you really sit with Hobson's choice.

author: Len
author_rank: Frequent Contributor
timestamp: August 06, 2016, 10:18:13 pm
content: Quote from: Jr460 on August 06, 2016, 01:52:36 pmYes, that's the way I heard it. The system worked fine against jet planes (which is what it was designed for), but when used against much faster missiles, the timing error caused it to lose track of the target.

author: boffin
author_rank: Unknown
timestamp: August 06, 2016, 10:31:58 pm
content: The Computer math error didn't cost 28 lives; the missile did.

author: Kalvin
author_rank: Super Contributor
timestamp: August 06, 2016, 10:34:07 pm
content: Quote from: SeanB on August 06, 2016, 07:51:59 pmBoeing 787 had also a timer bug that could potentially cause a loss of power of the four generators:http://embeddedgurus.com/stack-overflow/2015/05/boeing-dreamliner-bug/

author: johansen
author_rank: Super Contributor
timestamp: August 07, 2016, 04:00:21 am
content: Quote from: boffin on August 06, 2016, 10:31:58 pmyeah i was thinking the problem is a little higher up the food chain too.

author: timb
author_rank: Super Contributor
timestamp: August 07, 2016, 01:25:55 pm
content: Quote from: boffin on August 06, 2016, 10:31:58 pmActually, it wasn't even the missile that was ultimately responsible. As the (cliche) saying goes, "Guns don't kill people, people kill people." In this case, a software bug didn't kill 28 people, the assholes who shot the missile did!Anyway, I think the fact that they were able to repurpose and deploy the Patriot system for SCUD defense as quickly as they did actually saved countless lives.

author: vodka
author_rank: Frequent Contributor
timestamp: August 07, 2016, 02:42:28 pm
content: QuoteThat 's good to have childreen at house, they always have the guilt .This is a good excuse for avoiding the responsabilities from the Radar officers, because they could to give the alarm preventively to base for waking up the soldiers and they went into bunker or trenches,maybe today would have 4 or 5 KIA instead of 28 deaths

author: SkyMaster
author_rank: Frequent Contributor
timestamp: August 07, 2016, 02:43:55 pm
content: Quote from: wraper on August 06, 2016, 09:34:31 amYes, according to wikipedia, a Scud missile flies at March 5

author: StuUK
author_rank: Frequent Contributor
timestamp: August 07, 2016, 03:03:55 pm
content: Quote from: nfmax on August 06, 2016, 07:28:44 amThats why in my world the software developer does not get a vote!

author: SeanB
author_rank: Super Contributor
timestamp: August 07, 2016, 06:02:44 pm
content: Quote from: SkyMaster on August 07, 2016, 02:43:55 pmI will give you a LMG and a full box of preloaded link ammunition, and fly a slow Tomahawk at you . If you shoot it down you survive, otherwise the missile will destroy your area. Tomahawk missiles are not the fastest of missiles, they rely more on being 20m above the ground, and having a good terrain following RADAR and a built in terrain map to navigate to the target. This makes them near impossible to hit, as a ground based weapon has a very small window for engagement, and airborne has a very hard time trying to find the missile amongst ground clutter. Even the thermal signature is designed to be small.About the only way (and fixed in later versions) was to know the RADAR chirp rate and key, and to use a EW system active emitter to fool the onboard computer into thinking there was a building or a low hill coming up ahead, which then made the terrain follower fly higher to avoid what it saw as an obstacle, thus enabling a long enough time for a tracking system to acquire a lock and fire a countermissile. Hard to do, you had to know the path to be followed and had to have the EW system ready at the right time. The fix was a dither on the RADAR with a long PRNG key, kept by the missile alone, and used to both sign the pulses and vary the repetition rate over a range.

author: zapta
author_rank: Super Contributor
timestamp: August 07, 2016, 06:04:48 pm
content: Quote from: boffin on August 06, 2016, 10:31:58 pm+1. And we need to count also the lives that this software saved.

author: System Error Message
author_rank: Frequent Contributor
timestamp: August 07, 2016, 07:31:49 pm
content: Quote from: SkyMaster on August 07, 2016, 02:43:55 pmRadar can detect it but a high RPM gattling gun doesnt need to be accurate, it just has to fire all it can and hopefully a few bullets will hit and cause it to explode in midair.Missile interception isnt just a real life problem, its also a computer game problem where physics simulation is used, such as in from the depths where you can target projectiles and missiles (the information that you get is the targets heading and velocity, not some cheating game code). The issue is with deviation such as acceleration changes, wind changes and that the interception missile costs more than the scud itself. So you can either try to accurately intercept the missile or barrage the sky with bullets and explosions hoping to randomly hit it.The code if updated and made should take into account the sensors as well in the format that the sensors use and do as little conversions as possible. If accuracy is needed perhaps a faster CPU and using something similar to java's BigDecimal where accuracy can be controlled depending on how much CPU power you have. The problem isnt only coding but making sure that all the data sources use the same format or at least have the format used taken into account (such as absolute values or imperial or SI units).

author: wraper
author_rank: Unknown
timestamp: August 07, 2016, 08:11:55 pm
content: Quote from: System Error Message on August 07, 2016, 07:31:49 pmIt doesn't need to be super accurate because it sends swarm of bullets. But it need to do it in right time in the right direction. You just have no time to do this manually. At mach5 you are lucky if you have a few seconds to aim once the missile is detected.

author: System Error Message
author_rank: Frequent Contributor
timestamp: August 07, 2016, 08:42:09 pm
content: Quote from: wraper on August 07, 2016, 08:11:55 pmHence why those gattling guns on US ships are computer controlled and can move about really fast. Just saying if accuracy is a problem there are other ways other than just using an accurate missile. Cost is another issue as well as scuds are very cheap whereas the equipment for missile interception costs a lot more. In the cost of computer failure you can still use those guns manually with guidance from tracking. Odds may be low but its better than a missile interceptor missing with no other means of taking down the missile.Perhaps using those same gattling guns on the ground could be useful not only for scuds but also to shoot down jet fighters and ground units.

author: rollatorwieltje
author_rank: Unknown
timestamp: August 07, 2016, 11:16:53 pm
content: A GAU-8 gatling gun as used on the Thales Goalkeeper has a range of about 3km, the M61 Vulcan as used on the Raytheon Phalanx about the same. You can forget manually controlling them, the window of opportunity against a missile is literally seconds. Also completely pointless as AA defense, 3km range is useless. Missiles like the Exocet are dropped at 10+ km from the target.Realize a CIWS system is your last line of defense, when it activates it means you failed to guard your airspace, you failed to intercept the enemy, your electronic countermeasures (i.e. radar jamming) failed and your decoy systems (i.e. chaff, flares) failed.Ground self propelled AA guns do exist, but they mostly use autocannons instead of gatling guns. Much cheaper to operate, better range (although still just +/- 4km) and better high explosive munition. You don't need a cloud of lead, these things pretty much only exist to deter slow ground support helicopters and planes.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: August 08, 2016, 04:06:07 am
content: If you do a little physics/control simulation of these things with just a few bits of reality incorporated (transport lag due to speed of light, speed of bullets, alignment errors, solid angle of target and time of flight vs number of bullets per second and so on) you can see where the fundamental limits of these systems lie. That doesn't mean they can't work, it just means that no one system can solve everything. And that no system is a perfect solution to anything. But they can be useful. It does mean that getting them to work is extremely difficult and the problems lie far beyond the first order issues discussed here. As the applications for a system are expanded the problems just get harder.To the point of the thread, when you are juggling all of the balls involved, how do you keep track of all of them and avoid dropping them. Most of the comments here sound to me like "Well if you had only paid proper attention to ball 391 it would have been obvious." The trick is knowing that #391 needs more attention than balls #75 and #1369. Or having 2500 experienced and talented hands so that all of the balls are getting proper attention. Both of those solutions are hard to achieve.

author: vodka
author_rank: Frequent Contributor
timestamp: August 08, 2016, 05:25:54 am
content: QuoteAltough he don't shoot down the missile , he cans survive if he is at shooter hole or trenches.The missile might be shooted down if the gunners use ammo with RF proximitly fuze that doesn't need to impact with the objective for shooting down.https://youtu.be/6-D592VR4RU

author: chris_leyson
author_rank: Super Contributor
timestamp: August 08, 2016, 05:50:54 am
content: Would doing a divide by 10 in fixed point hardware be considered cheating, convert 24bit binary to 32bit BCD, shift four places to divide by 10, convert back to binary. You could do it serially with minimal hardware, albeit slowly, Xilinx XAPP029 for example.

author: David Hess
author_rank: Super Contributor
timestamp: August 08, 2016, 08:11:47 am
content: Quote from: chris_leyson on August 08, 2016, 05:50:54 amIt would not be much of a cheat since it would not fix the problem.It is the internal representation of the numbers which are the issue. The financial industry has been well aware of this since they first used computers if not earlier which is why some big systems support BCD math in hardware but BCD math is not a universal solution if you still pick the wrong divisors.Quote from: CatalinaWOW on August 08, 2016, 04:06:07 amThe problem is more insidious then unmanageable complexity. There is probably a better term for this in industry or education but saved state should never have a remainder.This means that any running sum needs to have no remainders. Ideally a representation is chosen which prevents this problem but if that is not possible, then the remainder needs to be truncated to an understood value at every sum but that is more difficult to enforce. Pick the right numerical representation.This applies to dynamic memory management as well. Ideally you do not use it at all. If you have to, then use it only where absolutely necessary and include real time check functions to monitor it. Personally I consider any delay caused by garbage collection to be a failure but in some applications this is acceptable which is a shame because at best it is annoying as hell.If you have a system where you have to periodically reset it (resetting state which cancels remainders) to maintain operation, then find out why and fix it. Resetting state is fine for unknown emergencies (by definition you cannot plan for the unknown) but it should not be relied on for normal operation.The automotive and home automation "internet of things" industries are preparing themselves to learn this the hard way but I doubt the lesson will every take with them. What happened to Toyota has happened to a couple of other vehicle makers as well and is only the beginning.