A bad day at work. The NOAA 19 satellite drop. - Page 1
Forum section: electronics / general

author: Homer J Simpson
author_rank: Super Contributor
timestamp: January 07, 2014, 06:59:37 pm
content: This happened back in 2004 but it is interesting if you have not heard about it before.Basically the accident happened because the bolts to the adaptor used to rotate the satellite were left out and know one noticed.Satellite cost was $220 million.Repairs cost $130 million."The Responsible Test Engineer (RTE) did not "assure" the turnover cart configuration through physical and visual verification as required by the procedures but rather through an examination of paperwork from a prior operation."NASA accident report.http://www.nasa.gov/pdf/65776main_noaa_np_mishap.pdfKT

author: Jon86
author_rank: Frequent Contributor
timestamp: January 07, 2014, 07:01:10 pm
content: Oops. Expensive mistake...

author: SeanB
author_rank: Super Contributor
timestamp: January 07, 2014, 07:19:20 pm
content: For want of a bolt......

author: pickle9000
author_rank: Super Contributor
timestamp: January 07, 2014, 07:53:08 pm
content: Man there must have been a great deal of upset stomachs that day, no matter who did it.

author: mdszy
author_rank: Unknown
timestamp: January 07, 2014, 08:12:33 pm
content: Oh man. I would've hated to be the guy responsible for that mishap!

author: SeanB
author_rank: Super Contributor
timestamp: January 07, 2014, 08:23:57 pm
content: I knew a guy who had something similar happen to him. Somebody neglected to unlock the nosewheel steering, and when the tractor turned the drawbar did so, but the nose gear did not. Shear bolts did as designed, and snapped off. Aircraft rolled down the gentle slope and 5m further stopped. No brakes as no hydraulics and pump was still being pumped for braking on impact. Nose cone left a 5cm hole in the 6mm steel wall bands, and in the rammed earth behind them. Too bad the nose cone is not the front most projection, that is a pitot static probe. That sheared the 16 rivets mounting it to the frame and went backwards, snapping off the 2 air lines and the 2 heater wires. Then it went through the main relay box about half way, at the level of the internal mounting plate, tearing it and all the wiring to pieces. not a cheap repair that one. Relay box was not repairable, and had to be remade. Quick fix was a new nose borrowed from another plane in for overhaul, it was not going anywhere in any case for the next year. Did lead to some fast retraining and testing for all flight line personnel.

author: Bored@Work
author_rank: Super Contributor
timestamp: January 07, 2014, 08:30:07 pm
content: And next time when you hear people here in the forum or at work bragging how cool they are for not following procedures or processes, that safety regulations are for wimps, that doing it the ghetto way is what real men do, that they don't need all that crap that is just slowing them down and limiting their creativity, and the establishment is just trying to deny them their freedom of expression ...... thats why they are wrong.

author: Maxlor
author_rank: Frequent Contributor
timestamp: January 07, 2014, 08:57:48 pm
content: I see you like procedure. But my first gut reaction when reading the accident report was, there's such a whole lot of procedure that people have stopped thinking about what they're doing. What in the report is described narrowing of focus (or similar.)I've experienced that attitude during my stint in the army; if procedure dictates everything I do, I stop thinking about what I'm doing. If I did otherwise, I might annoy someone, which usually has negative consequences. If I just concentrate on my tasks, I'll be fine, even if the guy next to me makes a mistake that I might have noticed... but hey, it's his problem and responsibility, not mine.I'm very glad that the work climate these days is a lot different. There's some overlap between us developers, and it happens every so often that I see something that seems odd to me, that I don't understand. So I question it. Most of the time, it's really nothing, and I might even learn something. Sometimes though issues are discovered (and promptly fixed.)People aren't robots. If you treat them like robots by giving them rigid instructions, you'll usually find a boat load of procedure violations after a while. It seems to me that instead of fighting against the negative aspects of being human (like us all being lazy to a degree), we should make use of the positive aspects, like giving people a reason to be there (as opposed to a robot), which makes them feel needed and important. And finally, let's give people training and schooling (which creates understanding) instead of just instructions (which is just procedure again, no understanding required.)

author: KJDS
author_rank: Super Contributor
timestamp: January 07, 2014, 08:59:57 pm
content: Quote from: Bored@Work on January 07, 2014, 08:30:07 pmRules are made for the guidance of the wise and the blind obedience of fools. I've met a lot of people that think they are in the "wise" category but aren't.

author: nctnico
author_rank: Super Contributor
timestamp: January 07, 2014, 09:09:34 pm
content: One of my personal experiences on my first job: the senior design engineer put 2 electrolytics in series to get a capacitor with a higher voltage rating. I wanted to put the proper capacitors in it but he insisted it would be OK. A couple of months later I went to install the equipment (worth an amount with 7 digits) at a major bank. So there I was in the equipment room of a bank (those are huge!) and doing some tests. Just as the bank's project manager walks in to see how things where going the capacitors blew... I don't think I ever looked unhappier. Fortunately the bank's manager made a joke from the situation.

author: vk6zgo
author_rank: Super Contributor
timestamp: January 08, 2014, 02:47:56 am
content: If the Electros are the same capacitance & voltage rating,it should work,& has worked on countless occasions.Of course,it will double the ESR,which may be critical in a particular application.What very few people do,however is to say,use two 25vwkg caps in series hoping to get 50vwkg.It is far more common to put two 40vwkg caps in series to give a bit more lee-way.

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: January 08, 2014, 03:28:05 am
content: "The Responsible Test Engineer (RTE) did not "assure" the turnover cart configuration through physical and visual verification as required by the procedures but rather through an examination of paperwork from a prior operation."I wonder how the QA manual got updated to make sure that did not happen again.Maybe an RTE overseer.

author: GEuser
author_rank: Frequent Contributor
timestamp: January 08, 2014, 03:35:35 am
content: Like Bored@Work mentions , these pictures in the OP also reminds me of those people that post @ new beginners or others just inquiring , thosespecial peoplepromoting their "intelligence" and opinion but without actually adding anything to the inquiries , like>Study before you ask questions! (recent)Try something easier in the hobby your too ahead of yourself , read books and study! (recent)Don't ask questions without studying first ! (recent)They are obviously sitting there in front of the computer all knowledgeable and superior in their perfect world with supposedly excellent advice yet seem to exist to post only that , the rant is nearly over btw , these "experts" are the one reason that I could get banned from this forum as thereisquite a few here imo and instead of enlightening (not feeding with spoon btw) or imparting a road to stay on or track they seem to actually deride those trying to learn like smartarse effing nitwits , I'd like to smack em one in real life but all I could say or type in this isolatedrelativeone way communication is drivel like this < .And look at those pictures nerds!

author: vk6zgo
author_rank: Super Contributor
timestamp: January 08, 2014, 03:50:34 am
content: Quote from: GEuser on January 08, 2014, 03:35:35 amAll quite fair answers!We get a few people who seem to think this forum is a free school.---well,sorry,it isn't!The knowledgeable people who volunteer part of their time to answer beginner's questions spent substantial parts of their life to acquire that knowledge,& know from bitter experience that shortcuts to proficiency in Electronics do not exist.For that reason,they fairly understandably,expect a little bit of effort from the questioner.

author: amyk
author_rank: Super Contributor
timestamp: January 08, 2014, 08:04:46 am
content: Quote from: Maxlor on January 07, 2014, 08:57:48 pmI gave up trying to decipher all the bureaucracy in the report and just skimmed over it. IMHO it's saying a whole lot of nothing... 100+ pages around something so simple? No wonder it cost $130 million to fix...QuoteApparently "this is a bloody expensive piece of kit, let's make damn sure it's not going to come off the mount when we turn it" was not in the minds of anyone working on it.

author: Kjelt
author_rank: Super Contributor
timestamp: January 08, 2014, 08:46:36 am
content: Quote from: nctnico on January 07, 2014, 09:09:34 pmQuote from: vk6zgo on January 08, 2014, 02:47:56 amThis needs an extra voltage devider across the caps (two high ohms resistors) to make sure the common connection in the middle has exactly half the voltage otherwise it could be that one elco gets to endure an higher voltage (higher then it is speced).

author: TMM
author_rank: Frequent Contributor
timestamp: January 08, 2014, 08:50:03 am
content: Quote from: nctnico on January 07, 2014, 09:09:34 pmIt would have been ok... if he added balance resistors appropriately

author: nctnico
author_rank: Super Contributor
timestamp: January 08, 2014, 09:35:18 am
content: Or just ordered the right capacitors. It was only 200V or so not needing low-ESR or anything special. It was a slip up from an otherwise very competent engineer.

author: grumpydoc
author_rank: Super Contributor
timestamp: January 08, 2014, 09:41:56 am
content: QuoteHadn't heard of that but I suppose it makes sense - more athttp://www.illinoiscapacitor.com/pdf/Papers/voltage_balancing_resistors.pdffor anyone who, like me, didn't know this was necessary.QuoteQuite but when you've something around that "will do" but is not ideal there is a high temptation to bodge it and get the lid back on there and then rather than leave the job around waiting for a delivery.

author: SeanB
author_rank: Super Contributor
timestamp: January 08, 2014, 09:43:07 am
content: Even the OHL power supplies have the resistors in, if they were not needed you know they would be cost cutted out. Of course they use a 0.25W unit there which fails after a short period as open and then the supply goes bang.

author: peter.mitchell
author_rank: Super Contributor
timestamp: January 08, 2014, 10:06:41 am
content: Quote from: SeanB on January 08, 2014, 09:43:07 amOnly if it makes it past the "plug it in and make sure it does pop" test, so if it does pop, it pops with the customer some time later and its their problem.

author: Kjelt
author_rank: Super Contributor
timestamp: January 08, 2014, 11:44:19 am
content: Quote from: SeanB on January 08, 2014, 09:43:07 amSaw applications that used for the balancing resistors two times three SMD 0603 in series which are even lower power, but the resistance is so high it should withstand mains 24/7 without problem, don't see the problem. Maybe a different topic, a schematic would help.

author: London Lad
author_rank: Regular Contributor
timestamp: January 08, 2014, 12:32:33 pm
content: Quote from: amyk on January 08, 2014, 08:04:46 am^ This

author: TerraHertz
author_rank: Super Contributor
timestamp: January 13, 2014, 08:17:03 am
content: Urrgh! I'd seen the first image but not the others or the NASA 'incident report'.Sigh. 90 pages of avoiding saying it straight up - "we forgot to put any bolts at all in, dammit". Every time they mention bolts they first go on about how they put in every second bolt on one adapter, and that's normal. But only in Fig 5-20 (page 37) is it perfectly clear that the 'turnover cart adapter plate' requires 24 bolts to the baseplate, and NONE were installed. So the thing was just sitting there like a paperweight.I like how one guy noticed, and pointed it out to the others, and everyone ignored him. Ha ha ha... So typical.Also, what a wimp. Didn't have the guts to start yelling "Hey you f*cks, STOP! There are NO BOLTS! Just LOOK you jerks!"Or words to that effect.How ironic, that in a 'how we screwed up and smashed a multi-million dollar satellite, many pages are 'export restricted' and 'proprietary'.The whole thing is an allegory of why centrally planned economies have never worked and can never work. Because they condition everyone to think like those guys working on the satellite that Saturday.Oh well. I suppose I'm just feeling extra cynical, having in the last week witnessed yet another instance of some great old Australian technological history trashed, via institutional incompetence bordering on criminal negligence. Or maybe even deliberate bloodymindednes for all I know. I'm not privy to the reasoning behind it. Nor can I talk about it. Very frustrating.(Edit: typo)

author: GEuser
author_rank: Frequent Contributor
timestamp: January 13, 2014, 11:03:38 am
content: Quote from: TerraHertz on January 13, 2014, 08:17:03 amIn bold above , WHAT! you mean that they have got rid of the short drop AND long drop nowbtw , I don't believe one word that one is overly cynical , its really going on , really really stooooopid stuff and it'll be getting worse as it has too imo .The analogy I use for the op's picture is nerdburglerizm at its best , and that is all the upperclass sat builders who are of course superiorgot rid of the nut and bolt guy as who needs a nut and bolt guy , anyone can do that even a monkey the monkeys say , so we'll do the nuts and bolts too as its sooooo easy and we can save $40 thou by getting rid of that pesky low intelligence nut and bolt guy , its easy ...btw , I have a bias in the above story real or not real as I'm a nut and bolt guy also and over the years have really screwed some nuts , ya see the nuts I gather think they are also superior to the bolts so have tried to rid themselves of the bolts but forgetting as they be nuts they'll have no existence in existence just being a Nut , so I say screw em! Ha Ha Ha Ha ...

author: EEVblog
author_rank: Unknown
timestamp: January 13, 2014, 11:28:14 am
content: Quote from: SeanB on January 07, 2014, 07:19:20 pmNot just one bolt, but all of them!That jig has only two operations, to rotate and tilt the satellite for working on. The only thing that holds it down is 24 bolts.How on earth do you press a button to tilt a $220M satellite without multiple people checking it's bolted in place?And not just checking, but probably being paranoid about having all 24 in place and torqued correctly etc.

author: rolycat
author_rank: Super Contributor
timestamp: January 13, 2014, 11:40:54 am
content: Quote from: KJDS on January 07, 2014, 08:59:57 pmSir Douglas Bader.QuoteProblem is we are all fools at one time or another.

author: John Coloccia
author_rank: Super Contributor
timestamp: January 13, 2014, 10:49:18 pm
content: Quote from: EEVblog on January 13, 2014, 11:28:14 amBecause of where I worked at the time, this incident made the rounds almost instantly and somehow I ended up with grainy, photocopied pictures of it. I also heard a good deal about how the incident occurred, right after it occurred. I don't know what the report says, but I remember reading and hearing an awful lot of BS in the months following. It's been a while, but my fuzzy recollection is that the accident was as simple as dope #1 "borrowing" the bolts for another project, and dope #2 not checking that the thing was actually bolted down. I guess it HAD been bolted down at one point. That's what I remember about it, anyway.I have to say that in my career, I've been involved with a number of expensive accidents like this (not this particular one, but ones like it). The thing that the final reports never seem to address is WHY so many things could go wrong...how does the environment get to that point. I'll tell you a story how:I remember one incident, and I won't get into details, but there was some big thing that moved, and any time it was powered up and could potentially move there was a guy holding "the pickle"...essentially an E-STOP. Any time motion was expected, someone would clearly indicate that, and the guys running the software had supposedly tight coordination with the guy manning the pickle. Well, things being what they are we fall behind schedule...shit, what am I saying...we were always behind schedule. We were falling DANGEROUSLY behind schedule on just about everything, and there was tremendous pressure to produce produce produce. Also, someone decided, "Well, I'll show these assholes...they'd better get on the ball and do everything right, and incidentally, anytime someone E-STOPS this thing, we're going to have a big meeting, and a root cause analysis, and someone's going to answer for what happened and why!!".That'll show'em, right? See where this is going? One day, they're running a test, and NO motion was supposed to occur. Things start moving...."Uhm, are we expecting motion?" AHHHHH!!! "NO NO NO!!!!!!!". Oh well, too late. Pickle man hesitated because he knew his ass would be on the hot seat if he made a boo boo. End result? Some damage to a very expensive piece of equipment.Investigative finding? The software was at fault. It did not adequately warn the operator that running the command could potentially make something move that wasn't in the right configuration to move. Solution? Put a warning on any command that could potentially cause damage. That goes down to the lead engineer for that part of the project. He asks for the list of commands that they feel could damage the system. No one can decide on which commands can cause damage, so they put it on the software lead's shoulders to figure out every possible interaction on this tremendously huge and complex system being built by multiple vendors and multiple teams.HIS solution? Well, he's not putting HIS ass on the line, so he makes EVERY command in the system pop up this message...some nonsense like, "Grave Damage May Occur!!!!". Not exactly that, but close. End result? Well, it conditioned everyone to completely IGNORE that message because it was absolutely meaningless.But lost somewhere in the mix was the real root cause: CRAPPY management punishing their employees just for doing their job the best they could, and employees making the very rationale decision that it's better to avoid getting in trouble than actually doing their job. THAT part never seems to make it into any of the fish bones or final reports.

author: vk6zgo
author_rank: Super Contributor
timestamp: January 14, 2014, 04:27:40 am
content: Back in the very late '80s,TVW7 moved the Studio Earth Station from over the road to the Stationproperty,& did an excellent job of it.A huge crane lifted the dish off the pedestal assembly & gently placed it out of harm's way,then picked up the (huge) pedestal assy & set it down on a pre-prepared concrete base.No disasters,no nothing,but there were a couple of guys from the original manufacturer who came over to help out.They told an interesting story of how in an anonymous city in the USA,they had been approached about moving the hut containing the Earth Station Electronics (Uplink Tx ,Downlink Rx.etc) from one side of a bulding roof to another.The inquirer ,on seeing the quote,decided they could do it themselves much cheaper.On the day,all went well,until the Uplink Tx PA,which was mounted on rollers in its cabinet came flying out of the hut door & went crashing to the pavement,many floors below.Luckily,no one was injured,but it all cost a lot more than the original quote,by the time all the lawyers were finished with it.As it is all hearsay,I don't know any other details.