Tips for diagnosing highly intermittent problems? - Page 1
Forum section: electronics / general

author: c4757p
author_rank: Super Contributor
timestamp: July 31, 2013, 05:01:37 pm
content: I'm considering selling my beloved Tek 2445A - but I don't want to sell it unless I know it's in perfect condition. About once out of every 20 or 30 power cycles (or more), it fails a startup test related to the frequency counter plugin. If I clear the error message, everything functions perfectly, and it certainly doesn't repeat on the next boot, so no use disassembling it to probe around, everything's back to proper function. It's test 82, which just seems to be a generic "signal path" check, and none of the components listed seem faulty. No cables or socketed chips are loose either, and I've resoldered a number of joints as well. Does anybody have any tricks for tracking down these damn errors?

author: Rufus
author_rank: Super Contributor
timestamp: July 31, 2013, 05:18:25 pm
content: Quote from: c4757p on July 31, 2013, 05:01:37 pmWhen you have a problem too intermittent to investigate one approach is to try to make the problem worse because it is so much quicker to tell that something made it worse than that something made it better.Obviously try to narrow down the area as much as possible then try warming it, cooling it, more and less supply voltage. Pull a signal up a bit or down a bit or load it with a bit of capacitance. That sort of thing.

author: c4757p
author_rank: Super Contributor
timestamp: July 31, 2013, 05:30:42 pm
content: Quote from: Rufus on July 31, 2013, 05:18:25 pmI didn't even think of that. Thanks!

author: ablacon64
author_rank: Frequent Contributor
timestamp: July 31, 2013, 06:20:50 pm
content: Quote from: Rufus on July 31, 2013, 05:18:25 pmInteresting approach. I've had some success by cooling and heating (coolant spray and hair dryers).

author: jeroen74
author_rank: Frequent Contributor
timestamp: July 31, 2013, 06:44:34 pm
content: The other day I used freeze spray and heat gun to pinpoint a timing problem between a FPGA and some external SPI device. Heating it up with a heatgun would expose the timing problem, subsequent cooling made it go away. On some PCBs self heating of the devices was enough to trigger the problem after a while.It took quite some debugging and hair-pulling to even come up with the hypothesis of a timing problemNot having any source code of the FPGA didn't make things any easier either. The configuration was basic enough (basically a softcore with run-off-the-mill peripherals) that it was easier and quicker to design a new FPGA rather than trying to have the external party fix it.

author: c4757p
author_rank: Super Contributor
timestamp: July 31, 2013, 06:51:40 pm
content: My gut suspicion is somewhere in the discrete level shift circuitry between the ECL and TTL sections; this is covered by that error code, I believe, and I can see if the levels coming out of the level shifters were right on a threshold that slight temperature or power supply variation could cause the occasional bit here and there to miss. I haven't gone that far, though, just because up until now I haven't really been interested in it...

author: Skimask
author_rank: Super Contributor
timestamp: August 01, 2013, 04:23:44 am
content: Quote from: Pippy on July 31, 2013, 08:35:01 pmWait a minute...So, is there a type of "Intermittent faults" that doesn't "only happen once in awhile"

author: TerraHertz
author_rank: Super Contributor
timestamp: August 01, 2013, 04:37:50 am
content: Quote from: Skimask on August 01, 2013, 04:23:44 amThere are the intermittent faults that only happen when your debugging equipment is not turned on.Or the ones that _never_ happen when you are trying to demonstrate them to someone else.Then there are the ones that no one has ever seen happen. Just the consequences.The very worst, are the ones that only happened in someone's imagination. But they insist you fix them.

author: notsob
author_rank: Frequent Contributor
timestamp: August 01, 2013, 04:40:51 am
content: long long age, when installing mainframes, we used to run them thru test routines at a slightly higher than nominal supply voltage and then vary the clock speed by a plus and minus value - fix any problems that resulted, then repeat at a lower than nominal supply voltage, then return them to nominal clock speed and supply voltage. You probably do not have the same luxury, but varying the supply may help to isolate it.

author: Stonent
author_rank: Super Contributor
timestamp: August 01, 2013, 06:01:48 am
content: Thump around with the plastic end of a screwdriver, try to flex the pcb.When Mike of Mikeselectricstuff was trying to fix a thermal imaging helmet he tapped around and flexed the board and found the fault.

author: dfmischler
author_rank: Frequent Contributor
timestamp: August 01, 2013, 10:25:15 am
content: Quote from: notsob on August 01, 2013, 04:40:51 amOh, yeah. I remember when I was using an early PDP-8 (straight 8 ) in college it had a "margin checking" power supply built in to it to facilitate this kind of testing. Yes, the machine was already quite obsolete at the time.On the other hand, in the mid 80's I had 23 PDP-11's with DEC communication processors that locked up frequently (a random unit every few days). They were spread out over 150 miles of Interstate 90 in northern Indiana. I wound up renting 4 logic analyzers to increase the event capture rate, and eventually isolated the problem to a fixable software/firmware interaction problem. But that approach won't help somebody who is trying to fix one misbehaving device.

author: Rufus
author_rank: Super Contributor
timestamp: August 01, 2013, 01:21:26 pm
content: Quote from: notsob on August 01, 2013, 04:40:51 amThat is a way to ensure you have some margin on voltages and timings.The better device programmers have options to verify at high and low supply voltages for the same reason. It is probably part of the programming specifications for some devices.

author: SArepairman
author_rank: Frequent Contributor
timestamp: August 01, 2013, 02:33:22 pm
content: what is the best way to reduce voltage on a supply rail for a piece of test equipment?Connect a dummy load to it?

author: AG6QR
author_rank: Frequent Contributor
timestamp: August 01, 2013, 06:30:08 pm
content: Quote from: SArepairman on August 01, 2013, 02:33:22 pmDepends, doesn't it? If the supply is overengineered, it's going to take quite a large load to drop the voltage much. If the current demands of the instrument vary from time to time, then a load that's large enough to drop the supply rail a bit during nominal conditions might blow a fuse or trigger some other overcurrent protection when the instrument demands its maximum current.If your goal is merely to lower the supply rail to a specific value, then substituting an adjustable bench supply for the instrument's built-in supply is the way to do this in a controlled way, assuming the built-in supply doesn't already have enough adjustment built-in.But yeah, depending on the fault, overstressing the supply with an added dummy load just might stress the supply enough to make an intermittent problem repeatable. If the power supply folds back to a low voltage when its current limit is exceeded, it could be that the intermittent problem you're seeing is caused by this lowered voltage at high current conditions. Drawing a bit more current might reproduce it more reliably.

author: c4757p
author_rank: Super Contributor
timestamp: August 01, 2013, 07:50:41 pm
content: Quote from: SArepairman on August 01, 2013, 02:33:22 pmThis is averysubstantial supply, you'd need a hell of a dummy load. I'll just connect an external supply.

author: brainwash
author_rank: Frequent Contributor
timestamp: August 01, 2013, 09:13:26 pm
content: I second the supply approach, if tapping and freezing does not do anything.It's handy to have a second scope around, those 5V you measure at the supply might actually be 2-8V. High-pitched noise might also pinpoint something.If the voltages are low enough I often use my finger to place 'capacitance' on the board. Normal production boards have no problem with that.

author: TerraHertz
author_rank: Super Contributor
timestamp: August 16, 2013, 11:49:20 am
content: Heating and cooling electronics tends to have a very similar effect to varying the supply voltage, and is easier and safer. One thing you have to be very careful about in some gear with multiple supply rails is that having individual supply rails drop out of even just come up out of sequence can destroy circuitry.Heating and cooling will also 'exercise' faults that are due to physical intermittents - bad jolder joints, ball joints under flatpacks, and I've even found thermal intermittents that seemed to be due to a flaky chip bonding wire contact in ceramic cavity packages.Of course heating is easy - a hairdryer will do the job.Cooling - if you don't have access to spray coolants, one trick I've often found useful is to crush some ice up and put it in a plastic bag (or two, for extra leak protection.) Then you can mold the icepack over the suspect circuit area. Cool a large area, if the fault appears, see if you can make it go away again by reheating specific small areas or components with a hot air gun with fine nozzle.Also as mentioned, manually flexing the board.

author: wraper
author_rank: Unknown
timestamp: August 16, 2013, 03:46:15 pm
content: Nokia instruction how to use cold spray.

author: c4757p
author_rank: Super Contributor
timestamp: August 16, 2013, 09:10:45 pm
content: Oh wow.... I forgot about this thread!I've been too busy with other things to even bother with it. I'll have to take it apart tomorrow and see what I can do. I'll have to bodge together some sort of cable to let me turn it on with the options bracket removed. IIRC Tek used to sell one...All I can say is.... thank god for the "continuous self-test" mode! I can only imagine if I had to keep rebooting the damn thing over and over...