Can not find root cause for field failures of my circuit - Page 1
Forum section: electronics / projects

author: rtm
author_rank: Contributor
timestamp: January 18, 2021, 11:13:49 pm
content: Hello all,I am new here. I am looking for help in trying to identify what could be causing the field failures of a fairly simple voltage reference circuit I designed a long time ago. Recently we have had a rash of failures in this circuit, but so far have been unable to identify a root cause.This circuit is a simple 68V reference developed from an existing 74V rail. The circuit is designed to fold back when when excessive current is drawn from the reference (limit is 350 ma). Please refer to the attached schematic.The failure mode we are seeing is that quad op-amp U4 section A is failing. It appears that the output driver for the op-amp no longer works. The other sections work fine. We have also seen a couple of instance where U4 section A failed along with resistor R18 being browned/blacked due to being over powered. R18 is a MIL specified 1/4 watt (so roughly commercial grade 1/2 watt) metal oxide resistor.PS1 is a fully isolated (to 3000V) DC-DC converter capable of 1 watt output. It is powered by the system 5V supply rail (Vcc). PS1 is running at low load, so it is actually developing a typical voltage of 5.5 VDC supply rail for U4.This is used in a heavy industrial application. Under normal circumstances there would be less than 50ma external load on this reference. The external load impedance is essentially resistive.Some testing that has been tried - but did not induce any failures.- Inject positive and negative going transients on the 74V supply rail of approximately 500V and less then 1 joule power.- Short the 68V reference to ground for an extended period of time.- Short the 68V reference to the 74V supply rail for an extended period of time.- Apply a 360ma restive load to hold the circuit in fold back mode for an extended period of time. Q1 gets hot, but the heat sink keeps things within limits. No failure occurs.- Attach a relay (74V coil) through one of it's own normally closedinterlocks so that it chatters when connected. No suppression was added across the relay, and the interlock was inserted in both the positive and negative sides of the coil connection. This relay was allowed to chatter for a least 1/2 hour. The idea was to inject some big inductive spikes directly onto the 68V reference. No failures occurred.- Tried removing the 5V Vcc supply while maintaining the 74V rail. No failure.I would appreciate any help anybody can offer.Thanks in advance!

author: S. Petrukhin
author_rank: Super Contributor
timestamp: January 18, 2021, 11:56:49 pm
content: In your + / - powered the op-amps fly in the air relative to Gnd, they have no potential relative to Gnd. After all, you have an isolated PS1. But you omit the lines to Gnd. This shouldn't work.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 12:06:36 am
content: Mr. Petrukhin,Your English is fine. Thank you for the reply.The negative side of the DC-DC supply is referenced to the 68V Reference itself. The whole circuit rides on top of the 68V line.The ground reference for the circuit is via R22 and RH1. The ground appears to U4 section D as a -68V input, and is scaled and inverted by U4D to become +2.5V. The output of U4D is compared to the 2.5V reference REF1 by U4A. U4A will drive the transistor ON if REF1 is higher than U4D output. U4A will drive the transistor off if REF1 is below the U4D output.The circuit can be a little hard to understand, because it essentially "floats" on the 68V reference voltage.

author: S. Petrukhin
author_rank: Super Contributor
timestamp: January 19, 2021, 12:31:53 am
content: Quote from: rtm on January 19, 2021, 12:06:36 amThank you, now I understand - you are forming a virtual Gnd near 74B... This scheme breaks my brain.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 19, 2021, 12:33:59 am
content: Quote from: rtm on January 18, 2021, 11:13:49 pmI assume Q1 (TIP49) on the failed boards is failing too. I'm wondering if the base-collector junction in Q1 is failing due to voltage/thermal overload, leading to burn-out of R18 and U4/A.Are the Q1 transistors on the failed boards genuine TIP49s, and not some fake parts? Do you always buy these from a reputable distributor, like Digi-Key? Apart from fake parts, another explanation would be aging/degradation of Q1's thermal mounting. I've seen the usual suspects selling fake "Arctic Silver thermal paste", in the past.Do you do heat-soak testing (load testing at elevated ambient temperature) of these units before shipping them? Anomalous voltage across R18 during heat-soak might reveal a problem with Q1.Can you say more about Q1's heatsinking arrangement ,and what suppliers you use.

author: rfclown
author_rank: Frequent Contributor
timestamp: January 19, 2021, 12:52:01 am
content: Say you have 1V across base emitter of Q1 and 1V across D5. MCP6024 datasheet says rail-to-rail, so say there is 5.5V at output of U4a with respect to VREF. (5.5-1-1)/(121+1) = 28.7 mA. Absolute max on MCP6024 datasheet is 30mA.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 12:56:31 am
content: Thermistor-guy,The odd thing is is that Q1 is NOT failed. Typically only U4 is failed. We've seen 8 field failures now, and only two have had U4 and the R18 failed.The failures have been on boards that are in service from a few days to a few months - some even years.I suspected some sort of collector-base breakdown. But I can't seem to make it happen on the bench. The fact that Q1 has never failed is puzzling.We use reputable distributors, like Mouser, Arrow and Digi-key. The heat sink on Q1 being used is Wakefield 647-15ABPE (Digi-key 647-15ABPE)I bench tested one of the transistors using a curve tracer and did not see any Collector Base breakdown until about 650V was applied (The part is rated at 350V).I didn't suspect the thermal paste, because the design of the heat sink keeps the leads away from the sink. But I will check to see if any has migrated between the leads. We do not do any sort of heat-soak testing. I have looked at the voltage across R18 during test and net seen anything unusual. I can try heating Q1 and looking again.All our boards are conformal coated. I did try a moisture test - allowing heavy condensation to form on the PCB and then testing. Did not see any issues.Thanks for your input. I really appreciate it.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 01:06:25 am
content: rfclown,The output of U4A would only reach the positive rail if the 68V reference is shorted to ground or heavily loaded so that the fold back circuit comes into play. I did suspect that maybe the op-amp was being overloaded, since as you calculated, it could be running near the maximum current during a fault. I tested this by shorting the 68V reference to ground for well over 1/2 hour. U4 did not even get slightly warm, and the circuit recovered just fine when the short was removed.In the application, the fault condition should be seen only rarely. This would be due to a miss-wire or possibly a failed component in some downstream module. From the field reports, the claim is that only this module was failed. No other modules were replaced.However, I will look into increasing the value of R18 to reduce the current from U4 during the fault condition.Thanks!

author: rfclown
author_rank: Frequent Contributor
timestamp: January 19, 2021, 01:17:56 am
content: I wasn't trying to understand everything at first, just looking up datasheet numbers and imagining worst case conditions. The 1V numbers for Q1 and D5 were datasheet numbers. Now that I see a little more what's going on, when fold-back occurs, U4C can only sink 30mA max also.

author: rfclown
author_rank: Frequent Contributor
timestamp: January 19, 2021, 01:33:31 am
content: When I pulled up the MCP6021 datasheet, the first things that stood out to me on page one for your case:Wide bandwidth 10 MHz. Thought: He doesn't need that kind of bandwidth. I've been burned by oscillations in using Opamps that were higher bandwidth than I needed.Power Supply Range 2.5-5.5V. Thought: Not much room for error here.

author: floobydust
author_rank: Super Contributor
timestamp: January 19, 2021, 02:50:03 am
content: I don't see coverage for -ve transients from the 74V rail. When you inject -500V at Q1-C, TZ2 turns on and then E-B junction of Q1 breaks down feeding the transient back into U4. D5 isn't blocking that.It depends on the inductance of field wiring and capacitance on the 74V rail, how bad that transient is.You could add a series diode to R18 to prevent backfeed and a E-B resistor to Q1 to keep speed up. Or just hard-clamp -ve transients by having a reverse-diode across either the 74/68V in and output.I'm not sure what the load is or how far away it is.

author: xavier60
author_rank: Super Contributor
timestamp: January 19, 2021, 03:24:29 am
content: Or just a diode in series with the 74V in, if it hasn't already been mentioned.There would be no need to worry about a possible negative transient reaching the reference output.I get a lot of 74VDC repair work off of rail locomotives.Some of the damage I see indicates that the supply is positive chassis but no one has confirmed it to me yet.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 03:32:12 am
content: Rfclown,The op-amp is one we’ve used in many other circuits. It was chosen because it is rail to rail and met the other requirements needed for the circuit. Bandwidth was not a critical parameter. The circuit is very stable until fold back occurs. Occasionally we have seen oscillation during fold back in our recent testing trying to find the failure cause. A 0.1 uF capacitor has been added at C11 to prevent any oscillation during fold back.The absolute max for the power rail is specified at 7 volts for the op-amp. I am fairly sure failure is not related to the power supply since no other sections of the chip have failed. It has always been U4 section A.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 03:49:16 am
content: Quote from: floobydust on January 19, 2021, 02:50:03 amFloobydust,There is another 1.5KE200CA suppression diode across the 74V supply at the module input. I had not considered that a negative transient could breakdown the E-B junction of Q1 causing over voltage of U4A output. The suppression diode at the 74V supply would l8mit this transient, but likely not enough to prevent E-B breakdown. I was considering a diode in series with R18 because I suspected some sort of backfeed to the op-amp. I could not see a way it was happening though.I think your E-B breakdown during negative transients explains a very likely possible path for the backfeed. I am going to try create a failure with negative transients. Then I’ll add the diode and E-B resistor to see if it prevents the failure ( assuming I can cause one).Thank you very much for your insight on this issue.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 03:55:43 am
content: Quote from: xavier60 on January 19, 2021, 03:24:29 amXavier60,This is in fact a railway application. Normally the 74 V supply is isolated from the chassis. But I have seen many cases where the positive side of the 74V supply gets shorted to chassis due to old wiring, chafed wiring, or failures of EMI suppression capacitors within electronics.I could add a series diode to the 74V supply for this module easily. I’ll see if I can induce a failure with negative transients. Then add the diode, and see if it prevents the failure.Thank you very much for your suggestion.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 19, 2021, 04:04:06 am
content: Quote from: rtm on January 19, 2021, 12:56:31 amThe fact that Q1 doesn't fail is an important clue.Suppose it takes 100 mA through R18 to burn it out (1.2 W). Suppose also that in the failure mode, U4/A output is trying to source that current. Depending on heatsinking, Q1 could survive that base current longer than U4/A could. U4 has an absolute maximum supply current rating of 30 mA.Your foldback circuit has an overcurrent threshold that depends on the output of isolation converter PS1. PS1 output can be higher than expected if the load on it is too low. Murata datasheet states a minimum load current of 10% rated load current (data sheet:https://www.mouser.com/datasheet/2/281/murata_kdc_nmv-1186528.pdf). One thing to check is that PS1 is loaded correctly.I'm trying to think of a circuit mode where U4/A tries to source over 30 mA into Q1 base, but not under normal test conditions (because you would have detected it otherwise).One possibility is under power cycling, where the circuit loses all power but later tries to start up with a load on the 68 V output. If the 74 V supply to Q1 is slow to come up, then U4/A will be forced to supply the 68 V output current by itself, initially. If U4/A fuses open during this transient, the R18 will be okay; if U4/A fuses short, then R18 will get overloaded. This is consistent with your fault symptoms, but I don't how if this failure mode is plausible.(edit: Also, if the 74 V is very slow to come up, U4/A might be trying to pull it up via Q1 base-collector junction).I imagine that power interruptions occur in the field, from time to time. Do you do power cycling tests before shipping the product? And do the tests mimic what happens in the field?

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 04:41:21 am
content: Quote from: thermistor-guy on January 19, 2021, 04:04:06 amPS1 is lightly loaded. It may be under the minimum specified by the DC-DC at some times. I’ll have to double check. But my old calculations showed the circuit would meet the minimum load when in operation. In any case, I’ve measured the DC-DC output on about 15 Modules and they fall into the range 5.5 volts +/- 0.2 volts.The fold back circuit is minimally dependent on supply voltage. The foldback reference is dominated by the scaled 68V feedback voltage. A shift of one volt in the power supply rail, would cause less than a 5ma increase in the initial fold back current.The way the system works, the 74V rail is always first to come up, since it is the source of power for the 5V Vcc rail. The 5V Vcc rail is generated on the module, so if 74V is lost, so is the 5V.The modules are power cycled during normal testing, it is similar to what is seen in the field, but not the same since in the field power is from batteries and an alternator. During test we use a DC supply. There is no specific power cycling test during manufacturing. I have tried multiple power cycles on the Module when trying to create a failure in recent testing, but have not induced a fault.I will try powering on into an overload condition. This is not something I’ve tried. I have powered on into the normal load many times.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 19, 2021, 04:57:23 am
content: Quote from: rtm on January 19, 2021, 04:41:21 amIf the 74 V is slow to ramp up, the 5V Vcc rail - and the circuit - might be operating before the 74 V rail exceeds 68 V. During that time, both junctions of Q1 would be forward-biased, and U4/A would be overloaded. Can you eliminate that as a possibility?

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 05:09:23 am
content: Quote from: thermistor-guy on January 19, 2021, 04:57:23 amThe 74V supply is basically switched on and off via a breaker. However it can either be just battery voltage or alternator voltage depending on if the engine is running. On batteries, the supply could be as low as 65 volts (drained batteries). The 5V Vcc rail has a low voltage shutdown when the input voltage is below about 55 volts. So there could be a time when the module is powered by a low battery , but not low enough for the 5V supply to shut down.I have tested in this condition without fault. But I can see where maybe U4 could be damaged long term by operation in this region. I will adjust the R18 value to prevent overload in this condition.

author: Vovk_Z
author_rank: Super Contributor
timestamp: January 19, 2021, 05:24:11 am
content: I would add a normally closed Shottky diode from U4a output to '+5V' power line (and possibly to the gnd line too). It'l help U4a to survive with transients at the output. It seems to me regulating transistor can withstand those transients but U4a not.

author: eblc1388
author_rank: Frequent Contributor
timestamp: January 19, 2021, 06:58:25 am
content: I would replace TIP49 by a high voltage Darlington like TIP150~TIP152 and suitably increase the value of R18.

author: Conrad Hoffman
author_rank: Super Contributor
timestamp: January 19, 2021, 02:25:19 pm
content: I'd be looking at process changes in the components, ECOs and ESD. Searchhttps://www.google.com/search?q=esd+the+walking+wounded&oq=esd&aqs=chrome.0.69i59j46i131i175i199i433j0i433j0j0i395j0i395i433j0i131i395i433.2006j1j1&sourceid=chrome&ie=UTF-8. After all, it worked for a long time and then didn't, so something changed and it wasn't the design.

author: Renate
author_rank: Super Contributor
timestamp: January 19, 2021, 04:35:27 pm
content: Doesn't U4B need a 4.02k resistor going from pin #5 to somewhere?

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 06:05:21 pm
content: Quote from: Renate on January 19, 2021, 04:35:27 pmRenate,U4B is in a non-inverting arrangement with a gain of 5.02. The 1K to the non-inverting terminal is to balance the 1K at the inverting terminal - and for some current limiting in case of a fault.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 06:15:39 pm
content: Quote from: Conrad Hoffman on January 19, 2021, 02:25:19 pmAgreed, I suspect some sort of process issue is a possibility. However, as far as I can tell nothing has changed process wise. The most likely process issue might be ESD damage. But I am hesitant to say that is the cause since it is such a consistent failure mode. Plus the two cases where R18 is burnt indicate that somehow excessive current was present, and normally that resistor would see at most (absolute worst case) 35ma which is only 148 milliwatts on an essentially 1/2 watt resistor.The other factor is that there is a very similar circuit on the module in another area, that has had zero failures of the same part number op-amp. That circuit also contains a stage that drives a TIP49 transistor in the same arrangement as this circuit. The main difference of that circuit is that it is powered from the 68V reference rather than the 74V supply rail.

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 06:19:44 pm
content: Quote from: Vovk_Z on January 19, 2021, 05:24:11 amI have had this same idea. But I really want to determine the failure root cause so that I can be confident that the fixes applied will solve the problem. Whatever fix is chosen will be applied to all modules, new production, field returns, and those returned for re-calibration.

author: jonpaul
author_rank: Super Contributor
timestamp: January 19, 2021, 06:36:15 pm
content: 75V PS source:Check for inductive spikes,Suggest to add transient absorbing zener or snubber and filter.Heave duty 28V avaition TAZ from Mot is MR2525 I think, perhaps 3 in seriesJon

author: Renate
author_rank: Super Contributor
timestamp: January 19, 2021, 10:21:15 pm
content: Quote from: rtm on January 19, 2021, 06:05:21 pmEh, I'd still have put one in. It would make it prettier and a regular differential amp. Anyway...I see that you have no output capacitor, C11 is marked as "open".Does your load have any capacitance snuck in?What happens when the 74 V collapse to zero?With the 68 V still hanging with capacitance you forward bias CB on Q1 and drive the right side of R18 to zero for ~60 V across it.In your disconnect experiments have you had a load resistor on the 74 V input and a bit of capacitance on the 68 V output?What was the philosophy of D5? Why wasn't it on the collector?

author: rtm
author_rank: Contributor
timestamp: January 19, 2021, 11:11:50 pm
content: Quote from: Renate on January 19, 2021, 10:21:15 pmD5 is in the emitter to give the fold back circuit via D4 and U4/C plenty of room cut off transistor Q1. Also it is to prevent reverse bias of the Base-Emitter junction in case of 74V collapse and 68V being held up by external capacitance.During recent testing I have added a 0.1uF capacitor at the C11 position to damp oscillations that occurred on some units when they entered the fold back current region. The external load is almost purely resistive in the application. There is no bulk capacitance connected to the 68V reference signal. The emitter is blocked from reverse current flow by D5, so I see no path for current to flow from the 68V line through emitter to base to R18 and then back thru the op-amp (other than leakage currents possibly).Since the entire circuit "floats" on top of the 68V reference, the ground pin of U4/A will always be at the same potential as the 68V reference signal. So I cannot see a way for high voltage to be across R18 from that path.During disconnect testing the 74V supply is switched on and off via a simple toggle switch. I have tested with an external load on the 68V reference, and with no external load. The module is powered by the 74V line, so the module power supply is the only load on the 74V when it is switched off.

author: Renate
author_rank: Super Contributor
timestamp: January 20, 2021, 12:10:29 am
content: If the 74 V goes to zero hard (i.e. not just disconnected):Q1 collector goes to zero.Q1 base goes to ~0.6 VU4A lower supply is still at 68 V because of some capacitance.U4A output is > 68 V.R18 has ~60 V across it.Lots of current flows.IC4A pops.Use your toggle switch, but put a load so that it goes down hard, not disconnected.Are you sure that your circuit is installed identically everywhere? Same load? Never shares a breaker with anything?Is that 5 V DC converter being fed from something off of the 74 V?

author: rtm
author_rank: Contributor
timestamp: January 20, 2021, 01:04:32 am
content: Quote from: Renate on January 20, 2021, 12:10:29 amThe circuit is part of a standard module configuration for a control system. I checked the overall schematics of the customer system where failures occurred, and they are identical. The entire set of modules for the control system are on one breaker. So all modules are powered on/off at the same time. Of course, there is bulk capacitance on the various modules across the supply rails, so I am sure there is some ramp down of the supply when power is removed.The 5 V DC converter is powered from the module 5V power supply which is derived from the 74V feed to the module. So when 74V is removed, the 5V supplies will loose input power. Again, there is bulk capacitance, so the 5V rails will fall after the 74V rail.I did not consider the transistor entering the reverse-active mode of operation. I'd have to see how the 68V line is being held up, but I can see now how there could be a large voltage across R18 if 74V collapses and the 68V reference is held up for any period of time.Thanks for your explanation. This is something I can bench test, and it looks like a simple diode in series with the 74V supply would solve the issue.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 20, 2021, 03:38:18 am
content: Quote from: Conrad Hoffman on January 19, 2021, 02:25:19 pmApparently, some of the failures had been working in field for a few years. So maybe something haschanged in the field.My background is telecoms, and I know what used to happen in telephone exchanges. It might be relevant here (railwayapplication), so I'll go into some detail in case it helpsBattery supply in an exchange was -48 V, powered by a dedicated room of lead-acid batteries. Busbars carrying hundreds,thousands of amps, ran though the multi-storey exchange building. The batteries were mains trickle-charged, with a dieselgenerator as backup. Battery supply could reach -65 V when the generator started up. The power went to aisle after aisleof equipment racks.Normally the exchange was very reliable, but when an installation crew arrived to fit/modify equipment, strange eventstypically occurred. "Mysterious" equipment outages would happen. Everything would settle down after the crew left. Why?Well, a crew member might accidentally drop a hand-tool across a busbar (high up near the ceiling), resulting in a smallexplosion of molten metal (the hand-tool) and a brown-out on a supply busbar, which had some inductance.The crew might temporarily connect to the power supply of a "suite" (an aisle of equipment racks), then accidentallyoverload the suite fuse, triggering alarms everywhere.A modern subrack, typically 3U to 6U high, used a switch-mode power supply module to convert -48 V to internally neededvoltages. The SMPS would turn off if the supply dropped to around -42 V or worse. The SMPS presented a negative incrementalresistance to the battery supply: the higher the busbar voltage, the lower the current draw by the subrack.Combine negative incremental resistance, busbar inductance, sudden load connects/disconnects, with maximum current drawat around -42 V, and you have the potential for unusual dynamic behavior in the -48 V supply within an equipment suite.Nearby suites would also see that transient.I mention this because something may have changed in the 74 V supply in the field. Installation activity, more loadadded, significant changes to the 74 V distribution arragement - something that isn't captured by normal benchtests.

author: floobydust
author_rank: Super Contributor
timestamp: January 20, 2021, 05:51:04 am
content: The branch circuit fed by the one breaker likely powers other (inductive) loads, making the 74V power dirty with transients, in the field compared with the lab.I would say it's easy to see -2kV impulses on a 74V DC power system. Using a relay-oscillator, the test voltage is clamped by the contact air-gap and there is no ringing. I would up the inductance, get some ringing happening and see if that induces failures. It's not just a single +ve or -ve blip, and 1N400x diodes are quite slow to turn off ~4usec and I would not use in series with devices.For problematic sites, sometimes I've added an MOV at the load generating the transients because other equipment will also be failing if you check with the customer.

author: Renate
author_rank: Super Contributor
timestamp: January 20, 2021, 12:05:39 pm
content: Is "74 V" even guaranteed to be greater than 68 V?Quote from: rtm on January 20, 2021, 01:04:32 amSo all the "modules" are this circuit plus individual (internal?) 68 V loads?The DC converters present a load to the input 74 V that will cause it to go to zero "hard".Any chance that 68 V outputs being fed from separate 74 V breakers are connected together (or a leakage path between them)?

author: xavier60
author_rank: Super Contributor
timestamp: January 20, 2021, 12:18:56 pm
content: Also if negative spikes are the cause, it would be mean that the DC-DC converters are tolerant to the negative spikes.

author: rtm
author_rank: Contributor
timestamp: January 21, 2021, 05:32:01 am
content: Quote from: Renate on January 20, 2021, 12:05:39 pmAll modules that are connected to the 68Vref are on the same circuit breaker. However, some off the modules do have I/O connected to other breakers (but off the same main 74V supply). Your questions got me thinking a bit, and I asked the customer if they had any third party equipment installed. It turns out they do. I have asked for a schematic of where that third party equipment is connected.In the meantime I have done some testing on the circuit. I connected a low ohm resistor (about 10) ohms across the 74V supply along with the 68 volt reference module, and one other module which is the main load for the 68V rail. I then monitored the 68V ref and 74V power rail on a scope. Using the math function I subtracted the two to see what the difference is during turn off.Initially I saw very little difference. I then added about 200uF of capacitance onto the 68V ref line and looked at the difference at turn off again. This time I did see a peak of about 15 volts difference that tailed off to zero in about 50ms.I repeatedly cycled power on the module under these conditions to see if I could cause a failure. It did not fail.I then tried connecting an external power supply to the 68V reference via a diode. I adjusted the external supply to 5 volts and then turned off the 74 volt supply for 30 seconds. I then restored the 74 volts supply, and checked if the 68 volt reference still worked. I repeated this at 2 volt increments until failure, which occurred at 25 volts on the external supply.Upon analysis, U4 had failed in the same way as seen in the field. This is good evidence that the something external is likely holding up the 68 V ref for a period of time after the 74 V is shut off. Or that extended exposure to this condition eventually causes the failure.I have applied a couple of diodes to the circuit, and will retest to see if the diodes solve the issue. One diode is in the collector of Q1. The other is from the 68V ref (anode) to the 74V supply. These diodes should prevent the damage caused if the 68V ref is ever above the 74V supply.I am a little unsure of the need for the second diode from 68V to the 74V supply, but it seems like some extra protection. The only worry is that if whatever is holding the 68V ref above the 74V is low impedance, it could supply enough current to destroy the diode, since it would in effect be powering all the modules in the system.Thank you again for your very insightful comments on this circuit. I really appreciate it.

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 21, 2021, 10:40:27 pm
content: Quote from: rtm on January 21, 2021, 05:32:01 amEdit: as you suggest, you may not need the second diode. What do your tests say?The possible surge current you highlight is a concern. As an alternative, can you connect the second diode from 68 V to cathode of the first?

author: rtm
author_rank: Contributor
timestamp: January 21, 2021, 11:45:07 pm
content: Quote from: thermistor-guy on January 21, 2021, 10:40:27 pmTesting has shown that the diode in the collector of Q1 is sufficient to prevent the failure from occurring.As before I connected a 10 ohm resistor in parallel with the module 74V supply inputs. I then connected the 68V reference to external power supply and ran it all the way up to 80 volts (the limit of the supply I had on hand) while cycling the 74V power to the module. I monitored the voltage across R18 and noted it was zero when the 74V supply was disconnected.I also injected some negative transients, but since I never was able to induce a failure without the diode, I did not expect to see anything different. But from an analysis standpoint, the added diode will protect against reverse current conduction in the collector of Q1 regardless of what causes the 74V power rail to dip below the 68V reference.When testing with the second diode from 68V ref to 74V rail I noted that the external supply was indeed powering the module through this diode when it got a few volts above the low voltage shutdown of the module's power supply. This confirms that if there was a hard fault to 74V on the 68V ref line, the entire set of modules would be supplied current through that second diode. It wouldn't last very long under in that situation.There is no "nice" place to connect the second diode to the collector of Q1 (i.e cathode of first diode). Given what I have seen in testing with just the one diode, I will be omitting the second.We will be sending a couple of modules with the modification to the customer where the rash of failures originated. I will update on the results of the field testing in case anyone is interested.Thank you again for all your input. I learned a valuable lesson - do not ignore the base-collector diode path on BJT transistors!

author: thermistor-guy
author_rank: Frequent Contributor
timestamp: January 22, 2021, 12:29:52 am
content: Quote from: rtm on January 21, 2021, 11:45:07 pmA one-diode modification makes sense.Please post an update on field testing, and any info on what has changed in the field to trigger these failures.Your thread is an good example of engineering in practice, where the real world changes and throws surprises at your initial solution.