Protection of equipment against rolling blackout "load shedding" - Page 1
Forum section: electronics / projects

author: Cicada
author_rank: Regular Contributor
timestamp: July 14, 2022, 03:28:52 pm
content: Hi,My company have bought commercial equipment from a big international manufacturer. We have had a lot of problems with this piece of equipment in the past caused by damage sustained due to poor power quality and overall poor design of the equipment. I must say I am shocked by the quality of design from such a big international company. But enough about that.Just some quick background before I explain the problem. This commercial equipment is used at a facility in South Africa (SA). In SA we have been experiencing terrible power quality problems due to what is called "load shedding" a.k.a. rolling blackouts. Load shedding means that power to some areas are switched off for periods of time because demand outstrip supply. We have lost lots of expensive equipment due to load shedding. There is a clear correlation between equipment damage and abruptly switching the power grid on and off.Fortunately load shedding is done on a schedule. This allows us to switch of equipment when not in use to avoid damage.Look at the attached block diagram. It shows the power routing inside the machine. The equipment represented with yellow blocks are not present in the commercial unit. I want to discuss possible inclusion of these yellow blocks in the machine to resolve the problems that we experience.Problem description:1. The equipment is switched off (powered down) by the power switch shown at (4). This shuts down the computer (5) and associated displays but not the power to the internal circuits shown at (11).2. These circuits at (11) contain many PCBs with microcontrollers used to control positioning systems. The machine allows for position calibrations to be done before the machine is used. The calibration data is stored on these microcontroller and are not communicated back to the control computer (5) In my opinion this is a flaw. The calibration data is lost if the 24Vdc to the circuits at (11) falls away because the equipment is powered down using the isolator switch (3). The manufacturer admits that this is the case. In their defence they argue that the machine should not be switched of at the isolater (3) but at the power switch (4).3. We clearly need to disconnect the machine at the isolator (3) to prevent damage to the electronics at (11) caused by poor power quality during load shedding. And yes load shedding has damaged this machine and the circuits at (11). Many boards had to be replaced already.4. So the manufacturer have suggested that we modify the internals of the machine to include a UPS as shown at position (6) They have indicated that they do operate some of their machine in the same way (UPS at position (6) ) for reasons other than load shedding.5. In our case we would like to avoid the UPS solution for various reasons. We are considering a protection circuit at position (9/10). Either a isolated DC to DC converter (9) or some protection circuit (10) to protect down stream equipment from any surges, spikes, swells, dips that could couple through the 24V AC/DC PSU at (7)6. It is this piece of protection equipment at (10) or the DC/DC converter at (9) that I am wondering about. What would such a protection circuit (a really good solid one) look like? Are there commercial solutions? Or would a good quality DC/DC with good isolation work better.I would like to hear a good solution to this.PS.I have very little faith in UPS's to protect against surges, spike and dips due to poor power quality. Purely because I have seen how supposed good quality UPS's do not provide protection.

author: ajb
author_rank: Super Contributor
timestamp: July 14, 2022, 04:44:34 pm
content: You can't really design a protection system without knowing what you're protecting against. How are the 'sensitive circuits' getting damaged by mains power quality issues on the other side of a 24V power supply? Are there transients appearing on the 24V side, or is there a series of rapid restarts that the sensitive stuff can't handle? How sure are you that the damage is being caused through the 24V supply and not some other connections between the sensitive circuits and other equipment?A UPS would probably be the simplest solution to keeping the calibration data alive, if you are relying on it to provide protection then you probably want a'line interactive' or 'online'double conversion UPS. These always run the load from the inverter and only use mains to charge the battery/power the DC bus, so there is never a direct connection between the input and output mains. You could also implement a UPS solution at the 24VDC rail, which would be more efficient. Depending on the amount of power you need there are off-the-shelf enclosed power supplies that have battery backup support built-in that might work, but that's probably more work in terms of modifying the machine.The problem with using an isolated DC-DC there is...what exactly are you isolating from what? If there are connections to other circuitry that aren't also isolated, then you could end up with even worse issues than continuing with the existing setup (which I would guess has the DC bus referenced to the equipment ground?).Another factor is that if the manufacturer has recommended a UPS, then going with that solution is probably going to make for an easier time if you need them to support the machine later on. Sounds like you're not impressed with the manufacturer so far, so whether or not that means anything to you IDK, but it's definitely an important consideration for big expensive industrial equipment.

author: David Hess
author_rank: Super Contributor
timestamp: July 14, 2022, 06:53:16 pm
content: Quote from: Cicada on July 14, 2022, 03:28:52 pmThe DC to DC converter solution comes down to a 24 volt output DC-to-DC converter which can handle a range of input voltages below and above 24 volts, and has a long or very long holdup time in case the input voltage fails for an extended time. An isolated converter might help, but isolation should not be necessary unless there is some flaw in the design. Maybe something like this:https://www.mouser.com/ProductDetail/MEAN-WELL/RSD-100B-24?qs=erfQA2AIGbXIthGbJSYrgw%3D%3DAlternatively the 24 volt power supply at (7) could be improved or replaced to provide better performance. Power supplies are available which accept a wider range of line input voltage for difficult applications. Something I might try if the existing power supply has a universal input, is to step the input up or down by 60 volts with a buck/boost transformer so any variation in the line voltage is not in the middle of its input range. Doing this is likely too clever by half so I do not recommend it.And there are battery backed up 24 volt UPSes like this, probably intended for exactly your application:https://www.newark.com/block/pvua-24-24-10/ups-24vdc-240w/dp/53X8854QuoteAn online or double conversion UPS cleans up poor power completely, since it is essentially a battery charger, battery, and inverter that runs continuously.Quote from: ajb on July 14, 2022, 04:44:34 pmMost line interactive UPSes include a buck/boost transformer with multiple taps, so they can step the line voltage up or down to maintain the correct output during extended periods of low and high voltage without using the inverter and drawing on the battery. They are not any better than offline UPSes at suppressing line disturbances.There are power conditioners which use a buck/boost transformer driven by an inverter to provide a high power regulated output, but finding them can be difficult. I think they have mostly been replaced by online UPSes.

author: mag_therm
author_rank: Frequent Contributor
timestamp: July 14, 2022, 07:21:02 pm
content: Yes, Double conversion UPS with DC link permanently on floating battery is best reliability but not so efficient.On the "High Powered AC Circuits" side I recall one case in a sudden unexpected utility outage where a DC drive on a big motor and an inverter both had device failure.Cause was never determined but it was suspected one rectifier failure put a short across 2 phases causing the other unit to go down.

author: ajb
author_rank: Super Contributor
timestamp: July 14, 2022, 08:32:50 pm
content: Ah yes, 'double conversion' is the UPS term I meant, thanks!

author: PaulAm
author_rank: Frequent Contributor
timestamp: July 15, 2022, 01:10:54 am
content: At one point a number of years ago, my wife worked for an auto supplier that was putting up small factories in Central America. Power was always an issue and their solution was to put in large UPSes, backed up by generators, to manage all of the critical infrastructure. We are not talking 5 kva units here. Expensive, but it worked.If you can't rely on power when you have to rely on it, you need to do whatever is necessary. Trying to cheap your way through it will probably not end well.

author: CaptDon
author_rank: Super Contributor
timestamp: July 15, 2022, 11:37:29 am
content: We learned early on with the Xilinx ZYNC and other devices they sell that power MUST be sequenced ON and OFF properly!!!! We had to go to extensive measures to have enough reserve energy to hold up the main supplies while each of the different voltage power rails were sequenced up or down. Our boards ended up having SuperCaps and elaborate charging circuits and delayed turn-on until the SuperCaps had enough stored energy to be able to perform a proper sequenced startup AND shutdown. The circuit boards were 14 layer with around 3000 total components with multiple chips costing over $200 each. The ZYNC and ARTIX-7 stuff tended to die early if not properly sequenced. Perhaps your equipment manufacturer has overlooked these failure modes in their design??

author: PeteH
author_rank: Unknown
timestamp: July 15, 2022, 12:37:28 pm
content: Quote from: CaptDon on July 15, 2022, 11:37:29 amZynq-7000 devices? - I haven't seen anything like this and have not used elaborate methods as you describe. The only real reliability issue is described in the Xilinx documentation (relationship between VCCAUX, 1.8V and 3.3V)

author: CaptDon
author_rank: Super Contributor
timestamp: July 15, 2022, 01:19:31 pm
content: We used the power rail sequencers from T.I. which were programmable in both on delays and off delays.

author: David Hess
author_rank: Super Contributor
timestamp: July 16, 2022, 07:12:43 pm
content: They used to do power sequencing with discrete designs, before ASICs were available for it. A common method was to have one "reference" supply, and make all of the other supplies track it so no violations in supply voltage could occur.

author: CaptDon
author_rank: Super Contributor
timestamp: July 16, 2022, 11:01:02 pm
content: Kind of reminds me of how Tektronix and others used one adjustable reference supply and then locked all the other supplies to it thereby guaranteeing if the reference went ape-shit "She'd blow up real good" damaging stuff on most of the other rails also!!! Just give me a bunch of LM317's running independent or a handful of 7805/7905/7812/7912 whatever.

author: David Hess
author_rank: Super Contributor
timestamp: July 16, 2022, 11:32:30 pm
content: Quote from: CaptDon on July 16, 2022, 11:01:02 pmThe reason Tektronix did that was to make sure none of the supplies reversed compared to the others, which applies just as much to modern ICs with multiple supply voltages. During power up if the high voltage supplies were lower than the low voltage supplies, things would be damaged anyway.Tektronix also used a "balance" input to the switching controller so if *any* of the outputs got pulled out of regulation, then the power supply would shut down.

author: Alti
author_rank: Frequent Contributor
timestamp: July 17, 2022, 06:42:02 pm
content: Quote from: Cicada on July 14, 2022, 03:28:52 pmI do not understand how exactly a load shedding can damage an equipment. Every appliance must be able to start and stop gracefully, that is not an option. Are you saying that if the machine is disconnected in some particular state, this leads to permanent damage?Or are you talking about a situation where the power quality goes beyond the standard that applies in your location? Like, frequency that is supposed to be 50Hz, becomes DC, voltage that is supposed to be 207-253Vrms reaches 400V for a minute, etc? Can you attach scope plot of those killing blows?

author: SeanB
author_rank: Super Contributor
timestamp: July 17, 2022, 07:21:40 pm
content: Problem is that, to implement the local load shedding, AKA rolling blackouts, the easiest method to implement it is to use the SCADA systems at the larger substations, and to use the 33kV and 66kV disconnectors on the SCADA to drop blocks of smaller substations, and load, as a unit. Then, on reconnect, they turn on the breaker again, leading to a massive start up surge, and then a large set of transient voltages propagate down the cascade of lower voltage transformers and switchgear. This results in kilovolt transients on your local 230V nominal/400V nominal single or 3 phase load.At a minimum you will need one of these per phase to neutral.https://www.mantech.co.za/ProductInfo.aspx?Item=14M3553This will at least provide some robust clamping of the transients, though a better thing to do is to take the output of your isolator (3) and use a 3 phase contactor, of appropriate current rating, and with a 230VAC operation coil, assuming you are using a neutral, or 400VAC if not. then to control the contactor you need to add in an on delay relay, so that your power is applied only 30 seconds after mains is applied, so that the major transients have died down, and that the local substation can finish blowing it's fuses if it is going to do so on power on. Link in series with the contactor coil a 3 phase phase loss relay, so that loss of a single phase will not cause motor malfunction in your process.Rhomberg R127 will give you the 3 phase delay timer, and R108 the voltage window monitor, so you can detect phase loss on power return. Yes a minute delay, but that will at least mean your equipment does not get all the transients applied to it, just make sure that you put those large MOV units on input and output of the contactor in star to neutral. Probably a fourth one from neutral to ground as well, though as you likely have your own on site transformer from 11kV this will not serve much, unless neutral is stolen during load shedding for the copper.With the time switches and voltage monitors they are plug in units, so you can keep a spare of each, in case they get cooked during power application, though the plug in units normally have small2VA iron core transformers in them, not switch mode supplies, so are very robust against surges, and being industrial lego they fit in a small DIN cabinet easily, so can easily be retrofitted in the power line. Not going to do much against your power going, but keeps that massive turn on surge manageable, and also the delay means if your local feedline is going to decide to blow up a transformer or switchgear, it will at least keep that nasty transient out.