A minute of electricity black-out that made into news headlines ... - Page 1
Forum section: electronics / general

author: BravoV
author_rank: Super Contributor
timestamp: January 01, 2020, 01:57:47 pm
content: ->Samsung Electronics chip output at South Korea plant partly halted due to short blackoutExcluding labor cost, curious how much the raw materials cost that will be disposed.Lots of cursing for sure.

author: Black Phoenix
author_rank: Super Contributor
timestamp: January 01, 2020, 02:11:38 pm
content: And again the prices of memories will go up...2018 again???Big coincidence isn't it (Corsair is known to use Samsung in most of his products).

author: NiHaoMike
author_rank: Super Contributor
timestamp: January 01, 2020, 02:13:31 pm
content: I wonder how many accountants rejected the plan to include UPSes as "too expensive"?

author: Cyberdragon
author_rank: Super Contributor
timestamp: January 01, 2020, 05:23:39 pm
content: Quote from: NiHaoMike on January 01, 2020, 02:13:31 pmThere are industrial UPS systems that go up into the megawatts range. A minute is well within the range of operation of the biggest systems. I can't believe Samsung, a tech giant, in Japan, the tech center of the world, didn't think of such things.Let's hope they learned their lesson.

author: coppice
author_rank: Super Contributor
timestamp: January 01, 2020, 05:29:23 pm
content: Quote from: NiHaoMike on January 01, 2020, 02:13:31 pmIf the public supply is highly reliable, and you have redundant wiring into the premises to reduce the risk from backhoe loaders, its questionable whether a UPS improves or degrades power reliability. No UPS is without its own problems.

author: Cyberdragon
author_rank: Super Contributor
timestamp: January 01, 2020, 05:36:56 pm
content: Quote from: coppice on January 01, 2020, 05:29:23 pmBut they don't necessarily have to go with the fancy flashy ones that have all the power conditioning and whatnot, just a battery backup. Companies of large systems usually have maintenace and checkup programs as well.

author: coppice
author_rank: Super Contributor
timestamp: January 01, 2020, 05:48:04 pm
content: Quote from: Cyberdragon on January 01, 2020, 05:36:56 pmHow do you build battery backup that cannot ever fail?

author: Gyro
author_rank: Super Contributor
timestamp: January 01, 2020, 06:17:33 pm
content: Quote from: Cyberdragon on January 01, 2020, 05:23:39 pmErrr, Samsung is South KoreaI suspect that it is far too easy to just sit back and criticise a huge smallest geometry fab facility like that for tripping out under power failure conditions without understanding the complexities (or the power consumption) involved.The lines will be chock full of fail-safes to protect the equipment of the line - rather than the product (which is disposable in this context). Think high vacuum systems, ultra-pure processing stages, hazardous chemicals, the risk of irreversible internal contamination or time consuming equipment cleanup risks etc. It doesn't take much to slash the yield of a fab. It must be quite an achievement to keep parts of a system like that from tripping out every time a mouse farts (not that a mouse is going to make it anywhere close to anything critical).The story does say "inspecting the production lines for a restart" which means, I suspect, that the protection systems actuallyworkedrather than failed.P.S. I wonder how you tell, at the start of a 1 minute power outage, that it is going to be just 1 minute long?

author: BravoV
author_rank: Super Contributor
timestamp: January 01, 2020, 06:30:50 pm
content: What interesting is in the report ..."A half-hour power outage at Samsung’s Pyeongtaek chip plant in 2018 resulted in estimated losses of about 50 billion won ($43.32 million) according to Yonhap."They've experienced this, and yet, it happened again, I guess the protection systems needed is much-much more expensive investment, compared to just throwing out <$50 million of dollars ?

author: Gyro
author_rank: Super Contributor
timestamp: January 01, 2020, 06:45:23 pm
content: Yes, I suspect that <$50M doesn't buy you too much additional protection in that context. You still have to plan on the line going down unless youknowexactly when power will be restored. It's probably already full of stuff to make the shutdown as graceful and least damaging as possible. That amount of money implies that they didn't actually break anything significant.

author: james_s
author_rank: Super Contributor
timestamp: January 01, 2020, 07:50:42 pm
content: Quote from: coppice on January 01, 2020, 05:29:23 pmAt a place I used to work we had outages on two different occasions that were caused by the huge UPS failing. One of those incidents filled the whole building with smoke.I got rid of the UPS's I had at home after I came home one day and smelled something, then found one of mine with the plastic top panel melting. Stored energy is dangerous and between that and the quiescent draw added by a UPS I decided it wasn't worth it for how rarely my power goes out.

author: Gyro
author_rank: Super Contributor
timestamp: January 01, 2020, 08:50:31 pm
content: Back in the '70s (miners and power workers strikes, 4 day week) all the sites in the company I worked for had diesel generators and fuel stores installed. They were still in place in the early '80s when our site started having 'issues' with the mains supply (cable capacity too small and phase load balancing poor) and started popping supply fuses with annoying regularity.Each time the generator would fire up automatically just fine. The only weakness with the scheme was that it relied on a guy from security going round to open the generator room doors within a couple of minutes to provide cooling (there were small louvres but insufficient). Of course one day there was a new guy on duty. By the time anyone noticed, there were vast clouds of steam and coolant spewing everywhere!There isalwaysa weakness somewhere.

author: Cyberdragon
author_rank: Super Contributor
timestamp: January 01, 2020, 09:03:19 pm
content: QuoteYes, as you've indentified...humans.Soon everything will be automated with just a very small skeleton crew that sit around drinking coffee unless an alarm goes off. Also, redundancy is key, just look at how many backups a nuclear plant has.

author: Gyro
author_rank: Super Contributor
timestamp: January 01, 2020, 09:11:58 pm
content: Quote from: Cyberdragon on January 01, 2020, 09:03:19 pmOnly in that particular (and trivial) case!QuoteAnd yet, they still fail catastrophically (taking Fukushima as the most recent case).

author: coppice
author_rank: Super Contributor
timestamp: January 01, 2020, 09:14:21 pm
content: Quote from: Gyro on January 01, 2020, 09:09:07 pmThere's usually some human incompetence somewhere when uninterruptible power proves highly interruptible. Redundant wiring that proves to be far less than redundant is a common one. It no use having an uninterruptible power source when minor damage to cables can separate all the sources from the load. Backhoe. Backhoe. Its off the work we go..........

author: james_s
author_rank: Super Contributor
timestamp: January 01, 2020, 09:22:30 pm
content: Quote from: Gyro on January 01, 2020, 08:50:31 pmHow on earth did that not get noticed during the design phase? It's not that hard to calculate the heat output and determine the amount of airflow needed. Certainly they could have retrofitted fans powered by the generators to exhaust hot air from the room.

author: Gyro
author_rank: Super Contributor
timestamp: January 01, 2020, 09:31:55 pm
content: It was a typical CEO 'I'm not going to be held to ransom by striking miners - do it now!' no-brain reflex. They were never properly designed in, just shoehorned into wherever they would fit on each site. At least the exhausts vented outside! After a few years, people pretty muchignoredforgot about them, I doubt they even got serviced.

author: David Hess
author_rank: Super Contributor
timestamp: January 02, 2020, 12:20:34 am
content: Quote from: coppice on January 01, 2020, 05:48:04 pmI think the most common problem is with the transfer switch failing catastrophically.To be reliable, the UPS needs to be regularly tested and that by itself tend to cause failure unless the designers, builders, and installers were all unusually competent.

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 02, 2020, 03:07:24 am
content: It boggles my mind how lot of critical applications that arn't life critical never bother with generator/UPSes. Even something like a restaurant can be completely shut down by a 15 minute outage. Get backup power, it will pay for itself the first time it's ever needed.In the case of a manufacturing plant, a fly wheel system with stand by generator would probably be the most cost effective, though a UPS with batteries running as a dual conversion setup is the most fail safe. It's what the telcos do and why land lines tend to be so reliable.

author: coppice
author_rank: Super Contributor
timestamp: January 02, 2020, 03:25:11 am
content: Quote from: Red Squirrel on January 02, 2020, 03:07:24 amLots of people haven't seen a single power interruption in a couple of decades. In places like this only life or death systems justify having backup power systems.

author: james_s
author_rank: Super Contributor
timestamp: January 02, 2020, 03:40:25 am
content: Not to mention having a backup system is an ongoing expense. Generators and battery backups both need regular maintenance and for large systems this is not cheap. Then as has already been mentioned the transfer switch adds another potential point of failure. Power outages downtown in many larger cities are virtually unheard of.

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 02, 2020, 03:40:32 pm
content: If it's setup right and is of good quality it should not require much upkeep. In the case of a generator you should run it once in a while and test and exercise it, but other than fuel to run it for a few hours a month it's not that big a deal. A battery based system is even less maintenance, just need to add water to them once in a while. Something that is online is running all the time too so if there is a problem you can deal with it before it becomes a big one. There are no transfer switches either to worry about in such a system. Even in places with reliable power, they sometimes need to do maintenance on the lines, or a car can hit a hydro pole etc... things happen.I eventually want to convert my home system to online, right now it's standby, so there is still a chance of the transfer relay failing.

author: james_s
author_rank: Super Contributor
timestamp: January 02, 2020, 07:29:56 pm
content: Don't underestimate the cost of upkeep on a large scale backup system like you'd need for a commercial building. It costs tens of thousands to install in the first place, consumes a significant amount of valuable space for which you pay rent/taxes, in Seattle for example commercial rent is over $41/sq-ft/yr so if we conservatively say an 8'x10' room for the backup generator or UPS that's over $3k/yr just to rent the space. Then you've got to either have a maintenance guy who is qualified to work on it or hire somebody out on a regular basis. Generators need oil changes, cooling system service, and other scheduled maintenance, repairs and regular test runs. UPS systems need battery maintenance and checks, occasionally need new batteries which cost many thousands of dollars. You've also got the additional hazards of stored energy on the premises, either fuel or electricity stored in batteries. Unless you deal with frequent outages, have life critical equipment or valuable inventory that will be ruined by a brief outage, it's more economical to just shut down when power is lost.We had a large UPS at a former job and it was expensive to maintain and caused at least two outages that I recall. At some point it was deemed inadequate for the load it was supporting and they spent another $50K or so replacing it with a larger unit. I'm not sure why they had it at all, I only remember there being one or two power outages in the more than 10 years I worked there. A UPS only provides power for 10-20 minutes in most cases anyway so it's only there to let you shut down equipment properly, you don't just keep going and hope the power comes back in time.

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 02, 2020, 08:14:35 pm
content: Oh yeah if you're renting space I can see it being an issue. In that case there's way more red tape as to if you can even have such setup in first place since you don't own the building.

author: james_s
author_rank: Super Contributor
timestamp: January 02, 2020, 08:21:42 pm
content: The vast majority of businesses rent the space they are in, at least in larger cities in the USA. Commercial buildings are enormously expensive. Even if you own the space you've still paid for it and you pay taxes on it. Space that is consumed by equipment and infrastructure is space that you can't use for other purposes.

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 02, 2020, 10:41:46 pm
content: Actually I wonder if cities tax generator cans as a building... knowing how stingy they are, they probably do. My city does not even allow sea cans at all (typically what generators are in) but not sure how that works for businesses, maybe they get an exception. We have two 110kw units in a seacan here. A lot of our buildings don't though which boggles us, especially the remote fly in sites. It costs around 30k to fly in a generator every time one is needed. There's one site where we had to fly one in like 20 times in a very short time span due to unstable power, a stationary one would have paid for itself and years of maintenance.

author: Black Phoenix
author_rank: Super Contributor
timestamp: January 03, 2020, 12:56:16 am
content: Quote from: Black Phoenix on January 01, 2020, 02:11:38 pmAnd as I expected:https://www.pcgamesn.com/ssd-nand-price-rise-40-percent-2020QuoteCoincidence that the news come after the power failure in the Samsung Factory? There is no coincidences. I can speculate that the supposed power failure was not really a power failure but something on propose... Lose some money now, activate the insurance (this companies have insurance for this kind of events, mind it) and the win double the amount in the following months, on sales alone and in the hike of stock market prices for shares...Now you say: "Ohh but they will be found for price gouging and market manipulation! Yes they were also found guilty in earlier times, and paid a fine that was way lower than the revenue earned during the years of practice under investigation before the fine:https://www.notebookcheck.net/Chinese-government-accuses-Samsung-Micron-and-SK-Hynix-of-fixing-DRAM-prices.365946.0.htmlQuote

author: coppice
author_rank: Super Contributor
timestamp: January 03, 2020, 01:02:42 am
content: Quote from: Black Phoenix on January 03, 2020, 12:56:16 amOf course its not a coincidence. Brokers have people specifically tracking the manufacturer's processes, looking for anything that might cause a supply hiccup. They then soak up any stock they can find before anyone else grabs it, and adjust ongoing prices accordingly.

author: SkyMaster
author_rank: Frequent Contributor
timestamp: January 03, 2020, 03:14:05 am
content: Quote from: Red Squirrel on January 02, 2020, 03:40:32 pmThe typical Canadian home has a 200 Amp main breaker. 200 A x 240 V = 48 000 WattsDo you have a 48 000 Watts standby power system?

author: 25 CPS
author_rank: Frequent Contributor
timestamp: January 03, 2020, 04:22:04 am
content: Quote from: james_s on January 01, 2020, 07:50:42 pmQuote from: David Hess on January 02, 2020, 12:20:34 amThis has been my experience too. In one of my work's locations, the building UPS itself would let the occasional nasty rip on the output side right into the technical equipment every so often and a few pieces of gear would get fried inevitably. It was worse than having the plant running straight off commercial power. There was one spectacular incident where it blew up more equipment than there were spares for on site and I ended up taking spares from my normal work location home on the commuter train and then jumped in the car at the crack of dawn the next morning and motored down the highway to repair the damage and get the place working again.More recently, closer to home, it was automatic transfer switch testing that went bad. The transfer switch threw fine but the UPS didn't shoulder the load while the generators spun up and the effect was as if a giant power bar feeding most of the place's equipment suddenly got yanked out of the wall. It was close to three days before the last of the mess that caused was all cleaned up.Quote from: SkyMaster on January 03, 2020, 03:14:05 amAre 200 amp services the standard now? I know insurance companies charge a hefty premium on old houses that still have 60 amp electric services and my neighbours ate some expensive premiums until they got an electrician in to install a new 100 amp breaker panel and service to the hydro pole. Most average houses I've seen have all been 100 amp services with 200 amps in big houses or where the homeowner's gone over the top. My house is a small 1000 square foot wartime bungalow and the previous guy who lived here had a 200 amp service installed which is total overkill for such a small house. On the other hand, my workshop isn't starved for juice to spin up equipment with!

author: james_s
author_rank: Super Contributor
timestamp: January 03, 2020, 04:39:59 am
content: I don't know about Canada but in the US a 200A service has been standard since at least 1979 when my house was built. It's a typical suburban split level, the sort that were at one time considered starter homes and sold with an unfinished basement. I've seen a few large (~3500sqft) houses with 400A and at least one ridiculously large house that had a 600A service but the vast majority of what I encounter from the late 70s on has been 200A.It occurred to me that things may be skewed somewhat in this region since the abundance of cheap hydro power has historically meant a lot of houses were fully electric without natural gas service and had resistance heat and cooking appliances. Gas is common now but the widespread use of electric heat up into the mid 80s may have driven adoption of higher capacity electrical service earlier than in other parts of the country.

author: Black Phoenix
author_rank: Super Contributor
timestamp: January 03, 2020, 09:15:35 am
content: Quote from: coppice on January 03, 2020, 01:02:42 amI'm not talking about brokers... I'm talking about Samsung, the company itself... Call me paranoic, but looking at the past practices and knowing the power and market share than Samsung, Micron and SK Hynix have in the "memory world", that it wasn't a fault of power that just happened. What really says to me than the power failure wasn't intentional?All than happen after that, is just the normal free market and speculation working.

author: coppice
author_rank: Super Contributor
timestamp: January 03, 2020, 12:17:18 pm
content: Quote from: Black Phoenix on January 03, 2020, 09:15:35 amSure. If a company experiences slack demand, or they just can't coax prices above break even, they may do many strange and devious things to avoid a clearly defined production cutback.

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 03, 2020, 02:02:27 pm
content: Quote from: SkyMaster on January 03, 2020, 03:14:05 amThat's just the max rating you can draw from the hydro drop, does not mean you're actually using that much power. In my case it's just my server stuff that's on UPS so I only have 750 watts of backup for around 4 hours. I eventually do want to upgrade that though.200 amp service was popular for a while when it was affordable to heat with hydro but now that everyone heats with gas most people get away with 100 amp service. I think for new builds they still put 200 though, as there's not really that much extra initial cost.

author: 25 CPS
author_rank: Frequent Contributor
timestamp: January 03, 2020, 07:52:24 pm
content: Quote from: Red Squirrel on January 03, 2020, 02:02:27 pmNow that you mention it, I do remember the home inspector mentioning that 200A was a 1980s thing. One of my friends who’s an electrician told me once that he only installs 200A services in new builds because the difference in cost between running a 100A service and a 200A one is pretty small but this way, the extra capacity is provided for up front without having to redo the service and panel. This was before battery electric cars were a thing to so these houses are covered capacity wise in case anybody wants to install a high rate car charger. For that matter, so it my bungalow.Electric heat. No thank you. Definitely not in Ontario, not with Dalton McGuinth/Kathleen Wynne hydro rates. Absolutely not. Maybe if I was living in Quebec with what Hydro-Quebec rates are, but definitely not here. I converted this place to natural gas for heat and hot water heating about a year and a half after I moved in from oil heat and an electric water heater. Electric heat was not on the table at all and junking the electric water heater was second from top of the list after replacing the furnace because all the electonics technologists at my workplace do shift work. A week of showers at the wrong time of day during the wrong season causing that 3,000 Watt heating element to come on and stay on to recover the water tank during on peak billing time with the highest of the high time of use rates was spiking the bill badly. Every time I read an Ars Technica article talking about how the state of California is thinking about time of use billing, I want to sit. Up so I can write comments saying, “No! Don’t. Don’t do it! Bad idea. As someone in Ontario stuck with it, don’t do it. You’ll be paying through the nose if you work shifts. You’ll be paying through the nose if you have kids and have to cook at a reasonable hour! You’ll be paying through the nose if you can’t strategically plan when you need to do your laundry!”

author: Red Squirrel
author_rank: Super Contributor
timestamp: January 03, 2020, 10:02:26 pm
content: Yeah can't imagine heating with hydro, which is too bad because it's green, but I literally would not be able to afford it, it would be like over a grand per month!

author: David Hess
author_rank: Super Contributor
timestamp: January 03, 2020, 10:41:17 pm
content: Where I lived in Orange County, California, about half of the houses were all electric with heat pumps for heating. At the time they were built, electricity "too cheap to meter" was suppose to become available "real soon now" and nuclear power was the future. They just recently shut down San Onofre nuclear power station which supplied that area and high prices and rolling blackouts have been accepted for 20 years now.

author: james_s
author_rank: Super Contributor
timestamp: January 04, 2020, 12:15:43 am
content: I currently pay a bit over 8c/kWh here which would make heating my house with electric resistance prohibitively expensive, but not THAT expensive. It was quite a bit cheaper in the 80s because it was subsidized somehow, something relating to the dams. IIRC the areas around the Grande Coulee hydro dam have power that costs about 2c/kWh even today.

author: AG6QR
author_rank: Frequent Contributor
timestamp: January 04, 2020, 12:20:02 am
content: Quote from: David Hess on January 02, 2020, 12:20:34 amI'm reminded of the (probably made-up, but told to me as true) story of the facility with a standby diesel generator that was regularly tested, and always worked perfectly until the utility power went out for real. The generator wouldn't start when needed. It turns out that the generator used an electric start motor, and the starter motor required power from the mains.On a separate note, I live in Northern California, where the local power company has started a policy of deliberately shutting down the power when we have high fire danger, to lessen the risk that one of the utility lines will spark a wild fire. This past October, my home county had a complete utility power shutdown for 45 hours. During the outage, I was listening to the fire dispatch radio channel. The outage started after sunset, and the following day around midday, after the power had been out for around 18 hours, I heard firefighters dealing with a call to a local elder-care nursing home. It seems the building had a standby generator that somehow overheated and caused a fire in the building. They had to evacuate residents. The fire was inside the walls of the structure, and they had to chop into the walls to extinguish the fire. Since I was only listening to the chatter over the radio, I never heard about the results of the investigation, if any. I'd guess that the generator lacked adequate ventilation and/or cooling system, but it seems to have lasted the better part of a full day before catching fire. It probably would have passed a test run of a reasonably short duration.It's often difficult to mitigate against rare problems.

author: SkyMaster
author_rank: Frequent Contributor
timestamp: January 04, 2020, 02:10:36 am
content: Quote from: james_s on January 04, 2020, 12:15:43 amIn the province of Quebec, the vast majority of the electricity is produced by hydropower. The rate is 6.08 cents/kWh.It is 25% less expensive than from where you are. We shall not convert the currency value, because well, we are paying our own electricity bill with our home currency.Here, heating with electricity is very popular. And it gets freaking cold in the winter.

author: SkyMaster
author_rank: Frequent Contributor
timestamp: January 04, 2020, 02:18:17 am
content: Quote from: Red Squirrel on January 03, 2020, 02:02:27 pmOh sorry Red Squirrel, I thought you had a whole home backup system. It is expensive, and then you need to maintain the system continuously; but there are people you want to be able to act like if there no power outage, when there is a power outage.The rest of us get by with a 4000 Watts generator.

author: james_s
author_rank: Super Contributor
timestamp: January 04, 2020, 06:46:14 am
content: I find a 2kW Honda inverter generator to be adequate. It will run my refrigerator, laptops, TV, internet router/modem/gear, gas stove and oven igniter, gas furnace blower and the lights which are all LED. The only thing I can't do is run the clothes dryer or run a big load like the vacuum cleaner while the dishwasher is running. With just a bit of management it's adequate while being extremely quiet and fuel efficient.

author: blacksheeplogic
author_rank: Frequent Contributor
timestamp: January 04, 2020, 07:16:49 am
content: I worked on a project where we took a fab line down. We were doing an out of cycle required update, fully tested the full upgrade procedure multiple times in test, fully documented with step by step instructions on how to perform the upgrade to the cluster while keeping the line up, but a mistake in not executing one of the steps caused the cluster to go down.It cost several million dollars. Even with the best plans and foresight sometimes things go wrong. You deal with the situation and move on.

author: NiHaoMike
author_rank: Super Contributor
timestamp: January 05, 2020, 02:20:31 am
content: Quote from: AG6QR on January 04, 2020, 12:20:02 amI suspect the generator was either meant to be used with a UPS system or someone put a big charger on the battery which masked the battery being weak/dead. ("They want $400 for that little charger? Let's replace it with this $300 Snap-on, the auto shop next door has been using them for over 20 years and they always work.")Quote from: blacksheeplogic on January 04, 2020, 07:16:49 amHave they looked into using an aerospace-like checklist system? It works really well to keep the team in sync during complicated procedures that have to be done right.