The BIG EEVblog Server Fire - Page 1
Forum section: electronics / general

author: gnif
author_rank: Unknown
timestamp: April 08, 2021, 03:26:37 am
content: After many days of stress, hair loss, and sleepless nights, we're back baby!Please note that there may still be some disruptions over the next few days as we are still running in a degraded state.

author: xrunner
author_rank: Super Contributor
timestamp: April 08, 2021, 03:33:27 am
content: Thank you.

author: CatalinaWOW
author_rank: Super Contributor
timestamp: April 08, 2021, 03:40:58 am
content: I am sure all of us would like an after action report when you finally get back to some semblance of normalcy.I am really curious about how use of backup generators caused a fire. Everything after that is just the dominoes falling, with an extra helping of bad luck for the EEVBlog servers.

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 03:41:53 am
content: I figured this event needed it's own thread, so moved it from the servere reports thread.HUGE thanks to gnif for handling this:https://hostfission.com/The server was down from 2021-04-04 21:13 UTC to 2021-04-08 03:36 UTCIt's currently still operating in a degraded state, and performance is surrently impacted until the caches catch up.Gorillaservers upgraded the server box (maybe the old box was water damaged?) from Dual Xeon 2620V2 from the older dual L5630Presumably they'll upgrade the other redundant box too to match, but the 2nd box is not currently online yet.The lesson here is, whilst it's great to have a fully redundant automatic backup server, it was kinda silly to have it in the same datacenter!We are going to ask Gorillaservers is they can provision one of the boxes in their LA data center, so if a whole city/state goes out the server will still operate.I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.

author: edpalmer42
author_rank: Super Contributor
timestamp: April 08, 2021, 03:44:45 am
content: The standby power system caused a fire that shut down the data center with some serversexpectedto be offline forseveral weeks!!https://www.gorillaservers.com/outage.htmlYou just can't make this stuff up!!

author: gnif
author_rank: Unknown
timestamp: April 08, 2021, 03:45:45 am
content: Quote from: CatalinaWOW on April 08, 2021, 03:40:58 amThis would have to be provided by GorillaServers first.We are not fully aware of all the details yet either as GS have given priority to restoring servers.

author: Tomorokoshi
author_rank: Super Contributor
timestamp: April 08, 2021, 03:46:42 am
content: I expected there was progress when the "502 Gateway Not Found" message came up!

author: gnif
author_rank: Unknown
timestamp: April 08, 2021, 03:49:40 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amScared the crap out me when my post went missing, I thought we had a major DB issue, lol

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 03:50:05 am
content: For those who weren't following along on Twitter, it was eventually confirmed that 2 of the three EEVblog boxes (the ones that handle the website and forum databases etc) were in the "splash zone".So they took longer to get back up and running than my email/management server box which was in another part of the datacenter and another subnet (it's a different type of box, single xeon instead of dual xeon).Given that they set us up on a new box (and presumably just pulling the old drives), it's likely the old boxes were either water damaged, or it was simply easier to give us a new box until such time as the old boxes can be evaluated properly.

author: Algoma
author_rank: Frequent Contributor
timestamp: April 08, 2021, 03:51:18 am
content: Good to see things back online, a bit at a time. I've certainly been there at the center of the NOC when things go sideways.

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 03:51:42 am
content: Quote from: gnif on April 08, 2021, 03:49:40 amSorry!

author: hamster_nz
author_rank: Super Contributor
timestamp: April 08, 2021, 03:54:10 am
content: Good to see things back to normal.I'm feeling for the DC guys... there but for the grace of god go I.

author: tautech
author_rank: Super Contributor
timestamp: April 08, 2021, 04:21:10 am
content: Will be interesting to see if they honor their pledge of 1 day free hosting for every 15 minutes down.96 free days for every 24 hours should give Dave a year or more free server hosting.gnif, you poor bugger go and get some sleep !

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 08, 2021, 04:21:51 am
content: Quote from: EEVblog on April 08, 2021, 03:50:05 amAre you going to make sure the "backup" will now be in another room or at least several racks away? (I think it was mentioned somewhere that there's enough bandwidth required for real time syncing that putting it in a separate building is not feasible.)

author: Whales
author_rank: Super Contributor
timestamp: April 08, 2021, 04:41:01 am
content: Quote from: tautech on April 08, 2021, 04:21:10 amI think I recall seeing a 30day cap.Super glad to see things back up. Sick today, needed a happy reading escape

author: tautech
author_rank: Super Contributor
timestamp: April 08, 2021, 04:45:14 am
content: Quote from: Whales on April 08, 2021, 04:41:01 amYou're right. I didn't go looking at the T&C's.https://webnx.com/sla/Credit shall not exceed 100% of billing in a thirty day cycle

author: graybeard
author_rank: Frequent Contributor
timestamp: April 08, 2021, 04:49:08 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amI have been relying on a single mail server for years. I have two servers running. One in a farm, and one on a fixed IP at home. I should configure the home server as a secondary mail server.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 08, 2021, 04:54:06 am
content: ˙uʍop ǝpᴉsdn plɹoʍ ǝloɥʍ ǝɯ pǝuɹn┴

author: tautech
author_rank: Super Contributor
timestamp: April 08, 2021, 04:56:41 am
content: For prosperity....including the bad spelling:As of approimately 2021-04-04 21:13:00 UTC there was a major outage at the datacenter in Odgen Utah where the EEVBlog servers are hosted. This outage was caused by a fire as a result of performing regular load testing of from the usage of the emergency generators as the result of a city power outage.Unfortuantly due to the nature of the failure it may be some time before the EEVBlog website and forums are restored to normal operation. Thankfully we maintain several off-site backups of the entire EEVBlog infrastructure, as a result there is no need to worry about any loss of data.At this time we simply have to sit and wait for more information from the datacenter, in the meantime you can follow Dave or HostFission on twitter for updates.If you cant wait you can keep yourself entertained by watching Dave's videos on Youtube or watch them over on Odysee.Alternatively you can join us on IRC on #EEVBlog over at irc.austnet.orgUpdates:2021-04-05 08:56:00 UTC - Current updates project that close to 90-95% of all hardware experienced zero damage. Electricians are working now to restore power to the Ogden Utah data center and are estimating power should return at approximately 3:00 pm MDT tomorrow.2021-04-05 13:18:00 UTC - Power restored to one of the three servers restoring Dave's email along with some of the EEVBlog management infrastructure.2021-04-06 03:45:43 UTC - Network restored to the management server2021-04-07 08:25:46 UTC - Informational update above, fire was not caused by genset test but rather genset use during a city wide power outage.2021-04-07 08:40:00 UTC - GorillaServers have confirmed that our servers "were located in the section with a high probability of water damage. So they will need to be physically inspected before they can be powered on."2021-04-07 18:21:21 UTC - A contact at GorillaServers has confirmed that it is almost certain that the two webservers will not be restored today in their current state and are looking at the possibility of issuing us a temporary server in the Los Angeles DC in the meantime2021-04-07 18:45:32 UTC - LA replacement hardware is not up to spec for our needs, GS has added our servers to a "quite extensive" priority list.2021-04-07 22:33:00 UTC - GorillaServers have moved the EEVBlog servers to the top of the priority list and are working on them now!

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 05:05:38 am
content: Quote from: EEVblog on April 08, 2021, 03:50:05 amI thought datacenters typically used Halon fire suppression systems? The last place I worked that had an onsite datacenter had one, there were warning strobes to indicate the system had discharged and asphyxiation warning signs.

author: bdunham7
author_rank: Super Contributor
timestamp: April 08, 2021, 05:05:52 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amThe thing is that if it is 20 feet away, a 10GBASE-T connection can maintain the sync with a very small initial investment and no recurring costs--and microseconds of latency. A 10Gb/s connection to another state would cost lotsa bucks and would still have 1000X or more latency.

author: bdunham7
author_rank: Super Contributor
timestamp: April 08, 2021, 05:15:34 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amI just noticed this--they're both pretty old tech, actually. Spinning drives too?

author: Halcyon
author_rank: Unknown
timestamp: April 08, 2021, 05:30:41 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amI'm actually surprised to learn that you weren't using Google Workspace or Office 365 Dave. For the sake of $8-9/month per user, you can have all of the Google services, redundancy, spam filtering and 30-something email aliases. I haven't run my own mail server for decades and it's a bit of a thing of the past.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 08, 2021, 05:33:10 am
content: Quote from: Halcyon on April 08, 2021, 05:30:41 amThat still seems like a lot of money.

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 05:36:41 am
content: Quote from: Halcyon on April 08, 2021, 05:30:41 amWe used the Google suite for a while at my job, absolutely hated it since it was all crippled browser based stuff, they don't even have a proper desktop email client. When we were acquired we went back to Microsoft, which while I'm not the biggest fan of Microsoft, their Outlook for email and calendar blows the doors off of Google's clunky offerings. Google Docs is nice for shared documents but the lack of a desktop version really kills it for most other uses.For email, yeah, I wouldn't bother hosting my own server, but for the client side, no way, I don't rent software, and I absolutely hate browser based productivity applications. They are a total pain in the ass and never offer the same functionality of a desktop application.

author: tautech
author_rank: Super Contributor
timestamp: April 08, 2021, 05:39:33 am
content: Better drop this here too:

author: hendorog
author_rank: Super Contributor
timestamp: April 08, 2021, 05:42:41 am
content: Quote from: james_s on April 08, 2021, 05:36:41 amPretty much 100% the opposite from me. Goes to show no one solution will please everyone.Been whoring myself to google full time now for 10 years. Became permanent after I found out that the cat was sneakily pissing on my home servers. That shortened the life of all concerned.

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 05:45:46 am
content: Quote from: Halcyon on April 08, 2021, 05:30:41 amI have Office 365, but only use it for Word/Excel etc. No idea it had other email server related stuff.

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 05:47:50 am
content: Quote from: hendorog on April 08, 2021, 05:42:41 amServer is one thing, but desktop productivity software? The browser based stuff is a joke, it doesn't even work at all if your internet goes down, and in my case I was constantly logged out every time I cleared my browser cache which is something I do often at work. Since the calendar was browser based it couldn't integrate into the OS and provide meeting reminders so I kept missing meetings, total pain in the ass. Not to mention Google is an advertising company, their products are almost all trojan horses in a sense, their function is to entice you into giving them your valuable personal information. I use them for email and nothing else, if I log into a browser I ALWAYS make sure I log out immediately when I'm done.

author: Halcyon
author_rank: Unknown
timestamp: April 08, 2021, 06:30:29 am
content: Quote from: Ed.Kloonk on April 08, 2021, 05:33:10 amI guess it depends on the individual but I use Google Workspace for my own private email on my own domain. It costs me AUD$8.32 per month and I have various email addresses for things, e.g,:payments@ - Links to my bank account (in Australia you can transfer money instantly between bank accounts using email addresses/phone numbers), also used for Paypalforums@ - Forum subscriptionsuber@ - Ubersonos@ - SonosEmail aliases are essentially "disposable" and I can create/delete them on a whim without having my private mailbox spammed because someone/a service has my personal email address.

author: TomS_
author_rank: Frequent Contributor
timestamp: April 08, 2021, 06:35:03 am
content: Quote from: james_s on April 08, 2021, 05:05:38 amI don't think anyone uses Halon any more, it would be something like FM200 these days. And yes, this would usually be installed in the server rooms.You would usually get something like 30 seconds from alarms and strobes activating to evacuate before it is dumped into the room because yes, it's whole purpose is to displace oxygen to starve a fire.But if the fire was caused by one of the generators, it's anyone's guess whether they installed the same fire suppression in that area. Presumably not if it got this bad!?But there seems to be a missing piece of the puzzle (or at least I've missed it)... They say the fire was caused by a generator. The generators would not normally be installed in the same room as servers. So how does a generator fire end up leading to water damage to servers?? Unless it burned through a wall??OVH also suffered a major fire recently. Interesting trend of events.

author: Ranayna
author_rank: Super Contributor
timestamp: April 08, 2021, 06:56:28 am
content: Yay, EEVBlog is backInterestingly the second datacenter fire that affected services that i use in only a couple of months. According to statistics the services i use should be save now for some time :pAnyway, always interesting to see what can happen. Makes me glad that my employer has definitly not skimped of the equipment and setup of our internal datacenter.Our internal datacenter is quite small, we only have 26 Racks. But it is set up very well: redundant cooling, redundant power (one leg UPS and generator protected). All the critical components are in separated rooms with at least 2 firewalls (the brick kind) separating them.The room with the racks and the UPS room are equipped with a Novec fire suppression system. The generator room, on the other hand, is not. It was explained that it was, first, not really needed, since the generator has its own room, the fuel has its own room, and the datacenter itself is two rooms away. And secondly, misfires of the required early detection system might trigger on the slightest leaks of the generator. And refilling the Novec would supposedly cost around 20.000 EuroThis kind of setup is expensive, but at least for us, it seems to be worth it: We had not a single outage beyond faults of individual servers. Yet...

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 07:18:29 am
content: Quote from: TomS_ on April 08, 2021, 06:35:03 amAh, yeah FM200 was what we had at that place, I just assumed it was a brand name for a Halon system, I never really looked into it.Yes I wondered about that too, I would have thought they'd have the generators in a separate structure, but who knows, maybe the datacenter is in a highrise or something. I actually spent a night in Ogden UT once but it was close to 15 years ago and I don't think I ever went into town.

author: peter-h
author_rank: Super Contributor
timestamp: April 08, 2021, 08:03:43 am
content: Well done on EEVBLOG for having kept good backups.Not many people do that.Also a lot of "hits" on forums are malicious, perhaps organised by people who got banned for behaving badly.

author: tszaboo
author_rank: Super Contributor
timestamp: April 08, 2021, 08:08:41 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amI was taught by the back-end people that your second server shall be on another continent, with a different government in charge of the company running it.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 08, 2021, 08:51:07 am
content: Quote from: tszaboo on April 08, 2021, 08:08:41 amIt might have sounded prudent some while ago but these days, with numerous jurisdictions in each other's pockets, I don't think it matters anymore.

author: peter-h
author_rank: Super Contributor
timestamp: April 08, 2021, 09:06:11 am
content: Also a "mirror" server won't protect you from a clever attack, where only older data is deleted, perhaps over time.One has to do several things at the same time, including a database snapshot every day or soand keep the snapshots for many months.Hacking is a big thing these days - much bigger than servers blowing up.

author: Towger
author_rank: Super Contributor
timestamp: April 08, 2021, 09:23:33 am
content: Quote from: TomS_ on April 08, 2021, 06:35:03 amFrom the sounds of it, I would not be surprised if the generators were on the roof. Normally I would have expected the generators to be in a separate building. But I have heard of similar issues with mission critical UPS systems failing. If you have a fire and call you call any local fire department, they will want all power turned off. They don't care about your business model etc, minimising risk to their firefighters is more important. The same goes if you call out the lifeboat, their job is to save life, saving the vessel is of secondary importance.

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 09:30:08 am
content: Quote from: james_s on April 08, 2021, 07:18:29 amit's here:https://www.google.com/maps/place/119+600+W+Bldg+3B,+Ogden,+UT+84404,+USA/@41.2631939,-111.994884,17z/data=!3m1!4b1!4m5!3m4!1s0x87530c31eeede3af:0xe0843e2258f1e68c!8m2!3d41.2631939!4d-111.9926953

author: EEVblog
author_rank: Unknown
timestamp: April 08, 2021, 09:31:39 am
content: Quote from: peter-h on April 08, 2021, 08:03:43 amDidn't need the backups in the end though.QuoteIt's happened.

author: Towger
author_rank: Super Contributor
timestamp: April 08, 2021, 09:33:07 am
content: Quote from: peter-h on April 08, 2021, 09:06:11 amRoyal Bank of Scotland learned that the hard way, it took years to properly sort out:https://en.wikipedia.org/wiki/2012_RBS_Group_computer_system_problemsIn their case it was incompetence. They could have pulled the plug on the main system before it was automatically replicated on the backup system, a couple of hours later.

author: MadScientist
author_rank: Frequent Contributor
timestamp: April 08, 2021, 09:49:25 am
content: I’d be really interested to know what happened and see a layout of the buildingGenset fires are not uncommon , ( IC engines etc) and typically are positioned well away from the devices they powered, airgapped by 100s of feet , walls , separate buildings etc. Gensets should be able to burn to the ground and not damage their associated systems. Not to mention comprehensive fire suppressionSomeone skimped here at gorilla methinks

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 08, 2021, 09:53:59 am
content: Quote from: MadScientist on April 08, 2021, 09:49:25 amMy money's on switchboxes. They really can and do indeed release magic black smoke.

author: Ian.M
author_rank: Super Contributor
timestamp: April 08, 2021, 09:59:10 am
content: EEVblog got an honorable mention in the article about the incident at 'The Register':https://www.theregister.com/2021/04/06/webnx_data_fire/The best unofficial live discussion thread about the incident is at:https://www.webhostingtalk.com/showthread.php?t=1842301It has a lot of stuff that WebNX and GorillaServers were either slow to, or did not, publicly release.

author: MadScientist
author_rank: Frequent Contributor
timestamp: April 08, 2021, 10:01:18 am
content: Quote from: Ed.Kloonk on April 08, 2021, 09:53:59 amHmm, normally the changeover systems are in the genset space not the supported devices space, the switchgear there would be in use daily using the grid supplies etc and wouldn’t or shouldn’t fail like it did. ( if it did ) , sometimes the changeover system burns out , but again this should be significantly airgapped

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 08, 2021, 10:06:35 am
content: Whatever the cause of the fire and resultant outage, congratulations to all concerned for getting EEVblog (and presumably many other systems) back up and running - even at mildly reduced capacity - so quickly.

author: tggzzz
author_rank: Super Contributor
timestamp: April 08, 2021, 10:18:16 am
content: Quote from: EEVblog on April 08, 2021, 03:41:53 amYou're not the first to notice that, and won't be the last.The Brown's Ferry fireverynearly caused two nuke reactors to meltdown. It was caused by a candleTalking about the control circuits (myemphasis):Quotehttps://hackaday.com/2018/12/06/fail-of-the-week-1975-the-browns-ferry-nuclear-incident/

author: tggzzz
author_rank: Super Contributor
timestamp: April 08, 2021, 10:24:42 am
content: Quote from: peter-h on April 08, 2021, 09:06:11 am... and test the backups.Old story: backups to tape drive completed successfully, but when needed the tapes were found to be blank. The tape head was electrically connected but hanging loose away from the tapes!

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 08, 2021, 10:34:46 am
content: Quote from: MadScientist on April 08, 2021, 10:01:18 amDon't underestimate the power of fire protection systems and the fire department.A plant near me recently had the control room catch fire. Such a mess. The panel where the fire started was vaporized. The adjacent panels had the wiring behind get so hot that all that is left looks like the wiring in a cheap mattress. The auto fire suppression system went nuts and sprayed yellow/orange gunk all over other panels not near the fire. And the boys that were sent to fix it all now need counseling, because of 60+? years of control mods that nobody kept track of.Management: "But will it be all working again by this time next week?"

author: MadScientist
author_rank: Frequent Contributor
timestamp: April 08, 2021, 10:35:35 am
content: All their reports say they had a mechanical genset failure resulting in a fire. So this isn’t a switch gear fault I suspect. The issue seems to be the proximity of servers to the Gensets. It sounds like they were in the same space

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 10:52:16 am
content: Quote from: james_s on April 08, 2021, 05:05:38 amHalon was banned, but old fire suppression systems may still use it. And there are a few alternatives, i.e. FM-200. Anyhow, you'll also find classic water based fire suppression systems in many data centers. Some of them create a misty spray.

author: YurkshireLad
author_rank: Frequent Contributor
timestamp: April 08, 2021, 11:38:03 am
content: My question is, are you staying with them or moving to another service provider?

author: Gyro
author_rank: Super Contributor
timestamp: April 08, 2021, 11:50:50 am
content: Quote from: YurkshireLad on April 08, 2021, 11:38:03 amThat's a question of whether you stick with the provider that's already had the fire, and hopefully learned something from the experience - or go with a new provider that hasn't had a fireyet!

author: coppice
author_rank: Super Contributor
timestamp: April 08, 2021, 11:55:01 am
content: Quote from: madires on April 08, 2021, 10:52:16 amIn the early 80s the company I worked for kept changing its insurers each year. These alternated between pro-halon insurers and pro-water insurers. We had halon and sprinkler systems alternately installed and ripped out of the computer room I oversaw for several years.

author: coppice
author_rank: Super Contributor
timestamp: April 08, 2021, 11:56:28 am
content: Quote from: YurkshireLad on April 08, 2021, 11:38:03 amWhy abandon one of the hottest operations in the business? Are you concerned their service may be cooling off now?

author: YurkshireLad
author_rank: Frequent Contributor
timestamp: April 08, 2021, 11:56:47 am
content: Quote from: Gyro on April 08, 2021, 11:50:50 amTrue, or do you stick with a provider that had a fire or move to one that hasn't.It is a business after all.

author: Ian.M
author_rank: Super Contributor
timestamp: April 08, 2021, 12:04:37 pm
content: Quote from: YurkshireLad on April 08, 2021, 11:38:03 amBetter the devil you know. At least you can be fairly certain they'll put measures in place to prevent or reduce the impact of similar future incidents, and have some idea of their actual disaster recovery capabilities.It would be nice if WebNX added a large NAS with its own power management in a shipping container at both their locations, with high bandwidth connectivity into their data centers. Its only a bit over 600 miles as the crow flies between their locations, (729 miles by road, est. 10.5H driving time), so if they had another extended outage, the whole NAS could be shipped to their other location to facilitate getting fall-back servers online without the bandwidth implications of either uploading from off-site backups, or live syncing over a significant geographical distance. They'd also need a priority standby contract with a local shipping company at each end to provide a self-loading container truck and two drivers with N hours notice 24/7.

author: wilfred
author_rank: Super Contributor
timestamp: April 08, 2021, 12:10:25 pm
content: Quote from: tautech on April 08, 2021, 04:56:41 amAlso for posterity.

author: wilfred
author_rank: Super Contributor
timestamp: April 08, 2021, 12:21:56 pm
content: Quote from: YurkshireLad on April 08, 2021, 11:56:47 amThe real question is were they prepared as they claimed they would be and did they recover as they promised they would. They can only control how they plan and respond. No-one knows which disaster scenario will play out. It could have been a Texas like power outage and the trucks delivering diesel for the generators may not have been able to get through to resupply.I do think Dave could (should?) make preparations to restart from offsite backups. From what I read there was only 10% of servers water affected. How long would it have taken to recover from a total loss of all the site. I can't imagine sufficient servers would be readily available.

author: cdev
author_rank: Super Contributor
timestamp: April 08, 2021, 12:44:14 pm
content: They put their backup server inLA?The city thats beencinematically annihilatedmore times than any other!Why one only needs to look at maps from not so long ago to realize thatCalifornia is likely to tear itself off of the mainand and strike out on its ownat it'searliest opportunity..Quote from: EEVblog on April 08, 2021, 03:41:53 amFission? Degradedstate?Quote from: EEVblog on April 08, 2021, 03:41:53 amThe city of LA is not a secure location. It's as insecure a location as there is in this country.They suppress all news to the contrary, as a favor to the real estate industry.Ive been told by a relative who was in a position to know.They have lots of tornadoes (and waterspouts, of salty water, which Ive seen between the city and Santa Cataline Island. )And substantial numbers of earthquakes.I aslo learned the importance of relying on a single email server. I was surprised at the stuff I couldn't do that relied on my primary email for confirmations etc.Its truly horrible how much we depend on easily broken computers and unreliable networks.What if (insert unspeakable tragedy here) ? Huh?what if.. We need more resilience and redundancy..

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 01:21:06 pm
content: Quote from: Ian.M on April 08, 2021, 12:04:37 pmFirst off, SAN, not NAS. Secondly, such an off-site backup requires fat pipes anyway for read and write, e.g. 100 Gbps Ethernet. So moving the SAN around won't speed up things. Actually it would cause a delay by the transport and also would add the risk of being damaged during transport, by a traffic accident for example.

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 08, 2021, 03:26:44 pm
content: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.— Andrew S Tanenbaum, 1989

author: Microdoser
author_rank: Frequent Contributor
timestamp: April 08, 2021, 03:33:27 pm
content: This has made me consider my own data safety and the importance of having an off-site backup of at leat the most critical things.

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 08, 2021, 03:40:44 pm
content: I have in front of me a 1TB microSD card, 15 x 11 x 1mm thick (fascinating hi-res X-Rays here). That works out at roughly 6GB per cubic millimetre.Assuming we have a VW Passat (which claims 1780 litres of cargo space with the rear seats down), our modern equivalent of the station-wagon-full of tapes is potentially something like 1780 x (100 x 100 x 100) x 6GB, or around 10 exabytes (10x10^9 GB).And yes I know we wouldn't get that capacity, they're slow to write to, and so on and so on, but it does make one think.

author: Ian.M
author_rank: Super Contributor
timestamp: April 08, 2021, 03:45:21 pm
content: FedEx Bandwidth:https://what-if.xkcd.com/31/

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 03:48:03 pm
content: Quote from: Ultrapurple on April 08, 2021, 03:26:44 pmOr the old fashioned sneakernet (https://en.wikipedia.org/wiki/Sneakernet)Anyhow, do your risk assessment!

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 08, 2021, 04:16:19 pm
content: Quote from: madires on April 08, 2021, 03:48:03 pmYup, that was the source of my Tanenbaum quote.

author: xmo
author_rank: Regular Contributor
timestamp: April 08, 2021, 04:23:51 pm
content: Day three without the forum....TEA user:"I have discovered something interesting."I seem to be living with a woman!Apparently, she is my wife.She seems nice."

author: tooki
author_rank: Super Contributor
timestamp: April 08, 2021, 04:31:22 pm
content: Quote from: Ultrapurple on April 08, 2021, 03:26:44 pmYes, or of actual messenger pigeons!With that said, the critical downside of the station wagon or pigeon is the latency!

author: Microdoser
author_rank: Frequent Contributor
timestamp: April 08, 2021, 04:34:22 pm
content: I have noticed the site, as we were told, seems a bit clunky and slow right now.Still, gives me more time to fix up the I7 6700K 24GB Ram 256GB SSD that I found in the street fully working (except for the graphics card). It has a legitimate Windows 7 Ultimate OEM on it which I am currently getting Windows updates up to date in preparation to upgrade to Win 10 Pro

author: Microdoser
author_rank: Frequent Contributor
timestamp: April 08, 2021, 04:35:58 pm
content: Quote from: tooki on April 08, 2021, 04:31:22 pmOr the time someone put a CD on the back of a snail and sent it across a table and got better data transmission rates than using the LAN (at the time)

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 08, 2021, 04:37:40 pm
content: Quote from: tooki on April 08, 2021, 04:31:22 pmI'm sorry I'm dragging this so far off-topic; I promise to sit on the Naughty Step for a while andThink About What I've Done. But meanwhile, have a ponder about the effective latency in even, say, a thousand parallel, error-free, get-the-label-speed 10Gbps direct linksfor that amount of data. How many miles apart do your data centres have to be before it's quicker to send the data via wires than wheels?And don't get me started on the effective data rate of a cargo plane worth of 6GB/mm3...OK, I've arrived at the Naughty Step. Sitting in 3... 2... 1...

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 04:49:49 pm
content: Quote from: tooki on April 08, 2021, 04:31:22 pm... or a Tesla with autopilot turned on, or Mrs. Eagle looking for a nice meal to feed her children. If that was your only backup you'll have problem.

author: Hamster
author_rank: Regular Contributor
timestamp: April 08, 2021, 04:54:52 pm
content: New Article :https://datacenterfrontier.com/generator-catches-fire-causes-lengthy-data-center-outage-at-webnx/Webhosting talk forum:https://www.webhostingtalk.com/showthread.php?t=1842301---I would abort continuing to host anything at this data center, here in Florida, most all data centers do NOT use water.. they use an FM200 Fire Suppression System, also, Generators are not co-located with the data center, they are usually detached just in case something like this happens, Generators due tend to go, and when the go, they go violently.. the worst part is, this data center probably had multiple gensets, covering the load, when one went down, it caused a cascade effect going down the line.. Fuel should of also been shut off and isolated, but it appears not? There should of been 0 water damage inside the facilities, this clearly is some kind of failed engineering design of a building that probably should of not been designed to be a data center.---Another popular site ( pinside.com ) is still down.. they have recovered the data and restored it to a lower tiered server, but still down.. they have to "order' new hardware for them.

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 05:05:20 pm
content: Quote from: Ultrapurple on April 08, 2021, 04:37:40 pmI'll take the bait.So your data center #1 has burned down, nothing left, no network connectivity, no servers, no power. But you have a backup in data center #2. To which servers do you want to restore the backup?

author: artag
author_rank: Super Contributor
timestamp: April 08, 2021, 05:14:52 pm
content: Quote from: Towger on April 08, 2021, 09:23:33 amAll the reports seem to say that the generators caught fire, the remaining power was taken down by the fire department, and that some servers were in danger of water damage.My conclusion is that the fire department applied plenty of water and a little of it got into a small area of servers. Perhaps near, but not colocated with the fiery genset.

author: ejeffrey
author_rank: Super Contributor
timestamp: April 08, 2021, 05:15:56 pm
content: Quote from: TomS_ on April 08, 2021, 06:35:03 amHalon and FM200 *do not work* by displacing oxygen, that is a persistent myth. They work by neutralizing the free radicals that allow fire to propagate. This allows them to be used in much smaller quantities than would be needed to extinguish a fire by oxygen displacement. Typically they are used at <10% concentration so they only lower the oxygen concentration slightly. You still want to evacuate because the distribution is not uniform and also because -- THERE IS A FIRE, but generally they are low risk to humans.CO2 fire extinguisher systems do work by displacing oxygen, and full-room CO2 systems are not generally used in occupied areas because of this.

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 05:23:55 pm
content: Quote from: coppice on April 08, 2021, 11:55:01 amSeems like they could just leave both systems in place and disable the one they weren't using.

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 05:33:27 pm
content: Quote from: MadScientist on April 08, 2021, 10:35:35 amThe pictures Dave posted look like a random light industrial park of the sort of place my friend's machine shop is in, there's even a sports bar & grill in one of the units. Looks like the genset has to be in the same space, there isn't anywhere else to put it.

author: madires
author_rank: Super Contributor
timestamp: April 08, 2021, 05:56:43 pm
content: Quote from: Hamster on April 08, 2021, 04:54:52 pmWishful thinking! In case of a fire the fire brigade decides what to do. If they think it's a good idea to hose down all servers despite a FM-200 fire suppression system you can't do much about that. Local regulations might force you to use water. Or the management board tells you to use water because it's cheap and the insurance will pay for any damages. Fire is one of the more likely events, but there are also many more, and in those cases it doesn't matter if the fire suppression system uses water or something more hardware friendly. If you need to keep your platform running 24x7 you have to design a redundant solution, i.e. you can't rely on a single data center. Complaining about a data center using a water based fire suppression system is laughable. Simply do your homework!

author: Bud
author_rank: Super Contributor
timestamp: April 08, 2021, 06:10:25 pm
content: I am questioning the details of the culprit, that is - what happened to the backup generatir, did they buy it on Alibaba?

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 06:15:58 pm
content: Quote from: Bud on April 08, 2021, 06:10:25 pmGenerators fail. Even well known name brand stuff like Caterpillar, Detroit, etc blow up now and then, especially older ones which it was mentioned this was. These are big diesel engines, they require maintenance and occasionally stuff breaks. If it spins a bearing, or has an injector problem, or something like a bad seal in a turbocharger can cause the engine to run away consuming its own lubricating oil until it throws a rod through the side of the block and spills oil and/or fuel all over the hot exhaust system and then you have a fire.

author: Bud
author_rank: Super Contributor
timestamp: April 08, 2021, 06:34:28 pm
content: A backup generator's keword is 'backup', isn't it. Some robustness is supposed to be embedded in it from the get go starting from specs.

author: PaulAm
author_rank: Frequent Contributor
timestamp: April 08, 2021, 06:53:39 pm
content: Don't forget that part of the backup power architecture is a humongous UPS. It's job is to keep everything up until the genset is up and stable. Quite possible something went wrong in that gear.My thought after the first day was that EEs everywhere across the world were showing an unexplained increase in productivity

author: SilverSolder
author_rank: Super Contributor
timestamp: April 08, 2021, 07:09:41 pm
content: Quote from: Ultrapurple on April 08, 2021, 03:40:44 pm1TB microSD still seems like sci-fi territory to me... an amazing milestone. Next: 10TB!

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 07:17:35 pm
content: Quote from: SilverSolder on April 08, 2021, 07:09:41 pmMakes me think of the old spy movies where somebody is trying to smuggle a microfilm containing information. These days you could fit an entire library on a thumbnail sized micro SD card that is easily hidden inside nearly anything.

author: SilverSolder
author_rank: Super Contributor
timestamp: April 08, 2021, 07:20:47 pm
content: I know of a sizeable datacenter in east coast US serving the financial industry that went down due to the backup systems many years ago. It is always Murphy that gets you:(1) External power failed, causing the building to switch to battery backup based on enormous banks of lead-acid batteries (a large room full) in anticipation of starting an array of big diesel generators.(2) One of those batteries blew up under the sudden load, spraying an employee with acid. The other men ran to his rescue and got him in a shower real fast (he was OK)...(3) ...but sadly, this cost crucial minutes... and the batteries ran out the second they tried to start the diesel generators... BLINK, the center went pitch black, with no way to start the diesels!

author: jonpaul
author_rank: Super Contributor
timestamp: April 08, 2021, 07:54:03 pm
content: Hello: Thanks for the info,Most pro server farms use a non-water fire suppression system eg nitrogen inerting.Wonder why EEV blog did not select a large host form like AWS, Ionos, GoDaddy etc as a host.I had never heard of this firm.Kind Regards,Jon

author: 3roomlab
author_rank: Frequent Contributor
timestamp: April 08, 2021, 08:06:03 pm
content: maybe it is time to move to other countries with less power interruptions

author: james_s
author_rank: Super Contributor
timestamp: April 08, 2021, 08:56:23 pm
content: Quote from: Bud on April 08, 2021, 06:34:28 pmComplex mechanical devices fail, it's just a fact of life. Turbine engines on aircraft are highly critical life safety items that are engineered to be extremely reliable and impeccably maintained, they still fail catastrophically now and then, sometimes with loss of life. Some robustness IS embedded in backup generators, it isn't like they blow up every day, but occasionally they still fail, especially if maintenance has been less than stellar, which is often the case if a company is not compelled to do it religiously the way aviation is. It's easy to look at the bottom line when the budget is tight and margins are slim and think "do we REALLY need to invest $$$ in rebuilding or replacing the genset or can we defer that to next year?" It happens, and it's much easier to predict and point fingers with the benefit of hindsight. If this place had a history of generators failing I would be more critical, but a single catastrophic failure is much too small of a sample to judge by. It could have been a total fluke where something just blew up due to a manufacturing defect that was never caught, or it could be it wasn't properly maintained, or it could be somebody monkeyed with it at some point, we don't know.

author: bd139
author_rank: Super Contributor
timestamp: April 08, 2021, 10:04:20 pm
content: Interesting thread.Also after looking at the photos and Google maps, an observation:Don’t go with a facility with no crash wall, no fence and parks a trailer probably with a nice propane bottle actually in the boundary of the building. If they do that then there’s probably 9000 even worse things inside the building you can’t see which are waiting to take your business out.May be better to leverage an IaaS cloud here. The whole availability zone concept makes these events into non events. If a facility burns then you lose an availability zone, not your entire deployment.I made this observation originally after mistakenly hosting half a rack of shit at a red neck provider here.Edit: also always host your email somewhere completely different as you might find it difficult to contact support if your email server is on fire (did that once)

author: CatalinaWOW
author_rank: Super Contributor
timestamp: April 09, 2021, 12:01:46 am
content: Quote from: james_s on April 08, 2021, 06:15:58 pmFailure is expected. Failure with catastrophic results is what is surprising. Aerospace uses something called FMECA, (Failure Modes, Effects and Criticallity Analysis) to try to prevent the latter. A spun bearing, or low oil or .... It isn't perfect. Things still slip through the cracks, but apparently we don't care that much about data loss, or respirator failure or anything else that depends on backup generators.

author: ve7xen
author_rank: Super Contributor
timestamp: April 09, 2021, 12:11:53 am
content: Quote from: james_s on April 08, 2021, 08:56:23 pmAnd like you say, sometimes you still get bitten. We're pretty good about maintaining our generators, every 6 months a guy comes from the service company and does all the scheduled maintenance items according to the manufacturers' recommendations, we do monthly full-load tests, and so on.A few years back we suffered a ~12h outage anyway. The genset fired up and took the load, no problem, and a few hours later, we got the overheat alarm and it shut itself down. Go to the bunker and find that one of the coolant hoses has burst and it's pumped all of its many litres of coolant onto the floor. We later learned that while the maintenance guy had been there two weeks before, and a 5-year replacement of the hoses was due, he happened to not have that one particular hose in his truck that day, and we allowed to be deferred to the next maintenance. Murphy gets you every time.

author: EEVblog
author_rank: Unknown
timestamp: April 09, 2021, 12:37:07 am
content: I just got one months credit. So they implemented the 30 day limit clause.

author: bdunham7
author_rank: Super Contributor
timestamp: April 09, 2021, 12:43:58 am
content: Quote from: EEVblog on April 09, 2021, 12:37:07 amAt least they didn't invokeforce majeureand weasel out entirely!

author: james_s
author_rank: Super Contributor
timestamp: April 09, 2021, 01:53:29 am
content: Quote from: CatalinaWOW on April 09, 2021, 12:01:46 amWe don't really know how much people care, because we know very little about the situation, only that a genset they refer to as "older" failed in some catastrophic manner and caused the outage. I've never shopped around for hosting of this sort but I would expect that prices vary widely and that to some extent, you get what you pay for in terms of reliability. There could be redundant sites, or at the very least, redundant gensets located in separate buildings, or newer/higher quality gensets, or any number of other things There is a lot of web hosting, this forum included, which I would not call mission critical. Nobody dies if it goes down for a few days and very few people are even particularly inconvenienced, is it worth paying more for higher reliability? Only Dave can answer that since it's his money, but given the rarity of outages I would vote no. I mean this is ONE outage, and it's impossible to measure reliability from a single failure. Maybe it was an accident waiting to happen and it is miraculous that things held on this long, or maybe it's all top notch high quality gear that is meticulously maintained and something still blew up because, sometimes no matter how careful you are shit happens, we don't know. I'm going to guess that the truth is somewhere in the middle.

author: james_s
author_rank: Super Contributor
timestamp: April 09, 2021, 01:58:04 am
content: Quote from: bd139 on April 08, 2021, 10:04:20 pmThe trailer wouldn't bother me particularly, it looks in good shape and those things don't often just go up unless someone is using one as a meth kitchen. Of all the details observable in the photos, restaurant in the building is the most eyebrow raising. It's not that rare for a restaurant to catch fire although even that doesn't happen every day.

author: tautech
author_rank: Super Contributor
timestamp: April 09, 2021, 02:19:58 am
content: Quote from: james_s on April 09, 2021, 01:53:29 amSure but a year or two back Dave changed hosts to this crowd due to server unreliability and in 8 years I've be aboard this one by far was the largest outage.We may have been inconvenienced some however Dave will have taken a financial hit due to his shop being down and loss of forum advertising.Dave's very lucky gnif does most of his admin on the cheap.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 09, 2021, 03:03:33 am
content: Quote from: bdunham7 on April 09, 2021, 12:43:58 amI was an act of God, I tell you.https://en.wikipedia.org/wiki/The_Man_Who_Sued_God

author: tooki
author_rank: Super Contributor
timestamp: April 09, 2021, 05:18:10 am
content: Quote from: Ultrapurple on April 08, 2021, 04:37:40 pmWell, that’s not really what latency means...(Latency is the minimum time for a single bit to go in one end of the “pipe” and come out the other end. As a metric, it isn’t dependent on the total size of the data you’re sending.)

author: tooki
author_rank: Super Contributor
timestamp: April 09, 2021, 05:23:34 am
content: Quote from: 3roomlab on April 08, 2021, 08:06:03 pmAs someone who has lived in both North Carolina (the worst-ranked place in those graphs) and Switzerland (the best-ranked in those graphs), anecdotally, my experience completely agrees with that! (When I moved from USA to Switzerland the second time, I didn’t even bother buying a UPS, since the power here never goes out. Good enough for my home computing.)

author: Nusa
author_rank: Super Contributor
timestamp: April 09, 2021, 05:31:38 am
content: Dave scares gnif.Sounds normal.

author: hamster_nz
author_rank: Super Contributor
timestamp: April 09, 2021, 05:44:04 am
content: Quote from: Ultrapurple on April 08, 2021, 04:37:40 pmWith a stupid number of media devices (e.g. tape drives) at either end the answer is:10km / 6 mileThat is about the limit for single-mode fibre running at 10G Ethernet....

author: peter-h
author_rank: Super Contributor
timestamp: April 09, 2021, 07:50:58 am
content: There is no easy solution to server backup, because you never know what common vulnerabilities there are.And how much do you want to pay? EEVBLOG probably doesn't make millionsA fully redundant solution isn't cheap.I run a few sites on virtual servers, with various backup policies (which I won't write about openly for obvious reasons) and if the virtual server company blew up and vanished for ever, I could start up a backup server which is a media PC running on an FTTP (80/30mbps) ADSL lineThat would actually be fast enough for EEVBLOG, on a bad day. Times have changed; this would not have been possible 10 years ago, and the bw required for EEVBLOG is probably of the order of 100-500GB/month which is nothing. And the whole real server is rsynced (only changed files copied) to this media PC every night.There are other backups of course because you cannot mirror the whole server, due to many open files etc.I would merely need to manually edit the DNS panel which is hosted by a company different to the server company. So the worst case is losing a day's data. It is practically impossible to lose everything, in this setup, and it is very cheap.

author: ve7xen
author_rank: Super Contributor
timestamp: April 09, 2021, 07:57:18 am
content: Quote from: hamster_nz on April 09, 2021, 05:44:04 am100G at 80km is trivial with off the shelf gear these days, and I don't imagine many datacentre interconnects are only 10G.Even so, the wheels win when it takes longer to transfer the data than it would to drive the media. At 100G you can transfer ~45TB/hr. That's only 4 or 5 hard drives or LTO-8 tapes so it's not hard for the car to win. If you fill a typicalstation wagonsmall SUV (~2000L cargo area) with LTO-8 tapes (~275cm^3), you can fit about 7,500 tapes (x12GB native, no compression here) for 90PB. At 45TB/hr on your 100Gbps that would take 2000 hours during which you should be able to drive/sail anywhere on the planet. Of course it's usually much more practical to transfer it; copying the data to/from the media becomes a significant time sink itself, but that may or may not matter.Fibre can only reasonably win this race when the data volume is relatively small, even when you start talking about 400G systems or multiple links, what you can transfer in an hour still fits in a suitcase.

author: Syntax Error
author_rank: Frequent Contributor
timestamp: April 09, 2021, 09:13:33 am
content: @eevblog : Welcome backQuote from: james_s on April 08, 2021, 05:33:27 pmI agree. I was wondering how water ended up on the racks. And there I was thinking the backup genny was in a shipping container some 100 feet from the main building. Just in case it caught fire. Nar, that is extra rental space.Maybe not odd in Utah, but certainly odd looking in the Uk, the power control boxes, transformer and what I assume is the deisel fuelling point, are not caged in collision resistant fencing - or even a crash carrier. All it would take is a truck driver choking on a hotdog from the grill... and back to square one.Morale of the story from both WebNX and OVH is, backup power systems are highly flammable! Just let the power go off and rebuild the filesystems, not the entire data center.

author: hamster_nz
author_rank: Super Contributor
timestamp: April 09, 2021, 09:19:10 am
content: Quote from: ve7xen on April 09, 2021, 07:57:18 amLOL Writing 7500 LTO-8 tapes and read them back in under 2,000 hours.Sure, the channel bandwidth of an SUV is high, but you can't transmit or receive the data at anything like that rate.Each LTO-8 tape can take 8 hours to write (source:https://en.wikipedia.org/wiki/Linear_Tape-Open) and I'm guessing about the same to read back, that is around 15 hours to move 30TB - about 2TB per hour per drive pair (about 5Gb/s), even just moving data across the room.To fill a SUV with tapes using a single drive will take about 12.8 years, and maybe another 12.8 years to read it back.I'll take the 2000 hours (85 days or so) using a 100Gb fibre...

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 09, 2021, 09:37:37 am
content: Whilst it's been fun discussing the merits and demerits of high-capacity microSD cards vs terabit fibre vs No 8 wire, I think we have lost sight of an important point.Dave provides the world - us - with a fantastic meeting place to discuss our ideas, and he does it without any cost to us. I salute him, and also all those who work with him to make this wonderful place happen.Thank you Dave.

author: madires
author_rank: Super Contributor
timestamp: April 09, 2021, 11:47:45 am
content: Quote from: hamster_nz on April 09, 2021, 09:19:10 amExactly! It's the same fallacy for moving a SAN with backups from one data center to another for restoring servers. The SAN has a limited read/write throughput, let's say 100 Gbps. So you would get a 100 Gbps link between both data centers to be able to backup with full throughput. The same is true for the other direction, i.e. restoring servers. Moving the SAN would cause a delay by the transport and would also add the risk of a traffic accident and possibly the complete loss of the backups. But it wouldn't speed up the restore process because its read/write throughput is still 100 Gbps.

author: Ian.M
author_rank: Super Contributor
timestamp: April 09, 2021, 12:01:34 pm
content: That's not what Amazon claim for their AWS Snowmobile 100PB data transfer storage in a 45' shipping container. They claim up to 1 TB/s aggregated over multiple 40Gb/s interfaces. See their FAQ for details:https://aws.amazon.com/snowmobile/faqs/

author: Algoma
author_rank: Frequent Contributor
timestamp: April 09, 2021, 12:14:21 pm
content: https://en.m.wikipedia.org/wiki/Terabit_Ethernet112Gb/s SerDes in a single channel.. I can only imagine the rate that signal is being switched on and off, let alone trying to sample such a frequency.

author: madires
author_rank: Super Contributor
timestamp: April 09, 2021, 12:23:55 pm
content: Quote from: Ian.M on April 09, 2021, 12:01:34 pmSorry, but I don't get your point.

author: SilverSolder
author_rank: Super Contributor
timestamp: April 09, 2021, 12:29:59 pm
content: Quote from: blueskull on April 09, 2021, 08:36:27 amSome old technology was built really well, and has stood the test of time and proven reliable and inexpensive. (Incandescent light bulbs?)The problem is - how to identify what technology of today, that will gain a good reputation and respect over the next several decades!

author: bd139
author_rank: Super Contributor
timestamp: April 09, 2021, 12:52:10 pm
content: Quote from: SilverSolder on April 09, 2021, 12:29:59 pmThat's fairly easy. Look out for stuff at conferences, then avoid the shit out of it. You want the stale unsexy things everyone takes for granted.My favourite tools this week are SQLite and Python. Both ancient in the scale of thingsQuote from: ve7xen on April 09, 2021, 07:57:18 amActually a surprising chunk of peering is only at 10G per link. The ISP I use has only got 320G aggregate across all links which isn't a lot in the scale of things and it only averages 100-200G. That has tens of thousands of leechers on it.As for transit, as mentioned AWS snowball/snowmobile type solutions are best for moving stuff around in large chunks. You can get 100G into one of them without having to dig up any roads.But better to boil the frog slowly. When I migrated 130TB over to S3 a couple of years back, we built a service abstraction over the SAN and S3 so it used S3 as read-write-through cache for the SAN. This allowed us to sling all the stuff up over a dedicated DirectConnect up to S3 over the space of a few months without introducing any link capacity problems or having to do any nasty switch overs.

author: schmitt trigger
author_rank: Super Contributor
timestamp: April 09, 2021, 01:13:03 pm
content: This incident clearly indicates how dependent we have become of technology in general, the WWW in particular.As such I am sure, well almost sure, that Dave will release a video of the incident, once that the investigation has been completed.It will be an extremely interesting episode.

author: BrokenYugo
author_rank: Super Contributor
timestamp: April 09, 2021, 03:52:03 pm
content: Quote from: Bud on April 08, 2021, 06:34:28 pmOn the other hand outside of some motorsports applications, where spectacular failure is just part of the game, I can't think of harder duty for an IC engine than standby generator service. It's always either sitting around or put on a heavy load from a cold start.I once met a guy who rebuilt big diesel engines, and asked what most of them came from, the answer was mostly generators. Im not sure how much was mandatory maintenance and how much were failures but either way they live a quite hard life.

author: jmelson
author_rank: Super Contributor
timestamp: April 09, 2021, 04:13:55 pm
content: Quote from: 3roomlab on April 08, 2021, 08:06:03 pmI'm in Missouri, current uptime is 177 days, no UPS. I run my web server out of my house.Jon

author: jmelson
author_rank: Super Contributor
timestamp: April 09, 2021, 04:18:08 pm
content: Quote from: Ultrapurple on April 09, 2021, 09:37:37 amYes, INDEED! Thanks, Dave, and all the hard workers at webNX and GorillaServers, and glad to see eevblog back up!Jon

author: james_s
author_rank: Super Contributor
timestamp: April 09, 2021, 06:12:50 pm
content: Quote from: tooki on April 09, 2021, 05:23:34 amI suspect averages like that are misleading, it really depends on where you live. In my location for example the downtown urban areas may go years without a single power interruption while the outlying rural areas may lose power several times a month over the winter and the average of two extremes just a few miles apart is not a very useful number. If you were to look at my whole state, power outages are probably common, however at my house they have generally been rare, this last winter being an unusual exception where I had 3 significant outages. Downtown Seattle which is only about 12 miles from here I have never seen a power outage other than a very localized one due to something like a transformer failure that knocks out a specific building.

author: ve7xen
author_rank: Super Contributor
timestamp: April 09, 2021, 07:14:52 pm
content: Quote from: hamster_nz on April 09, 2021, 09:19:10 amLOL yeah, of course, it's completely impractical, but really it's the same equation regardless of media, the practical limit is going to be how quickly you can read/write the media. You can swap the LTOs with hard drives (though probably can't take as many due to weight) if getting the data off the media faster is important, maybe you just plug them into a chassis on the other end that can use them directly, or even truck the entire chassis of storage in whatever form it's used in production. If you can provision the practical storage bandwidth on the fibre, then it's a wash, otherwise shipping it is going to win (though cost will likely push you to shipping way before you hit the practical limit). The bandwidth of the proverbial station wagon is still immense, even taking practical considerations into account.QuoteWe're not talking about Internet, we're talking about private datacentre interconnect. You're certainly not going to run your PB storage migration over a 10Gb peering link and the Internet. I can't imagine any service provider that would be running lots of uncoloured 10G over dark fibre these days, it's too costly; for your 10G peerings it's either a leased wavelength or more usually just a patch cable within the data centre between cheap ports because you don't need more to that peer (with several peers / transit at the POP, and likely Nx100G to your core). As an SP, you're either leasing a wave on someone else's OTN or running your own WDM system that can carry at least 100Gb per pair ('cheap' bog standard systems do 40x10G, state of the art off the shelf systems do 12x100G or more). Of course there are small SPs that only ever lease 10G or even 1G waves/EPLs, but I thought we were discussing the capacity of the fibre not what a small business actually buys on it.QuoteClever solution, I like it!Quote+1000! I actually had to get work done this week

author: SilverSolder
author_rank: Super Contributor
timestamp: April 09, 2021, 09:00:43 pm
content: Quote from: james_s on April 09, 2021, 06:12:50 pmThe main issue in North America is the "pioneering spirit" electrical system where wires are strung up among the trees in rural / suburban areas... what could possibly go wrong?

author: Red Squirrel
author_rank: Super Contributor
timestamp: April 09, 2021, 09:33:34 pm
content: Quite a crazy turn of events. Comes to show though that even professional commercial setups can and do fail. Always have to be prepared and have offsite backups. In this case they were not needed but things could have been worse.

author: eti
author_rank: Super Contributor
timestamp: April 09, 2021, 11:28:45 pm
content: Wow, EEVblog has a history of attracting water!

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 10, 2021, 01:13:17 am
content: Quote from: eti on April 09, 2021, 11:28:45 pmIt's from all the dissing of those poor, harmless God botherers.Having a gripe about the Hillsong peeps in the solar upgrade vid during their Easter festivus..well karma is a bitch.Thou shalt be baptised.Praise be. And God bless you, Dave.

author: Microdoser
author_rank: Frequent Contributor
timestamp: April 10, 2021, 02:40:23 am
content: Quote from: Ultrapurple on April 09, 2021, 09:37:37 amYou don't seem to have been here long, every thread descends into discussion of that type.If you are trying to change that behaviour, you will be as effective as King Canute.

author: jh15
author_rank: Frequent Contributor
timestamp: April 10, 2021, 03:44:15 am
content: Just put the servers in rice.A few days before this, I heard to put something in rice on the NCIS tv show, a rerun. Now I know where most people got it from.NO!Best part is about 13 mins in.

author: EEVblog
author_rank: Unknown
timestamp: April 10, 2021, 12:00:20 pm
content: Quote from: Syntax Error on April 09, 2021, 09:13:33 amI can confirm that the generator was only meters away from some of the racks in the same room. And that no servers were damaged due to the fire, it was all water.

author: madires
author_rank: Super Contributor
timestamp: April 10, 2021, 12:12:50 pm
content: Diesel generator and racks in the same room? That's insane for many reasons!

author: SilverSolder
author_rank: Super Contributor
timestamp: April 10, 2021, 12:54:48 pm
content: Quote from: madires on April 10, 2021, 12:12:50 pmIt does sound a bit how'ya doin'...

author: bd139
author_rank: Super Contributor
timestamp: April 10, 2021, 01:01:00 pm
content: I think that transcends that and into “what the hell were they thinking?” territory....Suitable ORA book attached

author: tggzzz
author_rank: Super Contributor
timestamp: April 10, 2021, 01:23:00 pm
content: Quote from: SilverSolder on April 09, 2021, 09:00:43 pmSo, completely the opposite of this example, 4 miles from the centre of a major UK city

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 10, 2021, 01:53:25 pm
content: Quote from: tggzzz on April 10, 2021, 01:23:00 pmWe are world famous here for our y-shaped gum trees on the side of the road.https://www.smh.com.au/national/nsw/ausgrid-accused-of-street-tree-vandalism-by-sydney-councils-20150807-giugg9.html

author: calzap
author_rank: Frequent Contributor
timestamp: April 10, 2021, 02:46:59 pm
content: I’m curious as to how many data center standby generators are powered by diesel versus propane. In the U.S., diesel is usually more expensive per kw-hr produced than propane, especially because EPA requires such generators to use ultralow sulfur diesel fuel. Diesel has shelf-life issues which require ongoing testing and maintenance, and replacement in some cases. Propane has an infinite shelf-life. Most building codes in the U.S. do not allow liquid propane inside buildings. So, if there is no outside storage area, propane can’t be used.I know of a biological lab near me that became so fed-up with diesel fuel issues with their generators that they switched to propane and have been happy with it.Nice link:https://www.csemag.com/articles/understanding-backup-power-system-fuel-choices/At my ranch, we have a small (8 kw continuous) backup generator. It’s just big enough to run one well plus a few refrigerators/freezers and a few low wattage items. Forced to run it 4-6 times per year. Longest run has been 2 days. It’s powered by gasoline, and there have been no fuel issues (we use a preservative). It’s in a “dog house” attached to the smallest building on the property. Only real worry is that building is where electrical panels for incoming power are located. In retrospect, I could have built the dog house 10 m away from any building and should have. We have propane available, and dual-fuel installation is on my to-do list … has been for 15 years!Mike in California

author: David Hess
author_rank: Super Contributor
timestamp: April 10, 2021, 03:24:27 pm
content: Quote from: calzap on April 10, 2021, 02:46:59 pmDiesel also has the disadvantage of requiring special containment provisions because it makes a hell of a mess if it spills, which also brings the EPA into it. For this reason and the others you mention, sites which are remote tend toward propane now.For tall buildings, diesel may not be stored at the upper levels so provisions must be made to pump it from tanks near the ground.In some cases requirements are so strict that backup generators of any type are precluded and the only solution is sufficient battery power.

author: David Hess
author_rank: Super Contributor
timestamp: April 10, 2021, 03:47:00 pm
content: Quote from: blueskull on April 09, 2021, 08:36:27 amI do not really consider that an abuse, except of course for the lack of proper decoupling. 723s do not make very good operational amplifiers but most applications do not require good operational amplifiers; consider all of the older regulator designs which only use discrete differential pairs for the voltage and current control loops. 723s also have several virtues including a built in reference and provisions for a high current output.QuoteThat is a common problem even when the same manufacturer changes processes. Unspecified characteristics are unspecified and cannot be relied on. Either test for them or design to handle them.QuoteNon-power analog parts are actually more susceptible to radiation damage than power analog parts, but as long as the NRC follows the rules no matter how stupid, they are safe.It is stupid but I am not surprised. The "safest" option when authority is divorced from responsibility is to do nothing, and make sure the blame will fall on someone else, which is why having nothing to do with that sort of project is the best thing to do. I have learned not to even notify them; they are not interested and doing so can create further jeopardy.

author: madires
author_rank: Super Contributor
timestamp: April 10, 2021, 03:47:52 pm
content: Quote from: David Hess on April 10, 2021, 03:24:27 pmSo propane doesn't need any special containment provisions?Diesel is easy to handle, to refill and to get hold of. Anyhow, you simply use the fuel which is easily available and allowed by local regulations. If you go for a battery system only you need deep pockets, because it has to provide several MW for a few days.

author: Tom45
author_rank: Frequent Contributor
timestamp: April 10, 2021, 04:02:04 pm
content: At a place I worked 50 years ago we had backup power for two large mainframes. The mainframes were fed by motor generator sets so switchover timing wasn't too critical. The top floor of the building was filled with lead acid batteries to hold over until the diesel generator got going.The generator was buried in a concrete pit under the parking lot with a diesel tank above ground.The whole thing worked well until one stormy period when power was so flaky they decided to just run continuously off the generator until the weather calmed down. No problems for a couple of days until someone forgot to order more diesel. Oops.No, it wasn't me.

author: calzap
author_rank: Frequent Contributor
timestamp: April 10, 2021, 04:19:34 pm
content: Quote from: madires on April 10, 2021, 03:47:52 pmAny fuel stored onsite obviously must have a primary container. In the U.S., propane tanks containing liquid propane must be outdoors. There are usually building setback requirements as well. So, if there is a leak, it wafts away as a gas. Large diesel tanks in most jurisdictions must have secondary containment in case of a leak. If indoors, there are usually fire sprinkler or other suppression and protection requirements as well.Mike in California

author: David Hess
author_rank: Super Contributor
timestamp: April 10, 2021, 04:26:34 pm
content: Quote from: madires on April 10, 2021, 03:47:52 pmPropane requires a pressure tank but is actually *safer* if there is a leak or fire. A diesel leak makes a hell of a mess which is where the EPA gets involved. A propane leaks leaves nothing to clean up.Propane tanks handle fire just fine. When the pressure relief valve activates, the propane exhaust is burned and evaporation cools the tank until the propane is exhausted. Just make sure that the exhaust is directed in a safe direction. I know of one case at a mountaintop repeater site where the exhaust was directed at the blockhouse. When personal showed up to find out why all of the repeaters had failed after a brush fire, they found that the blockhouse was completely destroyed (melted) by the jet of burning propane.

author: james_s
author_rank: Super Contributor
timestamp: April 11, 2021, 07:46:55 am
content: Quote from: calzap on April 10, 2021, 02:46:59 pmI've seen natural gas fired backup generators which have the obvious advantage of fuel being piped in rather than stored on the premises. Propane, that I have not seen other than for small portable generators and some that are used in RVs where you already have propane available. I think diesel pretty much owns the large backup generator market, the engines are the same as used for things like semi trucks, motor yachts and locomotives. I don't think anybody is making huge spark ignition engines anymore although there have been some really big ones in the past.

author: SilverSolder
author_rank: Super Contributor
timestamp: April 11, 2021, 11:00:21 am
content: The cost of diesel probably doesn't actually matter, since the generators are not used that often.

author: madires
author_rank: Super Contributor
timestamp: April 11, 2021, 11:27:21 am
content: Quote from: David Hess on April 10, 2021, 04:26:34 pmI'd think that propane is more prone to a nice BOOM than diesel. BTW, it's also heavier than air. And don't worry too much about the storage of diesel. Storage regulations for oil and oil based fuels take care about that.

author: TimFox
author_rank: Super Contributor
timestamp: April 11, 2021, 02:28:48 pm
content: In the recent Texas cold-weather catastrophe, the natural gas delivery failed due to lack of power at the wellhead compressors and frozen distribution pipes.

author: Ultrapurple
author_rank: Super Contributor
timestamp: April 11, 2021, 03:01:41 pm
content: Quote from: Microdoser on April 10, 2021, 02:40:23 amI invite you to checkmy profile info.

author: bdunham7
author_rank: Super Contributor
timestamp: April 11, 2021, 03:06:25 pm
content: Quote from: james_s on April 11, 2021, 07:46:55 amWaukesha Engine is still around, under different ownership.https://www.innio.com/en/products/waukesha

author: calzap
author_rank: Frequent Contributor
timestamp: April 12, 2021, 01:22:55 am
content: Quote from: james_s on April 11, 2021, 07:46:55 amI agree diesel-powered industrial generators are probably the most common now. However, I think that will change in favor of natural gas and propane where they are readily available. Energy costs, emissions limits and fuel stability will bring it about.Natural gas has the advantage of not having to store it on site. In fact, storing it on site, as CNG or LNG is an expensive proposition. However, as recent events in Texas have shown, natural disasters can stop the flow of natural gas. That low temperatures did it is primarily a reflection of inappropriate penny-pinching in engineering and building pumping stations. Even with appropriate design and installation, natural disasters can interrupt flow … like in an earthquake area where I live.Large industrial standby generators powered by propane or natural gas are already available. For example, Generac sells 150 kW propane generators and 500 kW dual fuel (natural gas/ diesel). And no, separate engines aren’t required for natural gas/diesel gensets. They are diesel engines modified to aspirate an air/gas mixture, which provides most of the energy. Small diesel injections provide ignition, but the engines can run on diesel alone if necessary. Generac’s largest natural gas generator is 1 MW and powered by a 12-cylinder, 49 L spark-ignited engine.Mike in California

author: tautech
author_rank: Super Contributor
timestamp: April 12, 2021, 01:34:40 am
content: Gas powered gensets are nothing special or new. Not far from me is a bank of 1MW gensets running on methane from a landfill.https://www.terracat.co.nz/power-systems/new-power-systems/epg/gas-generator-sets

author: james_s
author_rank: Super Contributor
timestamp: April 12, 2021, 01:53:26 am
content: Quote from: madires on April 11, 2021, 11:27:21 amThat's definitely true. You can throw a lighted match into a bucket of diesel fuel and it will go out. If you do manage to light the stuff, it burns pretty lethargically, it's similar to kerosene or salad oil as far as flammability. Propane on the other hand can be dangerous stuff. Unlike many fuels, it doesn't need a stoichiometric mixture to burn explosively, indeed a small engine will run pretty well if you just poke the end of a non-lit propane torch into the air intake, it's not like gasoline where the mixture has to be just right, propane will still go bang in very rich or very lean conditions.

author: floobydust
author_rank: Super Contributor
timestamp: April 12, 2021, 02:15:51 am
content: I looked atgoogle street viewand only one exhaust pipe for a generator, near the electrical room.The facility appears to be in some old warehouse (military?) district with brick exterior walls and a wooden roof? If true that's a problem.Had to laugh, not a solar panel in sight.

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 12, 2021, 03:04:00 am
content: How much can the backup power infrastructure be cut back if there's a system to force all CPUs to minimum frequency when running on backup power?

author: james_s
author_rank: Super Contributor
timestamp: April 12, 2021, 04:03:50 am
content: Quote from: floobydust on April 12, 2021, 02:15:51 amBuildings like that are all over the place in light industrial areas. As I mentioned my friends have a machine shop in a similar building, they've had all sorts of different neighbors and they've moved a couple of times too. Auto mechanic, sign company, importer warehouse, cabinet maker, in their current spot the place next door sells and services air compressors, that kind of stuff. In most of these industrial parks the business rents one or more bays and outfits the interior as needed. Solar probably isn't really an option in most of those places, they don't own the building and the landlord doesn't care about the power bill, they aren't the one paying it. They don't want a bunch of holes drilled in their roof.

author: schmitt trigger
author_rank: Super Contributor
timestamp: April 12, 2021, 03:05:16 pm
content: Very interesting thread!As I mentioned previously, hopefully Dave will do a video(s) regarding the subtle details of ensuring AC mains uptime.With everything nowadays tied to the web, this issue has become more critical than ever.Back in the mainframe days, I remember a motor-generator set where the blackout ride-through energy was stored in a gigantic flywheel.To my surprise, they are still being used.Google has plenty of examples..

author: duckduck
author_rank: Frequent Contributor
timestamp: April 12, 2021, 06:41:00 pm
content: Quote from: NiHaoMike on April 12, 2021, 03:04:00 amThat's a great idea. One issue I can see for a hosting company is that they allow their customers to manage/reinstall to OS and apps. It would be difficult to enforce the installation of power-management software. It would be great if servers had a (let's say) 5 volt input, and when it dropped to below 1 volt, the BIOS would throttle the CPU down. Then you "just" run a 5 volt line run off of non-UPS, non-generator mains to each server and you're golden.

author: schmitt trigger
author_rank: Super Contributor
timestamp: April 12, 2021, 07:02:11 pm
content: In the rotary UPS I mentioned above, it was explained to me that during the elapsed time between mains interruption and the diesel genset actually supplying power, certain non-critical processes were halted. Like printers and punched card readers.There may have been others.

author: bd139
author_rank: Super Contributor
timestamp: April 12, 2021, 07:10:03 pm
content: Quote from: NiHaoMike on April 12, 2021, 03:04:00 amThat probably wouldn't work. If you're running near your CPU provision, you can enter a thing called "load hysteresis" which may be irrecoverable. This is where your load average goes above the total capacity and the CPUs can never catch up with the workload. It requires adding much much more capacity than you had to start with before you can being the demand you had originally. Either that or breaking a huge chunk of your incoming load to recover.This is similar to halving the size of your cluster during peak demand, which never works out well. I was working for a very large stats company here a few years back and there was a sudden spike of packet loss over the inter-DC link they were running. The ops director at the time decided to fail the active-active over to active-standby and caused a 5 minute chunk of slowness into a 4 hour recovery job.

author: gnif
author_rank: Unknown
timestamp: April 13, 2021, 12:23:50 am
content: Quote from: peter-h on April 09, 2021, 07:50:58 amAnd how would you know how much bandwidth EEVBlog uses each day? Outgoing traffic averages 90mbps peaking out at 200mbps during high activity periods such as giveaways, etc.QuoteWhat if someone deletes all the files on your server and rsync does it's dutiful job and deletes them on your home machine? Or worse, some files are corrupted and you dont detect this for a few weeks? rsync is not a recommended enterprise grade backup solution, I suggest you look into BareOS (Free), or R1Soft (commercial).EEVBlog is backed up daily to two remote locations with a 6 month data retention policy, restoration to a bare metal server can be done in an hour or two (depending on network speed) and server configuration is performed via Puppet, this takes mere seconds. Total production ready stand-up time form bare metal is restore time + 10-20 seconds. With a warm backup (which we will be looking into) downtime would be nearly zero in the event of a failure like this in the future.One has to weigh up the time vs cost in standing up a new server when things like this happen. Sure, Dave could have paid for a server elsewhere (in fact, we had several offers of temp servers), however this makes things more complex when it comes to decommissioning these servers when they are no longer needed. Ie, sync temp to primary servers, change over DNS records and while waiting for DNS records to propagate proxy traffic to the primary servers from the temp server. At the end of the day, it's up to the site owner to decide on the best course of action for their business, even if there are technical solutions that could be implemented here and now.When you start hosting sites the size of EEVBlog you will quickly learn that you can't just cowboy things, because that 0.5s of downtime when you decided it wont hurt to just restart the HTTP service to make a config tweak, will impact people.

author: SL4P
author_rank: Super Contributor
timestamp: April 13, 2021, 01:27:39 am
content: Simple question.Why was there water damage in the datacenter? Surely the backup power was in an adjacent building or basement ?

author: gnif
author_rank: Unknown
timestamp: April 13, 2021, 01:29:20 am
content: Simple answer, read through this thread. GS are still recovering and are yet to release details.

author: Monkeh
author_rank: Super Contributor
timestamp: April 13, 2021, 01:37:39 am
content: Quote from: floobydust on April 12, 2021, 02:15:51 amWhat on earth makes you think that's a wooden roof?

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 13, 2021, 02:45:44 am
content: Quote from: duckduck on April 12, 2021, 06:41:00 pmI have hacked some PCs to do just that by using a small MOSFET to pull the PROCHOT line to ground.Quote from: bd139 on April 12, 2021, 07:10:03 pmThat doesn't sound like something that should happen with a robustly designed service, wouldn't that mean a hacker who wants to take it down can just DDoS it for a short time and let queue overflow continue it for much longer than the initial attack?

author: drussell
author_rank: Super Contributor
timestamp: April 13, 2021, 03:04:51 am
content: Quote from: duckduck on April 12, 2021, 06:41:00 pmNo, it is not.You don't randomly force-throttle a server...In many cases that would be just as bad of a scenario and you might as well just have pulled the power.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: April 13, 2021, 03:16:16 am
content: Quote from: drussell on April 13, 2021, 03:04:51 amSoftware should tell the hardware what to do, not the other way around.

author: NiHaoMike
author_rank: Super Contributor
timestamp: April 13, 2021, 03:36:03 am
content: Quote from: drussell on April 13, 2021, 03:04:51 amWhy not if the software could recover once things are back to normal? Obviously, you wouldn't do that for critical real time applications like a VoIP server, but for something like a web server, I don't see why the software couldn't be designed to handle it gracefully. Perhaps whether or not a server gets throttled on backup power be something that's reflected in the service fees (perhaps multiple levels that specify how much throttling and for how much time before partial refunds will be provided), I'd imagine that would do a lot to motivate the developers to make their programs able to tolerate it in order to take advantage of cheaper hosting.

author: drussell
author_rank: Super Contributor
timestamp: April 13, 2021, 04:07:33 am
content: Quote from: NiHaoMike on April 08, 2021, 04:21:51 amQuote from: NiHaoMike on April 13, 2021, 03:36:03 amOh, my....You've obviously never had the joy of running any "real" servers.(evenrealweb servers)

author: floobydust
author_rank: Super Contributor
timestamp: April 13, 2021, 04:10:30 am
content: Quote from: Monkeh on April 13, 2021, 01:37:39 amI see a (modified) bitumen roof with wood fascia. A "fire suppression system" does nothing if that lights up.It's common problem in building fires here, the roof and trusses (which are above all the sprinklers) can start burning and the fire crawls along the roof. Firefighters have to dump water despite the interior not being on fire at all.I would think buildings in the UK have similar issues, with flat tar roofs?

author: Monkeh
author_rank: Super Contributor
timestamp: April 13, 2021, 04:27:54 am
content: Quote from: floobydust on April 13, 2021, 04:10:30 amI see a structure which will have steel trusses. The deck could be anything up to and including pre-cast concrete panels.

author: ejeffrey
author_rank: Super Contributor
timestamp: April 13, 2021, 04:31:23 am
content: Most cloud compute platforms offer some form of preemptable / flexible instance class that the provider can shut down at any time. This allows them to reduce over-provisioning while serving peak loads to high paying customers but I'm sure it can also be used to throttle for power consumption purposes due to distribution capacity, backup generator size, or cooling. You then pay less -- possibly much less -- per CPU hour. It's not usually useful for a webserver since you can't control when users come to your site. You can't get an HTTPS request and just say "I'll respond to that when the spot price drops." There might be situations where a moderate amount of downtime isn't a big deal but it isn't the normal situation for commercial web hosting.Cancelling low priority or low urgency jobs is much more effective than force throttling CPU frequency. Even cancelling high priority jobs is sometimes the best approach if it gets you through a crunch without outright failure. Throttling CPUs not a great way to apply backpressure. At first it does nothing but reduce the sleep/idle time fraction. Then it generates congestion. Only when the congestion degrades the service enough people stop using it does it really reduce the workload. Essentially you put human behavior in your feedback loop, and require poor quality of service to be effective.

author: floobydust
author_rank: Super Contributor
timestamp: April 13, 2021, 04:49:21 am
content: Quote from: Monkeh on April 13, 2021, 04:27:54 amThe roof overhang on the loading docks is all wood, for many of the buildings. They might be cheaply built, whatever era they are from. There's no rooftop A/C on any of the buildings so they might not support any weight beyond snow load.Point is, a datacenter should be constructed entirely of non-combustible building materials IMHO.

author: james_s
author_rank: Super Contributor
timestamp: April 13, 2021, 06:06:40 am
content: Quote from: floobydust on April 13, 2021, 04:10:30 amWood roofs are typical on that sort of building, I don't recall the name they call buildings of that style but the walls are cast in place reinforced concrete with wooden beams running across, supporting a plywood roof that is covered in a weatherproof outer layer. All three of the light industrial complexes my friends' shop has been been built in that way.

author: bd139
author_rank: Super Contributor
timestamp: April 13, 2021, 07:40:22 am
content: Quote from: NiHaoMike on April 13, 2021, 02:45:44 amDDoS is not something you handle at the service level. There’s no way to sink one. I mean how do you scale up something (taking our shit as an example) from aggregate average 700Mbit out to say 40Gbit out which is your saturation boundary? Even if you can scale up enough nodes horizontally it’s unlikely to be sensible or even reasonable to add that capacity. I mean we’re not going to add 57x the database capacity on the fly when our customer base is fairly static.In this case we pay the provider to handle it. They do traffic analysis and if they detect an incoming DDoS then they drop the traffic from it at their network edge.Edit: there is a mid ground though which is perfectly valid where you are unexpectedly successful. This can make or break a product. I’ve seen both outcomes.

author: calzap
author_rank: Frequent Contributor
timestamp: April 13, 2021, 05:25:36 pm
content: Quote from: james_s on April 13, 2021, 06:06:40 amOften done as tilt-up construction. Floor slab is poured. After it hardens, forms for wall sections are put on the floor, and the wall section concrete is poured. After wall sections are hard, they are hoisted into place with a crane. Has been most popular in the U.S., Australia and NZ. Wikipedia describes it pretty well.Mike in California

author: tautech
author_rank: Super Contributor
timestamp: April 13, 2021, 06:20:50 pm
content: Quote from: calzap on April 13, 2021, 05:25:36 pmWould be nice if it were so but sadly not as few slab layers are capable of laying a perfectly flat slab so here we have had professional tilt slab companies for some decades. Finished and cured tilt slabs prefitted with lifting eyes are trucked on edge from the tilt slab plants to construction sites where they are lifted from trucks and placed straight into position were they are strapped to adjoining slabs and temporarily held vertical with adjustable building props until roofing trusses and 2nd story steelwork can be fitted at which time they are considered a safe structure.Mini tornado flattened some stood and braced slabs a couple years back killing some workers.

author: Nusa
author_rank: Super Contributor
timestamp: April 13, 2021, 06:24:25 pm
content: You're talking about modern construction techniques. The building in question was built during WWII using brick and steel construction. The whole complex (Defense Depot Ogden) remained military until about 25 years ago, and is now known as Business Depot Ogden. Some of the warehouses are still owned and operated by the military, but the complex as a whole is now commercial.There's an interior picture of one of the old warehouses here, clearly showing the steel construction inside the shell:https://www.boyerbdo.com/history/

author: floobydust
author_rank: Super Contributor
timestamp: April 13, 2021, 08:19:50 pm
content: Yes, wood roof on steel trusses mostly, there are huge timbers in old pics but a newer pic (brewery) shows them replaced with new concrete and steel. I imagine the buildings are all renovated. Back then lumber was plentiful.It's very difficult with old buildings to meet modern NFPA fire safety codes. They usually don't have enough exits or an updated fire-suppression system. Sprinklers being under the roof are useless and in the pic pipes extend outside, under the overhang - those pipes freeze here in winter.The military history of the area is incredible 5,000 POW's working there in WWII. B-17's being assembled."The soil and groundwater beneath Business Depot Ogden have been polluted with trichloroethylene, vinyl chloride, arsenic, lead, cadmium, mercury, barium and pesticides. The toxic mix of chemicals is the result of decades of cavalier disposal and burning of military-grade trash at the 1,100-acre former military facility, according to EPA records."

author: gnif
author_rank: Unknown
timestamp: April 14, 2021, 01:43:43 pm
content: See:https://www.eevblog.com/forum/chat/website-error-reports/msg3548995/#msg3548995

author: madires
author_rank: Super Contributor
timestamp: April 14, 2021, 02:05:14 pm
content: Looks like communication skills aren't a strength of GS.

author: YurkshireLad
author_rank: Frequent Contributor
timestamp: April 14, 2021, 02:08:02 pm
content: Quote from: madires on April 14, 2021, 02:05:14 pmTime to switch?