Le Cloud goes up in smoke - Page 1
Forum section: electronics / general

author: Syntax Error
author_rank: Frequent Contributor
timestamp: March 10, 2021, 11:24:03 pm
content: If you don't already have a disaster recovery plan for your web site, web service and mailboxes, maybe you should start thinking about having one. 555 > Internal server fireQuote

author: SilverSolder
author_rank: Super Contributor
timestamp: March 11, 2021, 01:25:42 am
content: Yeah, another downside of the "all eggs in a few baskets" cloud approach.The convenience comes at a cost...

author: Halcyon
author_rank: Unknown
timestamp: March 11, 2021, 02:58:05 am
content: Wouldn't that be poor planning on the DC operator?For someone like Google, Microsoft, Amazon, I'm sure if a DC or two went down, they would have high-availability in place so that downtime is minimal.

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 11, 2021, 06:16:26 am
content: As an OVH customer myself, I agree with Halcyon. (The services I pay for were not affected, though.)Unless it turns out it was not an accident, I do believe OVH dropped the ball here. The loss of a single rack is always possible, but proper automatic halon extinguishing systems should have stopped the fire. Apparently, it destroyed one hall, and partially spread to another hall.

author: hamster_nz
author_rank: Super Contributor
timestamp: March 11, 2021, 07:39:54 am
content: This is why proper 'clouds' have availability regions and availability zone. Most cloud providers don't promise 100% availability - it's the customer's responsibility to architect their systems correctly for the availability and redundancy they need.Gaseous fire suppression are not really practical in 'warehouse scale' data centers.... when triggered the rise in air pressure can literally blow the roof off or walls out, and the systems themselves present a risk (lots of gas in high pressure cylinders). Nor is localized inert gas fire suppression effective in Data Centers as they often don't have confined spaces and the have lots of fans and chillers moving air around to cool the equipment.In a lot of cases it's good site practices (e.g. no cardboard/paper, good cable management, hotwork processes, regular inspection with IR cameras), VESDA, temperature monitoring, and finally well-trained people with fire extinguishers as the last line of defense.Sure, you can engineer up to it, but having a workable BCP costs less.

author: Berni
author_rank: Super Contributor
timestamp: March 11, 2021, 07:43:53 am
content: Yep id say its the datacenters job to make sure things keep running when things go wrong.It's what people pay them for. If you wanted to deal with hosting a website or service yourself then you could buy your own server box and plug it into a fast internet connection. Or possibly pay someone to put your machine into a rack and feed it with power and internet for you, if its important pay someone else at another location to do the same too.With these cloud computing providers the whole point of using them is so that you don't have to care about keeping a physical machine running somewhere. You give them money and they give you access to a VM instance that is magically running on a machine somewhere on the internet, you don't know what machine is it or where exactly it is, it might be a different machine than it was yesterday, but you don't have to care because you are paying them to take care of it.But if you ask me a fire should not be this devastating on a data center in the first place. They have tons of airflow trough these server racks, so smoke can be detected quickly, they have halon fire suppression systems, they can close off air flow, some even put compartment walls around rack groups to better focus cooling... etc Or is the data center industry reaching a race for the bottom where they have to cut corners and minimize any non vital costs in order to turn a good enough profit.

author: ataradov
author_rank: Super Contributor
timestamp: March 11, 2021, 08:09:37 am
content: The only site that I use that was affected by this is lichess.org, and only part of it went down. They recovered in a few hours. So apparently they have a decent backup strategy.And yes, I'm surprised that it spread further than one rack. There may be more to this story.

author: TerraHertz
author_rank: Super Contributor
timestamp: March 13, 2021, 11:10:54 am
content: Quote from: ataradov on March 11, 2021, 08:09:37 amPossibly...From another site, not sure if these articles are visible without an account:* Early 10th MarchQuote* Later on 10th MarchQuoteGoogle 'OVH web hosting: SBG1, SBG2' for public articles.www.datacenterdynamics.com/en/news/ovh-fire-update-four-halls-sbg1-destroyed-well-all-sbg2/www.datacenterdynamics.com/en/news/fire-destroys-ovhclouds-sbg2-data-center-strasbourg/Both include aerial photo of site after fire, tweeted by Strasbourgh Bas-Rhin fire serviceHere it is:Search gmaps: OVH bas-rhin Strasbourg, FranceAltimat OVH 9 Rue du Bassin de L'Industrie ("Temporarily closed")goo.gl/maps/MnAtdRPBbKwRWYYF8The whole site is quite recently constructed. On google maps terrain view it is still under construction, and in street view it isn't there at all. The site is just across the Rhine river border from Germany.I wonder if they didn't get around to installing fire suppression systems yet? And the old favorite - "welders at work".Incidentally for any urbex enthusiasts, the vast industrial buildings just to the North on that peninsular, are clearly vacant.

author: kripton2035
author_rank: Super Contributor
timestamp: March 13, 2021, 12:22:34 pm
content: a recently maintained inverter seems to be at the origin of the fire ...12 to 16000 domain names are off. some have backups, some dont.some backups were in the same datacenter and also went in smoke.hope the inverter maintenance company has a good insurance policy ...

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 13, 2021, 02:20:22 pm
content: Quote from: kripton2035 on March 13, 2021, 12:22:34 pmI wonder if"oops, we disabled the fire suppression systems for the duration of the maintenance, and forgot to enable it afterwards/for the night"is involved...

author: Syntax Error
author_rank: Frequent Contributor
timestamp: March 13, 2021, 11:40:56 pm
content: Quote from: TerraHertz on March 13, 2021, 11:10:54 amBuilt remotely on what looks like a wind swept freight yard, there must be some serious backbone bandwidth crossing the Rhine at this point? There is on going development at the site; the tarmac on the road is barely a week old. A curious combination if hard buildings and shipping containers.Google Streetview is your friend. You can even peak in through the windows and see what was fried-funked.

author: TerraHertz
author_rank: Super Contributor
timestamp: March 14, 2021, 01:57:20 am
content: Interesting that your google streetview shows it present, ie taken fairly recently, while streetview accessed in Australia is old enough for the entire server site to be absent. Just the old entrance gate to the previous industrial complex.The streetview I've attached is the right place, you can see the 'end of rail line' buffer over at the right. That is visible in aerial views and very lower left in the 2nd firefighting shot I posted above.https://www.google.com/maps/@48.5851711,7.7974195,3a,75y,7.06h,83.32t/data=!3m6!1e1!3m4!1sB5ftKtUHPHChNHOG6JC57g!2e0!7i13312!8i6656

author: Syntax Error
author_rank: Frequent Contributor
timestamp: March 21, 2021, 12:06:01 pm
content: An update:OVH to shutter second strasbourg data center after  smoke incident:QuoteThe OVH Strasbourg realtime vms status monitor shows how much physical real estate is affected. The gray cloud:http://status.ovh.com/vms/sbg/

author: Simon
author_rank: Unknown
timestamp: March 23, 2021, 08:12:23 pm
content: Oh, OVH, yea I left them last year. Awful awful service. Nothing on there made sense and they tried to stop people taking their domains elsewhere by making the interface so complicated yet so non descript. I was saving a couple of bucks per domain but decided that the massive inconvenience of doing anything with them was not worth the saving after they left one of my domains in limbo during a transfer and at risk of expiring without me being able to access it for renewal. Have no sympathy for them. Never used any of their hosting, just running a few domains with them was enough to scare me off.Unfortunately you do have to think of everything these days in the cost cutting market where people will save you a few percent by not doing something that is only apparent when stuff like this happens. My provider make clear that they have off site backup storage. Backups are obviously not the same as guaranteeing uninterrupted service, to loose the service for some days whilst backups are put onto alternative equipment is unfortunate, but to not have any backups is incompetent.

author: Berni
author_rank: Super Contributor
timestamp: March 23, 2021, 08:20:30 pm
content: There is also a slovenian website that used there hosting that went down. They claim they had a hosting plan that included backup services, but those backups went offline along with all the rest of there site. Luckily they had backups of the whole site and database elsewhere too but they have to transfer it over to the new host before it can be brought back up.