Business users of Microsoft’s cloud-based Azure services had a hard time getting their work done Tuesday as the software giant reported widespread outages that resulted from temperatures overheating in one of its data centers.

Microsoft said the problem occurred at a facility in Texas and impacted Azure users in the south-central area of the United States.

According to Microsoft’s Azure Support Twitter feed, “A severe weather event, including lightning strikes,” occurred near the data center that caused a power surge, which impacted the cooling system.

As a result, the facility went into a shutdown process to keep the Azure servers and other data center gear from overheating. Among the services affected by the outage were Microsoft’s Office 365 and Active Directory offerings.

Azure is a set of cloud-based services used for building, testing, deploying and managing applications and services that Microsoft manages through a network of its own data centers. The services include website operation, storage and data networking management. Microsoft Azure competes in the cloud-services business with the likes of Amazon Web Services.
• Cybersecurity: Don’t be in a rush to click on links to fake sites possibly from Russia At 11:33 a.m. Pacific time, Microsoft’s Azure Support tweeted that engineers had successfully restored power to the data center and had recovered a majority of the network devices affected by the shutdown. The company said “mitigation efforts” were continuing in order to get all of the services involved back up and running. </s> Microsoft was scrambling to get a number of cloud-based services up and running this morning, after a lightning strike near one of its San Antonio data centres caused a voltage surge. This in turn forced a power-down when cooling equipment failed, triggering an Azure outage locally and issues with Office 365 globally.

The outage even took the Azure service status monitor offline.

The company said in an Azure status update: “Engineers are prioritizing the restoration of storage resources in order to recover all services with dependencies on these impacted resources. As storage mitigation continues to progress, a necessary extended mitigation phase is required.”

The company added that while power had been restored and software load balancers for Azure storage scale units recovered, work to recover a number of services was still ongoing. The outage appears to have also affected Office 365 users globally also: the enterprise software is not based on Azure, but does use Azure Active Directory authentication services.

Azure Outage: Enough Being Done on Resilience?

Pete Banham, cyber resilience expert at Mimecast, said: “Today’s incident at Azure was another clear reminder for the need for organisations to build in their own redundancy rather than rely on a single vendor. All organisations, including Microsoft, need to consider what downstream effects there may be from losing a critical service due to technical failure or human error.

“Should employees around the world using Office 365 be reliant on a single Azure DC in the US? Services will always fail and IT leaders need to ensure they have not outsourced responsibility to a lone cloud service.”

Speaking to Computer Business Review, he added: “It’s an open question how one data centre outage can cause so much disruption… clearly companies like Microsoft want to benefit from economies of scale, but from a mission-critical side of things, if users can’t even check support, that’s a challenge.”

Microsoft has been contacted for comment.

See also: Microsoft Office 365 Outages Reported Across UK </s> Microsoft is working to fully restore key services in its Azure cloud platform after an outage in a US region.

Since 9am GMT on Tuesday, Azure services have been intermittently unavailable to customers of Azure's South Central US region, due to an outage affecting Azure Storage and the wide range of services that rely on it, including Machine-Learning Studio, Cloud Services, Database for SQL, Functions and more.

While the outage initially resulted in online Microsoft services such as Office 365 and Azure Active Directory being unavailable for some customers in other regions, access has now been restored, with the problems attributed to congestion caused by automated failover procedures.

As of 11am GMT on Wednesday, some Azure services in the South Central US region remain affected, although Microsoft's support site says "engineers have restored storage availability for the majority of impacted services, and most customers should now be seeing signs of recovery".

Azure engineers are now working on recovering impacted Azure Storage scale units and the remaining storage-dependent services in South Central US, work Microsoft says is mostly complete.

Microsoft blamed the outage on a "severe weather event", which it said included lightning strikes that occured near one of the South Central US datacenters.

"This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process," according to the Azure status update.

An infrastructure outage at one of the major cloud platforms can take down the many of the high-level services these platforms offer, despite the redundancy and automated recovery built into these platforms.

In March last year, a wide range of Amazon Web Services offerings in the US-EAST-1 region were unavailable after an engineer inadvertently took a large number of Amazon Amazon Simple Storage Service (S3) servers offline.

While cloud outages hit the headlines because of the large numbers of people affected, some analysts advise the advantages of the public cloud over in-house infrastructure offset the risk of an occasional hiccup.

Modern firms are so reliant on the major cloud platforms that earlier this year insurance provider Lloyd's warned the failure of a top cloud service for just three days could cost the US economy $15bn. </s> A number of U.S.-based customers connecting to Microsoft cloud services, including Office 365, Azure Active Directory and Visual Studio Team Services, are reporting outages this morning, September 4. The issue seems to stem from problems in Microsoft's South Central U.S. datacenter.

The problems also are affecting users in other regions who are using services that authenticate with Azure Active Directory and who are attempting to use the Azure Resource Manager, I'm hearing from users on Twitter.

An Office 365 service health status page notes that some users have been unable to authenticate or connect to Office 365 services. The Office 365 admin center also has been down for these customers since at least 9:09 a.m. UTC (5:09 a.m. ET), according to that page.

"We've determined that a data center issue caused a subset of the Office 365 service to become degraded. We're connecting some of the affected services to an alternate infrastructure, while remediating the underlying issue within the datacenter," Microsoft's message says.

Office 365 doesn't run on Azure, but it does use Azure Active Directory authentication services.

An Azure Support message on Twitter acknowledged the problem in the South Central U.S. and said that "engineers were still working on the resolution of an issue affecting resources" there. Microsoft is "actively monitoring for impact to other regions also," the message said, directing customers to see status.azure.com for further updates. But as a number of Twitter users noted, they were unable to access that page.

Those who can see the Azure status page may see this message (as of 10:25 a.m. ET):

"Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Engineers have isolated an issue with cooling in one part of the data center, which caused a localized spike in temperature, as the preliminary root-cause, which has now been mitigated. Automated data center procedures to ensure data and hardware integrity went into effect when temperatures hit a specified threshold and critical hardware entered a structured power down process. Engineers are now in the process of restoring power to affected devices as part of the ongoing mitigation process. "Some services may also be experiencing intermittent authentication issues due to downstream Azure Active Directory impact, and engineers are separately working on mitigation options for this also. "The next update will be provided at 15:00 UTC or as events warrant."

I've heard from one contact that a lightning strike in the San Antonio, Texas, area may have caused the cooling system to go down in Microsoft's datacenter there.

Update (September 4, 11 a.m. ET): The Office 365 service health status page was updated, noting that Exchange, Power BI, SharePoint, Teams and Intune users all may be affected. The page says "This issue could potentially affect any of your users who are hosted out of the San Antonio data center," but I'm hearing from users outside the U.S. who say they are also affected. Next update from Microsoft is scheduled for 4:30 p.m. UTC (12:30 p.m. ET). </s> Microsoft Azure now supports NVIDIA GPU Cloud for AI, HPC workloads Microsoft and NVIDIA are targeting data scientists, developers and researchers with preconfigured containers with GPU-accelerated software for running their AI and high-performance computing tasks. Read more: https://zd.net/2MBmVD8

A number of U.S.-based customers connecting to Microsoft cloud services, including Office 365, Azure Active Directory and Visual Studio Team Services, are reporting outages this morning, September 4. The issue seems to stem from problems in Microsoft's South Central U.S. datacenter.

Also: Windows 10 how-to: Ed Bott's free tech support guide

The problems also are affecting users in other regions who are using services that authenticate with Azure Active Directory and who are attempting to use the Azure Resource Manager, I'm hearing from users on Twitter.

An Office 365 service health status page notes that some users have been unable to authenticate or connect to Office 365 services. The Office 365 admin center also has been down for these customers since at least 9:09 a.m. UTC (5:09 a.m. ET), according to that page.

Credit: Microsoft's Azure status page

"We've determined that a data center issue caused a subset of the Office 365 service to become degraded. We're connecting some of the affected services to an alternate infrastructure, while remediating the underlying issue within the datacenter," Microsoft's message says.

Also: Microsoft Windows U-turn removes warning about installing Chrome, Firefox CNET

Office 365 doesn't run on Azure, but it does use Azure Active Directory authentication services.

An Azure Support message on Twitter acknowledged the problem in the South Central U.S. and said that "engineers were still working on the resolution of an issue affecting resources" there. Microsoft is "actively monitoring for impact to other regions also," the message said, directing customers to see status.azure.com for further updates. But as a number of Twitter users noted, they were unable to access that page.

Those who can see the Azure status page may see this message (as of 10:25 a.m. ET):

"Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Engineers have isolated an issue with cooling in one part of the data center, which caused a localized spike in temperature, as the preliminary root-cause, which has now been mitigated. Automated data center procedures to ensure data and hardware integrity went into effect when temperatures hit a specified threshold and critical hardware entered a structured power down process. Engineers are now in the process of restoring power to affected devices as part of the ongoing mitigation process. "Some services may also be experiencing intermittent authentication issues due to downstream Azure Active Directory impact, and engineers are separately working on mitigation options for this also. "The next update will be provided at 15:00 UTC or as events warrant."

I've heard from one contact that a lightning strike in the San Antonio, Texas, area may have caused the cooling system to go down in Microsoft's datacenter there.

Update (September 4, 11 a.m. ET): The Office 365 service health status page was updated, noting that Exchange, Power BI, SharePoint, Teams and Intune users all may be affected. The page says "This issue could potentially affect any of your users who are hosted out of the San Antonio data center," but I'm hearing from users outside the U.S. who say they are also affected. Next update from Microsoft is scheduled for 4:30 p.m. UTC (12:30 p.m. ET).

Update (September 4, 1:45 p.m. ET): I'm hearing from a number of users worldwide that their Microsoft cloud services are coming back.

Also: Microsoft 365: A cheat sheet TechRepublic

Here's an update about why some customers not using U.S. Central datacenter may have been affected, from Azure Identity division's Alex Simons (via Twitter):

"Many of you are asking -- here's a quick update. Azure AD Service has been experiencing load issues in North America with several high volume tenant doing massive auth retries. Availability in some NA (North American) tenants has dropped to 70%. Tenants in Europe and Asia are not effected. "Mitigations are underway and capacity has now returned to >90% in North America and should be back to full availability very soon. We are deeply sorry for any negative impact to our customers are experiencing."

(Just to be clear, I've been hearing from Microsoft cloud customers outside the U.S. who have been affected, in spite of what Simons' tweet says.)

Update (September 4, 3:50 p.m. ET): I still am having problems seeing the Azure status page, but this is the latest from that page (forwarded to me by a Microsoft spokesperson), which does confirm the lightning strike and cooling-system impact -- plus the impact on services in other regions:

CUSTOMER IMPACT: Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Additionally, this issue impacts Azure Active Directory authentication, which may impact services in other regions.

PRELIMINARY ROOT CAUSE: A severe weather event, including lightning strikes, occurred near one of the South Central US datacenters. This resulted in a power voltage increase that impacted cooling systems. Automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a structured power down process.

ENGINEERING STATUS: Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices. While some services are starting to see signs of recovery, mitigation efforts are still ongoing.

NEXT UPDATE: The next update will be provided by 20:00 UTC or as events warrant.

Update (September 4, 4:50 pm. ET): The The Azure Support account on Twitter said, that as of 4:15 p.m. ET, engineers had restored power to the South Central U.S. datacenter. (Because of the cooling system problem, "automated datacenter procedures to ensure data and hardware integrity went into effect and critical hardware entered a shutdown process," they said.)

Non-regional services like Azure Active Directory "encountered an operational threshold for processing requests through the South Central US datacenter. Initial attempts to fail over into other datacenters resulted in temporary traffic congestion for those reasons."

Recovery efforts were still ongoing as of this update from Azure Support.

Update (September 5, 4:20 p.m. ET): Some users continued to experience lingering problems from yesterday's datacenter outage.

Some users were getting messages indicating their access to Outlook or Skype were being throttled. Microsoft officials were telling these users that an update was introduced to the components that manage authentication" which caused users to get this message. Microsof officials said they were reverting the update to remediate the problem. (Thanks to Rob Lavin for the info. on that one.)

Previous and related coverage:

Here's how you can still get a free Windows 10 upgrade

Microsoft's much-hyped free upgrade offer for Windows 10 ended in 2016, right? Not exactly. The GWX tool may be gone, but all the other upgrade tools still work. The end result is an apparently valid digital license, and there's no evidence that the free upgrades will end any time soon.

How to install, reinstall, upgrade and activate Windows 10

Here's everything you need to know before you repair, reinstall, or upgrade Windows 10, including details about activation and product keys.

After Windows 10 upgrade, do these seven things immediately

You've just upgraded to the most recent version of Windows 10. Before you get back to work, use this checklist to ensure that your privacy and security settings are correct and that you've cut annoyances to a bare minimum.

How to upgrade from Windows 10 Home to Pro for free

You've got a new PC running Windows 10 Home. You want to upgrade to Windows 10 Pro. Here's how to get that upgrade for free. All you need is a Pro/Ultimate product key from an older version of Windows.

Related stories: </s> A mysterious outage has hit Google Cloud, one of the biggest cloud service providers on the internet, and thousands of sites have gone down as a result, including both Google and non-Google services.

Affected companies include some of the biggest names around, such as Snapchat, Vimeo, Shopify, Discord, Pokemon GO; but also most of Google's own services, like YouTube, Gmail, Google Search, G Suite, Hangouts, Google Drive, Google Docs, Google Nest, and others.

In an extreme case of irony, according to a Google employee, the outage was so severe that it also took down internal tools Google engineers were using to communicate among each other about the outage, making recovery efforts even more difficult.

"We're having what appears to be a serious networking outage," the Google employee said in a HackerNews post.

"It's disrupting everything, including unfortunately the tooling we usually use to communicate across the company about outages."

The networking issue appears to be related to an outage at Level 3, an US-based ISP that provides connectivity and various other services to Google data centers.

In January, a similar Level 3 outage caused authentication services to go down for all Microsoft cloud services, with the most impacted being Office 365, Dynamics 365, and Azure Government Cloud.

Google Cloud Platform services have been down for at least two hours, at the time of writing. Users in North America are being impacted the most, although some European users also reported scarce problems.

This is a developing story. More updates will follow. </s> This story was originally published on ZDNet.

After reports emerged on Wednesday afternoon that Boeing had found itself the latest victim of the WannaCry ransomware attack, the company issued a statement to explain that its cybersecurity operations centre had merely detected a "limited intrusion" of malware that had affected a small number of its systems.

Although ZDNet has reached out to Boeing for comment, a statement issued to sister site CNET downplays the concerns over the company's production impact.

"A number of articles on a malware disruption are overstated and inaccurate," Linda Mills, vice president of Boeing commercial airplanes communications, told CNET.

"Our cybersecurity operations center detected a limited intrusion of malware that affected a small number of systems. Remediations were applied and this is not a production or delivery issue."

Claims Boeing had been struck by the virus that caused global panic when it spread last year were made after the company's commercial airplane chief engineer Mike VanderWel distributed a memo to staff that called for "all hands on deck".

"It is metastasizing rapidly out of North Charleston and I just heard 777 (automated spar assembly tools) may have gone down," the Seattle Times reported VanderWel as writing in the memo. He reportedly added that he was concerned the virus would hit equipment used in functional tests of planes ready to roll out and potentially "spread to airplane software".

The WannaCry ransomware, labelled the biggest challenge of 2017, saw ransomware spread with the help of a leaked NSA exploit and infect over 300,000 PCs at major organisations around the globe.

The United Kingdom's National Health Service was one of the highest-profile victims of the attack, with 47 trusts and foundation trusts affected. The ransomware forced a number of hospitals offline and some took weeks to recover.

The United States and its Five Eyes partners blamed the WannaCry ransomware attack on North Korea in December.
• A massive cyberattack is hitting organisations around the world (ZDNet)
• WannaCry ransomware was the biggest challenge of the year, says cybersecurity centre (ZDNet)
• WannaCry ransomware: Now the US says North Korea was to blame (ZDNet)
• Ransomware attacks: Here's what we need to learn from WannaCry and Petya (TechRepublic) </s> A widespread Google Cloud outage hit Sunday, rendering popular sites and apps like YouTube and Gmail unusable for many users, CNET reports. Google's Cloud Computing Engine and Cloud Networking services began experiencing issues as of 4 p.m. ET, according to the status page on company's website.

According to Down Detector, which logs outages and service interruptions, the outages appear to be primarily clustered along the East Coast of the United States as well as in parts of South America and Europe.

In addition to popular social media sites, Google Cloud is relied on by a variety of large-scale companies, including Weight Watchers, PayPal and Target. Even Apple admitted it uses Google Cloud for its iCloud storage services, The Verge reports.

Social media users have been quick to voice their frustrations. The hashtag #YouTubeDOWN is now among Twitter's trending topics in the U.S.

It was not immediately clear what caused Sunday's issue or when services might be restored, according to CNET. They have reached out to Google for comment. </s> Google services like YouTube, Nest and Gmail, as well as Cloud customers like Snapchat, Shopify, Vimeo and Discord were impacted by the problem, which began around 12:15 Pacific time.

"We are experiencing high levels of network congestion in the eastern USA, affecting multiple services in Google Cloud, G Suite and YouTube. Users may see slow performance or intermittent errors," the company said on its status page at the time.

While the congestion occurred in the US, its impact was felt globally, and was described by network monitoring company ThousandEyes as a "large scale" outage

The problem was resolved as of 4:00pm Pacific time, with Google promising to "conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."

In a statement, the company apologized for the inconvenience and thanked customers for their "patience and continued support." It added: "Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."

The outage which, among other things, meant that Nest users could not control their thermostats, comes after several major disruptions have impacted the largest cloud companies in recent years, highlighting the difficulty of building a resilient service, even with enormous resources.

Just last year saw Google Cloud go down due to a BGP error, Microsoft Azure be knocked out by a lightning strike, and Amazon Web Services be disrupted by a “power event.”
A Microsoft Azure datacenter in Texas goes down due to a cooling system failure. Outages in cloud services were reported worldwide (Zdnet)