Change:
Two weeks ago we were observing Windows nodes crashing and rebooting during our e2e test runs on GCE (). Analyzing the crash dumps showed a fault in the NTFS driver while the process was running. appears to be part of the Windows Defender anti-malware component. To address the crashes we did two things: 1) disabled Windows Defender on our Windows e2e test nodes (), and 2) included February's Windows updates on our Windows nodes. After these two things the crashes seem to have stopped. Two days ago we re-enabled Windows Defender on our Windows nodes: Unfortunately we started seeing the Windows node crashes again, starting with run 228 on (To determine if a node crash occurred we open the results link for the test run, open the artifacts for each Windows node and look for a file (e.g. )). We analyzed one of these crash dump files and it looked the same as what we saw a couple weeks ago: fault in NTFS while was running. For now we've merged we re-disables Windows Defender on our e2e Windows nodes. Since that PR merged early this morning (starting with run 246) none of our Windows nodes have crashed, strongly implicating Windows Defender as contributing to the crashes. Since the failure is happening in the NTFS driver we'll need help from Microsoft folks to debug it. The memory dump files are publicly available from the test artifacts (runs 228 - 245), and I've copied a WinDbg crash analysis below. Additionally, some Windows version information is available in the serial-1.log file for each Windows node, also copied below. I don't have any great ideas about why the combination of K8s e2e tests and Windows Defender seems to cause the crash; we don't have any other reports of this failure happening in Windows VMs on GCE. Version info: Analysis:
/assign pjh cc is there anyone on your side who can look at this? It's fairly easily reproducible in our e2e test environment. We're working around it for now by disabling Windows Defender; I think we're ok with this for now, but it would be good to find the cause.
/sig windows Not blocking for 1.14.
/cc /cc in case you've seen the same crash
did do some deeper analysis of one of the crash dumps early last week and found this which might be helpful:
cc as well
I fwd'ed the dmp internally
Thanks! also passed this along internally. I'll leave this issue assigned to me while awaiting updates from Microsoft.
The folks at Microsoft root-caused this issue and we've verified a fix for it. The fix will make it into Windows Server 2019 at some point, not sure about the timing for SAC releases.
/assign craiglpeters For tracking.
I believe this issue should be fixed via KB4501371, can you verify once it becomes available?
/assign pjh Yes, I'll verify the fix when that KB is available (end of June).
There's an optional fix out: It will be included in the July cumulative update too

Discussion:
Dump-DebugInfoToConsole Set-PrerequisiteOptions $kube_env = Fetch-KubeEnv <del> Disable-WindowsDefender </del> if (Test-IsTestCluster $kube_env) { Log-Output 'Test cluster detected, installing OpenSSH.'

Install-Module -Name powershell-yaml -Force } <del> # Disables Windows Defender realtime scanning. # TODO: remove this workaround once the fix is rolled out the Windows image # https://github.com/kubernetes/kubernetes/issues/75148 function Disable-WindowsDefender { # Windows Defender periodically consumes 100% of the CPU, so disable realtime # scanning. Uninstalling the Windows Feature will prevent the service from # starting after a reboot. # TODO(pjh): move this step to image preparation, since we don't want to do a # full reboot here. if ((Get-WindowsFeature -Name 'Windows-Defender').Installed) { Log-Output "Disabling Windows Defender service" Set-MpPreference -DisableRealtimeMonitoring $true Uninstall-WindowsFeature -Name 'Windows-Defender' } } </del> # Creates directories where other functions in this module will read and write # data. # Note: C:tmp is required for running certain kubernetes tests.