Repository: cockroachdb/cockroach
Pull request #144390: roachprod/roachtest: add detection for live VM migrations on Azure
Author: login: DarrylWong
Labels: name: v25.3.0-prerelease
Created: 2025-04-14T15:37:09Z | Merged: 2025-04-16T18:55:04Z

This change makes it so if a roachtest fails, we now check for a live migration on any of the VMs similar to how we check for preemptions and host errors. If one is found, the failure will be marked as a infra flake and routed to test-eng.

Fixes: #143397


Diff:
diff --git a/DEPS.bzl b/DEPS.bzl
index 3aa9dbda1e0b..bd8bb60678cd 100644
--- a/DEPS.bzl
+++ b/DEPS.bzl
@@ -927,6 +927,26 @@ def go_deps():
             "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_internal-v1.0.0.zip",
         ],
     )
+    go_repository(
+        name = "com_github_azure_azure_sdk_for_go_sdk_resourcemanager_internal_v2",
+        build_file_proto_mode = "disable_global",
+        importpath = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2",
+        sha256 = "ca118b980624e02a37bf1b312771344e665dee30b8e9dd64aba7cb19201dd348",
+        strip_prefix = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2@v2.0.0",
+        urls = [
+            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_internal_v2-v2.0.0.zip",
+        ],
+    )
+    go_repository(
+        name = "com_github_azure_azure_sdk_for_go_sdk_resourcemanager_monitor_armmonitor",
+        build_file_proto_mode = "disable_global",
+        importpath = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor",
+        sha256 = "97ad4d0a82444b35ce027cf64e2fe1f97e4fc6424165a13892cfe51bf599cda3",
+        strip_prefix = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor@v0.11.0",
+        urls = [
+            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_monitor_armmonitor-v0.11.0.zip",
+        ],
+    )
     go_repository(
         name = "com_github_azure_azure_sdk_for_go_sdk_resourcemanager_network_armnetwork",
         build_file_proto_mode = "disable_global",
@@ -941,10 +961,10 @@ def go_deps():
         name = "com_github_azure_azure_sdk_for_go_sdk_resourcemanager_resources_armresources",
         build_file_proto_mode = "disable_global",
         importpath = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources",
-        sha256 = "09d235afd45048829677351d042fba2c57754df4ae4dde8b25168e39e903db07",
-        strip_prefix = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.0.0",
+        sha256 = "859deb478c6150fcc8ab966b85be386ca0d0e0c385ace0bc5b3c73c14a877bcd",
+        strip_prefix = "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.1.1",
         urls = [
-            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_resources_armresources-v1.0.0.zip",
+            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_resources_armresources-v1.1.1.zip",
         ],
     )
     go_repository(
@@ -2705,10 +2725,10 @@ def go_deps():
         name = "com_github_dnaeon_go_vcr",
         build_file_proto_mode = "disable_global",
         importpath = "github.com/dnaeon/go-vcr",
-        sha256 = "d6d94a1c8471809db30c2979add32bac647120bc577ea30f7e8fcc06436483f0",
-        strip_prefix = "github.com/dnaeon/go-vcr@v1.1.0",
+        sha256 = "6d34b7e17c158d51ffc34f6ac64df05ab736b2ae50c0db07be4a9556dac10c52",
+        strip_prefix = "github.com/dnaeon/go-vcr@v1.2.0",
         urls = [
-            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dnaeon/go-vcr/com_github_dnaeon_go_vcr-v1.1.0.zip",
+            "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dnaeon/go-vcr/com_github_dnaeon_go_vcr-v1.2.0.zip",
         ],
     )
     go_repository(
diff --git a/build/bazelutil/distdir_files.bzl b/build/bazelutil/distdir_files.bzl
index b4e6028f4233..2bdbeece94a1 100644
--- a/build/bazelutil/distdir_files.bzl
+++ b/build/bazelutil/distdir_files.bzl
@@ -149,8 +149,10 @@ DISTDIR_FILES = {
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/keyvault/internal/com_github_azure_azure_sdk_for_go_sdk_keyvault_internal-v0.7.0.zip": "a3a79250f250d01abd0b402649ce9baf7eeebbbad186dc602eb011692fdbec24",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_compute_armcompute-v1.0.0.zip": "7f0b10080e81a23d259a4449509485c58d862c4ff4757f7c2a234bbf6e9ac6c4",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_internal-v1.0.0.zip": "e5a50bbc42b4be222b7bafd509316a14388ccc190947545df1abfbdd3727e54c",
+    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_internal_v2-v2.0.0.zip": "ca118b980624e02a37bf1b312771344e665dee30b8e9dd64aba7cb19201dd348",
+    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_monitor_armmonitor-v0.11.0.zip": "97ad4d0a82444b35ce027cf64e2fe1f97e4fc6424165a13892cfe51bf599cda3",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_network_armnetwork-v1.0.0.zip": "e09bacbe7fe4532f9887151a51e092ac89a143034da0fb1729126422f9e1212b",
-    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_resources_armresources-v1.0.0.zip": "09d235afd45048829677351d042fba2c57754df4ae4dde8b25168e39e903db07",
+    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources/com_github_azure_azure_sdk_for_go_sdk_resourcemanager_resources_armresources-v1.1.1.zip": "859deb478c6150fcc8ab966b85be386ca0d0e0c385ace0bc5b3c73c14a877bcd",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/com_github_azure_azure_sdk_for_go_sdk_storage_azblob-v1.0.0.zip": "9bb69aea32f1d59711701f9562d66432c9c0374205e5009d1d1a62f03fb4fdad",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/azure-storage-blob-go/com_github_azure_azure_storage_blob_go-v0.8.0.zip": "3b02b720c25bbb6cdaf77f45a29a21e374e087081dedfeac2700aed6147b4b35",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/Azure/go-ansiterm/com_github_azure_go_ansiterm-v0.0.0-20210617225240-d185dfc1b5a1.zip": "631ff4b167a4360e10911e475933ecb3bd327c58974c17877d0d4cf6fbef6c96",
@@ -441,7 +443,7 @@ DISTDIR_FILES = {
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dimchansky/utfbom/com_github_dimchansky_utfbom-v1.1.1.zip": "0c1a11101602d5f57ac3e790c0b72e09ff87d8d535535f43fbee9e6a42327350",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dimfeld/httptreemux/com_github_dimfeld_httptreemux-v5.0.1+incompatible.zip": "031da29a128234db595fdce84301cfe5ff13b4be03c1e344cfe7daadb68559e9",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/djherbis/atime/com_github_djherbis_atime-v1.1.0.zip": "195cebcceb6d76328f5e5d373154b5c46a6a9bf6b27a88f9c0158276a07c7c41",
-    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dnaeon/go-vcr/com_github_dnaeon_go_vcr-v1.1.0.zip": "d6d94a1c8471809db30c2979add32bac647120bc577ea30f7e8fcc06436483f0",
+    "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/dnaeon/go-vcr/com_github_dnaeon_go_vcr-v1.2.0.zip": "6d34b7e17c158d51ffc34f6ac64df05ab736b2ae50c0db07be4a9556dac10c52",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/docker/cli/com_github_docker_cli-v20.10.17+incompatible.zip": "595cf0e156e2d8afa7e736cbbed7f307685d103fc6109ea75f9ca6ebcafb767d",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/docker/distribution/com_github_docker_distribution-v2.7.1+incompatible.zip": "be78bc43d74873b67afe05a6b244490088680dab75bdfaf26d0fd4d054595bc7",
     "https://storage.googleapis.com/cockroach-godeps/gomod/github.com/docker/go-connections/com_github_docker_go_connections-v0.4.0.zip": "570ebcee7e6fd844e00c89eeab2b1922081d6969df76078dfe4ffacd3db56ada",
diff --git a/go.mod b/go.mod
index d590582341b5..43736521754b 100644
--- a/go.mod
+++ b/go.mod
@@ -92,6 +92,7 @@ require (
 	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0
 	github.com/Azure/azure-sdk-for-go/sdk/keyvault/azkeys v0.9.0
 	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute v1.0.0
+	github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.11.0
 	github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0
 	github.com/Azure/go-autorest/autorest/adal v0.9.15
 	github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c
diff --git a/go.sum b/go.sum
index 3e0707ea2161..a80487c1c6fe 100644
--- a/go.sum
+++ b/go.sum
@@ -111,10 +111,14 @@ github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute v1.0.0
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute v1.0.0/go.mod h1:gM3K25LQlsET3QR+4V74zxCsFAy0r6xMNN9n80SZn+4=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal v1.0.0 h1:lMW1lD/17LUA5z1XTURo7LcVG2ICBPlyMHjIUrcFZNQ=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal v1.0.0/go.mod h1:ceIuwmxDWptoW3eCqSXlnPsZFKh4X+R38dWPv7GS9Vs=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2 v2.0.0 h1:PTFGRSlMKCQelWwxUyYVEUqseBJVemLyqWJjvMyt0do=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2 v2.0.0/go.mod h1:LRr2FzBTQlONPPa5HREE5+RjSCTXl7BwOvYOaWTqCaI=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.11.0 h1:Ds0KRF8ggpEGg4Vo42oX1cIt/IfOhHWJBikksZbVxeg=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor v0.11.0/go.mod h1:jj6P8ybImR+5topJ+eH6fgcemSFBmU6/6bFF8KkwuDI=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.0.0 h1:nBy98uKOIfun5z6wx6jwWLrULcM0+cjBalBFZlEZ7CA=
 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/network/armnetwork v1.0.0/go.mod h1:243D9iHbcQXoFUtgHJwL7gl2zx1aDuDMjvBZVGr2uW0=
-github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.0.0 h1:ECsQtyERDVz3NP3kvDOTLvbQhqWp/x9EsGKtb4ogUr8=
-github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.0.0/go.mod h1:s1tW/At+xHqjNFvWU4G0c0Qv33KOhvbGNj0RCTQDV8s=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.1.1 h1:7CBQ+Ei8SP2c6ydQTGCCrS35bDxgTMfoP2miAwK++OU=
+github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.1.1/go.mod h1:c/wcGeGx5FUPbM/JltUYHZcKmigwyVLJlDq+4HdtXaw=
 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 h1:u/LLAOFgsMv7HmNL4Qufg58y+qElGOt5qv0z1mURkRY=
 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0/go.mod h1:2e8rMJtl2+2j+HXbTBwnyGpm5Nou7KhvSfxOq8JpTag=
 github.com/Azure/azure-storage-blob-go v0.8.0/go.mod h1:lPI3aLPpuLTeUwh1sViKXFxwl2B6teiRqI0deQUvsw0=
diff --git a/pkg/cmd/roachtest/cluster.go b/pkg/cmd/roachtest/cluster.go
index 6e4363ec78f6..07c88c0a5a13 100644
--- a/pkg/cmd/roachtest/cluster.go
+++ b/pkg/cmd/roachtest/cluster.go
@@ -1104,7 +1104,6 @@ func attachToExistingCluster(
 	if err := c.validate(clusterSpec, l); err != nil {
 		return nil, err
 	}
-	fmt.Printf("cloud: %s\n", c.cloud)
 	// Assert cloud was set.
 	if c.cloud == spec.AnyCloud {
 		return nil, errors.New("unable to validate cloud provider")
@@ -3352,6 +3351,33 @@ func (c *clusterImpl) GetHostErrorVMs(ctx context.Context, l *logger.Logger) ([]
 	return allHostErrorVMs, nil
 }
 
+func (c *clusterImpl) GetLiveMigrationVMs(l *logger.Logger) ([]string, error) {
+	if c.IsLocal() {
+		return nil, nil
+	}
+	cachedCluster, err := getCachedCluster(c.name)
+	if err != nil {
+		return nil, err
+	}
+
+	var liveMigrationVMs struct {
+		syncutil.Mutex
+		names []string
+	}
+	clusterErr := vm.FanOut(cachedCluster.VMs, func(p vm.Provider, vms vm.List) error {
+		names, err := p.GetLiveMigrationVMs(l, vms, cachedCluster.CreatedAt)
+		if err != nil {
+			return err
+		}
+		liveMigrationVMs.Lock()
+		defer liveMigrationVMs.Unlock()
+		liveMigrationVMs.names = append(liveMigrationVMs.names, names...)
+		return nil
+	})
+
+	return liveMigrationVMs.names, clusterErr
+}
+
 // RegisterClusterHook registers a hook to be run at a certain point as defined
 // by option.ClusterHookType. This exposes a way for test writers to run code
 // in between certain steps normally orchestrated by the framework, e.g. running
diff --git a/pkg/cmd/roachtest/github_test.go b/pkg/cmd/roachtest/github_test.go
index 285b5b223653..858aa92442de 100644
--- a/pkg/cmd/roachtest/github_test.go
+++ b/pkg/cmd/roachtest/github_test.go
@@ -214,6 +214,8 @@ func TestCreatePostRequest(t *testing.T) {
 							refError = vmPreemptionError("my_VM")
 						case "vm-host-error":
 							refError = vmHostError("my_VM")
+						case "live-migration-error":
+							refError = liveMigrationError("my_VM")
 						case "error-with-owner-sql-foundations":
 							refError = registry.ErrorWithOwner(registry.OwnerSQLFoundations, refError)
 						case "error-with-owner-test-eng":
diff --git a/pkg/cmd/roachtest/test_runner.go b/pkg/cmd/roachtest/test_runner.go
index 11fc56c30094..acddbf915429 100644
--- a/pkg/cmd/roachtest/test_runner.go
+++ b/pkg/cmd/roachtest/test_runner.go
@@ -109,6 +109,16 @@ var (
 		)
 	}
 
+	// liveMigrationError indicates that a test failed and also experienced
+	// a live migration. These errors are directed to Test Eng instead of owning teams.
+	liveMigrationError = func(liveMigrationVMs string) error {
+		return registry.ErrorWithOwner(
+			registry.OwnerTestEng, fmt.Errorf("liveMigrationError VMs: %s", liveMigrationVMs),
+			registry.WithTitleOverride("live_migration_error"),
+			registry.InfraFlake,
+		)
+	}
+
 	prng, _ = randutil.NewLockedPseudoRand()
 
 	runID string
@@ -1219,6 +1229,12 @@ func (r *testRunner) runTest(
 					t.resetFailures()
 					t.Error(vmHostError(hostErrorVMNames))
 				}
+				liveMigrationVMNames := getLiveMigrationVMNames(c, l)
+				if liveMigrationVMNames != "" {
+					failureMsg = fmt.Sprintf("VMs had live migrations during the test run: %s\n\n**Other Failures:**\n%s", liveMigrationVMNames, failureMsg)
+					t.resetFailures()
+					t.Error(liveMigrationError(hostErrorVMNames))
+				}
 
 				output := fmt.Sprintf("%s\ntest artifacts and logs in: %s", failureMsg, t.ArtifactsDir())
 				params := getTestParameters(t, github.cluster, github.vmCreateOpts)
@@ -1501,6 +1517,18 @@ func getHostErrorVMNames(ctx context.Context, c *clusterImpl, l *logger.Logger)
 	return getVMNames(hostErrorVMs)
 }
 
+// getLiveMigrationVMNames returns a comma separated list of VMs that
+// experienced a live migration over the duration of the test.
+func getLiveMigrationVMNames(c *clusterImpl, l *logger.Logger) string {
+	liveMigrationVMs, err := c.GetLiveMigrationVMs(l)
+	if err != nil {
+		l.Printf("failed to check live migrations:\n%+v", err)
+		return ""
+	}
+
+	return strings.Join(liveMigrationVMs, ", ")
+}
+
 // The assertions here are executed after each test, and may result in a test failure. Test authors
 // may opt out of these assertions by setting the relevant `SkipPostValidations` flag in the test spec.
 // An error caused by a timeout will not result in a failure.
diff --git a/pkg/cmd/roachtest/testdata/github/live_migration_error b/pkg/cmd/roachtest/testdata/github/live_migration_error
new file mode 100644
index 000000000000..b376e7403534
--- /dev/null
+++ b/pkg/cmd/roachtest/testdata/github/live_migration_error
@@ -0,0 +1,58 @@
+# Verify live migration failures are routed to test-eng and marked as infra-flake.
+
+add-failure name=(oops) type=(live-migration-error)
+----
+ok
+
+post
+----
+----
+roachtest.live_migration_error [failed]() on test_branch @ [test_SHA](). A Side-Eye cluster snapshot was captured on timeout: [https://app.side-eye.io/snapshots/1](https://app.side-eye.io/snapshots/1).
+
+
+```
+test github_test failed: liveMigrationError VMs: my_VM [owner=test-eng]
+```
+
+Parameters:
+ - <code>arch=amd64</code>
+ - <code>cloud=gce</code>
+ - <code>coverageBuild=false</code>
+ - <code>cpu=4</code>
+ - <code>encrypted=false</code>
+ - <code>fs=ext4</code>
+ - <code>localSSD=true</code>
+ - <code>runtimeAssertionsBuild=false</code>
+ - <code>ssd=0</code>
+<details><summary>Help</summary>
+<p>
+
+
+See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)
+
+
+
+See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
+
+
+
+See: [Grafana](https://go.crdb.dev/roachtest-grafana//github-test/1689957243000/1689957853000)
+
+</p>
+</details>
+/cc @cockroachdb/test-eng
+<sub>
+
+[This test on roachdash](https://roachdash.crdb.dev/?filter=status:open%20t:.*live_migration_error.*&sort=title+created&display=lastcommented+project) | [Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)
+
+</sub>
+
+------
+Labels:
+- <code>O-roachtest</code>
+- <code>X-infra-flake</code>
+- <code>T-testeng</code>
+Rendered:
+https://github.com/cockroachdb/cockroach/issues/new?body=roachtest.live_migration_error+%5Bfailed%5D%28%29+on+test_branch+%40+%5Btest_SHA%5D%28%29.+A+Side-Eye+cluster+snapshot+was+captured+on+timeout%3A+%5Bhttps%3A%2F%2Fapp.side-eye.io%2Fsnapshots%2F1%5D%28https%3A%2F%2Fapp.side-eye.io%2Fsnapshots%2F1%29.%0A%0A%0A%60%60%60%0Atest+github_test+failed%3A+liveMigrationError+VMs%3A+my_VM+%5Bowner%3Dtest-eng%5D%0A%60%60%60%0A%0AParameters%3A%0A+-+%3Ccode%3Earch%3Damd64%3C%2Fcode%3E%0A+-+%3Ccode%3Ecloud%3Dgce%3C%2Fcode%3E%0A+-+%3Ccode%3EcoverageBuild%3Dfalse%3C%2Fcode%3E%0A+-+%3Ccode%3Ecpu%3D4%3C%2Fcode%3E%0A+-+%3Ccode%3Eencrypted%3Dfalse%3C%2Fcode%3E%0A+-+%3Ccode%3Efs%3Dext4%3C%2Fcode%3E%0A+-+%3Ccode%3ElocalSSD%3Dtrue%3C%2Fcode%3E%0A+-+%3Ccode%3EruntimeAssertionsBuild%3Dfalse%3C%2Fcode%3E%0A+-+%3Ccode%3Essd%3D0%3C%2Fcode%3E%0A%3Cdetails%3E%3Csummary%3EHelp%3C%2Fsummary%3E%0A%3Cp%3E%0A%0A%0ASee%3A+%5Broachtest+README%5D%28https%3A%2F%2Fgithub.com%2Fcockroachdb%2Fcockroach%2Fblob%2Fmaster%2Fpkg%2Fcmd%2Froachtest%2FREADME.md%29%0A%0A%0A%0ASee%3A+%5BHow+To+Investigate+%5C%28internal%5C%29%5D%28https%3A%2F%2Fcockroachlabs.atlassian.net%2Fl%2Fc%2FSSSBr8c7%29%0A%0A%0A%0ASee%3A+%5BGrafana%5D%28https%3A%2F%2Fgo.crdb.dev%2Froachtest-grafana%2F%2Fgithub-test%2F1689957243000%2F1689957853000%29%0A%0A%3C%2Fp%3E%0A%3C%2Fdetails%3E%0A%2Fcc+%40cockroachdb%2Ftest-eng%0A%3Csub%3E%0A%0A%5BThis+test+on+roachdash%5D%28https%3A%2F%2Froachdash.crdb.dev%2F%3Ffilter%3Dstatus%3Aopen%2520t%3A.%2Alive_migration_error.%2A%26sort%3Dtitle%2Bcreated%26display%3Dlastcommented%2Bproject%29+%7C+%5BImprove+this+report%21%5D%28https%3A%2F%2Fgithub.com%2Fcockroachdb%2Fcockroach%2Ftree%2Fmaster%2Fpkg%2Fcmd%2Fbazci%2Fgithubpost%2Fissues%29%0A%0A%3C%2Fsub%3E%0A%0A------%0ALabels%3A%0A-+%3Ccode%3EO-roachtest%3C%2Fcode%3E%0A-+%3Ccode%3EX-infra-flake%3C%2Fcode%3E%0A-+%3Ccode%3ET-testeng%3C%2Fcode%3E%0A&template=none&title=roachtest%3A+live_migration_error+failed
+----
+----
diff --git a/pkg/roachprod/vm/aws/aws.go b/pkg/roachprod/vm/aws/aws.go
index db7e69a31477..553884a1ab64 100644
--- a/pkg/roachprod/vm/aws/aws.go
+++ b/pkg/roachprod/vm/aws/aws.go
@@ -363,6 +363,12 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	return nil, nil
+}
+
 // GetVMSpecs returns a map from VM.Name to a map of VM attributes, provided by AWS
 func (p *Provider) GetVMSpecs(
 	l *logger.Logger, vms vm.List,
diff --git a/pkg/roachprod/vm/azure/BUILD.bazel b/pkg/roachprod/vm/azure/BUILD.bazel
index f3ff8c3daff8..3035f668b834 100644
--- a/pkg/roachprod/vm/azure/BUILD.bazel
+++ b/pkg/roachprod/vm/azure/BUILD.bazel
@@ -24,6 +24,7 @@ go_library(
         "@com_github_azure_azure_sdk_for_go//profiles/latest/resources/mgmt/resources",
         "@com_github_azure_azure_sdk_for_go//profiles/latest/resources/mgmt/subscriptions",
         "@com_github_azure_azure_sdk_for_go_sdk_resourcemanager_compute_armcompute//:armcompute",
+        "@com_github_azure_azure_sdk_for_go_sdk_resourcemanager_monitor_armmonitor//:armmonitor",
         "@com_github_azure_go_autorest_autorest//:autorest",
         "@com_github_azure_go_autorest_autorest_adal//:adal",
         "@com_github_azure_go_autorest_autorest_azure_auth//:auth",
diff --git a/pkg/roachprod/vm/azure/azure.go b/pkg/roachprod/vm/azure/azure.go
index 012c1e23b7ec..2e7603f7588b 100644
--- a/pkg/roachprod/vm/azure/azure.go
+++ b/pkg/roachprod/vm/azure/azure.go
@@ -21,6 +21,7 @@ import (
 	"github.com/Azure/azure-sdk-for-go/profiles/latest/network/mgmt/network"
 	"github.com/Azure/azure-sdk-for-go/profiles/latest/resources/mgmt/resources"
 	"github.com/Azure/azure-sdk-for-go/profiles/latest/resources/mgmt/subscriptions"
+	"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/monitor/armmonitor"
 	"github.com/Azure/go-autorest/autorest"
 	"github.com/Azure/go-autorest/autorest/to"
 	"github.com/cockroachdb/cockroach/pkg/roachprod/config"
@@ -113,6 +114,75 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), p.OperationTimeout)
+	defer cancel()
+	sub, err := p.getSubscription(ctx)
+	if err != nil {
+		return nil, err
+	}
+	// Azure expects this exact format for --start-time.
+	startTime := since.Format("2006-01-02 15:04:05.999999999 -0700")
+
+	// Azure lets us query by either resourceID or resourceGroup. We don't keep track
+	// of either but can more easily reconstruct the latter through availability zones.
+	// Find all unique resourceGroups among the VMs; we will send a query to each.
+	resourceGroups := make(map[string]struct{})
+	for _, vm := range vms {
+		// Trim the trailing z we added to mock an availability zone.
+		zone := vm.Zone[:len(vm.Zone)-1]
+		clusterName, err := vm.ClusterName()
+		if err != nil {
+			return nil, err
+		}
+		resourceGroups[fmt.Sprintf("%s-%s", clusterName, zone)] = struct{}{}
+	}
+
+	var liveMigrationVMs []string
+	for group := range resourceGroups {
+		// Query for live migrations through the az CLI. The azure sdk does support an
+		// `armmonitor.NewActivityLogsClient`, but it does not appear to support JMESPath querying
+		// to filter for only migration events. Instead, examples suggest that such filtering should
+		// be done on the client side, so lets just use the CLI tool.
+		//
+		// One caveat is that the CLI does not directly support parallel calls, so we have to
+		// do our querying sequentially. This does not appear to be an issue so far, as a CLI call
+		// takes roughly < 1 second to return, but is something to reconsider if we see this stalling.
+		cmd := exec.CommandContext(ctx, "az", "monitor", "activity-log", "list",
+			"--start-time", startTime,
+			"--namespace", "Microsoft.Compute",
+			"--subscription", sub,
+			"--resource-group", group,
+			// Max number of events to return after filtering for resource group but _before_
+			// querying, i.e. we can't just set it to one to the find the latest migration, but we
+			// can set it to an arbitrarily high number to exhaustively search all events since
+			// one resource group isn't expected to have that many events.
+			"--max-events", "1000000",
+			"--query", "[?contains(to_string(properties.title), 'Migration')]",
+		)
+		res, err := cmd.Output()
+		if err != nil {
+			return nil, err
+		}
+		var resJson []*armmonitor.EventData
+		if err := json.Unmarshal(res, &resJson); err != nil {
+			return nil, err
+		}
+		for _, event := range resJson {
+			// The activity log does not have a vm name field so we have to parse it out from the ResourceID
+			_, vmName, found := strings.Cut(*event.ResourceID, "VIRTUALMACHINES/")
+			if !found {
+				l.Printf("GetLiveMigrationVMs: could not parse VM name from resource ID %s", *event.ResourceID)
+				vmName = *event.ResourceID
+			}
+			liveMigrationVMs = append(liveMigrationVMs, vmName)
+		}
+	}
+	return liveMigrationVMs, nil
+}
+
 func (p *Provider) GetVMSpecs(
 	l *logger.Logger, vms vm.List,
 ) (map[string]map[string]interface{}, error) {
diff --git a/pkg/roachprod/vm/flagstub/flagstub.go b/pkg/roachprod/vm/flagstub/flagstub.go
index 0b9588cfeb56..530880add9fb 100644
--- a/pkg/roachprod/vm/flagstub/flagstub.go
+++ b/pkg/roachprod/vm/flagstub/flagstub.go
@@ -52,6 +52,12 @@ func (p *provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	return nil, nil
+}
+
 func (p *provider) GetVMSpecs(
 	l *logger.Logger, vms vm.List,
 ) (map[string]map[string]interface{}, error) {
diff --git a/pkg/roachprod/vm/gce/gcloud.go b/pkg/roachprod/vm/gce/gcloud.go
index f40c2707bfbf..52cec51bc087 100644
--- a/pkg/roachprod/vm/gce/gcloud.go
+++ b/pkg/roachprod/vm/gce/gcloud.go
@@ -454,6 +454,12 @@ func (p *Provider) GetHostErrorVMs(
 	return hostErrorVMs, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	return nil, nil
+}
+
 // GetVMSpecs returns a map from VM.Name to a map of VM attributes, provided by GCE
 func (p *Provider) GetVMSpecs(
 	l *logger.Logger, vms vm.List,
diff --git a/pkg/roachprod/vm/local/local.go b/pkg/roachprod/vm/local/local.go
index 3647425264b4..7da1961bc55f 100644
--- a/pkg/roachprod/vm/local/local.go
+++ b/pkg/roachprod/vm/local/local.go
@@ -139,6 +139,12 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	return nil, nil
+}
+
 func (p *Provider) GetVMSpecs(
 	l *logger.Logger, vms vm.List,
 ) (map[string]map[string]interface{}, error) {
diff --git a/pkg/roachprod/vm/vm.go b/pkg/roachprod/vm/vm.go
index be2a47c223ac..5aaf34834170 100644
--- a/pkg/roachprod/vm/vm.go
+++ b/pkg/roachprod/vm/vm.go
@@ -519,6 +519,8 @@ type Provider interface {
 	GetPreemptedSpotVMs(l *logger.Logger, vms List, since time.Time) ([]PreemptedVM, error)
 	// GetHostErrorVMs returns a list of VMs that had host error since the time specified.
 	GetHostErrorVMs(l *logger.Logger, vms List, since time.Time) ([]string, error)
+	// GetLiveMigrationVMs checks a list of VMs if a live migration happened since the time specified.
+	GetLiveMigrationVMs(l *logger.Logger, vms List, since time.Time) ([]string, error)
 	// GetVMSpecs returns a map from VM.Name to a map of VM attributes, according to a specific cloud provider.
 	GetVMSpecs(l *logger.Logger, vms List) (map[string]map[string]interface{}, error)
 


Reviews:
user: login: herkolategan
body: Nice work! Out of curiosity I'm guessing there's no real good way to test this other than having it out in the wild. But, if we re-enable Azure + ARM I'm pretty sure we'll see if something is wrong since those machines migrate quite a bit. 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-15T09:40:32Z
user: login: DarrylWong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T17:38:07Z
user: login: herkolategan
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-15T18:48:47Z

Review comments:
diff_hunk: @@ -113,6 +114,71 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), p.OperationTimeout)
+	defer cancel()
+	sub, err := p.getSubscription(ctx)
+	if err != nil {
+		return nil, err
+	}
+	// Azure expects this exact format for --start-time.
+	startTime := since.Format("2006-01-02 15:04:05.999999999 -0700")
+
+	// Azure lets us query by either resourceID or resourceGroup. We don't keep track
+	// of either but can more easily reconstruct the latter through availability zones.
+	// Find all unique resourceGroups among the VMs; we will send a query to each.
+	resourceGroups := make(map[string]struct{})
+	for _, vm := range vms {
+		// Trim the trailing z we added to mock an availability zone.
+		zone := vm.Zone[:len(vm.Zone)-1]
+		clusterName, err := vm.ClusterName()
+		if err != nil {
+			return nil, err
+		}
+		resourceGroups[fmt.Sprintf("%s-%s", clusterName, zone)] = struct{}{}
+	}
+
+	var liveMigrationVMs []string
+	for group := range resourceGroups {
path: pkg/roachprod/vm/azure/azure.go
user: login: herkolategan
body: Perf Nit: Not sure how long this command takes to execute or if the Azure CLI would have an issue with executing commands in parallel, but could maybe do this in parallel to speed up execution a bit.
created_at: 2025-04-15T09:32:23Z
author_association: COLLABORATOR
diff_hunk: @@ -113,6 +114,71 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), p.OperationTimeout)
+	defer cancel()
+	sub, err := p.getSubscription(ctx)
+	if err != nil {
+		return nil, err
+	}
+	// Azure expects this exact format for --start-time.
+	startTime := since.Format("2006-01-02 15:04:05.999999999 -0700")
+
+	// Azure lets us query by either resourceID or resourceGroup. We don't keep track
+	// of either but can more easily reconstruct the latter through availability zones.
+	// Find all unique resourceGroups among the VMs; we will send a query to each.
+	resourceGroups := make(map[string]struct{})
+	for _, vm := range vms {
+		// Trim the trailing z we added to mock an availability zone.
+		zone := vm.Zone[:len(vm.Zone)-1]
+		clusterName, err := vm.ClusterName()
+		if err != nil {
+			return nil, err
+		}
+		resourceGroups[fmt.Sprintf("%s-%s", clusterName, zone)] = struct{}{}
+	}
+
+	var liveMigrationVMs []string
+	for group := range resourceGroups {
path: pkg/roachprod/vm/azure/azure.go
user: login: DarrylWong
body: From: https://learn.microsoft.com/en-us/cli/azure/use-azure-cli-successfully-tips?tabs=bash%2Ccmd2

```
Concurrent execution
If you run Azure CLI commands concurrently on the same machine, write conflicts can happen if multiple Azure CLI commands write to the same Microsoft Authentication Library (MSAL) token cache.

To avoid potential failures, isolate the Azure CLI configuration folder for each script by setting environment variable AZURE_CONFIG_DIR for each script to a separate directory. Azure CLI commands in that script save the configuration and token cache to the configured location instead of the default ~/.azure folder.
```

I'm not sure if that is a concern, but if it is I'm leaning toward not parallelizing since the command takes ~0.5 seconds to run + azure roachtests don't support multi region anyway.
created_at: 2025-04-15T17:38:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -113,6 +114,71 @@ func (p *Provider) GetHostErrorVMs(
 	return nil, nil
 }
 
+func (p *Provider) GetLiveMigrationVMs(
+	l *logger.Logger, vms vm.List, since time.Time,
+) ([]string, error) {
+	ctx, cancel := context.WithTimeout(context.Background(), p.OperationTimeout)
+	defer cancel()
+	sub, err := p.getSubscription(ctx)
+	if err != nil {
+		return nil, err
+	}
+	// Azure expects this exact format for --start-time.
+	startTime := since.Format("2006-01-02 15:04:05.999999999 -0700")
+
+	// Azure lets us query by either resourceID or resourceGroup. We don't keep track
+	// of either but can more easily reconstruct the latter through availability zones.
+	// Find all unique resourceGroups among the VMs; we will send a query to each.
+	resourceGroups := make(map[string]struct{})
+	for _, vm := range vms {
+		// Trim the trailing z we added to mock an availability zone.
+		zone := vm.Zone[:len(vm.Zone)-1]
+		clusterName, err := vm.ClusterName()
+		if err != nil {
+			return nil, err
+		}
+		resourceGroups[fmt.Sprintf("%s-%s", clusterName, zone)] = struct{}{}
+	}
+
+	var liveMigrationVMs []string
+	for group := range resourceGroups {
path: pkg/roachprod/vm/azure/azure.go
user: login: herkolategan
body: Makes sense, yeah considering the above I'd probably also opt to just go with serial execution as well.
created_at: 2025-04-15T18:48:47Z
author_association: COLLABORATOR

PR comments:
user: login: cockroach-teamcity
created_at: 2025-04-14T15:37:20Z
body: 

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/cockroachdb/cockroach/144390)
<!-- Reviewable:end -->

author_association: MEMBER
user: login: DarrylWong
created_at: 2025-04-15T17:40:03Z
body: > I'm guessing there's no real good way to test this other than having it out in the wild.

Yeah, I made sure the parsing worked by temporarily replacing the cli call with the example found in: https://github.com/cockroachdb/cockroach/issues/143397, but no clear way to me how to test this e2e either.
author_association: CONTRIBUTOR
user: login: DarrylWong
created_at: 2025-04-16T14:19:48Z
body: TFTR!

bors r=herkolategan
author_association: CONTRIBUTOR
user: login: DarrylWong
created_at: 2025-04-16T18:11:09Z
body: 🥺 

bors r=herkolategan
author_association: CONTRIBUTOR
user: login: craig[bot]
created_at: 2025-04-16T18:55:03Z
body: Build succeeded:
  * [examples_orms](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675768676)
  * [lint](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675769873)
  * [local_roachtest_fips](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675771774)
  * [docker_image_amd64](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675769317)
  * [linux_amd64_fips_build](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675771152)
  * [linux_amd64_build](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675770525)
  * [unit_tests](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675773131)
  * [acceptance](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675767507)
  * [check_generated_code](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675768201)
  * [local_roachtest](https://github.com/cockroachdb/cockroach/actions/runs/14499566024/job/40675772529)
author_association: CONTRIBUTOR