Repository: bpg/terraform-provider-proxmox

Issue:
virtio-scsi-single with agent.enabled = true causes indefinite timeout on VM creation
<h2>Description</h2>
<p>When creating a VM with <code>scsi_hardware = "virtio-scsi-single"</code> and <code>agent { enabled = true }</code>, Terraform hangs indefinitely during VM creation, continuously showing "Still creating..." messages. The VM is successfully created in Proxmox and qemu-guest-agent runs correctly, but Terraform never detects the agent and never completes.</p>
<p>Changing only <code>scsi_hardware = "virtio-scsi-pci"</code> (with all other settings identical) resolves the issue completely - Terraform completes successfully in ~2-3 minutes.</p>
<h2>Environment</h2>
<ul>
<li><strong>Provider version</strong>: <code>bpg/proxmox</code> v0.87.0</li>
<li><strong>Terraform version</strong>: 1.x.x (latest)</li>
<li><strong>Proxmox version</strong>: 9.0.11</li>
<li><strong>Proxmox setup</strong>: 3-node cluster</li>
<li><strong>Template</strong>: Ubuntu 24.04 cloud-init image with qemu-guest-agent pre-installed</li>
<li><strong>OS running Terraform</strong>: Linux/macOS/Windows</li>
</ul>
<h2>To Reproduce</h2>
<h3>Minimal Configuration (Reproduces the Bug)</h3>
<pre><code class="language-hcl">terraform {
  required_version = "&gt;= 1.0"
  
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~&gt; 0.87"
    }
  }
}

provider "proxmox" {
  endpoint  = "https://YOUR-PROXMOX-IP:8006"
  api_token = "YOUR_TOKEN_ID=YOUR_TOKEN_SECRET"
  insecure  = true
}

resource "proxmox_virtual_environment_vm" "test_vm" {
  name      = "test-scsi-single-bug"
  node_name = "YOUR_NODE_NAME"
  vm_id     = 999
  
  clone {
    vm_id = 9000  # Your cloud-init template with qemu-guest-agent
    full  = true
  }
  
  cpu {
    cores = 2
    type  = "host"
  }
  
  memory {
    dedicated = 2048
  }
  
  # THIS COMBINATION CAUSES THE HANG
  agent {
    enabled = true
  }
  
  # THIS LINE CAUSES THE ISSUE
  scsi_hardware = "virtio-scsi-single"
  
  disk {
    datastore_id = "local-lvm"
    interface    = "scsi0"
    size         = 20
    discard      = "on"
    iothread     = true
  }
  
  network_device {
    bridge = "vmbr0"
  }
  
  operating_system {
    type = "l26"
  }
  
  initialization {
    datastore_id = "local-lvm"
    
    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
    
    user_account {
      username = "testuser"
      keys     = ["ssh-rsa YOUR_SSH_KEY"]
    }
  }
  
  started = true
}
</code></pre>
<h3>Steps to Reproduce</h3>
<ol>
<li>Create a cloud-init template VM with qemu-guest-agent installed</li>
<li>Save the configuration above to <code>main.tf</code> (update with your details)</li>
<li>Run <code>terraform init</code></li>
<li>Run <code>terraform apply</code></li>
<li>Observe the hang:</li>
</ol>
<pre><code>proxmox_virtual_environment_vm.test_vm: Creating...
proxmox_virtual_environment_vm.test_vm: Still creating... [10s elapsed]
proxmox_virtual_environment_vm.test_vm: Still creating... [20s elapsed]
proxmox_virtual_environment_vm.test_vm: Still creating... [30s elapsed]
proxmox_virtual_environment_vm.test_vm: Still creating... [1m0s elapsed]
proxmox_virtual_environment_vm.test_vm: Still creating... [2m0s elapsed]
proxmox_virtual_environment_vm.test_vm: Still creating... [3m0s elapsed]
... continues indefinitely ...
</code></pre>
<h2>Expected Behavior</h2>
<p>Terraform should complete successfully after the VM boots and qemu-guest-agent becomes available, typically within 2-3 minutes.</p>
<h2>Actual Behavior</h2>
<p>Terraform hangs indefinitely with "Still creating..." messages.</p>
<p><strong>However, the VM is actually working:</strong></p>
<ul>
<li>✅ VM is created and running in Proxmox GUI</li>
<li>✅ VM boots successfully</li>
<li>✅ Can SSH into the VM</li>
<li>✅ qemu-guest-agent is running: <code>systemctl status qemu-guest-agent</code> shows "active (running)"</li>
<li>✅ Agent shows IP addresses in Proxmox GUI Summary tab</li>
<li>❌ But Terraform never detects the agent and never completes</li>
</ul>
<h2>Workaround (Confirmed Working)</h2>
<p>Change the SCSI controller to <code>virtio-scsi-pci</code>:</p>
<pre><code class="language-hcl"># Instead of:
scsi_hardware = "virtio-scsi-single"

# Use:
scsi_hardware = "virtio-scsi-pci"
</code></pre>
<p>With this single change (everything else identical), Terraform completes successfully in ~2-3 minutes.</p>
<h2>Comparison Test Results</h2>
<p>I tested both configurations with identical settings:</p>

Configuration | SCSI Controller | Agent Enabled | Result
-- | -- | -- | --
Config A | virtio-scsi-pci | true | ✅ Works - completes in ~2 min
Config B | virtio-scsi-single | true | ❌ Hangs indefinitely


<p><strong>Only difference</strong>: The <code>scsi_hardware</code> setting.</p>
<h2>Additional Context</h2>
<ul>
<li>The issue is specific to the combination of <code>virtio-scsi-single</code> + <code>agent.enabled = true</code></li>
<li>Using <code>agent { enabled = false }</code> with <code>virtio-scsi-single</code> works (but agent isn't enabled)</li>
<li>Using <code>agent { enabled = true }</code> with <code>virtio-scsi-pci</code> works perfectly</li>
<li>The VM and agent are fully functional - the issue is Terraform's detection mechanism</li>
<li>This may be related to how the provider waits for/detects the agent on different SCSI controllers</li>
</ul>
<h2>Related Issues</h2>
<p>This appears similar to #2091 but specific to the <code>virtio-scsi-single</code> controller type.</p>
<hr>
<p><strong>Note</strong>: Many users default to <code>virtio-scsi-single</code> for better performance with iothread, so this affects a common configuration pattern.</p></body></html>

Discussion:
@bpg: Hey @Ervikth 👋🏼 
I appreciate AI help in creating tickets, but have you did your own investigation? What is in the VM console? Does it boot? Can you try changing the same in the PVE UI, does it work? If you create a new VM from template that already has `virtio-scsi-single`, does it work?

I can't really do all those debugging steps for each ticket people are opening, it is simply too much.  @ervikth: Hi @bpg,

Yes, I use AI for typing. It cuts down on rambling and gets straight to the point.

Here are my findings:

## Investigation Results

**Manual Testing:**

1. ✅ Proxmox UI: Created VM manually with virtio-scsi-single + agent enabled → works perfectly
2. ✅ VM Console: Boots normally, shows login prompt
3. ✅ Agent Status: SSH'd into VM during Terraform hang - `systemctl status qemu-guest-agent` shows "active (running)"
4. ✅ Proxmox GUI: Agent working - shows VM IP address in Summary tab

**Key Finding:**

When I cancel Terraform during the "Still creating..." hang, the VM is already fully functional with the agent working.

## Debug Log Analysis

VM creation completes successfully:
```text
17:39:48: Clone complete ✓
17:39:49: Config updated (scsihw=virtio-scsi-single) ✓  
17:39:51: Disk resize complete ✓
17:39:52: VM started {"status":"running","qmpstatus":"running"} ✓
```

Then Terraform gets stuck polling the agent:
```text
17:39:55: GET /agent/network-get-interfaces
          Response: 500 "QEMU guest agent is not running"
17:40:00: Retry → 500 "QEMU guest agent is not running"  
17:40:05: Retry → 500 "QEMU guest agent is not running"
17:40:10: Retry → 500 "QEMU guest agent is not running"
(continues until Ctrl+C at 17:40:32)
```

## Comparison

**With virtio-scsi-pci:**
- Same template, same config
- Agent detected immediately
- Terraform completes in ~2 minutes

**With virtio-scsi-single:**
- VM works, agent runs
- Proxmox API returns "agent not running" 
- Terraform never completes

## Conclusion

The VM and agent are functional. Proxmox API reports the agent as "not running" specifically with virtio-scsi-single, but the agent IS running (verified in GUI and via SSH). This appears to be a detection/timing issue between the provider and Proxmox API when using this controller type.

Full debug log available if needed.@bpg: @ervikth I can't reproduce the issue. It is probably related to your base image, and it might have some incompatibility with `virtio-scsi-single` or perhaps there are some other dependencies that in case of `virtio-scsi-single` delays status of quemu agent for some whatever reason.
I'll add some extra error handling in the "wait for IP" logic, which could improve behaviour in your use case.

If you see this issue still reproducible on the new provider's version (just wait for the next release a bit), then please open a new ticket, and add more details about your template VM you're cloning, and attach full debug logs (be careful as they will have your credentials / api token in clear text)  

Solution patch:
diff --git a/proxmox/nodes/vms/vms.go b/proxmox/nodes/vms/vms.go
index 28ca25d2..c9cb80d1 100644
--- a/proxmox/nodes/vms/vms.go
+++ b/proxmox/nodes/vms/vms.go
@@ -598,6 +598,31 @@ func (c *Client) UpdateVMAsync(ctx context.Context, d *UpdateRequestBody) (*stri
 	return resBody.Data, nil
 }
 
+// isAgentNotReadyError checks if an HTTP error indicates the agent is not ready yet.
+// This includes HTTP 500 errors with messages like "QEMU guest agent is not running"
+// which can occur with certain SCSI controller types (e.g., virtio-scsi-single).
+func isAgentNotReadyError(err error) bool {
+	var httpError *api.HTTPError
+	if !errors.As(err, &httpError) {
+		return false
+	}
+
+	if httpError.Code == http.StatusBadRequest {
+		return true
+	}
+
+	if httpError.Code == http.StatusInternalServerError {
+		msg := strings.ToLower(httpError.Message)
+
+		return strings.Contains(msg, "qemu guest agent") &&
+			(strings.Contains(msg, "not running") ||
+				strings.Contains(msg, "not available") ||
+				strings.Contains(msg, "not ready"))
+	}
+
+	return false
+}
+
 // WaitForNetworkInterfacesFromVMAgent waits for a virtual machine's QEMU agent to publish the network interfaces.
 func (c *Client) WaitForNetworkInterfacesFromVMAgent(
 	ctx context.Context,
@@ -627,11 +652,8 @@ func (c *Client) WaitForNetworkInterfacesFromVMAgent(
 	data, err := retry.NewWithData[*GetQEMUNetworkInterfacesResponseData](
 		retry.Context(ctxWithTimeout),
 		retry.RetryIf(func(err error) bool {
-			var target *api.HTTPError
-			if errors.As(err, &target) {
-				if target.Code == http.StatusBadRequest {
-					return true
-				}
+			if isAgentNotReadyError(err) {
+				return true
 			}
 
 			return errors.Is(err, errNoIPsYet)
@@ -650,7 +672,7 @@ func (c *Client) WaitForNetworkInterfacesFromVMAgent(
 						return nil, err
 					}
 
-					if httpError.Code == http.StatusBadRequest {
+					if isAgentNotReadyError(err) {
 						return nil, errNoIPsYet
 					}
 				}



Test patch:
diff --git a/fwprovider/test/resource_vm_test.go b/fwprovider/test/resource_vm_test.go
index aadc5e71..9b70e654 100644
--- a/fwprovider/test/resource_vm_test.go
+++ b/fwprovider/test/resource_vm_test.go
@@ -1357,3 +1357,83 @@ func TestAccResourceVMClone(t *testing.T) {
 		})
 	}
 }
+
+func TestAccResourceVMVirtioSCSISingleWithAgent(t *testing.T) {
+	te := InitEnvironment(t)
+
+	resource.Test(t, resource.TestCase{
+		ProtoV6ProviderFactories: te.AccProviders,
+		Steps: []resource.TestStep{
+			{
+				Config: te.RenderConfig(`
+				resource "proxmox_virtual_environment_file" "cloud_config" {
+					content_type = "snippets"
+					datastore_id = "local"
+					node_name = "{{.NodeName}}"
+					source_raw {
+						data = <<-EOF
+						#cloud-config
+						runcmd:
+						  - apt update
+						  - apt install -y qemu-guest-agent
+						  - systemctl enable qemu-guest-agent
+						  - systemctl start qemu-guest-agent
+						EOF
+						file_name = "cloud-config.yaml"
+					}
+				}
+
+				resource "proxmox_virtual_environment_vm" "test_vm_scsi_single" {
+					node_name = "{{.NodeName}}"
+					started   = true
+					agent {
+						enabled = true
+					}
+					cpu {
+						cores = 2
+					}
+					memory {
+						dedicated = 2048
+					}
+					disk {
+						datastore_id = "local-lvm"
+						file_id      = proxmox_virtual_environment_download_file.ubuntu_cloud_image.id
+						interface    = "scsi0"
+						iothread     = true
+						discard      = "on"
+						size         = 20
+					}
+					scsi_hardware = "virtio-scsi-single"
+					initialization {
+						interface = "scsi1"
+						ip_config {
+							ipv4 {
+								address = "dhcp"
+							}
+						}
+						user_data_file_id = proxmox_virtual_environment_file.cloud_config.id
+					}
+					network_device {
+						bridge = "vmbr0"
+					}
+				}
+
+				resource "proxmox_virtual_environment_download_file" "ubuntu_cloud_image" {
+					content_type = "iso"
+					datastore_id = "local"
+					node_name    = "{{.NodeName}}"
+					url = "{{.CloudImagesServer}}/minimal/releases/noble/release/ubuntu-24.04-minimal-cloudimg-amd64.img"
+					overwrite_unmanaged = true
+				}`),
+				Check: resource.ComposeTestCheckFunc(
+					ResourceAttributes("proxmox_virtual_environment_vm.test_vm_scsi_single", map[string]string{
+						"scsi_hardware":             "virtio-scsi-single",
+						"agent.0.enabled":           "true",
+						"ipv4_addresses.#":          "2",
+						"network_interface_names.#": "2",
+					}),
+				),
+			},
+		},
+	})
+}
diff --git a/proxmox/nodes/vms/vms_agent_test.go b/proxmox/nodes/vms/vms_agent_test.go
new file mode 100644
index 00000000..0bf8a4ef
--- /dev/null
+++ b/proxmox/nodes/vms/vms_agent_test.go
@@ -0,0 +1,103 @@
+/*
+ * This Source Code Form is subject to the terms of the Mozilla Public
+ * License, v. 2.0. If a copy of the MPL was not distributed with this
+ * file, You can obtain one at https://mozilla.org/MPL/2.0/.
+ */
+
+package vms
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/bpg/terraform-provider-proxmox/proxmox/api"
+)
+
+func TestIsAgentNotReadyError(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		err     error
+		want    bool
+		comment string
+	}{
+		{
+			name:    "HTTP 400 BadRequest",
+			err:     &api.HTTPError{Code: http.StatusBadRequest, Message: "any message"},
+			want:    true,
+			comment: "HTTP 400 should be treated as agent not ready",
+		},
+		{
+			name:    "HTTP 500 with 'QEMU guest agent is not running'",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "QEMU guest agent is not running"},
+			want:    true,
+			comment: "HTTP 500 with agent not running message should be retried",
+		},
+		{
+			name:    "HTTP 500 with 'qemu guest agent not available'",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "qemu guest agent not available"},
+			want:    true,
+			comment: "HTTP 500 with agent not available message should be retried",
+		},
+		{
+			name:    "HTTP 500 with 'QEMU guest agent is not ready'",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "QEMU guest agent is not ready"},
+			want:    true,
+			comment: "HTTP 500 with agent not ready message should be retried",
+		},
+		{
+			name:    "HTTP 500 with case-insensitive matching",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "Qemu Guest Agent Is Not Running"},
+			want:    true,
+			comment: "Case-insensitive matching should work",
+		},
+		{
+			name:    "HTTP 500 with different error message",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "Internal Server Error"},
+			want:    false,
+			comment: "HTTP 500 without agent-related message should not be retried",
+		},
+		{
+			name:    "HTTP 500 with unrelated message",
+			err:     &api.HTTPError{Code: http.StatusInternalServerError, Message: "Something went wrong"},
+			want:    false,
+			comment: "HTTP 500 with unrelated message should not be retried",
+		},
+		{
+			name:    "HTTP 403 Forbidden",
+			err:     &api.HTTPError{Code: http.StatusForbidden, Message: "Forbidden"},
+			want:    false,
+			comment: "HTTP 403 should not be retried",
+		},
+		{
+			name:    "HTTP 404 Not Found",
+			err:     &api.HTTPError{Code: http.StatusNotFound, Message: "Not Found"},
+			want:    false,
+			comment: "HTTP 404 should not be retried",
+		},
+		{
+			name:    "non-HTTP error",
+			err:     assert.AnError,
+			want:    false,
+			comment: "Non-HTTP errors should not be treated as agent not ready",
+		},
+		{
+			name:    "nil error",
+			err:     nil,
+			want:    false,
+			comment: "Nil error should not be treated as agent not ready",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got := isAgentNotReadyError(tt.err)
+			assert.Equal(t, tt.want, got, tt.comment)
+		})
+	}
+}