Repository: ansible/ansible-modules-extras

Issue:
ARM deployment succeeds, Ansible Throws msrestazure.azure_exceptions.CloudError
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

Azure/azure_rm_deployment.py
##### ANSIBLE VERSION

```
ansible 2.1.1.0
```
##### CONFIGURATION

```
[defaults]
hostfile = ./inventories/development
roles_path = ./playbooks/roles
log_path = ./ansible.log

[ssh_connection]
ssh_args = -F ./ssh.config -o ControlMaster=auto -o ControlPersist=30m
control_path = ~/.ssh/ansible-%%r@%%h:%%p
```
##### OS / ENVIRONMENT

CentOS 7.2, Python 2.7
##### SUMMARY

When I run an ARM deployment of the template below, it seems that the deployment succeeds in azure according to the activity log, but there are some checks done by the module post-deployment, and these fail.

Template:

```
- name: Provision | Azure Virtual Machine
  azure_rm_deployment:
    state: "{{ desiredState }}"
    resource_group_name: "{{ resourceGroup }}"
    deployment_name: "{{ uuid | to_uuid }}"
    deployment_mode: incremental
    location: "{{ location }}"
    parameters:
      vmName:
        value: "{{ vmName }}"
      vmSize:
        value: "{{ vmSize }}"
      infraStorageResourceGroup:
        value: "{{ infraStorageResourceGroup }}"
      vhdsStorageAccountName:
        value: "{{ vhdsStorageAccountName }}"
      vhdsStorageContainerName:
        value: "{{ vhdsStorageContainerName }}"
      metricsStorageAccountName:
        value: "{{ metricsStorageAccountName }}"

      adminUsername:
        value: "{{ adminUsername }}"
      adminSSHPassword:
        value: "{{ adminSSHPassword }}"

      workspaceName:
        value: "{{ omsWorkspaceName }}"

      pubIpRG:
        value: "{{ pubIpRg }}"
      pubIpName:
        value: "{{ pubIpName }}"
      vnetRg:
        value: "{{ vnetRg }}"
      vnetName:
        value: "{{ vnetName }}"
      subnetName:
        value: "{{ subnetName }}"
      subnetNsgName:
        value: "{{ subnetNsgName }}"
      subnetNsgRg:
        value: "{{ subnetNsgRg }}"

    template:
      $schema: "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#"
      contentVersion: "1.0.0.0"
      parameters:
        vmName:
          type: "string"
        vmSize:
          type: "string"
        infraStorageResourceGroup:
          type: "string"
        vmSize:
          type: "string"
        infraStorageResourceGroup:
          type: "string"
        vhdsStorageAccountName:
          type: "string"
        metricsStorageAccountName:
          type: "string"
        vhdsStorageContainerName:
          type: "string"

        workspaceName:
          type: "string"
        adminUsername:
          type: "string"
        adminSSHPassword:
          type: "securestring"
        #adminSSHKey:
        #  type: "securestring"



        pubIpRG:
          type: "string"
        pubIpName:
          type: "string"
        vnetRg:
          type: "string"
        vnetName:
          type: "string"
        subnetName:
          type: "string"
        subnetNsgName:
          type: "string"
        subnetNsgRg:
          type: "string"

      variables:
        imagePublisher:                   "OpenLogic"
        imageOffer:                       "CentOS"
        sku:                              "7.2"
        version:                          "latest"
        sshKeyPath:                       "[concat('/home/',parameters('adminUsername'),'/.ssh/authorized_keys')]"
        #publicIpRef:                      "[resourceId(parameters('pubIpRG'), 'Microsoft.Network/publicIPAddresses', parameters('pubIpName'))]"
        nicName:                          "[concat( 'NIC-', parameters('vmName'), '-', parameters('subnetName'))]"
        vnetID:                           "[resourceId(parameters('vnetRg'), 'Microsoft.Network/virtualNetworks',parameters('vnetName'))]"
        subnetRef:                        "[concat(variables('vnetID'),'/subnets/', parameters('subnetName'))]"
        OSDiskName:                       "[concat(parameters('vmName'), '-osdisk')]"
        apiVersion:                       "2015-06-15"
        networkApiVersion:                "2015-05-01-preview"
        storageApiVersion:                "2016-01-01"
        extensionName:                    "vmDiagnostics"

      resources:
        - type: "Microsoft.Network/networkInterfaces"
          apiVersion: "[variables('apiVersion')]"
          name: "[variables('nicName')]"
          location: "[resourceGroup().location]"
          properties:
            ipConfigurations:
              - name: "ipconfig1"
                properties:
                  privateIPAllocationMethod: "Dynamic"
                  publicIPAddress:
                    id: "[ concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('pubIpRG'), '/providers/Microsoft.Network/publicIPAddresses/', parameters('pubIpName') ) ]"
                  subnet:
                    id: "[ concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('vnetRg'), '/providers/Microsoft.Network/virtualNetworks/', parameters('vnetName'), '/subnets/', parameters('subnetName') ) ]"
            networkSecurityGroup:
              id: "[ concat('/subscriptions/', subscription().subscriptionId, '/resourceGroups/', parameters('subnetNsgRg'), '/providers/Microsoft.Network/networkSecurityGroups/', parameters('subnetNsgName') ) ]"

        - type: "Microsoft.Compute/virtualMachines"
          apiVersion: "[variables('apiVersion')]"
          name: "[parameters('vmName')]"
          location: "[resourceGroup().location]"
          dependsOn:
          #  - "[reference(resourceId('Microsoft.Storage/storageAccounts', parameters('metricsStorageAccountName')), variables('storageApiVersion'))]"
            - "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
          properties:
            hardwareProfile:
              vmSize: "[parameters('vmSize')]"
            osProfile:
              computername: "[parameters('vmName')]"
              adminUsername: "[parameters('adminUsername')]"
              adminPassword: "[parameters('adminSSHPassword')]"
            storageProfile:
              imageReference:
                publisher: "[variables('imagePublisher')]"
                offer: "[variables('imageOffer')]"
                sku: "[variables('sku')]"
                version: "[variables('version')]"
              osDisk:
                name: "osdisk"
                vhd:
                  uri: "[concat('https://',parameters('vhdsStorageAccountName'),'.blob.core.windows.net/',parameters('vhdsStorageContainerName'),'/',variables('OSDiskName'), uniqueString(variables('OSDiskName')),'.vhd')]"
                caching: "ReadWrite"
                createOption: "FromImage"
            networkProfile:
              networkInterfaces:
                - id: "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
            diagnosticsProfile:
              bootDiagnostics:
                enabled: true
                storageUri: "[concat('http://',parameters('metricsStorageAccountName'),'.blob.core.windows.net')]"
          resources:
          - type: "Microsoft.Compute/virtualMachines/extensions"
            name: "[concat(parameters('vmName'), '/', variables('extensionName'))]"
            #apiVersion: "[variables('apiVersion')]"
            apiVersion: "[providers('Microsoft.Compute', 'virtualMachines/extensions').apiVersions[0]]"
            location: "[resourceGroup().location]"
            dependsOn:
              - "[concat('Microsoft.Compute/virtualMachines/', parameters('vmName'))]"
            properties:
              publisher: "Microsoft.EnterpriseCloud.Monitoring"
              type: "OmsAgentForLinux"
              typeHandlerVersion: "1.0"
              autoUpgradeMinorVersion: true
              settings:
                workspaceId: "[reference(resourceId('CENG-PRD-INFR-05','Microsoft.OperationalInsights/workspaces/', parameters('workspaceName')), providers('Microsoft.OperationalInsights', 'workspaces').apiVersions[0]).customerId]"
              protectedSettings:
                workspaceKey: "[listKeys(resourceId('CENG-PRD-INFR-05','Microsoft.OperationalInsights/workspaces/', parameters('workspaceName')), providers('Microsoft.OperationalInsights', 'workspaces').apiVersions[0]).primarySharedKey]"
```
##### STEPS TO REPRODUCE

Run the above task with some input parameters.

<!--- Paste example playbooks or commands between quotes below -->

```
- include: templates/virtual-machine.yaml
  vars:
    desiredState: present
    resourceGroup: "{{ client.virtualMachines[item].rgName }}"
    location: "{{ client.location }}"

    vmName: "{{ client.virtualMachines[item].name }}"
    vmSize: "{{ client.virtualMachines[item].size }}"
    adminUsername: "{{ client.virtualMachines[item].adminUser }}"
    adminSSHPassword:  "{{ client.virtualMachines[item].adminPassword }}"
    #adminSSHKey:  "{{ client.rgDmzVms.jumpServer.publicKey }}"

    infraStorageResourceGroup: "{{ client.rgInfraStorage.name }}"
    vhdsStorageAccountName:  "{{ client.rgInfraStorage.generalPurposeStorageAccounts.vhds.name }}"
    vhdsStorageContainerName:  "{{ client.rgInfraStorage.generalPurposeStorageAccounts.vhds.containerName }}"
    metricsStorageAccountName:  "{{ client.rgInfraStorage.generalPurposeStorageAccounts.metrics.name }}"

    omsWorkspaceName: "{{ client.omsLogAnalytics.prodWorkspace.name }}"

    pubIpRg: "{{ client.rgPublicIp.name }}"
    pubIpName: "{{ client.rgPublicIp.publicIps[client.virtualMachines[item].associatedPublicIP].name }}"
    vnetRg: "{{ client.rgVnet.name }}"
    vnetName: "{{ client.rgVnet.prodVnet.name }}"
    subnetName: "{{ client.rgVnet.prodVnet.dmzSubnet.name }}"
    subnetNsgName: "{{ client.rgVnet.prodVnet.dmzSubnet.nsgName }}"
    subnetNsgRg: "{{ client.rgVnet.rgNsgRules.name }}"
  when: client.virtualMachines[item].associatedPublicIP is defined
  with_items: "{{ client.virtualMachines }}"
```

Parameters:

```
  virtualMachines:
    jumpserver:
      name: CORE-JS-001 # <tttt>-<rrrr>-<nnn>
      rgName: CENG-PRD-INFR-07
      size: Standard_D1_v2
      adminUser: sshadminuser
      adminPassword: "supersecret"
      associatedPublicIP: pubip01
```
##### EXPECTED RESULTS

We should get little to no output showing that the deployment succeeded, since this is the case according to Azure.
##### ACTUAL RESULTS

The Deployment succeeds, but Ansible bombs out with the following message:

```
TASK [provision : Provision | Azure Virtual Machine] ***************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: msrestazure.azure_exceptions.CloudError: The Resource 'Microsoft.Network/publicIPAddresses/VIP-VM-001' under resource group 'CENG-PRD-INFR-07' was not found.
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 660, in <module>\n    main()\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 656, in main\n    AzureRMDeploymentManager()\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 438, in __init__\n    supports_check_mode=False)\n  File \"/tmp/ansible_UOqY2X/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py\", line 192, in __init__\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 455, in exec_module\n    instances=self._get_instances(deployment)\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 605, in _get_instances\n    for vm, nics in vms_and_nics]\n  File \"/tmp/ansible_UOqY2X/ansible_module_azure_rm_deployment.py\", line 652, in _nic_to_public_ips_instance\n    if ip_conf_instance.public_ip_address]]\n  File \"/usr/lib/python2.7/site-packages/azure/mgmt/network/operations/public_ip_addresses_operations.py\", line 179, in get\n    raise exp\nmsrestazure.azure_exceptions.CloudError: The Resource 'Microsoft.Network/publicIPAddresses/VIP-VM-001' under resource group 'CENG-PRD-INFR-07' was not found.\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}
```

Cheers for any help guys!



Discussion:
<!-- @mogthesprog: Thanks for filing this bug report! -->

@ansible ping, this issue is waiting for your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

FYI, I've done some further research, the error isn't thrown by the Azure SDK version 2.0.0.0rc5 when i make a normal deployment. If I update to rc6 the issue gets much worse... there are quite a few breaking changes in rc6, just a heads up. 

Looking at the code in the error above, it seems like ansible performs some checks to ensure that the deployment was successful? if this is the case I find myself asking why? should you not trust the return status and error code of the azure sdk/api? (I'm new to all this so feel free to tell me that this is a dumb thought/question).

Cheers,

Morgan


Resolving patch:
diff --git a/cloud/azure/azure_rm_deployment.py b/cloud/azure/azure_rm_deployment.py
--- a/cloud/azure/azure_rm_deployment.py
+++ b/cloud/azure/azure_rm_deployment.py
@@ -644,7 +644,7 @@ def _get_ip_dict(self, ip):
         return ip_dict
 
     def _nic_to_public_ips_instance(self, nics):
-        return [self.network_client.public_ip_addresses.get(self.resource_group_name, public_ip_id.split('/')[-1])
+        return [self.network_client.public_ip_addresses.get(public_ip_id.split('/')[4], public_ip_id.split('/')[-1])
                   for nic_obj in [self.network_client.network_interfaces.get(self.resource_group_name,
                                                                              nic['dep'].resource_name) for nic in nics]
                   for public_ip_id in [ip_conf_instance.public_ip_address.id


Test patch: