Repository: ansible/ansible

Issue:
azure_rm_virtualnetwork not handling DNS settings properly on existing virtual networks
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
azure_rm_virtualnetwork

##### ANSIBLE VERSION
```
ansible 2.5.1
  config file = /home/szponek/Documents/xyz/gitrepo/iaas-playbooks/ansible.cfg
  configured module search path = [u'/home/szponek/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/szponek/bin/ansible-venv/local/lib/python2.7/site-packages/ansible
  executable location = /home/szponek/bin/ansible-venv/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
ANSIBLE_NOCOWS(env: ANSIBLE_NOCOWS) = True
DEFAULT_REMOTE_USER(/home/szponek/Documents/xyz/gitrepo/iaas-playbooks/ansible.cfg) = ansible
DEFAULT_VAULT_PASSWORD_FILE(/home/szponek/Documents/xyz/gitrepo/iaas-playbooks/ansible.cfg) = /home/szponek/Documents/xyz/gitrepo/iaas-playbooks/support/ansible_vault_passwd
HOST_KEY_CHECKING(/home/szponek/Documents/xyz/gitrepo/iaas-playbooks/ansible.cfg) = False
RETRY_FILES_ENABLED(/home/szponek/Documents/xyz/gitrepo/iaas-playbooks/ansible.cfg) = False

##### OS / ENVIRONMENT
```
cat /etc/lsb-release 
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=18.1
DISTRIB_CODENAME=serena
DISTRIB_DESCRIPTION="Linux Mint 18.1 Serena"

pip list
Package                      Version  
---------------------------- ---------
adal                         0.5.1    
ansible                      2.5.1    
appdirs                      1.4.3    
applicationinsights          0.11.2   
argcomplete                  1.9.4    
asn1crypto                   0.24.0   
azure-cli-core               2.0.31   
azure-cli-nspkg              3.0.2    
azure-common                 1.1.9    
azure-mgmt-batch             4.1.0    
azure-mgmt-compute           2.1.0    
azure-mgmt-containerinstance 0.4.0    
azure-mgmt-containerregistry 1.0.1    
azure-mgmt-containerservice  2.0.0    
azure-mgmt-dns               1.2.0    
azure-mgmt-keyvault          0.40.0   
azure-mgmt-network           1.7.1    
azure-mgmt-nspkg             2.0.0    
azure-mgmt-rdbms             0.2.0rc1 
azure-mgmt-resource          1.2.2    
azure-mgmt-sql               0.7.1    
azure-mgmt-storage           1.5.0    
azure-mgmt-web               0.32.0   
azure-nspkg                  2.0.0    
azure-storage                0.35.1   
bcrypt                       3.1.4    
certifi                      2018.4.16
cffi                         1.11.5   
chardet                      3.0.4    
colorama                     0.3.9    
configparser                 3.5.0    
cryptography                 2.2.2    
decorator                    4.3.0    
deprecation                  2.0.2    
dnspython                    1.15.0   
dogpile.cache                0.6.5    
entrypoints                  0.2.3    
enum34                       1.1.6    
futures                      3.2.0    
humanfriendly                4.10     
idna                         2.6      
ipaddress                    1.0.22   
iso8601                      0.1.12   
isodate                      0.6.0    
Jinja2                       2.10     
jmespath                     0.9.3    
jsonpatch                    1.23     
jsonpointer                  2.0      
keyring                      12.0.1   
keystoneauth1                3.5.0    
knack                        0.3.2    
MarkupSafe                   1.0      
monotonic                    1.4      
msrest                       0.4.28   
msrestazure                  0.4.27   
munch                        2.3.1    
netifaces                    0.10.6   
oauthlib                     2.0.7    
openstacksdk                 0.12.0   
os-client-config             1.29.0   
os-service-types             1.2.0    
packaging                    17.1     
paramiko                     2.4.1    
pbr                          4.0.2    
pip                          10.0.1   
pkg-resources                0.0.0    
pyasn1                       0.4.2    
pycparser                    2.18     
Pygments                     2.2.0    
PyJWT                        1.6.1    
PyNaCl                       1.2.1    
pyOpenSSL                    17.5.0   
pyparsing                    2.2.0    
python-dateutil              2.7.2    
PyYAML                       3.12     
requests                     2.18.4   
requests-oauthlib            0.8.0    
requestsexceptions           1.4.0    
SecretStorage                2.3.1    
setuptools                   39.0.1   
shade                        1.27.1   
six                          1.11.0   
stevedore                    1.28.0   
tabulate                     0.8.2    
urllib3                      1.22     
wheel                        0.31.0   
zabbix-api                   0.5.3    
```
##### SUMMARY
Can't set or change custom DNS servers on an existing virtual network (with existing subnets and VMs in them)

##### STEPS TO REPRODUCE
On an existing Virtual Network resource in Azure (with subnets and VMs already running in them) specify custom DNS servers to be used instead of default Azure DNS servers.

The resulting error is:
```
TASK [virtual_networks : Manage virtual networks] ********************************************************************************************************************************************************************************************
failed: [localhost] (item={u'dns_servers': [u'10.90.2.10'], u'name': u'replication-prod-vnet', u'resource_group': u'network-prod-rg', u'address_prefixes_cidr': [u'10.90.2.64/26']}) => {"changed": false, "item": {"address_prefixes_cidr": ["10.90.2.64/26"], "dns_servers": ["10.90.2.10"], "name": "replication-prod-vnet", "resource_group": "network-prod-rg"}, "module_stderr": "No handlers could be found for logger \"msrestazure.azure_active_directory\"\nTraceback (most recent call last):\n  File \"/tmp/ansible_A3RaVd/ansible_module_azure_rm_virtualnetwork.py\", line 352, in <module>\n    main()\n  File \"/tmp/ansible_A3RaVd/ansible_module_azure_rm_virtualnetwork.py\", line 349, in main\n    AzureRMVirtualNetwork()\n  File \"/tmp/ansible_A3RaVd/ansible_module_azure_rm_virtualnetwork.py\", line 205, in __init__\n    supports_check_mode=True)\n  File \"/tmp/ansible_A3RaVd/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py\", line 345, in __init__\n  File \"/tmp/ansible_A3RaVd/ansible_module_azure_rm_virtualnetwork.py\", line 264, in exec_module\n    existing_dns_set = set(vnet.dhcp_options.dns_servers)\nAttributeError: 'NoneType' object has no attribute 'dns_servers'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
```
or on an existing Virtual Network resource in Azure (with subnets and VMs already running in them) try to change custom DNS server IP to a different one.

The resulting error is:
```
TASK [virtual_networks : Manage virtual networks] ********************************************************************************************************************************************************************************************
failed: [localhost] (item={u'dns_servers': [u'10.90.2.10'], u'name': u'primary-dr-vnet', u'resource_group': u'network-dr-rg', u'address_prefixes_cidr': [u'10.90.0.0/24', u'10.90.1.0/24', u'10.90.2.0/26', u'10.90.3.160/27']}) => {"changed": false, "item": {"address_prefixes_cidr": ["10.90.0.0/24", "10.90.1.0/24", "10.90.2.0/26", "10.90.3.160/27"], "dns_servers": ["10.90.2.10"], "name": "primary-dr-vnet", "resource_group": "network-dr-rg"}, "msg": "Error creating or updating virtual network primary-dr-vnet - Azure Error: InUseSubnetCannotBeDeleted\nMessage: Subnet GatewaySubnet is in use by /subscriptions/<redacted>/resourceGroups/network-dr-rg/providers/Microsoft.Network/virtualNetworkGateways/primary-dr-vgw1/ipConfigurations/default and cannot be deleted."}
```

##### EXPECTED RESULTS
I expect setting DNS server IPs in existing virtual networks not to fail


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @haroldwongms @nitzmahone @trstringer @xscript @yuwzho @zikalino
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@szponek Thanks for your reporting, for custom DNS, I add it into my backlog, and will fix it later. As for the deleting running virtual network, seems this is not allowed by Azure service
Hi @yuwzho,

I wasn't trying to delete my virtual network, I only wanted to change my custom DNS. Why ansible attempted to remove my subnet (and I suspect if it succeeded it would later try to re-create it) - I don't know - but it doesn't make sense why it would try to remove it.  I changed my custom DNS through Azure portal with no issues and without removing anything.
@yuwzho I test it and meet same problem as szponek said. It cannot change DNS setting( it has existing subnet and VM), But it can change DNS setting when the virtualnetwork has not deploy by subnet and VM. Does this is a problem? We can reproduce by following YML file. Thanks!
``` e.g.
---
# virtualnetwork has been deploy by subnet and VM
- name: For virtual network test
  hosts: localhost
  connection: local
  vars:
    resource_group: myRG
    location: westeurope
    virtualnetwork: virtualnetwork
    virtualnetwork01: virtualnetwork01
  tasks:
  - name: create virtual network01
    azure_rm_virtualnetwork:
      resource_group: "{{ resource_group }}"
      name: "{{ virtualnetwork01 }}"
      address_prefixes: "10.0.0.0/16"
      dns_servers:
          - "127.0.0.1"
  - name: create virtual network01
    azure_rm_virtualnetwork:
      resource_group: "{{ resource_group }}"
      name: "{{ virtualnetwork01 }}"
      address_prefixes: "10.0.0.0/16"
      dns_servers:
          - "128.0.0.1"
  - name: create virtual network
    azure_rm_virtualnetwork:
      resource_group: "{{ resource_group }}"
      name: "{{ virtualnetwork }}"
      address_prefixes: "10.0.0.0/16"
      dns_servers:
          - "128.0.0.1"

Get test result:
PLAY [For virtual network test] *************************************************************

TASK [Gathering Facts] **********************************************************************
ok: [127.0.0.1]

TASK [create virtual network01] *************************************************************
changed: [127.0.0.1]

TASK [create virtual network01] *************************************************************
changed: [127.0.0.1]

TASK [create virtual network] ***************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "module_stderr": "No handlers could be found for logger \"msrestazure.azure_active_directory\"\nTraceback (most recent call last):\n  File \"/tmp/ansible_XZpmFV/ansible_module_azure_rm_virtualnetwork.py\", line 352, in <module>\n    main()\n  File \"/tmp/ansible_XZpmFV/ansible_module_azure_rm_virtualnetwork.py\", line 349, in main\n    AzureRMVirtualNetwork()\n  File \"/tmp/ansible_XZpmFV/ansible_module_azure_rm_virtualnetwork.py\", line 205, in __init__\n    supports_check_mode=True)\n  File \"/tmp/ansible_XZpmFV/ansible_modlib.zip/ansible/module_utils/azure_rm_common.py\", line 345, in __init__\n  File \"/tmp/ansible_XZpmFV/ansible_module_azure_rm_virtualnetwork.py\", line 264, in exec_module\n    existing_dns_set = set(vnet.dhcp_options.dns_servers)\nAttributeError: 'NoneType' object has no attribute 'dns_servers'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
	to retry, use: --limit @/home/fred/task/39648/39648.retry

PLAY RECAP **********************************************************************************
127.0.0.1                  : ok=3    changed=2    unreachable=0    failed=1 





Resolving patch:
diff --git a/lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py b/lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py
--- a/lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py
+++ b/lib/ansible/modules/cloud/azure/azure_rm_virtualnetwork.py
@@ -257,7 +257,7 @@ def exec_module(self, **kwargs):
                     changed = True
 
                 if self.dns_servers:
-                    existing_dns_set = set(vnet.dhcp_options.dns_servers)
+                    existing_dns_set = set(vnet.dhcp_options.dns_servers) if vnet.dhcp_options else set([])
                     requested_dns_set = set(self.dns_servers)
                     if existing_dns_set != requested_dns_set:
                         self.log('CHANGED: replacing DNS servers')


Test patch:
diff --git a/test/integration/targets/azure_rm_virtualnetwork/tasks/main.yml b/test/integration/targets/azure_rm_virtualnetwork/tasks/main.yml
--- a/test/integration/targets/azure_rm_virtualnetwork/tasks/main.yml
+++ b/test/integration/targets/azure_rm_virtualnetwork/tasks/main.yml
@@ -4,6 +4,17 @@
     resource_group: "{{ resource_group }}"
     state: absent
 
+- name: Create virtual network
+  azure_rm_virtualnetwork:
+    name: my_test_network
+    address_prefixes_cidr:
+      - 10.1.0.0/16
+      - 172.100.0.0/16
+    tags:
+      testing: testing
+      delete: on-exit
+    resource_group: "{{ resource_group }}"
+
 - name: Create virtual network
   azure_rm_virtualnetwork:
     name: my_test_network