Repository: ansible/ansible
Pull request: docker_network IPAM test / general docker test cleanup
##### SUMMARY
The test had to be deactivated (in #50477) since it caused a lot of spurious failures; see also [here](https://github.com/ansible/ansible/pull/50354#issuecomment-450489875).

Fixes #50527, fixes #50478.

##### ISSUE TYPE
- Bugfix Pull Request

##### COMPONENT NAME
docker_network


Problem statement:
Unstable integration test docker_network
##### SUMMARY
The integration test `docker_network` is set to unstable as it sometimes fails.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
docker_network

##### ANSIBLE VERSION
```paste below
devel
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### STEPS TO REPRODUCE
`ansible-test integration docker_network -v --allow-unstable --docker ubuntu1604`

##### EXPECTED RESULTS
Tests pass

##### ACTUAL RESULTS
Test fail.

```
12:38 TASK [docker_network : Change subnet, gateway, iprange and auxiliary addresses of network with custom IPAM config] ***
12:38 task path: /root/ansible/test/integration/targets/docker_network/tasks/tests/ipam.yml:79
12:38 <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
12:38 <127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
12:38 <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723 `" && echo ansible-tmp-1546450863.08-79737007503723="` echo /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723 `" ) && sleep 0'
12:38 Using module file /root/ansible/lib/ansible/modules/cloud/docker/docker_network.py
12:38 <127.0.0.1> PUT /root/.ansible/tmp/ansible-local-7286KQz3At/tmphLBq5j TO /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py
12:38 <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/ /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py && sleep 0'
12:38 <127.0.0.1> EXEC /bin/sh -c '/tmp/ansible-test-coverage-f8L0Z_/coverage/injector.py /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py && sleep 0'
12:38 <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/ > /dev/null 2>&1 && sleep 0'
12:38 The full traceback is:
12:38 Traceback (most recent call last):
12:38   File "/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py", line 113, in <module>
12:38     _ansiballz_main()
12:38   File "/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py", line 105, in _ansiballz_main
12:38     invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
12:38   File "/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py", line 48, in invoke_module
12:38     imp.load_module('__main__', mod, module, MOD_DESC)
12:38   File "/tmp/ansible_docker_network_payload_TB7io9/__main__.py", line 618, in <module>
12:38   File "/tmp/ansible_docker_network_payload_TB7io9/__main__.py", line 613, in main
12:38   File "/tmp/ansible_docker_network_payload_TB7io9/__main__.py", line 347, in __init__
12:38   File "/tmp/ansible_docker_network_payload_TB7io9/__main__.py", line 546, in present
12:38   File "/tmp/ansible_docker_network_payload_TB7io9/__main__.py", line 480, in create_network
12:38   File "/usr/local/lib/python2.7/dist-packages/docker/api/network.py", line 152, in create_network
12:38     return self._result(res, json=True)
12:38   File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 252, in _result
12:38     self._raise_for_status(response)
12:38   File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 248, in _raise_for_status
12:38     raise create_api_error_from_http_exception(e)
12:38   File "/usr/local/lib/python2.7/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
12:38     raise cls(e, response=response, explanation=explanation)
12:38 docker.errors.APIError: 500 Server Error: Internal Server Error ("failed to allocate gateway (172.3.28.2): Address already in use")
12:38 
12:38 fatal: [testhost]: FAILED! => {
12:38     "changed": false, 
12:38     "module_stderr": "Traceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1546450863.08-79737007503723/AnsiballZ_docker_network.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_docker_network_payload_TB7io9/__main__.py\", line 618, in <module>\n  File \"/tmp/ansible_docker_network_payload_TB7io9/__main__.py\", line 613, in main\n  File \"/tmp/ansible_docker_network_payload_TB7io9/__main__.py\", line 347, in __init__\n  File \"/tmp/ansible_docker_network_payload_TB7io9/__main__.py\", line 546, in present\n  File \"/tmp/ansible_docker_network_payload_TB7io9/__main__.py\", line 480, in create_network\n  File \"/usr/local/lib/python2.7/dist-packages/docker/api/network.py\", line 152, in create_network\n    return self._result(res, json=True)\n  File \"/usr/local/lib/python2.7/dist-packages/docker/api/client.py\", line 252, in _result\n    self._raise_for_status(response)\n  File \"/usr/local/lib/python2.7/dist-packages/docker/api/client.py\", line 248, in _raise_for_status\n    raise create_api_error_from_http_exception(e)\n  File \"/usr/local/lib/python2.7/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\n    raise cls(e, response=response, explanation=explanation)\ndocker.errors.APIError: 500 Server Error: Internal Server Error (\"failed to allocate gateway (172.3.28.2): Address already in use\")\n", 
12:38     "module_stdout": "", 
12:38     "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
12:38     "rc": 1
12:38 }

```

Hints from issue discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/docker/docker_network.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/docker/docker_network.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @DBendit @akshay196 @chouseknecht @danihodovic @dariko @felixfontein @jwitko @kassiansun @keitwb @olsaki @tbouvet
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Test is set to unstable with https://github.com/ansible/ansible/pull/50526
Please do any discussion in #50499.

Commit message: Basic cleanup of docker daemon.

Patch to review:
diff --git a/test/integration/targets/docker_network/aliases b/test/integration/targets/docker_network/aliases
index 8bff534556ab79..80f0500dfffd11 100644
--- a/test/integration/targets/docker_network/aliases
+++ b/test/integration/targets/docker_network/aliases
@@ -3,4 +3,3 @@ skip/osx
 skip/freebsd
 destructive
 skip/rhel8.0
-unstable
diff --git a/test/integration/targets/setup_docker/tasks/main.yml b/test/integration/targets/setup_docker/tasks/main.yml
index 4f5a30512477fc..f0d1aa43c9c9a6 100644
--- a/test/integration/targets/setup_docker/tasks/main.yml
+++ b/test/integration/targets/setup_docker/tasks/main.yml
@@ -18,7 +18,12 @@
         name: 'docker{{ extra_packages }}'
         extra_args: "-c {{ role_path }}/../../../runner/requirements/constraints.txt"
 
-    # Detect docker API and docker-py versions
+    # Detect docker CLI, API and docker-py versions
+    - name: Check Docker CLI version
+      command: "docker version -f {% raw %}'{{.Client.Version}}'{% endraw %}"
+      register: docker_cli_version_stdout
+      ignore_errors: yes
+
     - name: Check Docker API version
       command: "{{ ansible_python.executable }} -c 'import docker; print(docker.from_env().version()[\"ApiVersion\"])'"
       register: docker_api_version_stdout
@@ -30,8 +35,52 @@
       ignore_errors: yes
 
     - set_fact:
+        docker_cli_version: "{{ (docker_cli_version_stdout.stdout | default('0.0')) or '0.0' }}"
         docker_api_version: "{{ docker_api_version_stdout.stdout or '0.0' }}"
         docker_py_version: "{{ docker_py_version_stdout.stdout or '0.0' }}"
 
     - debug:
-        msg: "Docker API version: {{ docker_api_version }}; docker-py library version: {{ docker_py_version }}"
+        msg: "Docker CLI version: {{ docker_cli_version }}; Docker API version: {{ docker_api_version }}; docker-py library version: {{ docker_py_version }}"
+
+    - block:
+      # Cleanup docker daemon
+      - name: "Remove all ansible-test-* docker containers"
+        shell: 'docker ps --no-trunc --format {% raw %}"{{.Names}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker rm -f'
+        register: docker_containers
+      - name: "Remove all ansible-test-* docker volumes"
+        shell: 'docker volume ls --format {% raw %}"{{.Name}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker volume rm -f'
+        register: docker_volumes
+      - name: "Remove all ansible-test-* docker networks"
+        shell: 'docker network ls --no-trunc --format {% raw %}"{{.Name}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker network rm'
+        register: docker_networks
+      - name: Cleaned docker resources
+        debug: var=docker_resources
+        vars:
+          docker_resources:
+            containers: "{{ docker_containers.stdout_lines }}"
+            volumes: "{{ docker_volumes.stdout_lines }}"
+            networks: "{{ docker_networks.stdout_lines }}"
+
+      # List all existing docker resources
+      - name: List all docker containers
+        command: docker ps --no-trunc -a
+        register: docker_containers
+      - name: List all docker volumes
+        command: docker volume ls
+        register: docker_volumes
+      - name: List all docker networks
+        command: docker network ls --no-trunc
+        register: docker_networks
+      - name: List all docker images
+        command: docker images --no-trunc -a
+        register: docker_images
+      - name: Still existing docker resources
+        debug: var=docker_resources
+        vars:
+          docker_resources:
+            containers: "{{ docker_containers.stdout_lines }}"
+            volumes: "{{ docker_volumes.stdout_lines }}"
+            networks: "{{ docker_networks.stdout_lines }}"
+            images: "{{ docker_images.stdout_lines }}"
+
+      when: docker_cli_version is version('0.0', '>')


Code review comments:
diff_hunk: @@ -30,8 +35,52 @@
       ignore_errors: yes
 
     - set_fact:
+        docker_cli_version: "{{ (docker_cli_version_stdout.stdout | default('0.0')) or '0.0' }}"
         docker_api_version: "{{ docker_api_version_stdout.stdout or '0.0' }}"
         docker_py_version: "{{ docker_py_version_stdout.stdout or '0.0' }}"
 
     - debug:
-        msg: "Docker API version: {{ docker_api_version }}; docker-py library version: {{ docker_py_version }}"
+        msg: "Docker CLI version: {{ docker_cli_version }}; Docker API version: {{ docker_api_version }}; docker-py library version: {{ docker_py_version }}"
+
+    - block:
+      # Cleanup docker daemon
+      - name: "Remove all ansible-test-* docker containers"
+        shell: 'docker ps --no-trunc --format {% raw %}"{{.Names}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker rm -f'
+        register: docker_containers
+      - name: "Remove all ansible-test-* docker volumes"
+        shell: 'docker volume ls --format {% raw %}"{{.Name}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker volume rm -f'
+        register: docker_volumes
+      - name: "Remove all ansible-test-* docker networks"
+        shell: 'docker network ls --no-trunc --format {% raw %}"{{.Name}}"{% endraw %} | grep "^ansible-test-" | xargs -r docker network rm'
+        register: docker_networks
+      - name: Cleaned docker resources
+        debug: var=docker_resources
path: test/integration/targets/setup_docker/tasks/main.yml
text: @user1:
It seems like the standard in these tests are using the yaml notation and not `key=value`, would be good to keep the same standard.

@author:
Fixed.