Repository: docker/docker-py

Issue:
The VolumeDriver was moved from config to HostConfig since Engine API v1.21
**Symptom:** When creating containers on a Swarm (1.2.6) cluster with named volumes using Docker SDK for Python, the named volumes will always be created using *local* driver whether a volume driver is specified or not. 
 
**Cause:** VolumeDriver config generated by current version (2.2.1) of Docker SDK for Python  to create/run a container is ignored/filtered by Swarm manager.

Please check docker/swarm#2669 for detailed description, thank you.


Discussion:
Thank you for the report. Does it work if you force the API  version to `1.20`?
No, forcing API version to 1.20 doesn't make it work. (details later)

Currently, Docker Engine for Linux is not affected by this issue because it handles backward compatibility for VolumeDriver. (https://github.com/docker/docker/blob/master/runconfig/config_unix.go)

That is, when I want to create a container with a named volume and specify a volume driver using Docker SDK for Python:
* it *works*<sup>[1](#myfootnote1)</sup> if DOCKER_HOST points to a Docker Engine 
* it *doesn't work*<sup>[2](#myfootnote2)</sup> if DOCKER_HOST points to a Docker Swarm (standalone) manager 

To illustrate:

* This *works*<sup>[1](#myfootnote1)</sup>:


  ```
                 create container
                with volume-driver
    docker-py  --------------------> Docker Engine   
  ```

* This *doesn't work*<sup>[2](#myfootnote2)</sup> because VolumeDriver got filtered/ignored at the Docker Swarm part:


  ```
                 create container
                with volume-driver
    docker-py  --------------------> Docker Swarm ------> Docker Engine   
  ```

<a name="myfootnote1">1</a>: *works*: a new named volume is created using the specified volume driver.

<a name="myfootnote2">2</a>: *doesn't work*: a new named volume is created using *local* driver.



### Test details of enforcing API version to 1.20

#### Testing with docker-py/2.2.1, swarm/1.2.6
```python
>>> import docker
>>> client = docker.from_env(version='1.20')
/usr/local/lib/python2.7/dist-packages/docker/api/client.py:155: UserWarning: The minimum API version supported is 1.21, but you are using version 1.20. It is recommended you either upgrade Docker Engine or use an older version of Docker SDK for Python.
  'Python.'.format(MINIMUM_DOCKER_API_VERSION, self._version)

>>> client.version()
{
 u'KernelVersion': u'4.4.0-66-generic',
 u'Os': u'linux',
 u'BuildTime': u'`date -u`',
 u'ApiVersion': u'1.22',
 u'Version': u'swarm/1.2.6',
 u'GitCommit': u'`git rev-parse --short HEAD`',
 u'Arch': u'amd64',
 u'GoVersion': u'go1.7.1'
}

>>> container = client.containers.create(image='ubuntu', command='uname', name='ubuntu-test', volume_driver='convoy', volumes={'testvol': {'bind': '/mnt/vol', 'mode': 'rw'}})
```

request dump of `client.containers.create()`:
```http
POST /v1.20/containers/create?name=ubuntu-test HTTP/1.1
Host: localhost:2375
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 314
Content-Type: application/json
User-Agent: docker-sdk-python/2.2.1

{
	"Tty": false,
	"NetworkDisabled": false,
	"Image": "ubuntu",
	"Cmd": [
		"uname"
	],
	"StdinOnce": false,
	"HostConfig": {
		"Binds": [
			"testvol:/mnt/vol:rw"
		],
		"NetworkMode": "default"
	},
	"AttachStdin": false,
	"Volumes": {
		"/mnt/vol": {}
	},
	"VolumeDriver": "convoy",
	"AttachStderr": true,
	"AttachStdout": true,
	"OpenStdin": false
}
```

result:
```
$ docker volume ls
DRIVER              VOLUME NAME
local               testvol
```

#### Testing with docker-py/1.10.6, swarm/1.2.6

```python
>>> import docker
>>> client = docker.from_env(version='1.20')
>>> client.version()
{
 u'KernelVersion': u'4.4.0-66-generic',
 u'Os': u'linux',
 u'BuildTime': u'`date -u`',
 u'ApiVersion': u'1.22',
 u'Version': u'swarm/1.2.6',
 u'GitCommit': u'`git rev-parse --short HEAD`',
 u'Arch': u'amd64',
 u'GoVersion': u'go1.7.1'
}

>>> container = client.create_container(image='ubuntu', command='uname', name='ubuntu-test', volume_driver='convoy', volumes=['/mnt/vol'],host_config=client.create_host_config(binds=['testvol:/mnt/vol']))
```

request dump of `client.create_container()`
```http
POST /v1.20/containers/create?name=ubuntu-test HTTP/1.1
Host: localhost:2375
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 311
Content-Type: application/json
User-Agent: docker-py/1.10.6

{
	"Tty": false,
	"NetworkDisabled": false,
	"Image": "ubuntu",
	"Cmd": [
		"uname"
	],
	"StdinOnce": false,
	"HostConfig": {
		"Binds": [
			"testvol:/mnt/vol"
		],
		"NetworkMode": "default"
	},
	"AttachStdin": false,
	"Volumes": {
		"/mnt/vol": {}
	},
	"VolumeDriver": "convoy",
	"AttachStderr": true,
	"AttachStdout": true,
	"OpenStdin": false
}
```

result:
```
$ docker volume ls
DRIVER              VOLUME NAME
local               testvol
```

### Reference

The `volume_driver` support was added to docker-py in docker/docker-py#628


Resolving patch:
diff --git a/docker/types/containers.py b/docker/types/containers.py
--- a/docker/types/containers.py
+++ b/docker/types/containers.py
@@ -118,7 +118,7 @@ def __init__(self, version, binds=None, port_bindings=None,
                  tmpfs=None, oom_score_adj=None, dns_opt=None, cpu_shares=None,
                  cpuset_cpus=None, userns_mode=None, pids_limit=None,
                  isolation=None, auto_remove=False, storage_opt=None,
-                 init=None, init_path=None):
+                 init=None, init_path=None, volume_driver=None):
 
         if mem_limit is not None:
             self['Memory'] = parse_bytes(mem_limit)
@@ -428,6 +428,11 @@ def __init__(self, version, binds=None, port_bindings=None,
                 raise host_config_version_error('init_path', '1.25')
             self['InitPath'] = init_path
 
+        if volume_driver is not None:
+            if version_lt(version, '1.21'):
+                raise host_config_version_error('volume_driver', '1.21')
+            self['VolumeDriver'] = volume_driver
+
 
 def host_config_type_error(param, param_value, expected):
     error_msg = 'Invalid type for {0} param: expected {1} but found {2}'
@@ -456,43 +461,27 @@ def __init__(
         stop_signal=None, networking_config=None, healthcheck=None,
         stop_timeout=None
     ):
-        if isinstance(command, six.string_types):
-            command = split_command(command)
-
-        if isinstance(entrypoint, six.string_types):
-            entrypoint = split_command(entrypoint)
-
-        if isinstance(environment, dict):
-            environment = format_environment(environment)
-
-        if labels is not None and version_lt(version, '1.18'):
-            raise errors.InvalidVersion(
-                'labels were only introduced in API version 1.18'
-            )
+        if version_gte(version, '1.10'):
+            message = ('{0!r} parameter has no effect on create_container().'
+                       ' It has been moved to host_config')
+            if dns is not None:
+                raise errors.InvalidVersion(message.format('dns'))
+            if volumes_from is not None:
+                raise errors.InvalidVersion(message.format('volumes_from'))
 
-        if cpuset is not None or cpu_shares is not None:
-            if version_gte(version, '1.18'):
+        if version_lt(version, '1.18'):
+            if labels is not None:
+                raise errors.InvalidVersion(
+                    'labels were only introduced in API version 1.18'
+                )
+        else:
+            if cpuset is not None or cpu_shares is not None:
                 warnings.warn(
                     'The cpuset_cpus and cpu_shares options have been moved to'
                     ' host_config in API version 1.18, and will be removed',
                     DeprecationWarning
                 )
 
-        if stop_signal is not None and version_lt(version, '1.21'):
-            raise errors.InvalidVersion(
-                'stop_signal was only introduced in API version 1.21'
-            )
-
-        if stop_timeout is not None and version_lt(version, '1.25'):
-            raise errors.InvalidVersion(
-                'stop_timeout was only introduced in API version 1.25'
-            )
-
-        if healthcheck is not None and version_lt(version, '1.24'):
-            raise errors.InvalidVersion(
-                'Health options were only introduced in API version 1.24'
-            )
-
         if version_lt(version, '1.19'):
             if volume_driver is not None:
                 raise errors.InvalidVersion(
@@ -513,6 +502,38 @@ def __init__(
                     'version 1.19'
                 )
 
+        if version_lt(version, '1.21'):
+            if stop_signal is not None:
+                raise errors.InvalidVersion(
+                    'stop_signal was only introduced in API version 1.21'
+                )
+        else:
+            if volume_driver is not None:
+                warnings.warn(
+                    'The volume_driver option has been moved to'
+                    ' host_config in API version 1.21, and will be removed',
+                    DeprecationWarning
+                )
+
+        if stop_timeout is not None and version_lt(version, '1.25'):
+            raise errors.InvalidVersion(
+                'stop_timeout was only introduced in API version 1.25'
+            )
+
+        if healthcheck is not None and version_lt(version, '1.24'):
+            raise errors.InvalidVersion(
+                'Health options were only introduced in API version 1.24'
+            )
+
+        if isinstance(command, six.string_types):
+            command = split_command(command)
+
+        if isinstance(entrypoint, six.string_types):
+            entrypoint = split_command(entrypoint)
+
+        if isinstance(environment, dict):
+            environment = format_environment(environment)
+
         if isinstance(labels, list):
             labels = dict((lbl, six.text_type('')) for lbl in labels)
 
@@ -566,14 +587,6 @@ def __init__(
                 attach_stdin = True
                 stdin_once = True
 
-        if version_gte(version, '1.10'):
-            message = ('{0!r} parameter has no effect on create_container().'
-                       ' It has been moved to host_config')
-            if dns is not None:
-                raise errors.InvalidVersion(message.format('dns'))
-            if volumes_from is not None:
-                raise errors.InvalidVersion(message.format('volumes_from'))
-
         self.update({
             'Hostname': hostname,
             'Domainname': domainname,


Test patch:
diff --git a/tests/unit/api_container_test.py b/tests/unit/api_container_test.py
--- a/tests/unit/api_container_test.py
+++ b/tests/unit/api_container_test.py
@@ -407,11 +407,8 @@ def test_create_container_with_volumes_from(self):
                          {'Content-Type': 'application/json'})
 
     def test_create_container_empty_volumes_from(self):
-        self.client.create_container('busybox', 'true', volumes_from=[])
-
-        args = fake_request.call_args
-        data = json.loads(args[1]['data'])
-        self.assertTrue('VolumesFrom' not in data)
+        with pytest.raises(docker.errors.InvalidVersion):
+            self.client.create_container('busybox', 'true', volumes_from=[])
 
     def test_create_named_container(self):
         self.client.create_container('busybox', 'true',
@@ -978,11 +975,11 @@ def test_create_container_with_named_volume(self):
         self.client.create_container(
             'busybox', 'true',
             host_config=self.client.create_host_config(
+                volume_driver='foodriver',
                 binds={volume_name: {
                     "bind": mount_dest,
                     "ro": False
                 }}),
-            volume_driver='foodriver',
         )
 
         args = fake_request.call_args
@@ -990,8 +987,8 @@ def test_create_container_with_named_volume(self):
             args[0][1], url_prefix + 'containers/create'
         )
         expected_payload = self.base_create_payload()
-        expected_payload['VolumeDriver'] = 'foodriver'
         expected_payload['HostConfig'] = self.client.create_host_config()
+        expected_payload['HostConfig']['VolumeDriver'] = 'foodriver'
         expected_payload['HostConfig']['Binds'] = ["name:/mnt:rw"]
         self.assertEqual(json.loads(args[1]['data']), expected_payload)
         self.assertEqual(args[1]['headers'],
diff --git a/tests/unit/dockertypes_test.py b/tests/unit/dockertypes_test.py
--- a/tests/unit/dockertypes_test.py
+++ b/tests/unit/dockertypes_test.py
@@ -1,14 +1,15 @@
 # -*- coding: utf-8 -*-
 
 import unittest
+import warnings
 
 import pytest
 
 from docker.constants import DEFAULT_DOCKER_API_VERSION
 from docker.errors import InvalidArgument, InvalidVersion
 from docker.types import (
-    EndpointConfig, HostConfig, IPAMConfig, IPAMPool, LogConfig, Mount,
-    ServiceMode, Ulimit,
+    ContainerConfig, EndpointConfig, HostConfig, IPAMConfig, IPAMPool,
+    LogConfig, Mount, ServiceMode, Ulimit,
 )
 
 try:
@@ -165,6 +166,26 @@ def test_create_host_config_invalid_mem_swappiness(self):
         with pytest.raises(TypeError):
             create_host_config(version='1.24', mem_swappiness='40')
 
+    def test_create_host_config_with_volume_driver(self):
+        with pytest.raises(InvalidVersion):
+            create_host_config(version='1.20', volume_driver='local')
+
+        config = create_host_config(version='1.21', volume_driver='local')
+        assert config.get('VolumeDriver') == 'local'
+
+
+class ContainerConfigTest(unittest.TestCase):
+    def test_create_container_config_volume_driver_warning(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            ContainerConfig(
+                version='1.21', image='scratch', command=None,
+                volume_driver='local'
+            )
+
+        assert len(w) == 1
+        assert 'The volume_driver option has been moved' in str(w[0].message)
+
 
 class UlimitTest(unittest.TestCase):
     def test_create_host_config_dict_ulimit(self):