Repository: iterative/dvc

Issue:
Version 0.23.2 does not work with Azure remote
Hi,

This is my first time writing a bug report, so please excuse me if I miss something. I am using DVC on Windows 10 with an Azure remote. After updating DVC to latest version (0.23.2) today, I tried to use 'dvc pull' which updated the state file version from '2' to '3' and then produced this error:

> [#########                     ] 30% Collecting informationClient-Request-ID=9c458912-0a4a-11e9-b07e-144f8a2a9eb3 Retry policy did not allow for a retry: Server-Timestamp=Fri, 28 Dec 2018 02:45:18 GMT, Server-Request-ID=2808e2e3-a01e-005c-3357-9e6a14000000, HTTP status code=400, Exception=The specifed resource name contains invalid characters. ErrorCode: InvalidResourceName<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters.RequestId:2808e2e3-a01e-005c-3357-9e6a14000000Time:2018-12-28T02:45:18.4898344Z</Message></Error>.
> Error: Failed to push data to the cloud: The specifed resource name contains invalid characters. ErrorCode: InvalidResourceName
> <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters.
> RequestId:2808e2e3-a01e-005c-3357-9e6a14000000
> Time:2018-12-28T02:45:18.4898344Z</Message></Error>

The issue is appearing even with a completely fresh dvc init. After reverting back to version 0.22.0, everything works again.

I suspect some non-ASCII characters are used with the new file version '3' which is causing the issue. Please let me know if any more info from my side can help.



Discussion:
Hi @raikhan !

Could you please run that command with `-v` flag added and show us the log?
Hi @efiop,

Thank you for a quick reply. I created a completely fresh dvc init (version 0.23.2) in a conda env with an empty Azure container as a remote. Here is the output of 'dvc pull -v' after setting up the remote:

> Debug: PRAGMA user_version;
> Debug: fetched: [(3,)]
> Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
> Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
> Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
> Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
> Debug: PRAGMA user_version = 3;
> Preparing to download data from 'azure://ContainerName=dvctest;'
> Preparing to collect status from azure://ContainerName=dvctest;
> [#########                     ] 30% Collecting informationClient-Request-ID=0a480fe4-0a50-11e9-a76f-144f8a2a9eb3 Retry policy did not allow for a retry: Server-Timestamp=Fri, 28 Dec 2018 03:24:10 GMT, Server-Request-ID=b9a22b83-e01e-0072-385c-9eead3000000, HTTP status code=400, Exception=The specifed resource name contains invalid characters. ErrorCode: InvalidResourceName<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters.RequestId:b9a22b83-e01e-0072-385c-9eead3000000Time:2018-12-28T03:24:10.5289140Z</Message></Error>.
> Debug: SELECT count from state_info WHERE rowid=1
> Debug: fetched: [(0,)]
> Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
> Traceback (most recent call last):
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\command\data_sync.py", line 29, in do_run
>     force=self.args.force)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\project.py", line 859, in pull
>     with_deps=with_deps)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\project.py", line 842, in fetch
>     show_checksums=show_checksums)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\data_cloud.py", line 127, in pull
>     show_checksums=show_checksums)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\local.py", line 671, in pull
>     download=True)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\local.py", line 632, in _process
>     show_checksums=show_checksums)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\local.py", line 561, in status
>     remote_exists = list(remote.cache_exists(md5s))
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\base.py", line 228, in cache_exists
>     return list(filter(lambda checksum: checksum in checksums, self.all()))
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\base.py", line 181, in <genexpr>
>     self.path_to_checksum(path)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\azure.py", line 86, in _list_paths
>     blob_service = self.blob_service
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\dvc\remote\azure.py", line 73, in blob_service
>     self.__blob_service.create_container(self.bucket)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\blob\baseblobservice.py", line 633, in create_container
>     _dont_fail_on_exist(ex)
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\common\_error.py", line 92, in _dont_fail_on_exist
>     raise error
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\blob\baseblobservice.py", line 630, in create_container
>     self._perform_request(request, expected_errors=[_CONTAINER_ALREADY_EXISTS_ERROR_CODE])
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\common\storageclient.py", line 381, in _perform_request
>     raise ex
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\common\storageclient.py", line 306, in _perform_request
>     raise ex
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\common\storageclient.py", line 292, in _perform_request
>     HTTPError(response.status, response.message, response.headers, response.body))
>   File "c:\users\raicevim\appdata\local\continuum\anaconda3\lib\site-packages\azure\storage\common\_error.py", line 119, in _http_error_handler
>     raise ex
> azure.common.AzureHttpError: The specifed resource name contains invalid characters. ErrorCode: InvalidResourceName
> <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters.
> RequestId:b9a22b83-e01e-0072-385c-9eead3000000
> Time:2018-12-28T03:24:10.5289140Z</Message></Error>
> 
> Error: Failed to pull data from the cloud: The specifed resource name contains invalid characters. ErrorCode: InvalidResourceName
> <?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidResourceName</Code><Message>The specifed resource name contains invalid characters.
> RequestId:b9a22b83-e01e-0072-385c-9eead3000000
> Time:2018-12-28T03:24:10.5289140Z</Message></Error>
> 
> Having any troubles? Hit us up at dvc.org/support, we are always happy to help!
> Debug: Analytics is enabled.
> Debug: Trying to spawn '['c:\\users\\raicevim\\appdata\\local\\continuum\\anaconda3\\python.exe', '-m', 'dvc', 'daemon', '-q', 'analytics', 'C:\\Users\\raicevim\\AppData\\Local\\Temp\\tmp78lk0n94']'
> Debug: Spawned '['c:\\users\\raicevim\\appdata\\local\\continuum\\anaconda3\\python.exe', '-m', 'dvc', 'daemon', '-q', 'analytics', 'C:\\Users\\raicevim\\AppData\\Local\\Temp\\tmp78lk0n94']'
> 
@raikhan And could you please try reconfiguring your remote url like so:
```
dvc remote add -d myremote azure://dvctest/
```
(you could also just edit it right in your .dvc/config)
and see if that works?
That seems to have done the trick! Here is the dvc pull -v again:

> Debug: PRAGMA user_version;
> Debug: fetched: [(3,)]
> Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
> Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
> Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
> Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
> Debug: PRAGMA user_version = 3;
> Preparing to download data from 'azure://dvctest/'
> Preparing to collect status from azure://dvctest/
> [##############################] 100% Collecting information
> [##############################] 100% Analysing status.
> Debug: SELECT count from state_info WHERE rowid=1
> Debug: fetched: [(0,)]
> Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
> Debug: PRAGMA user_version;
> Debug: fetched: [(3,)]
> Debug: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
> Debug: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
> Debug: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
> Debug: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
> Debug: PRAGMA user_version = 3;
> Debug: SELECT * FROM link_state
> Debug: SELECT count from state_info WHERE rowid=1
> Debug: fetched: [(0,)]
> Debug: UPDATE state_info SET count = 0 WHERE rowid = 1
> Debug: Analytics is enabled.
> Debug: Trying to spawn '['c:\\users\\raicevim\\appdata\\local\\continuum\\anaconda3\\python.exe', '-m', 'dvc', 'daemon', '-q', 'analytics', 'C:\\Users\\raicevim\\AppData\\Local\\Temp\\tmp74_h015e']'
> Debug: Spawned '['c:\\users\\raicevim\\appdata\\local\\continuum\\anaconda3\\python.exe', '-m', 'dvc', 'daemon', '-q', 'analytics', 'C:\\Users\\raicevim\\AppData\\Local\\Temp\\tmp74_h015e']'

My config is now:

> 
> ['remote "dvctest"']
> url = azure://ContainerName=dvctest;
> [core]
> remote = myremote
> ['remote "myremote"']
> url = azure://dvctest/

So the "ContainerName" in the Azure URL is the issue? I was using the example from [documentation](https://dvc.org/doc/commands-reference/remote) when setting it up...
Yes, we've introduced a more natural url structure for azure remotes, but, apparently, didn't test backward compatibility well enough. Looking into fixing this right now. Thank you for the feedback! :) 
My pleasure @efiop , glad I could be of service :) Thank you for explaining the issue!
Let's keep this opened until everything is fixed :) 

Resolving patch:
diff --git a/dvc/remote/azure.py b/dvc/remote/azure.py
--- a/dvc/remote/azure.py
+++ b/dvc/remote/azure.py
@@ -28,10 +28,10 @@ def __call__(self, current, total):
 
 class RemoteAzure(RemoteBase):
     scheme = 'azure'
-    REGEX = (r'^azure://((?P<path>[^=;]*)|('
+    REGEX = (r'azure://((?P<path>[^=;]*)?|('
              # backward compatibility
              r'(ContainerName=(?P<container_name>[^;]+);?)?'
-             r'(?P<connection_string>.+)?))$')
+             r'(?P<connection_string>.+)?)?)$')
     REQUIRES = {'azure-storage-blob': BlockBlobService}
     PARAM_ETAG = 'etag'
     COPY_POLL_SECONDS = 5
@@ -43,8 +43,9 @@ def __init__(self, project, config):
         self.url = config.get(Config.SECTION_REMOTE_URL)
         match = re.match(self.REGEX, self.url)  # backward compatibility
 
+        path = match.group('path')
         self.bucket = (
-            urlparse(self.url).netloc
+            urlparse(self.url if path else '').netloc
             or match.group('container_name')  # backward compatibility
             or os.getenv('AZURE_STORAGE_CONTAINER_NAME'))
 
@@ -68,8 +69,11 @@ def __init__(self, project, config):
     @property
     def blob_service(self):
         if self.__blob_service is None:
+            logger.debug('URL {}'.format(self.url))
+            logger.debug('Connection string {}'.format(self.connection_string))
             self.__blob_service = BlockBlobService(
                 connection_string=self.connection_string)
+            logger.debug('Container name {}'.format(self.bucket))
             self.__blob_service.create_container(self.bucket)
         return self.__blob_service
 


Test patch:
diff --git a/tests/test_data_cloud.py b/tests/test_data_cloud.py
--- a/tests/test_data_cloud.py
+++ b/tests/test_data_cloud.py
@@ -144,7 +144,11 @@ def get_gcp_url():
 
 
 def get_azure_url_compat():
-    return 'azure://'
+    container_name = os.getenv("AZURE_STORAGE_CONTAINER_NAME")
+    connection_string = os.getenv("AZURE_STORAGE_CONNECTION_STRING")
+    assert container_name is not None
+    return 'azure://ContainerName={};{}'.format(container_name,
+                                                connection_string)
 
 
 def get_azure_url():
@@ -165,11 +169,9 @@ def test(self):
                  ('gs://mybucket/', RemoteGS),
                  ('ssh://user@localhost:/', RemoteSSH),
                  ('http://localhost:8000/', RemoteHTTP),
+                 ('azure://ContainerName=mybucket;conn_string;', RemoteAzure),
                  (TestDvc.mkdtemp(), RemoteLOCAL)]
 
-        if _should_test_azure():
-            clist.append(('azure://ContainerName=', RemoteAzure))
-
         for scheme, cl in clist:
             remote_url = scheme + str(uuid.uuid4())
             config[TEST_SECTION][Config.SECTION_REMOTE_URL] = remote_url
diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py
new file mode 100644
diff --git a/tests/unit/remote/__init__.py b/tests/unit/remote/__init__.py
new file mode 100644
diff --git a/tests/unit/remote/azure.py b/tests/unit/remote/azure.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/remote/azure.py
@@ -0,0 +1,36 @@
+from unittest import TestCase
+
+from dvc.remote.azure import RemoteAzure
+
+
+class TestRemoteAzure(TestCase):
+    container_name = 'container-name'
+    connection_string = (
+        'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;'
+        'AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsu'
+        'Fq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;'
+        'BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;'
+    )
+
+    def test_init_compat(self):
+        url = (
+            'azure://ContainerName={container_name};{connection_string}'
+            .format(container_name=self.container_name,
+                    connection_string=self.connection_string)
+        )
+        config = {'url': url}
+        remote = RemoteAzure(None, config)
+        self.assertEqual(remote.url, url)
+        self.assertEqual(remote.prefix, '')
+        self.assertEqual(remote.bucket, self.container_name)
+        self.assertEqual(remote.connection_string, self.connection_string)
+
+    def test_init(self):
+        prefix = 'some/prefix'
+        url = 'azure://{}/{}'.format(self.container_name, prefix)
+        config = {'url': url, 'connection_string': self.connection_string}
+        remote = RemoteAzure(None, config)
+        self.assertEqual(remote.url, url)
+        self.assertEqual(remote.prefix, prefix)
+        self.assertEqual(remote.bucket, self.container_name)
+        self.assertEqual(remote.connection_string, self.connection_string)