Repository: avocado-framework/avocado

Issue:
utils/vmimage: get() when in parallel, sometimes returns a "Permission denied" error
When using `avocado.utils.vmimage.get` method and passing a wrong checksum, test state is ERROR but debug.log message doesn't help to figure out why. For instance:


```python
class TransientDomain(Test):
    def setUp(self):
        super().setUp()
        self.image = vmimage.get(name='fedora',
                                 version='33',
                                 cache_dir='/tmp/libvirt-scratch',
                                 checksum='67daa956d8c82ef799f8b0a191c1753c9bda3bca2')
```

(I have added a "2" at the end of checksum to force this error)

debug.log (for one specific test):
```
 13:55:34 INFO | INIT 1-TransientDomain.test_convert_transient_to_persistent
13:55:34 DEBUG| PARAMS (key=timeout, path=*, default=None) => None
13:55:34 DEBUG| Test metadata:
13:55:34 DEBUG|   filename: /home/local/src/beraldo/libvirt/tests/beraldo/tests/domain/test_transient.py
13:55:34 DEBUG|   teststmpdir: /var/tmp/avocado_bvytpc03
13:55:34 DEBUG|   workdir: /var/tmp/tmpre6s_9_h/test-results/tmp_dir1k5zauxr/1-TransientDomain.test_convert_transient_to_persistent
13:55:34 INFO | START 1-TransientDomain.test_convert_transient_to_persistent
13:55:34 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
13:55:35 INFO | Asset not in cache, fetching it.
13:55:35 INFO | Fetching https://dl.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 -> /tmp/libvirt-scratch/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2.6dlgj4_s
13:56:45 ERROR| PermissionError: [Errno 13] Permission denied: '/tmp/libvirt-scratch/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2'
13:56:45 ERROR| 
13:56:45 ERROR| Reproduced traceback from: /home/local/src/beraldo/avocado/avocado/core/test.py:770
13:56:45 ERROR| Traceback (most recent call last):
13:56:45 ERROR|   File "/home/local/src/beraldo/libvirt/tests/beraldo/tests/domain/test_transient.py", line 70, in test_convert_transient_to_persistent
13:56:45 ERROR|     domain = self.create_domain(arguments={'image': self.image.path},
13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 446, in path
13:56:45 ERROR|     return self._path or self.get()
13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 481, in get
13:56:45 ERROR|     self.download()
13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 472, in download
13:56:45 ERROR|     metadata=metadata).fetch()
13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/asset.py", line 381, in fetch
13:56:45 ERROR|     raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))
13:56:45 ERROR| OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/libvirt-scratch/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').
13:56:45 ERROR| 
13:56:45 DEBUG| Local variables:
13:56:45 DEBUG|  -> self <class 'test_transient.TransientDomain'>: 1-TransientDomain.test_convert_transient_to_persistent
13:56:45 ERROR| Traceback (most recent call last):

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/core/test.py", line 858, in _run_avocado
    raise test_exception

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/core/test.py", line 765, in _run_avocado
    testMethod()

13:56:45 ERROR|   File "/home/local/src/beraldo/libvirt/tests/beraldo/tests/domain/test_transient.py", line 70, in test_convert_transient_to_persistent
    domain = self.create_domain(arguments={'image': self.image.path},

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 446, in path
    return self._path or self.get()

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 481, in get
    self.download()

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 472, in download
    metadata=metadata).fetch()

13:56:45 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/asset.py", line 381, in fetch
    raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))

13:56:45 ERROR| OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/libvirt-scratch/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').

13:56:45 ERROR| ERROR 1-TransientDomain.test_convert_transient_to_persistent -> OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/libvirt-scratch/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').
```

IMO, this makes harder to find the issue, since PermissionDenied could be replaced by a proper avocado exception saying that CHECKSUM is not matching.


Discussion:
I think this `[Errno 13] Permission denied:` is related to another problem. I cannot reproduce this error with the asset module and a wrong hash.
@beraldoleal I can not reproduce this.  I get:

```
$ avocado run /tmp/t.py 
JOB ID     : 3191f9b83ff438a7e25e9d01f128f1020c9d7f14
JOB LOG    : /home/cleber/avocado/job-results/job-2021-06-01T19.16-3191f9b/job.log
 (1/1) /tmp/t.py:TransientDomain.test: ERROR: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 (Hash mismatch. Ignoring asset from the cache). (124.86 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2021-06-01T19.16-3191f9b/results.html
JOB TIME   : 125.26 s
```

And internally:

```
2021-06-01 19:18:40,104 test             L0917 ERROR| Traceback (most recent call last):

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 858, in _run_avocado
    raise test_exception

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/core/test.py", line 765, in _run_avocado
    testMethod()

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/tmp/t.py", line 11, in test
    self.image.path

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/utils/vmimage.py", line 447, in path
    return self._path or self.get()

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/utils/vmimage.py", line 482, in get
    self.download()

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/utils/vmimage.py", line 467, in download
    asset_path = asset.Asset(name=self.url,

2021-06-01 19:18:40,104 test             L0917 ERROR|   File "/home/cleber/src/avocado/avocado/avocado/utils/asset.py", line 382, in fetch
    raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))

2021-06-01 19:18:40,104 test             L0917 ERROR| OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 (Hash mismatch. Ignoring asset from the cache).

2021-06-01 19:18:40,104 test             L0932 ERROR| ERROR 1-/tmp/t.py:TransientDomain.test -> OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 (Hash mismatch. Ignoring asset from the cache).
```
I got this error again, and it is not related to the checksum indeed. Probably it is related to #4645, when downloading the same image multiple times in parallel.

Full log again, for debugging:

```
$ cat ~/avocado/job-results/job-2021-06-11T11.29-d0a4ccb/test-results/1-tests_domain_test_transient.py_TransientDomain.test_lifecycle/debug.log 
11:29:17 INFO | INIT 1-TransientDomain.test_lifecycle
11:29:17 DEBUG| PARAMS (key=timeout, path=*, default=None) => None
11:29:17 DEBUG| Test metadata:
11:29:17 DEBUG|   filename: /home/local/src/beraldo/lavocado/tests/domain/test_transient.py
11:29:17 DEBUG|   teststmpdir: /var/tmp/avocado_4dt6gn4t
11:29:17 DEBUG|   workdir: /var/tmp/tmpx4kgjuoy/test-results/tmp_dirqlbtf0dq/1-TransientDomain.test_lifecycle
11:29:17 INFO | START 1-TransientDomain.test_lifecycle
11:29:17 DEBUG| DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
11:29:17 INFO | Asset not in cache, fetching it.
11:29:17 INFO | Fetching https://dl.fedoraproject.org/pub/fedora/linux/releases/33/Cloud/x86_64/images/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 -> /tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2.4me7_o66
11:31:00 ERROR| PermissionError: [Errno 13] Permission denied: '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2'
11:31:00 ERROR| 
11:31:00 ERROR| Reproduced traceback from: /home/local/src/beraldo/avocado/avocado/core/test.py:770
11:31:00 ERROR| Traceback (most recent call last):
11:31:00 ERROR|   File "/home/local/src/beraldo/lavocado/tests/domain/test_transient.py", line 40, in test_lifecycle
11:31:00 ERROR|     domain = self.create_domain(arguments={'image': self.image.path},
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 450, in path
11:31:00 ERROR|     return self._path or self.get()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 486, in get
11:31:00 ERROR|     self.download()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 477, in download
11:31:00 ERROR|     metadata=metadata).fetch()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/asset.py", line 381, in fetch
11:31:00 ERROR|     raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))
11:31:00 ERROR| OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').
11:31:00 ERROR| 
11:31:00 DEBUG| Local variables:
11:31:00 DEBUG|  -> self <class 'test_transient.TransientDomain'>: 1-TransientDomain.test_lifecycle
11:31:00 INFO | Running '/usr/bin/qemu-img create -f qcow2 -b /tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 -F qcow2 /tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64-5eb8cadb.qcow2'
11:31:00 DEBUG| [stdout] Formatting '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64-5eb8cadb.qcow2', fmt=qcow2 size=4294967296 backing_file=/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
11:31:00 INFO | Command '/usr/bin/qemu-img create -f qcow2 -b /tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 -F qcow2 /tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64-5eb8cadb.qcow2' finished with 0 after 0.005259110999759287s
11:31:00 DEBUG| Formatting '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64-5eb8cadb.qcow2', fmt=qcow2 size=4294967296 backing_file=/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
11:31:00 ERROR| Traceback (most recent call last):
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/core/test.py", line 858, in _run_avocado
    raise test_exception
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/core/test.py", line 765, in _run_avocado
    testMethod()
11:31:00 ERROR|   File "/home/local/src/beraldo/lavocado/tests/domain/test_transient.py", line 40, in test_lifecycle
    domain = self.create_domain(arguments={'image': self.image.path},
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 450, in path
    return self._path or self.get()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 486, in get
    self.download()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/vmimage.py", line 477, in download
    metadata=metadata).fetch()
11:31:00 ERROR|   File "/home/local/src/beraldo/avocado/avocado/utils/asset.py", line 381, in fetch
    raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))
11:31:00 ERROR| OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').
11:31:00 ERROR| ERROR 1-TransientDomain.test_lifecycle -> OSError: Failed to fetch Fedora-Cloud-Base-33-1.2.x86_64.qcow2 ([Errno 13] Permission denied: '/tmp/lavocado-cache/by_location/67e23165d25d7f9934a909a12bc14277c9fb0b0d/Fedora-Cloud-Base-33-1.2.x86_64.qcow2').
```
I'm investigating this.

Resolving patch:
diff --git a/avocado/utils/asset.py b/avocado/utils/asset.py
--- a/avocado/utils/asset.py
+++ b/avocado/utils/asset.py
@@ -26,8 +26,8 @@
 import shutil
 import stat
 import sys
-import tempfile
 import time
+import uuid
 from datetime import datetime
 from urllib.parse import urlparse
 
@@ -125,12 +125,19 @@ def _download(self, url_obj, asset_path):
         """
         try:
             # Temporary unique name to use while downloading
-            temp = '%s.%s' % (asset_path,
-                              next(tempfile._get_candidate_names()))  # pylint: disable=W0212
-            url_download(url_obj.geturl(), temp)
-
-            # Acquire lock only after download the file
-            with FileLock(asset_path, 1):
+            temp = '%s.%s' % (asset_path, str(uuid.uuid4()))
+
+            # To avoid parallel downloads of the same asset, and errors during
+            # the write after download, let's get the lock before start the
+            # download.
+            with FileLock(asset_path, 120):
+                try:
+                    self.find_asset_file(create_metadata=True)
+                    return True
+                except OSError:
+                    LOG.info("Asset not in cache after lock, fetching it.")
+
+                url_download(url_obj.geturl(), temp)
                 shutil.copy(temp, asset_path)
                 self._create_hash_file(asset_path)
                 if not self._verify_hash(asset_path):
@@ -321,10 +328,11 @@ def _verify_hash(self, asset_path):
         return self._has_valid_hash(asset_path, self.asset_hash)
 
     def fetch(self):
-        """
-        Fetches the asset. First tries to find the asset on the provided
-        cache_dirs list. Then tries to download the asset from the locations
-        list provided.
+        """Try to fetch the current asset.
+
+        First tries to find the asset on the provided cache_dirs list.
+        Then tries to download the asset from the locations list
+        provided.
 
         :raise OSError: When it fails to fetch the asset
         :returns: The path for the file on the cache directory.
@@ -334,15 +342,10 @@ def fetch(self):
         asset_file = None
         error = "unknown"
         try:
-            asset_file = self.find_asset_file()
+            return self.find_asset_file(create_metadata=True)
         except OSError:
             LOG.info("Asset not in cache, fetching it.")
 
-        if asset_file is not None:
-            if self.metadata is not None:
-                self._create_metadata_file(asset_file)
-            return asset_file
-
         # If we get to this point, we have to download it from a location.
         # A writable cache directory is then needed. The first available
         # writable cache directory will be used.
@@ -380,10 +383,14 @@ def fetch(self):
 
         raise OSError("Failed to fetch %s (%s)." % (self.asset_name, error))
 
-    def find_asset_file(self):
+    def find_asset_file(self, create_metadata=False):
         """
         Search for the asset file in each one of the cache locations
 
+        :param bool create_metadata: Should this method create the
+                                     metadata in case asset file found
+                                     and metadata is not found?  Default
+                                     is False.
         :return: asset path, if it exists in the cache
         :rtype: str
         :raises: OSError
@@ -405,6 +412,9 @@ def find_asset_file(self):
             if not self._has_valid_hash(asset_file, self.asset_hash):
                 continue
 
+            if create_metadata:
+                self._create_metadata_file(asset_file)
+
             return asset_file
 
         raise OSError("File %s not found in the cache." % self.asset_name)


Test patch: