Repository: avocado-framework/avocado

Issue:
Functional test for requirements resolver fails easily
The PASS or FAIL result for the functional tests for the requirement resolver is very much a hit/miss, but it's somewhat easy to reproduce with:

```
$ export CI=1
$ dnf remove hello
$ dnf clean all
$ avocado run --test-runner=nrunner selftests/functional/test_requirements.py 
JOB ID     : 35d4cf58034c04eb47be0276197f8ae5f17af82b
JOB LOG    : /home/cleber/avocado/job-results/job-2021-05-26T23.55-35d4cf5/job.log
 (3/4) selftests/functional/test_requirements.py:BasicTest.test_multiple_success: STARTED
 (2/4) selftests/functional/test_requirements.py:BasicTest.test_single_fail: STARTED
 (1/4) selftests/functional/test_requirements.py:BasicTest.test_single_success: STARTED
 (4/4) selftests/functional/test_requirements.py:BasicTest.test_multiple_fails: STARTED
 (2/4) selftests/functional/test_requirements.py:BasicTest.test_single_fail: PASS (4.75 s)
 (1/4) selftests/functional/test_requirements.py:BasicTest.test_single_success: PASS (5.29 s)
 (3/4) selftests/functional/test_requirements.py:BasicTest.test_multiple_success: FAIL (14.22 s)
 (4/4) selftests/functional/test_requirements.py:BasicTest.test_multiple_fails: PASS (20.34 s)
RESULTS    : PASS 3 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB HTML   : /home/cleber/avocado/job-results/job-2021-05-26T23.55-35d4cf5/results.html
JOB TIME   : 21.52 s
```

The reason for the failure is that, multiple tests contain:

```
:avocado: requirement={"type": "package", "name": "hello"}
```

And because the nrunner will run those tests in parallel, it will also run those requirements in parallel.  But, given that dnf will hold a systemwide lock, the others can fail.  This is reproducible if you try to run multiple `avocado-runner-requirement-package` in parallel manually, such as this which succeeds:

```
{'status': 'running', 'time': 1345096.780925351}
{'status': 'running', 'time': 1345097.281267814}
{'type': 'stdout', 'log': b'Package(s) hello installed successfully', 'status': 'running', 'time': 1345097.281664319}
{'type': 'stderr', 'log': b'', 'status': 'running', 'time': 1345097.281689778}
{'result': 'pass', 'status': 'finished', 'time': 1345097.281706051}
```

And this one that ends up failing:

```
{'status': 'running', 'time': 1345097.18655851}
{'status': 'running', 'time': 1345097.687243183}
{'status': 'running', 'time': 1345098.187887105}
{'type': 'stdout', 'log': b'', 'status': 'running', 'time': 1345098.188174427}
{'type': 'stderr', 'log': b'Failed to install hello. Check the package(s) name and if sudo permission is granted.', 'status': 'running', 'time': 1345098.188197785}
{'result': 'error', 'status': 'finished', 'time': 1345098.188217684}
```

This issue became apparent because of the problems addressed in #4619 which causes test errors in nrunner to really be signaled as an exit code.


Discussion:
This problem was expected in this initial work of the Requirements Resolver and will happen in runners like package and asset when two or more instances of the runner try to fulfill the same requirement at the same time. We also discussed it in one of our scrum meetings last year. We were just lucky that it never happened in a way that caused a failure until now.

I expect to fix this with a more robust graph dependency architecture, where the same dependency of multiple tests will be fulfilled just once, and all the tests requiring it will depend on this unique task.
> This problem was expected in this initial work of the Requirements Resolver and will happen in runners like package and asset when two or more instances of the runner try to fulfill the same requirement at the same time. We also discussed it in one of our scrum meetings last year. We were just lucky that it never happened in a way that caused a failure until now.
> 

I understand that the *core* issue, the lack of unification of similar requirements was a caveat we were all aware of.  I did not realize (but should have) that the fact that we have multiple tests, running in parallel, asking for the same requirement to be fulfilled would cause very probable errors in our selftests.

And it wasn't luck... it was because of the bug I mention in #4619.  That bug needs urgent fixing, so we also need an urgent action here. 

> I expect to fix this with a more robust graph dependency architecture, where the same dependency of multiple tests will be fulfilled just once, and all the tests requiring it will depend on this unique task.

Agreed, but we need a short term solution too.
> > This problem was expected in this initial work of the Requirements Resolver and will happen in runners like package and asset when two or more instances of the runner try to fulfill the same requirement at the same time. We also discussed it in one of our scrum meetings last year. We were just lucky that it never happened in a way that caused a failure until now.
> 
> I understand that the _core_ issue, the lack of unification of similar requirements was a caveat we were all aware of. I did not realize (but should have) that the fact that we have multiple tests, running in parallel, asking for the same requirement to be fulfilled would cause very probable errors in our selftests.
> 
> And it wasn't luck... it was because of the bug I mention in #4619. That bug needs urgent fixing, so we also need an urgent action here.

Okay, got it!

> 
> > I expect to fix this with a more robust graph dependency architecture, where the same dependency of multiple tests will be fulfilled just once, and all the tests requiring it will depend on this unique task.
> 
> Agreed, but we need a short term solution too.

To have #4622 merged, I submitted #4623 skipping the functional tests for the requirements resolver for now.
This can also be reproduced running two instances of the `avocado-software-manager` at the same time:

```
[wrampazz@wrampazz avocado.dev]$ avocado-software-manager install hello
Running '/bin/sudo -n /bin/dnf -y install hello'
Command '/bin/sudo -n /bin/dnf -y install hello' finished with 0 after 4.3627714029862545s
Package(s) hello installed successfully
[wrampazz@wrampazz avocado.dev]$

[wrampazz@wrampazz avocado.dev]$ avocado-software-manager install hello
Running '/usr/bin/sudo -n /usr/bin/dnf -y install hello'
Command '/usr/bin/sudo -n /usr/bin/dnf -y install hello' finished with 1 after 3.968901346990606s
Failed to install hello. Check the package(s) name and if sudo permission is granted.
[wrampazz@wrampazz avocado.dev]$
```
Some more debugging to handle this problem correctly:

When running two instances of `process.system('/usr/bin/yum -y install hello', sudo=True)` at the same time, one fails not because it could not lock `yum`, but because one instance installed the package while the other instance thought the package was not installed yet. When the second process was able to lock `yum`, the package was already installed and it fails because of it.

First instance:
```
>>> process.system('/usr/bin/yum -y install hello', sudo=True)
0
>>>
```

Second instance:
```
>>> process.system('/usr/bin/yum -y install hello', sudo=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/utils/process.py", line 1142, in system
    cmd_result = run(cmd=cmd, timeout=timeout, verbose=verbose, ignore_status=ignore_status,
  File "/home/wrampazz/src/avocado/avocado.dev/avocado/utils/process.py", line 1082, in run
    raise CmdError(cmd, sp.result)
avocado.utils.process.CmdError: Command '/usr/bin/yum -y install hello' failed.
stdout: b"Waiting for process with pid 54186 to finish.\nLast metadata expiration check: 0:00:01 ago on Mon 31 May 2021 04:00:14 PM -03.\nDependencies resolved.\n================================================================================\n Package         Architecture     Version                Repository        Size\n================================================================================\nInstalling:\n hello           x86_64           2.10-5.fc34            fedora            71 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal size: 71 k\nInstalled size: 179 k\nDownloading Packages:\n[SKIPPED] hello-2.10-5.fc34.x86_64.rpm: Already downloaded                     \nRunning transaction check\nWaiting for process with pid 54186 to finish.\nTransaction check succeeded.\nRunning transaction test\nThe downloaded packages were saved in cache until the next successful transaction.\nYou can remove cached packages by executing 'yum clean packages'.\n"
stderr: b'Error: Transaction test error:\n  package hello-2.10-5.fc34.x86_64 is already installed\n\n'
additional_info: None
>>> 
```

Unfortunately, it seems there may be multiple causes for the process to fail when colliding. This is another reason, and this time it does not give a hint if the package is already installed or not:

```
stdout: b"Waiting for process with pid 57667 to finish.\nLast metadata expiration check: 0:00:22 ago on Mon 31 May 2021 04:29:37 PM -03.\nDependencies resolved.\n================================================================================\n Package         Architecture     Version                Repository        Size\n================================================================================\nInstalling:\n hello           x86_64           2.10-5.fc34            fedora            71 k\n\nTransaction Summary\n================================================================================\nInstall  1 Package\n\nTotal download size: 71 k\nInstalled size: 179 k\nDownloading Packages:\nWaiting for process with pid 57667 to finish.\n[SKIPPED] hello-2.10-5.fc34.x86_64.rpm: Already downloaded                     \nRunning transaction check\nWaiting for process with pid 57667 to finish.\nThe downloaded packages were saved in cache until the next successful transaction.\nYou can remove cached packages by executing 'dnf clean packages'.\n"
stderr: b"[Errno 2] No such file or directory: '/var/cache/dnf/fedora-e21c25ac3662d294/packages/hello-2.10-5.fc34.x86_64.rpm'\n"
additional_info: None
```

Resolving patch:
diff --git a/avocado/core/runners/requirement_package.py b/avocado/core/runners/requirement_package.py
--- a/avocado/core/runners/requirement_package.py
+++ b/avocado/core/runners/requirement_package.py
@@ -46,9 +46,14 @@ def _install(software_manager, cmd, package):
             if software_manager.install(package):
                 stdout = MESSAGES[cmd]['success'] % package
             else:
-                result = 'error'
-                stdout = ''
-                stderr = MESSAGES[cmd]['fail'] % package
+                # check if the error is a false negative because of package
+                # installation collision
+                if software_manager.check_installed(package):
+                    stdout = MESSAGES[cmd]['success'] % package
+                else:
+                    result = 'error'
+                    stdout = ''
+                    stderr = MESSAGES[cmd]['fail'] % package
         else:
             stdout = MESSAGES['check-installed']['success'] % package
         return result, stdout, stderr
@@ -61,9 +66,14 @@ def _remove(software_manager, cmd, package):
             if software_manager.remove(package):
                 stdout = MESSAGES[cmd]['success'] % package
             else:
-                result = 'error'
-                stdout = ''
-                stderr = MESSAGES[cmd]['fail'] % package
+                # check if the error is a false negative because of package
+                # installation collision
+                if not software_manager.check_installed(package):
+                    stdout = MESSAGES[cmd]['success'] % package
+                else:
+                    result = 'error'
+                    stdout = ''
+                    stderr = MESSAGES[cmd]['fail'] % package
         else:
             stdout = MESSAGES['check-installed']['fail'] % package
         return result, stdout, stderr


Test patch:
diff --git a/selftests/functional/test_requirements.py b/selftests/functional/test_requirements.py
--- a/selftests/functional/test_requirements.py
+++ b/selftests/functional/test_requirements.py
@@ -104,7 +104,6 @@ def test_single_success(self):
             self.assertIn('PASS 1', result.stdout_text,)
             self.assertNotIn('bash', result.stdout_text,)
 
-    @unittest.skip('Skipping until test collision is fixed (#4620).')
     @unittest.skipUnless(os.getenv('CI'), skip_package_manager_message)
     def test_single_fail(self):
         with script.Script(os.path.join(self.tmpdir.name,
diff --git a/selftests/functional/test_runner_requirement_package.py b/selftests/functional/test_runner_requirement_package.py
--- a/selftests/functional/test_runner_requirement_package.py
+++ b/selftests/functional/test_runner_requirement_package.py
@@ -10,7 +10,6 @@
 
 class RunnableRun(unittest.TestCase):
 
-    @unittest.skip('Skipping until test collision is fixed (#4620).')
     def test_no_kwargs(self):
         res = process.run("%s runnable-run -k requirement-package" % RUNNER,
                           ignore_status=True)
@@ -19,7 +18,6 @@ def test_no_kwargs(self):
         self.assertIn(b"'time': ", res.stdout)
         self.assertEqual(res.exit_status, 0)
 
-    @unittest.skip('Skipping until test collision is fixed (#4620).')
     def test_action_check_alone(self):
         action = 'action=check'
         res = process.run("%s runnable-run -k requirement-package %s"
@@ -31,7 +29,6 @@ def test_action_check_alone(self):
                       res.stdout)
         self.assertEqual(res.exit_status, 0)
 
-    @unittest.skip('Skipping until test collision is fixed (#4620).')
     @unittest.skipUnless(os.getenv('CI'), "This test runs on CI environments"
                          " only as it depends on the system package manager,"
                          " and some environments don't have it available.")
@@ -57,7 +54,6 @@ def test_recipe(self):
 
 class TaskRun(unittest.TestCase):
 
-    @unittest.skip('Skipping until test collision is fixed (#4620).')
     def test_no_kwargs(self):
         res = process.run("%s task-run -i XXXreq-pacXXX -k requirement-package"
                           % RUNNER, ignore_status=True)