Repository: microsoft/lisa

Issue:
Command not found (PATH does not contain /usr/sbin)
Getting errors when using LISAv3 to deploy and test CentOS 7_9 on Azure

`[ERROR] lisa.env[generated_0].node[0].cmd[7289] not found command: Command not found: modinfo. Check that modinfo is installed and on $PATH`

`[ERROR] lisa.env[generated_0].node[0].cmd[1038] not found command: Command not found: waagent. Check that waagent is installed and on $PATH`

`[ERROR] lisa.env[generated_0].node[0].cmd[8629] not found command: Command not found: lsmod. Check that lsmod is installed and on $PATH`

SSHing into the node confirms that all three of these commands are present and runnable on the node.

The error about modinfo missing appears to occur before any tests start running. These errors do not occur when deploying and testing Ubuntu 18.04-LTS.


Discussion:
@LiliDeng can you help on looking it?
Ack
@LiliDeng Those commands fallback to use sudo automatically. It needs to set no error level and info level log. If it's a tool, it should be handled at command detection phase, please check why it doesn't work.

```log
2021-09-07 22:37:21.095[3488][DEBUG] lisa.env[generated_0].node[0].cmd[2764] cmd: ['waagent', '-version'], cwd: None, shell: False, sudo: False, posix: True, remote: True
2021-09-07 22:37:21.178[3488][ERROR] lisa.env[generated_0].node[0].cmd[2764] not found command: Command not found: waagent. Check that waagent is installed and on $PATH
2021-09-07 22:37:21.178[3488][DEBUG] lisa.env[generated_0].node[0].cmd[3786] cmd: ['/usr/sbin/waagent', '-version'], cwd: None, shell: False, sudo: False, posix: True, remote: True
2021-09-07 22:37:21.440[15676][DEBUG] lisa.env[generated_0].node[0].cmd[3786].stdout WALinuxAgent-2.2.46 running on centos 7.9.2009
```
I do not understand why using sudo would be the solution here. waagent and modinfo should not require sudo and from the logs the fallback that works also doesn't use sudo
`cmd: ['/usr/sbin/waagent', '-version'], cwd: None, shell: False, sudo: False, posix: True, remote: True`
It simply uses the absolute path for the command since the PATH variable is not being properly set on CentOS.
> I do not understand why using sudo would be the solution here. waagent and modinfo should not require sudo and from the logs the fallback that works also doesn't use sudo
> `cmd: ['/usr/sbin/waagent', '-version'], cwd: None, shell: False, sudo: False, posix: True, remote: True`
> It simply uses the absolute path for the command since the PATH variable is not being properly set on CentOS.

You are right, the sudo is not necessary in this case.
# Updated Discussion
The problem is that the library being used to send commands to the node, spur, does so by calling `exec` on the provided command name and arguments. In this case, the PATH variable is only set by `/etc/environment`. No bash environment is created so setup scripts such as `bashrc` or `bash_profile` are not run. On CentOS and RHEL, this means that `/usr/sbin` (and other command locations) are never added to PATH.

In a bash environment, one would fix this by running `export` or `set`; however these are bash commands, not binary programs. For this reason, running `exec export...` will return command not found. This limits the possible solutions to this problem.

## Possible Solutions
1. Use `sudo` for any commands not found due to this misconfiguration of PATH. This works because when `sudo` is called, it sets the PATH to a specified sudo PATH. This solution is relatively easy to implement. However, it doesn't fix the more general problem of setup bash scripts not being run. This also requires running commands as sudo even when they do not require sudo permissions, which is bad practice. Lastly, this relies on the sudo PATH being properly set which may not be the case with every distro.
2. Modify /etc/environment to the correct PATH. This solution would only require changes to one place; it would be run once and the changes to PATH would persist. However, this requires manually changing the PATH which may have different requirements for every distro. This solution may also require the LISA system to re-login, which adds time and complexity to the running of tests. This solution also doesn't fix the more general problem of setup bash scripts not being run.
3. Run every command with `shell=True` by default. This would make every command take the form of `exec sh -c ...`. This would create an environment more congruent with running the commands through an terminal ssh session. For example, it would also fix parsing issues where spaces in a single argument gets broken into multiple arguments. This can reduce the number of bugs encountered, and decrease the amount of time needed to debug issues, because the behavior of LISA commands would more closely reflect the behavior of running commands in a terminal ssh sesison. However, this solution still might not run all of the setup scripts. This also has the most potential of causing unexpected errors across LISA because it changes how every command is run. Overall it would make the functioning of LISA more complex.

A better solution may exist that isn't listed in this discussion. At this time, we have decided to more forward with option 1 since it reliably fixes the issue encountered on CentOS and RHEL, it only requires changes to to the few commands in `/usr/sbin`, and all of the other considerations stated above.

Resolving patch:
diff --git a/lisa/tools/lsmod.py b/lisa/tools/lsmod.py
--- a/lisa/tools/lsmod.py
+++ b/lisa/tools/lsmod.py
@@ -36,6 +36,7 @@ def module_exists(
         no_error_log: bool = True,
     ) -> bool:
         result = self.run(
+            sudo=True,
             force_run=force_run,
             no_info_log=no_info_log,
             no_error_log=no_error_log,
diff --git a/lisa/tools/modinfo.py b/lisa/tools/modinfo.py
--- a/lisa/tools/modinfo.py
+++ b/lisa/tools/modinfo.py
@@ -30,20 +30,13 @@ def get_info(
     ) -> str:
         result = self.run(
             mod_name,
+            sudo=True,
             force_run=force_run,
             no_info_log=no_info_log,
             no_error_log=no_error_log,
+            expected_exit_code=0,
+            expected_exit_code_failure_message=f"Modinfo failed for module {mod_name}",
         )
-        if result.exit_code != 0:
-            # CentOS may not include the path when started,
-            # specify path and try again.
-            self._command = "/usr/sbin/modinfo"
-            result = self.run(
-                mod_name,
-                force_run=force_run,
-                no_info_log=no_info_log,
-                no_error_log=no_error_log,
-            )
         return result.stdout
 
     def get_version(


Test patch: