Repository: sosreport/sos

Issue:
[sos-collect] Adding a new transport
I'm attempting to add a transport for saltstack master. The transport connects to nodes but an issue occurs during the sosnodes.py `collect()` function. Comparing to **control_persist** transport with `-vvv` I can see both have identical behavior parsing and setting the hostname. Both transports succeed to `read_file(/etc/os-release)` with identical content and during `collect()` both have 4 jobs in (when I set 3 hosts w/ `--nodes`) . 

However at that point control_persist `collect()` have 4 jobs in `self.opts.jobs`. However `self.client_list` for transport saltstack always shows 0 or 1 depending on the `--no-local` flag, where transport control_persist always has the correct number in `self.client_list`.

All nodes connect and gather a hostname and read the `/etc/os-release` file. The `self.client_list` ends up with only the localhost or nobody in this transport as I can see the `connected()` property is True either via log_debug or even simply testing the class manually in python shell.

The `node` passed to `SosNode` is a tuple, and contains the node name and **None** for password for both transports. I've tracked it to function `determine_host_policy()` in file `sosnodef.py`. I'm at a loss to understand why creating the host object is failing

```
+       self.log_debug("Policy not local, loading host object")
        host = load(cache={}, sysroot=self.opts.sysroot, init=InitSystem(),
                    probe_runtime=True,
                    remote_exec=self._transport.remote_exec,
                    remote_check=self.read_file('/etc/os-release'))
+       self.log_debug("The host object successfully loaded")
```

1. The `log_debug("Policy is not local, loading host object")` appears
2. A few other messages but then then a successful `read_file('/etc/os-release')` message
3. The final `log_debug("The host object successfully loaded")`  never appears on the console. It goes directly back to `collect()` and reports back that the client_list is only the localhost.

```
[sos_collector:_connect_to_node] Attempting to connect to node: store-0
[store-0:_connect] Transport is locally supported and service running. 
[store-0:run_command] Running command hostname
[EL8_store-0:_get_hostname] Hostname set to EL8_store-0
[store-0:determine_host_policy] Attempting to determine host policy   <-- start of function
[store-0:determine_host_policy] Policy not local, loading host object     
[store-0:read_file] Reading file /etc/os-release
[EL8_store-0:read_file] Reading file /etc/os-release
[EL8_store-0:run_command] Running command cat /etc/os-release
[EL8_store-0:_read_file] success reading - fname: /etc/os-release            <-- when the status == 0
[sos_collector:collect] Number of nodes in client_list: 1                              <-- drops out of determine
[sos_collector:collect] Number of nodes to collect from: 1
Collection would only gather from localhost due to failure to either enumerate or connect to cluster nodes. Assuming single collection from localhost is not desired.
Aborting...
```

I then decided to tweak `load()` from file `policies/__init__.py` slightly to try and examine what happens when remote_exec is defined. 

```
def load(cache={}, sysroot=None, init=None, probe_runtime=True,
         remote_exec=None, remote_check=''):
    if 'policy' in cache:
        return cache.get('policy')
    print("remote_exec: %s" % remote_exec)
    import sos.policies.distros
    helper = ImporterHelper(sos.policies.distros)
    for module in helper.get_modules():
        for policy in import_policy(module):
            if policy.check(remote=remote_check):
                cache['policy'] = policy(sysroot=sysroot, init=init,
                                         probe_runtime=probe_runtime,
                                         remote_exec=remote_exec)
            else:
                if remote_exec is not None:
                    print("Policy %s does not match this remote system" %
                          policy)
                    print("Policy check returned: %s" % policy.check())
```

What is odd is non of the listed policies match the RockyLinux policy that the localhost immediately determines when included:

```
remote_exec: salt --out json --static store-0 cmd.shell 
Policy <class 'sos.policies.distros.amazon.AmazonPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.cos.CosPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.debian.DebianPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.openeuler.OpenEulerPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.CentOsAtomicPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.CentOsPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.FedoraPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.RHELPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.RedHatAtomicPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.RedHatCoreOSPolicy'> does not match this remote system
Policy check returned: False
Policy <class 'sos.policies.distros.redhat.RedHatPolicy'> does not match this remote system
Policy check returned: False
```

Clearly when `control_persist` is used however it does find a RockyLinux policy:

```
[store-0:_connect] Opening SSH session to create control socket
[store-0:_connect] Successfully created control socket at /var/tmp/sos.b1rp19fm/.sos-collector-store-0
[store-0:run_command] Running command hostname
[EL8_store-0:_get_hostname] Hostname set to EL8_store-0
[store-0:determine_host_policy] Attempting to determine host policy
[store-0:determine_host_policy] Policy not local, loading host object
[store-0:read_file] Reading file /etc/os-release
[EL8_store-0:read_file] Reading file /etc/os-release
[EL8_store-0:run_command] Running command cat /etc/os-release
[EL8_store-0:_read_file] read success for fname: /etc/os-release
[store-0:determine_host_policy] The host object contains: <sos.policies.distros.rocky.RockyPolicy object at 0x7f69690a9780>
[store-0:determine_host_policy] loaded policy Rocky Linux for host
[EL8_store-0:_load_sos_info] sos version is 4.2.0-20
```

Thanks in advance for any suggestions or pointers which might help.


Discussion:
Hmm, from your description so far nothing immediately jumps out at me - except that perhaps we're inadvertently squashing an exception during the file read, but I don't think that's all that likely to be the case.

Can you provide the salt transport that you have defined so far? Either as a WIP PR or just a pastebin of some fashion is fine.
I'll cleanup some of the extra debug_log I added and then submit it as a WIP PR shortly.

Thanks
@TurboTurtle Opened #3087 as a Draft. I left some of the debug_log in place but will cleanup the entire branch before converting from Draft to full PR.

Resolving patch:
diff --git a/sos/collector/exceptions.py b/sos/collector/exceptions.py
--- a/sos/collector/exceptions.py
+++ b/sos/collector/exceptions.py
@@ -104,6 +104,14 @@ def __init__(self, transport=None):
         super(InvalidTransportException, self).__init__(message)
 
 
+class SaltStackMasterUnsupportedException(Exception):
+    """Raised when SaltStack Master is unsupported locally"""
+
+    def __init__(self):
+        message = 'Master unsupported by local SaltStack installation'
+        super(SaltStackMasterUnsupportedException, self).__init__(message)
+
+
 __all__ = [
     'AuthPermissionDeniedException',
     'CommandTimeoutException',
@@ -113,6 +121,7 @@ def __init__(self, transport=None):
     'ControlSocketMissingException',
     'InvalidPasswordException',
     'PasswordRequestException',
+    'SaltStackMasterUnsupportedException',
     'TimeoutPasswordAuthException',
     'UnsupportedHostException',
     'InvalidTransportException'
diff --git a/sos/collector/sosnode.py b/sos/collector/sosnode.py
--- a/sos/collector/sosnode.py
+++ b/sos/collector/sosnode.py
@@ -21,6 +21,7 @@
 from sos.collector.transports.control_persist import SSHControlPersist
 from sos.collector.transports.local import LocalTransport
 from sos.collector.transports.oc import OCTransport
+from sos.collector.transports.saltstack import SaltStackMaster
 from sos.collector.exceptions import (CommandTimeoutException,
                                       ConnectionException,
                                       UnsupportedHostException,
@@ -29,7 +30,8 @@
 TRANSPORTS = {
     'local': LocalTransport,
     'control_persist': SSHControlPersist,
-    'oc': OCTransport
+    'oc': OCTransport,
+    'saltstack': SaltStackMaster
 }
 
 
diff --git a/sos/collector/transports/saltstack.py b/sos/collector/transports/saltstack.py
new file mode 100644
--- /dev/null
+++ b/sos/collector/transports/saltstack.py
@@ -0,0 +1,136 @@
+# Copyright Red Hat 2022, Trevor Benson <trevor.benson@gmail.com>
+
+# This file is part of the sos project: https://github.com/sosreport/sos
+#
+# This copyrighted material is made available to anyone wishing to use,
+# modify, copy, or redistribute it subject to the terms and conditions of
+# version 2 of the GNU General Public License.
+#
+# See the LICENSE file in the source distribution for further information.
+
+import contextlib
+import json
+import os
+import shutil
+from sos.collector.transports import RemoteTransport
+from sos.collector.exceptions import (ConnectionException,
+                                      SaltStackMasterUnsupportedException)
+from sos.utilities import (is_executable,
+                           sos_get_command_output)
+
+
+class SaltStackMaster(RemoteTransport):
+    """
+    A transport for collect that leverages SaltStack's Master Pub/Sub
+    functionality to send commands to minions.
+
+    This transport will by default assume the use cmd.shell module to
+    execute commands on the minions.
+    """
+
+    name = 'saltstack'
+
+    def _convert_output_json(self, json_output):
+        return list(json.loads(json_output).values())[0]
+
+    def run_command(
+            self, cmd, timeout=180, need_root=False, env=None, get_pty=False):
+        """
+        Run a command on the remote host using SaltStack Master.
+        If the output is json, convert it to a string.
+        """
+        ret = super(SaltStackMaster, self).run_command(
+            cmd, timeout, need_root, env, get_pty)
+        with contextlib.suppress(Exception):
+            ret['output'] = self._convert_output_json(ret['output'])
+        return ret
+
+    def _salt_retrieve_file(self, node, fname, dest):
+        """
+        Execute cp.push on the remote host using SaltStack Master
+        """
+        cmd = f"salt {node} cp.push {fname}"
+        res = sos_get_command_output(cmd)
+        if res['status'] == 0:
+            cachedir = f"/var/cache/salt/master/minions/{self.address}/files"
+            cachedir_file = os.path.join(cachedir, fname.lstrip('/'))
+            shutil.move(cachedir_file, dest)
+            return True
+        return False
+
+    @property
+    def connected(self):
+        """Check if the remote host is responding using SaltStack Master."""
+        up = self.run_command("echo Connected", timeout=10)
+        return up['status'] == 0
+
+    def _check_for_saltstack(self, password=None):
+        """Checks to see if the local system supported SaltStack Master.
+
+        This check relies on feedback from the salt binary. The command being
+        run should always generate stderr output, but depending on what that
+        output reads we can determine if SaltStack Master is supported or not.
+
+        For our purposes, a host that does not support SaltStack Master is not
+        able to run sos-collector.
+
+        Returns
+            True if SaltStack Master is supported, else raise Exception
+        """
+
+        cmd = 'salt-run manage.status'
+        res = sos_get_command_output(cmd)
+        if res['status'] == 0:
+            return res['status'] == 0
+        else:
+            raise SaltStackMasterUnsupportedException
+
+    def _connect(self, password=None):
+        """Connect to the remote host using SaltStack Master.
+
+        This method will attempt to connect to the remote host using SaltStack
+        Master. If the connection fails, an exception will be raised.
+
+        If the connection is successful, the connection will be stored in the
+        self._connection attribute.
+        """
+        if not is_executable('salt'):
+            self.log_error("salt command is not executable. ")
+            return False
+
+        try:
+            self._check_for_saltstack()
+        except ConnectionException:
+            self.log_error("Transport is not locally supported. ")
+            raise
+        self.log_info("Transport is locally supported and service running. ")
+        cmd = "echo Connected"
+        result = self.run_command(cmd, timeout=180)
+        return result['status'] == 0
+
+    def _disconnect(self):
+        return True
+
+    @property
+    def remote_exec(self):
+        """The remote execution command to use for this transport."""
+        salt_args = "--out json --static --no-color"
+        return f"salt {salt_args} {self.address} cmd.shell "
+
+    def _retrieve_file(self, fname, dest):
+        """Retrieve a file from the remote host using saltstack
+
+        Parameters
+            fname       The path to the file on the remote host
+            dest        The path to the destination directory on the master
+
+        Returns
+            True if the file was retrieved, else False
+        """
+        return (
+            self._salt_retrieve_file(self.address, fname, dest)
+            if self.connected
+            else False
+        )
+
+# vim: set et ts=4 sw=4 :


Test patch: