Repository: cobbler/cobbler

Issue:
cobbler status fails when log files are gzipped
### Describe the bug

After some time of running, the log file /var/log/cobbler/install.log gets rotated by logrotate.
Logrotate creates gzipped files, e.g. 'install.log-20220830.gz'

Then "cobbler status" fails with the following error:
````
### ERROR ###
Unexpected remote error, check the server side logs for further info
<class 'TypeError'>:a bytes-like object is required, not 'str'
````

<!--- A clear and concise description of ) owhat the bug is. -->

<!--- HINT: You can paste gist.github.com links for long logs or larger files -->

### Steps to reproduce

1. cd /var/log/cobbler 
2. rm -f install.log*.gz
3. cp install.log install.log-20220830
4. cobbler status => works OK
5. gzip install.log-20220830
6. cobbler status => fails

### Expected behavior

<!--- A clear and concise description of what you expected to happen. -->
Cobbler status should not fail when install.log files are gzipped.

### Cobbler version

Cobbler version 3.2.2 (from EPEL repository)
<!--- Paste output from `cobbler version` -->
````paste below
Cobbler 3.2.2
  source: ?, ?
  build time: Wed Mar  2 01:17:52 2022
````

### Operating system

<!--- On which operating system do you use Cobbler? -->
Linux RHEL 8.6

### Cobbler log

<!--- Paste (partial) output from `/var/log/cobbler/cobbler.log` -->
````paste below
2022-08-31T20:57:17 - INFO | Exception occurred: <class 'TypeError'>
2022-08-31T20:57:17 - INFO | Exception value: a bytes-like object is required, not 'str'
2022-08-31T20:57:17 - INFO | Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 3448, in _dispatch
    return method_handle(*params)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 3025, in get_status
    return self.api.status(mode=mode, logger=self.logger)

  File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 1373, in status
    return statusifier.run()

  File "/usr/lib/python3.6/site-packages/cobbler/actions/status.py", line 199, in run
    self.scan_logfiles()

  File "/usr/lib/python3.6/site-packages/cobbler/actions/status.py", line 86, in scan_logfiles
    for line in data.split("\n"):
````

### Screenshots

<!--- If applicable, add screenshots to help explain your problem. -->

### Additional information

Python 3.6.8
It looks like data read from gzip.open() need to be decoded first.
Proposed patch below:
[cobbler-3.2.2-log_gzip_decode.txt](https://github.com/cobbler/cobbler/files/9465048/cobbler-3.2.2-log_gzip_decode.txt)


<!--- Add any other context about the problem here. -->



Discussion:
@PunisherHD92 Thanks for your proposed patch. What we can instead is specify that the file should be opend in text-mode. We are already using `gzip.open()`, thus we can make your patch even smaller then it already is. Thanks for the amazingly understandable bug report!
Hello, thanks for replying.
You are correct about the gzip.open option.
Here is another proposed patch.
[cobbler-3.2.2-log_gzip_decode_2.txt](https://github.com/cobbler/cobbler/files/9467659/cobbler-3.2.2-log_gzip_decode_2.txt)

Good. I have come up with the same thing. However I am writing the missing tests atm. I will open a PR once I am finished with that.

Resolving patch:
diff --git a/cobbler/actions/status.py b/cobbler/actions/status.py
--- a/cobbler/actions/status.py
+++ b/cobbler/actions/status.py
@@ -11,14 +11,42 @@
 import gzip
 import re
 import time
+from typing import Dict, List, Union
 
-# ARRAY INDEXES
-MOST_RECENT_START = 0
-MOST_RECENT_STOP = 1
-MOST_RECENT_TARGET = 2
-SEEN_START = 3
-SEEN_STOP = 4
-STATE = 5
+
+class InstallStatus:
+    """
+    Helper class that represents the current state of the installation of a system or profile.
+    """
+
+    def __init__(self):
+        """
+        Default constructor.
+        """
+        self.most_recent_start = -1
+        self.most_recent_stop = -1
+        self.most_recent_target = ""
+        self.seen_start = -1
+        self.seen_stop = -1
+        self.state = "?"
+
+    def __eq__(self, other) -> bool:
+        """
+        Equality function that overrides the default behavior.
+
+        :param other: Other object.
+        :returns: True in case object is of the same type and all attributes are identical. False otherwise.
+        """
+        if isinstance(other, InstallStatus):
+            return (
+                self.most_recent_start == other.most_recent_start
+                and self.most_recent_stop == other.most_recent_stop
+                and self.most_recent_target == other.most_recent_target
+                and self.seen_start == other.seen_start
+                and self.seen_stop == other.seen_stop
+                and self.state == other.state
+            )
+        return False
 
 
 class CobblerStatusReport:
@@ -27,39 +55,47 @@ def __init__(self, api, mode: str):
         Constructor
 
         :param api: The API which holds all information.
-        :param mode: This describes how Cobbler should report. Currently there only the option ``text`` can be set
+        :param mode: This describes how Cobbler should report. Currently, there only the option ``text`` can be set
                      explicitly.
         """
         self.settings = api.settings()
-        self.ip_data = {}
+        self.ip_data: Dict[str, InstallStatus] = {}
         self.mode = mode
 
-    # -------------------------------------------------------
-
-    def scan_logfiles(self):
+    @staticmethod
+    def collect_logfiles() -> List[str]:
         """
-        Scan the install log-files - starting with the oldest file.
+        Collects all installation logfiles from ``/var/log/cobbler/``. This will also collect gzipped logfiles.
+
+        :returns: List of absolute paths that are matching the filepattern ``install.log`` or ``install.log.x``, where
+                  x is a number equal or greater than zero.
         """
         unsorted_files = glob.glob("/var/log/cobbler/install.log*")
-        files_dict = dict()
+        files_dict = {}
         log_id_re = re.compile(r"install.log.(\d+)")
         for fname in unsorted_files:
             id_match = log_id_re.search(fname)
             if id_match:
                 files_dict[int(id_match.group(1))] = fname
 
-        files = list()
+        files = []
         sorted_ids = sorted(files_dict, key=files_dict.get, reverse=True)
         for file_id in sorted_ids:
             files.append(files_dict[file_id])
         if "/var/log/cobbler/install.log" in unsorted_files:
             files.append("/var/log/cobbler/install.log")
 
-        for fname in files:
+        return files
+
+    def scan_logfiles(self):
+        """
+        Scan the installation log-files - starting with the oldest file.
+        """
+        for fname in self.collect_logfiles():
             if fname.endswith(".gz"):
-                fd = gzip.open(fname)
+                fd = gzip.open(fname, "rt")
             else:
-                fd = open(fname)
+                fd = open(fname, "rt")
             data = fd.read()
             for line in data.split("\n"):
                 tokens = line.split()
@@ -69,8 +105,6 @@ def scan_logfiles(self):
                 self.catalog(profile_or_system, name, ip, start_or_stop, ts)
             fd.close()
 
-    # ------------------------------------------------------
-
     def catalog(
         self, profile_or_system: str, name: str, ip, start_or_stop: str, ts: float
     ):
@@ -81,39 +115,35 @@ def catalog(
         :param name: The name of the object.
         :param ip: The ip of the system to watch.
         :param start_or_stop: This parameter may be ``start`` or ``stop``
-        :param ts: Don't know what this does.
+        :param ts: Timestamp as returned by ``time.time()``
         """
-        ip_data = self.ip_data
-
-        if ip not in ip_data:
-            ip_data[ip] = [-1, -1, "?", 0, 0, "?"]
-        elem = ip_data[ip]
+        if ip not in self.ip_data:
+            self.ip_data[ip] = InstallStatus()
+        elem = self.ip_data[ip]
 
         ts = float(ts)
 
-        mrstart = elem[MOST_RECENT_START]
-        mrstop = elem[MOST_RECENT_STOP]
-        mrtarg = elem[MOST_RECENT_TARGET]
+        mrstart = elem.most_recent_start
+        mrstop = elem.most_recent_stop
+        mrtarg = elem.most_recent_target
 
         if start_or_stop == "start":
             if mrstart < ts:
                 mrstart = ts
-                mrtarg = "%s:%s" % (profile_or_system, name)
-                elem[SEEN_START] += 1
+                mrtarg = f"{profile_or_system}:{name}"
+                elem.seen_start += 1
 
         if start_or_stop == "stop":
             if mrstop < ts:
                 mrstop = ts
-                mrtarg = "%s:%s" % (profile_or_system, name)
-                elem[SEEN_STOP] += 1
-
-        elem[MOST_RECENT_START] = mrstart
-        elem[MOST_RECENT_STOP] = mrstop
-        elem[MOST_RECENT_TARGET] = mrtarg
+                mrtarg = f"{profile_or_system}:{name}"
+                elem.seen_stop += 1
 
-    # -------------------------------------------------------
+        elem.most_recent_start = mrstart
+        elem.most_recent_stop = mrstop
+        elem.most_recent_target = mrtarg
 
-    def process_results(self):
+    def process_results(self) -> dict:
         """
         Look through all systems which were collected and update the status.
 
@@ -121,32 +151,31 @@ def process_results(self):
         """
         # FIXME: this should update the times here
         tnow = int(time.time())
-        for ip in list(self.ip_data.keys()):
+        for ip in self.ip_data:
             elem = self.ip_data[ip]
-            start = int(elem[MOST_RECENT_START])
-            stop = int(elem[MOST_RECENT_STOP])
+            start = int(elem.most_recent_start)
+            stop = int(elem.most_recent_stop)
             if stop > start:
-                elem[STATE] = "finished"
+                elem.state = "finished"
             else:
                 delta = tnow - start
-                min = delta // 60
-                sec = delta % 60
-                if min > 100:
-                    elem[STATE] = "unknown/stalled"
+                minutes = delta // 60
+                seconds = delta % 60
+                if minutes > 100:
+                    elem.state = "unknown/stalled"
                 else:
-                    elem[STATE] = "installing (%sm %ss)" % (min, sec)
+                    elem.state = f"installing ({minutes}m {seconds}s)"
 
         return self.ip_data
 
-    def get_printable_results(self):
+    def get_printable_results(self) -> str:
         """
-        Convert the status of Cobbler from a machine readable form to human readable.
+        Convert the status of Cobbler from a machine-readable form to human-readable.
 
         :return: A nice formatted representation of the results of ``cobbler status``.
         """
-        format = "%-15s|%-20s|%-17s|%-17s"
-        ip_data = self.ip_data
-        ips = list(ip_data.keys())
+        printable_status_format = "%-15s|%-20s|%-17s|%-17s"
+        ips = list(self.ip_data.keys())
         ips.sort()
         line = (
             "ip",
@@ -154,20 +183,18 @@ def get_printable_results(self):
             "start",
             "state",
         )
-        buf = format % line
+        buf = printable_status_format % line
         for ip in ips:
-            elem = ip_data[ip]
-            if elem[MOST_RECENT_START] > -1:
-                start = time.ctime(elem[MOST_RECENT_START])
+            elem = self.ip_data[ip]
+            if elem.most_recent_start > -1:
+                start = time.ctime(elem.most_recent_start)
             else:
                 start = "Unknown"
-            line = (ip, elem[MOST_RECENT_TARGET], start, elem[STATE])
-            buf += "\n" + format % line
+            line = (ip, elem.most_recent_target, start, elem.state)
+            buf += "\n" + printable_status_format % line
         return buf
 
-    # -------------------------------------------------------
-
-    def run(self):
+    def run(self) -> Union[dict, str]:
         """
         Calculate and print a automatic installation status report.
         """


Test patch:
diff --git a/tests/actions/status_test.py b/tests/actions/status_test.py
new file mode 100644
--- /dev/null
+++ b/tests/actions/status_test.py
@@ -0,0 +1,123 @@
+import pytest
+
+from cobbler.actions.status import InstallStatus
+from cobbler.actions import status
+
+
+def test_collect_logfiles(mocker):
+    # Arrange
+    mocker.patch(
+        "glob.glob",
+        return_value=[
+            "/var/log/cobbler/install.log",
+            "/var/log/cobbler/install.log1",
+            "/var/log/cobbler/install.log.1",
+        ],
+    )
+    expected_result = [
+        "/var/log/cobbler/install.log.1",
+        "/var/log/cobbler/install.log",
+    ]
+
+    # Act
+    result = status.CobblerStatusReport.collect_logfiles()
+
+    # Assert
+    assert result == expected_result
+
+
+def test_scan_logfiles(mocker, cobbler_api):
+    # Arrange
+    mocker.patch("gzip.open", mocker.mock_open(read_data="test test test test test"))
+    mocker.patch(
+        "builtins.open", mocker.mock_open(read_data="test test test test test")
+    )
+    test_status = status.CobblerStatusReport(cobbler_api, "text")
+    mocker.patch.object(
+        test_status, "collect_logfiles", return_value=["/test/test", "/test/test.gz"]
+    )
+    mock_catalog = mocker.patch.object(test_status, "catalog")
+
+    # Act
+    test_status.scan_logfiles()
+
+    # Assert
+    assert mock_catalog.call_count == 2
+
+
+def test_catalog(cobbler_api):
+    # Arrange
+    test_status = status.CobblerStatusReport(cobbler_api, "text")
+    expected_result = InstallStatus()
+    expected_result.most_recent_start = 0
+    expected_result.most_recent_stop = -1
+    expected_result.most_recent_target = "system:test"
+    expected_result.seen_start = 0
+    expected_result.seen_stop = -1
+    expected_result.state = "?"
+
+    # Act
+    test_status.catalog("system", "test", "192.168.0.1", "start", 0.0)
+
+    # Assert
+    assert "192.168.0.1" in test_status.ip_data
+    assert test_status.ip_data["192.168.0.1"] == expected_result
+
+
+@pytest.mark.parametrize(
+    "input_start,input_stop,expected_status",
+    [
+        (0, 5, "finished"),
+        (50, 0, "unknown/stalled"),
+        (99900, 0, "installing (1m 40s)"),
+    ],
+)
+def test_process_results(mocker, cobbler_api, input_start, input_stop, expected_status):
+    # Arrange
+    mocker.patch("time.time", return_value=100000)
+    test_status = status.CobblerStatusReport(cobbler_api, "text")
+    new_status = InstallStatus()
+    new_status.most_recent_start = input_start
+    new_status.most_recent_stop = input_stop
+    new_status.most_recent_target = ""
+    new_status.seen_start = -1
+    new_status.seen_stop = -1
+    new_status.state = "?"
+    test_status.ip_data["192.168.0.1"] = new_status
+
+    # Act
+    test_status.process_results()
+
+    # Assert
+    assert test_status.ip_data["192.168.0.1"].state == expected_status
+
+
+def test_get_printable_results(cobbler_api):
+    # Arrange
+    test_status = status.CobblerStatusReport(cobbler_api, "text")
+
+    # Act
+    result = test_status.get_printable_results()
+
+    # Assert
+    result_list = result.split("\n")
+    assert len(result_list) == 1
+
+
+@pytest.mark.parametrize(
+    "input_mode,expected_result", [("text", str), ("non-text", dict)]
+)
+def test_run(mocker, cobbler_api, input_mode, expected_result):
+    # Arrange
+    test_status = status.CobblerStatusReport(cobbler_api, input_mode)
+    mocker.patch.object(test_status, "scan_logfiles")
+    if input_mode == "text":
+        mocker.patch.object(test_status, "process_results", return_value="")
+    else:
+        mocker.patch.object(test_status, "process_results", return_value={})
+
+    # Act
+    result = test_status.run()
+
+    # Assert
+    assert isinstance(result, expected_result)