Repository: cobbler/cobbler

Issue:
Exception value: cannot use a string pattern on a bytes-like object
### Describe the bug

<!--- A clear and concise description of what the bug is. -->
```
Exception value: cannot use a string pattern on a bytes-like object,for each version is wrong
```

VMware-VMvisor-Installer-7.0.0.update01-16850804.x86_64-DellEMC_Customized-A00

<https://aliyun8.com/cobbler3.2_esxi/VMware-VMvisor-Installer-7.0.0.update01-16850804.x86_64-DellEMC_Customized-A00.png>
<https://aliyun8.com/cobbler3.2_esxi/VMware-VMvisor-Installer-7.0.0.update01-16850804.x86_64-DellEMC_Customized-A00_setversion_esxi70.png>


VMware-VMvisor-Installer-6.7.0.update03-16316930.x86_64-DellEMC_Customized-A06

<https://aliyun8.com/cobbler3.2_esxi/VMware-VMvisor-Installer-6.7.0.update03-16316930.x86_64-DellEMC_Customized-A06.png>

VMware-VMvisor-Installer-6.7.0.update03-14320388.x86_64

<https://aliyun8.com/cobbler3.2_esxi/VMware-VMvisor-Installer-6.7.0.update03-14320388.x86_64.png>

<!--- HINT: You can paste gist.github.com links for long logs or larger files -->


### Version

<!--- Paste output from `cobbler version` -->
```
[root@localhost ~]# cat /etc/redhat-release && cobbler version
CentOS Linux release 8.2.2004 (Core) 
Cobbler 3.2.0
  source: ?, ?
  build time: Sun Oct 25 21:46:20 2020
```
```
[root@localhost cobbler]# cobbler signature report
Currently loaded signatures:
debian:
	buster
	jessie
	squeeze
	stretch
	wheezy
freebsd:
	freebsd10.0
	freebsd10.1
	freebsd10.2
	freebsd10.3
	freebsd10.4
	freebsd11.0
	freebsd11.1
	freebsd11.2
	freebsd11.3
	freebsd12.0
	freebsd8.2
	freebsd8.3
	freebsd8.4
	freebsd9.0
	freebsd9.1
	freebsd9.2
	freebsd9.3
generic:
	generic26
	virtio26
powerkvm:
	2.1
redhat:
	cloudlinux6
	fedora16
	fedora17
	fedora18
	fedora19
	fedora20
	fedora21
	fedora22
	fedora23
	fedora24
	fedora25
	fedora26
	fedora27
	fedora28
	fedora29
	fedora30
	fedora31
	fedora32
	ovz7
	rhel4
	rhel5
	rhel6
	rhel7
	rhel8
suse:
	opensuse11.2
	opensuse11.3
	opensuse11.4
	opensuse12.1
	opensuse12.2
	opensuse12.3
	opensuse13.1
	opensuse13.2
	opensuse15.0
	opensuse15.1
	sles10generic
	sles11
	sles11generic
	sles11sp1
	sles11sp2
	sles11sp3
	sles11sp4
	sles12
	sles12generic
	sles12sp1
	sles12sp2
	sles12sp3
	sles12sp4
	sles12sp5
	sles15generic
	suse
ubuntu:
	artful
	bionic
	cosmic
	disco
	eoan
	focal
	lucid
	oneiric
	precise
	quantal
	raring
	saucy
	trusty
	utopic
	vivid
	wily
	xenial
	yakkety
	zesty
unix:
	(none)
vmware:
	esx4
	esxi4
	esxi5
	esxi51
	esxi55
	esxi60
	esxi65
	esxi67
	esxi70
windows:
	(none)
xen:
	xcp16
	xenserver620
	xenserver650
	xenserver700
	xenserver710
	xenserver720

11 breeds with 109 total signatures loaded
```


test esxi6.7X in cobbler-2.8.5-0.3,centos7 is ok



Discussion:
So the problem is that the file which Cobbler thinks is correct for detecting the distro is gziped. Thus the pattern which Cobbler tries is applied to a byte object like the exception is saying. Thus thus distro won't work for now... Due to the fact that this release (3.2.1) is more then full with work, I can't include it. However I would accept PR's for this. Otherwise please have patience.
Hi,
I was able to import esxi modifying modules/managers/import_signatures.py.
In get_file_lines, changing 

```
        if ftype.find("gzip") != -1:
            try:
                import gzip
                f = gzip.open(filename, 'r')
                lines = f.readlines()
                f.close()
            except:
                pass
```
with:
```
        if ftype.find("gzip") != -1:
            try:
                import gzip
                f = gzip.open(filename, 'r')
                while True:
                    line = f.readline()
                    if not line:
                        break
                    lines.add(line.decode(encoding='utf-8'))
                f.close()
            except:
                pass
```
I believe this has something to do with python2 to python3 change.
I am more a sysadmin than a python developer, but maybe this can help @qq8554650 as a temporary workaround.

Cheers,
Javier

@javier-angulo Thanks for that! I will prepare a patch with this later so we can fix this for everyone!
@javier-angulo Would you be able to test a PR I would prepare?
It is not fixed yet...
> It is not fixed yet...

OK.then fix I test it.I saved the problematic environment
```diff
diff --git a/cobbler/modules/managers/import_signatures.py b/cobbler/modules/managers/import_signatures.py
index fe308c4c..4ec8e2ec 100644
--- a/cobbler/modules/managers/import_signatures.py
+++ b/cobbler/modules/managers/import_signatures.py
@@ -19,19 +19,21 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
 02110-1301  USA
 """
 
-from builtins import str
-from builtins import object
 import glob
+import gzip
 import os
 import os.path
 import re
 import shutil
 import stat
 
+import magic
+
 # Import aptsources module if available to obtain repo mirror.
 try:
     from aptsources import distro as debdistro
     from aptsources import sourceslist
+
     apt_available = True
 except:
     apt_available = False
@@ -121,21 +123,19 @@ class ImportSignatureManager(object):
         :param filename: The name of the file to be read.
         :return: An array with all the lines.
         """
-        lines = []
-        ftype = utils.subprocess_get(self.logger, "/usr/bin/file %s" % filename)
-        if ftype.find("gzip") != -1:
+        ftype = magic.detect_from_filename(filename)
+        if ftype.mime_type == "application/gzip":
             try:
-                import gzip
-                f = gzip.open(filename, 'r')
-                lines = f.readlines()
-                f.close()
+                with gzip.open(filename, 'r') as f:
+                    return f.readlines()
             except:
                 pass
-        elif ftype.find("text") != -1:
-            f = open(filename, 'r')
-            lines = f.readlines()
-            f.close()
-        return lines
+        elif ftype.mime_type == "text/plain":
+            with open(filename, 'r') as f:
+                return f.readlines()
+        else:
+            self.logger.info("Could not detect the filetype and read the content. Returning nothing.")
+            return []
 
     def run(self, path, name, network_root=None, autoinstall_file=None, arch=None, breed=None, os_version=None):
         """
```

So please test this diff. It requires the installation of the package `python3-magic` or `pip install file-magic`. The dependency now calls directly to the library and does not use a subprocess call. Which is more secure and cleaner to read and interpret.

Resolving patch:
diff --git a/cobbler/modules/managers/import_signatures.py b/cobbler/modules/managers/import_signatures.py
--- a/cobbler/modules/managers/import_signatures.py
+++ b/cobbler/modules/managers/import_signatures.py
@@ -19,19 +19,21 @@
 02110-1301  USA
 """
 
-from builtins import str
-from builtins import object
 import glob
+import gzip
 import os
 import os.path
 import re
 import shutil
 import stat
 
+import magic
+
 # Import aptsources module if available to obtain repo mirror.
 try:
     from aptsources import distro as debdistro
     from aptsources import sourceslist
+
     apt_available = True
 except:
     apt_available = False
@@ -121,21 +123,20 @@ def get_file_lines(self, filename):
         :param filename: The name of the file to be read.
         :return: An array with all the lines.
         """
-        lines = []
-        ftype = utils.subprocess_get(self.logger, "/usr/bin/file %s" % filename)
-        if ftype.find("gzip") != -1:
+        ftype = magic.detect_from_filename(filename)
+        if ftype.mime_type == "application/gzip":
             try:
-                import gzip
-                f = gzip.open(filename, 'r')
-                lines = f.readlines()
-                f.close()
+                with gzip.open(filename, 'r') as f:
+                    return f.readlines()
             except:
                 pass
-        elif ftype.find("text") != -1:
-            f = open(filename, 'r')
-            lines = f.readlines()
-            f.close()
-        return lines
+        elif ftype.mime_type == "text/plain":
+            with open(filename, 'r') as f:
+                return f.readlines()
+        else:
+            self.logger.info("Could not detect the filetype and read the content of file \"%s\". Returning nothing." %
+                             filename)
+            return []
 
     def run(self, path, name, network_root=None, autoinstall_file=None, arch=None, breed=None, os_version=None):
         """
@@ -231,7 +232,8 @@ def scan_signatures(self):
                                                 break
                                         else:
                                             continue
-                                    self.logger.debug("Found a matching signature: breed=%s, version=%s" % (breed, version))
+                                    self.logger.debug(
+                                        "Found a matching signature: breed=%s, version=%s" % (breed, version))
                                     if not self.breed:
                                         self.breed = breed
                                     if not self.os_version:
@@ -267,9 +269,9 @@ def distro_adder(self, distros_added, dirname, fnames):
         """
         This is an import_walker routine that finds distributions in the directory to be scanned and then creates them.
 
-        :param distros_added: Unkown what this currently does.
-        :param dirname: Unkown what this currently does.
-        :param fnames: Unkown what this currently does.
+        :param distros_added: Unknown what this currently does.
+        :param dirname: Unknown what this currently does.
+        :param fnames: Unknown what this currently does.
         """
 
         re_krn = re.compile(self.signature["kernel_file"])
@@ -349,7 +351,7 @@ def add_entry(self, dirname, kernel, initrd):
             self.logger.error("No arch could be detected in %s, and none was specified via the --arch option" % dirname)
             return []
         elif len(archs) > 1:
-            self.logger.warning("- Warning : Multiple archs found : %s" % (archs))
+            self.logger.warning("- Warning : Multiple archs found : %s" % archs)
 
         distros_added = []
         for pxe_arch in archs:
@@ -514,7 +516,8 @@ def get_proposed_name(self, dirname, kernel=None):
         # remove any architecture name related string, as real arch will be appended later
         name = name.replace("chrp", "ppc64")
         for separator in ['-', '_', '.']:
-            for arch in ["i386", "x86_64", "ia64", "ppc64le", "ppc64el", "ppc64", "ppc32", "ppc", "x86", "s390x", "s390", "386", "amd"]:
+            for arch in ["i386", "x86_64", "ia64", "ppc64le", "ppc64el", "ppc64", "ppc32", "ppc", "x86", "s390x",
+                         "s390", "386", "amd"]:
                 name = name.replace("%s%s" % (separator, arch), "")
 
         return name
@@ -541,8 +544,9 @@ def configure_tree_location(self, distro):
                     os.symlink(base, dest_link)
                 except:
                     # FIXME: This shouldn't happen but I've seen it ... debug ...
-                    self.logger.warning("symlink creation failed: %(base)s, %(dest)s" % {"base": base, "dest": dest_link})
-            tree = "http://@@http_server@@/cblr/links/%s" % (distro.name)
+                    self.logger.warning(
+                        "symlink creation failed: %(base)s, %(dest)s" % {"base": base, "dest": dest_link})
+            tree = "http://@@http_server@@/cblr/links/%s" % distro.name
             self.set_install_tree(distro, tree)
         else:
             # Where we assign the automated installation file source is relative to our current directory and the input
@@ -561,8 +565,8 @@ def set_install_tree(self, distro, url):
         """
         distro.autoinstall_meta["tree"] = url
 
-# ==========================================================================
-# Repo Functions
+    # ==========================================================================
+    # Repo Functions
 
     def repo_finder(self, distros_added):
         """
@@ -655,7 +659,7 @@ def yum_process_comps_file(self, comps_path, distro):
         files = glob.glob("%s/%s/*comps*.xml" % (comps_path, masterdir))
         if len(files) == 0:
             self.logger.info("no comps found here: %s" % os.path.join(comps_path, masterdir))
-            return      # no comps xml file found
+            return  # no comps xml file found
 
         # pull the filename from the longer part
         comps_file = files[0].split("/")[-1]
@@ -673,7 +677,7 @@ def yum_process_comps_file(self, comps_path, distro):
             fname = os.path.join(self.settings.webdir, "distro_mirror", "config", "%s-%s.repo" % (distro.name, counter))
 
             repo_url = "http://@@http_server@@/cobbler/distro_mirror/config/%s-%s.repo" % (distro.name, counter)
-            repo_url2 = "http://@@http_server@@/cobbler/distro_mirror/%s" % (urlseg)
+            repo_url2 = "http://@@http_server@@/cobbler/distro_mirror/%s" % urlseg
 
             distro.source_repos.append([repo_url, repo_url2])
 
@@ -685,14 +689,13 @@ def yum_process_comps_file(self, comps_path, distro):
             # the @@http_server@@ left as templating magic.
             # repo_url2 is actually no longer used. (?)
 
-            config_file = open(fname, "w+")
-            config_file.write("[core-%s]\n" % counter)
-            config_file.write("name=core-%s\n" % counter)
-            config_file.write("baseurl=http://@@http_server@@/cobbler/distro_mirror/%s\n" % (urlseg))
-            config_file.write("enabled=1\n")
-            config_file.write("gpgcheck=0\n")
-            config_file.write("priority=$yum_distro_priority\n")
-            config_file.close()
+            with open(fname, "w+") as config_file:
+                config_file.write("[core-%s]\n" % counter)
+                config_file.write("name=core-%s\n" % counter)
+                config_file.write("baseurl=http://@@http_server@@/cobbler/distro_mirror/%s\n" % urlseg)
+                config_file.write("enabled=1\n")
+                config_file.write("gpgcheck=0\n")
+                config_file.write("priority=$yum_distro_priority\n")
 
             # Don't run creatrepo twice -- this can happen easily for Xen and PXE, when they'll share same repo files.
             if keeprepodata:
@@ -701,7 +704,8 @@ def yum_process_comps_file(self, comps_path, distro):
 
             elif comps_path not in self.found_repos:
                 utils.remove_yum_olddata(comps_path)
-                cmd = "createrepo %s --groupfile %s %s" % (self.settings.createrepo_flags, os.path.join(comps_path, masterdir, comps_file), comps_path)
+                cmd = "createrepo %s --groupfile %s %s" % (
+                    self.settings.createrepo_flags, os.path.join(comps_path, masterdir, comps_file), comps_path)
                 utils.subprocess_call(self.logger, cmd, shell=True)
                 self.found_repos[comps_path] = 1
                 # For older distros, if we have a "base" dir parallel with "repodata", we need to copy comps.xml up
@@ -710,7 +714,6 @@ def yum_process_comps_file(self, comps_path, distro):
                 p2 = os.path.join(comps_path, "base", "comps.xml")
                 if os.path.exists(p1) and os.path.exists(p2):
                     shutil.copyfile(p1, p2)
-
         except:
             self.logger.error("error launching createrepo (not installed?), ignoring")
             utils.log_exc(self.logger)
@@ -737,7 +740,7 @@ def apt_repo_adder(self, distro):
         repo.set_breed("apt")
         repo.set_arch(distro.arch)
         repo.set_keep_updated(True)
-        repo.set_apt_components("main universe")        # TODO: make a setting?
+        repo.set_apt_components("main universe")  # TODO: make a setting?
         repo.set_apt_dists("%s %s-updates %s-security" % ((distro.os_version,) * 3))
         repo.set_name(distro.name)
         repo.set_os_version(distro.os_version)
@@ -767,13 +770,10 @@ def get_repo_mirror_from_apt(self):
             mirrors = release.get_server_list()
             for mirror in mirrors:
                 if mirror[2]:
-                    mirror = mirror[1]
-                    break
+                    return mirror[1]
         except:
             return False
 
-        return mirror
-
     # ==========================================================================
     # rhn-specific
 
@@ -796,6 +796,7 @@ def rsync_repo_adder(self, distro):
         """
         return
 
+
 # ==========================================================================
 
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -524,6 +524,7 @@ def run(self):
             "ldap3",
             "dnspython",
             "tornado",
+            "file-magic"
         ],
         extras_require={"lint": ["pyflakes", "pycodestyle"], "test": ["pytest", "pytest-cov", "codecov"]},
         packages=find_packages(exclude=["*tests*"]),


Test patch: