Repository: cobbler/cobbler

Issue:
repo config files not deleted on distro delete
cobbler/modules/manage_import_signatures.py::yum_repo_adder -> yum_repo_scanner -> yum_process_comps_file creates files like:

```
        fname = os.path.join(self.settings.webdir, "distro_mirror", "config", "%s-%s.repo" % (distro.name, counter))
```

but cobbler/collection_distros.py::remove does not remove those files.



Discussion:
I did run into this issue as well with Cobbler 2.6.11 yesterday.

I have made a cobbler distro copy - and then changed initrd, kernel and ksmeta   values for the new distro.
Still the Rsyncd.conf -file had a path pointing to the old distro - and that's what was synced out to all the site-specific cobbler-servers.
I deleted the distros, and made a cobbler sync - but the faulty distros were still present in the rsync config. Took me a while to find out that the distros in the rsync-file is "built" from what's in /var/www/cobbler/links
After deleting the bad symlinks there - I could create new distros with the add-command that are working just fine.
But these links should really be managed both while copying/edit - but definitly when deleting the distro.
 
Examples:
````
cobbler distro copy --name=CentOS-7.2-x86_64 --newname=CentOS-7.3-x86_64
cobbler distro edit --name=CentOS-7.3-x86_64 --initrd="/var/www/cobbler/ks_mirror/CentOS-7.3-x86_64/images/pxeboot/initrd.img"
cobbler distro edit --name=CentOS-7.3-x86_64 --kernel="/var/www/cobbler/ks_mirror/CentOS-7.3-x86_64/images/pxeboot/vmlinuz"
..
cobbler sync
````
results in this snippet in `/etc/rsyncd.conf`
````
[distro-CentOS-7.3-x86_64]
    path    = /raid/cobbler/ks_mirror/CentOS-7.2-x86_64
    comment = Cobbler Distro CentOS-7.3-x86_64
````
````
ll /var/www/cobbler/links/
lrwxrwxrwx 1 root root 44 Dec 16 02:18 CentOS-7.3-x86_64 -> /var/www/cobbler/ks_mirror/CentOS-7.2-x86_64
````

As the link is not removed - it will not update it either. - and everything from within cobbler looks just fine. (which makes it really hard to find.)
Also in Cobbler 3.1.1.
Distro named "Test" will leave these behind when deleted.
```
/var/www/cobbler/images/Test
/var/www/cobbler/images/Test/initrd.gz
/var/www/cobbler/images/Test/linux
/var/www/cobbler/links/Test
```
@loreson :
Can you have a look at hbokh recent comment:

/var/www/cobbler/images/*/*

It would be great if you can have a closer look at possibilities to link linux/initrd if possible...
This is rather important for us for rolling releases like Tumbleweed or if in devel phase.
@loreson @watologo1 Will work on this happen or can I unassign him/you again?
@hbokh I was not able to reproduce this on `master` on openSUSE Leap 15.2. 

```shell
❯ mount
(...)
/home/dom/CentOS-8.3.2011-x86_64-dvd1.iso on /home/dom/centos_dvd type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048)

❯ cobbler version
Cobbler 3.2.0
  source: 4eef9800, Thu Mar 4 08:41:33 2021 +0100
  build time: Thu Mar  4 12:11:46 2021

~
❯ cobbler import --name=test --path=/home/dom/centos_dvd
task started: 2021-03-04_131800_import
task started (id=Media import, time=Thu Mar  4 13:18:00 2021)
running python triggers from /var/lib/cobbler/triggers/task/import/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/import/pre/*
shell triggers finished successfully
Found a candidate signature: breed=redhat, version=rhel8
Found a matching signature: breed=redhat, version=rhel8
Adding distros from path /srv/www/cobbler/distro_mirror/test:
creating new distro: test-x86_64
trying symlink: /srv/www/cobbler/distro_mirror/test -> /srv/www/cobbler/links/test-x86_64
creating new profile: test-x86_64
associating repos
checking for rsync repo(s)
checking for rhn repo(s)
checking for yum repo(s)
starting descent into /srv/www/cobbler/distro_mirror/test for test-x86_64
processing repo at : /srv/www/cobbler/distro_mirror/test/AppStream
need to process repo/comps: /srv/www/cobbler/distro_mirror/test/AppStream
looking for /srv/www/cobbler/distro_mirror/test/AppStream/repodata/*comps*.xml
Keeping repodata as-is :/srv/www/cobbler/distro_mirror/test/AppStream/repodata
processing repo at : /srv/www/cobbler/distro_mirror/test/BaseOS
need to process repo/comps: /srv/www/cobbler/distro_mirror/test/BaseOS
looking for /srv/www/cobbler/distro_mirror/test/BaseOS/repodata/*comps*.xml
Keeping repodata as-is :/srv/www/cobbler/distro_mirror/test/BaseOS/repodata
*** TASK COMPLETE ***

~ 48s
❯ cobbler distro list                                   
   test-x86_64

~
❯ cobbler profile list
   test-x86_64

~
❯ ll /srv/www/cobbler/distro_mirror 
total 0
drwxr-xr-x 1 root root  72 2021-03-04 13:18 config
dr-xr-xr-x 1 root root 152 2020-11-18 22:39 test

~
❯ ll /srv/www/cobbler/links        
total 4.0K
lrwxrwxrwx 1 root root 35 2021-03-04 13:18 test-x86_64 -> /srv/www/cobbler/distro_mirror/test

~
❯ cobbler distro copy --name=test-x86_64 --newname=test2-x86_64

~
❯ cobbler distro list                                          
   test-x86_64
   test2-x86_64

~
❯ ll /srv/www/cobbler/distro_mirror 
total 0
drwxr-xr-x 1 root root  72 2021-03-04 13:18 config
dr-xr-xr-x 1 root root 152 2020-11-18 22:39 test

~
❯ ll /srv/www/cobbler/links        
total 8.0K
lrwxrwxrwx 1 root root 35 2021-03-04 13:20 test2-x86_64 -> /srv/www/cobbler/distro_mirror/test
lrwxrwxrwx 1 root root 35 2021-03-04 13:18 test-x86_64 -> /srv/www/cobbler/distro_mirror/test

~
❯ cobbler distro remove --name=test-x86_64
exception on server: 'removal would orphan profile: test-x86_64'

~
❯ cobbler profile remove --name=test-x86_64

~
❯ cobbler distro remove --name=test-x86_64 

~
❯ cobbler distro list                     
   test2-x86_64

~
❯ ll /srv/www/cobbler/distro_mirror 
total 0
drwxr-xr-x 1 root root  72 2021-03-04 13:18 config
dr-xr-xr-x 1 root root 152 2020-11-18 22:39 test

~
❯ ll /srv/www/cobbler/links        
total 4.0K
lrwxrwxrwx 1 root root 35 2021-03-04 13:20 test2-x86_64 -> /srv/www/cobbler/distro_mirror/test

~
❯ cobbler distro remove --name=test2-x86_64

~
❯ ll /srv/www/cobbler/distro_mirror        
total 0
drwxr-xr-x 1 root root 72 2021-03-04 13:18 config

~
❯ ll /srv/www/cobbler/links                
total 0

~
```
When removing only `test-x86_64`, the folder `/srv/www/cobbler/distro_mirror/test` still exists due to the existing link of the copy `test2-x86_64`. The link for `test-x86_64` was correctly removed. After removing `test2-x86_64` the folder `/srv/www/cobbler/distro_mirror/test` was also deleted.
So I think the behaviour of `cobbler distro remove` seems to be correct. When editing the created copy like @JohanElmis did above, though, those changes are not correctly stored within the folders of the copy. But this is due to storing the new files inside `/srv/www/cobbler` which will get deleted when doing a `cobbler sync`. When storing the files outside the cobbler folder it works as expected.

@hbokh Could you do a quick test on `master`, please?
> @hbokh Could you do a quick test on `master`, please?

This has been over a year...
Unfortunately I don't have the tests / infra available anymore to do this **quick**. I have to set that all up again, which I am not planning to ATM. Sorry...
@hbokh No problem. Then we check at a later point. The next release will take a while definitively...
I did some quick testing on master and the behaviour is a little bit different now:

```bash
$ cobbler profile remove --name=RockyLinux-x86_64
$ cobbler distro remove --name=RockyLinux-x86_64
$ cobbler sync

task started: 2021-05-10_103450_sync
task started (id=Sync, time=Mon May 10 10:34:50 2021)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
syncing all
running pre-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/sync/pre/*
shell triggers finished successfully
cleaning trees
removing: /srv/www/cobbler/images/RockyLinux-x86_64
removing: /srv/tftpboot/pxelinux.cfg/default
removing: /srv/tftpboot/grub/images
removing: /srv/tftpboot/grub/system
removing: /srv/tftpboot/grub/system_link
removing: /srv/tftpboot/grub/grub.cfg
removing: /srv/tftpboot/grub/local_efi.cfg
removing: /srv/tftpboot/grub/local_legacy.cfg
removing: /srv/tftpboot/grub/local_powerpc-ieee1275.cfg
removing: /srv/tftpboot/grub/x86_64_menu_items.cfg
removing: /srv/tftpboot/images/RockyLinux-x86_64
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup', '/var/lib/cobbler/loaders/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig', '/var/lib/cobbler/grub_config/', '/srv/tftpboot']
received on stdout: 
received on stderr: 
copying distros to tftpboot
copying images
generating PXE configuration files
generating PXE menu structure
mkdir: /srv/tftpboot/pxelinux.cfg
mkdir: /srv/tftpboot/pxelinux.cfg
cleaning link caches
running post-sync triggers
running python triggers from /var/lib/cobbler/triggers/sync/post/*
running python trigger cobbler.modules.sync_post_restart_services
running shell triggers from /var/lib/cobbler/triggers/sync/post/*
shell triggers finished successfully
running python triggers from /var/lib/cobbler/triggers/change/*
running python trigger cobbler.modules.scm_track
running python trigger cobbler.modules.managers.genders
running shell triggers from /var/lib/cobbler/triggers/change/*
shell triggers finished successfully
*** TASK COMPLETE ***


$ cobbler report
distros:
==========

profiles:
==========

systems:
==========

repos:
==========

images:
==========

mgmtclasses:
==========

packages:
==========

files:
==========


$ cd /srv/tftpboot
$ tree -L 2
.
├── boot
├── etc
├── grub
│   ├── grub.cfg
│   ├── images -> ../images
│   ├── local_efi.cfg
│   ├── local_legacy.cfg
│   ├── local_powerpc-ieee1275.cfg
│   ├── system
│   └── system_link
├── grub.cfg
├── images
├── images2
├── ppc
├── pxelinux.cfg
│   └── default
└── s390x

11 directories, 6 files


$ cd /srv/www/cobbler
$  tree -L 3
.
├── distro_mirror
│   └── config
│       ├── RockyLinux-x86_64-0.repo
│       └── RockyLinux-x86_64-1.repo
├── images
├── links
│   └── RockyLinux-x86_64 -> /srv/www/cobbler/distro_mirror/RockyLinux
├── localmirror
├── misc
│   ├── anamon
│   └── anamon.init
├── pub
├── rendered
├── repo_mirror
└── svc
    └── services.py

10 directories, 6 files
```
There is a leftover `.repo` file from an earlier Rocky Linux import which was not removed, too after removing the earlier import a few days ago.

Resolving patch:
diff --git a/cobbler/actions/sync.py b/cobbler/actions/sync.py
--- a/cobbler/actions/sync.py
+++ b/cobbler/actions/sync.py
@@ -71,6 +71,8 @@ def __init__(self, collection_mgr, verbose: bool = True, dhcp=None, dns=None, tf
         self.images_dir = os.path.join(self.bootloc, "images")
         self.ipxe_dir = os.path.join(self.bootloc, "ipxe")
         self.rendered_dir = os.path.join(self.settings.webdir, "rendered")
+        self.links = os.path.join(self.settings.webdir, "links")
+        self.distromirror_config = os.path.join(self.settings.webdir, "distro_mirror/config")
         # FIXME: See https://github.com/cobbler/cobbler/issues/2453
         # Move __create_tftpboot_dirs() outside of sync.py.
         self.__create_tftpboot_dirs()
@@ -194,6 +196,10 @@ def __create_tftpboot_dirs(self):
             utils.mkdir(self.rendered_dir)
         if not os.path.exists(self.ipxe_dir):
             utils.mkdir(self.ipxe_dir)
+        if not os.path.exists(self.links):
+            utils.mkdir(self.links)
+        if not os.path.exists(self.distromirror_config):
+            utils.mkdir(self.distromirror_config)
 
     def clean_trees(self):
         """
@@ -353,6 +359,8 @@ def remove_single_distro(self, name):
         utils.rmtree(os.path.join(bootloc, "images", name))
         # delete potential symlink to tree in webdir/links
         utils.rmfile(os.path.join(self.settings.webdir, "links", name))
+        # delete potential distro config files
+        utils.rmglob_files(os.path.join(self.settings.webdir, "distro_mirror", "config"), name + "*.repo")
 
     def remove_single_image(self, name):
         """
diff --git a/cobbler/utils.py b/cobbler/utils.py
--- a/cobbler/utils.py
+++ b/cobbler/utils.py
@@ -37,6 +37,7 @@
 import urllib.parse
 import urllib.request
 from functools import reduce
+from pathlib import Path
 from typing import List, Optional, Pattern, Union
 
 import distro
@@ -1225,24 +1226,19 @@ def copyfile_pattern(pattern, dst, require_match: bool = True, symlink_ok: bool
         linkfile(file, dst1, symlink_ok=symlink_ok, cache=cache, api=api)
 
 
-def rmfile(path: str) -> bool:
+def rmfile(path: str):
     """
     Delete a single file.
 
     :param path: The file to delete.
-    :return: True if the action succeeded.
-    :raises CX
     """
     try:
-        logger.info("removing: %s", path)
-        os.unlink(path)
-        return True
-    except OSError as ioe:
-        # doesn't exist
-        if not ioe.errno == errno.ENOENT:
-            log_exc()
-            raise CX("Error deleting %s" % path)
-        return True
+        os.remove(path)
+        logger.info("Successfully removed \"%s\"", path)
+    except FileNotFoundError:
+        logger.warning("File to remove not found \"%s\"", path)
+    except OSError:
+        logger.warning("Could not remove file \"%s\"", path)
 
 
 def rmtree_contents(path: str):
@@ -1264,6 +1260,7 @@ def rmtree(path: str) -> Optional[bool]:
     :return: May possibly return true on success or may return None on success.
     :raises CX
     """
+    # TODO: Obsolete bool return value
     try:
         if os.path.isfile(path):
             return rmfile(path)
@@ -1277,6 +1274,17 @@ def rmtree(path: str) -> Optional[bool]:
         return True
 
 
+def rmglob_files(path: str, glob_pattern: str):
+    """
+    Deletes all files in ``path`` with ``glob_pattern`` with the help of ``rmfile()``.
+
+    :param path: The folder of the files to remove.
+    :param glob_pattern: The glob pattern for the files to remove in ``path``.
+    """
+    for p in Path(path).glob(glob_pattern):
+        rmfile(str(p))
+
+
 def mkdir(path, mode=0o755):
     """
     Create directory with a given mode.


Test patch:
diff --git a/tests/utils_test.py b/tests/utils_test.py
--- a/tests/utils_test.py
+++ b/tests/utils_test.py
@@ -508,19 +508,28 @@ def test_copyfile_pattern():
     assert False
 
 
-def test_rmfile():
+def test_rmfile(tmpdir: Path):
     # Arrange
-    filepath = "/dev/shm/testfile"
-    Path(filepath).touch()
+    tfile = tmpdir.join("testfile")
+
+    # Act
+    utils.rmfile(tfile)
+
+    # Assert
+    assert not os.path.exists(tfile)
 
-    assert os.path.exists(filepath)
+
+def test_rmglob_files(tmpdir: Path):
+    # Arrange
+    tfile1 = tmpdir.join("file1.tfile")
+    tfile2 = tmpdir.join("file2.tfile")
 
     # Act
-    result = utils.rmfile(filepath)
+    utils.rmglob_files(tmpdir, "*.tfile")
 
     # Assert
-    assert result
-    assert not os.path.exists(filepath)
+    assert not os.path.exists(tfile1)
+    assert not os.path.exists(tfile2)
 
 
 def test_rmtree_contents():