Repository: cobbler/cobbler

Issue:
Missing grub folders during cobbler sync
### Description
cobbler sync fails due to missing folders that seems to be deleted at some point in the code

Edit: the rsync failed quietly

### Cobbler Version
```
Cobbler 3.0.1
  source: b'527ed449', b'Thu Nov 7 09:11:18 2019 +0100'
  build time: Mon Nov 11 18:05:42 2019
```

### Operating System

CentOS 8

### Output
```
[root@master cobbler]# cobbler sync
task started: 2019-11-11_130706_sync
task started (id=Sync, time=Mon Nov 11 13:07:06 2019)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
running pre-sync triggers
cleaning trees
removing: /var/lib/tftpboot/pxelinux.cfg/01-f0-4d-a2-06-83-c5
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/images/centos8-x86_64
removing: /var/lib/tftpboot/images/centos8r-x86_64
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup /var/lib/cobbler/loaders/', '/var/lib/tftpboot']
received on stdout: drwxr-xr-x             90 2019/11/11 13:05:43 tftpboot
-rw-r--r--            429 2019/10/16 21:42:57 tftpboot/grub.cfg
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/etc
drwxr-xr-x             20 2019/11/11 13:03:00 tftpboot/grub
drwxr-xr-x             25 2019/11/11 13:07:07 tftpboot/grub/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/grub/images/.link_cache
drwxr-xr-x             25 2019/11/11 13:07:07 tftpboot/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/images/.link_cache
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/ppc
drwxr-xr-x              6 2019/11/11 13:07:07 tftpboot/pxelinux.cfg

received on stderr: 
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig /var/lib/cobbler/grub_config/', '/var/lib/tftpboot']
received on stdout: drwxr-xr-x             90 2019/11/11 13:05:43 tftpboot
-rw-r--r--            429 2019/10/16 21:42:57 tftpboot/grub.cfg
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/etc
drwxr-xr-x             20 2019/11/11 13:03:00 tftpboot/grub
drwxr-xr-x             25 2019/11/11 13:07:07 tftpboot/grub/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/grub/images/.link_cache
drwxr-xr-x             25 2019/11/11 13:07:07 tftpboot/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/images/.link_cache
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/ppc
drwxr-xr-x              6 2019/11/11 13:07:07 tftpboot/pxelinux.cfg

received on stderr: 
copying distros to tftpboot
copying files for distro: centos8-x86_64
trying hardlink /var/www/cobbler/distro_mirror/centos8-x86_64/images/pxeboot/vmlinuz -> /var/lib/tftpboot/images/centos8-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/centos8-x86_64/images/pxeboot/initrd.img -> /var/lib/tftpboot/images/centos8-x86_64/initrd.img
copying files for distro: centos8r-x86_64
trying hardlink /var/www/cobbler/distro_mirror/centos8r/images/pxeboot/vmlinuz -> /var/lib/tftpboot/images/centos8r-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/centos8r/images/pxeboot/initrd.img -> /var/lib/tftpboot/images/centos8r-x86_64/initrd.img
copying images
generating PXE configuration files
generating: /var/lib/tftpboot/pxelinux.cfg/01-f0-4d-a2-06-83-c5
generating: /var/lib/tftpboot/grub/system/f0:4d:a2:06:83:c5
Exception occured: <class 'FileNotFoundError'>
Exception value: [Errno 2] No such file or directory: '/var/lib/tftpboot/grub/system/f0:4d:a2:06:83:c5'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 84, in run
    rc = self._run(self)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 185, in runner
    self.remote.api.sync(self.options.get("verbose", False), logger=self.logger)

  File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 719, in sync
    sync.run()

  File "/usr/lib/python3.6/site-packages/cobbler/actions/sync.py", line 100, in run
    self.tftpd.sync(self.verbose)

  File "/usr/lib/python3.6/site-packages/cobbler/modules/managers/in_tftpd.py", line 166, in sync
    self.tftpgen.write_all_system_files(x, menu_items)

  File "/usr/lib/python3.6/site-packages/cobbler/tftpgen.py", line 238, in write_all_system_files
    self.write_pxe_file(grub_path, system, profile, distro, working_arch, format="grub")

  File "/usr/lib/python3.6/site-packages/cobbler/tftpgen.py", line 539, in write_pxe_file
    fd = open(filename, "w")

!!! TASK FAILED !!!
```

Maybe it's a packaging error. Let's create the folder then:

```
[root@master cobbler]# mkdir -p /var/lib/tftpboot/grub/system
[root@master cobbler]# mkdir -p /var/lib/tftpboot/grub/system_link
[root@master cobbler]# cobbler sync
task started: 2019-11-11_130741_sync
task started (id=Sync, time=Mon Nov 11 13:07:41 2019)
running python triggers from /var/lib/cobbler/triggers/task/sync/pre/*
running shell triggers from /var/lib/cobbler/triggers/task/sync/pre/*
shell triggers finished successfully
running pre-sync triggers
cleaning trees
removing: /var/lib/tftpboot/pxelinux.cfg/01-f0-4d-a2-06-83-c5
removing: /var/lib/tftpboot/grub/images
removing: /var/lib/tftpboot/grub/system
removing: /var/lib/tftpboot/grub/system_link
removing: /var/lib/tftpboot/images/centos8-x86_64
removing: /var/lib/tftpboot/images/centos8r-x86_64
copying bootloaders
running: ['rsync', '-rpt', '--copy-links', '--exclude=.cobbler_postun_cleanup /var/lib/cobbler/loaders/', '/var/lib/tftpboot']
received on stdout: drwxr-xr-x             90 2019/11/11 13:05:43 tftpboot
-rw-r--r--            429 2019/10/16 21:42:57 tftpboot/grub.cfg
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/etc
drwxr-xr-x             20 2019/11/11 13:07:42 tftpboot/grub
drwxr-xr-x             25 2019/11/11 13:07:42 tftpboot/grub/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/grub/images/.link_cache
drwxr-xr-x             25 2019/11/11 13:07:42 tftpboot/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/images/.link_cache
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/ppc
drwxr-xr-x              6 2019/11/11 13:07:42 tftpboot/pxelinux.cfg

received on stderr: 
running: ['rsync', '-rpt', '--copy-links', '--exclude=README.grubconfig /var/lib/cobbler/grub_config/', '/var/lib/tftpboot']
received on stdout: drwxr-xr-x             90 2019/11/11 13:05:43 tftpboot
-rw-r--r--            429 2019/10/16 21:42:57 tftpboot/grub.cfg
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/etc
drwxr-xr-x             20 2019/11/11 13:07:42 tftpboot/grub
drwxr-xr-x             25 2019/11/11 13:07:42 tftpboot/grub/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/grub/images/.link_cache
drwxr-xr-x             25 2019/11/11 13:07:42 tftpboot/images
drwxr-xr-x              6 2019/10/23 16:16:18 tftpboot/images/.link_cache
drwxr-xr-x              6 2019/10/17 15:35:40 tftpboot/ppc
drwxr-xr-x              6 2019/11/11 13:07:42 tftpboot/pxelinux.cfg

received on stderr: 
copying distros to tftpboot
copying files for distro: centos8-x86_64
trying hardlink /var/www/cobbler/distro_mirror/centos8-x86_64/images/pxeboot/vmlinuz -> /var/lib/tftpboot/images/centos8-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/centos8-x86_64/images/pxeboot/initrd.img -> /var/lib/tftpboot/images/centos8-x86_64/initrd.img
copying files for distro: centos8r-x86_64
trying hardlink /var/www/cobbler/distro_mirror/centos8r/images/pxeboot/vmlinuz -> /var/lib/tftpboot/images/centos8r-x86_64/vmlinuz
trying hardlink /var/www/cobbler/distro_mirror/centos8r/images/pxeboot/initrd.img -> /var/lib/tftpboot/images/centos8r-x86_64/initrd.img
copying images
generating PXE configuration files
generating: /var/lib/tftpboot/pxelinux.cfg/01-f0-4d-a2-06-83-c5
generating: /var/lib/tftpboot/grub/system/f0:4d:a2:06:83:c5
Exception occured: <class 'FileNotFoundError'>
Exception value: [Errno 2] No such file or directory: '/var/lib/tftpboot/grub/system/f0:4d:a2:06:83:c5'
Exception Info:
  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 84, in run
    rc = self._run(self)

  File "/usr/lib/python3.6/site-packages/cobbler/remote.py", line 185, in runner
    self.remote.api.sync(self.options.get("verbose", False), logger=self.logger)

  File "/usr/lib/python3.6/site-packages/cobbler/api.py", line 719, in sync
    sync.run()

  File "/usr/lib/python3.6/site-packages/cobbler/actions/sync.py", line 100, in run
    self.tftpd.sync(self.verbose)

  File "/usr/lib/python3.6/site-packages/cobbler/modules/managers/in_tftpd.py", line 166, in sync
    self.tftpgen.write_all_system_files(x, menu_items)

  File "/usr/lib/python3.6/site-packages/cobbler/tftpgen.py", line 238, in write_all_system_files
    self.write_pxe_file(grub_path, system, profile, distro, working_arch, format="grub")

  File "/usr/lib/python3.6/site-packages/cobbler/tftpgen.py", line 539, in write_pxe_file
    fd = open(filename, "w")

!!! TASK FAILED !!!
```

Nope, something is always wiping out the folders.
```
$ rsync -rvpt --copy-links --exclude=README.grubconfig /var/lib/cobbler/grub_config/ /var/lib/tftpboot
sending incremental file list
grub/
grub/grub.cfg
grub/local_efi.cfg
grub/local_legacy.cfg
grub/local_powerpc-ieee1275.cfg
grub/system/
grub/system_link/

sent 6,357 bytes  received 104 bytes  12,922.00 bytes/sec
total size is 6,395  speedup is 0.99

```
The rsync works, so it must be something afterwards...


Discussion:
Well the cobbler sync test is generally working so it has to be something which we are not testing or it is an environmental problem. Since I would say that you already know cobbler quite good I would go for the first guess...

What my gut tells me that I am not used to the fact that there is a `system` folder in the tftp directory.
@SchoolGuy You mean this test? https://github.com/cobbler/cobbler/blob/master/tests/cli/cobbler_cli_direct_test.py#L54
@SchoolGuy found the issue, creating PR...

Resolving patch:
diff --git a/cobbler/tftpgen.py b/cobbler/tftpgen.py
--- a/cobbler/tftpgen.py
+++ b/cobbler/tftpgen.py
@@ -65,9 +65,9 @@ def copy_bootloaders(self):
         # unfortunately using shutils copy_tree the dest directory must not exist,
         # but we must not delete an already partly synced /srv/tftp dir here.
         # rsync is very convenient here, being very fast on an already copied folder
-        utils.subprocess_call(self.logger, ["rsync", "-rpt", "--copy-links", "--exclude=.cobbler_postun_cleanup {src}/".format(src=src), dest], shell=False)
+        utils.subprocess_call(self.logger, ["rsync", "-rpt", "--copy-links", "--exclude=.cobbler_postun_cleanup", "{src}/".format(src=src), dest], shell=False)
         src = self.settings.grubconfig_dir
-        utils.subprocess_call(self.logger, ["rsync", "-rpt", "--copy-links", "--exclude=README.grubconfig {src}/".format(src=src), dest], shell=False)
+        utils.subprocess_call(self.logger, ["rsync", "-rpt", "--copy-links", "--exclude=README.grubconfig", "{src}/".format(src=src), dest], shell=False)
 
     def copy_images(self):
         """


Test patch: