Repository: thelastpickle/cassandra-medusa

Issue:
Listing backups fails on some installations against Azure
[Project board link](https://github.com/orgs/k8ssandra/projects/8/views/1?pane=issue&itemId=32514135)

While we couldn't reproduce the issue, we're seeing installations where backups are taken correctly but listing them fails with the following error:


```
Traceback (most recent call last):
 File "/usr/local/bin/medusa", line 8, in <module>
  sys.exit(cli())
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1128, in __call__
  return self.main(*args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1053, in main
  rv = self.invoke(ctx)
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1659, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1395, in invoke
  return ctx.invoke(self.callback, **ctx.params)
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
  return __callback(*args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 84, in new_func
  return ctx.invoke(f, obj, *args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/click/core.py", line 754, in invoke
  return __callback(*args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/medusa/medusacli.py", line 185, in list_backups
  medusa.listing.list_backups(medusaconfig, show_all)
 File "/usr/local/lib/python3.6/site-packages/medusa/listing.py", line 41, in list_backups
  cluster_backups = get_backups(config, show_all)
 File "/usr/local/lib/python3.6/site-packages/medusa/listing.py", line 29, in get_backups
  key=lambda b: b.started
 File "/usr/local/lib/python3.6/site-packages/medusa/storage/__init__.py", line 355, in list_cluster_backups
  key=lambda b: (b.name, b.started)
 File "/usr/local/lib/python3.6/site-packages/medusa/storage/__init__.py", line 190, in list_node_backups
  blobs_by_backup = self.group_backup_index_by_backup_and_node(backup_index_blobs)
 File "/usr/local/lib/python3.6/site-packages/medusa/storage/__init__.py", line 299, in group_backup_index_by_backup_and_node
  key=name_and_fqdn
 File "/usr/local/lib/python3.6/site-packages/medusa/storage/__init__.py", line 288, in name_and_fqdn
  return get_backup_name(blob), Storage.get_fqdn_from_any_index_blob(blob)
 File "/usr/local/lib/python3.6/site-packages/medusa/storage/__init__.py", line 285, in get_backup_name
  return blob.name.split('/')[2] if len(str(self.prefix_path)) <= 1 else blob.name.split('/')[3]
IndexError: list index out of range
```

After adding some debugging outputs, we see that the folders are returned when listing backups, giving a hierarchical view of the blobs instead of a flat one:

```
[2023-06-29 20:29:25,480] DEBUG: Loading storage_provider: azure_blobs
[2023-06-29 20:29:26,966] DEBUG: [Storage] Listing objects in index/backup_index
[2023-06-29 20:29:26,991] DEBUG: Found backup index blob: index/backup_index
[2023-06-29 20:29:26,991] DEBUG: Found backup index blob: index/backup_index/cassandra_backup_20230628
[2023-06-29 20:29:26,991] DEBUG: Found backup index blob: index/backup_index/cassandra_backup_20230628/differential....
...
```

This is unexpected and not dealt with correctly in the code.
We either need to filter out the folders when listing blobs, which should be made possible thanks to their 0 size.
We could also make the `get_backup_name()` method more resilient to such issues and do a precheck before trying to access the split array indices. If we don't have the index we're looking for, we can issue a warning and skip the file without erroring.

## Definition of Done
- [ ] when listing blobs in the list-backups command, empty files are filtered out
- [ ] `get_backup_name()` will detect blob that aren't named as expected and ignore them

Comments:

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7ef0044..698d56b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -107,7 +107,7 @@ jobs:
         matrix:
           #python-version: [3.6]
           python-version: [3.6, "3.10"]
-          it-backend: [local, s3, gcs, minio, azure]
+          it-backend: [local, s3, gcs, minio, azure, azure-hierarchical]
           # IBM not included by default due to lite plan quota being easily exceeded
           #it-backend: [local, s3, gcs, minio, ibm, azure]
           cassandra-version: [2.2.19, 3.11.11, 4.0.0, 'github:apache/trunk']
@@ -146,7 +146,15 @@ jobs:
               cassandra-version: 'github:apache/trunk'
             - it-backend: azure
               python-version: "3.10"
-            
+            - it-backend: azure-hierarchical
+              cassandra-version: 2.2.19
+            - it-backend: azure-hierarchical
+              cassandra-version: 3.11.11
+            - it-backend: azure-hierarchical
+              cassandra-version: 'github:apache/trunk'
+            - it-backend: azure-hierarchical
+              python-version: "3.10"
+
     runs-on: ubuntu-20.04
     services:
       minio:
@@ -180,7 +188,7 @@ jobs:
         pip install -r requirements-test.txt
         pip install ccm
         case '${{ matrix.it-backend }}' in
-          'azure')
+          'azure'|'azure-hierarchical')
             pip install -r requirements-azure.txt
             ;;
           'ibm'|'minio'|'s3')
@@ -198,7 +206,8 @@ jobs:
           || ( "${{ matrix.it-backend }}" == "minio" ) \
           || ( -n '${{ secrets.MEDUSA_GCS_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "gcs" ) \
           || ( -n '${{ secrets.MEDUSA_IBM_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "ibm" ) \
-          || ( -n '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure" ) ]];
+          || ( -n '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure" ) \
+          || ( -n '${{ secrets.MEDUSA_AZURE_HIERARCHICAL_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure-hierarchical" ) ]];
         then
           echo "IT_CAN_RUN=yes" >> $GITHUB_ENV
         else
@@ -263,6 +272,11 @@ jobs:
           # Azure Blob Storage tests
           printf "%s" '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' > ~/medusa_azure_credentials.json
           ./run_integration_tests.sh -v --azure --no-local --cassandra-version=${{ matrix.cassandra-version }} 
+        elif [ "${{ matrix.it-backend }}" == "azure-hierarchical" ]
+        then
+          # Azure Blob Storage with hierarchical namespace tests
+          printf "%s" '${{ secrets.MEDUSA_AZURE_HIERARCHICAL_CREDENTIALS }}' > ~/medusa_azure_credentials.json
+          ./run_integration_tests.sh -v --azure --no-local --cassandra-version=${{ matrix.cassandra-version }} 
         else
         # Local storage tests
           ./run_integration_tests.sh -v --cassandra-version=${{ matrix.cassandra-version }}
diff --git a/medusa/storage/__init__.py b/medusa/storage/__init__.py
index 9485aa2..30173d4 100644
--- a/medusa/storage/__init__.py
+++ b/medusa/storage/__init__.py
@@ -281,7 +281,8 @@ class Storage(object):
     def group_backup_index_by_backup_and_node(self, backup_index_blobs):
 
         def get_backup_name(blob):
-            return blob.name.split('/')[2] if len(str(self.prefix_path)) <= 1 else blob.name.split('/')[3]
+            blob_name_chunks = blob.name.split('/')
+            return blob_name_chunks[2] if len(str(self.prefix_path)) <= 1 else blob_name_chunks[3]
 
         def name_and_fqdn(blob):
             return get_backup_name(blob), Storage.get_fqdn_from_any_index_blob(blob)
@@ -292,9 +293,20 @@ class Storage(object):
         def group_by_fqdn(blobs):
             return itertools.groupby(blobs, Storage.get_fqdn_from_any_index_blob)
 
+        def has_proper_name(blob):
+            blob_name_chunks = blob.name.split('/')
+            is_proper = len(blob_name_chunks) == 4 if len(str(self.prefix_path)) <= 1 else len(blob_name_chunks) == 5
+            if not is_proper:
+                logging.warning('File {} in backup index has improper name'.format(blob.name))
+            return is_proper
+
         blobs_by_backup = {}
+        properly_named_index_blobs = filter(
+            has_proper_name,
+            backup_index_blobs
+        )
         sorted_backup_index_blobs = sorted(
-            backup_index_blobs,
+            properly_named_index_blobs,
             key=name_and_fqdn
         )
 
diff --git a/medusa/storage/abstract_storage.py b/medusa/storage/abstract_storage.py
index 61c5436..bd887c0 100644
--- a/medusa/storage/abstract_storage.py
+++ b/medusa/storage/abstract_storage.py
@@ -54,6 +54,8 @@ class AbstractStorage(abc.ABC):
         else:
             objects = self.driver.list_container_objects(self.bucket, ex_prefix=str(path))
 
+        objects = list(filter(lambda blob: blob.size > 0, objects))
+
         return objects
 
     @retry(stop_max_attempt_number=7, wait_exponential_multiplier=10000, wait_exponential_max=120000)
diff --git a/medusa/storage/local_storage.py b/medusa/storage/local_storage.py
index 0759d27..57da294 100644
--- a/medusa/storage/local_storage.py
+++ b/medusa/storage/local_storage.py
@@ -41,6 +41,8 @@ class LocalStorage(AbstractStorage):
         if path is not None:
             objects = list(filter(lambda blob: blob.name.startswith(path), objects))
 
+        objects = list(filter(lambda blob: blob.size > 0, objects))
+
         return objects
 
     def get_object_datetime(self, blob):


Test patch:
diff --git a/tests/storage_test.py b/tests/storage_test.py
index c7c0ded..94307d8 100644
--- a/tests/storage_test.py
+++ b/tests/storage_test.py
@@ -92,8 +92,10 @@ class RestoreNodeTest(unittest.TestCase):
     def test_list_objects(self):
         file1_content = "content of the test file1"
         file2_content = "content of the test file2"
+        file3_content = ""
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs1/file1.txt", file1_content)
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs2/file2.txt", file2_content)
+        self.storage.storage_driver.upload_blob_from_string("test_download_blobs3/file3.txt", file3_content)
         objects = self.storage.storage_driver.list_objects()
         self.assertEqual(len(objects), 2)
         one_object = self.storage.storage_driver.list_objects("test_download_blobs2")
@@ -284,6 +286,34 @@ class RestoreNodeTest(unittest.TestCase):
         self.assertTrue("node1" in blobs_by_backup["backup2"])
         self.assertFalse("node2" in blobs_by_backup["backup2"])
 
+    def test_parse_backup_index_with_wrong_names(self):
+        file_content = "content of the test file"
+        prefix_path = self.storage.prefix_path
+
+        # Index files for a backup
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/tokenmap_node1.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/schema_node1.cql".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/started_node1_1689598370.timestamp".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/finished_node1_1689598370.timestamp".format(prefix_path), file_content)
+        # Files that we want to see filtered out
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/extra_folder/backup3/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/missing_folder/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/missing_file".format(prefix_path), file_content)
+
+        path = '{}index/backup_index'.format(prefix_path)
+        backup_index = self.storage.storage_driver.list_objects(path)
+        blobs_by_backup = self.storage.group_backup_index_by_backup_and_node(backup_index)
+        self.assertEqual(1, len(blobs_by_backup.keys()))
+        self.assertEqual(1, len(blobs_by_backup['backup3'].keys()))
+        self.assertEqual(4, len(blobs_by_backup['backup3']['node1']))
+
     def test_remove_extension(self):
         self.assertEqual(
             'localhost',
diff --git a/tests/storage_test_with_prefix.py b/tests/storage_test_with_prefix.py
index 5bf5f73..618548a 100644
--- a/tests/storage_test_with_prefix.py
+++ b/tests/storage_test_with_prefix.py
@@ -93,8 +93,10 @@ class RestoreNodeTest(unittest.TestCase):
     def test_list_objects(self):
         file1_content = "content of the test file1"
         file2_content = "content of the test file2"
+        file3_content = ""
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs1/file1.txt", file1_content)
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs2/file2.txt", file2_content)
+        self.storage.storage_driver.upload_blob_from_string("test_download_blobs3/file3.txt", file3_content)
         objects = self.storage.storage_driver.list_objects()
         self.assertEqual(len(objects), 2)
         one_object = self.storage.storage_driver.list_objects("test_download_blobs2")
@@ -286,6 +288,34 @@ class RestoreNodeTest(unittest.TestCase):
         self.assertTrue("node1" in blobs_by_backup["backup2"])
         self.assertFalse("node2" in blobs_by_backup["backup2"])
 
+    def test_parse_backup_index_with_wrong_names(self):
+        file_content = "content of the test file"
+        prefix_path = self.storage.prefix_path
+
+        # Index files for a backup
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/tokenmap_node1.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/schema_node1.cql".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/started_node1_1689598370.timestamp".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/finished_node1_1689598370.timestamp".format(prefix_path), file_content)
+        # Files that we want to see filtered out
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/extra_folder/backup3/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/missing_folder/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/missing_file".format(prefix_path), file_content)
+
+        path = '{}index/backup_index'.format(prefix_path)
+        backup_index = self.storage.storage_driver.list_objects(path)
+        blobs_by_backup = self.storage.group_backup_index_by_backup_and_node(backup_index)
+        self.assertEqual(1, len(blobs_by_backup.keys()))
+        self.assertEqual(1, len(blobs_by_backup['backup3'].keys()))
+        self.assertEqual(4, len(blobs_by_backup['backup3']['node1']))
+
     def test_remove_extension(self):
         self.assertEqual(
             'localhost',