Repository: thelastpickle/cassandra-medusa

Issue:
Add `--expected-size` to the awscli cp commands
The awscli will use a default chunk size of 8MB but only authorizes 10000 chunks per multipart upload.
This will cause files bigger than 78GB to fail being uploaded.

We need to add the `--expected-size <size>` argument to allow the cli to autotune the chunk size and keep the number of chunks below the threshold.



[Project board link](https://github.com/orgs/k8ssandra/projects/8/views/1?pane=issue&itemId=18355355)

Discussion:
github-actions[bot]: No linked issues found. Please add the corresponding issues in the pull request description. <br/> [Use GitHub automation to close the issue when a PR is merged](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) <!-- metadata = {"action":"linked_issue"} -->
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) Report
> Merging [#596](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) (2740447) into [master](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/commit/ec425ec37f42cb5434644e9b84a353b5f49f3842?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) (ec425ec) will **increase** coverage by `0.38%`.
> The diff coverage is `87.50%`.

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596/graphs/tree.svg?width=650&height=150&src=pr&token=KTDCRD82NU&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)

```diff
@@            Coverage Diff             @@
##           master     #596      +/-   ##
==========================================
+ Coverage   80.56%   80.94%   +0.38%     
==========================================
  Files          58       58              
  Lines        4558     4597      +39     
  Branches      678      684       +6     
==========================================
+ Hits         3672     3721      +49     
+ Misses        860      849      -11     
- Partials       26       27       +1     
```


| [Impacted Files](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) | Coverage Δ | |
|---|---|---|
| [medusa/storage/s3\_compat\_storage/awscli.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvczNfY29tcGF0X3N0b3JhZ2UvYXdzY2xpLnB5) | `92.23% <87.50%> (-0.40%)` | :arrow_down: |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)

</details>
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) Report
> Merging [#596](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) (2740447) into [master](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/commit/ec425ec37f42cb5434644e9b84a353b5f49f3842?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) (ec425ec) will **increase** coverage by `0.38%`.
> The diff coverage is `87.50%`.

<details><summary>Additional details and impacted files</summary>


[![Impacted file tree graph](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596/graphs/tree.svg?width=650&height=150&src=pr&token=KTDCRD82NU&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)

```diff
@@            Coverage Diff             @@
##           master     #596      +/-   ##
==========================================
+ Coverage   80.56%   80.94%   +0.38%     
==========================================
  Files          58       58              
  Lines        4558     4597      +39     
  Branches      678      684       +6     
==========================================
+ Hits         3672     3721      +49     
+ Misses        860      849      -11     
- Partials       26       27       +1     
```


| [Impacted Files](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) | Coverage Δ | |
|---|---|---|
| [medusa/storage/s3\_compat\_storage/awscli.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvczNfY29tcGF0X3N0b3JhZ2UvYXdzY2xpLnB5) | `92.23% <87.50%> (-0.40%)` | :arrow_down: |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/596/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)

</details>
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)


sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=thelastpickle_cassandra-medusa&pullRequest=596)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=thelastpickle_cassandra-medusa&pullRequest=596&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=thelastpickle_cassandra-medusa&pullRequest=596&metric=new_duplicated_lines_density&view=list)



Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7ef0044..698d56b 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -107,7 +107,7 @@ jobs:
         matrix:
           #python-version: [3.6]
           python-version: [3.6, "3.10"]
-          it-backend: [local, s3, gcs, minio, azure]
+          it-backend: [local, s3, gcs, minio, azure, azure-hierarchical]
           # IBM not included by default due to lite plan quota being easily exceeded
           #it-backend: [local, s3, gcs, minio, ibm, azure]
           cassandra-version: [2.2.19, 3.11.11, 4.0.0, 'github:apache/trunk']
@@ -146,7 +146,15 @@ jobs:
               cassandra-version: 'github:apache/trunk'
             - it-backend: azure
               python-version: "3.10"
-            
+            - it-backend: azure-hierarchical
+              cassandra-version: 2.2.19
+            - it-backend: azure-hierarchical
+              cassandra-version: 3.11.11
+            - it-backend: azure-hierarchical
+              cassandra-version: 'github:apache/trunk'
+            - it-backend: azure-hierarchical
+              python-version: "3.10"
+
     runs-on: ubuntu-20.04
     services:
       minio:
@@ -180,7 +188,7 @@ jobs:
         pip install -r requirements-test.txt
         pip install ccm
         case '${{ matrix.it-backend }}' in
-          'azure')
+          'azure'|'azure-hierarchical')
             pip install -r requirements-azure.txt
             ;;
           'ibm'|'minio'|'s3')
@@ -198,7 +206,8 @@ jobs:
           || ( "${{ matrix.it-backend }}" == "minio" ) \
           || ( -n '${{ secrets.MEDUSA_GCS_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "gcs" ) \
           || ( -n '${{ secrets.MEDUSA_IBM_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "ibm" ) \
-          || ( -n '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure" ) ]];
+          || ( -n '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure" ) \
+          || ( -n '${{ secrets.MEDUSA_AZURE_HIERARCHICAL_CREDENTIALS }}' && "${{ matrix.it-backend }}" == "azure-hierarchical" ) ]];
         then
           echo "IT_CAN_RUN=yes" >> $GITHUB_ENV
         else
@@ -263,6 +272,11 @@ jobs:
           # Azure Blob Storage tests
           printf "%s" '${{ secrets.MEDUSA_AZURE_CREDENTIALS }}' > ~/medusa_azure_credentials.json
           ./run_integration_tests.sh -v --azure --no-local --cassandra-version=${{ matrix.cassandra-version }} 
+        elif [ "${{ matrix.it-backend }}" == "azure-hierarchical" ]
+        then
+          # Azure Blob Storage with hierarchical namespace tests
+          printf "%s" '${{ secrets.MEDUSA_AZURE_HIERARCHICAL_CREDENTIALS }}' > ~/medusa_azure_credentials.json
+          ./run_integration_tests.sh -v --azure --no-local --cassandra-version=${{ matrix.cassandra-version }} 
         else
         # Local storage tests
           ./run_integration_tests.sh -v --cassandra-version=${{ matrix.cassandra-version }}
diff --git a/medusa/storage/__init__.py b/medusa/storage/__init__.py
index 9485aa2..30173d4 100644
--- a/medusa/storage/__init__.py
+++ b/medusa/storage/__init__.py
@@ -281,7 +281,8 @@ class Storage(object):
     def group_backup_index_by_backup_and_node(self, backup_index_blobs):
 
         def get_backup_name(blob):
-            return blob.name.split('/')[2] if len(str(self.prefix_path)) <= 1 else blob.name.split('/')[3]
+            blob_name_chunks = blob.name.split('/')
+            return blob_name_chunks[2] if len(str(self.prefix_path)) <= 1 else blob_name_chunks[3]
 
         def name_and_fqdn(blob):
             return get_backup_name(blob), Storage.get_fqdn_from_any_index_blob(blob)
@@ -292,9 +293,20 @@ class Storage(object):
         def group_by_fqdn(blobs):
             return itertools.groupby(blobs, Storage.get_fqdn_from_any_index_blob)
 
+        def has_proper_name(blob):
+            blob_name_chunks = blob.name.split('/')
+            is_proper = len(blob_name_chunks) == 4 if len(str(self.prefix_path)) <= 1 else len(blob_name_chunks) == 5
+            if not is_proper:
+                logging.warning('File {} in backup index has improper name'.format(blob.name))
+            return is_proper
+
         blobs_by_backup = {}
+        properly_named_index_blobs = filter(
+            has_proper_name,
+            backup_index_blobs
+        )
         sorted_backup_index_blobs = sorted(
-            backup_index_blobs,
+            properly_named_index_blobs,
             key=name_and_fqdn
         )
 
diff --git a/medusa/storage/abstract_storage.py b/medusa/storage/abstract_storage.py
index 61c5436..bd887c0 100644
--- a/medusa/storage/abstract_storage.py
+++ b/medusa/storage/abstract_storage.py
@@ -54,6 +54,8 @@ class AbstractStorage(abc.ABC):
         else:
             objects = self.driver.list_container_objects(self.bucket, ex_prefix=str(path))
 
+        objects = list(filter(lambda blob: blob.size > 0, objects))
+
         return objects
 
     @retry(stop_max_attempt_number=7, wait_exponential_multiplier=10000, wait_exponential_max=120000)
diff --git a/medusa/storage/local_storage.py b/medusa/storage/local_storage.py
index 0759d27..57da294 100644
--- a/medusa/storage/local_storage.py
+++ b/medusa/storage/local_storage.py
@@ -41,6 +41,8 @@ class LocalStorage(AbstractStorage):
         if path is not None:
             objects = list(filter(lambda blob: blob.name.startswith(path), objects))
 
+        objects = list(filter(lambda blob: blob.size > 0, objects))
+
         return objects
 
     def get_object_datetime(self, blob):
diff --git a/medusa/storage/s3_compat_storage/awscli.py b/medusa/storage/s3_compat_storage/awscli.py
index 3d035a3..cc570a1 100644
--- a/medusa/storage/s3_compat_storage/awscli.py
+++ b/medusa/storage/s3_compat_storage/awscli.py
@@ -77,19 +77,34 @@ class AwsCli(object):
     def cmd():
         return [sys.executable, '-m', 'awscli']
 
+    # method to get current file size
+    @staticmethod
+    def get_file_size(file_path):
+        return os.stat(file_path).st_size
+
+    def _make_cp_cmd(self, src, bucket_name, dest):
+
+        cmd = self._create_s3_cmd()
+        cmd.extend(["s3", "cp"])
+
+        actual_size = self.get_file_size(src)
+
+        cmd.extend(["--expected-size", str(actual_size)])
+
+        if self._config.kms_id is not None:
+            cmd.extend(["--sse", "aws:kms", "--sse-kms-key-id", self._config.kms_id])
+
+        cmd.extend([str(src), "s3://{}/{}".format(bucket_name, dest)])
+
+        return cmd
+
     def cp_upload(self, *, srcs, bucket_name, dest, max_retries=5):
         job_id = str(uuid.uuid4())
         awscli_output = "/tmp/awscli_{0}.output".format(job_id)
+
         objects = []
         for src in srcs:
-            cmd = self._create_s3_cmd()
-            cmd.extend(["s3", "cp"])
-
-            if self._config.kms_id is not None:
-                cmd.extend(["--sse", "aws:kms", "--sse-kms-key-id", self._config.kms_id])
-
-            cmd.extend([str(src), "s3://{}/{}".format(bucket_name, dest)])
-
+            cmd = self._make_cp_cmd(src, bucket_name, dest)
             objects.append(self.upload_file(cmd, dest, awscli_output))
 
         return objects


Test patch:
diff --git a/tests/storage/awscli_test.py b/tests/storage/awscli_test.py
new file mode 100644
index 0000000..9825d78
--- /dev/null
+++ b/tests/storage/awscli_test.py
@@ -0,0 +1,55 @@
+import os
+import unittest
+from unittest import mock
+from unittest.mock import MagicMock
+from medusa.storage.s3_compat_storage.awscli import AwsCli
+
+
+class StorageMock:
+    def __init__(self):
+        self.config = MagicMock()
+        self.config.kms_id = None
+        self.config.secure = True
+
+
+class TestAwsCli(unittest.TestCase):
+
+    def create_test_file(self):
+        with open('test_file.txt', 'w') as test_file:
+            test_file.write('a' * 5001)
+        return 'test_file.txt'
+
+    def test_get_file_size(self):
+        # Arrange
+        instance = AwsCli(StorageMock())
+
+        # Act
+        test_file_size = instance.get_file_size(self.create_test_file())
+
+        # Assert
+        self.assertEqual(test_file_size, 5001)
+
+    @mock.patch.object(AwsCli, '_create_s3_cmd')
+    def test_make_cp_cmd(self, mock_create_s3_cmd):
+        # Arrange
+        instance = AwsCli(StorageMock())
+        mock_create_s3_cmd.return_value = ['awscli']
+        src = self.create_test_file()
+        bucket_name = "test_bucket"
+        dest = "test_dest"
+        actual_size = instance.get_file_size(src)
+
+        expected_cmd = ['awscli', 's3', 'cp', '--expected-size',
+                        str(actual_size), 'test_file.txt', 's3://test_bucket/test_dest']
+        # Act
+        actual_cmd = instance._make_cp_cmd(src, bucket_name, dest)
+        # Assert
+        self.assertEqual(expected_cmd, actual_cmd)
+
+    def tearDown(self):
+        if os.path.exists('test_file.txt'):
+            os.remove('test_file.txt')
+
+
+if __name__ == '__main__':
+    unittest.main()
diff --git a/tests/storage_test.py b/tests/storage_test.py
index c7c0ded..94307d8 100644
--- a/tests/storage_test.py
+++ b/tests/storage_test.py
@@ -92,8 +92,10 @@ class RestoreNodeTest(unittest.TestCase):
     def test_list_objects(self):
         file1_content = "content of the test file1"
         file2_content = "content of the test file2"
+        file3_content = ""
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs1/file1.txt", file1_content)
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs2/file2.txt", file2_content)
+        self.storage.storage_driver.upload_blob_from_string("test_download_blobs3/file3.txt", file3_content)
         objects = self.storage.storage_driver.list_objects()
         self.assertEqual(len(objects), 2)
         one_object = self.storage.storage_driver.list_objects("test_download_blobs2")
@@ -284,6 +286,34 @@ class RestoreNodeTest(unittest.TestCase):
         self.assertTrue("node1" in blobs_by_backup["backup2"])
         self.assertFalse("node2" in blobs_by_backup["backup2"])
 
+    def test_parse_backup_index_with_wrong_names(self):
+        file_content = "content of the test file"
+        prefix_path = self.storage.prefix_path
+
+        # Index files for a backup
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/tokenmap_node1.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/schema_node1.cql".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/started_node1_1689598370.timestamp".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/finished_node1_1689598370.timestamp".format(prefix_path), file_content)
+        # Files that we want to see filtered out
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/extra_folder/backup3/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/missing_folder/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/missing_file".format(prefix_path), file_content)
+
+        path = '{}index/backup_index'.format(prefix_path)
+        backup_index = self.storage.storage_driver.list_objects(path)
+        blobs_by_backup = self.storage.group_backup_index_by_backup_and_node(backup_index)
+        self.assertEqual(1, len(blobs_by_backup.keys()))
+        self.assertEqual(1, len(blobs_by_backup['backup3'].keys()))
+        self.assertEqual(4, len(blobs_by_backup['backup3']['node1']))
+
     def test_remove_extension(self):
         self.assertEqual(
             'localhost',
diff --git a/tests/storage_test_with_prefix.py b/tests/storage_test_with_prefix.py
index 5bf5f73..618548a 100644
--- a/tests/storage_test_with_prefix.py
+++ b/tests/storage_test_with_prefix.py
@@ -93,8 +93,10 @@ class RestoreNodeTest(unittest.TestCase):
     def test_list_objects(self):
         file1_content = "content of the test file1"
         file2_content = "content of the test file2"
+        file3_content = ""
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs1/file1.txt", file1_content)
         self.storage.storage_driver.upload_blob_from_string("test_download_blobs2/file2.txt", file2_content)
+        self.storage.storage_driver.upload_blob_from_string("test_download_blobs3/file3.txt", file3_content)
         objects = self.storage.storage_driver.list_objects()
         self.assertEqual(len(objects), 2)
         one_object = self.storage.storage_driver.list_objects("test_download_blobs2")
@@ -286,6 +288,34 @@ class RestoreNodeTest(unittest.TestCase):
         self.assertTrue("node1" in blobs_by_backup["backup2"])
         self.assertFalse("node2" in blobs_by_backup["backup2"])
 
+    def test_parse_backup_index_with_wrong_names(self):
+        file_content = "content of the test file"
+        prefix_path = self.storage.prefix_path
+
+        # Index files for a backup
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/tokenmap_node1.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/schema_node1.cql".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/started_node1_1689598370.timestamp".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/backup3/finished_node1_1689598370.timestamp".format(prefix_path), file_content)
+        # Files that we want to see filtered out
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/extra_folder/backup3/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/missing_folder/tokenmap_node2.json".format(prefix_path), file_content)
+        self.storage.storage_driver.upload_blob_from_string(
+            "{}index/backup_index/missing_file".format(prefix_path), file_content)
+
+        path = '{}index/backup_index'.format(prefix_path)
+        backup_index = self.storage.storage_driver.list_objects(path)
+        blobs_by_backup = self.storage.group_backup_index_by_backup_and_node(backup_index)
+        self.assertEqual(1, len(blobs_by_backup.keys()))
+        self.assertEqual(1, len(blobs_by_backup['backup3'].keys()))
+        self.assertEqual(4, len(blobs_by_backup['backup3']['node1']))
+
     def test_remove_extension(self):
         self.assertEqual(
             'localhost',