Repository: CS-SI/eodag (python)

Issue:
NITF files are not downloaded
**Describe the bug**
 I don't know if its intended, but some STAC catalog have.... .ntf files 🙄, and eodag doesn't download them
For example, all [SICD Capella products](https://www.stacindex.org/catalogs/capella-space-open-data#/item/2fCZzY4MUXqhFCKQyCHq5xs9yKEtav6SUMVtn8XotXoG3UL99iGKCe64PF/9mf3J4kA7dTBNBwB2H7AAELDsZgyRK3ZhmtatcuEpa2SJMPBZEqpXGtw18DkzPJyUhETs3tcojWc5YzonkuKMgu17/GaP9p2tVBHPe6Tbc1doYSwBxnLuUAvYQiKvA6dAjuAL4rB5EsFp78hVBQUhyVc97P1F4RxcTyEoZQZafjHKhp2DoSxm7n8pM161rhFwbhULdtxPM8xyAREZbFLy/i1dUGrDnaZaAinXySYJRbutg8rzYXw86fP3EZM1XfJXYRt1jQ1SinMio2tUZKfoh6wTfyqSBdYkaMGVDu8Pjf1ak6Qz2SwoMkYG1vGG82a6ca2Xym4afqaEM3EVVsmWo5pr5ApM86792RP8TWiH2FV2wdC1kLNnhs/MwmMC36qiGXLGqhttVLKJRJ9BSwas3qx4ANc8Me3QETXrNmDqKj5DkRtJCCteNJed3ZNpcTiHPdLtgoEvyWCn2WXuQVUYhJQcuDfeugmzFTDN2ZzWxTqumqz32vEdTHni7FKZ6cY18eJ9oTVo6J2kBevcUM9A8FV8ZPnnFQPdRPtudACqU7dC4cpyCTbd37D7k7VzDLmZ4QzCrucKBtPTPnFNgYMvDdk123dwmCYgBJLMEznP7VTSz8LynuauPiWPFhYzBP76sGu9vjv5SukzuzNuCHSXFGXSq3nFJu?si=0&t=2#12/62.954289/-150.694400).

![2022-12-08_17h29_41](https://user-images.githubusercontent.com/67311115/206507457-84322f0c-45ca-485e-b375-1baa2e384d17.png)



Feel free to close it if this is a feature and not a bug 😉 

**Code To Reproduce**
```py
from eodag import setup_logging

from eodag.api.core import EODataAccessGateway

if __name__ == "__main__":
    # Create an EODAG custom STAC provider
    setup_logging(verbose=3)
    dag = EODataAccessGateway()

    # Add the custom STAC provider + output + login an password
    catalog_path = 'https://capella-open-data.s3.us-west-2.amazonaws.com/stac/capella-open-data-by-datetime/capella-open-data-2022/capella-open-data-2022-3/capella-open-data-2022-3-20/catalog.json'
    outputs_prefix = r"D:\capella_eodag"

    dag.update_providers_config(r"""
    capella:
        search:
            type: StaticStacSearch
            api_endpoint: %s
            metadata_mapping:
                assets: '{$.assets#recursive_sub_str(r"https([^.]*)\.[^/]*\/(.*)",r"s3\1/\2")}'
            discover_product_types:
                fetch_url:
        products:
            GENERIC_PRODUCT_TYPE:
                productType: '{productType}'
        download:
            type: AwsDownload
            flatten_top_dirs: True
            outputs_prefix: %s
    """ % (catalog_path, outputs_prefix))

    # Set the custom STAC provider as preferred
    dag.set_preferred_provider("capella")

    # Query every product from inside the catalog
    capella_prod, _ = dag.search(**{"sensorMode":"spotlight", "sar:product_type": "SICD"})
    paths = dag.download_all(capella_prod)

```

**Output**
```
2022-12-08 17:26:27,628 eodag.config                     [INFO    ] (config           ) Loading user configuration from: C:\Users\rbraun\.config\eodag\eodag.yml
2022-12-08 17:26:27,745 eodag.core                       [INFO    ] (core             ) usgs: provider needing auth for search has been pruned because no crendentials could be found
2022-12-08 17:26:27,745 eodag.core                       [INFO    ] (core             ) aws_eos: provider needing auth for search has been pruned because no crendentials could be found
2022-12-08 17:26:27,841 eodag.core                       [DEBUG   ] (core             ) Opening product types index in C:\Users\rbraun\.config\eodag\.index
2022-12-08 17:26:27,849 eodag.core                       [DEBUG   ] (core             ) Out-of-date product types index removed from C:\Users\rbraun\.config\eodag\.index
2022-12-08 17:26:27,849 eodag.core                       [DEBUG   ] (core             ) Creating product types index in C:\Users\rbraun\.config\eodag\.index
2022-12-08 17:26:27,976 eodag.core                       [INFO    ] (core             ) Locations configuration loaded from C:\Users\rbraun\.config\eodag\locations.yml
2022-12-08 17:26:27,978 eodag.config                     [INFO    ] (config           ) capella: unknown provider found in user conf, trying to use provided configuration
2022-12-08 17:26:27,990 eodag.core                       [INFO    ] (core             ) No product type could be guessed with provided arguments
2022-12-08 17:26:27,990 eodag.core                       [DEBUG   ] (core             ) Fetching external product types sources to find None product type
2022-12-08 17:26:27,991 eodag.config                     [INFO    ] (config           ) Fetching external product types from https://cs-si.github.io/eodag/eodag/resources/ext_product_types.json
2022-12-08 17:26:28,217 eodag.core                       [DEBUG   ] (core             ) Added product type mod11a1 for astraea_eod
2022-12-08 17:26:28,218 eodag.core                       [DEBUG   ] (core             ) Added product type mod13a1 for astraea_eod
2022-12-08 17:26:28,218 eodag.core                       [DEBUG   ] (core             ) Added product type myd11a1 for astraea_eod
2022-12-08 17:26:28,218 eodag.core                       [DEBUG   ] (core             ) Added product type myd13a1 for astraea_eod
2022-12-08 17:26:28,219 eodag.core                       [DEBUG   ] (core             ) Added product type maxar_open_data for astraea_eod
2022-12-08 17:26:28,219 eodag.core                       [DEBUG   ] (core             ) Added product type spacenet7 for astraea_eod
2022-12-08 17:26:28,220 eodag.core                       [DEBUG   ] (core             ) Added product type Sentinel1RTC for creodias
2022-12-08 17:26:28,220 eodag.core                       [DEBUG   ] (core             ) Added product type Sentinel5P for creodias
2022-12-08 17:26:28,220 eodag.core                       [DEBUG   ] (core             ) Added product type Sentinel6 for creodias
2022-12-08 17:26:28,221 eodag.core                       [DEBUG   ] (core             ) Added product type Landsat5 for creodias
2022-12-08 17:26:28,221 eodag.core                       [DEBUG   ] (core             ) Added product type Landsat7 for creodias
2022-12-08 17:26:28,222 eodag.core                       [DEBUG   ] (core             ) Added product type Landsat8 for creodias
2022-12-08 17:26:28,222 eodag.core                       [DEBUG   ] (core             ) Added product type Envisat for creodias
2022-12-08 17:26:28,222 eodag.core                       [DEBUG   ] (core             ) Added product type SMOS for creodias
2022-12-08 17:26:28,223 eodag.core                       [DEBUG   ] (core             ) Added product type S2GLC for creodias
2022-12-08 17:26:28,223 eodag.core                       [DEBUG   ] (core             ) Added product type CopDem for creodias
2022-12-08 17:26:28,224 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2ard-st for usgs_satapi_aws
2022-12-08 17:26:28,224 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2l3-fsca for usgs_satapi_aws
2022-12-08 17:26:28,224 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2ard-bt for usgs_satapi_aws
2022-12-08 17:26:28,225 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2l3-ba for usgs_satapi_aws
2022-12-08 17:26:28,225 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2ard-sr for usgs_satapi_aws
2022-12-08 17:26:28,226 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2l3-dswe for usgs_satapi_aws
2022-12-08 17:26:28,226 eodag.core                       [DEBUG   ] (core             ) Added product type landsat-c2ard-ta for usgs_satapi_aws
2022-12-08 17:26:28,229 eodag.core                       [DEBUG   ] (core             ) Out-of-date product types index removed from C:\Users\rbraun\.config\eodag\.index
2022-12-08 17:26:28,229 eodag.core                       [DEBUG   ] (core             ) Creating product types index in C:\Users\rbraun\.config\eodag\.index
2022-12-08 17:26:29,158 eodag.core                       [INFO    ] (core             ) Searching product type 'None' on provider: capella
2022-12-08 17:26:29,158 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: StaticStacSearch
2022-12-08 17:26:30,004 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) Fetching 8 items
2022-12-08 17:26:30,004 eodag.utils.stac_reader          [DEBUG   ] (stac_reader      ) read_local_json is not the right STAC opener
2022-12-08 17:26:31,008 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting genric provider product type definition parameters for None
2022-12-08 17:26:31,009 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-12-08 17:26:31,009 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-12-08 17:26:31,044 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page URL could not be collected
2022-12-08 17:26:31,061 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page Query-object could not be collected
2022-12-08 17:26:31,078 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Next page merge could not be collected
2022-12-08 17:26:31,078 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 8 plugin results to eodag product representation
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_overlap [DEBUG   ] (filter_overlap   ) Start filtering for overlapping products
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_overlap [WARNING ] (filter_overlap   ) geometry not found in cruncher arguments, filtering disabled.
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sensorMode'], spotlight)
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 4 resulting products
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_property [DEBUG   ] (filter_property  ) Start filtering for products matching operator.eq(product.properties['sar:product_type'], SICD)
2022-12-08 17:26:31,106 eodag.plugins.crunch.filter_property [INFO    ] (filter_property  ) Finished filtering products. 1 resulting products
2022-12-08 17:26:31,109 eodag.core                       [INFO    ] (core             ) Found 1 result(s) on provider 'capella'
2022-12-08 17:26:31,109 eodag.core                       [INFO    ] (core             ) Downloading 1 products
Downloaded products:   0%|          | 0/1 [00:00<?, ?product/s]
0.00B [00:00, ?B/s]
CAPELLA_C05_SP_SICD_HH_20220320114010_20220320114013: 0.00B [00:00, ?B/s]2022-12-08 17:26:31,110 eodag.plugins.download.base      [INFO    ] (base             ) Download url: Not Available
2022-12-08 17:26:32,718 eodag.plugins.download.aws       [DEBUG   ] (aws              ) Auth using _get_authenticated_objects_unsigned succeeded
CAPELLA_C05_SP_SICD_HH_20220320114010_20220320114013: 100%|██████████| 699M/699M [01:20<00:00, 8.73MB/s]
2022-12-08 17:27:53,399 eodag.api.product                [DEBUG   ] (_product         ) Product location updated from 'Not Available' to 'file:///D:/capella_eodag/CAPELLA_C05_SP_SICD_HH_20220320114010_20220320114013'
2022-12-08 17:27:53,399 eodag.api.product                [INFO    ] (_product         ) Remote location of the product is still available through its 'remote_location' property: Not Available
Downloaded products: 100%|██████████| 1/1 [01:22<00:00, 82.29s/product]
```

**Environment:**
 - Python version: `Python 3.9.15`
 - EODAG version: `eodag (Earth Observation Data Access Gateway): version 2.7.0`



Pull request:
fix: fixed flatten top dirs mechanism
fixes #573 

Before this fix, some files could be deleted after download, during the product directory structure modification

Patch:
diff --git a/eodag/plugins/download/aws.py b/eodag/plugins/download/aws.py
index e5d9d2a9..8c28dcce 100644
--- a/eodag/plugins/download/aws.py
+++ b/eodag/plugins/download/aws.py
@@ -19,7 +19,6 @@
 import logging
 import os
 import re
-import shutil
 from pathlib import Path
 
 import boto3
@@ -36,6 +35,7 @@ from eodag.api.product.metadata_mapping import (
 from eodag.plugins.download.base import Download
 from eodag.utils import (
     ProgressCallback,
+    flatten_top_directories,
     get_bucket_name_and_prefix,
     path_to_uri,
     rename_subfolder,
@@ -421,13 +421,7 @@ class AwsDownload(Download):
             self.finalize_s2_safe_product(product_local_path)
         # flatten directory structure
         elif flatten_top_dirs:
-            tmp_product_local_path = "%s-tmp" % product_local_path
-            for d, dirs, files in os.walk(product_local_path):
-                if len(files) != 0:
-                    shutil.copytree(d, tmp_product_local_path)
-                    shutil.rmtree(product_local_path)
-                    os.rename(tmp_product_local_path, product_local_path)
-                    break
+            flatten_top_directories(product_local_path, common_prefix)
 
         if build_safe:
             self.check_manifest_file_list(product_local_path)
diff --git a/eodag/plugins/download/http.py b/eodag/plugins/download/http.py
index 9b9b66ba..e7061532 100644
--- a/eodag/plugins/download/http.py
+++ b/eodag/plugins/download/http.py
@@ -33,7 +33,12 @@ from eodag.plugins.download.base import (
     DEFAULT_STREAM_REQUESTS_TIMEOUT,
     Download,
 )
-from eodag.utils import ProgressCallback, path_to_uri, uri_to_path
+from eodag.utils import (
+    ProgressCallback,
+    flatten_top_directories,
+    path_to_uri,
+    uri_to_path,
+)
 from eodag.utils.exceptions import (
     AuthenticationError,
     MisconfiguredError,
@@ -421,13 +426,7 @@ class HTTPDownload(Download):
 
         # flatten directory structure
         if flatten_top_dirs:
-            tmp_product_local_path = "%s-tmp" % fs_dir_path
-            for d, dirs, files in os.walk(fs_dir_path):
-                if len(files) != 0:
-                    shutil.copytree(d, tmp_product_local_path)
-                    shutil.rmtree(fs_dir_path)
-                    os.rename(tmp_product_local_path, fs_dir_path)
-                    break
+            flatten_top_directories(fs_dir_path)
 
         # save hash/record file
         with open(record_filename, "w") as fh:
diff --git a/eodag/utils/__init__.py b/eodag/utils/__init__.py
index 40234376..b51b5f1e 100644
--- a/eodag/utils/__init__.py
+++ b/eodag/utils/__init__.py
@@ -31,6 +31,7 @@ import json
 import logging as py_logging
 import os
 import re
+import shutil
 import string
 import types
 import unicodedata
@@ -39,6 +40,7 @@ from collections import defaultdict
 from glob import glob
 from itertools import repeat, starmap
 from pathlib import Path
+from tempfile import mkdtemp
 
 # All modules using these should import them from utils package
 from urllib.parse import (  # noqa; noqa
@@ -1058,3 +1060,25 @@ def get_bucket_name_and_prefix(url=None, bucket_path_level=None):
         )
 
     return bucket, prefix
+
+
+def flatten_top_directories(nested_dir_root, common_subdirs_path=None):
+    """Flatten directory structure, removing common empty sub-directories
+
+    :param nested_dir_root: Absolute path of the directory structure to flatten
+    :type nested_dir_root: str
+    :param common_subdirs_path: (optional) Absolute path of the desired subdirectory to remove
+    :type common_subdirs_path: str
+    """
+    if not common_subdirs_path:
+        subpaths_list = [p for p in Path(nested_dir_root).glob("**/*") if p.is_file()]
+        common_subdirs_path = os.path.commonpath(subpaths_list)
+
+    if nested_dir_root != common_subdirs_path:
+        logger.debug(f"Flatten {common_subdirs_path} to {nested_dir_root}")
+        tmp_path = mkdtemp()
+        # need to delete tmp_path to prevent FileExistsError with copytree. Use dirs_exist_ok with py > 3.7
+        shutil.rmtree(tmp_path)
+        shutil.copytree(common_subdirs_path, tmp_path)
+        shutil.rmtree(nested_dir_root)
+        shutil.move(tmp_path, nested_dir_root)


Test patch:
diff --git a/tests/context.py b/tests/context.py
index dd8a54a7..dec2c197 100644
--- a/tests/context.py
+++ b/tests/context.py
@@ -73,6 +73,7 @@ from eodag.utils import (
     parse_qsl,
     urlsplit,
     GENERIC_PRODUCT_TYPE,
+    flatten_top_directories,
 )
 from eodag.utils.exceptions import (
     AddressNotFound,
diff --git a/tests/units/test_utils.py b/tests/units/test_utils.py
index 8616d6c2..04b4ad9b 100644
--- a/tests/units/test_utils.py
+++ b/tests/units/test_utils.py
@@ -16,15 +16,19 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
 import sys
 import unittest
 from contextlib import closing
 from datetime import datetime
 from io import StringIO
+from pathlib import Path
+from tempfile import TemporaryDirectory
 
 from tests.context import (
     DownloadedCallback,
     ProgressCallback,
+    flatten_top_directories,
     get_bucket_name_and_prefix,
     get_timestamp,
     merge_mappings,
@@ -215,3 +219,46 @@ class TestUtils(unittest.TestCase):
             ),
             ("bucket", "path/to/product"),
         )
+
+    def test_flatten_top_dirs(self):
+        """flatten_top_dirs must flatten directory structure"""
+        with TemporaryDirectory() as nested_dir_root:
+            os.makedirs(os.path.join(nested_dir_root, "a", "b", "c1"))
+            os.makedirs(os.path.join(nested_dir_root, "a", "b", "c2"))
+            # create empty files
+            open(os.path.join(nested_dir_root, "a", "b", "c1", "foo"), "a").close()
+            open(os.path.join(nested_dir_root, "a", "b", "c2", "bar"), "a").close()
+            open(os.path.join(nested_dir_root, "a", "b", "c2", "baz"), "a").close()
+
+            flatten_top_directories(nested_dir_root)
+
+            dir_content = list(Path(nested_dir_root).glob("**/*"))
+
+            self.assertEqual(len(dir_content), 5)
+            self.assertIn(Path(nested_dir_root) / "c1", dir_content)
+            self.assertIn(Path(nested_dir_root) / "c1" / "foo", dir_content)
+            self.assertIn(Path(nested_dir_root) / "c2", dir_content)
+            self.assertIn(Path(nested_dir_root) / "c2" / "bar", dir_content)
+            self.assertIn(Path(nested_dir_root) / "c2" / "baz", dir_content)
+
+    def test_flatten_top_dirs_given_subdir(self):
+        """flatten_top_dirs must flatten directory structure using given subdirectory"""
+        with TemporaryDirectory() as nested_dir_root:
+            os.makedirs(os.path.join(nested_dir_root, "a", "b", "c1"))
+            os.makedirs(os.path.join(nested_dir_root, "a", "b", "c2"))
+            # create empty files
+            open(os.path.join(nested_dir_root, "a", "b", "c1", "foo"), "a").close()
+            open(os.path.join(nested_dir_root, "a", "b", "c2", "bar"), "a").close()
+            open(os.path.join(nested_dir_root, "a", "b", "c2", "baz"), "a").close()
+
+            flatten_top_directories(nested_dir_root, os.path.join(nested_dir_root, "a"))
+
+            dir_content = list(Path(nested_dir_root).glob("**/*"))
+
+            self.assertEqual(len(dir_content), 6)
+            self.assertIn(Path(nested_dir_root) / "b", dir_content)
+            self.assertIn(Path(nested_dir_root) / "b" / "c1", dir_content)
+            self.assertIn(Path(nested_dir_root) / "b" / "c1" / "foo", dir_content)
+            self.assertIn(Path(nested_dir_root) / "b" / "c2", dir_content)
+            self.assertIn(Path(nested_dir_root) / "b" / "c2" / "bar", dir_content)
+            self.assertIn(Path(nested_dir_root) / "b" / "c2" / "baz", dir_content)