Repository: CS-SI/eodag
Pull request #1033 (2024-02-27 17:08:30)

feat: configurable assets filtering
Adds a new metadata-mapping convert method `convert_dict_filter()` to help filtering assets.
Method used to update providers configuration:
- `earth_search` only lists `s3` assets
- `earth_search_cog` only lists `http` assets

github-actions[bot]: ## Test Results
  1 files   -     3    1 suites   - 3   1m 10s :stopwatch: - 3m 33s
472 tests +    1  466 :white_check_mark:  -     2  3 :zzz: ± 0  3 :x: +3 
472 runs   - 1 412  466 :white_check_mark:  - 1 336  3 :zzz:  - 79  3 :x: +3 

For more details on these failures, see [this check](https://github.com/CS-SI/eodag/runs/22041925342).

Results for commit 3c45df72. ± Comparison against base commit b800cacb.

[test-results]:data:application/gzip;base64,H4sIAMUX3mUC/1WMSw7CIBQAr9KwdlF5FKyXMfAoyYttMXxWxrsLNZZ2OTPJvJmjeYrs3l0vHYuZ0g42B53IrwVVX7iUVJtQ/E+PmBGrkrKpJ72Kgl04TfNJTCH4UEydhry2Z4Xz8mfacePDcOPjD/2yUCrAAMVgneIDAI4oQKLkIJSBXnEzolbSWmdujn2+qzZ4MwQBAAA=

github-actions[bot]: ![badge](https://raw.githubusercontent.com/CS-SI/eodag/_xml_coverage_reports/data/assets-filtering/./badge.svg)


## Code Coverage (Ubuntu)

<details>


```
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      27  91.35%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      74  90.00%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                641      84  86.90%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 556-557, 581-582, 588-591, 606-607, 771, 871-872, 915-920, 1044, 1058-1078, 1098, 1103, 1213, 1235, 1249, 1262-1281, 1320, 1372, 1410-1414, 1433
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       8  74.19%   26-29, 93, 97, 162, 207
plugins/apis/cds.py                            204      46  77.45%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 514, 516
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      54  79.15%   57-60, 141, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       402      92  77.11%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 966, 984-1000
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      54  89.49%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1353
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290      36  87.59%   78-79, 104, 127-128, 237-239, 296-297, 309-325, 424-429, 464, 618-625, 661
rest/stac.py                                   417      78  81.29%   57-59, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 395, 430-431, 609, 791, 856-858, 1077, 1089-1101, 1114-1138, 1152-1199, 1358-1359
rest/utils.py                                  392      63  83.93%   76-78, 202-217, 241, 273, 299, 306, 316, 327-330, 338-339, 368, 500, 518, 546, 548, 581-584, 735-748, 757, 784-799, 824, 914, 933, 941, 1098-1099, 1102, 1104, 1107, 1110-1112, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      25  70.93%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165
rest/types/stac_queryables.py                   38       7  81.58%   28, 120-126, 129
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8289    1518  81.69%
```

</details>


### Diff against develop

```
Filename                           Stmts    Miss  Cover
-------------------------------  -------  ------  -------
api/product/metadata_mapping.py      +14       0  +0.30%
TOTAL                                +14       0  +0.03%
```


Results for commit: 62956fe633e3d4173046e897453f6a0650fd23bd


_Minimum allowed coverage is `70%`_


:recycle: This comment has been updated with latest results


<!-- Sticky Pull Request Commentartifacts/unit-test-results-python3.12-ubuntu-latest/coverage.xml -->
github-actions[bot]: ![badge](https://raw.githubusercontent.com/CS-SI/eodag/_xml_coverage_reports_win/data/assets-filtering/./badge.svg)


## Code Coverage (Windows)

<details>


```
Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      312      28  91.03%   80-82, 91, 99, 103-105, 178, 190, 385-387, 451-454, 501-502, 511-512, 591, 625, 660-665, 667
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    740      78  89.46%   100-108, 381, 592, 637-640, 678, 781, 785-790, 816, 886, 1128-1140, 1180, 1182, 1186, 1207-1209, 1213-1224, 1237-1243, 1333-1336, 1365-1385, 1433, 1439-1442, 1451, 1832-1838, 2101, 2105-2108, 2122-2124, 2166-2173, 2189, 2199, 2231, 2234
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          39       3  92.31%   27, 77, 124
api/product/_product.py                        200      30  85.00%   46-51, 55-57, 161-168, 252-253, 343, 379, 440, 454-457, 470, 490-493, 534-540
api/product/metadata_mapping.py                641      85  86.74%   64-66, 127-129, 230, 262-263, 315-327, 329, 340, 344-360, 399-402, 439, 460-463, 477, 556-557, 581-582, 588-591, 606-607, 771, 871-872, 915-920, 1044, 1058-1078, 1098, 1103, 1213, 1235, 1249, 1262-1281, 1320, 1372, 1395, 1410-1414, 1433
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             130      11  91.54%   49-51, 93-98, 142, 271-272
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                            31       9  70.97%   26-29, 93, 97, 110, 162, 207
plugins/apis/cds.py                            204      53  74.02%   71-74, 122-125, 172-184, 240, 305, 313-318, 331-338, 356-358, 382-384, 394-396, 410-411, 456, 459, 469, 481, 506-521
plugins/apis/ecmwf.py                           94      12  87.23%   48-51, 152-154, 199-200, 226-228
plugins/apis/usgs.py                           165      34  79.39%   59-61, 122, 195, 229, 264-266, 271, 297-298, 303, 331-338, 347-352, 375-381, 383-389, 412
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              82      17  79.27%   32-34, 149-150, 180-201, 227-232
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py        99      60  39.39%   34-36, 133-145, 149-167, 175-209, 215-224, 233-273, 278-286, 291-292
plugins/authentication/qsauth.py                35       2  94.29%   32, 80
plugins/authentication/sas_auth.py              47       2  95.74%   32, 71
plugins/authentication/token.py                 75       8  89.33%   35-37, 102, 127, 154-157
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        478     160  66.53%   74-79, 276, 343-346, 360-364, 407-409, 413, 441-442, 448-452, 482, 550-558, 562, 600-608, 615-617, 658-732, 750-811, 822-827, 843-856, 885, 900-902, 905, 915-923, 931-944, 954-973, 980-992, 1033, 1059, 1104-1106, 1326
plugins/download/base.py                       259      57  77.99%   57-60, 141, 176, 244-246, 313-314, 334-340, 371-375, 381-382, 426, 429-443, 455, 459, 532-536, 566-567, 575-592, 599-607, 609-613, 660, 683, 705, 713
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       402      92  77.11%   68-73, 107, 151, 171-178, 188-201, 234-239, 289, 321, 348-350, 360-368, 427-431, 488, 563-615, 628, 652-654, 684-685, 693, 696, 712, 736, 758-759, 766, 827-833, 883-884, 890, 900, 966, 984-1000
plugins/download/s3rest.py                     111      24  78.38%   53-54, 158, 185, 215-222, 225-227, 231, 244-250, 258-259, 262-266, 289, 309-312
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                          91      11  87.91%   43-44, 91, 95, 121-125, 252, 272
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   52       3  94.23%   55, 73, 104
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          190      59  68.95%   52, 87-90, 106, 118, 137, 143, 149-150, 163-166, 220-221, 225, 235-241, 246, 270, 280-290, 301, 303, 310-311, 313-314, 330-334, 367, 374, 385, 398, 404-416, 421
plugins/search/qssearch.py                     514      78  84.82%   73, 340-341, 435-441, 468-471, 543-544, 603, 618, 671, 692, 695-696, 705, 716, 725, 736-737, 748, 808-813, 817-818, 846, 909, 928-945, 980, 1114-1115, 1136, 1178-1179, 1189-1195, 1232, 1245, 1265, 1313-1382
plugins/search/static_stac_search.py            46       3  93.48%   39-40, 81
rest/__init__.py                                 1       0  100.00%
rest/server.py                                 290     290  0.00%    18-792
rest/stac.py                                   417     134  67.87%   57-59, 211, 226-228, 246, 273, 302, 306-315, 334-340, 368-370, 391, 395, 430-431, 609, 732-733, 791, 856-858, 875-877, 885-887, 900-902, 916-933, 943-964, 974-996, 1004-1021, 1044-1067, 1077, 1089-1101, 1114-1138, 1152-1199, 1352-1379
rest/utils.py                                  392     140  64.29%   76-78, 202-217, 241, 273, 299, 302-322, 327-330, 338-339, 368, 500, 518, 544, 546, 548, 575, 581-584, 605, 725-810, 817-825, 914, 928, 933, 935-942, 949, 962-1006, 1097-1127, 1132-1136
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                      86      29  66.28%   99-109, 115-117, 123-125, 133-140, 146-148, 154-155, 160-165, 170-173
rest/types/stac_queryables.py                   38      13  65.79%   28, 53-58, 120-126, 129, 133-134
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-60, 71-73, 84-86, 98-100, 112-114, 122
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              470      42  91.06%   76-78, 99-101, 179-180, 189-216, 219, 233, 315-319, 395-399, 420-422, 504, 509, 519, 557-558, 954-957, 965-966, 1007-1008, 1169, 1187, 1360
utils/constraints.py                           120      40  66.67%   81-90, 131, 136, 140, 151, 174, 185-193, 202, 216-232, 241-252
utils/exceptions.py                             36       1  97.22%   23
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            89      27  69.66%   54-55, 62-83, 90-92, 96, 135, 149-152
TOTAL                                         8289    1987  76.03%
```

</details>


### Diff against develop

```
Filename                           Stmts    Miss  Cover
-------------------------------  -------  ------  -------
api/product/metadata_mapping.py      +14       0  +0.30%
TOTAL                                +14       0  +0.04%
```


Results for commit: 62956fe633e3d4173046e897453f6a0650fd23bd


_Minimum allowed coverage is `70%`_


:recycle: This comment has been updated with latest results


<!-- Sticky Pull Request Commentartifacts/unit-test-results-python3.12-windows-latest/coverage.xml -->

Patch:
diff --git a/eodag/api/product/metadata_mapping.py b/eodag/api/product/metadata_mapping.py
index f63f3a0d..8a6fe9c4 100644
--- a/eodag/api/product/metadata_mapping.py
+++ b/eodag/api/product/metadata_mapping.py
@@ -513,6 +513,27 @@ def format_metadata(search_param: str, *args: Tuple[Any], **kwargs: Any) -> str:
 
             return dict(input_dict, **new_items_dict)
 
+        @staticmethod
+        def convert_dict_filter(
+            input_dict: Dict[Any, Any], jsonpath_filter_str: str
+        ) -> Dict[Any, Any]:
+            """Fitlers dict items using jsonpath"""
+
+            jsonpath_filter = string_to_jsonpath(jsonpath_filter_str, force=True)
+            if isinstance(jsonpath_filter, str) or not isinstance(input_dict, dict):
+                return {}
+
+            keys_list = list(input_dict.keys())
+            matches = jsonpath_filter.find(input_dict)
+            result = {}
+            for match in matches:
+                # extract key index from matched jsonpath
+                matched_jsonpath_str = str(match.full_path)
+                matched_index = int(matched_jsonpath_str.split(".")[-1][1:-1])
+                key = keys_list[matched_index]
+                result[key] = match.value
+            return result
+
         @staticmethod
         def convert_slice_str(string: str, args: str) -> str:
             cmin, cmax, cstep = [x.strip() for x in args.split(",")]
diff --git a/eodag/resources/providers.yml b/eodag/resources/providers.yml
index 08eff9ea..23ad297d 100644
--- a/eodag/resources/providers.yml
+++ b/eodag/resources/providers.yml
@@ -1770,6 +1770,7 @@
       tileIdentifier:
         - '{{"query":{{"mgrs:utm_zone":{{"eq":"{tileIdentifier#slice_str(0,2,1)}"}},"mgrs:latitude_band":{{"eq":"{tileIdentifier#slice_str(2,3,1)}"}},"mgrs:grid_square":{{"eq":"{tileIdentifier#slice_str(3,5,1)}"}}}}}}'
         - '{utmZone}{latitudeBand}{gridSquare}'
+      assets: '{$.assets#dict_filter($[?(href=~"^s3.*")])}'
   products:
     S1_SAR_GRD:
       productType: sentinel-1-grd
@@ -1862,6 +1863,7 @@
       platformSerialIdentifier: '$.id.`split(_, 0, -1)`'
       polarizationMode: '$.id.`sub(/.{14}([A-Z]{2}).*/, \\1)`'
       title: '{$.properties."s2:product_uri"#remove_extension}'
+      assets: '{$.assets#dict_filter($[?(href=~"^http.*")])}'
   products:
     S2_MSI_L2A_COG:
       productType: sentinel-2-l2a


Test patch:
diff --git a/tests/units/test_metadata_mapping.py b/tests/units/test_metadata_mapping.py
index 78aa672a..385c1045 100644
--- a/tests/units/test_metadata_mapping.py
+++ b/tests/units/test_metadata_mapping.py
@@ -239,6 +239,16 @@ class TestMetadataFormatter(unittest.TestCase):
             "{'a': {'title': 'foo'}, 'b': {'href': 'bar', 'title': 'baz'}}",
         )
 
+    def test_convert_dict_filter(self):
+        to_format = '{fieldname#dict_filter($[?(href=~"^s3.*")])}'
+        self.assertEqual(
+            format_metadata(
+                to_format,
+                fieldname={"a": {"href": "https://foo"}, "b": {"href": "s3://bar"}},
+            ),
+            "{'b': {'href': 's3://bar'}}",
+        )
+
     def test_convert_slice_str(self):
         to_format = "{fieldname#slice_str(1,12,2)}"
         self.assertEqual(