Repository: dcs4cop/xcube
Issue: xcube serve Datasets configuration to use s3 resources

**Describe the bug**
When configuring Datasets via xcube serve not using the DataStore framework, s3 resources cause errors. 

**To Reproduce**
Steps to reproduce the behavior:
Use the configuration: 

```yaml

base_dir: s3://xcube-viewer-app/ws/dev

Viewer:
  Configuration:
    Path: viewer

Datasets:
  - Identifier: app-l2c-cyanoalert-olci-balt-v2
    Path: water-services/Sweden/app-l2c-cyanoalert-olci-balt-Sweden-v2.zarr
    Title: Sverige app v2 - limited temporal length
    FileSystem: s3
    Endpoint: "https://s3.eu-central-1.amazonaws.com"
    Anonymous: false
    Style: default

  - Identifier: c-app-l2c-cyanoalert-olci-balt-v2-1D
    Path: api/augmentation/resample_in_time.py
    Title: Sverige app v2 1D- limited temporal length
    Style: default
    FileSystem: memory
    Function: compute_dataset
    InputDatasets: [app-l2c-cyanoalert-olci-balt-v2]
    InputParameters:
      period: 1D
```


Traceback:
```
[I 2023-03-03 12:06:50,791 xcube] Opened dataset 'app-l2c-cyanoalert-olci-balt-v2' from data store 's3_1': took 326.93ms
[W 2023-03-03 12:06:50,792 xcube] Skipping dataset app-l2c-cyanoalert-olci-balt-v2: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\\-]{1,63}$"
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/controllers.py", line 140, in get_datasets
    get_dataset(ctx, ds_id,
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/controllers.py", line 182, in get_dataset
    ml_ds = ctx.get_ml_dataset(ds_id)
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/context.py", line 183, in get_ml_dataset
    ml_dataset, _ = self._get_dataset_entry(ds_id)
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/context.py", line 602, in _get_dataset_entry
    self._set_dataset_entry(self._create_dataset_entry(ds_id))
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/context.py", line 615, in _create_dataset_entry
    ml_dataset = self._open_ml_dataset(dataset_config)
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/webapi/datasets/context.py", line 639, in _open_ml_dataset
    dataset = data_store.open_data(data_id, **open_params)
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/core/store/fs/store.py", line 333, in open_data
    return opener.open_data(fs_path,
  File "/opt/conda/lib/python3.9/site-packages/xcube-1.0.0.dev2-py3.9.egg/xcube/core/store/fs/impl/dataset.py", line 190, in open_data
    fs.exists(f'{data_id}/.zmetadata'))
  File "/opt/conda/lib/python3.9/site-packages/fsspec/asyn.py", line 114, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/fsspec/asyn.py", line 99, in sync
    raise return_result
  File "/opt/conda/lib/python3.9/site-packages/fsspec/asyn.py", line 54, in _runner
    result[0] = await coro
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 974, in _exists
    await self._info(path, bucket, key, version_id=version_id)
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 1238, in _info
    out = await self._call_s3(
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 339, in _call_s3
    return await _error_wrapper(
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 139, in _error_wrapper
    raise err
  File "/opt/conda/lib/python3.9/site-packages/s3fs/core.py", line 112, in _error_wrapper
    return await func(*args, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/aiobotocore/client.py", line 321, in _make_api_call
    request_dict = await self._convert_to_request_dict(
  File "/opt/conda/lib/python3.9/site-packages/aiobotocore/client.py", line 383, in _convert_to_request_dict
    api_params = await self._emit_api_params(
  File "/opt/conda/lib/python3.9/site-packages/aiobotocore/client.py", line 416, in _emit_api_params
    await self.meta.events.emit(
  File "/opt/conda/lib/python3.9/site-packages/aiobotocore/hooks.py", line 66, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/opt/conda/lib/python3.9/site-packages/botocore/handlers.py", line 275, in validate_bucket_name
    raise ParamValidationError(report=error_msg)
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "s3:": Bucket name must match the regex "^[a-zA-Z0-9.\\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\\-]{1,63}$"
```

**Expected behavior**
I would expect that the Datasets will be able to use the s3 resources the same way the DataStore configuration is. 


Changed files:
xcube/core/byoa/fileset.py, xcube/webapi/datasets/context.py

diff --git a/xcube/core/byoa/fileset.py b/xcube/core/byoa/fileset.py
index 41e4745b..77334518 100644
--- a/xcube/core/byoa/fileset.py
+++ b/xcube/core/byoa/fileset.py
@@ -257,9 +257,15 @@ class FileSet(JsonObject):
         if fs.isdir(root):
             temp_dir = new_temp_dir(prefix=TEMP_FILE_PREFIX,
                                     suffix=temp_file_suffix)
+            # Note, AbstractFileSystem.get() behaves
+            # unpredictably. Sometimes it will create a new directory
+            # named after root in temp_dir (e.g. this is the case
+            # if temp_dir exists). To avoid this, we add a non-existing
+            # directory here:
+            temp_dir += "/data"
             # TODO: replace by loop so we can apply includes/excludes
             #   before downloading actual files. See impl of fs.get().
-            fs.get(root, temp_dir + "/", recursive=True)
+            fs.get(root + "/*", temp_dir, recursive=True)
             return FileSet(temp_dir,
                            sub_path=self.sub_path,
                            includes=self.includes,
diff --git a/xcube/webapi/datasets/context.py b/xcube/webapi/datasets/context.py
index 8d9572dc..590173d0 100644
--- a/xcube/webapi/datasets/context.py
+++ b/xcube/webapi/datasets/context.py
@@ -316,17 +316,17 @@ class DatasetsContext(ResourcesContext):
             # Determine common prefixes of paths (and call them roots)
             roots = _get_roots(paths)
             for root in roots:
-                abs_root = root
                 fs_protocol = FS_TYPE_TO_PROTOCOL.get(file_system,
                                                       file_system)
-                if not is_absolute_path(abs_root):
-                    abs_root = f"{base_dir}/{root}"
-                store_params_for_root = store_params.copy()
-                store_params_for_root['root'] = abs_root
+                store_params_with_root = store_params.copy()
+                if fs_protocol == "file" and not is_absolute_path(root):
+                    store_params_with_root['root'] = f"{base_dir}/{root}"
+                else:
+                    store_params_with_root['root'] = root
                 # See if there already is a store with this configuration
                 data_store_config = DataStoreConfig(
                     store_id=fs_protocol,
-                    store_params=store_params_for_root
+                    store_params=store_params_with_root
                 )
                 store_instance_id = data_store_pool. \\
                     get_store_instance_id(data_store_config)
@@ -340,10 +340,11 @@ class DatasetsContext(ResourcesContext):
                     data_store_pool.add_store_config(store_instance_id,
                                                      data_store_config)
                 for config in config_list:
-                    if config['Path'].startswith(root) and \\
+                    path = config['Path']
+                    if path.startswith(root) and \\
                             config.get('StoreInstanceId') is None:
                         config['StoreInstanceId'] = store_instance_id
-                        new_path = config['Path'][len(root):]
+                        new_path = path[len(root):]
                         while new_path.startswith("/") or \\
                                 new_path.startswith("\\\\"):
                             new_path = new_path[1:]