Repository: CS-SI/eodag (python)

Issue:
EODataAccessGateway constructor fails on AWS Lambda
**Describe the bug**

`makedirs` [invocation](https://github.com/CS-SI/eodag/blob/develop/eodag/api/core.py#L98) within the EODataAccessGateway constructor causes `OSError: [Errno 30] Read-only file system` when run on AWS Lambda

**Code To Reproduce**
CLI commands or Python code snippet to reproduce the bug. Please use maximum verbosity using:

```py
from aws_lambda_powertools.logging import Logger
from aws_lambda_powertools.tracing import Tracer
from eodag import EODataAccessGateway

logger = Logger()
tracer = Tracer()


@tracer.capture_lambda_handler
@logger.inject_lambda_context
def lambda_handler(event, context):
    EODataAccessGateway()
```

**Output**
Compete output obtained with maximal verbosity.

```
[ERROR] OSError: [Errno 30] Read-only file system: '/home/sbx_user1051'
Traceback (most recent call last):
  File "/opt/python/aws_lambda_powertools/tracing/tracer.py", line 305, in decorate
    response = lambda_handler(event, context, **kwargs)
  File "/opt/python/aws_lambda_powertools/logging/logger.py", line 438, in decorate
    return lambda_handler(event, context, *args, **kwargs)
  File "/var/task/<my file, my line>, in lambda_handler
    EODataAccessGateway()
  File "/opt/python/eodag/api/core.py", line 91, in __init__
    makedirs(self.conf_dir)
  File "/opt/python/eodag/utils/__init__.py", line 496, in makedirs
    os.makedirs(dirpath)
  File "/var/lang/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/var/lang/lib/python3.9/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/var/lang/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
```

**Environment:**

 - Python version: 3.9
 - EODAG version: 2.5.2
 - Runtime: AWS Lambda

**Additional context**

File system on AWS Lambda is read-only. It has "ephemeral storage" that could be accessed in `/tmp` ([1](https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/), [2](https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/)), however this class does not accept any configuration to where to point to `makedirs`. 

The workaround that heavily relies on the [current implementation](https://github.com/CS-SI/eodag/blob/develop/eodag/api/core.py#L97) is to change `$HOME` env var before invoking the constructor. But it is barely acceptable and extremely inconvenient, since this variable is used by many tools locally and in automated tools along the way.


Pull request:
feat: setting conf_dir with an environment variable and fallback directory
- The default configuration directory can be set with the environment variable `EODAG_CFG_DIR`
- If the configuration directory can't be created, a temporary directory is used instead. The temporary directory is not removed.

Fixes #684 


Patch:
diff --git a/docs/_static/product_types_information.csv b/docs/_static/product_types_information.csv
index 21b10020..61464555 100644
--- a/docs/_static/product_types_information.csv
+++ b/docs/_static/product_types_information.csv
@@ -28,10 +28,10 @@ CLMS_GLO_GDMP_333M,"Gross dry matter Productivity (GDMP) is an indication of the
 CLMS_GLO_LAI_333M,"LAI was defined by CEOS as half the developed area of the convex hull wrapping the green canopy elements per unit horizontal ground. This definition allows accounting for elements which are not flat such as needles or stems. LAI is strongly non linearly related to reflectance. Therefore, its estimation from remote sensing observations will be scale dependant over heterogeneous landscapes. When observing a canopy made of different layers of vegetation, it is therefore mandatory to consider all the green layers. This is particularly important for forest canopies where the understory may represent a very significant contribution to the total canopy LAI. The derived LAI corresponds therefore to the total green LAI, including the contribution of the green elements of the understory. The product at 333m resolution is provided in Near Real Time and consolidated in the next six periods. ","OLCI,PROBA-V",Sentinel-3,,,"Land,Leaf-area-index,LAI,OLCI,PROBA-V,Sentinel-3",,proprietary,Global 10-daily Leaf Area Index 333m,2014-01-10T00:00:00Z,,,,,,,,,,,,,,,,,,,,,available
 CLMS_GLO_NDVI_1KM_LTS,"The Normalized Difference Vegetation Index (NDVI) is a proxy to quantify the vegetation amount. It is defined as NDVI=(NIR-Red)/(NIR+Red) where NIR corresponds to the reflectance in the near infrared band, and Red to the reflectance in the red band. The time series of dekadal (10-daily) NDVI 1km version 2 observations over the period 1999-2017 is used to calculate Long Term Statistics (LTS) for each of the 36 10-daily periods (dekads) of the year. The calculated LTS include the minimum, median, maximum, average, standard deviation and the number of observations in the covered time series period. These LTS can be used as a reference for actual NDVI observations, which allows evaluating whether vegetation conditions deviate from a 'normal' situation. ","VEGETATION,PROBA-V",SPOT,,,"Land,NDVI,LTS,SPOT,VEGETATION,PROBA-V",,proprietary,"Normalized Difference Vegetation Index: global Long Term Statistics (raster 1km) - version 2, Apr 2019",1999-01-01T00:00:00Z,,,,,,,,,,,,,,,,,,,,,available
 CLMS_GLO_NDVI_333M,"The Normalized Difference Vegetation Index (NDVI) is a proxy to quantify the vegetation amount. It is defined as NDVI=(NIR-Red)/(NIR+Red) where NIR corresponds to the reflectance in the near infrared band, and Red to the reflectance in the red band. It is closely related to FAPAR and is little scale dependant. ",PROBA-V,,,,"Land,NDVI,PROBA-V",,proprietary,Global 10-daily Normalized Difference Vegetation Index 333M,2014-01-01T00:00:00Z,,,,,,,,,,,,,,,,,,,,,available
-COP_DEM_GLO30_DGED,"Defence Gridded Elevation Data (DGED) formatted Copernicus DEM GLO-30 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-30 provides worldwide coverage at 30 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-30,DSM,GDGED",ALTIMETRIC,proprietary,Copernicus DEM GLO-30 DGED,2010-06-21T00:00:00Z,,,,,,available,,,,,,,,,,,,,,,available
-COP_DEM_GLO30_DTED,"Digital Terrain Elevation Data (DTED) formatted Copernicus DEM GLO-30 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-30 provides worldwide coverage at 30 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-30,DSM,DTED",ALTIMETRIC,proprietary,Copernicus DEM GLO-30 DTED,2010-06-21T00:00:00Z,,,,,,available,,,,,,,,,,,,,,,
-COP_DEM_GLO90_DGED,"Defence Gridded Elevation Data (DGED) formatted Copernicus DEM GLO-90 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-90 provides worldwide coverage at 90 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-90,DSM,GDGED",ALTIMETRIC,proprietary,Copernicus DEM GLO-90 DGED,2010-06-21T00:00:00Z,,,,,,available,,,,,,,,,,,,,,,available
-COP_DEM_GLO90_DTED,"Digital Terrain Elevation Data (DTED) formatted Copernicus DEM GLO-90 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-90 provides worldwide coverage at 90 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-90,DSM,DTED",ALTIMETRIC,proprietary,Copernicus DEM GLO-90 DTED,2010-06-21T00:00:00Z,,,,,,available,,,,,,,,,,,,,,,
+COP_DEM_GLO30_DGED,"Defence Gridded Elevation Data (DGED) formatted Copernicus DEM GLO-30 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-30 provides worldwide coverage at 30 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-30,DSM,GDGED",ALTIMETRIC,proprietary,Copernicus DEM GLO-30 DGED,2010-06-21T00:00:00Z,,,,,,available,available,,,,,,,,,,,,,,available
+COP_DEM_GLO30_DTED,"Digital Terrain Elevation Data (DTED) formatted Copernicus DEM GLO-30 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-30 provides worldwide coverage at 30 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-30,DSM,DTED",ALTIMETRIC,proprietary,Copernicus DEM GLO-30 DTED,2010-06-21T00:00:00Z,,,,,,available,available,,,,,,,,,,,,,,
+COP_DEM_GLO90_DGED,"Defence Gridded Elevation Data (DGED) formatted Copernicus DEM GLO-90 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-90 provides worldwide coverage at 90 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-90,DSM,GDGED",ALTIMETRIC,proprietary,Copernicus DEM GLO-90 DGED,2010-06-21T00:00:00Z,,,,,,available,available,,,,,,,,,,,,,,available
+COP_DEM_GLO90_DTED,"Digital Terrain Elevation Data (DTED) formatted Copernicus DEM GLO-90 data. The Copernicus Digital Elevation Model is a Digital Surface Model (DSM) that represents the surface of the Earth including buildings, infrastructure and vegetation. The Copernicus DEM is provided in 3 different instances: EEA-10, GLO-30 and GLO-90. GLO-90 provides worldwide coverage at 90 meters.Data were acquired through the TanDEM-X mission between 2011 and 2015. The datasets were made available for use in 2019 and will be maintained until 2026. ",,TerraSAR,,,"TerraSAR,TanDEM-X,DEM,surface,GLO-90,DSM,DTED",ALTIMETRIC,proprietary,Copernicus DEM GLO-90 DTED,2010-06-21T00:00:00Z,,,,,,available,available,,,,,,,,,,,,,,
 EEA_DAILY_SSM_1KM,"Surface Soil Moisture (SSM) is the relative water content of the top few centimetres soil, describing how wet or dry the soil is in its topmost layer, expressed in percent saturation. It is measured by satellite radar sensors and allows insights in local precipitation impacts and soil conditions. SSM is a key driver of water and heat fluxes between the ground and the atmosphere, regulating air temperature and humidity. Moreover, in its role as water supply, it is vital to vegetation health. Vice versa, SSM is very sensitive to external forcing in the form of precipitation, temperature, solar irradiation, humidity, and wind. SSM is thus both an integrator of climatic conditions and a driver of local weather and climate, and plays a major role in global water-, energy- and carbon- cycles. Knowledge on the dynamics of soil moisture is important in the understanding of processes in many environmental and socio-economic fields, e.g., its impact on vegetation vitality, crop yield, droughts or exposure to flood threats. ","C-SAR,Metop ASCAT",Sentinel-1,,,"SSM,C-SAR,Metop-ASCAT,Sentinel-1",RADAR,proprietary,"Surface Soil Moisture: continental Europe daily (raster 1km) - version 1, Apr 2019",2015-01-01T00:00:00Z,,,,,,,,,,,,,,,,,,,,,available
 EEA_DAILY_SWI_1KM,"The Soil Water Index (SWI) quantifies the moisture condition at various depths in the soil. It is mainly driven by the precipitation via the process of infiltration. Soil moisture is a very heterogeneous variable and varies on small scales with soil properties and drainage patterns. Satellite measurements integrate over relative large-scale areas, with the presence of vegetation adding complexity to the interpretation. Soil moisture is a key parameter in numerous environmental studies including hydrology, meteorology and agriculture, and is recognized as an Essential Climate Variable (ECV) by the Global Climate Observing System (GCOS). The SWI product provides daily information about moisture conditions in different soil layers. It includes a quality flag (QFLAG) indicating the availability of SSM measurements for SWI calculations, and a Surface State Flag (SSF) indicating frozen or snow covered soils. ","C-SAR,Metop ASCAT",Sentinel-1,,,"SWI,QFLAG,SSF,C-SAR,Metop-ASCAT,Sentinel-1",RADAR,proprietary,"Soil Water Index: continental Europe daily (raster 1km) - version 1, Apr 2019",2015-01-01T00:00:00Z,,,,,,,,,,,,,,,,,,,,,available
 EEA_DAILY_VI,"Vegetation Indices (VI) comprises four daily vegetation indices (PPI, NDVI, LAI and FAPAR) and quality information, that are part of the Copernicus Land Monitoring Service (CLMS) HR-VPP product suite. The 10m resolution, daily updated Plant Phenology Index (PPI), Normalized Difference Vegetation Index (NDVI), Leaf Area Index (LAI) and Fraction of Absorbed Photosynthetically Active Radiation (fAPAR) are derived from Copernicus Sentinel-2 satellite observations. They are provided together with a related quality indicator (QFLAG2) that flags clouds, shadows, snow, open water and other areas where the VI retrieval is less reliable. These Vegetation Indices are made available as a set of raster files with 10 x 10m resolution, in UTM/WGS84 projection corresponding to the Sentinel-2 tiling grid, for those tiles that cover the EEA38 countries and the United Kingdom and for the period from 2017 until today, with daily updates. The Vegetation Indices are part of the pan-European High Resolution Vegetation Phenology and Productivity (HR-VPP) component of the Copernicus Land Monitoring Service (CLMS). ",,Sentinel-2,"S2A, S2B",,"Land,Plant-phenology-index,Phenology,Vegetation,Sentinel-2,S2A,S2B",RADAR,proprietary,"Vegetation Indices, daily, UTM projection",,,,,,,,,,,,,,,,,,,,,,available
diff --git a/docs/getting_started_guide/configure.rst b/docs/getting_started_guide/configure.rst
index ac0559c5..4601b6ae 100644
--- a/docs/getting_started_guide/configure.rst
+++ b/docs/getting_started_guide/configure.rst
@@ -137,6 +137,7 @@ Core configuration using environment variables
 
 Some EODAG core settings can be overriden using environment variables:
 
+* ``EODAG_CFG_DIR`` customized configuration directory in place of ``~/.config/eodag``
 * ``EODAG_CFG_FILE`` for defining the desired path to the `user configuration file\
   <https://eodag.readthedocs.io/en/stable/getting_started_guide/configure.html#yaml-user-configuration-file>`_
 * ``EODAG_LOCS_CFG_FILE`` for defining the desired path to the
diff --git a/eodag/api/core.py b/eodag/api/core.py
index bcd5d3e5..81a69d27 100644
--- a/eodag/api/core.py
+++ b/eodag/api/core.py
@@ -21,6 +21,7 @@ import logging
 import os
 import re
 import shutil
+import tempfile
 from operator import itemgetter
 from typing import (
     TYPE_CHECKING,
@@ -129,8 +130,27 @@ class EODataAccessGateway:
         self.product_types_config_md5 = obj_md5sum(self.product_types_config.source)
         self.providers_config = load_default_config()
 
-        self.conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
-        makedirs(self.conf_dir)
+        env_var_cfg_dir = "EODAG_CFG_DIR"
+        self.conf_dir = os.getenv(
+            env_var_cfg_dir,
+            default=os.path.join(os.path.expanduser("~"), ".config", "eodag"),
+        )
+        try:
+            makedirs(self.conf_dir)
+        except OSError as e:
+            logger.debug(e)
+            tmp_conf_dir = os.path.join(tempfile.gettempdir(), ".config", "eodag")
+            logger.warning(
+                f"Cannot create configuration directory {self.conf_dir}. "
+                + f"Falling back to temporary directory {tmp_conf_dir}."
+            )
+            if os.getenv(env_var_cfg_dir) is None:
+                logger.warning(
+                    "You can set the path of the configuration directory "
+                    + f"with the environment variable {env_var_cfg_dir}"
+                )
+            self.conf_dir = tmp_conf_dir
+            makedirs(self.conf_dir)
 
         self._plugins_manager = PluginManager(self.providers_config)
         # use updated providers_config


Test patch:
diff --git a/tests/units/test_core.py b/tests/units/test_core.py
index f5f118c4..46fc1bed 100644
--- a/tests/units/test_core.py
+++ b/tests/units/test_core.py
@@ -22,6 +22,7 @@ import json
 import logging
 import os
 import shutil
+import tempfile
 import unittest
 import uuid
 from pathlib import Path
@@ -1178,6 +1179,20 @@ class TestCoreConfWithEnvVar(TestCoreBase):
 
 
 class TestCoreInvolvingConfDir(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        super(TestCoreInvolvingConfDir, cls).setUpClass()
+        cls.dag = EODataAccessGateway()
+        # mock os.environ to empty env
+        cls.mock_os_environ = mock.patch.dict(os.environ, {}, clear=True)
+        cls.mock_os_environ.start()
+
+    @classmethod
+    def tearDownClass(cls):
+        super(TestCoreInvolvingConfDir, cls).tearDownClass()
+        # stop os.environ
+        cls.mock_os_environ.stop()
+
     def setUp(self):
         super(TestCoreInvolvingConfDir, self).setUp()
         self.dag = EODataAccessGateway()
@@ -1194,14 +1209,15 @@ class TestCoreInvolvingConfDir(unittest.TestCase):
                 except OSError:
                     shutil.rmtree(old_path)
 
-    def execution_involving_conf_dir(self, inspect=None):
+    def execution_involving_conf_dir(self, inspect=None, conf_dir=None):
         """Check that the path(s) inspected (str, list) are created after the instantation
         of EODataAccessGateway. If they were already there, rename them (.old), instantiate,
         check, delete the new files, and restore the existing files to there previous name."""
         if inspect is not None:
+            if conf_dir is None:
+                conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
             if isinstance(inspect, str):
                 inspect = [inspect]
-            conf_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
             olds = []
             currents = []
             for inspected in inspect:
@@ -1233,6 +1249,49 @@ class TestCoreInvolvingConfDir(unittest.TestCase):
         """The core object must create a locations config file and a shp dir in standard user config location on instantiation"""  # noqa
         self.execution_involving_conf_dir(inspect=["locations.yml", "shp"])
 
+    def test_read_only_home_dir(self):
+        # standard directory
+        home_dir = os.path.join(os.path.expanduser("~"), ".config", "eodag")
+        self.execution_involving_conf_dir(inspect="eodag.yml", conf_dir=home_dir)
+
+        # user defined directory
+        user_dir = os.path.join(os.path.expanduser("~"), ".config", "another_eodag")
+        os.environ["EODAG_CFG_DIR"] = user_dir
+        self.execution_involving_conf_dir(inspect="eodag.yml", conf_dir=user_dir)
+        shutil.rmtree(user_dir)
+        del os.environ["EODAG_CFG_DIR"]
+
+        # fallback temporary folder
+        def makedirs_side_effect(dir):
+            if dir == os.path.join(os.path.expanduser("~"), ".config", "eodag"):
+                raise OSError("Mock makedirs error")
+            else:
+                return makedirs(dir)
+
+        with mock.patch(
+            "eodag.api.core.makedirs", side_effect=makedirs_side_effect
+        ) as mock_makedirs:
+            # backup temp_dir if exists
+            temp_dir = temp_dir_old = os.path.join(
+                tempfile.gettempdir(), ".config", "eodag"
+            )
+            if os.path.exists(temp_dir):
+                temp_dir_old = f"{temp_dir}.old"
+                shutil.move(temp_dir, temp_dir_old)
+
+            EODataAccessGateway()
+            expected = [unittest.mock.call(home_dir), unittest.mock.call(temp_dir)]
+            mock_makedirs.assert_has_calls(expected)
+            self.assertTrue(os.path.exists(temp_dir))
+
+            # restore temp_dir
+            if temp_dir_old != temp_dir:
+                try:
+                    shutil.rmtree(temp_dir)
+                except OSError:
+                    os.unlink(temp_dir)
+                shutil.move(temp_dir_old, temp_dir)
+
 
 class TestCoreGeometry(TestCoreBase):
     @classmethod