Repository: E3SM-Project/e3sm_to_cmip

Issue:
KeyError raised when any of the variable handler fails
Initially reported by @forsyth2 in e3sm-unified rc testing, when any of the cmip variable handler fails, KeyError raised and the program exist. The expected behavior is to process every available target cmip variables, as previous versions. 
My standalone e2c test, v1.9.0:
```
2023-08-01 20:45:20,343_343:INFO:main:input_path = /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/rgr
2023-08-01 20:45:20,343_343:INFO:main:output_path = /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190
2023-08-01 20:45:20,343_343:INFO:main:precheck_path = None
[*] Loaded `tas` handler for variable `tas`.
[*] Loaded `pr` handler for variable `pr`.
[*] Loaded `od550aer` handler for variable `od550aer`.
[*] Loaded `abs550aer` handler for variable `abs550aer`.
[*] Writing log output to: /p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/converter.log
[+] Running CMOR handlers in parallel
  0%|                                                     | 0/4 [00:00<?, ?it/s][-] abs550aer: Unable to find input files for AODABS
 25%|███████████▎                                 | 1/4 [00:01<00:03,  1.28s/it][-] Error running handler abs550aer
100%|█████████████████████████████████████████████| 4/4 [00:01<00:00,  3.10it/s]
[+] 3 of 4 handlers complete
[-] abs550aer failed to complete
```

v1.10.0rc:

```
2023-08-01 20:43:01,023_023:INFO:__init__:--------------------------------------
2023-08-01 20:43:01,024_024:INFO:__init__:| E3SM to CMIP Configuration
2023-08-01 20:43:01,024_024:INFO:__init__:--------------------------------------
2023-08-01 20:43:01,024_024:INFO:__init__:    * var_list='['tas', 'pr', 'od550aer', 'abs550aer']'
2023-08-01 20:43:01,024_024:INFO:__init__:    * input_path='/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190/rgr'
2023-08-01 20:43:01,024_024:INFO:__init__:    * output_path='/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data/test_reference_v190'
2023-08-01 20:43:01,024_024:INFO:__init__:    * precheck_path='None'
2023-08-01 20:43:01,024_024:INFO:__init__:    * freq='mon'
2023-08-01 20:43:01,025_025:INFO:__init__:    * realm='atm'
2023-08-01 20:43:01,025_025:INFO:__init__:    * Writing log output file to: logs/20230801_204301_015530
Traceback (most recent call last):
  File "/home/zhang40/miniconda3/envs/e2c_nco511/bin/e3sm_to_cmip", line 8, in <module>
    sys.exit(main())
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 792, in main
    app = E3SMtoCMIP(args)
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 152, in __init__
    self.handlers = self._get_handlers()
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/__main__.py", line 219, in _get_handlers
    handlers = derive_handlers(
  File "/home/zhang40/miniconda3/envs/e2c_nco511/lib/python3.9/site-packages/e3sm_to_cmip/cmor_handlers/utils.py", line 220, in derive_handlers
    raise KeyError(
KeyError: "No handlers could be derived for the variables: ['abs550aer']. Make sure the input E3SM datasets have the variables needed derivation."
```
Below is the script to reproduce on acme1:

```
#!/bin/bash

# Conda Environment
# -----------------
# Either comment out the lines below to use the E3SM Unified environment
# or use a local environment.
#source /usr/local/e3sm_unified/envs/load_latest_e3sm_unified_acme1.sh
#conda activate e2c_nco511_13

# -----------------
# SETUP
# -----------------
exp=historical
caseid=v2.LR.${exp}_0101
start=1850
end=1850
ypf=1

# -----------------
# PATHS
# -----------------
e2c_path=/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data
model_data=$e2c_path/model-output
result_dir=${e2c_path}/test_reference_v1100

rgr_dir=${result_dir}/rgr
rgr_dir_vert=${result_dir}/rgr_vert
rgr_dir_vert_plev=${result_dir}/rgr_vert_plev
native_dir=${result_dir}/native

map_file=${e2c_path}/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
tables_path=${e2c_path}/cmor/cmip6-cmor-tables/Tables/
metadata_path=${e2c_path}/user_metadata.json

# NOTE: Space is not accepted in nco var list
## ------------------------------------------------------
## TEST CASE - atm monthly h0
## ------------------------------------------------------
input_path=${model_data}/v2.eam_input
flags='-7 --dfl_lvl=1 --no_cll_msr'
raw_var_list="TREFHT,PRECC,PRECL,AODVIS"

# 1. atm 2D variables
#--------------------------
cmip_var_list="tas, pr, od550aer, abs550aer"
ncclimo -P eam -j 1 --map=${map_file} --start=$start --end=$end --ypf=$ypf --split -c $caseid -o ${native_dir} -O ${rgr_dir} -v ${raw_var_list} -i ${input_path} ${flags}
# CMORIZE Atmosphere monthly variables: 2D and model level 3D variables (CLOUD,CLDICE,CLDLIQ)
e3sm_to_cmip -i ${rgr_dir} -o $result_dir  -v ${cmip_var_list} -t $tables_path -u ${metadata_path}

Discussion:
chengzhuzhang: @tomvothecoder I attempted a fix for addressing #202. Please review or suggest change. If things work out, we should create a new release candidate for e3sm-unified rc10 including this fix and #201 . 
forsyth2: @chengzhuzhang Thanks for working on this. I'm testing out `zppy` with these changes. Tests are still running, but it looks like this gets the `ts` tasks working.
chengzhuzhang: @forsyth2 thank you for testing with zppy! I will work with @tomvothecoder to finalize this fix. Let me know if there are additional e2c errors with zppy test. Thank you.
tomvothecoder: I'll checkout this branch and review the changes right now.
chengzhuzhang: @tomvothecoder thank you for the review and the suggested change. They all look good and I batch committed them (first time to use this nice batch feature). also thanks for helping addressing failed tests..
tomvothecoder: @chengzhuzhang no problem! I just fixed the tests and will merge this PR, then release v1.10.0rc2 momentarily.

Patch:
diff --git a/e3sm_to_cmip/cmor_handlers/utils.py b/e3sm_to_cmip/cmor_handlers/utils.py
index dcbcb00..9f0240d 100644
--- a/e3sm_to_cmip/cmor_handlers/utils.py
+++ b/e3sm_to_cmip/cmor_handlers/utils.py
@@ -74,7 +74,7 @@ def load_all_handlers(
             handlers = handlers + var_handler
 
         if len(missing_handlers) > 0:
-            raise KeyError(
+            logger.warning(
                 f"No handlers are defined for the variables: {missing_handlers}. "
                 "Make sure at least one variable handler is defined for each of these "
                 f"variables in `{HANDLER_DEFINITIONS_PATH}`."
@@ -116,7 +116,7 @@ def _get_mpas_handlers(cmip_vars: List[str]):
         derived_handlers.append(var_handler[0])
 
     if len(missing_handlers) > 0:
-        raise KeyError(
+        logger.warning(
             f"No variable handlers are defined for {missing_handlers}. Make sure at "
             "least one variable handler is defined for each of these variables in "
             f"`{MPAS_HANDLER_DIR_PATH}`."
@@ -211,13 +211,13 @@ def derive_handlers(
         derived_handlers.append(derived_handler)
 
     if len(missing_handlers) > 0:
-        raise KeyError(
+        logger.warning(
             f"No handlers are defined for the variables: {missing_handlers}. "
             "Make sure handlers are defined for these variables in `handlers.yaml`."
         )
 
     if len(cannot_derive) > 0:
-        raise KeyError(
+        logger.warning(
             f"No handlers could be derived for the variables: {cannot_derive}. "
             "Make sure the input E3SM datasets have the variables needed derivation."
         )


Test patch:
diff --git a/tests/cmor_handlers/test_utils.py b/tests/cmor_handlers/test_utils.py
index c9da397..f4a351d 100644
--- a/tests/cmor_handlers/test_utils.py
+++ b/tests/cmor_handlers/test_utils.py
@@ -45,13 +45,17 @@ class TestLoadAllHandlers:
                 json_file,
             )
 
-    def test_raises_error_if_handler_does_not_exist_for_var(self):
-        with pytest.raises(KeyError):
-            load_all_handlers("lnd", ["invalid_var"])
+    def test_prints_logger_warning_if_handler_does_not_exist_for_var(self, caplog):
+        load_all_handlers("lnd", ["invalid_var"])
 
-    def test_raises_error_if_mpas_handler_does_not_exist_for_var(self):
-        with pytest.raises(KeyError):
-            load_all_handlers("mpaso", cmip_vars=["invalid_var"])
+        for record in caplog.records:
+            assert record.levelname == "WARNING"
+
+    def test_prints_logger_warning_if_mpas_handler_does_not_exist_for_var(self, caplog):
+        load_all_handlers("mpaso", cmip_vars=["invalid_var"])
+
+        for record in caplog.records:
+            assert record.levelname == "WARNING"
 
     def test_updates_CMIP_table_for_variable_based_on_freq_param(self):
         result = load_all_handlers("lnd", cmip_vars=["pr"])
@@ -201,19 +205,27 @@ class TestDeriveHandlers:
                 json_file,
             )
 
-    def test_raises_error_if_handler_is_not_defined_for_a_variable(self):
-        with pytest.raises(KeyError):
-            derive_handlers(
-                self.tables_path,
-                cmip_vars=["undefined_var"],
-                e3sm_vars=["incorrect_e3sm_var"],
-            )
+    def test_prints_logger_warning_if_handler_is_not_defined_for_a_variable(
+        self, caplog
+    ):
+        derive_handlers(
+            self.tables_path,
+            cmip_vars=["undefined_var"],
+            e3sm_vars=["incorrect_e3sm_var"],
+        )
 
-    def test_raises_error_if_handler_cannot_be_derived_from_input_e3sm_vars(self):
-        with pytest.raises(KeyError):
-            derive_handlers(
-                self.tables_path, cmip_vars=["pr"], e3sm_vars=["incorrect_e3sm_var"]
-            )
+        for record in caplog.records:
+            assert record.levelname == "WARNING"
+
+    def test_prints_logger_warning_if_handler_cannot_be_derived_from_input_e3sm_vars(
+        self, caplog
+    ):
+        derive_handlers(
+            self.tables_path, cmip_vars=["pr"], e3sm_vars=["incorrect_e3sm_var"]
+        )
+
+        for record in caplog.records:
+            assert record.levelname == "WARNING"
 
     def test_raises_error_if_CMIP6_table_entry_for_variable_and_freq_arg(self):
         with pytest.raises(KeyError):