Repository: open-sdg/sdg-build

Issue:
Error when using code_map in inputs
Pandas 1.4 introduced a breaking change in the [Series.replace](https://pandas.pydata.org/docs/reference/api/pandas.Series.replace.html) method. If you are using Pandas 1.4 or higher and try to use the code_map parameter, you'll see this error: `ValueError: Series.replace cannot use dict-like to_replace and non-None value`. More info on the bug here: https://github.com/pandas-dev/pandas/issues/46004

I'm not sure if Pandas is going to revert their breaking change (they should IMO, since it was introduced in a minor version), but we need to adjust our code in the meantime. The work-around in the meantime is to not specify the `value` parameter at all. But, we still need it for previous versions of Pandas... So we will need to do a try/except and use both.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/open-sdg/sdg-build/pull/322?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg) Report
> Merging [#322](https://codecov.io/gh/open-sdg/sdg-build/pull/322?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg) (05e29f3) into [2.1.0-dev](https://codecov.io/gh/open-sdg/sdg-build/commit/250b6a33c0560ad7e0374fcb70fd6aea9581ea94?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg) (250b6a3) will **decrease** coverage by `0.04%`.
> The diff coverage is `0.00%`.

```diff
@@              Coverage Diff              @@
##           2.1.0-dev     #322      +/-   ##
=============================================
- Coverage      70.32%   70.27%   -0.05%     
=============================================
  Files             74       74              
  Lines           4536     4539       +3     
=============================================
  Hits            3190     3190              
- Misses          1346     1349       +3     
```


| [Impacted Files](https://codecov.io/gh/open-sdg/sdg-build/pull/322?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg) | Coverage Δ | |
|---|---|---|
| [sdg/inputs/InputBase.py](https://codecov.io/gh/open-sdg/sdg-build/pull/322/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg#diff-c2RnL2lucHV0cy9JbnB1dEJhc2UucHk=) | `74.82% <0.00%> (-1.66%)` | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=open-sdg)


Patch:
diff --git a/sdg/inputs/InputBase.py b/sdg/inputs/InputBase.py
index 18f0a9a..8dcf40d 100644
--- a/sdg/inputs/InputBase.py
+++ b/sdg/inputs/InputBase.py
@@ -363,5 +363,8 @@ class InputBase(Loggable):
                 if row['Dimension'] not in code_dict:
                     code_dict[row['Dimension']] = {}
                 code_dict[row['Dimension']][row['Text']] = row['Value']
-            data.replace(to_replace=code_dict, value=None, inplace=True)
+            try:
+                data.replace(to_replace=code_dict, value=None, inplace=True)
+            except:
+                data.replace(to_replace=code_dict, inplace=True)
         return data
diff --git a/sdg/open_sdg.py b/sdg/open_sdg.py
index fd4ca0a..56185e3 100644
--- a/sdg/open_sdg.py
+++ b/sdg/open_sdg.py
@@ -45,7 +45,7 @@ def open_sdg_build(src_dir='', site_dir='_site', schema_file='_prose.yml',
                    docs_extra_disaggregations=None, docs_translate_disaggregations=False,
                    logging=None, indicator_export_filename='all_indicators',
                    datapackage=None, csvw=None, data_schema=None, docs_metadata_fields=None,
-                   alter_indicator=None):
+                   alter_indicator=None, indicator_callback=None):
     """Read each input file and edge file and write out json.
 
     Args:
@@ -66,6 +66,8 @@ def open_sdg_build(src_dir='', site_dir='_site', schema_file='_prose.yml',
         alter_data: function. A callback function that alters a data Dataframe (for each input)
         alter_meta: function. A callback function that alters a metadata dictionary (for each input)
         alter_indicator: function. A callback function that alters the full Indicator objects (for each output)
+        indicator_callback: function. A callback function that runs for each
+          indicator after the outputs have already been generated.
         indicator_options: Dict. Options to pass into each indicator.
         docs_branding: string. A heading for all documentation pages
         docs_intro: string. An introduction for the documentation homepage
@@ -144,6 +146,8 @@ def open_sdg_build(src_dir='', site_dir='_site', schema_file='_prose.yml',
     # Pass along our data/meta alterations.
     options['alter_data'] = alter_data
     options['alter_meta'] = alter_meta
+    # And other callbacks.
+    options['indicator_callback'] = indicator_callback
     options['alter_indicator'] = alter_indicator
 
     # Convert the indicator options.
@@ -159,6 +163,14 @@ def open_sdg_build(src_dir='', site_dir='_site', schema_file='_prose.yml',
         else:
             sys.exit('The data configuration must have a "languages" setting with at least one language. See the documentation here: https://open-sdg.readthedocs.io/en/latest/data-configuration/#languages')
 
+    # Perform per-indicator callbacks.
+    if callable(options['indicator_callback']):
+        indicator_callback = options['indicator_callback']
+        for output in outputs:
+            if isinstance(output, sdg.outputs.OutputOpenSdg):
+                for indicator in output.indicators.values():
+                    indicator_callback(indicator)
+
     # Output the documentation pages.
     documentation_service = sdg.OutputDocumentationService(outputs,
         folder=options['site_dir'],
diff --git a/sdg/outputs/OutputSdmxMl.py b/sdg/outputs/OutputSdmxMl.py
index 0ce04c5..edc13b4 100644
--- a/sdg/outputs/OutputSdmxMl.py
+++ b/sdg/outputs/OutputSdmxMl.py
@@ -459,7 +459,10 @@ class OutputSdmxMl(OutputBase):
                 if row['Dimension'] not in code_dict:
                     code_dict[row['Dimension']] = {}
                 code_dict[row['Dimension']][row['Text']] = row['Value']
-            data.replace(to_replace=code_dict, value=None, inplace=True)
+            try:
+                data.replace(to_replace=code_dict, value=None, inplace=True)
+            except:
+                data.replace(to_replace=code_dict, inplace=True)
 
             # Now put back the pseudo-unique strings mentioned above.
             data.replace(temp_nan_fix, np.nan, inplace=True)


Test patch:
diff --git a/tests/InputCsvData_test.py b/tests/InputCsvData_test.py
index 371a0e5..93e9407 100644
--- a/tests/InputCsvData_test.py
+++ b/tests/InputCsvData_test.py
@@ -26,8 +26,6 @@ def test_csv_input_with_duplicates():
     indicator = data_input.indicators['1-1-1']
     indicator.translate('en', translation_helper)
 
-    print(indicator.data)
-
     correct_data = """
         Year,SEX,SEX1,Value
         2020,,,100
@@ -38,3 +36,31 @@ def test_csv_input_with_duplicates():
         2021,F,F,80
     """
     inputs_common.assert_input_has_correct_data(indicator.language('en').data, correct_data)
+
+def test_csv_input_with_code_map():
+
+    translation_input = sdg.translations.TranslationInputYaml(
+        source=os.path.join('tests', 'assets', 'translations', 'yaml'),
+    )
+    translation_helper = sdg.translations.TranslationHelper([translation_input])
+
+    data_pattern = os.path.join('tests', 'assets', 'data', 'csv', '*.csv')
+    code_map = os.path.join('tests', 'assets', 'misc', 'code-map.csv')
+    data_input = sdg.inputs.InputCsvData(path_pattern=data_pattern, code_map=code_map)
+    indicator_options = sdg.IndicatorOptions()
+    data_input.execute(indicator_options=indicator_options)
+
+    correct_data = """
+        Year,SEX,Value
+        2020,,100
+        2021,,120
+        2020,Male,50
+        2021,Male,60
+        2020,Female,70
+        2021,Female,80
+    """
+
+    indicator = data_input.indicators['1-1-1']
+    indicator.translate('en', translation_helper)
+
+    inputs_common.assert_input_has_correct_data(indicator.language('en').data, correct_data)
diff --git a/tests/OutputSdmxMl_test.py b/tests/OutputSdmxMl_test.py
index 66a36fd..ae0aeec 100644
--- a/tests/OutputSdmxMl_test.py
+++ b/tests/OutputSdmxMl_test.py
@@ -12,15 +12,17 @@ def test_sdmx_ml_output():
     )
     schema_path = os.path.join('tests', 'assets', 'meta', 'metadata_schema.yml')
     schema = sdg.schemas.SchemaInputOpenSdg(schema_path=schema_path)
+    output_folder = '_site_sdmx'
     data_output = sdg.outputs.OutputSdmxMl([data_input], schema,
         structure_specific=True,
         meta_reporting_type='N',
         meta_ref_area=862,
+        output_folder=output_folder
     )
     assert data_output.validate()
     assert data_output.execute()
 
-    output_path = os.path.join('_site', 'sdmx', '1-1-1.xml')
+    output_path = os.path.join(output_folder, 'sdmx', '1-1-1.xml')
     output_input = sdg.inputs.InputSdmxMl_StructureSpecific(
         source=output_path,
         import_codes=True,
@@ -32,3 +34,48 @@ def test_sdmx_ml_output():
     input_df = data_input.indicators['1-1-1'].data
     output_df = output_input.indicators['1-1-1'].data
     pd.testing.assert_frame_equal(input_df, output_df)
+
+def test_sdmx_ml_output_with_code_map():
+
+    code_map = os.path.join('tests', 'assets', 'misc', 'code-map.csv')
+
+    data_path = os.path.join('tests', 'assets', 'data', 'sdmx', 'structure-specific', '1-1-1--structure-specific.xml')
+    data_input = sdg.inputs.InputSdmxMl_StructureSpecific(
+        source=data_path,
+        import_codes=True,
+        drop_singleton_dimensions=False,
+    )
+    schema_path = os.path.join('tests', 'assets', 'meta', 'metadata_schema.yml')
+    schema = sdg.schemas.SchemaInputOpenSdg(schema_path=schema_path)
+    output_folder = '_site_sdmx_mapped'
+    data_output = sdg.outputs.OutputSdmxMl([data_input], schema,
+        structure_specific=True,
+        meta_reporting_type='N',
+        meta_ref_area=862,
+        code_map=code_map,
+        output_folder=output_folder
+    )
+    assert data_output.validate()
+    assert data_output.execute()
+
+    output_path = os.path.join(output_folder, 'sdmx', '1-1-1.xml')
+    output_input = sdg.inputs.InputSdmxMl_StructureSpecific(
+        source=output_path,
+        import_codes=True,
+        drop_singleton_dimensions=False,
+    )
+    indicator_options = sdg.IndicatorOptions()
+    output_input.execute(indicator_options=indicator_options)
+
+    data_mapped_path = os.path.join('tests', 'assets', 'data', 'sdmx', 'structure-specific', '1-1-1--structure-specific.xml')
+    data_mapped_input = sdg.inputs.InputSdmxMl_StructureSpecific(
+        source=data_path,
+        import_codes=True,
+        drop_singleton_dimensions=False,
+        code_map=code_map
+    )
+    data_mapped_input.execute(indicator_options=indicator_options)
+
+    input_df = data_mapped_input.indicators['1-1-1'].data
+    output_df = output_input.indicators['1-1-1'].data
+    pd.testing.assert_frame_equal(input_df, output_df)
diff --git a/tests/assets/misc/code-map.csv b/tests/assets/misc/code-map.csv
new file mode 100644
index 0000000..bf44936
--- /dev/null
+++ b/tests/assets/misc/code-map.csv
@@ -0,0 +1,3 @@
+Dimension,Text,Value
+SEX,F,Female
+SEX,M,Male
diff --git a/tests/open_sdg_test.py b/tests/open_sdg_test.py
index b224068..e8cdffe 100644
--- a/tests/open_sdg_test.py
+++ b/tests/open_sdg_test.py
@@ -10,10 +10,12 @@ def test_open_sdg():
         return meta
     def alter_indicator(indicator, context):
         return indicator
+    def my_indicator_callback(indicator):
+        assert isinstance(indicator, sdg.Indicator)
 
     config_path = os.path.join('tests', 'assets', 'open-sdg', 'config_data.yml')
     assert sdg.open_sdg_check(config=config_path, alter_data=alter_data, alter_meta=alter_meta, alter_indicator=alter_indicator)
-    assert sdg.open_sdg_build(config=config_path, alter_data=alter_data, alter_meta=alter_meta, alter_indicator=alter_indicator)
+    assert sdg.open_sdg_build(config=config_path, alter_data=alter_data, alter_meta=alter_meta, alter_indicator=alter_indicator, indicator_callback=my_indicator_callback)
 
     OutputOpenSdg_test.test_open_sdg_output_comb()
     OutputOpenSdg_test.test_open_sdg_output_data_csv()