Repository: sdss/sdsstools

Issue:
Add !extends constructor
Add a PyYAML constructor to base a YAML file on another. For example

```yaml
# file: a.yml
a: 1
b:
  c: 3
  d: 4
```

```yaml
# file: b.yaml
!extends a.yml
b:
  c: 42
```

would be parsed into `{a: 1, b: {c: 42, d: 4}}`.

This would be useful for example for APO/LCO configuration files in which on can be made the template and the other can just extend and modify it.

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=h1) Report
> Merging [#14](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=desc) into [master](https://codecov.io/gh/sdss/sdsstools/commit/c75fa6dd59ccd5c4bd58fd7c2a82765389d31eab&el=desc) will **increase** coverage by `1.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/sdss/sdsstools/pull/14/graphs/tree.svg?width=650&height=150&src=pr&token=RXDhA7SpXB)](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #14      +/-   ##
==========================================
+ Coverage   95.60%   96.61%   +1.01%     
==========================================
  Files           3        3              
  Lines         250      265      +15     
  Branches       48       53       +5     
==========================================
+ Hits          239      256      +17     
+ Misses          1        0       -1     
+ Partials       10        9       -1     
```


| [Impacted Files](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/sdsstools/configuration.py](https://codecov.io/gh/sdss/sdsstools/pull/14/diff?src=pr&el=tree#diff-c3JjL3Nkc3N0b29scy9jb25maWd1cmF0aW9uLnB5) | `100.00% <100.00%> (+2.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=footer). Last update [c75fa6d...3ff7487](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 03b5541..63e6960 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,7 @@
 
 ## [Next version](https://github.com/sdss/sdsstools/compare/0.2.1...HEAD)
 
+- Feature [#7](https://github.com/sdss/sdsstools/issues/7): add support for a tag `!extends` in YAML files when read with `read_yaml_file`.
 - Bug [#8](https://github.com/sdss/sdsstools/issues/8): log `StreamHandler` to `stderr` when the record level is `ERROR` or greater.
 - Support [#11](https://github.com/sdss/sdsstools/issues/11): replaced `colored_formatter` with a propper `Formatter` and added tests for logging.
 - Support [#12](https://github.com/sdss/sdsstools/issues/12): move `color_print` to `sdsstools._vendor`.
diff --git a/README.md b/README.md
index 1f7e05f..9647d13 100644
--- a/README.md
+++ b/README.md
@@ -70,6 +70,36 @@ In addition to the (recommended) location `~/.config/sdss/<NAME>.yaml`, `get_con
 
 `sdsstools.configuration` includes two other tools, `merge_config`, that allows to merge dictionaries recursively, and `read_yaml_file`, to read a YAML file.
 
+### Extending a YAML file
+
+`read_yaml_file` provides a non-standard feature that allows you to extend one YAML file with another. To achieve this you need to add the tag `!extends <base-file>` at the top of the file that you want to extend. For example, if you have a file `base.yaml`
+
+```yaml
+cat1:
+    key1: value2
+
+cat2:
+    key2: 1
+```
+
+that you want to use as a template for `extendable.yaml`
+
+```yaml
+#!extends base.yaml
+
+cat1:
+    key1: value1
+```
+
+you can use `read_yaml_file` to parse the result
+
+```python
+>>> read_yaml_file('extendable.yaml')
+{'cat1': {'key1': 'value2'}, 'cat2': {'key2': 1}}
+```
+
+The path to the base file must be absolute, or relative to the location of the file to be extended.
+
 ## Metadata
 
 sdsscore provides tools to locate and parse metadata files (`pyproject.toml`, `setup.cfg`, `setup.py`). `get_metadata_files` locates the path of the metadata file relative to a given `path`. `get_package_version` tries to find the version of the package by looking for a version string in the metadata file or in the egg/wheel metadata file, if the package has been installed. To use it
diff --git a/src/sdsstools/configuration.py b/src/sdsstools/configuration.py
index cd8fd40..3782a94 100644
--- a/src/sdsstools/configuration.py
+++ b/src/sdsstools/configuration.py
@@ -43,15 +43,34 @@ yaml.add_implicit_resolver('!env', env_matcher)
 yaml.add_constructor('!env', env_constructor)
 
 
-def read_yaml_file(path):
+def read_yaml_file(path, use_extends=True, loader=yaml.FullLoader):
     """Read a YAML file and returns a dictionary."""
 
     if isinstance(path, (str, pathlib.Path)):
-        with open(path, 'r') as fp:
-            config = yaml.load(fp, Loader=yaml.FullLoader)
+        fp = open(path, 'r')
     else:
-        # Assume it's an stream
-        config = yaml.load(path, Loader=yaml.FullLoader)
+        fp = path
+
+    fp.seek(0)
+    config = yaml.load(fp, Loader=loader)
+
+    if use_extends:
+        fp.seek(0)
+        while True:
+            line = fp.readline()
+            if line.strip().startswith('#!extends'):
+                base_file = line.strip().split()[1]
+                if not os.path.isabs(base_file) and hasattr(fp, 'buffer'):
+                    base_file = os.path.join(os.path.dirname(path), base_file)
+                if not os.path.exists(base_file):
+                    raise FileExistsError(f'cannot find !extends file {base_file}.')
+                return merge_config(read_yaml_file(base_file),
+                                    read_yaml_file(path, use_extends=False))
+            elif line.strip().startswith('#') or line.strip() == '':
+                continue
+            else:
+                fp.seek(0)
+                break
 
     return config
 


Test patch:
diff --git a/test/test_configuration.py b/test/test_configuration.py
index 31a0788..d84ab84 100644
--- a/test/test_configuration.py
+++ b/test/test_configuration.py
@@ -7,18 +7,38 @@
 # @License: BSD 3-clause (http://www.opensource.org/licenses/BSD-3-Clause)
 
 import inspect
+import io
 import os
 import unittest.mock
 
 import pytest
 
 from sdsstools import Configuration, get_config
-from sdsstools.configuration import DEFAULT_PATHS
+from sdsstools.configuration import DEFAULT_PATHS, read_yaml_file
 
 
 BASE_CONFIG_FILE = os.path.join(os.path.dirname(__file__), 'etc/test.yml')
 
 
+BASE = """
+cat1:
+    key1: base_value
+
+cat2:
+    key2: 1
+"""
+
+EXTENDABLE = """
+#
+
+#!extends {base_path}
+
+cat1:
+    # test
+    key1: value1
+"""
+
+
 @pytest.fixture(autouse=True)
 def cleanup():
 
@@ -64,6 +84,15 @@ def set_envvar():
     del os.environ['A_TEST_VARIABLE']
 
 
+@pytest.fixture
+def extendable(tmp_path):
+
+    base_path = tmp_path / 'base.yaml'
+    base_path.write_text(BASE)
+
+    yield io.StringIO(EXTENDABLE.format(base_path=str(base_path))), base_path
+
+
 def test_configuration(config_file):
 
     config = Configuration()
@@ -155,3 +184,47 @@ def test_get_config_bad_module(mock_func):
 
     config = get_config('test')
     assert config == {}
+
+
+def test_extends(extendable):
+
+    stream, __ = extendable
+    data = read_yaml_file(stream)
+
+    assert data['cat1']['key1'] == 'base_value'
+    assert 'cat2' in data
+
+
+def test_extends_file_not_found(extendable):
+
+    stream, base_path = extendable
+    base_path.unlink()
+
+    with pytest.raises(FileExistsError):
+        read_yaml_file(stream)
+
+
+def test_dont_extend(extendable):
+
+    stream, __ = extendable
+    data = read_yaml_file(stream, use_extends=False)
+
+    assert data['cat1']['key1'] == 'value1'
+    assert 'cat2' not in data
+
+
+def test_extends_from_file(tmp_path):
+
+    base_path = tmp_path / 'subdir' / 'base.yaml'
+    (tmp_path / 'subdir').mkdir()
+    base_path.touch()
+    base_path.write_text(BASE)
+
+    extendable_path = tmp_path / 'extendable.yaml'
+    extendable_relative = EXTENDABLE.format(base_path='subdir/base.yaml')
+    extendable_path.write_text(extendable_relative)
+
+    data = read_yaml_file(extendable_path)
+
+    assert data['cat1']['key1'] == 'base_value'
+    assert 'cat2' in data