Repository: daviskirk/climatecontrol

Issue:
Add option for outputting status of configuration setting
During application startup if is often very helpful to see which settings are using the default value, and which were manually set or overridden using environment variables.  
Currently there can be cases where a user will inadvertently set an environment variable with the wrong name and expect it to be set when it is really using a default value. 

Discussion:

Patch:
diff --git a/.coveragerc b/.coveragerc
new file mode 100644
index 0000000..cc9c803
--- /dev/null
+++ b/.coveragerc
@@ -0,0 +1,2 @@
+[report]
+show_missing = true
diff --git a/.flake8 b/.flake8
index afaba8e..1291aa2 100644
--- a/.flake8
+++ b/.flake8
@@ -1,3 +1,3 @@
 [flake8]
-ignore = W391,D105,W504
+ignore = W391,D105,W503,W504
 max-line-length = 120
diff --git a/.gitignore b/.gitignore
index a05a7ae..e4204f5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,6 +9,7 @@ __pycache__/
 # Distribution / packaging
 .Python
 env/
+bin/
 build/
 develop-eggs/
 dist/
@@ -63,3 +64,6 @@ target/
 
 # Type Checking
 .mypy_cache
+
+# Misc
+*.swp
diff --git a/.travis.yml b/.travis.yml
index a56ae8c..c38d30c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,12 +2,14 @@ language: python
 python:
   - 3.5
   - 3.6
+  - 3.7
+  - 3.8
 install:
-  - pip install coveralls mypy==0.650 flake8==3.6.0 flake8-docstrings==1.3.1 pyyaml
+  - pip install coveralls mypy==0.750 flake8==3.7.9 flake8-docstrings==1.5.0
 script:
   - flake8
   - mypy climatecontrol
-  - pip install "pytest>=3.8.1,<4.0" && python setup.py test
+  - pip install "pytest" && python setup.py test
 after_success:
   - coveralls
 deploy:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 561bd07..43b9076 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,20 @@
 
 ## [Unreleased]
 
+### Added
+
+- Allow "_from_env" postfix in variables and load settings from the specified
+  environment variable (similar to "from_file").
+- Add "update_log" property which specifies a log of all variables loaded or
+  removed/replaced.
+- Allow overwriting / merging lists
+
+### Changed
+
+- Use fragments as base data structure for building settings map
+- Add python 3.7 and 3.8 to CI and update development environment config to 3.8
+
+
 ## [0.7.3] - 2019-01-17
 
 ### Fixed
diff --git a/README.rst b/README.rst
index 2760f6c..5285341 100644
--- a/README.rst
+++ b/README.rst
@@ -1,4 +1,4 @@
-|Build Status| |Coverage Status| |PyPi Status| |PyPI license|
+|Build Status| |Coverage Status| |PyPi Status| |PyPI license| |PyPI pyversions|
 
 
 .. image:: https://raw.githubusercontent.com/daviskirk/climatecontrol/logo/climatecontrol-text.svg?sanitize=true
@@ -8,6 +8,19 @@ CLIMATECONTROL controls your applications settings and configuration
 environment. It is a Python library for loading app configurations from files
 and/or namespaced environment variables.
 
+Features
+--------
+
+* Separation of settings and code
+* Loading from files (`.yaml`, `.json`, `.toml`)
+* Loading multiple files using glob syntax
+* Loading from environment variables, including loading of nested values
+* Freely reference nested configurations via files or environment variables
+* CLI integration
+* Validation using the Validation library of your choice
+* Logging configuration integration
+* Testing integration
+
 
 Install
 -------
@@ -88,37 +101,6 @@ settings. For this situation we can delimit sections using a double underscore:
    }
 
 
-Finally if you decide that your settings are simpler and you know that your
-section names do not have underscores, you can use the ``implicit_depth``
-option, which allows you to add a new section at every single underscore (up to
-the depth you specify).
-
-.. code:: sh
-
-   export MY_APP_SECTION1_VALUE1=test1
-   export MY_APP_SECTION2_VALUE2=test2
-   export MY_APP_SECTION2_VALUE3=test3
-   export MY_APP_SECTION2_SUBSECTION_VALUE4=test4
-
-.. code:: python
-
-   settings_map = Settings(prefix='MY_APP', implicit_depth=2)
-   print(dict(settings_map))
-
-   {
-       'section1': {
-           'value1': 'test1'
-       },
-       'section2': {
-           'value2': 'test2',
-           'value3': 'test3',
-           'subsection': {
-               'value4': 'test4'
-           }
-       }
-   }
-
-
 Settings file support
 ---------------------
 
@@ -163,8 +145,11 @@ In the following documentation examples, yaml files will be used, but any
 examples will work using the other file syntaxes as well.
 
 
-Setting variables whos values are saved in files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Advanced Features
+-----------------
+
+Setting variables from values saved in files
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Sometimes we don't want to save values in plain text in environment files or in
 the settings file itself. Instead we have a file that contains the value of the
@@ -191,6 +176,33 @@ will both write the content of the file at `"/home/myuser/supersecret.txt"`
 into the variable `section1 -> subsection1`.
 
 
+Setting variables from values saved in specific environment variables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Similarly, to read a value from an environment variable, add a `"_from_env"` to
+the variable name. For example if we wanted to obtain a value from the variable
+`SPECIFIC_ENV_VAR`:
+
+.. code-block:: sh
+
+   export SPECIFIC_ENV_VAR="some value"
+
+Using a settings file with the contents (in this case yaml):
+
+.. code-block:: yaml
+
+   section1:
+     subsection1_from_env: SPECIFIC_ENV_VAR
+
+or using an environment variable:
+
+.. code-block:: sh
+
+   export MY_APP_SECTION1_SUBSECTION1_FROM_FILE="/home/myuser/supersecret.txt"
+
+will both write "some value" into the variable `section1 -> subsection1`.
+
+
 Nested settings files
 ^^^^^^^^^^^^^^^^^^^^^
 
@@ -227,10 +239,11 @@ which would result in a settings structure:
    }
 
 
-
+Integrations
+------------
 
 Command line support using click
---------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The click_ library is a great tool for creating command line applications. If
 you don't want to have to use an environment to set your configuration file.
@@ -263,6 +276,32 @@ with the last file overriting any overlapping keys of the first file.
    python cli.py --settings ./my_settings_file.toml  --settings ./my_settings_file.yaml
 
 
+Logging
+^^^^^^^
+
+If you have a "logging" section in your settings files, you can configure
+python standard library logging using that section directly:
+
+.. code:: yml
+
+   logging:
+     formatters:
+       default:
+         format': "%(levelname)s > %(message)s"
+     root:
+       level: DEBUG
+
+
+.. code:: python
+
+   import logging
+
+   settings_map = Settings(prefix='MY_APP')
+   settings_map.setup_logging()
+   logging.debug('test')
+   # outputs: DEBUG > test
+
+
 Testing
 -------
 
@@ -286,12 +325,6 @@ temporarily:
    print(settings_map['a'])  # outputs: 1
 
 
-Python version support
-----------------------
-
-Do to the use of modern python features, only python 3.5 and above are supported.
-
-
 .. |Build Status| image:: https://travis-ci.org/daviskirk/climatecontrol.svg?branch=master
    :target: https://travis-ci.org/daviskirk/climatecontrol
 .. |Coverage Status| image:: https://coveralls.io/repos/github/daviskirk/climatecontrol/badge.svg?branch=master
diff --git a/climatecontrol/env_parser.py b/climatecontrol/env_parser.py
index 7b01707..ee9a7be 100644
--- a/climatecontrol/env_parser.py
+++ b/climatecontrol/env_parser.py
@@ -1,17 +1,17 @@
 """Environment variable parser."""
 
-import json
 import logging
 import os
-from typing import Any, Dict, Iterator, Mapping, NamedTuple, Sequence, Set
-
-from .file_loaders import load_from_filepath_or_content
-from .utils import update_nested
+import warnings
+from typing import Iterable, Iterator, NamedTuple, Tuple
 
+from . import file_loaders
+from .fragment import Fragment
+from .utils import parse_as_json_if_possible
 
 logger = logging.getLogger(__name__)
 
-EnvSetting = NamedTuple('EnvSetting', [('name', str), ('value', Mapping[str, Any])])
+EnvSetting = NamedTuple('EnvSetting', [('name', str), ('value', Fragment)])
 
 
 class EnvParser:
@@ -28,6 +28,8 @@ class EnvParser:
             occurrences of a single `split_char` character will be considered
             nested settings boundaries. Note that if a file is given, the
             maximum depth does not apply as the definition is clear.
+            **WARNING:** This feature is deprecated as of version 0.8 and will be
+            removed in a future version.
         settings_file_suffix: Suffix to identify an environment variable as a
             settings file.
 
@@ -57,29 +59,30 @@ class EnvParser:
     """
 
     def __init__(self,
-                 prefix: str = 'APP_SETTINGS',
+                 prefix: str = 'CLIMATECONTROL',
                  split_char: str = '_',
                  implicit_depth: int = 0,
                  settings_file_suffix: str = 'SETTINGS_FILE',
-                 exclude: Sequence[str] = ()) -> None:
+                 exclude: Iterable[str] = ()) -> None:
         """Initialize object."""
         self.settings_file_suffix = str(settings_file_suffix)
         self.implicit_depth = int(implicit_depth)
+        if implicit_depth != 0:
+            warnings.warn('Setting "implicit_depth" is deprecated and will be removed in a future version',
+                          DeprecationWarning)
         self.split_char = split_char
         self.prefix = prefix
-        self.exclude = exclude
+        self.exclude = exclude  # type: ignore
 
     @property
-    def exclude(self) -> Set[str]:
+    def exclude(self) -> Tuple[str, ...]:
         """Return excluded environment variables."""
         exclude = self._exclude.union({self.settings_file_env_var})
-        return set(s.lower() for s in exclude)
+        return tuple(set(s.lower() for s in exclude))
 
     @exclude.setter
-    def exclude(self, exclude: Sequence = ()) -> None:
+    def exclude(self, exclude: Iterable[str] = ()) -> None:
         """Set excluded environment variables."""
-        if isinstance(exclude, str):
-            exclude = (exclude,)
         self._exclude = set(exclude)
 
     @property
@@ -115,7 +118,7 @@ class EnvParser:
             raise ValueError('``split_char`` must be a single character')
         self._split_char = str(char)
 
-    def parse(self, include_vars=True, include_file: bool = True) -> Dict[str, Any]:
+    def iter_load(self, include_vars=True, include_file: bool = True) -> Iterator[Fragment]:
         """Convert environment variables to nested dict.
 
         Note that all string inputs are case insensitive and all resulting keys
@@ -142,31 +145,25 @@ class EnvParser:
             {'testgroup': {'testvar': 27}}
 
         """
-        settings_map = {}  # type: dict
-        for env_var, settings in self._iter_parse(include_vars=include_vars, include_file=include_file):
-            logger.info('Parsed setting from env var: %s.', env_var)
-            update_nested(settings_map, settings)
-        return settings_map
+        if include_file:
+            settings_file_str = os.getenv(self.settings_file_env_var, '')
+            settings_files = [s.strip() for s in settings_file_str.split(',')]
+            for settings_file in settings_files:
+                for fragment in file_loaders.iter_load(settings_file):
+                    fragment.source = 'ENV:' + str(self.settings_file_env_var) + ':' + fragment.source
+                    yield fragment
+        if include_vars:
+            for env_var, env_var_value in os.environ.items():
+                nested_keys = list(self._iter_nested_keys(env_var))
+                if not nested_keys:
+                    continue
+                value = parse_as_json_if_possible(env_var_value)
+                fragment = Fragment(value=value, path=nested_keys, source='ENV:' + env_var)
+                yield fragment
 
     def _build_env_var(self, *parts: str) -> str:
         return self.split_char.join(self._strip_split_char(p).upper() for p in parts)
 
-    def _build_settings_update(self, keys: Sequence[str], value: Any) -> dict:
-        """Build a settings update dictionary.
-
-        Args:
-            keys: Sequence of keys, each key representing a level in a nested dictionary
-            value: Value that is assigned to the key at the deepest level.
-
-        """
-        update = {}  # type: dict
-        u = update
-        for key in keys[:-1]:
-            u[key] = {}
-            u = u[key]
-        u[keys[-1]] = value
-        return update
-
     def _iter_nested_keys(self, env_var: str) -> Iterator[str]:
         """Iterate over nested keys of an environment variable name.
 
@@ -187,45 +184,9 @@ class EnvParser:
             elif section:
                 yield section
 
-    def _iter_parse(self, include_vars: bool = True, include_file: bool = True) -> Iterator[EnvSetting]:
-        """Use in ``parse``.
-
-        Iterate over valid environment variables and files defined by
-        environment variables and yieldan envirnment settings tuple for each
-        valid entry.
-
-        See also:
-            :meth:`parse`
-
-        """
-        if include_file:
-            settings_file = os.environ.get(self.settings_file_env_var)
-            if settings_file:
-                yield EnvSetting(self.settings_file_env_var, load_from_filepath_or_content(settings_file))
-        if include_vars:
-            for env_var in os.environ:
-                nested_keys = list(self._iter_nested_keys(env_var))
-                if not nested_keys:
-                    continue
-                value = self._get_env_var_value(env_var)
-                update = self._build_settings_update(nested_keys, value)
-                if update:
-                    yield EnvSetting(env_var, update)
-
     def _strip_split_char(self, s):
         if s.startswith(self.split_char):
             s = s[len(self.split_char):]
         elif s.endswith(self.split_char):
             s = s[:-len(self.split_char)]
         return s
-
-    @staticmethod
-    def _get_env_var_value(env_var: str) -> Any:
-        """Parse an environment variable value using the toml parser."""
-        v = os.environ[env_var]
-        if isinstance(v, str):
-            try:
-                return json.loads(v)
-            except json.JSONDecodeError:
-                pass
-        return v
diff --git a/climatecontrol/file_loaders.py b/climatecontrol/file_loaders.py
index 47e61cb..4951d6b 100644
--- a/climatecontrol/file_loaders.py
+++ b/climatecontrol/file_loaders.py
@@ -1,13 +1,15 @@
 """Module for loading various file formats."""
 
+import glob
 import json
 import os
 from abc import ABC, abstractmethod
 from collections import OrderedDict
-from typing import Tuple, List  # noqa: F401
+from typing import Tuple, Iterator, List  # noqa: F401
 from typing import Any, Dict, Mapping
 
 from .exceptions import NoCompatibleLoaderFoundError
+from .fragment import Fragment
 
 try:
     import toml
@@ -19,36 +21,13 @@ except ImportError:
     yaml = None  # type: ignore
 
 
-def load_from_filepath(filepath: str, allow_unknown_file_type=False):
-    """Read settings file from a filepath.
-
-    Returns:
-        Data structure loaded/parsed from a compatible `FileLoader`. If no
-        compatible file loaded can be found and `allow_unknown_file_type` is
-        set, return the raw file contents (if `allow_unknown_file_type` is not
-        set we raise an error on this case).
-
-    Raises:
-        SettingsLoadError
-
-    """
-    try:
-        return load_from_filepath_or_content(filepath, _allow_content=False)
-    except NoCompatibleLoaderFoundError:
-        if not allow_unknown_file_type:
-            raise
-        # Load the raw contents from file and assume that they are to be
-        # interpreted as a raw string.
-        with open(filepath) as f:
-            return f.read().strip()
-
-
-def load_from_filepath_or_content(path_or_content: str, _allow_content=True) -> Dict[str, Any]:
+def iter_load(path_or_content: str) -> Iterator[Fragment]:
     """Read settings file from a filepath or from a string representing the file contents.
 
-    If ``path_or_content`` is a valid filename, load the file. If
-    ``path_or_content`` represents a json, yaml or toml string instead (the
-    contents of a json/toml/yaml file), parse the string directly.
+    If ``path_or_content`` is a valid filename or glob expression, load the
+    file (or all matching files). If ``path_or_content`` represents a json,
+    yaml or toml string instead (the contents of a json/toml/yaml file), parse
+    the string directly.
 
     Note that json, yaml and toml files are read. If ``path_or_content`` is a
     string, we will try to guess what file type you meant. Note that this last
@@ -64,18 +43,61 @@ def load_from_filepath_or_content(path_or_content: str, _allow_content=True) ->
             this filepath or content type.
 
     """
-    file_data = {}  # type: Dict
     if not path_or_content:
+        return
+    expanded_path_or_content = os.path.expanduser(os.path.expandvars(path_or_content))
+    globbed_files = glob.glob(expanded_path_or_content)  # type: List[str]
+    if globbed_files:
+        for filepath in sorted(globbed_files):
+            yield Fragment(value=load_from_filepath(filepath), source=filepath)
+        return
+
+    # assume content if no files were found
+    yield Fragment(value=load_from_content(path_or_content), source='content')
+
+
+def load_from_content(content: str) -> Dict[str, Any]:
+    """Read settings from a content string."""
+    file_data = {}  # type: Dict
+    if not content:
         return file_data
     for loader in FileLoader.registered_loaders:
-        if loader.is_path(path_or_content):
-            file_data = loader.from_path(path_or_content)
+        if loader.is_content(content):
+            file_data = loader.from_content(content)
             break
-        if _allow_content and loader.is_content(path_or_content):
-            file_data = loader.from_content(path_or_content)
+    else:
+        raise NoCompatibleLoaderFoundError(
+            'Failed to load settings from content. No compatible loader: {}'
+            .format(content)
+        )
+    return file_data
+
+
+def load_from_filepath(filepath: str) -> Dict[str, Any]:
+    """Read settings file from a filepath or from a string representing the file contents.
+
+    Args:
+        filepath: Path to file or file contents
+
+    Raises:
+        FileLoadError: when an error occurs during the loading of a file.
+        ContentLoadError: when an error occurs during the loading of file contents.
+        NoCompatibleLoaderFoundError: when no compatible loader was found for
+            this filepath or content type.
+
+    """
+    file_data = {}  # type: Dict
+    if not filepath:
+        return file_data
+    for loader in FileLoader.registered_loaders:
+        if loader.is_path(filepath):
+            file_data = loader.from_path(filepath)
             break
     else:
-        raise NoCompatibleLoaderFoundError('Failed to load settings. No compatible loader: {}'.format(path_or_content))
+        raise NoCompatibleLoaderFoundError(
+            'Failed to load settings from filepath. '
+            'No compatible loader for file: {}'.format(filepath)
+        )
     return file_data
 
 
diff --git a/climatecontrol/fragment.py b/climatecontrol/fragment.py
new file mode 100644
index 0000000..2152d2b
--- /dev/null
+++ b/climatecontrol/fragment.py
@@ -0,0 +1,148 @@
+"""Module for defining settings fragments."""
+
+from itertools import zip_longest
+from typing import Any, Iterable, Iterator, Mapping, Sequence, Union  # noqa: F401
+
+from .utils import EMPTY, get_nested, merge_nested
+
+
+class FragmentPath(Sequence):
+    """Path indicating nested levels of a fragment value."""
+
+    def __init__(self, iterable: Iterable = ()) -> None:
+        """Assign initial iterable data."""
+        self._data = list(iterable)  # type: list
+
+    def __len__(self) -> int:
+        return len(self._data)
+
+    def __iter__(self) -> Iterator:
+        yield from self._data
+
+    def __getitem__(self, index) -> Any:
+        return self._data[index]
+
+    def __repr__(self) -> str:
+        return '{}({})'.format(type(self).__qualname__, repr(self._data))
+
+    def __eq__(self, other) -> bool:
+        return type(self) == type(other) and self._data == other._data
+
+    def expand(self, value: Any = None) -> Any:
+        """Expand path to object.
+
+        Depending on each entry of the path a dictionary or list is created.
+        Entries that are not defined are will with the :data:`EMPTY` object.
+
+        Example:
+            >>> FragmentPath(['a', 1, 'b']).expand()
+            {'a': [<EMPTY>, {'b': None}]}
+
+        """
+        if not self._data:
+            return value
+        if self._data and self._is_list_index(self._data[0]):
+            new_value = [EMPTY] * (self._data[0] + 1)  # type: Union[dict, list]
+        else:
+            new_value = {}
+        sub_value = new_value
+        for subpath, next_subpath in zip_longest(self._data[:-1], self._data[1:]):
+            if self._is_list_index(next_subpath):
+                sub_value[subpath] = [EMPTY] * (next_subpath + 1)
+            else:
+                sub_value[subpath] = {}
+            sub_value = sub_value[subpath]
+
+        # last path value holds the actual value
+        if self._data:
+            sub_value[self._data[-1]] = value
+
+        return new_value
+
+    def common(self, other: Sequence) -> 'FragmentPath':
+        """Given a second path, return the part of the sequence up to the point where they first differ."""
+        common_path = []
+        other_path = type(self)(other)  # type: FragmentPath
+        for subpath, subpath_other in zip(self._data, other_path):
+            if subpath == subpath_other:
+                common_path.append(subpath)
+            else:
+                break
+        return type(self)(common_path)
+
+    def _is_list_index(self, index) -> bool:
+        """Check if index is a list index."""
+        return isinstance(index, int)
+
+
+class Fragment:
+    """Data fragment for storing a value and metadata related to it."""
+
+    def __init__(self, value: Any,
+                 source: str = '',
+                 path: Sequence = ()) -> None:
+        """Initialize fragment."""
+        self.value = value
+        self.source = source
+        self.path = FragmentPath(path)
+
+    def __repr__(self):
+        return '{}(value={}, source={}, path={})'.format(
+            type(self).__qualname__,
+            repr(self.value),
+            repr(self.source),
+            repr(self.path),
+        )
+
+    def __eq__(self, other):
+        return (
+            type(self) == type(other) and
+            self.value == other.value and
+            self.source == other.source and
+            self.path == other.path
+        )
+
+    def iter_leaves(self) -> Iterator['Fragment']:
+        """Iterate over all leaves of a fragment.
+
+        A leaf is obtained by walking through any nested dictionaries until a
+        non-dictionary value is found.
+
+        """
+        if isinstance(self.value, Mapping):
+            items = self.value.items()  # type: Iterable[tuple]
+        elif isinstance(self.value, Sequence) and not isinstance(self.value, str):
+            items = enumerate(self.value)
+        else:
+            # Can't obtain any items so just assume this is a leaf
+            yield self
+            return
+
+        for k, v in items:
+            yield from self.clone(value=v, path=list(self.path) + [k]).iter_leaves()
+
+    def expand_value_with_path(self) -> Any:
+        """Create expanded dictionary where the fragments path acts as nested keys."""
+        return self.path.expand(self.value)
+
+    def merge(self, other: 'Fragment') -> 'Fragment':
+        """Merge with another fragment."""
+        expanded_value = self.expand_value_with_path()
+        other_expanded_value = other.expand_value_with_path()
+        merged_value = merge_nested(expanded_value, other_expanded_value)
+
+        new_path = self.path.common(other.path)
+        new_value = get_nested(merged_value, new_path)
+        new_source = ', '.join(str(s) for s in [self.source, other.source] if s)
+
+        return self.clone(value=new_value, source=new_source, path=new_path)
+
+    def clone(self, **kwargs):
+        """Clone fragment but using ``kwargs`` as alternative constructor arguments."""
+        defaults = {
+            'value': self.value,
+            'source': self.source,
+            'path': self.path,
+        }
+        updated_kwargs = {**defaults, **kwargs}
+        return type(self)(**updated_kwargs)
diff --git a/climatecontrol/settings_parser.py b/climatecontrol/settings_parser.py
index 31948ce..4d7dc76 100644
--- a/climatecontrol/settings_parser.py
+++ b/climatecontrol/settings_parser.py
@@ -1,14 +1,18 @@
 """Settings parser."""
 
-import json
+from enum import Enum
+import itertools
 import logging
+import warnings
 import os
 from contextlib import contextmanager
 from copy import deepcopy
 from pprint import pformat
-from typing import List, Type, TypeVar, Tuple # noqa F401
-from typing import (cast, Any, Callable, Sequence,
-                    Optional, Union, Mapping, Dict, Iterator, NamedTuple)
+from typing import (  # noqa: F401
+    Any, Callable, Dict, Iterable, Iterator, List,
+    Mapping, MutableMapping, MutableSequence, Optional,
+    Sequence, TypeVar, Union, cast
+)
 
 try:
     import click
@@ -17,18 +21,18 @@ except ImportError:
 
 from .env_parser import EnvParser
 from .exceptions import SettingsValidationError  # noqa: F401  # Import here for backwards compatability.
-from .file_loaders import FileLoader, load_from_filepath, load_from_filepath_or_content
+from .file_loaders import (
+    FileLoader, NoCompatibleLoaderFoundError, iter_load, load_from_filepath
+)
+from .fragment import Fragment
 from .logtools import DEFAULT_LOG_SETTINGS, logging_config
-from .utils import iter_hierarchy, update_nested
+from .utils import merge_nested, parse_as_json_if_possible
 
 
 logger = logging.getLogger(__name__)
 T = TypeVar('T')
 
 
-Fragment = NamedTuple('Fragment', [('data', Dict[str, Any]), ('source', str)])
-
-
 class Settings(Mapping):
     """A Settings instance allows settings to be loaded from a settings file or environment variables.
 
@@ -42,14 +46,14 @@ class Settings(Mapping):
             result of the settings. The function should take a single
             nested dictionary argument (the settings map) as an argument
             and output a nested dictionary.
-        update_on_init: If set to `False` no parsing is performed upon
+        update_on_init: (Deprecated) If set to `False` no parsing is performed upon
             initialization of the object. You will need to call update
             manually if you want load use any settings.
 
     Args:
         settings_files: See attribute
         parser: See attribute
-        update_on_init: If set to ``True``, read all configurations upon initialization.
+        update_on_init: (Deprecated) If set to ``True``, read all configurations upon initialization.
         **env_parser_kwargs: Arguments passed to :class:`EnvParser` constructor.
 
     Example:
@@ -58,8 +62,8 @@ class Settings(Mapping):
         >>> os.environ['MY_APP_SECTION1_SUB1'] = 'test1'
         >>> os.environ['MY_APP_SECTION2_SUB2'] = 'test2'
         >>> os.environ['MY_APP_SECTION2_SUB3'] = 'test3'
-        >>> settings_map = Settings(prefix='MY_APP', implicit_depth=1)
-        >>> dict(settings_map)
+        >>> settings_manager = SettingsManager(prefix='MY_APP')
+        >>> dict(settings_manager)
         {'value0': 'test0', 'section1': {'subsection1': 'test1'}, 'section2': {'sub2': 'test2', 'sub3': 'test3'}}
 
     See Also:
@@ -68,17 +72,21 @@ class Settings(Mapping):
     """
 
     def __init__(self,
-                 settings_files: Optional[Sequence[str]] = None,
-                 parser: Optional[Callable] = None,
-                 update_on_init: bool = True,
+                 settings_files: Sequence[str] = (),
+                 parser: Optional[Callable[[Mapping], Mapping]] = None,
+                 update_on_init: Optional[bool] = None,
                  **env_parser_kwargs) -> None:
         """Initialize settings object."""
         self.env_parser = EnvParser(**(env_parser_kwargs or {}))
         self.parser = parser
         self.settings_files = settings_files
         self.update_data = {}  # type: dict
+        self.fragments = []  # type: List[Fragment]
         self._data = {}  # type: Mapping
+        self._initialized = False  # type: bool
 
+        if update_on_init is not None:
+            warnings.warn('setting update_on_init is deprecated', DeprecationWarning)
         if update_on_init:
             self.update()
 
@@ -89,18 +97,22 @@ class Settings(Mapping):
         return len(self._data)
 
     def __getitem__(self, key: str) -> Any:
+        if not self._initialized:
+            self.update()
         return self._data[key]
 
     def __iter__(self) -> Iterator[str]:
+        if not self._initialized:
+            self.update()
         yield from self._data
 
     @property
-    def parser(self) -> Optional[Callable]:
+    def parser(self) -> Optional[Callable[[Mapping], Mapping]]:
         """Return settings parser function."""
         return self._parse
 
     @parser.setter
-    def parser(self, value: Optional[Callable]) -> None:
+    def parser(self, value: Optional[Callable[[Mapping], Mapping]]) -> None:
         """Set the settings parser function."""
         if value and not callable(value):
             raise TypeError('If given, ``parser`` must be a callable')
@@ -112,14 +124,28 @@ class Settings(Mapping):
         return self._settings_files
 
     @settings_files.setter
-    def settings_files(self, files: Sequence) -> None:
+    def settings_files(self, files: Union[str, Iterable[str]]) -> None:
         """Set settings files to use when parsing settings."""
         files = files or ()
         if isinstance(files, str):
             files = (files,)
         self._settings_files = list(files)
 
-    def update(self, d: Optional[Union[Mapping, Dict]] = None, clear: bool = False) -> None:
+    @property
+    def update_log(self) -> str:
+        """Log of all each loaded settings variable."""
+        def iter_fragment_lines(fragment: Fragment) -> Iterator[str]:
+            for leaf in fragment.iter_leaves():
+                action = 'removed' if leaf.value == REMOVED else 'loaded'
+                yield action + ' ' + '.'.join(str(p) for p in leaf.path) + ' from ' + str(leaf.source)
+
+        lines = itertools.chain.from_iterable(
+            iter_fragment_lines(fragment) for fragment in self.fragments
+        )
+        result = '\n'.join(lines)
+        return result
+
+    def update(self, d: Optional[Mapping] = None, clear: bool = False) -> None:
         """Update object settings and reload files and environment variables.
 
         Args:
@@ -141,29 +167,44 @@ class Settings(Mapping):
             {'section': {'value': 'test', 'new_value': 'new'}, 'section2': {'new_env_value': 'new_env_data'}}
 
         """
-        settings_map = {}  # type: dict
-
         # External data is any data that was explicitely assigned through a
         # call to :meth:`update`.
-        update_data = {} if clear else deepcopy(self.update_data)
+        update_data = {} if clear else self.update_data
         if d:
-            update_nested(update_data, d)
-
-        def update_fragment(fragment: Fragment) -> None:
-            preprocessed = Fragment(self.preprocess_fragment(fragment.data), fragment.source)
-            self._log_assignments(preprocessed)
-            update_nested(settings_map, preprocessed.data)
-
-        # Update settings
-        for fragment in self._iter_load_files():
-            update_fragment(fragment)
-        update_fragment(Fragment(self.env_parser.parse(), 'environment variables'))
-        update_fragment(Fragment(update_data, 'external'))
+            update_data = merge_nested(update_data, d)
+
+        # Compile a list of fragments
+        fragments = []
+
+        for fragment in itertools.chain(
+                self._iter_load_files(),
+                self.env_parser.iter_load(),
+                [Fragment(value=update_data, source='external')]
+        ):
+            fragments.append(fragment)
+            fragments.extend(self.process_fragment(fragment))
+
+        # Combine the fragments into one final fragment
+        combined_fragment = None  # type: Optional[Fragment]
+        for fragment in fragments:
+            if combined_fragment is None:
+                combined_fragment = fragment
+            else:
+                combined_fragment = combined_fragment.merge(fragment)
+
+        # Obtain settings map
+        if combined_fragment is None:
+            settings_map = {}  # type: dict
+        else:
+            settings_map = combined_fragment.expand_value_with_path()
+            clean_removed_items(settings_map)
 
         self._data = self.parse(settings_map)
 
-        # If parsing was successfull, update external data
+        # If parsing was successfull, update external data and fragments.
         self.update_data = update_data
+        self.fragments = fragments
+        self._initialized = True
 
     def parse(self, data: Mapping) -> Mapping:
         """Parse data into settings.
@@ -188,20 +229,26 @@ class Settings(Mapping):
         ``climatecontrol`` package.
 
         """
-        logging_settings = deepcopy(DEFAULT_LOG_SETTINGS)
-        logging_settings_update = self.get(logging_section, {})
+        logging_settings = DEFAULT_LOG_SETTINGS
+        logging_settings_update = self.get(logging_section)
         if logging_settings_update:
-            update_nested(logging_settings, logging_settings_update)
+            logging_settings = merge_nested(logging_settings, logging_settings_update)
         logging_config.dictConfig(logging_settings)
 
-    def click_settings_file_option(self, **kw) -> Callable:
+    def click_settings_file_option(self, **kw) -> Callable[..., Any]:
         """See :func:`cli_utils.click_settings_file_option`."""
         from . import cli_utils
         return cli_utils.click_settings_file_option(self, **kw)
 
-    def preprocess_fragment(self, fragment: T) -> T:
+    def process_fragment(self, fragment: Fragment) -> Iterator[Fragment]:
         """Preprocess a settings fragment and return the new version."""
-        return self._render_from_file_vars(fragment)
+        processors = [
+            replace_from_file_vars, replace_from_env_vars
+        ]  # type: List[Callable[[Fragment], Iterator[Fragment]]]
+        for process in processors:
+            for new_fragment in process(fragment):
+                yield new_fragment
+                yield from self.process_fragment(new_fragment)
 
     def to_config(self, *, save_to: str = None, style: str = '.json') -> Optional[str]:
         """Generate a settings file from the current settings."""
@@ -209,7 +256,7 @@ class Settings(Mapping):
             style = os.path.splitext(save_to)[1]
         for loader in FileLoader.registered_loaders:
             if style in loader.valid_file_extensions:
-                s = loader.to_content(self._data)
+                s = loader.to_content(dict(self))
                 break
         else:
             raise ValueError('Not a valid style / file extension: {}'.format(style))
@@ -239,70 +286,114 @@ class Settings(Mapping):
 
         """
         archived_settings = deepcopy(self._data)
-        archived_settings_files = deepcopy(self._settings_files)
+        archived_settings_files = deepcopy(self.settings_files)
         archived_update_data = deepcopy(self.update_data)
         yield self
         self._data = archived_settings
-        self._settings_files = archived_settings_files
+        self.settings_files = archived_settings_files
         self.update_data = archived_update_data
 
-    def _render_from_file_vars(self, data: T, postfix_trigger='_from_file') -> T:
-        """Read and replace settings values from content local files.
+    def _iter_load_files(self) -> Iterator[Fragment]:
+        for entry in self.settings_files:
+            yield from iter_load(entry)
 
-        Args:
-            data: Given subset of settings data (or entire settings mapping)
-            postfix_trigger: Optionally configurable string to trigger a local
-                file value. If a key is found which ends with this string, the
-                value is assumed to be a file path and the settings value will
-                be set to the content of the file.
 
-        Returns:
-            An updated copy of `data` with keys and values replaced accordingly.
+def replace_from_env_vars(fragment: Fragment, postfix_trigger: str = '_from_env') -> Iterator[Fragment]:
+    """Read and replace settings values from environment variables.
 
-        """
-        if not data:
-            return data
-        elif isinstance(data, Mapping):
-            new_data = {k: v for k, v in data.items()}  # type: Any
-            items = tuple(data.items())
-        elif isinstance(data, Sequence) and not isinstance(data, str):
-            new_data = [item for item in data]
-            items = tuple(enumerate(data))
-        else:
-            return cast(T, data)
-        for k, v in items:
-            if isinstance(v, str) and isinstance(k, str) and k.lower().endswith(postfix_trigger):
-                key_with_postfix = k
-                filepath = v
-                # Reassign value (v) using the contents of the file.
+    Args:
+        fragment: Fragment to process
+        postfix_trigger: Optionally configurable string to trigger a
+            replacement with an environment variable. If a key is found which
+            ends with this string, the value is assumed to be the name of an
+            environemtn variable and the settings value will be set to the
+            contents of that variable.
+
+    Yields:
+        Additional fragments to patch the original fragment.
+
+    """
+    for leaf in fragment.iter_leaves():
+        if not leaf.path or leaf.value == REMOVED:
+            continue
+        key, value = leaf.path[-1], leaf.value
+        if isinstance(value, str) and isinstance(key, str) and key.lower().endswith(postfix_trigger):
+            env_var = value
+            yield leaf.clone(value=REMOVED)
+            try:
+                env_var_value = os.environ[env_var]
+            except KeyError as e:
+                logger.info('Error while trying to load environment variable: %s. (%s) Skipping...',
+                            env_var, e)
+            else:
+                new_key = key[:-len(postfix_trigger)]
+                new_value = parse_as_json_if_possible(env_var_value)
+                yield leaf.clone(value=new_value, path=list(leaf.path[:-1]) + [new_key])
+
+
+def replace_from_file_vars(fragment: Fragment, postfix_trigger: str = '_from_file') -> Iterator[Fragment]:
+    """Read and replace settings values from content local files.
+
+    Args:
+        fragment: Fragment to process
+        postfix_trigger: Optionally configurable string to trigger a local
+            file value. If a key is found which ends with this string, the
+            value is assumed to be a file path and the settings value will
+            be set to the content of the file.
+
+    Yields:
+        Additional fragments to patch the original fragment.
+
+    """
+    for leaf in fragment.iter_leaves():
+        if not leaf.path or leaf.value == REMOVED:
+            continue
+        key, value = leaf.path[-1], leaf.value
+        if isinstance(value, str) and isinstance(key, str) and key.lower().endswith(postfix_trigger):
+            filepath = value
+            yield leaf.clone(value=REMOVED)
+            try:
                 try:
-                    v = load_from_filepath(filepath, allow_unknown_file_type=True)
-                except FileNotFoundError as e:
-                    logger.info('Error while trying to load variable from file: %s. (%s) Skipping...',
-                                filepath, e)
-                else:
-                    k = k[:-len(postfix_trigger)]  # Use the "actual" key from here on.
-                    new_data[k] = v
-                    logger.info('Settings key %s set to contents of file "%s"', k, filepath)
-                finally:
-                    del new_data[key_with_postfix]
-            if isinstance(v, (Mapping, Sequence)) and not isinstance(v, str):
-                parsed_v = self._render_from_file_vars(v)
-                new_data[k] = parsed_v
-        return new_data
+                    new_value = load_from_filepath(filepath)  # type: Any
+                except NoCompatibleLoaderFoundError:
+                    # just load as plain text file and interpret as string
+                    with open(filepath) as f:
+                        new_value = f.read().strip()
+            except FileNotFoundError as e:
+                logger.info('Error while trying to load variable from file: %s. (%s) Skipping...',
+                            filepath, e)
+            else:
+                new_key = key[:-len(postfix_trigger)]
+                yield leaf.clone(value=new_value, path=list(leaf.path[:-1]) + [new_key])
+
+
+def clean_removed_items(obj):
+    """Remove all keys that contain a removed key indicated by a :data:``REMOVED`` object."""
+    if isinstance(obj, MutableMapping):
+        items = obj.items()  # type: Any
+    elif isinstance(obj, MutableSequence):
+        items = enumerate(obj)
+    else:
+        return
+
+    keys_to_remove = []
+    for key, value in items:
+        if value == REMOVED:
+            keys_to_remove.append(key)
+        else:
+            clean_removed_items(value)
 
-    def _iter_load_files(self) -> Iterator[Fragment]:
-        for settings_file in self.settings_files:
-            file_update = load_from_filepath_or_content(settings_file)
-            yield Fragment(data=file_update, source=str(settings_file))
-
-    def _log_assignments(self, fragment: Fragment) -> None:
-        messages = []
-        for levels in iter_hierarchy(fragment.data):
-            if levels:
-                message = '.'.join(levels)
-                messages.append(message)
-        if messages:
-            logger.debug('Assigned settings%s: %s',
-                         ' from ' + str(fragment.source) if fragment.source else '',
-                         json.dumps(messages))
+    for key in keys_to_remove:
+        del obj[key]
+
+
+class _Removed(Enum):
+    """Object representing an empty item."""
+
+    REMOVED = None
+
+    def __repr__(self):
+        return '<REMOVED>'  # pragma: nocover
+
+
+REMOVED = _Removed.REMOVED
diff --git a/climatecontrol/utils.py b/climatecontrol/utils.py
index 90b72f2..5a85023 100644
--- a/climatecontrol/utils.py
+++ b/climatecontrol/utils.py
@@ -1,48 +1,78 @@
 """Utility functions."""
 
-from typing import Type  # noqa: F401
-from typing import Dict, Iterator, Mapping, Sequence, Union
+from copy import deepcopy
+from enum import Enum
+import json
+from itertools import zip_longest
+from typing import Any, Mapping, Sequence, Union
 
 
-def update_nested(d: Dict, u: Mapping) -> Dict:
-    """Update nested mapping ``d`` with nested mapping ``u``."""
-    for k, v in u.items():
-        if isinstance(v, Mapping):
-            r = update_nested(d.get(k, {}), v)
-            d[k] = r
-        else:
-            d[k] = u[k]
-    return d
+def get_nested(obj: Union[Mapping, Sequence], path: Sequence) -> Any:
+    """Get element of a sequence or map based on multiple nested keys.
 
+    Args:
+        obj: Object to index from
+        path: Sequence of nested keys.
 
-def iter_hierarchy(data: Union[Mapping, Sequence], levels: Sequence[str] = ()) -> Iterator[Sequence[str]]:
-    """Iterate over nested keys and yield a tuple describing the level of hierarchy of each leaf.
+    Example:
+        >>> get_nested({'a': {'b': [1, 2]}}, ['a', 'b', 0])
+        1
 
-    Args:
-        data: Nested mapping or non-string sequence.
+    """
+    result = obj
+    for subpath in path:
+        result = result[subpath]
+    return result
+
+
+def merge_nested(d: Any, u: Any) -> Any:
+    """Merge nested mapping or sequence ``d`` with nested mapping or sequence ``u``.
+
+    Dictionaries are merge recursively while sequences are merged by index (and
+    expanded automatically if longer). Note that the special value
+    :data:``EMPTY`` can be used in `u` to NOT overwrite a sequence item.
 
     Example:
-        >>> list(_iter_hierarchy({'a': {'b': [{'c': 'item'}, 123]}}))
-        [('a', 'b', '[0]', 'c'), ('a', 'b', '[1]')]
+        merge_nested({'a': {'b': [3, {'c': 4}, 5]}}, {'a': {'b': [EMPTY, {'d': 6}]}})
+        {'a': {'b': [3, {'c': 4, 'd': 6}, 5]}}
 
     """
-    levels = list(levels)
-    if not data:
-        yield levels
-        return data
-    elif isinstance(data, Mapping):
-        level_type = Mapping  # type: Type
-        items = tuple(data.items())
-    elif isinstance(data, Sequence) and not isinstance(data, str):
-        level_type = Sequence
-        items = tuple(enumerate(data))
-    else:
-        yield levels
-        return
-    for k, v in items:
-        new_level = '[{}]'.format(k) if level_type is Sequence else str(k)
-        new_levels = tuple(levels + [new_level])
-        if isinstance(v, (Mapping, Sequence)) and not isinstance(v, str):
-            yield from iter_hierarchy(v, new_levels)
-        else:
-            yield new_levels
+    if isinstance(d, Mapping):
+        new_dict = dict(**d)  # type: dict
+        if not isinstance(u, Mapping):
+            return deepcopy(u)
+        for k, u_v in u.items():
+            new_dict[k] = merge_nested(d.get(k), u_v)
+        return new_dict
+    elif isinstance(d, Sequence) and not isinstance(d, str):
+        if not isinstance(u, Sequence) or isinstance(u, str):
+            return deepcopy(u)
+        new_list = [
+            merge_nested(d_item, u_item) if u_item is not EMPTY else d_item
+            for d_item, u_item
+            in zip_longest(d, u, fillvalue=EMPTY)
+        ]
+        return new_list
+    return deepcopy(u)
+
+
+def parse_as_json_if_possible(v: str) -> Any:
+    """Parse a string value as json if possible, but fallback to the string if not."""
+    if isinstance(v, str):
+        try:
+            return json.loads(v)
+        except json.JSONDecodeError:
+            pass
+    return v
+
+
+class _Empty(Enum):
+    """Object representing an empty item."""
+
+    EMPTY = None
+
+    def __repr__(self):
+        return '<EMPTY>'  # pragma: nocover
+
+
+EMPTY = _Empty.EMPTY
diff --git a/environment.yml b/environment.yml
index c62b421..f49709f 100644
--- a/environment.yml
+++ b/environment.yml
@@ -1,7 +1,15 @@
 name: climatecontrol
 dependencies:
-  - python=3.6
+  - python=3.8
+  - pip=19
   - pyyaml
-  - pip
+  - click
+  - pytest
+  - pytest-mock
+  - pytest-cov
+  - toml
+  - flake8
+  - mypy
+  - ipython
   - pip:
-    - toml~=0.9.3.0
+    - pdbpp
diff --git a/setup.py b/setup.py
index 9377749..6423d60 100644
--- a/setup.py
+++ b/setup.py
@@ -5,12 +5,10 @@
 import os
 from setuptools import setup
 
-requirements = [
-    'typing'
-]
+requirements = []  # type: list
 
 test_requirements = [
-    'pytest>=3.2.1,<4.0.0',
+    'pytest',
     'pytest-mock',
     'pytest-cov>=2.5.1',
     'toml>=0.9.2',
@@ -20,23 +18,11 @@ test_requirements = [
 
 rootdir = os.path.abspath(os.path.dirname(__file__))
 
-
-def read(fname):
-    """Read a file from path.
-
-    Utility function to read the README file. Used for the long_description.
-    It's nice, because now 1) we have a top level README file and 2) it's
-    easier to type in the README file than to put a raw string in below ...
-
-    """
-    return open(os.path.join(rootdir, fname)).read()
-
-
 setup(
     name='climatecontrol',
     use_scm_version=True,
     description='Python library for loading app configurations from files and/or namespaced environment variables',
-    long_description=read('README.rst'),
+    long_description=open(os.path.join(rootdir, 'README.rst')).read(),
     author='Davis Kirkendall',
     author_email='davis.e.kirkendall@gmail.com',
     url='https://github.com/daviskirk/climatecontrol',
@@ -58,6 +44,7 @@ setup(
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
     ],
     setup_requires=['pytest-runner', 'setuptools_scm'],
     tests_require=test_requirements


Test patch:
diff --git a/tests/test_env_parser.py b/tests/test_env_parser.py
index 4464152..2ec3e34 100644
--- a/tests/test_env_parser.py
+++ b/tests/test_env_parser.py
@@ -1,10 +1,12 @@
 """Test settings."""
 
 import os
+import sys
 
 import pytest
 
 from climatecontrol.env_parser import EnvParser  # noqa: I100
+from climatecontrol.fragment import Fragment
 
 
 @pytest.mark.parametrize('attr, value, expected', [
@@ -12,7 +14,7 @@ from climatecontrol.env_parser import EnvParser  # noqa: I100
     ('settings_file_suffix', 'suffix2', 'suffix2'),
     ('settings_file_env_var', 'wrongval', None),
 ])
-def test_env_parser_assign(mock_empty_os_environ, attr, value, expected):
+def test_envparser_assign(mock_empty_os_environ, attr, value, expected):
     """Check that we can assign attributes of env parser."""
     s = EnvParser(prefix='this', settings_file_suffix='suffix')
     assert s.prefix == 'THIS_'
@@ -29,88 +31,64 @@ def test_env_parser_assign(mock_empty_os_environ, attr, value, expected):
             assert s.settings_file_env_var == 'THIS_' + expected.upper()
 
 
-@pytest.mark.parametrize('prefix, implicit_depth, split_char, expected', [
-    ('TEST_STUFF', 1, '_', {'testgroup': {'testvar': 7, 'test_var': 9}}),
-    ('TEST_STUFF', 1, '-', {}),
-    ('TEST_STUFF', 0, '_', {'testgroup': {'testvar': 7, 'test_var': 6}, 'testgroup_test_var': 9}),
-    ('TEST_STUFF_', 1, '_', {'testgroup': {'testvar': 7, 'test_var': 9}}),
-    ('TEST_STUFFING', 1, '_', {}),
+@pytest.mark.parametrize('prefix, implicit_depth, split_char, expected_kws', [
+    ('TEST_STUFF', 1, '_', [
+        dict(value=6, source='ENV:TEST_STUFF_TESTGROUP__TEST_VAR', path=['testgroup', 'test_var']),
+        dict(value=7, source='ENV:TEST_STUFF_TESTGROUP__TESTVAR', path=['testgroup', 'testvar']),
+        dict(value=9, source='ENV:TEST_STUFF_TESTGROUP_TEST_VAR', path=['testgroup', 'test_var'])
+    ]),
+    ('TEST_STUFF', 1, '-', []),
+    ('TEST_STUFF', 0, '_', [
+        dict(value=6, source='ENV:TEST_STUFF_TESTGROUP__TEST_VAR', path=['testgroup', 'test_var']),
+        dict(value=7, source='ENV:TEST_STUFF_TESTGROUP__TESTVAR', path=['testgroup', 'testvar']),
+        dict(value=9, source='ENV:TEST_STUFF_TESTGROUP_TEST_VAR', path=['testgroup_test_var'])
+    ]),
+    ('TEST_STUFF_', 1, '_', [
+        dict(value=6, source='ENV:TEST_STUFF_TESTGROUP__TEST_VAR', path=['testgroup', 'test_var']),
+        dict(value=7, source='ENV:TEST_STUFF_TESTGROUP__TESTVAR', path=['testgroup', 'testvar']),
+        dict(value=9, source='ENV:TEST_STUFF_TESTGROUP_TEST_VAR', path=['testgroup', 'test_var'])
+    ]),
+    ('TEST_STUFFING', 1, '_', []),
 ])
-def test_parse_environment_vars(mock_os_environ, prefix, implicit_depth, split_char, expected):
+def test_envparser_iter_load(mock_os_environ, prefix, implicit_depth, split_char, expected_kws):
     """Check that we can parse settings from variables."""
     env_parser = EnvParser(
         prefix=prefix,
         split_char=split_char,
         implicit_depth=implicit_depth)
-    result = env_parser.parse()
-    assert result == expected
+    expected = [Fragment(**kw) for kw in expected_kws]
+    results = list(env_parser.iter_load())
+    assert results == expected
 
 
-@pytest.mark.parametrize('implicit_depth, environ, expected', [
+@pytest.mark.parametrize('environ, expected_kw', [
     (
-        1,
-        {
-            'TEST_STUFF_TESTGROUP_TEST_INT': '6',
-            'TEST_STUFF_TESTGROUP_TEST_ARRAY': '[4, 5, 6]',
-            'TEST_STUFF_TESTGROUP_TEST_RAW_STR': 'al//asdjk',
-            'TEST_STUFF_TESTGROUP_TEST_STR': '"[4, 5, 6]"',
-            'TEST_STUFF_TESTGROUP_TEST_AMQP': 'amqp://guest:guest@127.0.0.1:5672//'
-        },
-        {
-            'testgroup': {
-                'test_int': 6,
-                'test_array': [4, 5, 6],
-                'test_raw_str': 'al//asdjk',
-                'test_str': '[4, 5, 6]',
-                'test_amqp': 'amqp://guest:guest@127.0.0.1:5672//',
-            }
-        }
-    ),
-    (
-        2,
-        {
-            'TEST_STUFF_TESTGROUP_TEST_INT': '6',
-            'TEST_STUFF_TESTGROUP_TEST_ARRAY': '[4, 5, 6]',
-            'TEST_STUFF_TESTGROUP_TEST_RAW_STR': 'al//asdjk',
-            'TEST_STUFF_TESTGROUP_TEST_STR': '"[4, 5, 6]"',
-            'TEST_STUFF_TESTGROUP_TEST_AMQP': 'amqp://guest:guest@127.0.0.1:5672//'
-        },
-        {
-            'testgroup': {
-                'test': {
-                    'int': 6,
-                    'array': [4, 5, 6],
-                    'raw_str': 'al//asdjk',
-                    'str': '[4, 5, 6]',
-                    'amqp': 'amqp://guest:guest@127.0.0.1:5672//',
-                }
-            }
-        }
-    ),
-    (
-        -1,
         {
             'TEST_STUFF_TESTGROUP__TEST_INT': '6',
             'TEST_STUFF_TESTGROUP__TEST_ARRAY': '[4, 5, 6]',
             'TEST_STUFF_TESTGROUP__TEST_RAW_STR': 'al//asdjk',
             'TEST_STUFF_TESTGROUP__TEST_STR': '"[4, 5, 6]"',
-            'TEST_STUFF_TESTGROUP__TEST_AMQP': 'amqp://guest:guest@127.0.0.1:5672//'
+            'TEST_STUFF_TESTGROUP__TEST_COMPLEX_RAW_STR': 'amqp://guest:guest@127.0.0.1:5672//'
         },
-        {
-            'testgroup': {
-                'test_int': 6,
-                'test_array': [4, 5, 6],
-                'test_raw_str': 'al//asdjk',
-                'test_str': '[4, 5, 6]',
-                'test_amqp': 'amqp://guest:guest@127.0.0.1:5672//',
-            }
-        }
+        [
+            {'value': 6, 'source': 'ENV:TEST_STUFF_TESTGROUP__TEST_INT', 'path': ['testgroup', 'test_int']},
+            {'value': [4, 5, 6], 'source': 'ENV:TEST_STUFF_TESTGROUP__TEST_ARRAY', 'path': ['testgroup', 'test_array']},
+            {'value': 'al//asdjk', 'source': 'ENV:TEST_STUFF_TESTGROUP__TEST_RAW_STR', 'path': ['testgroup', 'test_raw_str']},  # noqa: E501
+            {'value': '[4, 5, 6]', 'source': 'ENV:TEST_STUFF_TESTGROUP__TEST_STR', 'path': ['testgroup', 'test_str']},
+            {'value': 'amqp://guest:guest@127.0.0.1:5672//', 'source': 'ENV:TEST_STUFF_TESTGROUP__TEST_COMPLEX_RAW_STR', 'path': ['testgroup', 'test_complex_raw_str']},  # noqa: E501
+        ]
     ),
-
 ])
-def test_parse_toml(monkeypatch, implicit_depth, environ, expected):
+def test_parse_json(monkeypatch, environ, expected_kw):
     """Check that we can parse toml from environment variables."""
     monkeypatch.setattr(os, 'environ', environ)
-    env_parser = EnvParser(prefix='TEST_STUFF', implicit_depth=implicit_depth)
-    result = env_parser.parse()
-    assert result == expected
+    env_parser = EnvParser(prefix='TEST_STUFF')
+    expected = [Fragment(**kw) for kw in expected_kw]
+    result = list(env_parser.iter_load())
+    if sys.version_info[:2] >= (3, 6):
+        assert result == expected
+    else:
+        # python 3.5 doesn't order dicts so we can't test the exact order
+        def to_set(fragments):
+            return set(str(f) for f in fragments)
+        assert to_set(result) == to_set(expected)
diff --git a/tests/test_file_loaders.py b/tests/test_file_loaders.py
new file mode 100644
index 0000000..79bbc42
--- /dev/null
+++ b/tests/test_file_loaders.py
@@ -0,0 +1,1 @@
+"""Test file loaders."""
diff --git a/tests/test_fragment.py b/tests/test_fragment.py
new file mode 100644
index 0000000..0d25e40
--- /dev/null
+++ b/tests/test_fragment.py
@@ -0,0 +1,180 @@
+"""Tests for fragments."""
+
+import pytest
+import sys
+
+from climatecontrol.fragment import Fragment, FragmentPath, EMPTY, merge_nested
+
+
+def test_fragment_path():
+    """Test fragment path construction."""
+    assert not FragmentPath()  # empty path is falsy
+    assert FragmentPath([]) == FragmentPath(), 'empty fragment paths should be equal'
+    assert FragmentPath(['a', 'stuff', 1]) == FragmentPath(['a', 'stuff', 1]), 'Unexpected equality result'
+    assert FragmentPath(['a', 'stuff', 1]) != FragmentPath(['a', 'wrong', 1]), 'Unexpected inequality result'
+    assert list(FragmentPath(['a', 'stuff', 1])) == ['a', 'stuff', 1], 'Unexpected list conversion result'
+    assert str(FragmentPath(['a', 'stuff', 1])) == "FragmentPath(['a', 'stuff', 1])", 'Unexpected string representation'
+    assert FragmentPath(['a', 'stuff', 1])[1] == 'stuff', 'unexpected indexing'
+
+
+@pytest.mark.parametrize('path, expected', [
+    (
+        ['a'],
+        {'a': 'test'}
+    ),
+    (
+        ['a', 'stuff', 1, 'bla'],
+        {
+            'a': {
+                'stuff': [
+                    EMPTY,
+                    {'bla': 'test'}
+                ]
+            }
+        }
+    ),
+    (
+        [2, 'a'],
+        [EMPTY, EMPTY, {'a': 'test'}]
+    )
+])
+def test_fragment_path_expand(path, expected):
+    """Test expansion of fragment path."""
+    assert FragmentPath(path).expand('test') == expected
+
+
+@pytest.mark.parametrize('a,b,expected', [
+    (['a'], ['b'], []),
+    ([], [], []),
+    ([1, 2, 3], [1, 2, 4, 5], [1, 2]),
+    (['a', 'b', 'c'], ['a', 'b'], ['a', 'b']),
+    (['a', 'b', 'c'], ['a', 'b', 'd'], ['a', 'b']),
+    (['a', 'b', 'c'], ['wrong', 'b', 'c'], []),
+])
+def test_fragment_path_common(a, b, expected):
+    """Test common path."""
+    assert FragmentPath(a).common(b) == FragmentPath(expected)
+
+
+def test_fragment():
+    """Test fragment constructor and representation."""
+    fragment = Fragment(value='bla', source='test', path=['a', 'b'])
+    assert fragment
+    assert str(fragment) == \
+        "Fragment(value='bla', source='test', path=FragmentPath(['a', 'b']))", \
+        'unexpected string representation'
+
+
+def test_fragment_equality():
+    """Test fragment equality."""
+    assert (
+        Fragment(value='bla', source='test', path=['a', 'b']) ==
+        Fragment(value='bla', source='test', path=['a', 'b'])
+    )
+    assert Fragment(value='bla') == Fragment(value='bla')
+    assert Fragment(value='bla') != Fragment(value='blub')
+    assert Fragment(value='bla', source='a') != Fragment(value='bla', source='b')
+    assert Fragment(value='bla', path=['a']) != Fragment(value='bla', path=['b'])
+
+
+def test_fragment_clone():
+    """Test the fragments clone method."""
+    fragment = Fragment('bla')
+    assert fragment.clone() == fragment
+    assert fragment.clone() is not fragment
+    cloned = fragment.clone(source=['a'])
+    assert cloned != fragment
+    assert cloned == Fragment('bla', source=['a'])
+
+
+def test_fragment_iter_leaves():
+    """Test that iterating over fragment leaves gives expected result."""
+    fragment = Fragment('bla')
+    assert list(Fragment('bla').iter_leaves()) == [fragment]
+    actual = list(Fragment({
+        'a': 4,
+        'b': 'test',
+        'c': {
+            'd': [
+                2,
+                {
+                    'e': None,
+                    'f': 'test2'
+                }
+            ]
+        }
+    }).iter_leaves())
+
+    expected = [
+        Fragment(value=4, path=['a']),
+        Fragment(value='test', path=['b']),
+        Fragment(value=2, path=['c', 'd', 0]),
+        Fragment(value=None, path=['c', 'd', 1, 'e']),
+        Fragment(value='test2', path=['c', 'd', 1, 'f'])
+    ]
+
+    if sys.version_info[:2] >= (3, 6):
+        assert actual == expected
+    else:
+        def to_set(fragment_list):
+            return set((item.value, tuple(item.path)) for item in fragment_list)
+        assert to_set(actual) == to_set(expected)
+
+
+def test_expand_value_with_path():
+    """Test expanding fragment value with path."""
+    actual = Fragment('bla', path=['a', 'b']).expand_value_with_path()
+    expected = {'a': {'b': 'bla'}}
+    assert actual == expected
+
+
+@pytest.mark.parametrize('a, b, expected', [
+    pytest.param({'a': 5}, {'b': 6}, {'a': 5, 'b': 6},
+                 id='simple dict update'),
+    pytest.param({'a': 5}, {'b': 6, 'a': 4}, {'a': 4, 'b': 6},
+                 id='simple dict update with overwrite'),
+    pytest.param({'a': {'b': 1, 'c': 2}}, {'d': 3, 'a': {'b': 'new'}}, {'a': {'b': 'new', 'c': 2}, 'd': 3},
+                 id='nested dict update'),
+    pytest.param([1, 2, 3], [1, 4], [1, 4, 3],
+                 id='list update'),
+    pytest.param({'a': [1, {'b': 2}, 3]}, {'a': [EMPTY, 2]}, {'a': [1, 2, 3]},
+                 id='nested dict list update'),
+    pytest.param({'a': [1, {'b': 2}, 3]}, {'a': [EMPTY, {'c': 4}]}, {'a': [1, {'b': 2, 'c': 4}, 3]},
+                 id='nested dict list update'),
+    pytest.param(1, 2, 2,
+                 id='simple object overwrite'),
+    pytest.param([1], 2, 2,
+                 id='simple object overwrite'),
+])
+def test_merge_nested(a, b, expected):
+    """Testing nested merge."""
+    assert merge_nested(a, b) == expected
+
+
+@pytest.mark.parametrize('a_kw, b_kw, expected_kw', [
+    (
+        {'value': 4, 'path': ['a']},
+        {'value': {'a': 5}},
+        {'value': {'a': 5}}
+    ),
+    (
+        {'value': {'a': 4, 'b': ['c', 'd', 'e']}, 'path': ['root']},
+        {'value': {'bla': 2}, 'path': ['root', 'b', 1]},
+        {'value': {'a': 4, 'b': ['c', {'bla': 2}, 'e']}, 'path': ['root']}
+    ),
+    (
+        {'value': {'a': 4, 'b': {'c': 'd'}}, 'path': ['root']},
+        {'value': {'b': 2}, 'path': ['root']},
+        {'value': {'a': 4, 'b': 2}, 'path': ['root']}
+    ),
+    (
+        {'value': 3},
+        {'value': 5},
+        {'value': 5}
+    )
+])
+def test_fragment_merge(a_kw, b_kw, expected_kw):
+    """Test the apply method of the fragment class."""
+    actual = Fragment(**a_kw).merge(Fragment(**b_kw))
+    expected = Fragment(**expected_kw)
+    assert actual == expected
diff --git a/tests/test_settings.py b/tests/test_settings.py
index 39e09f8..a15bf24 100644
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -1,11 +1,10 @@
 """Test settings."""
 
 import json
-import logging
 import os
+import sys
 from collections.abc import Mapping
 from unittest.mock import MagicMock
-import sys
 
 import click
 from click.testing import CliRunner
@@ -14,6 +13,7 @@ import toml
 
 from climatecontrol import settings_parser, cli_utils  # noqa: E402
 from climatecontrol.exceptions import NoCompatibleLoaderFoundError
+from climatecontrol.fragment import Fragment
 
 
 def test_settings_empty(mock_empty_os_environ):
@@ -35,10 +35,7 @@ def test_settings(mock_os_environ, update_on_init):
         kwargs['update_on_init'] = update_on_init
     settings_map = settings_parser.Settings(**kwargs)
     assert isinstance(settings_map, Mapping)
-    if update_on_init is False:
-        expected = {}
-    else:
-        expected = {'testgroup': {'testvar': 7, 'test_var': 6}, 'testgroup_test_var': 9}
+    expected = {'testgroup': {'testvar': 7, 'test_var': 6}, 'testgroup_test_var': 9}
     assert dict(settings_map) == expected
 
 
@@ -48,9 +45,10 @@ def test_settings_parse(mock_os_environ):
     parser = MagicMock()
     parser.return_value = expected
     settings_map = settings_parser.Settings(prefix='TEST_STUFF', parser=parser)
-    assert parser.call_count == 1
+    assert parser.call_count == 0, 'Before accessing settings, the parser should not have been called'
     assert isinstance(settings_map, Mapping)
     assert dict(settings_map) == expected
+    assert parser.call_count == 1, 'After accessing settings, the parser should have been called'
 
 
 @pytest.mark.parametrize('original', [False, True])
@@ -62,7 +60,7 @@ def test_parse_from_file_vars(original, file_exists, mock_os_environ, tmpdir):
     the file and not directly.
 
     """
-    settings_map = settings_parser.Settings(update_on_init=False)
+    settings_map = settings_parser.Settings()
     filepath = tmpdir.join('testvarfile')
     filename = str(filepath)
     if file_exists:
@@ -82,34 +80,59 @@ def test_parse_from_file_vars(original, file_exists, mock_os_environ, tmpdir):
     assert actual == expected
 
 
+@pytest.mark.parametrize('settings_update, var_content, expected', [
+    (
+        {'test_var_from_env': 'MY_VAR'},
+        'apassword',
+        {'test_var': 'apassword'}
+    ),
+    (
+        {'a': {'test_var_from_env': 'MY_VAR'}},
+        '{"b": "apassword"}',
+        {'a': {'test_var': {'b': 'apassword'}}}
+    ),
+    (
+        {'a': [0, {'test_var_from_env': 'MY_VAR', 'test_var2_from_env': 'MY_VAR'}]},
+        '1',
+        {'a': [0, {'test_var': 1, 'test_var2': 1}]}
+    ),
+    (
+        {'test_var_from_env': 'MY_WRONG_VAR', 'b': 3},
+        'never seen',
+        {'b': 3}
+    ),
+])
+def test_parse_from_env_vars(mock_os_environ, settings_update, var_content, expected):
+    """Test replacing environment variables in settings."""
+    settings_map = settings_parser.Settings(update_on_init=False)
+    os.environ['MY_VAR'] = var_content
+    settings_map.update(settings_update)
+    actual = dict(settings_map)
+    assert actual == expected
+
+
 @pytest.mark.parametrize('settings_files', ['asd;kjhaflkjhasf', '.', '/home/', ['.', 'asd;kjhaflkjhasf']])
 def test_settings_files_fail(mock_empty_os_environ, settings_files):
     """Check that passing invalid settings files really results in errors."""
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF', settings_files=settings_files)
     with pytest.raises(NoCompatibleLoaderFoundError):
-        settings_parser.Settings(prefix='TEST_STUFF',
-                                 settings_files='asdlijasdlkjaa')
+        settings_map.update()
 
 
-def test_yaml_import_fail(mock_empty_os_environ, monkeypatch):
-    """Check that uninstalled yaml really results in an error."""
-    file_str = '---\na: 5'
-    # Check that without mocking everything is file:
-    settings_parser.Settings(prefix='TEST_STUFF', settings_files=file_str)
-    # Now fake not having imported yaml
-    monkeypatch.setattr('climatecontrol.file_loaders.yaml', None)
-    with pytest.raises(ImportError):
-        settings_parser.Settings(prefix='TEST_STUFF', settings_files='---\na: 5')
-
-
-def test_toml_import_fail(mock_empty_os_environ, monkeypatch):
-    """Check that uninstalled toml really results in an error."""
-    file_str = '[section]\na = 5'
+@pytest.mark.parametrize('file_str, mock_module', [
+    ('---\na: 5', 'climatecontrol.file_loaders.yaml'),
+    ('[section]\na = 5', 'climatecontrol.file_loaders.toml')
+])
+def test_file_loader_module_import_fail(mock_empty_os_environ, monkeypatch, file_str, mock_module):
+    """Check that uninstalled yaml or toml really results in an error."""
     # Check that without mocking everything is file:
-    settings_parser.Settings(prefix='TEST_STUFF', settings_files=file_str)
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF', settings_files=[file_str])
+    settings_map.update()
     # Now fake not having imported yaml
-    monkeypatch.setattr('climatecontrol.file_loaders.toml', None)
+    monkeypatch.setattr(mock_module, None)
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF', settings_files=[file_str])
     with pytest.raises(ImportError):
-        settings_parser.Settings(prefix='TEST_STUFF', settings_files=file_str)
+        settings_map.update()
 
 
 @pytest.mark.parametrize('settings_file_content', [
@@ -130,8 +153,9 @@ def test_settings_file_content(mock_empty_os_environ, settings_file_content):
 ])
 def test_settings_file_content_fail(mock_empty_os_environ, settings_file_content, error):
     """Check parsing file content from different file types raises an error on incorrect file content."""
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF', settings_files=settings_file_content)
     with pytest.raises(error):
-        settings_parser.Settings(prefix='TEST_STUFF', settings_files=settings_file_content)
+        settings_map.update()
 
 
 def test_settings_single_file(mock_empty_os_environ, mock_settings_file, tmpdir):
@@ -150,6 +174,16 @@ def test_settings_multiple_files(mock_empty_os_environ, mock_settings_files, tmp
     assert dict(settings_map) == mock_settings_files[1]
 
 
+def test_settings_multiple_files_with_glob(mock_empty_os_environ, mock_settings_files, tmpdir, file_extension):
+    """Check that setting multiple files as "settings_files" option works correctly."""
+    directory, _ = os.path.split(mock_settings_files[0][0])
+    glob_path = directory + os.path.sep + '*' + file_extension
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF',
+                                            settings_files=glob_path)
+    assert isinstance(settings_map, Mapping)
+    assert dict(settings_map) == mock_settings_files[1]
+
+
 def test_settings_env_file_and_env(mock_env_settings_file, tmpdir):
     """Check that a settings file from an env variable works together with other env variables settings.
 
@@ -168,7 +202,7 @@ def test_settings_env_file_and_env(mock_env_settings_file, tmpdir):
     }
 
 
-def test_settings_multiple_files_and_env(mock_os_environ, mock_settings_files, tmpdir, caplog):
+def test_settings_multiple_files_and_env(mock_os_environ, mock_settings_files, tmpdir):
     """Check that using multiple settings files together with settings parsed from env variables works.
 
     Each subsequent settings file should override the last and environment vars
@@ -178,7 +212,6 @@ def test_settings_multiple_files_and_env(mock_os_environ, mock_settings_files, t
     result.
 
     """
-    caplog.set_level(logging.DEBUG)
     settings_map = settings_parser.Settings(
         prefix='TEST_STUFF',
         settings_files=mock_settings_files[0])
@@ -196,38 +229,46 @@ def test_settings_multiple_files_and_env(mock_os_environ, mock_settings_files, t
         'testgroup_test_var': 9
     }
 
-    # Check that the logs are correctly printed.
-    record_tuples = caplog.record_tuples
-    secret_file_path_str = str(tmpdir / 'sub2' / 'secret.txt')
-    expected = [
-        ('climatecontrol.settings_parser', 20, 'Settings key testvar_inline_1 set to contents of file "{}"'.format(secret_file_path_str)),  # noqa: 501
-        ('climatecontrol.settings_parser', 20, 'Settings key testvar_inline_2 set to contents of file "{}"'.format(secret_file_path_str)),  # noqa: 501
-        ('climatecontrol.settings_parser', 10, 'Assigned settings from {}: ["testgroup.testvar", "testgroup.testvar_inline_1", "othergroup.blabla", "othergroup.testvar_inline_2"]'.format(mock_settings_files[0][0])),  # noqa: 501
-        ('climatecontrol.settings_parser', 10, 'Assigned settings from {}: ["othergroup.blabla", "othergroup.testvar_inline_2"]'.format(mock_settings_files[0][1])),  # noqa: 501
-        ('climatecontrol.env_parser', 20, 'Parsed setting from env var: TEST_STUFF_TESTGROUP__TEST_VAR.'),
-        ('climatecontrol.env_parser', 20, 'Parsed setting from env var: TEST_STUFF_TESTGROUP__TESTVAR.'),
-        ('climatecontrol.env_parser', 20, 'Parsed setting from env var: TEST_STUFF_TESTGROUP_TEST_VAR.'),
-        ('climatecontrol.settings_parser', 10, 'Assigned settings from environment variables: ["testgroup.test_var", "testgroup.testvar", "testgroup_test_var"]')  # noqa: 501
+    expected_fragments = [
+        Fragment(
+            value={
+                'testgroup': {'testvar': 123, 'testvar_inline_1_from_file': str(tmpdir / 'sub2' / 'secret.txt')},
+                'othergroup': {'blabla': 55, 'testvar_inline_2_from_file': str(tmpdir / 'sub2' / 'secret.txt')}
+            },
+            source=mock_settings_files[0][0], path=[]
+        ),
+        Fragment(value=settings_parser.REMOVED, source=mock_settings_files[0][0],
+                 path=['testgroup', 'testvar_inline_1_from_file']),
+        Fragment(value='foo', source=mock_settings_files[0][0],
+                 path=['testgroup', 'testvar_inline_1']),
+        Fragment(value=settings_parser.REMOVED, source=mock_settings_files[0][0],
+                 path=['othergroup', 'testvar_inline_2_from_file']),
+        Fragment(value='foo', source=mock_settings_files[0][0],
+                 path=['othergroup', 'testvar_inline_2']),
+        Fragment(value={'othergroup': {'blabla': 555, 'testvar_inline_2': 'bar'}},
+                 source=mock_settings_files[0][1]),
+        Fragment(value=6, source='ENV:TEST_STUFF_TESTGROUP__TEST_VAR',
+                 path=['testgroup', 'test_var']),
+        Fragment(value=7, source='ENV:TEST_STUFF_TESTGROUP__TESTVAR',
+                 path=['testgroup', 'testvar']),
+        Fragment(value=9, source='ENV:TEST_STUFF_TESTGROUP_TEST_VAR',
+                 path=['testgroup_test_var']),
+        Fragment(value={}, source='external')
     ]
-    assert len(record_tuples) == len(expected)
+
+    assert len(settings_map.fragments) == len(expected_fragments)
     if sys.version_info[:2] >= (3, 6):
-        assert record_tuples == expected
-    else:
-        # for python < 3.6 dictionaries are not ordered. For this reason it is
-        # very hard to figure out in what order the hierarchies actually come
-        # out.
-        unambiguous_indices = [0, 1, 4, 5, 6]
-        set(record_tuples[i] for i in unambiguous_indices) == set(expected[i] for i in unambiguous_indices)
-        for i in [2, 3, 7]:
-            assert 'Assigned settings from' in record_tuples[2][2]
-        assert '"testgroup.testvar"' in record_tuples[2][2]
-        assert str(mock_settings_files[0][0]) in record_tuples[2][2]
-
-
-@pytest.mark.parametrize('ending,content', [
+        # in python < 3.6 dicts are not ordered so we can't be sure what's up here in python 3.5
+        assert settings_map.fragments == expected_fragments
+
+
+@pytest.mark.parametrize('ending, content', [
     ('.json', '["this", "that"]\n'),
     ('.yml', '[this, that]\n'),
-    pytest.mark.xfail(('.toml', '[\'this\', \'that\']'), reason='toml literal lists are not supported')
+    pytest.param(
+        '.toml', '[\'this\', \'that\']',
+        marks=pytest.mark.xfail(reason='toml literal lists are not supported')
+    )
 ])
 def test_parse_from_file_list(ending, content, mock_os_environ, tmpdir):
     """Check that the "from_file" extension works as expected.
@@ -479,3 +520,21 @@ def test_setup_logging(monkeypatch, update, mock_empty_os_environ):
     assert 'handlers' in mock_dict_config.call_args[0][0]['root'], \
         'Expected default logging configuration to be updated, not overwritten.'
     assert mock_dict_config.call_args[0][0]['root']['level'] == 'DEBUG' if update else 'INFO'
+
+
+def test_update_log(caplog, mock_empty_os_environ, mock_settings_file):
+    """Test writing out an example configuration file."""
+    settings_file_path, expected = mock_settings_file
+    settings_map = settings_parser.Settings(prefix='TEST_STUFF',
+                                            settings_files=settings_file_path)
+    assert settings_map.update_log == '', 'before updating, the update log should be empty'
+    settings_map.update({'a': settings_parser.REMOVED, 'b': 2})
+    lines = settings_map.update_log.split('\n')
+    assert len(lines) == 4
+    expected_lines = [
+        'loaded testgroup.testvar from {}'.format(settings_file_path),
+        'loaded othergroup.blabla from {}'.format(settings_file_path),
+        'removed a from external',
+        'loaded b from external'
+    ]
+    assert set(lines) == set(expected_lines), 'Unexpected lines in update_log'