Repository: skypyproject/skypy
Pull request #255 (2020-08-27 12:14:51)

simplify config file syntax
## Description

This PR makes use of YAML tags (`!tag`) to define functions and references in config files in a non-hackish way.

```yaml
cosmology: !astropy.cosmology.default_cosmology.get
tables:
  merger_rates:
    redshift: !skypy.galaxy.redshift.herbel_redshift
      alpha: -0.5
      a_phi: -2.232
      b_phi: 0.0141
      a_m: -0.610
      b_m: -20.416
      size: 1000
      cosmology: !cosmology
    luminosity: !skypy.galaxy.luminosity.herbel_luminosities
      alpha: -0.5
      a_m: -0.610
      b_m: -20.416
      redshift: !merger_rates.redshift
    ns_ns_rate: !skypy.gravitational_wave.b_band_merger_rate
      population: NS-NS
      optimism: low
      luminosity: !merger_rates.luminosity
```

## Checklist
- [x] Follow the [Contributor Guidelines](https://github.com/skypyproject/skypy/blob/master/CONTRIBUTING.md)
- [x] Write unit tests
- [x] Write documentation strings
- [x] Assign someone from your working team to review this pull request
- [x] Assign someone from the infrastructure team to review this pull request

skypybot: Hello @ntessore! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`skypy/pipeline/tests/test_driver.py`](https://github.com/skypyproject/skypy/blob/432a53a41c4822a8bcb45dcad522a535258e8d5b/skypy/pipeline/tests/test_driver.py):

> [Line 51:80](https://github.com/skypyproject/skypy/blob/432a53a41c4822a8bcb45dcad522a535258e8d5b/skypy/pipeline/tests/test_driver.py#L51): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (88 > 79 characters)
> [Line 70:80](https://github.com/skypyproject/skypy/blob/432a53a41c4822a8bcb45dcad522a535258e8d5b/skypy/pipeline/tests/test_driver.py#L70): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 76:80](https://github.com/skypyproject/skypy/blob/432a53a41c4822a8bcb45dcad522a535258e8d5b/skypy/pipeline/tests/test_driver.py#L76): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)


rrjbca: Could you also update the docstrings for SkyPyDriver.execute and skypy/pipeline/scripts/skypy.py to reflect the new sytax

Patch:
diff --git a/examples/abundance_matching.yml b/examples/abundance_matching.yml
index ad6ffc8..9ed0355 100644
--- a/examples/abundance_matching.yml
+++ b/examples/abundance_matching.yml
@@ -1,49 +1,36 @@
-cosmology:
-  function: astropy.cosmology.default_cosmology.get
+cosmology: !astropy.cosmology.default_cosmology.get
 A_s: 2.1982e-09
 n_s: 0.969453
 redshift: 1
-growth_function:
-  function: skypy.power_spectrum.growth_function
-  args:
-    redshift: redshift
-    cosmology: cosmology
-wavenumber:
-  function: numpy.logspace
-  args: [-4, 2, 100]
-power_spectrum:
-  function: skypy.power_spectrum.eisenstein_hu
-  args:
-    wavenumber: wavenumber
-    A_s: A_s
-    n_s: n_s
-    cosmology: cosmology
+growth_function: !skypy.power_spectrum.growth_function
+  redshift: !redshift
+  cosmology: !cosmology
+wavenumber: !numpy.logspace [-4, 2, 100]
+power_spectrum: !skypy.power_spectrum.eisenstein_hu
+  wavenumber: !wavenumber
+  A_s: !A_s
+  n_s: !n_s
+  cosmology: !cosmology
 tables:
   halos:
-    mass:
-      function: skypy.halo.mass.press_schechter
-      args:
-        m_min: 1.0E+9
-        m_max: 1.0E+12
-        resolution: 1000
-        size: 1000
-        wavenumber: wavenumber
-        power_spectrum: power_spectrum
-        growth_function: growth_function
-        cosmology: cosmology
+    mass: !skypy.halo.mass.press_schechter
+      m_min: 1.0E+9
+      m_max: 1.0E+12
+      resolution: 1000
+      size: 1000
+      wavenumber: !wavenumber
+      power_spectrum: !power_spectrum
+      growth_function: !growth_function
+      cosmology: !cosmology
   galaxies:
-    luminosity:
-      function: skypy.galaxy.luminosity.herbel_luminosities
-      args:
-        alpha: -0.5
-        a_m: -0.610
-        b_m: -20.416
-        size: 100
-        redshift: redshift
+    luminosity: !skypy.galaxy.luminosity.herbel_luminosities
+      alpha: -0.5
+      a_m: -0.610
+      b_m: -20.416
+      size: 100
+      redshift: !redshift
   abundance_matching:
-    init:
-      function: skypy.halo.abundance_matching.vale_ostriker
-      args:
-        halos: halos
-        galaxies: galaxies
-      depends: [halos.complete, galaxies.complete]
+    .init: !skypy.halo.abundance_matching.vale_ostriker
+      .depends: [halos.complete, galaxies.complete]
+      halos: !halos
+      galaxies: !galaxies
diff --git a/examples/gravitational_wave_rates.yml b/examples/gravitational_wave_rates.yml
index eced095..49c92ae 100644
--- a/examples/gravitational_wave_rates.yml
+++ b/examples/gravitational_wave_rates.yml
@@ -1,39 +1,28 @@
-cosmology:
-  function: astropy.cosmology.default_cosmology.get
+cosmology: !astropy.cosmology.default_cosmology.get
 tables:
   merger_rates:
-    redshift:
-      function: skypy.galaxy.redshift.herbel_redshift
-      args:
-        alpha: -0.5
-        a_phi: -2.232
-        b_phi: 0.0141
-        a_m: -0.610
-        b_m: -20.416
-        size: 1000
-        cosmology: cosmology
-    luminosity:
-      function: skypy.galaxy.luminosity.herbel_luminosities
-      args:
-        alpha: -0.5
-        a_m: -0.610
-        b_m: -20.416
-        redshift: merger_rates.redshift
-    ns_ns_rate:
-      function: skypy.gravitational_wave.b_band_merger_rate
-      args:
-        population: '''NS-NS'''
-        optimism: '''low'''
-        luminosity: merger_rates.luminosity
-    ns_bh_rate:
-      function: skypy.gravitational_wave.b_band_merger_rate
-      args:
-        population: '''NS-BH'''
-        optimism: '''low'''
-        luminosity: merger_rates.luminosity
-    bh_bh_rate:
-      function: skypy.gravitational_wave.b_band_merger_rate
-      args:
-        population: '''BH-BH'''
-        optimism: '''low'''
-        luminosity: merger_rates.luminosity
+    redshift: !skypy.galaxy.redshift.herbel_redshift
+      alpha: -0.5
+      a_phi: -2.232
+      b_phi: 0.0141
+      a_m: -0.610
+      b_m: -20.416
+      size: 1000
+      cosmology: !cosmology
+    luminosity: !skypy.galaxy.luminosity.herbel_luminosities
+      alpha: -0.5
+      a_m: -0.610
+      b_m: -20.416
+      redshift: !merger_rates.redshift
+    ns_ns_rate: !skypy.gravitational_wave.b_band_merger_rate
+      population: NS-NS
+      optimism: low
+      luminosity: !merger_rates.luminosity
+    ns_bh_rate: !skypy.gravitational_wave.b_band_merger_rate
+      population: NS-BH
+      optimism: low
+      luminosity: !merger_rates.luminosity
+    bh_bh_rate: !skypy.gravitational_wave.b_band_merger_rate
+      population: BH-BH
+      optimism: low
+      luminosity: !merger_rates.luminosity
diff --git a/examples/herbel_galaxies.yaml b/examples/herbel_galaxies.yaml
index 73a4ee2..8689998 100644
--- a/examples/herbel_galaxies.yaml
+++ b/examples/herbel_galaxies.yaml
@@ -1,102 +1,72 @@
-cosmology:
-  function: astropy.cosmology.FlatLambdaCDM
-  args:
-    H0: 67.7
-    Om0: 0.307
+cosmology: !astropy.cosmology.FlatLambdaCDM
+  H0: 67.7
+  Om0: 0.307
 red_galaxy_sersic_index: 4
 blue_galaxy_sersic_index: 1
 tables:
   red_galaxies:
-    redshift:
-      function: skypy.galaxy.redshift.herbel_redshift
-      args:
-        alpha: -0.5
-        a_phi: -2.232
-        b_phi: 0.0141
-        a_m: -0.610
-        b_m: -20.416
-        size: 141
-        cosmology: cosmology
-    luminosity:
-      function: skypy.galaxy.luminosity.herbel_luminosities
-      args:
-        alpha: -0.5
-        a_m: -0.610
-        b_m: -20.416
-        redshift: red_galaxies.redshift
-    magnitude:
-      function: skypy.utils.astronomy.absolute_magnitude_from_luminosity
-      args:
-        luminosity: red_galaxies.luminosity
-    spectral_coefficients:
-      function: skypy.galaxy.spectrum.dirichlet_coefficients
-      args:
-        alpha0: [2.461, 2.358, 2.568, 2.268, 2.402]
-        alpha1: [2.410, 2.340, 2.200, 2.540, 2.464]
-        redshift: red_galaxies.redshift
-    physical_size:
-      function: skypy.galaxy.size.linear_lognormal
-      args:
-        a_mu: -0.243
-        b_mu: 0.954
-        sigma: 0.568
-        magnitude: red_galaxies.magnitude
-    angular_size:
-      function: skypy.galaxy.size.angular_size
-      args:
-        physical_size: red_galaxies.physical_size
-        redshift: red_galaxies.redshift
-        cosmology: cosmology
-    ellipticity:
-      function: skypy.galaxy.ellipticity.beta_ellipticity
-      args:
-        e_ratio: 0.45
-        e_sum: 3.5
-        size: 141
+    redshift: !skypy.galaxy.redshift.herbel_redshift
+      alpha: -0.5
+      a_phi: -2.232
+      b_phi: 0.0141
+      a_m: -0.610
+      b_m: -20.416
+      size: 141
+      cosmology: !cosmology
+    luminosity: !skypy.galaxy.luminosity.herbel_luminosities
+      alpha: -0.5
+      a_m: -0.610
+      b_m: -20.416
+      redshift: !red_galaxies.redshift
+    magnitude: !skypy.utils.astronomy.absolute_magnitude_from_luminosity
+      luminosity: !red_galaxies.luminosity
+    spectral_coefficients: !skypy.galaxy.spectrum.dirichlet_coefficients
+      alpha0: [2.461, 2.358, 2.568, 2.268, 2.402]
+      alpha1: [2.410, 2.340, 2.200, 2.540, 2.464]
+      redshift: !red_galaxies.redshift
+    physical_size: !skypy.galaxy.size.linear_lognormal
+      a_mu: -0.243
+      b_mu: 0.954
+      sigma: 0.568
+      magnitude: !red_galaxies.magnitude
+    angular_size: !skypy.galaxy.size.angular_size
+      physical_size: !red_galaxies.physical_size
+      redshift: !red_galaxies.redshift
+      cosmology: !cosmology
+    ellipticity: !skypy.galaxy.ellipticity.beta_ellipticity
+      e_ratio: 0.45
+      e_sum: 3.5
+      size: 141
   blue_galaxies:
-    redshift:
-      function: skypy.galaxy.redshift.herbel_redshift
-      args:
-        alpha: -1.3
-        a_phi: -0.264
-        b_phi: 0.0063
-        a_m: -0.417
-        b_m: -20.591
-        size: 63
-        cosmology: cosmology
-    luminosity:
-      function: skypy.galaxy.luminosity.herbel_luminosities
-      args:
-        alpha: -1.3
-        a_m: -0.417
-        b_m: -20.591
-        redshift: blue_galaxies.redshift
-    magnitude:
-      function: skypy.utils.astronomy.absolute_magnitude_from_luminosity
-      args:
-        luminosity: blue_galaxies.luminosity
-    spectral_coefficients:
-      function: skypy.galaxy.spectrum.dirichlet_coefficients
-      args:
-        alpha0: [2.079, 3.524, 1.917, 1.992, 2.536]
-        alpha1: [2.265, 3.862, 1.921, 1.685, 2.480]
-        redshift: blue_galaxies.redshift
-    physical_size:
-      function: skypy.galaxy.size.linear_lognormal
-      args:
-        a_mu: -0.243
-        b_mu: 0.954
-        sigma: 0.568
-        magnitude: blue_galaxies.magnitude
-    angular_size:
-      function: skypy.galaxy.size.angular_size
-      args:
-        physical_size: blue_galaxies.physical_size
-        redshift: blue_galaxies.redshift
-        cosmology: cosmology
-    ellipticity:
-      function: skypy.galaxy.ellipticity.beta_ellipticity
-      args:
-        e_ratio: 0.45
-        e_sum: 3.5
-        size: 63
+    redshift: !skypy.galaxy.redshift.herbel_redshift
+      alpha: -1.3
+      a_phi: -0.264
+      b_phi: 0.0063
+      a_m: -0.417
+      b_m: -20.591
+      size: 63
+      cosmology: !cosmology
+    luminosity: !skypy.galaxy.luminosity.herbel_luminosities
+      alpha: -1.3
+      a_m: -0.417
+      b_m: -20.591
+      redshift: !blue_galaxies.redshift
+    magnitude: !skypy.utils.astronomy.absolute_magnitude_from_luminosity
+      luminosity: !blue_galaxies.luminosity
+    spectral_coefficients: !skypy.galaxy.spectrum.dirichlet_coefficients
+      alpha0: [2.079, 3.524, 1.917, 1.992, 2.536]
+      alpha1: [2.265, 3.862, 1.921, 1.685, 2.480]
+      redshift: !blue_galaxies.redshift
+    physical_size: !skypy.galaxy.size.linear_lognormal
+      a_mu: -0.243
+      b_mu: 0.954
+      sigma: 0.568
+      magnitude: !blue_galaxies.magnitude
+    angular_size: !skypy.galaxy.size.angular_size
+      physical_size: !blue_galaxies.physical_size
+      redshift: !blue_galaxies.redshift
+      cosmology: !cosmology
+    ellipticity: !skypy.galaxy.ellipticity.beta_ellipticity
+      e_ratio: 0.45
+      e_sum: 3.5
+      size: 63
diff --git a/skypy/pipeline/driver.py b/skypy/pipeline/driver.py
index 2298564..c408202 100644
--- a/skypy/pipeline/driver.py
+++ b/skypy/pipeline/driver.py
@@ -4,12 +4,10 @@ This module provides methods to run pipelines of functions with dependencies
 and handle their results.
 """
 
-from collections.abc import Mapping
 from copy import deepcopy
 from importlib import import_module
 import builtins
 import networkx
-import re
 
 
 __all__ = [
@@ -17,23 +15,6 @@ __all__ = [
 ]
 
 
-def _items(a):
-    '''return keys and values for dict or list'''
-    if hasattr(a, 'items'):
-        return a.items()
-    return enumerate(a)
-
-
-def _is_string_argument(s):
-    # string arguments are enclosed in either single or double quotation marks
-    return s[0] in '"\'' and s[0] == s[-1]
-
-
-def _parse_string_argument(s):
-    # strip nested quotes from literal string
-    return s[1:-1]
-
-
 class SkyPyDriver:
     r'''Class for running pipelines.
 
@@ -62,16 +43,17 @@ class SkyPyDriver:
 
         Notes
         -----
-        Each step in the pipeline is configured by a dictionary specifying:
+        Each step in the pipeline is configured by a dictionary specifying
+        a variable name and the associated value.
 
-        - 'function' : the fully qualified name of the function
-        - 'args' : a list of positional arguments (by value), or a dictionary
-                   of keyword arguments
+        A value that is a tuple `(function_name, function_args)` specifies that
+        the value will be the result of a function call. The first item is the
+        fully qualified function name, and the second value specifies the
+        function arguments.
 
-        Note that 'args' either specifices keyword arguments by value, or the
-        names of previous steps in the pipeline and uses their return values as
-        keyword arguments. Literal strings (i.e. not field names) are escaped
-        by nested quotes: '"this is a literal string"' and "'another one'".
+        If a function argument is a tuple `(variable_name,)`, it refers to the
+        values of previous step in the pipeline. The tuple item must be the
+        name of the reference variable.
 
         'configuration' should contain the name and configuration of each
         variable and/or an entry named 'tables'. 'tables' should contain a set
@@ -94,61 +76,58 @@ class SkyPyDriver:
         # table_config contains settings for all table columns
         config = deepcopy(configuration)
         table_config = config.pop('tables', {})
-        default_table = {'function': 'astropy.table.Table'}
-        config.update({k: v.pop('init', default_table)
+        default_table = ('astropy.table.Table',)
+        config.update({k: v.pop('.init', default_table)
                       for k, v in table_config.items()})
 
         # Create a Directed Acyclic Graph of all jobs and dependencies
         dag = networkx.DiGraph()
 
-        # Variables initialised by value don't require function evaluations
-        def isfunction(f):
-            return isinstance(f, Mapping) and 'function' in f
-        variables = {k: v for k, v in config.items() if not isfunction(v)}
-        for v in variables:
-            dag.add_node(v)
-            setattr(self, v, config.pop(v))
-
-        # Add nodes for each variable, table and column
-        for job in config:
+        # - add nodes for each variable, table and column
+        # - add edges for the table dependencies
+        # - keep track where functions need to be called
+        #   functions are tuples (function name, [function args])
+        functions = {}
+        for job, settings in config.items():
             dag.add_node(job)
+            if isinstance(settings, tuple):
+                functions[job] = settings
         for table, columns in table_config.items():
             table_complete = '.'.join((table, 'complete'))
             dag.add_node(table_complete)
-            for column in columns.keys():
-                job = '.'.join((table, column))
-                dag.add_node(job)
-
-        # Add edges for all requirements and dependencies
-        def deps(settings):
-            deps = settings.get('depends', [])
-            args = settings.get('args', [])
-            for k, v in _items(args):
-                if isinstance(v, str) and not _is_string_argument(v):
-                    deps.append(v)
-            return deps
-        for job, settings in config.items():
-            dag.add_edges_from((d, job) for d in deps(settings))
-        for table, columns in table_config.items():
-            table_complete = '.'.join((table, 'complete'))
             dag.add_edge(table, table_complete)
             for column, settings in columns.items():
                 job = '.'.join((table, column))
+                dag.add_node(job)
                 dag.add_edge(table, job)
                 dag.add_edge(job, table_complete)
-                dag.add_edges_from((d, job) for d in deps(settings))
+                if isinstance(settings, tuple):
+                    functions[job] = settings
+
+        # go through functions and add edges for all references
+        for job, settings in functions.items():
+            # settings are tuple (function, [args])
+            args = settings[1] if len(settings) > 1 else None
+            # get dependencies from arguments
+            deps = self.get_deps(args)
+            # add edges for dependencies
+            for d in deps:
+                if dag.has_node(d):
+                    dag.add_edge(d, job)
+                else:
+                    raise KeyError(d)
 
         # Execute jobs in order that resolves dependencies
         for job in networkx.topological_sort(dag):
-            if job in variables or job.endswith('.complete'):
+            if job.endswith('.complete'):
                 continue
             elif job in config:
                 settings = config.get(job)
-                setattr(self, job, self._call_from_config(settings))
+                setattr(self, job, self.get_value(settings))
             else:
                 table, column = job.split('.')
                 settings = table_config[table][column]
-                getattr(self, table)[column] = self._call_from_config(settings)
+                getattr(self, table)[column] = self.get_value(settings)
 
         # Write tables to file
         if file_format:
@@ -156,10 +135,22 @@ class SkyPyDriver:
                 filename = '.'.join((table, file_format))
                 getattr(self, table).write(filename, overwrite=overwrite)
 
-    def _call_from_config(self, config):
+    def get_value(self, value):
+        '''return the value of a field
+
+        tuples specify function calls `(function name, function args)`
+        '''
+
+        # check for plain value
+        if not isinstance(value, tuple):
+            return value
+
+        # value is tuple (function name, [function args])
+        name = value[0]
+        args = value[1] if len(value) > 1 else []
 
         # Import function
-        function_path = config.get('function').split('.')
+        function_path = name.split('.')
         module = builtins
         for i, key in enumerate(function_path[:-1]):
             if not hasattr(module, key):
@@ -173,24 +164,62 @@ class SkyPyDriver:
         function = getattr(module, function_path[-1])
 
         # Parse arguments
-        args = config.get('args', [])
-        for k, v in _items(args):
-            if isinstance(v, str):
-                if _is_string_argument(v):
-                    args[k] = _parse_string_argument(v)
-                else:
-                    # get variable or table column
-                    args[k] = self[v]
+        parsed_args = self.get_args(args)
 
         # Call function
-        if isinstance(args, Mapping):
-            result = function(**args)
+        if isinstance(args, dict):
+            result = function(**parsed_args)
+        elif isinstance(args, list):
+            result = function(*parsed_args)
         else:
-            result = function(*args)
+            result = function(parsed_args)
 
         return result
 
+    def get_args(self, args):
+        '''parse function arguments
+
+        tuples specify references to container: `(field name,)`
+        '''
+
+        if isinstance(args, tuple):
+            # get reference
+            return self[args[0]]
+        elif isinstance(args, dict):
+            # recurse kwargs
+            return {k: self.get_args(v) for k, v in args.items()}
+        elif isinstance(args, list):
+            # recurse args
+            return [self.get_args(a) for a in args]
+        else:
+            # return value
+            return args
+
+    def get_deps(self, args):
+        '''get dependencies from function args
+
+        returns a list of all dependencies found
+        '''
+
+        if isinstance(args, tuple):
+            # reference
+            return [args[0]]
+        elif isinstance(args, dict):
+            # get explicit dependencies
+            deps = args.pop('.depends', [])
+            # turn a single value into a list
+            if isinstance(deps, str) or not isinstance(deps, list):
+                deps = [deps]
+            # recurse remaining kwargs
+            return deps + sum([self.get_deps(a) for a in args.values()], [])
+        elif isinstance(args, list):
+            # recurse args
+            return sum([self.get_deps(a) for a in args], [])
+        else:
+            # no reference
+            return []
+
     def __getitem__(self, label):
-        name, key = re.search(r'^(\w*)\.?(\w*)$', label).groups()
+        name, _, key = label.partition('.')
         item = getattr(self, name)
         return item[key] if key else item
diff --git a/skypy/pipeline/scripts/skypy.py b/skypy/pipeline/scripts/skypy.py
index 763a5eb..a32a256 100644
--- a/skypy/pipeline/scripts/skypy.py
+++ b/skypy/pipeline/scripts/skypy.py
@@ -39,6 +39,28 @@ from skypy.pipeline.driver import SkyPyDriver
 import sys
 
 
+def yaml_tag(loader, tag, node):
+    '''handler for generic YAML tags
+
+    tags are stored as a tuple `(tag, value)`
+    '''
+
+    import yaml
+
+    if isinstance(node, yaml.ScalarNode):
+        value = loader.construct_scalar(node)
+    elif isinstance(node, yaml.SequenceNode):
+        value = loader.construct_sequence(node)
+    elif isinstance(node, yaml.MappingNode):
+        value = loader.construct_mapping(node)
+
+    # tags without arguments have empty string value
+    if value == '':
+        return tag,
+
+    return tag, value
+
+
 def main(args=None):
 
     import yaml
@@ -55,6 +77,9 @@ def main(args=None):
     if args is None:
         args = sys.argv[1:]
 
+    # register custom tags
+    yaml.SafeLoader.add_multi_constructor('!', yaml_tag)
+
     args = parser.parse_args(args or ['--help'])
     config = yaml.safe_load(args.config)
     config = {} if config is None else config


Test patch:
diff --git a/skypy/pipeline/tests/data/test_config.yml b/skypy/pipeline/tests/data/test_config.yml
index 0513bc1..115802c 100644
--- a/skypy/pipeline/tests/data/test_config.yml
+++ b/skypy/pipeline/tests/data/test_config.yml
@@ -1,29 +1,21 @@
 test_int: 1
 test_float: 1.0
 test_str: hello world
-test_cosmology:
-  function: astropy.cosmology.FlatLambdaCDM
-  args:
-    H0: 67.74
-    Om0: 0.3075
+test_func: !numpy.random.uniform
+test_func_with_arg: !len 'hello world'
+test_cosmology: !astropy.cosmology.FlatLambdaCDM
+  H0: 67.74
+  Om0: 0.3075
 tables:
   test_table_1:
-    test_column_1:
-      function: numpy.random.uniform
-      args:
-        size: 10
-    test_column_2:
-      function: numpy.random.uniform
-      args:
-        low: test_table_1.test_column_1
-    test_column_3:
-      function: numpy.random.uniform
-      args:
-        low: test_table_1.test_column_1
-        high: test_table_1.test_column_2
+    test_column_1: !numpy.random.uniform
+      size: 10
+    test_column_2: !numpy.random.uniform
+      low: !test_table_1.test_column_1
+    test_column_3: !numpy.random.uniform
+      - !test_table_1.test_column_1  # <- low
+      - !test_table_1.test_column_2  # <- high
   test_table_2:
-    init:
-      function: astropy.table.QTable
-      args:
-        names: ['column_1', 'column_2']
-      depends: [test_table_1.complete]
+    .init: !astropy.table.QTable
+      .depends: test_table_1.complete
+      names: ['column_1', 'column_2']
diff --git a/skypy/pipeline/tests/test_driver.py b/skypy/pipeline/tests/test_driver.py
index d130857..395adbc 100644
--- a/skypy/pipeline/tests/test_driver.py
+++ b/skypy/pipeline/tests/test_driver.py
@@ -11,8 +11,7 @@ from skypy.pipeline.driver import SkyPyDriver
 def test_driver():
 
     # Evaluate and store the default astropy cosmology.
-    config = {'test_cosmology': {
-                'function': 'astropy.cosmology.default_cosmology.get'}}
+    config = {'test_cosmology': ('astropy.cosmology.default_cosmology.get',)}
 
     driver = SkyPyDriver()
     driver.execute(config)
@@ -21,21 +20,14 @@ def test_driver():
     # Generate a simple two column table with a dependency. Also write the
     # table to a fits file and check it's contents.
     size = 100
-    string = '"' + size*'a' + '"'
+    string = size*'a'
     config = {'tables': {
                 'test_table': {
-                  'column1': {
-                    'function': 'numpy.random.uniform',
-                    'args': {
-                      'size': size}},
-                  'column2': {
-                    'function': 'numpy.random.uniform',
-                    'args': {
-                      'low': 'test_table.column1'}},
-                  'column3': {
-                      'function': 'list',
-                      'args': [
-                        string]}}}}
+                  'column1': ('numpy.random.uniform', {
+                      'size': size}),
+                  'column2': ('numpy.random.uniform', {
+                      'low': ('test_table.column1',)}),
+                  'column3': ('list', [string])}}}
 
     driver = SkyPyDriver()
     driver.execute(config, file_format='fits')
@@ -51,33 +43,33 @@ def test_driver():
 
     # Check that the existing output files are modified if overwrite is True
     new_size = 2 * size
-    new_string = '"' + new_size*'a' + '"'
-    config['tables']['test_table']['column1']['args']['size'] = new_size
-    config['tables']['test_table']['column3']['args'][0] = new_string
+    new_string = new_size*'a'
+    config['tables']['test_table']['column1'][1]['size'] = new_size
+    config['tables']['test_table']['column3'][1][0] = new_string
     driver = SkyPyDriver()
     driver.execute(config, file_format='fits', overwrite=True)
     with fits.open('test_table.fits') as hdu:
         assert len(hdu[1].data) == new_size
 
     # Check for failure if 'column1' calls a nonexistant module
-    config['tables']['test_table']['column1']['function'] = 'nomodule.function'
+    config['tables']['test_table']['column1'] = ('nomodule.function',)
     with pytest.raises(ModuleNotFoundError):
         SkyPyDriver().execute(config)
 
     # Check for failure if 'column1' calls a nonexistant function
-    config['tables']['test_table']['column1']['function'] = 'builtins.nofunction'
+    config['tables']['test_table']['column1'] = ('builtins.nofunction',)
     with pytest.raises(AttributeError):
         SkyPyDriver().execute(config)
 
     # Check for failure if 'column1' requires itself creating a cyclic
     # dependency graph
-    config['tables']['test_table']['column1']['args'] = {'low': 'test_table.column1'}
+    config['tables']['test_table']['column1'] = ('list', ('test_table.column1',))
     with pytest.raises(networkx.NetworkXUnfeasible):
         SkyPyDriver().execute(config)
 
     # Check for failure if 'column1' and 'column2' both require each other
     # creating a cyclic dependency graph
-    config['tables']['test_table']['column1']['args'] = {'low': 'test_table.column2'}
+    config['tables']['test_table']['column1'] = ('list', ('test_table.column2',))
     with pytest.raises(networkx.NetworkXUnfeasible):
         SkyPyDriver().execute(config)
 
@@ -90,29 +82,32 @@ def test_driver():
     # Check variables intialised by value
     config = {'test_int': 1,
               'test_float': 1.0,
-              'test_string': 'hello world'}
+              'test_string': 'hello world',
+              'test_list': [0, 'one', 2.],
+              'test_dict': {'a': 'b'}}
     driver = SkyPyDriver()
     driver.execute(config)
     assert isinstance(driver.test_int, int)
     assert isinstance(driver.test_float, float)
     assert isinstance(driver.test_string, str)
+    assert isinstance(driver.test_list, list)
+    assert isinstance(driver.test_dict, dict)
     assert driver.test_int == 1
     assert driver.test_float == 1.0
     assert driver.test_string == 'hello world'
+    assert driver.test_list == [0, 'one', 2.]
+    assert driver.test_dict == {'a': 'b'}
 
     # Check variables intialised by function
-    config = {'test_func': {
-                'function': 'list',
-                'args': [
-                    '"hello world"']},
-              'test_func2': {
-                'function': 'len',
-                'args': [
-                    'test_func']}}
+    config = {'test_func': ('list', 'hello world'),
+              'len_of_test_func': ('len', ('test_func',)),
+              'nested_references': ('sum', [
+                [('test_func',), [' '], ('test_func',)], []])}
     driver = SkyPyDriver()
     driver.execute(config)
     assert driver.test_func == list('hello world')
-    assert driver.test_func2 == 11
+    assert driver.len_of_test_func == len('hello world')
+    assert driver.nested_references == list('hello world hello world')
 
 
 def teardown_module(module):