Repository: hu-macsy/simexpal

Issue:
Variation types
AFAICS simexpal variations have a name and several values which need to be written manually as well as `extra_args`. It would be nice to have cases where a variation has a specific type (i.e., int, bool), and a list (or even better a range) of values that the variation can have.
For example, at the moment if I want to execute an experiment with 1, 2, .. x threads I need to create a single variation `thread1`, `thread2`, ... for each number of threads. Wouldn't be better to just have a variation `number of threads` that takes values in [1, 2, ... x] ?
This would simplify the writing of the `experiments.yml` file, and also the creation of an output dataframe (see #57).

Discussion:
duylethanh: PR updated and ready for review:
- fixed merge conflicts
- fixed bug in ``schemes/experiments.json`` regarding the ``num_nodes``, ``procs_per_node`` and ``num_threads`` key
- fixed bug in ``launch/slurm.py`` regarding the substitution oof the  ``num_nodes``, ``procs_per_node`` and ``num_threads`` key
duylethanh: PR ready for review.
duylethanh: > LGTM, except for the comment below.

PR ready for review.

Added a check for consistent variants.

Example error message:

```
RuntimeError: Can not start experiment 'quick-sort ~ ba-bubble, bs32, num-cores:2 @ main' 
as Slurm job array. The experiment contains inconsistent variants.
```

Patch:
diff --git a/docs/at_variables_reference.rst b/docs/at_variables_reference.rst
index 2dcdbd3..a66821a 100644
--- a/docs/at_variables_reference.rst
+++ b/docs/at_variables_reference.rst
@@ -35,7 +35,7 @@ Below, we list all @-variables and where they can be used.
 - ``@THIS_COMPILE_DIR@``: :ref:`compilation directory <BuildDirectories>` of this build
 - ``@THIS_PREFIX_DIR@``: :ref:`installation directory <BuildDirectories>` of this build
 - ``@THIS_SOURCE_DIR@``: :ref:`source directory <BuildDirectories>` of this build
-
+- ``@VARIANT_VALUE:<axis_name>@``: variant value of the :ref:`dynamic variant <DynamicVariants>` axis ``<axis_name>``
 
 
 Builds
@@ -115,6 +115,29 @@ The following @-variables can be used in the ``args`` key:
 - ``@PREFIX_DIR_FOR:<build_name>@`` (``<build_name>`` has to be in ``used_builds`` or be required by a build in it)
 - ``@REPETITION@``
 - ``@SOURCE_DIR_FOR:<build_name>@`` (``<build_name>`` has to be in ``used_builds`` or be required by a build in it)
+- ``@VARIANT_VALUE:<axis_name>@``
+
+environ
+^^^^^^^
+
+The values of the ``environ`` key will be substituted and the @-variables are the same as for
+the :ref:`AtVariablesExperimentsArgs` key.
+
+num_nodes
+^^^^^^^^^
+
+Same as for the :ref:`AtVariablesExperimentsArgs` key `without` the ``@EXTRA_ARGS@`` variable.
+
+num_threads
+^^^^^^^^^^^
+
+Same as for the :ref:`AtVariablesExperimentsArgs` key `without` the ``@EXTRA_ARGS@`` variable.
+
+procs_per_node
+^^^^^^^^^^^^^^
+
+Same as for the :ref:`AtVariablesExperimentsArgs` key `without` the ``@EXTRA_ARGS@`` variable.
+
 
 workdir
 ^^^^^^^
@@ -125,6 +148,11 @@ Same as for the :ref:`AtVariablesExperimentsArgs` key `without` the ``@EXTRA_ARG
 Instances
 ---------
 
+extra_args
+^^^^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
+
 generator
 ^^^^^^^^^
 
@@ -141,3 +169,31 @@ url
 The following @-variables can be used in the ``url`` key:
 
 - ``@INSTANCE_FILENAME@``
+
+Variants
+--------
+
+environ
+^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
+
+extra_args
+^^^^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
+
+num_nodes
+^^^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
+
+num_threads
+^^^^^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
+
+procs_per_node
+^^^^^^^^^^^^^^
+
+Same as for the :ref:`experiments args <AtVariablesExperimentsArgs>` key `without` the ``@EXTRA_ARGS@`` variable.
diff --git a/docs/experiments_reference.rst b/docs/experiments_reference.rst
index 845498a..c86b985 100644
--- a/docs/experiments_reference.rst
+++ b/docs/experiments_reference.rst
@@ -87,6 +87,7 @@ This entry is a list of variants that will be used for experiments. The followin
 used for specifying variants:
 
 - ``axis``: name of the variant axis
+- ``enum``: list of variant values
 - ``environ``: dictionary of (environment variable, value)-pairs
 - ``extra_args``: list of variant arguments
 - ``items``: list of dictionaries, which specify variants belonging to the same axis.
@@ -94,6 +95,8 @@ used for specifying variants:
 - ``num_nodes``: number of nodes on which to run
 - ``num_threads``: number of cpus required per task
 - ``procs_per_node``: number of tasks to invoke on each node
+- ``range``: list with start and stop element
+- ``steps``: step size
 
 For detailed usage examples, see the :ref:`Variants` page.
 
diff --git a/docs/variants.rst b/docs/variants.rst
index 295c251..88017e0 100644
--- a/docs/variants.rst
+++ b/docs/variants.rst
@@ -18,7 +18,22 @@ you to duplicate the ``experiments`` stanza multiple times.
 Specifying Variants
 -------------------
 
-To specify variants we will use the following keys:
+There are two types of variants:
+
+1. static variants
+2. dynamic variants
+
+Static variants have to be specified individually and thus allow for more customization e.g.
+:ref:`setting different environment variables <VariantsSettingEnvironmentVariables>` for each individual
+variant. In cases where such customization is not necessary, dynamic variations can be used. By using
+:ref:`AtVariables` and syntactic sugar, dynamic variations also simplify writing the ``experiments.yml``.
+
+.. _StaticVariants:
+
+Static Variants
+^^^^^^^^^^^^^^^
+
+To specify static variants we will use the following keys:
 
 - ``axis``: name of the variant axis
 - ``items``: list of dictionaries, which specify variants belonging to the same axis.
@@ -32,7 +47,7 @@ keys.
 
 .. code-block:: YAML
    :linenos:
-   :caption: How to specify variants in the experiments.yml file.
+   :caption: How to specify static variants in the experiments.yml file.
 
     variants:
       - axis: 'block-algo'
@@ -65,6 +80,90 @@ for the ``experiments.yml`` above.
 
 Later we will see how to choose only certain combinations of variants (:ref:`RunMatrix`).
 
+.. _DynamicVariants:
+
+Dynamic Variants
+^^^^^^^^^^^^^^^^
+
+Dynamic variants work similar to :ref:`static variants <StaticVariants>`. Some differences are the syntax, automatic
+assignment of variant names of the form ``<axis_name>:<variant_value>`` to the variants and the introduction
+of the :ref:`@-variable <AtVariables>` ``@VARIANT_VALUE:<axis_name>@``.
+
+.. _RangeVariants:
+
+``range`` Variants
+~~~~~~~~~~~~~~~~~~
+
+In cases where the values of variants are integers with regular step sizes, we can use ``range``
+variants. In order to do so, we will use the
+
+- ``axis``:  name of the variant axis
+- ``range``: list with start and stop element
+- ``steps``: step size
+- ``extra_args``: list of variant arguments
+
+keys.
+
+The ``range`` and ``steps`` keys will generate values ``i`` equivalent to a
+``for(int i = start; i <= stop; i += steps)`` loop in C.
+
+The ``experiments.yml`` file in the :ref:`static variants <StaticVariants>` example can then be written in the
+following way:
+
+.. code-block:: YAML
+   :linenos:
+   :caption: How to specify range variants in the experiments.yml file.
+
+    variants:
+      - axis: 'block-algo'
+        items:
+          - name: 'ba-insert'
+            extra_args: ['insertion_sort']
+          - name: 'ba-bubble'
+            extra_args: ['bubble_sort']
+      - axis: 'block-size'
+        range: [32,64]
+        steps: 32
+        extra_args: ['@VARIANT_VALUE:block-size@']
+
+The only differences to the :ref:`static variants <StaticVariants>` example is that we created the variant axis
+``block-size`` with variants named ``block-size:32`` and ``block-size:64``. Also, we need to specify the
+``@VARIANT_VALUE:<axis_name>@`` :ref:`@-variable <AtVariables>` in the ``extra_args`` key to access the values of
+the variants.
+
+``enum`` Variants
+~~~~~~~~~~~~~~~~~
+
+``enum`` variants can be seen as a generalization of :ref:`range variants <RangeVariants>`. With ``enum``
+variants, we can specify arbitrary values explicitly. In order to so, we will use the
+
+- ``axis``:  name of the variant axis
+- ``enum``: list of variant values
+- ``extra_args``: list of variant arguments
+
+keys.
+
+The ``experiments.yml`` file in the :ref:`range variants <RangeVariants>` example can then be written in the
+following way:
+
+.. code-block:: YAML
+   :linenos:
+   :caption: How to specify enum variants in the experiments.yml file.
+
+    variants:
+      - axis: 'block-algo'
+        enum: ['insertion_sort', 'bubble_sort']
+        extra_args: ['@VARIANT_VALUE:block-algo@']
+      - axis: 'block-size'
+        range: [32,64]
+        steps: 32
+        extra_args: ['@VARIANT_VALUE:block-size@']
+
+The only differences to the :ref:`range variants <RangeVariants>` example is that we created the variant axis
+``block-algo`` with variants named ``block-algo:insertion_sort`` and ``block-algo:bubble_sort``.
+
+.. _VariantsSettingEnvironmentVariables:
+
 Setting Environment Variables
 -----------------------------
 
@@ -74,12 +173,12 @@ Setting Environment Variables
 
 Setting environment variables for variants works similar to
 :ref:`setting environment variables for experiments <ExperimentsSettingEnvironmentVariables>`.
-The difference is that we set the
+For static variants we set the
 
 - ``environ``: dictionary of (environment variable, value)-pairs
 
-key for each item in the ``items`` key. For example you can specify the
-``OMP_NUM_THREADS`` environment variable as follows:
+key for each item in the ``items`` key. For dynamic variants we set the ``environ`` key on the same
+level as the ``axis`` key. For example you can specify environment variables as follows:
 
 .. code-block:: YAML
    :linenos:
@@ -97,6 +196,24 @@ key for each item in the ``items`` key. For example you can specify the
            ...
            environ:
              OMP_NUM_THREADS: 4
+     - axis: block-algo
+       enum: ...
+       ...
+       environ:
+         foo1: 'bar'
+     - axis: block-size
+       range: ...
+       steps: ...
+       ...
+       environ:
+         foo2: 'baz'
+
+With dynamic variants it is also possible to use the :ref:`@-variable <AtVariables>` ``@VARIANT_VALUE:<axis_name>@``
+as value for the environment variables.
+
+.. note::
+   Dynamic variants of the same ``axis`` will share the environment variables set by ``environ``.
+
 
 Slurm: ``--ntasks-per-node``, ``-c``, ``-N``
 --------------------------------------------
@@ -106,7 +223,7 @@ Slurm: ``--ntasks-per-node``, ``-c``, ``-N``
    arguments specified in the ``experiments`` stanza (if the same slurm argument occurs).
 
 The :ref:`supported Slurm arguments in experiments <ExperimentsSupportedSlurmArgs>` are also
-supported for variants. Here, we can specify the
+supported for variants. For static variants we set the
 
 - ``procs_per_node``: number of tasks to invoke on each node (slurm: ``--ntasks-per-node=n``)
 - ``num_threads``: number of cpus required per task (slurm: ``-c``, ``--cpus-per-task=ncpus``)
@@ -114,9 +231,11 @@ supported for variants. Here, we can specify the
 
 keys for each item in the ``items`` key.
 
+
+.. _StaticSlurmVariants:
 .. code-block:: YAML
    :linenos:
-   :caption: How to specify supported Slurm parameters for variants in the experiments.yml file.
+   :caption: How to specify supported Slurm parameters for static variants in the experiments.yml file.
 
    variants:
      - axis: num_cores
@@ -126,7 +245,7 @@ keys for each item in the ``items`` key.
            procs_per_node: 24
            num_threads: 2
            extra_args: []           # empty extra_args as we only want to benchmark with
-                                    # different node/cpu settings; do NOT omit this key
+                                    # different node/cpu settings
          - name: c48
            num_nodes: 2
            procs_per_node: 24
@@ -137,6 +256,42 @@ When launching your experiments with slurm, the variant ``c24`` will append
 ``-N 1 --ntasks-per-node 24 -c 2`` to the sbatch command. Analogously for the experiment with
 the variant ``c48``.
 
+For dynamic variants we set ``procs_per_node``, ``num_threads`` and ``num_nodes`` on the same
+level as the ``axis`` key:
+
+.. code-block:: YAML
+   :linenos:
+   :caption: How to specify supported Slurm parameters for dynamic variants in the experiments.yml file.
+
+   variants:
+     - axis: 'block-size'
+       range: [32,64]
+       steps: 32
+       extra_args: ['@VARIANT_VALUE:block-size@']
+       num_nodes: 1
+       procs_per_node: 24
+       num_threads: 2
+
+.. note::
+   Dynamic variants of the same ``axis`` will share the values set by ``procs_per_node``, ``num_threads``
+   and ``num_nodes``.
+
+Also, the :ref:`@-variable <AtVariables>` ``@VARIANT_VALUE:<axis_name>@`` can be specified as value of
+these keys. E.g. the following is equivalent to the :ref:`static case above <StaticSlurmVariants>`:
+
+.. code-block:: YAML
+   :linenos:
+   :caption: How to specify supported Slurm parameters as dynamic variants in the experiments.yml file.
+
+   variants:
+   - axis: 'num-cores'
+     range: [1,2]
+     steps: 1
+     num_nodes: '@VARIANT_VALUE:num-cores@'
+     procs_per_node: 24
+     num_threads: 2
+
+
 Next
 ----
 
diff --git a/simexpal/base.py b/simexpal/base.py
index fc1866f..51001b7 100644
--- a/simexpal/base.py
+++ b/simexpal/base.py
@@ -118,10 +118,19 @@ class Config:
 				for axis_yml in self.yml['variants']:
 					check_for_reserved_name(axis_yml['axis'])
 
-					for variant_yml in axis_yml['items']:
-						check_for_reserved_name(variant_yml['name'])
-
-						yield Variant(self, axis_yml['axis'], variant_yml)
+					if 'items' in axis_yml:
+						for variant_yml in axis_yml['items']:
+							check_for_reserved_name(variant_yml['name'])
+
+							yield Variant(self, axis_yml['axis'], variant_yml)
+					elif 'range' in axis_yml:
+						for val in range(axis_yml['range'][0], axis_yml['range'][1] + 1, axis_yml['steps']):
+							yield Variant(self, axis_yml['axis'], axis_yml, is_dynamic=True, value=val)
+					elif 'enum' in axis_yml:
+						for val in axis_yml['enum']:
+							yield Variant(self, axis_yml['axis'], axis_yml, is_dynamic=True, value=val)
+					else:
+						raise RuntimeError('Unknown variant type: ', axis_yml)
 
 		if 'instdir' not in self.yml:
 			self.yml['instdir'] = './instances'
@@ -997,13 +1006,17 @@ def extract_thread_settings(yml):
 	}
 
 class Variant:
-	def __init__(self, cfg, axis, variant_yml):
+	def __init__(self, cfg, axis, variant_yml, is_dynamic=False, value=None):
 		self._cfg = cfg
 		self.axis = axis
 		self.variant_yml = variant_yml
+		self.is_dynamic = is_dynamic
+		self.value = value
 
 	@property
 	def name(self):
+		if self.is_dynamic:
+			return self.axis + ':' + str(self.value)
 		return self.variant_yml['name']
 
 	@property
@@ -1022,6 +1035,14 @@ class Variant:
 	def launcher(self):
 		return self.variant_yml.get('launcher', None)
 
+	@property
+	def environ(self):
+		return self.variant_yml.get('environ', {})
+
+	@property
+	def extra_args(self):
+		return self.variant_yml.get('extra_args', [])
+
 class ExperimentInfo:
 	def __init__(self, cfg, exp_yml):
 		self._cfg = cfg
diff --git a/simexpal/launch/common.py b/simexpal/launch/common.py
index 9711540..c88262b 100644
--- a/simexpal/launch/common.py
+++ b/simexpal/launch/common.py
@@ -138,6 +138,14 @@ class RunManifest:
 	def variants(self):
 		return [v['name'] for v in self.yml['variants']]
 
+	def get_variant_value(self, axis):
+		for variant in self.yml['variants']:
+			if axis == variant['axis']:
+				if not variant['is_dynamic']:
+					raise RuntimeError("The axis '{}' does not have dynamic variants".format(axis))
+				return variant['value']
+		raise RuntimeError("The experiment '{}' does not use the variant axis '{}'".format(self.yml['experiment'], axis))
+
 	def aux_file_path(self, ext):
 		return os.path.join(self.aux_subdir,
 				base.get_aux_file_name(ext, self.instance, self.repetition))
@@ -234,13 +242,16 @@ def compile_manifest(run):
 	variants_yml = []
 	for variant in exp.variation:
 		environ = {}
-		if 'environ' in variant.variant_yml:
-			for (k, v) in variant.variant_yml['environ'].items():
-				environ[k] = str(v)
+		for k, v in variant.environ.items():
+			environ[k] = str(v)
+
 		variants_yml.append({
+			'axis': variant.axis,
 			'name': variant.name,
-			'extra_args': variant.variant_yml.get('extra_args', []),
-			'environ': environ
+			'extra_args': variant.extra_args,
+			'environ': environ,
+			'is_dynamic': variant.is_dynamic,
+			'value': variant.value
 		})
 
 	environ = {}
@@ -359,12 +370,14 @@ def invoke_run(manifest):
 			return manifest.base_dir
 		elif p == 'INSTANCE_DIR':
 			return manifest.instance_dir
+		elif p.startswith('VARIANT_VALUE:'):
+			return str(manifest.get_variant_value(p.split(':')[1]))
 		else:
 			return None
 
 	def substitute_list(p):
 		if p == 'EXTRA_ARGS':
-			return manifest.get_extra_args()
+			return util.expand_at_params(manifest.get_extra_args(), substitute)
 		else:
 			return None
 
@@ -404,7 +417,12 @@ def invoke_run(manifest):
 	environ['PATH'] = prepend_env('PATH', extra_paths)
 	environ['LD_LIBRARY_PATH'] = prepend_env('LD_LIBRARY_PATH', manifest.get_ldso_paths())
 	environ['PYTHONPATH'] = prepend_env('PYTHONPATH', manifest.get_python_paths())
-	environ.update(manifest.environ)
+
+	manifest_environ = manifest.environ
+	for env_var in manifest_environ:
+		manifest_environ[env_var] = util.expand_at_params(manifest_environ[env_var], substitute)
+
+	environ.update(manifest_environ)
 
 	# Dumps data from an FD to the FS.
 	# Creates the output file only if something is written.
diff --git a/simexpal/launch/slurm.py b/simexpal/launch/slurm.py
index c96bfaf..2a8636a 100644
--- a/simexpal/launch/slurm.py
+++ b/simexpal/launch/slurm.py
@@ -48,13 +48,16 @@ class SlurmLauncher(common.Launcher):
 		use_array = len(locked) > 1
 
 		# Build the specfile.
+		manifest = common.compile_manifest(locked[0])
 		if not use_array:
 			specs = {
-				'manifest': common.compile_manifest(locked[0]).yml
+				'manifest': manifest.yml
 			}
 		else:
+			manifests = [manifest.yml]
+			manifests.extend([common.compile_manifest(run).yml for run in locked[1:]])
 			specs = {
-				'manifests': [common.compile_manifest(run).yml for run in locked]
+				'manifests': manifests
 			}
 
 		(specfd, specfile) = tempfile.mkstemp(prefix='', suffix='-spec.yml',
@@ -75,9 +78,48 @@ class SlurmLauncher(common.Launcher):
 
 		sbatch_script = util.expand_at_params(script_template, substitute)
 
+		def substitute_slurm_settings(p):
+			if p.startswith('VARIANT_VALUE:'):
+				# Verify that all runs have the same variants.
+				if use_array:
+					for run in locked[1:]:
+						if not locked[0].experiment.variation == run.experiment.variation:
+							raise RuntimeError("Can not start experiment '{}' as Slurm job array. The experiment "
+												"contains inconsistent variants.".format(locked[0].experiment.display_name))
+
+				return str(manifest.get_variant_value(p.split(':')[1]))
+			else:
+				return None
+
 		ps = experiment.effective_process_settings
 		ts = experiment.effective_thread_settings
 
+		def _represents_int(x):
+			try:
+				int(x)
+				return True
+			except ValueError:
+				return False
+
+		if ps is not None:
+			if ps['num_nodes'] is not None:
+				if isinstance(ps['num_nodes'], str):
+					ps['num_nodes'] = util.expand_at_params(ps['num_nodes'], substitute_slurm_settings)
+				if not _represents_int(ps['num_nodes']):
+					raise RuntimeError("The value of the 'num_nodes' key has to be an integer: {}".format(ps['num_nodes']))
+
+			if ps['procs_per_node'] is not None:
+				if isinstance(ps['procs_per_node'], str):
+					ps['procs_per_node'] = util.expand_at_params(ps['procs_per_node'], substitute_slurm_settings)
+				if not _represents_int(ps['procs_per_node']):
+					raise RuntimeError("The value of the 'procs_per_node' key has to be an integer: {}".format(ps['procs_per_node']))
+
+		if ts is not None:
+			if isinstance(ts['num_threads'], str):
+				ts['num_threads'] = util.expand_at_params(ts['num_threads'], substitute_slurm_settings)
+			if not _represents_int(ts['num_threads']):
+				raise RuntimeError("The value of the 'num_threads' key has to be an integer: {}".format(ts['num_threads']))
+
 		# Build the sbatch command to run the script.
 		# TODO: Support multiple queues
 		sbatch_args = ['sbatch', '-J', experiment.display_name]
diff --git a/simexpal/schemes/experiments.json b/simexpal/schemes/experiments.json
index 7e55bec..4595a35 100644
--- a/simexpal/schemes/experiments.json
+++ b/simexpal/schemes/experiments.json
@@ -20,6 +20,12 @@
 				"type": "object"
 			}
 		},
+		"int_list": {
+			"type": "array",
+			"items": {
+				"type": "integer"
+			}
+		},
 		"build_arguments": {
 			"type": "array",
 			"items": {
@@ -31,6 +37,24 @@
 				},
 				"additionalProperties": false
 			}
+		},
+		"slurm_settings": {
+			"type": "object",
+			"properties": {
+				"num_nodes": {
+					"type": ["integer", "string"],
+					"minimum": 1
+				},
+				"procs_per_node": {
+					"type":  ["integer", "string"],
+					"minimum": 1
+				},
+				"num_threads": {
+					"type": ["integer", "string"],
+					"minimum": 1
+				},
+				"slurm_args": {"$ref": "#/definitions/str_or_str_list"}
+			}
 		}
 	},
 
@@ -94,7 +118,7 @@
 						"if": {
 							"required": ["generator"],
 							"properties": {
-								"generator": {"type": "object"}
+								"generator": {}
 							}
 						},
 						"then": {
@@ -137,7 +161,7 @@
 						"if": {
 							"required": ["extensions"],
 							"properties": {
-								"extensions": {"$ref": "#/definitions/str_list"}
+								"extensions": {}
 							}
 						},
 						"then": {
@@ -180,7 +204,7 @@
 										"type": "object",
 										"required": ["files"],
 										"properties": {
-											"files": {"$ref": "#/definitions/str_list"}
+											"files": {}
 										}
 									}
 								}
@@ -213,7 +237,7 @@
 						"if": {
 							"required": ["method"],
 							"properties": {
-								"method": {"type": "string"}
+								"method": {}
 							}
 						},
 						"then": {
@@ -306,13 +330,13 @@
 									{
 										"required": ["generator"],
 										"properties": {
-											"generator": {"type": "object"}
+											"generator": {}
 										}
 									},
 									{
 										"required": ["extensions"],
 										"properties": {
-											"extensions": {"$ref": "#/definitions/str_list"}
+											"extensions": {}
 										}
 									},
 									{
@@ -324,7 +348,7 @@
 													"type": "object",
 													"required": ["files"],
 													"properties": {
-														"files": {"$ref": "#/definitions/str_list"}
+														"files": {}
 													}
 												}
 											}
@@ -333,7 +357,7 @@
 									{
 										"required": ["method"],
 										"properties": {
-											"method": {"type": "string"}
+											"method": {}
 										}
 									}
 								]
@@ -381,50 +405,48 @@
 			"type": "array",
 			"items": {
 				"type": "object",
-				"required": ["name", "args"],
-				"properties": {
-					"name": {"type": "string"},
-					"args": {"$ref": "#/definitions/str_list"},
-					"use_builds": {"$ref": "#/definitions/str_list"},
-					"output": {
-						"oneOf": [
-							{"type": "string"},
-							{
-								"type": "object",
-								"properties": {
-									"extensions": {"$ref": "#/definitions/str_list"}
-								},
-								"additionalProperties": false
-							}
-						]
-					},
-					"num_nodes": {
-						"type": "integer",
-						"minimum": 1
-					},
-					"procs_per_node": {
-						"type": "integer",
-						"minimum": 1
-					},
-					"num_threads": {
-						"type": "integer",
-						"minimum": 1
-					},
-					"timeout": {
-						"type": "integer",
-						"minimum": 0
-					},
-					"environ": {"type": "object"},
-					"repeat": {
-						"type": "integer",
-						"minimum": 0
+				"allOf": [
+					{
+						"$ref": "#/definitions/slurm_settings"
 					},
-					"slurm_args": {"$ref": "#/definitions/str_or_str_list"},
-					"workdir": {"type": "string"},
-					"launcher": {"type": "string"},
-					"stdout": {"type": "string"}
-				},
-				"additionalProperties": false
+					{
+						"required": ["name", "args"],
+						"properties": {
+							"name": {"type": "string"},
+							"args": {"$ref": "#/definitions/str_list"},
+							"use_builds": {"$ref": "#/definitions/str_list"},
+							"output": {
+								"oneOf": [
+									{"type": "string"},
+									{
+										"type": "object",
+										"properties": {
+											"extensions": {"$ref": "#/definitions/str_list"}
+										},
+										"additionalProperties": false
+									}
+								]
+							},
+							"timeout": {
+								"type": "integer",
+								"minimum": 0
+							},
+							"environ": {"type": "object"},
+							"repeat": {
+								"type": "integer",
+								"minimum": 0
+							},
+							"workdir": {"type": "string"},
+							"launcher": {"type": "string"},
+							"stdout": {"type": "string"},
+							"num_nodes": {},
+							"procs_per_node": {},
+							"num_threads": {},
+							"slurm_args": {}
+						},
+						"additionalProperties": false
+					}
+				]
 			}
 		},
 		"variants": {
@@ -432,38 +454,132 @@
 			"type": "array",
 			"items": {
 				"type": "object",
-				"required": ["axis", "items"],
-				"properties": {
-					"axis": {"type": "string"},
-					"items": {
-						"type": "array",
-						"items": {
-							"type": "object",
-							"required": ["name"],
+				"allOf": [
+					{
+						"if": {
+							"required": ["range", "steps"],
 							"properties": {
-								"name": {"type": "string"},
-								"extra_args": {"$ref":  "#/definitions/str_list"},
-								"num_nodes": {
-									"type": "integer",
-									"minimum": 1
+								"range": {},
+								"steps": {}
+							}
+
+						},
+						"then": {
+							"allOf": [
+								{
+									"$ref": "#/definitions/slurm_settings"
 								},
-								"procs_per_node": {
-									"type":  "integer",
-									"minimum": 1
+								{
+									"required": ["axis", "range", "steps"],
+									"properties": {
+										"axis": {"type": "string"},
+										"extra_args": {"$ref":  "#/definitions/str_list"},
+										"range": {"$ref": "#/definitions/int_list"},
+										"steps": {"type": "integer"},
+										"environ": {"type": "object"},
+										"launcher": {"type": "string"},
+										"num_nodes": {},
+										"procs_per_node": {},
+										"num_threads": {},
+										"slurm_args": {}
+									},
+									"additionalProperties": false
+								}
+							]
+						}
+					},
+					{
+						"if": {
+							"required": ["enum"],
+							"properties": {
+								"enum": {}
+							}
+						},
+						"then": {
+							"allOf": [
+								{
+									"$ref": "#/definitions/slurm_settings"
 								},
-								"num_threads": {
-									"type": "integer",
-									"minimum": 1
+								{
+									"required": ["axis", "enum"],
+									"properties": {
+										"axis": {"type": "string"},
+										"enum": {"type": "array"},
+										"extra_args": {"$ref":  "#/definitions/str_list"},
+										"environ": {"type": "object"},
+										"launcher": {"type": "string"},
+										"num_nodes": {},
+										"procs_per_node": {},
+										"num_threads": {},
+										"slurm_args": {}
+									},
+									"additionalProperties": false
+								}
+							]
+						}
+					},
+					{
+						"if": {
+							"not": {
+								"anyOf": [
+									{
+										"required": ["range", "steps"],
+										"properties": {
+											"range": {},
+											"steps:": {}
+										}
+									},
+									{
+										"required": ["enum"],
+										"properties": {
+											"enum": {}
+										}
+									}
+								]
+							}
+						},
+						"then": {
+							"allOf": [
+								{
+									"properties": {
+										"items": {
+											"type": "array",
+											"items": {
+												"$ref": "#/definitions/slurm_settings"
+											}
+										}
+									}
 								},
-								"environ": {"type": "object"},
-								"slurm_args": {"$ref": "#/definitions/str_or_str_list"},
-								"launcher": {"type": "string"}
-							},
-							"additionalProperties": false
+								{
+									"required": ["axis", "items"],
+									"properties": {
+										"axis": {"type": "string"},
+										"items": {
+											"type": "array",
+											"items": {
+												"type": "object",
+												"required": ["name"],
+												"properties": {
+													"name": {"type": "string"},
+													"extra_args": {"$ref":  "#/definitions/str_list"},
+													"environ": {"type": "object"},
+													"launcher": {"type": "string"},
+													"num_nodes": {},
+													"procs_per_node": {},
+													"num_threads": {},
+													"slurm_args": {}
+												},
+												"additionalProperties": false
+											}
+										}
+									},
+									"additionalProperties": false
+								}
+							]
 						}
 					}
-				},
-				"additionalProperties": false
+
+				]
 			}
 		},
 		"matrix": {


Test patch:
diff --git a/tests/validation/experiments_ymls/invalid/repo-invalid_value.yml b/tests/validation/experiments_ymls/invalid/instances/repo-invalid_value.yml
similarity index 100%
rename from tests/validation/experiments_ymls/invalid/repo-invalid_value.yml
rename to tests/validation/experiments_ymls/invalid/instances/repo-invalid_value.yml
diff --git a/tests/validation/experiments_ymls/invalid/variants/enum_additional_property.yml b/tests/validation/experiments_ymls/invalid/variants/enum_additional_property.yml
new file mode 100644
index 0000000..6c81539
--- /dev/null
+++ b/tests/validation/experiments_ymls/invalid/variants/enum_additional_property.yml
@@ -0,0 +1,8 @@
+# Invalid .yml file for util.validate_setup_file().
+# This file contains an additional property for 'enum' variants.
+
+variants:
+  - axis: 'block-algo'
+    enum: ['insertion_sort', 'bubble_sort']
+    extra_args: ['@VARIANT_VALUE:block-algo@']
+    foo: true
diff --git a/tests/validation/experiments_ymls/invalid/variants/range_additional_property.yml b/tests/validation/experiments_ymls/invalid/variants/range_additional_property.yml
new file mode 100644
index 0000000..f517adf
--- /dev/null
+++ b/tests/validation/experiments_ymls/invalid/variants/range_additional_property.yml
@@ -0,0 +1,9 @@
+# Invalid .yml file for util.validate_setup_file().
+# This file contains an additional property for 'range' variants.
+
+variants:
+  - axis: 'block-size'
+    range: [32, 64]
+    steps: 32
+    extra_args: ['@VARIANT_VALUE:block-size@']
+    foo: true
diff --git a/tests/validation/experiments_ymls/invalid/variants/range_str_list.yml b/tests/validation/experiments_ymls/invalid/variants/range_str_list.yml
new file mode 100644
index 0000000..8fa391b
--- /dev/null
+++ b/tests/validation/experiments_ymls/invalid/variants/range_str_list.yml
@@ -0,0 +1,8 @@
+# Invalid .yml file for util.validate_setup_file().
+# This file contains a string list for the 'range' key.
+
+variants:
+  - axis: 'block-size'
+    range: ['foo', 'bar']
+    steps: 32
+    extra_args: ['@VARIANT_VALUE:block-size@']
diff --git a/tests/validation/experiments_ymls/invalid/variants/steps_str_value.yml b/tests/validation/experiments_ymls/invalid/variants/steps_str_value.yml
new file mode 100644
index 0000000..94724d5
--- /dev/null
+++ b/tests/validation/experiments_ymls/invalid/variants/steps_str_value.yml
@@ -0,0 +1,8 @@
+# Invalid .yml file for util.validate_setup_file().
+# This file contains a string for the 'steps' key.
+
+variants:
+  - axis: 'block-size'
+    range: [32, 64]
+    steps: 'foo'
+    extra_args: ['@VARIANT_VALUE:block-size@']
diff --git a/tests/validation/experiments_ymls/valid/variants/enum.yml b/tests/validation/experiments_ymls/valid/variants/enum.yml
new file mode 100644
index 0000000..56ddadf
--- /dev/null
+++ b/tests/validation/experiments_ymls/valid/variants/enum.yml
@@ -0,0 +1,6 @@
+# This file contains an 'enum' variant.
+
+variants:
+  - axis: 'block-algo'
+    enum: ['insertion_sort', 'bubble_sort']
+    extra_args: ['@VARIANT_VALUE:block-algo@']
diff --git a/tests/validation/experiments_ymls/valid/variants/range.yml b/tests/validation/experiments_ymls/valid/variants/range.yml
new file mode 100644
index 0000000..c8697a2
--- /dev/null
+++ b/tests/validation/experiments_ymls/valid/variants/range.yml
@@ -0,0 +1,7 @@
+# This file contains a 'range' variant.
+
+variants:
+  - axis: 'block-size'
+    range: [32, 64]
+    steps: 32
+    extra_args: ['@VARIANT_VALUE:block-size@']
diff --git a/tests/validation/test_experiments_yml.py b/tests/validation/test_experiments_yml.py
index 550a3dc..c6d5a95 100644
--- a/tests/validation/test_experiments_yml.py
+++ b/tests/validation/test_experiments_yml.py
@@ -9,11 +9,17 @@ file_dir = os.path.abspath(os.path.dirname(__file__))
 valid_experiments_ymls = ['../../examples/sorting/experiments.yml',
                           '../../examples/sorting_cpp/experiments.yml',
                           '../../examples/download_instances/experiments.yml',
-                          'experiments_ymls/valid/instances/extra_args.yml']
+                          'experiments_ymls/valid/instances/extra_args.yml',
+                          'experiments_ymls/valid/variants/enum.yml',
+                          'experiments_ymls/valid/variants/range.yml']
 
 invalid_experiments_ymls = ['experiments_ymls/invalid/top_level-additional_property.yml',
-                            'experiments_ymls/invalid/repo-invalid_value.yml']
-
+                            'experiments_ymls/invalid/instances/repo-invalid_value.yml',
+                            'experiments_ymls/invalid/variants/enum_additional_property.yml',
+                            'experiments_ymls/invalid/variants/range_additional_property.yml',
+                            'experiments_ymls/invalid/variants/range_str_list.yml',
+                            'experiments_ymls/invalid/variants/steps_str_value.yml'
+                            ]
 
 @pytest.mark.parametrize('rel_yml_path', valid_experiments_ymls)
 def test_valid_experiments_yml(rel_yml_path):