Repository: spacetx/starfish

Issue:
Add validate-sptx fuzz command
cF: https://github.com/spacetx/sptx-format/pull/21

Approximate output:

```
validate_sptx/test_field_of_view.py::test_fuzz_input[0] 0    corrupt     b'{\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[1] 1    invalid     b'  "version": "0.0.0",\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[2] 2    invalid     b'  "fov": 0,\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[3] 3    corrupt     b'  "dimensions": [\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[4] 4    passes      b'      "x",\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[5] 5    passes      b'      "y",\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[6] 6    passes      b'      "z",\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[7] 7    passes      b'      "r",\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[8] 8    corrupt     b'      "c"\n'
validate_sptx/test_field_of_view.py::test_fuzz_input[9] 9    corrupt     b'  ],\n'
...
```

Discussion:
joshmoore: Workings towards this type of output:

```
test_fuzz.py::test_fuzz_codebook > Fuzzing unknown...
A D I S
X X X X	 - codeword:
X   X X	   - r:
      X	        0
X   X X	     c:
  X   X	        0
X   X X	     v:
    X X	        1
X   X X	   - r:
      X	        0
X   X X	     c:
  X   X	        1
X   X X	     v:
    X X	        1
X   X X	   - r:
      X	        1
X   X X	     c:
  X   X	        1
X   X X	     v:
    X X	        1
X X X X	   target:
  X X  	      SCUBE2
```

where:

 * A = add a new value
 * D = delete this value
 * I = change value to an int
 * S = change value to a str
joshmoore: Forced pushed now that gh-514 is merged. Suggestions on others tests that would be interesting welcome. ("Replace with object" and "replace with list"?)
codecov-io: # [Codecov](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=h1) Report
> Merging [#523](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=desc) into [master](https://codecov.io/gh/spacetx/starfish/commit/9b53c8a84d586af1f906c742b52617b036f14221?src=pr&el=desc) will **increase** coverage by `0.82%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/spacetx/starfish/pull/523/graphs/tree.svg?width=650&token=kliQHejt6T&height=150&src=pr)](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #523      +/-   ##
==========================================
+ Coverage   82.67%   83.49%   +0.82%     
==========================================
  Files          87       91       +4     
  Lines        3255     3411     +156     
==========================================
+ Hits         2691     2848     +157     
+ Misses        564      563       -1
```


| [Impacted Files](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [starfish/test/full\_pipelines/cli/test\_validate.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-c3RhcmZpc2gvdGVzdC9mdWxsX3BpcGVsaW5lcy9jbGkvdGVzdF92YWxpZGF0ZS5weQ==) | `100% <100%> (ø)` | |
| [starfish/starfish.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-c3RhcmZpc2gvc3RhcmZpc2gucHk=) | `83.63% <100%> (+0.94%)` | :arrow_up: |
| [validate\_sptx/test\_fuzz.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-dmFsaWRhdGVfc3B0eC90ZXN0X2Z1enoucHk=) | `100% <100%> (ø)` | |
| [validate\_sptx/validate\_sptx.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-dmFsaWRhdGVfc3B0eC92YWxpZGF0ZV9zcHR4LnB5) | `27.77% <23.07%> (+27.77%)` | :arrow_up: |
| [starfish/util/exec.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-c3RhcmZpc2gvdXRpbC9leGVjLnB5) | `78.37% <78.37%> (ø)` | |
| [validate\_sptx/cli.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-dmFsaWRhdGVfc3B0eC9jbGkucHk=) | `92.85% <92.85%> (ø)` | |
| [validate\_sptx/util.py](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree#diff-dmFsaWRhdGVfc3B0eC91dGlsLnB5) | `95.61% <97.22%> (+2.43%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/spacetx/starfish/pull/523/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=footer). Last update [9b53c8a...16d3591](https://codecov.io/gh/spacetx/starfish/pull/523?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

joshmoore: Ooh, I'm finally doing something _positive_ for coverage numbers.
joshmoore: Pushed a sufficient state for review. Example output (from the unit tests):

```
validate_sptx/test_fuzz.py::test_fuzz_experiment...

> Fuzzing unknown...
A D I S M L	If the letter is present, mutation is valid!
-----------	--------------------------------------------
. . . . . .	version:
. . . . . .	   0.0.0
. . . . . .	primary_images:
. . . . . .	   primary_images.json
. D . . M .	auxiliary_images:
. D . . . .	   nuclei:
. D . . . .	      nuclei.json
. . . . . .	codebook:
. . . . . .	   codebook.json
. D I S M L	extras:
A D I S M L	   is_space_tx_cool:
A D I S M L	      True
PASSED
```
joshmoore: > I've added a few suggestions that are mostly stylistic.

Wow. Now I know what to be ready for with an @ambrosejcarr review!

> Do you think it's worth pointing them at this?

Potentially. While setting it up, I did wonder if it would actually be better as a standalone tool that `starfish` consumes. If you'd prefer not to have it in this codebase, I understand and/or it can be migrated later.

Changes pushed.

Patch:
diff --git a/.dockerignore b/.dockerignore
new file mode 100644
index 00000000..6b8710a7
--- /dev/null
+++ b/.dockerignore
@@ -0,0 +1,1 @@
+.git
diff --git a/.travis.yml b/.travis.yml
index f1c6753e..77f8c4e9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,5 +23,7 @@ script:
     .venv/bin/pip install starfish;
     make PYTHON=.venv/bin/python run_notebooks;
   fi
+- |-
+  docker run -ti --rm spacetx/starfish build --fov-count 1 --hybridization-dimensions '{"z": 1}' /tmp/
 after_success:
 - bash <(curl -s https://codecov.io/bash)
diff --git a/Dockerfile b/Dockerfile
index 7c6850ca..0fbc37cc 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -36,18 +36,22 @@
 ##
 ##   $ docker run -e TEST_ISS_KEEP_DATA=true --entrypoint=pytest spacetx/starfish -vsxk TestWithIssData
 ##
-FROM python:3.6
-
-COPY REQUIREMENTS.txt /src/
-COPY REQUIREMENTS-DEV.txt /src/
-COPY REQUIREMENTS-NOTEBOOK.txt /src/
-RUN pip install -r /src/REQUIREMENTS-DEV.txt -r /src/REQUIREMENTS-NOTEBOOK.txt
-
+FROM continuumio/miniconda3
 RUN useradd -m starfish
-COPY . /src
-RUN chown -R starfish:starfish /src
 USER starfish
+
+# Set up the initial conda environment
+COPY --chown=starfish:starfish environment.yml /src/environment.yml
 WORKDIR /src
-RUN pip install --user -e .
-ENV PATH=${PATH}:/home/starfish/.local/bin
+RUN conda env create -f environment.yml
+
+# Prepare for build
+COPY --chown=starfish:starfish . /src
+RUN echo "source activate starfish" >> ~/.bashrc
+ENV PATH /home/starfish/.conda/envs/starfish/bin:$PATH
+
+# Build and configure for running
+RUN pip install -e .
+
+env MPLBACKEND Agg
 ENTRYPOINT ["starfish"]
diff --git a/environment.yml b/environment.yml
new file mode 100644
index 00000000..450fc052
--- /dev/null
+++ b/environment.yml
@@ -0,0 +1,30 @@
+name: starfish
+channels:
+- defaults
+- bioconda
+- conda-forge
+dependencies:
+- python>=3.6
+- click
+- flake8
+- jsonschema
+- matplotlib
+- mypy
+- numpy!=1.13.0
+- pandas>=0.23.4
+- pytest>=3.6.3
+- regional
+- requests
+- scikit-image>=0.14.0
+- scikit-learn
+- scipy
+- seaborn
+- semantic_version
+- showit>=1.1.4
+- slicedimage==0.0.4
+- tqdm
+- trackpy
+- validators
+- xarray
+- pip:
+  - jsonpath-rw
diff --git a/sptx-format/Dockerfile b/sptx-format/Dockerfile
deleted file mode 100644
index df84967f..00000000
--- a/sptx-format/Dockerfile
+++ /dev/null
@@ -1,23 +0,0 @@
-## Dockerfile for sptx
-##
-## Default entrypoint is the validate-sptx script.
-
-FROM continuumio/miniconda3
-RUN useradd -m sptx
-USER sptx
-
-# Set up the initial conda environment
-COPY --chown=sptx:sptx environment.yml /src/environment.yml
-WORKDIR /src
-RUN conda env create -f environment.yml
-
-# Prepare for build
-COPY --chown=sptx:sptx . /src
-RUN echo "source activate sptx" >> ~/.bashrc
-ENV PATH /home/sptx/.conda/envs/sptx/bin:$PATH
-
-# Build and configure for running
-RUN pip install -e .
-
-env MPLBACKEND Agg
-ENTRYPOINT ["validate-sptx"]
diff --git a/sptx-format/environment.yml b/sptx-format/environment.yml
deleted file mode 100644
index c2b420f7..00000000
--- a/sptx-format/environment.yml
+++ /dev/null
@@ -1,14 +0,0 @@
-# Also update REQUIREMENTS.txt
-name: sptx
-channels:
-- defaults
-- bioconda
-- conda-forge
-dependencies:
-- python>=3.6
-- click
-- flake8
-- jsonschema
-- mypy
-- pytest
-- slicedimage==0.0.1
diff --git a/starfish/starfish.py b/starfish/starfish.py
index 0b764490..02127a7d 100755
--- a/starfish/starfish.py
+++ b/starfish/starfish.py
@@ -4,6 +4,7 @@ import argparse
 import cProfile
 from pstats import Stats
 
+from validate_sptx.cli import Cli as ValidateCli
 from starfish.experiment.builder.cli import Cli as BuilderCli
 from starfish.image import (
     Filter,
@@ -47,6 +48,9 @@ def build_parser():
     build_group = subparsers.add_parser("build")
     BuilderCli.add_to_parser(build_group)
 
+    validate_group = subparsers.add_parser("validate")
+    ValidateCli.add_to_parser(validate_group)
+
     return parser
 
 
diff --git a/starfish/util/exec.py b/starfish/util/exec.py
new file mode 100644
index 00000000..aea84ddd
--- /dev/null
+++ b/starfish/util/exec.py
@@ -0,0 +1,118 @@
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+from typing import Callable, Sequence, Union
+
+import jsonpath_rw
+import numpy as np
+import pandas as pd
+
+from starfish.intensity_table import IntensityTable
+from starfish.types import Features
+from starfish.util import clock
+
+
+def stages(commands: Sequence[Sequence[Union[str, Callable]]],
+           subdirs: Sequence[str]=None,
+           keep_data:bool=False) -> None:
+    """
+    Execute a list of commands in a temporary directory
+    cleaning them up unless otherwise requested.
+
+    Parameters
+    ----------
+    commands : Sequence[Sequence[Union[str, Callable]]]
+        A collection of tuples of commands composed either of
+        str elements or callable objects which will be invoked
+        with the keyword argument "tempdir". The resulting list
+        of strings will be passed to subprocess.check_call.
+    subdirs : Sequence[str]
+        A collection of paths which should be created as subdirectories
+        within the temporary directory used by this invocation.
+    keep_data : bool
+        If not true, shutil.rmtree will be called on the temporary
+        directory used by this invocation.
+
+    Environment variables
+    ---------------------
+    STARFISH_COVERAGE:
+         If set, then command lists will have `coverage run ...` options
+         prepended before execution.
+    """
+    tempdir = tempfile.mkdtemp()
+
+    def callback(interval):
+        print(" ".join(stage[:2]), " ==> {} seconds".format(interval))
+
+    try:
+
+        if subdirs:
+            for subdir in subdirs:
+                os.makedirs("{tempdir}".format(
+                    tempdir=os.path.join(tempdir, subdir)))
+
+        for stage in commands:
+            cmdline = prepare_stage(stage, tempdir)
+            with clock.timeit(callback):
+                subprocess.check_call(cmdline)
+
+    finally:
+        if not keep_data:
+            shutil.rmtree(tempdir)
+
+
+def prepare_stage(stage: Sequence[Union[str, Callable]],
+                  tempdir: str) -> Sequence[str]:
+    """
+    Loop through elements of stage, building them into a commandline.
+    If an element is a callable, it will be invoked with the "tempdir"
+    keyword.
+
+    Parameters
+    ----------
+    stage: Sequence[Union[str, Callable]]
+        A collection of commands composed either of
+        str elements or callable objects which will be invoked
+        with the keyword argument "tempdir". The resulting list
+        of strings will be passed to subprocess.check_call.
+    tempdir: str
+        Temporary directory that will be used by the invoking method.
+
+    Return
+    ------
+    Sequence of strings for passing to subprocess.check_call
+
+    Environment variables
+    ---------------------
+    STARFISH_COVERAGE:
+         If set, then command lists will have `coverage run ...` options
+         prepended before execution.
+    """
+    coverage_enabled = "STARFISH_COVERAGE" in os.environ
+    cmdline = [
+        element(tempdir=tempdir) if callable(element) else element
+        for element in stage
+    ]
+    if cmdline[0] == "starfish" and coverage_enabled:
+        coverage_cmdline = [
+            "coverage", "run",
+            "-p",
+            "--source", "starfish",
+            "-m", "starfish",
+        ]
+        coverage_cmdline.extend(cmdline[1:])
+        cmdline = coverage_cmdline
+    elif cmdline[0] == "validate-sptx" and coverage_enabled:
+        coverage_cmdline = [
+            "coverage", "run",
+            "-p",
+            "--source", "validate_sptx",
+            "-m", "validate_sptx",
+        ]
+        coverage_cmdline.extend(cmdline[1:])
+        cmdline = coverage_cmdline
+    return cmdline
diff --git a/validate_sptx/cli.py b/validate_sptx/cli.py
new file mode 100644
index 00000000..8e8751fb
--- /dev/null
+++ b/validate_sptx/cli.py
@@ -0,0 +1,30 @@
+import argparse
+import json
+import sys
+
+from starfish.util.argparse import FsExistsType
+from validate_sptx.validate_sptx import validate
+
+
+class Cli:
+    parser_group = None
+
+    @staticmethod
+    def add_to_parser(parser):
+        """adds experiment-json and fuzz arguments to the given parser"""
+        parser.add_argument(
+            "--experiment-json",
+            required=True,
+            metavar="JSON_FILE_OR_URL")
+        parser.add_argument(
+            "--fuzz", action="store_true")
+        parser.set_defaults(starfish_command=Cli.run)
+        Cli.parser_group = parser
+
+    @staticmethod
+    def run(args, print_help=False):
+        """invokes validate with the parsed commandline arguments"""
+        try:
+            validate(args.experiment_json, fuzz=args.fuzz)
+        except KeyboardInterrupt:
+            sys.exit(3)
diff --git a/validate_sptx/util.py b/validate_sptx/util.py
index 127d4760..29f5b252 100644
--- a/validate_sptx/util.py
+++ b/validate_sptx/util.py
@@ -1,21 +1,28 @@
+import copy
 import json
 import os
+import sys
 import warnings
+
 from pkg_resources import resource_filename
-from typing import Dict, Iterator
+from typing import Any, Dict, IO, Iterator, List, Optional
 
 from jsonschema import RefResolver, Draft4Validator, ValidationError
 
 
 class SpaceTxValidator:
 
-    def __init__(self, schema: str) -> None:
+    def __init__(self, schema: str, fuzz: bool=False) -> None:
         """create a validator for a json-schema compliant spaceTx specification file
 
         Parameters
         ----------
         schema : str
             file path to schema
+        fuzz : bool
+            if true, then the json documents which are validated will
+            be modified piece-wise and a statement printed to standard
+            out about whether or not they are still valid.
 
         """
         self._schema: Dict = self.load_json(schema)
@@ -59,7 +66,7 @@ class SpaceTxValidator:
             iterator over ValidationErrors that occur during validation
         level : int
             current level of recursion
-        filename : str
+        filename: str
             informational string regarding the source file of the given object
 
         """
@@ -98,7 +105,7 @@ class SpaceTxValidator:
         target_object = self.load_json(target_file)
         return self.validate_object(target_object, target_file)
 
-    def validate_object(self, target_object: Dict, target_file: str=None) -> bool:
+    def validate_object(self, target_object: Dict, target_file: str=None, fuzz: bool=False) -> bool:
         """validate a loaded json object, returning True if valid, and False otherwise
 
         Parameters
@@ -107,16 +114,203 @@ class SpaceTxValidator:
             loaded json object to be validated against the schem passed to this object's constructor
         target_file : str
             informational string regarding the source file of the given object
+        fuzz: bool
+            whether or not to perform element-by-element fuzzing.
+            If true, will return true and will *not* use warnings.
 
         Returns
         -------
         bool :
-            True, if object valid, else False
+            True, if object valid or fuzz=True, else False
 
         """
+
+        if fuzz:
+            if target_file:
+                print(f"> Fuzzing {target_file}...")
+            else:
+                print("> Fuzzing unknown...")
+            fuzzer = Fuzzer(self._validator, target_object)
+            fuzzer.fuzz()
+            return True
+
         if self._validator.is_valid(target_object):
             return True
         else:
             es: Iterator[ValidationError] = self._validator.iter_errors(target_object)
             self._recurse_through_errors(es, filename=target_file)
             return False
+
+
+class Fuzzer(object):
+
+    def __init__(self, validator: Draft4Validator, obj: Any, out: IO=sys.stdout) -> None:
+        """create a fuzzer which will check different situations against the validator
+
+        Parameters
+        ----------
+        validator : SpaceTxValidator
+            validator which should match the given object type
+        obj : Any
+            JSON-like object which will be checked against the validator
+        out : IO
+            if true, then the json documents which are validated will
+            be modified piece-wise and a statement printed to standard
+            out about whether or not they are still valid.
+
+        """
+        self.validator = validator
+        self.obj = obj
+        self.out = out
+        self.stack: Optional[List[Any]] = None
+
+    def fuzz(self) -> None:
+        """prints to the out field the state of the object tree after types of fuzzing
+
+        Each line is prefixed by the output of {state()} followed by a YAML-like
+        representation of the branch of the object tree.
+        """
+        header = f"{self.state()}"
+        header += "If the letter is present, mutation is valid!"
+        self.out.write(f"{header}\n")
+        self.out.write("".join([x in ("\t", "\n") and x or "-" for x in header]))
+        self.out.write("\n")
+        self.stack: List[Any] = []
+        try:
+            self.descend(self.obj)
+        finally:
+            self.stack = None
+
+    def state(self) -> str:
+        """primary driver for the checks of individual trees
+
+        Returns
+        -------
+        str :
+            space-separated representation of the fuzzing conditions.
+            If a letter is present, then mutation leaves the tree in
+            a valid state:
+
+             A: inserting a fake key or appending to a list
+             D: deleting a key or index
+             I: converting value to an integer
+             I: converting value to a string
+             M: converting value to an empty dict
+             L: converting value to an empty list
+
+        """
+        rv = [
+            Add().check(self),
+            Del().check(self),
+            Change("I", lambda *args: 123456789).check(self),
+            Change("S", lambda *args: "fake").check(self),
+            Change("M", lambda *args: dict()).check(self),
+            Change("L", lambda *args: list()).check(self),
+        ]
+        return ' '.join(rv) + "\t"
+
+    def descend(self, obj: Any, depth: int=0, prefix: str="") -> None:
+        """walk a JSON-like object tree printing the state of the tree
+        at each level. A YAML representation is used for simplicity.
+
+        Parameters
+        ----------
+        obj : Any
+            JSON-like object tree
+        depth : int
+            depth in the tree that is currently being evaluated
+        prefix : str
+            value which should be prepended to printouts at this level
+        """
+        if self.stack is None:
+            raise Exception("invalid state")
+        if isinstance(obj, list):
+            for i, o in enumerate(obj):
+                depth += 1
+                self.stack.append(i)
+                self.descend(o, depth, prefix="- ")
+                self.stack.pop()
+                depth -= 1
+        elif isinstance(obj, dict):
+            for k in obj:
+                # This is something of a workaround in that we need a special
+                # case for object keys since no __getitem__ method will suffice.
+                self.stack.append((k,))
+                self.out.write(f"{self.state()}{' ' * depth}{prefix}{k}:\n")
+                self.stack.pop()
+                if prefix == "- ": prefix = "  "
+                depth += 1
+                self.stack.append(k)
+                self.descend(obj[k], depth, prefix="  "+prefix)
+                self.stack.pop()
+                depth -= 1
+        else:
+            self.out.write(f"{self.state()}{' ' * depth}{prefix}{obj}\n")
+
+class Checker(object):
+
+    LETTER : str = "?"
+
+    def check(self, fuzz: Fuzzer) -> str:
+        """create a copy of the current state of the object tree,
+        mutate it, and run it through is_valid on the validator.
+
+        Parameters
+        ----------
+        fuzz : Fuzzer
+            the containing instance
+
+        Returns
+        -------
+        str :
+            A single character string representation of the check
+
+        """
+        # Don't mess with the top level
+        if fuzz.stack is None: return self.LETTER
+        if not fuzz.stack: return "-"
+        # Operate on a copy for mutating
+        dupe = copy.deepcopy(fuzz.obj)
+        target = dupe
+        for level in fuzz.stack[0:-1]:
+            target = target.__getitem__(level)
+        self.handle(fuzz, target)
+        valid = fuzz.validator.is_valid(dupe)
+        return valid and self.LETTER or "."
+
+    def handle(self, fuzz, target):
+       raise NotImplementedError()
+
+class Add(Checker):
+
+    LETTER = "A"
+
+    def handle(self, fuzz, target):
+        if isinstance(target, dict):
+            target["fake"] = "!"
+        elif isinstance(target, list):
+            target.append("!")
+        else:
+            raise Exception("unknown")
+
+class Del(Checker):
+
+    LETTER = "D"
+
+    def handle(self, fuzz, target):
+        key = fuzz.stack[-1]
+        if isinstance(key, tuple):
+            key = key[0]
+        target.__delitem__(key)
+
+class Change(Checker):
+
+    def __init__(self, letter, call):
+        self.LETTER = letter
+        self.call = call
+
+    def handle(self, fuzz, target):
+        key = fuzz.stack[-1]
+        if isinstance(key, tuple):
+            key = key[0]
+        target.__setitem__(key, self.call())
diff --git a/validate_sptx/validate_sptx.py b/validate_sptx/validate_sptx.py
index db707d30..cafd0b75 100644
--- a/validate_sptx/validate_sptx.py
+++ b/validate_sptx/validate_sptx.py
@@ -19,9 +19,26 @@ def _get_absolute_schema_path(schema_name: str) -> str:
 
 @click.command()
 @click.option('--experiment-json', help='image metadata file to validate')
-def validate_sptx(experiment_json: str) -> None:
+def validate_sptx(experiment_json: str, fuzz: bool=False) -> bool:
+    return validate(experiment_json, fuzz)
+
+def validate(experiment_json: str, fuzz: bool=False) -> bool:
     """validate a spaceTx formatted experiment.
     Accepts local filepaths or files hosted at http links.
+
+    Parameters
+    ----------
+    experiment_json : str
+        path or URL to a target json object to be validated against the schema passed to this
+        object's constructor
+    fuzz : bool
+        whether or not to perform element-by-element fuzzing.
+        If true, will return true and will *not* use warnings.
+
+    Returns
+    -------
+    bool :
+        True, if object valid or fuzz=True, else False
     """
 
     valid = True
@@ -33,27 +50,27 @@ def validate_sptx(experiment_json: str) -> None:
 
     # validate experiment.json
     experiment_validator = SpaceTxValidator(_get_absolute_schema_path('experiment.json'))
-    valid &= experiment_validator.validate_object(experiment)
+    valid &= experiment_validator.validate_object(experiment, name, fuzz=fuzz)
 
     # validate manifests that it links to.
     possible_manifests = []
     manifest_validator = SpaceTxValidator(_get_absolute_schema_path('fov_manifest.json'))
     with backend.read_contextmanager(experiment['primary_images']) as fh:
-        possible_manifests.append(json.load(fh))
+        possible_manifests.append((json.load(fh), experiment['primary_images']))
 
     # loop over all the manifests that are stored in auxiliary images. Disallowed names will
     # have already been excluded by experiment validation.
     for manifest in experiment['auxiliary_images'].values():
         with backend.read_contextmanager(manifest) as fh:
-            possible_manifests.append(json.load(fh))
+            possible_manifests.append((json.load(fh), manifest))
 
     # we allow the objects linked from primary_images and auxiliary images to either be
     # manifests OR field_of_view files. We distinguish these by checking if they have a `contents`
     # flag, which indicates it is a manifest.
     fovs = []
-    for manifest in possible_manifests:
+    for manifest, filename in possible_manifests:
         if 'contents' in manifest:  # is a manifest; validate
-            valid &= manifest_validator.validate_object(manifest)
+            valid &= manifest_validator.validate_object(manifest, filename, fuzz=fuzz)
 
             # contains fields of view
             for key, fov in manifest['contents'].items():
@@ -67,7 +84,7 @@ def validate_sptx(experiment_json: str) -> None:
     assert len(fovs) != 0
     fov_validator = SpaceTxValidator(_get_absolute_schema_path('field_of_view/field_of_view.json'))
     for fov, filename in fovs:
-        valid &= fov_validator.validate_object(fov, filename)
+        valid &= fov_validator.validate_object(fov, filename, fuzz=fuzz)
 
     # validate codebook
     codebook_validator = SpaceTxValidator(_get_absolute_schema_path('codebook/codebook.json'))
@@ -76,13 +93,15 @@ def validate_sptx(experiment_json: str) -> None:
     if codebook_file is not None:
         with backend.read_contextmanager(codebook_file) as fh:
             codebook = json.load(fh)
-    valid &= codebook_validator.validate_object(codebook, codebook_file)
+    valid &= codebook_validator.validate_object(codebook, codebook_file, fuzz=fuzz)
+
+    return valid
+
+
+if __name__ == "__main__":
+    valid = validate_sptx()
 
     if valid:
         sys.exit(0)
     else:
         sys.exit(1)
-
-
-if __name__ == "__main__":
-    validate_sptx()


Test patch:
diff --git a/starfish/test/full_pipelines/cli/test_validate.py b/starfish/test/full_pipelines/cli/test_validate.py
new file mode 100644
index 00000000..16f3dd3e
--- /dev/null
+++ b/starfish/test/full_pipelines/cli/test_validate.py
@@ -0,0 +1,21 @@
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+import unittest
+
+from starfish.util import exec
+
+
+class TestValidateCommand(unittest.TestCase):
+
+    STAGES = (
+        [
+            "starfish", "validate", "--help"
+        ],
+    )
+
+    def test_run_pipeline(self):
+        exec.stages(TestValidateCommand.STAGES)
diff --git a/validate_sptx/test_fuzz.py b/validate_sptx/test_fuzz.py
new file mode 100644
index 00000000..0fb1b00b
--- /dev/null
+++ b/validate_sptx/test_fuzz.py
@@ -0,0 +1,98 @@
+import os
+from pkg_resources import resource_filename
+
+from .util import SpaceTxValidator
+from .util import Fuzzer
+
+codebook_schema_path = resource_filename("validate_sptx", "schema/codebook/codebook.json")
+experiment_schema_path = resource_filename("validate_sptx", "schema/experiment.json")
+codebook_validator = SpaceTxValidator(codebook_schema_path)
+experiment_validator = SpaceTxValidator(experiment_schema_path)
+
+
+def test_fuzz_mock():
+    """
+    Starting from the simple obj test class, the Fuzzer will
+    propose all the different mutations contained in the the
+    values list. The mocked validator will pop them off, in
+    order, and compare what it is being given. For testing
+    purposes, it will always return true.
+    """
+    obj = {
+        "list": [1, 2, 3]
+    }
+    values = [
+        { "list": [1, 2, 3], "fake": "!" },
+        { },
+        { "list": 123456789 },
+        { "list": "fake" },
+        { "list": {} },
+        { "list": [] },
+        { "list": [1, 2, 3, '!'] },
+        { "list": [2, 3] },
+        { "list": [123456789, 2, 3] },
+        { "list": ["fake", 2, 3] },
+        { "list": [{}, 2, 3] },
+        { "list": [[], 2, 3] },
+        { "list": [1, 2, 3, "!"]},
+        { "list": [1, 3] },
+        { "list": [1, 123456789, 3] },
+        { "list": [1, "fake", 3] },
+        { "list": [1, {}, 3] },
+        { "list": [1, [], 3] },
+        { "list": [1, 2, 3, '!'] },
+        { "list": [1, 2] },
+        { "list": [1, 2, 123456789] },
+        { "list": [1, 2, "fake"] },
+        { "list": [1, 2, {}] },
+        { "list": [1, 2, []] },
+    ]
+    class Validator(object):
+        def __init__(self):
+            self.called = 0
+        def is_valid(self, obj):
+            self.called += 1
+            assert obj == values.pop(0), self.called
+            return True
+    Fuzzer(Validator(), obj).fuzz()
+
+
+def test_fuzz_codebook():
+    """
+    simple validation of a hard-coded example codebook.
+
+    The actual values don't matter overly much, but it
+    provides a good example of the output.
+    """
+    codebook = [
+        {
+          "codeword": [
+            {"r": 0, "c": 0, "v": 1},
+            {"r": 0, "c": 1, "v": 1},
+            {"r": 1, "c": 1, "v": 1}
+          ],
+          "target": "SCUBE2"
+        }
+    ]
+    assert codebook_validator.validate_object(codebook, fuzz=True)
+
+
+def test_fuzz_experiment():
+    """
+    simple validation of a hard-coded example experiment.
+
+    The actual values don't matter overly much, but it
+    provides a good example of the output.
+    """
+    experiment = {
+        "version": "0.0.0",
+        "primary_images": "primary_images.json",
+        "auxiliary_images": {
+          "nuclei": "nuclei.json"
+        },
+        "codebook": "codebook.json",
+        "extras": {
+          "is_space_tx_cool": True
+        }
+      }
+    assert experiment_validator.validate_object(experiment, fuzz=True)