Repository: microsoft/pybryt

Issue:
PyBryt CLI Feature
PyBryt CLI 

Goal : The ability to use a command line interface to create a reference implementation and grade a students work.

**Describe the solution you'd like**
Goal here is to allow the easy onboarding of educators to using PyBryt 

So task would be 
1. Educator can create a reference implementation 
2. Educator uses PyBryt CLI to create a solution with reference implementations 

**Describe alternatives you've considered**
CLI Features bases MVP
- CLI to run a student code based on reference implementation (single submission)
Future features
- Pickle Reference implementation 
- Use Pickle reference to undertake assessment 
- Or CLI to pickle and compare 
 
**Additional context**
To be discussed. 

Discussion:

Patch:
diff --git a/.coveragerc b/.coveragerc
index ff6415d..7342ed4 100644
--- a/.coveragerc
+++ b/.coveragerc
@@ -1,3 +1,7 @@
+[run]
+concurrency = multiprocessing
+source = pybryt
+
 [report]
 exclude_lines =
     # Have to re-enable the standard pragma
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 62cc180..a594050 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -11,6 +11,7 @@ All notable changes to this project will be documented in this file, and this pr
 * Fixed bug in intermediate variable preprocessor for newer versions of Python
 * Added a context manager for checking code against a reference implementation from the same
   notebook
+* Added a CLI for interacting with PyBryt
 
 ## 0.0.5 - 2021-05-04
 
diff --git a/Makefile b/Makefile
index 8ecadd5..9529c8a 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,8 @@ test:
 	$(PYTEST) $(TESTPATH) $(PYTESTOPTS)
 
 testcov:
-	$(COVERAGE) run --source=pybryt -m pytest $(TESTPATH) $(PYTESTOPTS) 
+	$(COVERAGE) run -m pytest $(TESTPATH) $(PYTESTOPTS) 
+	$(COVERAGE) combine
 
 covhtml: testcov
 	$(COVERAGE) html
diff --git a/docs/cli_reference.rst b/docs/cli_reference.rst
new file mode 100644
index 0000000..4dc9ce5
--- /dev/null
+++ b/docs/cli_reference.rst
@@ -0,0 +1,93 @@
+Command-Line Interface
+======================
+
+PyBryt includes a small command-line interface for performing quick tasks using its functionality.
+The main tasks that the CLI currently supports are:
+
+- compiling reference implementations and saving them to a file
+- executing student implementations and saving them to a file
+- checking reference implementations against student implementations (as notebooks or pickled objects)
+  and saving/echoing the results
+
+Each of the commands is detailed in the :ref:`refererence<cli_reference>` below, but a short summary
+of each is provided here. The CLI can be invoked via the command ``pybryt`` or by calling PyBryt as a 
+Python module: ``python3 -m pybryt``. For simplicity, the convention of the former is used here.
+
+
+CLI Sub-commands
+----------------
+
+
+``check``
++++++++++
+
+``pybryt check`` is used to check a student implementation against a reference implementation. It
+takes two position arguments corresponding to the path to a reference implementation and the path to
+a student implementation. Both paths can lead either to notebooks (which will be executed/compiled
+if provided) or to pickle files.
+
+The output of this command comes in three forms:
+
+- a pickled :py:class:`ReferenceResult<pybryt.reference.ReferenceResult>` object, written to a file
+- a JSON file with a text-based representation of the ``ReferenceResult`` object
+- a report echoed to the console
+
+To set the output type, use the ``-t`` flag. If a file is written, use the ``-o`` flag to set the 
+output path (defaults to ``{stu.stem}_results.[pkl|json]``).
+
+.. code-block:: console
+
+    $ ls
+    reference.ipynb         subm.ipynb
+    $ pybryt check reference.ipynb subm.ipynb
+    $ ls
+    reference.ipynb         subm.ipynb              subm_results.pkl
+    $ pybryt check reference.pkl subm.pkl -t json
+    $ ls
+    reference.ipynb         subm.ipynb              subm_results.json       subm_results.pkl
+    $ pybryt check reference.ipynb subm.pkl -t report
+    REFERENCE: median
+    SATISFIED: True
+    MESSAGES:
+    ...
+
+
+``compile``
++++++++++++
+
+``pybryt compile`` is used to compile a reference implementation to a file. It takes a single positional
+argument, the path to the notebook to be compiled. To set the destination path, use the ``-d`` flag.
+If a name is needed for the reference implementation, it can be provided with the ``-n`` flag.
+
+.. code-block:: console
+
+    $ pybryt compile reference.ipynb
+    $ pybryt compile reference.ipynb -d ref.pkl -n foo
+
+
+``execute``
++++++++++++
+
+``pybryte execute`` is used to execute one or more student implementations and write the memory 
+footprints to files for futher processing. All paths passed as position arguments to this command
+are paths to student implementation notebooks.
+
+Because execution can be time-consuming, ``pybryt execute`` supports parallelism using Python's
+``multiprocessing`` library. To enable parallelism, use the ``-p`` flag.
+
+.. code-block:: console
+
+    $ pybryt execute submissions/*.ipynb
+    $ pybryt execute submissions/*.ipynb -p
+
+
+.. _cli_reference:
+
+Reference
+---------
+
+This section is a short reference for all of the commands and options for PyBryt's CLI.
+
+.. click:: pybryt.cli:click_cli
+    :prog: pybryt
+    :nested: full
diff --git a/docs/conf.py b/docs/conf.py
index 714b46a..35e25ca 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -33,6 +33,7 @@ extensions = [
     'sphinxarg.ext',
     'IPython.sphinxext.ipython_console_highlighting',
     'IPython.sphinxext.ipython_directive',
+    'sphinx_click',
 ]
 
 # extension configurations
diff --git a/docs/index.rst b/docs/index.rst
index 0c16ab0..999680e 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -14,6 +14,7 @@ PyBryt Documentation
    annotations/index
    reference_implementations
    student_implementations
+   cli_reference
    api_reference
    Changelog <https://github.com/microsoft/pybryt/tree/main/CHANGELOG.md>
 
@@ -31,7 +32,8 @@ benefits are:
 
 - Educators do not have to enforce the structure of the solution
 - Learners practice algorithm design, code design, and solution implementation
-- Learners receive quick and meaningful pedagogical feedback, which substantially contributes to the learning experience
+- Learners receive quick and meaningful pedagogical feedback, which substantially contributes 
+  to the learning experience
 - Complexity of the learner's solution can be analyzed
 - Plagiarism detection and support for reference implementations
 - Easy integration into existing organizational or institutional grading infrastructure
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 544a0a5..26dd287 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,3 +1,4 @@
 furo
 sphinxcontrib-apidoc
 sphinx-argparse
+sphinx-click
diff --git a/pybryt/__main__.py b/pybryt/__main__.py
new file mode 100644
index 0000000..d5e7dd1
--- /dev/null
+++ b/pybryt/__main__.py
@@ -0,0 +1,7 @@
+""""""
+
+from .cli import cli
+
+
+if __name__ == "__main__":
+    cli()
diff --git a/pybryt/cli.py b/pybryt/cli.py
new file mode 100644
index 0000000..520eaaa
--- /dev/null
+++ b/pybryt/cli.py
@@ -0,0 +1,139 @@
+"""Command-line interface for PyBryt"""
+
+import os
+import dill
+import json
+import click
+
+from . import (
+    generate_report, generate_student_impls, ReferenceImplementation, StudentImplementation, __version__
+)
+from .utils import get_stem
+
+
+@click.group()
+@click.version_option(__version__)
+def click_cli():
+    """
+    A command-line interface for PyBryt. See https://microsoft.github.io/pybryt for more information.
+    """
+    pass
+
+
+@click_cli.command("compile")
+@click.option("-d", "--dest", default=None, type=click.Path(dir_okay=False), 
+              help="Path at which to write the pickled reference implementation")
+@click.option("-n", "--name", default=None, type=click.STRING, 
+              help="Optional name for the reference implementation")
+@click.argument("src", type=click.Path(exists=True, dir_okay=False))
+def compile_reference(src, dest, name):
+    """
+    Compile the reference implementation SRC.
+
+    SRC must be a path to a Jupyter Notebook file which defines a reference implementation.
+    """
+    ref = ReferenceImplementation.compile(src, name=name)
+    ref.dump(dest)
+
+
+@click_cli.command()
+@click.option("-n", "--name", default=None, type=click.STRING, 
+              help="Optional name for the reference implementation")
+@click.option("--output-nb", default=None, type=click.Path(dir_okay=False), 
+              help="Path at which to write the output notebook from executing the student submission")
+@click.option("-o", "--output", default=None, type=click.Path(dir_okay=False), 
+              help="Path at which to write the results of the check")
+@click.option("-t", "--type", "output_type", default="pickle", show_default=True, 
+              type=click.Choice(["pickle", "json", "report"]), help="Type of output to write")
+@click.argument("ref", type=click.Path(exists=True, dir_okay=False))
+@click.argument("stu", type=click.Path(exists=True, dir_okay=False))
+def check(ref, stu, name, output_nb, output, output_type):
+    """
+    Run a student submission against a reference implementation.
+
+    REF can be a path to a pickled reference implementation or to a notebook to be compiled on-the-fly.
+    STU can be a path to a pickled student implementation or to a notebook to be executed.
+
+    If TYPE is "pickle" or "json", the output is a file. If TYPE is "report", a report is echoed to
+    the console and OUTPUT is ignored.
+    """
+    if os.path.splitext(ref)[1] == ".ipynb":
+        ref = ReferenceImplementation.compile(ref)
+    else:
+        try:
+            ref = ReferenceImplementation.load(ref)
+        except:
+            raise RuntimeError(f"Could not load the reference implementation {ref}")
+
+    if output is None:
+        output = get_stem(stu) + "_results" + (".pkl", ".json")[output_type == "json"]
+
+    if os.path.splitext(stu)[1] == ".ipynb":
+        stu = StudentImplementation(stu, output=output_nb)
+    else:
+        try:
+            stu = StudentImplementation.load(stu)
+        except:
+            raise RuntimeError(f"Could not load the student implementation {stu}")
+
+    res = stu.check(ref)
+
+    if output_type == "pickle":
+        if isinstance(res, list):
+            with open(output, "wb+") as f:
+                dill.dump(res, f)
+        else:
+            res.dump(output)
+    elif output_type == "json":
+        if isinstance(res, list):
+            d = [r.to_dict() for r in res]
+        else:
+            d = res.to_dict()
+        with open(output, "w+") as f:
+            json.dump(d, f, indent=2)
+    elif output_type == "report":
+        report = generate_report(res)
+        click.echo(report)
+
+
+@click_cli.command()
+@click.option("-p", "--parallel", is_flag=True, default=False, 
+              help="Execute notebooks in parallel using the multiprocessing library")
+@click.option("-o", "--output", default=None, type=click.Path(), 
+              help="Path at which to write the pickled student implementation")
+@click.argument("subm", nargs=-1, type=click.Path(exists=True, dir_okay=False))
+def execute(subm, parallel, output):
+    """
+    Execute student submissions to generate memory footprints.
+
+    Executes the student submission(s) SUBM at writes the pickled objects to some output file. If
+    OUTPUT is unspecified, this defaults to "./{SUBM.stem}.pkl" (e.g. for SUBM 
+    "submissions/subm01.ipynb", this is "./subm01.pkl").
+    """
+    if len(subm) == 0:
+        raise ValueError("You must specify at least one notebook to execute")
+
+    stus = generate_student_impls(subm, parallel=parallel)
+
+    if len(subm) == 1:
+        stus[0].dump(output)
+
+    else:
+        if output is None:
+            output = "./"
+
+        if not os.path.isdir(output):
+            raise ValueError(f"Output directory {output} does not exist or is not a directory")
+
+        for s, stu in zip(subm, stus):
+            stem = get_stem(s)
+            path = os.path.join(output, stem + ".pkl")
+            stu.dump(path)
+
+
+def cli(*args, **kwargs):
+    """
+    Wrapper for the click CLI that sets the prog name.
+    """
+    prog_name = kwargs.pop("prog_name", "pybryt")
+    return click_cli(*args, prog_name=prog_name, **kwargs)
diff --git a/pybryt/reference.py b/pybryt/reference.py
index 0d02e8c..89181c0 100644
--- a/pybryt/reference.py
+++ b/pybryt/reference.py
@@ -14,15 +14,16 @@ from textwrap import indent
 from typing import Any, Dict, List, NoReturn, Optional, Tuple, Union
 
 from .annotations import Annotation, AnnotationResult
-from .utils import get_stem, notebook_to_string
+from .utils import get_stem, notebook_to_string, Serializable
 
 
-class ReferenceImplementation:
+class ReferenceImplementation(Serializable):
     """
     A reference implementation class for managing collections of annotations. Defines methods for
     creating, running, and storing reference implementations.
 
     Args:
+        name (``str``): the name of the reference implementation
         annotations (``list[Annotation]``): the annotations comprising this reference implementation
     """
     
@@ -30,7 +31,7 @@ class ReferenceImplementation:
     """the annotations comprising this reference implementation"""
 
     name: str
-    """"""
+    """the name of the reference implementation"""
 
     def __init__(self, name: str, annotations: List[Annotation]):
         if not isinstance(annotations, list):
@@ -57,36 +58,9 @@ class ReferenceImplementation:
         return isinstance(other, type(self)) and self.annotations == other.annotations and \
             self.name == other.name
 
-    @staticmethod
-    def load(file: str) -> 'ReferenceImplementation':
-        """
-        Unpickles a reference implementation from a file.
-
-        Args:
-            file (``str``): the path to the file
-        
-        Returns:
-            :py:class:`ReferenceImplementation<pybryt.ReferenceImplementation>`: the unpickled
-            reference implementation
-        """
-        with open(file, "rb") as f:
-            instance = dill.load(f)
-        if not isinstance(instance, ReferenceImplementation):
-            raise TypeError(f"Unpickled reference implementation has type {type(instance)}")
-        return instance
-    
-    def dump(self, dest: Optional[str] = None) -> NoReturn:
-        """
-        Pickles this reference implementation to a file.
-
-        Args:
-            dest (``str``, optional): the path to the file; if unspecified, defaults to 
-                ``{self.name}.pkl``
-        """
-        if dest is None:
-            dest = f"{self.name}.pkl"
-        with open(dest, "wb+") as f:
-            dill.dump(self, f)
+    @property
+    def _default_dump_dest(self) -> str:
+        return f"{self.name}.pkl"
 
     def run(self, observed_values: List[Tuple[Any, int]], group: Optional[str] = None) -> 'ReferenceResult':        
         """
@@ -174,7 +148,7 @@ class ReferenceImplementation:
         return refs
 
 
-class ReferenceResult:
+class ReferenceResult(Serializable):
     """
     Class for wrangling and managing the results of a reference implementation. Collects a series of
     :py:class:`AnnotationResult<pybryt.AnnotationResult>` objects and provides an API for managing
@@ -203,11 +177,15 @@ class ReferenceResult:
         self.reference = reference
         self.results = annotation_results
         self.group = group
-    
+
     def __repr__(self):
         results = ',\n  '.join(repr(r) for r in self.results)
         return f"ReferenceResult([\n  {results}\n])"
 
+    @property
+    def _default_dump_dest(self) -> str:
+        return f"{self.name}_results.pkl"
+
     @property
     def correct(self) -> bool:
         """
diff --git a/pybryt/student.py b/pybryt/student.py
index cf47ef4..db6dad2 100644
--- a/pybryt/student.py
+++ b/pybryt/student.py
@@ -1,6 +1,6 @@
 """Student implementations for PyBryt"""
 
-__all__ = ["StudentImplementation", "check"]
+__all__ = ["StudentImplementation", "check", "generate_student_impls"]
 
 import os
 import dill
@@ -9,6 +9,7 @@ import nbformat
 import inspect
 
 from contextlib import contextmanager
+from multiprocessing import Process, Queue
 from types import FrameType
 from typing import Any, Dict, List, NoReturn, Optional, Tuple, Union
 
@@ -17,9 +18,10 @@ from .execution import (
     tracing_on, TRACING_VARNAME
 )
 from .reference import generate_report, ReferenceImplementation, ReferenceResult
+from .utils import Serializable
 
 
-class StudentImplementation:
+class StudentImplementation(Serializable):
     """
     A student implementation class for handling the execution of student work and manging the 
     memory footprint generated by that execution.
@@ -91,55 +93,19 @@ class StudentImplementation:
         stu.values = footprint
         return stu
 
-    def dump(self, dest: str = "student.pkl") -> NoReturn:
+    def __eq__(self, other: Any) -> bool:
         """
-        Pickles this student implementation to a file.
+        Checks whether a student implementation is equal to another object.
 
-        Args:
-            dest (``str``, optional): the path to the file
-        """
-        with open(dest, "wb+") as f:
-            dill.dump(self, f)
-
-    def dumps(self) -> str:
-        """
-        Pickles this student implementation to a base-64-encoded string.
-
-        Returns:
-           ``str``: the pickled and encoded student implementation
-        """
-        bits = dill.dumps(self)
-        return base64.b64encode(bits).decode("ascii")
-
-    @staticmethod
-    def load(file: str) -> Union['StudentImplementation']:
+        The object is considered equal if it is also a student implementation, has the same memory
+        footprint, the same number of steps, and the same source notebook.
         """
-        Unpickles a student implementation from a file.
+        return isinstance(other, type(self)) and self.values == other.values and \
+            self.steps == other.steps and self.nb == other.nb
 
-        Args:
-            file (``str``): the path to the file
-        
-        Returns:
-            :py:class:`StudentImplementation<pybryt.StudentImplementation>`: the unpickled student 
-            implementation
-        """
-        with open(file, "rb") as f:
-            instance = dill.load(f)
-        return instance
-
-    @staticmethod
-    def loads(data: str) -> "StudentImplementation":
-        """
-        Unpickles a student implementation from a base-64-encoded string.
-
-        Args:
-            data (``str``): the pickled and encoded student implementation
-        
-        Returns:
-            :py:class:`StudentImplementation<pybryt.StudentImplementation>`: the unpickled student 
-            implementation
-        """
-        return dill.loads(base64.b64decode(data.encode("ascii")))
+    @property
+    def _default_dump_dest(self) -> str:
+        return "student.pkl"
 
     def check(self, ref: Union[ReferenceImplementation, List[ReferenceImplementation]], group=None) -> \
             Union[ReferenceResult, List[ReferenceResult]]:
@@ -187,6 +153,55 @@ class StudentImplementation:
         return get_impl_results(refs[0], student_impls, **kwargs)
 
 
+def generate_student_impls(
+    paths_or_nbs: List[Union[str, nbformat.NotebookNode]], parallel: bool = False, **kwargs
+) -> List[StudentImplementation]:
+    """
+    Generates multiple student implementations from a list of file paths or notebooks.
+
+    Can optionally generate the student implementations in parallel processes using Python's 
+    ``multiprocessing`` library to reduce the runtime.
+
+    Args:
+        paths_or_nbs (``list[Union[str, nbformat.NotebookNode]]``): the notebooks or paths to them
+        parallel (``bool``, optional): whether to execute in parallel
+        **kwargs: additional keyword arguments passed to the 
+            :py:class:`StudentImplementation<pybryt.StudentImplementation>` constructor
+
+    Returns:
+        ``list[StudentImplementation]``: the student implementations
+    """
+    if parallel:
+        def create_and_collect_impl(impl, queue, **kwargs):
+            stu = StudentImplementation(impl, **kwargs)
+            queue.put((impl, stu))
+
+    impls = []
+    if parallel:
+        queue = Queue()
+
+    for stu in paths_or_nbs:
+        if parallel:
+            p = Process(target=create_and_collect_impl, args=(stu, queue), kwargs=kwargs)
+            p.start()
+            impls.append(p)
+        else:
+            impls.append(StudentImplementation(stu, **kwargs))
+
+    if parallel:
+        for p in impls:
+            p.join()
+
+        procs, impls = impls, {}
+        while not queue.empty():
+            t = queue.get()
+            impls[str(t[0])] = t[1]
+
+        impls = [impls[str(p)] for p in paths_or_nbs]
+
+    return impls
+
+
 class check:
     """
     A context manager for running a block of student code against a reference implementation.
diff --git a/pybryt/utils.py b/pybryt/utils.py
index 3614a58..1d53adc 100644
--- a/pybryt/utils.py
+++ b/pybryt/utils.py
@@ -4,13 +4,15 @@
 import os
 import json
 import random
+import base64
 import string
 import dill
 import hashlib
 import time
 import nbformat
 
-from typing import Any, List, NoReturn, Union
+from abc import ABC, abstractmethod
+from typing import Any, List, NoReturn, Optional, Union
 from IPython import get_ipython
 from IPython.display import publish_display_data
 
@@ -137,3 +139,72 @@ def get_stem(fp):
         ``str``: the stem of the filepath
     """
     return os.path.splitext(os.path.split(fp)[1])[0]
+
+
+class Serializable(ABC):
+    """
+    A class that implements serialization using the ``dill`` library.
+    """
+
+    @property
+    @abstractmethod
+    def _default_dump_dest(self) -> str:
+        """
+        Default destination path for the ``dump`` method
+        """
+        ... # pragma: no cover
+
+    def dump(self, dest: Optional[str] = None) -> NoReturn:
+        """
+        Pickles this object to a file.
+
+        Args:
+            dest (``str``, optional): the path to the file
+        """
+        if dest is None:
+            dest = self._default_dump_dest
+        with open(dest, "wb+") as f:
+            dill.dump(self, f)
+
+    def dumps(self) -> str:
+        """
+        Pickles this object to a base-64-encoded string.
+
+        Returns:
+           ``str``: the pickled and encoded object
+        """
+        bits = dill.dumps(self)
+        return base64.b64encode(bits).decode("ascii")
+
+    @classmethod
+    def load(cls, file: str) -> 'Serializable':
+        """
+        Unpickles an object from a file.
+
+        Args:
+            file (``str``): the path to the file
+        
+        Returns:
+            ``Serializable``: the unpickled object
+        """
+        with open(file, "rb") as f:
+            instance = dill.load(f)
+        if not isinstance(instance, cls):
+            raise TypeError(f"Unpickled object is not of type {cls}")
+        return instance
+
+    @classmethod
+    def loads(cls, data: str) -> "Serializable":
+        """
+        Unpickles an object from a base-64-encoded string.
+
+        Args:
+            data (``str``): the pickled and encoded object
+        
+        Returns:
+            ``Serializable``: the unpickled object
+        """
+        instance = dill.loads(base64.b64decode(data.encode("ascii")))
+        if not isinstance(instance, cls):
+            raise TypeError(f"Unpickled object is not of type {cls}")
+        return instance
diff --git a/setup.py b/setup.py
index 6cd7b70..2b886f7 100644
--- a/setup.py
+++ b/setup.py
@@ -29,4 +29,7 @@ setuptools.setup(
         "Operating System :: OS Independent",
 	],
 	install_requires=install_requires,
+	entry_points={
+		"console_scripts": ["pybryt=pybryt.cli:cli"]
+	},
 )


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
new file mode 100644
index 0000000..21a4450
--- /dev/null
+++ b/tests/test_cli.py
@@ -0,0 +1,164 @@
+""""""
+
+import pytest
+import nbformat
+import tempfile
+
+from click.testing import CliRunner
+from unittest import mock
+
+from pybryt import StudentImplementation
+from pybryt.cli import click_cli
+from pybryt.utils import get_stem
+
+from .test_reference import generate_reference_notebook
+from .test_student import generate_impl
+
+
+def test_check():
+    """
+    """
+    runner = CliRunner()
+    with mock.patch("pybryt.cli.ReferenceImplementation") as mocked_ref, \
+            mock.patch("pybryt.cli.StudentImplementation") as mocked_stu, \
+            mock.patch("pybryt.cli.dill") as mocked_dill, \
+            mock.patch("pybryt.cli.json") as mocked_json, \
+            mock.patch("pybryt.cli.open") as mocked_open:
+        mocked_stu.return_value = mock.create_autospec(StudentImplementation)
+
+        with tempfile.NamedTemporaryFile(mode="w+", suffix=".ipynb") as ref_ntf, \
+                tempfile.NamedTemporaryFile(mode="w+", suffix=".ipynb") as stu_ntf:
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name])
+            assert result.exit_code == 0
+            mocked_ref.compile.assert_called_with(ref_ntf.name)
+            mocked_stu.assert_called_with(stu_ntf.name, output=None)
+            mocked_stu.return_value.check.return_value.dump.assert_called_with(get_stem(stu_ntf.name) + "_results.pkl")
+
+        with tempfile.NamedTemporaryFile(mode="w+", suffix=".pkl") as ref_ntf, \
+                tempfile.NamedTemporaryFile(mode="w+", suffix=".pkl") as stu_ntf:
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name, "-t", "json"])
+            assert result.exit_code == 0
+            mocked_ref.load.assert_called_with(ref_ntf.name)
+            mocked_stu.load.assert_called_with(stu_ntf.name)
+            mocked_stu.load.return_value.check.return_value.to_dict.assert_called_with()
+            mocked_json.dump.assert_called()
+
+            with mock.patch("pybryt.cli.generate_report") as mocked_report:
+                result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name, "-t", "report"])
+                assert result.exit_code == 0
+                mocked_ref.load.assert_called_with(ref_ntf.name)
+                mocked_stu.load.assert_called_with(stu_ntf.name)
+                mocked_report.assert_called_with(mocked_stu.load.return_value.check.return_value)
+
+            # check with list of references
+            mocked_ref.load.return_value = []
+            mocked_stu.load.return_value.check.return_value = []
+            
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name])
+            assert result.exit_code == 0
+            mocked_ref.load.assert_called_with(ref_ntf.name)
+            mocked_stu.load.assert_called_with(stu_ntf.name)
+            mocked_dill.dump.assert_called()
+
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name, "-t", "json"])
+            assert result.exit_code == 0
+            mocked_ref.load.assert_called_with(ref_ntf.name)
+            mocked_stu.load.assert_called_with(stu_ntf.name)
+            mocked_json.dump.assert_called()
+
+            # check errors
+            mocked_stu.load.side_effect = Exception()
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name])
+            assert result.exit_code == 1
+            assert isinstance(result.exception, RuntimeError)
+            assert result.exception.args[0] == f"Could not load the student implementation {stu_ntf.name}"
+
+            mocked_ref.load.side_effect = Exception()
+            result = runner.invoke(click_cli, ["check", ref_ntf.name, stu_ntf.name])
+            assert result.exit_code == 1
+            assert isinstance(result.exception, RuntimeError)
+            assert result.exception.args[0] == f"Could not load the reference implementation {ref_ntf.name}"
+
+
+def test_compile():
+    """
+    """
+    runner = CliRunner()
+    with tempfile.NamedTemporaryFile(mode="w+", suffix=".ipynb") as ntf:
+        nb = generate_reference_notebook()
+        nbformat.write(nb, ntf)
+        ntf.seek(0)
+
+        with mock.patch("pybryt.cli.ReferenceImplementation") as mocked_ref:
+            result = runner.invoke(click_cli, ["compile", ntf.name])
+            assert result.exit_code == 0
+            mocked_ref.compile.assert_called_with(ntf.name, name=None)
+            mocked_ref.compile.return_value.dump.assert_called_with(None)
+
+            result = runner.invoke(click_cli, ["compile", ntf.name, "-n", "foo"])
+            assert result.exit_code == 0
+            mocked_ref.compile.assert_called_with(ntf.name, name="foo")
+            mocked_ref.compile.return_value.dump.assert_called_with(None)
+
+            result = runner.invoke(click_cli, ["compile", ntf.name, "-d", "foo.pkl"])
+            assert result.exit_code == 0
+            mocked_ref.compile.assert_called_with(ntf.name, name=None)
+            mocked_ref.compile.return_value.dump.assert_called_with("foo.pkl")
+
+
+def test_execute():
+    """
+    """
+    runner = CliRunner()
+    num_subms = 10
+    ntfs = []
+    nb = generate_reference_notebook()
+    for _ in range(num_subms):
+        ntf = tempfile.NamedTemporaryFile(mode="w+", suffix=".ipynb")
+        nbformat.write(nb, ntf)
+        ntf.seek(0)
+        ntfs.append(ntf)
+
+    fns = tuple(ntf.name for ntf in ntfs)
+
+    with mock.patch("pybryt.cli.generate_student_impls") as mocked_generate:
+        mocked_generate.return_value.__iter__.return_value = (
+            iter([mock.create_autospec(StudentImplementation) for _ in range(num_subms)])
+        )
+
+        result = runner.invoke(click_cli, ["execute", *fns])
+        assert result.exit_code == 0
+        mocked_generate.assert_called_with(fns, parallel=False)
+
+        result = runner.invoke(click_cli, ["execute", fns[0]])
+        assert result.exit_code == 0
+        mocked_generate.assert_called_with((fns[0], ), parallel=False)
+
+        result = runner.invoke(click_cli, ["execute", "-p", *fns])
+        assert result.exit_code == 0
+        mocked_generate.assert_called_with(fns, parallel=True)
+
+        # check for error on nonexistance output dir
+        result = runner.invoke(click_cli, ["execute", *fns, "-o", "/some/fake/path"])
+        assert result.exit_code == 1
+        assert isinstance(result.exception, ValueError)
+        assert result.exception.args[0] == "Output directory /some/fake/path does not exist or is not a directory"
+
+    # check other errors
+    result = runner.invoke(click_cli, ["execute"])
+    assert result.exit_code == 1
+    assert isinstance(result.exception, ValueError)
+    assert result.exception.args[0] == "You must specify at least one notebook to execute"
+
+
+def test_cli_func(capsys):
+    """
+    Checks that the prog name is set correctly.
+    """
+    from pybryt.__main__ import cli
+    try:
+        cli(["--help"])
+    except SystemExit:
+        pass
+    captured = capsys.readouterr()
+    assert captured.out.startswith("Usage: pybryt [")
diff --git a/tests/test_reference.py b/tests/test_reference.py
index a210733..e3c5be2 100644
--- a/tests/test_reference.py
+++ b/tests/test_reference.py
@@ -136,7 +136,7 @@ def test_construction_errors():
 
         ntf.seek(0)
 
-        with pytest.raises(TypeError, match="Unpickled reference implementation has type <class 'int'>"):
+        with pytest.raises(TypeError, match="Unpickled object is not of type <class 'pybryt.reference.ReferenceImplementation'>"):
             ReferenceImplementation.load(ntf.name)
 
     # check that loading an empty reference implementation gives an error
diff --git a/tests/test_student.py b/tests/test_student.py
index 28a86ce..438507e 100644
--- a/tests/test_student.py
+++ b/tests/test_student.py
@@ -8,9 +8,12 @@ import pkg_resources
 
 from functools import lru_cache
 from textwrap import dedent
+from types import MethodType
 from unittest import mock
 
-from pybryt import check, ReferenceImplementation, ReferenceResult, StudentImplementation
+from pybryt import (
+    check, generate_student_impls, ReferenceImplementation, ReferenceResult, StudentImplementation
+)
 
 from .test_reference import generate_reference_notebook
 
@@ -175,3 +178,38 @@ def test_check_cm(capsys):
 
     with pytest.raises(TypeError, match="Invalid values in the reference list"):
         check([ref, "path", 1])
+
+
+def test_generate_student_impls():
+    """
+    """
+    num_notebooks = 6
+    nb, stu = generate_impl()
+    nbs = [nb] * num_notebooks
+
+    with mock.patch("pybryt.student.execute_notebook") as mocked_execute:
+        mocked_execute.return_value = (stu.steps, stu.values)
+        stus = generate_student_impls(nbs)
+
+    assert all(s == stu for s in stus)
+
+    with mock.patch("pybryt.student.Process") as mocked_process:
+
+        class MockedQueue:
+            def __init__(self, *args, **kwargs):
+                self.calls = 0
+
+            def empty(self, *args, **kwargs):
+                if self.calls >= num_notebooks:
+                    return True
+                self.calls += 1
+                return False
+
+            def get(self, *args, **kwargs):
+                return (nb, stu)
+
+        with mock.patch("pybryt.student.Queue") as mocked_queue:
+            mocked_queue.return_value = mock.MagicMock(wraps=MockedQueue())
+            stus = generate_student_impls(nbs, parallel=True)
+
+    assert all(s == stu for s in stus)