Repository: microsoft/pybryt (language: python)

Issue:
Relative tolerance
Tolerances allow student's values to deviate from the reference ones. Having the ability to define only the absolute tolerance poses several difficulties:

- It is required for the teacher to inspect each output value separately and decide what is an acceptable tolerance.
- Values such as lists, tuples, arrays, can contain values with different orders of magnitude and specifying a single absolute tolerance value is not sufficient.

Having the ability to define relative tolerance would help to resolve those difficulties and bring the comparison of values closer to `numpy.allclose` (https://numpy.org/doc/stable/reference/generated/numpy.allclose.html).



Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8d37c20..bef13f9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file, and this pr
 ## Unreleased
 
 * Changed tracing function to only increment step counter when tracing student code
+* Added relative tolerance to value annotations
 
 ## 0.0.5 - 2021-05-04
 
diff --git a/docs/html/.buildinfo b/docs/html/.buildinfo
index 534488d..5b9ec8a 100644
--- a/docs/html/.buildinfo
+++ b/docs/html/.buildinfo
@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: 7cf7d65884e2e782bc5912c20cae2801
+config: 8309b206771faeb99d80531df9570704
 tags: 645f666f9bcd5a90fca523b33c5a78b7
diff --git a/docs/html/_sources/annotations/invariants.rst.txt b/docs/html/_sources/annotations/invariants.rst.txt
index 5a9916b..65fde8f 100644
--- a/docs/html/_sources/annotations/invariants.rst.txt
+++ b/docs/html/_sources/annotations/invariants.rst.txt
@@ -3,4 +3,4 @@
 Invariants
 ==========
 
-This page is current under construction.
+This page is currently under construction.
diff --git a/docs/html/_sources/api_reference.rst.txt b/docs/html/_sources/api_reference.rst.txt
index 5a46670..c45022b 100644
--- a/docs/html/_sources/api_reference.rst.txt
+++ b/docs/html/_sources/api_reference.rst.txt
@@ -66,3 +66,11 @@ Student Implementations
 .. autoclass:: pybryt.StudentImplementation
    :members:
    :undoc-members:
+
+
+Execution
++++++++++
+
+.. automodule:: pybryt.execution
+   :members:
+   :undoc-members:
diff --git a/docs/html/_sources/index.rst.txt b/docs/html/_sources/index.rst.txt
index b10ab32..7df8344 100644
--- a/docs/html/_sources/index.rst.txt
+++ b/docs/html/_sources/index.rst.txt
@@ -17,7 +17,10 @@ PyBryt Documentation
    api_reference
    Changelog <https://github.com/microsoft/pybryt/tree/main/CHANGELOG.md>
 
-PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_.
+PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to 
+empower students and educators to learn about technology through fun, guided, hands-on content aimed 
+at specific learning goals. PyBryt is designed to work with existing autograding solutions and 
+workflows such as `Otter Grader`_, `OkPy`_, and `Autolab`_.
 
 .. image:: _static/images/pybryt_goals.png
 
diff --git a/docs/html/_static/basic.css b/docs/html/_static/basic.css
index 2e3cf32..be19270 100644
--- a/docs/html/_static/basic.css
+++ b/docs/html/_static/basic.css
@@ -4,7 +4,7 @@
  *
  * Sphinx stylesheet -- basic theme.
  *
- * :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
  * :license: BSD, see LICENSE for details.
  *
  */
@@ -693,7 +693,7 @@ pre {
     overflow-y: hidden;  /* fixes display issues on Chrome browsers */
 }
 
-pre, div[class|="highlight"] {
+pre, div[class*="highlight-"] {
     clear: both;
 }
 
@@ -704,7 +704,7 @@ span.pre {
     hyphens: none;
 }
 
-div[class^="highlight-"] {
+div[class*="highlight-"] {
     margin: 1em 0;
 }
 
@@ -764,6 +764,7 @@ div.code-block-caption code {
 }
 
 table.highlighttable td.linenos,
+span.linenos,
 div.doctest > div.highlight span.gp {  /* gp: Generic.Prompt */
     user-select: none;
 }
diff --git a/docs/html/_static/doctools.js b/docs/html/_static/doctools.js
index daccd20..61ac9d2 100644
--- a/docs/html/_static/doctools.js
+++ b/docs/html/_static/doctools.js
@@ -4,7 +4,7 @@
  *
  * Sphinx JavaScript utilities for all documentation.
  *
- * :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
  * :license: BSD, see LICENSE for details.
  *
  */
@@ -29,9 +29,14 @@ if (!window.console || !console.firebug) {
 
 /**
  * small helper function to urldecode strings
+ *
+ * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL
  */
 jQuery.urldecode = function(x) {
-  return decodeURIComponent(x).replace(/\+/g, ' ');
+  if (!x) {
+    return x
+  }
+  return decodeURIComponent(x.replace(/\+/g, ' '));
 };
 
 /**
@@ -285,9 +290,10 @@ var Documentation = {
   initOnKeyListeners: function() {
     $(document).keydown(function(event) {
       var activeElementType = document.activeElement.tagName;
-      // don't navigate when in search box or textarea
+      // don't navigate when in search box, textarea, dropdown or button
       if (activeElementType !== 'TEXTAREA' && activeElementType !== 'INPUT' && activeElementType !== 'SELECT'
-          && !event.altKey && !event.ctrlKey && !event.metaKey && !event.shiftKey) {
+          && activeElementType !== 'BUTTON' && !event.altKey && !event.ctrlKey && !event.metaKey
+          && !event.shiftKey) {
         switch (event.keyCode) {
           case 37: // left
             var prevHref = $('link[rel="prev"]').prop('href');
diff --git a/docs/html/_static/language_data.js b/docs/html/_static/language_data.js
index d2b4ee9..863704b 100644
--- a/docs/html/_static/language_data.js
+++ b/docs/html/_static/language_data.js
@@ -5,7 +5,7 @@
  * This script contains the language-specific data used by searchtools.js,
  * namely the list of stopwords, stemmer, scorer and splitter.
  *
- * :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
  * :license: BSD, see LICENSE for details.
  *
  */
@@ -13,7 +13,8 @@
 var stopwords = ["a","and","are","as","at","be","but","by","for","if","in","into","is","it","near","no","not","of","on","or","such","that","the","their","then","there","these","they","this","to","was","will","with"];
 
 
-/* Non-minified version JS is _stemmer.js if file is provided */ 
+/* Non-minified version is copied as a separate JS file, is available */
+
 /**
  * Porter Stemmer
  */
@@ -199,7 +200,6 @@ var Stemmer = function() {
 
 
 
-
 var splitChars = (function() {
     var result = {};
     var singles = [96, 180, 187, 191, 215, 247, 749, 885, 903, 907, 909, 930, 1014, 1648,
diff --git a/docs/html/_static/searchtools.js b/docs/html/_static/searchtools.js
index ab56499..1a90152 100644
--- a/docs/html/_static/searchtools.js
+++ b/docs/html/_static/searchtools.js
@@ -4,7 +4,7 @@
  *
  * Sphinx JavaScript utilities for the full-text search.
  *
- * :copyright: Copyright 2007-2020 by the Sphinx team, see AUTHORS.
+ * :copyright: Copyright 2007-2021 by the Sphinx team, see AUTHORS.
  * :license: BSD, see LICENSE for details.
  *
  */
@@ -59,10 +59,10 @@ var Search = {
   _pulse_status : -1,
 
   htmlToText : function(htmlString) {
-      var htmlElement = document.createElement('span');
-      htmlElement.innerHTML = htmlString;
-      $(htmlElement).find('.headerlink').remove();
-      docContent = $(htmlElement).find('[role=main]')[0];
+      var virtualDocument = document.implementation.createHTMLDocument('virtual');
+      var htmlElement = $(htmlString, virtualDocument);
+      htmlElement.find('.headerlink').remove();
+      docContent = htmlElement.find('[role=main]')[0];
       if(docContent === undefined) {
           console.warn("Content block not found. Sphinx search tries to obtain it " +
                        "via '[role=main]'. Could you check your theme or template.");
@@ -166,8 +166,7 @@ var Search = {
           objectterms.push(tmp[i].toLowerCase());
       }
 
-      if ($u.indexOf(stopwords, tmp[i].toLowerCase()) != -1 || tmp[i].match(/^\d+$/) ||
-          tmp[i] === "") {
+      if ($u.indexOf(stopwords, tmp[i].toLowerCase()) != -1 || tmp[i] === "") {
         // skip this "word"
         continue;
       }
@@ -249,7 +248,7 @@ var Search = {
       // results left, load the summary and display it
       if (results.length) {
         var item = results.pop();
-        var listItem = $('<li style="display:none"></li>');
+        var listItem = $('<li></li>');
         var requestUrl = "";
         var linkUrl = "";
         if (DOCUMENTATION_OPTIONS.BUILDER === 'dirhtml') {
@@ -274,9 +273,9 @@ var Search = {
         if (item[3]) {
           listItem.append($('<span> (' + item[3] + ')</span>'));
           Search.output.append(listItem);
-          listItem.slideDown(5, function() {
+          setTimeout(function() {
             displayNextItem();
-          });
+          }, 5);
         } else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) {
           $.ajax({url: requestUrl,
                   dataType: "text",
@@ -286,16 +285,16 @@ var Search = {
                       listItem.append(Search.makeSearchSummary(data, searchterms, hlterms));
                     }
                     Search.output.append(listItem);
-                    listItem.slideDown(5, function() {
+                    setTimeout(function() {
                       displayNextItem();
-                    });
+                    }, 5);
                   }});
         } else {
           // no source available, just display title
           Search.output.append(listItem);
-          listItem.slideDown(5, function() {
+          setTimeout(function() {
             displayNextItem();
-          });
+          }, 5);
         }
       }
       // search finished, update title and status message
@@ -380,6 +379,13 @@ var Search = {
     return results;
   },
 
+  /**
+   * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions
+   */
+  escapeRegExp : function(string) {
+    return string.replace(/[.*+\-?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
+  },
+
   /**
    * search for full-text terms in the index
    */
@@ -403,13 +409,14 @@ var Search = {
       ];
       // add support for partial matches
       if (word.length > 2) {
+        var word_regex = this.escapeRegExp(word);
         for (var w in terms) {
-          if (w.match(word) && !terms[word]) {
+          if (w.match(word_regex) && !terms[word]) {
             _o.push({files: terms[w], score: Scorer.partialTerm})
           }
         }
         for (var w in titleterms) {
-          if (w.match(word) && !titleterms[word]) {
+          if (w.match(word_regex) && !titleterms[word]) {
               _o.push({files: titleterms[w], score: Scorer.partialTitle})
           }
         }
diff --git a/docs/html/_static/underscore.js b/docs/html/_static/underscore.js
index 5b55f32..166240e 100644
--- a/docs/html/_static/underscore.js
+++ b/docs/html/_static/underscore.js
@@ -1,31 +1,6 @@
-// Underscore.js 1.3.1
-// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
-// Underscore is freely distributable under the MIT license.
-// Portions of Underscore are inspired or borrowed from Prototype,
-// Oliver Steele's Functional, and John Resig's Micro-Templating.
-// For all details and documentation:
-// http://documentcloud.github.com/underscore
-(function(){function q(a,c,d){if(a===c)return a!==0||1/a==1/c;if(a==null||c==null)return a===c;if(a._chain)a=a._wrapped;if(c._chain)c=c._wrapped;if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return false;switch(e){case "[object String]":return a==String(c);case "[object Number]":return a!=+a?c!=+c:a==0?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source==
-c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if(typeof a!="object"||typeof c!="object")return false;for(var f=d.length;f--;)if(d[f]==a)return true;d.push(a);var f=0,g=true;if(e=="[object Array]"){if(f=a.length,g=f==c.length)for(;f--;)if(!(g=f in a==f in c&&q(a[f],c[f],d)))break}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return false;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&q(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c,
-h)&&!f--)break;g=!f}}d.pop();return g}var r=this,G=r._,n={},k=Array.prototype,o=Object.prototype,i=k.slice,H=k.unshift,l=o.toString,I=o.hasOwnProperty,w=k.forEach,x=k.map,y=k.reduce,z=k.reduceRight,A=k.filter,B=k.every,C=k.some,p=k.indexOf,D=k.lastIndexOf,o=Array.isArray,J=Object.keys,s=Function.prototype.bind,b=function(a){return new m(a)};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports)exports=module.exports=b;exports._=b}else r._=b;b.VERSION="1.3.1";var j=b.each=
-b.forEach=function(a,c,d){if(a!=null)if(w&&a.forEach===w)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e<f;e++){if(e in a&&c.call(d,a[e],e,a)===n)break}else for(e in a)if(b.has(a,e)&&c.call(d,a[e],e,a)===n)break};b.map=b.collect=function(a,c,b){var e=[];if(a==null)return e;if(x&&a.map===x)return a.map(c,b);j(a,function(a,g,h){e[e.length]=c.call(b,a,g,h)});if(a.length===+a.length)e.length=a.length;return e};b.reduce=b.foldl=b.inject=function(a,c,d,e){var f=arguments.length>2;a==
-null&&(a=[]);if(y&&a.reduce===y)return e&&(c=b.bind(c,e)),f?a.reduce(c,d):a.reduce(c);j(a,function(a,b,i){f?d=c.call(e,d,a,b,i):(d=a,f=true)});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(z&&a.reduceRight===z)return e&&(c=b.bind(c,e)),f?a.reduceRight(c,d):a.reduceRight(c);var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect=
-function(a,c,b){var e;E(a,function(a,g,h){if(c.call(b,a,g,h))return e=a,true});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(A&&a.filter===A)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(B&&a.every===B)return a.every(c,b);j(a,function(a,g,h){if(!(e=
-e&&c.call(b,a,g,h)))return n});return e};var E=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(C&&a.some===C)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return n});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;return p&&a.indexOf===p?a.indexOf(c)!=-1:b=E(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck=
-function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a))return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a))return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b<e.computed&&(e={value:a,computed:b})});
-return e.value};b.shuffle=function(a){var b=[],d;j(a,function(a,f){f==0?b[0]=a:(d=Math.floor(Math.random()*(f+1)),b[f]=b[d],b[d]=a)});return b};b.sortBy=function(a,c,d){return b.pluck(b.map(a,function(a,b,g){return{value:a,criteria:c.call(d,a,b,g)}}).sort(function(a,b){var c=a.criteria,d=b.criteria;return c<d?-1:c>d?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a,
-c,d){d||(d=b.identity);for(var e=0,f=a.length;e<f;){var g=e+f>>1;d(a[g])<d(c)?e=g+1:f=g}return e};b.toArray=function(a){return!a?[]:a.toArray?a.toArray():b.isArray(a)?i.call(a):b.isArguments(a)?i.call(a):b.values(a)};b.size=function(a){return b.toArray(a).length};b.first=b.head=function(a,b,d){return b!=null&&!d?i.call(a,0,b):a[0]};b.initial=function(a,b,d){return i.call(a,0,a.length-(b==null||d?1:b))};b.last=function(a,b,d){return b!=null&&!d?i.call(a,Math.max(a.length-b,0)):a[a.length-1]};b.rest=
-b.tail=function(a,b,d){return i.call(a,b==null||d?1:b)};b.compact=function(a){return b.filter(a,function(a){return!!a})};b.flatten=function(a,c){return b.reduce(a,function(a,e){if(b.isArray(e))return a.concat(c?e:b.flatten(e));a[a.length]=e;return a},[])};b.without=function(a){return b.difference(a,i.call(arguments,1))};b.uniq=b.unique=function(a,c,d){var d=d?b.map(a,d):a,e=[];b.reduce(d,function(d,g,h){if(0==h||(c===true?b.last(d)!=g:!b.include(d,g)))d[d.length]=g,e[e.length]=a[h];return d},[]);
-return e};b.union=function(){return b.uniq(b.flatten(arguments,true))};b.intersection=b.intersect=function(a){var c=i.call(arguments,1);return b.filter(b.uniq(a),function(a){return b.every(c,function(c){return b.indexOf(c,a)>=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1));return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e<c;e++)d[e]=b.pluck(a,""+e);return d};b.indexOf=function(a,c,
-d){if(a==null)return-1;var e;if(d)return d=b.sortedIndex(a,c),a[d]===c?d:-1;if(p&&a.indexOf===p)return a.indexOf(c);for(d=0,e=a.length;d<e;d++)if(d in a&&a[d]===c)return d;return-1};b.lastIndexOf=function(a,b){if(a==null)return-1;if(D&&a.lastIndexOf===D)return a.lastIndexOf(b);for(var d=a.length;d--;)if(d in a&&a[d]===b)return d;return-1};b.range=function(a,b,d){arguments.length<=1&&(b=a||0,a=0);for(var d=arguments[2]||1,e=Math.max(Math.ceil((b-a)/d),0),f=0,g=Array(e);f<e;)g[f++]=a,a+=d;return g};
-var F=function(){};b.bind=function(a,c){var d,e;if(a.bind===s&&s)return s.apply(a,i.call(arguments,1));if(!b.isFunction(a))throw new TypeError;e=i.call(arguments,2);return d=function(){if(!(this instanceof d))return a.apply(c,e.concat(i.call(arguments)));F.prototype=a.prototype;var b=new F,g=a.apply(b,e.concat(i.call(arguments)));return Object(g)===g?g:b}};b.bindAll=function(a){var c=i.call(arguments,1);c.length==0&&(c=b.functions(a));j(c,function(c){a[c]=b.bind(a[c],a)});return a};b.memoize=function(a,
-c){var d={};c||(c=b.identity);return function(){var e=c.apply(this,arguments);return b.has(d,e)?d[e]:d[e]=a.apply(this,arguments)}};b.delay=function(a,b){var d=i.call(arguments,2);return setTimeout(function(){return a.apply(a,d)},b)};b.defer=function(a){return b.delay.apply(b,[a,1].concat(i.call(arguments,1)))};b.throttle=function(a,c){var d,e,f,g,h,i=b.debounce(function(){h=g=false},c);return function(){d=this;e=arguments;var b;f||(f=setTimeout(function(){f=null;h&&a.apply(d,e);i()},c));g?h=true:
-a.apply(d,e);i();g=true}};b.debounce=function(a,b){var d;return function(){var e=this,f=arguments;clearTimeout(d);d=setTimeout(function(){d=null;a.apply(e,f)},b)}};b.once=function(a){var b=false,d;return function(){if(b)return d;b=true;return d=a.apply(this,arguments)}};b.wrap=function(a,b){return function(){var d=[a].concat(i.call(arguments,0));return b.apply(this,d)}};b.compose=function(){var a=arguments;return function(){for(var b=arguments,d=a.length-1;d>=0;d--)b=[a[d].apply(this,b)];return b[0]}};
-b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=J||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.defaults=function(a){j(i.call(arguments,
-1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return q(a,b,[])};b.isEmpty=function(a){if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=o||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)};
-b.isArguments=function(a){return l.call(a)=="[object Arguments]"};if(!b.isArguments(arguments))b.isArguments=function(a){return!(!a||!b.has(a,"callee"))};b.isFunction=function(a){return l.call(a)=="[object Function]"};b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"};
-b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,b){return I.call(a,b)};b.noConflict=function(){r._=G;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e<a;e++)b.call(d,e)};b.escape=function(a){return(""+a).replace(/&/g,"&amp;").replace(/</g,"&lt;").replace(/>/g,"&gt;").replace(/"/g,"&quot;").replace(/'/g,"&#x27;").replace(/\//g,"&#x2F;")};b.mixin=function(a){j(b.functions(a),
-function(c){K(c,b[c]=a[c])})};var L=0;b.uniqueId=function(a){var b=L++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var t=/.^/,u=function(a){return a.replace(/\\\\/g,"\\").replace(/\\'/g,"'")};b.template=function(a,c){var d=b.templateSettings,d="var __p=[],print=function(){__p.push.apply(__p,arguments);};with(obj||{}){__p.push('"+a.replace(/\\/g,"\\\\").replace(/'/g,"\\'").replace(d.escape||t,function(a,b){return"',_.escape("+
-u(b)+"),'"}).replace(d.interpolate||t,function(a,b){return"',"+u(b)+",'"}).replace(d.evaluate||t,function(a,b){return"');"+u(b).replace(/[\r\n\t]/g," ")+";__p.push('"}).replace(/\r/g,"\\r").replace(/\n/g,"\\n").replace(/\t/g,"\\t")+"');}return __p.join('');",e=new Function("obj","_",d);return c?e(c,b):function(a){return e.call(this,a,b)}};b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var v=function(a,c){return c?b(a).chain():a},K=function(a,c){m.prototype[a]=
-function(){var a=i.call(arguments);H.call(a,this._wrapped);return v(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return v(d,this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return v(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain=
-true;return this};m.prototype.value=function(){return this._wrapped}}).call(this);
+!function(n,r){"object"==typeof exports&&"undefined"!=typeof module?module.exports=r():"function"==typeof define&&define.amd?define("underscore",r):(n=n||self,function(){var t=n._,e=n._=r();e.noConflict=function(){return n._=t,e}}())}(this,(function(){
+//     Underscore.js 1.12.0
+//     https://underscorejs.org
+//     (c) 2009-2020 Jeremy Ashkenas, DocumentCloud and Investigative Reporters & Editors
+//     Underscore may be freely distributed under the MIT license.
+var n="1.12.0",r="object"==typeof self&&self.self===self&&self||"object"==typeof global&&global.global===global&&global||Function("return this")()||{},t=Array.prototype,e=Object.prototype,u="undefined"!=typeof Symbol?Symbol.prototype:null,o=t.push,i=t.slice,a=e.toString,f=e.hasOwnProperty,c="undefined"!=typeof ArrayBuffer,l="undefined"!=typeof DataView,s=Array.isArray,p=Object.keys,v=Object.create,h=c&&ArrayBuffer.isView,y=isNaN,g=isFinite,d=!{toString:null}.propertyIsEnumerable("toString"),b=["valueOf","isPrototypeOf","toString","propertyIsEnumerable","hasOwnProperty","toLocaleString"],m=Math.pow(2,53)-1;function j(n,r){return r=null==r?n.length-1:+r,function(){for(var t=Math.max(arguments.length-r,0),e=Array(t),u=0;u<t;u++)e[u]=arguments[u+r];switch(r){case 0:return n.call(this,e);case 1:return n.call(this,arguments[0],e);case 2:return n.call(this,arguments[0],arguments[1],e)}var o=Array(r+1);for(u=0;u<r;u++)o[u]=arguments[u];return o[r]=e,n.apply(this,o)}}function _(n){var r=typeof n;return"function"===r||"object"===r&&!!n}function w(n){return void 0===n}function A(n){return!0===n||!1===n||"[object Boolean]"===a.call(n)}function x(n){var r="[object "+n+"]";return function(n){return a.call(n)===r}}var S=x("String"),O=x("Number"),M=x("Date"),E=x("RegExp"),B=x("Error"),N=x("Symbol"),I=x("ArrayBuffer"),k=x("Function"),T=r.document&&r.document.childNodes;"function"!=typeof/./&&"object"!=typeof Int8Array&&"function"!=typeof T&&(k=function(n){return"function"==typeof n||!1});var D=k,R=x("Object"),F=l&&R(new DataView(new ArrayBuffer(8))),V="undefined"!=typeof Map&&R(new Map),P=x("DataView");var q=F?function(n){return null!=n&&D(n.getInt8)&&I(n.buffer)}:P,U=s||x("Array");function W(n,r){return null!=n&&f.call(n,r)}var z=x("Arguments");!function(){z(arguments)||(z=function(n){return W(n,"callee")})}();var L=z;function C(n){return O(n)&&y(n)}function K(n){return function(){return n}}function J(n){return function(r){var t=n(r);return"number"==typeof t&&t>=0&&t<=m}}function $(n){return function(r){return null==r?void 0:r[n]}}var G=$("byteLength"),H=J(G),Q=/\[object ((I|Ui)nt(8|16|32)|Float(32|64)|Uint8Clamped|Big(I|Ui)nt64)Array\]/;var X=c?function(n){return h?h(n)&&!q(n):H(n)&&Q.test(a.call(n))}:K(!1),Y=$("length");function Z(n,r){r=function(n){for(var r={},t=n.length,e=0;e<t;++e)r[n[e]]=!0;return{contains:function(n){return r[n]},push:function(t){return r[t]=!0,n.push(t)}}}(r);var t=b.length,u=n.constructor,o=D(u)&&u.prototype||e,i="constructor";for(W(n,i)&&!r.contains(i)&&r.push(i);t--;)(i=b[t])in n&&n[i]!==o[i]&&!r.contains(i)&&r.push(i)}function nn(n){if(!_(n))return[];if(p)return p(n);var r=[];for(var t in n)W(n,t)&&r.push(t);return d&&Z(n,r),r}function rn(n,r){var t=nn(r),e=t.length;if(null==n)return!e;for(var u=Object(n),o=0;o<e;o++){var i=t[o];if(r[i]!==u[i]||!(i in u))return!1}return!0}function tn(n){return n instanceof tn?n:this instanceof tn?void(this._wrapped=n):new tn(n)}function en(n){return new Uint8Array(n.buffer||n,n.byteOffset||0,G(n))}tn.VERSION=n,tn.prototype.value=function(){return this._wrapped},tn.prototype.valueOf=tn.prototype.toJSON=tn.prototype.value,tn.prototype.toString=function(){return String(this._wrapped)};var un="[object DataView]";function on(n,r,t,e){if(n===r)return 0!==n||1/n==1/r;if(null==n||null==r)return!1;if(n!=n)return r!=r;var o=typeof n;return("function"===o||"object"===o||"object"==typeof r)&&function n(r,t,e,o){r instanceof tn&&(r=r._wrapped);t instanceof tn&&(t=t._wrapped);var i=a.call(r);if(i!==a.call(t))return!1;if(F&&"[object Object]"==i&&q(r)){if(!q(t))return!1;i=un}switch(i){case"[object RegExp]":case"[object String]":return""+r==""+t;case"[object Number]":return+r!=+r?+t!=+t:0==+r?1/+r==1/t:+r==+t;case"[object Date]":case"[object Boolean]":return+r==+t;case"[object Symbol]":return u.valueOf.call(r)===u.valueOf.call(t);case"[object ArrayBuffer]":case un:return n(en(r),en(t),e,o)}var f="[object Array]"===i;if(!f&&X(r)){if(G(r)!==G(t))return!1;if(r.buffer===t.buffer&&r.byteOffset===t.byteOffset)return!0;f=!0}if(!f){if("object"!=typeof r||"object"!=typeof t)return!1;var c=r.constructor,l=t.constructor;if(c!==l&&!(D(c)&&c instanceof c&&D(l)&&l instanceof l)&&"constructor"in r&&"constructor"in t)return!1}o=o||[];var s=(e=e||[]).length;for(;s--;)if(e[s]===r)return o[s]===t;if(e.push(r),o.push(t),f){if((s=r.length)!==t.length)return!1;for(;s--;)if(!on(r[s],t[s],e,o))return!1}else{var p,v=nn(r);if(s=v.length,nn(t).length!==s)return!1;for(;s--;)if(p=v[s],!W(t,p)||!on(r[p],t[p],e,o))return!1}return e.pop(),o.pop(),!0}(n,r,t,e)}function an(n){if(!_(n))return[];var r=[];for(var t in n)r.push(t);return d&&Z(n,r),r}function fn(n){var r=Y(n);return function(t){if(null==t)return!1;var e=an(t);if(Y(e))return!1;for(var u=0;u<r;u++)if(!D(t[n[u]]))return!1;return n!==hn||!D(t[cn])}}var cn="forEach",ln="has",sn=["clear","delete"],pn=["get",ln,"set"],vn=sn.concat(cn,pn),hn=sn.concat(pn),yn=["add"].concat(sn,cn,ln),gn=V?fn(vn):x("Map"),dn=V?fn(hn):x("WeakMap"),bn=V?fn(yn):x("Set"),mn=x("WeakSet");function jn(n){for(var r=nn(n),t=r.length,e=Array(t),u=0;u<t;u++)e[u]=n[r[u]];return e}function _n(n){for(var r={},t=nn(n),e=0,u=t.length;e<u;e++)r[n[t[e]]]=t[e];return r}function wn(n){var r=[];for(var t in n)D(n[t])&&r.push(t);return r.sort()}function An(n,r){return function(t){var e=arguments.length;if(r&&(t=Object(t)),e<2||null==t)return t;for(var u=1;u<e;u++)for(var o=arguments[u],i=n(o),a=i.length,f=0;f<a;f++){var c=i[f];r&&void 0!==t[c]||(t[c]=o[c])}return t}}var xn=An(an),Sn=An(nn),On=An(an,!0);function Mn(n){if(!_(n))return{};if(v)return v(n);var r=function(){};r.prototype=n;var t=new r;return r.prototype=null,t}function En(n){return _(n)?U(n)?n.slice():xn({},n):n}function Bn(n){return U(n)?n:[n]}function Nn(n){return tn.toPath(n)}function In(n,r){for(var t=r.length,e=0;e<t;e++){if(null==n)return;n=n[r[e]]}return t?n:void 0}function kn(n,r,t){var e=In(n,Nn(r));return w(e)?t:e}function Tn(n){return n}function Dn(n){return n=Sn({},n),function(r){return rn(r,n)}}function Rn(n){return n=Nn(n),function(r){return In(r,n)}}function Fn(n,r,t){if(void 0===r)return n;switch(null==t?3:t){case 1:return function(t){return n.call(r,t)};case 3:return function(t,e,u){return n.call(r,t,e,u)};case 4:return function(t,e,u,o){return n.call(r,t,e,u,o)}}return function(){return n.apply(r,arguments)}}function Vn(n,r,t){return null==n?Tn:D(n)?Fn(n,r,t):_(n)&&!U(n)?Dn(n):Rn(n)}function Pn(n,r){return Vn(n,r,1/0)}function qn(n,r,t){return tn.iteratee!==Pn?tn.iteratee(n,r):Vn(n,r,t)}function Un(){}function Wn(n,r){return null==r&&(r=n,n=0),n+Math.floor(Math.random()*(r-n+1))}tn.toPath=Bn,tn.iteratee=Pn;var zn=Date.now||function(){return(new Date).getTime()};function Ln(n){var r=function(r){return n[r]},t="(?:"+nn(n).join("|")+")",e=RegExp(t),u=RegExp(t,"g");return function(n){return n=null==n?"":""+n,e.test(n)?n.replace(u,r):n}}var Cn={"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;","'":"&#x27;","`":"&#x60;"},Kn=Ln(Cn),Jn=Ln(_n(Cn)),$n=tn.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g},Gn=/(.)^/,Hn={"'":"'","\\":"\\","\r":"r","\n":"n","\u2028":"u2028","\u2029":"u2029"},Qn=/\\|'|\r|\n|\u2028|\u2029/g;function Xn(n){return"\\"+Hn[n]}var Yn=0;function Zn(n,r,t,e,u){if(!(e instanceof r))return n.apply(t,u);var o=Mn(n.prototype),i=n.apply(o,u);return _(i)?i:o}var nr=j((function(n,r){var t=nr.placeholder,e=function(){for(var u=0,o=r.length,i=Array(o),a=0;a<o;a++)i[a]=r[a]===t?arguments[u++]:r[a];for(;u<arguments.length;)i.push(arguments[u++]);return Zn(n,e,this,this,i)};return e}));nr.placeholder=tn;var rr=j((function(n,r,t){if(!D(n))throw new TypeError("Bind must be called on a function");var e=j((function(u){return Zn(n,e,r,this,t.concat(u))}));return e})),tr=J(Y);function er(n,r,t,e){if(e=e||[],r||0===r){if(r<=0)return e.concat(n)}else r=1/0;for(var u=e.length,o=0,i=Y(n);o<i;o++){var a=n[o];if(tr(a)&&(U(a)||L(a)))if(r>1)er(a,r-1,t,e),u=e.length;else for(var f=0,c=a.length;f<c;)e[u++]=a[f++];else t||(e[u++]=a)}return e}var ur=j((function(n,r){var t=(r=er(r,!1,!1)).length;if(t<1)throw new Error("bindAll must be passed function names");for(;t--;){var e=r[t];n[e]=rr(n[e],n)}return n}));var or=j((function(n,r,t){return setTimeout((function(){return n.apply(null,t)}),r)})),ir=nr(or,tn,1);function ar(n){return function(){return!n.apply(this,arguments)}}function fr(n,r){var t;return function(){return--n>0&&(t=r.apply(this,arguments)),n<=1&&(r=null),t}}var cr=nr(fr,2);function lr(n,r,t){r=qn(r,t);for(var e,u=nn(n),o=0,i=u.length;o<i;o++)if(r(n[e=u[o]],e,n))return e}function sr(n){return function(r,t,e){t=qn(t,e);for(var u=Y(r),o=n>0?0:u-1;o>=0&&o<u;o+=n)if(t(r[o],o,r))return o;return-1}}var pr=sr(1),vr=sr(-1);function hr(n,r,t,e){for(var u=(t=qn(t,e,1))(r),o=0,i=Y(n);o<i;){var a=Math.floor((o+i)/2);t(n[a])<u?o=a+1:i=a}return o}function yr(n,r,t){return function(e,u,o){var a=0,f=Y(e);if("number"==typeof o)n>0?a=o>=0?o:Math.max(o+f,a):f=o>=0?Math.min(o+1,f):o+f+1;else if(t&&o&&f)return e[o=t(e,u)]===u?o:-1;if(u!=u)return(o=r(i.call(e,a,f),C))>=0?o+a:-1;for(o=n>0?a:f-1;o>=0&&o<f;o+=n)if(e[o]===u)return o;return-1}}var gr=yr(1,pr,hr),dr=yr(-1,vr);function br(n,r,t){var e=(tr(n)?pr:lr)(n,r,t);if(void 0!==e&&-1!==e)return n[e]}function mr(n,r,t){var e,u;if(r=Fn(r,t),tr(n))for(e=0,u=n.length;e<u;e++)r(n[e],e,n);else{var o=nn(n);for(e=0,u=o.length;e<u;e++)r(n[o[e]],o[e],n)}return n}function jr(n,r,t){r=qn(r,t);for(var e=!tr(n)&&nn(n),u=(e||n).length,o=Array(u),i=0;i<u;i++){var a=e?e[i]:i;o[i]=r(n[a],a,n)}return o}function _r(n){var r=function(r,t,e,u){var o=!tr(r)&&nn(r),i=(o||r).length,a=n>0?0:i-1;for(u||(e=r[o?o[a]:a],a+=n);a>=0&&a<i;a+=n){var f=o?o[a]:a;e=t(e,r[f],f,r)}return e};return function(n,t,e,u){var o=arguments.length>=3;return r(n,Fn(t,u,4),e,o)}}var wr=_r(1),Ar=_r(-1);function xr(n,r,t){var e=[];return r=qn(r,t),mr(n,(function(n,t,u){r(n,t,u)&&e.push(n)})),e}function Sr(n,r,t){r=qn(r,t);for(var e=!tr(n)&&nn(n),u=(e||n).length,o=0;o<u;o++){var i=e?e[o]:o;if(!r(n[i],i,n))return!1}return!0}function Or(n,r,t){r=qn(r,t);for(var e=!tr(n)&&nn(n),u=(e||n).length,o=0;o<u;o++){var i=e?e[o]:o;if(r(n[i],i,n))return!0}return!1}function Mr(n,r,t,e){return tr(n)||(n=jn(n)),("number"!=typeof t||e)&&(t=0),gr(n,r,t)>=0}var Er=j((function(n,r,t){var e,u;return D(r)?u=r:(r=Nn(r),e=r.slice(0,-1),r=r[r.length-1]),jr(n,(function(n){var o=u;if(!o){if(e&&e.length&&(n=In(n,e)),null==n)return;o=n[r]}return null==o?o:o.apply(n,t)}))}));function Br(n,r){return jr(n,Rn(r))}function Nr(n,r,t){var e,u,o=-1/0,i=-1/0;if(null==r||"number"==typeof r&&"object"!=typeof n[0]&&null!=n)for(var a=0,f=(n=tr(n)?n:jn(n)).length;a<f;a++)null!=(e=n[a])&&e>o&&(o=e);else r=qn(r,t),mr(n,(function(n,t,e){((u=r(n,t,e))>i||u===-1/0&&o===-1/0)&&(o=n,i=u)}));return o}function Ir(n,r,t){if(null==r||t)return tr(n)||(n=jn(n)),n[Wn(n.length-1)];var e=tr(n)?En(n):jn(n),u=Y(e);r=Math.max(Math.min(r,u),0);for(var o=u-1,i=0;i<r;i++){var a=Wn(i,o),f=e[i];e[i]=e[a],e[a]=f}return e.slice(0,r)}function kr(n,r){return function(t,e,u){var o=r?[[],[]]:{};return e=qn(e,u),mr(t,(function(r,u){var i=e(r,u,t);n(o,r,i)})),o}}var Tr=kr((function(n,r,t){W(n,t)?n[t].push(r):n[t]=[r]})),Dr=kr((function(n,r,t){n[t]=r})),Rr=kr((function(n,r,t){W(n,t)?n[t]++:n[t]=1})),Fr=kr((function(n,r,t){n[t?0:1].push(r)}),!0),Vr=/[^\ud800-\udfff]|[\ud800-\udbff][\udc00-\udfff]|[\ud800-\udfff]/g;function Pr(n,r,t){return r in t}var qr=j((function(n,r){var t={},e=r[0];if(null==n)return t;D(e)?(r.length>1&&(e=Fn(e,r[1])),r=an(n)):(e=Pr,r=er(r,!1,!1),n=Object(n));for(var u=0,o=r.length;u<o;u++){var i=r[u],a=n[i];e(a,i,n)&&(t[i]=a)}return t})),Ur=j((function(n,r){var t,e=r[0];return D(e)?(e=ar(e),r.length>1&&(t=r[1])):(r=jr(er(r,!1,!1),String),e=function(n,t){return!Mr(r,t)}),qr(n,e,t)}));function Wr(n,r,t){return i.call(n,0,Math.max(0,n.length-(null==r||t?1:r)))}function zr(n,r,t){return null==n||n.length<1?null==r||t?void 0:[]:null==r||t?n[0]:Wr(n,n.length-r)}function Lr(n,r,t){return i.call(n,null==r||t?1:r)}var Cr=j((function(n,r){return r=er(r,!0,!0),xr(n,(function(n){return!Mr(r,n)}))})),Kr=j((function(n,r){return Cr(n,r)}));function Jr(n,r,t,e){A(r)||(e=t,t=r,r=!1),null!=t&&(t=qn(t,e));for(var u=[],o=[],i=0,a=Y(n);i<a;i++){var f=n[i],c=t?t(f,i,n):f;r&&!t?(i&&o===c||u.push(f),o=c):t?Mr(o,c)||(o.push(c),u.push(f)):Mr(u,f)||u.push(f)}return u}var $r=j((function(n){return Jr(er(n,!0,!0))}));function Gr(n){for(var r=n&&Nr(n,Y).length||0,t=Array(r),e=0;e<r;e++)t[e]=Br(n,e);return t}var Hr=j(Gr);function Qr(n,r){return n._chain?tn(r).chain():r}function Xr(n){return mr(wn(n),(function(r){var t=tn[r]=n[r];tn.prototype[r]=function(){var n=[this._wrapped];return o.apply(n,arguments),Qr(this,t.apply(tn,n))}})),tn}mr(["pop","push","reverse","shift","sort","splice","unshift"],(function(n){var r=t[n];tn.prototype[n]=function(){var t=this._wrapped;return null!=t&&(r.apply(t,arguments),"shift"!==n&&"splice"!==n||0!==t.length||delete t[0]),Qr(this,t)}})),mr(["concat","join","slice"],(function(n){var r=t[n];tn.prototype[n]=function(){var n=this._wrapped;return null!=n&&(n=r.apply(n,arguments)),Qr(this,n)}}));var Yr=Xr({__proto__:null,VERSION:n,restArguments:j,isObject:_,isNull:function(n){return null===n},isUndefined:w,isBoolean:A,isElement:function(n){return!(!n||1!==n.nodeType)},isString:S,isNumber:O,isDate:M,isRegExp:E,isError:B,isSymbol:N,isArrayBuffer:I,isDataView:q,isArray:U,isFunction:D,isArguments:L,isFinite:function(n){return!N(n)&&g(n)&&!isNaN(parseFloat(n))},isNaN:C,isTypedArray:X,isEmpty:function(n){if(null==n)return!0;var r=Y(n);return"number"==typeof r&&(U(n)||S(n)||L(n))?0===r:0===Y(nn(n))},isMatch:rn,isEqual:function(n,r){return on(n,r)},isMap:gn,isWeakMap:dn,isSet:bn,isWeakSet:mn,keys:nn,allKeys:an,values:jn,pairs:function(n){for(var r=nn(n),t=r.length,e=Array(t),u=0;u<t;u++)e[u]=[r[u],n[r[u]]];return e},invert:_n,functions:wn,methods:wn,extend:xn,extendOwn:Sn,assign:Sn,defaults:On,create:function(n,r){var t=Mn(n);return r&&Sn(t,r),t},clone:En,tap:function(n,r){return r(n),n},get:kn,has:function(n,r){for(var t=(r=Nn(r)).length,e=0;e<t;e++){var u=r[e];if(!W(n,u))return!1;n=n[u]}return!!t},mapObject:function(n,r,t){r=qn(r,t);for(var e=nn(n),u=e.length,o={},i=0;i<u;i++){var a=e[i];o[a]=r(n[a],a,n)}return o},identity:Tn,constant:K,noop:Un,toPath:Bn,property:Rn,propertyOf:function(n){return null==n?Un:function(r){return kn(n,r)}},matcher:Dn,matches:Dn,times:function(n,r,t){var e=Array(Math.max(0,n));r=Fn(r,t,1);for(var u=0;u<n;u++)e[u]=r(u);return e},random:Wn,now:zn,escape:Kn,unescape:Jn,templateSettings:$n,template:function(n,r,t){!r&&t&&(r=t),r=On({},r,tn.templateSettings);var e,u=RegExp([(r.escape||Gn).source,(r.interpolate||Gn).source,(r.evaluate||Gn).source].join("|")+"|$","g"),o=0,i="__p+='";n.replace(u,(function(r,t,e,u,a){return i+=n.slice(o,a).replace(Qn,Xn),o=a+r.length,t?i+="'+\n((__t=("+t+"))==null?'':_.escape(__t))+\n'":e?i+="'+\n((__t=("+e+"))==null?'':__t)+\n'":u&&(i+="';\n"+u+"\n__p+='"),r})),i+="';\n",r.variable||(i="with(obj||{}){\n"+i+"}\n"),i="var __t,__p='',__j=Array.prototype.join,"+"print=function(){__p+=__j.call(arguments,'');};\n"+i+"return __p;\n";try{e=new Function(r.variable||"obj","_",i)}catch(n){throw n.source=i,n}var a=function(n){return e.call(this,n,tn)},f=r.variable||"obj";return a.source="function("+f+"){\n"+i+"}",a},result:function(n,r,t){var e=(r=Nn(r)).length;if(!e)return D(t)?t.call(n):t;for(var u=0;u<e;u++){var o=null==n?void 0:n[r[u]];void 0===o&&(o=t,u=e),n=D(o)?o.call(n):o}return n},uniqueId:function(n){var r=++Yn+"";return n?n+r:r},chain:function(n){var r=tn(n);return r._chain=!0,r},iteratee:Pn,partial:nr,bind:rr,bindAll:ur,memoize:function(n,r){var t=function(e){var u=t.cache,o=""+(r?r.apply(this,arguments):e);return W(u,o)||(u[o]=n.apply(this,arguments)),u[o]};return t.cache={},t},delay:or,defer:ir,throttle:function(n,r,t){var e,u,o,i,a=0;t||(t={});var f=function(){a=!1===t.leading?0:zn(),e=null,i=n.apply(u,o),e||(u=o=null)},c=function(){var c=zn();a||!1!==t.leading||(a=c);var l=r-(c-a);return u=this,o=arguments,l<=0||l>r?(e&&(clearTimeout(e),e=null),a=c,i=n.apply(u,o),e||(u=o=null)):e||!1===t.trailing||(e=setTimeout(f,l)),i};return c.cancel=function(){clearTimeout(e),a=0,e=u=o=null},c},debounce:function(n,r,t){var e,u,o=function(r,t){e=null,t&&(u=n.apply(r,t))},i=j((function(i){if(e&&clearTimeout(e),t){var a=!e;e=setTimeout(o,r),a&&(u=n.apply(this,i))}else e=or(o,r,this,i);return u}));return i.cancel=function(){clearTimeout(e),e=null},i},wrap:function(n,r){return nr(r,n)},negate:ar,compose:function(){var n=arguments,r=n.length-1;return function(){for(var t=r,e=n[r].apply(this,arguments);t--;)e=n[t].call(this,e);return e}},after:function(n,r){return function(){if(--n<1)return r.apply(this,arguments)}},before:fr,once:cr,findKey:lr,findIndex:pr,findLastIndex:vr,sortedIndex:hr,indexOf:gr,lastIndexOf:dr,find:br,detect:br,findWhere:function(n,r){return br(n,Dn(r))},each:mr,forEach:mr,map:jr,collect:jr,reduce:wr,foldl:wr,inject:wr,reduceRight:Ar,foldr:Ar,filter:xr,select:xr,reject:function(n,r,t){return xr(n,ar(qn(r)),t)},every:Sr,all:Sr,some:Or,any:Or,contains:Mr,includes:Mr,include:Mr,invoke:Er,pluck:Br,where:function(n,r){return xr(n,Dn(r))},max:Nr,min:function(n,r,t){var e,u,o=1/0,i=1/0;if(null==r||"number"==typeof r&&"object"!=typeof n[0]&&null!=n)for(var a=0,f=(n=tr(n)?n:jn(n)).length;a<f;a++)null!=(e=n[a])&&e<o&&(o=e);else r=qn(r,t),mr(n,(function(n,t,e){((u=r(n,t,e))<i||u===1/0&&o===1/0)&&(o=n,i=u)}));return o},shuffle:function(n){return Ir(n,1/0)},sample:Ir,sortBy:function(n,r,t){var e=0;return r=qn(r,t),Br(jr(n,(function(n,t,u){return{value:n,index:e++,criteria:r(n,t,u)}})).sort((function(n,r){var t=n.criteria,e=r.criteria;if(t!==e){if(t>e||void 0===t)return 1;if(t<e||void 0===e)return-1}return n.index-r.index})),"value")},groupBy:Tr,indexBy:Dr,countBy:Rr,partition:Fr,toArray:function(n){return n?U(n)?i.call(n):S(n)?n.match(Vr):tr(n)?jr(n,Tn):jn(n):[]},size:function(n){return null==n?0:tr(n)?n.length:nn(n).length},pick:qr,omit:Ur,first:zr,head:zr,take:zr,initial:Wr,last:function(n,r,t){return null==n||n.length<1?null==r||t?void 0:[]:null==r||t?n[n.length-1]:Lr(n,Math.max(0,n.length-r))},rest:Lr,tail:Lr,drop:Lr,compact:function(n){return xr(n,Boolean)},flatten:function(n,r){return er(n,r,!1)},without:Kr,uniq:Jr,unique:Jr,union:$r,intersection:function(n){for(var r=[],t=arguments.length,e=0,u=Y(n);e<u;e++){var o=n[e];if(!Mr(r,o)){var i;for(i=1;i<t&&Mr(arguments[i],o);i++);i===t&&r.push(o)}}return r},difference:Cr,unzip:Gr,transpose:Gr,zip:Hr,object:function(n,r){for(var t={},e=0,u=Y(n);e<u;e++)r?t[n[e]]=r[e]:t[n[e][0]]=n[e][1];return t},range:function(n,r,t){null==r&&(r=n||0,n=0),t||(t=r<n?-1:1);for(var e=Math.max(Math.ceil((r-n)/t),0),u=Array(e),o=0;o<e;o++,n+=t)u[o]=n;return u},chunk:function(n,r){if(null==r||r<1)return[];for(var t=[],e=0,u=n.length;e<u;)t.push(i.call(n,e,e+=r));return t},mixin:Xr,default:tn});return Yr._=Yr,Yr}));
\ No newline at end of file
diff --git a/docs/html/annotations/index.html b/docs/html/annotations/index.html
index 94a4864..049e0a6 100644
--- a/docs/html/annotations/index.html
+++ b/docs/html/annotations/index.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Relational Annotations" href="relational_annotations.html" /><link rel="prev" title="Getting Started" href="../getting_started.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Annotations - PyBryt documentation</title>
       <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="../_static/pygments.css">
@@ -255,7 +255,5 @@ <h3>Invariants<a class="headerlink" href="#invariants" title="Permalink to this
     <script src="../_static/jquery.js"></script>
     <script src="../_static/underscore.js"></script>
     <script src="../_static/doctools.js"></script>
-    <script src="../_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/annotations/invariants.html b/docs/html/annotations/invariants.html
index ebac578..643c5ac 100644
--- a/docs/html/annotations/invariants.html
+++ b/docs/html/annotations/invariants.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Reference Implementations" href="../reference_implementations.html" /><link rel="prev" title="Relational Annotations" href="relational_annotations.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Invariants - PyBryt documentation</title>
       <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="../_static/pygments.css">
@@ -133,7 +133,7 @@
         </label>
         <div class="section" id="invariants">
 <span id="id1"></span><h1>Invariants<a class="headerlink" href="#invariants" title="Permalink to this headline">¶</a></h1>
-<p>This page is current under construction.</p>
+<p>This page is currently under construction.</p>
 </div>
 
       </article>
@@ -189,7 +189,5 @@
     <script src="../_static/jquery.js"></script>
     <script src="../_static/underscore.js"></script>
     <script src="../_static/doctools.js"></script>
-    <script src="../_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/annotations/relational_annotations.html b/docs/html/annotations/relational_annotations.html
index 247e5a2..f662ee9 100644
--- a/docs/html/annotations/relational_annotations.html
+++ b/docs/html/annotations/relational_annotations.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="../genindex.html" /><link rel="search" title="Search" href="../search.html" /><link rel="next" title="Invariants" href="invariants.html" /><link rel="prev" title="Annotations" href="index.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Relational Annotations - PyBryt documentation</title>
       <link rel="stylesheet" href="../_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="../_static/pygments.css">
@@ -291,7 +291,6 @@ <h2>Boolean Annotations<a class="headerlink" href="#boolean-annotations" title="
     <script src="../_static/jquery.js"></script>
     <script src="../_static/underscore.js"></script>
     <script src="../_static/doctools.js"></script>
-    <script src="../_static/language_data.js"></script>
     <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="../_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/api_reference.html b/docs/html/api_reference.html
index 40d6717..2aeff3a 100644
--- a/docs/html/api_reference.html
+++ b/docs/html/api_reference.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="prev" title="Student Implementations" href="student_implementations.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>API Reference - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -137,7 +137,7 @@
 <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to this headline">¶</a></h2>
 <dl class="py class">
 <dt id="pybryt.Annotation">
-<em class="property">class </em><code class="sig-name descname">Annotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">name</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em>, <em class="sig-param"><span class="n">limit</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>int<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em>, <em class="sig-param"><span class="n">group</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em>, <em class="sig-param"><span class="n">success_message</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em>, <em class="sig-param"><span class="n">failure_message</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.Annotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">Annotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">limit</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">group</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">success_message</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">failure_message</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.Annotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Abstract base class for annotating a reference implementation.</p>
 <p>Defines an API and global configurations for all annotations used for marking-up reference
 implementations. All instances of this class, or any of its subclasses, are added to a singleton
@@ -148,7 +148,8 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 <dt class="field-odd">Parameters</dt>
 <dd class="field-odd"><ul class="simple">
 <li><p><strong>name</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code>, optional) – the name of the annotation</p></li>
-<li><p><strong>limit</strong> (<code class="docutils literal notranslate"><span class="pre">int</span></code>, optional) – in <code class="docutils literal notranslate"><span class="pre">_TRACKED_ANNOTATIONS</span></code></p></li>
+<li><p><strong>limit</strong> (<code class="docutils literal notranslate"><span class="pre">int</span></code>, optional) – the maximum number of annotations with name <code class="docutils literal notranslate"><span class="pre">name</span></code> to track
+in <code class="docutils literal notranslate"><span class="pre">_TRACKED_ANNOTATIONS</span></code></p></li>
 <li><p><strong>group</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code>, optional) – the name of the group that this annotation belongs to</p></li>
 <li><p><strong>success_message</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code>, optional) – a message to relay to the student if satisfied</p></li>
 <li><p><strong>failure_message</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code>, optional) – a message to relay to the student if not satisfied</p></li>
@@ -157,7 +158,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.after">
-<code class="sig-name descname">after</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">other_annotation</span><span class="p">:</span> <span class="n">pybryt.annotations.annotation.Annotation</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span> → <a class="reference internal" href="#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation">pybryt.annotations.relation.BeforeAnnotation</a><a class="headerlink" href="#pybryt.Annotation.after" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">after</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">other_annotation</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">pybryt.annotations.annotation.Annotation</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span> → <a class="reference internal" href="#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation"><span class="pre">pybryt.annotations.relation.BeforeAnnotation</span></a><a class="headerlink" href="#pybryt.Annotation.after" title="Permalink to this definition">¶</a></dt>
 <dd><p>Creates an annotation asserting that this annotation is satisfied after another (i.e. that
 the satisfying timestamp of this annotation is greater than or equal to that of
 <code class="docutils literal notranslate"><span class="pre">other_annotation</span></code>).</p>
@@ -179,7 +180,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.before">
-<code class="sig-name descname">before</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">other_annotation</span><span class="p">:</span> <span class="n">pybryt.annotations.annotation.Annotation</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span> → <a class="reference internal" href="#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation">pybryt.annotations.relation.BeforeAnnotation</a><a class="headerlink" href="#pybryt.Annotation.before" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">before</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">other_annotation</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">pybryt.annotations.annotation.Annotation</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span> → <a class="reference internal" href="#pybryt.annotations.relation.BeforeAnnotation" title="pybryt.annotations.relation.BeforeAnnotation"><span class="pre">pybryt.annotations.relation.BeforeAnnotation</span></a><a class="headerlink" href="#pybryt.Annotation.before" title="Permalink to this definition">¶</a></dt>
 <dd><p>Creates an annotation asserting that this annotation is satisfied before another (i.e. that
 the satisfying timestamp of this annotation is less than or equal to that of
 <code class="docutils literal notranslate"><span class="pre">other_annotation</span></code>).</p>
@@ -201,7 +202,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.check">
-<em class="property">abstract </em><code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.Annotation.check" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">abstract</span> </em><code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.Annotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Runs the check on the condition asserted by this annotation and returns a results object.</p>
 <p>Checks that the condition required by this annotation is met using the list of tuples of
 observed values and timestamps <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>. Creates and returns an
@@ -222,7 +223,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.children">
-<em class="property">abstract property </em><code class="sig-name descname">children</code><a class="headerlink" href="#pybryt.Annotation.children" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">abstract</span> <span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">children</span></code><a class="headerlink" href="#pybryt.Annotation.children" title="Permalink to this definition">¶</a></dt>
 <dd><p>the child annotations of this annotation. If this annotation has no
 children, an empty list.</p>
 <dl class="field-list simple">
@@ -233,12 +234,12 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.Annotation.failure_message">
-<code class="sig-name descname">failure_message</code><em class="property">: Optional<span class="p">[</span>str<span class="p">]</span></em><a class="headerlink" href="#pybryt.Annotation.failure_message" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">failure_message</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.Annotation.failure_message" title="Permalink to this definition">¶</a></dt>
 <dd><p>a message to relay to the student if not satisfied</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.get_tracked_annotations">
-<em class="property">static </em><code class="sig-name descname">get_tracked_annotations</code><span class="sig-paren">(</span><span class="sig-paren">)</span> → List<span class="p">[</span>pybryt.annotations.annotation.Annotation<span class="p">]</span><a class="headerlink" href="#pybryt.Annotation.get_tracked_annotations" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">get_tracked_annotations</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.Annotation</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.Annotation.get_tracked_annotations" title="Permalink to this definition">¶</a></dt>
 <dd><p>Returns the list of tracked annotations.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Returns</dt>
@@ -251,36 +252,51 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.Annotation.group">
-<code class="sig-name descname">group</code><em class="property">: Optional<span class="p">[</span>str<span class="p">]</span></em><a class="headerlink" href="#pybryt.Annotation.group" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">group</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.Annotation.group" title="Permalink to this definition">¶</a></dt>
 <dd><p>the name of the group that this annotation belongs to</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.Annotation.limit">
-<code class="sig-name descname">limit</code><em class="property">: Optional<span class="p">[</span>int<span class="p">]</span></em><a class="headerlink" href="#pybryt.Annotation.limit" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">limit</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">int</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.Annotation.limit" title="Permalink to this definition">¶</a></dt>
 <dd><p>the maximum number of annotations with this name to track in <code class="xref py py-obj docutils literal notranslate"><span class="pre">_TRACKED_ANNOTATIONS</span></code></p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.Annotation.name">
-<code class="sig-name descname">name</code><em class="property">: Optional<span class="p">[</span>str<span class="p">]</span></em><a class="headerlink" href="#pybryt.Annotation.name" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">name</span></code><em class="property"><span class="pre">:</span> <span class="pre">str</span></em><a class="headerlink" href="#pybryt.Annotation.name" title="Permalink to this definition">¶</a></dt>
 <dd><p>the name of the annotation</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.Annotation.reset_tracked_annotations">
-<em class="property">static </em><code class="sig-name descname">reset_tracked_annotations</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.Annotation.reset_tracked_annotations" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">reset_tracked_annotations</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.Annotation.reset_tracked_annotations" title="Permalink to this definition">¶</a></dt>
 <dd><p>Resets the list of tracked annotations and the mapping of group names to indices in that
 list.</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.Annotation.success_message">
-<code class="sig-name descname">success_message</code><em class="property">: Optional<span class="p">[</span>str<span class="p">]</span></em><a class="headerlink" href="#pybryt.Annotation.success_message" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">success_message</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.Annotation.success_message" title="Permalink to this definition">¶</a></dt>
 <dd><p>a message to relay to the student if satisfied</p>
 </dd></dl>
+<dl class="py method">
+<dt id="pybryt.Annotation.to_dict">
+<code class="sig-name descname"><span class="pre">to_dict</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.Annotation.to_dict" title="Permalink to this definition">¶</a></dt>
+<dd><p>Converts this annotation’s details to a JSON-friendly dictionary format.</p>
+<p>Output dictionary contains the annotation’s name, group, limit number, success message, and
+failure message.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>the dictionary representation of this annotation</p>
+</dd>
+<dt class="field-even">Return type</dt>
+<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">dict[str,</span> <span class="pre">object]</span></code></p>
+</dd>
+</dl>
+</dd></dl>
 </dd></dl>
 <div class="section" id="module-pybryt.annotations.value">
 <span id="value-annotations"></span><h3>Value Annotations<a class="headerlink" href="#module-pybryt.annotations.value" title="Permalink to this headline">¶</a></h3>
 <dl class="py class">
 <dt id="pybryt.annotations.value.Attribute">
-<em class="property">class </em><code class="sig-name descname">Attribute</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">obj</span><span class="p">:</span> <span class="n">Any</span></em>, <em class="sig-param"><span class="n">attrs</span><span class="p">:</span> <span class="n">Union<span class="p">[</span>str<span class="p">, </span>List<span class="p">[</span>str<span class="p">]</span><span class="p">]</span></span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.value.Attribute" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">Attribute</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">obj</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Any</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">attrs</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.value.Attribute" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that an object has an attribute value matching that of another object.</p>
 <p>Uses a <code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code> annotation to check there exists an object in the
 students’ code which as a specific attribute, or a series of attributes, with values matching
@@ -298,7 +314,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.value.Attribute.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.value.Attribute.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.value.Attribute.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks whether any of the values in <code class="docutils literal notranslate"><span class="pre">observed_values</span></code> has all of the required attributes,
 each matching the values expected.</p>
 <dl class="field-list simple">
@@ -317,7 +333,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.annotations.value.Attribute.children">
-<em class="property">property </em><code class="sig-name descname">children</code><a class="headerlink" href="#pybryt.annotations.value.Attribute.children" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">children</span></code><a class="headerlink" href="#pybryt.annotations.value.Attribute.children" title="Permalink to this definition">¶</a></dt>
 <dd><p>the child annotations of this annotation. If this annotation has no
 children, an empty list.</p>
 <dl class="field-list simple">
@@ -326,20 +342,41 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd>
 </dl>
 </dd></dl>
+<dl class="py method">
+<dt id="pybryt.annotations.value.Attribute.to_dict">
+<code class="sig-name descname"><span class="pre">to_dict</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.annotations.value.Attribute.to_dict" title="Permalink to this definition">¶</a></dt>
+<dd><p>Converts this annotation’s details to a JSON-friendly dictionary format.</p>
+<p>Output dictionary contains the annotation’s name, group, limit number, success message, and
+failure message, as well as an <code class="docutils literal notranslate"><span class="pre">invariants</span></code> key with a list of the names of all invariants
+used in this value annotation, a <code class="docutils literal notranslate"><span class="pre">atol</span></code> key with this annotation’s tolerance, and a key for
+the attributes being checked. This dictionary does <em>not</em> contain the value being tracked.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>the dictionary representation of this annotation</p>
+</dd>
+<dt class="field-even">Return type</dt>
+<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">dict[str,</span> <span class="pre">object]</span></code></p>
+</dd>
+</dl>
+</dd></dl>
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.value.Value">
-<em class="property">class </em><code class="sig-name descname">Value</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">value</span><span class="p">:</span> <span class="n">Any</span></em>, <em class="sig-param"><span class="n">tol</span><span class="p">:</span> <span class="n">Union<span class="p">[</span>float<span class="p">, </span>int<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">0</span></em>, <em class="sig-param"><span class="n">invariants</span><span class="p">:</span> <span class="n">List<span class="p">[</span><a class="reference internal" href="#pybryt.annotations.invariants.invariant" title="pybryt.annotations.invariants.invariant">pybryt.annotations.invariants.invariant</a><span class="p">]</span></span> <span class="o">=</span> <span class="default_value">[]</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.value.Value" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">Value</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Any</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">atol</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rtol</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">invariants</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#pybryt.annotations.invariants.invariant" title="pybryt.annotations.invariants.invariant"><span class="pre">pybryt.annotations.invariants.invariant</span></a><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">[]</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.value.Value" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation class for asserting that a value should be observed.</p>
 <p>Indicates that a value passed to the constructor should be observed while tracing through the
 students’ code. Values can be of any type that is picklable by <code class="docutils literal notranslate"><span class="pre">dill</span></code>. Values can specify a
 list of <a class="reference internal" href="annotations/invariants.html#invariants"><span class="std std-ref">invariants</span></a> that will allow objects to be considered “equal.” For
 values that support arithemtic operators, absolute tolerances can be specified as well.</p>
+<p>Numeric tolerances are computing as with <code class="docutils literal notranslate"><span class="pre">numpy.allcose</span></code>, where the value is considered “equal
+enough” if it is within <span class="math notranslate nohighlight">\(v \pm (\texttt{atol} + \texttt{rtol} \cdot |v|)\)</span>, where
+<span class="math notranslate nohighlight">\(v\)</span> is the value of the annotation.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
 <dd class="field-odd"><ul class="simple">
 <li><p><strong>value</strong> (<code class="docutils literal notranslate"><span class="pre">object</span></code>) – the value that should be observed</p></li>
-<li><p><strong>tol</strong> (<code class="docutils literal notranslate"><span class="pre">float</span></code> or <code class="docutils literal notranslate"><span class="pre">int</span></code>, optional) – absolute tolerance for numeric values</p></li>
+<li><p><strong>atol</strong> (<code class="docutils literal notranslate"><span class="pre">float</span></code> or <code class="docutils literal notranslate"><span class="pre">int</span></code>, optional) – absolute tolerance for numeric values</p></li>
+<li><p><strong>rtol</strong> (<code class="docutils literal notranslate"><span class="pre">float</span></code> or <code class="docutils literal notranslate"><span class="pre">int</span></code>, optional) – absolute tolerance for numeric values</p></li>
 <li><p><strong>invariants</strong> (<code class="docutils literal notranslate"><span class="pre">list[invariant]</span></code>) – invariants for
 this value</p></li>
 <li><p><strong>**kwargs</strong> – additional keyword arguments passed to the
@@ -347,9 +384,14 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </ul>
 </dd>
 </dl>
+<dl class="py attribute">
+<dt id="pybryt.annotations.value.Value.atol">
+<code class="sig-name descname"><span class="pre">atol</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.annotations.value.Value.atol" title="Permalink to this definition">¶</a></dt>
+<dd><p>absolute tolerance for numeric values</p>
+</dd></dl>
 <dl class="py method">
 <dt id="pybryt.annotations.value.Value.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.value.Value.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.value.Value.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that the value tracked by this annotation occurs in the list of observed values.</p>
 <p>Checks that the value under the conditions of its invariants occurs in the list of tuples of
 observed values and timestamps <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>. Creates and returns an
@@ -369,8 +411,12 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 </dd></dl>
 <dl class="py method">
+<dt id="pybryt.annotations.value.Value.check_values_equal">
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">check_values_equal</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">value</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">other_value</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">atol</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rtol</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.value.Value.check_values_equal" title="Permalink to this definition">¶</a></dt>
+<dd></dd></dl>
+<dl class="py method">
 <dt id="pybryt.annotations.value.Value.children">
-<em class="property">property </em><code class="sig-name descname">children</code><a class="headerlink" href="#pybryt.annotations.value.Value.children" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">children</span></code><a class="headerlink" href="#pybryt.annotations.value.Value.children" title="Permalink to this definition">¶</a></dt>
 <dd><p>the child annotations of this annotation. If this annotation has no
 children, an empty list.</p>
 <dl class="field-list simple">
@@ -381,18 +427,35 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.annotations.value.Value.intial_value">
-<code class="sig-name descname">intial_value</code><em class="property">: Any</em><a class="headerlink" href="#pybryt.annotations.value.Value.intial_value" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">intial_value</span></code><em class="property"><span class="pre">:</span> <span class="pre">Any</span></em><a class="headerlink" href="#pybryt.annotations.value.Value.intial_value" title="Permalink to this definition">¶</a></dt>
 <dd><p>a copy of the value passed to the constructor</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.annotations.value.Value.invariants">
-<code class="sig-name descname">invariants</code><em class="property">: List<span class="p">[</span>invariant<span class="p">]</span></em><a class="headerlink" href="#pybryt.annotations.value.Value.invariants" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">invariants</span></code><em class="property"><span class="pre">:</span> <span class="pre">List</span><span class="p"><span class="pre">[</span></span><a class="reference internal" href="#pybryt.annotations.invariants.invariant" title="pybryt.annotations.invariants.invariant"><span class="pre">pybryt.annotations.invariants.invariant</span></a><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.annotations.value.Value.invariants" title="Permalink to this definition">¶</a></dt>
 <dd><p>the invariants for this value</p>
 </dd></dl>
 <dl class="py attribute">
-<dt id="pybryt.annotations.value.Value.tol">
-<code class="sig-name descname">tol</code><em class="property">: Union<span class="p">[</span>float<span class="p">, </span>int<span class="p">]</span></em><a class="headerlink" href="#pybryt.annotations.value.Value.tol" title="Permalink to this definition">¶</a></dt>
-<dd><p>absolute tolerance for numeric values</p>
+<dt id="pybryt.annotations.value.Value.rtol">
+<code class="sig-name descname"><span class="pre">rtol</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">float</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.annotations.value.Value.rtol" title="Permalink to this definition">¶</a></dt>
+<dd><p>relative tolerance for numeric values</p>
+</dd></dl>
+<dl class="py method">
+<dt id="pybryt.annotations.value.Value.to_dict">
+<code class="sig-name descname"><span class="pre">to_dict</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.annotations.value.Value.to_dict" title="Permalink to this definition">¶</a></dt>
+<dd><p>Converts this annotation’s details to a JSON-friendly dictionary format.</p>
+<p>Output dictionary contains the annotation’s name, group, limit number, success message, and
+failure message, as well as an <code class="docutils literal notranslate"><span class="pre">invariants</span></code> key with a list of the names of all invariants
+used in this value annotation and a <code class="docutils literal notranslate"><span class="pre">atol</span></code> key with this annotation’s tolerance. This
+dictionary does <em>not</em> contain the value being tracked.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>the dictionary representation of this annotation</p>
+</dd>
+<dt class="field-even">Return type</dt>
+<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">dict[str,</span> <span class="pre">object]</span></code></p>
+</dd>
+</dl>
 </dd></dl>
 </dd></dl>
 </div>
@@ -401,7 +464,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 <p>Relational annotations for asserting conditions on other annotations</p>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.AndAnnotation">
-<em class="property">class </em><code class="sig-name descname">AndAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.AndAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">AndAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.AndAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that a series of annotations are <strong>all</strong> satisfied.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -415,7 +478,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.AndAnnotation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.AndAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.AndAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that all child annotations are satisfied by the values in <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -434,7 +497,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.BeforeAnnotation">
-<em class="property">class </em><code class="sig-name descname">BeforeAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.BeforeAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">BeforeAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.BeforeAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that one annotation occurs before another.</p>
 <p>When being <code class="xref py py-meth docutils literal notranslate"><span class="pre">check</span></code> is called, ensures that all child
 annotations are satisfied and then checks that for the <span class="math notranslate nohighlight">\(i^\text{th}\)</span> annotation the
@@ -452,7 +515,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.BeforeAnnotation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.BeforeAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.BeforeAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that all child annotations are satisfied by the values in <code class="docutils literal notranslate"><span class="pre">observed_values</span></code> and
 that the timestamps of the satisfying values occur in non-decreasing order.</p>
 <dl class="field-list simple">
@@ -472,7 +535,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.NotAnnotation">
-<em class="property">class </em><code class="sig-name descname">NotAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.NotAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">NotAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.NotAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that a single annotation should <strong>not</strong> be satisfied.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -486,7 +549,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.NotAnnotation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.NotAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.NotAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that the child annotation is not satisfied by the values in <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -505,7 +568,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.OrAnnotation">
-<em class="property">class </em><code class="sig-name descname">OrAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.OrAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">OrAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.OrAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that, of a series of annotations, <strong>any</strong> are satisfied.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -519,7 +582,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.OrAnnotation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.OrAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.OrAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that any of the child annotations are satisfied by the values in <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -538,7 +601,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.RelationalAnnotation">
-<em class="property">class </em><code class="sig-name descname">RelationalAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">RelationalAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Abstract base class for annotations that assert some kind of condition (temporal or boolean)
 on one or more other annotations.</p>
 <p>All relational annotations should inherit from this class, which defines a constructor that
@@ -556,7 +619,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.RelationalAnnotation.check">
-<em class="property">abstract </em><code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">abstract</span> </em><code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Runs the check on the condition asserted by this annotation and returns a results object.</p>
 <p>Checks that the condition required by this annotation is met using the list of tuples of
 observed values and timestamps <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>. Creates and returns an
@@ -577,7 +640,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.RelationalAnnotation.children">
-<em class="property">property </em><code class="sig-name descname">children</code><a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation.children" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">children</span></code><a class="headerlink" href="#pybryt.annotations.relation.RelationalAnnotation.children" title="Permalink to this definition">¶</a></dt>
 <dd><p>the child annotations that this annotation operates on</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -588,7 +651,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.relation.XorAnnotation">
-<em class="property">class </em><code class="sig-name descname">XorAnnotation</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">annotations</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.XorAnnotation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">XorAnnotation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">annotations</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.relation.XorAnnotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>Annotation for asserting that, of two annotations, one is satisfied and the other is not.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -602,7 +665,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dl>
 <dl class="py method">
 <dt id="pybryt.annotations.relation.XorAnnotation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em><span class="sig-paren">)</span> → pybryt.annotations.annotation.AnnotationResult<a class="headerlink" href="#pybryt.annotations.relation.XorAnnotation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.annotations.annotation.AnnotationResult</span><a class="headerlink" href="#pybryt.annotations.relation.XorAnnotation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks that one child annotation is satisfied and one is not by the values in
 <code class="docutils literal notranslate"><span class="pre">observed_values</span></code>.</p>
 <dl class="field-list simple">
@@ -626,7 +689,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 <p>Invariants for value annotations (available as <code class="xref py py-mod docutils literal notranslate"><span class="pre">pybryt.invariants</span></code>)</p>
 <dl class="py class">
 <dt id="pybryt.annotations.invariants.invariant">
-<em class="property">class </em><code class="sig-name descname">invariant</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">args</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.invariants.invariant" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">invariant</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.invariants.invariant" title="Permalink to this definition">¶</a></dt>
 <dd><p>Abstract base class for invariants.</p>
 <p>All subclasses should implement the <a class="reference internal" href="#pybryt.annotations.invariants.invariant.run" title="pybryt.annotations.invariants.invariant.run"><code class="xref py py-meth docutils literal notranslate"><span class="pre">run</span></code></a> static method for generating
 values that this invariant accepts as “correct”. Invariants have a custom <code class="docutils literal notranslate"><span class="pre">__new__</span></code> method
@@ -634,7 +697,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 function as callables.</p>
 <dl class="py method">
 <dt id="pybryt.annotations.invariants.invariant.run">
-<em class="property">abstract static </em><code class="sig-name descname">run</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Any<span class="p">]</span></span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span> → List<span class="p">[</span>Any<span class="p">]</span><a class="headerlink" href="#pybryt.annotations.invariants.invariant.run" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">abstract</span> <span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">run</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span> → <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.annotations.invariants.invariant.run" title="Permalink to this definition">¶</a></dt>
 <dd><p>Returns a list of values that this invariant accepts as correct.</p>
 <p>Takes in a list of acceptable values from a <code class="xref py py-class docutils literal notranslate"><span class="pre">Value</span></code> annotation and
 returns a list of values that would evaluate as “the same” under the conditions of this
@@ -662,13 +725,13 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 </dd></dl>
 <dl class="py class">
 <dt id="pybryt.annotations.invariants.string_capitalization">
-<em class="property">class </em><code class="sig-name descname">string_capitalization</code><span class="sig-paren">(</span><em class="sig-param"><span class="o">*</span><span class="n">args</span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.invariants.string_capitalization" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">string_capitalization</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span><span class="n"><span class="pre">args</span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.annotations.invariants.string_capitalization" title="Permalink to this definition">¶</a></dt>
 <dd><p>An invariant that compares strings ignoring capitalization.</p>
 <p>Works by taking in a list of values and lowercasing them if they are strings and leaving them
 unchanged otherwise.</p>
 <dl class="py method">
 <dt id="pybryt.annotations.invariants.string_capitalization.run">
-<em class="property">static </em><code class="sig-name descname">run</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Any<span class="p">]</span></span></em><span class="sig-paren">)</span> → List<span class="p">[</span>Any<span class="p">]</span><a class="headerlink" href="#pybryt.annotations.invariants.string_capitalization.run" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">run</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.annotations.invariants.string_capitalization.run" title="Permalink to this definition">¶</a></dt>
 <dd><p>Returns a list of values in which all strings have been lowercased.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -676,7 +739,10 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 of the annotation or from the results of other invariants</p>
 </dd>
 <dt class="field-even">Returns</dt>
-<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">list[object]</span></code>: the elements of <code class="docutils literal notranslate"><span class="pre">values</span></code> with all strings lowercased</p>
+<dd class="field-even"><p>the elements of <code class="docutils literal notranslate"><span class="pre">values</span></code> with all strings lowercased</p>
+</dd>
+<dt class="field-odd">Return type</dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">list[object]</span></code></p>
 </dd>
 </dl>
 </dd></dl>
@@ -686,7 +752,7 @@ <h2>Annotations<a class="headerlink" href="#annotations" title="Permalink to thi
 <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Permalink to this headline">¶</a></h3>
 <dl class="py class">
 <dt id="pybryt.AnnotationResult">
-<em class="property">class </em><code class="sig-name descname">AnnotationResult</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">satisfied</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>bool<span class="p">]</span></span></em>, <em class="sig-param"><span class="n">annotation</span><span class="p">:</span> <span class="n">pybryt.annotations.annotation.Annotation</span></em>, <em class="sig-param"><span class="n">value</span><span class="p">:</span> <span class="n">Any</span> <span class="o">=</span> <span class="default_value">None</span></em>, <em class="sig-param"><span class="n">timestamp</span><span class="p">:</span> <span class="n">int</span> <span class="o">=</span> <span class="default_value">- 1</span></em>, <em class="sig-param"><span class="n">children</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>List<span class="p">[</span>AnnotationResult<span class="p">]</span><span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.AnnotationResult" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">AnnotationResult</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">satisfied</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">bool</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">annotation</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">pybryt.annotations.annotation.Annotation</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">value</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">timestamp</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">int</span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">-</span> <span class="pre">1</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">children</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.AnnotationResult</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">[]</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.AnnotationResult" title="Permalink to this definition">¶</a></dt>
 <dd><p>Class that manages and defines an API for interacting with the results of an annotation.</p>
 <p>Created when an annotation calls its <code class="xref py py-meth docutils literal notranslate"><span class="pre">check&lt;pybryt.Annotation.check()</span></code> method and
 wrangles the results of that annotation. Contains fields for tracking child annotation results,
@@ -706,17 +772,17 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dl>
 <dl class="py attribute">
 <dt id="pybryt.AnnotationResult.annotation">
-<code class="sig-name descname">annotation</code><em class="property">: Annotation</em><a class="headerlink" href="#pybryt.AnnotationResult.annotation" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">annotation</span></code><em class="property"><span class="pre">:</span> <span class="pre">pybryt.annotations.annotation.Annotation</span></em><a class="headerlink" href="#pybryt.AnnotationResult.annotation" title="Permalink to this definition">¶</a></dt>
 <dd><p>the annotation that this result is for</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.AnnotationResult.children">
-<code class="sig-name descname">children</code><em class="property">: Optional[List[‘AnnotationResult’]]</em><a class="headerlink" href="#pybryt.AnnotationResult.children" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">children</span></code><em class="property"><span class="pre">:</span> <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.AnnotationResult</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.AnnotationResult.children" title="Permalink to this definition">¶</a></dt>
 <dd><p>child annotation results of this annotation result</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.AnnotationResult.group">
-<em class="property">property </em><code class="sig-name descname">group</code><a class="headerlink" href="#pybryt.AnnotationResult.group" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">group</span></code><a class="headerlink" href="#pybryt.AnnotationResult.group" title="Permalink to this definition">¶</a></dt>
 <dd><p>the group name of the annotation that these results track</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -726,7 +792,7 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.AnnotationResult.messages">
-<em class="property">property </em><code class="sig-name descname">messages</code><a class="headerlink" href="#pybryt.AnnotationResult.messages" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">messages</span></code><a class="headerlink" href="#pybryt.AnnotationResult.messages" title="Permalink to this definition">¶</a></dt>
 <dd><p>The messages returned by this annotation and its
 children based on whether or not the annotations were satisfied. A list of tuples where the
 first element is the message, the second is the name of the annotation (or <code class="docutils literal notranslate"><span class="pre">None</span></code> if no
@@ -739,7 +805,7 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.AnnotationResult.name">
-<em class="property">property </em><code class="sig-name descname">name</code><a class="headerlink" href="#pybryt.AnnotationResult.name" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">name</span></code><a class="headerlink" href="#pybryt.AnnotationResult.name" title="Permalink to this definition">¶</a></dt>
 <dd><p>the name of the annotation that these results track</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -749,7 +815,7 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.AnnotationResult.satisfied">
-<em class="property">property </em><code class="sig-name descname">satisfied</code><a class="headerlink" href="#pybryt.AnnotationResult.satisfied" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">satisfied</span></code><a class="headerlink" href="#pybryt.AnnotationResult.satisfied" title="Permalink to this definition">¶</a></dt>
 <dd><p>whether this annotation was satisfied</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -759,7 +825,7 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.AnnotationResult.satisfied_at">
-<em class="property">property </em><code class="sig-name descname">satisfied_at</code><a class="headerlink" href="#pybryt.AnnotationResult.satisfied_at" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">satisfied_at</span></code><a class="headerlink" href="#pybryt.AnnotationResult.satisfied_at" title="Permalink to this definition">¶</a></dt>
 <dd><p>the step counter value at which this annotation was satisfied; if child results are
 present, this is the maximum satisfying timestamp of all child results; if this annotation
 was not satisfied, returns -1</p>
@@ -771,12 +837,27 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.AnnotationResult.timestamp">
-<code class="sig-name descname">timestamp</code><em class="property">: int</em><a class="headerlink" href="#pybryt.AnnotationResult.timestamp" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">timestamp</span></code><em class="property"><span class="pre">:</span> <span class="pre">int</span></em><a class="headerlink" href="#pybryt.AnnotationResult.timestamp" title="Permalink to this definition">¶</a></dt>
 <dd><p>the step counter value at which this annotation was satisfied</p>
 </dd></dl>
 <dl class="py method">
+<dt id="pybryt.AnnotationResult.to_dict">
+<code class="sig-name descname"><span class="pre">to_dict</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.AnnotationResult.to_dict" title="Permalink to this definition">¶</a></dt>
+<dd><p>Converts this annotation result’s details to a JSON-friendly dictionary format.</p>
+<p>Output dictionary contains the annotation, whether it was satisfied, when it was satisfied,
+and any child results.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>the dictionary representation of this annotation</p>
+</dd>
+<dt class="field-even">Return type</dt>
+<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">dict[str,</span> <span class="pre">object]</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+<dl class="py method">
 <dt id="pybryt.AnnotationResult.value">
-<em class="property">property </em><code class="sig-name descname">value</code><a class="headerlink" href="#pybryt.AnnotationResult.value" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">value</span></code><a class="headerlink" href="#pybryt.AnnotationResult.value" title="Permalink to this definition">¶</a></dt>
 <dd><p>the value that satisfied the condition of this annotation</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -791,7 +872,7 @@ <h3>Annotation Results<a class="headerlink" href="#annotation-results" title="Pe
 <h2>Reference Implementations<a class="headerlink" href="#reference-implementations" title="Permalink to this headline">¶</a></h2>
 <dl class="py class">
 <dt id="pybryt.ReferenceImplementation">
-<em class="property">class </em><code class="sig-name descname">ReferenceImplementation</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">annotations</span><span class="p">:</span> <span class="n">List<span class="p">[</span>pybryt.annotations.annotation.Annotation<span class="p">]</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceImplementation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">ReferenceImplementation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">annotations</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.Annotation</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceImplementation" title="Permalink to this definition">¶</a></dt>
 <dd><p>A reference implementation class for managing collections of annotations. Defines methods for
 creating, running, and storing reference implementations.</p>
 <dl class="field-list simple">
@@ -801,19 +882,19 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 </dl>
 <dl class="py attribute">
 <dt id="pybryt.ReferenceImplementation.annotations">
-<code class="sig-name descname">annotations</code><em class="property">: List<span class="p">[</span>Annotation<span class="p">]</span></em><a class="headerlink" href="#pybryt.ReferenceImplementation.annotations" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">annotations</span></code><em class="property"><span class="pre">:</span> <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.Annotation</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.ReferenceImplementation.annotations" title="Permalink to this definition">¶</a></dt>
 <dd><p>the annotations comprising this reference implementation</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceImplementation.compile">
-<em class="property">classmethod </em><code class="sig-name descname">compile</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">file</span><span class="p">:</span> <span class="n">str</span></em><span class="sig-paren">)</span> → Union<span class="p">[</span>pybryt.reference.ReferenceImplementation<span class="p">, </span>List<span class="p">[</span>pybryt.reference.ReferenceImplementation<span class="p">]</span><span class="p">]</span><a class="headerlink" href="#pybryt.ReferenceImplementation.compile" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">classmethod</span> </em><code class="sig-name descname"><span class="pre">compile</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path_or_nb</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">nbformat.notebooknode.NotebookNode</span><span class="p"><span class="pre">]</span></span></span></em><span class="sig-paren">)</span> → <span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">,</span> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.ReferenceImplementation.compile" title="Permalink to this definition">¶</a></dt>
 <dd><p>Compiles a notebook or Python script into a single or list of reference implementations.</p>
 <p>Creates a reference implementation by executing a Jupyter Notebook or Python script and
 collecting all <a class="reference internal" href="#pybryt.ReferenceImplementation" title="pybryt.ReferenceImplementation"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceImplementation</span></code></a> objects
 created, if any, or all <a class="reference internal" href="#pybryt.Annotation" title="pybryt.Annotation"><code class="xref py py-class docutils literal notranslate"><span class="pre">Annotation</span></code></a> objects otherwise.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
-<dd class="field-odd"><p><strong>file</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code>) – path to the file to be executed</p>
+<dd class="field-odd"><p><strong>path_or_nb</strong> (<code class="docutils literal notranslate"><span class="pre">str</span></code> or <code class="docutils literal notranslate"><span class="pre">nbformat.NotebookNode</span></code>) – the file to be executed</p>
 </dd>
 <dt class="field-even">Returns</dt>
 <dd class="field-even"><p>the reference(s)
@@ -826,7 +907,7 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceImplementation.dump">
-<code class="sig-name descname">dump</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">dest</span><span class="p">:</span> <span class="n">str</span> <span class="o">=</span> <span class="default_value">'reference.pkl'</span></em><span class="sig-paren">)</span> → NoReturn<a class="headerlink" href="#pybryt.ReferenceImplementation.dump" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">dump</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dest</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">str</span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">'reference.pkl'</span></span></em><span class="sig-paren">)</span> → <span class="pre">NoReturn</span><a class="headerlink" href="#pybryt.ReferenceImplementation.dump" title="Permalink to this definition">¶</a></dt>
 <dd><p>Pickles this reference implementation to a file.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -836,7 +917,7 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceImplementation.load">
-<em class="property">static </em><code class="sig-name descname">load</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">file</span><span class="p">:</span> <span class="n">str</span></em><span class="sig-paren">)</span> → pybryt.reference.ReferenceImplementation<a class="headerlink" href="#pybryt.ReferenceImplementation.load" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">load</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">file</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> → <span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">,</span> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.ReferenceImplementation.load" title="Permalink to this definition">¶</a></dt>
 <dd><p>Unpickles a reference implementation from a file.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -853,7 +934,7 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceImplementation.run">
-<code class="sig-name descname">run</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">observed_values</span><span class="p">:</span> <span class="n">List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></span></em>, <em class="sig-param"><span class="n">group</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em><span class="sig-paren">)</span> → pybryt.reference.ReferenceResult<a class="headerlink" href="#pybryt.ReferenceImplementation.run" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">run</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">observed_values</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">group</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.reference.ReferenceResult</span><a class="headerlink" href="#pybryt.ReferenceImplementation.run" title="Permalink to this definition">¶</a></dt>
 <dd><p>Runs the annotations tracked by this reference implementation against a memory footprint.</p>
 <p>Can run only specific annotations by specifying the <code class="docutils literal notranslate"><span class="pre">group</span></code> argument. Returns a
 <a class="reference internal" href="#pybryt.ReferenceResult" title="pybryt.ReferenceResult"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceResult</span></code></a> object.</p>
@@ -871,7 +952,7 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 <dd class="field-odd"><p><a class="reference internal" href="#pybryt.ReferenceResult" title="pybryt.ReferenceResult"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceResult</span></code></a></p>
 </dd>
 <dt class="field-even">Raises</dt>
-<dd class="field-even"><p><strong>ValueError</strong> – if <code class="docutils literal notranslate"><span class="pre">group</span></code> is specified but there are no annotations with that group:</p>
+<dd class="field-even"><p><strong>ValueError</strong> – if <code class="docutils literal notranslate"><span class="pre">group</span></code> is specified but there are no annotations with that group</p>
 </dd>
 </dl>
 </dd></dl>
@@ -880,7 +961,7 @@ <h2>Reference Implementations<a class="headerlink" href="#reference-implementati
 <h3>Reference Results<a class="headerlink" href="#reference-results" title="Permalink to this headline">¶</a></h3>
 <dl class="py class">
 <dt id="pybryt.ReferenceResult">
-<em class="property">class </em><code class="sig-name descname">ReferenceResult</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">reference</span><span class="p">:</span> <span class="n">pybryt.reference.ReferenceImplementation</span></em>, <em class="sig-param"><span class="n">annotation_results</span><span class="p">:</span> <span class="n">List<span class="p">[</span>pybryt.annotations.annotation.AnnotationResult<span class="p">]</span></span></em>, <em class="sig-param"><span class="n">group</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceResult" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">ReferenceResult</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">reference</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">pybryt.reference.ReferenceImplementation</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">annotation_results</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.AnnotationResult</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">group</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceResult" title="Permalink to this definition">¶</a></dt>
 <dd><p>Class for wrangling and managing the results of a reference implementation. Collects a series of
 <a class="reference internal" href="#pybryt.AnnotationResult" title="pybryt.AnnotationResult"><code class="xref py py-class docutils literal notranslate"><span class="pre">AnnotationResult</span></code></a> objects and provides an API for managing
 those results collectively.</p>
@@ -896,7 +977,7 @@ <h3>Reference Results<a class="headerlink" href="#reference-results" title="Perm
 </dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceResult.correct">
-<em class="property">property </em><code class="sig-name descname">correct</code><a class="headerlink" href="#pybryt.ReferenceResult.correct" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">correct</span></code><a class="headerlink" href="#pybryt.ReferenceResult.correct" title="Permalink to this definition">¶</a></dt>
 <dd><p>whether the reference implementation was satisfied</p>
 <dl class="field-list simple">
 <dt class="field-odd">Type</dt>
@@ -906,12 +987,12 @@ <h3>Reference Results<a class="headerlink" href="#reference-results" title="Perm
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.ReferenceResult.group">
-<code class="sig-name descname">group</code><em class="property">: Optional<span class="p">[</span>str<span class="p">]</span></em><a class="headerlink" href="#pybryt.ReferenceResult.group" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">group</span></code><em class="property"><span class="pre">:</span> <span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.ReferenceResult.group" title="Permalink to this definition">¶</a></dt>
 <dd><p>the name of the group of annotations executed, if applicable</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceResult.messages">
-<em class="property">property </em><code class="sig-name descname">messages</code><a class="headerlink" href="#pybryt.ReferenceResult.messages" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">property</span> </em><code class="sig-name descname"><span class="pre">messages</span></code><a class="headerlink" href="#pybryt.ReferenceResult.messages" title="Permalink to this definition">¶</a></dt>
 <dd><p>the list of messages returned by all annotations in the reference
 implementation; if <code class="docutils literal notranslate"><span class="pre">self.group</span></code> is not <code class="docutils literal notranslate"><span class="pre">None</span></code>, only messages from annotations in that
 group are returned</p>
@@ -923,17 +1004,17 @@ <h3>Reference Results<a class="headerlink" href="#reference-results" title="Perm
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.ReferenceResult.reference">
-<code class="sig-name descname">reference</code><em class="property">: ReferenceImplementation</em><a class="headerlink" href="#pybryt.ReferenceResult.reference" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">reference</span></code><em class="property"><span class="pre">:</span> <span class="pre">pybryt.reference.ReferenceImplementation</span></em><a class="headerlink" href="#pybryt.ReferenceResult.reference" title="Permalink to this definition">¶</a></dt>
 <dd><p>the reference implementation executed</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.ReferenceResult.results">
-<code class="sig-name descname">results</code><em class="property">: List<span class="p">[</span>AnnotationResult<span class="p">]</span></em><a class="headerlink" href="#pybryt.ReferenceResult.results" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">results</span></code><em class="property"><span class="pre">:</span> <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.annotations.annotation.AnnotationResult</span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.ReferenceResult.results" title="Permalink to this definition">¶</a></dt>
 <dd><p>the annotation results from running the reference implementation</p>
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.ReferenceResult.to_array">
-<code class="sig-name descname">to_array</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceResult.to_array" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">to_array</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.ReferenceResult.to_array" title="Permalink to this definition">¶</a></dt>
 <dd><p>Converts this result into a numpy array of integers, where <code class="docutils literal notranslate"><span class="pre">1</span></code> indicates a satisfied
 annotation and <code class="docutils literal notranslate"><span class="pre">0</span></code> is unsatisfied.</p>
 <dl class="field-list simple">
@@ -945,6 +1026,21 @@ <h3>Reference Results<a class="headerlink" href="#reference-results" title="Perm
 </dd>
 </dl>
 </dd></dl>
+<dl class="py method">
+<dt id="pybryt.ReferenceResult.to_dict">
+<code class="sig-name descname"><span class="pre">to_dict</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">Dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">Any</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.ReferenceResult.to_dict" title="Permalink to this definition">¶</a></dt>
+<dd><p>Converts this reference result’s details to a JSON-friendly dictionary format.</p>
+<p>Output dictionary contains the group name run, if present, and the dictionary representations
+of all child annotation results.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Returns</dt>
+<dd class="field-odd"><p>the dictionary representation of this annotation</p>
+</dd>
+<dt class="field-even">Return type</dt>
+<dd class="field-even"><p><code class="docutils literal notranslate"><span class="pre">dict[str,</span> <span class="pre">object]</span></code></p>
+</dd>
+</dl>
+</dd></dl>
 </dd></dl>
 </div>
 </div>
@@ -952,7 +1048,7 @@ <h3>Reference Results<a class="headerlink" href="#reference-results" title="Perm
 <h2>Student Implementations<a class="headerlink" href="#student-implementations" title="Permalink to this headline">¶</a></h2>
 <dl class="py class">
 <dt id="pybryt.StudentImplementation">
-<em class="property">class </em><code class="sig-name descname">StudentImplementation</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">path_or_nb</span><span class="p">:</span> <span class="n">Union<span class="p">[</span>str<span class="p">, </span>nbformat.notebooknode.NotebookNode<span class="p">]</span></span></em>, <em class="sig-param"><span class="n">addl_filenames</span><span class="p">:</span> <span class="n">List<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">[]</span></em>, <em class="sig-param"><span class="n">output</span><span class="p">:</span> <span class="n">Optional<span class="p">[</span>str<span class="p">]</span></span> <span class="o">=</span> <span class="default_value">None</span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.StudentImplementation" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">class</span> </em><code class="sig-name descname"><span class="pre">StudentImplementation</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">path_or_nb</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span> </span><span class="pre">nbformat.notebooknode.NotebookNode</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">addl_filenames</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">[]</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">output</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Optional</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.StudentImplementation" title="Permalink to this definition">¶</a></dt>
 <dd><p>A student implementation class for handling the execution of student work and manging the
 memory footprint generated by that execution.</p>
 <dl class="field-list simple">
@@ -967,7 +1063,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.check">
-<code class="sig-name descname">check</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">ref</span><span class="p">:</span> <span class="n">Union<span class="p">[</span>pybryt.reference.ReferenceImplementation<span class="p">, </span>List<span class="p">[</span>pybryt.reference.ReferenceImplementation<span class="p">]</span><span class="p">]</span></span></em>, <em class="sig-param"><span class="n">group</span><span class="o">=</span><span class="default_value">None</span></em><span class="sig-paren">)</span> → Union<span class="p">[</span>pybryt.reference.ReferenceResult<span class="p">, </span>List<span class="p">[</span>pybryt.reference.ReferenceResult<span class="p">]</span><span class="p">]</span><a class="headerlink" href="#pybryt.StudentImplementation.check" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">ref</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">,</span> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceImplementation</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="n"><span class="pre">group</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span> → <span class="pre">Union</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceResult</span><span class="p"><span class="pre">,</span> </span><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceResult</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.StudentImplementation.check" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks this student implementation against a single or list of reference implementations.
 Returns the <a class="reference internal" href="#pybryt.ReferenceResult" title="pybryt.ReferenceResult"><code class="xref py py-class docutils literal notranslate"><span class="pre">ReferenceResult</span></code></a> object(s) resulting from
 those checks.</p>
@@ -990,7 +1086,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.check_plagiarism">
-<code class="sig-name descname">check_plagiarism</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">student_impls</span><span class="p">:</span> <span class="n">List<span class="p">[</span>StudentImplementation<span class="p">]</span></span></em>, <em class="sig-param"><span class="o">**</span><span class="n">kwargs</span></em><span class="sig-paren">)</span> → List<span class="p">[</span>pybryt.reference.ReferenceResult<span class="p">]</span><a class="headerlink" href="#pybryt.StudentImplementation.check_plagiarism" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">check_plagiarism</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">student_impls</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.student.StudentImplementation</span><span class="p"><span class="pre">]</span></span></span></em>, <em class="sig-param"><span class="o"><span class="pre">**</span></span><span class="n"><span class="pre">kwargs</span></span></em><span class="sig-paren">)</span> → <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">pybryt.reference.ReferenceResult</span><span class="p"><span class="pre">]</span></span><a class="headerlink" href="#pybryt.StudentImplementation.check_plagiarism" title="Permalink to this definition">¶</a></dt>
 <dd><p>Checks this student implementation against a list of other student implementations for
 plagiarism. Uses <code class="xref py py-meth docutils literal notranslate"><span class="pre">create_references</span></code> to create
 a randomly-generated reference implementation from this student implementation and runs it
@@ -1017,7 +1113,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.dump">
-<code class="sig-name descname">dump</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">dest</span><span class="p">:</span> <span class="n">str</span> <span class="o">=</span> <span class="default_value">'student.pkl'</span></em><span class="sig-paren">)</span> → NoReturn<a class="headerlink" href="#pybryt.StudentImplementation.dump" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">dump</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">dest</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">str</span></span> <span class="o"><span class="pre">=</span></span> <span class="default_value"><span class="pre">'student.pkl'</span></span></em><span class="sig-paren">)</span> → <span class="pre">NoReturn</span><a class="headerlink" href="#pybryt.StudentImplementation.dump" title="Permalink to this definition">¶</a></dt>
 <dd><p>Pickles this student implementation to a file.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -1027,7 +1123,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.dumps">
-<code class="sig-name descname">dumps</code><span class="sig-paren">(</span><span class="sig-paren">)</span> → str<a class="headerlink" href="#pybryt.StudentImplementation.dumps" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">dumps</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span> → <span class="pre">str</span><a class="headerlink" href="#pybryt.StudentImplementation.dumps" title="Permalink to this definition">¶</a></dt>
 <dd><p>Pickles this student implementation to a base-64-encoded string.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Returns</dt>
@@ -1040,7 +1136,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.load">
-<em class="property">static </em><code class="sig-name descname">load</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">file</span><span class="p">:</span> <span class="n">str</span></em><span class="sig-paren">)</span> → pybryt.student.StudentImplementation<a class="headerlink" href="#pybryt.StudentImplementation.load" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">static</span> </em><code class="sig-name descname"><span class="pre">load</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">file</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.student.StudentImplementation</span><a class="headerlink" href="#pybryt.StudentImplementation.load" title="Permalink to this definition">¶</a></dt>
 <dd><p>Unpickles a student implementation from a file.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -1057,7 +1153,7 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py method">
 <dt id="pybryt.StudentImplementation.loads">
-<em class="property">classmethod </em><code class="sig-name descname">loads</code><span class="sig-paren">(</span><em class="sig-param"><span class="n">data</span><span class="p">:</span> <span class="n">str</span></em><span class="sig-paren">)</span> → pybryt.student.StudentImplementation<a class="headerlink" href="#pybryt.StudentImplementation.loads" title="Permalink to this definition">¶</a></dt>
+<em class="property"><span class="pre">classmethod</span> </em><code class="sig-name descname"><span class="pre">loads</span></code><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">data</span></span><span class="p"><span class="pre">:</span></span> <span class="n"><span class="pre">str</span></span></em><span class="sig-paren">)</span> → <span class="pre">pybryt.student.StudentImplementation</span><a class="headerlink" href="#pybryt.StudentImplementation.loads" title="Permalink to this definition">¶</a></dt>
 <dd><p>Unpickles a student implementation from a base-64-encoded string.</p>
 <dl class="field-list simple">
 <dt class="field-odd">Parameters</dt>
@@ -1074,20 +1170,68 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.StudentImplementation.nb">
-<code class="sig-name descname">nb</code><em class="property">: nbformat.NotebookNode</em><a class="headerlink" href="#pybryt.StudentImplementation.nb" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">nb</span></code><em class="property"><span class="pre">:</span> <span class="pre">nbformat.notebooknode.NotebookNode</span></em><a class="headerlink" href="#pybryt.StudentImplementation.nb" title="Permalink to this definition">¶</a></dt>
 <dd><p>the submission notebook</p>
 </dd></dl>
 <dl class="py attribute">
+<dt id="pybryt.StudentImplementation.nb_path">
+<code class="sig-name descname"><span class="pre">nb_path</span></code><em class="property"><span class="pre">:</span> <span class="pre">str</span></em><a class="headerlink" href="#pybryt.StudentImplementation.nb_path" title="Permalink to this definition">¶</a></dt>
+<dd><p>the path to the notebook file</p>
+</dd></dl>
+<dl class="py attribute">
 <dt id="pybryt.StudentImplementation.steps">
-<code class="sig-name descname">steps</code><em class="property">: int</em><a class="headerlink" href="#pybryt.StudentImplementation.steps" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">steps</span></code><em class="property"><span class="pre">:</span> <span class="pre">int</span></em><a class="headerlink" href="#pybryt.StudentImplementation.steps" title="Permalink to this definition">¶</a></dt>
 <dd><p>number of execution steps</p>
 </dd></dl>
 <dl class="py attribute">
 <dt id="pybryt.StudentImplementation.values">
-<code class="sig-name descname">values</code><em class="property">: List<span class="p">[</span>Tuple<span class="p">[</span>Any<span class="p">, </span>int<span class="p">]</span><span class="p">]</span></em><a class="headerlink" href="#pybryt.StudentImplementation.values" title="Permalink to this definition">¶</a></dt>
+<code class="sig-name descname"><span class="pre">values</span></code><em class="property"><span class="pre">:</span> <span class="pre">List</span><span class="p"><span class="pre">[</span></span><span class="pre">Tuple</span><span class="p"><span class="pre">[</span></span><span class="pre">Any</span><span class="p"><span class="pre">,</span> </span><span class="pre">int</span><span class="p"><span class="pre">]</span></span><span class="p"><span class="pre">]</span></span></em><a class="headerlink" href="#pybryt.StudentImplementation.values" title="Permalink to this definition">¶</a></dt>
 <dd><p>the memory footprint (a list of tuples of objects and their timestamps)</p>
 </dd></dl>
 </dd></dl>
+<div class="section" id="module-pybryt.execution">
+<span id="execution"></span><h3>Execution<a class="headerlink" href="#module-pybryt.execution" title="Permalink to this headline">¶</a></h3>
+<p>Submission execution internals for PyBryt</p>
+<dl class="py function">
+<dt id="pybryt.execution.tracing_off">
+<code class="sig-name descname"><span class="pre">tracing_off</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.execution.tracing_off" title="Permalink to this definition">¶</a></dt>
+<dd><p>Turns off PyBryt’s tracing if tracing is occurring in this call stack. If PyBryt is not tracing,
+takes no action.</p>
+<p>This method can be used in students’ notebooks to include code that shouldn’t be traced as part
+of the submission, e.g. demo code or ungraded code. In the example below, the call that creates
+<code class="docutils literal notranslate"><span class="pre">x2</span></code> is traced but the one to create <code class="docutils literal notranslate"><span class="pre">x3</span></code> is not.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">a</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">x</span> <span class="o">**</span> <span class="n">a</span>
+
+<span class="n">x2</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
+
+<span class="n">pybryt</span><span class="o">.</span><span class="n">tracing_off</span><span class="p">()</span>
+<span class="n">x3</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
+</pre></div>
+</div>
+</dd></dl>
+<dl class="py function">
+<dt id="pybryt.execution.tracing_on">
+<code class="sig-name descname"><span class="pre">tracing_on</span></code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pybryt.execution.tracing_on" title="Permalink to this definition">¶</a></dt>
+<dd><p>Turns tracing on if PyBryt was tracing the call stack. If PyBryt is not tracing or
+<code class="xref py py-meth docutils literal notranslate"><span class="pre">tracing_off</span></code> has not been called, no action is taken.</p>
+<p>This method can be used in students’ notebooks to turn tracing back on after deactivating tracing
+for ungraded code In the example below, <code class="docutils literal notranslate"><span class="pre">x4</span></code> is traced because <code class="docutils literal notranslate"><span class="pre">tracing_on</span></code> is used after
+<code class="docutils literal notranslate"><span class="pre">tracing_off</span></code> and the creation of <code class="docutils literal notranslate"><span class="pre">x3</span></code>.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">a</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">x</span> <span class="o">**</span> <span class="n">a</span>
+
+<span class="n">x2</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
+
+<span class="n">pybryt</span><span class="o">.</span><span class="n">tracing_off</span><span class="p">()</span>
+<span class="n">x3</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span>
+<span class="n">pybryt</span><span class="o">.</span><span class="n">tracing_on</span><span class="p">()</span>
+
+<span class="n">x4</span> <span class="o">=</span> <span class="nb">pow</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
+</pre></div>
+</div>
+</dd></dl>
+</div>
 </div>
 </div>
 
@@ -1149,7 +1293,10 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
 <li><a class="reference internal" href="#reference-results">Reference Results</a></li>
 </ul>
 </li>
-<li><a class="reference internal" href="#student-implementations">Student Implementations</a></li>
+<li><a class="reference internal" href="#student-implementations">Student Implementations</a><ul>
+<li><a class="reference internal" href="#module-pybryt.execution">Execution</a></li>
+</ul>
+</li>
 </ul>
 </li>
 </ul>
@@ -1166,7 +1313,6 @@ <h2>Student Implementations<a class="headerlink" href="#student-implementations"
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
     <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/genindex.html b/docs/html/genindex.html
index 2387a40..b0703d7 100644
--- a/docs/html/genindex.html
+++ b/docs/html/genindex.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="#" /><link rel="search" title="Search" href="search.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/><title>Index - PyBryt documentation</title>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/><title>Index - PyBryt documentation</title>
 <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
     <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="_static/pygments_dark.css">
@@ -147,13 +147,15 @@ <h2>A</h2>
 </li>
         <li><a href="api_reference.html#pybryt.AnnotationResult.annotation">annotation (AnnotationResult attribute)</a>
 </li>
-    </ul></td>
-    <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.Annotation">Annotation (class in pybryt)</a>
 </li>
+    </ul></td>
+    <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.AnnotationResult">AnnotationResult (class in pybryt)</a>
 </li>
         <li><a href="api_reference.html#pybryt.ReferenceImplementation.annotations">annotations (ReferenceImplementation attribute)</a>
+</li>
+        <li><a href="api_reference.html#pybryt.annotations.value.Value.atol">atol (Value attribute)</a>
 </li>
         <li><a href="api_reference.html#pybryt.annotations.value.Attribute">Attribute (class in pybryt.annotations.value)</a>
 </li>
@@ -204,6 +206,8 @@ <h2>C</h2>
     </ul></td>
     <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.StudentImplementation.check_plagiarism">check_plagiarism() (StudentImplementation method)</a>
+</li>
+        <li><a href="api_reference.html#pybryt.annotations.value.Value.check_values_equal">check_values_equal() (Value static method)</a>
 </li>
         <li><a href="api_reference.html#pybryt.AnnotationResult.children">children (AnnotationResult attribute)</a>
 </li>
@@ -328,6 +332,8 @@ <h2>M</h2>
           <li><a href="api_reference.html#module-pybryt.annotations.relation">pybryt.annotations.relation</a>
 </li>
           <li><a href="api_reference.html#module-pybryt.annotations.value">pybryt.annotations.value</a>
+</li>
+          <li><a href="api_reference.html#module-pybryt.execution">pybryt.execution</a>
 </li>
         </ul></li>
     </ul></td>
@@ -345,6 +351,8 @@ <h2>N</h2>
     </ul></td>
     <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.StudentImplementation.nb">nb (StudentImplementation attribute)</a>
+</li>
+        <li><a href="api_reference.html#pybryt.StudentImplementation.nb_path">nb_path (StudentImplementation attribute)</a>
 </li>
         <li><a href="api_reference.html#pybryt.annotations.relation.NotAnnotation">NotAnnotation (class in pybryt.annotations.relation)</a>
 </li>
@@ -387,6 +395,13 @@ <h2>P</h2>
 
         <ul>
           <li><a href="api_reference.html#module-pybryt.annotations.value">module</a>
+</li>
+        </ul></li>
+        <li>
+    pybryt.execution
+
+        <ul>
+          <li><a href="api_reference.html#module-pybryt.execution">module</a>
 </li>
         </ul></li>
     </ul></td>
@@ -405,11 +420,13 @@ <h2>R</h2>
 </li>
         <li><a href="api_reference.html#pybryt.annotations.relation.RelationalAnnotation">RelationalAnnotation (class in pybryt.annotations.relation)</a>
 </li>
-    </ul></td>
-    <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.Annotation.reset_tracked_annotations">reset_tracked_annotations() (Annotation static method)</a>
 </li>
+    </ul></td>
+    <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.ReferenceResult.results">results (ReferenceResult attribute)</a>
+</li>
+        <li><a href="api_reference.html#pybryt.annotations.value.Value.rtol">rtol (Value attribute)</a>
 </li>
         <li><a href="api_reference.html#pybryt.annotations.invariants.invariant.run">run() (invariant static method)</a>
 
@@ -451,11 +468,25 @@ <h2>T</h2>
     <td style="width: 33%; vertical-align: top;"><ul>
         <li><a href="api_reference.html#pybryt.AnnotationResult.timestamp">timestamp (AnnotationResult attribute)</a>
 </li>
+        <li><a href="api_reference.html#pybryt.ReferenceResult.to_array">to_array() (ReferenceResult method)</a>
+</li>
+        <li><a href="api_reference.html#pybryt.Annotation.to_dict">to_dict() (Annotation method)</a>
+
+        <ul>
+          <li><a href="api_reference.html#pybryt.AnnotationResult.to_dict">(AnnotationResult method)</a>
+</li>
+          <li><a href="api_reference.html#pybryt.annotations.value.Attribute.to_dict">(Attribute method)</a>
+</li>
+          <li><a href="api_reference.html#pybryt.ReferenceResult.to_dict">(ReferenceResult method)</a>
+</li>
+          <li><a href="api_reference.html#pybryt.annotations.value.Value.to_dict">(Value method)</a>
+</li>
+        </ul></li>
     </ul></td>
     <td style="width: 33%; vertical-align: top;"><ul>
-        <li><a href="api_reference.html#pybryt.ReferenceResult.to_array">to_array() (ReferenceResult method)</a>
+        <li><a href="api_reference.html#pybryt.execution.tracing_off">tracing_off() (in module pybryt.execution)</a>
 </li>
-        <li><a href="api_reference.html#pybryt.annotations.value.Value.tol">tol (Value attribute)</a>
+        <li><a href="api_reference.html#pybryt.execution.tracing_on">tracing_on() (in module pybryt.execution)</a>
 </li>
     </ul></td>
   </tr></table>
@@ -518,7 +549,5 @@ <h2>X</h2>
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/getting_started.html b/docs/html/getting_started.html
index 63edce5..7254575 100644
--- a/docs/html/getting_started.html
+++ b/docs/html/getting_started.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="Annotations" href="annotations/index.html" /><link rel="prev" title="PyBryt Documentation" href="index.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Getting Started - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -327,7 +327,5 @@ <h2>Next Steps<a class="headerlink" href="#next-steps" title="Permalink to this
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/index.html b/docs/html/index.html
index fc0eaa6..8c9c6db 100644
--- a/docs/html/index.html
+++ b/docs/html/index.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="Getting Started" href="getting_started.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Home - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -135,7 +135,10 @@
 <h1>PyBryt Documentation<a class="headerlink" href="#pybryt-documentation" title="Permalink to this headline">¶</a></h1>
 <div class="toctree-wrapper compound">
 </div>
-<p>PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to empower students and educators to learn about technology through fun, guided, hands-on content aimed at specific learning goals. PyBryt is designed to work with existing autograding solutions and workflows such as <a class="reference external" href="https://otter-grader.readthedocs.io">Otter Grader</a>, <a class="reference external" href="https://okpy.org">OkPy</a>, and <a class="reference external" href="https://autolab.readthedocs.io/">Autolab</a>.</p>
+<p>PyBryt is an open source  Python auto-assessment library for teaching and learning. Our goal is to
+empower students and educators to learn about technology through fun, guided, hands-on content aimed
+at specific learning goals. PyBryt is designed to work with existing autograding solutions and
+workflows such as <a class="reference external" href="https://otter-grader.readthedocs.io">Otter Grader</a>, <a class="reference external" href="https://okpy.org">OkPy</a>, and <a class="reference external" href="https://autolab.readthedocs.io/">Autolab</a>.</p>
 <img alt="_images/pybryt_goals.png" src="_images/pybryt_goals.png"/>
 <p>Educators and institutions can leverage PyBryt to integrate auto-assessment and reference models to hands-on labs and assessments.</p>
 <ul class="simple">
@@ -199,7 +202,5 @@ <h1>PyBryt Documentation<a class="headerlink" href="#pybryt-documentation" title
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/objects.inv b/docs/html/objects.inv
index fd1fd76..735a078 100644
Binary files a/docs/html/objects.inv and b/docs/html/objects.inv differ
diff --git a/docs/html/py-modindex.html b/docs/html/py-modindex.html
index 30a8f39..4e156a5 100644
--- a/docs/html/py-modindex.html
+++ b/docs/html/py-modindex.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title> - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -43,7 +43,5 @@
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/reference_implementations.html b/docs/html/reference_implementations.html
index ed84bc1..9092f50 100644
--- a/docs/html/reference_implementations.html
+++ b/docs/html/reference_implementations.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="Student Implementations" href="student_implementations.html" /><link rel="prev" title="Invariants" href="annotations/invariants.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Reference Implementations - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -359,7 +359,5 @@ <h2>Interacting with Reference Implementations<a class="headerlink" href="#inter
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/docs/html/search.html b/docs/html/search.html
index 3181c30..04e8e40 100644
--- a/docs/html/search.html
+++ b/docs/html/search.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="#" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/><title>Search - PyBryt  documentation</title><link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/><title>Search - PyBryt  documentation</title><link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
     <link media="(prefers-color-scheme: dark)" rel="stylesheet" href="_static/pygments_dark.css">
     
@@ -170,8 +170,6 @@
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     
 <script src="_static/searchtools.js"></script>
 <script src="_static/language_data.js"></script><script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script>
diff --git a/docs/html/searchindex.js b/docs/html/searchindex.js
index 732133b..cb214bd 100644
--- a/docs/html/searchindex.js
+++ b/docs/html/searchindex.js
@@ -1,1 +1,1 @@
-Search.setIndex({docnames:["annotations/index","annotations/invariants","annotations/relational_annotations","api_reference","getting_started","index","reference_implementations","student_implementations"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":1,sphinx:56},filenames:["annotations/index.rst","annotations/invariants.rst","annotations/relational_annotations.rst","api_reference.rst","getting_started.rst","index.rst","reference_implementations.rst","student_implementations.rst"],objects:{"pybryt.Annotation":{after:[3,1,1,""],before:[3,1,1,""],check:[3,1,1,""],children:[3,1,1,""],failure_message:[3,2,1,""],get_tracked_annotations:[3,1,1,""],group:[3,2,1,""],limit:[3,2,1,""],name:[3,2,1,""],reset_tracked_annotations:[3,1,1,""],success_message:[3,2,1,""]},"pybryt.AnnotationResult":{annotation:[3,2,1,""],children:[3,2,1,""],group:[3,1,1,""],messages:[3,1,1,""],name:[3,1,1,""],satisfied:[3,1,1,""],satisfied_at:[3,1,1,""],timestamp:[3,2,1,""],value:[3,1,1,""]},"pybryt.ReferenceImplementation":{annotations:[3,2,1,""],compile:[3,1,1,""],dump:[3,1,1,""],load:[3,1,1,""],run:[3,1,1,""]},"pybryt.ReferenceResult":{correct:[3,1,1,""],group:[3,2,1,""],messages:[3,1,1,""],reference:[3,2,1,""],results:[3,2,1,""],to_array:[3,1,1,""]},"pybryt.StudentImplementation":{check:[3,1,1,""],check_plagiarism:[3,1,1,""],dump:[3,1,1,""],dumps:[3,1,1,""],load:[3,1,1,""],loads:[3,1,1,""],nb:[3,2,1,""],steps:[3,2,1,""],values:[3,2,1,""]},"pybryt.annotations":{invariants:[3,3,0,"-"],relation:[3,3,0,"-"],value:[3,3,0,"-"]},"pybryt.annotations.invariants":{invariant:[3,0,1,""],string_capitalization:[3,0,1,""]},"pybryt.annotations.invariants.invariant":{run:[3,1,1,""]},"pybryt.annotations.invariants.string_capitalization":{run:[3,1,1,""]},"pybryt.annotations.relation":{AndAnnotation:[3,0,1,""],BeforeAnnotation:[3,0,1,""],NotAnnotation:[3,0,1,""],OrAnnotation:[3,0,1,""],RelationalAnnotation:[3,0,1,""],XorAnnotation:[3,0,1,""]},"pybryt.annotations.relation.AndAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.BeforeAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.NotAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.OrAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.RelationalAnnotation":{check:[3,1,1,""],children:[3,1,1,""]},"pybryt.annotations.relation.XorAnnotation":{check:[3,1,1,""]},"pybryt.annotations.value":{Attribute:[3,0,1,""],Value:[3,0,1,""]},"pybryt.annotations.value.Attribute":{check:[3,1,1,""],children:[3,1,1,""]},"pybryt.annotations.value.Value":{check:[3,1,1,""],children:[3,1,1,""],intial_value:[3,2,1,""],invariants:[3,2,1,""],tol:[3,2,1,""]},pybryt:{Annotation:[3,0,1,""],AnnotationResult:[3,0,1,""],ReferenceImplementation:[3,0,1,""],ReferenceResult:[3,0,1,""],StudentImplementation:[3,0,1,""]}},objnames:{"0":["py","class","Python class"],"1":["py","method","Python method"],"2":["py","attribute","Python attribute"],"3":["py","module","Python module"]},objtypes:{"0":"py:class","1":"py:method","2":"py:attribute","3":"py:module"},terms:{"abstract":[0,2,3,5,6,7],"boolean":[0,3],"case":[0,7],"class":[0,2,3,4,5,6,7],"default":[0,2,6,7],"float":3,"function":[3,4,6,7],"import":4,"int":[2,3,4,6],"new":6,"return":[2,3,4,6,7],"static":[3,6],"switch":2,"while":[0,3,7],For:[0,2,3,7],The:[0,2,3,4,6,7],There:[0,6],These:6,Uses:3,__init__:6,__new__:3,_tracked_annot:3,abl:2,about:[0,4,5],abov:2,absolut:[0,3,7],accept:[0,3],access:7,across:7,activ:4,actual:7,add:[4,6],added:3,addit:3,addl_filenam:[3,7],advanc:6,affect:0,after:[2,3,4,6],against:3,aim:5,algorithm:2,all:[0,2,3,6],allow:[0,2,3,7],also:[0,2,4,7],analysi:5,andannot:3,ani:[3,4,6],annot:[5,6,7],annotation_result:3,annotationresult:3,anoth:[2,3],ans:6,api:[4,6],appear:[0,2,3],append:6,applic:3,arg:[2,3],argument:[0,3,7],arithemt:3,arr:0,arrai:[2,3],assert:[2,3,4],assess:[4,5],assign:6,attr:3,attribut:3,auto:5,autograd:5,autolab:5,autom:4,automat:4,avail:3,awai:[6,7],bad:4,base:[3,7],basic:0,becaus:[6,7],been:[2,3,6],befor:[2,3,4,6],beforeannot:[2,3],begin:6,behavior:[0,5],being:[0,3,6,7],belong:3,below:[0,6],between:2,beyond:4,bin:4,bitwis:[2,3],black:4,block:0,bool:3,both:4,box:4,build:0,built:6,builtin:4,call:[0,2,3,6,7],callabl:3,can:[2,3,4,5,6,7],capit:3,check:[2,3,4,6],check_plagiar:3,child:[2,3],children:3,classmethod:3,code:[0,2,3,5,6,7],collect:[3,6],combin:0,come:2,compar:[3,5],compil:[3,6],complet:4,complex:[0,5],compris:3,comput:2,computation:7,conda:4,condit:[0,2,3,5,6],configur:3,consid:[0,2,3,6],constraint:2,construct:[0,1,6,7],constructor:[0,2,3,4,6,7],consum:7,contain:[2,3,7],content:5,convert:3,copi:[0,3],core:[5,6],correct:[2,3],correct_answ:0,correctli:6,correspond:[2,7],could:4,count:4,count_word:4,counter:3,creat:[0,2,3,4,5,7],create_refer:3,creation:3,curr_val:[2,6],current:[1,2],custom:[3,7],data:3,debug:7,decreas:3,def:[2,4,6],defin:[0,2,3,5,6],demo:4,describ:[0,2],design:5,dest:3,detail:4,detect:[4,5],determin:[3,6],differ:[0,2,7],dill:[3,6],directli:2,doesn:6,don:0,down:6,dtype:[2,6],dump:[3,6,7],dunder:2,dure:3,dynam:[2,6],each:[2,3,7],easi:5,easier:7,educ:5,either:[2,3,7],element:3,emploi:7,empow:5,empti:[3,4],encod:[3,6,7],enforc:[2,5],ensur:[2,3],environ:[4,6],equal:[0,2,3],evalu:3,everi:7,exact:7,exampl:[2,3,6,7],execut:[0,3,5,6],executepreprocessor:7,exist:[3,5],expect:[0,3,6],expens:7,fail:0,failure_messag:3,feedback:5,fib:[2,6],fib_map:6,fibonacci:[2,6],field:3,file:[3,6,7],filenam:[3,6],final_answ:6,find:6,first:[2,3,4,6],first_ref:6,five:4,folder:4,follow:[2,4],footprint:[3,7],form:0,found:[5,6],fragment:4,from:[0,2,3,4,6],fun:5,further:6,gener:[3,6,7],get_impl_result:3,get_tracked_annot:3,given:4,global:[3,6],goal:5,grade:5,grader:5,greater:[2,3],group:[3,6,7],guid:5,hand:5,handl:[3,7],har:7,has:[2,3,6],hash:6,have:[2,3,4,5,6],help:2,ignor:3,implemen:6,implement:[0,4,5],implenet:6,includ:0,indic:3,infrastructur:5,inherit:3,initi:3,input:4,insensit:0,insert:7,insid:[3,7],instanc:[0,2,3,4,5,6],instani:0,instanti:2,institut:5,instructor:5,integ:3,integr:5,interact:3,intial_valu:3,ipynb:[6,7],item:4,iter:[0,2],its:[3,4],itself:2,jig:4,judg:4,jupyt:[3,6,7],kei:4,keyword:3,kind:[0,2,3],kwarg:3,lab:5,lack:0,later:6,learn:[4,5],learner:5,leav:3,len:4,length:[3,4],less:3,leverag:5,librari:[5,6],limit:3,line:6,list:[0,3,4,6,7],load:[3,6,7],logic:[0,2,3],look:[4,5],loop:2,lowercas:3,made:6,magic:4,mai:2,main:0,maintain:[3,6],make:[3,4,7],manag:[3,7],mang:3,map:[3,6],mark:[3,6],match:3,math:0,matrix:3,maximum:3,mean:6,meaning:5,meet:0,memori:[2,3,7],messag:3,met:3,method:[2,3,4,6,7],might:7,model:[4,5],more:[2,3,4,6],most:0,much:4,multipl:[6,7],must:[3,6],mutabl:0,n_fib:6,name:[3,7],nbformat:[3,7],ndarrai:[2,3],necessari:[3,6],need:[0,3],next:2,njump:4,non:3,none:3,noreturn:3,normal:0,notannot:3,note:[0,2],notebook:[3,6],notebooknod:3,number:[2,3,4],numbr:2,numer:3,numpi:3,nymph:4,obj:3,object:[0,3,5,6,7],observ:[3,4,7],observed_valu:3,occur:[2,3,7],okpi:5,one:[2,3,6],onli:[0,2,3,6],open:5,oper:[2,3,5],operand:2,option:3,orannot:3,order:3,organiz:5,origin:3,other:3,other_annot:3,otherwis:3,otter:5,our:5,out:[0,4],output:[3,7],overrid:2,own:[3,4],page:1,paramet:3,pass:[0,3,4,6,7],passag:3,path:[2,3,6,7],path_or_nb:3,pattern:[5,7],pedagog:5,perspect:0,pickl:3,picklabl:3,piec:4,pip:4,pkl:[3,6],plagiar:[3,4,5],pointer:2,popul:3,possibl:[0,2],practic:5,preprocessor:7,presenc:[0,2],present:[2,3],print:7,problem:[2,5],proce:7,process:5,program:[2,5,6],project:4,properti:3,provid:[0,2,3,5,7],ps1:4,pybryt:[0,2,3,6,7],pybyrt:4,python3:4,python:[2,3,4,5,7],quartz:4,question:[3,7],quick:4,quickli:4,rais:3,random:0,randomli:3,rang:[2,6],read:[4,7],recent:4,recommend:2,reconcil:[6,7],ref1:6,ref2:6,ref:[3,6,7],refer:[0,4,5,7],referenceimplement:[3,5,6,7],referenceresult:[3,7],rel:2,relai:3,relat:0,relationalannot:[2,3],relationship:[0,2],remov:4,repres:0,requir:[3,4],reset:3,reset_tracked_annot:3,respond:0,result:[4,6],run:[3,6,7],same:[2,3,7],sampl:4,satisfi:[2,3],satisfied_at:3,save:6,script:[3,4,7],search:6,second:[2,3,6],second_ref:6,see:[0,3,7],seed:0,self:3,send:4,sequenc:[2,6],seri:[3,5,6],serial:[6,7],session:7,set:[0,3,6,7],should:[0,2,3,7],show:7,signatur:4,simpl:[3,4],simpli:[2,7],singl:[0,3,6,7],singleton:[3,6],size:0,solut:[2,5],some:[2,3,4,5,6],sourc:[4,5],space:4,special:7,specif:[0,2,3,5,7],specifi:3,sphinx:4,split:4,stage:7,statement:7,step:3,still:6,store:3,str:[3,4],string:[0,3,4,7],string_capit:[0,3],structur:[0,5],stu:7,student:[0,2,4,5,6],student_impl:3,studentimplement:[3,6,7],studi:3,subclass:[0,2,3,5],subm:7,submiss:[3,7],success_messag:3,support:[0,2,3,5],sure:4,surround:0,tak:3,take:[0,2,3,4,6,7],task:4,teach:5,technolog:5,tempor:[0,3],test:[4,7],text:[3,4],than:[2,3],thei:[0,3,6],them:[3,4,6,7],thereof:0,thi:[1,2,3,4,5,6,7],third:3,those:[0,3,5,6,7],through:[3,5],time:7,timestamp:[2,3,7],to_arrai:3,tol:[0,3],toler:3,trace:[3,7],track:[3,6,7],train:4,transpos:3,transposit:3,tupl:[3,6,7],turn:6,two:[0,2,3,4,6],type:[0,3,4,7],unchang:3,under:[1,3],union:3,unit:6,unless:6,unnecessari:6,unpickl:3,unsatisfi:3,updat:[2,4],use:[0,2,4,6,7],used:[2,3,4,7],useful:7,uses:6,using:[0,2,3,4,7],valu:[2,4,5,6,7],valueerror:3,variabl:[2,4,6],variou:7,vector:0,venv:4,version:[4,7],vex:4,virtual:4,vow:4,wai:6,waltz:4,want:4,well:3,were:[3,4,7],what:4,when:[0,2,3,6],where:3,whether:[3,6],which:[0,2,3,4,6,7],within:[4,5],wizard:4,word:4,work:[2,3,5,6],workflow:5,worri:0,would:[3,4,7],wrangl:3,write:[2,3,4,7],written:[5,7],xor:2,xorannot:[2,3],you:[0,4],your:4,zero:[0,2,6]},titles:["Annotations","Invariants","Relational Annotations","API Reference","Getting Started","PyBryt Documentation","Reference Implementations","Student Implementations"],titleterms:{"boolean":2,annot:[0,2,3,4],api:3,automat:6,check:7,command:4,creat:6,creation:6,custom:[],document:5,exampl:4,execut:7,exist:4,get:4,implement:[3,6,7],insid:4,instal:4,interact:6,invari:[0,1,3],jupyt:4,line:4,linux:4,maco:4,manual:6,next:4,notebook:[4,7],numer:0,powershel:4,pybryt:[4,5],refer:[3,6],relat:[2,3],result:3,start:4,step:4,store:[6,7],student:[3,7],tempor:2,toler:0,valu:[0,3],window:4,work:4}})
\ No newline at end of file
+Search.setIndex({docnames:["annotations/index","annotations/invariants","annotations/relational_annotations","api_reference","getting_started","index","reference_implementations","student_implementations"],envversion:{"sphinx.domains.c":2,"sphinx.domains.changeset":1,"sphinx.domains.citation":1,"sphinx.domains.cpp":3,"sphinx.domains.index":1,"sphinx.domains.javascript":2,"sphinx.domains.math":2,"sphinx.domains.python":2,"sphinx.domains.rst":2,"sphinx.domains.std":2,sphinx:56},filenames:["annotations/index.rst","annotations/invariants.rst","annotations/relational_annotations.rst","api_reference.rst","getting_started.rst","index.rst","reference_implementations.rst","student_implementations.rst"],objects:{"pybryt.Annotation":{after:[3,1,1,""],before:[3,1,1,""],check:[3,1,1,""],children:[3,1,1,""],failure_message:[3,2,1,""],get_tracked_annotations:[3,1,1,""],group:[3,2,1,""],limit:[3,2,1,""],name:[3,2,1,""],reset_tracked_annotations:[3,1,1,""],success_message:[3,2,1,""],to_dict:[3,1,1,""]},"pybryt.AnnotationResult":{annotation:[3,2,1,""],children:[3,2,1,""],group:[3,1,1,""],messages:[3,1,1,""],name:[3,1,1,""],satisfied:[3,1,1,""],satisfied_at:[3,1,1,""],timestamp:[3,2,1,""],to_dict:[3,1,1,""],value:[3,1,1,""]},"pybryt.ReferenceImplementation":{annotations:[3,2,1,""],compile:[3,1,1,""],dump:[3,1,1,""],load:[3,1,1,""],run:[3,1,1,""]},"pybryt.ReferenceResult":{correct:[3,1,1,""],group:[3,2,1,""],messages:[3,1,1,""],reference:[3,2,1,""],results:[3,2,1,""],to_array:[3,1,1,""],to_dict:[3,1,1,""]},"pybryt.StudentImplementation":{check:[3,1,1,""],check_plagiarism:[3,1,1,""],dump:[3,1,1,""],dumps:[3,1,1,""],load:[3,1,1,""],loads:[3,1,1,""],nb:[3,2,1,""],nb_path:[3,2,1,""],steps:[3,2,1,""],values:[3,2,1,""]},"pybryt.annotations":{invariants:[3,3,0,"-"],relation:[3,3,0,"-"],value:[3,3,0,"-"]},"pybryt.annotations.invariants":{invariant:[3,0,1,""],string_capitalization:[3,0,1,""]},"pybryt.annotations.invariants.invariant":{run:[3,1,1,""]},"pybryt.annotations.invariants.string_capitalization":{run:[3,1,1,""]},"pybryt.annotations.relation":{AndAnnotation:[3,0,1,""],BeforeAnnotation:[3,0,1,""],NotAnnotation:[3,0,1,""],OrAnnotation:[3,0,1,""],RelationalAnnotation:[3,0,1,""],XorAnnotation:[3,0,1,""]},"pybryt.annotations.relation.AndAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.BeforeAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.NotAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.OrAnnotation":{check:[3,1,1,""]},"pybryt.annotations.relation.RelationalAnnotation":{check:[3,1,1,""],children:[3,1,1,""]},"pybryt.annotations.relation.XorAnnotation":{check:[3,1,1,""]},"pybryt.annotations.value":{Attribute:[3,0,1,""],Value:[3,0,1,""]},"pybryt.annotations.value.Attribute":{check:[3,1,1,""],children:[3,1,1,""],to_dict:[3,1,1,""]},"pybryt.annotations.value.Value":{atol:[3,2,1,""],check:[3,1,1,""],check_values_equal:[3,1,1,""],children:[3,1,1,""],intial_value:[3,2,1,""],invariants:[3,2,1,""],rtol:[3,2,1,""],to_dict:[3,1,1,""]},"pybryt.execution":{tracing_off:[3,4,1,""],tracing_on:[3,4,1,""]},pybryt:{Annotation:[3,0,1,""],AnnotationResult:[3,0,1,""],ReferenceImplementation:[3,0,1,""],ReferenceResult:[3,0,1,""],StudentImplementation:[3,0,1,""],execution:[3,3,0,"-"]}},objnames:{"0":["py","class","Python class"],"1":["py","method","Python method"],"2":["py","attribute","Python attribute"],"3":["py","module","Python module"],"4":["py","function","Python function"]},objtypes:{"0":"py:class","1":"py:method","2":"py:attribute","3":"py:module","4":"py:function"},terms:{"100":0,"abstract":[0,2,3,5,6,7],"boolean":[0,3],"case":[0,7],"class":[0,2,3,4,5,6,7],"default":[0,2,6,7],"float":3,"function":[3,4,6,7],"import":4,"int":[2,3,4,6],"new":6,"null":[],"return":[2,3,4,6,7],"static":[3,6],"switch":2,"true":[],"while":[0,3,7],For:[0,2,3,7],The:[0,2,3,4,6,7],There:[0,6],These:6,Use:[],Uses:3,__init__:6,__len__:[],__new__:3,_collector_ret:[],_tracked_annot:3,abc:[],abl:2,about:[0,4,5],abov:2,absolut:[0,3,7],accept:[0,3],access:7,across:7,act:[],action:3,activ:4,actual:7,add:[4,6],added:3,addit:3,addl_filenam:[3,7],advanc:6,affect:0,after:[2,3,4,6],against:3,aim:5,algorithm:2,all:[0,2,3,6],allcos:3,allow:[0,2,3,7],also:[0,2,4,7],analysi:5,andannot:3,ani:[3,4,6],annot:[5,6,7],annotation_result:3,annotationresult:3,anoth:[2,3],ans:6,api:[4,6],appear:[0,2,3],append:6,applic:3,arang:[],architectur:[],arg:[2,3],argument:[0,3,7],arithemt:3,arr:0,arrai:[2,3],assert:[2,3,4],assess:[4,5],assign:6,atol:3,attr:3,attribut:3,auto:5,autograd:5,autolab:5,autom:4,automat:4,avail:3,awai:[6,7],back:3,bad:4,base:[3,7],basic:0,becaus:[3,6,7],been:[2,3,6],befor:[2,3,4,6],beforeannot:[2,3],begin:6,behavior:[0,5],being:[0,3,6,7],belong:3,below:[0,3,6],best:[],between:2,beyond:4,big_o:[],bin:4,bitwis:[2,3],black:4,block:0,bool:3,both:4,box:4,build:0,built:6,builtin:4,call:[0,2,3,6,7],callabl:3,can:[2,3,4,5,6,7],capit:3,captur:[],cdot:3,check:[2,3,4,6],check_plagiar:3,check_time_complex:[],check_values_equ:3,child:[2,3],children:3,classmethod:3,closest:[],code:[0,2,3,5,6,7],collect:[3,6],com:[],combin:0,come:2,compar:[3,5],compelx:[],compil:[3,6],complet:4,complex:[0,5],complexity_class:[],complexity_data:[],complexityannot:[],compris:3,comput:[2,3],computation:7,conda:4,condit:[0,2,3,5,6],configur:3,consid:[0,2,3,6],constant:[],constraint:2,construct:[0,1,6,7],constructor:[0,2,3,4,6,7],consum:7,contain:[2,3,7],content:5,context:[],convert:3,copi:[0,3],core:[5,6],correct:[2,3],correct_answ:0,correctli:6,correspond:[2,7],could:4,count:4,count_word:4,counter:3,cplx:[],creat:[0,2,3,4,5,7],create_collector:[],create_refer:3,creation:3,cubic:[],curr_val:[2,6],current:[1,2],custom:[3,7],data:3,deactiv:3,debug:7,decreas:3,def:[2,3,4,6],defin:[0,2,3,5,6],demo:[3,4],describ:[0,2],design:5,dest:3,detail:[3,4],detect:[4,5],determin:[3,6],dict:3,dictionari:3,differ:[0,2,7],dill:[3,6],directli:2,doe:3,doesn:6,don:0,down:6,dtype:[2,6],dump:[3,6,7],dunder:2,dure:3,dynam:[2,6],each:[2,3,7],easi:5,easier:7,educ:5,either:[2,3,7],element:3,emploi:7,empow:5,empti:[3,4],encod:[3,6,7],enforc:[2,5],enough:3,ensur:[2,3],environ:[4,6],equal:[0,2,3],error:[],etc:[],evalu:3,everi:7,exact:7,exampl:[2,3,6,7],execut:[0,5,6],executepreprocessor:7,exist:[3,5],exit:[],exp:[],expect:[0,3,6],expens:7,exponenti:[],fail:0,failur:3,failure_messag:3,feedback:5,fib:[2,6],fib_map:6,fiber:[],fibonacci:[2,6],field:3,file:[3,6,7],filenam:[3,6],final_answ:6,find:6,first:[2,3,4,6],first_ref:6,fit:[],five:4,folder:4,follow:[2,4],footprint:[3,7],form:0,format:3,found:[5,6],fragment:4,frame:[],friendli:3,from:[0,2,3,4,6],from_footprint:[],full:[],fun:5,further:6,gener:[3,6,7],get_impl_result:3,get_tracked_annot:3,github:[],given:4,global:[3,6],goal:5,grade:5,grader:5,greater:[2,3],group:[3,6,7],guid:5,hand:5,handl:[3,7],har:7,has:[2,3,6],hash:6,have:[2,3,4,5,6],help:2,here:[],http:[],ignor:3,implemen:6,implement:[0,4,5],implenet:6,includ:[0,3],incorpor:[],increas:[],indic:3,inform:[],infrastructur:5,inherit:3,initi:3,input:4,insensit:0,insert:7,insid:[3,7],instanc:[0,2,3,4,5,6],instani:0,instanti:2,institut:5,instructor:5,integ:3,integr:5,interact:3,intern:3,intial_valu:3,ipynb:[6,7],item:4,iter:[0,2],its:[3,4],itself:2,jig:4,json:3,judg:4,jupyt:[3,6,7],just:[],kei:[3,4],keyword:3,kind:[0,2,3],kwarg:3,lab:5,lack:0,later:6,launch:[],learn:[4,5],learner:5,leav:3,len:4,length:[3,4],less:3,leverag:5,librari:[5,6],like:[],limit:3,linalg:[],line:6,linear:[],linearithm:[],list:[0,3,4,6,7],load:[3,6,7],log:[],logarithm:[],logic:[0,2,3],look:[4,5],loop:2,lowercas:3,lstsq:[],made:6,magic:4,mai:2,main:0,maintain:[3,6],make:[3,4,7],manag:[3,7],mang:3,map:[3,6],mark:[3,6],match:3,math:0,mathcal:[],matrix:3,maximum:3,mean:6,meaning:5,meet:0,memori:[2,3,7],messag:3,met:3,method:[2,3,4,6,7],might:7,model:[4,5],modul:[],more:[2,3,4,6],most:0,much:4,multipl:[6,7],must:[3,6],mutabl:0,n_fib:6,name:[3,7],nb_path:3,nbformat:[3,7],ndarrai:[2,3],necessari:[3,6],need:[0,3],next:2,njump:4,non:3,none:3,noreturn:3,normal:0,notannot:3,note:[0,2],notebook:[3,6],notebooknod:3,number:[2,3,4],numbr:2,numer:3,numpi:3,nymph:4,obj:3,object:[0,3,5,6,7],observ:[3,4,7],observed_valu:3,occur:[2,3,7],off:3,okpi:5,one:[2,3,6],onli:[0,2,3,6],open:5,oper:[2,3,5],operand:2,option:3,orannot:3,order:3,organiz:5,origin:3,other:3,other_annot:3,other_valu:3,otherwis:3,otter:5,our:5,out:[0,4],output:[3,7],outsid:[],overrid:2,own:[3,4],packag:[],page:1,paramet:3,part:3,pass:[0,3,4,6,7],passag:3,path:[2,3,6,7],path_or_nb:3,pattern:[5,7],pberk:[],pedagog:5,perspect:0,pickl:3,picklabl:3,piec:4,pip:4,pkl:[3,6],place:[],plagiar:[3,4,5],pointer:2,popul:3,posit:[],possibl:[0,2],pow:3,practic:5,prefer:[],preprocessor:7,presenc:[0,2],present:[2,3],print:7,problem:[2,5],proce:7,process:5,program:[2,5,6],project:4,properti:3,provid:[0,2,3,5,7],ps1:4,pybryt:[0,2,3,6,7],pybyrt:4,python3:4,python:[2,3,4,5,7],quadrat:[],quartz:4,question:[3,7],quick:4,quickli:4,rais:3,random:0,randomli:3,rang:[2,6],rather:[],read:[4,7],recent:4,recommend:2,reconcil:[6,7],ref1:6,ref2:6,ref:[3,6,7],refer:[0,4,5,7],referenceimplement:[3,5,6,7],referenceresult:[3,7],rel:[2,3],relai:3,relat:0,relationalannot:[2,3],relationship:[0,2],remov:4,report:[],report_on_error:[],repres:0,represent:3,requir:[3,4],reset:3,reset_tracked_annot:3,residu:[],respond:0,result:[4,6],rtol:3,run:[3,6,7],runtim:[],same:[2,3,7],sampl:4,satisfi:[2,3],satisfied_at:3,satisifi:[],save:6,save_func:[],script:[3,4,7],search:6,second:[2,3,6],second_ref:6,see:[0,3,7],seed:0,select:[],self:3,send:4,sent:[],sequenc:[2,6],seri:[3,5,6],serial:[6,7],session:7,set:[0,3,6,7],should:[0,2,3,7],shouldn:3,show:7,show_onli:[],signatur:4,similar:[],simpl:[3,4],simpler:[],simpli:[2,7],simplic:[],singl:[0,3,6,7],singleton:[3,6],size:0,slight:[],smallest:[],solut:[2,5],some:[2,3,4,5,6],sort:[],sourc:[4,5],space:4,special:7,specif:[0,2,3,5,7],specifi:3,sphinx:4,split:4,stack:3,stage:7,statement:7,step:3,still:6,stolen:[],store:3,str:[3,4],string:[0,3,4,7],string_capit:[0,3],structur:[0,5],stu:7,student:[0,2,4,5,6],student_impl:3,studentimplement:[3,6,7],studi:3,subclass:[0,2,3,5],subm:7,submiss:[3,7],success:3,success_messag:3,support:[0,2,3,5],sure:4,surround:0,tak:3,take:[0,2,3,4,6,7],taken:3,task:4,teach:5,technolog:5,tempor:[0,3],test:[4,7],text:[3,4],texttt:3,than:[2,3],thei:[0,3,6],them:[3,4,6,7],thereof:0,thi:[1,2,3,4,5,6,7],third:3,those:[0,3,5,6,7],through:[3,5],thrown:[],time:7,timecomplex:[],timecomplexityresult:[],timestamp:[2,3,7],to_arrai:3,to_dict:3,tol:0,toler:3,trace:[3,7],tracing_func:[],tracing_off:3,tracing_on:3,track:[3,6,7],train:4,transform:[],transform_n:[],transform_t:[],transpos:3,transposit:3,tupl:[3,6,7],turn:[3,6],two:[0,2,3,4,6],type:[0,3,4,7],typecontextmanag:[],unchang:3,under:[1,3],ungrad:3,uniform:[],union:3,uniqu:[],unit:6,unless:6,unnecessari:6,unpickl:3,unsatisfi:3,unspecifi:[],updat:[2,4],upon:[],use:[0,2,4,6,7],used:[2,3,4,7],useful:7,uses:6,using:[0,2,3,4,7],valu:[2,4,5,6,7],valueerror:3,variabl:[2,4,6],variou:7,vector:0,venv:4,version:[4,7],vex:4,virtual:4,vow:4,wai:6,waltz:4,want:4,well:3,were:[3,4,7],what:4,when:[0,2,3,6],where:3,whether:[3,6],which:[0,2,3,4,6,7],within:[3,4,5],wizard:4,word:4,work:[2,3,5,6],workflow:5,worri:0,would:[3,4,7],wrangl:3,write:[2,3,4,7],written:[5,7],xor:2,xorannot:[2,3],you:[0,4],your:4,zero:[0,2,6]},titles:["Annotations","Invariants","Relational Annotations","API Reference","Getting Started","PyBryt Documentation","Reference Implementations","Student Implementations"],titleterms:{"boolean":2,"class":[],annot:[0,2,3,4],api:3,automat:6,check:7,code:[],command:4,complex:[],creat:6,creation:6,document:5,evalu:[],exampl:4,execut:[3,7],exist:4,get:4,implement:[3,6,7],insid:4,instal:4,interact:6,invari:[0,1,3],jupyt:4,line:4,linux:4,maco:4,manual:6,next:4,notebook:[4,7],numer:0,powershel:4,pybryt:[4,5],refer:[3,6],relat:[2,3],result:3,start:4,step:4,store:[6,7],student:[3,7],tempor:2,toler:0,track:[],valu:[0,3],window:4,work:4}})
\ No newline at end of file
diff --git a/docs/html/student_implementations.html b/docs/html/student_implementations.html
index d63b660..e3f81c4 100644
--- a/docs/html/student_implementations.html
+++ b/docs/html/student_implementations.html
@@ -3,7 +3,7 @@
   <head><meta charset="utf-8"/>
     <meta name="viewport" content="width=device-width,initial-scale=1"/><link rel="index" title="Index" href="genindex.html" /><link rel="search" title="Search" href="search.html" /><link rel="next" title="API Reference" href="api_reference.html" /><link rel="prev" title="Reference Implementations" href="reference_implementations.html" />
 
-    <meta name="generator" content="sphinx-3.1.2, furo 2021.02.21.beta25"/>
+    <meta name="generator" content="sphinx-3.5.1, furo 2021.02.21.beta25"/>
         <title>Student Implementations - PyBryt documentation</title>
       <link rel="stylesheet" href="_static/styles/furo.css?digest=33d2fc4f3f180ec1ffc6524e273e21d7d58cbe49">
     <link rel="stylesheet" href="_static/pygments.css">
@@ -279,7 +279,5 @@ <h2>Storing Implementations<a class="headerlink" href="#storing-implementations"
     <script src="_static/jquery.js"></script>
     <script src="_static/underscore.js"></script>
     <script src="_static/doctools.js"></script>
-    <script src="_static/language_data.js"></script>
-    <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     <script src="_static/scripts/main.js?digest=e931d09b2a40c1bb82b542effe772014573baf67"></script></body>
 </html>
\ No newline at end of file
diff --git a/pybryt/annotations/value.py b/pybryt/annotations/value.py
index 48189a4..ff421b8 100644
--- a/pybryt/annotations/value.py
+++ b/pybryt/annotations/value.py
@@ -8,7 +8,7 @@
 
 from collections.abc import Iterable
 from copy import copy
-from typing import Any, Dict, List, Tuple, Union
+from typing import Any, Dict, List, Optional, Tuple, Union
 
 from .annotation import Annotation, AnnotationResult
 from .invariants import invariant
@@ -24,9 +24,14 @@ class Value(Annotation):
     list of :ref:`invariants<invariants>` that will allow objects to be considered "equal." For 
     values that support arithemtic operators, absolute tolerances can be specified as well.
 
+    Numeric tolerances are computing as with ``numpy.allcose``, where the value is considered "equal 
+    enough" if it is within :math:`v \\pm (\\texttt{atol} + \\texttt{rtol} \cdot |v|)`, where 
+    :math:`v` is the value of the annotation.
+
     Args:
         value (``object``): the value that should be observed
-        tol (``float`` or ``int``, optional): absolute tolerance for numeric values
+        atol (``float`` or ``int``, optional): absolute tolerance for numeric values
+        rtol (``float`` or ``int``, optional): absolute tolerance for numeric values
         invariants (``list[invariant]``): invariants for 
             this value
         **kwargs: additional keyword arguments passed to the 
@@ -42,13 +47,19 @@ class Value(Annotation):
     specified in the constructor
     """
 
-    tol: Union[float, int]
+    atol: Optional[Union[float, int]]
     """absolute tolerance for numeric values"""
 
+    rtol: Optional[Union[float, int]]
+    """relative tolerance for numeric values"""
+
     invariants: List[invariant]
     """the invariants for this value"""
 
-    def __init__(self, value: Any, tol: Union[float, int] = 0, invariants: List[invariant] = [], **kwargs):
+    def __init__(
+        self, value: Any, atol: Optional[Union[float, int]] = None, rtol: Optional[Union[float, int]] = None, 
+        invariants: List[invariant] = [], **kwargs
+    ):
         try:
             dill.dumps(value)
         except Exception as e:
@@ -56,7 +67,8 @@ def __init__(self, value: Any, tol: Union[float, int] = 0, invariants: List[inva
 
         self.initial_value = copy(value)
         self._values = [self.initial_value]
-        self.tol = tol
+        self.atol = atol
+        self.rtol = rtol
         self.invariants = invariants
 
         for inv in self.invariants:
@@ -74,7 +86,7 @@ def to_dict(self) -> Dict[str, Any]:
 
         Output dictionary contains the annotation's name, group, limit number, success message, and
         failure message, as well as an ``invariants`` key with a list of the names of all invariants
-        used in this value annotation and a ``tol`` key with this annotation's tolerance. This 
+        used in this value annotation and a ``atol`` key with this annotation's tolerance. This 
         dictionary does *not* contain the value being tracked.
 
         Returns:
@@ -83,7 +95,8 @@ def to_dict(self) -> Dict[str, Any]:
         d = super().to_dict()
         d.update({
             "invariants": [inv.__name__ for inv in self.invariants],
-            "tol": self.tol,
+            "atol": self.atol,
+            "rtol": self.rtol,
         })
         return d
 
@@ -124,7 +137,8 @@ def __eq__(self, other: Any) -> bool:
             ``bool``: whether the objects are equal
         """
         return isinstance(other, type(self)) and self.invariants == other.invariants and \
-            self.check_values_equal(self.initial_value, other.initial_value) and self.tol == other.tol
+            self.check_values_equal(self.initial_value, other.initial_value) and \
+            self.atol == other.atol and self.rtol == other.rtol
 
     def _check_observed_value(self, observed_value: Tuple[Any, int]) -> bool:
         """
@@ -147,15 +161,20 @@ def _check_observed_value(self, observed_value: Tuple[Any, int]) -> bool:
             for other_value in other_values:
                 # if type(value) != type(other_value):
                 #     continue
-                if self.check_values_equal(value, other_value, self.tol):
+                if self.check_values_equal(value, other_value, self.atol, self.rtol):
                     return True
 
         return False
 
     @staticmethod
-    def check_values_equal(value, other_value, tol=0):
+    def check_values_equal(value, other_value, atol = None, rtol = None):
         """
         """
+        if atol is None:
+            atol = 0
+        if rtol is None:
+            rtol = 0
+
         if hasattr(value, 'shape'):
             try:
                 if value.shape != other_value.shape:
@@ -164,7 +183,9 @@ def check_values_equal(value, other_value, tol=0):
                 return False
 
         try:
-            ub, lb = value + tol, value - tol
+            ub, lb = value + atol, value - atol
+            ub += rtol * np.abs(value)
+            lb -= rtol * np.abs(value)
             numeric = True
         
         except:
@@ -275,7 +296,8 @@ class Attribute(Annotation):
 
     _annotations: List[_AttrValue]
     _invariants: List[invariant]
-    _tol: float
+    _atol: Optional[Union[float, int]]
+    _rtol: Optional[Union[float, int]]
 
     def __init__(self, obj: Any, attrs: Union[str, List[str]], **kwargs):
         if isinstance(attrs, str):
@@ -293,7 +315,8 @@ def __init__(self, obj: Any, attrs: Union[str, List[str]], **kwargs):
             self._annotations.append(_AttrValue(obj, attr, **kwargs))
         
         self._invariants = kwargs.pop("invariants", [])
-        self._tol = kwargs.pop("tol", 0)
+        self._atol = kwargs.pop("atol", None)
+        self._rtol = kwargs.pop("rtol", None)
     
         super().__init__(name=name, success_message=success_message, failure_message=failure_message,
             **kwargs)
@@ -323,7 +346,7 @@ def to_dict(self) -> Dict[str, Any]:
 
         Output dictionary contains the annotation's name, group, limit number, success message, and
         failure message, as well as an ``invariants`` key with a list of the names of all invariants
-        used in this value annotation, a ``tol`` key with this annotation's tolerance, and a key for 
+        used in this value annotation, a ``atol`` key with this annotation's tolerance, and a key for 
         the attributes being checked. This dictionary does *not* contain the value being tracked.
 
         Returns:
@@ -332,7 +355,8 @@ def to_dict(self) -> Dict[str, Any]:
         d = super().to_dict()
         d.update({
             "invariants": [inv.__name__ for inv in self._invariants],
-            "tol": self._tol,
+            "atol": self._atol,
+            "rtol": self._rtol,
             "attributes": [av._attr for av in self._annotations],
         })
         return d


Test patch:
diff --git a/tests/annotations/test_relation.py b/tests/annotations/test_relation.py
index 5eae43a..64ef949 100644
--- a/tests/annotations/test_relation.py
+++ b/tests/annotations/test_relation.py
@@ -49,7 +49,8 @@ def test_before_annotation():
                 'children': [], 
                 'type': 'value', 
                 'invariants': [], 
-                'tol': 0
+                'atol': None,
+                'rtol': None,
             }, {
                 'name': 
                 'Annotation 2', 
@@ -60,7 +61,8 @@ def test_before_annotation():
                 'children': [], 
                 'type': 'value', 
                 'invariants': [], 
-                'tol': 0
+                'atol': None,
+                'rtol': None,
             }
         ], 
         'type': 'before',
diff --git a/tests/annotations/test_value.py b/tests/annotations/test_value.py
index 5dbe1c3..292ed73 100644
--- a/tests/annotations/test_value.py
+++ b/tests/annotations/test_value.py
@@ -51,7 +51,8 @@ def test_value_annotation():
         "limit": None,
         "group": None,
         "invariants": [],
-        "tol": 0,
+        "atol": None,
+        "rtol": None,
         "type": "value",
     }
     assert repr(res) == "AnnotationResult(satisfied=False, annotation=pybryt.Value)"
@@ -111,7 +112,8 @@ def test_attribute_annotation():
                 'failure_message': None, 
                 'children': [], 
                 'invariants': [], 
-                'tol': 0,
+                'atol': None,
+                'rtol': None,
                 "type": None,
             }
         ],
@@ -120,7 +122,8 @@ def test_attribute_annotation():
         "limit": None,
         "group": None,
         "invariants": [],
-        "tol": 0,
+        "atol": None,
+        "rtol": None,
         "type": "attribute",
         "attributes": ['T'],
     }
diff --git a/tests/files/expected_ref.pkl b/tests/files/expected_ref.pkl
index 4d38f01..c58f68f 100644
Binary files a/tests/files/expected_ref.pkl and b/tests/files/expected_ref.pkl differ
diff --git a/tests/files/expected_ref2.pkl b/tests/files/expected_ref2.pkl
index d675e7e..59d98da 100644
Binary files a/tests/files/expected_ref2.pkl and b/tests/files/expected_ref2.pkl differ
diff --git a/tests/files/expected_result.json b/tests/files/expected_result.json
index bd31841..f50ee9f 100644
--- a/tests/files/expected_result.json
+++ b/tests/files/expected_result.json
@@ -13,7 +13,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -29,7 +30,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -45,7 +47,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -61,7 +64,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -77,7 +81,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -93,7 +98,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -109,7 +115,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -125,7 +132,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -141,7 +149,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -157,7 +166,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -173,7 +183,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -189,7 +200,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -205,7 +217,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -221,7 +234,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -237,7 +251,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -253,7 +268,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -269,7 +285,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -285,7 +302,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -301,7 +319,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -317,7 +336,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -333,7 +353,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -349,7 +370,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -365,7 +387,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -381,7 +404,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -397,7 +421,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -413,7 +438,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     },
@@ -429,7 +455,8 @@
         "children": [],
         "type": "value",
         "invariants": [],
-        "tol": 0
+        "atol": null,
+        "rtol": null
       },
       "children": []
     }
diff --git a/tests/integrations/test_otter.py b/tests/integrations/test_otter.py
index d605b50..9a20265 100644
--- a/tests/integrations/test_otter.py
+++ b/tests/integrations/test_otter.py
@@ -8,6 +8,7 @@
 import pathlib
 import dill
 import pkg_resources
+import pytest
 
 from otter.assign.assignment import Assignment
 from otter.test_files import GradingResults 
@@ -129,7 +130,9 @@ def test_from_notebook(capsys):
         ref_path = str(pathlib.Path(__file__).parent.parent / 'files' / 'expected_ref.pkl')
 
         plg = OtterPlugin(ntf.name, {}, {})
-        plg.from_notebook(ref_path)
+        with pytest.warns(UserWarning, match="ould not force-save notebook; the results of this "
+                                "call will be based on the last saved version of this notebook."):
+            plg.from_notebook(ref_path)
 
         captured = capsys.readouterr()
         expected = dedent("""\
@@ -147,7 +150,10 @@ def test_from_notebook(capsys):
         ntf.seek(0)
 
         plg = OtterPlugin(ntf.name, {}, {})
-        plg.from_notebook(ref_path)
+
+        with pytest.warns(UserWarning, match="ould not force-save notebook; the results of this "
+                                "call will be based on the last saved version of this notebook."):
+            plg.from_notebook(ref_path)
 
         captured = capsys.readouterr()
         expected = dedent("""\