Repository: microsoft/pybryt

Issue:
Forbidding certain type to be in memory
**Is your feature request related to a problem? Please describe.**
It is very common to ask students in exercises to do something "using lists", "using tuples", "not using numpy", etc. This makes them practice solving problems by not using some of the operations they normally use or not using the solutions they can find on the internet.

We believe that forbidding a certain variable type in memory without specifying the exact value would simplify writing reference solutions.

**Describe the solution you'd like**
For instance, `pybryt.ForbidType(numpy.ndarray)` would give an error to the student if a variable of `numpy.ndarray` type is found in the memory footprint.

Discussion:

Patch:
diff --git a/docs/annotations/index.rst b/docs/annotations/index.rst
index 1ea59f3..6202194 100644
--- a/docs/annotations/index.rst
+++ b/docs/annotations/index.rst
@@ -7,8 +7,10 @@ Annotations
     :maxdepth: 3
     :hidden:
 
+    value_annotations
     relational_annotations
     complexity_annotations
+    type_annotations
     invariants
 
 Annotations are the basic building blocks, out of which reference
@@ -21,11 +23,13 @@ student's code, including the temporal relationship of values and complex
 boolean logic surrounding the presence or absence of those values.
 
 All annotations are created by instantiating subclasses of the abstract
-:py:class:`Annotation<pybryt.Annotation>` class. There are two main types of
-annotations: 
+:py:class:`Annotation<pybryt.annotations.annotation.Annotation>` class. There 
+are four main types of annotations: 
  
-* value annotations, described below, and
-* :ref:`relational annotations<relational>`.
+* :ref:`value annotations<value>`
+* :ref:`relational annotations<relational>`
+* :ref:`complexity annotations<complexity>`
+* :ref:`type annotations<type>`
 
 
 Annotation Arguments
@@ -63,100 +67,3 @@ arguments.
       - ``failure_message``
       - ``str``
       - A message to be displayed to the student if the annotation is *not* satisfied
-
-
-Value Annotations
------------------
-
-Value annotation is the most basic type of annotation. It expects a specific
-value to appear while executing the student's code. To create a value
-annotation, we create an instance of :py:class:`Value<pybryt.Value>` and pass to
-its constructor the value we expect to see in the student's code:
-
-.. code-block:: python
-
-    arr = np.linspace(0, 10, 11)
-    pybryt.Value(arr)
-
-Note that when an instance of :py:class:`Value<pybryt.Value>` is created,
-:py:meth:`copy.copy` is called on the argument passed to it, so values cannot be
-affected by mutability.
-
-
-Numerical Tolerances
-++++++++++++++++++++
-
-For numerical values, or iterables of numerical values that support vectorized
-math, it is also possible to define an absolute tolerance (``atol``) and/or a
-relative tolerance (``rtol``) to allow the student's solution to deviate from
-the reference. Numerical tolerances are computed as with ``numpy.allcose``,
-where the value is "equal enough" if it is within :math:`v \pm (\texttt{atol}
-+ \texttt{rtol} \cdot |v|)`, where :math:`v` is the value of the annotation.
-Both ``atol`` and ``rtol`` default to zero and have to be specified when value
-annotation is defined:
-
-.. code-block:: python
-
-    pybryt.Value(arr, atol=1e-3, rtol=1e-5)
-
-
-Invariants
-++++++++++
-
-Similar to numerical tolerances, which allow the student's solution to deviate
-from the reference value, PyBryt allows specifying conditions when other data
-types should be considered equal. PyBryt supports defining these conditions
-using :ref:`invariants<invariants>`. To use invariants on a value, we need to
-pass the invariant objects as a list to the ``invariants`` argument of the
-:py:class:`Value<pybryt.Value>` constructor. For instance, let us say we want to
-allow the student's solution (string) to be case-insensitive.
-
-.. code-block:: python
-
-    correct_answer = "a CasE-inSensiTiVe stRING"
-    pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
-
-More information about invariants can be found :ref:`here<invariants>`.
-
-
-Attribute Annotations
----------------------
-
-PyBryt supports checking for the presence of an object with a specific attribute value using the
-:py:class:`Attribute<pybryt.Attribute>` annotation. This annotation takes in an object and one or more
-strings representing an instance variable, and asserts that the student's memory footprint should 
-contain an object with that attribute such that the value of the attribute equals the value of the 
-attribute in the annotation.
-
-For example, this can be useful in checking that students have correctly fitted in ``sklearn``
-regression model by checking that the coefficients are correct:
-
-.. code-block:: python
-
-    import sklearn.linear_model as lm
-
-    model = lm.LinearRegression()
-    model.fit(X, y)
-
-    pybryt.Attribute(model, "coef_",
-        failure_message="Your model doesn't have the correct coefficients.")
-
-Note that, by default, PyBryt doesn't check that the object satisfying the attribute annotation has
-the same type as the object the annotation was created for. If a student knew the coefficient values
-in the above example, the following student code would satisfy that annotation:
-
-.. code-block:: python
-
-    class Foo:
-
-        coef_ = ...  # the array of coefficients
-    
-    f = Foo()  # f will satisfy the annotation
-
-To ensure that the annotation is only satisfied when the object is of the same type, set 
-``enforce_type=True`` in the constructor:
-
-.. code-block:: python
-
-    pybryt.Attribute(model, "coef_", enforce_type=True,
-        failure_message="Your model doesn't have the correct coefficients.")
diff --git a/docs/annotations/type_annotations.rst b/docs/annotations/type_annotations.rst
new file mode 100644
index 0000000..d7d39aa
--- /dev/null
+++ b/docs/annotations/type_annotations.rst
@@ -0,0 +1,26 @@
+.. _type:
+
+Type Annotations
+================
+
+Type annotations are somewhat similar to value annotations, although they operate on types
+insted of individual values or objects. They can be used to check the presence, or lack thereof, of
+objects of a specific type in a memory footprint, or conditions on that presence.
+
+
+Forbid Type
+-----------
+
+The :py:class:`ForbidType<pybryt.annotations.type_.ForbidType>` annotation asserts that a student's
+memory footprint has no values of a specific type in it. For example, if students should be implementing
+an algorithm without vectorization, it is possible to create an annotation forbidding the use of
+arrays:
+
+.. code-block:: python
+
+    import numpy as np
+
+    pybryt.ForbidType(np.ndarray, failure_message="Arrays are not allowed!")
+
+Checking for values is performed using Python's built-in ``isinstance`` function. Types passed to 
+the constructor must be picklable with the ``dill`` library.
diff --git a/docs/annotations/value_annotations.rst b/docs/annotations/value_annotations.rst
new file mode 100644
index 0000000..0c98d96
--- /dev/null
+++ b/docs/annotations/value_annotations.rst
@@ -0,0 +1,97 @@
+.. _value:
+
+Value Annotations
+=================
+
+Value annotations are the most basic type of annotation. They expect a specific
+value to appear while executing the student's code. To create a value
+annotation, create an instance of :py:class:`Value<pybryt.annotations.value.Value>` and pass to
+its constructor the value expected in the student's code:
+
+.. code-block:: python
+
+    arr = np.linspace(0, 10, 11)
+    pybryt.Value(arr)
+
+Note that when an instance of :py:class:`Value<pybryt.annotations.value.Value>` is created,
+:py:meth:`copy.copy` is called on the argument passed to it, so values cannot be
+affected by mutability.
+
+
+Attribute Annotations
+---------------------
+
+PyBryt supports checking for the presence of an object with a specific attribute value using the
+:py:class:`Attribute<pybryt.annotations.value.Attribute>` annotation. This annotation takes in an 
+object and one or more strings representing an instance variable, and asserts that the student's 
+memory footprint should contain an object with that attribute such that the value of the attribute 
+equals the value of the  attribute in the annotation.
+
+For example, this can be useful in checking that students have correctly fitted in ``sklearn``
+regression model by checking that the coefficients are correct:
+
+.. code-block:: python
+
+    import sklearn.linear_model as lm
+
+    model = lm.LinearRegression()
+    model.fit(X, y)
+
+    pybryt.Attribute(model, "coef_",
+        failure_message="Your model doesn't have the correct coefficients.")
+
+Note that, by default, PyBryt doesn't check that the object satisfying the attribute annotation has
+the same type as the object the annotation was created for. If a student knew the coefficient values
+in the above example, the following student code would satisfy that annotation:
+
+.. code-block:: python
+
+    class Foo:
+
+        coef_ = ...  # the array of coefficients
+    
+    f = Foo()  # f will satisfy the annotation
+
+To ensure that the annotation is only satisfied when the object is of the same type, set 
+``enforce_type=True`` in the constructor:
+
+.. code-block:: python
+
+    pybryt.Attribute(model, "coef_", enforce_type=True,
+        failure_message="Your model doesn't have the correct coefficients.")
+
+
+Numerical Tolerances
+---------------------
+
+For numerical values, or iterables of numerical values that support vectorized
+math, it is also possible to define an absolute tolerance (``atol``) and/or a
+relative tolerance (``rtol``) to allow the student's solution to deviate from
+the reference. Numerical tolerances are computed as with ``numpy.allcose``,
+where the value is "equal enough" if it is within :math:`v \pm (\texttt{atol}
++ \texttt{rtol} \cdot |v|)`, where :math:`v` is the value of the annotation.
+Both ``atol`` and ``rtol`` default to zero and have to be specified when value
+annotation is defined:
+
+.. code-block:: python
+
+    pybryt.Value(arr, atol=1e-3, rtol=1e-5)
+
+
+Invariants
+---------------------
+
+Similar to numerical tolerances, which allow the student's solution to deviate
+from the reference value, PyBryt allows specifying conditions when other data
+types should be considered equal. PyBryt supports defining these conditions
+using :ref:`invariants<invariants>`. To use invariants on a value, we need to
+pass the invariant objects as a list to the ``invariants`` argument of the
+:py:class:`Value<pybryt.Value>` constructor. For instance, let's say we want to
+allow the student's solution (string) to be case-insensitive.
+
+.. code-block:: python
+
+    correct_answer = "a CasE-inSensiTiVe stRING"
+    pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
+
+More information about invariants can be found :ref:`here<invariants>`.
diff --git a/docs/api_reference.rst b/docs/api_reference.rst
index cdc4765..c3fc3d8 100644
--- a/docs/api_reference.rst
+++ b/docs/api_reference.rst
@@ -28,6 +28,14 @@ Relational Annotations
    :undoc-members:
 
 
+Invariants
+??????????
+
+.. automodule:: pybryt.annotations.invariants
+   :members:
+   :undoc-members:
+
+
 Complexity Annotations
 ++++++++++++++++++++++
 
@@ -46,10 +54,10 @@ Complexity Classes
    :undoc-members:
 
 
-Invariants
-++++++++++
+Type Annotations
+++++++++++++++++
 
-.. automodule:: pybryt.annotations.invariants
+.. automodule:: pybryt.annotations.type_
    :members:
    :undoc-members:
 
diff --git a/pybryt/annotations/__init__.py b/pybryt/annotations/__init__.py
index 01b7140..093e6b0 100644
--- a/pybryt/annotations/__init__.py
+++ b/pybryt/annotations/__init__.py
@@ -4,4 +4,5 @@ from . import invariants
 from .annotation import Annotation, AnnotationResult
 from .complexity import *
 from .relation import *
+from .type_ import *
 from .value import *
diff --git a/pybryt/annotations/type_.py b/pybryt/annotations/type_.py
new file mode 100644
index 0000000..ec6ca42
--- /dev/null
+++ b/pybryt/annotations/type_.py
@@ -0,0 +1,91 @@
+"""Annotations for checking type presence, or lack thereof, within memory footprints"""
+
+__all__ = ["ForbidType"]
+
+import dill
+
+from typing import Any, Dict, List, Tuple
+
+from .annotation import Annotation, AnnotationResult
+
+
+class ForbidType(Annotation):
+    """
+    Annotation class for asserting there are no objects of a specific type in the memory footprint.
+
+    Indicates that the student's memory footprint should not have any values of a certain type. Uses
+    the ``isinstance`` function to determine if any values match the specified type. The type itself
+    must be pickleable by ``dill``.
+
+    Args:
+        type\\_ (``type``): the type to forbid
+        **kwargs: additional keyword arguments passed to the 
+            :py:class:`Annotation<pybryt.annotations.annotation.Annotation>` constructor
+    """
+
+    type_: type
+    """the type to forbid"""
+
+    def __init__(self, type_, **kwargs):
+        if not isinstance(type_, type):
+            raise TypeError(f"{type_} is not a type")
+
+        try:
+            dill.dumps(type_)
+        except Exception as e:
+            raise ValueError(f"Types must be serializable but the following error was thrown during serialization:\n{e}")
+
+        self.type_ = type_
+
+        super().__init__(**kwargs)
+
+    @property
+    def children(self) -> List[Annotation]:
+        return []
+
+    def check(self, observed_values: List[Tuple[Any, int]]) -> AnnotationResult:
+        """
+        Checks that there are no values of type ``self.type_`` in the memory footprint.
+
+        Args:
+            observed_values (``list[tuple[object, int]]``): a list of tuples of values observed
+                during execution and the timestamps of those values
+        
+        Returns:
+            :py:class:`AnnotationResult`: the results of this annotation based on ``observed_values``
+        """
+        for val, _ in observed_values:
+            if isinstance(val, self.type_):
+                return AnnotationResult(False, self)
+        return AnnotationResult(True, self)
+
+    def __eq__(self, other: Any) -> bool:
+        """
+        Checks whether this annotation is equal to another object.
+
+        To be equal to a ``ForbidType``, the other object must also be a ``Forbidtype`` object, have
+        the same instance variables, and the same forbidden type.
+
+        Args:
+            other (``object``): the object to compare to
+
+        Returns:
+            ``bool``: whether the objects are equal
+        """
+        return super().__eq__(other) and self.type_ == other.type_
+
+    def to_dict(self) -> Dict[str, Any]:
+        """
+        Converts this annotation's details to a JSON-friendly dictionary format.
+
+        Output dictionary contains the annotation's name, group, limit number, success message, and
+        failure message, as well as the type as a string.
+
+        Returns:
+            ``dict[str, object]``: the dictionary representation of this annotation
+        """
+        d = super().to_dict()
+        d.update({
+            "type_": str(self.type_),
+        })
+        return d
diff --git a/pybryt/annotations/value.py b/pybryt/annotations/value.py
index dec3fa0..5937aba 100644
--- a/pybryt/annotations/value.py
+++ b/pybryt/annotations/value.py
@@ -34,7 +34,7 @@ class Value(Annotation):
         invariants (``list[invariant]``): invariants for 
             this value
         **kwargs: additional keyword arguments passed to the 
-            :py:class:`Annotation<pybryt.Annotation>` constructor
+            :py:class:`Annotation<pybryt.annotations.annotation.Annotation>` constructor
     """
 
     intial_value: Any
@@ -76,7 +76,7 @@ class Value(Annotation):
         super().__init__(**kwargs)
 
     @property
-    def children(self):
+    def children(self) -> List[Annotation]:
         return []
 
     def to_dict(self) -> Dict[str, Any]:
@@ -112,8 +112,7 @@ class Value(Annotation):
                 during execution and the timestamps of those values
         
         Returns:
-            :py:class:`AnnotationResult`: the results of this annotation based on 
-            ``observed_values``
+            :py:class:`AnnotationResult`: the results of this annotation based on ``observed_values``
         """
         satisfied = [self._check_observed_value(v) for v in observed_values]
         if not any(satisfied):


Test patch:
diff --git a/tests/annotations/test_type_.py b/tests/annotations/test_type_.py
new file mode 100644
index 0000000..9af54ab
--- /dev/null
+++ b/tests/annotations/test_type_.py
@@ -0,0 +1,64 @@
+"""Tests for PyBryt type annotations"""
+
+import time
+import pytest
+import numpy as np
+
+from unittest import mock
+
+from pybryt import *
+from pybryt.utils import pickle_and_hash
+
+from .utils import *
+
+
+def test_forbid_type():
+    """
+    """
+    mfp = generate_memory_footprint()
+    Annotation.reset_tracked_annotations()
+
+    a = ForbidType(bool)
+    res = a.check(mfp)
+
+    check_obj_attributes(a, {"children__len": 0})
+    check_obj_attributes(res, {
+        "children": [],
+        "satisfied": True,
+        "_satisfied": True,
+        "annotation": a,
+        "timestamp": -1,
+        "value": None,
+    })
+
+    assert a.to_dict() == {
+        "name": "Annotation 1",
+        "children": [],
+        "success_message": None,
+        "failure_message": None,
+        "limit": None,
+        "group": None,
+        "type": "forbidtype",
+        "type_": "<class 'bool'>",
+    }
+
+    a = ForbidType(np.ndarray)
+    res = a.check(mfp)
+    check_obj_attributes(res, {
+        "children": [],
+        "satisfied": False,
+        "_satisfied": False,
+        "annotation": a,
+        "timestamp": -1,
+        "value": None,
+    })
+
+    # check constructor errors
+    with pytest.raises(TypeError, match=f"1 is not a type"):
+        ForbidType(1)
+
+    with mock.patch("pybryt.annotations.type_.dill") as mocked_dill:
+        mocked_dill.dumps.side_effect = Exception()
+
+        with pytest.raises(ValueError, match="Types must be serializable but the following error was thrown during serialization:\n"):
+            ForbidType(np.ndarray)