Repository: microsoft/pybryt

Issue:
Add documentation for `Attribute` annotations
The docs currently don't mention the `Attribute` annotation (except for in the API reference), so this should be added.

Discussion:

Patch:
diff --git a/docs/annotations/index.rst b/docs/annotations/index.rst
index f25fa10..1ea59f3 100644
--- a/docs/annotations/index.rst
+++ b/docs/annotations/index.rst
@@ -4,12 +4,12 @@ Annotations
 ===========
 
 .. toctree::
-   :maxdepth: 3
-   :hidden:
+    :maxdepth: 3
+    :hidden:
 
-   relational_annotations
-   complexity_annotations
-   invariants
+    relational_annotations
+    complexity_annotations
+    invariants
 
 Annotations are the basic building blocks, out of which reference
 implementations are constructed. The annotations represent a single condition
@@ -36,36 +36,36 @@ constructor or by accessing the instance variables of that object. The table bel
 arguments.
 
 .. list-table::
-   :widths: 10 10 10 70
-
-   * - Field Name
-     - Keyword Argument
-     - Type
-     - Description
-   * - ``name``
-     - ``name``
-     - ``str``
-     - The name of the annotation; used to process ``limit``; automatically set if unspecified
-   * - ``limit``
-     - ``limit``
-     - ``str``
-     - The maximum number of annotations with a given ``name`` to track
-   * - ``group``
-     - ``group``
-     - ``str``
-     - The name of a group to which this annotation belongs; for running specific groups of annotations
-       from a reference implementation
-   * - ``success_message``
-     - ``success_message``
-     - ``str``
-     - A message to be displayed to the student if the annotation is satisfied
-   * - ``failure_message``
-     - ``failure_message``
-     - ``str``
-     - A message to be displayed to the student if the annotation is *not* satisfied
-
-
-Value annotations
+    :widths: 10 10 10 70
+
+    * - Field Name
+      - Keyword Argument
+      - Type
+      - Description
+    * - ``name``
+      - ``name``
+      - ``str``
+      - The name of the annotation; used to process ``limit``; automatically set if unspecified
+    * - ``limit``
+      - ``limit``
+      - ``str``
+      - The maximum number of annotations with a given ``name`` to track
+    * - ``group``
+      - ``group``
+      - ``str``
+      - The name of a group to which this annotation belongs; for running specific groups of annotations
+        from a reference implementation
+    * - ``success_message``
+      - ``success_message``
+      - ``str``
+      - A message to be displayed to the student if the annotation is satisfied
+    * - ``failure_message``
+      - ``failure_message``
+      - ``str``
+      - A message to be displayed to the student if the annotation is *not* satisfied
+
+
+Value Annotations
 -----------------
 
 Value annotation is the most basic type of annotation. It expects a specific
@@ -75,8 +75,8 @@ its constructor the value we expect to see in the student's code:
 
 .. code-block:: python
 
-   arr = np.linspace(0, 10, 11)
-   pybryt.Value(arr)
+    arr = np.linspace(0, 10, 11)
+    pybryt.Value(arr)
 
 Note that when an instance of :py:class:`Value<pybryt.Value>` is created,
 :py:meth:`copy.copy` is called on the argument passed to it, so values cannot be
@@ -97,7 +97,7 @@ annotation is defined:
 
 .. code-block:: python
 
-   pybryt.Value(arr, atol=1e-3, rtol=1e-5)
+    pybryt.Value(arr, atol=1e-3, rtol=1e-5)
 
 
 Invariants
@@ -113,5 +113,50 @@ allow the student's solution (string) to be case-insensitive.
 
 .. code-block:: python
 
-   correct_answer = "a CasE-inSensiTiVe stRING"
-   pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
+    correct_answer = "a CasE-inSensiTiVe stRING"
+    pybryt.Value(correct_answer, invariants=[pybryt.invariants.string_capitalization])
+
+More information about invariants can be found :ref:`here<invariants>`.
+
+
+Attribute Annotations
+---------------------
+
+PyBryt supports checking for the presence of an object with a specific attribute value using the
+:py:class:`Attribute<pybryt.Attribute>` annotation. This annotation takes in an object and one or more
+strings representing an instance variable, and asserts that the student's memory footprint should 
+contain an object with that attribute such that the value of the attribute equals the value of the 
+attribute in the annotation.
+
+For example, this can be useful in checking that students have correctly fitted in ``sklearn``
+regression model by checking that the coefficients are correct:
+
+.. code-block:: python
+
+    import sklearn.linear_model as lm
+
+    model = lm.LinearRegression()
+    model.fit(X, y)
+
+    pybryt.Attribute(model, "coef_",
+        failure_message="Your model doesn't have the correct coefficients.")
+
+Note that, by default, PyBryt doesn't check that the object satisfying the attribute annotation has
+the same type as the object the annotation was created for. If a student knew the coefficient values
+in the above example, the following student code would satisfy that annotation:
+
+.. code-block:: python
+
+    class Foo:
+
+        coef_ = ...  # the array of coefficients
+    
+    f = Foo()  # f will satisfy the annotation
+
+To ensure that the annotation is only satisfied when the object is of the same type, set 
+``enforce_type=True`` in the constructor:
+
+.. code-block:: python
+
+    pybryt.Attribute(model, "coef_", enforce_type=True,
+        failure_message="Your model doesn't have the correct coefficients.")
diff --git a/pybryt/__main__.py b/pybryt/__main__.py
index d5e7dd1..efa249a 100644
--- a/pybryt/__main__.py
+++ b/pybryt/__main__.py
@@ -1,4 +1,4 @@
-""""""
+"""Main module for PyBryt CLI"""
 
 from .cli import cli
 
diff --git a/pybryt/annotations/annotation.py b/pybryt/annotations/annotation.py
index bded61e..d6e9591 100644
--- a/pybryt/annotations/annotation.py
+++ b/pybryt/annotations/annotation.py
@@ -104,7 +104,7 @@ class Annotation(ABC):
         return _TRACKED_ANNOTATIONS
 
     @staticmethod
-    def reset_tracked_annotations():
+    def reset_tracked_annotations() -> NoReturn:
         """
         Resets the list of tracked annotations and the mapping of group names to indices in that
         list.
diff --git a/pybryt/annotations/complexity/__init__.py b/pybryt/annotations/complexity/__init__.py
index 6c8ef64..39e28be 100644
--- a/pybryt/annotations/complexity/__init__.py
+++ b/pybryt/annotations/complexity/__init__.py
@@ -1,4 +1,4 @@
-""""""
+"""Annotation scaffold for asserting the complexity of code"""
 
 from . import complexities
 from .annotation import *
diff --git a/pybryt/annotations/complexity/annotation.py b/pybryt/annotations/complexity/annotation.py
index b0e8755..91f9554 100644
--- a/pybryt/annotations/complexity/annotation.py
+++ b/pybryt/annotations/complexity/annotation.py
@@ -1,4 +1,4 @@
-""""""
+"""Annotation classes for complexity assertions"""
 
 __all__ = ["ComplexityAnnotation", "TimeComplexity"]
 
diff --git a/pybryt/annotations/complexity/complexities.py b/pybryt/annotations/complexity/complexities.py
index f961ab6..472f92a 100644
--- a/pybryt/annotations/complexity/complexities.py
+++ b/pybryt/annotations/complexity/complexities.py
@@ -1,4 +1,4 @@
-""""""
+"""Complexity classes for complexity annotations"""
 
 import numpy as np
 
diff --git a/pybryt/annotations/invariants.py b/pybryt/annotations/invariants.py
index 001feb1..1503215 100644
--- a/pybryt/annotations/invariants.py
+++ b/pybryt/annotations/invariants.py
@@ -1,13 +1,10 @@
-"""Invariants for value annotations (available as :py:mod:`pybryt.invariants`)"""
+"""Invariants for value annotations"""
 
 import numpy as np
 
 from abc import ABC, abstractmethod
 from collections.abc import Iterable
 from typing import Any, List, Optional, Union
-# from enum import Enum, auto
-
-# TODO: add iterable_type invariant
 
 
 class invariant(ABC):
@@ -50,14 +47,9 @@ class invariant(ABC):
         ... # pragma: no cover
 
 
-# TODO: if hashing, for all strings collect actual string and lowercased version (marked as such), 
-#       and compare against that if this invariant is used. 
 class string_capitalization(invariant):
     """
-    An invariant that compares strings ignoring capitalization.
-
-    Works by taking in a list of values and lowercasing them if they are strings and leaving them
-    unchanged otherwise.
+    An invariant that compares strings ignoring the capitalization of letters.
     """
 
     @staticmethod
diff --git a/pybryt/annotations/value.py b/pybryt/annotations/value.py
index b9e9d59..dec3fa0 100644
--- a/pybryt/annotations/value.py
+++ b/pybryt/annotations/value.py
@@ -1,5 +1,4 @@
-"""
-"""
+"""Annotations for asserting the presence of a value"""
 
 __all__ = ["Value", "Attribute"]
 
@@ -13,7 +12,6 @@ from typing import Any, Dict, List, Optional, Tuple, Union
 
 from .annotation import Annotation, AnnotationResult
 from .invariants import invariant
-# from ..utils import check_values_equal
 
 
 class Value(Annotation):
@@ -166,8 +164,19 @@ class Value(Annotation):
         return False
 
     @staticmethod
-    def check_values_equal(value, other_value, atol = None, rtol = None):
+    def check_values_equal(value, other_value, atol = None, rtol = None) -> bool:
         """
+        Checks whether two objects are equal. 
+        
+        If the values are both numeric (numerics, arrays, etc.) and ``atol`` and/or ``rtol`` are 
+        specified, the values are considered equal if ``other_value`` is within the tolerance
+        bounds of ``value``.
+
+        Args:
+            value (``object``): the first object to compare
+            other_value (``object``): the second object to compare
+            atol (``float``, optional): the absolute tolerance for numeric values
+            rtol (``float``, optional): the relative tolerance for numeric values
         """
         if isinstance(value, Iterable) ^ isinstance(other_value, Iterable):
             return False
@@ -230,6 +239,8 @@ class _AttrValue(Value):
     Args:
         obj (``object``): the object being checked
         attr (``str``): the attribute of the object being checked
+        enforce_type (``bool``, optional): whether to ensure that the satisfying value has the same 
+            type as the initial value
         **kwargs: additional keyword arguments passed to the :py:class:`Value<pybryt.Value>` 
             constructor
     """
@@ -237,9 +248,13 @@ class _AttrValue(Value):
     _object: Any
     _attr: str
 
-    def __init__(self, obj: Any, attr: str, **kwargs):
+    enforce_type: bool
+    """whether to ensure that the satisfying value has the same type as the initial value"""
+
+    def __init__(self, obj: Any, attr: str, enforce_type: bool = False, **kwargs):
         self._object = obj
         self._attr = attr
+        self.enforce_type = enforce_type
         val = getattr(obj, attr)
         super().__init__(val, **kwargs)
     
@@ -258,7 +273,7 @@ class _AttrValue(Value):
             ``bool``: whether the objects are equal
         """
         return super().__eq__(other) and self.check_values_equal(self._object, other._object) and \
-            self._attr == other._attr
+            self._attr == other._attr and self.enforce_type == other.enforce_type
     
     def check(self, observed_values: List[Tuple[Any, int]]) -> AnnotationResult:
         """
@@ -273,10 +288,15 @@ class _AttrValue(Value):
             :py:class:`AnnotationResult`: the results of this annotation based on 
             ``observed_values``
         """
+        if self.enforce_type:  # filter out values of wrong type if enforce_type is True
+            observed_values = [t for t in observed_values if isinstance(t[0], type(self._object))]
         vals = [t for t in observed_values if hasattr(t[0], self._attr)]
         attrs = [(getattr(obj, self._attr), t) for obj, t in vals]
         res = super().check(attrs)
-        satisfier = vals[attrs.index((res.value, res.timestamp))][0]
+        try:
+            satisfier = vals[attrs.index((res.value, res.timestamp))][0]
+        except ValueError:
+            satisfier = None
         return AnnotationResult(None, self, value=satisfier, children=[res])
 
 
@@ -292,6 +312,8 @@ class Attribute(Annotation):
     Args:
         obj (``object``): the object being checked
         attrs (``str`` or ``list[str]``): the attribute or attributes that should be checked
+        enforce_type (``bool``, optional): whether to ensure that the satisfying value has the same 
+            type as the initial value
         **kwargs: additional keyword arguments passed to the 
             :py:class:`Annotation<pybryt.Annotation>` constructor
     """
@@ -301,7 +323,10 @@ class Attribute(Annotation):
     _atol: Optional[Union[float, int]]
     _rtol: Optional[Union[float, int]]
 
-    def __init__(self, obj: Any, attrs: Union[str, List[str]], **kwargs):
+    enforce_type: bool
+    """whether to ensure that the satisfying value has the same type as the initial value"""
+
+    def __init__(self, obj: Any, attrs: Union[str, List[str]], enforce_type: bool = False, **kwargs):
         if isinstance(attrs, str):
             attrs = [attrs]
         if not isinstance(attrs, list) or not all(isinstance(a, str) for a in attrs):
@@ -314,17 +339,19 @@ class Attribute(Annotation):
         for attr in attrs:
             if not hasattr(obj, attr):
                 raise AttributeError(f"{obj} has not attribute '{attr}'")
-            self._annotations.append(_AttrValue(obj, attr, **kwargs))
+            self._annotations.append(_AttrValue(obj, attr, enforce_type=enforce_type, **kwargs))
         
         self._invariants = kwargs.pop("invariants", [])
         self._atol = kwargs.pop("atol", None)
         self._rtol = kwargs.pop("rtol", None)
+
+        self.enforce_type = enforce_type
     
         super().__init__(name=name, success_message=success_message, failure_message=failure_message,
             **kwargs)
 
     @property
-    def children(self):
+    def children(self) -> List[Annotation]:
         return self._annotations
     
     def __eq__(self, other: Any) -> bool:
@@ -360,6 +387,7 @@ class Attribute(Annotation):
             "atol": self._atol,
             "rtol": self._rtol,
             "attributes": [av._attr for av in self._annotations],
+            "enforce_type": self.enforce_type,
         })
         return d
 
diff --git a/pybryt/execution/complexity.py b/pybryt/execution/complexity.py
index 7fc84d1..352b963 100644
--- a/pybryt/execution/complexity.py
+++ b/pybryt/execution/complexity.py
@@ -9,6 +9,27 @@ _TRACKING_DISABLED = False
 
 
 class TimeComplexityResult:
+    """
+    A class for collecting the results of time complexity check blocks.
+
+    Args:
+        name (``str``): the name of the block
+        n (``int``): the input length
+        start (``int``): the step counter value at the start of the block
+        stop (``int``): the step counter value at the end of the block
+    """
+
+    name: str
+    """the name of the block"""
+
+    n: int
+    """the input length or the input itself"""
+
+    start: int
+    """the step counter value at the start of the block"""
+
+    stop: int
+    """the step counter value at the end of the block"""
 
     def __init__(self, name, n, start, stop):
         self.name = name
diff --git a/pybryt/execution/tracing.py b/pybryt/execution/tracing.py
index a45f047..c64b4dd 100644
--- a/pybryt/execution/tracing.py
+++ b/pybryt/execution/tracing.py
@@ -1,4 +1,4 @@
-""""""
+"""Tracing function and controls"""
 
 import re
 import linecache
diff --git a/pybryt/plagiarism.py b/pybryt/plagiarism.py
index e60dd22..72b503e 100644
--- a/pybryt/plagiarism.py
+++ b/pybryt/plagiarism.py
@@ -1,5 +1,4 @@
-"""
-"""
+"""Plagiarism checking for PyBryt"""
 
 import random
 import numpy as np
diff --git a/pybryt/utils.py b/pybryt/utils.py
index 1d53adc..fc5acaa 100644
--- a/pybryt/utils.py
+++ b/pybryt/utils.py
@@ -1,5 +1,4 @@
-"""
-"""
+"""Various utilities for PyBryt"""
 
 import os
 import json


Test patch:
diff --git a/tests/annotations/test_value.py b/tests/annotations/test_value.py
index e412609..e6f77a2 100644
--- a/tests/annotations/test_value.py
+++ b/tests/annotations/test_value.py
@@ -132,8 +132,24 @@ def test_attribute_annotation():
         "rtol": None,
         "type": "attribute",
         "attributes": ['T'],
+        "enforce_type": False,
     }
 
+    # check enforce type
+    class Foo:
+        T = val.T
+
+    mfp2 = [(Foo(), 1)]
+    res = v.check(mfp2)
+    assert res.satisfied
+
+    v = Attribute(val, "T", enforce_type=True)
+    res = v.check(mfp2)
+    assert not res.satisfied
+
+    res = v.check(mfp + mfp2)
+    assert res.satisfied
+
     # check error raising
     with pytest.raises(TypeError):
         Attribute(val, ["T", 1])