Repository: CORE-GATECH-GROUP/serpent-tools
Pull request #191 (2018-06-08 21:24:29)

[ENH] Update compare method with low/upper bounds
The compare method now accepts the following objects to control the comparison:
    - lower: lower bound on acceptable relative differences
    - upper: upper bound on acceptable relative differences

Values with relative difference less than lower will be treated as a success, greater than or equal to upper will be treated as a failure, and in the middle will be warned but not a failure.

Added a developer guide that details how our comparison methods should be implemented, and what types of messages should be given for different scenarios

Added test_base_compare.py to perform some basic checks regarding
    - input types to the compare method
    - input arguments

DanKotlyar: This is really great. Should be really useful to the serpent developers. 

Patch:
diff --git a/docs/develop/comparisons.rst b/docs/develop/comparisons.rst
new file mode 100644
index 0000000..6e221d2
--- /dev/null
+++ b/docs/develop/comparisons.rst
@@ -0,0 +1,82 @@
+.. |baseObj| replace:: ``BaseObject``
+
+.. |error| replace:: :func:`~serpentTools.messages.error`
+
+.. |warn| replace:: :func:`~serpentTools.messages.warning`
+
+.. |info| replace:: :func:`~serpentTools.messages.info`
+
+.. |debug| replace:: :func:`~serpentTools.messages.debug`
+
+.. _dev-comparisons:
+
+==================
+Comparison Methods
+==================
+
+We are currently developing methods for our readers and containers to be able
+for comparing between like objects. This could be used to compare the effect 
+of changing fuel enrichment on pin powers or criticality, or used to compare
+the effect of different ``SERPENT`` settings. The ``BaseObject`` that 
+every object **should** inherit from contains the bulk of the input checking,
+so each reader and object needs to implement a private ``_compare`` method with
+the following structure::
+
+    def _compare(self, other, lower, upper, sigma):
+        return <boolean output of comparison>
+
+.. note::
+
+    While these methods will iterate over many quantities, and some quantities
+    may fail early on in the test, the comparison method should continue
+    until all quantities have been tested.
+
+The value ``sigma`` should be used to compare quantities with uncertainties
+by constructing intervals bounded by :math:`x\pm S\sigma`, where
+``sigma``:math:`=S`. Quantities that do not have overlapping confidence
+windows will be considered too different and should result in a ``False`` 
+value being returned from the method.
+
+The ``lower`` and ``upper`` arguments should be used to compare values
+that do not have uncertainties. Both will be ``float`` values, with 
+``lower`` less than or equal to ``upper``. These should be be used
+when comparing the relative differences between quantities from ``self``
+and ``other`` with something like::
+
+   # selfV is the "reference" value from self
+   # otherV is the value from ``other``
+   >>> relDiff = abs(otherV - selfV)
+   >>> if selfV > 1E-6:
+   ...     relDiff /= selfV
+   >>> if relDiff < lower:
+   ...     # close enough, move along
+   >>> elif relDiff >= upper:
+   ...     # consider test a failure; print an error
+   >>> else:
+   ...     # close but not close enough; print a warning
+   ...     # but don't treat as a failure
+
+It is likely that the ``BaseObject`` or :mod:`serpentTools.utils` module
+will have some methods/functions for expediting this process, but the 
+procedure for all cases should follow this behavior.
+
+Use of messaging module
+=======================
+
+Below is a non-definitive nor comprehensive list of possible comparison cases
+and the corresponding message that should be printed. Using a range of message
+types allows the user to be able to easily focus on things that are really bad by
+using our :ref:`verbosity` setting.
+
+* Two objects contain different data sets, e.g. different dictionary values
+  - |warn| displaying the missing items, and then apply test to items in both objects
+* Two items are identically zero, or arrays of zeros - |debug|
+* Two items are outside of the ``sigma`` confidence intervals - |error|
+* Two items without uncertainties have relative difference
+
+    * less than ``lower`` - |info|
+    * greater than or equal to ``upper`` - |error|
+    * otherwise - |warn|
+
+* Two items are in good agreement - |info|
+* Two arrays are not of similar size - |error|
diff --git a/docs/develop/index.rst b/docs/develop/index.rst
index 2dbef65..d386f58 100644
--- a/docs/develop/index.rst
+++ b/docs/develop/index.rst
@@ -20,3 +20,4 @@ without any loss of comprehension.
     checklist.rst
     git.rst
     serpentVersions.rst
+    comparisons.rst
diff --git a/serpentTools/objects/base.py b/serpentTools/objects/base.py
index 1320d3c..510ea8d 100644
--- a/serpentTools/objects/base.py
+++ b/serpentTools/objects/base.py
@@ -16,24 +16,38 @@ from serpentTools.plot import (
 class BaseObject(object):
     """Most basic class shared by all other classes."""
 
-    def compare(self, other, rtol=0., atol=0., sigma=3, strictMissing=True):
+    def compare(self, other, lower=0, upper=10, sigma=3):
         """
         Compare the results of this reader to another.
 
+        For values without uncertainties, the upper and lower
+        arguments control what passes and what messages get
+        raised. If a quantity in ``other`` is less than
+        ``lower`` percent different that the same quantity
+        on this object, consider this allowable and make
+        no messages.
+        Quantities that are greater than ``upper`` percent
+        different will have a error messages printed and
+        the comparison will return ``False``, but continue.
+        Quantities with difference between these ranges will
+        have warning messages printed.
+
         Parameters
         ----------
         other:
             Other reader instance against which to compare.
             Must be a similar class as this one.
-        rtol: float
-            Relative tolerance for arrays.
-        atol: float
-            Absolute tolerance for arrays
+        lower: float or int
+            Lower limit for relative tolerances.
+            Differences below this will be considered allowable
+        upper: float or int
+            Upper limit for relative tolerances. Differences
+            above this will be considered failure and errors
+            messages will be raised
         sigma: int
             Size of confidence interval to apply to
-            potentially random values
-        strictMissing: bool
-            If ``True``, raise Exceptions for missing data
+            quantities with uncertainties. Quantities that do not
+            have overlapping confidence intervals will fail.
 
         Returns
         -------
@@ -46,9 +60,22 @@ class BaseObject(object):
         TypeError
             If ``other`` is not of the same class as this class
             nor a subclass of this class
-        :py:class:`~serpentTools.messages.MissingDataError`
-            If data is missing from ``other`` and ``strictMissing``
+        ValueError
+            If upper > lower,
+            If sigma, lower, or upper are negative
         """
+        upper = float(upper)
+        lower = float(lower)
+        sigma = float(sigma)
+        if upper < lower:
+            raise ValueError("Upper limit must be greater than lower. "
+                             "{} is not greater than {}"
+                             .format(upper, lower))
+        for item, key in zip((upper, lower, sigma),
+                             ('upper', 'lower', 'sigma')):
+            if item < 0:
+                raise ValueError("{} must be non-negative, is {}"
+                                 .format(key, item))
         if not (isinstance(other, self.__class__) or
                 issubclass(other.__class__, self.__class__)):
             oName = other.__class__.__name__
@@ -57,9 +84,9 @@ class BaseObject(object):
                     "Cannot compare against {} - not instance nor subclass "
                     "of {}".format(oName, name))
 
-            return self._compare(other, rtol, atol, sigma, strictMissing)
+            return self._compare(other, lower, upper, sigma)
 
-    def _compare(self, other, rtol, atol, sigma, strictMissing):
+    def _compare(self, other, lower, upper, sigma):
         """Actual comparison method for similar classes."""
         raise NotImplementedError
 


Test patch:
diff --git a/serpentTools/tests/test_base_compare.py b/serpentTools/tests/test_base_compare.py
new file mode 100644
index 0000000..82c5c19
--- /dev/null
+++ b/serpentTools/tests/test_base_compare.py
@@ -0,0 +1,56 @@
+"""Test the basic aspect of the comparison system."""
+
+from unittest import TestCase
+
+from serpentTools.objects.base import BaseObject
+from serpentTools.parsers.results import ResultsReader
+
+
+class SafeCompare(BaseObject):
+    """Object that can use the comparison method."""
+
+    def _compare(self, *args, **kwargs):
+        return True
+
+
+class SafeSubclass(SafeCompare):
+    pass
+
+
+class BaseCompareTester(TestCase):
+    """Class responsible for testing the basic compare system."""
+
+    def setUp(self):
+        self.obj = SafeCompare()
+
+    def test_badCompType(self):
+        """Verify that comparisons against bad-types raise errors."""
+        with self.assertRaises(TypeError):
+            self.obj.compare(ResultsReader(None))
+        with self.assertRaises(TypeError):
+            self.obj.compare(list())
+
+    def test_safeCompare(self):
+        """
+        Verify that the compare method doesn't fail when
+        comparing against similar objects or subclasses
+        """
+        self.obj.compare(SafeCompare())
+        self.obj.compare(SafeSubclass())
+
+    def test_badArgs(self):
+        """Verify that the compare method fails for bad arguments."""
+        other = SafeCompare()
+        with self.assertRaises(ValueError):
+            self.obj.compare(other, lower=-1)
+        with self.assertRaises(ValueError):
+            self.obj.compare(other, upper=-1)
+        with self.assertRaises(ValueError):
+            self.obj.compare(other, sigma=-1)
+        with self.assertRaises(ValueError):
+            self.obj.compare(other, lower=100, upper=1)
+
+
+if __name__ == '__main__':
+    from unittest import main
+    main()