Repository: CORE-GATECH-GROUP/serpent-tools (python)

Issue:
BUG ADFs in results file fail in numeric conversion
## Summary of issue
When reading a file with assembly discontinuity factors (ADFs), there is a string that serpentTools attempts to convert to a vector. This causes a ValueError
## Code for reproducing the issue
Add the following code to a section of a result file
```
% Assembly discontinuity factors (order: W-S-E-N / NW-NE-SE-SW):

DF_SURFACE                (idx, [1:  3])  = 'ADF' ;
DF_SYM                    (idx, 1)        = 1 ;
DF_N_SURF                 (idx, 1)        = 4 ;
DF_N_CORN                 (idx, 1)        = 4 ;
```
```
$ python -c "import serpentTools; serpentTools.read(\"adf_res.m\")"
```
## Actual outcome including console output and error traceback if applicable
```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/parsers/__init__.py", line 155, in read
    returnedFromLoader.read()
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/parsers/base.py", line 52, in read
    self._read()
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/parsers/results.py", line 158, in _read
    self._processResults(tline)
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/parsers/results.py", line 176, in _processResults
    self._storeUnivData(varNameSer, varVals)
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/parsers/results.py", line 198, in _storeUnivData
    values = str2vec(varVals)  # convert the string to float numbers
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/utils/core.py", line 59, in str2vec
    return out([of(xx) for xx in vec])
  File "/home/ajohnson400/.local/lib/python3.7/site-packages/serpentTools/utils/core.py", line 59, in <listcomp>
    return out([of(xx) for xx in vec])
ValueError: could not convert string to float: 'ADF'
```
## Expected outcome
Doesn't fail
## Versions

* Version  from ``serpentTools.__version__`` `0.7.0rc0`

* Python version - ``python --version`` `3.7`

Pull request:
BUG/DEV str2vec to handle spaceless strings, performance - #305
Closes #305 by allowing the conversion to handle string without spaces and create arrays of strings. When we perform `iterable.split()` on the string `'ADF'`, we get back a string that cannot be converted to floats or ints. This PR would instead return a direct array (or list or tuple) of the string.

## Performance

Utilize the python `map` function and `numpy.fromiter` to improve performance of the conversion. The following simple bash script compares two ways to construct a list of floats, and two ways to construct and array of floats. All commands work with the same range of values. 
```
#!/bin/bash
N=${1-10000}
echo Testing with $N values

python -m timeit -s "xr = range(${N})" "list(map(float, xr))"
python -m timeit -s "xr = range(${N})" "[float(xx) for xx in xr]"
python -m timeit -s "from numpy import fromiter; xr = range(${N})" "fromiter(map(float, xr), float)"
python -m timeit -s "from numpy import array; xr=range(${N})" "array([float(xx) for xx in xr])"
```
The times are ordered as:
1. List of floats using map
2. List of floats using list comprehension
3. Array of floats using map
4. Array of floats using list comprehension
```
Testing with 10 values
500000 loops, best of 5: 751 nsec per loop
500000 loops, best of 5: 913 nsec per loop
200000 loops, best of 5: 1.23 usec per loop
200000 loops, best of 5: 1.71 usec per loop
Testing with 100 values
50000 loops, best of 5: 5.3 usec per loop
50000 loops, best of 5: 7.54 usec per loop
50000 loops, best of 5: 6.9 usec per loop
20000 loops, best of 5: 10.8 usec per loop
Testing with 1000 values
5000 loops, best of 5: 59.5 usec per loop
5000 loops, best of 5: 83.3 usec per loop
5000 loops, best of 5: 69.6 usec per loop
2000 loops, best of 5: 111 usec per loop
Testing with 10000 values
500 loops, best of 5: 605 usec per loop
500 loops, best of 5: 837 usec per loop
500 loops, best of 5: 692 usec per loop
200 loops, best of 5: 1.08 msec per loop
```
We can see that, for small values, the two are comparable. As the number of entries increases the `map`-based approach becomes comparatively faster. This speed up is nice when people read universe data with 100s of energy groups, or detectors with thousands of groups. This function is called a lot, and this marginal speed up *should* make a difference in the scheme of reading. We don't have a profiling and timing suite yet, but we should.

Patch:
diff --git a/serpentTools/utils/core.py b/serpentTools/utils/core.py
index e56468e..c1b35ad 100644
--- a/serpentTools/utils/core.py
+++ b/serpentTools/utils/core.py
@@ -4,7 +4,7 @@ Core utilities
 
 from re import compile
 
-from numpy import array, ndarray
+from numpy import array, ndarray, fromiter
 
 
 # Regular expressions
@@ -15,28 +15,33 @@ SCALAR_REGEX = compile(r'=.+;')  # scalar
 FIRST_WORD_REGEX = compile(r'^\w+')  # first word in the line
 
 
-def str2vec(iterable, of=float, out=array):
+def str2vec(iterable, dtype=float, out=array):
     """
     Convert a string or other iterable to vector.
 
     Parameters
     ----------
     iterable: str or iterable
-        If string, will be split with ``split(splitAt)``
-        to create a list. Every item in this list, or original
+        If a string containing spaces, will be split using
+        ```iterable.split()``. If no spaces are found, the
+        outgoing type is filled with a single string, e.g.
+        a list with a single string as the first and only
+        entry. This is returned directly, avoiding conversion
+        with ``dtype``.
+        Every item in this split list, or original
         iterable, will be iterated over and converted accoring
         to the other arguments.
-    of: type
+    dtype: type
         Convert each value in ``iterable`` to this data type.
     out: type
         Return data type. Will be passed the iterable of
-        converted items of data dtype ``of``.
+        converted items of data dtype ``dtype``.
 
     Returns
     -------
     vector
         Iterable of all values of ``iterable``, or split variant,
-        converted to type ``of``.
+        converted to type ``dtype``.
 
     Examples
     --------
@@ -53,10 +58,19 @@ def str2vec(iterable, of=float, out=array):
         >>> str2vec(x)
         array([1., 2., 3., 4.,])
 
+        >>> str2vec("ADF")
+        array(['ADF', dtype='<U3')
+
     """
-    vec = (iterable.split() if isinstance(iterable, str)
-           else iterable)
-    return out([of(xx) for xx in vec])
+    if isinstance(iterable, str):
+        if ' ' in iterable:
+            iterable = iterable.split()
+        else:
+            return out([iterable])
+    cmap = map(dtype, iterable)
+    if out is array:
+        return fromiter(cmap, dtype)
+    return out(cmap)
 
 
 def splitValsUncs(iterable, copy=False):


Test patch:
diff --git a/serpentTools/tests/test_ResultsReader.py b/serpentTools/tests/test_ResultsReader.py
index 44c9514..001408b 100644
--- a/serpentTools/tests/test_ResultsReader.py
+++ b/serpentTools/tests/test_ResultsReader.py
@@ -1,10 +1,11 @@
 """Test the results reader."""
 
 from os import remove
+from shutil import copy
 from unittest import TestCase
 
 from numpy import array
-from numpy.testing import assert_equal
+from numpy.testing import assert_array_equal
 from six import iteritems
 
 from serpentTools.settings import rc
@@ -15,21 +16,33 @@ from serpentTools.messages import SerpentToolsException
 
 GCU_START_STR = "GCU_UNIVERSE_NAME"
 NO_GCU_FILE = "./pwr_noGcu_res.m"
-GOOD_FILE = getFile("pwr_noBU_res.m")
+ADF_FILE = "./pwr_adf_res.m"
+RES_NO_BU = getFile("pwr_noBU_res.m")
 
 
 def setUpModule():
     """Write the result file with no group constant data."""
-    with open(NO_GCU_FILE, 'w') as noGcu, open(GOOD_FILE) as good:
+    with open(NO_GCU_FILE, 'w') as noGcu, open(RES_NO_BU) as good:
         for line in good:
             if GCU_START_STR in line:
                 break
             noGcu.write(line)
+    copy(RES_NO_BU, ADF_FILE)
+    with open(ADF_FILE, 'a') as stream:
+        stream.write("""
+% Assembly discontinuity factors (order: W-S-E-N / NW-NE-SE-SW):
+
+DF_SURFACE                (idx, [1:  3])  = 'ADF' ;
+DF_SYM                    (idx, 1)        = 1 ;
+DF_N_SURF                 (idx, 1)        = 4 ;
+DF_N_CORN                 (idx, 1)        = 4 ;
+""")
 
 
 def tearDownModule():
     """Remove the noGcu file."""
     remove(NO_GCU_FILE)
+    remove(ADF_FILE)
 
 
 class Serp2129Helper(TestCase):
@@ -130,7 +143,7 @@ class TestGetUniv(TestCase):
     def test_validUniv(self):
         """Verify that getUniv returns the correct universe"""
         xsDict = self.reader.getUniv('0', burnup=0.0, index=0, timeDays=0.0)
-        assert_equal(xsDict.infExp['infAbs'], self.expectedinfValAbs)
+        assert_array_equal(xsDict.infExp['infAbs'], self.expectedinfValAbs)
 
 
 class TesterCommonResultsReader(TestCase):
@@ -179,14 +192,14 @@ class TesterCommonResultsReader(TestCase):
                 self.assertSetEqual(set(self.reader.metadata[key]),
                                     set(expectedValue))
             else:
-                assert_equal(self.reader.metadata[key], expectedValue)
+                assert_array_equal(self.reader.metadata[key], expectedValue)
 
     def test_resdata(self):
         """Verify that results data is properly stored."""
         expectedKeys = self.expectedResdata
         actualKeys = set(self.reader.resdata.keys())
         self.assertSetEqual(expectedKeys, actualKeys)
-        assert_equal(self.reader.resdata['absKeff'], self.expectedKeff)
+        assert_array_equal(self.reader.resdata['absKeff'], self.expectedKeff)
 
     def test_burnup(self):
         """Verify the burnup vector is properly stored."""
@@ -197,7 +210,7 @@ class TesterCommonResultsReader(TestCase):
                     "{} should have burnup, but does not".format(self))
             raise self.skipTest(
                 "{} does not, and should not, have burnup".format(self))
-        assert_equal(actualBurnDays, self.expectedDays)
+        assert_array_equal(actualBurnDays, self.expectedDays)
 
     def test_universes(self):
         """Verify that results for all states (univ, bu, buIdx, days) exist.
@@ -216,12 +229,12 @@ class TesterCommonResultsReader(TestCase):
         expUniv = self.reader.universes[expSt0]
         self.assertSetEqual(set(expUniv.infExp.keys()), self.expectedInfExp)
         self.assertSetEqual(set(expUniv.gc.keys()), self.expectedUnivgcData)
-        assert_equal(expUniv.infExp['infFlx'], self.expectedInfVals)
-        assert_equal(expUniv.infUnc['infFlx'], self.expectedInfUnc)
-        assert_equal(expUniv.gc['cmmTranspxs'], self.expectedCMM)
-        assert_equal(expUniv.gcUnc['cmmTranspxs'], self.expectedCMMunc)
-        assert_equal(expUniv.groups, self.expectedGroups)
-        assert_equal(expUniv.microGroups, self.expectedMicroGroups)
+        assert_array_equal(expUniv.infExp['infFlx'], self.expectedInfVals)
+        assert_array_equal(expUniv.infUnc['infFlx'], self.expectedInfUnc)
+        assert_array_equal(expUniv.gc['cmmTranspxs'], self.expectedCMM)
+        assert_array_equal(expUniv.gcUnc['cmmTranspxs'], self.expectedCMMunc)
+        assert_array_equal(expUniv.groups, self.expectedGroups)
+        assert_array_equal(expUniv.microGroups, self.expectedMicroGroups)
 
 
 class TestFilterResults(TesterCommonResultsReader):
@@ -599,7 +612,7 @@ class RestrictedResultsReader(Serp2129Helper):
 
     def _testUnivFlux(self, reader):
         univ = reader.getUniv('0', index=0)
-        assert_equal(self.expectedInfFlux_bu0, univ.get("infFlx"))
+        assert_array_equal(self.expectedInfFlux_bu0, univ.get("infFlx"))
 
     def test_justFlux(self):
         """Restrict the variables to gcu inf flux and verify their values"""
@@ -621,11 +634,25 @@ class RestrictedResultsReader(Serp2129Helper):
             rc['xs.variableExtras'] = ['ABS_KEFF', 'INF_FLX']
             r = readDataFile(self.dataFile)
         self._testUnivFlux(r)
-        assert_equal(self.expectedAbsKeff, r.resdata['absKeff'])
+        assert_array_equal(self.expectedAbsKeff, r.resdata['absKeff'])
 
 
 del TesterCommonResultsReader
 
+
+class ResADFTester(TestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        cls.reader = ResultsReader(ADF_FILE)
+        cls.reader.read()
+
+    def test_adf(self):
+        univ = self.reader.getUniv('0', index=0)
+        expected = array('ADF')
+        assert_array_equal(expected, univ.gc['dfSurface'])
+
+
 if __name__ == '__main__':
     from unittest import main
     main()
diff --git a/serpentTools/tests/test_utils.py b/serpentTools/tests/test_utils.py
index ce80756..ebb8e6e 100644
--- a/serpentTools/tests/test_utils.py
+++ b/serpentTools/tests/test_utils.py
@@ -45,8 +45,9 @@ class VariableConverterTester(TestCase):
 class VectorConverterTester(TestCase):
     """Class for testing the str2vec function"""
 
-    def setUp(self):
-        self.testCases = ("0 1 2 3", [0, 1, 2, 3], (0, 1, 2, 3), arange(4))
+    @classmethod
+    def setUpClass(cls):
+        cls.testCases = ("0 1 2 3", [0, 1, 2, 3], (0, 1, 2, 3), arange(4))
 
     def test_str2Arrays(self):
         """Verify that the str2vec converts to arrays."""
@@ -60,6 +61,13 @@ class VectorConverterTester(TestCase):
         expected = [0, 1, 2, 3]
         self._runConversionTest(int, expected, list)
 
+    def test_vecOfStr(self):
+        """Verify a single word can be converted with str2vec"""
+        key = 'ADF'
+        expected = array('ADF')
+        actual = str2vec(key)
+        assert_array_equal(expected, actual)
+
     def _runConversionTest(self, valType, expected, outType=None):
         if outType is None:
             outType = array
@@ -67,7 +75,7 @@ class VectorConverterTester(TestCase):
         else:
             compareType = outType
         for case in self.testCases:
-            actual = str2vec(case, of=valType, out=outType)
+            actual = str2vec(case, dtype=valType, out=outType)
             self.assertIsInstance(actual, compareType, msg=case)
             ofRightType = [isinstance(xx, valType) for xx in actual]
             self.assertTrue(all(ofRightType),
@@ -79,8 +87,9 @@ class VectorConverterTester(TestCase):
 class SplitValsTester(TestCase):
     """Class that tests splitValsUncs."""
 
-    def setUp(self):
-        self.input = arange(4)
+    @classmethod
+    def setUp(cls):
+        cls.input = arange(4)
 
     def test_splitVals(self):
         """Verify the basic functionality."""
@@ -321,8 +330,9 @@ class OverlapTester(TestCase):
 class SplitDictionaryTester(TestCase):
     """Class for testing utils.splitDictByKeys."""
 
-    def setUp(self):
-        self.map0 = {
+    @classmethod
+    def setUpClass(cls):
+        cls.map0 = {
             'hello': 'world',
             'missingFrom1': True,
             'infKeff': arange(2),
@@ -331,7 +341,7 @@ class SplitDictionaryTester(TestCase):
             'anaKeff': arange(6),
             'notBool': 1,
         }
-        self.map1 = {
+        cls.map1 = {
             'hello': 'world',
             'infKeff': arange(2),
             'float': 0.24,
@@ -340,9 +350,9 @@ class SplitDictionaryTester(TestCase):
             'anaKeff': arange(2),
             'absKeff': arange(2),
         }
-        self.badTypes = {'notBool': (int, bool)}
-        self.badShapes = {'anaKeff': ((6, ), (2, )), }
-        self.goodKeys = {'hello', 'absKeff', 'float', 'infKeff', }
+        cls.badTypes = {'notBool': (int, bool)}
+        cls.badShapes = {'anaKeff': ((6, ), (2, )), }
+        cls.goodKeys = {'hello', 'absKeff', 'float', 'infKeff', }
 
     def callTest(self, keySet=None):
         return splitDictByKeys(self.map0, self.map1, keySet)