Repository: qiime2/qiime2 (python)

Issue:
Allow Metadata constructor(s) to optionally strip whitespace
**Improvement Description**
A kwarg in the various Metadata constructors to strip whitespace in all columns could be really convenient for transformers et al.

**Current Behavior**
Whitespace is not stripped.

**Proposed Behavior**
If some kwarg (say, `strip_whitespace`) was True, the MD constructor would strip whitespace.

**Questions**
1. This could also just be the default behavior, right? Am I missing something here?


Pull request:
IMP: Metadata constructor now strips whitespace
Closes #478. Strips all white space from all pieces of the Metadata

Patch:
diff --git a/README.md b/README.md
index df9de93..ea2c51b 100644
--- a/README.md
+++ b/README.md
@@ -23,4 +23,4 @@ Please visit the [contributing page](https://github.com/qiime2/qiime2/blob/maste
 ## Citing QIIME 2
 If you use QIIME 2 for any published research, please include the following citation:
 
-> Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology. https://doi.org/10.1038/s41587-019-0209-9
+> Bolyen E, Rideout JR, Dillon MR, Bokulich NA, Abnet CC, Al-Ghalith GA, Alexander H, Alm EJ, Arumugam M, Asnicar F, Bai Y, Bisanz JE, Bittinger K, Brejnrod A, Brislawn CJ, Brown CT, Callahan BJ, Caraballo-Rodríguez AM, Chase J, Cope EK, Da Silva R, Diener C, Dorrestein PC, Douglas GM, Durall DM, Duvallet C, Edwardson CF, Ernst M, Estaki M, Fouquier J, Gauglitz JM, Gibbons SM, Gibson DL, Gonzalez A, Gorlick K, Guo J, Hillmann B, Holmes S, Holste H, Huttenhower C, Huttley GA, Janssen S, Jarmusch AK, Jiang L, Kaehler BD, Kang KB, Keefe CR, Keim P, Kelley ST, Knights D, Koester I, Kosciolek T, Kreps J, Langille MGI, Lee J, Ley R, Liu YX, Loftfield E, Lozupone C, Maher M, Marotz C, Martin BD, McDonald D, McIver LJ, Melnik AV, Metcalf JL, Morgan SC, Morton JT, Naimey AT, Navas-Molina JA, Nothias LF, Orchanian SB, Pearson T, Peoples SL, Petras D, Preuss ML, Pruesse E, Rasmussen LB, Rivers A, Robeson MS, Rosenthal P, Segata N, Shaffer M, Shiffer A, Sinha R, Song SJ, Spear JR, Swafford AD, Thompson LR, Torres PJ, Trinh P, Tripathi A, Turnbaugh PJ, Ul-Hasan S, van der Hooft JJJ, Vargas F, Vázquez-Baeza Y, Vogtmann E, von Hippel M, Walters W, Wan Y, Wang M, Warren J, Weber KC, Williamson CHD, Willis AD, Xu ZZ, Zaneveld JR, Zhang Y, Zhu Q, Knight R, and Caporaso JG. 2019. Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2. Nature Biotechnology 37:852–857. https://doi.org/10.1038/s41587-019-0209-9
diff --git a/qiime2/citations.bib b/qiime2/citations.bib
index 99d20a8..263863f 100644
--- a/qiime2/citations.bib
+++ b/qiime2/citations.bib
@@ -3,6 +3,9 @@
   title={Reproducible, interactive, scalable and extensible microbiome data science using QIIME 2},
   journal={Nature Biotechnology},
   year={2019},
+  volume={37},
+  number={8},
+  pages={852-857},
   issn={1546-1696},
   doi={10.1038/s41587-019-0209-9},
   url={https://doi.org/10.1038/s41587-019-0209-9}
diff --git a/qiime2/metadata/metadata.py b/qiime2/metadata/metadata.py
index 36fcee4..a6aa9f8 100644
--- a/qiime2/metadata/metadata.py
+++ b/qiime2/metadata/metadata.py
@@ -156,11 +156,6 @@ class _MetadataBase:
                     "Detected empty metadata %s. %ss must consist of at least "
                     "one character." % (label, label))
 
-            if value != value.strip():
-                raise ValueError(
-                    "Detected metadata %s with leading or trailing "
-                    "whitespace characters: %r" % (label, value))
-
             if axis == 'id' and value.startswith('#'):
                 raise ValueError(
                     "Detected metadata %s that begins with a pound sign "
@@ -362,13 +357,19 @@ class Metadata(_MetadataBase):
         super().__init__(dataframe.index)
 
         self._dataframe, self._columns = self._normalize_dataframe(dataframe)
+        self._validate_index(self._dataframe.columns, axis='column')
 
     def _normalize_dataframe(self, dataframe):
-        self._validate_index(dataframe.columns, axis='column')
-
         norm_df = dataframe.copy()
+
+        # Do not attempt to strip empty metadata
+        if not norm_df.columns.empty:
+            norm_df.columns = norm_df.columns.str.strip()
+
+        norm_df.index = norm_df.index.str.strip()
+
         columns = collections.OrderedDict()
-        for column_name, series in dataframe.items():
+        for column_name, series in norm_df.items():
             metadata_column = self._metadata_column_factory(series)
             norm_df[column_name] = metadata_column.to_series()
             properties = ColumnProperties(type=metadata_column.type)
@@ -870,8 +871,6 @@ class MetadataColumn(_MetadataBase, metaclass=abc.ABCMeta):
 
         super().__init__(series.index)
 
-        self._validate_index([series.name], axis='column')
-
         if not self._is_supported_dtype(series.dtype):
             raise TypeError(
                 "%s %r does not support a pandas.Series object with dtype %s" %
@@ -879,6 +878,8 @@ class MetadataColumn(_MetadataBase, metaclass=abc.ABCMeta):
 
         self._series = self._normalize_(series)
 
+        self._validate_index([self._series.name], axis='column')
+
     def __repr__(self):
         """String summary of the metadata column."""
         return '<%s name=%r id_count=%d>' % (self.__class__.__name__,
@@ -1125,6 +1126,7 @@ class CategoricalMetadataColumn(MetadataColumn):
     def _normalize_(cls, series):
         def normalize(value):
             if isinstance(value, str):
+                value = value.strip()
                 if value == '':
                     raise ValueError(
                         "%s does not support empty strings as values. Use an "
@@ -1132,11 +1134,6 @@ class CategoricalMetadataColumn(MetadataColumn):
                         "(e.g. `numpy.nan`) or supply a non-empty string as "
                         "the value in column %r." %
                         (cls.__name__, series.name))
-                elif value != value.strip():
-                    raise ValueError(
-                        "%s does not support values with leading or trailing "
-                        "whitespace characters. Column %r has the following "
-                        "value: %r" % (cls.__name__, series.name, value))
                 else:
                     return value
             elif pd.isnull(value):  # permits np.nan, Python float nan, None
@@ -1147,7 +1144,10 @@ class CategoricalMetadataColumn(MetadataColumn):
                     "%r of type %r in column %r." %
                     (cls.__name__, value, type(value), series.name))
 
-        return series.apply(normalize, convert_dtype=False)
+        norm_series = series.apply(normalize, convert_dtype=False)
+        norm_series.index = norm_series.index.str.strip()
+        norm_series.name = norm_series.name.strip()
+        return norm_series
 
 
 class NumericMetadataColumn(MetadataColumn):
diff --git a/qiime2/plugin/model/file_format.py b/qiime2/plugin/model/file_format.py
index e32f8a1..575ba03 100644
--- a/qiime2/plugin/model/file_format.py
+++ b/qiime2/plugin/model/file_format.py
@@ -8,6 +8,7 @@
 
 import abc
 
+from qiime2.core import transform
 from .base import FormatBase, ValidationError, _check_validation_level
 
 
@@ -39,6 +40,13 @@ class _FileFormat(FormatBase, metaclass=abc.ABCMeta):
             raise NotImplementedError("%r does not implement validate."
                                       % type(self))
 
+    def view(self, view_type):
+        from_type = transform.ModelType.from_view_type(self.__class__)
+        to_type = transform.ModelType.from_view_type(view_type)
+
+        transformation = from_type.make_transformation(to_type)
+        return transformation(self)
+
 
 class TextFileFormat(_FileFormat):
     def open(self):


Test patch:
diff --git a/qiime2/metadata/tests/test_metadata.py b/qiime2/metadata/tests/test_metadata.py
index 5050b05..9f3cab0 100644
--- a/qiime2/metadata/tests/test_metadata.py
+++ b/qiime2/metadata/tests/test_metadata.py
@@ -76,22 +76,6 @@ class TestInvalidMetadataConstruction(unittest.TestCase):
                 {'col': [1, 2, 3],
                  '': [4, 5, 6]}, index=pd.Index(['a', 'b', 'c'], name='id')))
 
-    def test_leading_trailing_whitespace_id(self):
-        with self.assertRaisesRegex(ValueError, "metadata ID.*leading or "
-                                                "trailing whitespace.*' b '"):
-            Metadata(pd.DataFrame(
-                {'col': [1, 2, 3]},
-                index=pd.Index(['a', ' b ', 'c'], name='id')))
-
-    def test_leading_trailing_whitespace_column_name(self):
-        with self.assertRaisesRegex(
-                ValueError, "metadata column name.*leading or trailing "
-                            "whitespace.*' col2 '"):
-            Metadata(pd.DataFrame(
-                {'col1': [1, 2, 3],
-                 ' col2 ': [4, 5, 6]},
-                index=pd.Index(['a', 'b', 'c'], name='id')))
-
     def test_pound_sign_id(self):
         with self.assertRaisesRegex(
                 ValueError, "metadata ID.*begins with a pound sign.*'#b'"):
@@ -157,15 +141,6 @@ class TestInvalidMetadataConstruction(unittest.TestCase):
                  'col2': ['foo', '', 'bar']},
                 index=pd.Index(['a', 'b', 'c'], name='id')))
 
-    def test_categorical_column_leading_trailing_whitespace_value(self):
-        with self.assertRaisesRegex(
-                ValueError, "CategoricalMetadataColumn.*leading or trailing "
-                            "whitespace characters.*Column 'col2'.*' bar '"):
-            Metadata(pd.DataFrame(
-                {'col1': [1, 2, 3],
-                 'col2': ['foo', ' bar ', 'baz']},
-                index=pd.Index(['a', 'b', 'c'], name='id')))
-
     def test_numeric_column_infinity(self):
         with self.assertRaisesRegex(
                 ValueError, "NumericMetadataColumn.*positive or negative "
@@ -402,6 +377,38 @@ class TestMetadataConstructionAndProperties(unittest.TestCase):
 
         self.assertEqual(set(metadata.columns), {'column', 'Column'})
 
+    def test_categorical_column_leading_trailing_whitespace_value(self):
+        md1 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3],
+             'col2': ['foo', ' bar ', 'baz']},
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+        md2 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3],
+             'col2': ['foo', 'bar', 'baz']},
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(md1, md2)
+
+    def test_leading_trailing_whitespace_id(self):
+        md1 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+            index=pd.Index(['a', ' b ', 'c'], name='id')))
+        md2 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(md1, md2)
+
+    def test_leading_trailing_whitespace_column_name(self):
+        md1 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3], ' col2 ': [4, 5, 6]},
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+        md2 = Metadata(pd.DataFrame(
+            {'col1': [1, 2, 3], 'col2': [4, 5, 6]},
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(md1, md2)
+
 
 class TestSourceArtifacts(unittest.TestCase):
     def setUp(self):
diff --git a/qiime2/metadata/tests/test_metadata_column.py b/qiime2/metadata/tests/test_metadata_column.py
index 26b3647..a763d95 100644
--- a/qiime2/metadata/tests/test_metadata_column.py
+++ b/qiime2/metadata/tests/test_metadata_column.py
@@ -96,21 +96,6 @@ class TestInvalidMetadataColumnConstruction(unittest.TestCase):
                 [1, 2, 3], name='',
                 index=pd.Index(['a', 'b', 'c'], name='id')))
 
-    def test_leading_trailing_whitespace_id(self):
-        with self.assertRaisesRegex(ValueError, "metadata ID.*leading or "
-                                                "trailing whitespace.*' b '"):
-            DummyMetadataColumn(pd.Series(
-                [1, 2, 3], name='col',
-                index=pd.Index(['a', ' b ', 'c'], name='id')))
-
-    def test_leading_trailing_whitespace_column_name(self):
-        with self.assertRaisesRegex(
-                ValueError, "metadata column name.*leading or trailing "
-                            "whitespace.*' col2 '"):
-            DummyMetadataColumn(pd.Series(
-                [1, 2, 3], name=' col2 ',
-                index=pd.Index(['a', 'b', 'c'], name='id')))
-
     def test_pound_sign_id(self):
         with self.assertRaisesRegex(
                 ValueError, "metadata ID.*begins with a pound sign.*'#b'"):
@@ -826,14 +811,6 @@ class TestCategoricalMetadataColumn(unittest.TestCase):
                 ['foo', '', 'bar'], name='col1',
                 index=pd.Index(['a', 'b', 'c'], name='id')))
 
-    def test_leading_trailing_whitespace_value(self):
-        with self.assertRaisesRegex(
-                ValueError, "CategoricalMetadataColumn.*leading or trailing "
-                            "whitespace characters.*Column 'col1'.*' bar '"):
-            CategoricalMetadataColumn(pd.Series(
-                ['foo', ' bar ', 'baz'], name='col1',
-                index=pd.Index(['a', 'b', 'c'], name='id')))
-
     def test_type_property(self):
         self.assertEqual(CategoricalMetadataColumn.type, 'categorical')
 
@@ -899,6 +876,36 @@ class TestCategoricalMetadataColumn(unittest.TestCase):
         pdt.assert_series_equal(obs, exp)
         self.assertEqual(obs.dtype, object)
 
+    def test_leading_trailing_whitespace_value(self):
+        col1 = CategoricalMetadataColumn(pd.Series(
+            ['foo', ' bar ', 'baz'], name='col1',
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+        col2 = CategoricalMetadataColumn(pd.Series(
+            ['foo', 'bar', 'baz'], name='col1',
+            index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(col1, col2)
+
+    def test_leading_trailing_whitespace_id(self):
+        col1 = CategoricalMetadataColumn(pd.Series(
+                ['foo', ' bar ', 'baz'], name='col',
+                index=pd.Index(['a', ' b ', 'c'], name='id')))
+        col2 = CategoricalMetadataColumn(pd.Series(
+                ['foo', ' bar ', 'baz'], name='col',
+                index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(col1, col2)
+
+    def test_leading_trailing_whitespace_column_name(self):
+        col1 = CategoricalMetadataColumn(pd.Series(
+                ['foo', ' bar ', 'baz'], name=' col ',
+                index=pd.Index(['a', 'b', 'c'], name='id')))
+        col2 = CategoricalMetadataColumn(pd.Series(
+                ['foo', ' bar ', 'baz'], name='col',
+                index=pd.Index(['a', 'b', 'c'], name='id')))
+
+        self.assertEqual(col1, col2)
+
 
 class TestNumericMetadataColumn(unittest.TestCase):
     def test_unsupported_dtype(self):
diff --git a/qiime2/plugin/model/tests/test_file_format.py b/qiime2/plugin/model/tests/test_file_format.py
index 4495781..65185d1 100644
--- a/qiime2/plugin/model/tests/test_file_format.py
+++ b/qiime2/plugin/model/tests/test_file_format.py
@@ -11,6 +11,7 @@ import unittest
 import tempfile
 
 import qiime2.plugin.model as model
+from qiime2.core.testing.plugin import SingleIntFormat
 
 
 class TestTextFileFormat(unittest.TestCase):
@@ -59,5 +60,28 @@ class TestTextFileFormat(unittest.TestCase):
             self.assertEqual(b'S', fh.read(1))
 
 
+class TestFileFormat(unittest.TestCase):
+    def setUp(self):
+        self.test_dir = tempfile.TemporaryDirectory(prefix='qiime2-test-temp-')
+
+        path = os.path.join(self.test_dir.name, 'int')
+        with open(path, 'w') as fh:
+            fh.write('1')
+
+        self.format = SingleIntFormat(path, mode='r')
+
+    def tearDown(self):
+        self.test_dir.cleanup()
+
+    def test_view_expected(self):
+        number = self.format.view(int)
+        self.assertEqual(1, number)
+
+    def test_view_invalid_type(self):
+        with self.assertRaisesRegex(
+                Exception, "No transformation.*SingleIntFormat.*float"):
+            self.format.view(float)
+
+
 if __name__ == '__main__':
     unittest.main()