Repository: NeurodataWithoutBorders/nwbinspector (language: python)

Issue:
[New Check] Entire column of Timeseries is NaN
Solves half of #155

Only applies to 1D and 2D TimeSeries data - any higher than that and the multi-axis structure could itself contain information; *e.g.*, if writing an ImageSeries `(frames x width x height x 3 color channels)`, if a single `(x,y)` pixel is `NaN` for all frames and color channels, you'd still want to keep it in the dataset to maintain the video structure.

But for 1D data, if the entire data is `NaN`, I can't think of a case why this data was even written to the NWBFile.

Likewise for 2D data, it would be suggested for example, on an ElectricalSeries, to simply not write those channels that never seem to specify any actual data.

Note that it is fairly common to find blocks of `NaN` data interspersed within TimeSeries - the `subframe_selection` method thus attempts to sample an even spread of the `nelems` over the range of the number of frames.
[Add Check]: entire col/row of table is not NaN, and entire axis of TimeSeries.data is not NaN
### What would you like to see added to the NWBInspector?

We were in the process of making something like this inspired by the previous commented code, but was also discovered in a recent manual inspection.

Basically, an entire row or column of a DynamicTable, or an entire axis of a TimeSeries, should not all be NaN.

There are some exceptions that we have to observe such as `x,y,z` for ElectrodeTable. Basically any field that is required schematically but unknown in practice, we'd have to skip. But for axes optionally added or certainly custom fields, they should not be included if they are unknown.

### Pseudocode or other logic of the check

```shell
all(np.isnan(getattr(obj, "data")))
```


### Do you have any interest in helping implement the check function?

Yes.

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/NeurodataWithoutBorders/nwbinspector/blob/dev/.github/CODE_OF_CONDUCT.rst)
- [X] Have you ensured this check was not already [requested](https://github.com/neurodatawithoutborders/nwbinspector/issues)?


Discussion / hints:

Fix patch:
diff --git a/nwbinspector/checks/tables.py b/nwbinspector/checks/tables.py
index 3d28a4728..df83a3bb5 100644
--- a/nwbinspector/checks/tables.py
+++ b/nwbinspector/checks/tables.py
@@ -188,3 +188,22 @@ def check_table_values_for_dict(table: DynamicTable, nelems: int = 200):
                 if is_string_json_loadable(string=string):
                     message += " This string is also JSON loadable, so call `json.loads(...)` on the string to unpack."
                 yield InspectorMessage(message=message)
+
+
+@register_check(importance=Importance.BEST_PRACTICE_SUGGESTION, neurodata_type=DynamicTable)
+def check_col_not_nan(table: DynamicTable, nelems: Optional[int] = 200):
+    """Check if all of the values in a single column of a table are NaN."""
+    for column in table.columns:
+        if not hasattr(column, "data") or isinstance(column, VectorIndex) or isinstance(column.data[0], str):
+            continue
+        if nelems is not None and not all(np.isnan(column[:nelems]).flatten()):
+            continue
+
+        if all(
+            np.isnan(
+                column[slice(0, None, np.ceil(len(column.data) / nelems).astype(int) if nelems else None)]
+            ).flatten()
+        ):
+            yield InspectorMessage(
+                message=f"Column {column.name} has all NaN values. Consider removing it from the table."
+            )


Test patch:
diff --git a/tests/unit_tests/test_tables.py b/tests/unit_tests/test_tables.py
index 4f1a47279..d25f98483 100644
--- a/tests/unit_tests/test_tables.py
+++ b/tests/unit_tests/test_tables.py
@@ -17,6 +17,7 @@
     check_column_binary_capability,
     check_single_row,
     check_table_values_for_dict,
+    check_col_not_nan,
 )
 from nwbinspector.utils import get_package_version
 
@@ -237,8 +238,8 @@ def test_check_single_row_ignore_units():
 
 def test_check_single_row_ignore_electrodes():
     table = ElectrodeTable(
-        name="electrodes",  # default name when building through nwbfile
-    )
+        name="electrodes",
+    )  # default name when building through nwbfile
     if get_package_version(name="pynwb") >= version.Version("2.1.0"):
         table.add_row(
             location="unknown",
@@ -291,7 +292,7 @@ def test_check_table_values_for_dict_pass():
     assert check_table_values_for_dict(table=table) is None
 
 
-def test_check_table_values_for_dict():
+def test_check_table_values_for_dict_fail():
     table = DynamicTable(name="test_table", description="")
     table.add_column(name="test_column", description="")
     table.add_row(test_column=str(dict(a=1)))
@@ -308,19 +309,59 @@ def test_check_table_values_for_dict():
     )
 
 
-def test_check_table_values_for_dict_json_case():
+def test_check_table_values_for_dict_json_case_fail():
     table = DynamicTable(name="test_table", description="")
     table.add_column(name="test_column", description="")
     table.add_row(test_column=json.dumps(dict(a=1)))
-    assert check_table_values_for_dict(table=table)[0] == InspectorMessage(
-        message=(
-            "The column 'test_column' contains a string value that contains a dictionary! Please unpack "
-            "dictionaries as additional rows or columns of the table. This string is also JSON loadable, so call "
-            "`json.loads(...)` on the string to unpack."
+    assert check_table_values_for_dict(table=table) == [
+        InspectorMessage(
+            message=(
+                "The column 'test_column' contains a string value that contains a dictionary! Please unpack "
+                "dictionaries as additional rows or columns of the table. This string is also JSON loadable, so call "
+                "`json.loads(...)` on the string to unpack."
+            ),
+            importance=Importance.BEST_PRACTICE_VIOLATION,
+            check_function_name="check_table_values_for_dict",
+            object_type="DynamicTable",
+            object_name="test_table",
+            location="/",
+        )
+    ]
+
+
+def test_check_col_not_nan_pass():
+    table = DynamicTable(name="test_table", description="")
+    for name in ["test_column_not_nan", "test_column_string"]:
+        table.add_column(name=name, description="")
+    table.add_row(test_column_not_nan=1.0, test_column_string="abc")
+    assert check_col_not_nan(table=table) is None
+
+
+def test_check_col_not_nan_fail():
+    table = DynamicTable(name="test_table", description="")
+    for name in ["test_column_not_nan_1", "test_column_nan_1", "test_column_not_nan_2", "test_column_nan_2"]:
+        table.add_column(name=name, description="")
+    for _ in range(400):
+        table.add_row(
+            test_column_not_nan_1=1.0, test_column_nan_1=np.nan, test_column_not_nan_2=1.0, test_column_nan_2=np.nan
+        )
+    assert check_col_not_nan(table=table) == [
+        InspectorMessage(
+            message="Column test_column_nan_1 has all NaN values. Consider removing it from the table.",
+            importance=Importance.BEST_PRACTICE_SUGGESTION,
+            check_function_name="check_col_not_nan",
+            object_type="DynamicTable",
+            object_name="test_table",
+            location="/",
+            file_path=None,
         ),
-        importance=Importance.BEST_PRACTICE_VIOLATION,
-        check_function_name="check_table_values_for_dict",
-        object_type="DynamicTable",
-        object_name="test_table",
-        location="/",
-    )
+        InspectorMessage(
+            message="Column test_column_nan_2 has all NaN values. Consider removing it from the table.",
+            importance=Importance.BEST_PRACTICE_SUGGESTION,
+            check_function_name="check_col_not_nan",
+            object_type="DynamicTable",
+            object_name="test_table",
+            location="/",
+            file_path=None,
+        ),
+    ]