Repository: ARM-DOE/ACT

Issue:
Streaking with missing data in 2D plots
### Description

There is a new issue with pcolormesh() causing streaking when there is larger data gaps between data periods. If you look closely at the plot there is a part with some structure at the start and end of the plotted periods. That is the actual data. The extended streaking part is an artifact of pcolormesh(). The issue is that we use add_nan=True to add one NaN value in the middle of the missing data period. pcolormesh() will extend the plot to ½ that window to the inserted NaN. All attempts at changing this behavior with shading keyword (set_shading in .plot() from timeseriesdisplay()) did not fix this issue.
![epcwsacrS2 a1 vpt 20231102](https://github.com/ARM-DOE/ACT/assets/8261054/90fd62d8-fd13-440d-a864-eb7c55d1158f)


### What I Did
If I modify data_utils.add_in_nan() to add two values during the missing data period, one right after last value and one just before next value, the issue is resolved.
![Figure_1](https://github.com/ARM-DOE/ACT/assets/8261054/4328e4c6-3da2-40a3-a6b5-3aab4d4b2a36)

I have some code ready to go. It will add one NaN value in the middle of the data period for 1D data, same as normal. But will add a value right after last value and right before next value. With two values added the streaking is removed. With v2.0 coming will need to integrate correctly.


Discussion:
That's interesting, yeah I'm not sure the best approach for this, I think your solution is fine, if you want to share the code or submit and PR and we can take a look!

Resolving patch:
diff --git a/act/utils/data_utils.py b/act/utils/data_utils.py
--- a/act/utils/data_utils.py
+++ b/act/utils/data_utils.py
@@ -427,6 +427,7 @@ def add_in_nan(time, data):
         # Leaving code in here in case we need to update.
         # diff = np.diff(time.astype('datetime64[s]'), 1)
         diff = np.diff(time, 1)
+
         # Wrapping in a try to catch error while switching between numpy 1.10 to 1.11
         try:
             mode = stats.mode(diff, keepdims=True).mode[0]
@@ -437,10 +438,24 @@ def add_in_nan(time, data):
         offset = 0
         for i in index[0]:
             corr_i = i + offset
-            time_added = time[corr_i] + (time[corr_i + 1] - time[corr_i]) / 2.0
-            time = np.insert(time, corr_i + 1, time_added)
-            data = np.insert(data, corr_i + 1, np.nan, axis=0)
-            offset += 1
+
+            if len(data.shape) == 1:
+                # For line plotting adding a NaN will stop the connection of the line
+                # between points. So we just need to add a NaN anywhere between the points.
+                corr_i = i + offset
+                time_added = time[corr_i] + (time[corr_i + 1] - time[corr_i]) / 2.0
+                time = np.insert(time, corr_i + 1, time_added)
+                data = np.insert(data, corr_i + 1, np.nan, axis=0)
+                offset += 1
+            else:
+                # For 2D plots need to add a NaN right after and right before the data
+                # to correctly mitigate streaking with pcolormesh.
+                time_added_1 = time[corr_i] + 1  # One time step after
+                time_added_2 = time[corr_i + 1] - 1  # One time step before
+                time = np.insert(time, corr_i + 1, [time_added_1, time_added_2])
+                data = np.insert(data, corr_i + 1, np.nan, axis=0)
+                data = np.insert(data, corr_i + 2, np.nan, axis=0)
+                offset += 2
 
         if time_is_DataArray:
             time = xr.DataArray(time, attrs=time_attributes, dims=time_dims)


Test patch:
diff --git a/act/tests/test_utils.py b/act/tests/test_utils.py
--- a/act/tests/test_utils.py
+++ b/act/tests/test_utils.py
@@ -82,6 +82,17 @@ def test_add_in_nan():
     index = np.where(time_filled == np.datetime64('2019-01-01T01:40'))[0]
     assert index.size == 0
 
+    # Test for 2D data
+    time = np.arange('2019-01-01T01:00', '2019-01-01T02:00', dtype='datetime64[m]')
+    data = np.random.random((len(time), 25))
+
+    time = np.delete(time, range(3, 8))
+    data = np.delete(data, range(3, 8), axis=0)
+    time_filled, data_filled = act.utils.add_in_nan(time, data)
+
+    assert np.count_nonzero(np.isnan(data_filled[3, :])) == 25
+    assert len(time_filled) == len(time) + 2
+
 
 def test_get_missing_value():
     ds = act.io.arm.read_arm_netcdf(act.tests.sample_files.EXAMPLE_EBBR1)