Repository: openradar/xradar

Issue:
Bugs in NEXRAD L2 reader
* xradar version: 0.5.0
* Python version: 3.11
* Operating System: Linux

### Description
 
I identified two bugs affecting the NEXRAD L2 reader. 

- The first bug cause the `time` coordinates and `time_coverage` variables to be shifted by 1 day compared to the actual time. **This impacts all NEXRAD L2 files**.
- The second bug is related to an indexing error arising with some NEXRAD files. 

Can someone have a look at it ? Maybe @mgrover1 who implemented the reader? 

Here below I provide MCVE and error logs. 

----------------------------------------------------------------------------
**MCVE - Time is shifted by 1 day**

```
import xradar as xd
import fsspec # s3fs need to be installed 
filepath = 'noaa-nexrad-level2/2023/08/20/KNKX/KNKX20230820_220808_V06' 
file = fsspec.open_local(
    f"simplecache::s3://{filepath}",
    s3={"anon": True},
    filecache={"cache_storage": "."},
dt_gr = xd.io.open_nexradlevel2_datatree(file)
print(dt_gr["time_coverage_start"])  # '2023-08-21T22:08:08Z'
print(dt_gr["time_coverage_end"]) # '2023-08-21T22:13:33Z'
print(dt_gr["sweep_0"]["time"]) # you will see the date is +1 day
```

----------------------------------------------------------------------------
**MCVE - `IndexError: list index out of range`**

```
import xradar as xd
import fsspec # s3fs need to be installed 
filepath = 'noaa-nexrad-level2/2023/08/19/KNKX/KNKX20230819_220909_V06'
file = fsspec.open_local(
    f"simplecache::s3://{filepath}",
    s3={"anon": True},
    filecache={"cache_storage": "."},
dt_gr = xd.io.open_nexradlevel2_datatree(file)
```
Error log:
```
  File ~/micromamba/envs/gpm-api-dev/lib/python3.11/site-packages/xradar/io/backends/nexrad_level2.py:485 in get_sweep
    moments = self.msg_31_data_header[sweep_number]["msg_31_data_header"].keys()

IndexError: list index out of range
```

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/openradar/xradar/pull/181?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) Report
Attention: Patch coverage is `90.00000%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 77.19%. Comparing base [(`2d266bb`)](https://app.codecov.io/gh/openradar/xradar/commit/2d266bb46e7ac95dd4df9c1d8dcf16578c8c5566?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) to head [(`bfb6f41`)](https://app.codecov.io/gh/openradar/xradar/commit/bfb6f415c43dc6afe7d330e4e1cf511da27f4e84?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar).

| [Files](https://app.codecov.io/gh/openradar/xradar/pull/181?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) | Patch % | Lines |
|---|---|---|
| [xradar/io/backends/nexrad\_level2.py](https://app.codecov.io/gh/openradar/xradar/pull/181?src=pr&el=tree&filepath=xradar%2Fio%2Fbackends%2Fnexrad_level2.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar#diff-eHJhZGFyL2lvL2JhY2tlbmRzL25leHJhZF9sZXZlbDIucHk=) | 90.00% | [2 Missing :warning: ](https://app.codecov.io/gh/openradar/xradar/pull/181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) |

> :exclamation:  There is a different number of reports uploaded between BASE (2d266bb) and HEAD (bfb6f41). Click for more details.
> 
> <details><summary>HEAD has 6 uploads less than BASE</summary>
>
>| Flag | BASE (2d266bb) | HEAD (bfb6f41) |
>|------|------|------|
>|unittests|6|0|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##             main     #181       +/-   ##
===========================================
- Coverage   91.70%   77.19%   -14.51%     
===========================================
  Files          22       22               
  Lines        4231     4236        +5     
===========================================
- Hits         3880     3270      -610     
- Misses        351      966      +615     
```

| [Flag](https://app.codecov.io/gh/openradar/xradar/pull/181/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) | Coverage Δ | |
|---|---|---|
| [notebooktests](https://app.codecov.io/gh/openradar/xradar/pull/181/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) | `77.19% <90.00%> (-0.03%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/openradar/xradar/pull/181/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/openradar/xradar/pull/181?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openradar).

ghiggi: Cool @kmuehlbauer ! Thanks to have solved this.
I guess that for the end of next week, on my side, I can make a new release of gpm-api with a tutorial on how to perform SR/GR calibration "on-the-fly" using xradar and various radar datasets available on S3 (nexrad, columbian and fmi). I will ping you when ready and then we can discuss what to use for the ERAD short course ;) 
kmuehlbauer: I think this is good to go. I'll merge tomorrow, if no one beats me to it.
mgrover1: This looks great! Good to merge!
mgrover1: @kmuehlbauer - is it okay if I go ahead and merge? This looks great!
kmuehlbauer: Oh my, I thought I've pressed the merge button already. Yes, please go ahead!

Patch:
diff --git a/docs/history.md b/docs/history.md
index 4d2de54..cce02e2 100644
--- a/docs/history.md
+++ b/docs/history.md
@@ -6,6 +6,8 @@
 * ADD: Add Alfonso to citation doc ({pull}`169`) by [@mgrover1](https://github.com/mgrover1).
 * ENH: Adding global variables and attributes to iris datatree ({pull}`166`) by [@aladinor](https://github.com/aladinor).
 * FIX: Set fillvalue before applying scale/offset when exporting to odim ({issue}`122`) by [@pavlikp](https://github.com/pavlikp), ({pull}`173`) by [@kmuehlbauer](https://github.com/kmuehlbauer).
+* FIX: Nexrad level2 time offset of 1 day, skip reading missing elevations, introduce new radial_status of 5
+ ({issue}`180`) by [@ghiggi](https://github.com/ghiggi), ({pull}`180`) by [@kmuehlbauer](https://github.com/kmuehlbauer).
 * ADD: Reader for Halo Photonics Doppler lidar data by [@rcjackson](https://github.com/rcjackson)
 
 ## 0.5.0 (2024-03-28)
diff --git a/xradar/io/backends/nexrad_level2.py b/xradar/io/backends/nexrad_level2.py
index 74bb8a0..5a951eb 100644
--- a/xradar/io/backends/nexrad_level2.py
+++ b/xradar/io/backends/nexrad_level2.py
@@ -644,18 +644,26 @@ class NEXRADLevel2File(NEXRADRecordFile):
                 msg_31_header["filepos"] = self.filepos
                 # retrieve data/const headers from msg 31
                 # check if this is a new sweep
-                if msg_31_header["radial_status"] in [0, 3]:
-                    # do not run for the first time
-                    if current_sweep > -1:
-                        sweep["record_end"] = self.rh.recnum - 1
-                        sweep["intermediate_records"] = sweep_intermediate_records
-                        self._data[current_sweep] = sweep
-                        # create new sweep object
-                        sweep = OrderedDict()
-                        _msg_31_header.append(sweep_msg_31_header)
-                        sweep_msg_31_header = []
-                        sweep_intermediate_records = []
+                status = msg_31_header["radial_status"]
+                if status == 1:
+                    # 1 - intermediate radial
+                    pass
+                if status in [2, 4]:
+                    # 2 - end of elevation
+                    # 4 - end of volume
+                    sweep["record_end"] = self.rh.recnum
+                    sweep["intermediate_records"] = sweep_intermediate_records
+                    self._data[current_sweep] = sweep
+                    _msg_31_header.append(sweep_msg_31_header)
+                if status in [0, 3, 5]:
+                    # 0 - start of new elevation
+                    # 3 - start of new volume
+                    # 5 - start of new elevation, last elevation in VCP
                     current_sweep += 1
+                    # create new sweep object
+                    sweep = OrderedDict()
+                    sweep_msg_31_header = []
+                    sweep_intermediate_records = []
                     sweep["record_number"] = self.rh.recnum
                     sweep["filepos"] = self.filepos
                     sweep["msg_31_header"] = msg_31_header
@@ -699,11 +707,6 @@ class NEXRADLevel2File(NEXRADRecordFile):
             else:
                 sweep_intermediate_records.append(msg_header)
 
-        # finalize last sweep
-        sweep["record_end"] = self.rh.recnum
-        sweep["intermediate_records"] = sweep_intermediate_records
-        self._data[current_sweep] = sweep
-        _msg_31_header.append(sweep_msg_31_header)
         return data_header, _msg_31_header, _msg_31_data_header
 
     def _check_record(self):
@@ -1333,7 +1336,8 @@ class NexradLevel2Store(AbstractDataStore):
         elevation = np.array([ms["elevation_angle"] for ms in msg_31_header])
 
         # time
-        date = np.array([ms["collect_date"] for ms in msg_31_header]) * 86400e3
+        # date is 1-based (1 -> 1970-01-01T00:00:00), so we need to subtract 1
+        date = (np.array([ms["collect_date"] for ms in msg_31_header]) - 1) * 86400e3
         milliseconds = np.array([ms["collect_ms"] for ms in msg_31_header])
         rtime = date + milliseconds
         time_prefix = "milli"
@@ -1533,10 +1537,15 @@ def open_nexradlevel2_datatree(filename_or_obj, **kwargs):
 
     engine = NexradLevel2BackendEntrypoint
 
-    ds = [
-        xr.open_dataset(filename_or_obj, group=swp, engine=engine, **kwargs)
-        for swp in sweeps
-    ]
+    # todo: only open once! Needs new xarray built in datatree!
+    ds = []
+    for swp in sweeps:
+        try:
+            dsx = xr.open_dataset(filename_or_obj, group=swp, engine=engine, **kwargs)
+        except IndexError:
+            break
+        else:
+            ds.append(dsx)
 
     ds.insert(0, xr.Dataset())
 


Test patch:
diff --git a/tests/io/test_io.py b/tests/io/test_io.py
index 291919d..3417c57 100644
--- a/tests/io/test_io.py
+++ b/tests/io/test_io.py
@@ -940,8 +940,8 @@ def test_open_nexradlevel2_datatree(nexradlevel2_files):
     assert rvars["volume_number"] == 0
     assert rvars["platform_type"] == "fixed"
     assert rvars["instrument_type"] == "radar"
-    assert rvars["time_coverage_start"] == "2016-06-02T15:00:25Z"
-    assert rvars["time_coverage_end"] == "2016-06-02T15:06:06Z"
+    assert rvars["time_coverage_start"] == "2016-06-01T15:00:25Z"
+    assert rvars["time_coverage_end"] == "2016-06-01T15:06:06Z"
     np.testing.assert_almost_equal(rvars["latitude"].values, np.array(33.65414047))
     np.testing.assert_almost_equal(rvars["longitude"].values, np.array(-101.81416321))
     np.testing.assert_almost_equal(rvars["altitude"].values, np.array(1029))
diff --git a/tests/io/test_nexrad_level2.py b/tests/io/test_nexrad_level2.py
index 54307be..f46842a 100644
--- a/tests/io/test_nexrad_level2.py
+++ b/tests/io/test_nexrad_level2.py
@@ -24,6 +24,9 @@ def test_open_nexradlevel2_datatree(nexradlevel2_files):
     dtree = open_nexradlevel2_datatree(nexradlevel2_files)
     ds = dtree["sweep_0"].ds
     assert ds.attrs["instrument_name"] == "KLBB"
+    assert ds["time"].min() == np.array(
+        "2016-06-01T15:00:25.232000000", dtype="datetime64[ns]"
+    )
     assert ds["DBZH"].shape == (720, 1832)
     assert ds["DBZH"].dims == ("azimuth", "range")
     assert int(ds.sweep_number.values) == 0