Repository: ARM-DOE/ACT

Issue:
ENH: Enable Hour/Minute Level of Searching with ARMLive API
Currently, the hour/minute/second field of the ARM data access API is not being passed to the query, preventing the ability to search at this level. This is an issue when working with larger datasets (ex. x-band radar) where a day of data is 10s of GB in size and a user wants a small subset.

I put together some new logic over in this PR (https://github.com/pangeo-forge/staged-recipes/pull/153) for the Pangeo Forge recipe.

This would be good to bring into ACT + update so users can use more refined searches.


Discussion:
@mgrover1 there is a way to set a time to download a specific file but not files within a specific time range.  Having that option would be useful.

https://github.com/ARM-DOE/ACT/blob/2e598c8f0aec1b36a773f16c65783e01f3fdb7b2/act/discovery/get_armfiles.py#L19 
The question is how do we add this into the API...

Right now, the function takes in `startdate` and `enddate`, which are both strings. We basically have three options:

1. Allow start date and end date to be datetime objects, which allows us to pass seconds directly in there.
```python
startdate = datetime(2022, 7, 19, 12, 0)
enddate = datetime(2022, 7, 19, 14, 0)

data = download_data(...,  startdate, enddate)
```

2. Add a starttime and endtime (formatted as `HHMMSS` as in `time`) field into the function

```python
startdate = "2022-07-09"
starttime = "120000"
enddate = "2022-07-09"
endtime = "140000"

data = download_data(...,  startdate, enddate)
``` 

3. Add additional options to the `date_parser` function to allow seconds as well `(['%Y-%m-%d', '%d.%m.%Y',
                 '%d/%m/%Y', '%Y%m%d', '%Y/%m/%d'])`
```python
startdate = "2022-07-09T12:00:00"
enddate = "2022-07-09T14:00:00"

data = download_data(...,  startdate, enddate)
``` 

And with that last option, we would need to decide which valid strings to accept.

My personal preference would be option 1, since this is more generic + extensible

@mgrover1 I would lean towards option 3 and limit the options mainly because it's less change for the user and sticks with the ARM date convention YYYMMDD.  For all the formats we currently accept (YYYY-MM-DD, DD.MM.YYYY, DD/MM/YYYY, YYYYMMDD or YYYY/MM/DD), could we just add the option to look for 'THHMMSS' onto it?


We can - so should we support:
* HH:MM:SS
* THH:MM:SS
* HH-MM-SS

Should we just append onto each? Should we allow spaces in between?
ex. DD.MM.YYYYTHH:MM:SS, DD.MM.YYYYTHH:MM:SS

@mgrover1 I think we should try and support what we currently have 

- YYYY-MM-DD
- DD.MM.YYYY
- DD/MM/YYYY
- YYYYMMDD
- YYYY/MM/DD

and then allow the addition of THH:MM:SS onto each.  Would this be feasible?
Yup! I can submit a PR with that!
So to confirm, the support formats for starttime and endtime would be:
- YYYY-MM-DD
- DD.MM.YYYY
- DD/MM/YYYY
- YYYYMMDD
- YYYY/MM/DD
- YYYY-MM-DDTHH:MM:SS
- DD.MM.YYYYTHH:MM:SS
- DD/MM/YYYYTHH:MM:SS
- YYYYMMDDTHH:MM:SS
- YYYY/MM/DDTHH:MM:SS
Just trying to think how we would this word this with the doc string/API... 

Resolving patch:
diff --git a/act/discovery/get_armfiles.py b/act/discovery/get_armfiles.py
--- a/act/discovery/get_armfiles.py
+++ b/act/discovery/get_armfiles.py
@@ -31,10 +31,14 @@ def download_data(username, token, datastream, startdate, enddate, time=None, ou
         The name of the datastream to acquire.
     startdate : str
         The start date of the data to acquire. Formats accepted are
-        YYYY-MM-DD, DD.MM.YYYY, DD/MM/YYYY, YYYYMMDD or YYYY/MM/DD.
+        YYYY-MM-DD, DD.MM.YYYY, DD/MM/YYYY, YYYYMMDD, YYYY/MM/DD or
+        any of the previous formats with THH:MM:SS added onto the end
+        (ex. 2020-09-15T12:00:00).
     enddate : str
         The end date of the data to acquire. Formats accepted are
-        YYYY-MM-DD, DD.MM.YYYY, DD/MM/YYYY, YYYYMMDD or YYYY/MM/DD.
+        YYYY-MM-DD, DD.MM.YYYY, DD/MM/YYYY, YYYYMMDD or YYYY/MM/DD, or
+        any of the previous formats with THH:MM:SS added onto the end
+        (ex. 2020-09-15T13:00:00).
     time: str or None
         The specific time. Format is HHMMSS. Set to None to download all files
         in the given date interval.
@@ -92,10 +96,10 @@ def download_data(username, token, datastream, startdate, enddate, time=None, ou
     # start and end strings for query_url are constructed
     # if the arguments were provided
     if startdate:
-        start = date_parser(startdate, output_format='%Y-%m-%d')
+        start = date_parser(startdate, output_format='%Y-%m-%dT%H:%M:%SZ')
         start = f'&start={startdate}'
     if enddate:
-        end = date_parser(enddate, output_format='%Y-%m-%d')
+        end = date_parser(enddate, output_format='%Y-%m-%dT%H:%M:%SZ')
         end = f'&end={enddate}'
     # build the url to query the web service using the arguments provided
     query_url = (
diff --git a/act/utils/datetime_utils.py b/act/utils/datetime_utils.py
--- a/act/utils/datetime_utils.py
+++ b/act/utils/datetime_utils.py
@@ -168,9 +168,8 @@ def datetime64_to_datetime(time):
     return datetime_array
 
 
-def date_parser(date_string, output_format='%Y%m%d',
-                return_datetime=False):
-    """ Converts one datetime string to another or to
+def date_parser(date_string, output_format='%Y%m%d', return_datetime=False):
+    """Converts one datetime string to another or to
     a datetime object.
 
     Parameters
@@ -192,8 +191,18 @@ def date_parser(date_string, output_format='%Y%m%d',
         A datetime object.
 
     """
-    date_fmts = ['%Y-%m-%d', '%d.%m.%Y',
-                 '%d/%m/%Y', '%Y%m%d', '%Y/%m/%d']
+    date_fmts = [
+        '%Y-%m-%d',
+        '%d.%m.%Y',
+        '%d/%m/%Y',
+        '%Y%m%d',
+        '%Y/%m/%d',
+        '%Y-%m-%dT%H%M%S',
+        '%d.%m.%YT%H%M%S',
+        '%d/%m/%YT%H%M%S',
+        '%Y%m%dT%H%M%S',
+        '%Y/%m/%dT%H%M%S',
+    ]
     for fmt in date_fmts:
         try:
             datetime_obj = dt.datetime.strptime(date_string, fmt)
@@ -204,6 +213,4 @@ def date_parser(date_string, output_format='%Y%m%d',
         except ValueError:
             pass
     fmt_strings = ', '.join(date_fmts)
-    raise ValueError(
-        'Invalid Date format, please use one of these formats '
-        + fmt_strings)
+    raise ValueError('Invalid Date format, please use one of these formats ' + fmt_strings)


Test patch:
diff --git a/act/tests/test_discovery.py b/act/tests/test_discovery.py
--- a/act/tests/test_discovery.py
+++ b/act/tests/test_discovery.py
@@ -1,8 +1,9 @@
 import glob
 import os
+from datetime import datetime
+
 import numpy as np
 import requests
-from datetime import datetime
 from requests.packages.urllib3.exceptions import InsecureRequestWarning
 
 import act
@@ -98,6 +99,53 @@ def test_get_armfile():
         assert len(results) == 1
 
 
+def test_get_armfile_hourly():
+    if not os.path.isdir(os.getcwd() + '/data/'):
+        os.makedirs(os.getcwd() + '/data/')
+
+    # Place your username and token here
+    username = os.getenv('ARM_USERNAME')
+    token = os.getenv('ARM_PASSWORD')
+
+    if username is not None and token is not None:
+        if len(username) == 0 and len(token) == 0:
+            return
+        datastream = 'sgpmetE13.b1'
+        startdate = '2020-01-01T00:00:00'
+        enddate = '2020-01-01T12:00:00'
+        outdir = os.getcwd() + '/data/'
+
+        results = act.discovery.get_armfiles.download_data(
+            username, token, datastream, startdate, enddate, output=outdir
+        )
+        files = glob.glob(outdir + datastream + '*20200101*cdf')
+        if len(results) > 0:
+            assert files is not None
+            assert 'sgpmetE13' in files[0]
+
+        if files is not None:
+            if len(files) > 0:
+                os.remove(files[0])
+
+        datastream = 'sgpmeetE13.b1'
+        act.discovery.get_armfiles.download_data(
+            username, token, datastream, startdate, enddate, output=outdir
+        )
+        files = glob.glob(outdir + datastream + '*20200101*cdf')
+        assert len(files) == 0
+
+        with np.testing.assert_raises(ConnectionRefusedError):
+            act.discovery.get_armfiles.download_data(
+                username, token + '1234', datastream, startdate, enddate, output=outdir
+            )
+
+        datastream = 'sgpmetE13.b1'
+        results = act.discovery.get_armfiles.download_data(
+            username, token, datastream, startdate, enddate
+        )
+        assert len(results) == 1
+
+
 def test_airnow():
     token = os.getenv('AIRNOW_API')
     if token is not None:
@@ -108,7 +156,9 @@ def test_airnow():
         assert results['AQI'].values[2] == -1
         assert results['ReportingArea'][3] == 'Chicago'
 
-        results = act.discovery.get_airnow_forecast(token, '2022-05-01', distance=50, latlon=[41.958, -88.12])
+        results = act.discovery.get_airnow_forecast(
+            token, '2022-05-01', distance=50, latlon=[41.958, -88.12]
+        )
         assert results['CategoryName'].values[3] == 'Good'
         assert results['AQI'].values[2] == -1
         assert results['ReportingArea'][3] == 'Aurora and Elgin'
@@ -128,46 +178,73 @@ def test_airnow():
         with np.testing.assert_raises(NameError):
             results = act.discovery.get_airnow_forecast(token, '2022-05-01')
 
-        results = act.discovery.get_airnow_obs(token, date='2022-05-01', distance=50, latlon=[41.958, -88.12])
+        results = act.discovery.get_airnow_obs(
+            token, date='2022-05-01', distance=50, latlon=[41.958, -88.12]
+        )
         assert results['AQI'].values[0] == 30
         assert results['ParameterName'].values[1] == 'PM2.5'
         assert results['CategoryName'].values[0] == 'Good'
 
         lat_lon = '-88.245401,41.871346,-87.685099,42.234359'
-        results = act.discovery.get_airnow_bounded_obs(token, '2022-05-01T00', '2022-05-01T12', lat_lon,
-                                                       'OZONE,PM25', data_type='B')
+        results = act.discovery.get_airnow_bounded_obs(
+            token, '2022-05-01T00', '2022-05-01T12', lat_lon, 'OZONE,PM25', data_type='B'
+        )
         assert results['PM2.5'].values[-1, 0] == 1.8
         assert results['OZONE'].values[0, 0] == 37.0
         assert len(results['time'].values) == 13
 
 
 def test_noaa_psl():
-    result = act.discovery.download_noaa_psl_data(site='ctd', instrument='Parsivel', startdate='20211231',
-                                                  enddate='20220101', output='./data/')
+    result = act.discovery.download_noaa_psl_data(
+        site='ctd',
+        instrument='Parsivel',
+        startdate='20211231',
+        enddate='20220101',
+        output='./data/',
+    )
     assert len(result) == 48
 
-    result = act.discovery.download_noaa_psl_data(site='ctd', instrument='Pressure', startdate='20220101',
-                                                  hour='00')
+    result = act.discovery.download_noaa_psl_data(
+        site='ctd', instrument='Pressure', startdate='20220101', hour='00'
+    )
     assert len(result) == 1
 
-    result = act.discovery.download_noaa_psl_data(site='ctd', instrument='GpsTrimble', startdate='20220104',
-                                                  hour='00')
+    result = act.discovery.download_noaa_psl_data(
+        site='ctd', instrument='GpsTrimble', startdate='20220104', hour='00'
+    )
     assert len(result) == 6
 
-    types = ['Radar S-band Moment', 'Radar S-band Bright Band', '449RWP Bright Band',
-             '449RWP Wind', '449RWP Sub-Hour Wind', '449RWP Sub-Hour Temp', '915RWP Wind',
-             '915RWP Temp', '915RWP Sub-Hour Wind', '915RWP Sub-Hour Temp']
+    types = [
+        'Radar S-band Moment',
+        'Radar S-band Bright Band',
+        '449RWP Bright Band',
+        '449RWP Wind',
+        '449RWP Sub-Hour Wind',
+        '449RWP Sub-Hour Temp',
+        '915RWP Wind',
+        '915RWP Temp',
+        '915RWP Sub-Hour Wind',
+        '915RWP Sub-Hour Temp',
+    ]
     for t in types:
-        result = act.discovery.download_noaa_psl_data(site='ctd', instrument=t, startdate='20220601', hour='01')
+        result = act.discovery.download_noaa_psl_data(
+            site='ctd', instrument=t, startdate='20220601', hour='01'
+        )
         assert len(result) == 1
 
     types = ['Radar FMCW Moment', 'Radar FMCW Bright Band']
     files = [3, 1]
     for i, t in enumerate(types):
-        result = act.discovery.download_noaa_psl_data(site='bck', instrument=t, startdate='20220101', hour='01')
+        result = act.discovery.download_noaa_psl_data(
+            site='bck', instrument=t, startdate='20220101', hour='01'
+        )
         assert len(result) == files[i]
 
     with np.testing.assert_raises(ValueError):
-        result = act.discovery.download_noaa_psl_data(instrument='Parsivel', startdate='20220601', hour='01')
+        result = act.discovery.download_noaa_psl_data(
+            instrument='Parsivel', startdate='20220601', hour='01'
+        )
     with np.testing.assert_raises(ValueError):
-        result = act.discovery.download_noaa_psl_data(site='ctd', instrument='dongle', startdate='20220601', hour='01')
+        result = act.discovery.download_noaa_psl_data(
+            site='ctd', instrument='dongle', startdate='20220601', hour='01'
+        )