Repository: hydroshare/hydroshare

Issue:
Python strftime can't handle dates before 1900
A user has tried to upload data with a legitimate start date of 1890.

However, when trying to upload it into HydroShare, they were met with the following error:

```
temporal_coverage_data_dict['start_date'] = start_date.strftime('%Y-%m-%d')
ValueError: year=1890 is before 1900; the datetime strftime() methods require year >= 1900
```

It looks like there are [numerous options](https://stackoverflow.com/questions/10263956/use-datetime-strftime-on-years-before-1900-require-year-1900) to fix this.


Discussion:
grep returns the following results for strftime usage.

- [ ] `./django_irods/views.py: daily_date = datetime.datetime.today().strftime('%Y-%m-%d')`
- [x] `./hs_collection_resource/views.py:                       value_dict = {'start': time_start.strftime(UI_DATETIME_FORMAT),`
- [x] `./hs_collection_resource/views.py:                      'end': time_end.strftime(UI_DATETIME_FORMAT)}`
- [x] `./hs_core/discovery_parser.py:                           thisday = thisday_object.strftime("%Y-%m-%dT%H:%M:%SZ")`
- [x] `./hs_core/discovery_parser.py:                           nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")`
- [x] `./hs_core/discovery_parser.py:                           nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")`
- [x] `./hs_core/discovery_parser.py:                           nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")`
- [x] `./hs_core/irods.py:                                      formatted = timeout.strftime("%Y-%m-%d.%H:%M")`
- [x] `./hs_core/page_processors.py:                            temporal_coverage_data_dict['start_date'] = start_date.strftime('%Y-%m-%d')`
- [x] `./hs_core/page_processors.py:                            temporal_coverage_data_dict['end_date'] = end_date.strftime('%Y-%m-%d')`
- [x] `./hs_core/page_processors.py:                             temporal_coverage_data_dict['start'] = start_date.strftime('%m-%d-%Y')`
- [x] `./hs_core/page_processors.py:        temporal_coverage_data_dict['end'] = end_date.strftime('%m-%d-%Y')`
- [x] `./hs_core/views/utils.py:            temporal_coverage_dict['start'] = start_date.strftime('%m-%d-%Y')`
- [x] `./hs_core/views/utils.py:            temporal_coverage_dict['end'] = end_date.strftime('%m-%d-%Y')`
- [x] `./hs_core/serialization.py:                val['start'] = c.start_date.strftime('%m/%d/%Y')`
- [x] `./hs_core/serialization.py:                val['end'] = c.end_date.strftime('%m/%d/%Y')`
- [x] `./hs_core/models.py:                            td(start_date.strftime('%m/%d/%Y'))`
- [x] `./hs_core/models.py:                            td(end_date.strftime('%m/%d/%Y'))`
- [x] `./hs_core/models.py:            coverage_data_dict['start'] = start_date.strftime('%m/%d/%Y')`
- [x] `./hs_core/models.py:            coverage_data_dict['end'] = end_date.strftime('%m/%d/%Y')`
- [x] `./hs_core/tasks.py:    date_folder = (date.today() - timedelta(2)).strftime('%Y-%m-%d')`
- [x] `./hs_core/tasks.py:            pub_date = pub_date.start_date.strftime('%m/%d/%Y')`
- [x] `./hs_core/tasks.py:            pub_date = pub_date.start_date.strftime('%m/%d/%Y')`
- [x] `./hs_file_types/models/reftimeseries.py:            st_date = st_date.strftime('%m-%d-%Y')`
- [x] `./hs_file_types/models/reftimeseries.py:            end_date = end_date.strftime('%m-%d-%Y')`
- [x] `./hs_file_types/nc_functions/nc_meta.py:                    period_info['start'] = limit_meta['start'].strftime('%Y-%m-%d %H:%M:%S')`
- [x] `./hs_file_types/nc_functions/nc_meta.py:                    period_info['end'] = limit_meta['end'].strftime('%Y-%m-%d %H:%M:%S')`
- [x] `./hs_file_types/utils.py:        temp_cov.value['start'] = start_date.strftime('%m/%d/%Y')`
- [x] `./hs_file_types/utils.py:        temp_cov.value['end'] = end_date.strftime('%m/%d/%Y')`
- [x] `./hs_model_program/views.py:            dt = datetime.datetime.strftime(mpmeta.modelReleaseDate,'%m/%d/%Y')`
- [x] `./hs_tracking/management/commands/stats.py:            last_login = up.user.last_login.strftime('%m/%d/%Y') if up.user.last_login else ""`
- [x] `./hs_tracking/management/commands/stats.py:                up.user.date_joined.strftime('%m/%d/%Y %H:%M:%S.%f'),`
- [x] `./hs_tracking/management/commands/stats.py:                    start_date.strftime("%m/%d/%Y %H:%M:%S.%f"),`
Recreating the bug on the Python shell

```
>>> from datetime import datetime
>>> pre_date = datetime(1890, 6, 6)
>>> pre_date.strftime("%Y-%m-%d")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: year=1890 is before 1900; the datetime strftime() methods require year >= 1900
```
Drop-in replacement:

```
>>> '{0.year:4d}-{0.month:02d}-{0.day:02d}'.format(pre_date)
'1890-06-06'
```

@aphelionz Perhaps it is better to make a simple utility function for your strftime() replacement to be called in various places. Alternatively, ```isoformat()``` can be used in place of strftime(), e.g., ```pre_date.isoformat(' ').split(' ')[0]``` 
I saw that approach but I don't think I like the idea of relying on `split` and an array key for this. I looked at the utility function approach but I think in this case it would be a one-line function that takes a datetime object and a format string and returns the formatted string I posted above... so I dont think that saves us much code.
Dropping this into a utility function doesn't save us code but it increases reuse and helps to standardize date formatting.
@aphelionz The only argument I'd make regarding making a simple utility function is that if in the future we want to change date format in HydroShare, e.g., to use ```/``` rather than ```-```, we only need to change that utility function without the need to make the changes in all those places where this formatting is used. In any case, I don't have strong opinion on this. It is your call. 
Date formatting isn't standard across the UI and backend though - check out the grep results.
We could create a utility function that returns a standard date format but also accepts another date format as an optional keyword argument?
I'd argue it is probably a good idea to standardize date format across the board in HydroShare.
@sblack-usu @hyi Here is the current breakdown of the different datetime formats we use.. It looks like we may be able to consolidate the dates into a format like YYYY-mm-dd and then the datetimes to some standard format like ISO or some such thing...

```
YYYY-mm-dd
./django_irods/views.py: daily_date = datetime.datetime.today().strftime('%Y-%m-%d')
./hs_core/page_processors.py: temporal_coverage_data_dict['start_date'] = start_date.strftime('%Y-%m-%d')
./hs_core/page_processors.py: temporal_coverage_data_dict['end_date'] = end_date.strftime('%Y-%m-%d')
./hs_core/tests/api/rest/test_folder_download_zip.py: date_folder = (date.today()).strftime('%Y-%m-%d')
./hs_core/tasks.py: date_folder = (date.today() - timedelta(2)).strftime('%Y-%m-%d')

mm-dd-YYYY
./hs_core/views/utils.py: temporal_coverage_dict['start'] = start_date.strftime('%m-%d-%Y')
./hs_core/views/utils.py: temporal_coverage_dict['end'] = end_date.strftime('%m-%d-%Y')
./hs_core/page_processors.py: temporal_coverage_data_dict['start'] = start_date.strftime('%m-%d-%Y')
./hs_core/page_processors.py: temporal_coverage_data_dict['end'] = end_date.strftime('%m-%d-%Y')
./hs_file_types/models/reftimeseries.py: st_date = st_date.strftime('%m-%d-%Y')
./hs_file_types/models/reftimeseries.py: end_date = end_date.strftime('%m-%d-%Y')

mm/dd/YYYY
./hs_core/tests/api/native/test_create_resource.py: self.assertTrue(res.created.strftime('%m/%d/%Y') == dtime.datetime.today().strftime('%m/%d/%Y'))
./hs_core/tests/api/native/test_create_resource.py: self.assertTrue(new_res.created.strftime('%m/%d/%Y') == dtime.datetime.today().strftime('%m/%d/%Y'))
./hs_core/serialization.py: val['start'] = c.start_date.strftime('%m/%d/%Y')
./hs_core/serialization.py: val['end'] = c.end_date.strftime('%m/%d/%Y')
./hs_core/models.py: td(start_date.strftime('%m/%d/%Y'))
./hs_core/models.py: td(end_date.strftime('%m/%d/%Y'))
./hs_core/models.py: coverage_data_dict['start'] = start_date.strftime('%m/%d/%Y')
./hs_core/models.py: coverage_data_dict['end'] = end_date.strftime('%m/%d/%Y')
./hs_core/tasks.py: pub_date = pub_date.start_date.strftime('%m/%d/%Y')
./hs_core/tasks.py: pub_date = pub_date.start_date.strftime('%m/%d/%Y')
./hs_file_types/utils.py: temp_cov.value['start'] = start_date.strftime('%m/%d/%Y')
./hs_file_types/utils.py: temp_cov.value['end'] = end_date.strftime('%m/%d/%Y')
./hs_model_program/views.py: dt = datetime.datetime.strftime(mpmeta.modelReleaseDate,'%m/%d/%Y')
./hs_tracking/management/commands/stats.py: last_login = up.user.last_login.strftime('%m/%d/%Y') if up.user.last_login else ""
./hs_collection_resource/views.py: value_dict = {'start': time_start.strftime(UI_DATETIME_FORMAT),
./hs_collection_resource/views.py: 'end': time_end.strftime(UI_DATETIME_FORMAT)}

YYYY-mm-ddTHH-MM-SSZ (or other)
./hs_core/discovery_parser.py: thisday = thisday_object.strftime("%Y-%m-%dT%H:%M:%SZ")
./hs_core/discovery_parser.py: nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")
./hs_core/discovery_parser.py: nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")
./hs_core/discovery_parser.py: nextday = nextday_object.strftime("%Y-%m-%dT%H:%M:%SZ")
./hs_core/irods.py: formatted = timeout.strftime("%Y-%m-%d.%H:%M")
./hs_file_types/nc_functions/nc_meta.py: period_info['start'] = limit_meta['start'].strftime('%Y-%m-%d %H:%M:%S')
./hs_file_types/nc_functions/nc_meta.py: period_info['end'] = limit_meta['end'].strftime('%Y-%m-%d %H:%M:%S')
./hs_tracking/management/commands/stats.py: up.user.date_joined.strftime('%m/%d/%Y %H:%M:%S.%f'),
./hs_tracking/management/commands/stats.py: start_date.strftime("%m/%d/%Y %H:%M:%S.%f"),
```
So, perhaps, a function like this would be useful.

```
def format_datetime(dt, template='{0.year:4d}-{0.month:02d}-{0.day:02d}'):
    return template.format(dt)
```
Finally, note that I'm ignoring JS for now, which also has a number of strftime-type calls.
That's a clever function.  It self documents how to do custom formatting while defaulting to a standard format.  👍 
@Lizabrazil reminded me that @horsburgh's powerpoint reports standard metadata formatting guidelines: "For new versions of files, use date format ISO 8601: YYYYMMDD", and the HIS system displays dates as 7/16/2017 0:00
@aphelionz Agreed with @sblack-usu. That is a clever function indeed.
For automated testing we will need:
1. To detect `strftime` in new commits and fail.
2. To test four digit years
3. To test 2 digit years > 69
4. To test 2 digit years < 70

What else?
detecting strftime in new commits seems pretty heavy handed.  I think it's enough to write a unit test with a pre-1900 date where the issue bubbled up.  

i'm not following why you have the 2 digit year tests... why 69 and 70?
How else should we prevent strftime getting back into the code, which can/will break the display of any pre-1900 dates? This throws a proper ValueError and would leave pages breaking with 500 errors if not handled correctly.

w/r/t 1969 and 1970 - the reason strftime breaks is beacuse of Y2K-ery, which is to say that that all two digit dates under 70 are treated as the 1900s, and all two digit dates after 69 are treated as the 2000s.

After doing some further testing though it looks like our new method will handle any dates down to year 0, so that's good :)

```
>>> mydate = datetime(18,12,12)
>>> mydate
datetime.datetime(18, 12, 12, 0, 0)
>>> mydate.strftime("%Y")
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ValueError: year=18 is before 1900; the datetime strftime() methods require year >= 1900
>>> mydate.isoformat()
'0018-12-12T00:00:00'
```
Fwiw, 7/16/18 is pretty clear, but mm/dd/yy can be ambiguous. Europeans and military folks often interpret numeric dates like 7/4/18 as April 7 not July 4. It's least ambiguous & most general to stick with yyyy-mm-dd or spell out the month abbreviation. 
I also like YYYY first.
@aphelionz I agree with @sblack-usu that detecting strftime in new commits seems pretty heavy handed. I think it'd work to detect this via code review and a bit of developer education. Also, 1900 year problem has been fixed in strftime in python 3.3 and above, so after HydroShare were upgraded to python 3, strftime would be good to be used. 
I will buy everybody on the team a round of drinks at whatever conference we are all next at if and when HydroShare gets updated to Python 3.
This work was not closed out during transition of one of the developers. Any future work may benefit from the first pass that was made at the time to glean code hints or copy paste, where applicable. https://github.com/hydroshare/hydroshare/pull/2839
@cuahsimarko @mraocuahsi 
This is still an issue on www
I created a new composite resource and tried to set its dates as 01/01/1890 to 01/02/1890
The metadata failed to update; The whole res landing page failed to open.
![image](https://user-images.githubusercontent.com/10070599/56509224-a62f9200-64e3-11e9-94b4-b0c7a9079ddc.png)

![image](https://user-images.githubusercontent.com/10070599/56509244-b0519080-64e3-11e9-8cbe-1e49c6aea14e.png)

@cuahsimarko @mraocuahsi 
Test REST API: hs_restclient was able to update coverages with 01/01/1890 through rest api, the error only happens on resource landing page
![image](https://user-images.githubusercontent.com/10070599/56510958-95cde600-64e8-11e9-941d-a6e3fc0f2b68.png)


Resolving patch:
diff --git a/hs_core/page_processors.py b/hs_core/page_processors.py
--- a/hs_core/page_processors.py
+++ b/hs_core/page_processors.py
@@ -123,8 +123,10 @@ def get_page_context(page, user, resource_edit=False, extended_metadata_layout=N
         if temporal_coverage:
             start_date = parser.parse(temporal_coverage.value['start'])
             end_date = parser.parse(temporal_coverage.value['end'])
-            temporal_coverage_data_dict['start_date'] = start_date.strftime('%Y-%m-%d')
-            temporal_coverage_data_dict['end_date'] = end_date.strftime('%Y-%m-%d')
+            temporal_coverage_data_dict['start_date'] = \
+                '{0.year:4d},{0.month:02d},{0.day:02d},{0.hour},{0.minute}'.format(start_date)
+            temporal_coverage_data_dict['end_date'] = \
+                '{0.year:4d},{0.month:02d},{0.day:02d},{0.hour},{0.minute}'.format(end_date)
             temporal_coverage_data_dict['name'] = temporal_coverage.value.get('name', '')
 
         spatial_coverage = content_model.metadata.spatial_coverage
@@ -235,8 +237,10 @@ def get_page_context(page, user, resource_edit=False, extended_metadata_layout=N
     if temporal_coverage:
         start_date = parser.parse(temporal_coverage.value['start'])
         end_date = parser.parse(temporal_coverage.value['end'])
-        temporal_coverage_data_dict['start'] = start_date.strftime('%m-%d-%Y')
-        temporal_coverage_data_dict['end'] = end_date.strftime('%m-%d-%Y')
+        temporal_coverage_data_dict['start'] = \
+            '{0.year:4d},{0.month:02d},{0.day:02d},{0.hour},{0.minute}'.format(start_date)
+        temporal_coverage_data_dict['end'] = \
+            '{0.year:4d},{0.month:02d},{0.day:02d},{0.hour},{0.minute}'.format(end_date)
         temporal_coverage_data_dict['name'] = temporal_coverage.value.get('name', '')
         temporal_coverage_data_dict['id'] = temporal_coverage.id
 


Test patch: