Repository: obspy/obspy

Issue:
UTCDateTime parsing of dashed separated format
In the past I have been able to use `UTCDateTime` to parse strings whose hour, minute, and second are separated by `'-'` as well as `:`. I am not sure if this is officially supported but I haven't run into any issues until today. However, I found that if only an hour and minute is provided it does not work. For example,

```python
from obspy import UTCDateTime
print(UTCDateTime('2019-01-01T10-33-00'))  #  2019-01-01T10:33:00.000000Z
print(UTCDateTime('2019-01-01T10-33'))  #  2019-01-02T19:00:00.000000Z
print(UTCDateTime('2019-01-01T10-33-00'))  #  2019-01-01T10:33:00.000000Z
```

I am using Ubuntu 18, python 3.7 and obspy master. I think we should consider this a bug and either 1) fix it, or 2) raise an error if users (like me) are abusing the `UTCDateTime` string parsing. 

Edit: fixed third code example




Discussion:
the second print line is not a bug as ```2019-01-01T10-33``` is a valid ISO 8601 formated date/time string and gets interpreted as such - its basically the same as ```2019-01-01T10:00:00-33:00```, which results into ```2019-01-02T19:00:00.000000Z```

ofc a time zone of ```-33:00``` does not exist and hours in the time string are limited to 00 to 23, however I couldn't find a limitation of the hour given within the timezone indicator

anyway I suggest to either remove the ```T``` within the string you are using or use a proper ISO 8601 format, like having a ```:``` as divider in the time string, e.g. ```2019-01-01T10:33```

I don't get the third print line in your example ....
Thanks @barsch,

I had forgotten that ISO 8601 specifies time zones in this way, and it didn't even cross my mind that it might be the case as a timezone of -33 doesn't make a lot of sense.  I suspect specifying a timezone in a `UTCDateTime` constructor is uncommon, but thanks for clearing this up.

I guess the confusion comes from `UTCDateTime`'s extremely flexible (and sometimes surprising) parsing of time strings which can change significantly based on whether the string complies with ISO 8601. A few examples:

```python 
import obspy
# These three strings are all very similar, and at first glance they appear to indicate the
# same time. However, the first string has a time zone indication, the second one 
# indicates minutes, the third a time zone of -2:33
print(obspy.UTCDateTime('2019-01-01T02:02-33'))  #  2019-01-02T11:02:00.000000Z
print(obspy.UTCDateTime('2019-01-01T02-02-33'))  # 2019-01-01T02:02:33.000000Z
print(obspy.UTCDateTime('2019-01-01T02-02:33'))  # 2019-01-01T04:33:00.000000Z

# The meaning of the last char group is set to seconds if a decimal is found (skipping minutes)
print(obspy.UTCDateTime('2019-01-01T02-02.33'))   # 2019-01-01T02:00:02.330000Z

#  This one is particularly confusing as the T character enables the interpretation of time zones 
print(obspy.UTCDateTime('2019-01-01T02-02'))  #  2019-01-01T04:00:00.000000Z
print(obspy.UTCDateTime('2019-01-01 02-02'))  #  2019-01-01T02:02:00.000000Z
```
The `iso8601` keyword is handy for bringing a bit more discipline to the parsing. One just has to be aware, as I am now thanks to your reply, of the possibility of part of the string indicating a time zone rather than a minute value.

In hindsight, `UTCDateTime`s constructor might be a bit too flexible. However, the cost of changing anything now certainly outweighs the benefits. Do you think it would be worth adding a note about these potential gotchas in the docstring of `UTCDateTime`? 


it is less confusing if you look at the following lines: https://github.com/obspy/obspy/blob/master/obspy/core/utcdatetime.py#L333-L339

So as soon as a single ```T``` is in the datetime string, ObsPy assumes its a ISO8601 formated string - and those strings have a fixed format, e.g. there are only ```:``` and ```.``` allowed in the time part. However, I agree that more checks if its a valid ISO8601 datetime string would be useful, and being more verbose if something is wrong is always good - but on the other side such checks increase the memory footprint and initialization time of UTCDateTime objects - which is already very heavy ... 

So I again suggest just to remove the ```T``` (which is the ISO8601 indicator) from the datetime string and everything works as it should


btw. I don't have a working ObsPy instance here to test, but I assume that setting 
```iso8601=False``` and having a ```T``` in the string is not working correctly - if so, this need to be fixed

I would expect:
```
print(obspy.UTCDateTime('2019-01-01T02-02:33', iso8601=True))  # 2019-01-01T04:33:00.000000Z
print(obspy.UTCDateTime('2019-01-01T02-02:33', iso8601=False)) # 2019-01-01T02:02:33.000000Z
```




On current master:
```
In [1]: print(UTCDateTime('2019-01-01T02-02:33', iso8601=True))  # 2019-01-01T04:33:00.000000Z
   ...: print(UTCDateTime('2019-01-01T02-02:33', iso8601=False)) # 2019-01-01T02:02:33.000000Z
   ...: 
2019-01-01T04:33:00.000000Z
2019-01-01T04:33:00.000000Z
```
thanks @megies 

so there is at least this bug (inclusive test above) which needs fixing - otherwise I don't feel like we should further improve ISO8601 handling, or?
> Do you think it would be worth adding a note about these potential gotchas in the docstring of UTCDateTime?

sure - we should at least mention that a ```T``` as divider between a date and time string will automatically assume its ISO8601

Resolving patch:
diff --git a/obspy/core/utcdatetime.py b/obspy/core/utcdatetime.py
--- a/obspy/core/utcdatetime.py
+++ b/obspy/core/utcdatetime.py
@@ -30,6 +30,25 @@
 # is called a lot. Thus pre-compile it.
 _YEAR0REGEX = re.compile(r"^(\d{1,3}[-/,])(.*)$")
 
+# based on https://www.myintervals.com/blog/2009/05/20/iso-8601, w/ week 53 fix
+_ISO8601_REGEX = re.compile(r"""
+    ^
+    ([\+-]?\d{4}(?!\d{2}\b))
+    ((-?)
+     ((0[1-9]|1[0-2])
+      (\3([12]\d|0[1-9]|3[01]))?
+      |W([0-4]\d|5[0-3])(-?[1-7])?
+      |(00[1-9]|0[1-9]\d|[12]\d{2}|3([0-5]\d|6[1-6]))
+     )
+     ([T\s]
+      ((([01]\d|2[0-3])((:?)[0-5]\d)?|24\:?00)([\.,]\d+(?!:))?)?
+      (\17[0-5]\d([\.,]\d+)?)?
+      ([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?
+     )?
+    )?
+    $
+    """, re.VERBOSE)
+
 TIMESTAMP0 = datetime.datetime(1970, 1, 1, 0, 0)
 # XXX the strftime problem seems to be specific to Python < 3.2
 # XXX so this can be removed after dropping Python 2 support
@@ -66,9 +85,9 @@ class UTCDateTime(object):
     :param args: The creation of a new `UTCDateTime` object depends from the
         given input parameters. All possible options are summarized in the
         `Examples`_ section below.
-    :type iso8601: bool, optional
-    :param iso8601: Enforce `ISO8601:2004`_ detection. Works only with a string
-        as first input argument.
+    :type iso8601: bool or None, optional
+    :param iso8601: Enforce/disable `ISO8601:2004`_ mode. Defaults to ``None``
+        for auto detection. Works only with a string as first input argument.
     :type strict: bool, optional
     :param strict: If True, Conform to `ISO8601:2004`_ limits on positional
         and keyword arguments. If False, allow hour, minute, second, and
@@ -268,7 +287,7 @@ def __init__(self, *args, **kwargs):
             self._ns = ns
             return
         # iso8601 flag
-        iso8601 = kwargs.pop('iso8601', False) is True
+        iso8601 = kwargs.pop('iso8601', None)
         # check parameter
         if len(args) == 0 and len(kwargs) == 0:
             # use current date/time if no argument is given
@@ -331,13 +350,17 @@ def __init__(self, *args, **kwargs):
                         "'%s' does not start with a 4 digit year" % value)
 
                 # check for ISO8601 date string
-                if value.count("T") == 1 or iso8601:
+                if iso8601 is True or (iso8601 is None and
+                                       re.match(_ISO8601_REGEX, value)):
                     try:
                         self._from_iso8601_string(value)
                         return
                     except Exception:
+                        # raise here if iso8601 is enforced otherwise fallback
+                        # to non iso8601 detection by continuing below
                         if iso8601:
                             raise
+
                 # try to apply some standard patterns
                 value = value.replace('T', ' ')
                 value = value.replace('_', ' ')


Test patch:
diff --git a/obspy/core/tests/test_utcdatetime.py b/obspy/core/tests/test_utcdatetime.py
--- a/obspy/core/tests/test_utcdatetime.py
+++ b/obspy/core/tests/test_utcdatetime.py
@@ -1552,6 +1552,24 @@ def test_leap_years(self):
         expected = UTCDateTime(2016, 12, 31)
         self.assertEqual(got, expected)
 
+    def test_issue_2447(self):
+        """
+        Setting iso8601=False should disable ISO8601 parsing.
+
+        See issue #2447.
+        """
+        # auto detection
+        self.assertEqual(UTCDateTime('2019-01-01T02-02:33'),
+                         UTCDateTime(2019, 1, 1, 4, 33, 0))
+        self.assertEqual(UTCDateTime('2019-01-01 02-02:33'),
+                         UTCDateTime(2019, 1, 1, 2, 2, 33))
+        # enforce ISO8601 mode
+        self.assertEqual(UTCDateTime('2019-01-01T02-02:33', iso8601=True),
+                         UTCDateTime(2019, 1, 1, 4, 33, 0))
+        # skip ISO8601 mode
+        self.assertEqual(UTCDateTime('2019-01-01T02-02:33', iso8601=False),
+                         UTCDateTime(2019, 1, 1, 2, 2, 33))
+
 
 def suite():
     return unittest.makeSuite(UTCDateTimeTestCase, 'test')