Repository: urllib3/urllib3

Issue:
parse_retry_after() in utils/retry.py computes seconds incorrectly when Retry-After value is a http-date and local time is not in the GMT timezone
For the given Retry-After value `Thu, 25 Jun 2020 21:30:12 GMT`, with the local time being `Thu, 25 Jun 2020 17:28:12 EST`, parse_retry_after() will return `14564.671981096268`.  This value is incorrect, and should be `~120`.

parse_retry_after() doesn't correctly handle the time difference between time zones.


Discussion:
Here's a MRE that pulls out the code of parse_retry_after().

```
import re
import time
import email.utils
import datetime



def parse_retry_after(retry_after):
    """
    Pulled from retry.py and minimally modified to change exception raised.
    """
    # Whitespace: https://tools.ietf.org/html/rfc7230#section-3.2.4
    if re.match(r"^\s*[0-9]+\s*$", retry_after):
        seconds = int(retry_after)
    else:
        retry_date_tuple = email.utils.parsedate(retry_after)
        if retry_date_tuple is None:
            raise RuntimeError('Invalid header!')
        retry_date = time.mktime(retry_date_tuple)
        seconds = retry_date - time.time()

    if seconds < 0:
        seconds = 0

    return seconds


if __name__ == '__main__':
    # Create a Retry-After value (http-date) that's 180 seconds after now.  Note this will only
    # demonstrate the issue when the local timezone is *not* GMT.
    nowutc = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(seconds=180)
    retryafter = nowutc.strftime('%a, %d %b %Y %H:%M:%S GMT ')

    print(f'got {parse_retry_after(retryafter)} for {retryafter}, '
          f'but should be {nowutc.timestamp() - time.time()}')
```
Thanks for opening this, will look at it later. Going to summon the timezone wizard @pganssle for assistance making sense of this. 
This seems to fix the issue, but I've not done near enough testing or tried any python version other than 3.6.8:

```
   def parse_retry_after(self, retry_after):
        """Fix TZ conversion"""
        # Whitespace: https://tools.ietf.org/html/rfc7230#section-3.2.4
        if re.match(r"^\s*[0-9]+\s*$", retry_after):
            seconds = int(retry_after)
        else:
            retry_date_tuple = email.utils.parsedate_tz(retry_after)
            if retry_date_tuple is None:
                raise InvalidHeader("Invalid Retry-After header: %s" % retry_after)
            retry_date = email.utils.mktime_tz(retry_date_tuple)
            seconds = retry_date - time.time()
        if seconds < 0:
            seconds = 0
        return seconds
```
@cblades-tc Would you be willing to submit a patch with a previously failing test case?
I think the `parsedate_tz` and `mktime_tz` is probably the right thing to do here. It looks like the current logic ignores time zone offsets, and assumes that *all* RFC 2822 datetimes are in system local time (which seems like a not great default for a wire format, TBH).

That said, I do not know how people currently use this, but it *may* be a breaking change to the semantics of unspecified time zones (which could legitimately be said to reflect local time):

```python
>>> from time import mktime
>>> from email.utils import parsedate, parsedate_tz, mktime_tz
>>> from datetime import datetime

>>> print(dt := datetime.fromtimestamp(1593160100))
2020-06-26 04:28:20
>>> (dt_str := dt.strftime("%a, %d %b %Y %H:%M:%S"))
'Fri, 26 Jun 2020 04:28:20'

>>> (ts := mktime(parsedate(dt_str)))
1593160100.0
>>> (ts_tz := mktime_tz(parsedate_tz(dt_str)))
1593145700

>>> print(datetime.fromtimestamp(ts))
2020-06-26 04:28:20
>>> print(datetime.fromtimestamp(ts_tz))
2020-06-26 00:28:20
```

(Note: the above was executed on a computer set to EDT. You won't see the problem on a computer with a system time set to any zone with an offset of 0).

From what I can tell, the output of `parsedate_tz` is no different between a string that is explicitly marked as GMT / UTC and one that has no time zone specified. If you care about maintaining the old behavior for timezone-unspecified strings, you may have to actively detect non-UTC time zones being parsed with time zone of 0.

There are a good number of ways to specify UTC in [RFC 2822](https://tools.ietf.org/html/rfc2822#section-3.3)  — the main ways are, I think `0000` and `+0000` (`-0000` is actually supposed to refer to local time¹), and then there's also the [obsolete methods](https://tools.ietf.org/html/rfc2822#section-4.3): `GMT` `UT` and "Z" and "z". You'll probably want to carefully look at the test cases for `email.utils.parsedate_tz` to see if I've missed anything, but if not, then I think this might work (assuming it doesn't allow for random whitespace at the end or something):

```python
def parse_rfc2822_datetime_to_timestamp(dt_str):
    time_t = email.utils.parsedate_tz(dt_str)
    if time_t is None:
        raise WhateverException()
    if time_t[-1] == 0 and dt_str.endswith("-0000") or not dt_str.endswith(
        ("+0000", "0000", "GMT", "UT", "Z", "z")):
            naive_time_t = email.utils.parsedate(dt_str)
            return time.mktime(naive_time_t)
    return email.utils.mktime_tz(time_t)
```

I don't know how much efficiency matters to you here, there are almost certainly many optimizations you can do to make the "check if it's UTC" part more efficient.

¹RFC 2822 is sorta vague about how `-0000` should be treated, but it seems like they are saying that it should be treated identically to "no offset specified":

> Though "-0000" also indicates Universal Time, it is used to indicate that the time was generated on a system that may be in a local time zone other than Universal Time and therefore indicates that the date-time contains no information about the local time zone.

Thanks for all this information @pganssle, I'm thinking this is the direction we should head is exactly in line with the example you provided. Maybe we can use a regex for the "check if its UTC".
If it simplifies things, I believe https://tools.ietf.org/html/rfc7231#section-7.1.1.1 suggests that http-dates should _always_ be in UTC.

`An HTTP-date value represents time as an instance of Coordinated
   Universal Time (UTC).`
That's good to know. Since most dates will be GMT we can optimize for that in our UTC TZ detection.
> That's good to know. Since most dates will be GMT we can optimize for that in our UTC TZ detection.

I think the more important sentence is this one:

> values in the asctime format are assumed to be in UTC

That's what `parsedate_tz` already does, so it seems you may be able to use it without changing.

The real question is whether you want to follow the spec or try and keep the old behavior. I think everyone will agree that it's a bug if `'Fri, 26 Jun 2020 04:28:20 GMT'` is parsed as representing local time, but local time is at least a reasonable choice for `'Fri, 26 Jun 2020 04:28:20'`, so you may want to maintain backwards compatibility there. Still — given that RFC 7231¹ explicitly says that the second one should be interpreted as UTC, there's a very reasonable case to be made that "no offset supplied" is just another manifestation of the same bug.

¹Assuming that RFC 7231 is the one that applies here.
Oh, another thing to mention is that if RFC 7231 actually *is* the spec that applies here, it allows only a subset of datetime formats currently parsed by this function:

```
   An example of the preferred format is

     Sun, 06 Nov 1994 08:49:37 GMT    ; IMF-fixdate

   Examples of the two obsolete formats are

     Sunday, 06-Nov-94 08:49:37 GMT   ; obsolete RFC 850 format
     Sun Nov  6 08:49:37 1994         ; ANSI C's asctime() format
```

I don't know how strict you want to be with `urllib3` but if you're trying to implement RFC 7231, you probably want to throw an exception if something other than one of those formats is found. Might be too late to do that without breaking the world, though. 😅
@pganssle Since this issue is being reported only now after existing for over 4 years means I can safely assume there are very few users relying on the "old" (buggy) behavior. Let's assume UTC the same way that `parsedate_tz` does and not worry about the old behavior.

And yeah RFC 7231 is the spec to apply, so maybe we can be stricter about `Retry-After` dates that we throw into `parsedate_tz()`. I wonder what formats it accepts outside of those three, maybe some research to do.

Resolving patch:
diff --git a/src/urllib3/util/retry.py b/src/urllib3/util/retry.py
--- a/src/urllib3/util/retry.py
+++ b/src/urllib3/util/retry.py
@@ -266,10 +266,10 @@ def parse_retry_after(self, retry_after):
         if re.match(r"^\s*[0-9]+\s*$", retry_after):
             seconds = int(retry_after)
         else:
-            retry_date_tuple = email.utils.parsedate(retry_after)
+            retry_date_tuple = email.utils.parsedate_tz(retry_after)
             if retry_date_tuple is None:
                 raise InvalidHeader("Invalid Retry-After header: %s" % retry_after)
-            retry_date = time.mktime(retry_date_tuple)
+            retry_date = email.utils.mktime_tz(retry_date_tuple)
             seconds = retry_date - time.time()
 
         if seconds < 0:


Test patch:
diff --git a/test/test_retry.py b/test/test_retry.py
--- a/test/test_retry.py
+++ b/test/test_retry.py
@@ -1,7 +1,5 @@
-import datetime
 import mock
 import pytest
-import time
 
 from urllib3.response import HTTPResponse
 from urllib3.packages import six
@@ -313,6 +311,7 @@ def test_respect_retry_after_header_propagated(self, respect_retry_after_header)
         new_retry = retry.new()
         assert new_retry.respect_retry_after_header == respect_retry_after_header
 
+    @pytest.mark.freeze_time("2019-06-03 11:00:00", tz_offset=0)
     @pytest.mark.parametrize(
         "retry_after_header,respect_retry_after_header,sleep_duration",
         [
@@ -326,6 +325,11 @@ def test_respect_retry_after_header_propagated(self, respect_retry_after_header)
             ("Mon, 3 Jun 2019 11:00:00 UTC", True, None),
             # Won't sleep due to current time reached + not respecting header
             ("Mon, 3 Jun 2019 11:00:00 UTC", False, None),
+            # Handle all the formats in RFC 7231 Section 7.1.1.1
+            ("Mon, 03 Jun 2019 11:30:12 GMT", True, 1812),
+            ("Monday, 03-Jun-19 11:30:12 GMT", True, 1812),
+            # Assume that datetimes without a timezone are in UTC per RFC 7231
+            ("Mon Jun  3 11:30:12 2019", True, 1812),
         ],
     )
     def test_respect_retry_after_header_sleep(
@@ -333,17 +337,7 @@ def test_respect_retry_after_header_sleep(
     ):
         retry = Retry(respect_retry_after_header=respect_retry_after_header)
 
-        # Date header syntax can specify an absolute date; compare this to the
-        # time in the parametrized inputs above.
-        current_time = mock.MagicMock(
-            return_value=time.mktime(
-                datetime.datetime(year=2019, month=6, day=3, hour=11).timetuple()
-            )
-        )
-
-        with mock.patch("time.sleep") as sleep_mock, mock.patch(
-            "time.time", current_time
-        ):
+        with mock.patch("time.sleep") as sleep_mock:
             # for the default behavior, it must be in RETRY_AFTER_STATUS_CODES
             response = HTTPResponse(
                 status=503, headers={"Retry-After": retry_after_header}