Repository: boto/botocore
Pull request #2664 (2022-04-21 15:40:07)

Add support for IMDS credential extension
Adding support for server side credential validation during IMDS outage scenarios. This patch will defer validation of credential expiry to the service for cases where IMDS may not be capable of receiving new metadata updates.

codecov-commenter: # [Codecov](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto) Report
> Merging [#2664](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto) (aa11b4e) into [develop](https://codecov.io/gh/boto/botocore/commit/b7a6f50cca3dcfdab7022a8e1ebe9e31edf57581?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto) (b7a6f50) will **increase** coverage by `0.00%`.
> The diff coverage is `96.15%`.

```diff
@@           Coverage Diff            @@
##           develop    #2664   +/-   ##
========================================
  Coverage    95.00%   95.00%           
========================================
  Files           60       60           
  Lines        12237    12258   +21     
========================================
+ Hits         11626    11646   +20     
- Misses         611      612    +1     
```


| [Impacted Files](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto) | Coverage Δ | |
|---|---|---|
| [botocore/utils.py](https://codecov.io/gh/boto/botocore/pull/2664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto#diff-Ym90b2NvcmUvdXRpbHMucHk=) | `93.66% <95.45%> (+0.01%)` | :arrow_up: |
| [botocore/credentials.py](https://codecov.io/gh/boto/botocore/pull/2664/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto#diff-Ym90b2NvcmUvY3JlZGVudGlhbHMucHk=) | `98.65% <100.00%> (+<0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto). Last update [b7a6f50...aa11b4e](https://codecov.io/gh/boto/botocore/pull/2664?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=boto).


Patch:
diff --git a/.changes/next-release/feature-IMDS-63815.json b/.changes/next-release/feature-IMDS-63815.json
new file mode 100644
index 000000000..fc7a27888
--- /dev/null
+++ b/.changes/next-release/feature-IMDS-63815.json
@@ -0,0 +1,5 @@
+{
+  "type": "feature",
+  "category": "IMDS",
+  "description": "Added resiliency mechanisms to IMDS Credential Fetcher"
+}
diff --git a/botocore/credentials.py b/botocore/credentials.py
index ca504b40f..9227fe16b 100644
--- a/botocore/credentials.py
+++ b/botocore/credentials.py
@@ -56,6 +56,9 @@ logger = logging.getLogger(__name__)
 ReadOnlyCredentials = namedtuple('ReadOnlyCredentials',
                                  ['access_key', 'secret_key', 'token'])
 
+_DEFAULT_MANDATORY_REFRESH_TIMEOUT = 10 * 60  # 10 min
+_DEFAULT_ADVISORY_REFRESH_TIMEOUT = 15 * 60  # 15 min
+
 
 def create_credential_resolver(session, cache=None, region_name=None):
     """Create a default credential resolver.
@@ -74,7 +77,8 @@ def create_credential_resolver(session, cache=None, region_name=None):
         'ec2_metadata_service_endpoint': session.get_config_variable(
             'ec2_metadata_service_endpoint'),
         'ec2_metadata_service_endpoint_mode': resolve_imds_endpoint_mode(
-            session)
+            session),
+        'ec2_credential_refresh_window': _DEFAULT_ADVISORY_REFRESH_TIMEOUT,
     }
 
     if cache is None:
@@ -342,10 +346,10 @@ class Credentials(object):
     """
     Holds the credentials needed to authenticate requests.
 
-    :ivar access_key: The access key part of the credentials.
-    :ivar secret_key: The secret key part of the credentials.
-    :ivar token: The security token, valid only for session credentials.
-    :ivar method: A string which identifies where the credentials
+    :param str access_key: The access key part of the credentials.
+    :param str secret_key: The secret key part of the credentials.
+    :param str token: The security token, valid only for session credentials.
+    :param str method: A string which identifies where the credentials
         were found.
     """
 
@@ -382,18 +386,20 @@ class RefreshableCredentials(Credentials):
     Holds the credentials needed to authenticate requests. In addition, it
     knows how to refresh itself.
 
-    :ivar access_key: The access key part of the credentials.
-    :ivar secret_key: The secret key part of the credentials.
-    :ivar token: The security token, valid only for session credentials.
-    :ivar method: A string which identifies where the credentials
+    :param str access_key: The access key part of the credentials.
+    :param str secret_key: The secret key part of the credentials.
+    :param str token: The security token, valid only for session credentials.
+    :param function refresh_using: Callback function to refresh the credentials.
+    :param str method: A string which identifies where the credentials
         were found.
+    :param function time_fetcher: Callback function to retrieve current time.
     """
     # The time at which we'll attempt to refresh, but not
     # block if someone else is refreshing.
-    _advisory_refresh_timeout = 15 * 60
+    _advisory_refresh_timeout = _DEFAULT_ADVISORY_REFRESH_TIMEOUT
     # The time at which all threads will block waiting for
     # refreshed credentials.
-    _mandatory_refresh_timeout = 10 * 60
+    _mandatory_refresh_timeout = _DEFAULT_MANDATORY_REFRESH_TIMEOUT
 
     def __init__(self, access_key, secret_key, token,
                  expiry_time, refresh_using, method,
diff --git a/botocore/utils.py b/botocore/utils.py
index 652bada80..244658b80 100644
--- a/botocore/utils.py
+++ b/botocore/utils.py
@@ -378,7 +378,10 @@ class IMDSFetcher(object):
                  env=None, user_agent=None, config=None):
         self._timeout = timeout
         self._num_attempts = num_attempts
+        if config is None:
+            config = {}
         self._base_url = self._select_base_url(base_url, config)
+        self._config = config
 
         if env is None:
             env = os.environ.copy()
@@ -394,9 +397,6 @@ class IMDSFetcher(object):
         return self._base_url
 
     def _select_base_url(self, base_url, config):
-        if config is None:
-            config = {}
-
         requires_ipv6 = config.get(
             'ec2_metadata_service_endpoint_mode') == 'ipv6'
         custom_metadata_endpoint = config.get('ec2_metadata_service_endpoint')
@@ -550,13 +550,15 @@ class InstanceMetadataFetcher(IMDSFetcher):
             role_name = self._get_iam_role(token)
             credentials = self._get_credentials(role_name, token)
             if self._contains_all_credential_fields(credentials):
-                return {
+                credentials = {
                     'role_name': role_name,
                     'access_key': credentials['AccessKeyId'],
                     'secret_key': credentials['SecretAccessKey'],
                     'token': credentials['Token'],
                     'expiry_time': credentials['Expiration'],
                 }
+                self._evaluate_expiration(credentials)
+                return credentials
             else:
                 # IMDS can return a 200 response that has a JSON formatted
                 # error message (i.e. if ec2 is not trusted entity for the
@@ -623,6 +625,35 @@ class InstanceMetadataFetcher(IMDSFetcher):
                 return False
         return True
 
+    def _evaluate_expiration(self, credentials):
+        expiration = credentials.get("expiry_time")
+        if expiration is None:
+            return
+        try:
+            expiration = datetime.datetime.strptime(
+                expiration, "%Y-%m-%dT%H:%M:%SZ"
+            )
+            refresh_interval = self._config.get(
+                "ec2_credential_refresh_window", 60 * 10
+            )
+            refresh_interval_with_jitter = refresh_interval + random.randint(120, 600)
+            current_time = datetime.datetime.utcnow()
+            refresh_offset = datetime.timedelta(seconds=refresh_interval_with_jitter)
+            extension_time = expiration - refresh_offset
+            if current_time >= extension_time:
+                new_time = current_time + refresh_offset
+                credentials["expiry_time"] = new_time.strftime("%Y-%m-%dT%H:%M:%SZ")
+                logger.info(
+                    f"Attempting credential expiration extension due to a "
+                    f"credential service availability issue. A refresh of "
+                    f"these credentials will be attempted again within "
+                    f"the next {refresh_interval_with_jitter/60:.0f} minutes."
+                )
+        except ValueError:
+            logger.debug(
+                f"Unable to parse expiry_time in {credentials['expiry_time']}"
+            )
+
 
 class IMDSRegionProvider(object):
     def __init__(self, session, environ=None, fetcher=None):


Test patch:
diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py
index 0f4509cd9..737c0b18b 100644
--- a/tests/unit/test_utils.py
+++ b/tests/unit/test_utils.py
@@ -13,6 +13,7 @@
 import copy
 import datetime
 import io
+import operator
 
 import pytest
 from dateutil.tz import tzoffset, tzutc
@@ -88,7 +89,10 @@ from botocore.utils import (
     switch_to_virtual_host_style,
     validate_jmespath_for_set,
 )
-from tests import RawResponse, create_session, mock, unittest
+from tests import FreezeTime, RawResponse, create_session, mock, unittest
+
+DATE = datetime.datetime(2021, 12, 10, 00, 00, 00)
+DT_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
 
 
 class TestEnsureBoolean(unittest.TestCase):
@@ -2704,6 +2708,144 @@ class TestInstanceMetadataFetcher(unittest.TestCase):
             user_agent=user_agent).retrieve_iam_role_credentials()
         self.assertEqual(result, {})
 
+    def _get_datetime(
+        self, dt=None, offset=None, offset_func=operator.add
+    ):
+        if dt is None:
+            dt = datetime.datetime.utcnow()
+        if offset is not None:
+            dt = offset_func(dt, offset)
+
+        return dt
+
+    def _get_default_creds(self, overrides=None):
+        if overrides is None:
+            overrides = {}
+
+        creds = {
+            'AccessKeyId': 'access',
+            'SecretAccessKey': 'secret',
+            'Token': 'token',
+            'Expiration': '1970-01-01T00:00:00'
+        }
+        creds.update(overrides)
+        return creds
+
+    def _convert_creds_to_imds_fetcher(self, creds):
+        return {
+            'access_key': creds['AccessKeyId'],
+            'secret_key': creds['SecretAccessKey'],
+            'token': creds['Token'],
+            'expiry_time': creds['Expiration'],
+            'role_name': self._role_name
+        }
+
+    def _add_default_imds_response(self, status_code=200, creds=''):
+        self.add_get_token_imds_response(token='token')
+        self.add_get_role_name_imds_response()
+        self.add_imds_response(
+            status_code=200,
+            body=json.dumps(creds).encode('utf-8')
+        )
+
+    def mock_randint(self, int_val=600):
+        randint_mock = mock.Mock()
+        randint_mock.return_value = int_val
+        return randint_mock
+
+    @FreezeTime(module=botocore.utils.datetime, date=DATE)
+    def test_expiry_time_extension(self):
+        current_time = self._get_datetime()
+        expiration_time = self._get_datetime(
+            dt=current_time, offset=datetime.timedelta(seconds=14*60)
+        )
+        new_expiration = self._get_datetime(
+            dt=current_time, offset=datetime.timedelta(seconds=20*60)
+        )
+
+        creds = self._get_default_creds(
+            {"Expiration": expiration_time.strftime(DT_FORMAT)}
+        )
+        expected_data = self._convert_creds_to_imds_fetcher(creds)
+        expected_data["expiry_time"] = new_expiration.strftime(DT_FORMAT)
+
+        self._add_default_imds_response(200, creds)
+
+        with mock.patch("random.randint", self.mock_randint()):
+            fetcher = InstanceMetadataFetcher()
+            result = fetcher.retrieve_iam_role_credentials()
+            assert result == expected_data
+
+    @FreezeTime(module=botocore.utils.datetime, date=DATE)
+    def test_expired_expiry_extension(self):
+        current_time = self._get_datetime()
+        expiration_time = self._get_datetime(
+            dt=current_time,
+            offset=datetime.timedelta(seconds=14*60),
+            offset_func=operator.sub
+        )
+        new_expiration = self._get_datetime(
+            dt=current_time, offset=datetime.timedelta(seconds=20*60)
+        )
+        assert current_time > expiration_time
+        assert new_expiration > current_time
+
+        creds = self._get_default_creds(
+            {"Expiration": expiration_time.strftime(DT_FORMAT)}
+        )
+        expected_data = self._convert_creds_to_imds_fetcher(creds)
+        expected_data["expiry_time"] = new_expiration.strftime(DT_FORMAT)
+
+        self._add_default_imds_response(200, creds)
+
+        with mock.patch("random.randint", self.mock_randint()):
+            fetcher = InstanceMetadataFetcher()
+            result = fetcher.retrieve_iam_role_credentials()
+            assert result == expected_data
+
+    @FreezeTime(module=botocore.utils.datetime, date=DATE)
+    def test_expiry_extension_with_config(self):
+        current_time = self._get_datetime()
+        expiration_time = self._get_datetime(
+            dt=current_time,
+            offset=datetime.timedelta(seconds=14*60),
+            offset_func=operator.sub
+        )
+        new_expiration = self._get_datetime(
+            dt=current_time, offset=datetime.timedelta(seconds=25*60)
+        )
+        assert current_time > expiration_time
+        assert new_expiration > current_time
+
+        creds = self._get_default_creds(
+            {"Expiration": expiration_time.strftime(DT_FORMAT)}
+        )
+        expected_data = self._convert_creds_to_imds_fetcher(creds)
+        expected_data["expiry_time"] = new_expiration.strftime(DT_FORMAT)
+
+        self._add_default_imds_response(200, creds)
+
+        with mock.patch("random.randint", self.mock_randint()):
+            fetcher = InstanceMetadataFetcher(
+                config={"ec2_credential_refresh_window": 15*60}
+            )
+            result = fetcher.retrieve_iam_role_credentials()
+            assert result == expected_data
+
+    @FreezeTime(module=botocore.utils.datetime, date=DATE)
+    def test_expiry_extension_with_bad_datetime(self):
+        bad_datetime = "May 20th, 2020 19:00:00"
+        creds = self._get_default_creds(
+            {"Expiration": bad_datetime}
+        )
+        self._add_default_imds_response(200, creds)
+
+        fetcher = InstanceMetadataFetcher(
+            config={"ec2_credential_refresh_window": 15*60}
+        )
+        results = fetcher.retrieve_iam_role_credentials()
+        assert results['expiry_time'] == bad_datetime
+
 
 class TestIMDSRegionProvider(unittest.TestCase):
     def setUp(self):