Repository: activeloopai/deeplake

Issue:
[BUG] S3GetError:  An error occurred (InvalidAccessKeyId)
## 🐛🐛 Bug Report


### ⚗️ Current Behavior
Trying to train a simple model on fashionpedia dataset. After 7 epochs during training (approximately 9 hours later) the below error was thrown.

**Input Code**

```python
# data
fashionpedia_train = deeplake.load("hub://activeloop/fashionpedia-train")

train_set_size = int(len(fashionpedia_train) * 0.8)
train_set = fashionpedia_train[:train_set_size]
valid_set = fashionpedia_train[train_set_size:]

train_loader = train_set.pytorch(
    tensors=[
        "images",
        "masks",
        "categories",
        "boxes",
    ],  # Specify the tensors that are needed, so we don't load unused data
    transform=transform_train,
    batch_size=16,
    collate_fn=collate_fn,
)

val_loader = valid_set.pytorch(
    tensors=[
        "images",
        "masks",
        "categories",
        "boxes",
    ],  # Specify the tensors that are needed, so we don't load unused data
    transform=transform_eval,
    batch_size=16,
    collate_fn=collate_fn,
)

# model
simple_model = SimpleModel()

# train
trainer = pl.Trainer.from_argparse_args(args)
trainer.fit(model=simple_model, train_dataloaders=train_loader, val_dataloaders=val_loader)

```

### ⚙️ Environment

- `Python`: 3.9
- `OS`: Ubuntu 18.04
- `Packages`: [ `albumentations==1.2.1`, `pytorch-lightning==1.7.5`, `deeplake==3.0.10`, `libdeeplake==0.0.20`]


### 🖼 Additional context/Screenshots (optional)


```python
  File "/repo/train_coco_tutorial.py", line 301, in <module>
    trainer.fit(
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
    results = self._run_stage()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
    return self._run_train()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train
    self.fit_loop.run()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 271, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 174, in advance
    batch = next(data_fetcher)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
    return self.fetching_function()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 263, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 277, in _fetch_next_batch
    batch = next(iterator)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 557, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/trainer/supporters.py", line 569, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/pytorch_lightning/utilities/apply_func.py", line 99, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1356, in _next_data
    return self._process_data(data)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception

deeplake.util.exceptions.S3GetError: Caught S3GetError in DataLoader worker process 1.

Original Traceback (most recent call last):
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 250, in get_bytes
    return self._get_bytes(path, start_byte, end_byte)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 223, in _get_bytes
    resp = self.client.get_object(Bucket=self.bucket, Key=path, Range=range)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 934, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 257, in get_bytes
    return self._get_bytes(path, start_byte, end_byte)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 223, in _get_bytes
    resp = self.client.get_object(Bucket=self.bucket, Key=path, Range=range)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 934, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/integrations/pytorch/dataset.py", line 479, in __iter__
    for data in stream:
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/io.py", line 307, in read
    yield from self.stream(block)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/io.py", line 346, in stream
    chunk = engine.get_chunk(c_key)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/chunk_engine.py", line 517, in get_chunk
    chunk = self.cache.get_deeplake_object(
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/lru_cache.py", line 142, in get_deeplake_object
    item = self[path]
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/lru_cache.py", line 188, in __getitem__
    result = self.next_storage[path]
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 210, in __getitem__
    return self.get_bytes(path)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 257, in get_bytes
    return self._get_bytes(path, start_byte, end_byte)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 76, in __exit__
    raise self.error_class(exc_value).with_traceback(exc_traceback)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 257, in get_bytes
    return self._get_bytes(path, start_byte, end_byte)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/deeplake/core/storage/s3.py", line 223, in _get_bytes
    resp = self.client.get_object(Bucket=self.bucket, Key=path, Range=range)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/anaconda3/envs/segmentation/lib/python3.9/site-packages/botocore/client.py", line 934, in _make_api_call
    raise error_class(parsed_response, operation_name)
deeplake.util.exceptions.S3GetError: An error occurred (InvalidAccessKeyId) when calling the GetObject operation: The AWS Access Key Id you provided does not exist in our records.
```


Discussion:
Thanks for reporting the issue @rizavelioglu. Short term solution is to cache locally the dataset [docs](https://docs.deeplake.ai/en/latest/deeplake.html#deeplake.dataset) while we are looking into the issue.

```python
deeplake.load("hub://activeloop/fashionpedia-train", access_method="download") # then switch to "local"
```

Resolving patch:
diff --git a/deeplake/core/storage/s3.py b/deeplake/core/storage/s3.py
--- a/deeplake/core/storage/s3.py
+++ b/deeplake/core/storage/s3.py
@@ -42,7 +42,7 @@
     pass
 
 
-class S3ReloadCredentialsManager:
+class S3ResetReloadCredentialsManager:
     """Tries to reload the credentials if the error is due to expired token, if error still occurs, it raises it."""
 
     def __init__(self, s3p, error_class: S3Error):
@@ -50,25 +50,11 @@ def __init__(self, s3p, error_class: S3Error):
         self.s3p = s3p
 
     def __enter__(self):
-        self.s3p._locate_and_load_creds()
-        self.s3p._set_s3_client_and_resource()
-        return self
-
-    def __exit__(self, exc_type, exc_value, exc_traceback):
-        if exc_type is not None:
-            raise self.error_class(exc_value).with_traceback(exc_traceback)
-
-
-class S3ResetClientManager:
-    """Resets the client, if error still occurs, it raises it."""
-
-    def __init__(self, s3p, error_class: S3Error):
-        self.error_class = error_class
-        self.s3p = s3p
-
-    def __enter__(self):
-        self.s3p._check_update_creds(force=True)
-        self.s3p._initialize_s3_parameters()
+        if self.s3p.expiration:
+            self.s3p._check_update_creds(force=True)
+        else:
+            self.s3p._locate_and_load_creds()
+            self.s3p._set_s3_client_and_resource()
         return self
 
     def __exit__(self, exc_type, exc_value, exc_traceback):
@@ -166,9 +152,7 @@ def __setitem__(self, path, content):
         try:
             self._set(path, content)
         except botocore.exceptions.ClientError as err:
-            reload = self.need_to_reload_creds(err)
-            manager = S3ReloadCredentialsManager if reload else S3ResetClientManager
-            with manager(self, S3SetError):
+            with S3ResetReloadCredentialsManager(self, S3SetError):
                 self._set(path, content)
         except CONNECTION_ERRORS as err:
             tries = self.num_tries
@@ -251,9 +235,7 @@ def get_bytes(
         except botocore.exceptions.ClientError as err:
             if err.response["Error"]["Code"] == "NoSuchKey":
                 raise KeyError(err) from err
-            reload = self.need_to_reload_creds(err)
-            manager = S3ReloadCredentialsManager if reload else S3ResetClientManager  # type: ignore
-            with manager(self, S3GetError):  # type: ignore
+            with S3ResetReloadCredentialsManager(self, S3GetError):  # type: ignore
                 return self._get_bytes(path, start_byte, end_byte)
         except CONNECTION_ERRORS as err:
             tries = self.num_tries
@@ -293,9 +275,7 @@ def __delitem__(self, path):
         try:
             self._del(path)
         except botocore.exceptions.ClientError as err:
-            reload = self.need_to_reload_creds(err)
-            manager = S3ReloadCredentialsManager if reload else S3ResetClientManager
-            with manager(self, S3DeletionError):
+            with S3ResetReloadCredentialsManager(self, S3DeletionError):
                 self._del(path)
         except CONNECTION_ERRORS as err:
             tries = self.num_tries
@@ -379,9 +359,7 @@ def clear(self, prefix=""):
                 bucket = self.resource.Bucket(self.bucket)
                 bucket.objects.filter(Prefix=path).delete()
             except Exception as err:
-                reload = self.need_to_reload_creds(err)
-                manager = S3ReloadCredentialsManager if reload else S3ResetClientManager
-                with manager(self, S3DeletionError):
+                with S3ResetReloadCredentialsManager(self, S3DeletionError):
                     bucket = self.resource.Bucket(self.bucket)
                     bucket.objects.filter(Prefix=self.path).delete()
 
@@ -494,14 +472,13 @@ def _check_update_creds(self, force=False):
                 org_id,
                 ds_name,
                 mode,
-                force,
+                True,
             )
             self.expiration = expiration
-            self._set_s3_client_and_resource(
-                creds.get("aws_access_key_id"),
-                creds.get("aws_secret_access_key"),
-                creds.get("aws_session_token"),
-            )
+            self.aws_access_key_id = creds.get("aws_access_key_id")
+            self.aws_secret_access_key = creds.get("aws_secret_access_key")
+            self.aws_session_token = creds.get("aws_session_token")
+            self._set_s3_client_and_resource()
 
     def _locate_and_load_creds(self):
         session = boto3.session.Session(profile_name=self.profile_name)._session
@@ -520,16 +497,11 @@ def _locate_and_load_creds(self):
                 self.aws_region, self.client_config
             )
 
-    def _set_s3_client_and_resource(
-        self, aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None
-    ):
-        key = aws_access_key_id or self.aws_access_key_id
-        secret = aws_secret_access_key or self.aws_secret_access_key
-        token = aws_session_token or self.aws_session_token
+    def _set_s3_client_and_resource(self):
         args = {
-            "aws_access_key_id": key,
-            "aws_secret_access_key": secret,
-            "aws_session_token": token,
+            "aws_access_key_id": self.aws_access_key_id,
+            "aws_secret_access_key": self.aws_secret_access_key,
+            "aws_session_token": self.aws_session_token,
             "region_name": self.aws_region,
             "endpoint_url": self.endpoint_url,
             "config": self.client_config,
@@ -596,9 +568,7 @@ def get_object_from_full_url(self, url: str):
         except botocore.exceptions.ClientError as err:
             if err.response["Error"]["Code"] == "NoSuchKey":
                 raise KeyError(err) from err
-            reload = self.need_to_reload_creds(err)
-            manager = S3ReloadCredentialsManager if reload else S3ResetClientManager  # type: ignore
-            with manager(self, S3GetError):  # type: ignore
+            with S3ResetReloadCredentialsManager(self, S3GetError):  # type: ignore
                 return self._get(path, bucket)
         except CONNECTION_ERRORS as err:
             tries = self.num_tries


Test patch: