Repository: ethyca/fides
Pull request #5922: LJ-372 Updated Attachments/uploads to handle large files.
Author: login: JadeCara
Created: 2025-03-21T21:59:12Z | Merged: 2025-03-25T14:55:44Z

### Description Of Changes

Made some updates to Attachments, as well as the generic s3 storage functions that will allow them to handle larger files. This will allow us to upload files without having to read into memory. 

### Code Changes
 
* Updated the Attachment class to expect attachment_file as IO[bytes].
* Added chunking for uploads
* Added filesize checking for downloads. 
* Added some docs and additional error handling functionality to s3 upload functions. 

### Steps to Confirm

1.  Fidesplus should load without any overlap warnings. 
2. The rest of the functionality is still not linked to the front end and can be verified through test coverage/passing tests.

### Pre-Merge Checklist

* [ ] Issue requirements met
* [ ] All CI pipelines succeeded
* [ ] `CHANGELOG.md` updated
  * [ ] Add a https://github.com/ethyca/fides/labels/db-migration label to the entry if your change includes a DB migration
  * [ ] Add a https://github.com/ethyca/fides/labels/high-risk label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
* Followup issues:
  * [ ] Followup issues created (include link)
  * [ ] No followup issues
* Database migrations:
  * [ ] Ensure that your downrev is up to date with the latest revision on `main`
  * [ ] Ensure that your `downgrade()` migration is correct and works
    * [ ] If a downgrade migration is not possible for this change, please call this out in the PR description!
  * [ ] No migrations
* Documentation:
  * [ ] Documentation complete, [PR opened in fidesdocs](https://github.com/ethyca/fidesdocs/pulls)
  * [ ] Documentation [issue created in fidesdocs](https://github.com/ethyca/fidesdocs/issues/new/choose)
  * [ ] If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  * [ ] No documentation updates required



Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3fc54614872..543b978e7be 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -38,6 +38,7 @@ Changes can also be flagged with a GitHub label for tracking purposes. The URL o
 - Added option for disabling consent notice toggles [#5872](https://github.com/ethyca/fides/pull/5872)
 - Added UI to manually update Assets in the system asset view [#5914](https://github.com/ethyca/fides/pull/5914)
 - Use the experience's `tcf_publisher_country_code` when building TC strings [#5921](https://github.com/ethyca/fides/pull/5921)
+- Added size thresholds to S3 upload and retrieval methods for more efficient document processing. [#5922](https://github.com/ethyca/fides/pull/5922)
 
 ### Changed
 - Changed discovered asset "system" cell to use `user_assigned_system_key` property [#5908](https://github.com/ethyca/fides/pull/5908)
diff --git a/src/fides/api/models/attachment.py b/src/fides/api/models/attachment.py
index 353d47f287c..0ff8f2b27a9 100644
--- a/src/fides/api/models/attachment.py
+++ b/src/fides/api/models/attachment.py
@@ -1,6 +1,6 @@
 import os
 from enum import Enum as EnumType
-from typing import Any, Optional
+from typing import IO, Any, Optional
 
 from loguru import logger as log
 from sqlalchemy import Column
@@ -110,7 +110,7 @@ class Attachment(Base):
         uselist=False,
     )
 
-    def upload(self, attachment: bytes) -> None:
+    def upload(self, attachment: IO[bytes]) -> None:
         """Uploads an attachment to S3 or local storage."""
         if self.config.type == StorageType.s3:
             bucket_name = f"{self.config.details[StorageDetails.BUCKET.value]}"
@@ -128,7 +128,7 @@ def upload(self, attachment: bytes) -> None:
         if self.config.type == StorageType.local:
             filename = get_local_filename(self.id)
             with open(filename, "wb") as file:
-                file.write(attachment)
+                file.write(attachment.read())
             return
 
         raise ValueError(f"Unsupported storage type: {self.config.type}")
@@ -148,7 +148,7 @@ def retrieve_attachment(self) -> Optional[bytes]:
         if self.config.type == StorageType.local:
             filename = f"{LOCAL_FIDES_UPLOAD_DIRECTORY}/{self.id}"
             with open(filename, "rb") as file:
-                return file.read()
+                return file.read(), filename
 
         raise ValueError(f"Unsupported storage type: {self.config.type}")
 
@@ -178,7 +178,7 @@ def create_and_upload(
         db: Session,
         *,
         data: dict[str, Any],
-        attachment_file: bytes,
+        attachment_file: IO[bytes],
         check_name: bool = False,
     ) -> "Attachment":
         """Creates a new attachment record in the database and uploads the attachment to S3."""
diff --git a/src/fides/api/service/storage/s3.py b/src/fides/api/service/storage/s3.py
index 77eb8945d7c..a82076d6272 100644
--- a/src/fides/api/service/storage/s3.py
+++ b/src/fides/api/service/storage/s3.py
@@ -1,7 +1,6 @@
 from __future__ import annotations
 
-from io import BytesIO
-from typing import Any, Dict, Optional
+from typing import IO, Any, Dict, Optional, Tuple
 
 from boto3.s3.transfer import TransferConfig
 from botocore.exceptions import ClientError, ParamValidationError
@@ -12,11 +11,34 @@
 from fides.api.util.aws_util import get_s3_client
 from fides.config import CONFIG
 
+LARGE_FILE_THRESHOLD = 5 * 1024 * 1024  # 5 MB threshold for large files
+LARGE_FILE_WARNING_TEXT = (
+    "File is too large to display directly. Please use the provided link to download."
+)
+
+
+def maybe_get_s3_client(
+    auth_method: str, storage_secrets: Dict[StorageSecrets, Any]
+) -> Any:
+    """
+    Returns an S3 client if the client can be created successfully, otherwise raises an exception.
+    """
+    try:
+        return get_s3_client(auth_method, storage_secrets)
+    except ClientError as e:
+        logger.error(
+            "Encountered error while uploading and generating link for S3 object: {}", e
+        )
+        raise e
+    except ParamValidationError as e:
+        raise ValueError(f"The parameters you provided are incorrect: {e}")
+
 
 def create_presigned_url_for_s3(
     s3_client: Any, bucket_name: str, file_key: str
 ) -> AnyHttpUrlString:
-    """ "Generate a presigned URL to share an S3 object
+    """
+    Generates a presigned URL to share an S3 object
 
     :param s3_client: s3 base client
     :param bucket_name: string
@@ -38,52 +60,68 @@ def generic_upload_to_s3(  # pylint: disable=R0913
     bucket_name: str,
     file_key: str,
     auth_method: str,
-    document: BytesIO,
+    document: IO[bytes],
+    size_threshold: int = LARGE_FILE_THRESHOLD,  # 5 MB threshold
 ) -> Optional[AnyHttpUrlString]:
     """
     Uploads arbitrary data to S3 returned from an access request.
     Handles both small and large uploads.
+
+    :param storage_secrets: S3 storage secrets
+    :param bucket_name: Name of the S3 bucket
+    :param file_key: Key of the file in the bucket
+    :param auth_method: Authentication method for S3
+    :param document: File contents to upload
     """
     logger.info("Starting S3 Upload of {}", file_key)
 
-    try:
-        s3_client = get_s3_client(auth_method, storage_secrets)
+    s3_client = maybe_get_s3_client(auth_method, storage_secrets)
+
+    # Define a transfer configuration for multipart uploads
+    transfer_config = TransferConfig(
+        multipart_threshold=size_threshold,
+        multipart_chunksize=size_threshold,
+    )
 
-        # Define a transfer configuration for multipart uploads
-        transfer_config = TransferConfig(
-            multipart_threshold=5 * 1024 * 1024,  # 5 MB threshold for multipart uploads
-            multipart_chunksize=5 * 1024 * 1024,  # 5 MB chunk size
+    # Use upload_fileobj for efficient uploads (handles both small and large files)
+    try:
+        # Use upload_fileobj for efficient uploads (handles both small and large files)
+        s3_client.upload_fileobj(
+            Fileobj=document,
+            Bucket=bucket_name,
+            Key=file_key,
+            Config=transfer_config,
         )
+    except Exception as e:
+        logger.error(f"Failed to upload file {file_key} to bucket {bucket_name}: {e}")
+        raise e  # Re-raise the exception if you want it to propagate
 
-        try:
-            # Check if the document is a file-like object or raw bytes
-            if isinstance(document, bytes):
-                document = BytesIO(document)  # Wrap raw bytes in a file-like object
-
-            # Use upload_fileobj for efficient uploads (handles both small and large files)
-            s3_client.upload_fileobj(
-                Fileobj=document,
-                Bucket=bucket_name,
-                Key=file_key,
-                Config=transfer_config,
-            )
-        except Exception as e:
-            logger.error("Encountered error while uploading S3 object: {}", e)
-            raise e
+    # Generate a presigned URL for the uploaded file
+    logger.info(f"Successfully uploaded file {file_key} to bucket {bucket_name}")
+    return create_presigned_url_for_s3(s3_client, bucket_name, file_key)
 
-        # Generate a presigned URL for the uploaded file
-        presigned_url: AnyHttpUrlString = create_presigned_url_for_s3(
-            s3_client, bucket_name, file_key
-        )
 
-        return presigned_url
-    except ClientError as e:
-        logger.error(
-            "Encountered error while uploading and generating link for S3 object: {}", e
-        )
-        raise e
-    except ParamValidationError as e:
-        raise ValueError(f"The parameters you provided are incorrect: {e}")
+def generic_download_from_s3(
+    storage_secrets: Dict[StorageSecrets, Any],
+    bucket_name: str,
+    file_key: str,
+    auth_method: str,
+) -> Optional[AnyHttpUrlString]:
+    """
+    Returns a presigned URL for downloading an S3 object.
+
+    :param storage_secrets: S3 storage secrets
+    :param bucket_name: Name of the S3 bucket
+    :param file_key: Key of the file in the bucket
+    :param auth_method: Authentication method for S3
+    :return: A presigned URL (str) for the S3 object
+    """
+    logger.info("Generating presigned URL for downloading file: {}", file_key)
+
+    s3_client = maybe_get_s3_client(auth_method, storage_secrets)
+
+    # Generate a presigned URL for the file
+    return create_presigned_url_for_s3(s3_client, bucket_name, file_key)
 
 
 def generic_retrieve_from_s3(
@@ -91,23 +129,44 @@ def generic_retrieve_from_s3(
     bucket_name: str,
     file_key: str,
     auth_method: str,
-) -> Optional[BytesIO]:
-    """Retrieves arbitrary data from s3"""
+    size_threshold: int = LARGE_FILE_THRESHOLD,  # 5 MB threshold
+) -> Tuple[str, AnyHttpUrlString]:
+    """
+    Retrieves arbitrary data from S3.
+    Returns the file contents and presigned url to download the file if the file is small,
+    or a warning message with the presigned URL to download the file if it is large.
+
+    :param storage_secrets: S3 storage secrets
+    :param bucket_name: Name of the S3 bucket
+    :param file_key: Key of the file in the bucket
+    :param auth_method: Authentication method for S3
+    :param size_threshold: Size threshold in bytes to determine small vs large files
+    :return: File contents (BytesIO) for small files or a presigned URL (str) for large files
+    """
     logger.info("Starting S3 Retrieve of {}", file_key)
 
+    s3_client = maybe_get_s3_client(auth_method, storage_secrets)
     try:
-        s3_client = get_s3_client(auth_method, storage_secrets)
-        try:
+        # Get the object metadata to check its size
+        head_response = s3_client.head_object(Bucket=bucket_name, Key=file_key)
+        file_size = head_response["ContentLength"]
+
+        if file_size <= size_threshold:
+            # File is small, return its contents and presigned url
             response = s3_client.get_object(Bucket=bucket_name, Key=file_key)
-            return response["Body"].read()
-        except Exception as e:
-            logger.error("Encountered error while retrieving s3 object: {}", e)
-            raise e
-    except ClientError as e:
-        logger.error("Encountered error while retrieving s3 object: {}", e)
+            return response["Body"].read(), create_presigned_url_for_s3(
+                s3_client, bucket_name, file_key
+            )
+
+        # File is large, return warning text and a presigned URL
+        return (
+            LARGE_FILE_WARNING_TEXT,
+            create_presigned_url_for_s3(s3_client, bucket_name, file_key),
+        )
+
+    except Exception as e:
+        logger.error("Encountered error while retrieving S3 object: {}", e)
         raise e
-    except ParamValidationError as e:
-        raise ValueError(f"The parameters you provided are incorrect: {e}")
 
 
 def generic_delete_from_s3(
@@ -116,18 +175,19 @@ def generic_delete_from_s3(
     file_key: str,
     auth_method: str,
 ) -> None:
-    """Deletes arbitrary data from s3"""
+    """
+    Deletes arbitrary data from s3
+
+    :param storage_secrets: S3 storage secrets
+    :param bucket_name: Name of the S3 bucket
+    :param file_key: Key of the file in the bucket
+    :param auth_method: Authentication method for S3
+    """
     logger.info("Starting S3 Delete of {}", file_key)
 
+    s3_client = maybe_get_s3_client(auth_method, storage_secrets)
     try:
-        s3_client = get_s3_client(auth_method, storage_secrets)
-        try:
-            s3_client.delete_object(Bucket=bucket_name, Key=file_key)
-        except Exception as e:
-            logger.error("Encountered error while deleting s3 object: {}", e)
-            raise e
-    except ClientError as e:
+        s3_client.delete_object(Bucket=bucket_name, Key=file_key)
+    except Exception as e:
         logger.error("Encountered error while deleting s3 object: {}", e)
         raise e
-    except ParamValidationError as e:
-        raise ValueError(f"The parameters you provided are incorrect: {e}")
diff --git a/tests/ctl/models/conftest.py b/tests/ctl/models/conftest.py
index f0ffb57ec8a..9e11df61a12 100644
--- a/tests/ctl/models/conftest.py
+++ b/tests/ctl/models/conftest.py
@@ -1,3 +1,5 @@
+from io import BytesIO
+
 import pytest
 
 from fides.api.models.attachment import (
@@ -21,17 +23,17 @@ def mock_get_s3_client(auth_method, storage_secrets):
     attachment_data["user_id"] = user.id
     attachment_data["file_name"] = "file_1.txt"
     attachment_1 = Attachment.create_and_upload(
-        db, data=attachment_data, attachment_file=b"file content 1"
+        db, data=attachment_data, attachment_file=BytesIO(b"file content 1")
     )
 
     attachment_data["file_name"] = "file_2.txt"
     attachment_2 = Attachment.create_and_upload(
-        db, data=attachment_data, attachment_file=b"file content 2"
+        db, data=attachment_data, attachment_file=BytesIO(b"file content 2")
     )
 
     attachment_data["file_name"] = "file_3.txt"
     attachment_3 = Attachment.create_and_upload(
-        db, data=attachment_data, attachment_file=b"file content 3"
+        db, data=attachment_data, attachment_file=BytesIO(b"file content 3")
     )
 
     yield attachment_1, attachment_2, attachment_3
diff --git a/tests/ctl/models/test_attachment.py b/tests/ctl/models/test_attachment.py
index d928f0edcaa..fb482ea7467 100644
--- a/tests/ctl/models/test_attachment.py
+++ b/tests/ctl/models/test_attachment.py
@@ -1,3 +1,5 @@
+from io import BytesIO
+
 import pytest
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.orm import Session
@@ -11,6 +13,7 @@
 from fides.api.models.fides_user import FidesUser
 from fides.api.models.storage import StorageConfig
 from fides.api.schemas.storage.storage import StorageDetails
+from fides.api.service.storage.util import LOCAL_FIDES_UPLOAD_DIRECTORY
 
 
 @pytest.fixture(
@@ -18,11 +21,29 @@
         ("testfile.pdf", b"%PDF-1.4 Test PDF content"),
         ("testfile.txt", b"Test text content"),
         ("testfile.jpeg", b"\xff\xd8\xff\xe0\x00\x10JFIF Test JPEG content"),
-    ]
+    ],
+    scope="function",
 )
 def attachment_file(request):
+    """
+    Fixture to provide file-like objects for testing.
+    Each parameter is a tuple of (filename, file content as bytes).
+    A new BytesIO object is created for each test to avoid reuse issues.
+    """
     file_name, file_content = request.param
-    return file_name, file_content
+    return file_name, BytesIO(file_content)
+
+
+def verify_attachment_created_uploaded_s3(attachment, attachment_file_copy):
+    retrieved_file, download_url = attachment.retrieve_attachment()
+    assert retrieved_file == attachment_file_copy
+    assert attachment.config.details[StorageDetails.BUCKET.value] in download_url
+
+
+def verify_attachment_created_uploaded_local(attachment, attachment_file_copy):
+    retrieved_attachment, download_path = attachment.retrieve_attachment()
+    assert retrieved_attachment == attachment_file_copy
+    assert download_path == f"{LOCAL_FIDES_UPLOAD_DIRECTORY}/{attachment.id}"
 
 
 def test_create_attachment_without_attachement_file_raises_error(db, attachment_data):
@@ -35,6 +56,8 @@ def test_create_attachment_with_S3_storage(
     s3_client, db, user, attachment_data, attachment_file, monkeypatch
 ):
     """Test creating an attachment."""
+    # Create a copy of the file-like object
+    attachment_file_copy = attachment_file[1].getvalue()
 
     def mock_get_s3_client(auth_method, storage_secrets):
         return s3_client
@@ -60,7 +83,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
         )
         is not None
     )
-    assert retrieved_attachment.retrieve_attachment() == attachment_file[1]
+    verify_attachment_created_uploaded_s3(attachment, attachment_file_copy)
     attachment.delete(db)
 
 
@@ -68,6 +91,7 @@ def test_create_attachment_with_local_storage(
     db, attachment_data, attachment_file, storage_config_local
 ):
     """Test creating an attachment."""
+    attachment_file_copy = attachment_file[1].getvalue()
 
     attachment_data["storage_key"] = storage_config_local.key
     attachment = Attachment.create_and_upload(
@@ -80,7 +104,7 @@ def test_create_attachment_with_local_storage(
     assert retrieved_attachment.file_name == attachment.file_name
     assert retrieved_attachment.attachment_type == attachment.attachment_type
 
-    assert retrieved_attachment.retrieve_attachment() == attachment_file[1]
+    verify_attachment_created_uploaded_local(attachment, attachment_file_copy)
     attachment.delete(db)
 
 
@@ -102,6 +126,8 @@ def test_retrieve_attachment_from_s3(
     s3_client, db, attachment_data, attachment_file, monkeypatch
 ):
     """Test retrieving an attachment (bytes) from S3."""
+    # Create a copy of the file-like object
+    attachment_file_copy = attachment_file[1].getvalue()
 
     def mock_get_s3_client(auth_method, storage_secrets):
         return s3_client
@@ -111,8 +137,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
     attachment = Attachment.create_and_upload(
         db, data=attachment_data, attachment_file=attachment_file[1]
     )
-    retrieved_file = attachment.retrieve_attachment()
-    assert retrieved_file == attachment_file[1]
+    verify_attachment_created_uploaded_s3(attachment, attachment_file_copy)
     attachment.delete(db)
 
 
@@ -120,13 +145,14 @@ def test_retrieve_attachment_from_local(
     db, attachment_data, attachment_file, storage_config_local
 ):
     """Test retrieving an attachment locally."""
-
+    # Create a copy of the file-like object
+    attachment_file_copy = attachment_file[1].getvalue()
     attachment_data["storage_key"] = storage_config_local.key
     attachment = Attachment.create_and_upload(
         db=db, data=attachment_data, attachment_file=attachment_file[1]
     )
 
-    assert attachment.retrieve_attachment() == attachment_file[1]
+    verify_attachment_created_uploaded_local(attachment, attachment_file_copy)
     attachment.delete(db)
 
 
@@ -134,6 +160,8 @@ def test_delete_attachment_from_s3(
     s3_client, db, attachment_data, attachment_file, monkeypatch
 ):
     """Test deleting an attachment from S3."""
+    # Create a copy of the file-like object
+    attachment_file_copy = attachment_file[1].getvalue()
 
     def mock_get_s3_client(auth_method, storage_secrets):
         return s3_client
@@ -143,8 +171,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
     attachment = Attachment.create_and_upload(
         db, data=attachment_data, attachment_file=attachment_file[1]
     )
-
-    assert attachment.retrieve_attachment() == attachment_file[1]
+    verify_attachment_created_uploaded_s3(attachment, attachment_file_copy)
 
     # Delete the file using the method
     attachment.delete_attachment_from_storage()
@@ -161,12 +188,13 @@ def test_delete_attachment_from_local(
     db, attachment_data, attachment_file, storage_config_local
 ):
     """Test deleting an attachment locally."""
+    # Create a copy of the file-like object
+    attachment_file_copy = attachment_file[1].getvalue()
     attachment_data["storage_key"] = storage_config_local.key
     attachment = Attachment.create_and_upload(
         db=db, data=attachment_data, attachment_file=attachment_file[1]
     )
-
-    assert attachment.retrieve_attachment() == attachment_file[1]
+    verify_attachment_created_uploaded_local(attachment, attachment_file_copy)
 
     # Delete the file using the method
     attachment.delete_attachment_from_storage()
@@ -195,7 +223,7 @@ def test_attachment_storageconfig_foreign_key_constraint(
     """Test that deleting storage config cascades."""
     attachment_data["storage_key"] = storage_config_local.key
     attachment = Attachment.create_and_upload(
-        db=db, data=attachment_data, attachment_file=b"test file content"
+        db=db, data=attachment_data, attachment_file=BytesIO(b"test file content")
     )
 
     config = db.query(StorageConfig).filter_by(key=attachment.storage_key).first()
diff --git a/tests/ctl/models/test_comment.py b/tests/ctl/models/test_comment.py
index 42023501ec1..e83c1ca319a 100644
--- a/tests/ctl/models/test_comment.py
+++ b/tests/ctl/models/test_comment.py
@@ -261,15 +261,3 @@ def test_comment_reference_foreign_key_constraint(db):
     db.add(comment_reference)
     with pytest.raises(IntegrityError):
         db.commit()
-
-
-def test_delete_comment_reference_cascades(db, comment, comment_reference):
-    """Test that deleting a comment reference does not delete the comment."""
-    retrieved_comment = db.query(Comment).filter_by(id=comment.id).first()
-    assert retrieved_comment is not None
-
-    comment_reference.delete(db=db)
-
-    retrieved_comment = db.query(Comment).filter_by(id=comment.id).first()
-    assert retrieved_comment is not None
-    assert len(retrieved_comment.references) == 0
diff --git a/tests/fixtures/application_fixtures.py b/tests/fixtures/application_fixtures.py
index 0198da3233a..f5d69392768 100644
--- a/tests/fixtures/application_fixtures.py
+++ b/tests/fixtures/application_fixtures.py
@@ -2,6 +2,7 @@
 import uuid
 from copy import deepcopy
 from datetime import datetime, timedelta, timezone
+from io import BytesIO
 from typing import Dict, Generator, List, Optional
 from unittest import mock
 from uuid import uuid4
@@ -3887,7 +3888,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
         "fides.api.service.storage.s3.get_s3_client", mock_get_s3_client
     )
     attachment = Attachment.create_and_upload(
-        db, data=attachment_data, attachment_file=b"file content"
+        db, data=attachment_data, attachment_file=BytesIO(b"file content")
     )
     yield attachment
     attachment.delete(db)
diff --git a/tests/ops/models/privacy_request/test_privacy_request.py b/tests/ops/models/privacy_request/test_privacy_request.py
index a4dc23dc2b8..e183422eaf0 100644
--- a/tests/ops/models/privacy_request/test_privacy_request.py
+++ b/tests/ops/models/privacy_request/test_privacy_request.py
@@ -1,4 +1,5 @@
 from datetime import datetime, timedelta, timezone
+from io import BytesIO
 from time import sleep
 from typing import Any, Dict, List
 from uuid import uuid4
@@ -1161,17 +1162,17 @@ def mock_get_s3_client(auth_method, storage_secrets):
     attachment1 = Attachment.create_and_upload(
         db=db,
         data=attachment_data,
-        attachment_file=b"contents of test file 1",
+        attachment_file=BytesIO(b"contents of test file 1"),
     )
     attachment2 = Attachment.create_and_upload(
         db=db,
         data=attachment_data,
-        attachment_file=b"contents of test file 2",
+        attachment_file=BytesIO(b"contents of test file 2"),
     )
     attachment3 = Attachment.create_and_upload(
         db=db,
         data=attachment_data,
-        attachment_file=b"contents of test file 3",
+        attachment_file=BytesIO(b"contents of test file 3"),
     )
 
     # Associate Attachments with the PrivacyRequest
diff --git a/tests/ops/service/storage/test_s3.py b/tests/ops/service/storage/test_s3.py
index 742ed1dcb40..4821bc2b2e5 100644
--- a/tests/ops/service/storage/test_s3.py
+++ b/tests/ops/service/storage/test_s3.py
@@ -1,40 +1,116 @@
+from io import BytesIO
+from tempfile import SpooledTemporaryFile
+
 import boto3
 import pytest
+from botocore.exceptions import ClientError, ParamValidationError
 from moto import mock_aws
+from pytest import param
 
 from fides.api.schemas.storage.storage import StorageDetails
 from fides.api.service.storage.s3 import (
+    LARGE_FILE_THRESHOLD,
+    LARGE_FILE_WARNING_TEXT,
     generic_delete_from_s3,
     generic_retrieve_from_s3,
     generic_upload_to_s3,
+    maybe_get_s3_client,
 )
 
+TEST_DOCUMENT = b"This is a test document."
+TEST_SPOOLED_DOC = SpooledTemporaryFile()
+
+
+@pytest.mark.parametrize(
+    "mock_get_s3_client, expected_exception, expected_message",
+    [
+        param(
+            lambda auth_method, storage_secrets: "mock_s3_client",
+            None,
+            None,
+            id="success",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(
+                ClientError(
+                    {"Error": {"Code": "403", "Message": "Access Denied"}}, "GetObject"
+                )
+            ),
+            ClientError,
+            "Access Denied",
+            id="client_error",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(ParamValidationError(report="Invalid parameters")),
+            ValueError,
+            "The parameters you provided are incorrect",
+            id="param_validation_error",
+        ),
+    ],
+)
+def test_maybe_get_s3_client(
+    monkeypatch, mock_get_s3_client, expected_exception, expected_message
+):
+    """
+    Test maybe_get_s3_client for all potential outcomes:
+    - Successful return of s3_client
+    - Raising ClientError
+    - Raising ParamValidationError
+    """
+
+    # Mock the get_s3_client function
+    monkeypatch.setattr(
+        "fides.api.service.storage.s3.get_s3_client", mock_get_s3_client
+    )
 
-@pytest.fixture
-def s3_client(storage_config):
-    with mock_aws():
-        session = boto3.Session(
-            aws_access_key_id="fake_access_key",
-            aws_secret_access_key="fake_secret_key",
-            region_name="us-east-1",
-        )
-        s3 = session.client("s3")
-        s3.create_bucket(Bucket=storage_config.details[StorageDetails.BUCKET.value])
-        yield s3
-
-
-def test_generic_upload_to_s3(s3_client, storage_config, monkeypatch):
+    auth_method = "test_auth_method"
+    storage_secrets = {"key": "value"}  # Example secrets
+
+    if expected_exception:
+        # Test for exceptions
+        with pytest.raises(expected_exception) as excinfo:
+            maybe_get_s3_client(auth_method, storage_secrets)
+        assert expected_message in str(excinfo.value)
+    else:
+        result = maybe_get_s3_client(auth_method, storage_secrets)
+        assert result == "mock_s3_client"
+
+
+@pytest.mark.parametrize(
+    "document",
+    [
+        BytesIO(TEST_DOCUMENT),
+        TEST_SPOOLED_DOC,
+    ],
+)
+def test_generic_upload_to_s3(s3_client, document, storage_config, monkeypatch):
+    """
+    Test that generic_upload_to_s3 can handle both BytesIO and SpooledTemporaryFile.
+    """
 
     def mock_get_s3_client(auth_method, storage_secrets):
         return s3_client
 
     monkeypatch.setattr("fides.api.tasks.storage.get_s3_client", mock_get_s3_client)
 
+    # Prepare the document content
+    if isinstance(document, SpooledTemporaryFile):
+        document.write(TEST_DOCUMENT)
+        document.seek(0)  # Reset the file pointer to the beginning
+        copy_document = document.read()
+        document.seek(0)  # Reset the file pointer for the upload
+
+    else:
+        copy_document = document.getvalue()
+
     bucket_name = storage_config.details[StorageDetails.BUCKET.value]
     storage_secrets = storage_config.secrets
     file_key = "test-file"
     auth_method = storage_config.details[StorageDetails.AUTH_METHOD.value]
-    document = b"This is a test document."
 
     presigned_url = generic_upload_to_s3(
         storage_secrets=storage_secrets,
@@ -46,10 +122,9 @@ def mock_get_s3_client(auth_method, storage_secrets):
 
     # Verify the file was uploaded
     response = s3_client.get_object(Bucket=bucket_name, Key=file_key)
-    assert response["Body"].read() == document
+    assert response["Body"].read() == copy_document
 
     # Verify the presigned URL is generated
-    print(presigned_url)
     assert bucket_name in presigned_url
 
 
@@ -60,7 +135,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
 
     monkeypatch.setattr("fides.api.tasks.storage.get_s3_client", mock_get_s3_client)
 
-    s3, bucket_name = s3_client, "test-bucket"
+    bucket_name = "test-bucket"
     storage_secrets = {
         "aws_access_key_id": "fake_access",
         "aws_secret_access_key": "fake",
@@ -82,6 +157,39 @@ def mock_get_s3_client(auth_method, storage_secrets):
         assert "NoSuchBucket" in str(e.value)
 
 
+def test_generic_download_from_s3_with_large_file(
+    s3_client, storage_config, monkeypatch
+):
+
+    def mock_get_s3_client(auth_method, storage_secrets):
+        return s3_client
+
+    monkeypatch.setattr("fides.api.tasks.storage.get_s3_client", mock_get_s3_client)
+
+    bucket_name = storage_config.details[StorageDetails.BUCKET.value]
+    storage_secrets = storage_config.secrets
+    file_key = "test-file"
+    auth_method = storage_config.details[StorageDetails.AUTH_METHOD.value]
+    document = BytesIO(b"0" * (LARGE_FILE_THRESHOLD))  # 5 MB document
+
+    copy_document = document.getvalue()
+
+    presigned_url = generic_upload_to_s3(
+        storage_secrets=storage_secrets,
+        bucket_name=bucket_name,
+        file_key=file_key,
+        auth_method=auth_method,
+        document=document,
+    )
+
+    # Verify the file was uploaded
+    response = s3_client.get_object(Bucket=bucket_name, Key=file_key)
+    assert response["Body"].read() == copy_document
+
+    # Verify the presigned URL is generated
+    assert bucket_name in presigned_url
+
+
 def test_generic_retrieve_from_s3(s3_client, storage_config, monkeypatch):
 
     def mock_get_s3_client(auth_method, storage_secrets):
@@ -99,7 +207,7 @@ def mock_get_s3_client(auth_method, storage_secrets):
     s3.put_object(Bucket=bucket_name, Key=file_key, Body=document)
 
     # Retrieve the document
-    retrieved_document = generic_retrieve_from_s3(
+    retrieved_document, download_link = generic_retrieve_from_s3(
         storage_secrets=storage_secrets,
         bucket_name=bucket_name,
         file_key=file_key,
@@ -108,6 +216,38 @@ def mock_get_s3_client(auth_method, storage_secrets):
 
     # Verify the document was retrieved correctly
     assert retrieved_document == document
+    assert bucket_name in download_link
+
+
+def test_generic_retrieve_from_s3_with_large_file(
+    s3_client, storage_config, monkeypatch
+):
+
+    def mock_get_s3_client(auth_method, storage_secrets):
+        return s3_client
+
+    monkeypatch.setattr("fides.api.tasks.storage.get_s3_client", mock_get_s3_client)
+
+    s3, bucket_name = s3_client, storage_config.details[StorageDetails.BUCKET.value]
+    storage_secrets = storage_config.secrets
+    file_key = "test-file"
+    auth_method = storage_config.details[StorageDetails.AUTH_METHOD.value]
+    document = b"0" * (LARGE_FILE_THRESHOLD + 1)  # 5 MB document
+
+    # Upload the document first
+    s3.put_object(Bucket=bucket_name, Key=file_key, Body=document)
+
+    # Retrieve the document
+    warning_text, presigned_url = generic_retrieve_from_s3(
+        storage_secrets=storage_secrets,
+        bucket_name=bucket_name,
+        file_key=file_key,
+        auth_method=auth_method,
+    )
+
+    # Verify the document was retrieved correctly
+    assert warning_text == LARGE_FILE_WARNING_TEXT
+    assert bucket_name in presigned_url
 
 
 def test_generic_retrieve_file_not_found(s3_client, storage_config, monkeypatch):


Reviews:
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:11:41Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:12:54Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:13:18Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:24:17Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:24:36Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:25:16Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T17:25:45Z
user: login: erosselli
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T18:59:57Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T23:13:18Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T23:21:13Z
user: login: erosselli
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:42:26Z
user: login: erosselli
body: Thanks for making the changes! I approved with one question (non-blocking), and I think you're missing a changelog entry 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:52:02Z
user: login: JadeCara
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T14:15:40Z

Review comments:
diff_hunk: @@ -191,6 +199,7 @@ def create_and_upload(
             return attachment_model
         except Exception as e:
             log.error(f"Failed to upload attachment: {e}")
+            log.error(f"{e}")
path: src/fides/api/models/attachment.py
user: login: JadeCara
body: ```suggestion
```
created_at: 2025-03-24T17:11:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,11 +12,34 @@
 from fides.api.util.aws_util import get_s3_client
 from fides.config import CONFIG
 
+LARGE_FILE_THRESHOLD = 5 * 1024 * 1024  # 5 MB threshold for large files
+LARGE_FILE_WARNING_TEXT = (
+    "File is too large to display directly. Please use the provided link to download."
+)
+
+
+def maybe_get_s3_client(
path: src/fides/api/service/storage/s3.py
user: login: JadeCara
body: This new function replaces the `try`/`except` functionality in each function below it. 
created_at: 2025-03-24T17:12:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,76 +61,111 @@ def generic_upload_to_s3(  # pylint: disable=R0913
     bucket_name: str,
     file_key: str,
     auth_method: str,
-    document: BytesIO,
+    document: BytesIO | IO[bytes],
 ) -> Optional[AnyHttpUrlString]:
     """
     Uploads arbitrary data to S3 returned from an access request.
     Handles both small and large uploads.
+
+    :param storage_secrets: S3 storage secrets
+    :param bucket_name: Name of the S3 bucket
+    :param file_key: Key of the file in the bucket
+    :param auth_method: Authentication method for S3
+    :param document: File contents to upload
     """
     logger.info("Starting S3 Upload of {}", file_key)
 
-    try:
-        s3_client = get_s3_client(auth_method, storage_secrets)
+    s3_client = maybe_get_s3_client(auth_method, storage_secrets)
 
-        # Define a transfer configuration for multipart uploads
-        transfer_config = TransferConfig(
-            multipart_threshold=5 * 1024 * 1024,  # 5 MB threshold for multipart uploads
-            multipart_chunksize=5 * 1024 * 1024,  # 5 MB chunk size
-        )
+    # Define a transfer configuration for multipart uploads
+    transfer_config = TransferConfig(
+        multipart_threshold=LARGE_FILE_THRESHOLD,  # 5 MB threshold for multipart uploads
+        multipart_chunksize=LARGE_FILE_THRESHOLD,  # 5 MB chunk size
path: src/fides/api/service/storage/s3.py
user: login: JadeCara
body: ```suggestion
        multipart_threshold=LARGE_FILE_THRESHOLD,
        multipart_chunksize=LARGE_FILE_THRESHOLD,
```
created_at: 2025-03-24T17:13:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -11,18 +13,37 @@
 from fides.api.models.fides_user import FidesUser
 from fides.api.models.storage import StorageConfig
 from fides.api.schemas.storage.storage import StorageDetails
+from fides.api.service.storage.util import LOCAL_FIDES_UPLOAD_DIRECTORY
 
 
 @pytest.fixture(
     params=[
         ("testfile.pdf", b"%PDF-1.4 Test PDF content"),
         ("testfile.txt", b"Test text content"),
         ("testfile.jpeg", b"\xff\xd8\xff\xe0\x00\x10JFIF Test JPEG content"),
-    ]
+    ],
+    scope="function",
 )
 def attachment_file(request):
+    """
+    Fixture to provide file-like objects for testing.
+    Each parameter is a tuple of (filename, file content as bytes).
+    A new BytesIO object is created for each test to avoid reuse issues.
+    """
     file_name, file_content = request.param
-    return file_name, file_content
+    return file_name, BytesIO(file_content)
+
+
+def verify_attachment_created_uploaded_s3(attachment, attachment_file_copy):
+    retrieved_file, download_url = attachment.retrieve_attachment()
+    assert retrieved_file == attachment_file_copy
+    assert attachment.config.details[StorageDetails.BUCKET.value] in download_url
+
+
+def verify_attachment_created_uploaded_local(attachment, attachment_file_copy):
+    retrieved_attachment, download_path = attachment.retrieve_attachment()
+    assert retrieved_attachment == attachment_file_copy
+    assert download_path == f"{LOCAL_FIDES_UPLOAD_DIRECTORY}/{attachment.id}"
path: tests/ctl/models/test_attachment.py
user: login: JadeCara
body: Simplifies below tests which want to verify the attachments have uploaded. 
created_at: 2025-03-24T17:24:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -11,18 +13,37 @@
 from fides.api.models.fides_user import FidesUser
 from fides.api.models.storage import StorageConfig
 from fides.api.schemas.storage.storage import StorageDetails
+from fides.api.service.storage.util import LOCAL_FIDES_UPLOAD_DIRECTORY
 
 
 @pytest.fixture(
     params=[
         ("testfile.pdf", b"%PDF-1.4 Test PDF content"),
         ("testfile.txt", b"Test text content"),
         ("testfile.jpeg", b"\xff\xd8\xff\xe0\x00\x10JFIF Test JPEG content"),
-    ]
+    ],
+    scope="function",
 )
 def attachment_file(request):
+    """
+    Fixture to provide file-like objects for testing.
+    Each parameter is a tuple of (filename, file content as bytes).
+    A new BytesIO object is created for each test to avoid reuse issues.
+    """
     file_name, file_content = request.param
-    return file_name, file_content
+    return file_name, BytesIO(file_content)
path: tests/ctl/models/test_attachment.py
user: login: JadeCara
body: Updates datatype to be file like object.
created_at: 2025-03-24T17:24:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,40 +1,116 @@
+from io import BytesIO
+from tempfile import SpooledTemporaryFile
+
 import boto3
 import pytest
+from botocore.exceptions import ClientError, ParamValidationError
 from moto import mock_aws
+from pytest import param
 
 from fides.api.schemas.storage.storage import StorageDetails
 from fides.api.service.storage.s3 import (
+    LARGE_FILE_THRESHOLD,
+    LARGE_FILE_WARNING_TEXT,
     generic_delete_from_s3,
     generic_retrieve_from_s3,
     generic_upload_to_s3,
+    maybe_get_s3_client,
 )
 
+TEST_DOCUMENT = b"This is a test document."
+TEST_SPOOLED_DOC = SpooledTemporaryFile()
+
+
+@pytest.mark.parametrize(
+    "mock_get_s3_client, expected_exception, expected_message",
+    [
+        param(
+            lambda auth_method, storage_secrets: "mock_s3_client",
+            None,
+            None,
+            id="success",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(
+                ClientError(
+                    {"Error": {"Code": "403", "Message": "Access Denied"}}, "GetObject"
+                )
+            ),
+            ClientError,
+            "Access Denied",
+            id="client_error",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(ParamValidationError(report="Invalid parameters")),
+            ValueError,
+            "The parameters you provided are incorrect",
+            id="param_validation_error",
+        ),
+    ],
+)
+def test_maybe_get_s3_client(
+    monkeypatch, mock_get_s3_client, expected_exception, expected_message
+):
+    """
+    Test maybe_get_s3_client for all potential outcomes:
+    - Successful return of s3_client
+    - Raising ClientError
+    - Raising ParamValidationError
+    """
+
+    # Mock the get_s3_client function
+    monkeypatch.setattr(
+        "fides.api.service.storage.s3.get_s3_client", mock_get_s3_client
+    )
 
-@pytest.fixture
-def s3_client(storage_config):
-    with mock_aws():
-        session = boto3.Session(
-            aws_access_key_id="fake_access_key",
-            aws_secret_access_key="fake_secret_key",
-            region_name="us-east-1",
-        )
-        s3 = session.client("s3")
-        s3.create_bucket(Bucket=storage_config.details[StorageDetails.BUCKET.value])
-        yield s3
path: tests/ops/service/storage/test_s3.py
user: login: JadeCara
body: This is available through conftest. 
created_at: 2025-03-24T17:25:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,40 +1,116 @@
+from io import BytesIO
+from tempfile import SpooledTemporaryFile
+
 import boto3
 import pytest
+from botocore.exceptions import ClientError, ParamValidationError
 from moto import mock_aws
+from pytest import param
 
 from fides.api.schemas.storage.storage import StorageDetails
 from fides.api.service.storage.s3 import (
+    LARGE_FILE_THRESHOLD,
+    LARGE_FILE_WARNING_TEXT,
     generic_delete_from_s3,
     generic_retrieve_from_s3,
     generic_upload_to_s3,
+    maybe_get_s3_client,
 )
 
+TEST_DOCUMENT = b"This is a test document."
+TEST_SPOOLED_DOC = SpooledTemporaryFile()
+
+
+@pytest.mark.parametrize(
+    "mock_get_s3_client, expected_exception, expected_message",
+    [
+        param(
+            lambda auth_method, storage_secrets: "mock_s3_client",
+            None,
+            None,
+            id="success",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(
+                ClientError(
+                    {"Error": {"Code": "403", "Message": "Access Denied"}}, "GetObject"
+                )
+            ),
+            ClientError,
+            "Access Denied",
+            id="client_error",
+        ),
+        param(
+            lambda auth_method, storage_secrets: (
+                _ for _ in ()  # Empty generator to raise exception
+            ).throw(ParamValidationError(report="Invalid parameters")),
+            ValueError,
+            "The parameters you provided are incorrect",
+            id="param_validation_error",
+        ),
+    ],
+)
+def test_maybe_get_s3_client(
+    monkeypatch, mock_get_s3_client, expected_exception, expected_message
+):
+    """
+    Test maybe_get_s3_client for all potential outcomes:
+    - Successful return of s3_client
+    - Raising ClientError
+    - Raising ParamValidationError
+    """
+
+    # Mock the get_s3_client function
+    monkeypatch.setattr(
+        "fides.api.service.storage.s3.get_s3_client", mock_get_s3_client
+    )
 
-@pytest.fixture
-def s3_client(storage_config):
-    with mock_aws():
-        session = boto3.Session(
-            aws_access_key_id="fake_access_key",
-            aws_secret_access_key="fake_secret_key",
-            region_name="us-east-1",
-        )
-        s3 = session.client("s3")
-        s3.create_bucket(Bucket=storage_config.details[StorageDetails.BUCKET.value])
-        yield s3
-
-
-def test_generic_upload_to_s3(s3_client, storage_config, monkeypatch):
+    auth_method = "test_auth_method"
+    storage_secrets = {"key": "value"}  # Example secrets
+
+    if expected_exception:
+        # Test for exceptions
+        with pytest.raises(expected_exception) as excinfo:
+            maybe_get_s3_client(auth_method, storage_secrets)
+        assert expected_message in str(excinfo.value)
+    else:
+        result = maybe_get_s3_client(auth_method, storage_secrets)
+        assert result == "mock_s3_client"
+
+
+@pytest.mark.parametrize(
+    "document",
+    [
+        BytesIO(TEST_DOCUMENT),
+        TEST_SPOOLED_DOC,
+    ],
+)
path: tests/ops/service/storage/test_s3.py
user: login: JadeCara
body: Added test params for both BytesIO and Spooled files. 
created_at: 2025-03-24T17:25:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,6 +98,7 @@ class Comment(Base):
         secondaryjoin="Attachment.id == AttachmentReference.attachment_id",
         order_by="Attachment.created_at",
         uselist=True,
+        viewonly=True,
     )
 
path: src/fides/api/models/comment.py
user: login: erosselli
body: I'm not too SQLAlchemy-savvy , for my own benefit can you explain why we have to add `viewonly=True` here? I understand it gets rid of the fidesplus warnings but not _why_ it does
created_at: 2025-03-24T18:51:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,6 +98,7 @@ class Comment(Base):
         secondaryjoin="Attachment.id == AttachmentReference.attachment_id",
         order_by="Attachment.created_at",
         uselist=True,
+        viewonly=True,
     )
 
path: src/fides/api/models/comment.py
user: login: erosselli
body: I also don't know if I follow all the models here, can you explain the `Comment` , `CommentReference` , `AttachmentReference` and `Attachment` models and how they relate to each other? 
created_at: 2025-03-24T18:53:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -174,13 +174,15 @@ class PrivacyRequest(
         primaryjoin="PrivacyRequest.id == AttachmentReference.reference_id",
         secondaryjoin="Attachment.id == AttachmentReference.attachment_id",
         order_by="Attachment.created_at",
+        viewonly=True,
path: src/fides/api/models/privacy_request/privacy_request.py
user: login: erosselli
body: same question here
created_at: 2025-03-24T18:53:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,76 +61,112 @@ def generic_upload_to_s3(  # pylint: disable=R0913
     bucket_name: str,
     file_key: str,
     auth_method: str,
-    document: BytesIO,
+    document: BytesIO | IO[bytes],
path: src/fides/api/service/storage/s3.py
user: login: erosselli
body: what's the difference between `BytesIO` and `IO[Bytes]` ?
created_at: 2025-03-24T18:56:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,6 +98,7 @@ class Comment(Base):
         secondaryjoin="Attachment.id == AttachmentReference.attachment_id",
         order_by="Attachment.created_at",
         uselist=True,
+        viewonly=True,
     )
 
path: src/fides/api/models/comment.py
user: login: JadeCara
body: Both totally fair questions! 
I looked at all the models and decided to be much clearer on how things link for both sqlalchemy and for us going forward!
I have updated the models with very clear relationship and delete paths. 

Comments and Attachments have reference tables. This is designed so that in the future when we decide we want to add a comment or attachment option to any other object it will be much simpler/require no migrations etc to add a new reference type to either. 
These reference tables mean that we are using them as a secondary join to link up to Privacy request. It also means we have that extra added layer of abstraction when doing deletes. I updated all the backrefs and added some very clear `delete` functionality and tests for it! 

I am sorry this added extra lines but I think its a really good addition for clarity!
created_at: 2025-03-24T23:13:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,76 +61,112 @@ def generic_upload_to_s3(  # pylint: disable=R0913
     bucket_name: str,
     file_key: str,
     auth_method: str,
-    document: BytesIO,
+    document: BytesIO | IO[bytes],
path: src/fides/api/service/storage/s3.py
user: login: JadeCara
body: Another really good question! (And honestly this is a bit redundant. I am going to adjust - read on for why!)
They are pretty similar. 

BytesIO:
-  is a concrete class from the io module in Python.
- It creates an in-memory binary stream that behaves like a file object.
- You can read from and write to it just like you would with a regular file.
- It is primarily used when you want to manipulate binary data in memory without creating actual files.
- BytesIO behaves like a binary file.
- You can perform file-like operations (read, write, seek, etc.).
- It stores the data in memory rather than on disk.

IO[Bytes]: 
- is a type hint used in Python type annotations.
- It indicates that a function or variable expects an I/O stream that handles Bytes.
- It is part of Python’s type hinting system and comes from the typing module.
- It is not an actual class or object but a type alias used for static typing.
- IO[bytes] is only used for type checking and documentation.
- It indicates that the function takes any binary stream (not just BytesIO).
- The stream could be a BytesIO, a socket, or any object that adheres to the binary I/O interface.

TLDR:
BytesIO → Concrete class: For working with in-memory binary streams.
IO[Bytes] → Type hint: For annotating functions that accept any bytes-based I/O stream.
Since BytesIO is already a valid IO[bytes] stream, explicitly including it in the union is unnecessary.
created_at: 2025-03-24T23:21:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,76 +61,112 @@ def generic_upload_to_s3(  # pylint: disable=R0913
     bucket_name: str,
     file_key: str,
     auth_method: str,
-    document: BytesIO,
+    document: BytesIO | IO[bytes],
path: src/fides/api/service/storage/s3.py
user: login: erosselli
body: thanks for the super detailed explanation!
created_at: 2025-03-25T12:42:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,11 +11,34 @@
 from fides.api.util.aws_util import get_s3_client
 from fides.config import CONFIG
 
+LARGE_FILE_THRESHOLD = 5 * 1024 * 1024  # 5 MB threshold for large files
path: src/fides/api/service/storage/s3.py
user: login: erosselli
body: where does this number come from? would we want to to be configurable, or is that overkill? 
created_at: 2025-03-25T12:43:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,11 +11,34 @@
 from fides.api.util.aws_util import get_s3_client
 from fides.config import CONFIG
 
+LARGE_FILE_THRESHOLD = 5 * 1024 * 1024  # 5 MB threshold for large files
path: src/fides/api/service/storage/s3.py
user: login: JadeCara
body: It honestly might be a little small! I wanted to put something in place so we can iterate on it, but as a starting place. I was passing it into the retrieval function but updated the upload function to take it as an argument as well. That way both functions can take another value as an argument :) 
created_at: 2025-03-25T14:15:40Z
author_association: CONTRIBUTOR

PR comments:
user: login: vercel[bot]
created_at: 2025-03-21T21:59:17Z
body: [vc]: #GJy8IhydjEpd08PSj0UDRiBGkHFsJ8Jv/lmphxB4uYw=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJmaWRlcy1wbHVzLW5pZ2h0bHkiLCJyb290RGlyZWN0b3J5IjoiY2xpZW50cy9hZG1pbi11aSIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiIifSwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2V0aHljYS9maWRlcy1wbHVzLW5pZ2h0bHkvN1Q0OTFTd245VXZYTWd2ZEZFQ3JLazN5aHY0cCIsIm5leHRDb21taXRTdGF0dXMiOiJJR05PUkVEIiwicHJldmlld1VybCI6ImZpZGVzLXBsdXMtbmlnaHRseS1naXQtbGotMzcyLWJlLXVwZGF0ZS1wcml2YWN5LXItZmVhZGJhLWV0aHljYS52ZXJjZWwuYXBwIn0seyJuYW1lIjoiZmlkZXMtcHJpdmFjeS1jZW50ZXIiLCJyb290RGlyZWN0b3J5IjoiY2xpZW50cy9wcml2YWN5LWNlbnRlciIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJmaWRlcy1wcml2YWN5LWNlbnRlci1naXQtbGotMzcyLWJlLXVwZGF0ZS1wcml2YWN5LWI1MGI1ZS1ldGh5Y2EudmVyY2VsLmFwcCJ9LCJpbnNwZWN0b3JVcmwiOiJodHRwczovL3ZlcmNlbC5jb20vZXRoeWNhL2ZpZGVzLXByaXZhY3ktY2VudGVyLzJtd3NmdlNxVmQ4TTJkZlJwa2ltdEtkOFprQ04iLCJwcmV2aWV3VXJsIjoiZmlkZXMtcHJpdmFjeS1jZW50ZXItZ2l0LWxqLTM3Mi1iZS11cGRhdGUtcHJpdmFjeS1iNTBiNWUtZXRoeWNhLnZlcmNlbC5hcHAiLCJuZXh0Q29tbWl0U3RhdHVzIjoiREVQTE9ZRUQifV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **fides-privacy-center** | ✅ Ready ([Inspect](https://vercel.com/ethyca/fides-privacy-center/2mwsfvSqVd8M2dfRpkimtKd8ZkCN)) | [Visit Preview](https://fides-privacy-center-git-lj-372-be-update-privacy-b50b5e-ethyca.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/fides-privacy-center-git-lj-372-be-update-privacy-b50b5e-ethyca.vercel.app?via=pr-comment-feedback-link) | Mar 25, 2025 2:21pm |

<details><summary>1 Skipped Deployment</summary>

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **fides-plus-nightly** | ⬜️ Ignored ([Inspect](https://vercel.com/ethyca/fides-plus-nightly/7T491Swn9UvXMgvdFECrKk3yhv4p)) | [Visit Preview](https://fides-plus-nightly-git-lj-372-be-update-privacy-r-feadba-ethyca.vercel.app) |  | Mar 25, 2025 2:21pm |
</details>



author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-21T22:30:01Z
body: ## [Codecov](https://app.codecov.io/gh/ethyca/fides/pull/5922?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca) Report
Attention: Patch coverage is `89.74359%` with `4 lines` in your changes missing coverage. Please review.
> Project coverage is 87.02%. Comparing base [(`0dc62dc`)](https://app.codecov.io/gh/ethyca/fides/commit/0dc62dc1ab974ac40d2aec0d8e1cdebe08bb859a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca) to head [(`0c31495`)](https://app.codecov.io/gh/ethyca/fides/commit/0c31495b754c01e3bbe632b026730297b291daaf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca).
> Report is 1 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/ethyca/fides/pull/5922?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca) | Patch % | Lines |
|---|---|---|
| [src/fides/api/service/storage/s3.py](https://app.codecov.io/gh/ethyca/fides/pull/5922?src=pr&el=tree&filepath=src%2Ffides%2Fapi%2Fservice%2Fstorage%2Fs3.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca#diff-c3JjL2ZpZGVzL2FwaS9zZXJ2aWNlL3N0b3JhZ2UvczMucHk=) | 88.57% | [4 Missing :warning: ](https://app.codecov.io/gh/ethyca/fides/pull/5922?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca) |

:x: Your patch status has failed because the patch coverage (89.74%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the [target](https://docs.codecov.com/docs/commit-status#target) coverage.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #5922      +/-   ##
==========================================
+ Coverage   86.99%   87.02%   +0.02%     
==========================================
  Files         416      416              
  Lines       25537    25534       -3     
  Branches     2760     2760              
==========================================
+ Hits        22216    22221       +5     
+ Misses       2720     2714       -6     
+ Partials      601      599       -2     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/ethyca/fides/pull/5922?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ethyca).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: cypress[bot]
created_at: 2025-03-25T15:13:24Z
body: 
<h3>
  <a href="https://cloud.cypress.io/projects/bauzeh?utm_source=github&utm_medium=PASSED&utm_campaign=project">fides</a>&nbsp;
  <img src="https://assets.cypress.io/github-pr-comment-icons/chevron.svg" alt="" height="14" width="14" />&nbsp;
  <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?utm_source=github&utm_medium=PASSED&utm_campaign=run%20number">Run #12711</a>
</h3>

<details>
  <summary>
    <strong>Run Properties:</strong>&nbsp;
    <code><img src="https://assets.cypress.io/github-pr-comment-icons/passed.svg" alt="status check passed" height="10" width="10" />&nbsp;Passed&nbsp;<a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?utm_source=github&utm_medium=PASSED&utm_campaign=run%20number">#12711</a></code> &nbsp;&bull;&nbsp;
    <code><img src="https://assets.cypress.io/github-pr-comment-icons/git-commit.svg" alt="git commit" width="10" height="10" /> <a href="https://github.com/ethyca/fides/commit/60fcd78999d14067e4a3c55bee36c3aba090de6e">60fcd78999</a>: LJ-372 Updated Attachments/uploads to handle large files. (#5922)</code>
  </summary><br />

  <table width="744">
    <tbody>
      <tr />
      <tr height="48">
        <td width="224">
          <strong>Project</strong>
        </td>
        <td width="520">
          <code><a href="https://cloud.cypress.io/projects/bauzeh?utm_source=github&utm_medium=PASSED&utm_campaign=project">fides</a></code>
        </td>
      </tr>
      <tr />
      <tr height="48">
        <td width="224">
          <strong>Branch Review</strong>
        </th>
        <td>
          <code><a href="https://cloud.cypress.io/projects/bauzeh/branches/main/review?utm_source=github&utm_medium=PASSED&utm_campaign=review%20changes&changeRequest=5922">main</a></code>
        </td>
      </tr>
      <tr />
      <tr height="48">
        <td><strong>Run status</strong></td>
        <td>
          <code><img src="https://assets.cypress.io/github-pr-comment-icons/passed.svg" alt="status check passed" height="10" width="10" />&nbsp;Passed&nbsp;<a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?utm_source=github&utm_medium=PASSED&utm_campaign=run%20number">#12711</a></code>
        </td>
      </tr>
      <tr />
      <tr height="48">
        <td><strong>Run duration</strong></td>
        <td><code>00m 49s</code></td>
      </tr>
      <tr />
      <tr height="48">
        <td><strong>Commit</strong></td>
        <td>
          <code><img src="https://assets.cypress.io/github-pr-comment-icons/git-commit.svg" alt="git commit" width="10" height="10" /> <a href="https://github.com/ethyca/fides/commit/60fcd78999d14067e4a3c55bee36c3aba090de6e">60fcd78999</a>: LJ-372 Updated Attachments/uploads to handle large files. (#5922)</code>
        </td>
      </tr>
      <tr />
      <tr height="48">
        <td><strong>Committer</strong></td>
        <td><code>JadeWibbels</code></td>
      </tr>
    </tbody>
    <tfoot>
      <tr />
      <tr height="48">
        <td colspan="2">
          <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/properties?utm_source=github&utm_medium=PASSED&utm_campaign=properties">View all properties for this run ↗︎</a>
        </td>
      </tr>
    </tfoot>
  </table>
  <hr />
</details>

<table width="744">
  <thead>
    <tr />
    <tr height="48">
      <th colspan="2" align="left" width="744">Test results</th>
    </tr>
  </thead>
  <tbody>
    <tr height="48">
      <td width="224">
        <img src="https://assets.cypress.io/github-pr-comment-icons/failed.svg" alt="Tests that failed" width="10" height="10" />&nbsp;
        <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?reviewViewBy=FAILED&utm_source=github&utm_medium=PASSED&utm_campaign=failed%20tests">
          <strong>Failures</strong>
        </a>
      </td>
      <td>
        <code><strong>0</strong></code>
      </td>
    </tr>
    <tr />
    <tr height="48">
      <td width="224">
        <img src="https://assets.cypress.io/github-pr-comment-icons/flaky-v3.svg" alt="Tests that were flaky" width="10" height="10" />&nbsp;
        <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?reviewViewBy=FLAKY&utm_source=github&utm_medium=PASSED&utm_campaign=flaky%20tests">
          <strong>Flaky</strong>
        </a>
      </td>
      <td>
        <code><strong>0</strong></code>
      </td>
    </tr>
    <tr />
    <tr height="48">
      <td width="224">
        <img src="https://assets.cypress.io/github-pr-comment-icons/pending.svg" alt="Tests that did not run due to a developer annotating a test with .skip" width="10" height="10" />&nbsp;
        <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/test-results?utm_source=github&utm_medium=PASSED&utm_campaign=pending%20tests&statuses=%5B%7B%22value%22%3A%22PENDING%22%2C%22label%22%3A%22PENDING%22%7D%5D">
          <strong>Pending</strong>
        </a>
      </td>
      <td>
        <code><strong>0</strong></code>
      </td>
    </tr>
    <tr />
    <tr height="48">
      <td width="224">
        <img src="https://assets.cypress.io/github-pr-comment-icons/skipped-v3.svg" alt="Tests that did not run due to a failure in a mocha hook" width="10" height="10" />&nbsp;
        <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/test-results?utm_source=github&utm_medium=PASSED&utm_campaign=skipped%20tests&statuses=%5B%7B%22value%22%3A%22SKIPPED%22%2C%22label%22%3A%22SKIPPED%22%7D%5D">
          <strong>Skipped</strong>
        </a>
      </td>
      <td>
        <code><strong>0</strong></code>
      </td>
    </tr>
    <tr />
    <tr height="48">
      <td width="224">
        <img src="https://assets.cypress.io/github-pr-comment-icons/passed.svg" alt="Tests that passed" width="10" height="10" />&nbsp;
        <a href="https://cloud.cypress.io/projects/bauzeh/runs/12711/overview?utm_source=github&utm_medium=PASSED&utm_campaign=passed%20tests">
          <strong>Passing</strong>
        </a>
      </td>
      <td>
        <code><strong>5</strong></code>
      </td>
    </tr><tfoot>
    <tr height="48"><td colspan="2"><a href="https://cloud.cypress.io/projects/bauzeh/branches/main/review?utm_source=github&utm_medium=PASSED&utm_campaign=review%20changes&changeRequest=5922">View all changes introduced in this branch ↗︎</a></td></tr></tfoot>
  </tbody>
</table>






author_association: NONE