Repository: CTFd/CTFd

Issue:
Upload to S3 Failing
- CTFd Version/Commit: 3.6.1
- Operating System: Linux (Docker container)
- Web Browser and Version: Chrome

**What happened?**

Upgrading CTFd resulting in S3 file uploads beginning to return 400 (bad request) status codes. I see one of the fixes for 3.6.1 was for S3, so perhaps a new bug was introduced.

Here are some additional facts which may be helpful:

  - The files are successfully making there way into S3, despite the error
  - The timezone I have configured for my server is CST

I can also confirm that my deployment had working file upload before upgrade to version 3.6.1 (file upload was working for 3.6.0).

**What did you expect to happen?**

File upload to continue working.

**How to reproduce your issue**

Deploy CTFd free version using version 3.6.1 with S3 file upload configured.

**Any associated stack traces or error logs**

The browser request returns error (400 status code):

```
{
    "success": false,
    "errors": {
        "location": [
            "I/O operation on closed file."
        ]
    }
}
```

The backend error is:

```
[ERROR] Error handling request
Traceback (most recent call last):
File "/opt/venv/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 113, in handle_request
resp.write_file(respiter)
File "/opt/venv/lib/python3.9/site-packages/gunicorn/http/wsgi.py", line 385, in write_file
if not self.sendfile(respiter):
File "/opt/venv/lib/python3.9/site-packages/gunicorn/http/wsgi.py", line 375, in sendfile
self.sock.sendfile(respiter.filelike, count=nbytes)
File "/opt/venv/lib/python3.9/site-packages/gevent/_socket3.py", line 486, in sendfile
return self._sendfile_use_send(file, offset, count)
File "/opt/venv/lib/python3.9/site-packages/gevent/_socket3.py", line 416, in _sendfile_use_send
self._check_sendfile_params(file, offset, count)
File "/opt/venv/lib/python3.9/site-packages/gevent/_socket3.py", line 461, in _check_sendfile_params
raise ValueError(
ValueError: count must be a positive integer (got 0)
```


Discussion:
This could be related to https://github.com/CTFd/CTFd/pull/2432 but I am unsure since I have not seen S3 uploads breaking. Can you perhaps set your timezone to UTC to see if that allows uploads to work? That will help confirm the source of the issue. 


@ColdHeat I did some digging. It looks like the boto3 `upload_fileobj` function silently closes the file object you give it after upload (see https://github.com/boto/boto3/issues/929).

This unfortunate undocumented behavior, along with the addition of https://github.com/CTFd/CTFd/blob/3536c4c423aab5b0b9c164af73f59fe29a4d7dcf/CTFd/utils/uploads/__init__.py#L50 in the last release it causing the issue, since this line occurs after the called to `upload_fileobj`. This explains why the returned error is `"I/O operation on closed file."`.

I guess this would only be a problem if using S3 to store files for CTFd.
Thank you for digging into this and figuring out the issue. This is an issue in a feature for 3.7 which explains why I haven't seen it before. 

You're welcome to PR a fix but I will tackle this as part of 3.7 as well. 
Likely the simplest fix is to calculate the SHA before uploading because we don't close the file and re-seek to the beginning as well. 
@ColdHeat I can confirm that moving the hash calculated to before the file upload fixes the issue. I can submit a PR. Any idea when this might be pushed to a new hotfix version (maybe 3.6.2?)? This issue is currently causing me some trouble in preparing for an upcoming event.

Resolving patch:
diff --git a/CTFd/utils/uploads/__init__.py b/CTFd/utils/uploads/__init__.py
--- a/CTFd/utils/uploads/__init__.py
+++ b/CTFd/utils/uploads/__init__.py
@@ -44,11 +44,12 @@ def upload_file(*args, **kwargs):
         model = PageFiles
         model_args["page_id"] = page_id
 
+    # Hash is calculated before upload since S3 file upload closes file object
+    sha1sum = hash_file(fp=file_obj)
+
     uploader = get_uploader()
     location = uploader.upload(file_obj=file_obj, filename=filename, path=parent)
 
-    sha1sum = hash_file(fp=file_obj)
-
     model_args["location"] = location
     model_args["sha1sum"] = sha1sum
 


Test patch: