Repository: kiwitcms/Kiwi (language: python)

Issue:
Unable to Upload Large Files despite Increasing FILE_UPLOAD_MAX_SIZE
### Description:
We encountered an issue while attempting to increase the maximum file upload size (**FILE_UPLOAD_MAX_SIZE**) for larger files in our project. Despite adjusting the parameter to a higher value, we were unable to successfully upload files larger than 2MB. This is perplexing since the default **FILE_UPLOAD_MAX_SIZE** is set to 5MB.

### Steps to Reproduce:

1. Attempt to upload a file larger than 2MB.
2. Observe that the file upload fails.



### Expected Behavior:
The file upload should succeed for files larger than 2MB, as the **FILE_UPLOAD_MAX_SIZE** parameter was increased.

### Actual Behavior:
The file upload fails, indicating that the increased **FILE_UPLOAD_MAX_SIZE** value is not being recognized.

### Troubleshooting and Solution:

1. We referred to the [Django Documentation](https://docs.djangoproject.com/en/2.1/ref/settings/#data-upload-max-memory-size) to investigate the issue further.
3. After reviewing the documentation, we discovered that the correct variable to configure for managing file upload size is **DATA_UPLOAD_MAX_MEMORY_SIZE**, not **FILE_UPLOAD_MAX_SIZE**.
4. We replaced **FILE_UPLOAD_MAX_SIZE** with **DATA_UPLOAD_MAX_MEMORY_SIZE** in the configuration.
5. Following this adjustment, we were able to successfully upload larger files without any issues.

### Additional Feedback:
We found it frustrating that it was not immediately apparent where to change the settings for the Nginx configuration within the Docker container. It would be helpful if this information were documented in a more accessible location, such as the **common.py** file, rather than solely in the **/Kiwi/etc/nginx.conf** file.

### Screenshot:
![grafik](https://github.com/kiwitcms/Kiwi/assets/50086322/69d70fb2-e13d-47db-affa-da663f35de32)

Thank you for your attention to this matter. We hope this report assists in improving the project's documentation and helps resolve the issue with file uploads.


Discussion / hints:
> ### Description:
> 
> We encountered an issue while attempting to increase the maximum file upload size (**FILE_UPLOAD_MAX_SIZE**) for larger files in our project. Despite adjusting the parameter to a higher value, we were unable to successfully upload files larger than 2MB. This is perplexing since the default **FILE_UPLOAD_MAX_SIZE** is set to 5MB.
> ### Steps to Reproduce:
> 
>     1. Attempt to upload a file larger than 2MB.
> 
>     2. Observe that the file upload fails.
> 

Can you specify exactly what failure do you see? 

Are you seeing an error message `File exceeds maximum size of 5.0 MB` or are you seeing `HttpBadRequest (status_code=400)` response ? 


FTR I am not able to reproduce with v12.7, see https://public.tenant.kiwitcms.org/plan/8530/testing-maximum-file-upload-size. I was able to upload two files which are exactly 5MiB large:

```
-rw-r--r--. 1 atodorov atodorov 5242880 Dec  4 22:40 large-file.txt
-rw-r--r--. 1 atodorov atodorov 5242880 Dec  4 22:28 large-file.zero
```

Trying with a file which is 1 byte larger:
```
-rw-r--r--. 1 atodorov atodorov 5242881 Dec  4 22:44 large-file.log
```
results in the `File exceeds maximum size of 5.0 MB` error message as expected.





> 
> ### Expected Behavior:
> 
> The file upload should succeed for files larger than 2MB, as the **FILE_UPLOAD_MAX_SIZE** parameter was increased.
> ### Actual Behavior:
> 
> The file upload fails, indicating that the increased **FILE_UPLOAD_MAX_SIZE** value is not being recognized.
> ### Troubleshooting and Solution:
> 
>     1. We referred to the [Django Documentation](https://docs.djangoproject.com/en/2.1/ref/settings/#data-upload-max-memory-size) to investigate the issue further.
> 
>     2. After reviewing the documentation, we discovered that the correct variable to configure for managing file upload size is **DATA_UPLOAD_MAX_MEMORY_SIZE**, not **FILE_UPLOAD_MAX_SIZE**.
> 

Indeed Django's configuration variable is **DATA_UPLOAD_MAX_MEMORY_SIZE**. **FILE_UPLOAD_MAX_SIZE** is the configuration for the django-attachments library which handles uploaded files.



> * We replaced **FILE_UPLOAD_MAX_SIZE** with **DATA_UPLOAD_MAX_MEMORY_SIZE** in the configuration.
> 
> * Following this adjustment, we were able to successfully upload larger files without any issues.

Django's own [source code](https://github.com/django/django/blob/main/django/core/exceptions.py#L79-L85) says:

```
class RequestDataTooBig(SuspiciousOperation):
    """
    The size of the request (excluding any file uploads) exceeded
    settings.DATA_UPLOAD_MAX_MEMORY_SIZE.
    """
```


So this change doesn't sound like it has any effect. Maybe it was something else which made it possible for you to upload larger files but it's highly unlikely it was adjusting the `DATA_UPLOAD_MAX_MEMORY_SIZE` setting.


Fix patch:
diff --git a/tcms/settings/common.py b/tcms/settings/common.py
index 558de77588..46e5c66037 100644
--- a/tcms/settings/common.py
+++ b/tcms/settings/common.py
@@ -131,6 +131,9 @@
 # Maximum upload file size, default set to 5MB.
 FILE_UPLOAD_MAX_SIZE = 5242880
 
+# DO NOT MODIFY! Maximum size of POST request
+# x1.5 to be able to upload base64 encoded attachments via RPC
+DATA_UPLOAD_MAX_MEMORY_SIZE = int(FILE_UPLOAD_MAX_SIZE * 1.5)
 
 # Controls if django-attachments deletes files from disk
 DELETE_ATTACHMENTS_FROM_DISK = True


Test patch:
diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml
index ea302a1561..e50b23c3b9 100644
--- a/.github/workflows/testing.yml
+++ b/.github/workflows/testing.yml
@@ -287,3 +287,10 @@ jobs:
         with:
           name: wrk-logs
           path: ./wrk-logs-*/
+
+      - name: Upload Docker logs
+        uses: actions/upload-artifact@v3
+        if: always()
+        with:
+          name: docker-logs
+          path: "*docker.log"
diff --git a/tests/test_http.sh b/tests/test_http.sh
index 005f00b924..7238659574 100755
--- a/tests/test_http.sh
+++ b/tests/test_http.sh
@@ -71,7 +71,7 @@ username = testadmin
 password = password
 _EOF_
 
-        rlRun -t -c "./tests/test_utf8_uploads.py -v"
+        rlRun -t -c "PYTHONPATH=../Kiwi ./tests/test_utf8_uploads.py -v"
     rlPhaseEnd
 
     rlPhaseStartTest "Should send ETag header"
@@ -148,6 +148,7 @@ _EOF_
     rlPhaseEnd
 
     rlPhaseStartCleanup
+        rlRun -t -c "docker-compose logs --no-color > test_http_docker.log"
         rlRun -t -c "docker-compose down"
         rm -f /tmp/testcookies.txt
         rm -f /tmp/testdata.txt
diff --git a/tests/test_utf8_uploads.py b/tests/test_utf8_uploads.py
index 35f7c35444..7503fd936e 100755
--- a/tests/test_utf8_uploads.py
+++ b/tests/test_utf8_uploads.py
@@ -1,17 +1,22 @@
 #!/usr/bin/env python3
 
 #
-# Copyright (c) 2022 Kiwi TCMS project. All rights reserved.
+# Copyright (c) 2022-2023 Kiwi TCMS project. All rights reserved.
 # Author: Alexander Todorov <info@kiwitcms.org>
 #
 
+import base64
 import ssl
+import tempfile
 import unittest
 from unittest.mock import patch
+from xmlrpc.client import Fault as XmlRPCFault
 
 import requests
 from tcms_api import TCMS
 
+from tcms.settings.common import FILE_UPLOAD_MAX_SIZE
+
 try:
     _create_unverified_https_context = ssl._create_unverified_context
 except AttributeError:
@@ -29,14 +34,37 @@ def get(self, url, **kwargs):
 
 
 class Utf8UploadTestCase(unittest.TestCase):
+    @staticmethod
+    def create_file(size, name):
+        """
+        Create a temporary file with specific size.
+        """
+        (_handle, file_name) = tempfile.mkstemp(prefix=f"kiwitcms-test-{name}.")
+        with open(file_name, "wb") as file:
+            for _ in range(size):
+                file.write(b"T")
+
+        return file_name
+
+    def b64_file(self, size, name):
+        """
+        Create a temporary file and return it's contents in Base64,
+        which can be passed directly to rpc.User.add_attachment().
+        """
+        file_name = self.create_file(size, name)
+        with open(file_name, "rb") as file_handle:
+            file_content = file_handle.read()
+            file_content = base64.b64encode(file_content).decode()
+            return (file_name, file_content)
+
     @classmethod
     def setUpClass(cls):
         with patch("requests.sessions.Session") as session:
             session.return_value = DoNotVerifySSLSession()
             cls.rpc = TCMS().exec
 
-    def upload_and_assert(self, filename):
-        result = self.rpc.User.add_attachment(filename, "a2l3aXRjbXM=")
+    def upload_and_assert(self, filename, contents="a2l3aXRjbXM="):
+        result = self.rpc.User.add_attachment(filename, contents)
         self.assertIn("url", result)
         self.assertIn("filename", result)
 
@@ -46,6 +74,15 @@ def test_with_filename_in_french(self):
     def test_with_filename_in_bulgarian(self):
         self.upload_and_assert("Screenshot_2021-12-07 Плащане.png")
 
+    def test_uploading_file_with_maximum_allowed_size_should_work(self):
+        file_name, file_content = self.b64_file(FILE_UPLOAD_MAX_SIZE, "5mb-file")
+        self.upload_and_assert(file_name, file_content)
+
+    def test_uploading_file_with_size_greater_than_maximum_should_fail(self):
+        file_name, file_content = self.b64_file(FILE_UPLOAD_MAX_SIZE + 1, "file-gt-5mb")
+        with self.assertRaisesRegex(XmlRPCFault, "File exceeds maximum size"):
+            self.rpc.User.add_attachment(file_name, file_content)
+
 
 if __name__ == "__main__":
     unittest.main()