Repository: pypi/warehouse

Issue:
Periodically get a cgi.FieldStorage instead of a string on uploads
For some reason, and I'm not sure why it happens, sometimes the POST dictionary contains cgi.FieldStorage classes instead of strings (I think). We should figure out what cases this happens in, and guard against them however we can.

---

https://sentry.io/python-software-foundation/warehouse-production/issues/187224559/

```
ProgrammingError: can't adapt type 'cgi_FieldStorage'
  File "sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)

ProgrammingError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.ProgrammingError) can't adapt type 'cgi_FieldStorage' [SQL: 'INSERT INTO releases (name, version, author, author_email, maintainer, maintainer_email, home_page, license, summary, description, keywords, platform, download_url, _pypi_ordering, _pypi_hidden, cheesecake_installability_id, cheesecake_documentation_id, cheesecake_code_kwalitee_id, requires_python, description_from_readme) VALUES (%(name)s, %(version)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(home_page)s, %(license)s, %(summary)s, %(description)s, %(keywords)s, %(platform)s, %(download_url)s, %(_pypi_ordering)s, %(_pypi_hidden)s, %(cheesecake_installability_id)s, %(cheesecake_documentation_id)s, %(cheesecake_code_kwalitee_id)s, %(requires_python)s, %(description_from_readme)s)'] [parameters: {'name': 'mdsplus', 'version': '7.7.6', 'author': 'Tom Fredian,Josh Stillerman,Gabriele Manduchi', 'author_email': 'twf@www.mdsplus.org', 'maintainer': '', 'maintainer_email': '', 'home_page': 'http://www.mdsplus.org/', 'license': '', 'summary': 'MDSplus Python Objects', 'description': '\n      This module provides all of the functionality of MDSplus TDI natively in python.\n      All of the MDSplus data types such as signal are represented as python classes.\n      ', 'keywords': FieldStorage('keywords', 'physics'), 'platform': '', 'download_url': 'http://www.mdsplus.org/mdsplus_download/python', '_pypi_ordering': None, '_pypi_hidden': False, 'cheesecake_installability_id': None, 'cheesecake_documentation_id': None, 'cheesecake_code_kwalitee_id': None, 'requires_python': '', 'description_from_readme': None}]
(51 additional frame(s) were not displayed)
...
  File "warehouse/raven.py", line 41, in raven_tween
    return handler(request)
  File "warehouse/cache/http.py", line 69, in conditional_http_tween
    response = handler(request)
  File "warehouse/cache/http.py", line 33, in wrapped
    return view(context, request)
  File "warehouse/csrf.py", line 38, in wrapped
    return view(context, request)
  File "warehouse/forklift/legacy.py", line 791, in file_upload
    .filter(Release.project == project)

ProgrammingError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.ProgrammingError) can't adapt type 'cgi_FieldStorage' [SQL: 'INSERT INTO releases (name, version, author, author_email, maintainer, maintainer_email, home_page, license, summary, description, keywords, platform, download_url, _pypi_ordering, _pypi_hidden, cheesecake_installability_id, cheesecake_documentation_id, cheesecake_code_kwalitee_id, requires_python, description_from_readme) VALUES (%(name)s, %(version)s, %(author)s, %(author_email)s, %(maintainer)s, %(maintainer_email)s, %(home_page)s, %(license)s, %(summary)s, %(description)s, %(keywords)s, %(platform)s, %(download_url)s, %(_pypi_ordering)s, %(_pypi_hidden)s, %(cheesecake_installability_id)s, %(cheesecake_documentation_id)s, %(cheesecake_code_kwalitee_id)s, %(requires_python)s, %(description_from_readme)s)'] [parameters: {'name': 'mdsplus', 'version': '7.7.6', 'author': 'Tom Fredian,Josh Stillerman,Gabriele Manduchi', 'author_email': 'twf@www.mdsplus.org', 'maintainer': '', 'maintainer_email': '', 'home_page': 'http://www.mdsplus.org/', 'license': '', 'summary': 'MDSplus Python Objects', 'description': '\n      This module provides all of the functionality of MDSplus TDI natively in python.\n      All of the MDSplus data types such as signal are represented as python classes.\n      ', 'keywords': FieldStorage('keywords', 'physics'), 'platform': '', 'download_url': 'http://www.mdsplus.org/mdsplus_download/python', '_pypi_ordering': None, '_pypi_hidden': False, 'cheesecake_installability_id': None, 'cheesecake_documentation_id': None, 'cheesecake_code_kwalitee_id': None, 'requires_python': '', 'description_from_readme': None}]
```


Discussion:
Quick attempt at debugging this, according to the Sentry traceback the error happens when trying to insert a new `Release` including `'keywords': FieldStorage('keywords', 'physics')` as a parameter for the query.

I believe the sequence would be:
1. a POST request is made
2. a WSGI environment is created and parsed by Webob
3. a Webob `Request` object contains a `Multidict` in its POST attribute from the parsed body of the request which somehow causes keywords to be a `FieldStorage` instance
4. The view [constructs a MetadataForm from request.POST](https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L630) and successfully validates the form including the FieldStorage as a StringField.
5. The view then [constructs and adds a new `Release`](https://github.com/pypa/warehouse/blob/master/warehouse/forklift/legacy.py#L738) which causes the error.

Would it be possible to check the body of the request?
Seems like we could do something similar to #2655 here.

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -675,21 +675,24 @@ def file_upload(request):
     if request.POST.get("protocol_version", "1") != "1":
         raise _exc_with_message(HTTPBadRequest, "Unknown protocol version.")
 
+    # Check if any fields were supplied as a tuple and have become a
+    # FieldStorage. The 'content' field _should_ be a FieldStorage, however.
+    # ref: https://github.com/pypa/warehouse/issues/2185
+    # ref: https://github.com/pypa/warehouse/issues/2491
+    for field in set(request.POST) - {'content'}:
+        values = request.POST.getall(field)
+        if any(isinstance(value, FieldStorage) for value in values):
+            raise _exc_with_message(
+                HTTPBadRequest,
+                f"{field}: Should not be a tuple.",
+            )
+
     # Look up all of the valid classifiers
     all_classifiers = request.db.query(Classifier).all()
 
     # Validate and process the incoming metadata.
     form = MetadataForm(request.POST)
 
-    # Check if the classifiers were supplied as a tuple
-    # ref: https://github.com/pypa/warehouse/issues/2185
-    classifiers = request.POST.getall('classifiers')
-    if any(isinstance(classifier, FieldStorage) for classifier in classifiers):
-        raise _exc_with_message(
-            HTTPBadRequest,
-            "classifiers: Must be a list, not tuple.",
-        )
-
     form.classifiers.choices = [
         (c.classifier, c.classifier) for c in all_classifiers
     ]


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -839,7 +839,19 @@ def test_fails_invalid_version(self, pyramid_config, pyramid_request,
                     "filetype": "sdist",
                     "classifiers": FieldStorage(),
                 },
-                "classifiers: Must be a list, not tuple.",
+                "classifiers: Should not be a tuple.",
+            ),
+
+            # keywords are a FieldStorage
+            (
+                {
+                    "metadata_version": "1.2",
+                    "name": "example",
+                    "version": "1.0",
+                    "filetype": "sdist",
+                    "keywords": FieldStorage(),
+                },
+                "keywords: Should not be a tuple.",
             ),
         ],
     )