Repository: pypi/warehouse

Issue:
Package upload fails with twine and setuptools
Attempting to add a new release with twine or setuptools results in a unhelpful `Received "500: Internal Server Error"`. 

**OS:** Ubuntu 16.04.2 LTS
**Twine:** `twine version 1.11.0 (pkginfo: 1.4.2, requests: 2.18.4, setuptools: 39.0.1,
requests-toolbelt: 0.8.0, tqdm: 4.20.0)`

I have tried multiple repository URLs and configurations. The following is the last configuration and logs attempted: 

**cat ~/.pypirc**
```
[distutils]
index-servers =
  pypi

[pypi]
username: 0xgiddi
password: <redacted>
```
**twine upload dist/***
```Uploading distributions to https://upload.pypi.org/legacy/
Uploading pymbr-1.0.3-py2-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.7k/13.7k [00:01<00:00, 8.43kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 1 of 5
Uploading pymbr-1.0.3-py2-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.7k/13.7k [00:01<00:00, 11.5kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 2 of 5
Uploading pymbr-1.0.3-py2-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.7k/13.7k [00:00<00:00, 16.2kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 3 of 5
Uploading pymbr-1.0.3-py2-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.7k/13.7k [00:00<00:00, 16.9kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 4 of 5
Uploading pymbr-1.0.3-py2-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.7k/13.7k [00:00<00:00, 18.2kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 5 of 5
HTTPError: 500 Server Error: Internal Server Error for url: https://upload.pypi.org/legacy/
```

For fun, here is the log of a classic  `python setup.py sdist upload`
```
running sdist
running egg_info
writing pymbr.egg-info/PKG-INFO
writing top-level names to pymbr.egg-info/top_level.txt
writing dependency_links to pymbr.egg-info/dependency_links.txt
reading manifest file 'pymbr.egg-info/SOURCES.txt'
writing manifest file 'pymbr.egg-info/SOURCES.txt'
running check
creating pymbr-1.0.3
creating pymbr-1.0.3/pymbr
creating pymbr-1.0.3/pymbr.egg-info
copying files to pymbr-1.0.3...
copying README.rst -> pymbr-1.0.3
copying setup.py -> pymbr-1.0.3
copying pymbr/__init__.py -> pymbr-1.0.3/pymbr
copying pymbr/bootcode.py -> pymbr-1.0.3/pymbr
copying pymbr/bootrecord.py -> pymbr-1.0.3/pymbr
copying pymbr/filesystem.py -> pymbr-1.0.3/pymbr
copying pymbr/partition.py -> pymbr-1.0.3/pymbr
copying pymbr.egg-info/PKG-INFO -> pymbr-1.0.3/pymbr.egg-info
copying pymbr.egg-info/SOURCES.txt -> pymbr-1.0.3/pymbr.egg-info
copying pymbr.egg-info/dependency_links.txt -> pymbr-1.0.3/pymbr.egg-info
copying pymbr.egg-info/top_level.txt -> pymbr-1.0.3/pymbr.egg-info
Writing pymbr-1.0.3/setup.cfg
Creating tar archive
removing 'pymbr-1.0.3' (and everything under it)
running upload
Submitting dist/pymbr-1.0.3.tar.gz to https://upload.pypi.org/legacy/
Upload failed (500): Internal Server Error
error: Upload failed (500): Internal Server Error
```
Package source: https://github.com/0xGiddi/pymbr


Discussion:
@0xGiddi Could you upload, link, or email me (di@python.org) the `pymbr-1.0.3.tar.gz` file you're trying to upload?
@di   Here you go: 
[pymbr-1.0.3.tar.gz](https://github.com/pypa/warehouse/files/1873528/pymbr-1.0.3.tar.gz)

Interesting, this seems to be caused by your PyPI username starting with `0x`. Investigating...

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -718,12 +718,20 @@ def file_upload(request):
             "Invalid or non-existent authentication information.",
         )
 
-    # distutils "helpfully" substitutes unknown, but "required" values with the
-    # string "UNKNOWN". This is basically never what anyone actually wants so
-    # we'll just go ahead and delete anything whose value is UNKNOWN.
+    # Do some cleanup of the various form fields
     for key in list(request.POST):
-        if str(request.POST.get(key)).strip() == "UNKNOWN":
-            del request.POST[key]
+        value = request.POST.get(key)
+        if isinstance(value, str):
+            # distutils "helpfully" substitutes unknown, but "required" values
+            # with the string "UNKNOWN". This is basically never what anyone
+            # actually wants so we'll just go ahead and delete anything whose
+            # value is UNKNOWN.
+            if value.strip() == "UNKNOWN":
+                del request.POST[key]
+
+            # Escape NUL characters, which psycopg doesn't like
+            if '\x00' in value:
+                request.POST[key] = value.replace('\x00', '\\x00')
 
     # We require protocol_version 1, it's the only supported version however
     # passing a different version should raise an error.


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -1096,6 +1096,22 @@ def test_upload_cleans_unknown_values(
 
         assert "name" not in db_request.POST
 
+    def test_upload_escapes_nul_characters(self, pyramid_config, db_request):
+        pyramid_config.testing_securitypolicy(userid=1)
+        db_request.POST = MultiDict({
+            "metadata_version": "1.2",
+            "name": "testing",
+            "summary": "I want to go to the \x00",
+            "version": "1.0",
+            "filetype": "sdist",
+            "md5_digest": "a fake md5 digest",
+        })
+
+        with pytest.raises(HTTPBadRequest):
+            legacy.file_upload(db_request)
+
+        assert "\x00" not in db_request.POST["summary"]
+
     @pytest.mark.parametrize(
         ("has_signature", "digests"),
         [