Repository: pypi/warehouse

Issue:
Validate Blake2 Uniqueness prior to saving
For some reason sometimes people are uploading files that already exist with different filenames. I'm not sure if there is a valid use case for this, but if there is we need to either remove the ``UNIQUE`` constraint on the blake2 hash or if there isn't we should validate this before we attempt to save the row.

---

https://sentry.io/python-software-foundation/warehouse-production/issues/147880375/

```
IntegrityError: duplicate key value violates unique constraint "release_files_blake2_256_digest_key"
DETAIL:  Key (blake2_256_digest)=(1d31e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c) already exists.

  File "sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)

IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "release_files_blake2_256_digest_key"
DETAIL:  Key (blake2_256_digest)=(1d31e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c) already exists.
 [SQL: 'INSERT INTO release_files (name, version, python_version, requires_python, packagetype, comment_text, filename, path, size, has_signature, md5_digest, sha256_digest, blake2_256_digest) VALUES (%(name)s, %(version)s, %(python_version)s, %(requires_python)s, %(packagetype)s, %(comment_text)s, %(filename)s, %(path)s, %(size)s, %(has_signature)s, %(md5_digest)s, %(sha256_digest)s, %(blake2_256_digest)s) RETURNING release_files.id'] [parameters: {'name': 'treelite', 'version': '0.1a1', 'python_version': 'cp27', 'requires_python': None, 'packagetype': 'bdist_wheel', 'comment_text': '', 'filename': 'treelite-0.1a3-cp27-cp27m-macosx_10_6_intel.whl', 'path': '1d/31/e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c/treelite-0.1a3-cp27-cp27m-macosx_10_6_intel.whl', 'size': 1574670, 'has_signature': False, 'md5_digest': '32229792ce7968ecd29f9b099cb69644', 'sha256_digest': 'eea762a38b556c0a1f76640be5fcea73eb897d67dc5006dbe590162fbdc5bdea', 'blake2_256_digest': '1d31e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c'}]
(44 additional frame(s) were not displayed)
...
  File "warehouse/config.py", line 85, in require_https_tween
    return handler(request)
  File "warehouse/static.py", line 78, in whitenoise_tween
    return handler(request)
  File "warehouse/utils/compression.py", line 93, in compression_tween
    response = handler(request)
  File "warehouse/raven.py", line 41, in raven_tween
    return handler(request)
  File "warehouse/cache/http.py", line 69, in conditional_http_tween
    response = handler(request)

IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "release_files_blake2_256_digest_key"
DETAIL:  Key (blake2_256_digest)=(1d31e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c) already exists.
 [SQL: 'INSERT INTO release_files (name, version, python_version, requires_python, packagetype, comment_text, filename, path, size, has_signature, md5_digest, sha256_digest, blake2_256_digest) VALUES (%(name)s, %(version)s, %(python_version)s, %(requires_python)s, %(packagetype)s, %(comment_text)s, %(filename)s, %(path)s, %(size)s, %(has_signature)s, %(md5_digest)s, %(sha256_digest)s, %(blake2_256_digest)s) RETURNING release_files.id'] [parameters: {'name': 'treelite', 'version': '0.1a1', 'python_version': 'cp27', 'requires_python': None, 'packagetype': 'bdist_wheel', 'comment_text': '', 'filename': 'treelite-0.1a3-cp27-cp27m-macosx_10_6_intel.whl', 'path': '1d/31/e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c/treelite-0.1a3-cp27-cp27m-macosx_10_6_intel.whl', 'size': 1574670, 'has_signature': False, 'md5_digest': '32229792ce7968ecd29f9b099cb69644', 'sha256_digest': 'eea762a38b556c0a1f76640be5fcea73eb897d67dc5006dbe590162fbdc5bdea', 'blake2_256_digest': '1d31e5fd783d7ce96471b6d66ba3dd59e3e0722811ecbf5c19173a5194be514c'}]
```
Check if another file has been uploaded with the same blake2 digest
This tries to resolve  #2490

I'm not sure if the uniqueness of the blake2 digest should be validated on file uploading or before, or if hijacking on the same "File already exists" message is helpful enough.

Also, I'm getting a weird error on my test, making it fail. Something about the request object missing a 'user' attribute.

Thanks for your feedback y'all.
🐍 


Discussion:

The error is because your test is getting to this line:

```python
            request.db.add(
                JournalEntry(
                    name=release.project.name,
                    version=release.version,
                    action="add {python_version} file {filename}".format(
                        python_version=file_.python_version,
                        filename=file_.filename,
                    ),
>                   submitted_by=request.user,
                    submitted_from=request.remote_addr,
                ),
            )
```

Which means it's trying to add the file being uploaded, so it seems like either something is wrong with your test, or with the new logic you added.
Hey @alanbato, need any help or more direction here?
I think @alanbato [is having computer problems](https://twitter.com/alanbato/status/972266164855410688).
Yes I am 😢
If anyone wants to continue and build on this PR, please do so. I'll ping this thread when my computer gets better 😓 

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -660,7 +660,9 @@ def _is_valid_dist_file(filename, filetype):
 
 def _is_duplicate_file(db_session, filename, hashes):
     """
-    Check to see if file already exists, and if it's content matches
+    Check to see if file already exists, and if it's content matches.
+    A file is considered to exist if its filename *or* blake2 digest are
+    present in a file row in the database.
 
     Returns:
     - True: This file is a duplicate and all further processing should halt.
@@ -670,14 +672,19 @@ def _is_duplicate_file(db_session, filename, hashes):
 
     file_ = (
         db_session.query(File)
-                  .filter(File.filename == filename)
+                  .filter(
+                        (File.filename == filename) |
+                        (File.blake2_256_digest == hashes["blake2_256"]))
                   .first()
     )
 
     if file_ is not None:
-        return (file_.sha256_digest == hashes["sha256"] and
-                file_.md5_digest == hashes["md5"] and
-                file_.blake2_256_digest == hashes["blake2_256"])
+        return (
+            file_.filename == filename and
+            file_.sha256_digest == hashes["sha256"] and
+            file_.md5_digest == hashes["md5"] and
+            file_.blake2_256_digest == hashes["blake2_256"]
+        )
 
     return None
 
@@ -1075,7 +1082,7 @@ def file_upload(request):
             return Response()
         elif is_duplicate is not None:
             raise _exc_with_message(
-                HTTPBadRequest, "File already exists. "
+                HTTPBadRequest, "The filename or contents already exist. "
                                 "See " +
                                 request.route_url(
                                     'help', _anchor='file-name-reuse'


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -639,10 +639,52 @@ def test_is_duplicate_none(self, pyramid_config, db_request):
             ),
         )
 
+        hashes["blake2_256"] = "another blake2 digest"
+
         assert legacy._is_duplicate_file(
             db_request.db, requested_file_name, hashes
         ) is None
 
+    def test_is_duplicate_false_same_blake2(self, pyramid_config, db_request):
+        pyramid_config.testing_securitypolicy(userid=1)
+
+        user = UserFactory.create()
+        EmailFactory.create(user=user)
+        project = ProjectFactory.create()
+        release = ReleaseFactory.create(project=project, version="1.0")
+        RoleFactory.create(user=user, project=project)
+
+        filename = "{}-{}.tar.gz".format(project.name, release.version)
+        requested_file_name = "{}-{}-1.tar.gz".format(project.name,
+                                                      release.version)
+        file_content = io.BytesIO(b"A fake file.")
+        file_value = file_content.getvalue()
+
+        hashes = {
+            "sha256": hashlib.sha256(file_value).hexdigest(),
+            "md5": hashlib.md5(file_value).hexdigest(),
+            "blake2_256": hashlib.blake2b(
+                file_value, digest_size=256 // 8
+            ).hexdigest()
+        }
+        db_request.db.add(
+            File(
+                release=release,
+                filename=filename,
+                md5_digest=hashes["md5"],
+                sha256_digest=hashes["sha256"],
+                blake2_256_digest=hashes["blake2_256"],
+                path="source/{name[0]}/{name}/{filename}".format(
+                    name=project.name,
+                    filename=filename,
+                ),
+            ),
+        )
+
+        assert legacy._is_duplicate_file(
+            db_request.db, requested_file_name, hashes
+        ) is False
+
     def test_is_duplicate_false(self, pyramid_config, db_request):
         pyramid_config.testing_securitypolicy(userid=1)
 
@@ -1792,7 +1834,69 @@ def test_upload_fails_with_existing_filename_diff_content(self,
         ]
         assert resp.status_code == 400
         assert resp.status == (
-            "400 File already exists. "
+            "400 The filename or contents already exist. "
+            "See /the/help/url/"
+        )
+
+    def test_upload_fails_with_diff_filename_same_blake2(self,
+                                                         pyramid_config,
+                                                         db_request):
+        pyramid_config.testing_securitypolicy(userid=1)
+
+        user = UserFactory.create()
+        project = ProjectFactory.create()
+        release = ReleaseFactory.create(project=project, version="1.0")
+        RoleFactory.create(user=user, project=project)
+
+        filename = "{}-{}.tar.gz".format(project.name, release.version)
+        file_content = io.BytesIO(b"A fake file.")
+
+        db_request.POST = MultiDict({
+            "metadata_version": "1.2",
+            "name": project.name,
+            "version": release.version,
+            "filetype": "sdist",
+            "md5_digest": hashlib.md5(file_content.getvalue()).hexdigest(),
+            "content": pretend.stub(
+                filename="{}-fake.tar.gz".format(project.name),
+                file=file_content,
+                type="application/tar",
+            ),
+        })
+
+        db_request.db.add(
+            File(
+                release=release,
+                filename=filename,
+                md5_digest=hashlib.md5(file_content.getvalue()).hexdigest(),
+                sha256_digest=hashlib.sha256(
+                    file_content.getvalue()
+                ).hexdigest(),
+                blake2_256_digest=hashlib.blake2b(
+                    file_content.getvalue(),
+                    digest_size=256 // 8
+                ).hexdigest(),
+                path="source/{name[0]}/{name}/{filename}".format(
+                    name=project.name,
+                    filename=filename,
+                ),
+            ),
+        )
+        db_request.route_url = pretend.call_recorder(
+            lambda route, **kw: "/the/help/url/"
+        )
+
+        with pytest.raises(HTTPBadRequest) as excinfo:
+            legacy.file_upload(db_request)
+
+        resp = excinfo.value
+
+        assert db_request.route_url.calls == [
+            pretend.call('help', _anchor='file-name-reuse')
+        ]
+        assert resp.status_code == 400
+        assert resp.status == (
+            "400 The filename or contents already exist. "
             "See /the/help/url/"
         )