Repository: pypi/warehouse

Issue:
Fully implement PEP 527
We still retain the ability for some projects to upload the legacy filetypes listed in [PEP 527](https://www.python.org/dev/peps/pep-0527/): 

https://github.com/pypa/warehouse/blob/aa0d54019c322b52cb6428780808816d417abbd1/warehouse/packaging/models.py#L124

https://github.com/pypa/warehouse/blob/e1d0e4e41738fd07a0edeb77f95fedba0fdd41f8/warehouse/forklift/legacy.py#L463-L480

https://github.com/pypa/warehouse/blob/e1d0e4e41738fd07a0edeb77f95fedba0fdd41f8/warehouse/forklift/legacy.py#L1167-L1174

We should audit which projects currently have this ability, and whether they are still publishing deprecated filetypes. For example, [Pillow is no longer publishing `bdist_wininst` files](https://github.com/python-pillow/Pillow/pull/4029).


Discussion:
IMO we should also push these projects to drop usage of the legacy formats, and if not, at least get a good understanding if the issue is a "the current toolchain doesn't satisfy workflows like the legacy formats" or "oh, we don't need it" or something else entirely.


How long is the list? 

If not too long, let's create issues at these projects, it's likely they're not aware of the deprecation. I'd expect many are already using wheels and can ditch the legacy formats.

For example with Pillow, I didn't know they were deprecated until it came up at https://discuss.python.org/t/deprecate-bdist-wininst/1929/12?u=hugovk, and we already distribute wheels so it was "oh, we don't need it".

And how about adding a deprecation warning to Twine when uploading them?
> How long is the list?

IIRC the "list" was any project that had previously uploaded one of these filetypes, essentially we only blocked new projects.

A better list would be every project that has this ability that has actually published one of these filetypes in the last `N` months.

I'm guessing that this list is short enough that adding a deprecation notice to Twine would be unnecessary, but hard to say until we actually make an audit.
There are currently 4,678 projects that have `allow_legacy_files` set:

```
warehouse=> select count(*) from projects where allow_legacy_files;
 count
-------
  4678
(1 row)
```

Recent uploads for individual deprecated filetypes:

### `bdist_dmg`:
```
warehouse=> select filename, upload_time from release_files where packagetype='bdist_dmg' order by upload_time desc limit 10;
                    filename                    |        upload_time
------------------------------------------------+----------------------------
 python_igraph-0.7.1.post6-py2.7-macosx10.9.dmg | 2015-06-05 20:58:15.702734
 python_igraph-0.7.1.post6-py2.6-macosx10.9.dmg | 2015-06-05 20:56:47.202244
 python_igraph-0.7.1_4-py2.7-macosx10.9.dmg     | 2015-03-05 21:11:20.376493
 python_igraph-0.7.1_4-py2.6-macosx10.9.dmg     | 2015-03-05 21:10:22.376507
 python_igraph-0.7.1_3-py2.7-macosx10.9.dmg     | 2015-03-05 20:30:28.362479
 python_igraph-0.7.1_3-py2.6-macosx10.9.dmg     | 2015-03-05 20:28:18.772667
 python_igraph-0.7.1_2-py2.7-macosx10.9.dmg     | 2015-02-10 20:29:57.860577
 python_igraph-0.7.1_2-py2.6-macosx10.9.dmg     | 2015-02-10 20:12:25.660451
 python_igraph-0.7.1_1-py2.7-macosx10.9.dmg     | 2015-02-10 07:56:59.793387
 python_igraph-0.7.1_1-py2.6-macosx10.9.dmg     | 2015-02-09 20:48:42.300196
(10 rows)
```

### `bdist_dumb`:
```
warehouse=> select filename, upload_time from release_files where packagetype='bdist_dumb' order by upload_time desc limit 10;
                    filename                     |        upload_time
-------------------------------------------------+----------------------------
 airspeed-0.5.13.macosx-10.14-x86_64.tar.gz      | 2019-10-22 00:49:01.646779
 py_nifty_cloud-0.9.5.macosx-10.14-x86_64.tar.gz | 2019-09-28 14:32:32.022959
 algorithmia-1.2.0.linux-x86_64.tar.gz           | 2019-08-02 19:12:17.972642
 htrc-0.1.51.macosx-10.7-x86_64.tar.gz           | 2019-07-30 14:19:25.724787
 htrc-0.1.51b1.macosx-10.7-x86_64.tar.gz         | 2019-07-24 17:42:03.053786
 htrc-0.1.51b0.macosx-10.7-x86_64.tar.gz         | 2019-07-24 15:36:59.304947
 airspeed-0.5.12.macosx-10.14-x86_64.tar.gz      | 2019-07-24 06:32:12.329112
 pysodium-0.7.2.linux-x86_64.tar.gz              | 2019-06-25 14:30:59.086794
 htrc-0.1.50.macosx-10.7-x86_64.tar.gz           | 2019-06-21 14:43:12.68958
 htrc-0.1.50b0.macosx-10.7-x86_64.tar.gz         | 2019-06-20 17:23:49.411197
(10 rows)
```

### `bdist_msi`:
```
warehouse=> select filename, upload_time from release_files where packagetype='bdist_msi' order by upload_time desc limit 10;
              filename               |        upload_time
-------------------------------------+----------------------------
 pywincffi-0.5.0.win32-py3.6.msi     | 2017-11-18 18:55:27.694295
 pywincffi-0.5.0.win32-py3.5.msi     | 2017-11-18 18:55:26.221545
 pywincffi-0.5.0.win32-py3.4.msi     | 2017-11-18 18:55:25.067028
 pywincffi-0.5.0.win32-py3.3.msi     | 2017-11-18 18:55:23.828515
 pywincffi-0.5.0.win32-py2.7.msi     | 2017-11-18 18:55:22.319013
 pywincffi-0.5.0.win-amd64-py3.6.msi | 2017-11-18 18:55:21.128075
 pywincffi-0.5.0.win-amd64-py3.5.msi | 2017-11-18 18:55:20.016371
 pywincffi-0.5.0.win-amd64-py3.4.msi | 2017-11-18 18:55:18.597794
 pywincffi-0.5.0.win-amd64-py3.3.msi | 2017-11-18 18:55:17.357411
 pywincffi-0.5.0.win-amd64-py2.7.msi | 2017-11-18 18:55:16.060902
(10 rows)
```

### `bdist_rpm`:
```
warehouse=> select filename, upload_time from release_files where packagetype='bdist_rpm' order by upload_time desc limit 10;
              filename               |        upload_time
-------------------------------------+----------------------------
 Aglyph-3.0.0-1.noarch.rpm           | 2018-03-16 02:39:14.304743
 Aglyph-3.0.0-1.src.rpm              | 2018-03-16 02:39:08.195363
 toughradius-5.0.0.6-1.noarch.rpm    | 2017-11-19 04:45:14.203082
 toughradius-5.0.0.6-1.src.rpm       | 2017-11-19 04:45:09.635322
 python-otopi-mdp-0.2.2-1.noarch.rpm | 2017-10-02 10:23:32.79819
 python-otopi-mdp-0.2.2-1.src.rpm    | 2017-10-02 10:23:23.707941
 toughradius-5.0.0.5-1.noarch.rpm    | 2017-08-20 07:48:30.370971
 toughradius-5.0.0.5-1.src.rpm       | 2017-08-20 07:48:26.664831
 cx_Oracle-6.0rc1-py35-1.x86_64.rpm  | 2017-06-17 00:14:38.838786
 cx_Oracle-6.0rc1-py27-1.x86_64.rpm  | 2017-06-17 00:14:20.067593
(10 rows)
```

### `bdist_wininst`:
```
warehouse=> select filename, upload_time from release_files where packagetype='bdist_wininst' order by upload_time desc limit 10;
                filename                 |        upload_time
-----------------------------------------+----------------------------
 GPy-1.9.9.win-amd64-py3.7.exe           | 2019-10-17 08:37:05.494866
 GPy-1.9.9.win-amd64-py3.6.exe           | 2019-10-17 08:28:54.341581
 GPy-1.9.9.win-amd64-py3.5.exe           | 2019-10-17 08:10:01.336052
 GPy-1.9.9.win-amd64-py2.7.exe           | 2019-10-17 08:01:52.12953
 Trac-1.0.19.win-amd64.exe               | 2019-10-15 00:36:35.161141
 Trac-1.0.19.win32.exe                   | 2019-10-15 00:36:30.520495
 xrayutilities-1.5.3.win-amd64-py3.7.exe | 2019-10-09 10:12:48.375835
 xrayutilities-1.5.3.win-amd64-py3.6.exe | 2019-10-09 10:12:44.698438
 xrayutilities-1.5.3.win-amd64-py3.5.exe | 2019-10-09 10:12:41.362779
 xrayutilities-1.5.3.win-amd64-py2.7.exe | 2019-10-09 10:12:38.179182
(10 rows)
```

Of these it looks like `bdist_dmg`,  `bdist_msi`, and `bdist_rpm` can just be shut off.

The `bdist_wininst` filetype is still getting a lot of uploads, but PEP 527 says that this is misleading:

> It's quite easy to look at the low usage of `bdist_dmg` and `bdist_msi` and conclude that removing them will be fairly low impact, however `bdist_wininst` has several orders of magnitude more usage. This is somewhat misleading though, because although it has more people uploading those files the actual usage of those uploaded files is fairly low. Taking a look at the previous 30 days, we can see that 90% of all downloads of `bdist_winist` files from PyPI were generated by the mirroring infrastructure and 7% of them were generated by setuptools (which can currently be better covered by `bdist_egg` files).

Also `bdist_dumb` is still getting the occasional upload, but these projects would probably be better served by uploading wheels if they want platform-specific built distributions.
Thanks. From the PEP's [removal process](https://www.python.org/dev/peps/pep-0527/#removal-process):

> Finally, an email will be generated to the maintainers of all projects still given the legacy flag, which will inform them of the upcoming new restrictions on uploads and tell them that these restrictions will be applied to future uploads to their projects starting in 1 month. Finally, after 1 month all projects will have the legacy file type flag removed, and support for uploading these types of files will cease to exist on PyPI.

Would now be a good time to send out the email? 

And to just bdist_dmg, bdist_msi, and bdist_rpm users first, or to all legacy format users? 
ooooh! Nice find @hugovk! ^>^

I'm in favor of dropping all legacy formats if the PEP has a clear mechanism to do so.
I don't really think it's necessary to email all 4,678 projects when only a small fraction have actually used their legacy flag recently. Perhaps we should set a timeframe instead: if they've uploaded a deprecated distribution type in the last year?
> We still retain the ability for some projects to upload the legacy filetypes listed in [PEP 527](https://www.python.org/dev/peps/pep-0527/):
> 
> https://github.com/pypa/warehouse/blob/aa0d54019c322b52cb6428780808816d417abbd1/warehouse/packaging/models.py#L124
> 
> https://github.com/pypa/warehouse/blob/e1d0e4e41738fd07a0edeb77f95fedba0fdd41f8/warehouse/forklift/legacy.py#L463-L480
> 
> https://github.com/pypa/warehouse/blob/e1d0e4e41738fd07a0edeb77f95fedba0fdd41f8/warehouse/forklift/legacy.py#L1167-L1174
> 
> We should audit which projects currently have this ability, and whether they are still publishing deprecated filetypes. For example, [Pillow is no longer publishing `bdist_wininst` files](https://github.com/python-pillow/Pillow/pull/4029).


Yes, that sound reasonable.

Gentle nudge on this, given https://discuss.python.org/t/3115.
OK, next steps would be:
1. querying to get the subset of the 4,678 projects that have `allow_legacy_files` set which have had an upload in the last year
2. querying for email addresses for all their maintainers
3. drafting the email
4. sending a bulk email

Would anyone like to help with #​3?
3. Something along the lines of this?

> Hello,
> 
> We're emailing because you're listed as the maintainer for a package that has uploaded a legacy file type to PyPI in the past year:
> 
> bdist_dmg
> bdist_dumb
> bdist_msi
> bdist_rpm
> bdist_wininst
> 
> Following PEP 527, it will soon not be possible to upload legacy file types. Existing uploads will remain on PyPI, but soon new ones cannot be uploaded.
> 
> https://www.python.org/dev/peps/pep-0527/
> 
> This restriction will apply to new uploads after 2020-04-01 [TODO decide exact date, must be at least 1 month from email date].
> 
> See PEP 527 for suggestions of replacement file types, and if you have any questions, please visit https://github.com/pypa/warehouse/issues/6792 [TODO or https://discuss.python.org/somewhere or somewhere else?].
> 
> Thank you,
> 
> [TODO]
> 
OK, I've sent the notices to everyone that's uploaded one of these packages in the last year. The shutoff date is 30 days from today (2020-04-12).

For posterity, here's the SQL script I used to generate the affected users/projects:
```sql
SELECT 
  user_id, 
  projects.name as project_name, 
  packagetype 
FROM 
  (
    SELECT 
      roles.user_id as user_id, 
      roles.project_id as project_id, 
      packagetype 
    FROM 
      (
        SELECT 
          project_id, 
          packagetype 
        FROM 
          (
            SELECT 
              release_id, 
              packagetype 
            FROM 
              release_files 
            WHERE 
              (
                packagetype IN (
                  'bdist_dmg', 'bdist_dumb', 'bdist_msi', 
                  'bdist_rpm', 'bdist_wininst'
                ) 
                AND "upload_time" > (
                  localtimestamp - interval '365 days'
                )
              ) 
            GROUP BY 
              release_id, 
              packagetype
          ) f 
          JOIN releases ON releases.id = f.release_id 
        GROUP BY 
          project_id, 
          packagetype
      ) release 
      JOIN roles ON release.project_id = roles.project_id 
    GROUP BY 
      user_id, 
      roles.project_id, 
      packagetype
  ) p1 
  JOIN projects ON p1.project_id = projects.id;
```
Ran that like so:
```
psql service=pypi -t -A -F"," -f pep527.sql > pep527.csv
```
Then used the following script to turn that output into a CSV of mass emails:
```python
import csv
from collections import defaultdict

users = defaultdict(list)
subject = "[PyPI] Notice: Deprecation of underused file types/extensions"
body_template = """
Hello,

We're emailing because you're listed as a maintainer or owner for a package that has uploaded a legacy file type to PyPI in the past year:

{project_list}

Following PEP 527, it will soon not be possible to upload legacy file types.

https://www.python.org/dev/peps/pep-0527/

This restriction will apply to new uploads after 30 days from today (2020-04-12). Existing uploads will remain on PyPI, but soon new ones cannot be uploaded.

See PEP 527 for suggestions of replacement file types, and if you have any questions, please comment on the tracking issue for this deprecation:

https://github.com/pypa/warehouse/issues/6792

Thank you,
The PyPI Administrators
"""

with open("pep527.csv") as f:
    reader = csv.DictReader(f)

    for row in reader:
        users[row["user_id"]].append((row["project_name"], row["packagetype"]))

with open("pep527-complete.csv", "w") as f:
    writer = csv.DictWriter(f, fieldnames=["user_id", "subject", "body_text"])
    writer.writeheader()

    for user_id, projects in users.items():
        project_list = "\n".join(
            f"* Project: {project_name}, package type: {packagetype}"
            for project_name, packagetype in projects
        )

        writer.writerow(
            {
                "user_id": user_id,
                "subject": subject,
                "body_text": body_template.format(project_list=project_list),
            }
        )
```

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -142,11 +142,7 @@ def _valid_platform_tag(platform_tag):
 _error_message_order = ["metadata_version", "name", "version"]
 
 
-_dist_file_regexes = {
-    # True/False is for legacy or not.
-    True: re.compile(r".+?\.(exe|tar\.gz|bz2|rpm|deb|zip|tgz|egg|dmg|msi|whl)$", re.I),
-    False: re.compile(r".+?\.(tar\.gz|zip|whl|egg)$", re.I),
-}
+_dist_file_re = re.compile(r".+?\.(tar\.gz|zip|whl|egg)$", re.I)
 
 
 _wheel_file_re = re.compile(
@@ -500,17 +496,7 @@ class MetadataForm(forms.Form):
         validators=[
             wtforms.validators.DataRequired(),
             wtforms.validators.AnyOf(
-                [
-                    "bdist_dmg",
-                    "bdist_dumb",
-                    "bdist_egg",
-                    "bdist_msi",
-                    "bdist_rpm",
-                    "bdist_wheel",
-                    "bdist_wininst",
-                    "sdist",
-                ],
-                message="Use a known file type.",
+                ["bdist_egg", "bdist_wheel", "sdist"], message="Use a known file type.",
             ),
         ]
     )
@@ -1171,7 +1157,7 @@ def file_upload(request):
         )
 
     # Make sure the filename ends with an allowed extension.
-    if _dist_file_regexes[project.allow_legacy_files].search(filename) is None:
+    if _dist_file_re.search(filename) is None:
         raise _exc_with_message(
             HTTPBadRequest,
             "Invalid file extension: Use .egg, .tar.gz, .whl or .zip "
@@ -1193,16 +1179,6 @@ def file_upload(request):
     ):
         raise _exc_with_message(HTTPBadRequest, "Invalid distribution file.")
 
-    # Ensure that the package filetype is allowed.
-    # TODO: Once PEP 527 is completely implemented we should be able to delete
-    #       this and just move it into the form itself.
-    if not project.allow_legacy_files and form.filetype.data not in {
-        "sdist",
-        "bdist_wheel",
-        "bdist_egg",
-    }:
-        raise _exc_with_message(HTTPBadRequest, "Unknown type of file.")
-
     # The project may or may not have a file size specified on the project, if
     # it does then it may or may not be smaller or larger than our global file
     # size limits.
diff --git a/warehouse/migrations/versions/b265ed9eeb8a_fully_deprecate_legacy_distribution_.py b/warehouse/migrations/versions/b265ed9eeb8a_fully_deprecate_legacy_distribution_.py
new file mode 100644
--- /dev/null
+++ b/warehouse/migrations/versions/b265ed9eeb8a_fully_deprecate_legacy_distribution_.py
@@ -0,0 +1,31 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Fully deprecate legacy distribution types
+
+Revision ID: b265ed9eeb8a
+Revises: c4cb2d15dada
+Create Date: 2020-03-12 17:51:08.447903
+"""
+
+from alembic import op
+
+revision = "b265ed9eeb8a"
+down_revision = "c4cb2d15dada"
+
+
+def upgrade():
+    op.drop_column("projects", "allow_legacy_files")
+
+
+def downgrade():
+    raise RuntimeError("Order No. 227 - Ни шагу назад!")
diff --git a/warehouse/packaging/models.py b/warehouse/packaging/models.py
--- a/warehouse/packaging/models.py
+++ b/warehouse/packaging/models.py
@@ -115,7 +115,6 @@ class Project(SitemapMixin, db.Model):
     has_docs = Column(Boolean)
     upload_limit = Column(Integer, nullable=True)
     last_serial = Column(Integer, nullable=False, server_default=sql.text("0"))
-    allow_legacy_files = Column(Boolean, nullable=False, server_default=sql.false())
     zscore = Column(Float, nullable=True)
 
     total_size = Column(BigInteger, server_default=sql.text("0"))


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -1513,7 +1513,10 @@ def test_upload_fails_with_legacy_type(self, pyramid_config, db_request):
         resp = excinfo.value
 
         assert resp.status_code == 400
-        assert resp.status == "400 Unknown type of file."
+        assert (
+            resp.status
+            == "400 Invalid value for filetype. Error: Use a known file type."
+        )
 
     def test_upload_fails_with_legacy_ext(self, pyramid_config, db_request):
         pyramid_config.testing_securitypolicy(userid=1)
@@ -2519,106 +2522,6 @@ def storage_service_store(path, file_path, *, meta):
             )
         ]
 
-    def test_upload_succeeds_with_legacy_ext(
-        self, tmpdir, monkeypatch, pyramid_config, db_request, metrics
-    ):
-        monkeypatch.setattr(tempfile, "tempdir", str(tmpdir))
-
-        pyramid_config.testing_securitypolicy(userid=1)
-
-        user = UserFactory.create()
-        EmailFactory.create(user=user)
-        project = ProjectFactory.create(allow_legacy_files=True)
-        release = ReleaseFactory.create(project=project, version="1.0")
-        RoleFactory.create(user=user, project=project)
-
-        filename = "{}-{}.tar.bz2".format(project.name, release.version)
-
-        db_request.user = user
-        db_request.remote_addr = "10.10.10.30"
-        db_request.user_agent = "warehouse-tests/6.6.6"
-        db_request.POST = MultiDict(
-            {
-                "metadata_version": "1.2",
-                "name": project.name,
-                "version": release.version,
-                "filetype": "sdist",
-                "pyversion": "source",
-                "md5_digest": _TAR_BZ2_PKG_MD5,
-                "content": pretend.stub(
-                    filename=filename,
-                    file=io.BytesIO(_TAR_BZ2_PKG_TESTDATA),
-                    type="application/tar",
-                ),
-            }
-        )
-
-        def storage_service_store(path, file_path, *, meta):
-            with open(file_path, "rb") as fp:
-                assert fp.read() == _TAR_BZ2_PKG_TESTDATA
-
-        storage_service = pretend.stub(store=storage_service_store)
-        db_request.find_service = lambda svc, name=None, context=None: {
-            IFileStorage: storage_service,
-            IMetricsService: metrics,
-        }.get(svc)
-
-        monkeypatch.setattr(legacy, "_is_valid_dist_file", lambda *a, **kw: True)
-
-        resp = legacy.file_upload(db_request)
-
-        assert resp.status_code == 200
-
-    def test_upload_succeeds_with_legacy_type(
-        self, tmpdir, monkeypatch, pyramid_config, db_request, metrics
-    ):
-        monkeypatch.setattr(tempfile, "tempdir", str(tmpdir))
-
-        pyramid_config.testing_securitypolicy(userid=1)
-
-        user = UserFactory.create()
-        EmailFactory.create(user=user)
-        project = ProjectFactory.create(allow_legacy_files=True)
-        release = ReleaseFactory.create(project=project, version="1.0")
-        RoleFactory.create(user=user, project=project)
-
-        filename = "{}-{}.tar.gz".format(project.name, release.version)
-
-        db_request.user = user
-        db_request.remote_addr = "10.10.10.30"
-        db_request.user_agent = "warehouse-tests/6.6.6"
-        db_request.POST = MultiDict(
-            {
-                "metadata_version": "1.2",
-                "name": project.name,
-                "version": release.version,
-                "filetype": "bdist_dumb",
-                "pyversion": "3.5",
-                "md5_digest": _TAR_GZ_PKG_MD5,
-                "content": pretend.stub(
-                    filename=filename,
-                    file=io.BytesIO(_TAR_GZ_PKG_TESTDATA),
-                    type="application/tar",
-                ),
-            }
-        )
-
-        def storage_service_store(path, file_path, *, meta):
-            with open(file_path, "rb") as fp:
-                assert fp.read() == _TAR_GZ_PKG_TESTDATA
-
-        storage_service = pretend.stub(store=storage_service_store)
-        db_request.find_service = lambda svc, name=None, context=None: {
-            IFileStorage: storage_service,
-            IMetricsService: metrics,
-        }.get(svc)
-
-        monkeypatch.setattr(legacy, "_is_valid_dist_file", lambda *a, **kw: True)
-
-        resp = legacy.file_upload(db_request)
-
-        assert resp.status_code == 200
-
     @pytest.mark.parametrize("plat", ["linux_x86_64", "linux_x86_64.win32"])
     def test_upload_fails_with_unsupported_wheel_plat(
         self, monkeypatch, pyramid_config, db_request, plat