internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pypi/warehouse

Issue:
Add a "shortlink" for projects
**From user testing:**

When viewing projects on PyPI, some users type the URL directly if they know the project name.

We should create a shortlink like`pypi.org/p/myproject` which would redirect to `pypi.org/projects/myproject`

cc @di for feedback / guidance.

---

**Good First Issue**: This issue is good for first time contributors. If you've already contributed to Warehouse, please work on [another issue without this label](https://github.com/pypa/warehouse/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+-label%3A%22good+first+issue%22) instead. If there is not a corresponding pull request for this issue, it is up for grabs. For directions for getting set up, see our [Getting Started Guide](https://warehouse.pypa.io/development/getting-started/). If you are working on this issue and have questions, please feel free to ask them here, [`#pypa-dev` on Freenode](https://webchat.freenode.net/?channels=%23pypa-dev), or the [pypa-dev mailing list](https://groups.google.com/forum/#!forum/pypa-dev).



Discussion:
Going to remove the "good first issue" label from this. IMO while this would be nice, it's not likely to happen.

We have a number of other routes that sit at the top level path here, and this would make packages conflict with them, e.g. would https://pypi.org/help/ go to our "Help" page (as it does now) or to the `help` project (currently at https://pypi.org/project/help/)?

We would need to 'audit' the existing routes for conflicting projects and either rename our routes or "reserve" the project name for our own routes.

This would also make adding new routes harder in the future, because we'd need to go through the same process, rather than just adding it as we do now.
What about a short route? E.g. https://pypi.org/p/foobar which would replace, or be an alias to https://pypi.org/project/foobar . It would be a shorter URL to type directly, while not messing up "non project" routes (like https://pypi.org/help).
I'd be fine with a shorter router as an alias. I don't think we gain much by making it the canonical location, but a blanket redirect of ``/p/`` to ``/project/`` seems like it'd be fine, and let people type out something shorter.
Sounds good to me, this is probably a "good first issue" then. I've updated the title/description/labels accordingly.

Resolving patch:
diff --git a/warehouse/routes.py b/warehouse/routes.py
--- a/warehouse/routes.py
+++ b/warehouse/routes.py
@@ -178,6 +178,7 @@ def includeme(config):
     )
 
     # Packaging
+    config.add_redirect('/p/{name}/', '/project/{name}/', domain=warehouse)
     config.add_route(
         "packaging.project",
         "/project/{name}/",


Test patch:
diff --git a/tests/unit/test_routes.py b/tests/unit/test_routes.py
--- a/tests/unit/test_routes.py
+++ b/tests/unit/test_routes.py
@@ -269,6 +269,7 @@ def add_policy(name, filename):
     ]
 
     assert config.add_redirect.calls == [
+        pretend.call("/p/{name}/", "/project/{name}/", domain=warehouse),
         pretend.call("/pypi/{name}/", "/project/{name}/", domain=warehouse),
         pretend.call(
             "/pypi/{name}/{version}/",
Repository: pypi/warehouse

Issue:
License metadata seems to be ignored
When I made the latest release of `python-dateutil`, it came along with a license change from BSD to Apache / BSD dual licensed.  I updated the `license=` metadata in `setup.py`, but I forgot to update the trove classifiers.

[The page on PyPI](https://pypi.python.org/pypi/python-dateutil/2.7.0) shows the license as "Apache 2.0" as I would expect. [The page on warehouse](https://pypi.org/project/python-dateutil/) shows the license as "BSD License". I'm assuming it's pulling that from the trove classifier? Shouldn't it pull it from the `license` field if that is populated?


Discussion:

Resolving patch:
diff --git a/warehouse/packaging/views.py b/warehouse/packaging/views.py
--- a/warehouse/packaging/views.py
+++ b/warehouse/packaging/views.py
@@ -122,16 +122,21 @@ def release_detail(release, request):
         )
     ]
 
-    # Get the license from the classifiers or metadata, preferring classifiers.
-    license = None
-    if release.license:
-        # Make a best effort when the entire license text is given
-        # by using the first line only.
-        license = release.license.split('\n')[0]
-    license_classifiers = [c.split(" :: ")[-1] for c in release.classifiers
-                           if c.startswith("License")]
-    if license_classifiers:
-        license = ', '.join(license_classifiers)
+    # Get the license from both the `Classifier` and `License` metadata fields
+    license_classifiers = ', '.join(
+        c.split(" :: ")[-1]
+        for c in release.classifiers
+        if c.startswith("License")
+    )
+
+    # Make a best effort when the entire license text is given by using the
+    # first line only.
+    short_license = release.license.split('\n')[0] if release.license else None
+
+    if license_classifiers and short_license:
+        license = f'{license_classifiers} ({short_license})'
+    else:
+        license = license_classifiers or short_license or None
 
     return {
         "project": project,


Test patch:
diff --git a/tests/unit/packaging/test_views.py b/tests/unit/packaging/test_views.py
--- a/tests/unit/packaging/test_views.py
+++ b/tests/unit/packaging/test_views.py
@@ -206,11 +206,11 @@ def test_license_from_classifier(self, db_request):
             classifier="License :: OSI Approved :: BSD License")
         release = ReleaseFactory.create(
             _classifiers=[other_classifier, classifier],
-            license="Will not be used")
+            license="Will be added at the end")
 
         result = views.release_detail(release, db_request)
 
-        assert result["license"] == "BSD License"
+        assert result["license"] == "BSD License (Will be added at the end)"
 
     def test_license_with_no_classifier(self, db_request):
         """With no classifier, a license is used from metadata."""
Repository: pypi/warehouse

Issue:
Missing Purges
Noticed this while setting up new mirror. We don't seem to be purging `project/<normalized_name>` key when projects are deleted.

This leads bandersnatch to get confused and fall behind until the key is purged so the JSON api returns a 404


Discussion:
Also appears to miss purges due to maintainership changes and _pypi_hidden (legacy) changes.
Legacy doesn't purge Warehouse at all
I think we should just disable editing anything but pypi_hidden on legacy, and disable the journal entry for it then @dstufft ?
fine with me, pypi_hidden is useless in warehouse anyways.
👍
pypi-legacy interference resolved in https://github.com/pypa/pypi-legacy/pull/785

Still need maintainer changes to purge `project/` keys though.
Should just require adding:
```
    config.register_origin_cache_keys(
         Role,
         purge_keys=[
             key_factory("user/{obj.user.username}"),
             key_factory("project/{obj.project.normalized_name}")
         ],
     )
```

Resolving patch:
diff --git a/warehouse/packaging/__init__.py b/warehouse/packaging/__init__.py
--- a/warehouse/packaging/__init__.py
+++ b/warehouse/packaging/__init__.py
@@ -17,7 +17,7 @@
 from warehouse.accounts.models import User, Email
 from warehouse.cache.origin import key_factory, receive_set
 from warehouse.packaging.interfaces import IFileStorage
-from warehouse.packaging.models import Project, Release
+from warehouse.packaging.models import Project, Release, Role
 from warehouse.packaging.tasks import compute_trending
 
 
@@ -60,6 +60,13 @@ def includeme(config):
             key_factory("all-projects"),
         ],
     )
+    config.register_origin_cache_keys(
+        Role,
+        purge_keys=[
+            key_factory("user/{obj.user.username}"),
+            key_factory("project/{obj.project.normalized_name}")
+        ],
+    )
     config.register_origin_cache_keys(
         User,
         cache_keys=["user/{obj.username}"],


Test patch:
diff --git a/tests/unit/packaging/test_init.py b/tests/unit/packaging/test_init.py
--- a/tests/unit/packaging/test_init.py
+++ b/tests/unit/packaging/test_init.py
@@ -18,7 +18,7 @@
 from warehouse import packaging
 from warehouse.accounts.models import Email, User
 from warehouse.packaging.interfaces import IFileStorage
-from warehouse.packaging.models import Project, Release
+from warehouse.packaging.models import Project, Release, Role
 from warehouse.packaging.tasks import compute_trending
 
 
@@ -71,6 +71,13 @@ def key_factory(keystring, iterate_on=None):
                 key_factory("all-projects"),
             ],
         ),
+        pretend.call(
+            Role,
+            purge_keys=[
+                key_factory("user/{obj.user.username}"),
+                key_factory("project/{obj.project.normalized_name}")
+            ],
+        ),
         pretend.call(
             User,
             cache_keys=["user/{obj.username}"],
Repository: pypi/warehouse

Issue:
Remove X-Forwarded-Proto/X-Forwarded-Port h2o workaround
[In `warehouse/utils/wsgi.py`](https://github.com/pypa/warehouse/blob/master/warehouse/utils/wsgi.py#L51-L58) we note that once https://github.com/h2o/h2o/issues/883 is solved, we can remove a few lines about `X-Forwarded-Port` and `X-Forwarded-Proto`. They resolved that issue in May 2016 and [have released several new versions since then](https://h2o.examp1e.net/). OK to remove workaround?

Followup to b8b9f385382cd659750c694cf8b1b3db6f1f6d35 .


Discussion:
We don't even use h2o anymore, so it can just be removed I think

Resolving patch:
diff --git a/warehouse/utils/wsgi.py b/warehouse/utils/wsgi.py
--- a/warehouse/utils/wsgi.py
+++ b/warehouse/utils/wsgi.py
@@ -48,15 +48,6 @@ def __call__(self, environ, start_response):
             )
             host = environ.get("HTTP_X_FORWARDED_HOST", "")
 
-            # If we have a X-Forwarded-Port and it disagreed with
-            # X-Forwarded-Proto then we're going to listen to X-Forwarded-Port
-            # instead. This is because h2o overwrites X-Forwarded-Proto but not
-            # X-Forwarded-Port
-            # TODO: Note, this can go away if/once h2o/h2o#883 is solved.
-            port = environ.get("HTTP_X_FORWARDED_PORT", "")
-            if port == "443":
-                proto = "https"
-
         # Put the new header values into our environment.
         if remote_addr:
             environ["REMOTE_ADDR"] = remote_addr


Test patch:
diff --git a/tests/unit/utils/test_wsgi.py b/tests/unit/utils/test_wsgi.py
--- a/tests/unit/utils/test_wsgi.py
+++ b/tests/unit/utils/test_wsgi.py
@@ -149,30 +149,6 @@ def test_selects_right_x_forwarded_value(self):
             ),
         ]
 
-    def test_accepts_x_forwarded_port(self):
-        response = pretend.stub()
-        app = pretend.call_recorder(lambda e, s: response)
-
-        environ = {
-            "HTTP_X_FORWARDED_PROTO": "http",
-            "HTTP_X_FORWARDED_PORT": "443",
-            "HTTP_SOME_OTHER_HEADER": "woop",
-        }
-        start_response = pretend.stub()
-
-        resp = wsgi.ProxyFixer(app, token=None)(environ, start_response)
-
-        assert resp is response
-        assert app.calls == [
-            pretend.call(
-                {
-                    "HTTP_SOME_OTHER_HEADER": "woop",
-                    "wsgi.url_scheme": "https",
-                },
-                start_response,
-            ),
-        ]
-
 
 class TestVhmRootRemover:
Repository: pypi/warehouse

Issue:
Set samesite=lax on session cookies
This is a strong defense-in-depth mechanism for protecting against CSRF. It's currently only respected by Chrome, but Firefox will add it as well.


Discussion:
This may be blocked on https://github.com/Pylons/pyramid/issues/2733
Thanks, @alex.

Yeah, after Pyramid supports this, Warehouse should add it.
I think we're blocked on a WebOb release, not pyramid support.
I think this is unblocked now that #3554 has been merged.

Resolving patch:
diff --git a/warehouse/sessions.py b/warehouse/sessions.py
--- a/warehouse/sessions.py
+++ b/warehouse/sessions.py
@@ -263,6 +263,7 @@ def _process_response(self, request, response):
                 max_age=self.max_age,
                 httponly=True,
                 secure=request.scheme == "https",
+                samesite=b"lax"
             )
 
 


Test patch:
diff --git a/tests/unit/test_sessions.py b/tests/unit/test_sessions.py
--- a/tests/unit/test_sessions.py
+++ b/tests/unit/test_sessions.py
@@ -497,7 +497,7 @@ def test_invalidated_deletes_save_non_secure(self, monkeypatch,
         )
         response = pretend.stub(
             set_cookie=pretend.call_recorder(
-                lambda cookie, data, max_age, httponly, secure: None
+                lambda cookie, data, max_age, httponly, secure, samesite: None
             )
         )
         session_factory._process_response(pyramid_request, response)
@@ -532,6 +532,7 @@ def test_invalidated_deletes_save_non_secure(self, monkeypatch,
                 max_age=12 * 60 * 60,
                 httponly=True,
                 secure=False,
+                samesite=b"lax",
             ),
         ]
Repository: pypi/warehouse

Issue:
Missing legacy redirection from pypi.python.org/pypi/
**Describe the bug**
Redirections from `https://pypi.python.org/pypi/` are not handled (only redirected to `https://pypi.org/pypi/` by varnish (fastly)).

As https://pypi.org/pypi/ does not exists, it creates some broken links.

**Expected behavior**
A 301 to `https://pypi.org/`, simply.

**To Reproduce**
```$ curl -sI https://pypi.python.org/pypi/
HTTP/2 301 
server: Varnish
retry-after: 0
location: https://pypi.org/pypi/
[...redacted for readability...]
```




Discussion:
Per @JulienPalard:

>  it's a missing route in warehouse, .... probably in `warehouse/routes.py` around line 327.
I can add this legacy redirect, thanks!

Resolving patch:
diff --git a/warehouse/routes.py b/warehouse/routes.py
--- a/warehouse/routes.py
+++ b/warehouse/routes.py
@@ -330,6 +330,7 @@ def includeme(config):
         "/project/{name}/{version}/",
         domain=warehouse,
     )
+    config.add_redirect("/pypi/", "/", domain=warehouse)
     config.add_redirect("/packages/{path:.*}", files_url, domain=warehouse)
 
     # Legacy Action Redirects


Test patch:
diff --git a/tests/unit/test_routes.py b/tests/unit/test_routes.py
--- a/tests/unit/test_routes.py
+++ b/tests/unit/test_routes.py
@@ -292,6 +292,7 @@ def add_policy(name, filename):
             "/project/{name}/{version}/",
             domain=warehouse,
         ),
+        pretend.call("/pypi/", "/", domain=warehouse),
         pretend.call(
             "/packages/{path:.*}",
             "https://files.example.com/packages/{path}",
Repository: pypi/warehouse

Issue:
use CSP: sandbox on /simple/ pages
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/sandbox + https://www.youtube.com/watch?v=fbhW37JZtSA&feature=youtu.be

I believe this is a great fit for /simple/, which don't need any ability to do anthing but have a simple HTML structure.

I _think_ we can replace the whole current header with `Content-Security-Policy: sandbox allow-top-navigations`.


Discussion:
@devd -- I just watched your talk which inspired me to file this. Do you mind weighing in on if the policy I suggested is enough to fully lock down the page (https://pypi.org/simple/cryptography/ is an example of what these pages look like), or do we also need a `default-src: 'none'`?
I think that makes sense. noodling on this, I suspect `default-src: 'none'` is enough but why not both? Good defense in depth. 
Thanks @devd!

@dstufft Can you suggest where in the codebase the right place to _override_ the CSP for `/simple/` pages is?

Resolving patch:
diff --git a/warehouse/csp.py b/warehouse/csp.py
--- a/warehouse/csp.py
+++ b/warehouse/csp.py
@@ -34,6 +34,12 @@ def content_security_policy_tween(request):
         except ValueError:
             policy = collections.defaultdict(list)
 
+        # Replace CSP headers on /simple/ pages.
+        if request.path.startswith("/simple/"):
+            policy = collections.defaultdict(list)
+            policy["sandbox"] = ["allow-top-navigation"]
+            policy["default-src"] = [NONE]
+
         # We don't want to apply our Content Security Policy to the debug
         # toolbar, that's not part of our application and it doesn't work with
         # our restrictive CSP.


Test patch:
diff --git a/tests/unit/test_csp.py b/tests/unit/test_csp.py
--- a/tests/unit/test_csp.py
+++ b/tests/unit/test_csp.py
@@ -163,6 +163,34 @@ def test_devel_csp(self):
             "Content-Security-Policy": "script-src https://example.com",
         }
 
+    def test_simple_csp(self):
+        settings = {
+            "csp": {
+                "default-src": ["'none'"],
+                "sandbox": ["allow-top-navigation"],
+            }
+        }
+        response = pretend.stub(headers={})
+        registry = pretend.stub(settings=settings)
+        handler = pretend.call_recorder(lambda request: response)
+
+        tween = csp.content_security_policy_tween_factory(handler, registry)
+
+        request = pretend.stub(
+            scheme="https",
+            host="example.com",
+            path="/simple/",
+            find_service=pretend.call_recorder(
+                lambda *args, **kwargs: settings["csp"],
+            ),
+        )
+
+        assert tween(request) is response
+        assert response.headers == {
+            "Content-Security-Policy":
+                "default-src 'none'; sandbox allow-top-navigation",
+        }
+
 
 class TestCSPPolicy:
     def test_create(self):
Repository: pypi/warehouse

Issue:
Redirect loop when requesting missing package
**Describe the bug**
When I `curl -v https://pypi.org/pypi/bogus/1.0.0/json` with an invalid package/version combination, I get put in an infinite redirect loop. If the package/version combination is correct, then the data is served as expected.

I noticed this because the `license_finder` gem just now started having issues when looking for my team's internal (non-PyPi) packages. Previously, it hadn't complained.

**Expected behavior**
For an invalid package/version combination, I'd expect a 404 (or whatever the previous behavior was).

**To Reproduce**
`curl -v https://pypi.org/pypi/bogus/1.0.0/json`, note that the `Location` header leads you to the same URL.

**My Platform**
Pretty sure my platform isn't relevant. I've reproduced the problem both from my personal internet connection as well as within AWS.

**Additional context**
Adding @dstufft @di as suggested on IRC.



Discussion:
I'm guessing this is likely fallout from https://github.com/pypa/warehouse/pull/3971.
Looks like this is because we use `@notfound_view_config(append_slash=HTTPMovedPermanently)` with our exception view, so when it's not found, it gets a redirect to the same URL with the trailing slash.

Resolving patch:
diff --git a/warehouse/routes.py b/warehouse/routes.py
--- a/warehouse/routes.py
+++ b/warehouse/routes.py
@@ -325,17 +325,6 @@ def includeme(config):
 
     # Legacy Redirects
     config.add_redirect("/pypi/{name}/", "/project/{name}/", domain=warehouse)
-    config.add_redirect(
-        "/pypi/{name}/{version}/json/",
-        "/pypi/{name}/{version}/json",
-        domain=warehouse,
-    )
-    config.add_redirect(
-        "/pypi/{name}/json/",
-        "/pypi/{name}/json",
-        domain=warehouse,
-    )
-
     config.add_redirect(
         "/pypi/{name}/{version}/",
         "/project/{name}/{version}/",


Test patch:
diff --git a/tests/unit/test_routes.py b/tests/unit/test_routes.py
--- a/tests/unit/test_routes.py
+++ b/tests/unit/test_routes.py
@@ -287,16 +287,6 @@ def add_policy(name, filename):
     assert config.add_redirect.calls == [
         pretend.call("/p/{name}/", "/project/{name}/", domain=warehouse),
         pretend.call("/pypi/{name}/", "/project/{name}/", domain=warehouse),
-        pretend.call(
-            "/pypi/{name}/{version}/json/",
-            "/pypi/{name}/{version}/json",
-            domain=warehouse,
-        ),
-        pretend.call(
-            "/pypi/{name}/json/",
-            "/pypi/{name}/json",
-            domain=warehouse,
-        ),
         pretend.call(
             "/pypi/{name}/{version}/",
             "/project/{name}/{version}/",
Repository: pypi/warehouse

Issue:
400 Client Error: Invalid value for provides
**Describe the bug**
Attempting to upload an updated version of an existing python package results in a HTTPError 400 "Invalid value for provides" response.

A condensed example below.

```python
# Fail
setup(
    name='example',
    provides=['test.plugins']
)

# Success
setup(
    name='example',
    provides=['test_plugins']
)
```
**Expected behavior**
The ability to upload a revision to an existing package after bumping the version number in the setup.py file.

**To Reproduce**
```
# Bump version in setup.py file.
$ python setup.py build sdist
$ twine upload -r testpypi dist/*
```

**My Platform**
```
python 2.7.15
twine  1.13.0  

```
**Additional context**
<!-- Add any other context, links, etc. about the feature here. -->
The last release of my package was in 2016 and requirements for the provides kwarg could have changed. If that's the case, this can be closed & I'll update my setup.py.

[Pypi release history](https://pypi.org/project/PyUpdater-s3-Plugin/#history) of the package I noticed the issue on.


Discussion:

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -225,7 +225,10 @@ def _validate_legacy_non_dist_req(requirement):
             "Can't direct dependency: {!r}".format(requirement)
         )
 
-    if not req.name.isalnum() or req.name[0].isdigit():
+    if any(
+        not identifier.isalnum() or identifier[0].isdigit()
+        for identifier in req.name.split(".")
+    ):
         raise wtforms.validators.ValidationError("Use a valid Python identifier.")
 
 


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -97,7 +97,9 @@ def test_validates_invalid_pep440_specifier(self, specifier):
         with pytest.raises(ValidationError):
             legacy._validate_pep440_specifier(specifier)
 
-    @pytest.mark.parametrize("requirement", ["foo (>=1.0)", "foo", "_foo", "foo2"])
+    @pytest.mark.parametrize(
+        "requirement", ["foo (>=1.0)", "foo", "_foo", "foo2", "foo.bar"]
+    )
     def test_validates_legacy_non_dist_req_valid(self, requirement):
         legacy._validate_legacy_non_dist_req(requirement)
 
@@ -111,6 +113,7 @@ def test_validates_legacy_non_dist_req_valid(self, requirement):
             "☃ (>=1.0)",
             "☃",
             "name @ https://github.com/pypa",
+            "foo.2bar",
         ],
     )
     def test_validates_legacy_non_dist_req_invalid(self, requirement):