Repository: gratipay/gratipay.com

Issue:
Broken URL links for npm packages with special characters? 
From @calvinhp on [Slack](https://gratipay.slackarchive.io/gratipay/page-100/ts-1496325029088097): 

> I just found a 404 :slightly_smiling_face:
> If I search for Plone, the first matching project is @plone/plone-react
> but it is a 404

[Search results](https://gratipay.com/search?q=Plone): 

![screen shot 2017-06-01 at 8 00 50 pm](https://cloud.githubusercontent.com/assets/3893573/26684619/1246a946-4705-11e7-821f-04d03028cbdb.png)

Broken Link: https://gratipay.com/on/npm/@plone/plone-react/

I tried URL encoding it to be https://gratipay.com/on/npm/%40plone%2Fplone-react/, didn't work. I've encountered similar problems with common routing libraries where a `/` can't be URL-encoded and placed in the URL, although the spec supports it - might be related. 


Discussion:
Blorg. At a minimum I guess we need to support these under `/on/npm/`. We don't allow forward slash in Gratipay project slugs (for, ahem, obvious reasons). So what are we going to do with projects claimed from these packages?
We also [don't](https://github.com/gratipay/gratipay.com/blob/2099/gratipay/models/team/__init__.py#L23) allow `@`.
> https://www.npmjs.com/package/@plone/plone-react/

It looks to me like this is a namespacing hack in npm. We plan to implement namespacing directly (#4246), at which point the obvious thing to do is undo npm's hack and simply name the project `plone-react`. We could do the same now against the promise of #4246.
I think it's safer to consume the remainder of the string under `/on/npm/` since we're not expecting to use that URL space for other things.
Can npm packages have `/` as the final character in their name? If so we've got a trailing slash race.
> simply name the project `plone-react`

That's what they're already doing on GitHub anyway: https://github.com/plone/plone-react.
> If so we've got a trailing slash race.

I guess we can do an extra db query if we don't find a match under `.../` or vice-versa.
Though the trailing slash case should be so rare that we should always try the non-trailing db query first I think. Much more likely to send `/on/npm/foo/` and want `/foo` than the other way around.
Docs: https://docs.npmjs.com/misc/scope. Maybe we don't have to worry about trailing slashes? Is there something interesting to be done with scope endpoints? [They don't](https://www.npmjs.com/package/@plone/).
I think the name itself can't include a `/`:

> The name ends up being part of a URL, an argument on the command line, and a folder name. Therefore, the name can't contain any non-URL-safe characters.

https://docs.npmjs.com/files/package.json
How are scoped packages stored in our packages table?
With the scope in the name. 👍 

```
gratipay::DATABASE=> select * from packages where name like '@%' limit 10;
┌────────┬─────────────────┬──────────────────────────────────────────┬──────────────────────────────────
│   id   │ package_manager │                   name                   │                                 d
├────────┼─────────────────┼──────────────────────────────────────────┼──────────────────────────────────
│ 872909 │ npm             │ @zeke/scopey                             │ A package for testing npm scopes 
│ 873868 │ npm             │ @scdhhs/document-services                │ A collection of services that pro
│ 874294 │ npm             │ @lib/tsargs                              │ Adapter for using tsconfig.json f
│ 874527 │ npm             │ @asaayers/react-a11y                     │ Warns about potential accessibili
│ 874540 │ npm             │ @dmail/test                              │                                  
│ 874829 │ npm             │ @smikes/bletch                           │                                  
│ 874996 │ npm             │ @planeshifter/tweet-sentiment            │ SVM Classifier to Detect Sentimen
│ 875207 │ npm             │ @jasonfill/lambda-cloud-watch-log-parser │ Parses event data from cloud watc
│ 875344 │ npm             │ @stevegill/test-module                   │ testing out scoped modules       
│ 875568 │ npm             │ @netapps/netapps-session                 │ netapps-api session store for Exp
└────────┴─────────────────┴──────────────────────────────────────────┴──────────────────────────────────
(10 rows)

```
In that case we should add redirect from trailing slash to non-trailing slash just for `on/npm/$package.spt`. We'll never want a trailing slash as part of a name, but we don't want to 404 if we get one in the URL.

Resolving patch:
diff --git a/gratipay/models/package/__init__.py b/gratipay/models/package/__init__.py
--- a/gratipay/models/package/__init__.py
+++ b/gratipay/models/package/__init__.py
@@ -16,7 +16,7 @@ class Package(Model, Emails, Team):
 
     Packages are entities on open source package managers; `npm
     <https://www.npmjs.com/>`_ is the only one we support so far. Each package
-    on npm has a page on Gratipay with an URL of the form ``/on/npm/foo/``.
+    on npm has a page on Gratipay with an URL of the form ``/on/npm/foo``.
     Packages can be claimed by Gratipay participants, at which point we create
     a :py:class:`~gratipay.models.team.Team` for them under the hood so they
     can start accepting payments.
@@ -40,7 +40,7 @@ def __ne__(self, other):
     def url_path(self):
         """The path part of the URL for this package on Gratipay.
         """
-        return '/on/{}/{}/'.format(self.package_manager, self.name)
+        return '/on/{}/{}'.format(self.package_manager, self.name)
 
 
     @property
diff --git a/gratipay/models/package/team.py b/gratipay/models/package/team.py
--- a/gratipay/models/package/team.py
+++ b/gratipay/models/package/team.py
@@ -40,9 +40,12 @@ def get_or_create_linked_team(self, cursor, owner):
             return team
 
         def slug_options():
-            yield self.name
+            # Having analyzed existing names, we should never get `@` without
+            # `/`. Be conservative in what we accept! Oh, wait ...
+            base_name = self.name.split('/')[1] if self.name.startswith('@') else self.name
+            yield base_name
             for i in range(1, 10):
-                yield '{}-{}'.format(self.name, i)
+                yield '{}-{}'.format(base_name, i)
             yield uuid.uuid4().hex
 
         for slug in slug_options():


Test patch:
diff --git a/tests/py/test_packages.py b/tests/py/test_packages.py
--- a/tests/py/test_packages.py
+++ b/tests/py/test_packages.py
@@ -64,9 +64,9 @@ def test_can_be_deleted_in_a_transaction(self):
 
 class Linking(Harness):
 
-    def link(self):
+    def link(self, package_name='foo'):
         alice = self.make_participant('alice')
-        package = self.make_package()
+        package = self.make_package(name=package_name)
         with self.db.get_cursor() as c:
             team = package.get_or_create_linked_team(c, alice)
         return alice, package, team
@@ -124,6 +124,10 @@ def test_linking_team_drops_back_to_uuid4_eventually(self, uuid):
         _, _, team = self.link()
         assert team.slug == 'deadbeef-aaaa-aaaa-aaaa-aaaaaaaaaaaa'
 
+    def test_linking_team_ignores_scope(self):
+        _, _, team = self.link(package_name='@foo/bar')
+        assert team.slug == 'bar'
+
     def test_linked_team_takes_package_description(self):
         _, _, team = self.link()
         assert team.product_or_service == 'Foo fooingly.'
diff --git a/tests/py/test_pages.py b/tests/py/test_pages.py
--- a/tests/py/test_pages.py
+++ b/tests/py/test_pages.py
@@ -34,8 +34,12 @@ def browse(self, setup=None, **kw):
         exchange_id = self.make_exchange('braintree-cc', 19, 0, alice, address=address)
 
         # for npm page
-        self.db.run("INSERT INTO packages (package_manager, name, description, emails) "
-                    "VALUES ('npm', 'foo-package', 'A package', ARRAY[]::text[])")
+        self.db.run("""
+            INSERT
+              INTO packages (package_manager, name, description, emails)
+            VALUES ('npm', 'foo-package', 'A package', ARRAY[]::text[])
+                 , ('npm', '@foo/package', 'A scoped package', ARRAY[]::text[])
+        """)
 
         if setup:
             setup(alice)
@@ -47,7 +51,7 @@ def browse(self, setup=None, **kw):
                            .replace('/~/%username/', '/~alice/') \
                            .replace('/for/%slug/', '/for/wonderland/') \
                            .replace('/%platform/', '/github/') \
-                           .replace('/%package/', '/foo-package/') \
+                           .replace('/%package', '/foo-package') \
                            .replace('/%user_name/', '/gratipay/') \
                            .replace('/%to', '/1') \
                            .replace('/%country', '/TT') \
@@ -63,6 +67,7 @@ def browse(self, setup=None, **kw):
            /about/me
            /about/me/
            /about/me/history
+           /on/npm/@foo/package
         """.split())
         for url in urls:
             try:
diff --git a/tests/py/test_www_npm_package.py b/tests/py/test_www_npm_package.py
--- a/tests/py/test_www_npm_package.py
+++ b/tests/py/test_www_npm_package.py
@@ -10,27 +10,32 @@ class Tests(Harness):
     def setUp(self):
         self.make_package()
 
+    def test_trailing_slash_redirects(self):
+        response = self.client.GxT('/on/npm/foo/')
+        assert response.code == 302
+        assert response.headers['Location'] == '/on/npm/foo'
+
     def test_anon_gets_signin_page_from_unclaimed(self):
-        body = self.client.GET('/on/npm/foo/').body
+        body = self.client.GET('/on/npm/foo').body
         assert 'foo</a> npm package on Gratipay:' in body
 
     def test_auth_gets_send_confirmation_page_from_unclaimed(self):
         self.make_participant('bob', claimed_time='now')
-        body = self.client.GET('/on/npm/foo/', auth_as='bob').body
+        body = self.client.GET('/on/npm/foo', auth_as='bob').body
         assert 'foo</a> npm package:' in body
         assert 'alice@example.com' in body
 
     def test_auth_gets_multiple_options_if_present(self):
         self.make_package(NPM, 'bar', 'Bar', ['alice@example.com', 'alice@example.net'])
         self.make_participant('bob', claimed_time='now')
-        body = self.client.GET('/on/npm/bar/', auth_as='bob').body
+        body = self.client.GET('/on/npm/bar', auth_as='bob').body
         assert 'alice@example.com' in body
         assert 'alice@example.net' in body
 
     def test_auth_gets_something_if_no_emails(self):
         self.make_package(NPM, 'bar', 'Bar', [])
         self.make_participant('bob', claimed_time='now')
-        body = self.client.GET('/on/npm/bar/', auth_as='bob').body
+        body = self.client.GET('/on/npm/bar', auth_as='bob').body
         assert "No email addresses on file" in body
 
 
@@ -46,7 +51,7 @@ def claim_package(self):
 
     def test_package_redirects_to_project_if_claimed(self):
         self.claim_package()
-        response = self.client.GxT('/on/npm/foo/')
+        response = self.client.GxT('/on/npm/foo')
         assert response.code == 302
         assert response.headers['Location'] == '/foo/'
 
diff --git a/tests/ttw/test_package_claiming.py b/tests/ttw/test_package_claiming.py
--- a/tests/ttw/test_package_claiming.py
+++ b/tests/ttw/test_package_claiming.py
@@ -12,7 +12,7 @@ class Test(BrowserHarness):
     def check(self, choice=0):
         self.make_participant('alice', claimed_time='now')
         self.sign_in('alice')
-        self.visit('/on/npm/foo/')
+        self.visit('/on/npm/foo')
         self.css('#content label')[0].click() # activate select
         self.css('#content label')[choice].click()
         self.css('#content button')[0].click()
@@ -44,7 +44,7 @@ def test_disabled_items_are_disabled(self):
         alice = self.make_participant('alice', claimed_time='now')
         self.add_and_verify_email(alice, 'alice@example.com', 'bob@example.com')
         self.sign_in('alice')
-        self.visit('/on/npm/foo/')
+        self.visit('/on/npm/foo')
         self.css('#content label')[0].click()            # activate select
         self.css('#content label')[1].click()            # click second item
         self.css('#content li')[0].has_class('selected') # first item is still selected
@@ -66,7 +66,7 @@ def test_claimed_packages_can_be_given_to(self):
 
         self.make_participant('bob', claimed_time='now')
         self.sign_in('bob')
-        self.visit('/on/npm/foo/')
+        self.visit('/on/npm/foo')
 
         self.css('.your-payment button.edit').click()
         self.wait_for('.your-payment input.amount').fill('10')
@@ -91,7 +91,7 @@ def test_visiting_verify_link_shows_helpful_information(self):
     def test_deleted_packages_are_404(self):
         self.make_package()
         Package.from_names(NPM, 'foo').delete()
-        self.visit('/on/npm/foo/')
+        self.visit('/on/npm/foo')
         assert self.css('#content h1').text == '404'