Repository: jupyterhub/oauthenticator

Issue:
Add team whitelisting for GitHubOAuthenticator and  GitLabOAuthenticator 
BitBucketOAuthenticator has team whitelisting, so it would seem reasonable to also add this for Gitlab and Github.

I am already working towards submitting a pull request, but am creating this issue here to serve as a centre for discussion and to avoid spamming the jupyterhub gitter channel.

Discussion:
jbweston: Apologies for the rebase -- I only noticed that there were conflicts after pushing
minrk: @jbweston sorry for leaving you hanging so long. I've started with some testing utilities in #66, which should give you a place to test what you are doing here. It's not the easiest to use, I realize, but it might be enough.
manics: `GitHubOAuthenticator` with `github_organization_whitelist` set fails for me due to insufficient permissions on the Github token:

```
[E 2017-03-30 14:42:41.766 JupyterHub web:1548] Uncaught exception GET /jupyter/hub/oauth_callback?code=XXXXXXXXXXXXXXXXXXXX (::ffff:192.168.2.9)
    HTTPServerRequest(protocol='http', host='XXXXXXXXXX', method='GET', uri='/jupyter/hub/oauth_callback?code=XXXXXXXXXXXXXXXXXXXX', version='HTTP/1.1', ...
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/tornado/web.py", line 1469, in _execute
        result = yield result
      File "/opt/conda/lib/python3.5/site-packages/oauthenticator/oauth2.py", line 58, in get
        username = yield self.authenticator.get_authenticated_user(self, None)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/auth.py", line 171, in get_authenticated_user
        username = yield self.authenticate(handler, data)
      File "/opt/conda/lib/python3.5/site-packages/oauthenticator/github.py", line 134, in authenticate
        user_in_org = yield self._check_organization_whitelist(username, access_token)
      File "/opt/conda/lib/python3.5/site-packages/oauthenticator/github.py", line 149, in _check_organization_whitelist
        resp = yield http_client.fetch(req)
    tornado.httpclient.HTTPError: HTTP 403: Forbidden
```

If I add some additional logging to `GitHubOAuthenticator._check_organization_whitelist` I can see the GitHub API request:
`[D 2017-03-30 14:46:11.719 JupyterHub github:148] url:https://api.github.com/user/orgs headers:[('User-Agent', 'JupyterHub'), ('Authorization', 'token XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'), ('Accept', 'application/json')]
`

Checking this with curl fails:
```
$ curl -H 'Authorization: token XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' https://api.github.com/user/orgs
{
  "message": "You need at least read:org scope or user scope to list your organizations.",
  "documentation_url": "https://developer.github.com/v3/orgs/#list-your-organizations"
}
```

The default scope for the GitHub oauth token [doesn't allow listing of a user's organisations](https://developer.github.com/v3/orgs/#list-your-organizations).

It's also worth noting that [users can change the scope](https://developer.github.com/v3/oauth/#requested-scopes-vs-granted-scopes):

> It is important to handle error cases where a user chooses to grant you less access than you originally requested. For example, applications can warn or otherwise communicate with their users that they will see reduced functionality or be unable to perform some actions.

jbweston: @manics thanks for the feedback. I haven't been busy with other stuff the past few weeks, but will sit down to look at this an evening next week.

Patch:
diff --git a/oauthenticator/bitbucket.py b/oauthenticator/bitbucket.py
index 0f4beab..f453e56 100644
--- a/oauthenticator/bitbucket.py
+++ b/oauthenticator/bitbucket.py
@@ -46,6 +46,13 @@ class BitbucketOAuthenticator(OAuthenticator):
         help="Automatically whitelist members of selected teams",
     )
 
+    bitbucket_team_whitelist = team_whitelist
+
+
+    headers = {"Accept": "application/json",
+               "User-Agent": "JupyterHub",
+               "Authorization": "Bearer {}"
+               }
 
     @gen.coroutine
     def authenticate(self, handler, data=None):
@@ -95,7 +102,7 @@ class BitbucketOAuthenticator(OAuthenticator):
 
         # Check if user is a member of any whitelisted teams.
         # This check is performed here, as the check requires `access_token`.
-        if self.team_whitelist:
+        if self.bitbucket_team_whitelist:
             user_in_team = yield self._check_team_whitelist(username, access_token)
             return username if user_in_team else None
         else:  # no team whitelisting
@@ -109,16 +116,18 @@ class BitbucketOAuthenticator(OAuthenticator):
         # We verify the team membership by calling teams endpoint.
         next_page = url_concat("https://api.bitbucket.org/2.0/teams",
                                {'role': 'member'})
-        user_teams = set()
         while next_page:
             req = HTTPRequest(next_page, method="GET", headers=headers)
             resp = yield http_client.fetch(req)
             resp_json = json.loads(resp.body.decode('utf8', 'replace'))
             next_page = resp_json.get('next', None)
 
-            user_teams |= \
+            user_teams = \
                 set([entry["username"] for entry in resp_json["values"]])
-        return len(self.team_whitelist & user_teams) > 0
+            # check if any of the organizations seen thus far are in whitelist
+            if len(self.bitbucket_team_whitelist & user_teams) > 0:
+                return True
+        return False
 
 
 class LocalBitbucketOAuthenticator(LocalAuthenticator,
diff --git a/oauthenticator/github.py b/oauthenticator/github.py
index 8bf8623..05a2f9e 100644
--- a/oauthenticator/github.py
+++ b/oauthenticator/github.py
@@ -7,25 +7,49 @@ Most of the code c/o Kyle Kelley (@rgbkrk)
 
 import json
 import os
+import re
 
 from tornado.auth import OAuth2Mixin
 from tornado import gen, web
 
+import requests
 from tornado.httputil import url_concat
 from tornado.httpclient import HTTPRequest, AsyncHTTPClient
 
 from jupyterhub.auth import LocalAuthenticator
 
-from traitlets import Unicode
+from traitlets import Unicode, Set
 
 from .oauth2 import OAuthLoginHandler, OAuthenticator
 
 # Support github.com and github enterprise installations
 GITHUB_HOST = os.environ.get('GITHUB_HOST') or 'github.com'
 if GITHUB_HOST == 'github.com':
-    GITHUB_API = 'api.github.com/user'
+    GITHUB_API = 'api.github.com'
 else:
-    GITHUB_API = '%s/api/v3/user' % GITHUB_HOST
+    GITHUB_API = '%s/api/v3' % GITHUB_HOST
+
+
+def _api_headers(access_token):
+    return {"Accept": "application/json",
+            "User-Agent": "JupyterHub",
+            "Authorization": "token {}".format(access_token)
+           }
+
+
+def _get_next_page(response):
+    # Github uses Link headers for pagination.
+    # See https://developer.github.com/v3/#pagination
+    link_header = response.headers.get('Link')
+    if not link_header:
+        return
+    for link in requests.utils.parse_header_links(link_header):
+        if link.get('rel') == 'next':
+            return link['url']
+    # if no "next" page, this is the last one
+    return None
+
+
 
 class GitHubMixin(OAuth2Mixin):
     _OAUTH_AUTHORIZE_URL = "https://%s/login/oauth/authorize" % GITHUB_HOST
@@ -53,6 +77,12 @@ class GitHubOAuthenticator(OAuthenticator):
     client_id_env = 'GITHUB_CLIENT_ID'
     client_secret_env = 'GITHUB_CLIENT_SECRET'
     login_handler = GitHubLoginHandler
+
+    github_organization_whitelist = Set(
+        config=True,
+        help="Automatically whitelist members of selected organizations",
+    )
+
     
     @gen.coroutine
     def authenticate(self, handler, data=None):
@@ -88,18 +118,45 @@ class GitHubOAuthenticator(OAuthenticator):
         access_token = resp_json['access_token']
         
         # Determine who the logged in user is
-        headers={"Accept": "application/json",
-                 "User-Agent": "JupyterHub",
-                 "Authorization": "token {}".format(access_token)
-        }
-        req = HTTPRequest("https://%s" % GITHUB_API,
+        req = HTTPRequest("https://%s/user" % GITHUB_API,
                           method="GET",
-                          headers=headers
+                          headers=_api_headers(access_token)
                           )
         resp = yield http_client.fetch(req)
         resp_json = json.loads(resp.body.decode('utf8', 'replace'))
 
-        return resp_json["login"]
+        username =  resp_json["login"]
+
+        # Check if user is a member of any whitelisted organizations.
+        # This check is performed here, as it requires `access_token`.
+        if self.github_organization_whitelist:
+            for org in self.github_organization_whitelist:
+                user_in_org = yield self._check_organization_whitelist(org, username, access_token)
+                if user_in_org:
+                    return username
+            else:  # User not found in member list for any organisation
+                return None
+        else:  # no organization whitelisting
+            return username
+
+
+    @gen.coroutine
+    def _check_organization_whitelist(self, org, username, access_token):
+        http_client = AsyncHTTPClient()
+        headers = _api_headers(access_token)
+        # Get all the members for organization 'org'
+        next_page = "https://%s/orgs/%s/members" % (GITHUB_API, org)
+        while next_page:
+            req = HTTPRequest(next_page, method="GET", headers=headers)
+            resp = yield http_client.fetch(req)
+            resp_json = json.loads(resp.body.decode('utf8', 'replace'))
+            next_page = _get_next_page(resp)
+            org_members = set(entry["login"] for entry in resp_json)
+            # check if any of the organizations seen thus far are in whitelist
+            if username in org_members:
+                return True
+        return False
+
 
 
 class LocalGitHubOAuthenticator(LocalAuthenticator, GitHubOAuthenticator):
diff --git a/oauthenticator/gitlab.py b/oauthenticator/gitlab.py
index 14edda7..6ffb732 100644
--- a/oauthenticator/gitlab.py
+++ b/oauthenticator/gitlab.py
@@ -13,16 +13,41 @@ import sys
 from tornado.auth import OAuth2Mixin
 from tornado import gen, web
 
+import requests
+from tornado.escape import url_escape
 from tornado.httputil import url_concat
 from tornado.httpclient import HTTPRequest, AsyncHTTPClient
 
 from jupyterhub.auth import LocalAuthenticator
 
+from traitlets import Set
+
 from .oauth2 import OAuthLoginHandler, OAuthenticator
 
 # Support gitlab.com and gitlab community edition installations
 GITLAB_HOST = os.environ.get('GITLAB_HOST') or 'https://gitlab.com'
-GITLAB_API = '%s/api/v3/user' % GITLAB_HOST
+GITLAB_API = '%s/api/v3' % GITLAB_HOST
+
+
+def _api_headers(access_token):
+    return {"Accept": "application/json",
+            "User-Agent": "JupyterHub",
+            "Authorization": "token {}".format(access_token)
+           }
+
+
+def _get_next_page(response):
+    # Gitlab uses Link headers for pagination.
+    # See https://docs.gitlab.com/ee/api/README.html#pagination-link-header
+    link_header = response.headers.get('Link')
+    if not link_header:
+        return
+    for link in requests.utils.parse_header_links(link_header):
+        if link.get('rel') == 'next':
+            return link['url']
+    # if no "next" page, this is the last one
+    return None
+
 
 class GitLabMixin(OAuth2Mixin):
     _OAUTH_AUTHORIZE_URL = "%s/oauth/authorize" % GITLAB_HOST
@@ -41,6 +66,12 @@ class GitLabOAuthenticator(OAuthenticator):
     client_secret_env = 'GITLAB_CLIENT_SECRET'
     login_handler = GitLabLoginHandler
 
+    gitlab_group_whitelist = Set(
+        config=True,
+        help="Automatically whitelist members of selected groups",
+    )
+
+
     @gen.coroutine
     def authenticate(self, handler, data=None):
         code = handler.get_argument("code", False)
@@ -81,18 +112,57 @@ class GitLabOAuthenticator(OAuthenticator):
         access_token = resp_json['access_token']
 
         # Determine who the logged in user is
-        headers={"Accept": "application/json",
-                 "User-Agent": "JupyterHub",
-        }
-        req = HTTPRequest("%s?access_token=%s" % (GITLAB_API, access_token),
+        req = HTTPRequest("%s/user" % GITLAB_API,
                           method="GET",
                           validate_cert=validate_server_cert,
-                          headers=headers
+                          headers=_api_headers(access_token)
                           )
         resp = yield http_client.fetch(req)
         resp_json = json.loads(resp.body.decode('utf8', 'replace'))
 
-        return resp_json["username"]
+        username = resp_json["username"]
+        user_id = resp_json["id"]
+        is_admin = resp_json["is_admin"]
+
+        # Check if user is a member of any whitelisted organizations.
+        # This check is performed here, as it requires `access_token`.
+        if self.gitlab_group_whitelist:
+            user_in_group = yield self._check_group_whitelist(
+                username, user_id, is_admin, access_token)
+            return username if user_in_group else None
+        else:  # no organization whitelisting
+            return username
+
+
+    @gen.coroutine
+    def _check_group_whitelist(self, username, user_id, is_admin, access_token):
+        http_client = AsyncHTTPClient()
+        headers = _api_headers(access_token)
+        if is_admin:
+            # For admins, /groups returns *all* groups. As a workaround
+            # we check if we are a member of each group in the whitelist
+            for group in map(url_escape, self.gitlab_group_whitelist):
+                url = "%s/groups/%s/members/%d" % (GITLAB_API, group, user_id)
+                req = HTTPRequest(url, method="GET", headers=headers)
+                resp = yield http_client.fetch(req, raise_error=False)
+                if resp.code == 200:
+                    return True  # user _is_ in group
+        else:
+            # For regular users we get all the groups to which they have access
+            # and check if any of these are in the whitelisted groups
+            next_page = url_concat("%s/groups" % GITLAB_API,
+                                   dict(all_available=True))
+            while next_page:
+                req = HTTPRequest(next_page, method="GET", headers=headers)
+                resp = yield http_client.fetch(req)
+                resp_json = json.loads(resp.body.decode('utf8', 'replace'))
+                next_page = _get_next_page(resp)
+                user_groups = set(entry["path"] for entry in resp_json)
+                # check if any of the organizations seen thus far are in whitelist
+                if len(self.gitlab_group_whitelist & user_groups) > 0:
+                    return True
+            return False
+
 
 
 class LocalGitLabOAuthenticator(LocalAuthenticator, GitLabOAuthenticator):


Test patch:
diff --git a/oauthenticator/tests/test_bitbucket.py b/oauthenticator/tests/test_bitbucket.py
index 8537e3f..764453c 100644
--- a/oauthenticator/tests/test_bitbucket.py
+++ b/oauthenticator/tests/test_bitbucket.py
@@ -41,7 +41,7 @@ def test_no_code(bitbucket_client):
 def test_team_whitelist(bitbucket_client):
     client = bitbucket_client
     authenticator = BitbucketOAuthenticator()
-    authenticator.team_whitelist = ['blue']
+    authenticator.bitbucket_team_whitelist = ['blue']
 
     teams = {
         'red': ['grif', 'simmons', 'donut', 'sarge', 'lopez'],
@@ -80,7 +80,3 @@ def test_team_whitelist(bitbucket_client):
     handler = client.handler_for_user(user_model('donut'))
     name = yield authenticator.authenticate(handler)
     assert name == 'donut'
-
-
-
-    
\ No newline at end of file
diff --git a/oauthenticator/tests/test_github.py b/oauthenticator/tests/test_github.py
index 247b710..7979b9c 100644
--- a/oauthenticator/tests/test_github.py
+++ b/oauthenticator/tests/test_github.py
@@ -1,4 +1,12 @@
+import re
+import functools
+import json
+from io import BytesIO
+
 from pytest import fixture, mark
+from urllib.parse import urlparse, parse_qs
+from tornado.httpclient import HTTPRequest, HTTPResponse
+from tornado.httputil import HTTPHeaders
 
 from ..github import GitHubOAuthenticator
 
@@ -33,3 +41,84 @@ def test_github(github_client):
 @mark.gen_test
 def test_no_code(github_client):
     yield no_code_test(GitHubOAuthenticator())
+
+
+def make_link_header(urlinfo, page):
+    return {'Link': '<{}://{}{}?page={}>;rel="next"'
+                    .format(urlinfo.scheme, urlinfo.netloc, urlinfo.path, page)}
+
+
+@mark.gen_test
+def test_org_whitelist(github_client):
+    client = github_client
+    authenticator = GitHubOAuthenticator()
+
+    ## Mock Github API
+
+    teams = {
+        'red': ['grif', 'simmons', 'donut', 'sarge', 'lopez'],
+        'blue': ['tucker', 'caboose', 'burns', 'sheila', 'texas'],
+    }
+
+    member_regex = re.compile(r'/orgs/(.*)/members')
+
+    def team_members(paginate, request):
+        urlinfo = urlparse(request.url)
+        team = member_regex.match(urlinfo.path).group(1)
+
+        if team not in teams:
+            return HTTPResponse(400, request)
+
+        if not paginate:
+            return [user_model(m) for m in teams[team]]
+        else:
+            page = parse_qs(urlinfo.query).get('page', ['1'])
+            page = int(page[0])
+            return team_members_paginated(
+                team, page, urlinfo, functools.partial(HTTPResponse, request))
+
+    def team_members_paginated(team, page, urlinfo, response):
+        if page < len(teams[team]):
+            headers = make_link_header(urlinfo, page + 1)
+        elif page == len(teams[team]):
+            headers = {}
+        else:
+            return response(400)
+
+        headers.update({'Content-Type': 'application/json'})
+
+        ret = [user_model(teams[team][page - 1])]
+
+        return response(200,
+                        headers=HTTPHeaders(headers),
+                        buffer=BytesIO(json.dumps(ret).encode('utf-8')))
+
+    ## Perform tests
+
+    for paginate in (False, True):
+        client.hosts['api.github.com'].append(
+            (member_regex, functools.partial(team_members, paginate)),
+        )
+
+        authenticator.github_organization_whitelist = ['blue']
+
+        handler = client.handler_for_user(user_model('caboose'))
+        name = yield authenticator.authenticate(handler)
+        assert name == 'caboose'
+
+        handler = client.handler_for_user(user_model('donut'))
+        name = yield authenticator.authenticate(handler)
+        assert name is None
+
+        # reverse it, just to be safe
+        authenticator.github_organization_whitelist = ['red']
+
+        handler = client.handler_for_user(user_model('caboose'))
+        name = yield authenticator.authenticate(handler)
+        assert name is None
+
+        handler = client.handler_for_user(user_model('donut'))
+        name = yield authenticator.authenticate(handler)
+        assert name == 'donut'
+
+        client.hosts['api.github.com'].pop()
diff --git a/oauthenticator/tests/test_gitlab.py b/oauthenticator/tests/test_gitlab.py
index 7ae9aea..2c4d09d 100644
--- a/oauthenticator/tests/test_gitlab.py
+++ b/oauthenticator/tests/test_gitlab.py
@@ -1,3 +1,12 @@
+import re
+import json
+from io import BytesIO
+import functools
+import collections
+from urllib.parse import urlparse, parse_qs
+
+from tornado.httpclient import HTTPResponse
+from tornado.httputil import HTTPHeaders
 from pytest import fixture, mark
 
 from ..gitlab import GitLabOAuthenticator
@@ -5,10 +14,12 @@ from ..gitlab import GitLabOAuthenticator
 from .mocks import setup_oauth_mock, no_code_test
 
 
-def user_model(username):
+def user_model(username, id=1, is_admin=False):
     """Return a user model"""
     return {
         'username': username,
+        'id': id,
+        'is_admin': is_admin
     }
 
 @fixture
@@ -32,3 +43,109 @@ def test_gitlab(gitlab_client):
 @mark.gen_test
 def test_no_code(gitlab_client):
     yield no_code_test(GitLabOAuthenticator())
+
+
+def make_link_header(urlinfo, page):
+    return {'Link': '<{}://{}{}?page={}>;rel="next"'
+                    .format(urlinfo.scheme, urlinfo.netloc, urlinfo.path, page)}
+
+@mark.gen_test
+def test_group_whitelist(gitlab_client):
+    client = gitlab_client
+    authenticator = GitLabOAuthenticator()
+
+    ## set up fake Gitlab API
+
+    user_groups = collections.OrderedDict({
+        'grif': ['red', 'yellow'],
+        'simmons': ['red', 'yellow'],
+        'caboose': ['blue', 'yellow'],
+        'burns': ['blue', 'yellow'],
+    })
+
+    def user_model(username, is_admin=False):
+        return {
+            'username': username,
+            'id': list(user_groups.keys()).index(username) + 1,
+            'is_admin': is_admin
+        }
+
+
+    member_regex = re.compile(r'/api/v3/groups/(.*)/members/(.*)')
+    def is_member(request):
+        urlinfo = urlparse(request.url)
+        group, uid = member_regex.match(urlinfo.path).group(1, 2)
+        uname = list(user_groups.keys())[int(uid) - 1]
+        if group in user_groups[uname]:
+            return HTTPResponse(request, 200)
+        else:
+            return HTTPResponse(request, 404)
+
+    def groups(paginate, request):
+        urlinfo = urlparse(request.url)
+        _, token = request._headers.get('Authorization').split()
+        user = client.access_tokens[token]['username']
+        if not paginate:
+            return [{'path': group} for group in user_groups[user]]
+        else:
+            page = parse_qs(urlinfo.query).get('page', ['1'])
+            page = int(page[0])
+            return groups_paginated(user, page, urlinfo,
+                                    functools.partial(HTTPResponse, request))
+
+    def groups_paginated(user, page, urlinfo, response):
+        if page < len(user_groups[user]):
+            headers = make_link_header(urlinfo, page + 1)
+        elif page == len(user_groups[user]):
+            headers = {}
+        else:
+            return response(400)
+
+        headers.update({'Content-Type': 'application/json'})
+
+        ret = [{'path': user_groups[user][page - 1]}]
+
+        return response(200, headers=HTTPHeaders(headers),
+                        buffer=BytesIO(json.dumps(ret).encode('utf-8')))
+
+    client.hosts['gitlab.com'].append(
+        (member_regex, is_member)
+    )
+
+    ## actual tests
+
+    for paginate in (False, True):
+        client.hosts['gitlab.com'].append(
+            ('/api/v3/groups', functools.partial(groups, paginate))
+        )
+
+        authenticator.gitlab_group_whitelist = ['blue']
+
+        handler = client.handler_for_user(user_model('caboose'))
+        name = yield authenticator.authenticate(handler)
+        assert name == 'caboose'
+
+        handler = client.handler_for_user(user_model('burns', is_admin=True))
+        name = yield authenticator.authenticate(handler)
+        assert name == 'burns'
+
+        handler = client.handler_for_user(user_model('grif'))
+        name = yield authenticator.authenticate(handler)
+        assert name is None
+
+        handler = client.handler_for_user(user_model('simmons', is_admin=True))
+        name = yield authenticator.authenticate(handler)
+        assert name is None
+
+        # reverse it, just to be safe
+        authenticator.gitlab_group_whitelist = ['red']
+
+        handler = client.handler_for_user(user_model('caboose'))
+        name = yield authenticator.authenticate(handler)
+        assert name is None
+
+        handler = client.handler_for_user(user_model('grif'))
+        name = yield authenticator.authenticate(handler)
+        assert name == 'grif'
+
+        client.hosts['gitlab.com'].pop()