Repository: gratipay/gratipay.com

Issue:
move caching out of memory
There's one or two places in the app where we have an in-memory cache that will prevent us from scaling. I believe the two places are:
- /on/twitter/ - we store state in memory during the round-trip to Twitter for oauth
- /on/confirm.html - something about "connect tokens"

move caching out of memory
There's one or two places in the app where we have an in-memory cache that will prevent us from scaling. I believe the two places are:
- /on/twitter/ - we store state in memory during the round-trip to Twitter for oauth
- /on/confirm.html - something about "connect tokens"



Discussion:
/on/bitbucket also uses website.oauth_cache. I am very curious how it is that our tie to Github avoids this. Someday I'll have to learn our elsewhere related code.

Let's just hope that #505 (#1369) will make these go away :)

[IRC](https://botbot.me/freenode/gittip/msg/8152390/)

@whit537 do we currently have a mechanism to store a secure cookie in the browser?

@seanlinsley Not really?

I was just peaking at the oauth_cache stuff. We send a bunch of cookies already. I'm not quite sure what the sequence of events is where we need the cache, but it appears to be just hanging on to a little bit of data during the out-and-back to the "elsewhere" OAuth service. It's the sort of thing we should be able to do with a cookie, maybe even a short-lived cookie. We do need to be careful that we don't create a security hole.

Yeah I've been re-learning how OAuth works to see what exactly is needed at each stage.

@seanlinsley @bruceadams Would you two be willing to own this ticket and land it before the retreat (Jan 3)?

@whit537 yes!

of course :)

Awesome! Thank you! :-)

:heart: :rocket:

Looks like i cannot assign this to two people. I assigned it to me, mostly to show my focus. I don't intend to show exclusive ownership.

Thinking and looking at this a bit more. We can make it work with a cookie, but I still don't know what the security implications are there. We could do server-side caching into the database instead into memory. That keeps security clean (as least as clean as it is now). We'd want to keep whatever little cache table we setup in the database clean. Each secret we need to hang on to is very short lived, no more than a few minutes, typically just a few seconds.

@bruceadams Let's understand what exactly it is we're storing. If I remember right it's only a redirect URL or something non-sensitive like that.

... in which case storing it in a non-secure cookie would be fine. Even if it's sensitive I like the idea of storing it in a secure cookie somehow instead of cluttering up the database.

I don't know how OAuth works but is this thing that we store in memory secret from the user? If it is not sensitive we can just plug it to the url as a param and be done.

@zwn For GitHub I was able to pass the data through the URL, for Twitter I wasn't for some reason.

Obviously if that can be figured out it's preferable.

I certainly agree with both of that carrying it along in the URL or using a cookie is preferable to using the database (or any other server-side mechanism) so long as we don't mess up security. I'm (slowly) learning more about OAuth.

stealing this from @bruceadams :octocat: ([IRC](https://botbot.me/freenode/gittip/msg/8885539/))

@seanlinsley Looking at Huboard, I see this in Ready to Start with your name on it. I believe I closed the item you had under Work in Progress (#986), as discussed at the retreat (apologies). Does that mean you'll be going here next, or over to #986? As mentioned on #986, I think we may be able take care of that with a simple configuration change.

@seanlinsley I guess what I'm saying is that I'd love to see this ticket get done. :-)

Sure, this can be the next thing I do.

Thanks @seanlinsley! :-)

I've nominated this to the 'DevX ★' label.

We now have the same issue with OpenStreetMap #1848. I merged it anyway thinking once we know what to do we will just do it for both. 

I've been trying to read up on OAuth. Could the solution be just to cache the `oauth_token_secret` in the database? I am getting the feeling that this might be done just once per application and not for each sign in. The other stuff could be just passed around in some nonsafe way since it is not security related (`action` and `then`).

@seanlinsley I'd like to be your coworker on this one. Feel free to ping me.

Check out http://requests-oauthlib.readthedocs.org/en/latest/examples/bitbucket.html. It seems that no state is carried from step 3 to step 4. The only thing there is the OAuth1Session object that is constructed from a static data.

OAuth is so hard to grok :sweat:. Now I think that request token secret really needs to be stored securely on our side. I'd do this in the db for now, since we do not have any memcached or redis going but I'd model the schema like we had (just creating a `cache` table with a `key` and `data`, the data possibly being `hstore` type?).

Maybe we could use [tokenlib](https://github.com/mozilla-services/tokenlib) to store the token secret in a cookie or redirect url? I am afraid there is still something I am not getting about OAuth1. Why would someone design a protocol that forces the use of server side permanent storage if it could be done without it?

Nice overview for Twitter OAuth1 flow https://github.com/inueni/birdy#great-what-about-authorization-how-do-i-get-my-access-tokens that is somewhat readable (the best I have found so far).

It seems there is really no way to add custom params to any of the requests (nowhere to hide even encrypted token_secret). So secure cookie with tokenlib or cache in db are the two available options. I am reasonably sure there are no other.

I was looking into using tokenlib for this, but I can't make sense of the code as it is. I'm biting the bullet, diving in to #1369

It's okay to store `oauth_token` and the related thing from OAuth2 in an unsecured cookie ([IRC](https://botbot.me/freenode/gittip/msg/11473794/)). On #1369 we're doing this for the oauth caches, but for this ticket we still need to address the merge confirmation workflow.

/on/bitbucket also uses website.oauth_cache. I am very curious how it is that our tie to Github avoids this. Someday I'll have to learn our elsewhere related code.

Let's just hope that #505 (#1369) will make these go away :)

[IRC](https://botbot.me/freenode/gittip/msg/8152390/)

@whit537 do we currently have a mechanism to store a secure cookie in the browser?

@seanlinsley Not really?

I was just peaking at the oauth_cache stuff. We send a bunch of cookies already. I'm not quite sure what the sequence of events is where we need the cache, but it appears to be just hanging on to a little bit of data during the out-and-back to the "elsewhere" OAuth service. It's the sort of thing we should be able to do with a cookie, maybe even a short-lived cookie. We do need to be careful that we don't create a security hole.

Yeah I've been re-learning how OAuth works to see what exactly is needed at each stage.

@seanlinsley @bruceadams Would you two be willing to own this ticket and land it before the retreat (Jan 3)?

@whit537 yes!

of course :)

Awesome! Thank you! :-)

:heart: :rocket:

Looks like i cannot assign this to two people. I assigned it to me, mostly to show my focus. I don't intend to show exclusive ownership.

Thinking and looking at this a bit more. We can make it work with a cookie, but I still don't know what the security implications are there. We could do server-side caching into the database instead into memory. That keeps security clean (as least as clean as it is now). We'd want to keep whatever little cache table we setup in the database clean. Each secret we need to hang on to is very short lived, no more than a few minutes, typically just a few seconds.

@bruceadams Let's understand what exactly it is we're storing. If I remember right it's only a redirect URL or something non-sensitive like that.

... in which case storing it in a non-secure cookie would be fine. Even if it's sensitive I like the idea of storing it in a secure cookie somehow instead of cluttering up the database.

I don't know how OAuth works but is this thing that we store in memory secret from the user? If it is not sensitive we can just plug it to the url as a param and be done.

@zwn For GitHub I was able to pass the data through the URL, for Twitter I wasn't for some reason.

Obviously if that can be figured out it's preferable.

I certainly agree with both of that carrying it along in the URL or using a cookie is preferable to using the database (or any other server-side mechanism) so long as we don't mess up security. I'm (slowly) learning more about OAuth.

stealing this from @bruceadams :octocat: ([IRC](https://botbot.me/freenode/gittip/msg/8885539/))

@seanlinsley Looking at Huboard, I see this in Ready to Start with your name on it. I believe I closed the item you had under Work in Progress (#986), as discussed at the retreat (apologies). Does that mean you'll be going here next, or over to #986? As mentioned on #986, I think we may be able take care of that with a simple configuration change.

@seanlinsley I guess what I'm saying is that I'd love to see this ticket get done. :-)

Sure, this can be the next thing I do.

Thanks @seanlinsley! :-)

I've nominated this to the 'DevX ★' label.

We now have the same issue with OpenStreetMap #1848. I merged it anyway thinking once we know what to do we will just do it for both. 

I've been trying to read up on OAuth. Could the solution be just to cache the `oauth_token_secret` in the database? I am getting the feeling that this might be done just once per application and not for each sign in. The other stuff could be just passed around in some nonsafe way since it is not security related (`action` and `then`).

@seanlinsley I'd like to be your coworker on this one. Feel free to ping me.

Check out http://requests-oauthlib.readthedocs.org/en/latest/examples/bitbucket.html. It seems that no state is carried from step 3 to step 4. The only thing there is the OAuth1Session object that is constructed from a static data.

OAuth is so hard to grok :sweat:. Now I think that request token secret really needs to be stored securely on our side. I'd do this in the db for now, since we do not have any memcached or redis going but I'd model the schema like we had (just creating a `cache` table with a `key` and `data`, the data possibly being `hstore` type?).

Maybe we could use [tokenlib](https://github.com/mozilla-services/tokenlib) to store the token secret in a cookie or redirect url? I am afraid there is still something I am not getting about OAuth1. Why would someone design a protocol that forces the use of server side permanent storage if it could be done without it?

Nice overview for Twitter OAuth1 flow https://github.com/inueni/birdy#great-what-about-authorization-how-do-i-get-my-access-tokens that is somewhat readable (the best I have found so far).

It seems there is really no way to add custom params to any of the requests (nowhere to hide even encrypted token_secret). So secure cookie with tokenlib or cache in db are the two available options. I am reasonably sure there are no other.

I was looking into using tokenlib for this, but I can't make sense of the code as it is. I'm biting the bullet, diving in to #1369

It's okay to store `oauth_token` and the related thing from OAuth2 in an unsecured cookie ([IRC](https://botbot.me/freenode/gittip/msg/11473794/)). On #1369 we're doing this for the oauth caches, but for this ticket we still need to address the merge confirmation workflow.


Resolving patch:
diff --git a/gratipay/models/account_elsewhere.py b/gratipay/models/account_elsewhere.py
--- a/gratipay/models/account_elsewhere.py
+++ b/gratipay/models/account_elsewhere.py
@@ -1,19 +1,26 @@
 from __future__ import absolute_import, division, print_function, unicode_literals
 
+from datetime import timedelta
 import json
 from urlparse import urlsplit, urlunsplit
+import uuid
 import xml.etree.ElementTree as ET
 
 from aspen import Response
+from aspen.utils import utcnow
 from postgres.orm import Model
 from psycopg2 import IntegrityError
 import xmltodict
 
 import gratipay
 from gratipay.exceptions import ProblemChangingUsername
+from gratipay.security.crypto import constant_time_compare
 from gratipay.utils.username import safely_reserve_a_username
 
 
+CONNECT_TOKEN_TIMEOUT = timedelta(hours=24)
+
+
 class UnknownAccountElsewhere(Exception): pass
 
 
@@ -29,6 +36,16 @@ def __init__(self, *args, **kwargs):
     # Constructors
     # ============
 
+    @classmethod
+    def from_id(cls, id):
+        """Return an existing AccountElsewhere based on id.
+        """
+        return cls.db.one("""
+            SELECT elsewhere.*::elsewhere_with_participant
+              FROM elsewhere
+             WHERE id = %s
+        """, (id,))
+
     @classmethod
     def from_user_id(cls, platform, user_id):
         """Return an existing AccountElsewhere based on platform and user_id.
@@ -128,6 +145,31 @@ def upsert(cls, i):
         return account
 
 
+    # Connect tokens
+    # ==============
+
+    def check_connect_token(self, token):
+        return (
+            self.connect_token and
+            constant_time_compare(self.connect_token, token) and
+            self.connect_expires > utcnow()
+        )
+
+    def make_connect_token(self):
+        token = uuid.uuid4().hex
+        expires = utcnow() + CONNECT_TOKEN_TIMEOUT
+        return self.save_connect_token(token, expires)
+
+    def save_connect_token(self, token, expires):
+        return self.db.one("""
+            UPDATE elsewhere
+               SET connect_token = %s
+                 , connect_expires = %s
+             WHERE id = %s
+         RETURNING connect_token, connect_expires
+        """, (token, expires, self.id))
+
+
     # Random Stuff
     # ============
 


Test patch:
diff --git a/tests/py/test_elsewhere.py b/tests/py/test_elsewhere.py
--- a/tests/py/test_elsewhere.py
+++ b/tests/py/test_elsewhere.py
@@ -4,6 +4,7 @@
 from base64 import b64encode
 
 import mock
+
 from gratipay.elsewhere import UserInfo
 from gratipay.models.account_elsewhere import AccountElsewhere
 from gratipay.models.participant import Participant
@@ -55,9 +56,6 @@ def test_logging_in_doesnt_reset_goal(self):
         alice = AccountElsewhere.from_user_name('twitter', 'alice').opt_in('alice')[0].participant
         assert alice.goal == 100
 
-    def test_hitting_confirm_plain_results_in_404(self):
-        assert self.client.GxT('/on/confirm.html').code == 404
-
     @mock.patch('requests_oauthlib.OAuth2Session.fetch_token')
     @mock.patch('gratipay.elsewhere.Platform.get_user_self_info')
     @mock.patch('gratipay.elsewhere.Platform.get_user_info')
@@ -73,8 +71,8 @@ def test_connect_might_need_confirmation(self, gui, gusi, ft):
         response = self.client.GxT('/on/github/associate?state=deadbeef',
                                    auth_as='alice',
                                    cookies={b'github_deadbeef': cookie})
-        assert response.code == 200
-        assert "Please Confirm" in response.body
+        assert response.code == 302
+        assert response.headers['Location'].startswith('/on/confirm.html?id=')
 
     def test_redirect_csrf(self):
         response = self.client.GxT('/on/github/redirect')
@@ -135,3 +133,42 @@ def test_failure_page_accepts_valid_username(self):
         self.client.GET('/on/twitter/Gratipay/')  # normal case will have the db primed
         response = self.client.GET('/on/twitter/Gratipay/failure.html?action')
         assert response.code == 200
+
+
+class TestConfirmTakeOver(Harness):
+
+    def setUp(self):
+        Harness.setUp(self)
+        self.alice_elsewhere = self.make_elsewhere('twitter', -1, 'alice')
+        token, expires = self.alice_elsewhere.make_connect_token()
+        self.connect_cookie = {b'connect_%s' % self.alice_elsewhere.id: token}
+        self.bob = self.make_participant('bob', claimed_time='now')
+
+    def test_confirm(self):
+        url = '/on/confirm.html?id=%s' % self.alice_elsewhere.id
+
+        response = self.client.GxT(url)
+        assert response.code == 403
+
+        response = self.client.GxT(url, auth_as='bob')
+        assert response.code == 400
+        assert 'bad connect token' in response.body
+
+        response = self.client.GET(url, auth_as='bob', cookies=self.connect_cookie)
+        assert response.code == 200
+        assert 'Please Confirm' in response.body
+
+    def test_take_over(self):
+        data = {'account_id': self.alice_elsewhere.id, 'should_transfer': 'yes'}
+
+        response = self.client.PxST('/on/take-over.html', data=data)
+        assert response.code == 403
+
+        response = self.client.PxST('/on/take-over.html', data=data, auth_as='bob')
+        assert response.code == 400
+        assert 'bad connect token' in response.body
+
+        response = self.client.PxST('/on/take-over.html', data=data, auth_as='bob',
+                                    cookies=self.connect_cookie)
+        assert response.code == 302
+        assert response.headers['Location'] == '/bob/'