Repository: gratipay/gratipay.com

Issue:
someone managed to orphan their participant!
Check this out:

https://www.gittip.com/c9b2c1e22b5d/

I discovered this by expanding the leaderboard, like so:

https://www.gittip.com/?limit=20

Investigate!

someone managed to orphan their participant!
Check this out:

https://www.gittip.com/c9b2c1e22b5d/

I discovered this by expanding the leaderboard, like so:

https://www.gittip.com/?limit=20

Investigate!



Discussion:
Another one:

https://www.gittip.com/d6b62b6b95c9/

https://www.gittip.com/?limit=1000 

woooooahhh. Is there potential to make this feature a drop down menu option as a sort of primitive search engine?

@abnor Oooh! That's neat with the new design, good find. :)

BTW, what's your Twitter?

I do not have a Twitter... Should I?

@abnor Yes. :)

Okayyyy

https://twitter.com/LyndyPalmer

https://www.gittip.com/99df04871a57/

> I am making the world better by actually being https://www.gittip.com/whit537/.

https://www.gittip.com/6a2ab9485317/

wtf?

https://www.gittip.com/bf05b6fe935e/

https://www.gittip.com/dd36ea2607b2/

https://www.gittip.com/1a36b29a7161/

+1 from @mvdkleijn on [Twitter](https://twitter.com/mvdkleijn/status/375261191133593600).

Solution seems simple to me... delete (on a regular basis) all profiles that don't have any accounts connected and don't have any bank or credit card account connected.

Also: are we sure these are accidentally orphaned accounts or could these be some form of spam?

Note: https://www.gittip.com/1a36b29a7161/ has a credit card connected... 

@mvdkleijn The db repair is straightforward, yes. The hard part is understanding how these happen in the first place and closing any loopholes.

@whit537 I’ll see if I can run / think of some test scenarios to replicate the problem. If any orphaned accounts have funds escrowed with us we'll need to figure out what to do with that.

Possibly related to how Twitter handles accounts (and deactivation)? See screenshot in #54 and the remark by @sprice at https://github.com/gittip/www.gittip.com/issues/54#issuecomment-24417209

https://www.gittip.com/b0ee8f2a621b/ (#1509)

I am looking into this issue (ref. #1549, #1598, #650). Right now we have 559 orphaned accounts.

The code in question is in https://github.com/gittip/www.gittip.com/blob/master/gittip/models/_mixin_elsewhere.py#L119. The orphaned participant [is just renamed to a random username](https://github.com/gittip/www.gittip.com/blob/master/gittip/models/_mixin_elsewhere.py#L341) and kept in the participants table. Since I do not know the history of the code I don't know if it has ever worked differently.

If this is the correct way to count orphaned participants

``` sql
select count(*) from participants where not exists (select * from elsewhere where elsewhere.participant=username);
```

then we actually have 6182 of them (and not 559) out of almost 50k. The `absorptions` table has 6169 rows so we have 13 unacounted orphans that I am not sure where they come from or what to do with them.
https://dataclips.heroku.com/buljjtubfplplynvcchxsdsqewyf
One of them has been created just a couple hours ago. Interesting about them is that `ctime` and `session_expires` are identical.

Steps to create orphaned participant:
1. start with empty db (`make schema`)
2. visit /on/[platform]/[name] of at least 2 of your identities - creates 2 participants with random usernames
3. sign in with one - renames one of the participants to a [name]
4. connect the other identity to the currently signed in participant - creates entry in `absorptions` but leaves renamed orphaned participant behind

@whit537 Why do we keep orphans around? Is there some danger in deleting the orphans that have corresponding entry in `absorptions`? Do you rework this part of the code in the `elsewhere` branch or can/should this be a separate fix?

The query to show the unexplained orphans:

``` sql
select username, ctime, claimed_time, session_expires
   from participants
  where not exists (select * from elsewhere where elsewhere.participant=username)
    and not exists (select * from absorptions where archived_as=username)
order by ctime desc;
```

Dropping from Infrastructure per [IRC](https://botbot.me/freenode/gittip/msg/7109990/).

I decided to revisit this issue now when we fixed #1704. The accounts that were involved in manifestation of that bug are:

```
c9b2c1e22b5d - first reported
a5ad0ebe3dff
182eed83c351
4c074000c7bc
187cb554c30a
bb670f195cd3
43cafe74bc45
81d042613b23
```

I have run github searches for all of them but except the first one, none were involved in any bugs.

I am closing this issue on the grounds that keeping orphans in the `participants` table is "by design" so there is nothing surprising that we have them. On the other hand orphans should not be involved in any part of the site and if they are let's track that as a separate issue. Should anyone feel differently, please reopen.

Oh well, that was too quick. There are still unexplained orphans from https://github.com/gittip/www.gittip.com/issues/617#issuecomment-26659054. 

Continuing in #1768.

I have a new theory on where the unexplained orphans might be coming from. Recently we got a new orphan #1856. It has the same `ctime` as `session_expire` - both default to `now` in Postgres. This likely means that the account has been created but never used for anything. The only 2 places where we create accounts are

[gittip/elsewhere/__init__.py#L97](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/elsewhere/__init__.py#L97)
[gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327)

Both places immediately attach `elsewhere` account. Both are run in a transaction that should either commit both or rollback both. The catch might be the transaction isolation level http://www.postgresql.org/docs/9.1/static/transaction-iso.html. Default level is _read committed_. The higher levels are _repeatable read_ and _serializable_. 

When the site is under a stress (read _slow_) user might hit refresh in the take over page trying to get it working. It does not cancel the currently running request handling and starts another one with the same data. Both transaction can get to a point [gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327) where they would be wrestling with trying to attach a single account elsewhere to two newly created participants, obviously only one of them winning.

I'll try to prepare a simplified example to illustrate the situation and actually confirm if something like that is possible and if it gets fixed by using higher levels of transaction isolation.

Why do we actually do [gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327)? Why don't we just dump the account elsewhere? We are not using it for anything (not attaching any information to it). If the proper '/on/...' url gets visited the elsewhere account gets recreated if missing anyway. @whit537 anything I am missing here? I love deleting code :stuck_out_tongue:

To recreate the problem by hand, follow the these steps:
1. `make schema`
2. create participant 'a'
3. attach elsewhere platform='g', user_id='a'
4. run fist psql session - start transaction, create participant 'a1'
5. run second psql session - start transaction, create participant 'a2'
6. in first session run - `update elsewhere set participant='a1' where platform='g' and participant='a'` - this will update 1 row - commit
7. in second session try to do the same for 'a2' - update will not match any rows (`where participant='a'`) because first transaction is already commited, thus creating an orphan

Which is what is happening. If we were to upgrade the isolation level at least to 'repeatable read' we would not have this problem because the whole transaction would get snapshot of the db at the beginning of the transaction (update would find the row to update, which would conflict and raise error). To be able to upgrade the isolation level of the transactions we need to be able to deal with serialization errors. Also further (deeper) thought needs to go into API review - for example current take_over function if called after serialization failure would unexpectedly undo itself. 

TL DR; 
- We need to increase transaction isolation (possible in the whole app). 
- To be able to do that, all places need to be able to deal with serialization errors. 
- To be able to do _that_, the parts of code we rerun need to be [idempotent](http://en.wikipedia.org/wiki/Idempotent).

Looking at papertrail what was happening at the time the orphan was created I have only one relevant line which seems to point to the `take_over` direction.

```
Jan 09 12:16:55 gittip heroku/router:  at=info method=POST path=/on/take-over.html host=www.gittip.com fwd="x.x.x.x" dyno=web.1 connect=1ms service=40ms status=302 bytes=0 
```

Another one:

https://www.gittip.com/d6b62b6b95c9/

https://www.gittip.com/?limit=1000 

woooooahhh. Is there potential to make this feature a drop down menu option as a sort of primitive search engine?

@abnor Oooh! That's neat with the new design, good find. :)

BTW, what's your Twitter?

I do not have a Twitter... Should I?

@abnor Yes. :)

Okayyyy

https://twitter.com/LyndyPalmer

https://www.gittip.com/99df04871a57/

> I am making the world better by actually being https://www.gittip.com/whit537/.

https://www.gittip.com/6a2ab9485317/

wtf?

https://www.gittip.com/bf05b6fe935e/

https://www.gittip.com/dd36ea2607b2/

https://www.gittip.com/1a36b29a7161/

+1 from @mvdkleijn on [Twitter](https://twitter.com/mvdkleijn/status/375261191133593600).

Solution seems simple to me... delete (on a regular basis) all profiles that don't have any accounts connected and don't have any bank or credit card account connected.

Also: are we sure these are accidentally orphaned accounts or could these be some form of spam?

Note: https://www.gittip.com/1a36b29a7161/ has a credit card connected... 

@mvdkleijn The db repair is straightforward, yes. The hard part is understanding how these happen in the first place and closing any loopholes.

@whit537 I’ll see if I can run / think of some test scenarios to replicate the problem. If any orphaned accounts have funds escrowed with us we'll need to figure out what to do with that.

Possibly related to how Twitter handles accounts (and deactivation)? See screenshot in #54 and the remark by @sprice at https://github.com/gittip/www.gittip.com/issues/54#issuecomment-24417209

https://www.gittip.com/b0ee8f2a621b/ (#1509)

I am looking into this issue (ref. #1549, #1598, #650). Right now we have 559 orphaned accounts.

The code in question is in https://github.com/gittip/www.gittip.com/blob/master/gittip/models/_mixin_elsewhere.py#L119. The orphaned participant [is just renamed to a random username](https://github.com/gittip/www.gittip.com/blob/master/gittip/models/_mixin_elsewhere.py#L341) and kept in the participants table. Since I do not know the history of the code I don't know if it has ever worked differently.

If this is the correct way to count orphaned participants

``` sql
select count(*) from participants where not exists (select * from elsewhere where elsewhere.participant=username);
```

then we actually have 6182 of them (and not 559) out of almost 50k. The `absorptions` table has 6169 rows so we have 13 unacounted orphans that I am not sure where they come from or what to do with them.
https://dataclips.heroku.com/buljjtubfplplynvcchxsdsqewyf
One of them has been created just a couple hours ago. Interesting about them is that `ctime` and `session_expires` are identical.

Steps to create orphaned participant:
1. start with empty db (`make schema`)
2. visit /on/[platform]/[name] of at least 2 of your identities - creates 2 participants with random usernames
3. sign in with one - renames one of the participants to a [name]
4. connect the other identity to the currently signed in participant - creates entry in `absorptions` but leaves renamed orphaned participant behind

@whit537 Why do we keep orphans around? Is there some danger in deleting the orphans that have corresponding entry in `absorptions`? Do you rework this part of the code in the `elsewhere` branch or can/should this be a separate fix?

The query to show the unexplained orphans:

``` sql
select username, ctime, claimed_time, session_expires
   from participants
  where not exists (select * from elsewhere where elsewhere.participant=username)
    and not exists (select * from absorptions where archived_as=username)
order by ctime desc;
```

Dropping from Infrastructure per [IRC](https://botbot.me/freenode/gittip/msg/7109990/).

I decided to revisit this issue now when we fixed #1704. The accounts that were involved in manifestation of that bug are:

```
c9b2c1e22b5d - first reported
a5ad0ebe3dff
182eed83c351
4c074000c7bc
187cb554c30a
bb670f195cd3
43cafe74bc45
81d042613b23
```

I have run github searches for all of them but except the first one, none were involved in any bugs.

I am closing this issue on the grounds that keeping orphans in the `participants` table is "by design" so there is nothing surprising that we have them. On the other hand orphans should not be involved in any part of the site and if they are let's track that as a separate issue. Should anyone feel differently, please reopen.

Oh well, that was too quick. There are still unexplained orphans from https://github.com/gittip/www.gittip.com/issues/617#issuecomment-26659054. 

Continuing in #1768.

I have a new theory on where the unexplained orphans might be coming from. Recently we got a new orphan #1856. It has the same `ctime` as `session_expire` - both default to `now` in Postgres. This likely means that the account has been created but never used for anything. The only 2 places where we create accounts are

[gittip/elsewhere/__init__.py#L97](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/elsewhere/__init__.py#L97)
[gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327)

Both places immediately attach `elsewhere` account. Both are run in a transaction that should either commit both or rollback both. The catch might be the transaction isolation level http://www.postgresql.org/docs/9.1/static/transaction-iso.html. Default level is _read committed_. The higher levels are _repeatable read_ and _serializable_. 

When the site is under a stress (read _slow_) user might hit refresh in the take over page trying to get it working. It does not cancel the currently running request handling and starts another one with the same data. Both transaction can get to a point [gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327) where they would be wrestling with trying to attach a single account elsewhere to two newly created participants, obviously only one of them winning.

I'll try to prepare a simplified example to illustrate the situation and actually confirm if something like that is possible and if it gets fixed by using higher levels of transaction isolation.

Why do we actually do [gittip/models/_mixin_elsewhere.py#L327](https://github.com/gittip/www.gittip.com/blob/854ccfbba4f52b8a0153b66f9658026d7d318108/gittip/models/_mixin_elsewhere.py#L327)? Why don't we just dump the account elsewhere? We are not using it for anything (not attaching any information to it). If the proper '/on/...' url gets visited the elsewhere account gets recreated if missing anyway. @whit537 anything I am missing here? I love deleting code :stuck_out_tongue:

To recreate the problem by hand, follow the these steps:
1. `make schema`
2. create participant 'a'
3. attach elsewhere platform='g', user_id='a'
4. run fist psql session - start transaction, create participant 'a1'
5. run second psql session - start transaction, create participant 'a2'
6. in first session run - `update elsewhere set participant='a1' where platform='g' and participant='a'` - this will update 1 row - commit
7. in second session try to do the same for 'a2' - update will not match any rows (`where participant='a'`) because first transaction is already commited, thus creating an orphan

Which is what is happening. If we were to upgrade the isolation level at least to 'repeatable read' we would not have this problem because the whole transaction would get snapshot of the db at the beginning of the transaction (update would find the row to update, which would conflict and raise error). To be able to upgrade the isolation level of the transactions we need to be able to deal with serialization errors. Also further (deeper) thought needs to go into API review - for example current take_over function if called after serialization failure would unexpectedly undo itself. 

TL DR; 
- We need to increase transaction isolation (possible in the whole app). 
- To be able to do that, all places need to be able to deal with serialization errors. 
- To be able to do _that_, the parts of code we rerun need to be [idempotent](http://en.wikipedia.org/wiki/Idempotent).

Looking at papertrail what was happening at the time the orphan was created I have only one relevant line which seems to point to the `take_over` direction.

```
Jan 09 12:16:55 gittip heroku/router:  at=info method=POST path=/on/take-over.html host=www.gittip.com fwd="x.x.x.x" dyno=web.1 connect=1ms service=40ms status=302 bytes=0 
```


Resolving patch:
diff --git a/gittip/models/__init__.py b/gittip/models/__init__.py
--- a/gittip/models/__init__.py
+++ b/gittip/models/__init__.py
@@ -108,29 +108,7 @@ def _check_orphans(self):
               where not exists (select * from elsewhere where elsewhere.participant=username)
                 and not exists (select * from absorptions where archived_as=username)
         """)
-        known = set((
-            "4c46cc22afdd",
-            "82b0d81fe9e8",
-            "6b6527ac6c02",
-            "cbde8c31c11c",
-            "bcfc65158eaf",
-            "31d54a0c19ae",
-            "afbddadaac3c",
-            "a78c4e42bb93",
-            "42eb93b3ab89",
-            "b1bc5e47fe8e",
-            "3ac515cc8da6",
-            "14a79340c40d",
-            "14d60c6884e7",
-            "0c783dee50ed",
-            "e2020536ef6d",
-            "60a5099d49c7",
-            "64f4f959b322",
-            "0bdf90d51786",
-            "880ac6873f7f"
-        ))
-        real = set(orphans) - known
-        assert len(real) == 0, "missing elsewheres: {}".format(list(real))
+        assert len(orphans) == 0, "missing elsewheres: {}".format(list(orphans))
 
     def _check_orphans_no_tips(self):
         """
diff --git a/gittip/models/_mixin_elsewhere.py b/gittip/models/_mixin_elsewhere.py
--- a/gittip/models/_mixin_elsewhere.py
+++ b/gittip/models/_mixin_elsewhere.py
@@ -269,6 +269,10 @@ def take_over(self, account_elsewhere, have_confirmation=False):
 
             other_username = rec.participant
 
+            if self.username == other_username:
+                # this is a no op - trying to take over itself
+                return
+
 
             # Make sure we have user confirmation if needed.
             # ==============================================


Test patch:
diff --git a/tests/test_participant.py b/tests/test_participant.py
--- a/tests/test_participant.py
+++ b/tests/test_participant.py
@@ -167,6 +167,14 @@ def test_do_not_take_over_zero_tips_receiving(self):
         assert 2 == ntips
         self.db.self_check()
 
+    def test_idempotent(self):
+        alice = TwitterAccount(self.db, 1, dict(screen_name='alice'))
+        bob   = GitHubAccount(self.db, 2, dict(screen_name='bob'))
+        alice_participant = alice.opt_in('alice')[0].participant
+        alice_participant.take_over(bob, have_confirmation=True)
+        alice_participant.take_over(bob, have_confirmation=True)
+        self.db.self_check()
+
 
 class TestParticipant(Harness):
     def setUp(self):