Repository: gratipay/gratipay.com

Issue:
Failed ACH transfers aren't reflected in user balance
When payday initiates an ACH transfer for a user's balance, that user's balance is immediately reduced by the amount of the transfer. The problem come on the rare occasions where the ACH fails. The only notification we get (that I know about) is an email from Balanced about ACH failures. We need to restore funds to the user's account when the transfer did not, in fact, occur.

Failed ACH transfers aren't reflected in user balance
When payday initiates an ACH transfer for a user's balance, that user's balance is immediately reduced by the amount of the transfer. The problem come on the rare occasions where the ACH fails. The only notification we get (that I know about) is an email from Balanced about ACH failures. We need to restore funds to the user's account when the transfer did not, in fact, occur.



Discussion:
Possible +1 from @gulopine on [Twitter](https://twitter.com/Gulopine/status/423901295523553281).

I hadn't planned on commenting on this, but yes, it would be nice to see the reality reflected in Gittip, so that at least there's an explanation. Crediting the account would be better still, but even just having an explanation would be better than having no idea what happened or who to contact.

Oh, and another problem with this is the email we get from Balanced (in the Gittip support system) requires logging into the Balanced to figure out which Gittip user the failure applied to.

cc: @pjf

@bruceadams I believe Balanced has a webhooks implementation. Can you (or someone else) find out if they have a hook for this use case? If we can get a POST from them to an endpoint back on www.gittip.com, then we stand a chance of being able to automate the correction to the user's balance in Gittip.

@bruceadams As I believe you've noticed, the emails we get from Balanced for outstanding failed credits contain bad links (ticketed as https://github.com/balanced/balanced-dashboard/issues/951). I've figured out steps to go from Balanced's email to a useful page on their dashboard:
1. Copy and paste the reference number for the transaction in question (this is the link text for the link in the email&mdash;don't click on it, highlight and copy it).
2. Go to https://dashboard.balancedpayments.com/.
3. Paste the transaction id into the search box at the very top of the dashboard. It should pop up a link to the transaction.

As you mention, this requires access to Balanced, something that you and @clone1018 and @pjf now have.

https://docs.balancedpayments.com/1.1/api/callbacks/

> Callbacks are a means of registering to receive payloads of information to a specific URL of your choice. You may create multiple callbacks. Events will be sent to each callback URL. In the event that a 20x response is not received when the payload is sent to the callback URL, the callback will be retried up to 10 times with an initial retry delay of 10 minutes which will increase exponentially with each failure for the next 7 days.

How do I get access into Balanced? I am not aware of having credentials for Balanced.

@clone1018 helped me through it. I can get to the Balanced dashboard now. :dancer: 

I've successfully handled the recent _Balanced - Outstanding failed credits_ emails to Support. We have one Gittip user that has had many failures throughout the life of their account. The first failure we saw was December 5, 2013. What I find very surprising is that a few transactions (a Gittip deposit to their bank account) appear to have gone through successfully. How is that possible? Can we ask Balanced to look deeper into the history on that account? How do we do that in a privacy preserving way? (I do not think it is appropriate to name the Gittip user in a public forum.)

> Can we ask Balanced to look deeper into the history on that account? How do we do that in a privacy preserving way?

Email support@balancedpayments.com? Maybe if you initiate that email from inside Freshdesk we can manage the conversation there?

These continue to accumulate. I'm collecting them on [FD373](http://gittip.freshdesk.com/helpdesk/tickets/373).

@cczona Picking up over here from https://github.com/gittip/www.gittip.com/issues/2085#issuecomment-47149686.

@cczona Looking back through the emails from Balanced mentioned above on this ticket, I _am_ seeing you among the failed payouts.

@cczona Ditto CallbackWomen.

Okay, thanks. How do I proceed? I don't have a Balanced account, though am fine with opening one if that solves a problem.

@cczona Since Gittip uses Balanced as it's payment processor, we end up creating an implicit Balanced account for you when you enter financial details. Here's the message we got from Balanced on February 18:

> Greetings, 
> 
> The bank has notified us that the tokenization for bank account [redacted] is no longer valid. In response, we have marked this bank account as invalid. The reason given for this is: 
> 
> Incorrect DFI Account number. 
> 
> The bank account may be found at: [redacted] 
> 
> Please have your customer update their bank account information before attempting to debit or credit them again. 
> 
> Thanks, 
> Balanced

@cczona So I guess let's have you re-enter your bank account details? Sounds like @bruceadams has looked at your case and perhaps double-checked that we haven't corrupted your balance. I'll triple-check ...

I didn't know about this issue, or maybe I just forgot, anyway I've now added it to the list of bugs that the payday rewrite should fix (#2508).

@Changaco Cool, thank you. Isn't the action item here an endpoint for a Balanced webhook? That's out of band for payday itself, right?

@cczona Here's where you can re-enter your bank account details: https://www.gittip.com/bank-account.html.

Both accounts continued to make transfers between Feb 18th through May 15th. So the error message is flawed.

I used that URL last Thursday to add (back) the banking info for CallbackWomen's gittip account.

The cczona gittip account shows that banking is still in place, but I'll go ahead and re-do.

> Both accounts continued to make transfers between Feb 18th through May 15th. So the error message is flawed.

@cczona Good point. Let's look into that ...

> I'll go ahead and re-do.

Thanks. :-)

@whit537 Possibly, but #2508 is where I have my checklist of payday-related bugs to fix, so might as well add it there to make sure I don't forget.

@Changaco Fair enough.

> I used that URL last Thursday to add (back) the banking info for CallbackWomen's gittip account.

@cczona Depending on what time you added that on Thursday we may not have paid out to you during the payday run that day. I'm seeing the same #2085 issue w/ CallbackWomen as w/ cczona. Let me see what CallbackWomen looks like in Balanced ... 

I _do_ see a bank account on file at Balanced for CallbackWomen.

Says it was created at June 20 2014, 3:19 AM, which (assuming UTC and PDT) would be last Thursday at 8:19 PM your time. So that lines up, at least. :-)

@cczona I'm also now seeing that you've re-added a bank account to your cczona account. So we should have what we need to do the payouts. Now let's make sure we've got your balance right on both accounts.

The crucial question is whether the payouts we've recorded in Gittip sync up with the payouts that are recorded in Balanced. If those line up then your balance should be good, and we can get you squared away here, regardless of secondary issues like this present one, #2085, and the "flawed error message" issue [above](https://github.com/gittip/www.gittip.com/issues/1811#issuecomment-47151716).

The payouts for CallbackWomen are the same in both Gittip and Balanced.

The payouts for cczona are the same in both Gittip and Balanced.

@cczona BTW, you should be able to cross-check my result by looking at your Gittip history and your bank statements. Admittedly that's going to be more tedious, however.

Okay! So your Gittip balance is not corrupt, and we now have a bank account on file, for both CallbackWomen and cczona.

@cczona If we do nothing else right now, I would expect your Gittip balance to clear to your bank account tomorrow (less outbound tips). Is that sufficient for today? What else do you need from me to resolve this?

Thanks! I'll check on it tomorrow.

Success! Both accounts resumed transfers this morning. Thank you.

@cczona Yay! Thanks for closing the loop. BTW, this ticket will stay open until we fix the underlying issue, so if you don't want spam down the road feel free to unsubscribe. :-)

Possible +1 from @gulopine on [Twitter](https://twitter.com/Gulopine/status/423901295523553281).

I hadn't planned on commenting on this, but yes, it would be nice to see the reality reflected in Gittip, so that at least there's an explanation. Crediting the account would be better still, but even just having an explanation would be better than having no idea what happened or who to contact.

Oh, and another problem with this is the email we get from Balanced (in the Gittip support system) requires logging into the Balanced to figure out which Gittip user the failure applied to.

cc: @pjf

@bruceadams I believe Balanced has a webhooks implementation. Can you (or someone else) find out if they have a hook for this use case? If we can get a POST from them to an endpoint back on www.gittip.com, then we stand a chance of being able to automate the correction to the user's balance in Gittip.

@bruceadams As I believe you've noticed, the emails we get from Balanced for outstanding failed credits contain bad links (ticketed as https://github.com/balanced/balanced-dashboard/issues/951). I've figured out steps to go from Balanced's email to a useful page on their dashboard:
1. Copy and paste the reference number for the transaction in question (this is the link text for the link in the email&mdash;don't click on it, highlight and copy it).
2. Go to https://dashboard.balancedpayments.com/.
3. Paste the transaction id into the search box at the very top of the dashboard. It should pop up a link to the transaction.

As you mention, this requires access to Balanced, something that you and @clone1018 and @pjf now have.

https://docs.balancedpayments.com/1.1/api/callbacks/

> Callbacks are a means of registering to receive payloads of information to a specific URL of your choice. You may create multiple callbacks. Events will be sent to each callback URL. In the event that a 20x response is not received when the payload is sent to the callback URL, the callback will be retried up to 10 times with an initial retry delay of 10 minutes which will increase exponentially with each failure for the next 7 days.

How do I get access into Balanced? I am not aware of having credentials for Balanced.

@clone1018 helped me through it. I can get to the Balanced dashboard now. :dancer: 

I've successfully handled the recent _Balanced - Outstanding failed credits_ emails to Support. We have one Gittip user that has had many failures throughout the life of their account. The first failure we saw was December 5, 2013. What I find very surprising is that a few transactions (a Gittip deposit to their bank account) appear to have gone through successfully. How is that possible? Can we ask Balanced to look deeper into the history on that account? How do we do that in a privacy preserving way? (I do not think it is appropriate to name the Gittip user in a public forum.)

> Can we ask Balanced to look deeper into the history on that account? How do we do that in a privacy preserving way?

Email support@balancedpayments.com? Maybe if you initiate that email from inside Freshdesk we can manage the conversation there?

These continue to accumulate. I'm collecting them on [FD373](http://gittip.freshdesk.com/helpdesk/tickets/373).

@cczona Picking up over here from https://github.com/gittip/www.gittip.com/issues/2085#issuecomment-47149686.

@cczona Looking back through the emails from Balanced mentioned above on this ticket, I _am_ seeing you among the failed payouts.

@cczona Ditto CallbackWomen.

Okay, thanks. How do I proceed? I don't have a Balanced account, though am fine with opening one if that solves a problem.

@cczona Since Gittip uses Balanced as it's payment processor, we end up creating an implicit Balanced account for you when you enter financial details. Here's the message we got from Balanced on February 18:

> Greetings, 
> 
> The bank has notified us that the tokenization for bank account [redacted] is no longer valid. In response, we have marked this bank account as invalid. The reason given for this is: 
> 
> Incorrect DFI Account number. 
> 
> The bank account may be found at: [redacted] 
> 
> Please have your customer update their bank account information before attempting to debit or credit them again. 
> 
> Thanks, 
> Balanced

@cczona So I guess let's have you re-enter your bank account details? Sounds like @bruceadams has looked at your case and perhaps double-checked that we haven't corrupted your balance. I'll triple-check ...

I didn't know about this issue, or maybe I just forgot, anyway I've now added it to the list of bugs that the payday rewrite should fix (#2508).

@Changaco Cool, thank you. Isn't the action item here an endpoint for a Balanced webhook? That's out of band for payday itself, right?

@cczona Here's where you can re-enter your bank account details: https://www.gittip.com/bank-account.html.

Both accounts continued to make transfers between Feb 18th through May 15th. So the error message is flawed.

I used that URL last Thursday to add (back) the banking info for CallbackWomen's gittip account.

The cczona gittip account shows that banking is still in place, but I'll go ahead and re-do.

> Both accounts continued to make transfers between Feb 18th through May 15th. So the error message is flawed.

@cczona Good point. Let's look into that ...

> I'll go ahead and re-do.

Thanks. :-)

@whit537 Possibly, but #2508 is where I have my checklist of payday-related bugs to fix, so might as well add it there to make sure I don't forget.

@Changaco Fair enough.

> I used that URL last Thursday to add (back) the banking info for CallbackWomen's gittip account.

@cczona Depending on what time you added that on Thursday we may not have paid out to you during the payday run that day. I'm seeing the same #2085 issue w/ CallbackWomen as w/ cczona. Let me see what CallbackWomen looks like in Balanced ... 

I _do_ see a bank account on file at Balanced for CallbackWomen.

Says it was created at June 20 2014, 3:19 AM, which (assuming UTC and PDT) would be last Thursday at 8:19 PM your time. So that lines up, at least. :-)

@cczona I'm also now seeing that you've re-added a bank account to your cczona account. So we should have what we need to do the payouts. Now let's make sure we've got your balance right on both accounts.

The crucial question is whether the payouts we've recorded in Gittip sync up with the payouts that are recorded in Balanced. If those line up then your balance should be good, and we can get you squared away here, regardless of secondary issues like this present one, #2085, and the "flawed error message" issue [above](https://github.com/gittip/www.gittip.com/issues/1811#issuecomment-47151716).

The payouts for CallbackWomen are the same in both Gittip and Balanced.

The payouts for cczona are the same in both Gittip and Balanced.

@cczona BTW, you should be able to cross-check my result by looking at your Gittip history and your bank statements. Admittedly that's going to be more tedious, however.

Okay! So your Gittip balance is not corrupt, and we now have a bank account on file, for both CallbackWomen and cczona.

@cczona If we do nothing else right now, I would expect your Gittip balance to clear to your bank account tomorrow (less outbound tips). Is that sufficient for today? What else do you need from me to resolve this?

Thanks! I'll check on it tomorrow.

Success! Both accounts resumed transfers this morning. Thank you.

@cczona Yay! Thanks for closing the loop. BTW, this ticket will stay open until we fix the underlying issue, so if you don't want spam down the road feel free to unsubscribe. :-)


Resolving patch:
diff --git a/gittip/security/csrf.py b/gittip/security/csrf.py
--- a/gittip/security/csrf.py
+++ b/gittip/security/csrf.py
@@ -87,15 +87,14 @@ def inbound(request):
     """Given a Request object, reject it if it's a forgery.
     """
     if request.line.uri.startswith('/assets/'): return
+    if request.line.uri.startswith('/callbacks/'): return
 
     try:
-        csrf_token = request.headers.cookie.get('csrf_token')
-        csrf_token = '' if csrf_token is None else csrf_token.value
-        csrf_token = _sanitize_token(csrf_token)
+        csrf_token = _sanitize_token(request.headers.cookie['csrf_token'].value)
     except KeyError:
-        csrf_token = _get_new_csrf_key()
+        csrf_token = None
 
-    request.context['csrf_token'] = csrf_token
+    request.context['csrf_token'] = csrf_token or _get_new_csrf_key()
 
     # Assume that anything not defined as 'safe' by RC2616 needs protection
     if request.line.method not in ('GET', 'HEAD', 'OPTIONS', 'TRACE'):


Test patch:
diff --git a/gittip/testing/__init__.py b/gittip/testing/__init__.py
--- a/gittip/testing/__init__.py
+++ b/gittip/testing/__init__.py
@@ -35,8 +35,10 @@ def build_wsgi_environ(self, *a, **kw):
         """
 
         # csrf - for both anon and authenticated
-        self.cookie[b'csrf_token'] = b'sotokeny'
-        kw[b'HTTP_X-CSRF-TOKEN'] = b'sotokeny'
+        csrf_token = kw.get('csrf_token', b'sotokeny')
+        if csrf_token:
+            self.cookie[b'csrf_token'] = csrf_token
+            kw[b'HTTP_X-CSRF-TOKEN'] = csrf_token
 
         # user authentication
         auth_as = kw.pop('auth_as', None)
diff --git a/tests/py/test_callbacks.py b/tests/py/test_callbacks.py
new file mode 100644
--- /dev/null
+++ b/tests/py/test_callbacks.py
@@ -0,0 +1,58 @@
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import json
+
+from mock import patch
+
+from gittip.billing.exchanges import record_exchange
+from gittip.testing import Harness
+
+
+class TestBalancedCallbacks(Harness):
+
+    def callback(self, *a, **kw):
+        kw.setdefault(b'HTTP_X_FORWARDED_FOR', b'50.18.199.26')
+        kw.setdefault('content_type', 'application/json')
+        kw.setdefault('raise_immediately', False)
+        return self.client.POST('/callbacks/balanced', **kw)
+
+    def test_simplate_checks_source_address(self):
+        r = self.callback(HTTP_X_FORWARDED_FOR=b'0.0.0.0')
+        assert r.code == 403
+
+    def test_simplate_doesnt_require_a_csrf_token(self):
+        r = self.callback(body=b'{"events": []}', csrf_token=False)
+        assert r.code == 200, r.body
+
+    @patch('gittip.billing.exchanges.record_exchange_result')
+    def test_credit_callback(self, rer):
+        alice = self.make_participant('alice')
+        for status in ('succeeded', 'failed'):
+            error = 'FOO' if status == 'failed' else None
+            e_id = record_exchange(self.db, 'ach', 10, 0, alice, 'pre')
+            body = json.dumps({
+                "events": [
+                    {
+                        "type": "credit."+status,
+                        "entity": {
+                            "credits": [
+                                {
+                                    "failure_reason": error,
+                                    "meta": {
+                                        "participant_id": alice.id,
+                                        "exchange_id": e_id,
+                                    },
+                                    "status": status,
+                                }
+                            ]
+                        }
+                    }
+                ]
+            })
+            r = self.callback(body=body, csrf_token=False)
+            assert r.code == 200, r.body
+            assert rer.call_count == 1
+            assert rer.call_args[0][:-1] == (self.db, e_id, status, error)
+            assert rer.call_args[0][-1].id == alice.id
+            assert rer.call_args[1] == {}
+            rer.reset_mock()