Repository: gratipay/gratipay.com

Issue:
charts.json returns null for `widthdrawals`
Expected:

``` json
[
    {
        "active_users": 13,
        "charges": 27.02,
        "date": "2014-01-15",
        "total_gifts": 251383.81,
        "total_users": 100,
        "weekly_gifts": 382.95,
        "withdrawals": 3.53
    },
...
```

Received:

``` json
[
    {
        "active_users": 13,
        "charges": 27.02,
        "date": "2014-01-15",
        "total_gifts": 251383.81,
        "total_users": 100,
        "weekly_gifts": 382.95,
        "withdrawals": null
    },
...
```

2012-08-17 and every week before it is broken:

``` json
    {
        "active_users": 313,
        "charges": 964.40,
        "date": "2012-08-17",
        "total_gifts": 5812.20,
        "total_users": 4067,
        "weekly_gifts": 1052.09,
        "withdrawals": null
    },

```



Discussion:
such [IRC](https://botbot.me/freenode/gittip/msg/9892221/)

From http://www.postgresql.org/docs/9.3/static/functions-aggregate.html

> It should be noted that except for count, these functions return a null value when no rows are selected. In particular, sum of no rows returns null, not zero as one might expect, and array_agg returns null rather than an empty array when there are no input rows. The coalesce function can be used to substitute zero or an empty array for null when necessary.

When recalculating `ach_volume` in #1902 when there are no exchanges for the particular payday, nulls sneaked in because of the above. To guard against problems like this the schema needs to be updated to add 'not null' constrain. Also self checks added need coalesce to fix sum of no rows to return zero, otherwise they always pass (null in sql is 'unknown' and never equals to anything). One needs to be on guard when moving between python and sql since none and null are different beasts.


Resolving patch:
diff --git a/gittip/models/__init__.py b/gittip/models/__init__.py
--- a/gittip/models/__init__.py
+++ b/gittip/models/__init__.py
@@ -147,7 +147,7 @@ def _check_paydays_volumes(self):
         charge_volume = self.all("""
             select * from (
                 select id, ts_start, charge_volume, (
-                        select sum(amount+fee)
+                        select coalesce(sum(amount+fee), 0)
                         from exchanges
                         where timestamp > ts_start
                         and timestamp < ts_end
@@ -163,7 +163,7 @@ def _check_paydays_volumes(self):
         charge_fees_volume = self.all("""
             select * from (
                 select id, ts_start, charge_fees_volume, (
-                        select sum(fee)
+                        select coalesce(sum(fee), 0)
                         from exchanges
                         where timestamp > ts_start
                         and timestamp < ts_end
@@ -179,7 +179,7 @@ def _check_paydays_volumes(self):
         ach_volume = self.all("""
             select * from (
                 select id, ts_start, ach_volume, (
-                        select sum(amount)
+                        select coalesce(sum(amount), 0)
                         from exchanges
                         where timestamp > ts_start
                         and timestamp < ts_end
@@ -195,7 +195,7 @@ def _check_paydays_volumes(self):
         ach_fees_volume = self.all("""
             select * from (
                 select id, ts_start, ach_fees_volume, (
-                        select sum(fee)
+                        select coalesce(sum(fee), 0)
                         from exchanges
                         where timestamp > ts_start
                         and timestamp < ts_end


Test patch: