Repository: gratipay/gratipay.com

Issue:
repair paydays table
This isn't really used much yet but it should be (nice historical data). Two repairs needed:
- ~~recalculate nparticipants for paydays <= #3~~
- recalculate exchange_volume and exchange_fees_volume for paydays <= #8 (see #202)



Discussion:
Here's how I fixed nparticipants:

```
update paydays set nparticipants=(select count(*) from participants where claimed_time < ts_start);
```

In addition to the early issue where we were counting unclaimed accounts as registered users, it turns out that rerunning the paydays script throws this number off as well (#261). We got bit by that in week 7 (#169) but it wasn't enough to notice. We got bit by it again this week (#257) and it _was_ enough to notice. :-)

Am I missing something major here or is updating the exchange_volume (that's actually ach_volume, right?) as simple as:

``` sql
UPDATE paydays
SET ach_volume = (
        SELECT count(amount) 
        FROM exchanges 
        WHERE timestamp > ts_start 
        AND timestamp < ts_end
        AND amount < 0
    );
```

And the same for fees:

``` sql
UPDATE paydays
SET ach_fees_volume = (
        SELECT count(fee) 
        FROM exchanges 
        WHERE timestamp > ts_start 
        AND timestamp < ts_end
        AND amount < 0
    );
```

Following the self check #1705 idea I have

``` sql
select count(id)
from paydays
where charge_volume != (
    select sum(amount+fee) 
    from exchanges 
    where timestamp > ts_start 
    and timestamp < ts_end 
    and amount > 0
)
```

returning 0.

There are two bugs in `charge_fees_volume`. See

``` sql
select * from (
    select id, ts_start, charge_fees_volume, (
            select sum(fee) 
            from exchanges 
            where timestamp > ts_start 
            and timestamp < ts_end 
            and amount > 0
        ) as ref
    from paydays
    order by id
) as foo
where charge_fees_volume != ref;
```

returning 2 rows

```
 id |           ts_start            | charge_fees_volume |  ref  
----+-------------------------------+--------------------+-------
 13 | 2012-07-27 20:51:21.754544+00 |              71.36 | 99.31
 14 | 2012-08-03 14:57:42.935184+00 |              45.27 | 56.83
```

There seems to be one bug in `ach_volume`. See

``` sql
select * from (
    select id, ts_start, ach_volume, (
            select sum(amount) 
            from exchanges 
            where timestamp > ts_start 
            and timestamp < ts_end 
            and amount < 0
        ) as ref
    from paydays
    order by id
) as foo
where ach_volume != ref;
```

returning

```
 id |           ts_start            | ach_volume |   ref    
----+-------------------------------+------------+----------
 74 | 2013-06-20 12:12:53.292609+00 |   -1885.73 | -2024.73
(1 row)
```

`ach_fees_volume` seems to be fine:

``` sql
select * from (
    select id, ts_start, ach_fees_volume, (
            select sum(fee) 
            from exchanges 
            where timestamp > ts_start 
            and timestamp < ts_end 
            and amount < 0
        ) as ref
    from paydays
    order by id
) as foo
where ach_fees_volume != ref;
```

Returns 0 rows.

@whit537 Is it worth to start looking for a bugs in the payday code or do we just fix it for now, add it to the self checks and wait for it to break? That would be my suggestion.

Agreed, add it to self-checks and wait for it to break. Once we're comfortable with web app db usage we'll turn our attention to payday.


Resolving patch:
diff --git a/gittip/models/__init__.py b/gittip/models/__init__.py
--- a/gittip/models/__init__.py
+++ b/gittip/models/__init__.py
@@ -18,6 +18,7 @@ def self_check(self):
         self._check_tips()
         self._check_orphans()
         self._check_orphans_no_tips()
+        self._check_paydays_volumes()
 
     def _check_tips(self):
         """
@@ -138,4 +139,72 @@ def _check_orphans_no_tips(self):
         known = set([25206]) # '4c074000c7bc', 'naderman', '3.00'
         real = set(tips_with_orphans) - known
         assert len(real) == 0, real
+
+    def _check_paydays_volumes(self):
+        """
+        Recalculate *_volume fields in paydays table using exchanges table.
+        """
+        charge_volume = self.all("""
+            select * from (
+                select id, ts_start, charge_volume, (
+                        select sum(amount+fee)
+                        from exchanges
+                        where timestamp > ts_start
+                        and timestamp < ts_end
+                        and amount > 0
+                    ) as ref
+                from paydays
+                order by id
+            ) as foo
+            where charge_volume != ref
+        """)
+        assert len(charge_volume) == 0
+
+        charge_fees_volume = self.all("""
+            select * from (
+                select id, ts_start, charge_fees_volume, (
+                        select sum(fee)
+                        from exchanges
+                        where timestamp > ts_start
+                        and timestamp < ts_end
+                        and amount > 0
+                    ) as ref
+                from paydays
+                order by id
+            ) as foo
+            where charge_fees_volume != ref
+        """)
+        assert len(charge_fees_volume) == 0
+
+        ach_volume = self.all("""
+            select * from (
+                select id, ts_start, ach_volume, (
+                        select sum(amount)
+                        from exchanges
+                        where timestamp > ts_start
+                        and timestamp < ts_end
+                        and amount < 0
+                    ) as ref
+                from paydays
+                order by id
+            ) as foo
+            where ach_volume != ref
+        """)
+        assert len(ach_volume) == 0
+
+        ach_fees_volume = self.all("""
+            select * from (
+                select id, ts_start, ach_fees_volume, (
+                        select sum(fee)
+                        from exchanges
+                        where timestamp > ts_start
+                        and timestamp < ts_end
+                        and amount < 0
+                    ) as ref
+                from paydays
+                order by id
+            ) as foo
+            where ach_fees_volume != ref
+        """)
+        assert len(ach_fees_volume) == 0
 #


Test patch: