Repository: gratipay/gratipay.com

Issue:
history and members pages are wonky during payday
See also #92 (stats page) and #125 (balance on profile page).

The history page counts paydays wrong. I'm seeing #5 labeled #6, down to #0 mislabeled #1.



Discussion:
Balance is also miscalculated. Amounts from currently running payday appear to be applied to balance.

Desired behavior would be to have the current payday represented accurately as run so far, or not on there until completed.

If you run the whole payday in a transaction this should fix it. Nothing else should see the updates until you commit the transaction because Postgres is [ACID](http://en.wikipedia.org/wiki/ACID). 

@mjallday The reason that payday isn't one transaction is so that if it crashes we don't have to start all over again. I've used this feature once already (see #169). I also intended this to set us up to parallelize payday in the future.

Am I thinking about this wrong?

That makes sense. Could we display balanced + pending in that case? Pending is zeroed out before and after payday and balance is adjusted in the same query that zeros out pending.

+1 from @indirect over on #1093.

+1 from @greggles in [IRC](https://botbot.me/freenode/gittip/msg/5218272/).

+1 from @Daxter over on #1388.

Also the members page is broken while payday is in progress. All last week values are zero. See https://www.gittip.com/Gittip/members/

+1 from @hawkowl in [IRC]().

+1 from @ehmatthes in [IRC](https://botbot.me/freenode/gittip/msg/10115208/).

How easily could wonky pages/ page elements be cached right before starting the payday script?

It seems that the payday script takes long enough to run now, that it might be worth figuring out a short-term solution to this issue.
- Run a script that caches all pages/ page elements affected by running the payday script.
- Run payday script.
- Clear out cache.


Resolving patch:
diff --git a/gittip/models/_mixin_team.py b/gittip/models/_mixin_team.py
--- a/gittip/models/_mixin_team.py
+++ b/gittip/models/_mixin_team.py
@@ -81,9 +81,12 @@ def get_take_last_week_for(self, member):
             SELECT amount
               FROM transfers
              WHERE tipper=%s AND tippee=%s AND context='take'
-               AND timestamp >
-                (SELECT ts_start FROM paydays ORDER BY ts_start DESC LIMIT 1)
-          ORDER BY timestamp DESC LIMIT 1
+               AND timestamp > (
+                       SELECT ts_start
+                         FROM paydays
+                        WHERE ts_end > ts_start
+                     ORDER BY ts_start DESC LIMIT 1
+                   )
 
         """, (self.username, membername), default=Decimal('0.00'))
 
diff --git a/gittip/utils/history.py b/gittip/utils/history.py
new file mode 100644
--- /dev/null
+++ b/gittip/utils/history.py
@@ -0,0 +1,123 @@
+from decimal import Decimal
+
+from aspen import log
+
+
+def iter_payday_events(db, participant):
+    """Yields payday events for the given participant username.
+
+    Each payday is expected to encompass 0 or 1 exchanges and 0 or more
+    transfers per participant. Here we knit them together along with start
+    and end events for each payday. If we have exchanges or transfers that
+    fall outside of a payday, then we have a logic bug. I am 50% confident
+    that this will manifest some day.
+
+    """
+    username = participant.username
+    paydays = db.all("""
+        SELECT ts_start, ts_end
+          FROM paydays
+      ORDER BY ts_start DESC
+    """, back_as=dict)
+    npaydays = len(paydays)
+    exchanges = db.all("""
+        SELECT *
+          FROM exchanges
+         WHERE participant=%s
+      ORDER BY timestamp ASC
+    """, (username,), back_as=dict)
+    transfers = db.all("""
+        SELECT *
+          FROM transfers
+         WHERE tipper=%(username)s OR tippee=%(username)s
+      ORDER BY timestamp ASC
+    """, locals(), back_as=dict)
+    balance = participant.balance
+    for i, payday in enumerate(paydays, 1):
+
+        if not (exchanges or transfers):
+            # Show all paydays since the user started really participating.
+            break
+
+        payday_start = { 'event': 'payday-start'
+                       , 'timestamp': payday['ts_start']
+                       , 'number': npaydays - i
+                       , 'balance': Decimal('0.00')
+                        }
+        payday_end = { 'event': 'payday-end'
+                     , 'timestamp': payday['ts_end']
+                     , 'number': npaydays - i
+                      }
+
+        events = []
+        while (exchanges or transfers):
+
+            # Take the next event, either an exchange or transfer.
+            # ====================================================
+            # We do this by peeking at both lists, and popping the list
+            # that has the next event.
+
+            exchange = exchanges[-1] if exchanges else None
+            transfer = transfers[-1] if transfers else None
+
+            if exchange is None:
+                event = transfers.pop()
+            elif transfer is None:
+                event = exchanges.pop()
+            elif transfer['timestamp'] > exchange['timestamp']:
+                event = transfers.pop()
+            else:
+                event = exchanges.pop()
+
+            if 'fee' in event:
+                if event['amount'] > 0:
+                    event['event'] = 'charge'
+                else:
+                    event['event'] = 'credit'
+            else:
+                event['event'] = 'transfer'
+
+            # Record the next event.
+            # ======================
+
+            if event['timestamp'] < payday_start['timestamp']:
+                if event['event'] == 'exchange':
+                    back_on = exchanges
+                else:
+                    back_on = transfers
+                back_on.append(event)
+                break
+
+            events.append(event)
+
+        if not events:
+            continue
+
+        # Calculate balance.
+        # ==================
+
+        prev = events[0]
+        prev['balance'] = balance
+        for event in events[1:] + [payday_start]:
+            if prev['event'] == 'charge':
+                balance = prev['balance'] - prev['amount']
+            elif prev['event'] == 'credit':
+                balance = prev['balance'] - prev['amount'] + prev['fee']
+            elif prev['event'] == 'transfer':
+                if prev['tippee'] == username:
+                    balance = prev['balance'] - prev['amount']
+                else:
+                    balance = prev['balance'] + prev['amount']
+            event['balance'] = balance
+            prev = event
+        balance = payday_start['balance']
+
+        yield payday_start
+        for event in reversed(events):
+            yield event
+        yield payday_end
+
+    # This should catch that logic bug.
+    if exchanges or transfers:
+        log("These should be empty:", exchanges, transfers)
+        raise Exception("Logic bug in payday timestamping.")


Test patch:
diff --git a/tests/py/test_history.py b/tests/py/test_history.py
new file mode 100644
--- /dev/null
+++ b/tests/py/test_history.py
@@ -0,0 +1,30 @@
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+from decimal import Decimal
+
+from gittip.billing.payday import Payday
+from gittip.testing import Harness
+from gittip.utils.history import iter_payday_events
+
+
+class TestHistory(Harness):
+
+    def test_iter_payday_events_counts_correctly(self):
+        team = self.make_participant('team', number='plural', claimed_time='now', balance=10000)
+        alice = self.make_participant('alice', claimed_time='now', balance=10000)
+        self.db.run("""
+            INSERT INTO exchanges
+                        (amount, fee, participant)
+                 VALUES (10000, 0, 'team')
+                      , (10000, 0, 'alice');
+        """)
+        bob = self.make_participant('bob', claimed_time='now')
+        team.add_member(bob)
+        team.set_take_for(bob, Decimal('1.00'), team)
+        alice.set_tip_to(bob, Decimal('5.00'))
+        for i in range(3):
+            Payday(self.db).run()
+        Payday(self.db).start()
+        event = next(iter_payday_events(self.db, bob))
+        assert event['event'] == 'payday-start'
+        assert event['number'] == 2
diff --git a/tests/py/test_take.py b/tests/py/test_take.py
--- a/tests/py/test_take.py
+++ b/tests/py/test_take.py
@@ -33,6 +33,7 @@ def make_participant(self, username, **kw):
                          "VALUES (now(), %(tipper)s, %(tippee)s, %(amount)s, 'take')"
                        , dict(tipper=team_name, tippee=username, amount=take_last_week,)
                         )
+            self.db.run("UPDATE paydays SET ts_end=now()")
         return participant
 
     def test_we_can_make_a_team(self):
@@ -197,3 +198,13 @@ def test_changes_to_others_take_can_increase_members_take(self):
         team.set_take_for(bob, D('30.00'), bob)
         alice = Participant.from_username('alice')
         assert alice.receiving == alice.taking == 42
+
+    # get_take_last_week_for - gtlwf
+
+    def test_gtlwf_works_during_payday(self):
+        team = self.make_team()
+        alice = self.make_participant('alice', take_last_week='30.00', claimed_time='now')
+        team.add_member(alice)
+        team.set_take_for(alice, D('42.00'), alice)
+        self.db.run("INSERT INTO paydays DEFAULT VALUES")
+        assert team.get_take_last_week_for(alice) == 30