Repository: liberapay/liberapay.com

Issue:
Income who are not easy divisible are wrong
A little bug, but it can be problematic in team. Excellent example here : https://en.liberapay.com/Khaganat/income/

Team have 0,85€/week and 7 people. It divide in 7 ( 0,121428 etc), rounded to the decimal point top (0,13), and the last member have less (0,85-(6*0,13)=0.07). Because people are alphabetical ordered, it's for me ;) but, also, it's always the same people who haven't got all of his part. 

Better to rounded to the decimal point lower and so have "Leftover". And when leftover is divisible by the number of people, it's shared. 

Example : like now, Team have 0,85€/week and 7 people. 0.85/7 and rounded lower = 0,12, rest 1 centime. So leftover have 1 centime per week ; after 7 week, if they are always 7 people in the team, all people have 1 centime in addition. And all people have the same income if all have the same part :)
Income who are not easy divisible are wrong
A little bug, but it can be problematic in team. Excellent example here : https://en.liberapay.com/Khaganat/income/

Team have 0,85€/week and 7 people. It divide in 7 ( 0,121428 etc), rounded to the decimal point top (0,13), and the last member have less (0,85-(6*0,13)=0.07). Because people are alphabetical ordered, it's for me ;) but, also, it's always the same people who haven't got all of his part. 

Better to rounded to the decimal point lower and so have "Leftover". And when leftover is divisible by the number of people, it's shared. 

Example : like now, Team have 0,85€/week and 7 people. 0.85/7 and rounded lower = 0,12, rest 1 centime. So leftover have 1 centime per week ; after 7 week, if they are always 7 people in the team, all people have 1 centime in addition. And all people have the same income if all have the same part :)


Discussion:
Originally this was supposed to be solved by processing team takes in random order, so it wouldn't be the same member who gets less each time, but this is linked to #434 and I haven't gotten around to that yet.

Keeping track of unused team income is also a good idea though. I've been planning to implement it as well, but more for #81.
I was wondering if this issue was still open & needed to be fixed. I am interested in working on it/looking into it further! 
I think this can still happen. I recommend adding a test in the `tests/py/test_payday.py` file to reproduce the problem: a team with €0.85 income and 7 members each taking €1. This should confirm that the `Payday.resolve_takes()` function needs to be improved.

The issue can probably be fixed by replacing the simplistic `takes_ratio` ([payday.py#L472-L477](https://github.com/liberapay/liberapay.com/blob/593/liberapay/billing/payday.py#L472-L477)) with a call to the generic `resolve_amounts()` function ([common.py#L493](https://github.com/liberapay/liberapay.com/blob/593/liberapay/payin/common.py#L493)).
We’ve been working on this issue and have successfully made a test to reproduce the problem and have it failing on the current version of the code. 

Fixing the issue itself has been a bit more difficult though; we are trying to call `resolve_amounts()`(common.py#L493), but are not sure what exact arguments to pass in. As of right now, we think that `available_amount` would be `fuzzy_income_sum` and `convergence_amounts` could be a dictionary of IDs or indices and corresponding takes. However, we aren’t sure what should be passed in for `base_amounts`. 

Also in the context of payday.py#L472-L477 what exactly would `resolve_amounts` be returning to? Any advice would be helpful. 
> As of right now, we think that `available_amount` would be `fuzzy_income_sum` and `convergence_amounts` could be a dictionary of IDs or indices and corresponding takes. However, we aren’t sure what should be passed in for `base_amounts`.

Hmm… Take amounts are maximums, but `resolve_amounts` is designed to always distribute the entire available amount, so we have to give it an amount that's never greater than the sum of the takes. This should work:

```python
resolved_takes = resolve_amounts(
    min(fuzzy_income_sum, fuzzy_takes_sum),
    {take.member: take.amount for take in takes}
)
```

For testing purposes, is there an easy way to only run test_payday.py without running the full "make tests" command? On our computers the "make tests" command takes quite a bit of time, and we were trying to find a way to streamline our debugging process! Any advice would be helpful, thank you! 
You can do `make pytest PYTEST=payday` to run only the tests in `test_payday.py`, or make sure that your test is failing and do `make pytest-re` to rerun only that test.
Following the example given & using the resolved_amounts function, the end result would be that one person would receive 0.01 extra every week (versus having one person with 0.06 less than everyone else). We were wondering if this is the desired outcome because if everyone were to get equal parts and then store leftovers, we would probably need to write another function to implement this outcome. Let us know what option/outcome you would prefer! 
Giving an extra cent to one of the members is a simple and acceptable solution. Ideally the cent should go to a different member every week.
Originally this was supposed to be solved by processing team takes in random order, so it wouldn't be the same member who gets less each time, but this is linked to #434 and I haven't gotten around to that yet.

Keeping track of unused team income is also a good idea though. I've been planning to implement it as well, but more for #81.
I was wondering if this issue was still open & needed to be fixed. I am interested in working on it/looking into it further! 
I think this can still happen. I recommend adding a test in the `tests/py/test_payday.py` file to reproduce the problem: a team with €0.85 income and 7 members each taking €1. This should confirm that the `Payday.resolve_takes()` function needs to be improved.

The issue can probably be fixed by replacing the simplistic `takes_ratio` ([payday.py#L472-L477](https://github.com/liberapay/liberapay.com/blob/593/liberapay/billing/payday.py#L472-L477)) with a call to the generic `resolve_amounts()` function ([common.py#L493](https://github.com/liberapay/liberapay.com/blob/593/liberapay/payin/common.py#L493)).
We’ve been working on this issue and have successfully made a test to reproduce the problem and have it failing on the current version of the code. 

Fixing the issue itself has been a bit more difficult though; we are trying to call `resolve_amounts()`(common.py#L493), but are not sure what exact arguments to pass in. As of right now, we think that `available_amount` would be `fuzzy_income_sum` and `convergence_amounts` could be a dictionary of IDs or indices and corresponding takes. However, we aren’t sure what should be passed in for `base_amounts`. 

Also in the context of payday.py#L472-L477 what exactly would `resolve_amounts` be returning to? Any advice would be helpful. 
> As of right now, we think that `available_amount` would be `fuzzy_income_sum` and `convergence_amounts` could be a dictionary of IDs or indices and corresponding takes. However, we aren’t sure what should be passed in for `base_amounts`.

Hmm… Take amounts are maximums, but `resolve_amounts` is designed to always distribute the entire available amount, so we have to give it an amount that's never greater than the sum of the takes. This should work:

```python
resolved_takes = resolve_amounts(
    min(fuzzy_income_sum, fuzzy_takes_sum),
    {take.member: take.amount for take in takes}
)
```

For testing purposes, is there an easy way to only run test_payday.py without running the full "make tests" command? On our computers the "make tests" command takes quite a bit of time, and we were trying to find a way to streamline our debugging process! Any advice would be helpful, thank you! 
You can do `make pytest PYTEST=payday` to run only the tests in `test_payday.py`, or make sure that your test is failing and do `make pytest-re` to rerun only that test.
Following the example given & using the resolved_amounts function, the end result would be that one person would receive 0.01 extra every week (versus having one person with 0.06 less than everyone else). We were wondering if this is the desired outcome because if everyone were to get equal parts and then store leftovers, we would probably need to write another function to implement this outcome. Let us know what option/outcome you would prefer! 
Giving an extra cent to one of the members is a simple and acceptable solution. Ideally the cent should go to a different member every week.

Resolving patch:
diff --git a/liberapay/billing/payday.py b/liberapay/billing/payday.py
--- a/liberapay/billing/payday.py
+++ b/liberapay/billing/payday.py
@@ -16,6 +16,7 @@
 from liberapay.billing.transactions import Money, transfer
 from liberapay.exceptions import NegativeBalance
 from liberapay.i18n.currencies import MoneyBasket
+from liberapay.payin.common import resolve_amounts
 from liberapay.utils import group_by
 from liberapay.website import website
 
@@ -149,7 +150,7 @@ def shuffle(self, log_dir='.'):
             assert self.stage == 1
             with self.db.get_cursor() as cursor:
                 self.prepare(cursor, self.ts_start)
-                self.transfer_virtually(cursor, self.ts_start)
+                self.transfer_virtually(cursor, self.ts_start, self.id)
                 self.check_balances(cursor)
                 cursor.run("""
                     UPDATE paydays
@@ -372,13 +373,13 @@ def prepare(cursor, ts_start):
         log("Prepared the DB.")
 
     @staticmethod
-    def transfer_virtually(cursor, ts_start):
+    def transfer_virtually(cursor, ts_start, payday_id):
         cursor.run("SELECT settle_tip_graph();")
         teams = cursor.all("""
             SELECT id, main_currency FROM payday_participants WHERE kind = 'group';
         """)
         for team_id, currency in teams:
-            Payday.transfer_takes(cursor, team_id, currency)
+            Payday.transfer_takes(cursor, team_id, currency, payday_id)
         cursor.run("""
             SELECT settle_tip_graph();
             UPDATE payday_tips SET is_funded = false WHERE is_funded IS NULL;
@@ -386,7 +387,7 @@ def transfer_virtually(cursor, ts_start):
         Payday.pay_invoices(cursor, ts_start)
 
     @staticmethod
-    def transfer_takes(cursor, team_id, currency):
+    def transfer_takes(cursor, team_id, currency, payday_id):
         """Resolve and transfer takes for the specified team
         """
         args = dict(team_id=team_id)
@@ -416,7 +417,7 @@ def transfer_takes(cursor, team_id, currency):
               JOIN payday_participants p ON p.id = t.member
              WHERE t.team = %(team_id)s;
         """, args)
-        transfers, leftover = Payday.resolve_takes(tips, takes, currency)
+        transfers, leftover = Payday.resolve_takes(tips, takes, currency, payday_id)
         for t in transfers:
             context = 'leftover-take' if t.is_leftover else 'partial-take' if t.is_partial else 'take'
             cursor.run("SELECT transfer(%s, %s, %s, %s, %s, NULL)",
@@ -425,7 +426,7 @@ def transfer_takes(cursor, team_id, currency):
                    (leftover, team_id))
 
     @staticmethod
-    def resolve_takes(tips, takes, ref_currency):
+    def resolve_takes(tips, takes, ref_currency, payday_id):
         """Resolve many-to-many donations (team takes)
         """
         for tip in tips:
@@ -469,12 +470,13 @@ def resolve_takes(tips, takes, ref_currency):
         tip_currencies = set(t.full_amount.currency for t in tips)
         takes_by_preferred_currency = group_by(takes, lambda t: t.main_currency)
         takes_by_secondary_currency = {c: [] for c in tip_currencies}
-        if fuzzy_takes_sum:
-            takes_ratio = min(fuzzy_income_sum / fuzzy_takes_sum, 1)
-        else:
-            takes_ratio = 0
+        resolved_takes = resolve_amounts(
+            min(fuzzy_income_sum, fuzzy_takes_sum),
+            {take.member: take.amount.convert(ref_currency) for take in takes},
+            payday_id=payday_id,
+        )
         for take in takes:
-            take.amount = (take.amount * takes_ratio).round_up()
+            take.amount = resolved_takes.get(take.member) or take.amount.zero()
             if take.paid_in_advance is None:
                 take.paid_in_advance = take.amount.zero()
             if take.accepted_currencies is None:
@@ -599,7 +601,7 @@ def resolve_takes(tips, takes, ref_currency):
                     take.amount -= transfer_amount.convert(take.amount.currency)
                 if in_advance_amount:
                     tip.paid_in_advance -= in_advance_amount
-                    take.paid_in_advance -= in_advance_amount.convert(take.amount.currency)
+                    take.paid_in_advance -= in_advance_amount.convert(take.paid_in_advance.currency)
                 if on_time_amount:
                     tip.balances -= on_time_amount
                 tip.amount -= transfer_amount
@@ -950,8 +952,15 @@ def update_cached_amounts(cls):
         now = pando.utils.utcnow()
         with cls.db.get_cursor() as cursor:
             cursor.run("LOCK TABLE takes IN EXCLUSIVE MODE")
+            payday_id = cursor.one("""
+                SELECT id
+                  FROM paydays
+                 WHERE ts_start IS NOT NULL
+              ORDER BY id DESC
+                 LIMIT 1
+            """, default=0) + 1
             cls.prepare(cursor, now)
-            cls.transfer_virtually(cursor, now)
+            cls.transfer_virtually(cursor, now, payday_id)
             cursor.run("""
             CREATE INDEX ON payday_transfers (tippee);
             CREATE INDEX ON payday_transfers (team) WHERE team IS NOT NULL;
diff --git a/liberapay/models/_mixin_team.py b/liberapay/models/_mixin_team.py
--- a/liberapay/models/_mixin_team.py
+++ b/liberapay/models/_mixin_team.py
@@ -260,7 +260,7 @@ def recompute_actual_takes(self, cursor, member=None):
                AND ( t.is_funded OR t.paid_in_advance > 0 )
                AND p.is_suspended IS NOT true
         """, dict(team_id=self.id, use_mangopay=mangopay.sandbox))
-        takes = (cursor or self.db).all("""
+        takes = cursor.all("""
             SELECT t.*, p.main_currency, p.accepted_currencies
               FROM current_takes t
               JOIN participants p ON p.id = t.member
@@ -268,7 +268,16 @@ def recompute_actual_takes(self, cursor, member=None):
                AND p.is_suspended IS NOT true
         """, (self.id,))
         # Recompute the takes
-        transfers, new_leftover = Payday.resolve_takes(tips, takes, self.main_currency)
+        next_payday_id = cursor.one("""
+            SELECT id
+              FROM paydays
+             WHERE ts_start IS NOT NULL
+          ORDER BY id DESC
+             LIMIT 1
+        """, default=0) + 1
+        transfers, new_leftover = Payday.resolve_takes(
+            tips, takes, self.main_currency, next_payday_id,
+        )
         transfers_by_member = group_by(transfers, lambda t: t.member)
         takes_sum = {k: MoneyBasket(t.amount for t in tr_list if not t.is_leftover)
                      for k, tr_list in transfers_by_member.items()}
diff --git a/liberapay/payin/common.py b/liberapay/payin/common.py
--- a/liberapay/payin/common.py
+++ b/liberapay/payin/common.py
@@ -1,5 +1,6 @@
 from collections import namedtuple
 from datetime import timedelta
+import itertools
 
 from pando.utils import utcnow
 from psycopg2.extras import execute_batch
@@ -451,7 +452,7 @@ def resolve_take_amounts(payment_amount, takes):
         t.resolved_amount = tr_amounts.get(t.member, payment_amount.zero())
 
 
-def resolve_amounts(available_amount, base_amounts, convergence_amounts=None):
+def resolve_amounts(available_amount, base_amounts, convergence_amounts=None, payday_id=1):
     """Compute transfer amounts.
 
     Args:
@@ -461,6 +462,9 @@ def resolve_amounts(available_amount, base_amounts, convergence_amounts=None):
             a map of IDs to raw transfer amounts
         convergence_amounts (Dict[Any, Money]):
             an optional map of IDs to ideal additional amounts
+        payday_id (int):
+            the ID of the current or next payday, used to rotate who receives
+            the remainder when there is a tie
 
     Returns a copy of `base_amounts` with updated values.
     """
@@ -488,7 +492,7 @@ def resolve_amounts(available_amount, base_amounts, convergence_amounts=None):
 
     # Compute the prorated amounts
     base_sum = Money.sum(base_amounts.values(), amount_left.currency)
-    base_ratio = amount_left / base_sum
+    base_ratio = 0 if base_sum == 0 else amount_left / base_sum
     for key, base_amount in sorted(base_amounts.items()):
         if base_amount == 0:
             continue
@@ -500,10 +504,17 @@ def resolve_amounts(available_amount, base_amounts, convergence_amounts=None):
     # Deal with rounding errors
     if amount_left > 0:
         # Try to distribute in a way that doesn't skew the percentages much.
+        # If there's a tie, use the payday ID to rotate the winner every week.
+        i = itertools.count(1)
+        n = len(r)
+
         def compute_priority(item):
             key, current_amount = item
             base_amount = base_amounts[key] * base_ratio
-            return (current_amount - base_amount) / base_amount
+            return (
+                (current_amount - base_amount) / base_amount if base_amount else 2,
+                (next(i) - payday_id) % n
+            )
 
         for key, amount in sorted(r.items(), key=compute_priority):
             r[key] += min_transfer_amount


Test patch:
diff --git a/tests/py/test_payday.py b/tests/py/test_payday.py
--- a/tests/py/test_payday.py
+++ b/tests/py/test_payday.py
@@ -288,7 +288,7 @@ def test_payday_doesnt_process_tips_when_goal_is_negative(self):
         payday = Payday.start()
         with self.db.get_cursor() as cursor:
             payday.prepare(cursor, payday.ts_start)
-            payday.transfer_virtually(cursor, payday.ts_start)
+            payday.transfer_virtually(cursor, payday.ts_start, payday.id)
             new_balances = self.get_new_balances(cursor)
             assert new_balances == {
                 'david': [],
@@ -313,7 +313,7 @@ def test_transfer_tips(self):
         payday = Payday.start()
         with self.db.get_cursor() as cursor:
             payday.prepare(cursor, payday.ts_start)
-            payday.transfer_virtually(cursor, payday.ts_start)
+            payday.transfer_virtually(cursor, payday.ts_start, payday.id)
             new_balances = self.get_new_balances(cursor)
             assert new_balances == {
                 'david': [],
@@ -341,7 +341,7 @@ def test_transfer_tips_whole_graph(self):
         payday = Payday.start()
         with self.db.get_cursor() as cursor:
             payday.prepare(cursor, payday.ts_start)
-            payday.transfer_virtually(cursor, payday.ts_start)
+            payday.transfer_virtually(cursor, payday.ts_start, payday.id)
             new_balances = self.get_new_balances(cursor)
             assert new_balances == {
                 'alice': [],
@@ -784,6 +784,101 @@ def test_unfunded_tip_to_team_doesnt_cause_NegativeBalance(self):
         }
         assert d == expected
 
+    def get_payday_transfers(self):
+        return self.db.all("""
+            SELECT tippee, amount
+              FROM transfers tr
+             WHERE tr.timestamp > (
+                       SELECT ts_start
+                         FROM paydays
+                     ORDER BY ts_start DESC
+                        LIMIT 1
+                   )
+        """)
+
+    def test_indivisible_team_income(self):
+        # Create a team with 7 members
+        member_1 = self.make_participant('member_1')
+        member_2 = self.make_participant('member_2')
+        member_3 = self.make_participant('member_3')
+        member_4 = self.make_participant('member_4')
+        member_5 = self.make_participant('member_5')
+        member_6 = self.make_participant('member_6')
+        member_7 = self.make_participant('member_7')
+        self.add_payment_account(member_1, 'stripe')
+        self.add_payment_account(member_2, 'stripe')
+        self.add_payment_account(member_3, 'stripe')
+        self.add_payment_account(member_4, 'stripe')
+        self.add_payment_account(member_5, 'stripe')
+        self.add_payment_account(member_6, 'stripe')
+        self.add_payment_account(member_7, 'stripe')
+        team = self.make_participant('team', kind='group')
+        team.set_take_for(member_1, EUR('1.00'), team)
+        team.set_take_for(member_2, EUR('1.00'), team)
+        team.set_take_for(member_3, EUR('1.00'), team)
+        team.set_take_for(member_4, EUR('1.00'), team)
+        team.set_take_for(member_5, EUR('1.00'), team)
+        team.set_take_for(member_6, EUR('1.00'), team)
+        team.set_take_for(member_7, EUR('1.00'), team)
+
+        # Fund the team
+        charlie = self.make_participant('charlie')
+        charlie.set_tip_to(team, EUR('0.85'))
+        charlie_card = self.upsert_route(charlie, 'stripe-card')
+        payin_transfers = self.make_payin_and_transfer(charlie_card, team, EUR('85.00'))[1]
+        transfer_amounts = {pt.recipient: pt.amount for pt in payin_transfers}
+        assert transfer_amounts == {
+            member_1.id: EUR('12.15'),
+            member_2.id: EUR('12.15'),
+            member_3.id: EUR('12.14'),
+            member_4.id: EUR('12.14'),
+            member_5.id: EUR('12.14'),
+            member_6.id: EUR('12.14'),
+            member_7.id: EUR('12.14'),
+        }
+
+        # First payday
+        Payday.start().run()
+        transfer_amounts = dict(self.get_payday_transfers())
+        assert transfer_amounts == {
+            member_1.id: EUR('0.13'),
+            member_2.id: EUR('0.12'),
+            member_3.id: EUR('0.12'),
+            member_4.id: EUR('0.12'),
+            member_5.id: EUR('0.12'),
+            member_6.id: EUR('0.12'),
+            member_7.id: EUR('0.12'),
+        }
+        self.db.run("UPDATE notifications SET ts = ts - interval '1 week'")
+
+        # Second payday
+        Payday.start().run()
+        transfer_amounts = dict(self.get_payday_transfers())
+        assert transfer_amounts == {
+            member_1.id: EUR('0.12'),
+            member_2.id: EUR('0.13'),
+            member_3.id: EUR('0.12'),
+            member_4.id: EUR('0.12'),
+            member_5.id: EUR('0.12'),
+            member_6.id: EUR('0.12'),
+            member_7.id: EUR('0.12'),
+        }
+        self.db.run("UPDATE notifications SET ts = ts - interval '1 week'")
+
+        # Third payday, with an income increased by four cents
+        charlie.set_tip_to(team, EUR('0.89'))
+        Payday.start().run()
+        transfer_amounts = dict(self.get_payday_transfers())
+        assert transfer_amounts == {
+            member_1.id: EUR('0.12'),
+            member_2.id: EUR('0.12'),
+            member_3.id: EUR('0.13'),
+            member_4.id: EUR('0.13'),
+            member_5.id: EUR('0.13'),
+            member_6.id: EUR('0.13'),
+            member_7.id: EUR('0.13'),
+        }
+
     # Two currencies
     # ==============
 
diff --git a/tests/py/test_payins.py b/tests/py/test_payins.py
--- a/tests/py/test_payins.py
+++ b/tests/py/test_payins.py
@@ -167,9 +167,9 @@ def test_resolve_team_donation(self):
             alice_card, team, EUR('3.30'), fee=EUR('0.30')
         )
         assert len(payin_transfers) == 2
-        assert payin_transfers[0].amount == EUR('1.02')
+        assert payin_transfers[0].amount == EUR('1.00')
         assert payin_transfers[0].destination == stripe_account_bob.pk
-        assert payin_transfers[1].amount == EUR('1.98')
+        assert payin_transfers[1].amount == EUR('2.00')
         assert payin_transfers[1].destination == stripe_account_carl.pk