Incentivizing Time-Aware Fairness in Data Sharing
Authors: Jiangwei Chen
Kieu Thao Nguyen Pham
Rachael Hwee Ling Sim
Arun Verma
Zhaoxuan Wu
Chuan-Sheng Foo
Bryan Kian Hsiang Low
Venue: NeurIPS 2025 poster ()
Keywords: Cooperative Game Theory
Incentives
Data Sharing
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A novel time-aware data sharing framework that fairly decides and distributes model rewards to participating parties based on their value of data and time of participation to satisfy incentives.

Abstract: In collaborative data sharing and machine learning, multiple parties aggregate their data resources to train a machine learning model with better model performance. 
However, as the parties incur data collection costs, they are only willing to do so when guaranteed incentives, such as fairness and individual rationality. 
Existing frameworks assume that all parties join the collaboration simultaneously, which does not hold in many real-world scenarios. 
Due to the long processing time for data cleaning, difficulty in overcoming legal barriers, or unawareness, the parties may join the collaboration at different times. 
In this work, we propose the following perspective: 
As a party who joins earlier incurs higher risk and encourages the contribution from other wait-and-see parties, that party should receive a reward of higher value for sharing data earlier. 
To this end, we propose a fair and time-aware data sharing framework, including novel time-aware incentives.
We develop new methods for deciding reward values to satisfy these incentives. 
We further illustrate how to generate model rewards that realize the reward values and empirically demonstrate the properties of our methods on synthetic and real-world datasets.

Incentivizing Time-Aware Fairness in Data Sharing
Jiangwei Chen 12 Kieu Thao Nguyen Pham'
Rachael Hwee Ling Sim'
Arun Verma Zhaoxuan Wu Chuan-Sheng Foo? Bryan Kian Hsiang Low'
Department Computer Science, National University of Singapore, Singapore
Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore "Singapore-MIT Alliance for Research and Technology, Singapore {chenj nguyen pkt, rachaellca....... simm @u. mus.edu {arun. verma, wwaoxuan.n..mmm.oao wu) @smart. wim.iiide.. foo_chuan, shenu@n 2r a-star edu. 8g lowkh@comp.. nus edd..gduugggg
Abstract
In collaborative data sharing and machine learning, multiple parties aggregate their data resources to train machine learning model with better model performance. However, as the parties incur data collection costs, they are only willing to do so when guaranteed incentives, such as fairness and individual rationality. Existing frameworks assume that all parties join the collaboration simultaneously, which does not hold in many real-world scenarios. Due to the long processing time for data cleaning, difficulty in overcoming legal barriers, or unawareness, the parties may join the collaboration different times. In this work, we propose the following perspective: As party who joins earlier incurs higher risk and encourages the contribution from other wait-and-sed see parties, that party should receive a reward of higher value for sharing data earlier. To this end, we propose fair and time-aware data sharing framework, including novel time-aware incentives. We develop new methods for deciding reward values to satisfy these incentives. We further illustrate how to generate model rewards that realize the reward values and empirically demonstrate the properties of our methods on synthetic and real-world datasets.
Introduction
Collaborative machine learning (CML) presents appealing paradigm where multiple parties can aggregate their data to train a machine learning (ML) model with better model performance [49, 63]. This paradigm has been widely adopted in various real-world applications, including clinical trials [8, 33, 17]. data marketplaces [45, 18], precision agriculture [53, 34j. However, parties, who incur non-trivial data collection and processing costs [22], may be unwilling to collaborate without fair reward. For example, hospitals may be reluctant to share their data with academic institutions due to the substantial investment required to conduct clinical trials and generate medical data [19]. To incentivize such parties to collaborate, existing frameworks [43, 49] have proposed two main steps: In data valuation, a party is assigned value for its shared data. reward realization, party will receive money, synthetic data, or an ML model as reward whose value satisfies incentives like fairness and individual rationality adapted from cooperative game theory (CGT). These incentives respectively entail that a party's 's reward value should be higher than that of others sharing less valuable data [54, 41] and higher than what can achieve alone.
In this work, we consider how data valuation and reward realization should change if parties join the data sharing process at different times due to the long processing time for data cleaning. delay in overcoming legal barriers, or unawareness of the collaboration opportunities [33] (see App. A for more justification and description of our setting). As a concrete example, hospitals sharing their data
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

from clinical trials often take heterogeneous time to convert these medical data into standardized format and secure informed consent from their patients [33]. Additionally, a data marketplace mediator, who wishes to encourage participation, may allow parties to freely/asynchronously join the collaboration and continuously update the MI. model with new data until target accuracy is met [1]. Then, the mediator closes the collaboration and rewards the contributing parties. In these examples, how should a party's reward value change it joins the collaboration earlier? Should parties contributing data of the same value at different joining times receive equal rewards?
We propose the following perspective as illustrated by Fig. 1: When parties take different times to join the collaboration, a party should receive a reward of higher value for contributing data earlier to incen- tivize early contribution. 2 Firstly, parties who join earlier incur higher risks and hence require rewards of higher value. In the data marketplace example, a party who contributes data early risks receiving late or no reward, unlike another party who contributes just before the target accuracy is met. Next, the early party's 's contribution may also encourage the contri- bution from other wait-and-see parties who assess the likelihood of reaching the target accuracy before committing [4]. Our perspective also aligns with the pocio-psychologicaa equity theory [16, 47] and the signaling effect [3] observed in economics: Given identical financial contributions, the one contributing earlier is entitled to a reward of higher value.
Without dime-oware incentives (16-40
Partles late
Walt
Tirpet
Time
f
Without faimuss (F1-6)
Partics collect submitt lean
Time
4..5
Oues (with both)
Purtics join ASAP
Process dara
Wast
Time
Figure 1: Overview of our data sharing prob- lem setting (App. A) and the impact of fair- ness and our time-aware incentives (Sec. 4).
Our work addresses two key challenges. The first challenge is mathematically formalize time-aware incentives that realize our above perspective. Naively, our perspective would conflict with fairness incentives which dictates that parties contributing data of equal value receive equal rewards regardless of their joining times. In Sec. 4, we resolve these conflicts by (i) adding a pre- pre-condition (c.g., equal joining times) such that existing incentives (c.g.. fairness) should hold and (ii) only requiring party's reward value to not decrease (instead of increase) from contributing carlier in some cases. The second challenge is to decide the reward values that satisfy the existing and new time-aware incentives. At first glance, one can just divide data value, such as the Shapley value [48, 13], by the joining time. However, this simple solution may overly reduce the reward value of party who joins late, thus violating individual rationality. To tackle this challenge, we introduce two new methods in Sec. 6. Our first time-aware reward cumulation method divides the entire duration of collaboration into multiple time intervals and considers each time interval as a separate collaboration among the parties present during that interval. A party's 's final reward value is a sum of the reward values assigned in each interval, weighted by the time interval importance. Our second method leverages novel time-aware data valuation function and the Shapley value to determine rewards that satisfy all incentives. We then realize the rewards by likelihood tempering [51] and training models only partially on the aggregate data. We empirically validate our proposed time-aware methods in Sec. 7.
2 Related Work
Time Consideration in FL. Federated learning (FL) [38] a form of collaborative machine learning (and an alternative to data sharing) that involves training on data from multiple parties (clients) without centralizing them. Instead, in each round, collaborating parties share parameters' updates for improving the model (e.g., gradients). The bulk of the FL works [26, 65] has focused on improving the model's utility and operated under the assumption that all parties are fully cooperative and do not require external incentives for contributing their data. While there is some research on providing incentives in Fl., such as encouraging collaborative fairness [30, 61] and egalitarian fairness [27, 28), these works do not guarantee that party's 's reward (across rounds) improves from contributing data earlier. To provide such guarantees, we consider the data sharing setting as a first step.
'This Lime-sensitive setting is realistic e-commerce marketplaces have used similar techniques (e.g.. threshold discounting [36], offering discounts to early customers [42]) to encourage prompt actions [31]. We consider settings where data retain value over time (e.g.. medical data), but earlier access enables better utility. We exclude time series data (e.g., stock prices), where recent observations are inherently more predictive.

---

Data Valuation. While existing data valuation methods have employed a range of metrics to value data, such as validation accuracy [21, 13], diversity [64], generalization performance [60], and cost [15], they lack consideration of the parties' joining times (and hence when they contributed their data). [32] can incorporate the parties' joining times based on the assumption that the permutation of parties affects the value of data, but there is no guarantee that party would receive reward of higher value for contributing its data earlier. Although data valuation methods in FL can integrate sequential information by considering the weighted average of the parties" contributions in each round ge utilizing uniform weights [57] or decaying weights [52]), they do not explicitly address nor theoretically guarantee individual rationality F2 and time-aware incentives F6 to F8.
Incentives in CML. [49, 54, 63] have prescribed how to give out data or model rewards that satisfy incentives (such as fairness) based on the popular cooperative game theory concept, Shapley value [48]. However, these works have also assumed that all parties are present at the start of the collaboration and hence do not realize the perspective that party should receive reward of higher value for contributing data earlier. Additionally, while these works suggest how to give out model rewards, our focus is on deciding the reward values for fair and time-aware incentives. [12] is the only other work that studies how to incentivize early arrival in cooperative games. However, our settings, incentives and solutions differ significantly. The key distinctions are discussed at length in App. B.
3 Problem Formulation
We consider problem setting where a trusted mediator (e.g., data-sharing frameworks implemented by government agencies [20]) identifies a common MI task (e.g., disease prediction) of interest to multiple parties (e.g., hospitals). The mediator invites parties to freely/asynchronously join the collaboration by contributing data and continuously updating the MI. model with the newly shared data. The mediator closes the collaboration after a target model performance/accuracy is met [1].
Let n denote the number of parties who have joined the collaboration. We denote the set of all parties
(i..,, grand coalition) as N _ (1...., n) and their respective datasets as D1,..., Dn. Any subset of parties, C S N, is a coalition of parties with an aggregated dataset Dc & Uiec D.. In time-aware data sharing, we further consider that each party i joins at different time value t; Zzo due to differences in processing times, urgency, and risk levels (see App. A for a detailed justification.. A larger time value indicates a later joining time, and the party joining earliest is always assigned time value of O. Collectively, the joining time values are denoted by the vector A (l,,...y l.)). After the collaboration closes, the trusted mediator values the shared data, decides the reward value ri, and trains an ML model as a reward (in short, model reward) valued r2 for every party E N.
party's 's data should be valued relative to the data contributed by the other parties [50]. So, to value data, the mediator uses data valuation function e that maps every coalition C' to its value vo. For example, ee can be the model performance (e.g., validation accuracy) achieved by training on the aggregated dataset De. We also denote v _ vtg-- As in the works of [49, 30), we do not check the truthfulness of the parties' data and value datasets as-is. Based the value VC for each coalition C' C N and the joining time values t, the mediator must decide the reward value r, for every party i E N that satisfies existing fairness and our time-aware incentives to encourage early contribution. We will outline the incentives in Sec. 4, the necessary characteristics for data valuation function e in Sec. 5, and two methods to decide reward values (r:)ieN that satisfy our incentives in Sec. 6.
4 Incentives in Time-Aware Data Sharing
In our time-aware setting, we encourage the parties share their data as early as possible. To this end, we incorporate existing incentives F1 to F5 (e.g., fairness) from prior works [49] and novelly consider the impact of joining time values when defining time-aware incentives F6 to F8. In this section, we formally define the incentive conditions that should hold for reward values (ri)GEN based on the value Ve of aggregated dataset De for all coalitions C' S N and the joining time values t. We also justify how these incentives encourage parties to join early instead of withholding their data with
Foor clarity, we discretize time into non negative integers: Given any earliest joining time t' and positive interval length f, time t maps to [(t - t')/E,. For instance, Windows systems [39] use t' = 1 January 1601 00:00 UTC and l = 100 ns.
3

---

a wait-and-see attitude. We use to mark incentives where we introduce time-aware considerations and # to mark our new incentives that specify how (Ti) should vary for different joining times.
Fl Non-negativity. The value of reward each party receives should be non-negative: Vi N r3 2 0. F2 Individual Rationality, Each party should receive a reward whose value is at least that of its own data: Vi E N 2 0/
F3 Equal-Time Symmetry". If parties and j enter the collaboration simultancously, and the inclusion of party i's data results in the same improvement to the model quality as that of party when trained on the aggregated data of any coalition, then both parties should receive rewards of equal value: Vi,j E N s.t. i + j.
(4=4) (VC C ((s))) routh rou()) = ri ri
F4 Equal-Time Desirability". parties i and j enter the collaboration simultaneously. and the inclusion of party i's data improves the model quality more than that of party j when trained on the aggregated data of at least one coalition, without the reverse being true, then party should receive a reward of higher value than party j: Vi, N i. A J.
(+: tt) A (3B N {i.}} (BU(() > vBUG)) A(AC N (i.5} vcugs 2 )outs) - ri T..
F5 Uselessness'. If the inclusion of party i's data fails to improve the model quality when trained on the aggregated data of any coalition, then party i is useless and should he assigned a reward with no value: Vi E N.
(VC S N \ {i ecu(s vc)
=
In F3 and F4, we add a pre-condition of equal joining time values to accommodate our perspective that a party who joins earlier may receive higher reward value. However, our adaptation of F5 ignores the joining time values to prevent useless party from unfairly increasing its reward to > 0 by joining earlier, which will disincentivize other parties from sharing data.
F6 Necessity If the exclusion of data from either party i or party renders the model trained on the aggregated data of any remaining coalitions useless, then both parties are necessary and should receive rewards of equal value: Vi,j N s.t. i j.
(VC NN {i.j) C - vc = - --
F6 guarantees that parties essential to the success of the collaboration are treated equally despite having different joining time values and datasets.
Justification. A party (c.g., a specialist hospital) may uniquely possess high-quality medical data such that the ML model trained without its data cannot achieve the required accuracy threshold and is untrustworthy and impractical to use [10, 24]. Necessity ensures that necessary party j is not penalized for joining later. Without necessity, j's reward could decrease to O over time, disincentivizing j from curating bigh-quality data, joining the collaboration, causing the collaboration to only have value UN O.
F7 Time-based Monotonicity When a party i joins the collaboration earlier,* ceteris paribus, the value of its reward should not decrease. Let r be the new reward received by party upon the new joining time values t' A (t'..... t.)). Then, Vi N.
( < < E N (i} E, t,) - n
Remark 4.1 (Time- (Timee-ased Equal- Value Desirability ). A natural consequence of F3 and F7 is time-based desirability. That is, if party i joins the collaboration earlier than party j. and party i's data yields the same improvement in model quality that of party j. then the value of reward received by party i should not be less than party j: Vi, E N s.t. i i j.
(44 < (2) (VC C N (i.s} eCuff) = ccun))
ri T
To see this, suppose that t', < ty. Then, 7/ r, by F3. and incentive complements and contrasts with the equal-time symmetry F3.
r/ 2 rj by F7. This
'The earliest party j is always assigned 1, 0. Thus, its reward value need not improve from joining earlier.

---

Remark 4.2 (Reason for weak inequality in F7 and Remark 4.1). F5 requires that any useless party receive reward of value 0 despite their joining times, i... r/ r; 0. F6 requires that necessary parties i, receive equal rewards despite j joining earlier. Hence, r > ri and ri > rj would not always hold in F7 and Remark 4.1. Instead, we need an additional pre-condition on party to define time-based strict monotonicity F8:
F8 Time- Strict Monotonicity # Let II indicate if party i's data yields additional improvement in model quality to that of some coalition comprising only i's predecessors who joined earlier (i... {j : t < t. Formally, 1, _ (3C S {j:t, :t < tt) VCu()) > VC + us). Any party who joins the collaboration earlier can receive a reward of higher value if I, holds: Vi E N.
A (4 < A E N \ (i) ( = (2)
r/
Remark 4.3 (Significance of Iı). Ti guarantees that party i is useless. When the data valuation function is superadditive (Sec. 5), I guarantees that party i's Shapley value (at an earlier joining time) would be positive, facilitating the guarantee of F8 in Sec.
In this section, we have designed time-aware incentive conditions F6 to F8 while resolving conflicts (e.g., introduce equal time pre-condition in F3 and F4, only require time-based strict monotonicity in certain cases). A party may get a higher reward from taking time to curate higher quality dataset instead of sharing a less valuable dataset as early as possible. This holds in two key scenarios: (i) when greater emphasis is placed on data quality relative joining time in the reward schemes; and (ii) when the parties are necessary parties, i.c., their data are essential for achieving non-zero value collaboration. The next step design reward schemes (Sec. 6) that satisfy all these incentives. This is non-trivial as existing frameworks fail to satisfy all incentives.
Failure of the Shapley Value. The Shapley value [48] is popular CGT concept that rewards each party with its average marginal contribution (Vcu(i) - vc) across every possible coalition C' S (td. Given a utility function v 2 + E that measures the value of a coalition,
IC |C| 1)! (Deut)) - -c)
N) &
(I)
[N]
CCNVM
is the Shapley value of within the grand coalition N. When v time-agnostic (e.g., the accuracy of
a model trained on coalition C),, the Shapley value (and CML frameworks [49, 54] that use it directly)
and other weighted sum of marginal contributions [25], fails to satisfy the time-aware incentive F8.
Although dividing the Shapley value by the joining time, i.e., (,(u,N) + 1) satisfies F8, other incentives may be violated. To illustrate, consider a two-party collaboration with 1
2, 2(1.) = 1 and f = 4 and f22 = O. Party I's time-weighted Shapley value would be .5/5 = v1. violating F2. As another example, consider
1/2 0 instead. The Shapley values are both .5
(satisfying F6) but the weighted Shapley values are r22 = o r = .1, thus violating F6.
5 Data Valuation in Time-Aware Data Sharing
To achieve the incentives Fl to F8, it is sufficient for the data valuation function to satisfy:
A1 Non-negativity. The value of aggregated data in any coalition is non-negative: VC S N wc 2 0.
A2 Monotonicity. The inclusion of more parties into a coalition does not decrease the value of the aggregated data: VB C C' C N vo 2 UB
A3 Superadditivity. The value of the aggregated data from two non-overlapping coalitions is no less than the sum of their individual values when the two coalitions are not collaborating: VB,C C N s.t. B C' 0, UBuc 2 UB + VC
Al and A2 align with the norms [50] in CGT and ML, and hold when non- mon-maiiiouus parties contribute valuable data. Crucially, A3 ensures (i) the formation of the grand coalition with all parties [5], (ii) the Shapley value satisfies individual rationality (Lemma F.1) and (iii) our methods (Sec. 6) satisfy the time-aware incentives Sec. 4. A superadditive data valuation function also makes intuitive sense: When parties with less diverse datasets jointly train and deploy a model, the revenue generated by their model with much-improved performance may exceed the total revenue achievable by the parties operating independently. Examples of data valuation function that satisfy Al to A3 include:
5

---

Conditional Information Gain (IG). The conditional IG measures the informativeness of coalition's data Dc conditioned upon the aggregated data from other parties D D_c _ DN App. D proves that the conditional IG (Eq. 2) satisfies A1 to A3. The IG [49] (Eq. 3) is the reduction in the uncertainty/entropy of the model parameters after observing De.
vc - I(00; - Dcl I(;; De, D-c) 1(0; D-c)
(2)
I(0: Dc) & ( (c)
(3)
Dual of Submodular Valuation Function. Submodular" functions are prevalent in ML [2, 55] and data valuation [50]. For example, IG [49] (3) is monotone submodular. The dual function v of a (submodular) function v' is defined by
(C) - v' (N) - (NV \ C), VCEN.
and measures the impportance/contiiutionn of coalition C to the overall collaboration. In App. E. we show that the dual of any monotone submodular valuation function satisfies A1 to A3, This connection is significant as the Shapley value for a valuation function and its dual are equal, i.e., (pi('' NN) = spi (v, N) our proposed time-aware framework can be used to decide reward values (e.g., monetary payments) and satisfy time-aware incentives for submodular functions. Remark 5.1 (Unlearning Data Valuation). The dual valuation function also admits an interpretation from machine unlearning perspective. Machine unlearning [62] aims to obtain a model trained on subset of the original training data, excluding the data to be unlearned (c.g., harmful or privacy- sensitive data). The dual data value of C' is then the difference in performance between the original model and the model obtained after unlearning C. This performance-drop notion of data value can better capture the indispensability of certain data points, as these data are often crucial for pushing model performance beyond a high baseline (e.g.. from 95% to 98%), whereas most data suffice to reach decent performance (e.g., from 10% to 70%). Although the dual data value differs from the original data value, as we show in App. E.2, the Shapley values computed using the dual valuation function and the original valuation function are identical. This equivalence makes it possible to utilize efficient machine unlearning methods [40] to approximate Shapley-based data values without retraining.
Remark 5.2 (Incentives for the Dual). The dual valuation function provides alternative insights into the incentives proposed in Sec. For example, instead of receiving a more valuable reward than party's 's original data, individual rationality F2 can be interpreted as a party's reward must be more indispensable (to the grand coalition with value v' (N(N) than its original data. We defer the relevant proofs and interpretations regarding the dual valuation function to App. E.3.
6 Time-Aware Reward Schemes
This section introduces two time-aware methods (Fig. 2) that consider the joining time values t. The two methods: time-aware reward cumulation (Sec. 6.1) and time-aware data valuation (Sec. 6.2) differ on whether the time consideration is introduced after or before using the Shapley value.
6.1 Time-Aware Reward Cumulation
To incorporate the time information into the reward distribution stage, we consider each time interval as a separate collaboration among parties present during that interval. Specifically, let N, & {i E N t; S 7} denote the set of parties who join the collaboration before time value E Zz0. At time value T, any party & N, is assumed to train model alone and the collaboration
is described by the valuation function restricted to the coalitions in N,, that is, (() 2N, + R such that v/6 UC, VC' N,.
Theorem 6.1. For each party i E N, its Shapley value at time value T, sp pilo N₂) if N, and iú otherwise. Let the weight of time interval t be w(t) & B E/ ST where 3 0 is a tunable parameter and T A maX4eN t, The reward value r; A DT ((())p? satisfies Fl to F8.
(7)
The weights are normalized to ensure feasibility (i.e., r; S UN Vi E N as the model rewards cannot be better than the model trained on all parties' data). The mediator can set 3 to control
A valuation function is submodular if N AC C² C CNN (i) vorut vo (Cu()) EC

---

(Bvery) time
Value
Dota
Value
Leorning Algoritaa
Learning Algocituu
(Every)
Valtan
sulled of owasues with f, S.
Value
(Every) subset of
Dara Valcation Function Nq V Sepertziive
Time-Avars Data Valiotton Punction of SSparddiiiee
Valum Cea
Data D,
S
Data
Weighted
Shopley valion
(A) Time-Aware Reward Cumulation
(B) Time-Aware Valuation Function
Figure 2: Overview of our proposed methods. (A) We partition the collaboration period into time intervals and consider separate cooperative games for each, rewarding parties via weighted sum of the corresponding Shapley values (Sec. 6.1). (B) We propose new time-aware data valuation function and directly the resulting Shapley values the reward values (Sec. 6.2).
the emphasis on joining times and time-aware incentives. As increases, the weight of (T() increases for later time intervals; thus, there is less emphasis on the joining times of parties. Conversely, as 33 decreases, more weight is placed on the earlier time intervals. In fact, as 3 - oc, the reward value r; - s (2)) which coincides with the Shapley value when all parties have joined, thus no time information is accounted for. The proof of Theorem 6.1 can found in App. F.I. Remark 6.2 (Efficient Estimation due to Linearity). Instead of computing the Shapley value T times, possible to exploit the linearity property of the Shapley value to express r; as the Shapley value of valuation function defined all subsets of N: VC
(r) UCON, Ejecr,, Vj, i.e.,
A :p: <E: w(()) .N). This reduction allows our time aware reward cumulation method to be combined with other Shapley value approximation methods [46, 21] for efficient estimation. Our time-aware setting only increases the computational complexity by a factor of at most n (the maximum number of unique joining times).
6.2 Time-Aware Data Valuation
The next method replaces the data valuation function v(-) with a time-aware data valuation function "(.ee) and makes use of propositions from [35] in the CGT literature. [35] has proposed that when e is superadditive and parties have different cooperative levels (denoted by the vector À & (x:) [0, 1)"), the Shapley value computed based on
Σ d(v, (Ad) + Ed(e, (i)) TCC,TT12
UC,A
FET
IEC
satisfies monotonicity (i.e., A; increases, ()) N) also increases). Here, the Harsanyi div- idend [14] d(v, removing the contributions of its sub-coalitions [56].6 6 We provide a detailed discussion of their results in App. F.2. In our work, we define party i's cooperative level by how early it joins. Formally,
Esgr d(v. S) measures the unique contribution of coalition T after
Theorem 6.3. For each party i E N, let party i's cooperative ability X, e where is the time value when joined and y E (0, 1] is a tunable parameter. The tine-aware data valuation funcctiono is
VC,t A
Σ
d(v, min(e NE
11 dd(, ((),(it)
(5)
TCC, TI22
IEC
for all C C N. When v is non-negative and superadditive, V(.)) is also superadditive. Rewarding based on the Shapley value, i.e., ri 2 p.(vi.,),, satisfies Fl to F8.
The mediator can set y to control the emphasis on joining times and time-aware incentives. When 7 0, the joining time does not matter and every party will have the maximum cooperative level of 1. As increases, the joining time has a larger influence on data values.
'The dividend 0 for empty coalitions, i.e., d(v, 0)

---

100
8-01
100
20
Roward
Reward
75
105 pl
R-07
10
yl
Pe1
25
P
(a) r with varying 8 (b) r with varying
(a) 72
(b) r
Figure 3: Graphs of r vs. l with the Friedman Figure 4: Graphs of differences between reward
dataset using methods in (a) Sec. 6.1 (b) Sec. 6.2. values with the Friedman dataset.
For y > 0. party i only attains the maximum cooperative level of 1 when 4, O. As & increases, party i's cooperative level shrinks towards 0. Thus, the monotonicity property implies that the reward value would decrease. We provide proof of Theorem 6.3 and efficient computation of Eq. (5) in App. F. Similar to the previous approach, this method retains the computational benefits of efficient approximation, and the time-aware setting only increases the complexity by a factor of at most n.
6.3 Reward Realization
Multiplying the reward values (ri)kEN by positive factor 2 1 still satisfies incentives Fl to F8. This allows us to exploit the freely replicable nature of data by training models with varying qualities to realize values r ê pr, where p = uN/ max,e (u N). The adjusted values (ri)seN ensure that in the aannagtootic case, at least one party is guaranteed to receive reward equivalent to the best-performing model trained on all aggregated data, i.c., when t; = 0 Vi e Ni e N rf = UN (weak efficiency incentive in [49]).
We adopt two approaches to realize the reward values: the likelihood tempering method and the subset selection method. The former assigns party model reward (posterior) p. that is updated using its own likelihood and the tempered likelihood of other parties' data (as in [51]). This method can realize the rewards exactly and is suitable when conditional IG is used as the data valuation function. The latter assigns party i a model reward that is only updated on a discrete subset the aggregated data. Although this latter method can only realize the rewards approximately, it is applicable to all data valuation functions. More detailed descriptions of both methods can be found in App. G.
7 Experiments and Discussion
This section empirically illustrates the properties of our proposed reward schemes using (a) the synthetic Friedman dataset with 6 input features [11]. (b) the Californian housing (CaliH) dataset [44] with 8 input features, and (c) the MNIST dataset [7] of handwritten digit images (28 x 28 pixels). We employ the Gaussian process (GP) regression [59] model for Friedman and CaliH datasets and neural network (NN) for the MNIST dataset. In (a) and (b), each party's s data value is measured by conditional IG. In (c), each party's s data value is the dual? of the validation accuracy, i.e., given the validation accuracy v' oD) of the model trained on D, (D.)) = vDDn)) - v'(Dr \ D.). Next, each party gets model reward P generated by likelihood tempering (a. b) or subset selection (c). We also report the model performance evaluated by mean negative log probubility (MNLP), defined as v/alval Σ (x,)) log (v/x) where D is the validation set. In (c), MNLP equals to the cross-entropy loss D A lower MNLP indicates better model performance. Following [49, 51], we consider n = 3 parties in our main paper (and n = 10 parties in App. H.6). Party i's reward depends on both its joining time and data value 0; To explore the impact of joining times on rewards, we increase f from O while keeping to = f3- = 0. For (a) and (b), we indirectly control party i's valuation 1; by varying the number of data points r4;- Each party's 's data D; is randomly sampled (without replacement) from Drain- When all parties draw data uniformly from the same distribution (but do not have sufficient data to achieve the best model performance), the number of data points is positively correlated with the data values, i.c., > T22 typically leads to U1 > U2, allowing us to demonstrate desirability (F4). For (c), we vary each party's data value by restricting the labels of their data. For baselines, we compare against the Shapley value, the standard approach in
7 As the accuracy is approximately submodular, its dual satisfies the properties outlined in Sec. 5.
8

---

200
200
L18
1.18
Reward
150
Reward
150
MNLP
116
Rob1
MNLP
1.16
8:37
1.14
100
100
112
1.12
50-
0
1
2
5
6
8
0
3
4
8
8
(a) rI with varying
(b) r with varying 7
(c) MNLP with varying 3 (d) MNLP with varying 7
Figure 5: Graphs of (a, b) reward values and (c, d) MNLP vs. joining time 4 on the CaliH dataset.
incentive-aware CML without incorporating joining time information [49, 43]. It is special case of our methods when y = 0 for time-aware data valuation and | - oo for time-aware reward accumulation and correspond to horizontal lines in experimental results (c.g., Fig. 3b) Shapley value-based rewards remain constant regardless of joining time and do not satisfy incentive F8.
Overview of Observations. All experiments using both reward schemes show the satisfaction of incentives: (i) All parties benefit from collaboration and receive rewards more valuable than their own data (F2). (ii) When a party delays its participation, ceteris paribus, it receives model with worse performance (F8). In the time-agnostic case, (iii) parties with higher Shapley val- ues sps (usually reflected as higher data values v;) receive models with better performance (F4) and (iv) there is always one party who receives the best model with value ey (weak efficiency). Friedman Dataset (n1 n2 na). In Figs. 3-6, we sample parties' data from the Friedman dataset, creating a scenario where 11 = 10.58
is close to 1/2 9.43, while Vty 5.48 is significantly smaller. When t = 1/2 = t3 = 0, the Shapley values are Φ1 = 35.88, 62 = 34.64, Ps = 28.40, with party receiving model with the highest possible value UN, as seen in Fig. 3. As party joins later (t, increases), Fig. 3 shows that its reward ri decreases as a disincentive for joining late. While F8 only stipulates a decrease in ri. we also see a drop in rī and r3. This is because other parties receive benefits from party I's collaboration for shorter duration, resulting in lower total rewards. However, each party is still guaranteed individual rationality (F2), i.e., all parties receive rewards at least as valuable as their own data (plotted as grey horizontal lines in Fig. 3).
MNLP
0.36
0.17-
-0.18
(a) MNLP with varying /
-0, 14
MNLP
00.15
0.16
Pery
-0.1
Paryy
Pury
Next, we investigate the impact of a party's joining time value (e.g.. (1) on the difference of its reward value with others (e.g., r and r rz). When the gap between parties' data values are small and 7 > 09, the effect of the joining time values is dominant. Although party 1 possesses more valuable data (v) > 02), in Fig. 4a, we observe that party receives a lower reward than party 2 (ri - r < 0) when it joins too late. When 3 = 1 and y = 0.5, party 1 receives lower reward than party 2 if it joins at t1 3. When 3 0.7 and 7 1, party receives lower reward if joins at t 2 2 and ri rš decreases faster. Thus, we observe that using smaller 3 or larger y will increase the emphasis on earlier participation and time-based desirability.
-0.18
(b) MNLP with varying y
Figure 6: Graphs of MNLP vs. ti on Fried- man dataset using (a) time-aware reward cu- mulation and (b) time- aware data valuation.
However, when there is significant gap between the data values of two parties (e.g.. U3 & vi), Fig. 4b shows that party I would always receive a higher reward than party 3 despite joining later (i.e., rī - r > 0). Thus, our time-aware framework balances the consideration of data values and time values and encourages all parties to both curate high-quality data and join earlier to receive higher rewards. We further verify in Fig. 6 that models with higher reward values r; also have better predictive performance.
CaliH Dataset (n1 (n))))>)g). We construct different scenario with CaliH dataset with significant gaps between the data values (v1 = 83.15, 822 = 57.17, V3y = 26.86). We report conditional IG and MNLP of the received rewards using both reward schemes in Fig. 5. We observe that the value of
'When 7 = 0, vC = vc. The reward values of all parties are time-agnostic and constant across the time values, resulting in the horizontal 0 lines in Figs. 3-4.

---

MNLLP
MNLP
Reward
0.6
Reward
0.6
0.4
0.4
0.2
2010
0.2
8-20
(a) r. with varying B
(b) ri with varying (c) MNLP with varying 8 (d) MNLP with varying
Figure 7: Graphs of (a, b) reward values r and (c, d) MNLP (averaged over 6 independent realization
with standard deviation shaded) vs. different joining time value t the MNIST dataset.
rewards always exceeds the value of each party's own data. In addition, the model performance of party 1 decreases as it joins later. These observations are in line with incentives F2 and F8. Lastly, there is no change in the ranking of each party's 's rewards across all joining times, as the large gaps between the data values outweigh the benefits of earlier participation.
MNIST Dataset. Each party has access only data with limited subset of labels (see App. H) and has data values U1 01=0.16, 0y=0.19, 2g=0.26. Fig. 7 reports assigned rewards and reward model performance under both reward schemes. Our reward values adhere to the desiderata in Sec. 4: all assigned rewards outperform individual values (F2), and rewards decrease when joining later (F8). Reward model performance generally declines with later joining times, and party 3, receiving the highest reward, achieves the lowest MNLP, thereby penalizing late joiners and encouraging high- quality data curation. However, the trend is not strictly monotonic, likely due to approximation errors in subset selection and randomness in NN training. Nevertheless, this does not contradict theoretical results. Our theoretical results are only for the reward values (Fig. 7a-b) and in Fig. 7c-d, we are optionally examining the impact MNLP (another measure of model performance) when using the subset selection reward realization mechanism.
Conclusion
This paper seeks to encourage parties to join data sharing collaboration early and curate high-quality data. To this end, we define time-aware incentives that complement existing fairness incentives and propose two time-aware reward schemes that satisfy all incentives, and have parameters to control the emphasis on joining times and earlier participation. Our empirical evaluations show that the incentives hold with respect to the data valuation function and the model predictive performance. We discuss the limitations of our work in App. C, focusing computational efficiency and privacy.
A Acknowledgments and Disclosure of Funding
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: the National Research Foundation (NRF). Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) is an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre. Jiangwei Chen is supported by the Institute for Infocomm Research RI²R), Agency for Science, Technology and Research (A*STAR). We would like to thank the anonymous reviewers and the AC for their helpful and constructive feedback.
This research is supported by
References
[1] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In Proc. EC, pages 701-726, 2019.
[2] Jeff A. Bilmes. Submodularity and adaptation. In IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 249-249, 2007.
10

---

[3] Sylvain Béal, Mare Deschamps, Catherine Alexande, and Guillaume Sekli. Early contributors, cooperation and fair rewards in crowdfunding, Working Papers 2022-07, CRESE, 2022.
[4] Timothy N. Cason, Alex Tabarrok, and Robertas Zubrickas. Early refund bonuses increase successful crowdfunding. Games and Economic Behavior, 129:78-95, September 2021.
[5] Georgios Chalkiadakis, Edith Elkind, and Michael Wooldridge. Computational aspects of cooperative game theory. Morgan & Claypool Publishers, 2011. Synthesis Lectures on Artificial Intelligence and Machine Learning.
[6] Thomas M. Cover. Elements of information theory. John Wiley & Sons, 1999.
[7] Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[8] Jeffrey M. Drazen, Stephen Morrissey, Debra Malina, Mary Beth Hamel, and Edward W. Campion. The importance and the complexities of data sharing. New England Journal of Medicine, 375(12):1 11122))::11111111 1183, September 2016.
[9] Bradley Efron, Trevor Hastie, Jain Johnstone, and Robert Tibshirani. Least angle regression. Annals of Statistics, 32(2):407-499 April 2004.
[10] Mehran Farajollahi and Ali Modaberi. Can ChatGPT pass the "iranian endodontics specialist board" exam? Iranian Endodontic Journal, 18(3):192, June 2023.
[11] Jerome H. Friedman. Multivariate adaptive regression splines, Annals of Statistics, 19(1):1-67, March 1991.
[12] Yaoxin Ge, Yao Zhang, Dengji Zhao, Zhihao Gavin Tang, Hu Fu, and Pinyan Lu. Incentives for early arrival in cooperative games. In Proc. AAMAS, pages 651-659, 2024.
[13] Amirata Ghorbani and James Zou. Data Shapley: Equitable valuation of data for machine learning. In Proc. ICML, pages 2242-2251, 2019.
[14] John C. Harsanyi. A bargaining model for the cooperative n-person game. In Albert William Tucker and Robert Duncan Luce, editors, Contributions to the theory of games, volume IV, pages 325-356.
Princeton Univ. Press, 1959.
[15] Judd Randolph Heckman, Erin Laurel Bochmer, Elizabeth Hope Peters, Milad Davaloo, and Nikhil Gopinath Kurup. A pricing model for data markets. In iConference 2015 Proceedings, 2015.
[16] George C. Homans. Social behavior: Its elementary forms. Harcourt Brace Jovanovich, revised edition, 1974.
[17] Ashley M. Hopkins, Andrew Rowland, and Michael J. Sorich. Data sharing from pharmaceu- tical industry sponsored clinical studies: Audit of data availability. BMC Medicine, 16:165, September 2018.
[18] Lihua Huang, Yifan Dou, Yezheng Liu, Jinzhao Wang. Gang Chen, Xiaoyang Zhang, and Runyin Wang. Toward a research framework conceptualize data as a factor production: The data marketplace perspective. Fundamental Research, 1(5):586-594, September 2021.
[19] Tim Hulsen. Sharing is caring data sharing initiatives in healthcare. International Journal of Environmental Research and Public Health, 17(9):0046, April 2020.
[20] IMDA. Trusted data sharing framework. Technical report, Infocomm Media Development Authority of Singapore, 2019.
[21] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas Spanos. Towards efficient data valuation based on the Shapley value. In Proc. AISTATS, pages 1167-1176, 2019.

---

[22] Sai Praneeth Karimireddy, Wenshuo Guo, and Michael Jordan. Mechanisms that incentivize data sharing in federated learning. In Proc. NeurIPS Workshop on Federated Learning: Recent Advances and New Challenges, 2022.
[23] Alex Krizhevsky. Learning multiple layers of features from tiny images. Techni- cal report. University of Toronto, 2009. URL https://www cs. torontoo......m edu/"kriz/ learning faaming-featurese features- TR. pdf.
[24] Jakub Kufel, Iga Paszkiewiez, Michal Bielówka, Wiktoria Bartnikowska, Michal Janik, Mag- dalena Stencel, Lukasz Czogalik, Katarzyna Gruszczyúska, and Sylwia Mielcarska. Will ChatGPT pass the polish specialty exam in radiology and diagnostic imaging? insights into strengths and limitations. Polish Journal of Radiology, 88:e430-c434, 2023.
[25] Yongchan Kwon and James Zou. Beta Shapley: A unified and noise-reduced data valuation framework for machine learning. In Proc. AISTATS, pages 1780-8802, 2022.
[26] Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin. A review of applications in federated learning. Computers Industrial Engineering, 149:106854 November 2020.
[27] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In Proc. ICLR, 2020.
[28] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In Proc. ICMI,, pages 6357-6368, 2021.
[29] Weida Li and Yaoliang Yu. Faster approximation of probabilistic and distributional values via least squares. In Proc. ICLR, 2024.
[30] Xiaoqiang Lin, Xinyi Xu, See-Kiong Ng, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Fair yet asymptotically equal collaborative learning. In Proc. ICML, pages 21223-21259, 2023.
[31] Ying-Chun Lin, Chi-Hsuan Huang, Chu-Cheng Hsieh, Yu-Chen Shu, and Kun-Ta Chuang. Mon- etary discount strategies for real-time promotion campaign. In Proc. WWW, page 1123-1132, 2017.
[32] Jie Liu, Peizheng Wang. and Chao Wu. Data valuation: The partial ordinal Shapley value for machine learning, arxiv:2305.01660, 2023.
[33] Bernard Lo and Steven N. Goodman. Sharing clinical research data finding the right balance. September 2017.
JAMA Internal Medicine,
[34] Greg Manson. WSU to lead new research institute to apply artificial intelligence innovations to farming. The Baakkesman-Reviieee July 2021.
[35] Conrado M. Manuel and Daniel Martín. A monotonic weighted Shapley value. Group Decision and Negotiation, 29:627-654, 2020.
[36] Simone Marinesi, Karan Girotra, and Serguei Netessine. The operational advantages of threshold discounting offers. Management Science, (666):269027788 2018.
[37] Michael Maschler, Bezalel Peleg, and Lloyd S. Shapley. Characterization of the kernels of convex games. Research program in game theory and mathematical economics, research memorandum no. 36, Department of Mathematics, Hebrew University of Jerusalem, July 1968.
[38] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-cfticcaa learning of deep networks from decentralized data. In Proc. AISTATS, pages 1273 1282,, 2017.
[39] Microsoft.
[MS-DTYP]:
FILETIME.
https://learn.
mi crosoftt com/en womm/enus//ee wtt.uuu/ppenspee/viiioooo http:///wioooospoooooossee/////e wwwpocols//monoooommo pestocoosmmmsdtyp//p
2c57429b- -5fc- ce4fff20f 2024. Accessed: 2024-01-23.
[40] Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajert Descent-to-delete: Gradient-based methods for machine unlearning. In Proc. ALT, volume 132, pages 931-962 June 2021.
12

---

[41] Quoe Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Trade-off between payoff and model rewards in Shapley-fair collaborative machine learning. In Proc. NeurIPS, pages 30542-30553, 2022.
[42] Mina Nouri-Harzvili and Seyyed-Mahdi Hosseini-Motlagh Dynamic discount pricing in online retail systems: Effects of post-discount dynamic forces. Expert Syst. Appl., 232((20864)) December 2023.
[43] Olga Ohrimenko, Shruti Tople, and Sebastian Tschiatschek. Collaborative machine learning markets with nntaarrpllication-rbuus payments. arXiv:1911.09052 2019.
[44] Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-297, May 1997.
[45] Gowri Sankar Ramachandran, Rahul Radhakrishnan, and Bhaskar Krishnamachari. Towards decentralized data marketplace for smart cities. In IEEE International Smart Cities Conference (ISC2), pages 1-8, 2018.
[46] Benedek Rozemberezki, Lauren Watson, Péter Bayer, Hao-Tsung Yang. Oliver Kiss, Sebastian Nilsson, and Rik Sarkar. The Shapley value in machine learning. In Proc. IJCAI, pages 5572-5579, 2022.
[47] Reinhard Selten. The equity principle economic behavior. In H. W. Gottinger and W. Le- infellner, editors, Decision Theory and Social Ethics: Issues in Social Choice, volume 17 of Theory and Decision Library, pages 289-301. Springer, Dordrecht, 1978.
[48] Lloyd S. Shapley, A value for n-person games. In H. W. Kuhn and A. W. Tucker, editors, Contributions to the Theory of Games, volume 2, pages 307-317. Princeton Univ. Press, 1953.
[49] Rachael Hwee Ling Sim, Yehong Zhang, Mun Choon Chan, and Bryan Kian Hsiang Low. Collaborative machine learning with incentive-aware model rewards. In Proc. ICML, pages 8927-8936, 2020.
[50] Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning: "ingredients", strategies, and open challenges. In Proc. IJCAI, pages 5607-5614, 2022.
[51] Rachael Hwee Ling Sim, Yehong Zhang, Trong Nghia Hoang, Xinyi Xu, Bryan Kian Hsiang Low, and Patrick Jaillet. Incentives in private collaborative machine learning, In Proc. NeurIPS, 2023.
[52] Tianshu Song, Yongxin Tong, and Shuyue Wei. Profit allocation for federated learning. IEEE International Conference on Big Data, pages 2577-2586, 2019.
[53] Konstantina Spanaki, Erisa Karafili, and Stella Despoudi. Ai applications of data sharing in agriculture 4.0: A framework for role-based data access control. International Journal of Information Management, 59: 19:100350, August 2021.
[54] Sebastian Shenghong Tay, Xinyi Xu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Incen- tivizing collaboration in machine learning via synthetic data rewards. In Proc. AAAI, pages 9448 9456, 2022.
[55) Ehsan Tohidi, Rouhollah Amiri, Mario Coutino, David Gesbert, Geert Leus, and Amin Karbasi. Submodularity in action: From machine learning to signal processing applications. IEEE Signal Processing Magazine, 37(5):120-133 September 2020.
[56] René van den Brink, René Levínský, and Miroslav Zelený. The shapley value, the proper shapley value, and sharing rules for cooperative ventures. Operations Research Letters, 48(1): 55 60, January 2020.
[57] Tianhao Wang, Johannes Rausch, Ce Zhang, Ruoxi Jia, and Dawn Song. A principled approach to data valuation for federated learning. In Q. Yang, L. Fan, and H. Yu, editors, Federated Learning: Privacy and Incentive, volume 12500 of Lecture Notes in Computer Science, pages 153-167. Springer, Cham, 2020.
13

---

[58] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 15:3454-3469, 2020.
[59] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes procesessr machine learning. MIT press, 2006.
[60] Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. DAVINZ: Data valuation using deep neural networks at initialization. In Proc. ICML, pages 24150-24176, 2022.
[61] Zhaoxuan Wu, Mohammad Mohammadi Amiri, Ramesh Raskar, and Bryan Kian Hsiang Low. Incentive-aware federated learning with training-time model rewards. In Proc. ICLR, 2024.
[62] Heng Xu, Tianqing Zhu, Lefeng Zhang. Wanlei Zhou, and Philip Yu. Machine unlearning: survey. ACM Comput. Surv., (6(1):1133300 August 2023.
[63] Xinyi Xu. Lingjuan Lyu, Xingjun Ma, Chenglin Miao, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Gradient driven rewards to guarantee fairness in collaborative machine learning. In Proc. NeurlPS, pages 16104-16117, 2021.
[64] Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-based data valuation. In Proc. NeurlPS, pages 10837-10848, 2021.
[65] Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge
Maased Based Systems, 216:106775, March 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Our claims are supported by both theoretical and empirical evidence. The theoretical evidence is provided in Sec. 6 and App. F. The empirical evidence is provided in Sec. 7 and App. H.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in App.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: Most of our theoretical results are related to the Shapley value and charac- teristic/valuation function. The sufficient conditions of the characteristic functions for our methods are discussed in Sec. 5. The relevant proofs about the characteristic functions are provided in App. D and App. E. The complete proofs of theorems are provided in App. which do not require more assumptions than what have been discussed in Sec. 5.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All datasets used in the paper are publicly accessible, we have also uploaded the code and instructions needed to reproduce the results in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used in the paper publicly accessible, we have also uploaded the code and instructions needed to reproduce the results in the supplementary materials. Once the blind review period is over, will open-source code and instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cublic/guides. /ogglicc/uiids//oeeetttiiiss oessonnnllcy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //niss. cctpublic/guieeeeeccccccccocccce gwwcc//g///////eeeeeeeeceee.ceee /ttpg///c///ueummmiiiic/e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe our experimental settings in Sec. 7, with more details on data splits, hyperparameters, models chosen in App. H.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the standard deviation of different realization for the subset selection method. For the likelihood tempering reward realization method, we did not report the error bars since the results are exact.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the specifications of the hardware used for running the experiments in App. H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eurrrppeeeee http///wwrsspppuuuueeeeeee htcppppbccc//icccccudeeee.l...el
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and made sure that our paper conforms
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
18

---

The purpose of this paper is to propose a framework that incentivizes early collaboration between ML parties, without negative societal impacts.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have provided references for all the datasets used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: /tt////eerippppepe http://pppp..ccomeeeeeree/////// ccc.Confeeenncc///25//////////// for what should or should not be described.
21

---

A Further Justification on the Time-Aware Data Sharing Setting
A party may not want or be able to share their data early because:
They may be waiting for confirmation that the collaboration benefits them.
In footnoee 1, we describe how e-commerce marketplaces restrict opportunities to customers to create urgency and encourage prompt actions. Similarly, data sharing mediator may announce that it will try to collect data but only give out model rewards if the model has good enough performance (e.g.. 90% accuracy) such that it is worth the effort of all parties and extra costs ge.g,, legal fees). The mediator may regularly update parties about the current performance and a wait-and-see party might only start contributing when the accuracy is near 80%.
They may take time to process the data and the time can be sped up.
For example, an incentivized healthcare firm may be proactive in seeking consent from their patients, legal consent, and anonymizing it.
They may take more time to collect more data.
In situations where longer time means more data collection, the healthcare firm can submit multiple datasets and submit each dataset as soon as possible to maximize its reward, i.e. it becomes multiple parties. Our framework can also be modified to support the case where it is just one party. We are not incentivizing smaller datasets over larger ones. Instead, our goal is to incentivize submission as soon available.
Thus, we design incentives to incentivize each party to share their data early possible. Note that while we incentivize parties to share the same dataset as early as possible, our consideration of fairness, F6 and data valuation ensures that we do not incentivize parties to submit smaller, lower quality, less diverse datasets to join earlier in the collaboration. Our experiments (e.g.. Fig. 4b) also empirically demonstrate that parties with significantly more valuable data will still receive better rewards despite joining late.
While it may sharing, data valuation, and reward allocation, it may not be possible to ensure our strong thcoretical properties on the reward values. Thus, we consider one-time data sharing as first step and leave federated learning to future work.
more practical to consider the online federated learning setting with repeated data
[49] described some realistic use cases of data sharing. In precision agriculture, a farmer with limited land area and sensors can combine his collected data with the other farmers to improve the modeling of the effect of various influences (e.g., weather, pest) his crop yield. In real estate, property agency can pool together its limited transactional data with that of the other agencies to improve the prediction of property prices. Additionally, we note that firms and other parties have and will be willing to share their data with one another using secure sharing platforms (such as X-road [https: ptttps:/xx// //wdd global] which is currently implemented in over 20 countries) which prevent data from being accessed by unauthorized individuals.
It is also worth noting that the centralerl training (e.g.. federated learning), while appealing, does not strictly guarantee privacy either. For example, [58] shows that "private information can still be divulged by analyzing uploaded parameters from clients". . Hence, the challenge of preserving privacy non-trivial in both data-sharing collaboration (CML) and non-data-sharing collaboration (FL.) and lies outside the scope of our contributions.
B Key Differences with [12]
Settings. [12] considers the online setting (i.c., distributing rewards every time party joins), whereas we consider the offline setting (i.c., distributing rewards after all parties join). The online setting poses challenges in adjusting rewards to ensure overall fairesss if one party's data value changes when additional parties join, as it is difficult to claw back rewards that have already been distributed. In contrast, focus the offline setting, where reward distribution occurs only after the collaboration terminates under a predetermined condition (Sec. 3), ensuring fair evaluation and compensation.
Additionally, we consider the differences in joining times while [12] considers only the order or permutation of arrivals. Our setting is more realistic since in CML, day's and month's 's wait have
22

---

different impacts. We introduce this notion by discretizing joining time values (Sec. 3) and propose equal-time incentives F3, F4 and strict time monotonicity F8 (Sec. 4).
Incentives. [12] enforces online individual rationality (OIR) which ensures that parties' rewards will not decrease when new parties join. However, this could be unfair in CML and other data valuation applications. For instance, if the validation accuracy is used as the valuation function, the first contributor could receive credit for the majority of the accuracy improvement, say, from 0 to 0.8, and this credit would not decrease under OIR, even if a later contributor with more valuable data increases the accuracy from 0.8 to 0.99. In this scenario, a fairer evaluation would reduce the first contributor's ss attributed contribution. However, in the online setting, as previously discussed, there is no mechanism to retrieve rewards that have already been distributed.
Instead, we build upon faireess incentives from existing CML literature to incorporate joining time values when defining time-aware incentives (Sec. 4), aiming encourage both early participation and the curation of high-quality data. Specifically, the method proposed in [12] fails to satisfy our F3, F4, F6 and F8 incentives.
Solutions. Our solutions incentive early participation across a broader range of valuation functions. We outline the necessary properties of the valuation function to satisfy all of our proposed incentives and provide a sufficient example (conditional IG) in Sec. 5. In contrast, Corollary 5.7 in [12] indicates that their solution is restricted to symmetric monotone valuation functions (i.e., vSS) v(T) if |S| = [T], and v(S) S (TT) if S T"), which significantly limits its practical applicability.
C Limitations
Computational Efficiency. One limitation of our work is that the exact computation of the proposed methods requires enumerating all possible subsets of the grand coalition, resulting in exponential computational complexity. However, the scalability challenge not unique this work; It applies to any approach [54, 32, 13, 50] that computes data values using the Shapley value. The only difference is that, incorporate the additional joining time information, our proposed methods increase the computational complexity at most linearly, i.e., by the maximum number of unique joining times,
This challenge can be avoided mitigated for two reasons. Firstly, in our collaborative data sharing and machine learning setting [49], there are are fewer participating parties, each contributing higher- quality data sources (e.g.. hospitals holding larger datasets). Secondly, discussed in Remark 6.2 and App. F4,, our theoretical results would also hold for unbiased Shapley value estimators, allowing the use of any Shapley value approximation method [21]. On average, the approximate Shapley values still incentivize parties to curate high-quality data and join the collaboration early. Using approximate Shapley values would significantly reduce the number of coalitions evaluate O log n) and scalability would also improve with the discovery of more efficient approximation techniques [29]. We evaluate on 10 parties App. H.6 using Monte Carlo approximation methods and demonstrate that our results remain consistent with an increased number of parties and Shapley value estimation.
Privacy Issues of Non-FL Setting, Another limitation of work is that data sharing poses privacy risks due to the sensitive nature of the data. While it may seem more practical to consider the online federated learning (FL.) setting with repeated data sharing, data valuation, and reward allocation, it may not be possible to ensure our strong theoretical properties on the reward values. Therefore, we consider one-time data sharing as a first step and leave the extension FL for future work.
is also important to note that the non-centralized training (c.g., FL), while appealing, does not strictly guarantec privacy either. For example, [58] shows that "private information can still be divulged by analyzing uploaded parameters from clients". Hence, the challenge of preserving privacy is non-trivial in both data-sharing collaboration (CML) and non-data- -haaiin collaboration (FL), and lies outside the scope of our contributions.
Additionally, we note that firms and other parties have and will be willing to share their data with one another using secure sharing platforms such as X-road (https: //x-roxdd gllobal) which currently deployed in over 20 countries and prevents unauthorized access to shared data.
23

---

D Proof of Characteristics of Conditional IG
To ease notations, we define v(.) 2 + R by v(A) = Da AA S N where VA = T(0; Da(D-A) is the conditional IG (2). We need show that is non-negative, superadditive, and monotonic.
Al Non-negativity. The non-negativity of v is apparent from the fact that conditional mutual information is never negative [6].
A3 Superadditivity. We need show AA, B N such that An B 0.
B) 1(0; DAUB (D-(AB))
2 1(0; DA D-(D=A)
= (A) + (B))
By chain rule of mutual information, T(0: D DAUB
T(0: DAID DAID--(UB))
+
10; D-BB) It is then sufficient to show
T(0; (A)LLAB)))) 2 ((0:DA|D-A)
(6)
which is equivalent H(Da D(DABD-(AUB)) D(AUU) 2 H(DA(D-A) H(,, D-A) by the definition of conditional mutual information. If we assume" that DA is conditionally independent of D given O, then
H(DA D-(AUB)) HDDA 0) H(DA)O.D-A)
2 ((DA(D-A) since additional information cannot increase uncer- tainty (entropy). Thus,
Finally,
(AUB)) 2 I(e; D-A)). We have successfully shown (6),
which concludes the proof.
A2 Monotonicity. We need show vB S C S v(B) < v(C). Let A C\ B, then AnB and JUB = C. By the superadditivity of x(C) = w (BBU A) u(B) "(A) 2 BBB), where the last inequality because v(A)) 2 0.
E Properties of Dual Valuation Function
E.1 Dual of a Monotonic and Submodular Function
We show that the dual of the monotonic, submodular function vf non-negative, monotonic, and superadditive.
Al Non- negativviy, VC C N v(C) = v'(N) C) 0 where the inequality is due to the monotonicity of o'.
A2 Monotonicity, VB CCCN v(C) = C(N) vN () 2 ('(N) o'N B) v(B) since (N (C) < VN B) by the monotonicity o'.
A3 Superadditivity. By the definition of submodularity, for any (. ''Cn'n
D) < (C(C)= + dDD). For any A, B C N such that An B 0, we can choose C = N A. D = N B, so CUD N since A n B @ and CnD N (AUB), B). Therefore,
V(CU D) s
+NN B)
((A(N)(AA (A)) < (AUB))
d'(N) v' A) ( ((N))
v(A) s
proving the superadditivity of
This assumption is also made [49].
24

---

E.2
Equivalence of Dual when Calculating the Shapley Value
For dual valuation function v, its Shapley value calculated by
C
|C|
1)!
()(
Σ
(v(Cu =(C))
|N!
C
|C
C 1)!
Σ
-
N
CSNVO
C
- CC 1)!
Σ CENA()
(N
N!
(
where the last equality is because of the bijection between the sets {c C S N (i) {N (CU C (i)).
and
E.3 Dual Interpretations of Incentives
As we define the incentives in Sec. 4 with superadditive valuation function v, which can be the dual of a submodular valuation function v', we investigate the interpretations of these incentives by expressing in terms of of in the definitions. We illustrate this using the subset selection reward realization method as applicable to dual of any submodular functions. Generally, the subset selection method assigns model trained on D: C Dj C DN reward to party such that (D;) = r (see App. G for more details). Remarkably, we find that F3 to F5 are equivalent to the corresponding incentives defined in terms of v' while alternative insights can be drawn from interpreting F2 and F6 using the original submodular valuation function v'.
F2 Individual Rationality, By the definition the dual valuation function (4), the condition of F2 translates to
v(D)) 2 v(Da)
J(N) - -NN D 2 (NN - NN D.)
J(NN D 2 (F))
This translation suggests that the value (using the original submodular valuation function) of the grand coalition without party i's reward is less than the value of the grand coalition without party i's original data. Therefore, the dual interpretation of F2 is that with the assigned reward, party i's importaaceindiisennbbiittyy to the overall collaboration is increased compared to its importance without the reward.
F3 Equal- Time Symmetry. The condition in F3 can be translated to VC C (i.j).
(C' (i}) e (i)
J(N) - 2' Y (C'U = (/( (CUG))
For any C N (i.j), we can take N (V,)) S N (i.j) so that N \ (C)
CU (i) and N (C' (()) C (j}. Since we are considering all C S NN) j), the dual
interpretation of F3 equivalent to the original submodular case.
F4 Tiue Desirability. The condition of F4 holds when 3B C (i,j) ((Bu(i))
v(B U (i)) and AC N [i,j} ((Cu(i)) 2 oCC'u (i)). Translating the inequality after the first quantifier:
1 (iB) (B)) ((N
J'N (Bu
If we can find B C N NV such that ( we can also find B' C (i, such that vBB J (i)) (B (i)) by taking B'= =y= = {i,j). Similarly, we can show equivalence between dual and original valuation functions for the condition after the second quantifier. Thus, the dual interpretation of F4 is equivalent to the original submodular case.
25

---

FS Uselessness, A party is useless if CN (i) ((C = =((C). By the definition of dual, VCC'U (t}) = v(C)
'' 0))=
(i)). Noting the bijection between (i)}. so useless party also satisfies
the sets (C:CCN [i] and (N (CU AC S N ( (1)=
F6 Necessity. A party is necessary if AC C [i.j] g - v(C) O. i.e., v'(N) = v'(N \ C). It suggests that if either or j absent from a coalition C, then C. is of no importance to the overall collaboration as it makes no difference when excluding C from the collaboration. Hence, both i and j are vital for the importance coalition and should be assigned the same reward.
F Proofs of Theorems
We first prove some useful lemmas regarding the properties of the Shapley value (I). Lemma F.1. grand coalition N, if superadditive, then y,(u, 2 v.. for all i N.
Proof. The proof follows directly from the definition of the Shapley value (1) and the superadditivity of o:
|C| 1)!
N)(u, = E
(oour) - vc)
|N
CENV()
CCI(N
|C| - 1)!
Σ
CCNYO
1/. Corollary F.2. If superadditive and non-negative, then Vi N (i(u, 2 0.
Proof. Following Lemma F.l: 2)
2 20
Lemma F.3 (Necessity). The Shapley value satisfies the necessity property, i.e,, for all i,j E such that i + j. if Ç N [i,j} & = vc 0, then (p,(v, N) = (2 N).
Proof. Suppose i and j are both necessary parties. We split the calculation of (u, N) into coalitions that do not contain j and coalitions that contain j:
Pi N)
ICIU(IN] -CI 1)!
Σ
(VOUTH vC)
[NI!
ICHI(IN
- -01 1)!
(IC1 + D)((IN |C|- -2)
(VOUTH) vc)+
(roulr,s) - (OU(t
|N]!
NN
(EN((L))
CEN((1))
B
(IC) + 1)!(IN) - -01- - 2)! (Cu(i.
Σ CENVAE) where the last equality is because A = 0 and B ((u())))- As í and j necessary parties, for any coalition C S N \ (i,j), ve = (C)( "Cu(} = 0. Similarly,
(v,
C -1C1 -1)! (VCUB) -v))
Σ
NI
CENAU
ICH([NN -1C -CI-1)! (VouG) - vc)+ + Σ
(IC) 1II(NI -CI-2)!
= =
(uou(Ls) - ((u())
INI!
CENVES
CENY(AA)
(ICI
+ 1|)(IIINNI |C| ECu(LJ)
Σ
|N
CEN(EL) which is exactly the same (V.
26

---

is also well known that the Shapley value satisfies the symmetry, uselessness [5] and desirability [37] properties. For ease of reading and exposition, we list these results as lemmas below:
Lemma F.4 (Symmetry). The Shapley value satisfies the symmetry property, i.e, for all i, E such that + j. ifVV C N {i,j} vou(s) vcuus), then si(v,, N) (),
Lemma E.5 (Uselessness). The Shapley value satisfies the uselessness property, i.e., for all i if VC' E N Xi) tCu(s) = Vc.. then (;(u,, = O.
Lemma F6 (Desirability). The Shapley value satisfies the desirability property, ie., for all i, E N such that + j. if _ N {i.j} vou(d) 2 tCuu), then yi(u,, N) 2 (, N). Further, if C N \ {i,j} vouus eouts) and 3B N (i.j) (Bu(i} (Buls} then pilu, (),
F1 Proof of Theorem 6.1
Since we are considering each time interval 7 as a separate collaboration in which new valuation function (y) is defined, we first note a crucial observation that (r) preserves the non-negativity and superadditivity of e::
(T)
ot)
Lemma E.7. If 2N + R is non-negative (superadditive), then v/ : 2N + IR as defined in Sec. 6.1 is non-negative (superadditive) for all 0 S T where T maxieN ti.
Proof. This clear as we define have
(()) to be restricted to the coalitions in N,. For any 0 S S ST
(T) Cc = dc 20 NC
and
(7) (7) BUC UBuC 2 un - Dc = =0 + Uc
(r)
for all B,C C N, N BnC= 0.
F1 Non-negativity. This follows from F2 and non-negativity of v. Individual Rationality. For all E N.
w (T)))
w (r)ppp
(T)
Σ (1))pp/7)) (7)
T=4
T
w
(7) +
Σ
w
Pi
(1(7))
(1() .(,,N,) .N..)
T=
T
(
Σ
(T)
r=t
Σ (1))
where the inequality is due to Lemma F.1 and Lemma F.7.
27

---

F3 Equal-Time Symmetry. For i, E N s.t. i if = and AA N {4,)) ecu()
(cu()) then
w
w
Pi
(T)
+
Σ
w(
T=h
T
w w( 04 +
Σ
w
( 8i
(1) N;
T=4
T
w(/ +
w
Σ
w
(1))
4j
(u .N.)
T=4
T
Σ w
(7)
Σ
w
rr
where the third equality is by Lemma F.4. F4 Equal-Time Desirability. For all i.j N s.t. i # j, j.ift; tt and the following condition holds:
(BB S N {i,j} Unu()) > vBL (AC NN {i.s} vou()) 2 vcu(s))
then by Lemma F.6,
w
(T))g)7)
(T)
+
Σ
w(t)
T
(0 +
Σ
(r) Pi (1/5,10 .N.
r=
Σ w
Σ
w(t)
...)
4gj
F5 Uselessness. By definition 8, 0 if i is useless. N [i} vCu(s) DC,
Following Lemma F.5, Vi N, if C
4-1
(o(() (() 4ps (1) [0 N.) 0.
1
T=0
F6 Necessity, If both i and j are necessary, then 8; = Uy = 0, and no coalition can generate value until both of them join the collaboration, which results in the Shapley values of 0 for all parties. Also, Lemma F3 guarantees that the Shapley values of and j are always equal after both of them join. Let t max(ti, f2), then
T
(T)
w
+
T=1
(F) 0+ Σ
w 4i
(eg) .N,,))
T=1
Σ
5
+ Σ w
(1
=Ty-
28

---

F7 Tase- Monotonicity. Suppose t' is the new time value vector such that t2 < t and t2 t Vj i. We divide the entire collaboration duration into three segments, and let 4p kp!""" be defined based on and t', respectively. Then,
I
w
w
w of 1
+
Σ
w
(p
T=4
Σ w (+)) (T)
1 +
Σ
pp Iry
T=t
(1) 95
(T)
(r)
w xx (r)
Pi
where the inequality is due to Lemma F.l since party i has joined the collaboration based on but not t for & S < f..
F8 base- Striet Monotonicity, By L. 3C S bi:tt ty < &) such that (Cu()) > ve + 2/-
Thus, there exists 14 S F < 4 such that 3C C N₂ with (ou()) > ve + v. By similar argument in Lemma F.I. pi (F) of F7. Therefore,
> 2s = p (f) Also note that yp; 2 4pi
(r)
(r) Vr from the proof
Ti w (p
(f) 4j
w w( (7)
w (t)
(F)
Σ 1w
(TV r/.
+
Tf7
F.2 Discussion on [35]
[35] has shown that ((:A) (defined in Sec. 6.2) inherits the superadditivity from v, i.e., if is superadditive, then ((..A) is also superadditive. This is useful because if ((..) (defined in Sec. 6.2) inherits the desirable properties of (e.g., A1, A2, A3 in Sec. 5), it can be shown that the preconditions of Lemma F. 1-F.6 are fulfilled, allowing us to use these lemmas to prove Theorem 6.3 in App. F.3.
Indeed, additionally show the inheritance of non-negativity (A1) and monotonicity (A2), in addition to superaddivity (A3), as outlined by the following propsition:
Proposition E. ([35)). Given a valuation function for the aggregated data of a coalition v(.) : 2N
I. and joining time values t 2 the time-aware valuation function v(,)) : 2 x 2 + IR as defined in (5) satisfies the following:
v() is non-negative, then ((,)) is non-negative.
Uff is monotonic, then V(.a) is monotonic.
. v() is superadditive, then ((.t) is superadditive.
Proof. The inheritance of superadditivity follows directly from Proposition 4.1 in [35]. The preserva- tion of -egaativtv and monotonicity can be shown similarly following the proof of Proposition 4.
Following the definitions in [35], we define list of increasing values yh: for h = 0, 1, 1...., r where r is the number of distinct cooperation abilities in (Ai hee.. Specifically, yo = 0, yh. = min N (Aa A; > 1/ We can interpret (yk))=o) = based on the order of arrival, i.e., Un. arranged in increasing order of cooperative ability (parties arriving earlier have higher cooperative ability). We also denote the valuation function restricted to a coalition S C N as (T) A v(T n S) for all C N. It shown in [35] that
= Σ (yh+1 - yn) UINN + (1- t A=0
((A))
SEN
where N₁-1 = e X, 2 3h+1). for h = o,...,,- 1. No+1 represents the set of parties with cooperative ability above a certain threshold (arriving carlier than specific time). By noting that
29

---

if v is non -on-nntttive (monotonic), then d|s is non-negative (monotonic) AA _ N, the inheritance of non-negativity and monotonicity by V(-,e) follows from (7) and X; = - E (0, 10 Vi N. L
F,3 Proof of Theorem 6.3
F1 Non-negativity, By Proposition F.8, ((.t) is non-negative and superadditive, then by Corol- lary E.2, for all i N,
rs ())) N) 20.
F2 Individual Rationality, By Proposition F.8, ((.)) is superadditive, then by Lemma F.1, Vi E N,
r = (i( 2 ((D))) (i)) U.S.
F3 Equal-Time Symmetry. For all i,j E N s.t. i # j,if4=4 and AA VCCNN (i,1) voug ecuts then X; = >g. Also, by (7), for all C N (i.j}
(ye+t - (CU())) + Σ (1-A))())(CU(())
((CU((,))
AEN
Σ (uh+1 U (CU()) E (1 (1
KENWiiH
+ (1 (CU))) -1)) (CU(((())
(yh+1 3nd UI (CU +
Σ (1- (1-A))(CC(k))
KEN
A3) + (1-11) -1
Σ (tn+1 - ynd UIN (CU()) +
Σ (1 (1n
KENYOU
Σ (yn+1 - UIN (CU)),, h=
(CU()) + Σ (1 (CU
KEN
= ((Cup)
Hence, we can use Lemma F.4 as the preconditions are met,
r (((().)) N)
=)=
F4 Equal-Time Desirability, First note that VC C N \ have vis (i)) 2 mi(CCUu (i) VS c N. By (7), since = ly implies ); Ays we have vouti), 2 t.
{i.j} (cu()) 2 (cu())- then we
30

---

Following the notations in App. F.2, N1 N since X = exp(-1t4) 0 = yo vi E N. Hence,
by (7), if = & and 3 _ {i.j} BBu(+) (Bu()> then
(h++1 0 ( (BU Σ (1 As) (B)
((BU)))))
KEN
wy++1 UINN (BU
(
Σ (1- (1
KEN
(ur yo) UM (BU (B)
(Bu(f)
Wh++1 ynd N (BU - (1- (BU
REN
+ (y) - yo) VnN (1))
(BU()
((B
Thus, have shown that the conditions in Lemma F.6 also hold for ((,4), so
()() > ())) ry
F5 Uselessness. First note that VC S {1} (cu()) = vc. then i (C)) = (is(C) AS S
N, which implies vCu((),+ vC by (7). Hence, by Lemma F.5, for all E N, if VC S
N \ {i} BCu(() tc.
(;(() N) = 0.
F6 Necessity, By Proposition 5.1 in [35], VC C N if (i,j) C = vc = 0, then
= (,((,,), =rj =)rg
F7 Time-based Monotonicity. If (f < 14) A E N (i) = 11)) then by Proposition 4.2
in [35] and the superadditivity of u.
= pi N) 2 ()) NN =r.
F8 Time-based Striet Monotonicity. The proof of time-based strict monotonicity follows closely from the proof of Proposition 4.2 in [35] except that we show under TI, the equality can be removed to achieve strict monotonicity. We only outline the significant changes and refer the readers to [35] for more details.
Notations. We follow the notations defined in App. F.2. Additionally, we define the zero- normalized valuation function U, as vS( & u(S) -KES v(k) AS S N. Also recall that in time-aware CML setting, A = exp(-744).
Proof Sketch of [35]. [35] showed that if 12 < f (X > A₂) and tt Vj Vj = A i), then the difference yp (P...e)) .N (P ( .N can be expressed as linear combinations of the Shapley values, where the Shapley values calculated with different valuation functions under separate conditions. There are a total of six possibilities. 10 Specifically:
Under Condition 3,
(P(() N) - (api +100)++6/(((00/) (V N);
Under Condition 6,
p.(P).)) N) - (p:(u) = Top; N) + 16 N):
Under Condition for E (1,2,4,5)
(' N) (M(..)) N) = (1oc
1To be more precise, there is one more possibility whose proof is omitted in [35], but this possibility is covered in F7, The readers refer the footnote of F7 in Sec. 4 for details.
31

---

Here, Na. T/g T/ > O are positive coefficients and Co. Cj C. S N are coalitions for all a E {1, 3, 4, 5, 6). We refer the readers to [35] for actual values of Ta. C., as well as the specifics about the conditions. The monotonicity then follows from the non-negativity of the Shapley values. Striet Monotonicity, To enforce strict inequality, we need to show
(i NN 0 Ya E (1,
(8)
and
p:(oolc : N) 0 Pi(Colo .N) > 0 {3.6).
(9)
- and T" = T J {a}. It is easy to check that T" C C. Va E 21, and T' C. C C. Va E 63. This relieves our burden of checking every single possible condition, as we can restrict ourselves to only consider S C T' {i). In fact, To show both (8)
and (9) hold (positive Shapley value), it sufficient to show Vo/T (SU (i)) (S) > 0 for
some _ T {i). i.e.
- > 0. of vo. as long as 35 C T" (i) such that
ES NO
(10)
Since (S (i)) nT' =
- (i], by the inherited superadditivity
> wT(STT) + ((()), (10) is
satisfied.
By the definition of rero-normalization, to ((+) 0, so we need (on(SSSn) (S(TT)
Again, by the definition of to. we require
((SNT))UR)
Σ JE
v((k) > =(STT)
Σ v(()))
(11)
RESHTT
Since i & S. (11) equivalent to =((SNT) (i)) v (ft)) "S
(i)
be the coalition that satisfies I., Then, if we take S = B, we have S = B because TT. Hence, v((S T') U (i)) v(fi)) = (Bu()) - U; > uB by L. Therefore, (11) is also satisfied so
both (8) and (9) are true and
r ri = pi (v) N) - (p(()) N) > = == M
F.4 Efficient Computation of Equation (5)
We can skip computing the Harsanyi dividend [14] by using Eq. 7 in App. F.2. In the following, we explain how to implement this efficient calculation in practice and provide insight into this approach.
We obtain é by sorting the weights (e (e=7t)) JEE in descending order and appending 0. We obtain Ó by sorting the list of parties in coalition C' ascending order of their joining time values. Then,
[C|
[C|
UC,
Σ j=1
Co (éw éyyty +
Σ l j=1
(12)
only involves summing |C| terms."" 11
We observe that for the keth ranked party = CIN). , the dividend d(u, (i)) contributes to any se where
i C. Thus, the dividend has a weight of Σ E é éb+tj] + +1 [1] in Eq. (12) which is the same as in Eq. (5).
In addition, the dividend d(v,, T") where [T] 2 2 contributes to ue only T C C. Let be the rank of the latest party in T. The dividend would have a weight of > éb) eb++) = éNN minjer et) in Eq. (12) which is the same as in Eq. (5).
G Reward Realization Methods
Likelihood Tempering. One way to realize (rT) exactly when using conditional IG (2) for data valuation is the likelihood tempering [51] method. The mediator assigns party model reward
"I [b] and [: jl denote indexing the j-th element and up to and inclusive of the j-th element.
32

---

(posterior) p; that is updated by its likelihood pDD, O) and the tempered likelihood of others' data xc p(DN)(()) (1) 0): where the tempering factor N; E [0. 1]. Note that when K; and 1, party i's model is trained on D; and DN. respectively. We can view DNiti as partitioned into two random variables R& and R_ with likelihood proportional to p(D 10) and p( (o respectively. Note that as D follows a Gaussian distribution, R2 and R_; ; also follow Gaussian distributions (but have larger variances). The mediator then uses any root-finding algorithm to exactly solve for K2 such that [ D, U ( ri.
Subset Selection. The mediator generate model rewards of different values for each party d E N by training on different subsets of DN Since r exceeds vy (i.e., F2 is satisfied), party d receives a model reward trained on D D, U R, where R3 S DN is selected such that (DD = rī. However, as it is intractable to enumerate an exponential number of subsets to find D whose value v(D)) is the closest to r,. we resort to approximation method. For example, when conditional IG (2) is used as the valuation function, we shuffle Di and incrementally add points to R. only until I(0; D \ DJ) > ri.
H Additional Experiment Details and Results
All experiments were performed on system equipped with an NVIDIA A16 GPU with 10 GB of VRAM. The system was configured with NVIDIA driver version 515.43.04 and CUDA version 11.7.
H.1
Experiment Setup
Part of our experiment setup is adapted from [49]. We empirically verify our results on an additional diabetes progression (DiaP) dataset [9] with conditional IG as the valuation function. We also demonstrate our results on the CIFAR-100 dataset [23] with n = 10 parties, and the dual of validation accuracy is used as the valuation function. We use the Gaussian Process (GP) regression model for all datasets except MNIST and CIFAR-100, 100, for which we train neural networks (NNs). For all GP models, we use automatic relevance determination such that each input feature has a different lengthscale parameter. For the Friedman, CaliH and DiaP datasets, we use 80-20 train-test split to obtain Danin and Diea, all parties' data are randomly sampled without replacement from D
Synthetic Friedman Dataset (m n2 > ni)) We generate data based the Friedman function: y = 10 (Ta.1) + 20( - 0.5)² + 10 + 52 (d.4) + Ozqa.o 10,
where d is the index of data point (x, y) and its 6 input features (i.c., (=)1:a))==0.... are independent and uniformly distributed over the input domain [0, 1). The last input feature a T|a,3] is an independent variable which does not affect the output y.
We standardize the values of output y and train a GP model with a squared exponential kernel. We consider a test set with 200 points and parties 1, and 3 having nı 300,
n2 = 300 and ng = 200 training points, respectively.
CaliH Dataset (n, > n2 > n2y) We standardize each input feature X and the output vector y to have variance of 1 and train GP model with a squared exponential kernel.
We consider test set with 4128 points and parties 1, 2 and 3 having nı 600, n2 400 and n3 = 200 training points, respectively.
DiaP Dataset (n1 n2 T2g) We use the diabetes progression (DiaP) dataset [9] with scaled fea- tures from sklearn. We remove the gender feature and standardize the output vector y to have variance of
We train a GP model with a composite kernel comprising the squared exponential kernel and the exponential kernel.
We use an 80-20 train-test split to obtain test set with 88 points and parties 1, 2 and 3 having 711 = 75, n2 = 75 and n23 = 125 training points, respectively.
MNIST Dataset Since NNs are highly effective on the MNIST dataset, we first create a subsampled version by randomly selecting 20000 data points from the original training set, which serves as the aggregated data Drmin for all parties. Daain is then distributed among the three parties based on data point labels. Specifically, the 10 classes are randomly divided into three
33

---

groups of size 3, 3 and 4. Party i receives all the data points in Duain with labels in the ith group.
We train NN with one hidden layer of 16 neurons using ReLU activation functions and stochastic gradient descent. The model with the highest validation accuracy over 10 training epochs is selected.
CIFAR-100 Dataset We use all 50,000 training images as the aggregated dataset for all parties. Prior data assignment, preprocess the images using pretrained ResNet50 backbone to extract embedding features. Each the 10 parties is then assigned all the transformed data from 10 randomly selected classes. There is no data overlap between parties, and collectively they cover the entire set 50,000 training images.
We train a NN with two hidden layers containing 1024 and 512 neurons each. We use ReLU as the activation function, and the NN is trained using dropout and the Adam optimizer. The model with the highest validation accuracy over 20 training epochs is selected.
H.2 Metrics
Information Gain Let X be the input matrix and y is the output vector. In a GP model with function
f. the latent output vector is f f(X). We assume that is generated from by adding independent Gaussian noise with noise variance o². The information gain of the GP from evaluating at X is given by
T(: T(f; = 0.5 1o5log([ Xxx/o²1
(13)
where Xxx is covariance matrix with components k(x,, x') for all ar, ar in X and l is kernel function.
When calculating the IG for heteroscedastic data (i.e., data points with different noise variances), instead of dividing by a², we multiply by the diagonal matrix K poise such that each diagonal component of Knoise represents the noise variance corresponding to a data point in D.
Mean Negative Log Probability The model reward P is Gaussian process posterior after observing the dataset D; (and some subset of Dwy)). To compute the MNLP, we compute the model reward pf's (predictive) mean Mæz and variance o2 each test point æ. Then,
(10g(2m02)) + (ez y)² ) 0
MNLP
Σ 2
(14)
Diest
(
A lower MNLP is better as indicates that the model is more confident in its prediction (small first term) and is not overconfident in predictions with larger squared error (small second term).
H.3 Reward Values on the DiaP Dataset
In Fig, 8, we create a scenario where U1 36.35 close to 1/2 37.11, while V3 61.79 is
significantly larger. When i t i3 = 0, the Shapley values are y0 = 38.69, 64.53, with party receiving model with the highest possible value vv. As party joins later (t) increases), Fig. 8 shows that its reward rf decreases as disincentive for joining late. While F8 only stipulates decrease rf, we also see drop ri and rÿ. This because other parties receive benefits from party I's collaboration for shorter duration, resulting in lower total rewards. However, each party is still guaranteed individual rationality (F2), i.e., all parties receive rewards at least as valuable their own data (plotted as grey horizontal lines in Fig, 8).
39.46, P3
34

---

125
125
Reward
Reward
100
100
75
8-0.7
ym0.5
A-1
81
(a) r with varying /
(b) 1? with varying y
Figure 8: Graphs of reward values rf vs. 1 with the DiaP dataset using (a) time-aware reward cumulation and (b) time-aware data valuation.
The trend in Fig. is nearly horizontal and thus almost unnoticeable. This is because the changes in reward values are small compared the largest reward value uN. To illustrate the trend more clearly, we restrict the range of reward values. In Figs. 9a-b, we can clearly observe that the reward values of parties and decrease with the joining time when the reward values are restricted the range [80, 90]. Similarly, the reward value of party 3 also decreases, as shown in Figs. 9c-d, when the range restricted [135, 145].
90
88
88
Reward
Reward
86
84
84
82
80
80
2
01
(a) ri, r2 with varying
(b) ri, r2 with varying 7
Reward
142
Reward
142
140
140
138
138
136
136
2
4
8
(c) rg with varying 3
(d) r with varying
Figure 9: Graphs of reward values ri (green), r (blue), r3 (orange) vs. t using (a.c) time-aware reward cumulation and (b,d) time-aware data valuation with the DiaP dataset.
H.4 Reward Realization with Likelihood Tempering Method on the DiaP Dataset
We verify that when using likelihood tempering, models with higher reward values r also have better predictive performance (lower MNLP). Thus, Fig. 10, we observe that: (i) Each party receives model reward with lower MNLP than the model it trained alone. (ii) When party delays its participation, it receives a model with worse MNLP. (iii) Parties with higher Shapley or data values receive models with better MNLP. In addition, using smaller 3 and larger 7 leads to decrease in rī and an increase in MNLP. We also note that the increasing trend of MNLP not obvious in Fig. 10. This is because the change in reward values is small, explained in the App. H.3.
35

---

Party
1.050
f=07
Party
1.050
Y-0.5
Party 2
MNLP
8-1
Party
MNLP
Farty
1.045
1.045
1.040
1.040
1.035
1.035
1
2
4
11
(a) MNLP with varying 3
(b) MNLP with varying Y
Figure 10: Graphs of MNLP vs. & using (a) time-aware reward cumulation and (b) time-aware data valuation using likelihood tempering for reward realization the DiaP dataset.
H.S Reward Realization with Subset Selection Method on the DiaP Dataset
When using subset selection, we similarly observe that models with higher reward values r; have better or similar predictive performance. Based on Fig. 11, we conclude that: (i) Each party receives a model reward with lower MNLP than the model it trained alone. (ii) When a party delays its participation, it does receive model with better MNLP. (iii) Parties with higher Shapley or data values receive models with better MNLP.
1.055
1.055
1.050
1.050
MNLP
#=0.1
Party
y=0
Party
1.045
f-0.7
1.045
Party
1.040
8-1
Party
1.040
1
Party
1.035
1.035
1
f
(a) MNLP with varying
(b) MNLP with varying Y
Figure 11: Graphs of MNLP vs. & using (a) time-ar aware reward cumulation and (b) time-aware data valuation using subset selection for reward realization on the DiaP dataset.
time-aware
In Sec. H.4, we observe that the MNLP increases smoothly as t increases. However, for subset selection, the MNLP exhibits less smooth behavior. This is because the reward values can only be realized approximately, and the process is sensitive to the quality of the data points included in the subset. For example, the MNLP would increase by larger extent with the addition of an outlier or the removal of all data from a region of the input space."
H.6 Reward Values with 10 Parties on the CIFAR-100 Dataset
As in Sec. 7, we investigate how each party's reward changes with the joining time of Party 1. We set 3 r for both methods. Fig. 12 shows that Party I's reward decreases with later joining times under both methods, consistent with our theoretical guarantees and serving as a disincentive for joining late. More importantly. Fig. 12 demonstrates that approaches remain consistent when an efficient approximation method is adopted, even as the number of parties increases and a larger real-world dataset is used.
In contrast, likelihood tempering always use all data and only vary their impact by varying the tempering factor and variance.
36

---

B
0.750
0.725
0.6
0.700
Reward
0.675
Reward
0.650
0.4
0.625
0.600
0.2
0.575
0.1
2
3
4
t1
tj
Party 1
Party 2
Party
Party
Party 5
Party
Party 7
Party
Party 9
Party 10
Figure 12: Graphs of reward values rf vs. 4 with the CIFAR-100 dataset using (a) time-aware reward cumulation and (b) time-aware data valuation.
37


Reviews:
[{"review_id": "2HKDZJQbxU", "reviewer": "Reviewer_acWA", "summary": "The paper looks at time aware incentives in collaborative learning. For instance, suppose a group of agents, each having their own data-set want to jointly train a model by pooling their datasets then we call this collaborative learning. Time-awareness looks at the aspect where the agents choose to participate (or not) at different time stamps. In such a setting, the authors look at rewarding agents based on two parameters: (1) The quality of the data contributed by the agent and (2) The time at which the agents choose to participate -- Early contributions should be incentivized (while ensuring that late, high-quality data contributions are also incentivized).\n\nTo this effect, the authors set up eight desirable properties that an incentive should satisfy. Out of these, five have been established in previous works on non-time-aware fairness in collaborative learning. They introduce three new that essentially state that agents who have same quality datasets, with one agent joining before the other agent, the one that joins earlier should get higher reward -- provided the agent's data is not useless. They also have a property that says that if an agent's data is highly valuable (which they call necessary) this agent should get a high reward irrespective of when she joins the collaboration.", "questions": "1) The way Property F6 is defined, it seems that if agents i, j have the same data set that is necessary to the other agents (but not necessary to each other i.e., it suffices for either i or j to join the collaboration to get a non-zero value model) and i and j join at different times, they get an equal reward -- shouldn't a more reasonable requirement be that the i gets a higher reward? \n\n2) On line 189, you have a sentence in bold that says that \"a party get higher reward by waiting longer to curate a better data set due to F(6)...\" however, F(6) is very strong and this will be true only if the data set contributed by the agent who waits longer is the only data set that provides any non-zero value. This seems rather stronger than what the statement is claiming -- simply better datasets that join later may not be compensated highly -- this will depend (in a complicated fashion) on how much \"better\" the data set is and how the parameters \\beta (or \\gamma) are tuned. Is this correct?\n\n3) Can you elaborate more on the aspect of rewards -- as far as I understand, you are rewarding agents with models whose accuracy is proportional the the rewards you get. If this is the case then the fact that you can compute a model with any desired accuracy is a rather strong assumption and should be mentioned in the paper.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The authors have defined and addressed an important problem that has surprisingly not been studied before.\n2) The set of properties the authors define are very reasonable (although I have some questions about this stated below).\n3) The paper is well-written and easy to follow. I checked proofs for one section and they seemed correct to me.\n\nWeaknesses:\nFrom the technical side, the paper seems pretty direct. At the end of the section where the properties are defined, the authors demonstrate how Shapley value fails to provide rewards that satisfy the properties outlined (which is itself not surprising) however, their incentives are eventually based on Shapley value. For instance, one reward model is computing the Shapley value at each time point based on only the subset of agents that have joined until then and then takes a convex combination of these values. This seems a relatively straightforward extension.\n\nHowever, given that the model is new, highly relevant to the real-world setting, I would say that overall I think this is a decent submission to NeurIPS.", "was_revised": true, "final_justification": "I have read the authors' response, they have addressed my questions. I maintain my opinion of the paper -- the technical aspects are direct but the question addressed is interesting. In the light of this, I will maintain my original score.", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work and for recognizing that our problem is new and highly relevant to the real-world. We address your questions below:\n\n> Q1. ...it suffices for either i or j to join the collaboration to get a non-zero value model...\n\nWe believe that there may be a misinterpretation of F6 (necessity). The precondition in F6 requires **both** i and j to be in coalition C for its value $v_C$ to be non-zero. This means that i and j are necessary to one another, and neither can achieve a non-zero value collaboration without the other. Therefore, it is appropriate to assign equal rewards to party i and j, as it ensures that both parties are incentivized to participate, regardless of who joins earlier.\n\n> Q2. ...this will depend (in a complicated fashion) on how much \"better\" the data set is and how the parameters \\beta (or \\gamma) are tuned. Is this correct?\n\nYes, this is correct, and we appreciate the reviewer's clarification. We will revise the sentence by adding conditions under which it is true after the original bolded sentence, as follows:\n\n**A party may get a higher reward from taking time to curate a higher quality dataset instead of sharing a less valuable dataset as early as possible.** This holds in two key scenarios: (i) When a greater emphasis is placed on data quality relative to joining time, e.g., by setting a large $\\beta$ or small $\\gamma$. (ii) When the parties are *necessary* parties, i.e., their data is essential for achieving non-zero value collaboration.\n\n> Q3. Can you elaborate more on the aspect of rewards...\n\nWe first decide a target reward value and seek to realize a model with the target reward value (Sec. 6.3). Note that **the reward value is not always measured using the accuracy function**.\n\nFor example, in Fig. 3,6 and Fig. 5, we seek to control the conditional information gain. This is possible to achieve **exactly** and **efficiently** based on the likelihood tempering approach proposed in [49], which we outline in App. G.\n\nIn Fig. 7, we seek to control the model validation accuracy. This is slightly harder to control and we propose to use the subset selection method (details in App. G) to achieve this **approximately**. [39] has also proposed early stopping to achieve different accuracies when training a neural network.\n\nWe will add the clarification in our revision.\n\n> From the technical side, the paper seems pretty direct.\n\nWe would like to clarify that while the use of Shapley value may appear straightforward in isolation, mathematically formulating all the proposed incentives in our time-aware setting (Sec. 4) is non-trivial. This includes both generalizing existing fairness notions to our time-aware setting and formalizing a new time-aware fairness perspective. Furthermore, we propose two schemes for deciding reward values (Sec. 6) that provably satisfy all the proposed incentives, this also poses additional technical challenges. We also wish to reiterate that a key contribution of our work lies in posing the problem itself. The time-aware data sharing and collaborative learning problem is underexplored despite its strong relevance to many real-world applications. \n\n> However, given that the model is new, highly relevant to the real-world setting, I would say that overall I think this is a decent submission to NeurIPS.\n\nWe are encouraged that the reviewer also recognizes the importance of this problem, and we sincerely appreciate the acknowledgment.\n\nThank you for your comments! We hope our clarifications above can improve your opinion of our work."}, {"review_id": "tYiP4y7YBX", "reviewer": "Reviewer_5syp", "summary": "This paper studied the problem of data sharing in collaborative machine learning. Traditional frameworks typically assume all the parties are joining the data sharing mechanism at the same time. However, different parties may have different preparation time, and earlier joiners suffer from higher risk. Motivated by these practical concerns, this paper proposed a new framework of data sharing, taking into the time-aware incentives into consideration.\n\nThe paper proposed new methods for deciding reward values to satisfy these time-aware incentives. The, the paper demonstrated how to generate model rewards that realize the reward values and empirically illustrate the properties of the proposed methods on synthetic and real-world datasets.", "questions": "- Is there any redundancy among the proposed eight time-aware reward incentives? Does complying with a subset of them imply certain compliance on others?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The time-aware incentives are motivated by practical concerns, and are interesting and novel considerations. It is reasonable that the data valuation and reward realization should change if parties join the data sharing process at different times due to various reasons. Thus the paper  provides explorations on this less-studied aspect.\n - The paper proposed a detailed framework with eight incentive conditions. These conditions are illustrated in detail to incorporate the time-aware considerations. \n- Theoretically, the two proposed reward scheme satisfied the proposed list of incentive conditions. Empirical results further demonstrated that the two schemes comply with the incentive conditions.\n\nWeakness:\n- there is no comparison between the proposed time-aware reward scheme and classical reward schemes that do not take the time-awareness into account. Although it's shown that the proposed two schemes comply with time-aware incentives, the benefit comparing to existing non time-aware reward schemes is unclear", "was_revised": true, "final_justification": "The author response was helpful in addressing my concerns, I remain my score to this paper as a Weak acceptance.", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work and for recognizing that our problem is motivated by practical concerns, and the claims are backed by both theoretical and empirical results. We address your questions below:\n\n> there is no comparison between the proposed time-aware reward scheme and classical reward schemes that do not take the time-awareness into account\n\nWe would like to clarify here and in our revised paper that **comparisons with classical non-time-aware reward schemes are already included in our experiments** (lines 318--322). Specifically, existing methods that do not account for joining time [41, 47] can be seen as special cases of our methods when $\\beta \\to \\infty$ and $\\gamma=0$. The baselines correspond to the horizontal lines in our experiment results (e.g., Fig. 3b and Fig. 7a-b), which suggest that they assign constant rewards regardless of joining time, thereby fail to satisfy incentive F8 (time-based strict monotonicity).\n\n> Is there any redundancy among the proposed eight time-aware reward incentives? Does complying with a subset of them imply certain compliance on others?\n\nNot with additional assumptions. If we assume non-negativity of the valuation function (line 205), then F2 (individual rationality) implies F1 (non-negativity of rewards). We still include F1 as one of the incentives as it is only implied under this extra assumption in a later section (Sec. 5), and it is convention in the literature as seen in [47, 52].\n\nAdditionally, in Remark 4.1, we show that F3 (equal-time symmetry) and F7 (time-based monotonicity) together imply time-based equal-value desirability, a generalization of the classical desirability incentive [47] to the time-aware setting. We did not include this as a separate incentive to avoid redundancy.\n\nThank you for your comments! We hope our clarifications above can improve your opinion of our work."}, {"review_id": "hFQTKcLnyv", "reviewer": "Reviewer_E1vJ", "summary": "This paper tackles the problem of incentivizing parties in collaborative machine learning who join at different times. Unlike prior work assuming simultaneous participation, the authors introduce a time-aware fairness framework that rewards earlier contributors more, reflecting their higher risk and contribution to collaboration momentum. They propose two reward mechanisms—time-aware reward cumulation and time-aware data valuation—both designed to satisfy fairness and new time-sensitive incentive conditions. It is novel to effectively balance data quality and joining time, encouraging early, valuable participation.", "questions": "1. I am curious about how to set the parameters in practice to control the emphasis on joining times and early participation. A user may want to know these parameters to decide whether to submit noisy data early or spend more time cleaning and submit high-quality data later. \n\n2. I understand that it may not be possible to guarantee the strong theoretical properties of the reward values under the online FL setting. However, it would be helpful to see a simple extension of your method to this setting along with empirical results. This is important to confirm the practical applicability of your work. If time constraints prevent experimentation, a discussion of your plans to extend the approach to online FL would be valuable.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Introduces a novel time-aware fairness framework for CML, addressing asynchronous participation—an underexplored but realistic setting. \n- Proposes two theoretically sound methods with formal guarantees for satisfying fairness and time-aware incentives \n- Well-structured with clear motivation, incentive definitions, and illustrative examples\n\nWeaknesses\n- The paper has already discussed some limitations, such as computational efficiency and the choice of a non-Federated Learning (non-FL) setting. Regarding computational efficiency, I agree that this issue is not unique to this work and therefore should not be considered a limitation specific to this paper. However, I have some questions about the non-FL setting, which I will raise later.\n- While I acknowledge the strong theoretical contributions of this paper, I would like to see more experiments on large-scale real-world datasets if time permits. I will elaborate on this point later.", "was_revised": true, "final_justification": "My final justification for the recommended score is still 4 (borderline accept). The main thing holding me back is parameters like β or γ, I agree with the plan proposed by the author (introducing a mediator), but I am also a bit concerned about its interpretability in practice. I mean, maybe under a different dataset and a different number of participants, the same number of β may have different effects. But considering this is new work proposing the problem of time-aware, I would prefer to keep 'borderline accept'. Meanwhile, I appreciate the author's time and effort in providing more experiments, so I keep my rate of 'borderline accept'.", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work and for recognizing that we are tackling a realistic setting with clear motivation. We address your questions below:\n\n> Q1. I am curious about how to set the parameters in practice to control the emphasis on joining times and early participation. A user may want to know these parameters...\n\nWe agree that the parameters should be broadcasted before the collaboration starts. In practice, the trusted mediators (e.g., a data sharing platform) can test out different values of $\\beta$ and $\\gamma$ to guide future decisions when the valuation function is the same (e.g., dual of accuracy).\n\nConcretely, the mediator can start by setting $\\beta=\\gamma=1$, which is shown to provide a reasonable trade-off between incentivizing early participation and encouraging high-quality data in our experiments (Sec. 7). **For future collaboration**, the mediator can then adjust these values based on observed behavior of parties: decrease $\\beta$/increase $\\gamma$ if early participation is sparse; increase $\\beta$/decrease $\\gamma$ if the majority of data received seems rushed and of low-quality. In addition, mediators can share their empirical experiences with parameter tuning (without exposing underlying data) to form best practices over time.\n\n> Q2. I understand that it may not be possible to guarantee the strong theoretical properties of the reward values under the online FL setting. However, it would be helpful to see a simple extension of your method to this setting along with empirical results.\n\nWe appreciate the reviewer's understanding that extending to the FL setting may violate the theoretical properties of our methods (lines 560--563). Notably, in an online FL setting, client contributions must be evaluated **dynamically**, and **reward values must be updated and distributed during each communication round**. This introduces a key practical issue: **once rewards are distributed, it becomes difficult to claw back or revise them to ensure overall fairness** if the reward values change when new clients join (lines 580--582).\n\nNevertheless, we show how to **adapt our time-aware reward cumulation method (Sec. 6.1) to dynamically update the reward values** of each client during each communication round of the online FL setting. We wish to clarify that this adaptation experiment is primarily intended to **showcase the feasibility of extending our method to other settings**, and the **empirical results may not fully satisfy the theoretical properties established in the main paper**.\n\n**Adapting Time-Aware Reward Cumulation to online FL**\n\nDenote $r_i^{(t)}$ as the reward value of client $i$ at communication round $t$. We update $r_i^{(t)}$ via a running average of Shapley values weighted by the penalizing factor $\\beta$ on joining time:\n$$r_i^{(t)} = \\frac{r_i^{(t-1)} \\cdot t + \\phi_i^{(t)} \\cdot \\beta^t}{t + 1}$$\nwhere $\\phi_i^{(t)}$ is the modified Shapley value defined in line 246. For computing $\\phi_i^{(t)}$ in the FL setting, the valuation function we use is the dual of $v_{FL}^{(t)}$, the validation accuracy of the weighted averaged model at round $t$, analogous to FedAvg [38]:\n$$v_{FL}^{(t)}(C) = acc \\left( \\sum_{i \\in C} \\theta_i^{(t)} \\cdot \\frac{n_i}{\\sum_{j \\in C} n_j} \\right).$$\n\n**Experiment Setup**\n\nWe conduct preliminary experiments on CIFAR-10 using FedAvg with ResNet-18 for 8 communication rounds. The training data is assigned to 3 clients as follows:\n- Client 1 receives a random half of the dataset;\n- Client 2 receives all data from 7 randomly chosen labels in the remaining half;\n- Client 3 receives the rest.\n\nClients 1 and 2 always join at round $t=0$. We vary the joining time of client 3 and assume, following [60], that clients remain in the FL process once joined. Client 3 will train on its own data until it participates in the FL process. We set $\\beta=0.98$ for a moderate penalization on late participation.\n\n**Experiment Results**\n\nWe first compare the reward values of each client at each communication round when all clients join at $t=0$:\n\n|$t$|1|2|3|4|5|6|7|8|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|$r_1^{(t)}$|0.044590|0.090728|0.124992|0.155551|0.177175|0.196051|0.207676|0.216027|\n|$r_2^{(t)}$|0.025888|0.071292|0.104497|0.127404|0.143289|0.155083|0.164143|0.171478|\n|$r_3^{(t)}$|-0.020531|-0.004606|0.003255|0.014589|0.024737|0.031186|0.037289|0.041416|\n\nWe observe that $r_1^{(t)}$ > $r_2^{(t)}$ > $r_3^{(t)}$, which is consistent with how we assign each client with their data: Client 1 has the most valuable data and client 3 the least. This reflects **fairness and equal-time desirability (F4)**. As we are working with non-IID setting, a client's gradient update may degrade the global model performance in early communication rounds. Hence, we can see some negative entries early on which may violate non-negativity (F1) and individual rationality (F2), but this is expected as the adapation no longer enjoys the full theoretical guarantees, and the results improve in later communication rounds/near convergence.\n\nWe then investigate how delaying client 3' participation affects its reward values:\n\n|$t$|1|2|3|4|5|6|7|8|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|$r_3^{(t)}$ (joins at $t=0$)|-0.020531|-0.004606|0.003255|0.014589|0.024737|0.031186|0.037289|0.041416|\n|$r_3^{(t)}$ (joins at $t=1$)|-0.025382|-0.008411|0.001155|0.013080|0.021434|0.028065|0.034989|0.039743|\n|$r_3^{(t)}$ (joins at $t=2$)|0.000000|0.007859|0.006993|0.014581|0.022988|0.029317|0.035224|0.040227|\n|$r_3^{(t)}$ (joins at $t=3$)|0.001813|-0.096688|-0.066300|-0.041649|-0.023047|-0.007794|0.001940|0.010262|\n\nWe observe that **as client 3 joins later, its reward value generally decreases**, reflecting **time-based strict monotonicity (F8)** and demonstrates that our method disincentize clients from delaying their participation. Some deviations (e.g., when client 3 joins at $t=2$) are expected, as the adaptation no longer enjoys the full theoretical guarantees. However, the overall trend remains consistent.\n\nWhile our adapation method demonstrates promising empirical results, we view this as a first step toward broader applicability. Future directions include (i) developing a theoretically sound method for the online FL setting, and (ii) designing mechanisms to retroactively adjust rewards to address the aforementioned \"claw back rewards\" challenge.\n\nThank you for your comments! We hope our clarifications above can improve your opinion of our work."}, {"review_id": "bbyiPwslPz", "reviewer": "Reviewer_XdcU", "summary": "This work considers the problem of heterogeneous arrivals of data contributors. It proposes to provide higher incentives to the ones contributing earlier to motivate participation. The authors proposed two methods to determine the reward values.", "questions": "-  How is the data modeled? How does the data distribution affect model quality? This is important because this work emphasizes data sharing rather than game design. \n- Why is it reasonable to set higher rewards for earlier participation? I understand the intuition, while a formal mathematical formulation is needed to motivate this setting. That is, there should be an objective with physical meaning, solving which we can obtain that \"higher rewards for earlier participation\".\n- How is fairness modeled in this work? Fairness should be a formal mathematical metric rather than an intuition.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The heterogeneous time arrival problem is interesting.\n- The analysis of this work is complete. \n\nWeakness: \n- The data was not actually characterized in this work. Specifically, whether non-IID or IID data is considered? If non-IID, then how do the data distributions of different parties affect the reward? If IID, then only the data quantity matters, i.e., no need to distinguish between parties. Further, if the data distribution does not matter, then it basically means the modeling is not related to the data themselves. In this case, the proposed methods should be able to apply to other scenarios without modifications. This is not good as the main feature of data sharing was not characterized. The modeling is too general.\n- F1 and F2 are basically the same concept. People usually use individual rationality without mentioning non-negativity.\n- In F5, \"improving model quality\" is non-rigorous. As mentioned earlier, the data together with the model was not characterized in the formulation.\n- F7 and F8 are human-invented properties. A more formal way should be formulating a problem and then showing that earlier participation should impose a higher reward. However, in this work, the authors impose the setting that earlier participation should have a higher reward. This may not be optimal sometimes. \n- The approach proposed in this work is relatively standard. Again, it is not actually related to data sharing, because those key features were not characterized in the formulation. The proposed approaches are simply existing approaches in coalition games without much modification associated with data sharing. \n- How is gamma tuned?\n- The approach should be compared with existing works. Although there may not be works considered similar settings, some incentive works should be compared with to show the enhancement for addressing heterogeneous arrivals. \n- Fairness is not modeled in this work.", "was_revised": true, "final_justification": "The authors have addressed my comments, and I have a better understanding of the properties and other system settings. To improve this work, I believe it is beneficial to explicitly model the non-IID data   (although it was modelled implicitly using the \"conditional information gain\"  and can be addressed empirically). It is still more promising to model and analyze the party arrival behavior (rather than assuming their behavior using a reward function). Overall, this work has made theoretical contributions in incentivizing sharing (considering asynchronous arrival behaviors) and provided interesting insights.", "rebuttal": "We sincerely thank the reviewer for taking the time to review our work and for recognizing our interesting problem and complete analysis. We address your questions below:\n\n> ...whether non-IID or IID data is considered?\n>\n> How is the data modeled? How does the data distribution affect model quality?\n\n**The data distribution can be either IID or non-IID.** Our work adopts the perspective that the value of the data depends on the learned model. That is, both data quantity and data quality matter. Hence, we consider valuing the data based on the conditional information gain on the model parameters or the dual of the accuracy function, which inherently captures the dependency on models and are applicable to all data distributions. Note that this perspective is shared many data valuation works [13, 25] that also value data by the model performance when trained on specific subsets. **In our experiments, we explicitly include both IID data with different data quantities** (Friedman and CaliH datasets, lines 312--314) **and non-IID data distributions** (MNIST and CIFAR-100 datasets, lines 317--318 and lines 910--914). The experiment results in Section 7 demonstrate that our methods consistently capture the relative value of each party under both settings.\n\n> Fairness is not modeled in this work.\n>\n> How is fairness modeled in this work? Fairness should be a formal mathematical metric rather than an intuition.\n\nIn our work, **fairness is modeled through satisfying formal incentive-based criteria** (F1--F5 in Sec. 4). As noted in line 128, our work adopts a well-established view in the literature that fairness can be defined in terms of satisfying certain axiomatic properties. For example, in [47], fairness is defined as satisfying incentives such as symmetry, desirability, uselessness which are inspired by intuition. We explicitly include these incentives or generalizations thereof in our formulation (Sec. 4), and our proposed methods are designed to satisfy them, hence achieving fairness.\n\nIn addition, we introduce the novel time-aware incentives (F6--F8) to capture time-aware fairness, which mathematically formalize the perspective that earlier parties should be rewarded accordingly. It is a widely recognized perspective in psychology, economics (lines 54--57) and motivated by practical scenarios (lines 45--53). These incentives are formal mathematical/logical constriants that are inspired by intuition like individual rationality and desirability in the literature [37, 47, 52].\n\n> F7 and F8 are human-invented properties.\n>\n> Why is it reasonable to set higher rewards for earlier participation? I understand the intuition, while a formal mathematical formulation is needed to motivate this setting. That is, there should be an objective with physical meaning, solving which we can obtain that \"higher rewards for earlier participation\".\n\nWe respectfully suggest that a formal mathematical formulation or objective with physical meaning may not be strictly necessary to motivate the incentives and setting considered in our work. Incentives and other properties desirable to humans may not be _naturally_ occurring (i.e., solution to objective with physical meaning) but should be _artificially_ guaranteed to realize benefits through well-designed solutions, as often done in economics and game theory. For example, in the seminal work on Shapley value [46], the author proposed the axioms of symmetry, efficiency (total payoff must be the value of the game) and additivity (when two independent games are combined, their values must be added party by party) and showed that the Shapley value satisfies them. These axioms turn out to be useful in many applications such as in data valuation [13] where we want to attribute model accuracy to individual data points. As another example, fairness (equal rewards for equal contribution; higher reward for contributing more valuable data) [39, 47] does not naturally occur but satisfying it would encourage parties to contribute more.\n\nIt is \"reasonable\" to set higher rewards for earlier participation because it is **desirable** in ML applications (as we have explained in Sec. 1 and App. A). Incentives for earlier participation must be designed and provided precisely because parties will not naturally do so. We do define the incentives formally as math equations in Sec. 4.\n\n> F1 and F2 are basically the same concept. People usually use individual rationality without mentioning non-negativity.\n\nWe wish to clarify that F1 (non-negativity) and F2 (individual rationality) are not the same concept as F1 ensures all parties receive non-negative rewards while F2 ensures the rewards are at least as valuable as each party's own data. However, if we assume non-negativity of the valuation function (line 205), then F2 implies F1. We still include F1 as one of the incentives as it is only implied under this extra assumption in a later section (Sec. 5), and **it is convention in the literature** as seen in [47, 52].\n\n> In F5, \"improving model quality\" is non-rigorous. As mentioned earlier, the data together with the model was not characterized in the formulation.\n\nWe wish to clarify that in lines 118--120, we define $v_C$ as the model performance/quality achieved by training on the aggregated dataset $D_C$. Accordingly, in F5 (uselessness), the condition $v_{C \\cup \\{i\\}} = v_C$ implies that $i$'s data **fail to improve the model quality**.\n\n> How is gamma tuned?\n\nWe demonstrate the effects of different hyperparameters ($\\beta$ and $\\gamma$) in our experiments (Sec. 7). In practice, the trusted mediators (e.g., a data sharing platform) can test out different values of $\\beta$ and $\\gamma$ to guide future decisions when the valuation function is the same (e.g., dual of accuracy).\n\nConcretely, the mediator can start by setting $\\beta=\\gamma=1$, which is shown to provide a reasonable trade-off between incentivizing early participation and encouraging high-quality data in our experiments (Sec. 7). **For future collaboration**, the mediator can then adjust these values based on observed behavior of parties: decrease $\\beta$/increase $\\gamma$ if early participation is sparse; increase $\\beta$/decrease $\\gamma$ if the majority of data received seems rushed and of low-quality. In addition, mediators can share their empirical experiences with parameter tuning (without exposing underlying data) to form best practices over time.\n\n> The approach should be compared with existing works. Although there may not be works considered similar settings, some incentive works should be compared with to show the enhancement for addressing heterogeneous arrivals.\n\nWe appreciate the reviewer's understanding that there may not be works tackling similar settings. We would like to clarify here and in our revised paper that **comparisons with classical non-time-aware reward schemes are already included in our experiments** (lines 318--322). Specifically, existing methods that do not account for joining time [41, 47] can be seen as special cases of our methods when $\\beta \\to \\infty$ and $\\gamma=0$. The baselines correspond to the horizontal lines in our experiment results (e.g., Fig. 3b and Fig. 7a-b), which suggest that they assign constant rewards regardless of joining time, thereby fail to satisfy incentive F8 (time-based strict monotonicity).\n\n> The approach proposed in this work is relatively standard. Again, it is not actually related to data sharing, because those key features were not characterized in the formulation.\n\nWe have **addressed the concern regarding IID/non-IID data and improving model quality** by clarifying that **data is valued relative to the model** in our formulation. If there are remaining concerns, we would appreciate if the reviewer could clarify which specific \"key features\" of data sharing they believe are missing from our formulation, and how these differ from the key incentives in data sharing outlined in Sec. 4, or the various data distributions (both IID and non-IID) considered in our experiments (Sec.7), so that we can respond more precisely.\n\nWe would like to emphasize that **our approach is related to data sharing as it addresses a key underexplored practical problem in data sharing**: When parties share their data at heterogeneous times, how should we fairly reward each party based on both their time of arrival and value of data? A key contribution of our work lies in formulating and posing this problem itself.\n\nWe would also like to clarify that we propose two reward schemes that incorporate the joining time information to address this problem. These schemes **generalize the standard coalition game approaches** (e.g., the Shapley value) by embedding the time information in the reward cumulation stage (Sec. 6.1) and the valuation stage (Sec. 6.2). Importantly, both schemes provably satisfy all the proposed incentives, this also poses additional technical challenges.\n\nThank you for your comments! We hope our clarifications above can improve your opinion of our work."}]

Author rebuttal:

Meta-review: This paper formalizes a new problem in collaborative data sharing, where agents are rewarded differently based on the time of their participation in addition to their quality of participation. Eight incentives are proposed for this problem, with five borrowed from prior studies and three newly designed to encourage early participation. New methods are proposed to decide reward values that satisfy these incentives with theoretical guarantees. These methods are also empirically validated on synthetic and real-world data sets. 

---

All reviewers find the new problem interesting and practical. Some reviewers compliment on the theoretical analysis and clear writing. All reviewers are on the positive side. 

A notable concern shared by two reviewers (acWA, XdcU) is that the proposed methods are relatively straightforward. Authors defend that formalizing the problem is the key contribution and the presented incentives are non-trivial. I agree the merits outweigh the weakness on this aspect, and encourage authors to deepen this linear of research in future.   

Another notable concern shared by two reviewers (5syp, XdcU) is the lack of baselines in the experiment. Authors explain the baselines of non-time aware reward schemes are compared with. Their choice look reasonable, considering the problem is newly formalized. 

---

Overall, I think this is a neat and decent work to be presented at NeurIPS. Authors are encouraged to incorporate reviewers’ feedback in revision, and consider advancing the work with more depth in future.
Decision: Accept (poster)