Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a data valuation mechanism based on modern property rights theory aiming to address the problem of data rights and data integration. The framework gives the power of integrating data and allocating compensations to the agent that contributes most significantly. Their framework uses a contribution evaluation method based on the Data Shapley to decide who has the power to integrate the data and give compensation. And they assign the compensation using the Least-core method. Finally, they demonstrate experimentally that the compensation mechanism is feasible in solving the data pricing problem.
strength_and_weaknesses: Strength:
The paper tries to address an important problem through a new lens. The main contribution is the conceptual contribution of adopting the property rights theory in the problem of data valuation. Basically, they adopt the theory to give the agent with the highest contribution the power of integrating user data and allocating payments. 

Weakness:
1. The main techniques come from previous literature. The method of deciding the contribution is basically the Data Shapley method from (Ghorbani & Zou, 2019), and the method of allocating compensations is just the least-core method from (Yan & Procaccia, 2021).
2. The motivation for using property rights theory to assign integration power is not clear. How is this method fundamentally different from having a trusted central planner? It seems to me that the method is basically choosing a central planner among the agents based on their contributions. Furthermore, it is not clear how to prevent the integrator from malicious behaviors of giving non-truthful payments. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, but it is a bit difficult to grasp the exact contribution of the work. The conceptual contribution is good, but the methodological contribution is limited based on the previous work. The contribution is original as far as I know.
summary_of_the_review: The main contribution of the paper is the conceptual contribution of using the modern property right theory in the problem of data valuation. The methodological contribution is not super strong based on the previous work. The motivation and the practicability of their method are not clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a data valuation framework based on property right theory. 
strength_and_weaknesses: Strength:

- The paper takes an inter-disciplinary approach and couples property right theory, cooperative game theory, machine learning to address the problem of data valuation

- The questions examined by the paper are crucial: how to address the ownership problem on a data chain? How to compensate each agent?

Weaknesses:
-Section 4.3: it’s not clear how to determine $B_0^i$ in Eqn (8)
- The efficiency aspect of the pricing mechanism in (9)-(10) needs to be further discussed. Specifically, to determine $\gamma$, one needs to solve Shapley value, which has exponential complexity in the number of agents. The optimization in (9)-(10) per se is also combinatorial in the number of agents. 
- One of the most intriguing part of the proposed framework is the ability to deal with data chain (i.e., different agents take different roles in a data pipeline line: some are data pre-processor, some are model trainer). However, this part is not empirically validated in the experiment 
- Section 5.1 aims at validating Theorem 1. However, this theorem was not the contribution of this paper but proposed in Yan et al. It’ll be great if the paper can focus on evaluating the differentiating factors of the paper.
- The writing of the paper can be significantly improved. This manuscript contains a lot of typos and grammar issues. 
- If the paper’s final target venue is ML conference, I would suggest the authors to significantly revise Section 3 “Preliminaries and Background” to contextualize the property right theory and other economic concepts in the machine learning and data analytics setup. 

clarity,_quality,_novelty_and_reproducibility: The clarity and quality of writing can be improved. Novelty is great. Reproducibility can be improved by providing some hyper parameters of the experiment.
summary_of_the_review: The paper takes an interdisciplinary approach to address an important question in data market. However, the experiment part does not sufficiently validate the key contributions of the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper discusses the data ownership problem from an economic perspective. The authors propose to use a compensation mechanism in the Modern Property Right Theory for solving data pricing problems in machine learning. Transferring the property right or ownership requires compensation, which is calculated by a modified least-core calculation. The formulation also incorporates fairness notions by introducing a parameter related to the contribution rate calculated through Shapley values. The paper also shows empirical experiments on data removal to demonstrate the usefulness of the proposed mechanism.
strength_and_weaknesses: Strength:
1. Provides data ownership from an economist’s perspective. It is an insightful contribution to formulate data ownership and pricing problem in the property right theory.
2. The time value introduction to contribution calculation is interesting.

Weakness:
1. The writing of the paper needs major improvement. Proofread to make the paper flow consistently. Also, providing brief explanations for concepts used as well as intuitive interpretations of the theoretical results could significantly improve the readability of the paper.
2. Some details of the method are not clearly described, partially due to the flow of the paper.
3. The experiments are lacking.

I have provided more detailed explanations of the weaknesses below.
clarity,_quality,_novelty_and_reproducibility: 1. How do you determine $W_i$ for each of the agents? Especially, in the experiments, if disrupting the orders of $W_i$ greatly changes the least-core’s result, then the $W_i$’s must be different for different agents. This essential part of the pricing mechanism is not clearly explained and specified in the paper.

2. It is unclear why stability is naturally preferred over fairness in the allocations discussed at the beginning of Section 3.3. Suppose stability is what we care about, why is it necessary then to again incorporate fairness concepts in the final Core-cs formulation in Equation (10)? Is it right to say that it is easier to add elements of fairness into the core solution concept?

3. I am confused about the rounds of investments described in Section 3.3. Can an agent make multiple rounds of investments? Also, I believe the compound interest should be related to the contribution rate of each agent. However, why is there a discrepancy between the notation $\gamma_t^i$ and $\gamma_i$. Please clarify this part and also the $\gamma$ used in Equation (10).

4. I need clarification about the problem setting. The paper’s title restricts the method to data pricing problems where I suppose data samples are the resource that each agent invests in and collects. However, it does seem that the setting can be more general such as each agent can perform different functions in the process, such as data cleaning, pre-processing, or even training as briefly mentioned in the example in Section 4.2. My understanding is that as long as the utility can be clearly specified, the contribution rate and then the final compensation can be computed. Please clarify this point as it might mean wider applicability of the proposed method.

5. There is confusion between the relaxation deficit parameter e in the relaxed core in (1) and the standard exponential number in the softmax function in (7). Also, is there any other stronger justification for using the softmax instead of the linear normalization? For example, how does it affect the experiments discussed in Section 5.2?

6. The experiments are insufficient. 1) The figures do not strongly support the necessity of Core-cs, since the performance is not significantly different from Core in three of the four plots. 2) The effectiveness of the valuation is questionable on the breast cancer dataset (Figure 3), why does the accuracy increase after removing the first 10-20% of the highest valued data? 3) For the data pricing scheme to be practical, a much larger scale experiment with more complex datasets should be performed, e.g., on MNIST, CIFAR-10/100 or ImageNet.

7. For the writing: There lacks an intuitive and clear description of the property right integration in the introduction and motivation section. I feel this is necessary to understand property right transfer as well the overall idea of the paper.

Minor:

8. Use of abbreviations. What is ICT on the first line? It was never spelled out anywhere else.
9. Inconsistent capitalization. “Even” in the first paragraph. “High” in the second bullet point of Section 3.2. “when” in the last paragraph of Section 4.3.
10. Inconsistent in-text citation. Some citations come with a space in front of the parenthesis but some don’t. Please make it consistent.
11. Typo. “propsed’ in the second bullet point of Section 3.2. “effct” in the second paragraph of Section 5.2.
12. Careless. Missing punctuation before “This allows” after Lemma 2 in Appendix A. Missing parenthesis in (12) in Appendix A.
13. Figure captions. Are the captions for Figures 2 and 3 reversed? According to the experiment section, Figure 2 is for (Synthetic) dataset while Figure 3 is for the breast cancer (Natural) dataset.
summary_of_the_review: This paper is a nice addition to the machine learning data pricing and ownership literature from an economist’s perspective. It is new to me that an agent contributing the most could perform integration through compensation, and subsequently gain income from all integrated data property rights. This process could be efficient from a societal point of view. However, careless writing and the lack of clarity in the presentation affects the readability of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in proposing a novel data valuation method based on property rights theory.

All the reviewers have agreed that the contributions of this work are novel and interesting.

On the flip side, there is a fairly large number of clarification and writing issues to be resolved. The authors have provided a detailed rebuttal to clarify most of them as well as revised the paper to rectify some writing issues. The authors are encouraged to do another proof-check to clear up the remaining issues with writing.

(A) The main concern shared by the reviewers during the discussion is the lack of empirical evaluation on data chain valuation which is the central idea/motivation of this work. In this regards, the authors need to set up appropriate experiments for empirically validating the performance and benefits of their proposed approach.
justification_for_why_not_higher_score: See the main concern (A).
justification_for_why_not_lower_score: N/A.
Decision: Reject