Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
This paper formulates a very interesting and novel problem (IDA, induced domain adapatation) in transfer learning. Consider the supervised classification setting where one usually trains a classifier $h : X \mapsto Y$ from samples $\{(X, Y)\}$ drawn from some distribution $\sim \mathcal D$. Oftentimes, when the data are generated from human input, the human could possibly modify $(X, Y)$ to adapt to $h$, resulting in a distribution shift over $\mathcal D$. Therefore, when training $h$, it is important to take this distribution shift into consideration. However, this can become complicated and interactive if the human further adapt to $h$.

This paper conducts a rather detailed study on this problem. It proves upper and lower bounds for the transfer risks for several important fundamental questions in this setting, as outlined in page 2. Besides, the paper realizes their bounds by both showing how to compute them in practice and computing them on real datasets.

strength_and_weaknesses: 
### Strength

1. The setting studied in this paper is important, interesting, and novel.
2. This paper studies most fundamental questions in this setting and presents satisfactory results that covers several fundamental lower and upper bounds in this setting.
3. As a mainly theoretical paper, this paper is aware of the practical impact of their theoretical results. To this end, this paper shows how to compute their upper bounds in real-world problems and conducts experiments to demonstrate their results. Furthermore, the experimental results corroborates their theoretical results and suggests that the theoretical results could give meaningful upper and lower bounds for the transfer risks.

### Weakness

1. The proof techniques look simple and the techniques themselves might not inspire broader community.
2. The authors could possibly conduct experiments on more datasets to strengthen their experimental results.

clarity,_quality,_novelty_and_reproducibility: 
The writing is good. No apparaent typos are noticed. The authors supplies sufficient materials (codes) for reproducing the experimental results in this paper.

summary_of_the_review: This paper has good theoretical and empirical results, yet the techniques might not be exciting and the experiments might not be convincing enough due to paucity of datasets evaluated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This papers studies the performance of models on the distribution induced by the model itself. The paper first provides examples of when this situation arises. Then, they develop formal definitions of metrics that measures the performance of the model on the induced distributions. These new quantities are inspired by the theory of domain adaptation (DA), like the induced risks that the model should minimize, which is similar to the target risk. Then authors show lower and upper bounds of these quantities. Afterwards, authors use different examples to illustrate their bounds. Finally, their bounds are computed on a real-world dataset.
strength_and_weaknesses: This paper studies a new problem and defines new quantities to understand and study it. It makes the paper mostly theoretical and related to the statistic field. Authors define many quantities inspired from DA. However as in this problem we do not have access to target samples unlike in DA, this problem seems more related to domain generalization (DG). I wonder if some theoretical or training procedure developed in DG could be applied to this problem and maybe this should be discussed in the paper. 

Regarding the bounds, they are explained and applied to several examples. They are also evaluated on the FICO credit score dataset.
clarity,_quality,_novelty_and_reproducibility:  The related work is, to the best of my knowledge, complete. Regarding discussion, I think a short discussion about the challenge to minimize induced risk should be added in the main paper and it should refer to appendix D for full discussion. The paper is well written. A short paragraph about the difference in setting between domain adaptation and the considered problem could also be useful. In particular, the lack of access to target samples. 
summary_of_the_review: I agree that the problem is appealing. The defined quantites and theorems are also interesting. The main downside is that I would have expected a training procedure to minimize the induced risk. While authors discuss the challenges of this question in appendix, it should also be discussed in the main paper.

The lack of training procedure to minimize the induced risk and the lack of empirical contributions make that I recommend 'marginally below the acceptance threshold'. I would be happy to increase my score if authors provide additionnal details/toy experiments on how to minimize the induced risk.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies model transferability when human decision subjects respond to the deployed model. It provides a series of lower / upper bound between empirical risk minimizer classifier and the optimal classifier. In particular, teh paper studies two common cases: covariate shift and target shift.
strength_and_weaknesses: pros：
the paper is clearly written, and the introduction of the problem is well motivated. The organization of the paper also makes it easy to follow, with clear notations / explanations to the theorems etc. 

cons:
As the authors write in Section 3.3, the bounds in the paper is mainly of theoretic interest. I would like to see some disucssions on the tightness of these bounds, even a single example would be helpful. The mathematical tools used are also not deep.

clarity,_quality,_novelty_and_reproducibility: Overall the paper is pretty clear. My main concern is still the use of these bounds. In the scenarios described in Section3.3, I would be interested in knowing
1. if the provided bound tight in any sense for the empirical risk minimizing model h_S?
2. as the authors write in Section 6  "indicating the suboptimality of training on D_S", is there any other learning algorithm that provides better bounds? Intuitively, training with a penalty term would likely give better results.

Of course it may involve more assumptions, but I think a tightness result, even a asymmpototic one would be very useful.
summary_of_the_review: The paper gives a series of bounds for the model transferability, the proofs I checked (non-exhaustive) are all good. It would be nice to see more discussions on the tightness of these bounds, as in the general case, the positiveness of the lower bound is not even clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors explore what guarantees we can identify on the performance of classifiers in response to classification subjects which act strategically - that is, attempt to update their inputs so as to achieve a better output. They call this loss the "induced risk": the risk the model incurs on the input distribution which its deployment induces. Their results are mostly of two forms: upper bounds on the induced risk above the optimal induced risk, and lower bounds on the minimum of the source risk and the induced risk. The machinery used for their proofs is mostly domain-adaptation-inspired, where the source distribution is the original data, and the target distribution is the strategically modified data.
strength_and_weaknesses: Strengths:
- These are useful results and I think more general than what has previously been shown in strategic classification
- paper is mostly pretty clear and exposition is good

Feedback: 
-Sec 4 and 5: it isn't clear to me why Covariate/Target shift would be useful assumptions for the strategic classification setting. One would imagine that the induced distribution will have a much lower P(Y = 1 | X) for X which are right over the decision boundary, for instance; or that P(X | Y = 1) would cluster more closely near the decision boundary in the induced distribution. I'm happy to be convinced otherwise, but it seems to me that by the definition of strategic classification, these results aren't so useful.
-Sec 4.2: it would be good to explain why these might be natural assumptions: I think I understand it has something to do with people choosing to move their inputs towards the desirable outcome Y=1, but not totally sure
Sec 4.3, Setup 3/4: I think there needs to be a little more definition around how the adapted feature is generated. For instance, if \tau = 0.5, x = 0.4, and B = 0.2, then the probability of a successful update is 0.5, but it is drawn from the distribution U(0.5, 0,3), which doesn't obviously make sense to me.
-Prop 4.7: I don't quite understand why as Err_D_S(h*_T) goes up in this bound, the gap increases - it would be nice to get some intuition on this relationship
Replicator Dynamics: It would be good to have a better description of this in the body of the paper, I'm not so clear on exactly how this works
Experiments: I don't totally understand the data generation process, even after looking at the supplement. It would be good to have a clearer explanation of the exact strategic modification model here (e.g. what is \epsilon or \alpha, as defined in the supplement)

Notes:
Eq (3) - would be good to define the optimal model here: it's not clear at this point in the paper if h*_T is the model with the optimal induced risk, or optimal source risk (you explain later, but should explain here)
Sec 4.2 - should define X_+(h) and X_-(h)
Sec 5: a little bit confusing to me to overload w here as the induced positive rate, when it was the weighting coefficient earlier
Thm 5.2: what is p here?



clarity,_quality,_novelty_and_reproducibility: Clarity: decent clarity around the exposition and proofs, some confusion around replicator dynamics and experiments
Quality: I think the work is solid, however I'm not sure that the assumptions relied on by the proofs are all appropriate
Novelty: I think this is novel because it uses lighter assumptions (mostly) than previous strategic classification work
Reproducibility: I have a little confusion around the experiments at the moment, with a bit more information this could be reproducible
summary_of_the_review: Overall, this paper provides some useful results in the strategic classification space. I have some doubts about the applicability of the covariate shift/label shift assumptions to this setting, and therefore not sure how useful those results are. Additionally, I have some confusion around other assumptions made around the data generative process with replicator dynamics and the experiments. Due to some good results and exposition alongside these concerns, I'm recommending a Weak Reject for now.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at the following property. When a ML algorithm outputs something (a classifier or whatever), the environment reacts to it and this induces a change in the data-generating process. As a consequence, the expectation of the loss after the output is not the same before. This paper basically looks at how this change occurs and quantifies it.


This idea of response to output is not new and is precisely the purpose and motivation of performative predictions. I find it weird that this paper only discuss in a few lines the differences with it. I would have expected a lot more. So I cannot give credit to that paper of this idea.

Then, the reviewers are also lukewarm concerning the other contributions (the proofs are certainly not straightforward, but techniques are quite common) and I have to agree with them.

In conclusion, this is a nice paper, but too incremental to reach the high bar of ICLR. This said, I am pretty confident that it will face a better fate once revised and resubmitted elsewhere.
justification_for_why_not_higher_score: The contributions are interesting but too incremental
Decision: Reject