Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a multi-stage influence function for transfer learning to identify the impact of source samples to the performance of the learned target model on the target domain. It considers two cases: fixed pretrained parameters and fine-tuned parameters.

Why not to directly add a scaled identity matrix to problem (15) to avoid the non-PSD issue?

How to use the proposed method to identify source samples that cause negative transfer as discussed in the introduction?

Even using the conjugate gradient method to reduce the complexity, the total complexity is still high as the number of parameters in a deep neural network is large. It is better to report the running time to see the efficiency of the proposed method.

In transfer learning, there is a setting that source data are not accessible due to, for example, the purpose of the privacy protection. In this case, can influence function be used?

For presentation, I think it is not correct to use ‘pretrain’ or ‘finetune’ before a noun. They should be replaced with ‘pretrained’ and ‘finetuned’.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This is an analysis paper of pretraining with the tool “influence function”. First, the authors calculate the influence score for the models with/without pretraining, and then propose some implementation details (i.e., use CG to estimate the inversed Hessian). To calculate the influence function of a model with pretraining, the authors use an approximation f(w)+||w-w*||, where w* is pretrained. 
The experiments are conducted on MNIST and CIFAR. 

1.	The idea of converting a pre-trained model with f(w)+||w-w*|| is interesting. But I do not think the conclusion is very promising and convincing. The authors leverage Pearson correlation to measure the similarity between “true loss difference” and “score value”. However, i do not think the value $0.62$ is significant. As shown in Figure (2), intuitively, the linear correlation between these two values do not hold. Also, I am not quite sure about the practical value of calculating influence scores.
2.	The experiments are conduct on small-scale datasets. I am not sure whether the conclusion holds for larger dataset.
3.	Page 7, last paragraph, “we replace all inverse Hessians in (11) with identity matrice”=>why?
4.	In figure 3, what is the relationship between the two MNIST images, and the relationship between the two CIFAR images?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors derive the influence function of models that are first pre-trained and then fine-tuned. This extends influence functions beyond the standard supervised setting that they have been primarily considered in. To do so, the authors make two methodological contributions: 1) working through the calculus for the pre-training setting and deriving a corresponding efficient algorithm, and 2) adding $L_2$ regularization to approximate the effect of fine-tuning for a limited number of gradient steps.

I believe that these are useful technical contributions that will help to broaden the applicability of influence functions beyond the standard supervised setting. For that reason, I recommend a weak accept. I have some questions and reservations about the current paper:

1) Does pretraining actually help in the MNIST/CIFAR settings considered? These seem to be non-standard pretraining settings. More generally, can we relate influence to some objective measure that we care about (say test accuracy), for example by showing that removing the top X% of influential pretraining data hurts test accuracy as much as predicted? Minor: section 4.2 also seems non-standard. Are the exact same bird vs. frog examples being used for both pretraining and finetuning?

2) In what situations might we want to examine the influence of pretraining data, and can we design experiments that show those situations? For example, perhaps we're wondering if different types of sentences in the one-billion-word dataset might be more or less useful. Can we verify those claims using these multi-stage influence functions? It is otherwise difficult to assess the utility of the qualitative results (e.g., Figure 3 and Appendix C).

3) It'd be helpful to get a better understanding of the technical contributions of this paper. Specifically, 
a. What is the impact of $\alpha$ in equation 12 and how does it interact with the number of fine-tuning steps taken?
b. If the Hessian has negative eigenvalues, we can still take $H^{-1}b$ by solving CG with $H^2$, but what does this correspond to? Is the influence equation well defined (or the Taylor approximation justified) if $H$ is not positive definite? 


review_title: Official Blind Review #3

Meta-review: This paper extends the idea of influence functions (aka the implicit function theorem) to multi-stage training pipelines, and also adds an L2 penalty to approximate the effect of training for a limited number of iterations.

I think this paper is borderline.  I also think that R3 had the best take and questions on this paper.

Pros:
 - The main idea makes sense, and could be used to understand real training pipelines better.
 - The experiments, while mostly small-scale, answer most of the immediate questions about this model.

Cons:
 - The paper still isn't all that polished.  E.g. on page 4: "Algorithm 1 shows how to compute the influence score in (11). The pseudocode for computing the influence function in (11) is shown in Algorithm 1"
 - I wish the image dataset experiments had been done with larger images and models.

Ultimately, the straightforwardness of the extension and the relative niche applications mean that although the main idea is sound, the quality and the overall impact of this paper don't quite meet the bar.
Decision: Reject