Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a statistic from information theory as a tool to analyze several aspects in machine learning: data, model, etc. 
use the codelength of a dataset subtracting the cross-entropy of the final model. The results are interesting, but slightly fall short of novelty, and do not lead to any significant implication/improvement to the existing machine learning pipeline/model. 

Clarity&Originality&Significance: 

Below I will take Section 3 as a specific example to elaborate.
In Section 3, the authors try to use L(M_D,D) to "define" task complexity, and argues that noise and task complexity are two independent directions. 

It seems what the authors try to define, is simply the mutual information between X and Y, where {x_i,y_i} is drawn from the joint distribution of (X,Y). The quantity L(M_D,D) is an approximation (lower bound) of the mutual information. This is a commonly known fact, which makes the results fall short of novelty.

When the authors define the approximation as the task complexity, it becomes dependent on both model and the dataset, not task alone. Therefore, L(M_D,D) will be strongly dependent on how one trains the model, and the size of dataset. For example, with/without dropout, L(M_D,D) will differ significantly. 

As a result, it is difficult to get useful take-away from the results of Section 3, such as Figure 3. 

In addition, it is unclear what the implications would be if we have an approximation of mutual information between X and Y. Could we use it to improve the learning of the task?

In authors' response, I hope the authors can clarify the dependence of their metric on model, dataset, besides the task. Also, please clarify how it relates to mutual information, whether there is any related work that studies the approximation of mutual information between X and Y, and their relationship with the current work. Also, it would be interesting to know if there are any potential application for the proposed descriptive statistic on task complexity.

review_title: An interesting survey, but being short of significant novel contributions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper examines different use-cases of a quantity proposed in prior works which is said to capture the model information. It shows that this quantity behaves as expected overall. Quantifying the amount of information a deep neural network is a very interesting question for the community with both theoretical and practical appeal (from an analytical perspective but also from a model selection perspective, for example).

Overall, I’m scoring the paper with a reject. While the model information seemingly behaves as expected, the weaknesses in the exposition, as well as potential mistakes and a lack of rigour in the paper, will require careful editing.

Quantifying model information can be an important tool. This paper sets out to show that quantities defined by Voita & Titov (2020) and Zhang et al (2020), respectively, seem meaningful and behave as one would expect.

However, the examples lack falsification and depth. 

The structure of the ResNet in 7(a) is either wrong or non-standard: The residual connection is usually just an identity function: `out = in + block(in)`. Is this intentional? And if it is, what does this tell us about regular ResNets then?

Similarly, for the model ablation, it is not clear why resetting the parameters to their initial parameters is the appropriate method of ablation.

Another potential issue is that the definitions in Section 4 pretend that $L(M_A, B) = L(M_B, A)$ when it is clearly not. See also [Xu 2020](https://arxiv.org/abs/2002.10689) for example.

This reviewer would wish for a more in-depth treatment of the various examples in order to be thoroughly convinced.

### Minor concerns

Figure 1 uses a line plot, even though the x-axis refers to unordered categories.

---
### Rebuttal

I thank the authors for their detailed reply. I still consider the contribution to be too high-level and to cover too much ground without going into sufficient depth. I am not sure I can follow the argument about Kolmogorov complexity. Its chain rule is also only equal up to a logarithmic factor. I will keep my score the same.
review_title: Promising paper of case-studies for model information. Reservations about rigour and superficial treatment arise, however.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The central concept of this paper is model information, a description length of a discriminative model. The authors advocate usage of model information for analyzing several aspects in deep learning. In particular, they show how model information can be used to judge about difficulty of supervised tasks, domain similarity, model capacity, roles of different network components, and knowledge distillation. All of these are important topics and are relevant to the ICLR community. While most of the definitions and interpretations seem intuitive and valid, there are a few concerns and questions, and in some cases, it is hard to decide whether the conclusions of the interpretations should be trusted.

**Model information**
- The term *model information* is vague and may create associations with other quantities like the Shannon mutual information model has about the training dataset, $I(\theta : D)$. I suggest to clarify early that by model information it is meant some kind of description length of the discriminative model $p_\theta(Y \mid X)$.
- As model information is not a well-defined concept, it is not appropriate to say "an approximation of model information". Better to say "an instance/a variant/a definition of model information".
- As all experiments are done using the instance of model information defined by Zhang et al. [1] (called Information Transfer), the conclusions may not hold for other instances of model information. For example, if one defines model information with the prequential coding of Bleir and Ollivier [2], then increasing the number of examples with noisy labels will make the task more difficult. 
- Please clarify whether the $k$ examples in equation (2) are from the training set $D$ for not.

**Task difficulty**   
I really like using model information for assessing task difficulty. The only concern is that the task difficulty depends on the network and the training algorithm. Additionally, I suggest to plot task difficulty versus the number of training examples. Will we see that it plateaus after enough number of training examples are given? How would task difficulty behave in the small data regime?

**Domain similarity**
- What do union and intersection signs mean for datasets in equation (4)? If they mean the same as in equation (3), you can just repeat the same notation.
- Please explain how the right-hand-side of equation (4) fits into the framework of equation (3).
- When reading that $S^\text{uni}(A, B)$ is asymmetric, one might imply $S(A,B)$ is symmetric, which is not true. Please clarify that both measures are unidirectional.
- "For example, if $S^\text{uni}(A, B) < 1$ and $S^\text{uni}(B, A) = 1$, then one can tell that A is a subset of B." Please explain why is this sentence true?
- The definitions of domain similarity of equations (4) and (5) seem a little bit arbitrary. Why should they be defined like that? How do these domain similarity measures compare to other measures, such as domain similarity computed by Task2Vec [3].
- For easier interpretation of Fig. 5, you can use the same color map in all subplots.

**Model capacity**
- As I understand model capacity defined in equation (6) depends on the task. Why shouldn't one take supremum over tasks too?
- In the experiments of Fig. 6, do all datasets have the same size?
- The straight lines in Fig. 6c don't approximate the curves well. To have more points on the horizontal axis you can consider Resnet-[18/35/50]-k networks and vary $k$.

**Ablation**
- How do we know that quantity defined in equation (7) truly captures how important a component is and that we should trust the conclusions of experiments presented in Fig. 7?

**Minor notes**
- To make the paper more self-contained, please mention in the main text which definition of model information is used in the experiments. The same applies for model confidence.
- In eq. (7), $M_D^{\bar{c}}$ should be defined beforehand.

*References*  
[````1] Xiao Zhang, Xingjian Li, Dejing Dou, and Ji Wu. Measuring information transfer in neural networks. arXiv preprint arXiv:2009.07624, 2020.
[2] Leonard Blier and Yann Ollivier. The description length of deep learning models. NeurIPS 2018.
[3] Achille, Alessandro, et al. "Task2vec: Task embedding for meta-learning." Proceedings of the IEEE International Conference on Computer Vision. 2019.

review_title: Good analysis paper, but a few concerns and questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper discusses the use of model information to assess properties of deep learning models and problems. In particular, it illustrates how model information may be used to capture the difficulty of a task, the degree of similarity between domains, the capacity of a model 

The idea of using a quantity related to codelength to assess properties of a learning model is interesting; indeed quantifying the properties chosen by the authors (task complexity, domain similarity) would be certainly useful. However, I am dubious whether the assessment using model information would be reliable.

To my understanding model information evaluates the amount of information a trained model M_D contains about dataset D as a difference in codelength of the data and codelenghth of model+data (as explained in Section 2). Critically, it seems to me that this quantity depends both on the data AND the model. It is therefore a particular measure for a specific model (or at most a family of models) and a dataset. (Incidentally, although guessable, it would be proper to define all the symbols in the equations).

In Section 3, it is not clear to me how this measure may be used to evaluate, for instance, the difficulty of a task: how is M_D chosen? Is there an underlying claim that this measure would be independent from the choice of model? What is the sensitivity of information measure to the choice of the model? I am also perplexed by the results of the simulations: it seems that dropping samples or scrambling labels makes the task easier; this, though, is quite counterintuitive as it would suggest that to simplify the problem we could literally drop samples or scramble labels (I guess, but I may be wrong, that this outcome is due to the fact that "difficulty" is not evaluated in terms of generalization).

Similarly, in Section 4, it seems to me that a discussion on M_A and M_B is lacking. It is stated that a property of informational similarity is its independence from particular representations/modelling. This seems to me not to hold for model information where we have to rely on M_A and M_B.

In Section 5, how is a given task T defined? This particularly important as it defines the set over which the maximization is computed and with respect to which capacity is defined. My intuition is that computing this quantity on a small set of tasks would return at best a lower bound.

I have some doubts also in Section 7, on Equation (8). It would seem to that the model information of the student should be relative to a different dataset, one enriched with the output probabilities generated by the teacher. Why is it not?
review_title: Original approach requiring more rigorous foundation and justification

Meta-review: This work presented a broad set of interesting applications of model information toward understanding task difficulty, domain similarity, and more. However, reviewers were concerned around the validity and rigor of the conclusions. Going into more depth in a subset of the areas presented would strengthen the paper, as would further discussions and experiments around the limitations of model information with regards to specific models and dataset sizes (as you have begun to discuss in Section 8). Additionally, reviewers found the updated paper with connections to Kolmogorov complexity interesting, but reviewers wanted a more formal treatment and analysis of the relationship. 
Decision: Reject