Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper contributes to the growing body of work on self-supervised representation learning approaches. It appears to have a strong theoretical foundation based on the concept of predictive information. The results seem competitive and there are visible efforts to place the method in the context of existing work.

Pros:
- Strong theoretical foundation, well explored.
- Competitive results in multiple contexts.
- Mostly clear with adequate detail for the method and experiments.
- Originality: combining DCA estimation of PI with additional regularization terms + shifted masked reconstruction.

Cons:
- Although there is plenty of discussion of related work and how the current method fits in with existing approaches, it would be helpful to tie these other approaches back more explicitly to the compared methods in Fig 3 / Tab 1 / Tab 2. Perhaps it would be possible even to add a table comparing the methods discussed across various dimensions (e.g. discriminative/generative, contrastive, mutual information entities [representation vs input, past vs future], model size, and other important characteristics in which they differ).
- It is not clear whether the results for other published methods are taken from their respective publications or reproduced. If taken from publications, are the models used comparable?
- I would like more discussion on the pros and cons of using masked reconstruction versus full reconstruction (as in a canonical auto-encoder) as a way to avoid learning degenerate representations.

Overall, I think this paper would be of interest to the community.
review_title: Accept

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper builds on the prior work of Dynamical Components Analysis (DCA) which maximizes the mutual information between past and future temporal windows around the current time step, referred to as the the Predictive Information (PI) loss.  
In this paper, the PI loss is used to train a neural encoder that learns continuous latent representations of input sequences. The PI loss is regularized to have orthogonal latent space. It is further improved by summing the PI loss applied at multiple scales, and by adding a masked reconstruction (MR) loss. The paper presents results on three domains, with speech recognition as the main one. 
The paper presents a nice extension of DCA, as well as a probabilistic interpretation motivated by the variational autoencoder (VAE) framework. It is clearly written with good citations of previous work. 

On the other hand, The experimental section requires more work. Here are some directions for improvements: 
1) Adding the MR loss, combines another effective pre-training mechanize from previous work. I'm assuming that in tables 1 and 2, DAPC (written alone) refers to models optimizing the PI loss without MR (please correct me if this is not true). If my understanding is correct, the PI loss doesn't require MR to avoid degenerate solutions. Further analysis for the ASR models would help the readers understand different cases when the MR is required. 
2) Following on the previous point, table 1 is missing the MR only results in the upper section, and missing the DAPC only in the section section. This is important to understand the relative contribution of each pre-training loss.
3) In table 2, the reported results are using a 30M parameter model compared to larger previously published models (~150M parameters). Given that the results of the proposed model is worse than those larger models, it is not clear why the authors didn't pre-train larger models of similar capacity for fairer comparison to their proposed approach. It is known that pre-training larger models yield better final representations for downstream tasks. Would it be the case for DAPC? 
4) The pre-training on librispeech only is done for a 1 epoch. This seems pretty short compared to prior pre-training work on the same dataset (correct me if this is not accurate). Do the learned representations using DPAC stop improving after 1 epoch of pre-training? If possible, please share results of pre-training for larger number of epochs for the DPAC only, MR only, and combined loss. 

review_title: A neural extension of Dynamical Components Analysis

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is an interesting paper with several proposed theories on how to improve self supervised learning objectives through a "predictive information" type objective, combined with a masked reconstruction loss. The hypothesis is interesting, and has some benefits, specifically that it does not require contrasting with negatives unlike many recent SSL methods. 

Most experiments performed on the speech recognition task. While the theory and discussion looks plausible, the experiments are somewhat on the weak side. Specifically:

1) the differences in word error rates in the ablations are pretty small. The authors claim to run 3 seeds for each result and report the mean - can we see the standard deviation?
2) why not compare to wsj results from cpc-style training such as wav2vec and vq-wav2vec?
3) eval on Librispeech is only on the clean set. Why not show results on noisy, where differences between techniques should be somewhat more apparent?
4) there are newer models to compare against, e.g. wav2vec 2.0
5) the authors claim that they perform worse than e.g. BERT + vq-wav2vec because their model is smaller. But they use standard transformers as their backbone - it should be straight forward to scale this up and have apples to apples comparison.
6) this is positioned as a general SSL technique - what about experiments in other modalities like nlp or vision?
7) is "DAPC" just the PI loss without masked reconstruction? (i am not sure if "shifted recon" just adds the shifting, or it adds the entire reconstruction dimension). I think that is the case, but i am not 100% sure - maybe its worth to make this more clear.

Overall I liked the premise of the paper but unfortunately the experiments left me unconvinced in the value of this approach and its various components

Update:
thanks for your reply. i remain not fully convinced of the improvements with the proposed method and i look forward to additional experiments in NLP. i do think this approach is valuable for additional study however and updated my rating to reflect this
review_title: Decent analysis; weak experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes Deep Autoencoding Predictive Components (DAPC), a self-supervised representation learning approach for sequential data. In this approach, the model learns to maximize the predictive information, which is the mutual information between past and future time windows. In order to avoid degenerate solutions, the proposed approach relies on a second loss that optimizes masked reconstructions.

Strengths:
* The paper is written clearly and is easy to follow. 
* The proposed method seems to outperform the baselines across all considered tasks.
* The usage of predictive information as an objective on the latent representations is well motivated and seems to improve performance over a non-regularized autoencoder.

Weaknesses:
* The paper focuses exclusively on one interpretation of their approach, in which the optimization of the predictive information is the main objective and the reconstruction loss merely pushes the model to avoid degenerate solutions. Explicitly considering the reverse interpretation, in which the predictive information is a regularizer applied to an autoencoder to enforce a better latent structure, could provide more depth to the paper: 
    1) Following this interpretation, it would make sense to include more references in the related works section describing autoencoding approaches for representation learning.
    2) Based on this interpretation, it would also be interesting to investigate the effect that the predictive information regularizer has on the autoencoding model. This is partially done in section 4.3, but missing from the other two experiments. 
    3) Based on this interpretation, the probabilistic interpretation of DAPC (sec 2.3) can be reduced to describing a standard VAE with additional regularization on the latent space in the form of predictive information.

* The experiments show strong results for the proposed DAPC. However, it remains unclear how much fine-tuning has been done for the DAPC, especially in comparison to the baselines. For example, I would imagine that the CPC model might perform better on the lorenz attractor (Fig. 2) when optimizing the parameter k (which influences the temporal lag between the positive samples). The experiments would provide a much stronger point for the superiority of DAPC if there was a clear outline of the considered hyperparameters for all models. Additionally, details, such as the employed model architectures, that would be needed for reproducing the results are missing.

Additional comment:
In the related work section, the paper states “Unlike prior work, our principle for sequence representation learning is to maximize the MI between the past and future latent representations, rather than the MI between representations and inputs.”. That characterization of previous work is not entirely correct: CPC, for example, optimizes the MI $I(x_{t+k}, c_t)$ between the future input $x_{t+k}$ and the current representation $c_t$. Through the data processing inequality, this will result in the model implicitly also optimizing the MI $I(c_{t+k}, c_t)$ between the future and current representation (Tschannen et al., 2019).
review_title: Interesting approach, lacking some depth in the interpretation and experiments

Meta-review: The paper combines a few different ideas for representation learning on sequential data and is able to achieve competitive WER on the Librispeech ASR dataset. I appreciate the fact that the authors engaged with reviewers and tried to improve the paper. While I get a sense that the final system has many moving parts, I believe the paper meets the bar for acceptance at ICLR.
Decision: Accept (Poster)