Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed TextTN that applied Tensor Network on text input. They effectively solved the high dimensionality problem and achieved the state of the art on multiple text classification problems. The paper also proved the bond dimension and showed experiment results to justify their theory.

In the experiments in table 1, was there a reason why you don't use the pretrained BERT embeddings? Can you compare with the BERT baseline as well?

In your BERT + TextTN experiment in table 2, do you finetune the pretrained BERT embeddings?

Could you please analysis on the efficiency of your model? It could be a benefit of TextTN over the baseline models.

The results on some experiments over the baseline models are not very significant (by ~0.1). I am concerned how much of them comes from hyper-parameter tuning vs. model structure. It's possible that the tasks are not very hard so there are not too much headroom. Can you evaluate it on harder tasks such as other tasks in the GLUE benchmark?


review_title: Official Blind Review #5

final_score: rating: 4: Ok but not good enough - rejection
review_content: After reading author replies:
I would like to thank the authors to respond to my doubts on some of the results. But I decide to keep the review and the score, because Theorem 1 and Claim 1 are still not well explained. In particular, the explanation like "if the 2nd inequality in Eq. 11 is violated, the network can not capture the amount of information measured by the entanglement entropy " still looks like a conjecture or intuition rather than a mathematical statement. 

---------------------------------------------

Originality: High.
The proposed tensor network (TN) based text classification model looks new and interesting.
It consists of two parts: a word-level generative TN model, used to find concise representation of each word; and a sentence-level discriminative TN model, used to classify the sentence based on the outputs of the word-level TNs.
By using the word-level TN for input representation, the dimension explosion problem may be effectively avoided.


Clarity of model description: OK.


Clarity of training method: Low.
The training method of the overall model is not presented. Since there are two TNs, how to train the overall model may not be a trivial problem.


Clarity of analysis: Low.
The major problem of this paper is the analysis presented in Sec 3.2 and Sec 3.3.
I understand that the authors may want to find some theoretical justification on how to choose the bond dimension in the sentence-level TN, as a function of the bond dimension of the word-level TN and the so called entanglement entropy.
However, the result in Theorem 1 is incomprehensible. Only two inequalities are presented, without stating any condition or implication of the inequalities. Since both m and the bond dimension are hyper parameters, what does the 1st inequality mean? Is it a necessary condition on how to choose their values? What happens if the inequality is violated? Even the proof of this theorem does not answer these questions. Same doubts are on the 2nd inequality as well. Additionally, how does one even know what the entangle entropy of a model is before training the model?

The statement of Claim 1 is even more problematic. Not able to understand what it says.


Clarity of expeiremtal results: OK, but not clear enough.
The authors claim that when combined with BERT for word embedding, the proposed model can outperform the SOTA methods.
However, there are several things that are not clear.
1. It would be more fair to compared the combined model with BERT with some similar models that also use BERT for word embedding.
2. It is claimed that the proposed method is better than word-GTN. But word-GTN is a unsupervised learning model. Why is it meaningful to make such a comparison?
3. In the introduction, another TN based method "TSLM" is mentioned. Would it be more fair to compare the proposed method with TSLM, as both of them use TN for modeling?


Overall, the proposed model looks interesting and shows some potential improvements over SOTA. However, the quality of the paper is degraded by the unclear claims and some questionable experimental results.
review_title: Interesting model; unclear claims; questionable experiments

final_score: rating: 7: Good paper, accept
review_content: Summary:
A tensor network model for text classification is introduced, which is constructed as the concatenation of a generative matrix product state (MPS) model for low-dimensional word embedding and a discriminative MPS model for classification. This model and different variants are assessed on multiple text classification datasets, with decent performance shown against a range of benchmarks.

Strengths:
The TextTN model seems to be the first tensor network model applied to text classification. In that sense, the experimental results given are important for assessing the usefulness of tensor network models for real-world ML challenges, a question which has seen limited study so far.

The close correspondence shown between the accuracy in classification tasks and the entanglement entropy of the models (Figure 3) is interesting, and hints at the possibility of a compelling link between theoretical quantum many-body physics and practical considerations in ML.

Critiques and Questions:
The word-GTN encoder (which converts high-dimensional word embeddings into low-dimensional inputs to the sentence-DTN) strikes me as being unnecessarily complicated, considering the small role it plays in the model. Given that this just outputs very low-dimensional word embeddings ($d=2$ in the paper), it would make sense to use a trainable word embedding here (or alternately a linear function of the original word embedding) in place of the word-GTN. On that note...

The "TextTN w/o word-GTNs" baseline in the ablation study (Table 3) seems rather misleading, as the paper text describes this as "we directly average the word vectors of words in a sentence to obtain the sentence representation". In other words, not only is the word-GTN removed, but the order of words is lost as well! I would request that this w/o word-GTN baseline to instead use a sentence-DTN whose inputs are low-dimensional vectors given by trainable low-dimensional word embeddings (or alternately a linear function of the word2vec word embeddings). This alternate baseline would make the TextTN significantly simpler (while still remaining a tensor network model), and would also bring it closer to the model of (Stoudenmire & Schwab 2016).

Recommendation:
I would recommend acceptance, owing to the new experimental evidence presented for the performance of tensor networks in NLP. However I do have some doubts about the encoding used for the model architecture, and request that the authors improve their ablation study to better justify the additional complexity coming from two separate linked matrix product state models.
review_title: Timely work, experimental setup has some issues

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary 
The paper proposes a tensor network for text classification. There are two components: (i) word-GTNs convert word embeddings to m-d probability encoding vectors, and (ii) a sentence-DND takes the word probability encoding vectors as input, combining them using matrix product state (MPS). Experiments on several text classification datasets e.g. SST, CR, MPQA show that the proposed method outperforms existing ones when using word2vec and BERT word embeddings. 

### Contributions 
The contributions are two-fold:

* The paper proves an effective lower-bound for the bond-dimension. This is then confirmed by experimental results.

* The used sentence-DND combines all possible locations inducing a distribution over the class set. Existing sentence-DNDs employs only location. 

###  Pros
I do like theorem 1 about a lower-bound of the bond-dimension based on entanglement entropy. It is very interesting to see how experimental results support the theorem nicely (Figure 3). 

The experimental results are a plus. 

### Cons

* Clarity: 
   - Although the appendices help a lot, the paper isn't easy to read, especially to people who are not familiar with tensor. For instance, what is a "tensor contraction" (right after equation 7)? 
   - The paper also doesn't show the complexity. For examples, each d-dim word vector is represented by a \phi(w) tensors, which should have 2^d parameters. In this case, what is the number of parameters of W in equation 7? what is the complexity of the whole model?
   - How to compute entanglement entropy (Appendix D4) isn't provided.

* Originality: In general, the paper doesn't seem to propose significant ideas. The two components are already used. 
   - In fact word-GTNs are just a classifier with a fix-sized input. These GTNs are used in computer vision. However, why don't just use a neural network mapping d-dim word embeddings to a distribution over m latent features. 
   - The extension of sentence-DTN seems to be trivial (actually I was wondering why no-one hadn't proposed that before). However, the extension should come with more complexity, which wasn't shown in the paper. 

* Motivation: The paper isn't convincing why using tensor networks is a good idea. It is unclear how tensors help the task (e.g. what is compositionality here?)



review_title: review

Meta-review: While the submission has promising components, the reviewers were not able to reach a consensus to recommend acceptance. The main concerns is that (1) theorem statements and assumptions are not clearly explained, and (2) the novelty of the approach is not made clear, and (3) there remain concerns on whether the experimental results are due to hyperparameter search or improvements due to the model.  
Decision: Reject