Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to embed static feature maps into a larger linear space and shows that the proposed method achieves good performance on standard benchmarks. Detailed proofs and theoretical results are given in the appendices. The use of free algebras in ML seems novel and under-explored, although it is classical in mathematics. This paper shows that algebraic structure can significantly elevate the performance of existing models empirically. 

Pros:
+This paper shows that the free algebra T(V) can capture the non-commutativity among sequential data theoretically and experimentally, which is under-explored in ML area.
 
+To address the heavy computational complexity of the free algebra, compositions of low-rank tensor projections(CP decomposition) are employed. 


Cons:
-The article is not clearly presented. The introduction is short and without illustrations to show the motivations. There is no intuitive explanation to show how the free algebra improve the models' performance.

-Some notations used in paper are confusing.
a)In Section 3((Stacked) Seq2Tens layers.), what is the difference between D and $D^`$;
b)In Algorithm 1, the truncation degree M and the tensor M share the same notation;
c)In Figure 3, why have the 64$\times$2 S2T and the Seq2Tens blocks different names and shapes? Aren't they the same module？

-A lot of important information, including the details of architecture, algorithms and experiments etc, needs to be obtained from the appendix. This makes the article difficult to understand.

Some typos:
Sec2: a sequences
Appendix D.3, line 5 : be $n_l$ be rank-1 tensors.
Algorithm 1, line8: ·R

review_title: The paper uses a classical mathematical tool, i.e., the free algebra to capture this non-commutativity of sequential data, and then utilizes compositions of low-rank tensor projections to reduce the computational complexity.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper introduces the free algebra, a classical mathematical concept as a generic tool to represent sequential data of arbitrary length. The proposed method has attractive theoretical property, such as preserving universality of static feature mapping, and convergence in the continuous setting. The author further proposes using stacked rank-1 projection of the free algebra as an approximation to the sequence representation in order to make it computationally feasible neural networks layers. The author illustrated the flexibility and effectiveness of the proposed method by combining the NN implementation with FCN to benchmark on multivariate time series classification problem, and GP-VAE model to benchmark on sequential data imputation problem. The proposed methods shows improved results over previous state-of-the-art. 


Significance: This paper provides the community an extension of the universal approximation theorem of NN on sequential data, as well as a generic method to transform static feature maps into sequence features. The experiment shows the proposed method and its implementation is flexible and effective in both discriminative and generative problems. Some questions/feedback: 1. In Proposition 2.3, while the rank-1 projection makes the method computationally feasible, taking sum over all non-contiguous subsequences of x cannot be too cheap? Would the author add analysis on computation complexity here? 2. In the experiment section, as the main motivation of stacking Seq2Tens layers is to mitigate the limitation of the representation power, how does different number of stacked Seq2Tens layers change the model performance? 
 
Clarity: While the paper is highly technical, the author did a good job explaining the idea, concepts and objectives. 

Originality: I am not aware of any other work explore free algebra and its usage on sequential data representation. 
review_title: Interesting theoretical work!

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an interesting low-rank tensor representation learning model called Seq2Tens for sequential data. The proposed model can be plugged into existing state-of-the-art neural network models as Seq2Tens layers to improve the performance, which has been demonstrated on several benchmark datasets in this paper. 

On one hand, there are some solid contributions made in this paper. On the other hand, I do find it is difficult for the readers to appreciate its merits. Please see my detailed comments below.

1. Possibly due to my limited mathematically background, "the free algebra" in the abstract of the PDF paper is a jargon for me (and possibly other AI/ML readers). Although I could google and learn from Wiki, I am not sure whether it is necessary. In the text abstract online, the authors use "the tensor algebra" in the online abstract of this paper.  Are they interchangeable? Which one is more appropriate? 

2. There is no citation to existing works in the first two pages, including the Introduction section. In fact, there is only one citation (on page 3) on the universality in the first three pages of this paper. Does this mean that all other materials are original? Because I am not familiar with so called free algebra, I expect some references to be provided. 

3. In the Introduction, after stating the problem of interest, the authors start describing the contributions immediately, without mentioning any existing approaches to solve the problems and giving proper context in judging or appreciating the claimed contributions.

4. The name Seq2Tens is not properly defined, although I can tell it means "sequence to tensors". In fact, in the second paragraph on contributions in Sec. 1 Introduction, the authors did not mention the word "tensor" at all.

5. Section 2 is quite abstract and mathematically, providing some visual illustrations will make the readers' life easier (e.g. cutting example frames in Fig. 3).

6. Equation (3), the degree m is defined to be m>=0. Is it a user-defined hyperparameter? Is there a maximum value for m to take? If yes, what is it and how to determine it in practice? If not, does it mean that m can go to infinity (not likely)?

7. Notations are not standard or consistent. A matrix is denoted as \mathbf(t)_2 and a 3rd-order tensor as \mathbf(t)_3, but the authors also use \mathbf(t) without subscript, in which case I am wondering what is the order (or degree in the authors' chosen terminology). Then in Equation (7), the authors use a non-bold font ell to represent a tensor. In the equation between (1) and (2) (defining Seq2Tens), the right hand side is a capital V without bold, which I assume is a tensor. 

8. Acronyms are not always defined before use, e.g. TSC appears in Fig. 1 caption on page 5, but it is only defined on page 6. 

9. Section 4 refers too frequent to the appendix and it does not seem to be self-contained. Essential information about the data is missing so readers have to navigate through the 21-page appendix to find information necessary for them to appreciate the methods. 

10. Is there any hyperparameter to set? Is there a demonstration of convergence and sensitivity to hyperparameter settings? 

11. In Related work, the authors pointed out the vast literature on tensor-based models. However, there is no comparison against any of them. For example, many tensor-based machine learning algorithms have been applied to gait sequences for gait recognition so they model sequences as tensors too (google "sequence tensor gait recognition" w/o quote). These related works should be evaluated against the proposed model, many with public code available. 

12. As mentioned above, with many important information put in the appendix, I think the 18 frames displayed in Fig. 3 are more important than those essential information about the data, hyperparameter setting, etc. I am not sure whether human eyes can tell the improvements achieved in much less than 0.001 in MSE on Sprites and whether it matters in practice. 
review_title: This paper contains some solid contributions but the writing and design need big improvement to communicate and demonstrate the key ideas and benefits.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a method to map static feature space to a space containing sequences of different lengths. The idea is worth of  interest and the appendix gives large amount of information on both theoretical and experimental sides of the work.

The experiments are the main drawback for me of the paper. Indeed, the experiments for sequence discrimination are not convincing enough due to the lack of datasets and framework employed (only the multivariate time series classification problem), and the results obtained. The main results achieved during the TSC classification task are located with moderate or high prior (> 0.6). Moreover, the experiments are conduced with both small models and datasets.

The experiment for sequential data imputation for time series and videos is not clear, and the way of writing and giving explanation is quite confusing for the reader. Since the appendix is large, more details from the appendix in the paper itself will help the reader to more easily follow the idea.

Overall, the experiments have to be larger and considering trickier tasks such as language processing related experiments (speech recognition, language modeling, etc.) that consider sequence mapping to extract robust features.

review_title: An Efficient Representation of Sequences by Low-Rank Tensor Projections

Meta-review: The paper uses free algebras for sequential data representation, and two of the reviewers and the AC find this highly innovative. There were numerous small issues brought up by reviewers (and reviewers disagreed some on the presentation), in particular R3 asking about the experiments, some of which were addressed in the rebuttal.  Overall, because the idea was unusual, it's a bit hard to place and judge this paper. In the end, in the opinion of this AC, the ideas are very creative and there is enough of a chance that this paper could become a very highly cited work, hence we recommend its acceptance.
Decision: Accept (Poster)