Low-Rank Constraints for Fast Inference in Structured Models

Abstract: Structured distributions, i.e. distributions over combinatorial spaces, are commonly used to learn latent probabilistic representations from observed data. However, scaling these models is bottlenecked by the high computational and memory complexity with respect to the size of the latent representations. Common models such as Hidden Markov Models (HMMs) and Probabilistic Context-Free Grammars (PCFGs) require time and space quadratic and cubic in the number of hidden states respectively. This work demonstrates a simple approach to reduce the computational and memory complexity of a large class of structured models. We show that by viewing the central inference step as a matrix-vector product and using a low-rank constraint, we can trade off model expressivity and speed via the rank.  Experiments with neural parameterized structured models for language modeling, polyphonic music modeling, unsupervised grammar induction, and video modeling show that our approach matches the accuracy of standard models at large state spaces while providing practical speedups.


Reviews and author responses:

 Appreciate your response. Decided to modify my score to reflect this.

This work proposes to reduce the runtime of the marginalization computation needed for inference in structured (graphical) models. Some examples of such models are: Hidden-Markov Models (HMM), Hidden Semi-Markov Models (HSMM), and Probabilistic Context Free Grammars (PCFG). In order to infer in this models, multiple matrix-vector multiplications are needed, cause of the bottleneck computation. The authors proposed to solve this problem by approximating the matrix-vector multiplication in the computation bottleneck by an approximated low rank factorization. This is achieved by considering kernelized approximations, yielding a reduced runtime complexity by a factor of the number of states. Such reduction allows for scaling approximate methods to larger models (more states). Lastly, the method is tested on a set of sequential data tasks showing similar accuracy to standard models but with increased runtime performance.
  This paper uses combinations of ideas to reduce the runtime complexity for the marginalization of a (structured) graphical model. Kernel methods and kernelized approximations are not new topics. Nevertheless, such application to structured models for runtime improvement is novel to the best of my knowledge.

In general, the submission covers all the technical details properly. However, several important details have been relegated to the supplementary material (like the definition of $K$ used for the models). Further the authors proposed a "neural" parameterization without much explanation or reference. These neural parameterizations required to train weights and are non-linear. Details on how to train them are missing. The claim the the property of linearization of $\Psi$ (line 139) has no reference or proof. Further, it is unclear why $\Psi$ is a square matrix (assuming 2 tail nodes, it looks rectangular). It could be interesting to try data with exponential decay to evaluate the HMM models as PTB and other types of data have a power law decay that cannot be captured with HMMs (see [1]). The quality of the approximation of $\Psi$ with different kernels was not explored and as a function of the number of states. Such experiment could share insights about the quality of the final result. 

The paper is well organized and most of it reads nicely. Hypergraphs and the subsequent definition of inference in each of the models, introduces a lot of notation that is hard to follow. One or more figures could help to reduce this burden on the reader. Note that each model has a different final solution given by the selected kernels. Important details are missing that can help understand the paper and reproduce it (see comments above and questions below).

The problem is an important bottleneck problem in graphical models and interesting to the NeurIPS community. The work has valuable practical aspects, and demonstrates the capability of scaling to a higher number of states. 

Questions for the authors:
1. Your work relies on kernelized approximations of $\Psi$ matrices. Can you explain in more detail why directly approximating each matrix $\Psi$ as a low rank decomposition is not a good option?  
2. How good is the approximation of the kernelized $\Psi$ matrices? What properties hold by using the kernels for the system?
3. What hardware is used for the experiments?


Minor:
* Formatting of captions in tables is incorrect.

References

[1] Mahto et al. "Multi-timescale Representation Learning in LSTM Language Models", ICLR 2021. Negative societal impact was included with the supplementary material instead of the main paper.
Error bars/standard deviations were not included in the paper. Is hard to assess the significance of some results.

The paper studies the problem of marginalization for a class of latent variable models defined over a hypergraph, which includes HMMs and PCFGs. The paper proposes a method to reduce the time and memory complexity of marginalization by a factor of $L/N$, where $L$ is the size of the label set and $N$ is a feature dimension to be chosen. In particular, the larger $N$ is, the more expressive the distribution of the model is. The idea of the method is to use kernel linearization to parameterize the conditional probabilities of the model.  The theoretical part is in general clear. A small thing that confused me was that section 3 never said how $K$ or $\phi$ was chosen. It took me a while to realize that this is a “user choice”, and to see how you choose it in the experiments section. Maybe this could be clarified earlier.

In the empirical part, my main concern is that the paper does not report the speed of the algorithm for the experiments other than the first one. There is no explanation for this — the text just shifts from talking about speed to talking about $L/N$ and time complexity. This implicitly suggests that the speedups are not significant for these experiments. If this interpretation is wrong and there are actual speedups, I think it would be very important to include the evidence for them. Practical speedups seem to be a very important ingredient of this paper, and the paper claims such speedups (e.g., at lines 49-59, it claims that “the decomposition allows us to greatly increase the speed of inference”). If there are no practical speedups, it is not clear that the method is very useful (although the memory improvements could still be useful). The improvement in time complexity is, in my opinion, not very relevant for this paper, since it is already mainly a heuristic (because of the kernel linearization with small $N$), so I feel that it should be evaluated based on its practical merits.

It is also not clear why ratios smaller than 1/8 were not chosen. It seems that it would have been helpful to look into this in order to understand better the limitations of the proposed method.

Minor comments:

In figure 1, why does a ratio of 2:1 consistently outperform softmax in perplexity, even with small number of states? Maybe a comment about this would be useful.

Also, the order in which the models are discussed in section 2 differs from section 3.2. (But this is not really important.)

Typos:
 - 44: "attention attention"
 - 114: "generate process"
 - 148: missing space between "." and "Algorithm"
 - 199: "to nonterminal to nonterminal"

Conclusion: I recommend rejection, mainly because of the omission of evidence for practical speedups, which to me looks like a major limitation.

----

UPDATE: The authors added some more evidence for practical speedups, as I originally requested. However, as another reviewer notices, these speedups are fairly modest, and as such they do not convince me that the proposed technique leads to significant practical speedups. I maintain my original score. Some of the technical limitations were discussed in the main review. Other than that, I think the paper is fine.

 It looks like I am the one high grade.

Normally, I like to see an advocate and a second and no serious objection.  After reading the other reviews, and the author rebuttal to my review, I'm not prepared to be the advocate.  I might be a second if there is an advocate, but that seems unlikely.

In my review, I raised a few concerns: (1) the parameterization and (2) poor practices in the community involving the Penn Treebank.  Both of these concerns could weaken the take-away message that I had hoped to see: a plausibility case for reviving traditional methods.

The author response didn't help me much with either concern.  As for (1), it was suggested that parameterization doesn't matter much, and as for (2) there was even more discussion of even more poor practices in the community.  While that may be true, this discussion would seem to further undermine the take away msg above.  If the parameterization doesn't matter, and if the experiments don't mean much, then maybe we can't leave with much of any take-away msg.  I'm sure that's not what the authors were trying to say, but this discussion doesn't give me much help if I wanted to push back on the other reviewers.  So, I'm inclined to go along with the other reviewers unless one of them decides to be an advocate.  I'd like to leave with the take-away msg above, but that may be too much to ask for.


 Thank you very much for the response to include the speed comparison, which I think is very important for the paper.

However, I feel the speed-up present here is not very large practically (mostly less than 2x). I like the idea in the paper and can see the value of this work, and it seems the proposed method would demonstrate its advantages more clearly with a large number of hidden states. Thus I think more convincing experiments would be when $L$ or $|\mathcal{N}|$ are so large that the vanilla model is not trainable (due to out-of-memory errors or too-slow training), yet the proposed linearized model is able to be trained and reflect the advantage of scaling. 

The HSMM experiment is a good attempt on this, but I would like to see similar setting on other tasks as well.

 Thank you for your thoughtful feedback, we provide responses below.

**Parameterizations of $\phi$**: We found in practice that the choice of the feature mapping $\phi$ did not have a large impact on the end performance, and mainly use the chosen set of $\phi$ to ensure positivity as required for a probability mass function. While there were some slight variations in the feature mappings across experiments, they were not integral to the results.

**Low-rank constraint vs approximation**: Our proposed method is not to approximate another kernel, but instead we define the kernel by defining the feature map $\phi$ explicitly.  Using a feature map $\phi$ with finite output dimensions (unlike the infinite implicit feature dimension of kernels like RBF) imposes a low-rank constraint to the scoring matrix $\Psi$ which, after using a low-rank factorization, allows us to apply the associative property of matrix-multiplication to turn a large matrix-vector product into two smaller ones. In practice, this finds quite different solutions than an unconstrained model (see the rank analysis in Appendix E for further details). We apologize for any confusion, and will clarify this point.

**Parameterization meaning**: By parameterization, we refer to the family of functions that define our hypothesis class in each experiment, e.g. the exact form of $\phi$. Roughly, the parameterization answers “how do we produce a scoring matrix $\Psi$”? Since a model can have different scoring matrices for different parts, e.g. the transition and emission matrices of an HMM, we introduce rank constraints for a subset of the parameters. This constraint limits expressivity in order to improve the speed of marginal inference and subsequently training as well.

**Papers with code results**: Many of the results on Penn Treebank in “papers with code” use external data (such as GPT- and BERT-based models). The best-performing models that do not use external data are variants of AWD-LSTM. However, even the baseline AWD-LSTM greatly outperforms traditional HMMs for language modeling in general [scalehmm]. As such, we only report the AWD-LSTM as a baseline.

**References**:

[scalehmm] Justin Chiu and Alexander Rush. 2020. Scaling hidden Markov language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1341–1349, Online. Association for Computational Linguistics.


 Thank you for your thoughtful feedback, we provide responses below.

**Speed for other models**: Thank you for your suggestion of including actual speed numbers for other models (in addition to the theoretic reduction of computational complexity)! We will add the below speed numbers to our next version.

PCFG Speed Numbers (measured during training):

\\begin{array} {|r|r|r|r|r|r|r|}\\hline
|\mathcal{N}| & |\mathcal{P}| & Model & N &  PPL & Batch/Second \\\\\\hline
30  & 60    & PCFG & - & 252.60 & 4.37 \\\\
    &       & LPCFG & 8 &  247.02    & 3.75     \\\\
    &       & LPCFG & 16 & 250.59    & 3.74   \\\\
\\hline
60  & 120   & PCFG & - & 234.01 & 2.99\\\\
    &       & LPCFG & 16& 217.24 & 3.55 \\\\
    &       & LPCFG & 32& 213.81 & 3.35\\\\
\\hline
100 & 200   & PCFG & - &  191.08   & 0.98 \\\\
    &       & LPCFG & 32& 203.47 & 1.56 \\\\
    &       & LPCFG & 64& 194.25 & 1.24 \\\\\hline
\\end{array}

LPCFG is faster than PCFG for the configuration $|\mathcal{N}|=60$ nonterminals, $|\mathcal{P}|=120$ preterminals and $|\mathcal{N}|=100$, $|\mathcal{P}|=200$. Note that LPCFG is faster even when the number of features $N>\frac{|\mathcal{N}|}{2}$ (such as $|\mathcal{N}|=60$ and $N=32$), because different from the HMM case, here the linearized kernel approach does not have an extra factor of 2 due to the score matrix not being square: we are essentially replacing the multiplication of a matrix of size $(T^2,|\mathcal{N}|^2)$ with a matrix of size $(|\mathcal{N}|^2, |\mathcal{N}|)$ (complexity $T^2|\mathcal{N}|^3$) by two separate (smaller) matrix multiplications:

1. the multiplication of a matrix of size $(T^2,|\mathcal{N}|^2)$ with a matrix of size $(|\mathcal{N}|^2, N)$ (complexity $T^2|\mathcal{N}|^2N$);
2. the multiplication of a matrix of size $(T^2,N)$ with a matrix of size $(N, |\mathcal{N}|)$ (complexity $T^2N|\mathcal{N}|$).

Note that here the first step dominates the complexity of the second step as $|\mathcal{N}|^2>>|\mathcal{N}|$, unlike the HMM case where these two steps have the same complexity due to having a square scoring matrix.

HSMM speed numbers (measured during training):

\\begin{array} {|r|r|r|r|r|r|r|}\\hline
Model & L & N & NLL & Batch/Second \\\\
\\hline
HSMM & 2^6 & - & 1.428\times 10^5 & 1.28 \\\\
HSMM & 2^7 & - & 1.427\times 10^5  & 0.45\\\\
HSMM & 2^8 & - & 1.426\times 10^5 & 0.13 \\\\
\\hline
LHSMM & 2^7 & 2^7 & 1.427\times 10^5 & 0.24 \\\\
LHSMM & 2^8 & 2^6 & 1.426\times 10^5 & 0.20 \\\\
LHSMM & 2^9 & 2^5 & 1.424\times 10^5 & 0.18 \\\\
LHSMM & 2^{10} & 2^4 & 1.423\times 10^5 & 0.10 \\\\
\\hline
\\end{array}

When the number of states $L=2^7$, the number of features $N=2^7$, LHSMM is slower than HSMM, which is not surprising given that $N=L$. However, when $L=2^8$, $N=2^7$, LHSMM is faster than HSMM ($\times1.53$). When $L\ge2^9$, HSMM would throw out-of-memory errors, but LHSMM can still train if we halve $N$ as we double $L$ (note that even LHSMM with $L=2^9$ and $N=2^5$ is faster than HSMM with $L=2^8$), and the best log-likelihood is achieved at $L=2^{10}$ and $N=2^4$.

**How $K$ (actually $\phi$) is chosen**: sorry for the confusion, we will clarify that the form of the feature map $\phi$ is chosen, but we learn its parameters, and $K$ is defined via $\phi$.

**State-feature ratios**: We provide the perplexity results of HMM experiments with larger state-feature ratios in Table 1, e.g. 1:16 and 1:32 at 16k states. We will also include the speed numbers for these models (843 and 626 secs/epoch respectively at 16k states). Additionally, in HSMM experiments the largest state-feature ratio is 64 at $L=2^{10}$ states (Table 2 right). Larger state-feature ratios are only viable at large state spaces, as extremely low rank transitions result in a large loss in model expressivity. 

**2:1 ratio outperforming softmax**: It is possible that the 2:1 ratio is a sweet spot for this particular model + dataset where the regularizing effects of the low-rank constraint outweigh the loss in flexibility due to a fewer number of parameters.

**Inconsistent presentation order between section 2 and section 3.2**: thanks for catching this! We will change the order to make them consistent in our next version.

 Thank you for your thoughtful feedback, we provide responses below.

**Speed for other models**: 

PCFG Speed Numbers (measured during training):

\\begin{array} {|r|r|r|r|r|r|r|}\\hline
|\mathcal{N}| & |\mathcal{P}| & Model & N &  PPL & Batch/Second \\\\\\hline
30  & 60    & PCFG & - & 252.60 & 4.37 \\\\
    &       & LPCFG & 8 &  247.02    & 3.75     \\\\
    &       & LPCFG & 16 & 250.59    & 3.74   \\\\
\\hline
60  & 120   & PCFG & - & 234.01 & 2.99\\\\
    &       & LPCFG & 16& 217.24 & 3.55 \\\\
    &       & LPCFG & 32& 213.81 & 3.35\\\\
\\hline
100 & 200   & PCFG & - &  191.08   & 0.98 \\\\
    &       & LPCFG & 32& 203.47 & 1.56 \\\\
    &       & LPCFG & 64& 194.25 & 1.24 \\\\\hline
\\end{array}

LPCFG is faster than PCFG for the configuration $|\mathcal{N}|=60$ nonterminals, $|\mathcal{P}|=120$ preterminals and $|\mathcal{N}|=100$, $|\mathcal{P}|=200$. Note that LPCFG is faster even when the number of features $N>\frac{|\mathcal{N}|}{2}$ (such as $|\mathcal{N}|=60$ and $N=32$), because different from the HMM case, here the linearized kernel approach does not have an extra factor of 2 due to the score matrix not being square: we are essentially replacing the multiplication of a matrix of size $(T^2,|\mathcal{N}|^2)$ with a matrix of size $(|\mathcal{N}|^2, |\mathcal{N}|)$ (complexity $T^2|\mathcal{N}|^3$) by two separate (smaller) matrix multiplications:

1. the multiplication of a matrix of size $(T^2,|\mathcal{N}|^2)$ with a matrix of size $(|\mathcal{N}|^2, N)$ (complexity $T^2|\mathcal{N}|^2N$);
2. the multiplication of a matrix of size $(T^2,N)$ with a matrix of size $(N, |\mathcal{N}|)$ (complexity $T^2N|\mathcal{N}|$).

Note that here the first step dominates the complexity of the second step as $|\mathcal{N}|^2>>|\mathcal{N}|$, unlike the HMM case where these two steps have the same complexity due to having a square scoring matrix.


HSMM speed numbers (measured during training):

\\begin{array} {|r|r|r|r|r|r|r|}\\hline
Model & L & N & NLL & Batch/Second \\\\
\\hline
HSMM & 2^6 & - & 1.428\times 10^5 & 1.28 \\\\
HSMM & 2^7 & - & 1.427\times 10^5  & 0.45\\\\
HSMM & 2^8 & - & 1.426\times 10^5 & 0.13 \\\\
\\hline
LHSMM & 2^7 & 2^7 & 1.427\times 10^5 & 0.24 \\\\
LHSMM & 2^8 & 2^6 & 1.426\times 10^5 & 0.20 \\\\
LHSMM & 2^9 & 2^5 & 1.424\times 10^5 & 0.18 \\\\
LHSMM & 2^{10} & 2^4 & 1.423\times 10^5 & 0.10 \\\\
\\hline
\\end{array}

When the number of states $L=2^7$, the number of features $N=2^7$, LHSMM is slower than HSMM, which is not surprising given that $N=L$. However, when $L=2^8$, $N=2^7$, LHSMM is faster than HSMM ($\times1.53$). When $L\ge2^9$, HSMM would throw out-of-memory errors, but LHSMM can still train if we halve $N$ as we double $L$ (note that even LHSMM with $L=2^9$ and $N=2^5$ is faster than HSMM with $L=2^8$), and the best log-likelihood is achieved at $L=2^{10}$ and $N=2^4$.

**LHMM+band speed**: Unfortunately, our current implementation of the LHMM+band is quite a bit slower than LHMM without the band, due to our unoptimized implementation of converting from banded to dense and vice versa. A practically faster implementation would require a customized CUDA kernel for banded matrix-vector multiplication, which we leave for future work.


 Thank you for your thoughtful feedback, we provide responses below.

**Neural parameterization**: The parameterization of the HMM follows prior state-of-the-art HMM works [scalehmm], and the parameterization of the PCFG follows prior work [cpcfg]. 

**Training details**: We train models by optimizing the exact log marginal likelihood with stochastic gradient ascent, obtained via dynamic programming, such as forward algorithm for HMMs and inside algorithm for PCFGs. This is possible as we focus on models where exact marginalization is tractable. We will clarify this in the manuscript.

**The linearization property of kernels (line 139)**: this is the definition of kernels (Eq. 3 in [kernel]), with references given in the manuscript on line 136 [kernel-orig].

**It is unclear why $\Psi$ is a square matrix**: $\Psi$ is not a square matrix. $\Psi = UV^\top$, where $U\in\mathbb{R}^{L\times N},V\in\mathbb{R}^{L^{|e|}\times N}$, so $\Psi$ is of size $L\times L^{|e|}$. Note that the score matrix $\Psi$ (generally not a square matrix) in this paper is not the same as the kernel matrix $K$ (a square matrix) in other kernel works whose $i, j$-th entry measures $K(x_i, x_j)$: the rows and the columns in the score matrix $\Psi$ are different (heads vs tails), and it doesn’t have to be positive semidefinite.

**Data with exponential decay for HMMs**: Thanks for the reference! This is a great idea.

**Kernel approximation of $\Psi$**: Our proposed method is not trying to approximate another kernel, but instead we are directly parameterizing the kernel $\Psi$ by parameterizing the feature maps $\phi$, so there is no approximation involved. We instead directly constrain the scoring matrix $\Psi$ to be low rank and train it from scratch, rather than approximate an existing high rank $\Psi$. This allows us to improve the speed of training.

**Why directly approximating each matrix as a low rank decomposition is not a good option**: The reason we did not directly parameterize the two lower-rank matrices $U$ and $V$ such that $\\Psi=UV^\top$ is because the kernel method we used is more parameter-efficient. The aproach we used generalizes low rank decompositions when the feature map $\phi$ is defined such that the mapping for heads are the rows in the corresponding left low-rank matrix, and the mapping for tails are the columns in the corresponding right row-rank matrix. This generalization allows us to be more parameter-efficient but still work in higher feature dimensions: for example, in the case of HMMs, we use embedding size 256 for each state, but the feature map can project those embeddings to up to a feature space of size $N=8192$: if we instead directly factorize $\\Psi\\in \\mathbb{R}^{L\\times L}$ ($L$ states) as $\\mathbb{R}^{L\\times N} \\times \\mathbb{R}^{N\\times L}$, it would take $2NL=2\\times 8192 \\times L$ parameters as opposed to $256\\times L$ plus $256\\times 8192$ (parameters of the feature map) parameters, which is much smaller when $L$ is large. This reduction in learnable parameters is why we use neural parameterizations in general.

**Hardware**: We give the hardware used in Appendix D, lines 634-636: for HMMs we run experiments on 8 Titan RTX GPUs with 24G of memory on an internal cluster. For PCFGs and HSMMs we run experiments on 1 Nvidia V100 GPU with 32G of memory on an internal cluster.

**References**

[linearattn] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2020.

[scalehmm] Justin Chiu and Alexander Rush. 2020. Scaling hidden Markov language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1341–1349, Online. Association for Computational Linguistics.

[cpcfg] Yoon Kim, Chris Dyer, and Alexander Rush. 2019. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2369–2385, Florence, Italy. Association for Computational Linguistics.

[kernel] Kernel Methods in Machine Learning. 2008. In The annals of statistics 36, no. 3: 1171-1220.

[kernel-orig] M. A. Aizerman, E. A. Braverman, and L. Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning.  In Automation and Remote Control, number 25, page 821–837, 1964.


This paper focuses on scaling unsupervised latent structure models such as HMM and PCFG, whose training/inference time complexity scales quadratically or cubically with the hidden state size. Specifically, the authors view the central inference step in the dynamic program as a matrix-vector product and apply a kernel linearization constraint, and as a result reduce the complexity of inference by a factor of the hidden state size. Experiments are conducted on language modeling, music modeling, and video modeling, covering HMM, PCFG, and HSMM models to evaluate the proposed technique. 
  I’ll detail the strengths, weaknesses, and questions below:

### Strengths


1. The problem studied in this paper is important. Reducing the quadratic or cubic complexity in latent structure models is critical for them to scale up to large datasets/models and potentially compete with typical neural models.


2. The proposed method is effective theoretically, reducing the complexity by a factor of hidden state size.


3. The experiments are extensive, evaluating three different models (HMM, PCFG, HSMM) on three different tasks. Experimental results indicate that the proposed technique is able to achieve speed-up while obtaining comparable PPL or NLL to the vanilla version.





### Weaknesses

1. I think that efficiency should be evaluated empirically as one of the main metrics in most settings in this paper (currently the only efficiency evaluation is Figure 1), given that improving efficiency/scaling is the main goal. While the speed-up is clear on the theoretical side, the practical benefits it brings (e.g. training and inference time) are related to the model size or data size. Therefore, it is important to demonstrate the total training/inference time in every setting/dataset and how it compares with the baselines, otherwise, it is difficult to judge whether the theoretical speed-up would really make a difference in practice. 


2. The proposed method limits the rank of the transition probability matrix to be a relatively small value, which clearly limits the capacity as acknowledged by the authors. It is good that the paper proposes possible solutions to mitigate this issue (Table 1), but the efficiency of the “LHMM + band” method should be reported empirically to understand its actual overhead. 
 Yes

HMMs used to be the method of choice in speech (and other apps) before nets took over.  This paper proposes a way to think about HMMs (and more) in a way that might make them more competitive again.  This perspective offers the hope that we can come up with a unified framework that includes HMMs and nets and more.  It would be very nice to have such a unified framework.

With a trigram model, for example, the number of states grows quickly.  The number of trigrams could be V^3 , where V is the size of the vocabulary, though actually, it is also limited by N_train, the size of the training set.  In practice, V^3 >> N_train, but even so, the number of states can easily become problematic.

This paper proposes a linearization so inference time is me O(TLN), where T is the length of the input, L is the size of the label space and N is the feature dimension.  There are obvious consequences for time here.  

It is less clear though exactly how the parameterization is done.  Appendix C suggests that there are different parameterizations for each task, which is somewhat disappointing.  One might expect that these parameterizations introduce certain approximations.  Is it possible to estimate bounds on these approximations?  If each task is approximated differently, maybe the proposed unified framework is not all that unified.  This paper is mainly a theoretical paper.  It is intended to provide an understanding of traditional methods such as HMMs and PCFGs in a way that is more comparable with modern methods.  This is an admirable goal.

The crux of the suggestion is referred to as parameterization, as illustrated in equations (5) and (6).   These equations are presented as facts, but I assume they are approximations.

Figure 1 suggests that the proposed parameterization has some advantages over HMMs in terms of speed; 8:1 state-to-rank ratio has comparable in performance (PPL) to HMMs, and considerable advantages in terms of speed.

Figure 2 is similar to Figure 1, but for another task. I would really like to understand better what is meant by parameterization.  Factoring?  Approximation?  Can we say more about the properties that the parameterization should have?

I really hate papers with code, but are there ways to compare the results in this paper with those results?  Superficially, these results seem to be far from SOTA.  Maybe that is beside the point, but if these results are not competitive, the paper should make that clear.

I assume the point of this paper is to give some insight as to why modern methods are better than traditional methods, and to suggest a direction to move in that could help understand this gap.  As such, the point of the experiments is more to show that the proposed parameterization improves traditional methods than to claim that they are better than the best alternatives.

One more flame... I really really hate the practice that has become all too standard of reporting perplexity on Penn Treebank, but actually replacing most of the content in the Penn Treebank with <unk>, as described in the appendix.    The Penn Treebank becomes unreadable after these replacements.  If we replaced all words with <unk>, PPL would be very impressive (but useless in any practical setting).  What does it mean to report perplexity when <unk> is one of the more common words in the vocabulary?  As a reviewer, I can't fault a paper for following precedent, but this precedent is not likely to advance the field.



Meta-review: This paper presents an an approach to speeding up marginalization in structured unsupervised models by using a low-rank approximation to corresponding matrix (HMM) and tensor multiplies (PCFG). Experiments are conducted on a variety of tasks. Reviewers are split -- two in favor and two opposed. Generally, reviewers appreciated the paper's clarity, viewed the underlying algorithmic idea positively, and saw the paper's goal of speeding up structured unsupervised models as important and well-motivated. The main concern shared by many reviewers relates to the empirical results and practicality of the proposed approach. While the proposed approach does reduce asymptotic bounds on computation, it also reduces model capacity. The question of whether the proposed approach offers a practical speedup (or reduction in memory consumption) at useful performance levels is an important one. The draft included very few experiments measuring runtime. Author response provided more complete runtime comparisons with baselines, but unfortunately showed very modest gains (and in some cases, none at all). Thus, taking these points in balance, I lean towards rejection. However, I do encourage authors to revise experiments and resubmit if more practical gains can be achieved.  

Decision: accept