Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This work on symbolisation and word embedding on symbolised sequences shares similar principles with the recently introduced work on PixelRNNs (quantization of the continuous signal, classification instead of regression tasks). Could you elaborate on similarities and differences between your method and PixelRNN?
review_title: References and comparison to PixelRNN

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper compare three representation learning algorithms over symbolized sequences. Experiments are executed on several prediction tasks. The approach is potentially very important but the proposed algorithm is rather trivial. Besides detailed analysis on hyper parameters are not described. 

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: In absence of authors' responses, the rating is maintained.

---

This paper introduces an approach for learning predictive time series models that can handle heterogenous multivariate sequence. The first step is in three possible ways to perform embedding of the d-dimensional sequences into d-character words, or a sum of d character embeddings, or a concatenation of d character embeddings. The embedding layer is the first layer of a deep architecture such as LSTM. The models are then trained to perform event prediction at a fixed horizon, with temporal weighting, and applied to hard disk or heating system failures or seizures.

The approach is interesting and the results seem to outperform an LSTM baseline, but need additional clarification.

The experimental section on seizure prediction is very short and would need to be considerably extended, in an appendix. What are the results obtained using LSTM vs. RNN? What is the state-of-the-art on that dataset? Given that EEG data contain mostly frequential information, how is this properly handled in per-sample embeddings?

Please also extend your reference and previous work section to include PixelRNN as well as: 
* van den Oord, et al. (2016)
WaveNet: A Generative Model for Raw Audio
arXiv 1609.03499
* Huang et al. (2013)
"Learning deep structured semantic models for web search using clickthrough data"
CIKM
In the latter paper, the authors embedded 3-gram hashes of the input sequences (e.g., text), which is somewhat similar to a time-delay embedding of the input sequence.

review_title: Interesting approach for sequence quantization and embedding

final_score: rating: null
review_content: Can you comment on the following:
- did you investigate how sensitive model architecture (# hidden layers, # units in each layer, etc.) was to choice of partition size? If your claim is "not very sensitive," can you offer empirical evidence to back this up?
- for your non-symbolic LSTM baselines, how did you preprocess the continuous channels? Did you do any kind of normalization or rescaling? If not, could the inferior performance of the non-symbolic LSTM be explained by poor conditioning of the inputs?
- did you try using an embedding layer for the raw continuous values, rather than feeding them directly into the LSTM?
review_title: Is the ad hoc symbolization step necessary?

final_score: rating: null
review_content: I guess the alphabet size is a magic number. How does the performance change over different alphabet sizes? Do you pick 4 or 50 which maximizes the performance? 
review_title: Alphabet size

final_score: rating: 3: Clear rejection
review_content: Because the authors provided no further responses to reviewer feedback, I maintained my original review score.

-----

This paper takes a unique approach to the modeling of heterogeneous sequence data. They first symbolize continuous inputs using a previously described approach (histograms or maximum entropy), the result being a multichannel discrete sequence (of symbolized time series or originally categorical data) of "characters." They then investigate three different approaches to learning an embedding of the characters at each time step (which can be thought of as a "word"):
1) Concatenate characters into a "word" and then apply standard lookup-based embeddings from language modeling (WDE)
2) Embed each character independently and then sum over the embeddings (SCE)
3) Embed each character as a scalar and concatenate the scalar embeddings (ICE)
The resulting embeddings can be used as inputs to any architecture, e.g., LSTM. The paper applies these methods primarily to event detection tasks, such as hard drive failures and seizures in EEG data. Empirical results largely suggest the a recurrent model combined with symbolization/embedding outperforms a comparable recurrent model applied to raw data. Results are inconclusive as to which embedding layer works best.

Strengths:
- The different embedding approaches, while simple, are designed to tackle a very interesting problem where the input consists of multivariate discrete sequences, which makes it different from standard language modeling and related domains. The proposed approaches offer several different interesting perspectives on how to approach this problem.
- The empirical results suggest that symbolizing the continuous input space can improve results for some problems. This is an interesting possibility as it enables the direct application of a variety of language modeling tools (e.g., embeddings).

Weaknesses:
- The LSTMs (one layer each of 8, 16, and 15 cells, respectively) used in the three experiments sound *very* under capacity given the complexity of the tasks and the sizes of the data sets (tens to hundreds of thousands of sequences). That might explain both the relatively small gap between the LSTMs and logistic regression *and* the improvement of the embedding-based LSTMs. Hypothetically, if quantizing the inputs is really useful, the raw data LSTMs should be able to learn this transformation, but if they are under capacity, they might not be able to dos. What is more, using the same architecture (# layers, # units, etc.) for very different kinds of inputs (raw, WdE, SCE, ICE, hand-engineered features) is poor methodology. Obviously, hyperparameters should be tuned independently for each type of input.
- The experiments omit obvious baselines, such as trying to directly learn an embedding of the continuous inputs.
- The experimental results offer an incomplete, mixed conclusion. First, no one embedding approach performs best across all tasks and metrics, and the authors offer no insights into why this might be. Second, the current set of experiments are not sufficiently thorough to conclude that quantization and embedding is superior to working with the raw data.
- The temporal weighting section appears out of place: it is unrelated to the core of the paper (quantizing and embedding continuous inputs), and there are no experiments to demonstrate its impact on performance.
- The paper omits a large number of related works: anything by Eamonn Keogh's lab (e.g., Symbolic Aggregate approXimation or SAX), work on modifying loss functions for RNN classifiers (Dai and Le. Semi-supervised sequence learning. NIPS 2015; Lipton and Kale, et al. Learning to Diagnose with LSTM Recurrent Neural Networks. ICLR 2016), work on embedding non-traditional discrete sequential inputs (Choi, et al. Multi-layer Representation Learning for Medical Concepts. KDD 2016).

This is an interesting direction for research on time series modeling with neural nets, and the current work is a good first step. The authors need to perform more thorough experiments to test their hypotheses (i.e., that embedding helps performance). My intuition is that a continuous embedding layer + proper hyperparameter tuning will work just as well. If quantization proves to be beneficial, then I encourage them to pursue some direction that eliminates the need for ad hoc quantization, perhaps some kind of differentiable clustering layer?
review_title: Unique angle for modeling heterogeneous sequences

Meta-review: The main strengths and weaknesses pointed out by the reviewers were:
 
 Strengths
 -Domain is interesting, problem is important (R2, R1)
 -Discretization of continuous domain may enable leveraging of advanced tools developed for discrete domains, e.g. NLP (R1)
 
 Weaknesses
 -Issues with experiments: models under-capacity, omission of obvious baselines (R1, R3)
 -Unclear conclusion: is quantization and embedding superior to working with the raw data? (R1)
 -Fair amount of relevant work omitted (R1, R3)
 
 While the authors engaged in the pre-review discussion, they did not respond to the official reviews. Therefore I have decided to align with the reviewers who are in consensus that the paper does not meet the acceptance bar.
Decision: Reject