Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a multi-level abstraction for representing logs that appear in a wide range of application domains and a Transformer-based model for embedding those representations in a vector space. The authors show that a variety of log processing tasks (anomaly detection, predictive analysis, search, etc.) can be implemented on top of this foundation along with empirical results for some of them based on two real-world log datasets.

Strong points:
- The paper is well written. It nicely educates the reader about the prior state of the art, motivates how the contributions of the work fit in, and then presents those contributions in a systematic manner with empirical evidence where applicable.
- Log processing is a widespread application with increasingly large and diverse datasets and need for a range of automated data analysis tasks over them. The multi-level abstraction to provide a common representation for this domain as a whole is a neat idea that can help organize existing knowledge into a common framework as well as facilitating future innovation based on that framework.
- The paper shows how transformer networks can be extended with the notion of time, which can be useful for time-oriented data in general, including logs.
- The proposed models are implemented in practice as part of a transformer library in PyTorch and are applied to a variety of log processing tasks with good results.

Weak points:
- The levels of abstraction could have been connected more clearly with the transformer model and the log processing applications in the experimental part. The mapping between these parts is not as easy to understand as it could have been based on the current writing.
- It would be good to add a figure to Section 3 to illustrate the various abstraction levels and how they are related (e.g., in the form of a pipeline together with an example log entry representation for each).

Additional comments:
- Releasing the telecommunications dataset would be a good contribution to the research community.
- Section 4.1, "Given log sequence <(l_1, t_1), ..": Please make sure to use consistent terminology. "log sequence" is defined slightly differently in Section 3.
- Figure 4: In the caption, it looks like "Middle" and "Right" should be in the opposite order. Also, the colors in the third graph are not easily readable.
- Typos:
learnt -> learned
anomaies -> anomalies
useful applications -> useful (for?) applications
oh HDFS dataset -> on HDFS dataset
pytorch -> PyTorch
figure x -> Figure x

review_title: Log representation as an interface for log processing applications

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary**
Logs are widely used in computer systems to record their events. The recorded logs can be applied to a wide variety of diagnostic applications such as anomaly detection, root cause analysis, and causal analysis. This paper proposes levels of abstraction for log representation. There are in total 5 levels of abstraction, which are log sequences, parsed logs, field embeddings, log embeddings, and sequence embeddings. Each of the aforementioned levels can be derived from its proceeding levels. The paper uses the transformer model to generate a log representation. Moreover, the authors propose a time encoding method and add the encoding time to the transformer to improve the representation ability of the proposed log representation method. In the evaluation section, the authors apply the generated log representation to various downstream applications to test the effectiveness of the proposed method.

**Strengths**:
1. The idea of using different levels of abstraction to generate log embedding is interesting.
2. The paper conducts a thorough experiment over various downstream tasks based on the learned log representation.

**Weaknesses**:
1. The motivation for adding time encoding to the transformer is not clear. It would be better if the intuition behind this variation could be discussed rather than simply using the result of the ablation study.
2. It seems that the paper just applies the Transformer model to the log representation task and there is no comparison between the proposed method and other log representation methods in the evaluation part. In the related work section, the authors mentioned a recently proposed log embedding work (Logsy). It would be better if there is a comparison between the two different methods in the evaluation section.
3. There is little discussion about the results of the experiments. It would be better if some discussion of why the use of the proposed representation can or cannot improve the performance could be mentioned at the end of each experiment.

**Minor Weaknesses**:
The presentation of the paper could be improved. 
1. The figures for the experiment results are too far away from the section that describes and discusses the experiment.
2. The “oh” in the caption of figure 3 should be “on”.

review_title: Review for Log Representation as an Interface for Log Processing Applications

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper deals with log data representation and analysis. Logs are important to understand the status of a system and do various root cause analysis. This paper proposes a transformer based approach to obtain vector representation of log data, in various levels such as for key-value pair with a log entry, a log entry within a sequence of logs and various blocks of log sequences. The usefulness of log embeddings are shown on multiple log downstream tasks.

The topic of the paper is very interesting and relatively less studied in machine learning domain, though important for various applications. However, it has several drawbacks, as follows. 

1. The technical contribution of the paper is very limited. The transformer based approach is not novel. The specific format of time encoding in Section 4.1 is not well motivated.

2. In Section 3, is it reasonable to consider a log entry as a sequence of characters, instead of sequence of words? Most of the log entries consist of a set of key words (presented in a readable format) and some arguments (can be numeric).

3. One key difficulty in handling log data is the presence of large number of arguments (or parameters) within each log entry. Thus, it becomes difficult to mine logs to different templates. The exact parameter values can be very different in different logs. So, applying text modeling approaches directly on log data may not be the best approach.

4. In Section 3, What is the granularity used for sequence embedding in logs? Is it for a whole log file, or a block of logs? How to determine the appropriate granularity for some downstream application?

5. Fir unsupervised anomaly detection, it is not clear why anomalous points would fall into a small cluster. Rather, they would probably be distributed over multiple clusters, but still will be far away from the respective cluster center. The paper could have also used some standard anomaly detection algorithm such as Isolation Forest once the vector embeddings are generated.

6. For supervised anomaly detection, what is the % of anomalous logs used in the training?

7. Being an application-oriented paper, more importance should be given on hyperparameter setup and tuning for all the experiments. The lack of availability of source code also makes the reproducibility of the results difficult.
review_title: The paper addresses an interesting application, but lacks novelty in the approach

final_score: rating: 3: Clear rejection
review_content: This paper proposes to represent system logs at five levels of abstraction (including log sequence, parsed log, field embedding, log embedding, and sequence embedding). The representation at each level can be computed from the previous level. Transformer Network is utilized for time encoding. The paper also describes various log-related applications based on the proposed log representation. Some experiments were conducted to evaluate the proposed approach. 

Logs are useful for understanding and diagnosing software intensive systems. It is good to see that this paper proposes a new neural representation of log data. The authors also suggested various applications of the proposed log representation.

In section 4, the authors only described the proposed time encoding technique, while other parts of the log representation (such as encoding a log entry) were not described. Also, it is not clear if the proposed time encoding technique is better than the related methods (there are many related methods for encoding/representing time). 

The evaluation of the proposed approach is very weak. In section 5, the authors mentioned the Radio datasets. However, the use of Radio dataset is not described. The authors only evaluated the anomaly detection model on the HDFS log dataset, which is not enough. Also, the obtained results on HDFS were not very different from the results of the related work (DeepLog). Furthermore, no experiments were conducted for the causal analysis task. Therefore, the effectiveness and generalizability of the proposed approach are not clear. 

The paper only compared with a few related methods for log-related tasks. Actually, this area has been widely studied and there are a lot more research work (some also utilized deep learning and language models). The authors could discuss and compare with them. Just a few examples:
Zhang et al., Robust log-based anomaly detection on unstable log data. In Proc. ESEC/FSE 2019, 807-817.
Zhu et al., Learning to log: Helping developers make informed logging decisions, in Proc. ICSE 2015. pp. 415–425.
P. He et al., “Characterizing the natural language descriptions in software logging statements,”  in Proc. ASE 2018, pp. 178–189.

review_title: Log Representation as An Interface for Log Processing Applications

Meta-review: Most of the reviewers had concerns on the model being considered and there are additional concerns in the paper's discussion on the experimental results. 
Decision: Reject