# Online Intrusion Scenario Discovery and Prediction Based on Hierarchical Temporal Memory (HTM)
## 1 Introduction  
Since the beginning of this century, cyberattacks have threatened users and organizations. They have become more complex and sophisticated with the development of computer networks. Nowadays, multistep attacks mostly aim at stealing important data from organizations and leads to serious data security problems. A multistep attack, also called a multistage attack, intrusion scenario, refers to the ensemble of steps taken by one or several attackers with a single specific objective inside the network, containing at least two distinct actions [1]. The multistep attack consists of multiple single-step attacks which can be detected by the IDS (intrusion detection system), however, the multistep attack is not easy to discover, having some difficulties:  
1. Although the IDS is efficient in detecting the single-step attack, it can hardly reveal the logical relations among these single-step attacks. 2. The real attack is a small probability event, and the most dangerous attacks happen only rarely, which mean that in each dataset we have only a few examples of attacks. This problem in intrusion detection research, called the \"rare data problem\" [2], seems even worse when multistep attacks are considered. IDS sensors could miss some steps of a multistep attack because these steps may appear normal to them [3]. Thus, we cannot assume that the important alerts related to the real attack will always be there. 4. High data redundancy and volumes will slow down the analysis process, which is a significant challenge for the development of systems working in real time. In this paper, an intrusion scenario discovery framework is proposed to tackle these difficulties by data preprocessing and online correlation analysis. The proposed framework was designed based on the following considerations:  
First, the framework is a totally online system and has no offline mining or learning part; it is different from other multistep attack detection systems [4; 5] that need offline processing. The HTM algorithm can learn the association patterns of intrusion actions in real time and the learned patterns will evolve with the incoming data over time, which is a good characteristic for online learning because it provides a continuous learning mechanism for our framework. However, the \"short-term memory\" has its negative influence on the prediction accuracy, so, an auxiliary method is proposed to mitigate the influence. Second, the framework is designed based on the processing of streaming data and does not limit the boundary or the quantity of data, which means the model should be constructed cumulatively. Third, the framework needs to predict the intrusion at an early stage without waiting for the model to be completely built. Given any intrusion action, the proposed framework will output the predicted action sequence that may happen with a higher probability in the future. This is the first time to our best knowledge that the HTM algorithm is combined with intrusion scenario discovery. HTM has some attractive features, such as strong noise resistance, continuous learning without retraining, and an unsupervised prediction-driven learning mechanism. The proposed framework can benefit from these features. In this section, four categories of existing approaches for constructing the intrusion scenarios are discussed and, after that, some HTM use cases are introduced. 1. Some early prediction models based on prior knowledge are created manually by security experts. The domain knowledge, such as known cyber threats or intrusion scenarios, are leveraged in the modeling process. These models can highly represent real intrusion scenarios, and provide a high accuracy of prediction with no ambiguity, which is the reason why they are popular in the industry. However, it is heavy work for the experts to keep the models always available for attack strategies that are changing all the time.