A Data Mining and CIDF Based Approach for Detecting Novel and Distributed Intrusions
As the recent distributed Denial-of-Service (DDOS) attacks on several major Internet sites have shown us, no open computer network is immune from intrusions. Furthermore, intrusion detection systems (IDSs) need to be updated timely whenever a novel intrusion surfaces; and geographically distributed IDSs need to cooperate to detect distributed and coordinated intrusions. In this paper, we describe an experimental system, based on the Common Intrusion Detection Framework (CIDF), where multiple IDSs can exchange attack information to detect distributed intrusions. The system also includes an ID model builder, where a data mining engine can receive audit data of a novel attack from an IDS, compute a new detection model, and then distribute it to other IDSs. We describe our experiences in implementing such system and the preliminary results of deploying the system in an experimental network.

title: Statistical Causality Analysis of INFOSEC Alert Data
abstract: With the increasingly widespread deployment of security mechanisms, such as firewalls, intrusion detection systems (IDSs), antiviras software and authentication services, the problem of alert analysis has become very important. The large amount of alerts can overwhelm security administrators and prevent them from adequately understanding and analyzing the security state of the network, and initiating appropriate response in a timely fashion. Recently, several approaches for alert correlation and attack scenario analysis have been proposed. However, these approaches all have limited capabilities in detecting new attack scenarios. In this paper, we study the problem of security alert correlation with an emphasis on attack scenario analysis. In our framework, we use clustering techniques to process low-level alert data into high-level aggregated alerts, and conduct causal analysis based on statistical tests to discover new relationships among attacks. Our statistical causality approach complements other approaches that use hard-coded prior knowledge for pattern matching. We perform a series of experiments to validate our method using DARPA’s Grand Challenge Problem (GCP) datasets, the 2000 DARPA Intrusion Detection Scenario datasets, and the DEF CON 9 datasets. The results show that our approach can discover new patterns of attack relationships when the alerts of attacks are statistically correlated.

title: HoneyStat: Local Worm Detection Using Honeypots
abstract: Worm detection systems have traditionally used global strategies and focused on scan rates. The noise associated with this approach requires statistical techniques and large data sets (e.g., 220 monitored machines) to yield timely alerts and avoid false positives. Worm detection techniques for smaller local networks have not been fully explored.

title: Worm detection, early warning and response based on local victim information
abstract: Worm detection systems have traditionally focused on global strategies. In the absence of a global worm detection system, we examine the effectiveness of local worm detection and response strategies. This paper makes three contributions: (1) we propose a simple two-phase local worm victim detection algorithm, DSC (Destination-Source Correlation), based on worm behavior in terms of both infection pattern and scanning pattern. DSC can detect zero-day scanning worms with a high detection rate and very low false positive rate. (2) We demonstrate the effectiveness of early worm warning based on local victim information. For example, warning occurs with 0.19% infection of all vulnerable hosts on Internet when using a /12 monitored network. (3) Based on local victim information, we investigate and evaluate the effectiveness of an automatic real-time local response in terms of slowing down the global Internet worms propagation. (2) and (3) are general results, not specific to certain detection algorithm like DSC. We demonstrate (2) and (3) with both analytical models and packet-level network simulator experiments.

title: A framework for constructing features and models for intrusion detection systems
abstract: Intrusion detection (ID) is an important component of infrastructure protection mechanisms. Intrusion detection systems (IDSs) need to be accurate, adaptive, and extensible. Given these requirements and the complexities of today's network environments, we need a more systematic and automated IDS development process rather that the pure knowledge encoding and engineering approaches. This article describes a novel framework, MADAM ID, for Mining Audit Data for Automated Models for Instrusion Detection. This framework uses data mining algorithms to compute activity patterns from system audit data and extracts predictive features from the patterns. It then applies machine learning algorithms to the audit records taht are processed according to the feature definitions to generate intrusion detection rules.  Evaluation showed that our ID model was one of the best performing of all the participating systems. We also briefly discuss our experience in converting the detection models produced by off-line data mining programs to real-time modules of existing IDSs.

title: Mining Audit Data to Build Intrusion Detection Models
abstract: In this paper we discuss a data mining framework for constructing intrusion detection models. The key ideas are to mine system audit data for consistent and useful patterns of program and user behavior, and use the set of relevant system features presented in the patterns to compute (inductively learned) classifiers that can recognize anomalies and known intrusions. Our past experiments showed that classifiers can be used to detect intrusions, provided that sufficient audit data is available for training and the right set of system features are selected. We propose to use the association rules and frequent episodes computed from audit data as the basis for guiding the audit data gathering and feature selection processes. We modify these two basic algorithms to use axis attribute(s) as a form of item constraints to compute only the relevant ("useful") patterns, and an iterative level-wise approximate mining procedure to uncover the low frequency (but important) patterns. We report our experiments in using these algorithms on real-world audit data.

title: Personalized face and speech communication over the Internet
abstract: We present our system for personalized face and speech communication over the Internet. The overall system consists of three parts: the cloning of real human faces to use as the representative avatars; the Networked Virtual Environment System performing the basic task of network and device management; and the speech system which includes a text-to-speech engine and a real time phoneme extraction engine from natural speech. The combination of these three elements provides a system to allow real humans, represented by their virtual counterparts, to communicate with each other even when they are geographically remote. In addition to this, all elements present use MPEG-4 as a common communication and animation standard and were designed and tested on the Windows operating system (OS). The paper presents the main aim of the work, the methodology and the resulting communication system.

title: Trees, windows and tiles for wavelet image compression
abstract: We investigate the task of compressing an image by using different probability models for compressing different regions of the image. In an earlier paper, we introduced a class of probability models for images, the k-rectangular tiling of an image, which is formed by partitioning the image into k rectangular regions and generating the coefficients within each region by using a probability model selected from a finite class C of probability models. We also describe a computationally efficient sequential probability assignment algorithm that is able to code an image with a code length that is close to the code length produced by the best model in the class. In this paper, we investigate the performance of the algorithm experimentally on the task of compressing wavelet subbands. We compare the method with compression methods that aim to compress as well as the best pruning of a quadtree and compression methods that exploit the local statistics in a window around the coefficient being compressed. For a class C consisting of a small number of Laplacian distributions and the uniform distribution, we find that the best tiling method works best, but the difference in performance is significant for only a few of the images tested.

title: Extracting key-substring-group features for text classification
abstract: In many text classification applications, it is appealing to take every document as a string of characters rather than a bag of words. Previous research studies in this area mostly focused on different variants of generative Markov chain models. Although discriminative machine learning methods like Support Vector Machine (SVM) have been quite successful in text classification with word features, it is neither effective nor efficient to apply them straightforwardly taking all substrings in the corpus as features. In this paper, we propose to partition all substrings into statistical equivalence groups, and then pick those groups which are important (in the statistical sense) as features (named key-substring-group features) for text classification. In particular, we propose a suffix tree based algorithm that can extract such features in linear time (with respect to the total number of characters in the corpus). Our experiments on English, Chinese and Greek datasets show that SVM with key-substring-group features can achieve outstanding performance for various text classification tasks.

title: A robust codec for transmission of very low bit-rate video over channels with bursty errors
abstract: We describe a robust codec for the transmission of very low bit-rate video over channels with a variety of errors, including random and bursty bit errors and packet loss. The codec exploits adaptivity to give good performance with a low overhead. By only protecting macroblocks which would otherwise be poorly concealed by the decoder the codec allows adaptive selection of the parts of video to protect. For protection, it uses multiple description codes which indirectly provide frequency-based adaptivity by protecting the more significant DCT coefficients. Simulations show significant improvements in the performance of the codec when compared to codecs which use intra macroblock updating (raster scan and random) at the same overhead. The codec is efficient in its use of bits and has good error resilience properties both objectively and subjectively over a wide range of conditions. Further, transcoding of the received bit stream to the standard H.263 syntax is relatively easy.

title: Exponential index: a parameterized distributed indexing scheme for data on air
abstract: Wireless data broadcast has received a lot of attention from industries and academia in recent years. Access efficiency and energy conservation are two critical performance concerns in a wireless data broadcast environment. To improve the efficiency of energy consumption on mobile devices, traditional disk-based indexing techniques such as B$^+$-tree have been extended to index broadcast data on a wireless channel. However, existing designs are mostly based on centralized tree structures. Most of these indexing techniques are not flexible in the sense that the trade-off between access efficiency and energy conservation is not adjustable based on application specific requirements. We propose in this paper a novel parameterized index, called the exponential index, which can be tuned to optimize the access latency with the tuning time bounded by a given limit, and vice versa. The proposed index is very efficient because it facilitates replication naturally by sharing links in multiple search trees and thus minimizes storage overhead. Experimental results show that the exponential index not only achieves better performance than the state-of-the-art indexes but also enables great flexibility in trade-offs between access latency and tuning time.