IDGraphs: intrusion detection and analysis using histographs
Traffic anomalies and attacks are commonplace in today's networks and identifying them rapidly and accurately is critical for large network operators. For a statistical intrusion detection system (IDS), it is crucial to detect at the flow-level for accurate detection and mitigation. However, existing IDS systems offer only limited support for: 1) interactively examining detected intrusions and anomalies; 2) analyzing worm propagation patterns; 3) and discovering correlated attacks. These problems are becoming even more acute as the traffic on today's high-speed routers continues to grow. IDGraphs is an interactive visualization system for intrusion detection that addresses these challenges. The central visualization in the system is a flow-level trace plotted with time on the horizontal axis and aggregated number of unsuccessful connections on the vertical axis. We then summarize a stack of tens or hundreds of thousands of these traces using the histographs (Pin Ren and Watson, 2005) technique, which maps data frequency at each pixel to brightness. Users may then interactively query the summary view, performing analysis by highlighting subsets of the traces. For example, brushing a linked correlation matrix view highlights traces with similar patterns, revealing distributed attacks that are difficult to detect using standard statistical analysis. We apply IDGraphs system to a real network router data-set with 179M flow-level records representing a total traffic of 1.16TB. The system successfully detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN floodings, and some distributed attacks.

title: IDGraphs: intrusion detection and analysis using stream compositing
abstract: IDGraphs is an interactive visualization system, supporting intrusion detection over massive network traffic streams. It features a novel time-versus-failed-connections mapping that aids in discovery of attack patterns. The number of failed connections (SYN-SYN/ACK) is a strong indicator of suspicious network flows. IDGraphs offers several flow aggregation methods that help reveal different attack patterns. The system also offers high visual scalability through the use of Histographs. The IDGraphs intrusion detection system detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN flooding, and some distributed attacks. In this article, we demonstrate IDGraphs using a single day of NetFlow network traffic traces collected at edge routers at Northwestern University which has several OC-3 links.

title: Efficient interprocedural analysis for program parallelization and restructuring
abstract: An approach to efficient interprocedural analysis for program parallelization and restructuring is presented. Such analysis is needed for parallelizing loops which contain procedure calls. Our approach captures call effect on data dependencies by propagating the precise information of array subscripts from the called procedure. This allows the optimizing compiler to choose an efficient yet precise data dependence test scheme depending on the complexity of array reference patterns. The other existing methods do not provide such flexibility, hence may suffer from either imprecision or inefficiency. The paper also discusses usage of classical summary information in several important transformations for program parallelization. Experimental results are reported.

title: An iterative method for hypothetical reference decoder
abstract: In this paper, we propose a simple iterative method to verify whether a coded bitstream conforms to a hypothetical reference decoder (HRD). A concept of maximum tolerate delay (MTD) is introduced to study the possible low-delay operation such that we can bound the degree of incorrect motion rendition caused by the variation in end to end delay in the neighborhood of big pictures. Our method can be used in the design of a rate control algorithm to improve the possibility for the coded bitstream that conforms to the HRD.

title: Detection and repair of software errors in hierarchical sensor networks
abstract: Sensor networks are being increasingly deployed for collecting critical data in various applications. Once deployed, a sensor network may experience faults at the individual node level or at an aggregate network level due to design errors in the protocol, implementation errors, or deployment conditions that are significantly different from the target environment. In many applications, the deployed system may fail to collect data in an accurate, complete, and timely manner due to such errors. If the network produces incorrect data, the resulting decisions on the data may be incorrect, and negatively impact the application. Hence, it is important to detect and diagnose these faults through run-time observation. Existing technologies face difficulty with wireless sensor networks due to the large scale of the networks, the resource constraints of bandwidth and energy on the sensing nodes, and the unreliability of the observation channels for recording the behavior. This paper presents a semi-automatic approach named H-SEND (hierarchical sensor network debugging) to observe the health of a sensor network and to remotely repair errors by reprogramming through the wireless network. In H-SEND, a programmer specifies correctness properties of the protocol ("invariants"). These invariants are associated with conditions (the "observed variables") of individual nodes or the network. The compiler automatically inserts checking code to ensure that the observed variables satisfy the invariants. The checking can be done locally or remotely, depending on the nature of the invariant. In the latter case, messages are generated automatically. If an error is detected at run-time, the logs of the observed variables are examined to analyze and correct the error. After errors are corrected, new programs or patches can be uploaded to the nodes through the wireless network. We construct a prototype to demonstrate the benefit of run-time detection and correction

title: Mining block correlations to improve storage performance
abstract: Block correlations are common semantic patterns in storage systems. They can be exploited for improving the effectiveness of storage caching, prefetching, data layout, and disk scheduling. Unfortunately, information about block correlations is unavailable at the storage system level. Previous approaches for discovering file correlations in file systems do not scale well enough for discovering block correlations in storage systems.In this article, we propose two algorithms, C-Miner and C-Miner*, that use a data mining technique called frequent sequence mining to discover block correlations in storage systems. Both algorithms run reasonably fast with feasible space requirement, indicating that they are practical for dynamically inferring correlations in a storage system. C-Miner is a direct application of a frequent-sequence mining algorithm with a few modifications; compared with C-Miner, C-Miner* is redesigned for mining block correlations by making concessions for the specific problem of long sequences in storage system traces. Therefore, C-Miner* can discover 7--109% more correlation rules within 2--15 times shorter time than C-Miner. Moreover, we have also evaluated the benefits of block correlation-directed prefetching and data layout through experiments. Our results using real system workloads show that correlation-directed prefetching and data layout can reduce average I/O response time by 12--30% compared to the base case, and 7--25% compared to the commonly used sequential prefetching scheme for most workloads.

title: Data locality enhancement by memory reduction
abstract: In this paper, we propose memory reduction as a new approach to data locality enhancement. Under this approach, we use the compiler to reduce the size of the data repeatedly referenced in a collection of nested loops. Between their reuses, the data will more likely remain in higher-speed memory devices, such as the cache. Specifically, we present an optimal algorithm to combine loop shifting, loop fusion and array contraction to reduce the temporary array storage required to execute a collection of loops. When applied to 20 benchmark programs, our technique reduces the memory requirement, counting both the data and the code, by 51% on average. The transformed programs gain a speedup of 1.40 on average, due to the reduced footprint and, consequently, the improved data locality.