A DoS Resilient Flow-level Intrusion Detection Approach for High-speed Networks
Global-scale attacks like viruses and worms are increasing in frequency, severity and sophistication, making it critical to detect outbursts at routers/gateways instead of end hosts. In this paper we leverage data streaming techniques such as the reversible sketch to obtain HiFIND, a High-speed Flow-level Intrusion Detection system. In contrast to existing intrusion detection systems, HiFIND I ) is scalable to flow-level detection on high-speed networks; 2) zs DoS resilient; 3) can distinguish SYN flooding and various port scans (mostly for worm propagation) for effective mitigation; 4 ) enables aggregate detection over multiple routers/gateways; and 5) separates anomalies to limit false positives in detection. Both theoretical analysis and evaluation with several router traces show that HiFIND achieves these properties. To the best of our knowledge, HiFIND is the first online DoS resilient flow-level intrusion detection system for high-speed networks (approximately 10s of Gigabit/second), even for the worst case trafic of 40-byte-packet streams with each packet forming a flow.

title: IDGraphs: intrusion detection and analysis using histographs
abstract: Traffic anomalies and attacks are commonplace in today's networks and identifying them rapidly and accurately is critical for large network operators. For a statistical intrusion detection system (IDS), it is crucial to detect at the flow-level for accurate detection and mitigation. However, existing IDS systems offer only limited support for: 1) interactively examining detected intrusions and anomalies; 2) analyzing worm propagation patterns; 3) and discovering correlated attacks. These problems are becoming even more acute as the traffic on today's high-speed routers continues to grow. IDGraphs is an interactive visualization system for intrusion detection that addresses these challenges. The central visualization in the system is a flow-level trace plotted with time on the horizontal axis and aggregated number of unsuccessful connections on the vertical axis. We then summarize a stack of tens or hundreds of thousands of these traces using the histographs (Pin Ren and Watson, 2005) technique, which maps data frequency at each pixel to brightness. Users may then interactively query the summary view, performing analysis by highlighting subsets of the traces. For example, brushing a linked correlation matrix view highlights traces with similar patterns, revealing distributed attacks that are difficult to detect using standard statistical analysis. We apply IDGraphs system to a real network router data-set with 179M flow-level records representing a total traffic of 1.16TB. The system successfully detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN floodings, and some distributed attacks.

title: Reverse Hashing for High-Speed Network Monitoring: Algorithms, Evaluation, and Applications
abstract: A key function for network traffic monitoring and analysis is the ability to perform aggregate queries over multiple data streams. Change detection is an important primitive which can be extended to construct many aggregate queries. The recently proposed sketches (Krishnamurthy, 2003) are among the very few that can detect heavy changes online for high speed links, and thus support various aggregate queries in both temporal and spatial domains. However, it does not preserve the keys (e.g., source IP address) of flows, making it difficult to reconstruct the desired set of anomalous keys. In an earlier abstract we proposed a framework for a reversible sketch data structure that offers hope for efficient extraction of keys (Schweller, 2004). However, this scheme is only able to detect a single heavy change key and places restrictions on the statistical properties of the key space. To address these challenges, we propose an efficient reverse hashing scheme to infer the keys of culprit flows from reversible sketches. There are two phases. The first operates online, recording the packet stream in a compact representation with negligible extra memory and few extra memory accesses. Our prototype single FPGA board implementation can achieve a throughput of over 16 Gbps for 40-byte-packet streams (the worst case). The second phase identifies heavy changes and their keys from the representation in nearly real time. We evaluate our scheme using traces from large edge routers with OC-12 or higher links. Both the analytical and experimental results show that we are able to achieve online traffic monitoring and accurate change/intrusion detection over massive data streams on high speed links, all in a manner that scales to large key space size. To the best of our knowledge, our system is the first to achieve these properties simultaneously.

title: IDGraphs: intrusion detection and analysis using stream compositing
abstract: IDGraphs is an interactive visualization system, supporting intrusion detection over massive network traffic streams. It features a novel time-versus-failed-connections mapping that aids in discovery of attack patterns. The number of failed connections (SYN-SYN/ACK) is a strong indicator of suspicious network flows. IDGraphs offers several flow aggregation methods that help reveal different attack patterns. The system also offers high visual scalability through the use of Histographs. The IDGraphs intrusion detection system detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN flooding, and some distributed attacks. In this article, we demonstrate IDGraphs using a single day of NetFlow network traffic traces collected at edge routers at Northwestern University which has several OC-3 links.

title: Longest-path selection for delay test under process variation
abstract: Under manufacturing process variation, a path through a fault site is called longest for delay test if there exists a process condition under which the path has the maximum delay among all paths through that fault site. There are often multiple longest paths for each fault site in the circuit, due to different process conditions. To detect the smallest delay fault, it is necessary to test all longest paths through the fault site. However, previous methods are either inefficient or their results include too many paths that are not longest.This paper presents an efficient method to generate the longest path set for delay test under process variation. To capture both structural and systematic process correlation, we use linear delay functions to express path delays under process variation. A novel path-pruning technique is proposed to discard paths that are not longest, resulting in a significantly reduction in the number of paths compared with the previous best method. The new method can be applied to any process variation as long as its impact on delay is linear.

title: Detection and repair of software errors in hierarchical sensor networks
abstract: Sensor networks are being increasingly deployed for collecting critical data in various applications. Once deployed, a sensor network may experience faults at the individual node level or at an aggregate network level due to design errors in the protocol, implementation errors, or deployment conditions that are significantly different from the target environment. In many applications, the deployed system may fail to collect data in an accurate, complete, and timely manner due to such errors. If the network produces incorrect data, the resulting decisions on the data may be incorrect, and negatively impact the application. Hence, it is important to detect and diagnose these faults through run-time observation. Existing technologies face difficulty with wireless sensor networks due to the large scale of the networks, the resource constraints of bandwidth and energy on the sensing nodes, and the unreliability of the observation channels for recording the behavior. This paper presents a semi-automatic approach named H-SEND (hierarchical sensor network debugging) to observe the health of a sensor network and to remotely repair errors by reprogramming through the wireless network. In H-SEND, a programmer specifies correctness properties of the protocol ("invariants"). These invariants are associated with conditions (the "observed variables") of individual nodes or the network. The compiler automatically inserts checking code to ensure that the observed variables satisfy the invariants. The checking can be done locally or remotely, depending on the nature of the invariant. In the latter case, messages are generated automatically. If an error is detected at run-time, the logs of the observed variables are examined to analyze and correct the error. After errors are corrected, new programs or patches can be uploaded to the nodes through the wireless network. We construct a prototype to demonstrate the benefit of run-time detection and correction

title: An Efficient Sort-Based DDM Matching Algorithm for HLA Applications with a Large Spatial Environment
abstract: Simulation is a low cost and safe alternative to solve complex problems in various areas. To promote reuse and interoperability of simulation applications and link geographically dispersed simulation components, distributed simulation was introduced. The high level architecture (HLA) is the IEEE standard for distributed simulation. To optimize communication efficiency between simulation components, HLA defines a data distribution management (DDM) service group for filtering out unnecessary data exchange. It relies on the computation of overlap between update and subscription regions, which is called matching. In this paper, we propose an efficient sort-based DDM matching algorithm for HLA applications with a large spatial environment. A theoretical analysis of our algorithm concludes that it should have good storage and computational scalability. The experimental results have verified the theoretical conclusions by showing that our algorithm has much less storage requirement than the original sort-based matching algorithm and generally has the best computational performance when compared with region-based and the original sort-based matching algorithms.

title: Improving Data Locality by Array Contraction
abstract: Array contraction is a program transformation which reduces array size while preserving the correct output. In this paper, we present an aggressive array-contraction technique and study its impact on memory system performance. This technique, called controlled SFC, combines loop shifting and controlled loop fusion to maximize opportunities for array contraction within a given loop nesting. A controlled fusion scheme is used to prevent overfusing loops and to avoid excessive pressure on the cache and the registers. Reducing the array size increases data reuse because of the increased average number of memory operations on the same memory addresses. Furthermore, if the data size of a loop nest fits in the cache after array contraction, then repeated references to the same variable in the loop nest generate cache hits, assuming set conflicts are eliminated successfully.

title: On increasing end-to-end throughput in wireless ad hoc networks
abstract: One of the main characteristics of wireless ad hoc networks is their node-centric broadcast nature of communication, leading to interferences and spatial contention between adjacent wireless links. Due to such interferences, pessimistic concerns have been recently raised with respect to the decreasing network capacity in wireless ad hoc networks when the number of nodes scales to several orders of magnitude higher. In this paper, we argue that in all cases of end-to-end data communications - including one-to-k unicast and multicast data dissemination as well as k-to-one data aggregation - the maximum achievable end-to-end data throughput (measured on the sources) heavily depends on the strategy of arranging the topology of transmission between sources and destinations, as well as possible per-node operations such as coding. An optimal strategy achieves better end-to-end throughput than an arbitrary one. We present theoretical studies and critical insights with respect to how these strategies may be designed so that end-to-end throughput may be increased