Reverse Hashing for High-Speed Network Monitoring: Algorithms, Evaluation, and Applications
A key function for network traffic monitoring and analysis is the ability to perform aggregate queries over multiple data streams. Change detection is an important primitive which can be extended to construct many aggregate queries. The recently proposed sketches (Krishnamurthy, 2003) are among the very few that can detect heavy changes online for high speed links, and thus support various aggregate queries in both temporal and spatial domains. However, it does not preserve the keys (e.g., source IP address) of flows, making it difficult to reconstruct the desired set of anomalous keys. In an earlier abstract we proposed a framework for a reversible sketch data structure that offers hope for efficient extraction of keys (Schweller, 2004). However, this scheme is only able to detect a single heavy change key and places restrictions on the statistical properties of the key space. To address these challenges, we propose an efficient reverse hashing scheme to infer the keys of culprit flows from reversible sketches. There are two phases. The first operates online, recording the packet stream in a compact representation with negligible extra memory and few extra memory accesses. Our prototype single FPGA board implementation can achieve a throughput of over 16 Gbps for 40-byte-packet streams (the worst case). The second phase identifies heavy changes and their keys from the representation in nearly real time. We evaluate our scheme using traces from large edge routers with OC-12 or higher links. Both the analytical and experimental results show that we are able to achieve online traffic monitoring and accurate change/intrusion detection over massive data streams on high speed links, all in a manner that scales to large key space size. To the best of our knowledge, our system is the first to achieve these properties simultaneously.

title: IDGraphs: intrusion detection and analysis using histographs
abstract: Traffic anomalies and attacks are commonplace in today's networks and identifying them rapidly and accurately is critical for large network operators. For a statistical intrusion detection system (IDS), it is crucial to detect at the flow-level for accurate detection and mitigation. However, existing IDS systems offer only limited support for: 1) interactively examining detected intrusions and anomalies; 2) analyzing worm propagation patterns; 3) and discovering correlated attacks. These problems are becoming even more acute as the traffic on today's high-speed routers continues to grow. IDGraphs is an interactive visualization system for intrusion detection that addresses these challenges. The central visualization in the system is a flow-level trace plotted with time on the horizontal axis and aggregated number of unsuccessful connections on the vertical axis. We then summarize a stack of tens or hundreds of thousands of these traces using the histographs (Pin Ren and Watson, 2005) technique, which maps data frequency at each pixel to brightness. Users may then interactively query the summary view, performing analysis by highlighting subsets of the traces. For example, brushing a linked correlation matrix view highlights traces with similar patterns, revealing distributed attacks that are difficult to detect using standard statistical analysis. We apply IDGraphs system to a real network router data-set with 179M flow-level records representing a total traffic of 1.16TB. The system successfully detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN floodings, and some distributed attacks.

title: IDGraphs: intrusion detection and analysis using stream compositing
abstract: IDGraphs is an interactive visualization system, supporting intrusion detection over massive network traffic streams. It features a novel time-versus-failed-connections mapping that aids in discovery of attack patterns. The number of failed connections (SYN-SYN/ACK) is a strong indicator of suspicious network flows. IDGraphs offers several flow aggregation methods that help reveal different attack patterns. The system also offers high visual scalability through the use of Histographs. The IDGraphs intrusion detection system detects and analyzes a variety of attacks and anomalies, including port scanning, worm outbreaks, stealthy TCP SYN flooding, and some distributed attacks. In this article, we demonstrate IDGraphs using a single day of NetFlow network traffic traces collected at edge routers at Northwestern University which has several OC-3 links.

title: Detection and repair of software errors in hierarchical sensor networks
abstract: Sensor networks are being increasingly deployed for collecting critical data in various applications. Once deployed, a sensor network may experience faults at the individual node level or at an aggregate network level due to design errors in the protocol, implementation errors, or deployment conditions that are significantly different from the target environment. In many applications, the deployed system may fail to collect data in an accurate, complete, and timely manner due to such errors. If the network produces incorrect data, the resulting decisions on the data may be incorrect, and negatively impact the application. Hence, it is important to detect and diagnose these faults through run-time observation. Existing technologies face difficulty with wireless sensor networks due to the large scale of the networks, the resource constraints of bandwidth and energy on the sensing nodes, and the unreliability of the observation channels for recording the behavior. This paper presents a semi-automatic approach named H-SEND (hierarchical sensor network debugging) to observe the health of a sensor network and to remotely repair errors by reprogramming through the wireless network. In H-SEND, a programmer specifies correctness properties of the protocol ("invariants"). These invariants are associated with conditions (the "observed variables") of individual nodes or the network. The compiler automatically inserts checking code to ensure that the observed variables satisfy the invariants. The checking can be done locally or remotely, depending on the nature of the invariant. In the latter case, messages are generated automatically. If an error is detected at run-time, the logs of the observed variables are examined to analyze and correct the error. After errors are corrected, new programs or patches can be uploaded to the nodes through the wireless network. We construct a prototype to demonstrate the benefit of run-time detection and correction

title: Chromatic sums of 2-edge-connected maps on the plane
abstract: This paper provides the chromatic sum function equation of rooted 2-edge-connected maps on the plane. The enumerating function equations of rooted biloopless maps, rooted nonseparable maps, rooted 2-edge-connected bipartite maps and rooted loopless Eulerian maps on the plane are derived by the chromatic sum function equation of rooted 2-edge-connected maps on the plane. Moreover, some explicit expressions of enumerating functions are also derived.

title: WDEE: Web Data Extraction by Example
abstract: Web data extraction systems in use today transform semi-structured Web documents and deliver structured documents to end users. Some systems provide a visual interface to users to generate the extraction rules. However, to end users, the visual effect of Web documents is lost during the transformation process. In this paper, we propose an approach that allows a user to query extracted documents without knowledge of formal query language. We bridge the gap between visual effect of Web documents and structured documents extracted by providing a QBE-like (Query by Example) interface named Wdee. The principle component of our method is the notion of a document schema. Document schemata are patterns of structures embedded in documents. Wdee generates tree skeletons based on schema information and a user may execute queries by input condition in the skeltons. By maintaining the mapping relation among schemata of Web documents and extracted documents, a visual example may be presented to end users. With the example, Wdee allows a user to construct tree skeletons in a manner that resembles the browsing of Web pages.

title: An Efficient Sort-Based DDM Matching Algorithm for HLA Applications with a Large Spatial Environment
abstract: Simulation is a low cost and safe alternative to solve complex problems in various areas. To promote reuse and interoperability of simulation applications and link geographically dispersed simulation components, distributed simulation was introduced. The high level architecture (HLA) is the IEEE standard for distributed simulation. To optimize communication efficiency between simulation components, HLA defines a data distribution management (DDM) service group for filtering out unnecessary data exchange. It relies on the computation of overlap between update and subscription regions, which is called matching. In this paper, we propose an efficient sort-based DDM matching algorithm for HLA applications with a large spatial environment. A theoretical analysis of our algorithm concludes that it should have good storage and computational scalability. The experimental results have verified the theoretical conclusions by showing that our algorithm has much less storage requirement than the original sort-based matching algorithm and generally has the best computational performance when compared with region-based and the original sort-based matching algorithms.

title: Grasp analysis as linear matrix inequality problems
abstract: Three important problems in the study of grasping and manipulation by multi-fingered robotic hands are: 1) given a grasp characterised by a set of contact points and the associated contact models, determine if the grasp has force closure; 2) if the grasp does not have force closure, determine if the fingers are able to apply a specified resultant wrench on the object; and 3) compute "optimal" contact forces if the answer to problem (2) is affirmative. In this paper, based on an early result by Buss-Hashimoto-Moore (1996), which transforms the nonlinear friction cone constraints into positive definiteness of certain symmetric matrices, we further cast the friction cone constraints into linear matrix inequalities (LMIs) and formulate all three of the problems stated above as a set of convex optimization problems involving LMIs. We perform simulation studies to show the simplicity and efficiency of the LMI formulation to the three problems.