The chaos router chip: design and implementation of an adaptive router

title: Efficient video similarity measurement and search
abstract: We consider the use of meta-data and/or video-domain methods to detect similar videos on the Web. Meta-data is extracted from the textual and hyperlink information associated with each video clip. In the video domain, we apply an efficient similarity detection algorithm called video signature. The idea is to form a signature for each clip by selecting a small number of its frames that are most similar to a set of random seed images. We then apply a statistical pruning algorithm to allow fast detection on very large databases. Using a small ground-truth set, we achieve 90% recall and 95% precision using only 8% of the total number of operations required without pruning. For a database of around 46,000 video clips crawled from the Web, the video signature technique significantly outperforms meta-data in precision and recall. We show that even better performance can be achieved by combining them together. Based on our measurements, each video clip in our database has, on average, 1.53 similar copies.

title: Efficient video similarity measurement with video signature
abstract: The video signature method has previously been proposed as a technique to summarize video efficiently for visual similarity measurements (see Cheung, S.-C. and Zakhor, A., Proc. SPIE, vol.3964, p.34-6, 2000; ICIP2000, vol.1, p.85-9, 2000; ICIP2001, vol.1, p.649-52, 2001). We now develop the necessary theoretical framework to analyze this method. We define our target video similarity measure based on the fraction of similar clusters shared between two video sequences. This measure is too computationally complex to be deployed in database applications. By considering this measure geometrically on the image feature space, we find that it can be approximated by the volume of the intersection between Voronoi cells of similar clusters. In the video signature method, sampling is used to estimate this volume. By choosing an appropriate distribution to generate samples, and ranking the samples based upon their distances to the boundary between Voronoi cells, we demonstrate that our target measure can be well approximated by the video signature method. Experimental results on a large dataset of Web video and a set of MPEG-7 test sequences with artificially generated similar versions are used to demonstrate the retrieval performance of our proposed techniques.

title: Video similarity detection with video signature clustering
abstract: The proliferation of video content on the Web makes similarity detection an indispensable tool in Web data management, searching, and navigation. We have previously proposed a compact representation of video clips, called video signature, for retrieving similar video clips in large databases. In this paper, we propose a new signature clustering algorithm to further improve retrieval performance. The algorithm treats all the signatures as an abstract threshold graph, where the threshold is determined based on local data statistics. Similar clusters are identified as highly connected regions in the graph. This algorithm outperforms simple thresholding and hierarchical clustering techniques in identifying a set of manually-determined similar clusters from a dataset of 46,356 Web video clips. At 95% precision, our algorithm attains 85% recall while simple thresholding and complete-link hierarchical scheme attain 67% and 75% recall respectively. Applying our algorithm to the entire dataset, 6,900 similar clusters are identified, with an average cluster size of 2.81 video clips. The distribution of cluster sizes follows a power-law distribution, which has been shown to describe many Web phenomena.

title: Fast similarity search and clustering of video sequences on the world-wide-web
abstract: We define similar video content as video sequences with almost identical content but possibly compressed at different qualities, reformatted to different sizes and frame-rates, undergone minor editing in either spatial or temporal domain, or summarized into keyframe sequences. Building a search engine to identify such similar content in the World-Wide Web requires: 1) robust video similarity measurements; 2) fast similarity search techniques on large databases; and 3) intuitive organization of search results. In a previous paper, we proposed a randomized technique called the video signature (ViSig) method for video similarity measurement. In this paper, we focus on the remaining two issues by proposing a feature extraction scheme for fast similarity search, and a clustering algorithm for identification of similar clusters. Similar to many other content-based methods, the ViSig method uses high-dimensional feature vectors to represent video. To warrant a fast response time for similarity searches on high dimensional vectors, we propose a novel nonlinear feature extraction scheme on arbitrary metric spaces that combines the triangle inequality with the classical Principal Component Analysis (PCA). We show experimentally that the proposed technique outperforms PCA, Fastmap, Triangle-Inequality Pruning, and Haar wavelet on signature data. To further improve retrieval performance, and provide better organization of similarity search results, we introduce a new graph-theoretical clustering algorithm on large databases of signatures. This algorithm treats all signatures as an abstract threshold graph, where the distance threshold is determined based on local data statistics. Similar clusters are then identified as highly connected regions in the graph. By measuring the retrieval performance against a ground-truth set, we show that our proposed algorithm outperforms simple thresholding, single-link and complete-link hierarchical clustering techniques.

title: Efficient video similarity measurement with video signature
abstract: The proliferation of video content on the Web makes similarity detection an indispensable tool in Web data management, searching, and navigation. We propose a number of algorithms to efficiently measure video similarity. We define video as a set of frames, which are represented as high dimensional vectors in a feature space. Our goal is to measure ideal video similarity (IVS), defined as the percentage of clusters of similar frames shared between two video sequences. Since IVS is too complex to be deployed in large database applications, we approximate it with Voronoi video similarity (VVS), defined as the volume of the intersection between Voronoi cells of similar clusters. We propose a class of randomized algorithms to estimate VVS by first summarizing each video with a small set of its sampled frames, called the video signature (ViSig), and then calculating the distances between corresponding frames from the two ViSigs. By generating samples with a probability distribution that describes the video statistics, and ranking them based upon their likelihood of making an error in the estimation, we show analytically that ViSig can provide an unbiased estimate of IVS. Experimental results on a large dataset of Web video and a set of MPEG-7 test sequences with artificially generated similar versions are provided to demonstrate the retrieval performance of our proposed techniques.

title: Alert-driven e-service management
abstract: Process management technology has recently been employed not only within businesses but also in provision of e-services over the Internet. Urgent requests and critical messages in these systems (referred to as alerts) should be delivered and handled timely. In particular, humans are often involved in critical conditions. Presently, most e-service systems cannot address these requirements, and alerts are often handled in an ad-hoc manner. In this paper, we propose a sophisticated alert management system for effective e-service integration under urgent constraints. We develop a model for specifying alerts, in which alerts are associated with service requests and a set of parameters are captured for their routing. The alert monitor matches the service provider to receive an alert, based on the alert specification. We then propose a routing mechanism that is initiated when the alert message is not acknowledged or serviced within the deadline, so that the alert can be forwarded to other suitable services if necessary. Monitoring is especially essential to ensure timeliness and availability of services, otherwise suitable exceptions should be raised and handled. We outline our implementation framework of an alert management system (AMS) which includes Web services for B2B interactions, together with multiple-platform support for human users. We demonstrate our approach with an example medical house-call AMS.

title: Dependable Intrusion Tolerance: technology demo
abstract: The Dependable Intrusion Tolerance (DIT) architecture is a flexible, adaptive, and intrusion-tolerant server design. We briefly discuss its prototype implementation and validation, and demonstrate how it resists sample attacks.

title: Data-driven methodology to extending workflows to e-services over the Internet
abstract: E-services refer to the services offered over the Internet. The globalization of economy accelerates the provision of e-services across organizations. Instead of being built from scratch, e-services are mostly extended from existing internal workflows or information systems. In this paper, we examine the requirements of extending a workflow to the provision of e-services, in order to fulfill predefined business processes and data requirements. We also discuss the support of exception handling and asynchronous events across organizational boundaries, through an event publish-and-subscribe mechanism. We demonstrate the feasibility of this approach with the E-ADOME extension layer of our ADOME-WFMS, further illustrated with a system integration example. Our event-driven execution model provides a unified framework for both synchronous execution of workflow and asynchronous handling of events and exceptions. Illustrations are also given on the provision of such e-services over the infrastructure of Web services and Enterprise JavaBeans.

title: A Framework for Distributed Object-Oriented Testing
abstract: Distributed programming and object-oriented programming are two popular programming paradigms. The former is driven by advances in networking technology whereas the latter provides vigorous software principles needed in developing complex software systems. While more and more distributed object-oriented software has appeared, not much work exists on the testing of these systems in an integrated manner. Instead, the distributed and object features have been tested separately. In this paper, we propose an integrated framework known as DOOT for incremental testing of distributed object-oriented software systems. It combines various testing techniques to provide comprehensive test coverage at four levels — class testing, intra-cluster testing, inter-cluster testing and system testing. Each level uses a specific fault model, test strategy and test case generation that build on the previous test level to reduce the overall test effort. They are designed to handle the distinct requirements of the two paradigms at each level. Moreover, a reduction algorithm for testing the inherited class is also included in the framework. The approach is illustrated using a real life example of a conferencing system.

title: An Architecture for an Adaptive Intrusion-Tolerant Server
abstract: We describe a general architecture for intrusion-tolerant enterprise systems and the implementation of an intrusion-tolerant Web server as a specific instance. The architecture comprises functionally redundant COTS servers running on diverse operating systems and platforms, hardened intrusion-tolerance proxies that mediate client requests and verify the behavior of servers and other proxies, and monitoring and alert management components based on the EMERALD intrusion-detection framework. Integrity and availability are maintained by dynamically adapting the system configuration in response to intrusions or other faults. The dynamic configuration specifies the servers assigned to each client request, the agreement protocol used to validate server replies, and the resources spent on monitoring and detection. Alerts trigger increasingly strict regimes to ensure continued service, with graceful degradation of performance, even if some servers or proxies are compromised or faulty. The system returns to less stringent regimes as threats diminish. Servers and proxies can be isolated, repaired, and reinserted without interrupting service.