nils-herrmann/clean_scirepeval huggingface dataset:
FGS+: optimizing the joint SNR-temporal video quality in MPEG-4 fine grained scalable coding

To enable video transmission over heterogeneous wireless networks, a highly scalable compression and streaming framework that can adapt to large and rapid bandwidth variations in real-time is necessary. MPEG-4 fine grained scalability (FGS) provides fine-grained SNR and temporal scalabilities, but these scalabilities are implemented and performed independently, thereby neglecting the gains that can be made from making joint SNR-temporal decisions to maximize quality. A novel fine grained SNR-temporal scalability framework called FGS+ that provides a new level of performance by considering SNR and temporal scalability jointly is presented. This new solution uses the results of our subjective tests, which indicate the levels to which SNR-quality needs to be enhanced before motion-smoothness should be improved. The study also reveals that these SNR-temporal tradeoff points vary among videos, and depend on the characteristics of the video. Based on these observations, our solution uses reference frames, enhanced relative to FGS, for prediction, improving visual quality over MPEG-4 FGS by up to 1.5 dB.
Generating semantic visual templates for video databases

We describe a system that generates semantic visual templates (SVTs) for video databases. From a single query sketch, new queries are automatically generated with each one representing a different view of the initial sketch. The combination of the original and new queries forms a large set of potential queries for a content-based video retrieval system. Through Bayesian relevance feedback, the user narrows the choices to an exemplar set. This exemplar set, or SVTs, represents personalized views of a concept and an effective set of queries to retrieve a general category of images and videos. We have generated SVTs for several classes of videos, including sunsets, high jumpers, and slalom skiers. Our experiments show that the user can quickly converge upon SVTs with optimal performance, achieving over 85% of the precision from icons chosen by exhaustive search.
Sample-Specific Late Fusion for Visual Category Recognition

Late fusion addresses the problem of combining the prediction scores of multiple classifiers, in which each score is predicted by a classifier trained with a specific feature. However, the existing methods generally use a fixed fusion weight for all the scores of a classifier, and thus fail to optimally determine the fusion weight for the individual samples. In this paper, we propose a sample-specific late fusion method to address this issue. Specifically, we cast the problem into an information propagation process which propagates the fusion weights learned on the labeled samples to individual unlabeled samples, while enforcing that positive samples have higher fusion scores than negative samples. In this process, we identify the optimal fusion weights for each sample and push positive samples to top positions in the fusion score rank list. We formulate our problem as a L∞ norm constrained optimization problem and apply the Alternating Direction Method of Multipliers for the optimization. Extensive experiment results on various visual categorization tasks show that the proposed method consistently and significantly beats the state-of-the-art late fusion methods. To the best knowledge, this is the first method supporting sample-specific fusion weight learning.
Muti-scale temporal segmentation and outlier detection in sensor networks

Monitoring multimodal data generated by sensor networks for extracting information is a challenging task for the human observer. To manage the barrage of data, one needs to create mechanisms for identifying only those time intervals which are informative and worthy of further highlevel analysis either by machine or the human observer. We regard a time interval to be informative and contain an event if it is uncommon or distinct from routine background. Different events in general may unfold at different temporal scales. Here, we present a non-parametric distribution based approach for event detection in sensor network data. In this approach we employ multiple sliding windows at different scales to obtain the distribution of the data. We segment the temporal data stream and identify the potential event bearing candidates by comparing the present and past statistical behavior of the data. In the experiments we demonstrate the effect of optimum bin width selection on accuracy and the range of allowable window sizes and therefore time scales. We analyze the computational speed as well as the supporting empirical results on the bin width.
Structure analysis of sports video using domain models

In this paper, we present an effective framework for scene detection and structure analysis for sports videos, using tennis and baseball as examples. Sports video can be characterized by its predictable temporal syntax, recurrent events with consistent features, and a fixed number of views. Our approach combines domain-specific knowledge, supervised machine learning techniques, and automatic feature analysis at multiple levels. Real time processing performance is achieved by utilizing compressed-domain processing techniques. High accuracy in view recognition is achieved by using compressed-domain global features as prefilters and object-level refined analysis in the latter verification stage. Applications include high-level structure browsing/navigation, highlight generation, and mobile media filtering.
A distributed system for editing and browsing compressed video over the network

We present a new framework for distributed video editing and browsing over the network. This framework uses a distributed client-server model including a server engine for content analysis/editing, and clients for interactive controls of video browsing/editing. It utilizes several unique features, including compressed-domain video manipulation, multi-resolution video access, content based video browsing/retrieval, and a distributed network architecture. We have developed a complete functional prototype, CVEPS, which includes a compressed domain video analysis and editing engine, and a JAVA based user interface. It has been incorporated into a World Wide Web application, WebClip, for editing compressed video over the Web.
An Introduction to the Special Issue on Event Analysis in Videos

I NTEREST from industry and academia has increased dramatically over recent years in the challenging area of event analysis and recognition from various video sources including sports, surveillance, user-generated video, etc. Video event analysis and recognition is a critical task in many applications such as detection of sporting highlights, incident detection in surveillance video, indexing, retrieval and summarization of video databases, and human-computer interaction. This special issue aims to capture the latest advances by the research community working in the area of video event analysis. The call for papers was enthusiastically greeted by the research community and we received over seventy submissions. The special issue presents 16 articles which provide fundamental contributions in a wide range of topics in video event analysis: 1) human action and activity recognition; 2) motion trajectory analysis; 3) video content analysis and pattern mining; 4) audio-visual multi-modal analysis; and 5) video analysis applications. An overview of the organization and a brief summary of the articles selected for publication in the special issue are provided below.
Watermarking capacity of digital images based on domain-specific masking effects

Our objective is to find a theoretical watermarking capacity bound of digital images based on domain-specific masking effects. We first show the capacity of private watermarking in which the power constraints are not uniform. Then, we apply several domain-specific Human Vision System approximation models to estimate the power constraints and then show the theoretical watermarking capacity of an image in a general noisy environment. Note that we consider all pixels, watermarks and noises to be discrete values, which occur in realistic cases.
Query-Adaptive Image Search With Hash Codes

Scalable image search based on visual similarity has been an active topic of research in recent years. State-of-the-art solutions often use hashing methods to embed high-dimensional image features into Hamming space, where search can be performed in real-time based on Hamming distance of compact hash codes. Unlike traditional metrics (e.g., Euclidean) that offer continuous distances, the Hamming distances are discrete integer values. As a consequence, there are often a large number of images sharing equal Hamming distances to a query, which largely hurts search results where fine-grained ranking is very important. This paper introduces an approach that enables query-adaptive ranking of the returned images with equal Hamming distances to the queries. This is achieved by firstly offline learning bitwise weights of the hash codes for a diverse set of predefined semantic concept classes. We formulate the weight learning process as a quadratic programming problem that minimizes intra-class distance while preserving inter-class relationship captured by original raw image features. Query-adaptive weights are then computed online by evaluating the proximity between a query and the semantic concept classes. With the query-adaptive bitwise weights, returned images can be easily ordered by weighted Hamming distance at a finer-grained hash code level rather than the original Hamming distance level. Experiments on a Flickr image dataset show clear improvements from our proposed approach.
A robust content based digital signature for image authentication

A methodology for designing content based digital signatures which can be used to authenticate images is presented. A continuous measure of authenticity is presented which forms the basis of this methodology. Using this methodology signature systems can be designed which allow certain types of image modification (e.g. lossy compression) but which prevent other types of manipulation. Some experience with content based signatures is also presented. The idea of signature based authentication is extended to video, and a system to generate signatures for video sequences is presented. This signature also allows smaller segments of the secured video to be verified as unmanipulated.
Uploader Intent for Online Video: Typology, Inference, and Applications

We investigate automatic inference of uploader intent for online video, i.e., prediction of the reason for which a user has uploaded a particular video to the Internet. Users upload video for specific reasons, but rarely state these reasons explicitly in the video metadata. Information about the reasons motivating uploaders has the potential ultimately to benefit a wide range of application areas, including video production, video-based advertising , and video search. In this paper, we apply a combination of social-Web mining and crowdsourcing to arrive at a typology that characterizes the uploader intent of a broad range of videos. We then use a set of multimodal features, including visual semantic features, found to be indicative of uploader intent in order to classify videos automatically into uploader intent classes. We evaluate our approach on a dataset containing ca. 3K crowdsourcing-annotated videos and demonstrate its usefulness in prediction tasks relevant to common application areas.
In a Blink of an Eye and a Switch of a Transistor: Cortically Coupled Computer Vision

Our society's information technology advancements have resulted in the increasingly problematic issue of information overload-i.e., we have more access to information than we can possibly process. This is nowhere more apparent than in the volume of imagery and video that we can access on a daily basis-for the general public, availability of YouTube video and Google Images, or for the image analysis professional tasked with searching security video or satellite reconnaissance. Which images to look at and how to ensure we see the images that are of most interest to us, begs the question of whether there are smart ways to triage this volume of imagery. Over the past decade, computer vision research has focused on the issue of ranking and indexing imagery. However, computer vision is limited in its ability to identify interesting imagery, particularly as ¿interesting¿ might be defined by an individual. In this paper we describe our efforts in developing brain-computer interfaces (BCIs) which synergistically integrate computer vision and human vision so as to construct a system for image triage. Our approach exploits machine learning for real-time decoding of brain signals which are recorded noninvasively via electroencephalography (EEG). The signals we decode are specific for events related to imagery attracting a user's attention. We describe two architectures we have developed for this type of cortically coupled computer vision and discuss potential applications and challenges for the future.
Hierarchical global motion estimation/compensation in low bitrate video coding

In this paper, we develop a hierarchical global motion estimation/compensation system which is used to improve the block-based motion compensation in the current low bitrate video coding standard H.263. The system first detects the background motion by examining the local motion information using hierarchical block matching and global motion estimation algorithm. If the background motion exists, the global motion compensation (GMC) is applied to the image before the H.263 block-based coding. Experiments are done under the typical low bitrate video conditions: low resolution and low frame rate. The results show that prediction errors on both motion vectors and texture are reduced. For sequences containing lots of background motion, the proposed system shows savings of up to 30% in total bitrate.
A Framework for Sub-Window Shot Detection

Browsing a digital video library can be very tedious especially with an ever expanding collection of multimedia material. We present a novel framework for extracting sub-window shots from MPEG encoded news video with the expectation that this will be another tool that can be used by retrieval systems. Sub-windows shots are also useful for tying in relevant material from multiple video sources. The system makes use of Macroblock parameters to extract visual features, which are then combined to identify possible sub-windows in individual frames. The identified sub-widows are then filtered by a non-linear Spatial-Temporal filter to produce sub-window shots. By working only on compressed domain information, this system avoids full frame decoding of MPEG sequences and hence achieves high speeds of up to 11 times real time.
A robust and secure media signature scheme for JPEG images

We have introduced a robust and secure digital signature solution for multimedia content authentication, by integrating content feature extraction, error correction coding (ECC), watermarking, and cryptographic hashing into a unified framework. We have successfully applied it to JPEG2000 as well as generic wavelet transform based applications. In this paper, we shall introduce a new JPEG-compliant solution under our proposed framework but with different ECC and watermarking methods. System security analysis as well as system robustness evaluation will also be given to further demonstrate the practicability of our method.
Joint adaptive space and frequency basis selection

We develop a new method for building a representation of an image from a library of basis elements that is facilitated by a joint adaptive space and frequency (JASF) graph. The JASF graph combines partitionable frequency expansion and spatial segmentation of the image, symmetrically. We demonstrate by using a rate-distortion framework for basis selection that the JASF graph improves compression performance over recent wavelet packet and double-tree methods by offering exponentially more bases in which to represent the images.
Model-Driven Feedforward Prediction for Manipulation of Deformable Objects

Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high-dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a precomputed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and efficient retrieval. To validate this approach, we developed a comprehensive pipeline for manipulating clothing as in a typical laundry task. First, the database is used for category and the pose estimation is used for a garment in an arbitrary position. A fully featured 3-D model of the garment is constructed in real time, and volumetric features are then used to obtain the most similar model in the database to predict the object category and pose. Second, the database can significantly benefit the manipulation of deformable objects via nonrigid registration, providing accurate correspondences between the reconstructed object model and the database models. Third, the accurate model simulation can also be used to optimize the trajectories for the manipulation of deformable objects, such as the folding of garments. Extensive experimental results are shown for the above tasks using a variety of different clothings. Note to Practitioners—This paper provides an open source, extensible, 3-D database for dissemination to the robotics and graphics communities. Model-driven methods are proliferating, and they need to be applied, tested, and validated in real environments. A key idea we have exploited is to have an innovative and novel use of simulation. This database will serve as infrastructure for developing advanced robotic machine learning algorithms. We want to address this machine learning idea ourselves, but we expect the dissemination of the database to other researchers with different agendas and task applications, which will bring wide progress in this area. Our proposed methods, as mentioned earlier, can be easily applied to interrelated areas. One example is that the 3-D shape-based matching algorithm can be used for other objects, such as bottles, papers, and food. After integrating with other robotic systems, the use of the robot can be easily extended to other tasks, such as making food, cleaning room, and fetching objects, to assist our daily life.
Using Relevance Feedback in Content-Based Image Metasearch

MetaSeek is an image metasearch engine developed to explore the querying of large, distributed, online visual information systems. The current implementation integrates user feedback into a performance-ranking mechanism. MetaSeek selects and queries the target image search engines according to their success under similar query conditions in previous searches. The current implementation keeps track of each target engine's performance by integrating user feedback for each visual query into a performance database. We begin with a review of the issues in content-based visual query, then describe the current MetaSeek implementation. We present the results of experiments that evaluated the implementation in comparison to a previous version of the system and a baseline engine that randomly selects the individual search engines to query. We conclude by summarizing open issues for future research.
News video story segmentation using fusion of multi-level multi-modal features in TRECVID 2003

We present our new results in news video story segmentation and classification in the context of the TRECVID video retrieval benchmarking event 2003. We applied and extended the maximum entropy statistical model to fuse diverse features effectively from multiple levels and modalities, including visual, audio, and text. We have included various features such as motion, face, music/speech types, prosody, and high-level text segmentation information. The statistical fusion model is used to discover automatically relevant features contributing to the detection of story boundaries. One novel aspect of our method is the use of a feature wrapper to address different types of features - asynchronous, discrete, continuous and delta ones. We also developed several novel features related to prosody. Using the large news video set from the TRECVID 2003 benchmark, we demonstrate satisfactory performance (F1 measure up to 0.76) and, more importantly, observe an interesting opportunity for further improvement.
AMOS: an active system for MPEG-4 video object segmentation

Object segmentation and tracking is a fundamental step for many digital video applications. In this paper, we present an active system (AMOS) which combines low level automatic region segmentation with an active method for defining and tracking high-level semantic video objects. The system contains two stages: an initial object segmentation stage where user input in the starting frame is used to create a semantic object; and an object tracking stage where underlying regions of the semantic object are tracked and grouped through successive frames. Experiments with different types of videos show very good performance.
Single-view recaptured image detection based on physics-based features

In daily life, we can see images of real-life objects on posters, television, or virtually any type of smooth physical surfaces. We seldom confuse these images with the objects per se mainly with the help of the contextual information from the surrounding environment and nearby objects. Without this contextual information, distinguishing an object from an image of the object becomes subtle; it is precisely an effect that a large immersive display aims at achieving. In this work, we study and address a problem that mirrors the above-mentioned recognition problem, i.e., distinguishing images of true natural scenes and those from recapturing. Being able to detect recaptured images, robot vision can be more intelligent and a single-image-based counter-measure for re-broadcast attack on a face authentication system becomes feasible. This work is timely as the face authentication system is getting common on consumer mobile devices such as smart phones and laptop computers. In this work, we present a physical model for image recapturing and the features derived from the model are used in a recaptured image detector. Our physics-based method out-performs a statistics-based method by a significant margin on images of VGA (640×480) and QVGA (320×240) resolutions which are common for mobile devices. In our study, we find that apart from the contextual information, the unique properties for the recaptured image rendering process are crucial for the recognition problem.
Image quality assessment and color difference

An average healthy person does not perceive the world in just black and white. Moreover, the perceived world is not composed of pixels and through vision humans perceive structures. However, the acquisition and display systems discretize the world. Therefore, we need to consider pixels, structures and colors to model the quality of experience. Quality assessment methods use the pixel-wise and structural metrics whereas color science approaches use the patch-based color differences. In this work, we combine these approaches by extending CIEDE2000 formula with perceptual color difference to assess image quality. We examine how perceptual color difference-based metric (PCDM) performs compared to PSNR, CIEDE2000, SSIM, MS-SSIM and CW-SSIM on the LIVE database. In terms of linear correlation, PCDM obtains compatible results under white noise (97.9%), Jpeg (95.9%) and Jp2k (95.6%) with an overall correlation of 92.7%. We also show that PCDM captures color-based artifacts that can not be captured by structure-based metrics.