nils-herrmann/clean_scirepeval huggingface dataset:
Video-server retrieval scheduling and resource reservation for variable bit rate scalable video

State-of-the-art digital video compression produces bursty, variable bit rate video. The bursty nature of compressed video raises challenges in the design of video servers. In this paper, we first present a method for the efficient retrieval of bursty video data from the disk system to the memory of a digital video server. For a single video stream, the proposed retrieval schedule minimizes the buffer requirement for continuous retrieval, given that a fixed disk bandwidth is reserved for the entire duration of retrieval. Secondly, we present an optimal resource-reservation algorithm for multiple video streams based on the proposed retrieval schedule. The resource-reservation algorithm maximizes the number of bursty video streams that ran be supported by a video server given any disk bandwidth and memory resource. Thirdly, we present a progressive display scheme for scalable video that is based on the retrieval schedule and resource-reservation algorithm. Performance evaluations based on simulations using MPEG-2 trace data are presented. For a personal computer with four disks and a memory resource of 120 MB, our approach can support 50%-275% more video streams than previously proposed approaches, depending on the pre-fetch delay that users are willing to tolerate in interactive viewing of videos.
Content-based indexing and retrieval of visual information

The ease of capturing and encoding digital images has produced a massive amount of visual information online. As a consequence, grand challenges have emerged in the areas of storage, indexing, and retrieval of visual information in large archives. How does one find a photograph from a large archive that contains millions of pictures? How does a CNN video journalist find a specific clip from the myriad of video tapes, ranging from historical to contemporary, from sports to humanities? Efficient, real-time algorithms and systems are needed to address these needs of not only professionals but for users who want to find visual information online.
Embedding visible video watermarks in the compressed domain

Digital visible or invisible watermarks are increasingly in demand for protecting or verifying the original image or video ownership. We propose a novel compressed-domain approach to embedding visible watermarks in MPEG-1 and MPEG-2 video streams. Our algorithms operate on the DCT coefficients which are obtained with minimal parsing of input video. The embedded watermarks adapt to the local video features such as brightness and complexity to achieve consistent perceptual visibility. The embedded watermarks are robust against attempts of removal since clear artifacts remain after the possible attacks.
Generative, discriminative, and ensemble learning on multi-modal perceptual fusion toward news video story segmentation

News video story segmentation is a critical task for automatic video indexing and summarization. Our prior work has demonstrated promising performance by using a generative model, called maximum entropy (ME), which models the posterior probability given the multi-modal perceptual features near the candidate points. In this paper, we investigate alternative statistical approaches based on discriminative models, i.e. support vector machine (SVM), and ensemble learning, i.e. boosting. In addition, we develop a novel approach, called BoostME, which uses the ME classifiers and the associated confidence scores in each boosting iteration. We evaluated these different methods using the TRECVID 2003 broadcast news data set. We found that SVM-based and ME-based techniques both outperformed the pure boosting techniques, with the SVM-based solutions achieving even slightly higher accuracy. Moreover, we summarize extensive analysis results of error sources over distinctive news story types to identify future research opportunities
Interactive Segmentation on RGBD Images via Cue Selection

Interactive image segmentation is an important problem in computer vision with many applications including image editing, object recognition and image retrieval. Most existing interactive segmentation methods only operate on color images. Until recently, very few works have been proposed to leverage depth information from low-cost sensors to improve interactive segmentation. While these methods achieve better results than color-based methods, they are still limited in either using depth as an additional color channel or simply combining depth with color in a linear way. We propose a novel interactive segmentation algorithm which can incorporate multiple feature cues like color, depth, and normals in an unified graph cut framework to leverage these cues more effectively. A key contribution of our method is that it automatically selects a single cue to be used at each pixel, based on the intuition that only one cue is necessary to determine the segmentation label locally. This is achieved by optimizing over both segmentation labels and cue labels, using terms designed to decide where both the segmentation and label cues should change. Our algorithm thus produces not only the segmentation mask but also a cue label map that indicates where each cue contributes to the final result. Extensive experiments on five large scale RGBD datasets show that our proposed algorithm performs significantly better than both other color-based and RGBD based algorithms in reducing the amount of user inputs as well as increasing segmentation accuracy.
Camera Response Functions for Image Forensics: An Automatic Algorithm for Splicing Detection

We present a fully automatic method to detect doctored digital images. Our method is based on a rigorous consistency checking principle of physical characteristics among different arbitrarily shaped image regions. In this paper, we specifically study the camera response function (CRF), a fundamental property in cameras mapping input irradiance to output image intensity. A test image is first automatically segmented into distinct arbitrarily shaped regions. One CRF is estimated from each region using geometric invariants from locally planar irradiance points (LPIPs). To classify a boundary segment between two regions as authentic or spliced, CRF-based cross fitting and local image features are computed and fed to statistical classifiers. Such segment level scores are further fused to infer the image level authenticity. Tests on two data sets reach performance levels of 70% precision and 70% recall, showing promising potential for real-world applications. Moreover, we examine individual features and discover the key factor in splicing detection. Our experiments show that the anomaly introduced around splicing boundaries plays the major role in detecting splicing. Such finding is important for designing effective and efficient solutions to image splicing detection.
Exploring image functionalities in WWW applications development of image/video search and editing engines

Image technologies provide critical components for achieving various Web-based multimedia applications. We identify key technical challenges in today's image/video applications on the WWW. We present two prototype systems, a video search engine (WebSEEk) and a compressed video editor (WebClip), to demonstrate technical challenges and viable solutions. We also discuss major issues in developing next-generation, scalable solutions for large-scale, distributed on-line environments.
Cross-domain learning methods for high-level visual concept classification

Exploding amounts of multimedia data increasingly require automatic indexing and classification, e.g. training classifiers to produce high-level features, or semantic concepts, chosen to represent image content, like car, person, etc. When changing the applied domain (i.e. from news domain to consumer home videos), the classifiers trained in one domain often perform poorly in the other domain due to changes in feature distributions. Additionally, classifiers trained on the new domain alone may suffer from too few positive training samples. Appropriately adapting data/models from an old domain to help classify data in a new domain is an important issue. In this work, we develop a new cross-domain SVM (CDSVM) algorithm for adapting previously learned support vectors from one domain to help classification in another domain. Better precision is obtained with almost no additional computational cost. Also, we give a comprehensive summary and comparative study of the state- of-the-art SVM-based cross-domain learning methods. Evaluation over the latest large-scale TRECVID benchmark data set shows that our CDSVM method can improve mean average precision over 36 concepts by 7.5%. For further performance gain, we also propose an intuitive selection criterion to determine which cross-domain learning method to use for each concept.
Visual Event Recognition in News Video using Kernel Methods with Multi-Level Temporal Alignment

In this work, we systematically study the problem of visual event recognition in unconstrained news video sequences. We adopt the discriminative kernel-based method for which video clip similarity plays an important role. First, we represent a video clip as a bag of orderless descriptors extracted from all of the constituent frames and apply Earth mover's distance (EMD) to integrate similarities among frames from two clips. Observing that a video clip is usually comprised of multiple sub-clips corresponding to event evolution over time, we further build a multilevel temporal pyramid. At each pyramid level, we integrate the information from different sub-clips with Integer-value-constrained EMD to explicitly align the sub-clips. By fusing the information from the different pyramid levels, we develop temporally aligned pyramid matching (TAPM) for measuring video similarity. We conduct comprehensive experiments on the Trecvid 2005 corpus, which contains more than 6,800 clips. Our experiments demonstrate that 1) the TAPM multi-level method clearly outperforms single-level EMD, and 2) single-level EMD outperforms by a large margin (43.0% in Mean Average Precision) basic detection methods that use only a single key-frame. Extensive analysis of the results also reveals an intuitive interpretation of subclip alignment at different levels.
Long-term moving object segmentation and tracking using spatio-temporal consistency

The success of object-based media representation and description (e.g., MPEG-4 and -7) depends largely on effective object segmentation tools. We expand our previous work on automatic video region tracking and develop a robust-moving objects detection system. In our system, we first utilize innovative methods of combining color and edge information in improving the object motion estimation results. Then we use the long-term spatio-temporal constraints to achieve reliable object tracking over long sequences. Our extensive experiments demonstrate excellent results in handling challenging cases in general domains (e.g., stock footage) including depth-varying multi-layer background and fast camera motion.
Mobile product search with Bag of Hash Bits and boundary reranking

Rapidly growing applications on smartphones have provided an excellent platform for mobile visual search. Most of previous visual search systems adopt the framework of ”Bag of Words”, in which words indicate quantized codes of visual features. In this work, we propose a novel visual search system based on ”Bag of Hash Bits” (BoHB), in which each local feature is encoded to a very small number of hash bits, instead of quantized to visual words, and the whole image is represented as bag of hash bits. The proposed BoHB method offers unique benefits in solving the challenges associated with mobile visual search, e.g., low transmission cost, cheap memory and computation on the mobile side, etc. Moreover, our BoHB method leverages the distinct properties of hashing bits such as multi-table indexing, multiple bucket probing, bit reuse, and hamming distance based ranking to achieve efficient search over gigantic visual databases. The proposed method significantly outperforms state-of-the-art mobile visual search methods like CHoG, and other (conventional desktop) visual search approaches like bag of words via vocabulary tree, or product quantization. The proposed BoHB approach is easy to implement on mobile devices, and general in the sense that it can be applied to different types of local features, hashing algorithms and image databases. We also incorporate a boundary feature in the reranking step to describe the object shapes, complementing the local features that are usually used to characterize the local details. The boundary feature can further filter out noisy results and improve the search performance, especially at the coarse category level. Extensive experiments over large-scale data sets up to 400k product images demonstrate the effectiveness of our approach.
Discovering recurrent visual semantics in consumer photographs

We present techniques to semi-automatically discover recurrent visual semantics (RVS)-the repetitive appearance of visually similar elements such as objects and scenes-in consumer photographs. First, we introduce the detection of "bracketing" (very similar photographs) using an edge-correlation metric, which outperforms the color histogram. Then, we use color and novel composition features (based on automatic region segmentation) to perform scene-level clustering of images. We use a novel sequence-weighted technique, which uses the structure of standard film (only image sequence information), to perform hierarchical clustering. We show performance results of bracketing, explore clustering evaluation, and discuss STELLA, an interactive albuming and story telling application that uses these techniques to assist users in building digital albums. The STELLA system uses a new approach to album creation: instead of automatically creating albums, it provides an interactive environment that assists users in digital album creation.
Perceptual knowledge construction from annotated image collections

This paper presents and evaluates new methods for extracting perceptual knowledge from collections of annotated images. The proposed methods include automatic techniques for constructing perceptual concepts by clustering the images based on visual and text feature descriptors, and for discovering perceptual relationships among the concepts based on descriptor similarity and statistics between the clusters. There are two main contributions of this work. The first lies on the support and the evaluation of several techniques for visual and text feature descriptor extraction, for visual and text feature descriptor integration, and for data clustering in the extraction of perceptual concepts. The second contribution is in proposing novel ways for discovering perceptual relationships among concepts. Experiments show extraction of useful knowledge from visual and text feature descriptors, high independence between visual and text feature descriptors, and potential performance improvement by integrating both kinds of descriptors compared to using either kind of descriptor alone.
Detection of non-identical duplicate consumer photographs

Consumers often make more than one photograph of the same scene, creating non-identical "duplicates" and similar "non-duplicates". In Kodak's consumer photography database, 19% of the images fall into this category. Automatic detection of duplicates, therefore, is extremely useful in consumer applications. In this paper, first we develop a model of the problem and introduce a new classification of different types of duplicates. Then, we introduce a novel framework that automatically distinguishes between non-identical duplicate and very similar non-duplicate images. Our approach is based on a multiple strategy framework that combines our knowledge about the geometry of multiple views of the same scene, the extraction of low-level features, the detection of a limited number of semantic objects, and domain knowledge. The approach consists of three stages: (1) global alignment, (2) detection of change areas, and (3) local analysis of change areas. We present a novel and extensive image duplicate database (255 image pairs from 60 rolls from 54 real consumers, labeled by 10 other people). We analyze labeling subjectivity in detail and present experiments using our approach.
Modeling Scene and Object Contexts for Human Action Retrieval With Few Examples

The use of context knowledge is critical for understanding human actions, which typically occur under particular scene settings with certain object interactions. For instance, driving car usually happens outdoors, and kissing involves two people moving toward each other. In this paper, we investigate the problem of context modeling for human action retrieval. We first identify ten simple object-level action atoms relevant to many human actions, e.g., people getting closer. With the action atoms and several background scene classes, we show that action retrieval can be improved through modeling action-scene-object dependency. An algorithm inspired by the popular semi-supervised learning paradigm is introduced for this purpose. One important contribution of this paper is to show that modeling the dependencies among actions, objects, and scenes can be efficiently achieved with very few examples. Such a solution has tremendous potential in practice as it is often expensive to acquire large sets of training data. Experiments were performed on the challenging Hollywood2 dataset containing 89 movies. The results validate the effectiveness of our approach, achieving a mean average precision of 26% with just ten examples per action.
Overview of the MPEG-7 standard

MPEG-7, formally known as the Multimedia Content Description Interface, includes standardized tools (descriptors, description schemes, and language) enabling structural, detailed descriptions of audio-visual information at different granularity levels (region, image, video segment, collection) and in different areas (content description, management, organization, navigation, and user interaction). It aims to support and facilitate a wide range of applications, such as media portals, content broadcasting, and ubiquitous multimedia. We present a high-level overview of the MPEG-7 standard. We first discuss the scope, basic terminology, and potential applications. Next, we discuss the constituent components. Then, we compare the relationship with other standards to highlight its capabilities.
Using Geometry Invariants for Camera Response Function Estimation

In this paper, we present a new single-image camera response function (CRF) estimation method using geometry invariants (GI). We derive mathematical properties and geometric interpretation for GI, which lend insight to addressing various algorithm implementation issues in a principled way. In contrast to the previous single-image CRF estimation methods, our method provides a constraint equation for selecting the potential target data points. Comparing to the prior work, our experiment is conducted over more extensive data and our method is flexible in that its estimation accuracy and stability can be improved whenever more than one image is available. The geometry invariance theory is novel and may be of wide interest.
Automatic view recognition in echocardiogram videos using parts-based representation

Indexing echocardiogram videos at different levels of structure is essential for providing efficient access to their content for browsing and retrieval purposes. We present a novel approach for the automatic identification of the views of the heart from the content of the echocardiogram videos. In this approach the structure of the heart is represented by the constellation of its parts (chambers) under the different views. The statistical variations of the parts in the constellation and their spatial relationships are modeled using Markov random field models. A discriminative method is then used for view recognition, which fuses the assessments of a test image, by all the view-models. To the best of our knowledge, this is the first work addressing the analysis of the echocardiogram videos for the purpose of indexing their content. The method presented could be used for multiple-object recognition when their parts represent the objects and there are structural similarities between them.
Transform features for texture classification and discrimination in large image databases

Proposes a method for classification and discrimination of textures based on the energies of image subbands. The authors show that with this relatively simple feature set, effective texture discrimination can be achieved. In the paper, subband-energy feature sets extracted from the following typical image decompositions are compared: wavelet subband, uniform subband, discrete cosine transform (DCT), and spatial partitioning. The authors report that over 90% correct classification was attained using the feature set in classifying the full Brodatz [1965] collection of 112 textures. Furthermore, the subband energy-based feature set can be readily applied to a system for indexing images by texture content in image databases, since the features can be extracted directly from spatial-frequency decomposed image data. The authors also show that to construct a suitable space for discrimination, Fisher discrimination analysis (Dillon and Goldstein, 1984) can be used to compact the original features into a set of uncorrelated linear discriminant functions. This procedure makes it easier to perform texture-based searches in a database by reducing the dimensionality of the discriminant space. The authors also examine the effects of varying training class size, the number of training classes, the dimension of the discriminant space and number of energy measures used for classification. The authors hope that the performance for texture discrimination of these simple energy-based features will allow images in a database to be efficiently and effectively indexed by contents of their textured regions.<<ETX>>
Combining text and audio-visual features in video indexing

We discuss the opportunities, state of the art, and open research issues in using multi-modal features in video indexing. Specifically, we focus on how imperfect text data obtained by automatic speech recognition (ASR) may be used to help solve challenging problems, such as story segmentation, concept detection, retrieval, and topic clustering. We review the frameworks and machine learning techniques that are used to fuse the text features with audio-visual features. Case studies showing promising performance are described, primarily in the broadcast news video domain.
Domain adaptive semantic diffusion for large scale context-based video annotation

Learning to cope with domain change has been known as a challenging problem in many real-world applications. This paper proposes a novel and efficient approach, named domain adaptive semantic diffusion (DASD), to exploit semantic context while considering the domain-shift-of-context for large scale video concept annotation. Starting with a large set of concept detectors, the proposed DASD refines the initial annotation results using graph diffusion technique, which preserves the consistency and smoothness of the annotation over a semantic graph. Different from the existing graph learning methods which capture relations among data samples, the semantic graph treats concepts as nodes and the concept affinities as the weights of edges. Particularly, the DASD approach is capable of simultaneously improving the annotation results and adapting the concept affinities to new test data. The adaptation provides a means to handle domain change between training and test data, which occurs very often in video annotation task. We conduct extensive experiments to improve annotation results of 374 concepts over 340 hours of videos from TRECVID 2005-2007 data sets. Results show consistent and significant performance gain over various baselines. In addition, the proposed approach is very efficient, completing DASD over 374 concepts within just 2 milliseconds for each video shot on a regular PC.