nils-herrmann/clean_scirepeval huggingface dataset:
A review on multimodal video indexing

Efficient and effective handling of video documents depends on the availability of indexes. Manual indexing is unfeasible for large video collections. Efficient, single modality based, video indexing methods have appeared in literature. Effective indexing, however, requires a multimodal approach in which either the most appropriate modality is selected or the different modalities are used in collaborative fashion. We present a framework for multimodal video indexing, which views a video document from the perspective of its author. The framework serves as a blueprint for a generic and flexible multimodal video indexing system, and generalizes different state-of-the-art video indexing methods. It furthermore forms the basis for categorizing these different methods.
Bootstrapping Visual Categorization With Relevant Negatives

Learning classifiers for many visual concepts are important for image categorization and retrieval. As a classifier tends to misclassify negative examples which are visually similar to positive ones, inclusion of such misclassified and thus relevant negatives should be stressed during learning. User-tagged images are abundant online, but which images are the relevant negatives remains unclear. Sampling negatives at random is the de facto standard in the literature. In this paper, we go beyond random sampling by proposing Negative Bootstrap. Given a visual concept and a few positive examples, the new algorithm iteratively finds relevant negatives. Per iteration, we learn from a small proportion of many user-tagged images, yielding an ensemble of meta classifiers. For efficient classification, we introduce Model Compression such that the classification time is independent of the ensemble size. Compared with the state of the art, we obtain relative gains of 14% and 18% on two present-day benchmarks in terms of mean average precision. For concept search in one million images, model compression reduces the search time from over 20 h to approximately 6 min. The effectiveness and efficiency, without the need of manually labeling any negatives, make negative bootstrap appealing for learning better visual concept classifiers.
Locality in Generic Instance Search from One Example

This paper aims for generic instance search from a single example. Where the state-of-the-art relies on global image representation for the search, we proceed by including locality at all steps of the method. As the first novelty, we consider many boxes per database image as candidate targets to search locally in the picture using an efficient point-indexed representation. The same representation allows, as the second novelty, the application of very large vocabularies in the powerful Fisher vector and VLAD to search locally in the feature space. As the third novelty we propose an exponential similarity function to further emphasize locality in the feature space. Locality is advantageous in instance search as it will rest on the matching unique details. We demonstrate a substantial increase in generic instance search performance from one example on three standard datasets with buildings, logos, and scenes from 0.443 to 0.620 in mAP.
Mediamill: Searching Multimedia Archives Based on Learned Semantics

Video is about to conquer the Internet. Real-time delivery of video content is technically possible to any desktop and mobile device, even with modest connections. The main problem hampering massive (re)usage of video content today is the lack of effective content-based tools that provide semantic access. In this contribution, we discuss systems for both video analysis and video retrieval that facilitate semantic access to video sources. Both systems were evaluated in the 2004 TRECVID benchmark as top performers in their task
Spatial-Aware Object Embeddings for Zero-Shot Localization and Classification of Actions

We aim for zero-shot localization and classification of human actions in video. Where traditional approaches rely on global attribute or object classification scores for their zero-shot knowledge transfer, our main contribution is a spatial-aware object embedding. To arrive at spatial awareness, we build our embedding on top of freely available actor and object detectors. Relevance of objects is determined in a word embedding space and further enforced with estimated spatial preferences. Besides local object awareness, we also embed global object awareness into our embedding to maximize actor and object interaction. Finally, we exploit the object positions and sizes in the spatial-aware embedding to demonstrate a new spatiotemporal action retrieval scenario with composite queries. Action localization and classification experiments on four contemporary action video datasets support our proposal. Apart from state-of-the-art results in the zero-shot localization and classification settings, our spatial-aware embedding is even competitive with recent supervised action localization alternatives.
Content-Based Analysis Improves Audiovisual Archive Retrieval

Content-based video retrieval is maturing to the point where it can be used in real-world retrieval practices. One such practice is the audiovisual archive, whose users increasingly require fine-grained access to broadcast television content. In this paper, we take into account the information needs and retrieval data already present in the audiovisual archive, and demonstrate that retrieval performance can be significantly improved when content-based methods are applied to search. To the best of our knowledge, this is the first time that the practice of an audiovisual archive has been taken into account for quantitative retrieval evaluation. To arrive at our main result, we propose an evaluation methodology tailored to the specific needs and circumstances of the audiovisual archive, which are typically missed by existing evaluation initiatives. We utilize logged searches, content purchases, session information, and simulators to create realistic query sets and relevance judgments. To reflect the retrieval practice of both the archive and the video retrieval community as closely as possible, our experiments with three video search engines incorporate archive-created catalog entries as well as state-of-the-art multimedia content analysis results. A detailed query-level analysis indicates that individual content-based retrieval methods such as transcript-based retrieval and concept-based retrieval yield approximately equal performance gains. When combined, we find that content-based video retrieval incorporated into the archive's practice results in significant performance increases for shot retrieval and for retrieving entire television programs. The time has come for audiovisual archives to start accommodating content-based video retrieval methods into their daily practice.
TagBook: A Semantic Video Representation Without Supervision for Event Detection

We consider the problem of event detection in video for scenarios where only a few, or even zero, examples are available for training. For this challenging setting, the prevailing solutions in the literature rely on a semantic video representation obtained from thousands of pretrained concept detectors. Different from existing work, we propose a new semantic video representation that is based on freely available social tagged videos only, without the need for training any intermediate concept detectors. We introduce a simple algorithm that propagates tags from a video's nearest neighbors, similar in spirit to the ones used for image retrieval, but redesign it for video event detection by including video source set refinement and varying the video tag assignment. We call our approach TagBook and study its construction, descriptiveness, and detection performance on the TRECVID 2013 and 2014 multimedia event detection datasets and the Columbia Consumer Video dataset. Despite its simple nature, the proposed TagBook video representation is remarkably effective for few-example and zero-example event detection, even outperforming very recent state-of-the-art alternatives building on supervised representations.
The Role of Visual Content and Style for Concert Video Indexing

This paper contributes to the automatic indexing of concert video. In contrast to traditional methods, which rely primarily on audio information for summarization applications, we explore how a visual-only concept detection approach could be employed. We investigate how our recent method for news video indexing -which takes into account the role of content and style -generalizes to the concert domain. We analyze concert video on three levels of visual abstraction, namely: content, style, and their fusion. Experiments with 12 concept detectors, on 45 hours of visually challenging concert video, show that the automatically learned best approach is concept-dependent. Moreover, these results suggest that the visual modality provides ample opportunity for more effective indexing and retrieval of concert video when used in addition to the auditory modality.
Active Transfer Learning with Zero-Shot Priors: Reusing Past Datasets for Future Tasks

How can we reuse existing knowledge, in the form of available datasets, when solving a new and apparently unrelated target task from a set of unlabeled data? In this work we make a first contribution to answer this question in the context of image classification. We frame this quest as an active learning problem and use zero-shot classifiers to guide the learning process by linking the new task to the the existing classifiers. By revisiting the dual formulation of adaptive SVM, we reveal two basic conditions to choose greedily only the most relevant samples to be annotated. On this basis we propose an effective active learning algorithm which learns the best possible target classification model with minimum human labeling effort. Extensive experiments on two challenging datasets show the value of our approach compared to the state-of-the-art active learning methodologies, as well as its potential to reuse past datasets with minimal effort for future tasks.
A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval

Effective video retrieval is the result of interplay between interactive query selection, advanced visualization of results, and a goal-oriented human user. Traditional interactive video retrieval approaches emphasize paradigms, such as query-by-keyword and query-by-example, to aid the user in the search for relevant footage. However, recent results in automatic indexing indicate that query-by-concept is becoming a viable resource for interactive retrieval also. We propose in this paper a new video retrieval paradigm. The core of the paradigm is formed by first detecting a large lexicon of semantic concepts. From there, we combine query-by-concept, query-by-example, query-by-keyword, and user interaction into the MediaMill semantic video search engine. To measure the impact of increasing lexicon size on interactive video retrieval performance, we performed two experiments against the 2004 and 2005 NIST TRECVID benchmarks, using lexicons containing 32 and 101 concepts, respectively. The results suggest that from all factors that play a role in interactive retrieval, a large lexicon of semantic concepts matters most. Indeed, by exploiting large lexicons, many video search questions are solvable without using query-by-keyword and query-by-example. In addition, we show that the lexicon-driven search engine outperforms all state-of-the-art video retrieval systems in both TRECVID 2004 and 2005
Fast algorithms for optimal two-description scalar quantizer design

New efficient algorithms are presented to design globally optimal two-description quantizers of fixed rate. The optimization objective is to minimize the expected distortion at the receiver side. We formulate the problem as one of shortest path in a directed acyclic graph. The fixed rate requirement puts constraints on the number and type of edges of the shortest path, which leads to an O(K/sub 1/K/sub 2/N/sup 3/) time design algorithm, where N is the cardinality of the source alphabet, and K/sub 1/, K/sub 2/ are the number of codecells, respectively, of the two side quantizers. This complexity is reduced to O(K/sub 1/K/sub 2/N/sup 2/) by exploiting a so-called Monge property of the objective function. Furthermore, if K/sub 1/=K/sub 2/=K and the two descriptions are subject to the same channel statistics, then the optimal description quantizer design problem can be solved in O(KN/sup 2/) time.
GPU-aided real-time image/video super resolution based on error feedback

Super resolution is a process to generate high-resolution images from their low-resolution versions. In many applications such as super-HD (4K) TV, super resolution has to be performed in real time. In this paper we propose a real-time image/video super-resolution algorithm, which achieves good performance at low computational cost via off-line learning of interpolation errors in different pixel contexts. The proposed algorithm consists of three stages: fast edge-guided interpolation to generate an initial HR estimation, GPU-aided de-convolution, and error feedback compensation. All three stages can be implemented with GPU to support real-time applications. Experiments demonstrate the competitive performance of the new real-time super-resolution algorithm in both PSNR and visual quality.
Real-Time High-Fidelity Compression for Extremely High Frame Rate Video Cameras

The single most debilitating bottleneck for sustained capturing of ultrahigh speed high-resolution video is the limited bandwidth of the on-camera mass storage device. We propose a novel downsampling-based real-time compression algorithms together with a suited camera architecture to overcome the bandwidth problem. The encoder generates and embeds into the compression code stream some useful side information to assist the decompression at the decoder end. The fidelity of recovered videos is made scalable to the decoder complexity, ranging from real-time decoding for instant playback at lesser quality to offline high-quality reconstruction by sophisticated sparsity and learning-based recovery algorithms, which are aided by the side information.
Image coding by adaptive tree-structured segmentation

A new segmentation-based image coding method is proposed. The encoder recursively partitions an image into convex n-gons, 3<or=n<or=8, until the pixels satisfy a uniformity criterion. The partition is adaptive to image semantics by aligning these n-gons with image edges. The compressed image is sent as a labeled pointerless binary tree, and decoding is simply polygon filling. The adaptability of the partition is achieved by an optimized 2-dimensional piecewise constant approximation of the image. This seemingly expensive optimization process is made computationally feasible by a novel preprocessing technique. High compression ratios are obtained through a good compromise between the segmentation validity and segmentation complexity.<<ETX>>
Monotonicity-based fast algorithms for MAP estimation of Markov sequences over noisy channels

In this correspondence, we study algorithmic approach to solving the problem of maximum a posteriori (MAP) estimation of Markov sequences transmitted over noisy channels, which is also known as the MAP decoding problem. For the class of memoryless binary channels that produce independent substitution and erasure errors, the MAP sequence estimation problem can be formulated and solved as one of the longest path in a weighted directed acyclic graph. But for algorithm efficiency, we transform the graph problem to one of matrix search. If the underlying matrix is totally monotone, then the complexity of MAP sequence estimation can be greatly reduced. We give a sufficient condition for the matrix induced by MAP sequence estimation to be totally monotone, which is indeed the case if the input sequence is Gaussian Markov. Under this condition, the complexity of MAP decoding can be reduced from O(N/sup 2/M) to O(NM), where N is the size of source alphabet and M is the length of input sequence. Furthermore, for Markov sequences of fixed-length code we propose a block parsing strategy to reduce the complexity of MAP sequence estimation to O(M+N/sup 2/M/logM) or to O(M+NM/logM), depending on if the total monotonicity holds. Another significance of this correspondence lies in the applicability of the presented algorithmic approach, which has been thoroughly studied in computer science literature, to many other discrete optimization problems encountered in both source and channel coding, ranging from optimal multiresolution and multiple-description quantizer design, to context quantization for minimum conditional entropy, and to optimal packetization with uneven error protection.
Low Bit-Rate Image Compression via Adaptive Down-Sampling and Constrained Least Squares Upconversion

Recently, many researchers started to challenge a long-standing practice of digital photography: oversampling followed by compression and pursuing more intelligent sparse sampling techniques. In this paper, we propose a practical approach of uniform down sampling in image space and yet making the sampling adaptive by spatially varying, directional low-pass prefiltering. The resulting down-sampled prefiltered image remains a conventional square sample grid, and, thus, it can be compressed and transmitted without any change to current image coding standards and systems. The decoder first decompresses the low-resolution image and then upconverts it to the original resolution in a constrained least squares restoration process, using a 2-D piecewise autoregressive model and the knowledge of directional low-pass prefiltering. The proposed compression approach of collaborative adaptive down-sampling and upconversion (CADU) outperforms JPEG 2000 in PSNR measure at low to medium bit rates and achieves superior visual quality, as well. The superior low bit-rate performance of the CADU approach seems to suggest that oversampling not only wastes hardware resources and energy, and it could be counterproductive to image quality given a tight bit budget.
Improvement of H.264 SVC by model-based adaptive resolution upconversion

H.264 SVC extension, as the state of art scalable video coding standard, can offer a single code stream to serve diverse communication bandwidths and display resolutions. However, the rate-distortion performance of H.264 SVC is still inferior to the non-scalable H.264 AVC. To reduce the performance gap between H.264 SVC and H.264 AVC, we propose a model-based adaptive resolution upconversion algorithm to improve the precision of the H.264 SVC inter-layer prediction. The new algorithm treats the up-sampling of video frames as an inverse problem of initial H.264 SVC down-sampling operation, and it significantly improves the performance of current H.264 SVC by optimally reversing the down-sampling filter.
Primary-consistent soft-decision color demosaicking for digital cameras (patent pending)

Color mosaic sampling schemes are widely used in digital cameras. Given the resolution of CCD sensor arrays, the image quality of digital cameras using mosaic sampling largely depends on the performance of the color demosaicking process. A common problem with existing color demosaicking algorithms is an inconsistency of sample interpolations in different primary color channels, which is the cause of the most objectionable color artifacts. To cure the problem, we propose a new primary-consistent soft-decision framework (PCSD) of color demosaicking. In the PCSD framework, we make multiple estimates of a missing color sample under different hypotheses on edge or texture directions. The estimates are made via a primary consistent interpolation, meaning that all three primary components of a color are interpolated in the same direction. The final estimate of a color sample is obtained by testing different interpolation hypotheses in the reconstructed full-resolution color image and selecting the best via an optimal statistical decision or inference process. A concrete color demosaicking method of the PCSD framework is presented. This new method eliminates certain types of color artifacts of existing color demosaicking methods. Extensive experimental results demonstrate that the PCSD approach can significantly improve the image quality of digital cameras in both subjective and objective measures. In some instances, our gain over the competing methods can be as much as 7dB.
Hierarchical modeling via optimal context quantization

Optimal context quantization with respect to the minimum conditional entropy (MCECQ) is proven to be an efficient way for high order statistical modeling and model complexity reduction in data compression systems. The MCECQ merges together contexts that have similar statistics to reduce the size of the original model. In this technique, the number of output clusters (the model size) must be set before quantization. Optimal model size for the given data is not usually known in advance. We extend the MCECQ technique to a multi-model approach for context modeling, which overcomes this problem and gives the possibilities for better fitting the model to the actual data. The method is primarily intended for image compression algorithms. In our experiments, we applied the proposed technique to embedded conditional bit-plane entropy coding of wavelet transform coefficients. We show that the performance of the proposed modeling achieves the performance of the optimal model of fixed size found individually for given data using MCECQ (and in most cases it is even slightly better).
Image interpolation using texture orientation map and kernel Fisher discriminant

We propose a new non-linear approach of high order context to image interpolation. A global texture orientation map is generated by directional Gabor filters to estimate the edge directions in subpixel precision. The interpolation direction is further refined by a kernel Fisher discriminant that exploits prior knowledge gained from a training set. Experiments show that the proposed method can preserve edge sharpness and subdue the ringing artifacts better than the existing methods, and it obtains higher PSNR as well.
Near-lossless image compression by combining wavelets and CALIC

In this paper we report experimental work on L/sub /spl infin//-constrained high-fidelity image compression. In our experiments minimum-entropy pre-quantization and minimum-entropy trellis quantization did not lead to competitive L/sub /spl infin//-constrained near-lossless compression performance against our previous work of L/sub /spl infin//-constrained CALIC (Wu et al. 1997). But we were able to improve L/sub /spl infin//-constrained CALIC, the best L/sub /spl infin//-constrained image coder for error tolerance below 7 so far by a hybrid approach. First, an input image is wavelet transformed, the wavelet coefficients are quantized in an L/sub /spl infin// criterion and entropy coded. Then a wavelet approximation of the image is constructed by inverse transform based on the quantized wavelet coefficients. Finally, L/sub /spl infin//-constrained CALIC is used to compress the residue image between the wavelet approximation and the original.
Progressive coding of medical volumetric data using three-dimensional integer wavelet packet transform

We examine progressive lossy to lossless compression of medical volumetric data using three-dimensional (3D) integer wavelet packet transforms and set partitioning in hierarchical trees (SPIHT). To achieve good lossy coding performance, we describe a 3D integer wavelet packet transform that allows implicit bit shifting of wavelet coefficients to approximate a 3D unitary transformation. We also address context modeling for efficient entropy coding within the SPIHT framework. Both lossy and lossless coding performance are better than those previously reported.