nils-herrmann/clean_scirepeval huggingface dataset:
Semi-supervised distance metric learning for Collaborative Image Retrieval

Typical content-based image retrieval (CBIR) solutions with regular Euclidean metric usually cannot achieve satisfactory performance due to the semantic gap challenge. Hence, relevance feedback has been adopted as a promising approach to improve the search performance. In this paper, we propose a novel idea of learning with historical relevance feedback log data, and adopt a new paradigm called ldquoCollaborative Image Retrievalrdquo (CIR). To effectively explore the log data, we propose a novel semi-supervised distance metric learning technique, called ldquoLaplacian Regularized Metric Learningrdquo (LRML), for learning robust distance metrics for CIR. Different from previous methods, the proposed LRML method integrates both log data and unlabeled data information through an effective graph regularization framework. We show that reliable metrics can be learned from real log data even they may be noisy and limited at the beginning stage of a CIR system. We conducted extensive evaluation to compare the proposed method with a large number of competing methods, including 2 standard metrics, 3 unsupervised metrics, and 4 supervised metrics with side information.
Structural and semantic analysis of video

We discuss our recent research and open issues in structural and semantic analysis of digital video. Specifically, we focus on segmentation, summarization and classification of digital video. In each area, we also emphasize the importance of understanding domain-specific characteristics. In scene segmentation, we introduce the idea of a computable scene as a chunk of audio-visual data that exhibits long-term consistency with regard to several audio-visual properties. In summarization, we discuss shot and program level summaries. We describe classification schemes based on Bayesian networks, which model interaction of multiple classes at different levels using multimedia. We also discuss classification techniques that exploit domain-specific spatial structural constraints as well as temporal transitional models.
Complexity Adaptive H.264 Encoding for Light Weight Streams

Emerging video coding standard H.264 achieves significant efficiency improvements, at the expense of greatly increased computational complexity at both the encoder and the decoder. Most prior works focus on reducing the encoder complexity only. In this paper, we develop a novel approach to reduce the decoder complexity without changing any implementation of standard-compliant decoders. Our solution, called complexity adaptive motion estimation and mode decision (CAMED), involves several core components, including a rigorous rate-distortion-complexity optimization framework, complexity cost modeling, and a complexity control algorithm. Such components are incorporated in the encoder side. Experiments over diverse videos and bit rates have shown that our method can achieve significant decoding complexity reduction (up to 60% of interpolation computation) with little degradation in video quality (les 0.2 dB) and very minor impact on the encoder complexity. Results also show that our method can be readily combined with other methods to reduce the complexity at both encoder and decoder
Subjective preference of spatio-temporal rate in video adaptation using multi-dimensional scalable coding

Video adaptation allows for direct manipulation of existing encoded video streams to meet new resource constraints without having to encode the video from scratch. Multi-dimensional scalable coding, such as motion-compensated subband coding (MCSBC), offers an effective and flexible representation for video adaptation. In order to develop robust criteria for selecting optimal spatio-temporal rates used in adaptation, knowledge about subjective preference of spatio-temporal rates is needed. We study the optimal temporal frame rate over a wide range of bandwidth (50 kbps to 1 Mbps) using subjective quality evaluation with 128 clips and 31 subjects. We analyze the results using statistical testing methods and investigate the dependence of optimal frame rate on user, bandwidth, and video content characteristics. Our findings indicate agreement among most users and the existence of switching bandwidths at which preferred frame rates change. Dependence of the preference on video content types is also revealed.
Visual saliency with side information

We propose novel algorithms for organizing large image and video datasets using both the visual content and the associated side-information, such as time, location, authorship, and so on. Earlier research have used side-information as pre-filter before visual analysis is performed, and we design a machine learning algorithm to model the join statistics of the content and the side information. Our algorithm, Diverse-Density Contextual Clustering (D2C2), starts by finding unique patterns for each sub-collection sharing the same side-info, e.g., scenes from winter. It then finds the common patterns that are shared among all subsets, e.g., persistent scenes across all seasons. These unique and common prototypes are found with Multiple Instance Learning and subsequent clustering steps. We evaluate D2C2 on two web photo collections from Flickr and one news video collection from TRECVID. Results show that not only the visual patterns found by D2C2 are intuitively salient across different seasons, locations and events, classifiers constructed from the unique and common patterns also outperform state-of-the-art bag-of-features classifiers.
Algorithms and system for segmentation and structure analysis in soccer video

In this paper, we present a novel system and effective algorithms for soccer video segmentation. The output, about whether the ball is in play, reveals high-level structure of the content. The first step is to classify each sample frame into 3 kinds of view using a unique domain-specific feature, grass-area-ratio. Here the grass value and classification rules are learned and automatically adjusted to each new clip. Then heuristic rules are used in processing the view label sequence, and obtain play/break status of the game. The results provide good basis for detailed content analysis in next step. We also show that low-level features and mid-level view classes can be combined to extract more information about the game, via the example of detecting grass orientation in the field. The results are evaluated under different metrics intended for different applications; the best result in segmentation is 86.5%.
A quantitative semi-fragile JPEG2000 image authentication system

We propose a novel integrated approach to quantitative semi-fragile authentication of JPEG2000 images under a generic framework which combines ECC and PKI infrastructures. Firstly acceptable manipulations (e.g., re-encoding) which should pass authentication are defined based on considerations of some target applications. We propose a unique concept of lowest authenticable bit rate - images undergoing repetitive re-encoding are guaranteed to pass authentication provided the re-encoding rates are above the lowest allowable bit rate. Our solutions include computation of content-based features derived from the EBCOT encoding procedure of JPEG2000, error correction coding of the derived features, PKI cryptographic signing, and finally robust embedding of the feature codes into image as watermarks.
Understanding and modeling user interests in consumer videos

The paper analyzes the interests of users in viewing and organizing consumer videos. It proposes a taxonomy of relevant concepts with three basic dimensions of interests (DOIs) and effective models to predict the user interests in each dimension. The three DOIs correspond to the objects, the scenes and the events. Our conclusions are backed with an extensive study, in which users were asked to annotate and score the importance of each DOI in short clips of diverse and real consumer videos. Analysis of the user study data reveals high consistency (70%) of the scores across different users and independence between objects and events. In addition, we show how heuristic rules and neural networks can accurately predict these scores using camera motion, foreground object and audio information. The automatic and effective prediction of user interests has the potential for improving applications for annotating and summarizing consumer videos
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos

Temporal action localization is an important yet challenging problem. Given a long, untrimmed video consisting of multiple action instances and complex background contents, we need not only to recognize their action categories, but also to localize the start time and end time of each instance. Many state-of-the-art systems use segment-level classifiers to select and rank proposal segments of pre-determined boundaries. However, a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. To this end, we design a novel Convolutional-De-Convolutional (CDC) network that places CDC filters on top of 3D ConvNets, which have been shown to be effective for abstracting action semantics but reduce the temporal length of the input data. The proposed CDC filter performs the required temporal upsampling and spatial downsampling operations simultaneously to predict actions at the frame-level granularity. It is unique in jointly modeling action semantics in space-time and fine-grained temporal dynamics. We train the CDC network in an end-to-end manner efficiently. Our model not only achieves superior performance in detecting actions in every frame, but also significantly boosts the precision of localizing temporal boundaries. Finally, the CDC network demonstrates a very high efficiency with the ability to process 500 frames per second on a single GPU server. Source code and trained models are available online at https://bitbucket.org/columbiadvmm/cdc.
Single color extraction and image query

We propose a method for automatic color extraction and indexing to support color queries of image and video databases. This approach identifies the regions within images that contain colors from predetermined color sets. By searching over a large number of color sets, a color index for the database is created in a fashion similar to that for file inversion. This allows very fast indexing of the image collection by the color contents of the images. Furthermore, information about the identified regions, such as the color set, size, and location, enables a rich variety of queries that specify both color content and spatial relationships of regions. We present the single color extraction and indexing method and contrast it to other color approaches. We examine single and multiple color extraction and image query on a database of 3000 color images.
Frequency and spatially adaptive wavelet packets

We consider a method for image compression based on frequency and spatially adaptive wavelet packets. We present a new fast directed acyclic graph (DAG) structured decomposition, with both spatial segmentation and orthogonal frequency branching from each node. Whereas traditional wavelet packet decomposition adapts to a global frequency distribution, this technique finds the best joint spatial segmentation and local frequency basis. The algorithm is derived from the fast double tree algorithm proposed by Herley, et. al. (see IEE Transactions on Signal Processing, December 1993), for 1-D signals, with an extension to 2-D and modification to include spatial segmentation of frequency nodes. By collecting redundant nodes in this full adaptive tree, we have derived a directed acyclic graph (DAG) structure which contains the same number of nodes as the double tree, but includes new connections between nodes. We present the adaptive wavelet packet DAG algorithm and examine image compression performance on test images.
Video transcoding for resilience in wireless channels

We describe a method to maintain an acceptable quality for video transported over wireless networks under time-varying conditions. We use a transcoder to modify the resilience of the encoded bitstream by using source coding techniques to provide the appropriate level of resilience for the prevailing channel conditions. We develop a statistical model for image loss versus resilience and wireless conditions. Simulation results indicate that using a transcoder to adjust the resilience can improve video quality when errors occur without significantly sacrificing quality when there are no errors. Also, simulation results compare favorably to the analytical model.
Bi-directional object-based coding in video compression at very low bitrate

An algorithmic architecture of video compression at a very low bitrate is presented. This architecture incorporates block-based as well as object-based coding algorithms and manages them in a cooperative way in which different algorithms are applied to different image frames iteratively: I and P frames are coded by a block-based algorithm and B frames are coded by a bi-directional object-based algorithm. An extra degree of freedom in video coding is introduced in order to achieve higher compression ratios while maintaining good visual quality. We emphasize the details of the object-based algorithm which is based on the source model of rigid 2D-objects with 3-D motions. Simulations are done for MPEG-4 Class A/B test sequences which contain low to medium spatial detail and a low amount of movement but is not restricted to head and shoulder images. The results are encouraging in that the coder provides a higher image quality with a satisfactory motion effect.
A practical methodology for guaranteeing quality of service for video-on-demand

A novel and simple approach for defining end-to-end quality of service (QoS) in video-on-demand (VoD) services is presented. Using this approach, we derive a schedulable region for a video server which guarantees end-to-end QoS, where a specific QoS required in the video client translates into a QoS specification for the video server. Our methodology is based on a generic model for VoD services, which is extendible to any VoD system. In this kind of system, both the network and the video server are potential sources of QoS degradation. Specifically, we examine the effect that impairments in the video server and video client have on the video quality perceived by the end user. The Columbia VoD testbed is presented as an example to validate the model through experimental results. Our model can be connected to network QoS admission control models to create a unified approach for admission control of incoming video requests in the video server and network.
Real-time content-based adaptive streaming of sports videos

We present a real-time software system for filtering live video content and then adoptively streaming the video over resource limited networks or devices. The rate adaptation is content-based and dynamically varied according to the event structure and the video content. The system includes a content analysis module for detecting important segments in the video, a variable-rate encoding module, and a buffer management module for streaming the variable rate video over low bandwidth networks. We have implemented a prototype system, which performs real-time content adaptation for tennis and baseball with promising results.
Topic Tracking Across Broadcast News Videos with Visual Duplicates and Semantic Concepts

Videos from distributed sources (e.g., broadcasts, podcasts, blogs, etc.) have grown exponentially. Topic threading is very useful for organizing such large-volume information sources. Current solutions primarily rely on text features only but encounter difficulty when text is noisy or unavailable. In this paper, we propose new representations and similarity measures for news videos based on low-level features, visual near-duplicates, and high-level semantic concepts automatically detected from videos. We develop a multi-modal fusion framework for estimating relevance of a new story to a known topic. Our extensive experiments using TRECVID 2005 data set (171 hours, 6 channels, 3 languages) confirm that near-duplicates consistently and significantly boost the tracking performance by up to 25%. In addition, we present information-theoretic analysis to assess the complexity of each semantic topic and determine the best subset of concepts for tracking each topic.
From fragments to salient closed boundaries: an in-depth study

This paper conducts an in-depth study on a classical perceptual-organization problem: finding salient closed boundaries from a set of boundary fragments detected in a noisy image. In this problem, a saliency boundary is formed by identifying and connecting a subset of fragments according to the simple Gestalt laws of closure, continuity, and proximity. Our specific interest is focused on the methods that aim to achieve boundary closure, an important global property of perceptual salient boundaries. In this paper, we analyze and compare three such methods that are developed in recent years: (a) Elder and Zucker's method based on the shortest-path algorithm, (b) Williams and Thornber's method combining the spectral-analysis and the strongly-connected-component algorithms, and (c) Wang, Kubota, and Siskind's method based on ratio-contour algorithm. Both theoretic analysis and experimental study show that, with a unified setting of fragment saliency, Wang, Kubota and Siskind's method more appropriately constrains the search space for the closed boundaries, and usually produces better performance than or at least comparable performance as the other two methods. Particularly, Wang, Kubota, and Siksind's method can always guarantee the boundary closure and simplicity, which may not be always hold in the other two methods. We construct and collect a variety of synthesized and real images for this comparison.
Predicting Evoked Emotions in Video

Understanding how human emotion is evoked from visual content is a task that we as people do every day, but machines have not yet mastered. In this work we address the problem of predicting the intended evoked emotion at given points within movie trailers. Movie Trailers are carefully curated to elicit distinct and specific emotional responses from viewers, and are therefore well-suited for emotion prediction. However, current emotion recognition systems struggle to bridge the "affective gap", which refers to the difficulty in modeling high-level human emotions with low-level audio and visual features. To address this problem, we propose a mid-level concept feature, which is based on detectable movie shot concepts which we believe to be tied closely to emotions. Examples of these concepts are "Fight", "Rock Music", and "Kiss". We also create 2 datasets, the first with shot-level concept annotations for learning our concept detectors, and a separate, second dataset with emotion annotations taken throughout the trailers using the two dimensional arousal and valence model for emotion annotation. We report the performance of our concept detectors, and show that by using the output of these detectors as a mid-level representation for the movie shots we are able to more accurately predict the evoked emotion throughout a trailer than by using low-level features.
Discovering meaningful multimedia patterns with audio-visual concepts and associated text

The work presents the first effort to automatically annotate the semantic meanings of temporal video patterns obtained through unsupervised discovery processes. This problem is interesting in domains where neither perceptual patterns nor semantic concepts have simple structures. The patterns in video are modeled with hierarchical hidden Markov models (HHMM), with efficient algorithms to learn the parameters, the model complexity and the relevant features; the meanings are contained in words of the speech transcript of the video. The pattern-word association is obtained via cooccurrence analysis and statistical machine translation models. Promising results are obtained through extensive experiments on 20+ hours of TRECVID news videos: video patterns that associate with distinct topics such as el-nino and politics are identified; the HHMM temporal structure model compares favorably to a nontemporal clustering algorithm.
Hash Bit Selection: A Unified Solution for Selection Problems in Hashing

Recent years have witnessed the active development of hashing techniques for nearest neighbor search over big datasets. However, to apply hashing techniques successfully, there are several important issues remaining open in selecting features, hashing algorithms, parameter settings, kernels, etc. In this work, we unify all these selection problems into a hash bit selection framework, i.e., selecting the most informative hash bits from a pool of candidate bits generated by different types of hashing methods using different feature spaces and/or parameter settings, etc. We represent the bit pool as a vertex- and edge-weighted graph with the candidate bits as vertices. The vertex weight represents the bit quality in terms of similarity preservation, and the edge weight reflects independence (non-redundancy) between bits. Then we formulate the bit selection problem as quadratic programming on the graph, and solve it efficiently by replicator dynamics. Moreover, a theoretical study is provided to reveal a very interesting insight: the selected bits actually are the normalized dominant set of the candidate bit graph. We conducted extensive large-scale experiments for three important application scenarios of hash techniques, i.e., hashing with multiple features, multiple hashing algorithms, and multiple bit hashing. We demonstrate that our bit selection approach can achieve superior performance over both naive selection methods and state-of-the-art hashing methods under each scenario, with significant accuracy gains ranging from 10% to 50% relatively.
Robust visual domain adaptation with low-rank reconstruction

Visual domain adaptation addresses the problem of adapting the sample distribution of the source domain to the target domain, where the recognition task is intended but the data distributions are different. In this paper, we present a low-rank reconstruction method to reduce the domain distribution disparity. Specifically, we transform the visual samples in the source domain into an intermediate representation such that each transformed source sample can be linearly reconstructed by the samples of the target domain. Unlike the existing work, our method captures the intrinsic relatedness of the source samples during the adaptation process while uncovering the noises and outliers in the source domain that cannot be adapted, making it more robust than previous methods. We formulate our problem as a constrained nuclear norm and ℓ2, 1 norm minimization objective and then adopt the Augmented Lagrange Multiplier (ALM) method for the optimization. Extensive experiments on various visual adaptation tasks show that the proposed method consistently and significantly beats the state-of-the-art domain adaptation methods.
Automated binary texture feature sets for image retrieval

Digital image and video libraries require new algorithms for the automated extraction and indexing of salient image features. Texture features provide one important cue for the visual perception and discrimination of image content. We propose a new approach for automated content extraction that allows for efficient database searching using texture features. The algorithm automatically extracts texture regions from image spatial-frequency data which are represented by binary texture feature vectors. We demonstrate that the binary texture features provide excellent performance in image query response time while providing highly effective texture discriminability, accuracy in spatial localization and capability for extraction from compressed data representations. We present the binary texture feature extraction and indexing technique and examine searching by texture on a database of 500 images.