nils-herrmann/clean_scirepeval huggingface dataset:
Multi-task deep visual-semantic embedding for video thumbnail selection

Given the tremendous growth of online videos, video thumbnail, as the common visualization form of video content, is becoming increasingly important to influence user's browsing and searching experience. However, conventional methods for video thumbnail selection often fail to produce satisfying results as they ignore the side semantic information (e.g., title, description, and query) associated with the video. As a result, the selected thumbnail cannot always represent video semantics and the click-through rate is adversely affected even when the retrieved videos are relevant. In this paper, we have developed a multi-task deep visual-semantic embedding model, which can automatically select query-dependent video thumbnails according to both visual and side information. Different from most existing methods, the proposed approach employs the deep visual-semantic embedding model to directly compute the similarity between the query and video thumbnails by mapping them into a common latent semantic space, where even unseen query-thumbnail pairs can be correctly matched. In particular, we train the embedding model by exploring the large-scale and freely accessible click-through video and image data, as well as employing a multi-task learning strategy to holistically exploit the query-thumbnail relevance from these two highly related datasets. Finally, a thumbnail is selected by fusing both the representative and query relevance scores. The evaluations on 1,000 query-thumbnail dataset labeled by 191 workers in Amazon Mechanical Turk have demonstrated the effectiveness of our proposed method.
Highly parallel mode decision method for HEVC

High Efficiency Video Coding (HEVC) standard achieves double compression efficiency compared to H.264/AVC with the adoption of more flexible coding structure and advanced coding tools. On the other hand, the coding mode space is too large and it's very time consuming for an HEVC encoder to search for the best coding mode. With the development of multi-core or many-core computing architecture, parallelizing HEVC encoding on such platforms is an efficient approach to fulfill the high computational requirement. In this paper, we exploit the potential parallelism in HEVC mode decision (MD) process and propose a highly parallel MD method which works in a motion estimation region (MER). Specifically, we analyze and remove data dependencies that hinder parallel MD, including motion estimation (ME) dependencies and entropy coding dependencies, and then the MD computation for different blocks within the same MER can be computed concurrently. Experimental results show that our proposed parallel MD method gets an overall speed up of more than 14x with negligible quality loss (1.79% bit rate increasing), compared with the non-parallel baseline.
A Fast Coefficients Convertion Method for the Transform Domain MPEG-2 to H.264 Transcoding

Converting MPEG-2 8-tap discrete cosine transform (DCT) coefficients to H.264/AVC 4-tap integer transform (IT) coefficients is one of the indispensable operations for transcoding MPEG-2 to H.264 in the transform domain. Motion information in the input MPEG-2 sequence should be reused as much as possible to speed up the transcoding process. Motivated by this, we first exploits the distribution information of non-zero DCT coefficients to classify the input 8times8 DCT block into three types, then proposes a fast method to perform coefficients conversion based on the type of 8times8 DCT block. The simulation results show that the proposed method significantly reduces the computational complexity, while maintains almost the same video quality compared with the existing method. Hence, it can be efficiently used in the real time transform domain MPEG-2 to H.264 transcoding
Fast and robust spatial matching for object retrieval

Spatial matching for visual words based object retrieval often involves generating affine transformation hypotheses and then choosing the best hypothesis to measure the spatial consistency. In existing methods, generating an affine transformation hypothesis either requires three correspondences or assumes images are taken in restricted range of viewpoints in using a single correspondence. In this paper, we propose a novel spatial matching method, in which the transformation hypothesis can be estimated from only a single correspondence without the assumption of the viewpoints from which the images are taken. Firstly, affine covariant neighborhoods(ACNs) of features are used to eliminate possible false matches. Secondly, we decompose the affine transformation into three sub-transforms and conquer each sub-transform by exploiting the shape information and the ACNs of a single pair of corresponding features. Experiment results demonstrate that this method improves the average retrieval precision evidently with less computation in comparison with the previous methods.
Growth and Characteristics of Yb-doped ${\rm Y}_{3}{\rm Ga}_{5}{\rm O}_{12}$ Laser Crystal

Ytterbium (Yb) doped rare-earth garnet Y3Ga5O12 (YGG) single crystal was grown by the optical floating zone method for the first time, to our knowledge. Its structure and cell parameter were determined by X-ray powder diffraction. The thermal properties of Yb:YGG, including specific heat, thermal expansion coefficient, thermal diffusion coefficient, and thermal conductivity, were investigated. The optical properties of the crystal were also studied and the effective gain cross sections were calculated. With the crystal cut along the 〈111〉 direction, laser performance was also demonstrated by using a laser diode as the pump source. All the results show that Yb:YGG can be an excellent laser medium for applications to tunable and ultrafast pulsed lasers.
KNSC: A novel local classification method for multimedia semantic analysis

The local classification methods try to simplify the complex global modeling problem by decomposing it into a set of local classification sub-problems, which is a potential key to overcome the semantic gap in multimedia content analysis. In this paper we proposed a Sample-Balancing Clustering segmentation method and an effective local classification framework named K-Nearest Sub-classifiers (KNSC). In KNSC the final prediction is an ensemble of the predictions made by K nearest local classifiers. We experimentally compare the effect of different sub-domain segmentation methods, different types of sub-classifiers and different classification/ensemble strategies. The applications on semantic analysis of TRECVID data show the good performance of our method.
Fast Picture and Macroblock Level Adaptive Frame/Field Coding for H.264

Picture adaptive frame/field (PAFF) coding and macroblock adaptive frame/field (MBAFF) coding in the new video coding standard H.264 can improve compression efficiency for interlaced video coding. It is assumed that the optimal solution can be found by employing a multi-pass strategy, which encodes a macroblock (MB) or picture with all possible modes and chooses the one with smallest rate-distortion cost. However, this coding strategy needs high computational complexity. In this paper, a novel single-pass algorithm based on motion activity detection is proposed to predict the mode of PAFF and MBAFF coding in an analysis stage. The simulation results show that the proposed scheme can reduce computational complexity by approximate 60%~75% compared to the 3-pass encoder of reference software, while maintaining grate coding efficiency
A Hierarchical Scheme for Rapid Video Copy Detection

Today with the rapid increasing popularity of web video sharing, digital copyright protection encounters many troubles. Video copy detection schemes are emerging to cope with the digital video piracy and illegal distribution problems. But the large amount of video data and diversity of copy attacks pose difficulties on copy detection. This paper presents a hierarchical scheme to detect video copies, especially the temporal attacked and re-encoded ones. Our algorithm which is based on the ordinal signature of intra frames and effective R*-tree indexing structure archives real time performance. Comparison experiments are conducted on the benchmarked database of CIVR 2007 copy detection showcase and demonstrate the promising results of the proposed approach.
Restricted H.264/AVC video coding for privacy region scrambling

Scrambling is widely used to protect privacy in surveillance video. However, as a critical issue in privacy protected video scrambling, drift error has not been adequately studied. In this paper, we focus on drift error prevention for different elements scrambling in privacy protected H.264/AVC video, which is the prevailing coding standard. A restricted coding scheme is proposed to prevent drift error in Transform Coefficient (TC), Intra Prediction Mode (IPM) and Motion Vector (MV) scrambling, respectively. Experiments show that the proposed scheme effectively prevents drift error with coding efficiency dramatically improved.
An New Coefficients Transform Matrix for the Transform Domain MPEG-2 TO H.264/AVC Transcoding

In this paper, a fast transform method is proposed to convert MPEG-2 8-tap discrete cosine transform (DCT) coefficients to H.264/AVC 4-tap integer transform coefficients directly in the transform domain. The proposed transform method saves 16 operations for each 8times8 DCT block by utilizing a novel transform kernel matrix and a fast computing method for multiplication of this new matrix. The simulation results show that the proposed method causes only a very little quality degradation, which is completely negligible in practice with the maximum value lower than 8times10-33dB, as compared with Jun Xin' s method. Hence, it can be efficiently used in the transform-domain MPEG-2 to H.264 transcoding
Data driven multi-index hashing

Binary representation for large scale nearest neighbor search received more and more concern recently. Although binary codes can be directly used as indices of the hash tables, correlations between the bits may lead to non-uniform codes distribution and reduce the performance of the hash table. In this paper, we propose a data driven multi-index hashing method for exact nearest neighbor search in Hamming space. By exploring the statistics properties of the dataset, we can separate the correlated bits into different segments during the process of building multiple hash tables, and thus make binary codes distributed as uniformly as possible in each hash table. Experiments conducted on a huge amount of binary codes extracted from the UK Bench dataset show that our method can achieve significant acceleration in searching speed for large scale dataset.
Spatio-temporal visual consistency for video copy detection

Video copy detection is essentially a problem of large scale pattern matching. Various copy attacks which change the visual appearance impose hazard on this task. Based on the spatio-temporal consistency, our algorithm aims to utilize the invariant pattern of visual information for video matching. Position correlation of trajectory feature points is calculated as the signature for fast detection. Experiments using benchmarked dataset and commonly happened copy attacks verify the robustness and efficiency of our algorithm.
Pseudo relevance feedback with incremental learning for high level feature detection

Pseudo Relevance Feedback (PRF) has shown effective performance in information retrieval, but it has seldom been applied in the area of high level feature detection (HLF). In this paper, we explicitly propose to introduce PRF into HLF. Our contributions mainly lie in two-fold: (1) proposing three novel PRF approaches to extract pseudo positive samples, i.e., Nearest-Neighbor (NN) based PRF, Score-Evaluation (SE) based PRF and Multi-Classifier Decision (MCD) based PRF; (2) utilizing incremental learning to reduce the re-training time. We evaluate our approaches on the benchmark of TRECVID2008. Reported results have shown that MCD based approach outperforms the other two and obtain an excellent gain in average precision with respect to the baseline without PRF.
Attention Model Based SIFT Keypoints Filtration for Image Retrieval

Effective feature extraction is a fundamental component of content-based image retrieval. Scale Invariant Feature Transform (SIFT) has been proven to be the most robust local invariant feature descriptor. However, SIFT algorithm generates hundreds of thousands of keypoints per image, and most of them comes from background. This has seriously affected the application of SIFT in real-time image retrieval. This paper addresses this problem and proposes a novel method to filter the SIFT keypoints using attention model. Based on visual attention analysis, all of the keypoints in an image are ranked with their attention saliency, and only the most distinctive keypoints will be reserved. Then we use Bag of words to efficiently index these features. Experiments demonstrate that the attention model based SIFT keypoints filtration algorithm provides significant benefits both in retrieval accuracy and matching speed.
An Innovative Model of Tempo and Its Application in Action Scene Detection for Movie Analysis

In this paper, we present an innovative model of tempo and its application in action scene detection for movie analysis. For the first time, we clearly propose that tempo indicates the rhythm of both movie scenarios and human perception. By thoroughly analyzing both aspects, we classify the factors of tempo into two sorts. The first is based on the film grammar and we use the low level features of shot length and camera motion to describe filmmaking by directors. The second is based on the human perception and we originally propose the information measure for perception depending on the cognitive informatics, a newly emerging and significative subject. With the information in both visual and auditory modalities, the low level features of motion intensity, motion complexity, audio energy and audio pace are integrated for the formulation of information to describe the viewers' emotional changes to continuously developing storyline. With both aspects, tempo is defined and tempo flow plot is derived as the clue of storyline. On the basis of video structuralization and movie tempo analysis, we build a system for hierarchical browse and edit with action scene annotation. The large-scale experiments demonstrate the effectiveness and generality of tempo for action movie analysis.In this paper, we present an innovative model of tempo and its application in action scene detection for movie analysis. For the first time, we clearly propose that tempo indicates the rhythm of both movie scenarios and human perception. By thoroughly analyzing both aspects, we classify the factors of tempo into two sorts. The first is based on the film grammar and we use the low level features of Shot Length and Camera Motion to describe filmmaking by directors. The second is based on the human perception and we originally propose the information measure for perception depending on the cognitive informatics, a newly emerging and significative subject. With the information in both visual and auditory modalities, the low level features of Motion Intensity, Motion Complexity, Audio Energy and Audio Pace are integrated for the formulation of information to describe the viewers' emotional changes to continuously developing storyline. With both aspects, tempo is defined and tempo flow plot is derived as the clue of storyline. On the basis of video structuralization and movie tempo analysis, we build a system for hierarchical browse and edit with action scene annotation. The large-scale experiments demonstrate the effectiveness and generality of tempo for action movie analysis.
Hierarchical threshold technique oriented to video object segmentation

This paper proposed a motion vector error function to segment the objects in video sequence. The key point of the segmentation technique is how to set the appropriate threshold to distinguish the global motion region from the local motion region precisely. This paper also proposed a hierarchical threshold technique based on global motion estimation to solve this problem. Experimental results show that the proposed techniques are robust techniques which refine the set of global motion pixels hierarchically and segment the video objects effectively.
Deep Fusion of Multiple Semantic Cues for Complex Event Recognition.

We present a deep learning strategy to fuse multiple semantic cues for complex event recognition. In particular, we tackle the recognition task by answering how to jointly analyze human actions (who is doing what), objects (what), and scenes (where). First, each type of semantic features (e.g., human action trajectories) is fed into a corresponding multi-layer feature abstraction pathway, followed by a fusion layer connecting all the different pathways. Second, the correlations of how the semantic cues interacting with each other are learned in an unsupervised cross-modality autoencoder fashion. Finally, by fine-tuning a large-margin objective deployed on this deep architecture, we are able to answer the question on how the semantic cues of who, what, and where compose a complex event. As compared with the traditional feature fusion methods (e.g., various early or late strategies), our method jointly learns the essential higher level features that are most effective for fusion and recognition. We perform extensive experiments on two real-world complex event video benchmarks, MED'11 and CCV, and demonstrate that our method outperforms the best published results by 21% and 11%, respectively, on an event recognition task.
Web Video Geolocation by Geotagged Social Resources

This paper considers the problem of web video geolocation: we hope to determine where on the Earth a web video was taken. By analyzing a 6.5-million geotagged web video dataset, we observe that there exist inherent geography intimacies between a video with its relevant videos (related videos and same-author videos). This social relationship supplies a direct and effective cue to locate the video to a particular region on the earth. Based on this observation, we propose an effective web video geolocation algorithm by propagating geotags among the web video social relationship graph. For the video that have no geotagged relevant videos, we aim to collect those geotagged relevant images that are content similar with the video (share some visual or textual information with the video) as the cue to infer the location of the video. The experiments have demonstrated the effectiveness of both methods, with the geolocation accuracy much better than state-of-the-art approaches. Finally, an online web video geolocation system: Video2Locatoin (V2L) is developed to provide public access to our algorithm.
Fusing Audio-Words with Visual Features for Pornographic Video Detection

The traditional approach of filtering pornographic videos on the Internet is based on visual features of key frames. However, it cannot meet users' needs owing to the proliferation of low-resolution videos. To improve the filtering performance, we propose a novel framework of fusing audio-words with visual features for pornographic video detection. Our intention is not only to fuse the two modalities of visual images and audio signals, but also to narrow down the semantic gap between low-level features and high-level concepts by using the mid-level feature "audio-words". To further improve the performance, we present the segmentation algorithm based on units of energy envelope and the decision algorithm based on periodic patterns. The results show that our approach outperforms the traditional one which is based on visual features and achieves satisfactory performance. Moreover, the proposed segmentation algorithm is better than the conventional one using the same length and the proposed decision algorithm exceeds the conventional one using thresholds.
Human attention model for semantic scene analysis in movies

In this paper, we specifically propose the Weber-Fechner Law-based human attention model for semantic scene analysis in movies. Different from traditional video processing techniques, we pay more attention on bringing in the related subjects, such as psychology, physiology and cognitive informatics, for content-based video analysis. The innovation of our work has two aspects. Firstly, we originally construct the human attention model with temporal information instructed by the Weber-Fechner Law. Secondly, motivated by cognitive informatics, we formulate the computational methodology of features in visual, audio and textual modalities in the uniform metric of information quantity. With human attention analysis and semantic scene detection, we build a system for hierarchical browse and edit with semantics annotation. Large-scale experiments demonstrate the effectiveness and generality of the proposed human attention model for movie analysis.
Deep saliency map estimation of hand-crafted features

Saliency detection that utilizes deep convolutional neural networks to obtain high level features from original images has achieved considerable progress during the past years. However, few methods consider learning saliency cues from hand-crafted features. In this paper, we demonstrate that deep learning can produce good enough saliency detection results using only hand-crafted features. We propose a novel multi-context deep learning saliency detection algorithm, where only hand-crafted features are taken into account and modeled in a unified deep learning framework. Extensive experiments on benchmark datasets indicate significant and consistent improvements over the representative deep learning framework based saliency detection methods.
A New Rate Control Scheme for H.264/AVC

A novel achievement to optimum bit allocation for H.264/AVC is presented, which is different from JVT-O016 by using a simple linear rate instead of the well-known MPEG-4 Q2 model. To find the global optimum, we resort to Lagrange optimization technique and develop a close-form formula to the optimum problem. It is shown via extensive experiments that the new rate control (RC) scheme exceeds JVT-G012, the current standardized RC scheme and is comparative to JVT-O016 in coding performance
Salient region detection : Integrate both global and local cues

Visual saliency detection provides an alternative methodology to semantic image understanding in many applications such as region-based image retrieval and adaptive compression of images. In this paper, we propose an approach which utilizes both global and local cues to extract saliency information. Our method can achieve better performance than existing saliency detection methods in terms of precision and recall rates. The main contributions are threefold: 1) a new model which can better describe the color perception of human beings is proposed. Based on this model, a global color contrast cue is also presented. 2) as supplements, two other global cues and one local cues are also presented to capture as much saliency information as we can. 3) a CRF model is used to integrate these cues and generate the final saliency map. Experimental results indicate that our proposed approach is effective and practicable.