nils-herrmann/clean_scirepeval huggingface dataset:
Efficient hybrid coding of video for low bitrate applications

A novel video codec for low-bit-rate applications is proposed. The main characteristics of the codec are the use of a space-variant wavelet packet decomposition for energy compaction in the spatial domain and the introduction of a locally adaptive multigrid motion estimation technique. The use of a mesh adaptive multigrid block matching technique for motion estimation reduces both the energy of the prediction error and the amount of the side information for transmitting the appropriate motion field. Accordingly, an adaptive compaction of the energy in the spatial domain using wavelet packets makes it possible to decide the nature of the desired transformation depending on the nature of the incoming signal (intraframe or prediction error). Simulation results show that a significant gain in compression is obtained with this coding scheme compared to a codec based on a dyadic wavelet decomposition and a fixed block size block matching, while maintaining the same quality of reconstructed data.<<ETX>>
The JPEG2000 still image coding system: an overview

With the increasing use of multimedia technologies, image compression requires higher performance as well as new features. To address this need in the specific area of still image encoding, a new standard is currently being developed, the JPEG2000. It is not only intended to provide rate-distortion and subjective image quality performance superior to existing standards, but also to provide features and functionalities that current standards can either not address efficiently or in many cases cannot address at all. Lossless and lossy compression, embedded lossy to lossless coding, progressive transmission by pixel accuracy and by resolution, robustness to the presence of bit-errors and region-of-interest coding, are some representative features. It is interesting to note that JPEG2000 is being designed to address the requirements of a diversity of applications, e.g. Internet, color facsimile, printing, scanning, digital photography, remote sensing, mobile applications, medical imagery, digital library and E-commerce.
Temporal synchronization in stereoscopic video: Influence on quality of experience and automatic asynchrony detection

In this paper, we analyze the influence of temporal asynchrony on the subjective quality of stereoscopic video. Based on our recently created 3D video database, different levels of asynchrony were simulated and a comprehensive subjective test was conducted to determine the associated degradations in quality of experience. Furthermore, we developed a method to detect asynchrony between left and right video streams based on canonical correlation analysis. Experiments demonstrate the robustness of this method with respect to different amounts of asynchrony and scene depth, which makes it suitable to predict quality of experience or automatic resynchronization.
A framework for the validation of privacy protection solutions in video surveillance

The issue of privacy protection in video surveillance has drawn a lot of interest lately. However, thorough performance analysis and validation is still lacking, especially regarding the fulfillment of privacy-related requirements. In this paper, we put forward a framework to assess the capacity of privacy protection solutions to hide distinguishing facial information and to conceal identity. We then conduct rigorous experiments to evaluate the performance of face recognition algorithms applied to images altered by privacy protection techniques. Results show the ineffectiveness of naïve privacy protection techniques such as pixelization and blur. Conversely, they demonstrate the effectiveness of more sophisticated scrambling techniques to foil face recognition.
Metadata-driven multimedia access

With the growing ubiquity and mobility of multimedia-enabled devices, universal multimedia access (UMA) is emerging as one of the important components for the next generation of multimedia applications. The basic concept underlying UMA is universal or seamless access to multimedia content, by automatic selection and adaptation of content based on the user's environment. UMA promises an integration of these different perspectives into a new class of content adaptive applications that could allow users to access multimedia content without concern for specific coding formats, terminal capabilities, or network conditions. We discuss methods that support UMA and the tools provided by MPEG-7 to achieve this. We also discuss the inclusion of metadata in JPEG 2000 encoded images. We present these methods in the typical order that they may be used in an actual application. Therefore, we first discuss the (personalized) selection of desired content from all available content, followed by the organization of related variations of a single piece of content. Then, we discuss segmentation and summarization of audio video (AV) content, and finally, transcoding of AV content.
On the performance of objective metrics for omnidirectional visual content

Omnidirectional image and video have gained popularity thanks to availability of capture and display devices for this type of content. Recent studies have assessed performance of objective metrics in predicting visual quality of omnidirectional content. These metrics, however, have not been rigorously validated by comparing their prediction results with ground-truth subjective scores. In this paper, we present a set of 360-degree images along with their subjective quality ratings. The set is composed of four contents represented in two geometric projections and compressed with three different codecs at four different bitrates. A range of objective quality metrics for each stimulus is then computed and compared to subjective scores. Statistical analysis is performed in order to assess performance of each objective quality metric in predicting subjective visual quality as perceived by human observers. Results show the estimated performance of the state-of-the-art objective metrics for omnidirectional visual content. Objective metrics specifically designed for 360-degree content do not outperform conventional methods designed for 2D images.
Quality assessment of multidimensional video scalability

Scalability is a powerful concept for adaptive video content delivery to many end users having heterogeneous and dynamic characteristics of networks and devices. In order to maximize users' quality of experience by selecting appropriate combinations of multiple scalability parameters, it is crucial to understand and model the relationship between multidimensional scalability and perceived quality. In this article, we address the latest advances in subjective and objective quality evaluation of multidimensional video scalability for optimal content distribution, present their applications, and discuss future trends and challenges.
Image privacy protection with secure JPEG transmorphing

Thanks to advancements in smart mobile devices and social media platforms, sharing photos and experiences has significantly bridged our lives, allowing us to stay connected despite distance and other barriers. However, concern on privacy has also been raised, due to not only mistakes or ignorance of impact of careless sharing but also complex infrastructures and cross-use of social media content. In this paper, we present secure JPEG Transmorphing, a flexible framework for protecting image visual privacy in a secure, reversible, fun and personalized manner. With secure JPEG Transmorphing, the protected image is also backwards compatible with JPEG, the most commonly used image format. Experiments have been performed and results show that the proposed method provides a near lossless image reconstruction, a controllable level of storage overhead, and a good degree of privacy protection and subjective pleasantness.
Quality Assessment of an HDR Dual-Layer Backward-Compatible Codec Compared to Uncompromised SDR and HDR Solutions

Broadcasting high dynamic range (HDR) video has been demonstrated as largely preferred when compared to standard dynamic range (SDR), mainly due to its capability of representing more details in dark and bright regions. Additionally, progress over the last decades on creation, compression, transmission and rendering of HDR content signals a forthcoming deployment of HDR broadcasting services. Lack of a widely supported recommendation regarding bandwidth allocation for HDR compressed streams or a unique compression approach, prevent faster deployment of such services. This paper investigates the performance of a dual-layer backward-compatible compression codec, when compared to state-of-the-art HDR compression strategies, in terms of perceived quality. The evaluated system is a dual-layer compression scheme enabling the transmission of a backward-compatible SDR stream along with an HDR stream, reconstructed from the residual-based enhancement layer and SDR mapping (i.e., prediction). Comparison is made to two compression strategies realizing uncompromised SDR or HDR through the use of single-layer systems multiplexed with metadata. Metadata contains information necessary to map HDR into SDR or SDR into HDR streams. Our conclusion provides guidance regarding the compression strategy to use as well as bandwidth allocation for HDR delivery, ensuring both SDR and HDR contents with perceptually acceptable quality.
Impact of interactivity on the assessment of quality of experience for light field content

The recent advances in light field imaging are changing the way in which visual content is captured, processed and consumed. Storage and delivery systems for light field images rely on efficient compression algorithms. Such algorithms must additionally take into account the feature-rich rendering for light field content. Therefore, a proper evaluation of visual quality is essential to design and improve coding solutions for light field content. Consequently, the design of subjective tests should also reflect the light field rendering process. This paper aims at presenting and comparing two methodologies to assess the quality of experience in light field imaging. The first methodology uses an interactive approach, allowing subjects to engage with the light field content when assessing it. The second, on the other hand, is completely passive to ensure all the subjects will have the same experience. Advantages and drawbacks of each approach are compared by relying on statistical analysis of results and conclusions are drawn. The obtained results provide useful insights for future design of evaluation techniques for light field content.
Toward Intelligent Product Retrieval for TV-to-Online (T2O) Application: A Transfer Metric Learning Approach

It is desired (especially for young people) to shop for the same or similar products shown in the multimedia contents (such as online TV programs). This indicates an urgent demand for improving the experience of TV-to-Online (T2O). In this paper, a transfer learning approach as well as a prototype system for effortless T2O experience is developed. In the system, a key component is high-precision product search, which is to fulfill exact matching between a query item and the database ones. The matching performance primarily relies on distance estimation, but the data characteristics cannot be well modeled and exploited by a simple Euclidean distance. This motivates us to introduce distance metric learning (DML) for improving the distance estimation. However, in traditional DML methods, the side information (such as the similar/dissimilar constraints or relevance/irrelevance judgements) in the target domain is leveraged. These methods may fail due to limited side information. Fortunately, this issue can be alleviated by utilizing transfer metric learning (TML) to exploit information from other related domains. In this paper, a novel manifold regularized heterogeneous multitask metric learning framework is proposed, in which each domain is treated equally. The proposed approach allows us to simultaneously exploit the information from other domains and the unlabeled information. Furthermore, the ranking-based loss is adopted to make our model more appropriate for search. Experiments on two challenging real-world datasets demonstrate the effectiveness of the proposed method. This TML approach is expected to impact the transformation of the emerging T2O trend in both TV and online video domains.
Incorporating intra-class variance to fine-grained visual recognition

Fine-grained visual recognition aims to capture discriminative characteristics amongst visually similar categories. The state-of-the-art research work has significantly improved the fine-grained recognition performance by deep metric learning using triplet network. However, the impact of intra-category variance on the performance of recognition and robust feature representation has not been well studied. In this paper, we propose to leverage intra-class variance in metric learning of triplet network to improve the performance of fine-grained recognition. Through partitioning training images within each category into a few groups, we form the triplet samples across different categories as well as different groups, which is called Group Sensitive TRiplet Sampling (GS-TRS). Accordingly, the triplet loss function is strengthened by incorporating intra-class variance with GS-TRS, which may contribute to the optimization objective of triplet network. Extensive experiments over benchmark datasets CompCar and VehicleID show that the proposed GS-TRS has significantly outperformed state-of-the-art approaches in both classification and retrieval tasks.
GPU Based fast MPEG-CDVS encoder

The compact descriptors for visual search (CDVS) standard from ISO/IEC Moving Picture Experts Group (MPEG) has received increasing attentions due to its effectiveness in mobile visual search related applications. To explore the efficiency of CDVS in real-time applications, we implement the first optimized CDVS encoder based on the graphics processing unit (GPU). In particular, the CDVS feature extraction is implemented in a CPU and GPU collaborative architecture, and the most of the computation-intensive operations are transferred to GPU platform, which achieves significant speedup for CDVS compared with CPU implementation. Extensive evaluations on standard datasets provide promising results of the proposed scheme for real applications scenarios.
Towards compact topical descriptors

We introduce a Compact Topical Descriptor to learn a compact yet discriminative image signature from the reference image corpus. This descriptor is deployed over the well used bag-of-words image histogram, with two merits over the traditional topical features: First, we propose to directly control the topical sparsity to achieve the descriptor compactness. Second, we ensure the descriptor discriminability by minimizing the bag-of-words reconstruction errors during the topical histogram encoding. To this end, we have a generative viewpoint of the topical feature extraction, which is estimated as a sparse MAP estimation over the original bag-of-words. We learn such estimation by a bi-convex optimization, iterating between both hierarchical sparse coding from words to topical histograms and dictionary learning of the corresponding word-to-topic transform. Especially, supervised labels such as image ranking list can be also incorporated into our descriptor learning paradigm. We quantize our performance in both Im-ageNet 10K and NUS-WIDE, with comparisons to bag-of-words, LDA, miniBoF, and Aggregated Local Descriptors. In practice, we also implement our descriptor for a low bit rate mobile visual search application, i.e. sending compact descriptors instead of the image to reduce the query delivery latency. Our descriptor has significantly outperformed the state-of-the-art compact descriptors by quantitative evaluations over 10 million reference images.
A Multi-Block N-ary trie structure for exact r-neighbour search in hamming space

This paper proposes a novel algorithm to solve the exact r-neighbour search problem in Hamming space. Existing r-neighbour search methods typically adopt hash table to index binary codes. Given a query, existing approaches search the nearest neighbours by checking all buckets of a Hamming ball centered at the query. The problem is these methods spend most of search time visiting empty buckets (lookup misses). In this paper, we adopt trie structure to index binary codes. We consider several continuous bits of a binary string as a block and use it as an atomic indexing element in trie structure, which is efficient in access speed and memory usage. Our method searches the nearest neighbours of a query by utilizing the records of nodes in trie structure to avoid lookup misses. We name the proposed indexing structure as Multi-Block N-ary Trie (MBNT). A theoretical analysis is given to prove that MBNT has less time cost than other hash-based methods. Extensive results show that MBNT outperforms state-of-the-art algorithms on several large scale benchmarks.
Compact descriptors for mobile visual search and MPEG CDVS standardization

In this paper, we present the state-of-the-art compact descriptors for mobile visual search. In particular, we introduce our MPEG contributions in global descriptor aggregation and local descriptor compression, which have been adopted by the ongoing MPEG standardization of compact descriptor for visual search (CDVS). Standardization progress will be introduced. Other issues including visual object databases and MPEG CDVS impact on visual search industry will be discussed as well.
Adaptive Weighted Matching of Deep Convolutional Features for Painting Retrieval

We focus on painting retrieval problem, and our motivation is to find out similar paintings and assist painting plagiarism identification. Similar painting retrieval is much more challenging than natural image retrieval, since different paintings have different styles and the similarity of paintings is difficult to measure. In this paper, we define the similarity of paintings from the perspectives of both semantics and structure rather than pixel level color, texture and shape. Specifically, we use the pooling activations of Convolutional Neural Network (CNN) to represent painting features, which preserves both semantic information and structure information. We propose an adaptive weighted matching approach to measure the similarity of paintings, and embed it into a painting retrieval framework. Furthermore, we propose a feature map selection approach to reduce redundancy based on the weights. We collect a new paintings dataset to evaluate painting similarity, which consists of 324 query-reference image pairs from China Artists Association, and 7200 distracted painting images from the Internet, which contains the most common similarity cases. Our approach obtains promising results on the dataset, confirming the superiority of our approach.
When codeword frequency meets geographical location

When codeword frequency meets geographical location in landmark search applications, is it still discriminative for the search procedure. In this paper, we give a systematic investigation about how geographical location affects the effectiveness of codeword frequency. We explain why the standard IDF in the BoW models is less effective in location related search applications [11][12]. Consequently, we propose a “location discriminative codeword frequency” strategy to introduce the location context into the codeword discriminability measurement. This new codeword frequency is calculated in each geographical region, for which a spectral clustering scheme is proposed to partition the geographical map of each city into distinct regions. Extensive comparisons over the standard codeword frequency in state-of-the-art landmark search systems [1][1] demonstrates our approach's effectiveness.
Group-Sensitive Triplet Embedding for Vehicle Reidentification

The widespread use of surveillance cameras toward smart and safe cities poses the critical but challenging problem of vehicle reidentification (Re-ID). The state-of-the-art research work performed vehicle Re-ID relying on deep metric learning with a triplet network. However, most existing methods basically ignore the impact of intraclass variance-incorporated embedding on the performance of vehicle reidentification, in which robust fine-grained features for large-scale vehicle Re-ID have not been fully studied. In this paper, we propose a deep metric learning method, group-sensitive-triplet embedding (GS-TRE), to recognize and retrieve vehicles, in which intraclass variance is elegantly modeled by incorporating an intermediate representation “group” between samples and each individual vehicle in the triplet network learning. To capture the intraclass variance attributes of each individual vehicle, we utilize an online grouping method to partition samples within each vehicle ID into a few groups, and build up the triplet samples at multiple granularities across different vehicle IDs as well as different groups within the same vehicle ID to learn fine-grained features. In particular, we construct a large-scale vehicle database “PKU-Vehicle,” consisting of 10 million vehicle images captured by different surveillance cameras in several cities, to evaluate the vehicle Re-ID performance in real-world video surveillance applications. Extensive experiments over benchmark datasets VehicleID, VeRI, and CompCar have shown that the proposed GS-TRE significantly outperforms the state-of-the-art approaches for vehicle Re-ID.
Real-Time Tracking with Selective DoP-RIEF Features for Augmented Reality

Real-time, accurate and robust target tracking on mobile devices is an important problem which can facilitate applications such as augmented reality. However, it is still unsolved, partly due to the mobile's computing limitations. Compressive tracker performs favorably against state-of-the-art algorithms in terms of efficiency, accuracy and robustness, but as limited by the speed of feature matching, it cannot achieve real-time tracking in mobile applications. In this paper, we propose a fast feature, i.e., Selective Difference of Patch Robust Independent Elementary Features (DoP-RIEF). DoP-RIEF is a global feature which is related to BRIEF. It uses histogram to fit feature distribution because it is more flexible than Gaussian, and intermediate results for subsequent classification can be stored, avoiding duplication of operations. Feature selection further deletes features which are less discriminative and improves the feature quality. Through these two steps, the feature matching can be accelerated significantly and at the same time tracking accuracy and robustness are improved. Compared with compressive tracker on 17 publicly available sequences, our method outperforms it in terms of both robustness and accuracy. In addition, the speed is about 270 frames per second which is 8 times faster than the compressive tracker. To further evaluate our algorithm in natural scenes with obvious scale, rotation, and illumination variations, we test it on Stanford datasets and Peking University landmark datasets, and the accuracy is above 90%.