nils-herrmann/clean_scirepeval huggingface dataset:
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

Single modality action recognition on RGB or depth sequences has been extensively explored recently. It is generally accepted that each of these two modalities has different strengths and limitations for the task of action recognition. Therefore, analysis of the RGB+D videos can help us to better study the complementary properties of these two types of modalities and achieve higher levels of performance. In this paper, we propose a new deep autoencoder based shared-specific feature factorization network to separate input multimodal signals into a hierarchy of components. Further, based on the structure of the features, a structured sparsity learning machine is proposed which utilizes mixed norms to apply regularization within components and group selection between them for better classification performance. Our experimental results show the effectiveness of our cross-modality feature analysis framework by achieving state-of-the-art accuracy for action classification on five challenging benchmark datasets.
Camera response function signature for digital forensics - Part I: Theory and data selection

Camera response function (CRF) is a form of camera signatures which can be extracted from a single image and provides a natural basis for image forensics. CRF extraction from a single-image is in theory ill-posed. It relies on specific structures in an image that offer glimpses of the CRF. Therefore, the challenges in CRF extraction are first in identifying structures of such property, second in locating such structures in an image, and third in extracting the CRF attributes from the selected structures. In our past work, we proposed that CRF attributes can be found on linear structures in an image and extracted using linear geometric invariants. In this paper, we show additional properties on linear geometric invariants, propose a more robust way to select linear structures in an image, and provide a model-based method to extract CRF attributes from the linear structures. This paper is divided into two parts. Part I is devoted to the theory of linear geometric invariants and the robust selection of linear structures. The linear structure candidates obtained from the method in Part I are used to instantiate the edge profiles for CRF extraction in Part II. The paper as a whole presents a reliable method for CRF extraction, together with rigorous analysis which gives useful insights into the method. In the first half of Part I, a simpler proof that links the equality of linear geometric invariants to a linear-isophote surface is given. As a by-product, the proof leads to an additional way to detect linear-isophote surfaces which uses only the first-order partial derivatives and improves detection reliability. In the second half of Part I, the variance of linear geometric invariants is shown to have a structure which can be used to improve the robustness in detecting linear-isophote surfaces.
Multi-modal feature fusion for action recognition in RGB-D sequences

Microsoft Kinect's output is a multi-modal signal which gives RGB videos, depth sequences and skeleton information simultaneously. Various action recognition techniques focused on different single modalities of the signals and built their classifiers over the features extracted from one of these channels. For better recognition performance, it's desirable to fuse these multi-modal information into an integrated set of discriminative features. Most of current fusion methods merged heterogeneous features in a holistic manner and ignored the complementary properties of these modalities in finer levels. In this paper, we proposed a new hierarchical bag-of-words feature fusion technique based on multi-view structured spar-sity learning to fuse atomic features from RGB and skeletons for the task of action recognition.
Rake transform and edge statistics for image forgery detection

In this paper, an effective framework for passive-blind color image forgery detection is proposed. It is a combination of image features extracted from image luminance by applying a rake-transform and from image chroma by using edge statistics. The efficacy of the image features has been tested over two color image datasets established for tampering detection. The proposed framework outweighs the state of the arts over the small-scale dataset, and performs well on the newly established large-scaled dataset (likely the first reported test result on this dataset). The initial tests on some real image forgery cases available in the website and those reported in the literature on image composition with advanced image and vision technologies indicate the promise possessed as well as the challenge faced by the community of image forgery detection.
Calibration of depth cameras using denoised depth images

Depth sensing devices have created various new applications in scientific and commercial research with the advent of Microsoft Kinect and PMD (Photon Mixing Device) cameras. Most of these applications require the depth cameras to be pre-calibrated. However, traditional calibration methods using a checkerboard do not work very well for depth cameras due to the low image resolution. In this paper, we propose a depth calibration scheme which excels in estimating camera calibration parameters when only a handful of corners and calibration images are available. We exploit the noise properties of PMD devices to denoise depth measurements and perform camera calibration using the denoised depth as additional set of measurements. Our synthetic and real experiments show that our depth denoising and depth based calibration scheme provides significantly better results than traditional calibration methods.
Camera response function signature for digital forensics - Part II: Signature extraction

Part I of this two-part paper proposed a robust way to detect local points on linear-isophote surface in an image. Only a subset of these points corresponds to linear surface in image irradiance and provides useful information about the camera response function (CRF). In Part II, we show that, for some images, this subset of linear points could constitute a very small portion of the candidate set and the remaining points are often considered as noise. Our previous approach was to eliminate the noise using a learning-based method. The learning-based method could only reduce the noise but not eliminate it completely. Hence, it fails when the proportion of linear points is too small. As a main contribution in Part II, we introduce the concept of edge profile and consider the candidate points as discrete samples of an edge profile. Instead of eliminating the unwanted candidate points as noise, we use them to instantiate the edge profiles. Assuming that every edge profile has a linear component in image irradiance, the interactions of the edge profiles in the space of linear geometric invariants may correspond to the linear part which is CRF-indicating. Such a model is shown to be sound and effective in both simulation images and real camera images.
A Paradigm for Building Generalized Models of Human Image Perception through Data Fusion

In many sub-fields, researchers collect datasets of human ground truth that are used to create a new algorithm. For example, in research on image perception, datasets have been collected for topics such as what makes an image aesthetic or memorable. Despite high costs for human data collection, datasets are infrequently reused beyond their own fields of interest. Moreover, the algorithms built from them are domain-specific (predict a small set of attributes) and usually unconnected to one another. In this paper, we present a paradigm for building generalized and expandable models of human image perception. First, we fuse multiple fragmented and partially-overlapping datasets through data imputation. We then create a theoretically-structured statistical model of human image perception that is fit to the fused datasets. The resulting model has many advantages. (1) It is generalized, going beyond the content of the constituent datasets, and can be easily expanded by fusing additional datasets. (2) It provides a new ontology usable as a network to expand human data in a cost-effective way. (3) It can guide the design of a generalized computational algorithm for multi-dimensional visual perception. Indeed, experimental results show that a model-based algorithm outperforms state-of-the-art methods on predicting visual sentiment, visual realism and interestingness. Our paradigm can be used in various visual tasks (e.g., video summarization).
Reconstruction of depth and normals from interreflections

While geometry reconstruction has been extensively studied, several shortcomings still exist. First, traditional geometry reconstruction methods such as geometric or photometric stereo only recover either surface depth or normals. Second, such methods require calibration. Third, such methods cannot recover accurate geometry in the presence of interreflections. In order to address these problems in a single system, we propose an approach to reconstruct geometry from light transport data. Specifically, we investigate the problem of geometry reconstruction from interreflections in a light transport matrix. We show that by solving a system of polynomial equations derived directly from the interreflection matrix, both surface depth and normals can be fully reconstructed. Our system does not require projector-camera calibration, but only make use of a calibration object such as a checkerboard in the scene to pre-determine a few known points to simplify the polynomial solver. Our experimental results show that our system is able to reconstruct accurate geometry from interreflections up to a certain noise level. Our system is easy to set up in practice.
Revealing the Trace of High-Quality JPEG Compression Through Quantization Noise Analysis

To identify whether an image has been JPEG compressed is an important issue in forensic practice. The state-of-the-art methods fail to identify high-quality compressed images, which are common on the Internet. In this paper, we provide a novel quantization noise-based solution to reveal the traces of JPEG compression. Based on the analysis of noises in multiple-cycle JPEG compression, we define a quantity called forward quantization noise. We analytically derive that a decompressed JPEG image has a lower variance of forward quantization noise than its uncompressed counterpart. With the conclusion, we develop a simple yet very effective detection algorithm to identify decompressed JPEG images. We show that our method outperforms the state-of-the-art methods by a large margin especially for high-quality compressed images through extensive experiments on various sources of images. We also demonstrate that the proposed method is robust to small image size and chroma subsampling. The proposed algorithm can be applied in some practical applications, such as Internet image classification and forgery detection.
Image Visual Realism: From Human Perception to Machine Computation

Visual realism is defined as the extent to which an image appears to people as a photo rather than computer generated. Assessing visual realism is important in applications like computer graphics rendering and photo retouching. However, current realism evaluation approaches use either labor-intensive human judgments or automated algorithms largely dependent on comparing renderings to reference images. We develop a reference-free computational framework for visual realism prediction to overcome these constraints. First, we construct a benchmark dataset of 2,520 images with comprehensive human annotated attributes. From statistical modeling on this data, we identify image attributes most relevant for visual realism. We propose both empirically-based (guided by our statistical modeling of human data) and deep convolutional neural network models to predict visual realism of images. Our framework has the following advantages: (1) it creates an interpretable and concise empirical model that characterizes human perception of visual realism; (2) it links computational features to latent factors of human image perception.
Statistical Model of JPEG Noises and Its Application in Quantization Step Estimation

In this paper, we present a statistical analysis of JPEG noises, including the quantization noise and the rounding noise during a JPEG compression cycle. The JPEG noises in the first compression cycle have been well studied; however, so far less attention has been paid on the statistical model of JPEG noises in higher compression cycles. Our analysis reveals that the noise distributions in higher compression cycles are different from those in the first compression cycle, and they are dependent on the quantization parameters used between two successive cycles. To demonstrate the benefits from the analysis, we apply the statistical model in JPEG quantization step estimation. We construct a sufficient statistic by exploiting the derived noise distributions, and justify that the statistic has several special properties to reveal the ground-truth quantization step. Experimental results demonstrate that the proposed estimator can uncover JPEG compression history with a satisfactory performance.
COVERAGE — A novel database for copy-move forgery detection

We present COVERAGE - a novel database containing copy-move forged images and their originals with similar but genuine objects. COVERAGE is designed to highlight and address tamper detection ambiguity of popular methods, caused by self-similarity within natural images. In COVERAGE, forged-original pairs are annotated with (i) the duplicated and forged region masks, and (ii) the tampering factor/similarity metric. For benchmarking, forgery quality is evaluated using (i) computer vision-based methods, and (ii) human detection performance. We also propose a novel sparsity-based metric for efficiently estimating forgery quality. Experimental results show that (a) popular forgery detection methods perform poorly over COVERAGE, and (b) the proposed sparsity based metric best correlates with human detection performance. We release the COVERAGE database to the research community.
Smoothly varying affine stitching

Traditional image stitching using parametric transforms such as homography, only produces perceptually correct composites for planar scenes or parallax free camera motion between source frames. This limits mosaicing to source images taken from the same physical location. In this paper, we introduce a smoothly varying affine stitching field which is flexible enough to handle parallax while retaining the good extrapolation and occlusion handling properties of parametric transforms. Our algorithm which jointly estimates both the stitching field and correspondence, permits the stitching of general motion source images, provided the scenes do not contain abrupt protrusions.
New developments in color image tampering detection

In this paper, an efficient framework for passive-blind color image tampering detection is presented. Statistical features are extracted from a given test image and a set of 2-D arrays derived by applying multi-size block discrete cosine transform to the given test image. Image features are extracted from Cr channel, a chroma channel in YCbCr color space, because of its observed sensitivity to color image tampering. A support vector machine is employed to evaluate the effectiveness of image features over a color image dataset recently established for tampering detection. Boosting feature selection is applied to having feature dimensionality reduced so as to make detection accuracy generalizable and computational complexity decreased. Experimental results have demonstrated that the proposed framework applied to the aforementioned dataset outperforms the state of the arts by distinct margins.
Multimodal Multipart Learning for Action Recognition in Depth Videos

The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart features are formulated into the learning framework via the proposed hierarchical mixed norm, to regularize the structured features of each part and to apply sparsity between them, in favor of a group feature selection. Our experimental results expose the effectiveness of the proposed learning method in which it outperforms other methods in all three tested datasets while saturating one of them by achieving perfect accuracy.
Recaptured photo detection using specularity distribution

Detection of planar surfaces in a generic scene is difficult when the illumination is complex and less intense, and the surfaces have non-uniform colors (e.g., a movie poster). As a result, the specularity, if appears, is superimposed with the surface color pattern, and hence the observation of uniform specularity is no longer sufficient for identifying planar surfaces in a generic scene as it does under a distant point light source. In this paper, we address the problem of planar surface recognition in a single generic-scene image. In particular, we study the problem of recaptured photo recognition as an application in image forensics. We discover that the specularity of a recaptured photo is modulated by the mesostructure of the photo surface, and its spatial distribution can be used for differentiating recaptured photos from the original photos. We validate our findings in real images of generic scenes. Experimental results show that there is a distinguishable feature of natural scene and recaptured images. Given the definition of specular ratio as the percentage of specularity in the overall measured intensity, the distribution of specular ratio image's gradient of natural images is Laplacian-like while that of recaptured images is Rayleigh-like.
Factorization towards a classifier

In practice, nonnegative data factorization is often performed for data dimensionality reduction prior to a classification task using a classifier which is effective in low dimensional space such as nearest neighbor classifier. In this work, we propose a novel formulation to learn a multi-class classifier directly through a supervised nonnegative data factorization. This new formulation has the following properties: 1) the nonnegative data matrix is approximated as the product of a nonnegative basis matrix and a coefficient matrix where the nonnegative bases distinctively capture the common characteristics of all classes apart from that specific to individual classes; 2) a regularization term is imposed on nonnegative data factorization so that each datum can be predominantly reconstructed by the common basis vectors and its corresponding class-specific basis vectors; and 3) the coefficient vector for each datum is assumed to be transformed from a mapped kernel space, and the l2 norm of the class-specific coefficients reveals the relative confidence of classes, which then directly leads to a multi-class classifier. We also present an iterative optimization technique for our formulation and analytically show its convergence property. Extensive experiments on face recognition, head pose estimation, and handwritten digit recognition tasks clearly demonstrate the advantages of the proposed classifier over the conventional two-step approach of nonnegative data factorization followed by a classic classifier.
Copy-move forgery detection in the presence of similar but genuine objects

Images often contain Similar but Genuine Objects (SGO), such as two beverage bottles, similar windows, etc. This poses a natural but unexplored challenge to existing copy-move forgery detection methods with an assumption that similar regions are always manipulated for forgery purpose. In this work, we investigate the limitations of the existing CMFD methods under the SGO setting, and propose a new one with performance improvement. Our method consists of the following key steps: first, pyramid scale space and orientation assignment are used for feature extraction to ensure scaling and rotation invariance; second, combined features are applied for effective texture description; third, similar features between two points are matched through RANSAC to reduce false matches; last, tampered regions are located and correlation coefficient are computed on those regions. The experimental results indicate that the proposed algorithm is effective in detecting SGO and copy-move forgery, and compares favorably to existing methods. Furthermore, our method exhibits robustness under geometric transformation and certain forms of post-processing.
Radiometric compensation using stratified inverses

Through radiometric compensation, a projector-camera system can project a desired image onto a non-flat and non-white surface. This can be achieved by computing the inverse light transport of a scene. A light transport matrix is in general large and on the order of 106 × 106 elements. Therefore, computing the inverse light transport matrix is computationally and memory intensive. Two prior methods were proposed to simplify matrix inversion by ignoring scene inter-reflection between individual or clusters of camera pixels. However, compromising scene inter-reflection in spatial domain introduces spatial artifacts and how to systematically adjust the compensation quality is not obvious. In this work, we show how scene inter-reflection can be systematically approximated by stratifying the light transport of a scene. The stratified light transport enables a similar stratification in the inverse light transport. We can show that the stratified inverse light transport converges to the true inverse. For radiometric compensation, the set of stratified inverse light transport provides a systematic way of quantifying the tradeoff between computational efficiency and accuracy. The framework of stratified matrix inversion is general and can have other applications, especially for applications that involve large-size sparse matrices.
Incomplete 3D motion trajectory segmentation and 2D-to-3D label transfer for dynamic scene analysis

The knowledge of the static scene parts and the moving objects in a dynamic scene plays a vital role for scene modelling, understanding, and landmark-based robot navigation. The key information for these tasks lies on semantic labels of the scene parts and the motion trajectories of the dynamic objects. In this work, we propose a method that segments the 3D feature trajectories based on their motion behaviours, and assigns them semantic labels using 2D-to-3D label transfer. These feature trajectories are constructed by using the proposed trajectory recovery algorithm which takes the loss of feature tracking into account. We introduce a complete framework for static-map and dynamic objects' reconstruction, as well as semantic scene understanding for a calibrated and moving 2D-3D camera setup. Our motion segmentation approach is faster by two orders of magnitude, while performing better than the state-of-the-art 3D motion segmentation methods, and successfully handles the previously discarded incomplete trajectory scenarios.
Supershape Recovery from 3D Data Sets

In this paper, we apply supershapes and R-functions to surface recovery from 3D data sets. Individual supershapes are separately recovered from a segmented mesh. R-functions are used to perform Boolean operations between the reconstructed parts to obtain a single implicit equation of the reconstructed object that is used to define a global error reconstruction function. We present surface recovery results ranging from single synthetic data to real complex objects involving the composition of several supershapes and holes.