nils-herrmann/clean_scirepeval huggingface dataset:
An optimal quadtree-based motion estimation and motion-compensated interpolation scheme for video compression

We propose an optimal quadtree (QT)-based motion estimator for video compression. It is optimal in the sense that for a given bit budget for encoding the displacement vector field (DVF) and the QT segmentation, the scheme finds a DVF and a QT segmentation which minimizes the energy of the resulting displaced frame difference (DFD). We find the optimal QT decomposition and the optimal DVF jointly using the Lagrangian multiplier method and a multilevel dynamic program. We introduce a new, very fast convex search for the optimal Lagrangian multiplier lambda(*), which results in a very fast convergence of the Lagrangian multiplier method. The resulting DVF is spatially inhomogeneous, since large blocks are used in areas with simple motion and small blocks in areas with complex motion. We also propose a novel motion-compensated interpolation scheme which uses the same mathematical tools developed for the QT-based motion estimator. One of the advantages of this scheme is the globally optimal control of the tradeoff between the interpolation error energy and the DVF smoothness. Another advantage is that no interpolation of the DVF is required since we directly estimate the DVF and the QT-segmentation for the frame which needs to be interpolated. We present results with the proposed QT-based motion estimator which show that for the same DFD energy the proposed estimator uses about 25% fewer bits than the commonly used block matching algorithm. We also experimentally compare the interpolated frames using the proposed motion compensated interpolation scheme with the reconstructed original frames.
Optical flow estimation from noisy data using differential techniques

Many optical flow estimation techniques are based on the differential optical flow equation. These algorithms involve solving over-determined systems of optical flow equations. Least squares (LS) estimation is usually used to solve these systems even though the underlying noise does not conform to the model implied by LS estimation. To ameliorate this problem, work has been done using the total least squares (TLS) method instead. However, the noise model presumed by TLS is again different from the noise present in the system of optical flow equations. A proper way to solve the system of optical flow equations is the constrained total least squares (CTLS) technique. The derivation and analysis of the CTLS technique for optical flow estimation is presented in this paper. It is shown that CTLS outperforms TLS and LS optical flow estimation.
An optimal polygonal boundary encoding scheme in the rate distortion sense

In this paper, we present fast and efficient methods for the lossy encoding of object boundaries that are given as eight-connect chain codes. We approximate the boundary by a polygon, and consider the problem of finding the polygon which leads to the smallest distortion for a given number of bits. We also address the dual problem of finding the polygon which leads to the smallest bit rate for a given distortion. We consider two different classes of distortion measures. The first class is based on the maximum operator and the second class is based on the summation operator. For the first class, we derive a fast and optimal scheme that is based on a shortest path algorithm for a weighted directed acyclic graph. For the second class we propose a solution approach that is based on the Lagrange multiplier method, which uses the above-mentioned shortest path algorithm. Since the Lagrange multiplier method can only find solutions on the convex hull of the operational rate distortion function, we also propose a tree-pruning-based algorithm that can find all the optimal solutions. Finally, we present results of the proposed schemes using objects from the Miss America sequence.
Audio-visual continuous speech recognition using MPEG-4 compliant visual features

We utilize facial animation parameters (FAPs), supported by the MPEG-4 standard for the visual representation of speech, in order to improve automatic speech recognition (ASR) significantly. We describe a robust and automatic algorithm for extraction of FAPs from visual data that requires no hand labeling or extensive training procedures. Multi-stream hidden Markov models (HMM) are used to integrate audio and visual information. ASR experiments are performed under both clean and noisy audio conditions using a relatively large vocabulary (approximately 1000 words). The proposed system reduces the word error rate (WER) by 20% to 23% relative to audio-only ASR WERs, at various SNRs with additive white Gaussian noise, and by 19% relative to the audio-only ASR WER under clean audio conditions.
Multichannel restoration of single channel images using a wavelet-based subband decomposition

We present a new matrix vector formulation of a wavelet-based subband decomposition. This formulation allows for the decomposition of both the convolution operator and the signal in the subband domain. With this approach, any single channel linear space-invariant filtering problem can be cast into a multichannel framework. We apply this decomposition to the linear space-invariant image restoration problem and propose a family of multichannel linear minimum mean square error (LMMSE) restoration filters. These filters explicitly incorporate both within and between subband (channel) relations of the decomposed image. Since only within channel stationarity is assumed in the image model, this approach presents a new method for modeling the nonstationarity of images. Experimental results are presented which test the proposed multichannel LMMSE filters. These experiments show that if accurate estimates of the subband statistics are available, the proposed multichannel filters provide major improvements over the traditional single channel filters.
An optimal shape encoding scheme using skeleton decomposition

This paper presents an operational rate-distortion (ORD) optimal approach for skeleton-based boundary encoding. The boundary information is first decomposed into skeleton and distance signals, by which a more efficient representation of the original boundary results. Curves of arbitrary order are utilized for approximating the skeleton and distance signals. For a given bit budget for a video frame, we solve the problem of choosing the number and location of the control points for all skeleton and distance signals and for all boundaries within a frame, so that the overall distortion is minimized. The problem is solved with the use of Lagrangian relaxation and a shortest path algorithm in a 4D directed acyclic graph (DAG) we propose. By defining a path selection pattern, we reduce the computational complexity of the 4D DAG shortest path algorithm from O(N/sup -5/) to O(N/sup -4/), where N is the number of admissible control points for a skeleton. A suboptimal solution is also presented for further reducing the computational complexity of the algorithm to O(N/sup -2/). The proposed algorithm outperforms experimentally other competing algorithms.
Variational Gaussian process for sensor fusion

In this paper, we introduce a new Gaussian Process (GP) classification method for multisensory data. The proposed approach can deal with noisy and missing data. It is also capable of estimating the contribution of each sensor towards the classification task. We use Bayesian modeling to build a GP-based classifier which combines the information provided by all sensors and approximates the posterior distribution of the GP using variational Bayesian inference. During its training phase, the algorithm estimates each sensor's weight and then uses this information to assign a label to each new sample. In the experimental section, we evaluate the classiication performance of the proposed method on both synthetic and real data and show its applicability to different scenarios.
Joint optimal object shape estimation and encoding

A major problem in object-oriented video coding and MPEG-4 is the encoding of object boundaries. Traditionally this problem is treated separately from the texture encoding problem. In this paper, we present a vertex-based shape coding method which is optimal in the operational rate-distortion sense and takes into account the texture information of the video frames. This is accomplished by utilizing a variable-width tolerance band whose width is a function of the texture profile. As an example, this width is inversely proportional to the magnitude of the image gradient. Thus, in areas where the confidence in the estimation of the boundary is low and/or coding errors in the boundary will not affect the application (e.g., object-oriented coding and MPEG-4) significantly, a larger boundary approximation error is allowed. We present experimental results which demonstrate the effectiveness of the proposed algorithm.
Joint source coding and packet classification for real-time video transmission over differentiated services networks

Differentiated Services (DiffServ) is one of the leading architectures for providing quality of service in the Internet. We propose a scheme for real-time video transmission over a DiffServ network that jointly considers video source coding, packet classification, and error concealment within a framework of cost-distortion optimization. The selections of encoding parameters and packet classification are both used to manage end-to-end delay variations and packet losses within the network. We present two dual formulations of the proposed scheme: the minimum distortion problem, in which the objective is to minimize the end-to-end distortion subject to cost and delay constraints, and the minimum cost problem, which minimizes the total cost subject to end-to-end distortion and delay constraints. A solution to these problems using Lagrangian relaxation and dynamic programming is given. Simulation results demonstrate the advantage of jointly adapting the source coding and packet classification in DiffServ networks.
Operational rate-distortion optimal bit allocation between shape and texture for MPEG-4 video coding

MPEG-4 is the first multimedia standard that supports the decoupling of a video object into object shape and object texture information, which consequently brings up the optimal encoding problem for object-based video. In this paper, we present an operational rate-distortion optimal bit allocation scheme between shape and texture for MPEG-4 encoding. Our approach is based on the Lagrange multiplier method, while the adoption of dynamic programming techniques enables its higher efficiency over the exhaustive search algorithm. Our work will not only benefit the further study of joint shape and texture encoding, but also make possible the deeper study of optimal joint source-channel coding of object-based video.
Operationally optimal vertex-based shape coding

In this article, we address the issue of operationally optimal shape encoding, which is a step in the direction of globally optimal resource allocation in object-oriented video. After an overview of shape-based coding and algorithms, we define the problem mathematically, introduce the necessary notation, and then present the basic idea behind the proposed algorithms. We then discuss the constraints imposed on the code used to encode the approximation. We then introduce a definition of distortion that fits into the proposed framework and introduce the directed acyclic graph (DAG) formulation of the problem, which results in a fast solution approach. We also show how the DAG algorithm can be used to find the approximation with the minimum-maximum segment distortion for a given rate as well as to find the approximation with the smallest total distortion for a given rate. We then present experimental results and point out directions for future research.
Optimal object-based video communications over differentiated services networks

In this paper, we propose an optimal unequal error protection scheme for object-based video communications over differentiated services networks. Our goal is to achieve the best video quality (minimum total expected distortion) with constraints on transmission cost and delay. An end-to-end distortion estimation approach for object-based video is proposed, which can be used for different packetization schemes. The problem is solved using Lagrangian relaxation and dynamic programming. Experimental results indicate that the proposed unequal error protection schemes can significantly outperform equal error protection methods.
Spatially adaptive high-resolution image reconstruction of DCT-based compressed images

The problem of recovering a high-resolution image from a sequence of low-resolution DCT-based compressed observations is considered in this paper. The introduction of compression complicates the recovery problem. We analyze the DCT quantization noise and propose to model it in the spatial domain as a colored Gaussian process. This allows us to estimate the quantization noise at low bit-rates without explicit knowledge of the original image frame, and we propose a method that simultaneously estimates the quantization noise along with the high-resolution data. We also incorporate a nonstationary image prior model to address blocking and ringing artifacts while still preserving edges. To facilitate the simultaneous estimate, we employ a regularization functional to determine the regularization parameter without any prior knowledge of the reconstruction procedure. The smoothing functional to be minimized is then formulated to have a global minimizer in spite of its nonlinearity by enforcing convergence and convexity requirements. Experiments illustrate the benefit of the proposed method when compared to traditional high-resolution image reconstruction methods. Quantitative and qualitative comparisons are provided.
ADP: Automatic differentiation ptychography

Ptychography is an imaging technique which aims to recover the complex-valued exit wavefront of an object from a set of its diffraction pattern magnitudes. Ptychography is one of the most popular techniques for sub-30 nanometer imaging as it does not suffer from the limitations of typical lens based imaging techniques. The object can be reconstructed from the captured diffraction patterns using iterative phase retrieval algorithms. Over time many algorithms have been proposed for iterative reconstruction of the object based on manually derived update rules. In this paper, we adapt automatic differentiation framework to solve practical and complex ptychographic phase retrieval problems and demonstrate its advantages in terms of speed, accuracy, adaptability and generalizability across different scanning techniques.
Spatially adaptive high-resolution image reconstruction of low-resolution DCT-based compressed images

The problem of recovering a high-resolution image from a sequence of low-resolution DCT-based compressed images is considered. The presence of the compression system complicates the recovery problem, as the operation reduces the amount of frequency aliasing in the low-resolution frames and introduces a non-linear quantization process, The effect of the quantization error and resulting inaccurate sub-pixel motion information is modeled as a zero-mean additive correlated Gaussian noise. A regularization functional is introduced, not only to reflect the relative amount of registration error in each low-resolution image, but also to determine the regularization parameter without any prior knowledge in the reconstruction procedure. The effectiveness of the proposed algorithm is demonstrated experimentally.
Lip tracking for MPEG-4 facial animation

It is very important to accurately track the mouth of a talking person for many applications, such as face recognition and human computer interaction. This is in general a difficult problem due to the complexity of shapes, colors, textures, and changing lighting conditions. We develop techniques for outer and inner lip tracking. From the tracking results FAPs are extracted which are used to drive an MPEG-4 decoder. A novel method consisting of a Gradient Vector Flow (GVF) snake with a parabolic template as an additional external force is proposed. Based on the results of the outer lip tracking, the inner lip is tracked using a similarity function and a temporal smoothness constraint. Numerical results are presented using the Bernstein database.
Retrieval Efficiency of DNA-Based Databases of Digital Signals

Using DNA to store digital signals, or data in general, offers significant advantages when compared to other media. The DNA molecule, especially in its double-stranded form, is very stable, compact, and inexpensive. In the past, we have shown that DNA can be used to store and retrieve digital signals encoded and stored in DNA. We have also shown that DNA hybridization can be used as a similarity criterion for retrieving digital signals encoded and stored in a DNA database. Retrieval is achieved through hybridization of ¿query¿ and ¿data¿ DNA molecules. In this paper, we present a mathematical framework to simulate single-query and parallel-query scenarios, and to estimate hybridization efficiency. Our framework allows for exact numerical solutions as well as closed-form approximations under certain conditions. Similarly to the digital domain, we define a DNA SNR measure to assess the performance of the DNA-based retrieval scheme in terms of database size and source statistics. With approximations, we show that the SNR of any finite-sized DNA-based database is upper bounded by the SNR of an infinitely large DNA-based database that has the same source distribution. Computer simulations are presented to validate our results.
Variational Bayesian inference image restoration using a product of total variation-like image priors

In this paper a new image prior is introduced and used in image restoration. This prior is based on products of spatially weighted Total Variations (TV). These spatial weights provide this prior with the flexibility to better capture local image features than previous TV based priors. Bayesian inference is used for image restoration with this prior via the variational approximation. The proposed algorithm is fully automatic in the sense that all necessary parameters are estimated from the data. Numerical experiments are shown which demonstrate that image restoration based on this prior compares favorably with previous state-of-the-art restoration algorithms.
Audio-Visual Biometrics

Biometric characteristics can be utilized in order to enable reliable and robust-to-impostor-attacks person recognition. Speaker recognition technology is commonly utilized in various systems enabling natural human computer interaction. The majority of the speaker recognition systems rely only on acoustic information, ignoring the visual modality. However, visual information conveys correlated and complimentary information to the audio information and its integration into a recognition system can potentially increase the system's performance, especially in the presence of adverse acoustic conditions. Acoustic and visual biometric signals, such as the person's voice and face, can be obtained using unobtrusive and user-friendly procedures and low-cost sensors. Developing unobtrusive biometric systems makes biometric technology more socially acceptable and accelerates its integration into every day life. In this paper, we describe the main components of audio-visual biometric systems, review existing systems and their performance, and discuss future research and development directions in this area
Joint source-channel coding for scalable video using models of rate-distortion functions

A joint source-channel coding scheme for scalable video is developed in this paper. An SNR scalable video coder is used and unequal error protection (UEP) is allowed for each scalable layer. Our problem is to allocate the available bit rate across scalable layers and, within each layer, between source and channel coding, while minimizing the end-to-end distortion of the received video sequence. The resulting optimization algorithm we propose utilizes universal rate-distortion characteristic plots. These plots show the contribution of each layer to the total distortion as a function of the source rate of the layer and the residual bit error rate (the error rate that remains after the use of channel coding). Models for these plots are proposed in order to reduce the computational complexity of the solution. Experimental results demonstrate the effectiveness of the proposed approach.
Stochastic methods for joint registration, restoration, and interpolation of multiple undersampled images

Using a stochastic framework, we propose two algorithms for the problem of obtaining a single high-resolution image from multiple noisy, blurred, and undersampled images. The first is based on a Bayesian formulation that is implemented via the expectation maximization algorithm. The second is based on a maximum a posteriori formulation. In both of our formulations, the registration, noise, and image statistics are treated as unknown parameters. These unknown parameters and the high-resolution image are estimated jointly based on the available observations. We present an efficient implementation of these algorithms in the frequency domain that allows their application to large images. Simulations are presented that test and compare the proposed algorithms.
Sequential construction of 3-D-based scene description

Binocular camera systems are commonly used to construct 3-D-based scene description. However, there is a tradeoff between the length of the camera baseline and the difficulty of the matching problem and the extent of the field of view of the 3-D scene. A large baseline system provides better depth resolution than a smaller baseline system at the expense of a narrower field of view. To increase the depth resolution without increasing the difficulty of the matching problem and decreasing the field of view of the 3-D scene, a sequential 3-D-based scene description technique is proposed. Multiple small-baseline 3-D scene descriptions from a single moving camera or an array of cameras are used to sequentially construct a large baseline 3-D scene description while maintaining the field of view of a small-baseline system. A Bayesian framework using a disparity-space image (DSI) technique for disparity estimation is presented. The cost function for large baseline image matching is designed based not only on the photometric matching error, the smoothness constraint, and the ordering constraint, but also on the previous disparity estimates from smaller baseline stereo image pairs as a prior model. Texture information is registered along the scan path of the camera(s). Experimental results demonstrate the effectiveness of this technique in visual communication applications.
A rate-distortion optimal coding alternative to matching pursuit

This paper presents a method to find the operational rate-distortion optimal solution for an overcomplete signal decomposition. The idea of using overcomplete dictionaries, or frames, is to get a sparse representation of the signal. Traditionally, suboptimal algorithms, such as Matching Pursuit (MP), is used for this purpose. When using frames in a lossy compression scheme, the major issue is to find the best possible rate-distortion (RD) tradeoff. Given the frame and the Variable Length Code (VLC) table embedded in the entropy coder, the solution of the problem of establishing the best RD tradeoff has a very high complexity. The proposed approach reduces this complexity significantly by structuring the solution approach such that the dependent quantizer allocation problem reduces into an independent one. It is important to note that this large reduction in complexity is achieved without sacrificing optimality. The optimal rate-distortion solution depends on the VLC table embedded in the entropy coder. Thus, VLC optimization is part of this work. We show experimentally that the new approach outperforms Rate-Distortion Optimized Matching Pursuit, previously proposed in .