nils-herrmann/clean_scirepeval huggingface dataset:
Sparse Representation-Based Multiple Frame Video Super-Resolution

In this paper, we propose two multiple-frame super-resolution (SR) algorithms based on dictionary learning (DL) and motion estimation. First, we adopt the use of video bilevel DL, which has been used for single-frame SR. It is extended to multiple frames by using motion estimation with sub-pixel accuracy. We propose a batch and a temporally recursive multi-frame SR algorithm, which improves over single-frame SR. Finally, we propose a novel DL algorithm utilizing consecutive video frames, rather than still images or individual video frames, which further improves the performance of the video SR algorithms. Extensive experimental comparisons with the state-of-the-art SR algorithms verify the effectiveness of our proposed multiple-frame video SR approach.
Rate-distortion optimal skeleton-based shape coding

We present a new shape-coding approach, which decouples the shape information into two independent data sets, the skeleton and the distance of the boundary from the skeleton. The major benefit of this approach is that it allows a more flexible trade-off between accuracy of the approximation and bit-allocation cost, and thus, provides the possibility of better performance in the operational rate-distortion (ORD) optimal sense than other reported techniques. The characteristics of these data sets are studied and various approximation approaches are applied on each of them to reach an ORD optimal result. We apply, for example, polygonal approximation on both the skeleton and distance data. We demonstrate that the resulting approach outperforms existing ORD optimal approaches.
Minimizing transmission energy in wireless video communications

A key constraint in mobile communications is the reliance on a battery with a limited energy supply. Efficiently utilizing the available energy is therefore an important design consideration. We consider a situation where a video sequence is to be compressed and transmitted over a wireless channel. The goal is to limit the amount of distortion in the received video sequence while using the minimum required transmission energy. To accomplish this goal, we consider error resilience and concealment techniques, at the source coding level, as well as the dynamic allocation of physical layer communication resources. We consider these approaches jointly in a novel framework. We formulate an optimization problem that corresponds to minimizing the energy required to transmit a video frame with an acceptable level of distortion. We present methods for solving this problem and other extensions.
Computational Imaging for Cultural Heritage: Recent developments in spectral imaging, 3-D surface measurement, image relighting, and X-ray mapping

Because art is inherently visual, the use of imaging has long been an important way to understand its structure, form, and history. Recently, new ways of engaging with objects from our shared cultural heritage are possible with advances in computation and imaging that allow scientists to analyze art noninvasively, historians to pose new social questions about the art, and the public to explore and interact with art in ways never before possible. There is a rich history in applying image processing techniques to conventional photographic images of works of art, many of which have been highlighted in previous special issues of IEEE Signal Processing Magazine (e.g., the 2008 and 2015 July issues). Building on these contributions, this article comprises a survey of techniques where computation is central to the image acquisition process. Known as computational imaging, the methods being pioneered in this field are increasingly relevant to cultural heritage applications because they leverage advances in image processing, acquisition, and display technologies that make scientific data readily comprehensible to a broad cohort of nontechnical researchers interested in understanding the visual content of art. Presently, only a small research community undertakes computational imaging of cultural heritage. Here we aim to introduce this growing new field to a larger research community by discussing: 1) the historic background of imaging of art, 2) the burgeoning present day community of researchers interested in computational imaging in the arts, and finally, 3) our vision for the future of this new field.
Ptychnet: CNN based fourier ptychography

Fourier ptychography is an imaging technique that overcomes the diffraction limit of conventional cameras with applications in microscopy and long range imaging. Diffraction blur causes resolution loss in both cases. In Fourier ptychography, a coherent light source illuminates an object, which is then imaged from multiple viewpoints. The reconstruction of the object from these set of recordings can be obtained by an iterative phase retrieval algorithm. However, the retrieval process is slow and does not work well under certain conditions. In this paper, we propose a new reconstruction algorithm that is based on convolutional neural networks and demonstrate its advantages in terms of speed and performance.
Automatic facial expression recognition using facial animation parameters and multistream HMMs

The performance of an automatic facial expression recognition system can be significantly improved by modeling the reliability of different streams of facial expression information utilizing multistream hidden Markov models (HMMs). In this paper, we present an automatic multistream HMM facial expression recognition system and analyze its performance. The proposed system utilizes facial animation parameters (FAPs), supported by the MPEG-4 standard, as features for facial expression classification. Specifically, the FAPs describing the movement of the outer-lip contours and eyebrows are used as observations. Experiments are first performed employing single-stream HMMs under several different scenarios, utilizing outer-lip and eyebrow FAPs individually and jointly. A multistream HMM approach is proposed for introducing facial expression and FAP group dependent stream reliability weights. The stream weights are determined based on the facial expression recognition results obtained when FAP streams are utilized individually. The proposed multistream HMM facial expression system, which utilizes stream reliability weights, achieves relative reduction of the facial expression recognition error of 44% compared to the single-stream HMM system.
Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods

Traditionally, analytical methods have been used to solve imaging problems such as image restoration, inpainting, and superresolution (SR). In recent years, the fields of machine and deep learning have gained a lot of momentum in solving such imaging problems, often surpassing the performance provided by analytical approaches. Unlike analytical methods for which the problem is explicitly defined and domain-knowledge carefully engineered into the solution, deep neural networks (DNNs) do not benefit from such prior knowledge and instead make use of large data sets to learn the unknown solution to the inverse problem. In this article, we review deep-learning techniques for solving such inverse problems in imaging. More specifically, we review the popular neural network architectures used for imaging tasks, offering some insight as to how these deep-learning tools can solve the inverse problem. Furthermore, we address some fundamental questions, such as how deeplearning and analytical methods can be combined to provide better solutions to the inverse problem in addition to providing a discussion on the current limitations and future directions of the use of deep learning for solving inverse problem in imaging.
A rate-distortion optimized error control scheme for scalable video streaming over the Internet

Video streaming over the Internet is a challenging task due, in part to the wide range of bandwidth variations caused by network congestion. To deal with this challenge, we propose an optimal error control scheme for scalable video transmission over the Internet. The three major components of error controlerror resilience, forward error correction (FEC), and error concealment- are considered in the proposed framework. Rate-distortion (R-D) optimization is carried out to determine the encoding mode for each packet and the channel coding rates, in order to minimize the overall expected end-to-end distortion. Our simulation study demonstrates that the proposed approach is robust to the wide range channel bandwidth variations and greatly outperforms the classical R-D optimization scheme.
Exploiting temporal correlation in shape coding

This paper investigates ways to explore the between frame correlation of shape information within the framework of an operationally rate-distortion (ORD) optimal coder. Contours are approximated both by connected second-order spline segments, each defined by three consecutive control points, and by segments of the motion-compensated reference contours. Consecutive control points are then encoded predictively using angle and run temporal contexts. We utilize a novel criterion for selecting global object motion vectors, which further improves efficiency. Formulating this problem as Lagrangian minimization, we employ an iterative technique to remove dependency on a particular VLC and jointly arrive at the ORD optimal solution and its underlying conditional parameter distribution.
A bayesian multi-frame image super-resolution algorithm using the Gaussian Information Filter

Multi-frame image super-resolution (SR) is an image processing technology applicable to any digital, pixilated camera that is limited, by construction, to a certain number of pixels. The objective of SR is to utilize signal processing to overcome the physical limitation and emulate the “capabilities” of a camera with a higher-density pixel array. SR is well known to be an ill-posed problem and, consequently, state-of-the-art solutions approach it statistically, typically making use of Bayesian inference. Unfortunately, direct marginalization of the posterior distribution resulting from the Bayesian modeling is not analytically tractable. An approximation method, such as Variational Bayesian Inference (VBI), is a powerful tool that retains the advantages of statistical modeling. However, its derivation is tedious and model specific. In this paper, we propose an alternative approximate inference methodology, based upon the well-established, Gaussian Information Filter, which offers a much simpler mathematical derivation while retaining the statistical advantages of VBI.
Energy-efficient wireless video coding and delivery

Transmitting video over wireless channels from mobile devices has gained increased popularity in a wide range of applications. A major obstacle in these types of applications is the limited energy supply in mobile device batteries. For this reason, efficiently utilizing energy is a critical issue in designing wireless video communication systems. This article highlights recent advances in joint source coding and optimal energy allocation. We present a general framework that takes into account multiple factors, including source coding, channel resource allocation, and error concealment, for the design of energy-efficient wireless video communication systems. This framework can take various forms and be applied to achieve the optimal trade-off between energy consumption and video delivery quality during wireless video transmission.
Super-resolution of compressed videos using convolutional neural networks

Convolutional neural networks (CNN) have been successfully applied to image super-resolution (SR) as well as other image restoration tasks. In this paper, we consider the problem of compressed video super-resolution. Traditional SR algorithms for compressed videos rely on information from the encoder such as frame type or quantizer step, whereas our algorithm only requires the compressed low resolution frames to reconstruct the high resolution video. We propose a CNN that is trained on both the spatial and the temporal dimensions of compressed videos to enhance their spatial resolution. Consecutive frames are motion compensated and used as input to a CNN that provides super-resolved video frames as output. Our network is pretrained with images, which significantly improves the performance over random initialization. In extensive experimental evaluations, we trained the state-of-the-art image and video superresolution algorithms on compressed videos and compared their performance to our proposed method.
Source fidelity over fading channels: erasure codes versus scalable codes

We consider the transmission of a Gaussian source through a block fading channel. Assuming each block is decoded independently, the received distortion depends on the tradeoff between quantization accuracy and probability of outage. Namely, higher quantization accuracy requires a higher channel code rate, which increases the probability of outage. Here we evaluate the received mean distortion with erasure coding across blocks as a function of the code length. We also evaluate the performance of scalable, or multi-resolution coding in which coded layers are superimposed, and the layers are sequentially decoded. In addition to analyzing a finite number of layers, we evaluate the mean distortion at high signal-to-noise ratios as the number of layers becomes infinite. As the block length of the erasure code increases to infinity, the received distortion converges to a deterministic limit, which is less than the mean distortion with an infinite-layer scalable coding scheme. However, for the same standard deviation in received distortion, infinite layer scalable coding performs slightly better than erasure coding
Image and video compression algorithms based on recovery techniques using mean field annealing

Image and video coding algorithms have found a number of applications ranging from video telephony on the public switched telephone networks (PSTN) to HDTV. However, as the bit rate is lowered, most of the existing techniques, as well as current standards, such as JPEG, H. 261, and MPEG-1 produce highly visible degradations in the reconstructed images primarily due to the information loss caused by the quantization process. In this paper, we propose an iterative technique to reduce the unwanted degradations, such as blocking and mosquito artifacts while keeping the necessary detail present in the original image. The proposed technique makes use of a priori information about the original image through a nonstationary Gauss-Markov model. Utilizing this model, a maximum a posteriori (MAP) estimate is obtained iteratively using mean field annealing. The fidelity to the data is preserved by projecting the image onto a constraint set defined by the quantizer at each iteration. The proposed solution represents an implementation of a paradigm we advocate, according to which the decoder is not simply undoing the operations performed by the encoder, but instead it solves an estimation problem based on the available bitstream and any prior knowledge about the source image. The performance of the proposed algorithm was tested on a JPEG, as well as on an H.261-type video codec. It is shown to be effective in removing the coding artifacts present in low bit rate compression.
Recursive map displacement field estimation and its applications

We briefly describe some of our work on the use of stochastic models to describe the displacement vector field (DVF) in an image sequence. Specifically, autoregressive models are used which describe the abrupt transitions in the DVF with the use of a line process, but also result in spatio-temporally recursive structures. The use of such models in developing maximum a posteriori estimators for the DVF and the line process is subsequently described. Finally, the extension and application of the resulting estimator to the problems of object tracking, video compression and restoration of video sequences is reviewed.
PMMW image super resolution from compressed sensing observations

In this paper we propose a novel optimization framework to obtain High Resolution (HR) Passive Millimeter Wave (P-MMW) images from multiple Low Resolution (LR) observations captured using a simulated Compressed Sensing (CS) imaging system. The proposed CS Super Resolution (CSS-R) approach combines existing CS reconstruction algorithms with the use of Super Gaussian (SG) regularization terms on the image to be reconstructed, smoothness constraints on the registration parameters to be estimated and the use of the Alternate Direction Methods of Multipliers (ADMM) to link the CS and SR problems. The image estimation subproblem is solved using Majorization-Minimization (MM), registration is tackled minimizing a quadratic function and CS reconstruction is approached as an l1-minimization problem subject to a quadratic constraint. The performed experiments, on simulated and real PMMW observations, validate the used approach.
Motion compensated shape error concealment

The introduction of Video Objects (VOs) is one of the innovations of MPEG-4. The /spl alpha/-plane of a VO defines its shape at a given instance in time and hence determines the boundary of its texture. In packet-based networks, shape, motion, and texture are subject to loss. While there has been considerable attention paid to the concealment of texture and motion errors, little has been done in the field of shape error concealment. In this paper we propose a post-processing shape error concealment technique that uses the motion compensated boundary information of the previously received /spl alpha/-plane. The proposed approach is based on matching received boundary segments in the current frame to the boundary in the previous frame. This matching is achieved by finding a maximally smooth motion vector field. After the current boundary segments are matched to the previous boundary, the missing boundary pieces are reconstructed by motion compensation. Experimental results demonstrating the performance of the proposed motion compensated shape error concealment method, and comparing it with the previously proposed weighted side matching method are presented.
Dense Disparity Estimation with a Divide-and-Conquer Disparity Space Image Technique

A new divide-and-conquer technique for disparity estimation is proposed in this paper. This technique performs feature matching following the high confidence first principle, starting with the strongest feature point in the stereo pair of scanlines. Once the first matching pair is established, the ordering constraint in disparity estimation allows the original intra-scanline matching problem to be divided into two smaller subproblems. Each subproblem can then be solved recursively until there is no reliable feature point within the subintervals. This technique is very efficient for dense disparity map estimation for stereo images with rich features. For general scenes, this technique can be paired up with the disparity-space image (DSI) technique to compute dense disparity maps with integrated occlusion detection. In this approach, the divide-and-conquer part of the algorithm handles the matching of stronger features and the DSI-based technique handles the matching of pixels in between feature points and the detection of occlusions. An extension to the standard disparity-space technique is also presented to compliment the divide-and-conquer algorithm. Experiments demonstrate the effectiveness of the proposed divide-and-conquer DSI algorithm.
Maximum likelihood blur identification and image restoration using the EM algorithm

The authors describe an algorithm for the identification of the blur and the restoration of a noisy blurred image. The original image and the additive noise are modeled as zero-mean Gaussian random processes. Their covariance matrices are unknown parameters. The blurring process is specified by its point spread function, which is also unknown. Maximum likelihood estimation is used to find these unknown parameters. In turn, the expectation-maximization (EM) algorithm is exploited in computing the maximum likelihood estimates. In applying the EM algorithm, the original image is part of the complete data; its estimate is computed in the E-step of the EM iterations. Explicit iterative expressions are derived for the estimation. Experimental results on simulated and photographically blurred images are shown.
Resampling for Spatial Scalability

Resampling is a fundamental issue in the design of a spatially scalable video codec. The resampling procedure is responsible for down-sampling the high-resolution video sequence to generate lower resolution data, as well as upsampling the transmitted lower resolution data to predict the original high-resolution frames. In both cases, the resampling operation must make trade-offs between coding efficiency, image quality and computational complexity. In this paper, we consider the resampling design problem within an optimization framework.
Regularized motion estimation using robust entropic functionals

In this paper, regularized estimation of the displacement vector field (DVF) of a dynamic image sequence is considered. A new class of non-quadratic convex regularization functionals is employed to estimate the motion field in the presence of motion discontinuities and occlusions. The derivation of the functionals is based on entropy considerations and does not require parameter tuning as in previously proposed methods. This new class of functionals is both robust and convex making it possible to preserve motion boundaries and obtain a globally optimum solution. The performance of entropic functionals is compared to previously suggested functionals for motion estimation using real and synthetic image sequences.
Packet Scheduling for Scalable Video Streaming Over Lossy Packet Access Networks

Video streaming applications have gained in popularity in recent years. The quality of service offered by such applications is limited by the available transmission rates as well as time-varying conditions, such as, channel fading and network congestion, which lead to packet losses. Scalable video coding techniques that allow for the flexible adaptation of temporal resolution as well as quality of an encoded bitstream can be immensely useful in developing video streaming applications that can adapt to time-varying network and channel conditions. Scalable coding techniques, however, are generally designed to offer progressive refinement, which introduces dependencies between encoded video packets. Therefore, when determining a packet scheduling technique for scalable coded video, the possibility of random packet losses, which might affect the decodability of subsequent packets, must be taken into account. In this paper, we take into account the available transmission rate, possibly time-varying channel conditions, and the possibility of random packet losses, to design a scheduling technique for video packets in a scalable bit-stream. Since the optimal solution to the scheduling problem requires an exhaustive, and therefore, intractable computation, we propose a greedy algorithm that will schedule the optimal packet for transmission at a given transmission opportunity based on the encoded content and the available channel state information. Simulation results show significant gains in performance when the proposed technique is compared to content and channel independent packet scheduling techniques.