nils-herrmann/clean_scirepeval huggingface dataset:
2D/3D freeview video generation for 3DTV system

In this paper, we propose a new approach of synthesizing novel views in multiview camera configurations. We introduce the semi iV-view & iV-depth framework in order to estimate disparity maps efficiently and correctly. This framework reduces redundancy on disparity estimation by using information from neighboring views. The occlusion problem is handled by using cost functions computed with multiview images. The proposed method provides a 2D/3D freeview video. User can select 2D/3D modes of freeview video and control 3D depth perception by adjusting several parameters in 3D freeview video. Experimental results show that the proposed method yields the accurate disparity maps and provides seamless freeview videos.
Freeview rendering with trinocular camera

The paper presents a method for synthesizing novel view from the virtual camera with trinocular camera configuration. We propose a cost aggregation method with weighted least square for stereo matching, and address the occlusion problem by using cost functions computed with multiview images. We avoid the redundancy of estimating disparity maps for all the images by using simple geometry transfer method. The novel view is synthesized by view-dependent geometries on the 3D translation of virtual camera. Experimental results show that the proposed method yields the accurate disparity maps and the synthesized novel view is satisfactory enough to provide a viewer freeview videos.
DCTM: Discrete-Continuous Transformation Matching for Semantic Flow

Techniques for dense semantic correspondence have provided limited ability to deal with the geometric variations that commonly exist between semantically similar images. While variations due to scale and rotation have been examined, there is a lack of practical solutions for more complex deformations such as affine transformations because of the tremendous size of the associated solution space. To address this problem, we present a discrete-continuous transformation matching (DCTM) framework where dense affine transformation fields are inferred through a discrete label optimization in which the labels are iteratively updated via continuous regularization. In this way, our approach draws solutions from the continuous space of affine transformations in a manner that can be computed efficiently through constant-time edge-aware filtering and a proposed affine-varying CNN-based descriptor. Experimental results show that this model outperforms the state-of-the-art methods for dense semantic correspondence on various benchmarks.
Patch Match Filter: Efficient Edge-Aware Filtering Meets Randomized Search for Fast Correspondence Field Estimation

Though many tasks in computer vision can be formulated elegantly as pixel-labeling problems, a typical challenge discouraging such a discrete formulation is often due to computational efficiency. Recent studies on fast cost volume filtering based on efficient edge-aware filters have provided a fast alternative to solve discrete labeling problems, with the complexity independent of the support window size. However, these methods still have to step through the entire cost volume exhaustively, which makes the solution speed scale linearly with the label space size. When the label space is huge, which is often the case for (sub pixel-accurate) stereo and optical flow estimation, their computational complexity becomes quickly unacceptable. Developed to search approximate nearest neighbors rapidly, the Patch Match method can significantly reduce the complexity dependency on the search space size. But, its pixel-wise randomized search and fragmented data access within the 3D cost volume seriously hinder the application of efficient cost slice filtering. This paper presents a generic and fast computational framework for general multi-labeling problems called Patch Match Filter (PMF). For the very first time, we explore effective and efficient strategies to weave together these two fundamental techniques developed in isolation, i.e., Based-based randomized search and efficient edge-aware image filtering. By decompositing an image into compact super pixels, we also propose super pixel-based novel search strategies that generalize and improve the original Patch Match method. Focusing on dense correspondence field estimation in this paper, we demonstrate PMF's applications in stereo and optical flow. Our PMF methods achieve state-of-the-art correspondence accuracy but run much faster than other competing methods, often giving over 10-times speedup for large label space cases.
FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondence

We present a descriptor, called fully convolutional self-similarity (FCSS), for dense semantic correspondence. To robustly match points among different instances within the same object class, we formulate FCSS using local self-similarity (LSS) within a fully convolutional network. In contrast to existing CNN-based descriptors, FCSS is inherently insensitive to intra-class appearance variations because of its LSS-based structure, while maintaining the precise localization ability of deep neural networks. The sampling patterns of local structure and the self-similarity measure are jointly learned within the proposed network in an end-to-end and multi-scale manner. As training data for semantic correspondence is rather limited, we propose to leverage object candidate priors provided in existing image datasets and also correspondence consistency between object pairs to enable weakly-supervised learning. Experiments demonstrate that FCSS outperforms conventional handcrafted descriptors and CNN-based descriptors on various benchmarks.
2D-plus-depth based resolution and frame-rate up-conversion technique for depth video

We propose a novel framework for upconversion of depth video resolution in both spatial and time domains considering spatial and temporal coherences. Although the Time-of-Flight (TOF) sensor which is widely used in computer vision fields provides depth video in realtime, it also provides a low resolution and a low frame-rate depth video. We propose a cheaper solution that enhances depth video obtained from a TOF sensor by combining it with a Charge-coupled Device (CCD) camera in 3D contents which consist of 2D-plus-depth. Temporal fluctuation problems are also considered for temporally consistent framerate up-conversion. It is important to maintain temporal coherence in depth video, because temporal fluctuation problems may cause eye fatigue and increase bit rates on video coding. We propose a Motion Compensated Frame Interpolation (MCFI) using reliable and rich motion information from a CCD camera and 3-dimensional Joint Bilateral Up-sampling (3D JBU) extended into the temporal domain of depth video. Experimental results show that depth video obtained by the proposed method provides satisfactory quality.
Deep Monocular Depth Estimation via Integration of Global and Local Predictions

Recent works on machine learning have greatly advanced the accuracy of single image depth estimation. However, the resulting depth images are still over-smoothed and perceptually unsatisfying. This paper casts depth prediction from single image as a parametric learning problem. Specifically, we propose a deep variational model that effectively integrates heterogeneous predictions from two convolutional neural networks (CNNs), named global and local networks. They have contrasting network architecture and are designed to capture the depth information with complementary attributes. These intermediate outputs are then combined in the integration network based on the variational framework. By unrolling the optimization steps of Split Bregman iterations in the integration network, our model can be trained in an end-to-end manner. This enables one to simultaneously learn an efficient parameterization of the CNNs and hyper-parameter in the variational method. Finally, we offer a new data set of 0.22 million RGB-D images captured by Microsoft Kinect v2. Our model generates realistic and discontinuity-preserving depth prediction without involving any low-level segmentation or superpixels. Intensive experiments demonstrate the superiority of the proposed method in a range of RGB-D benchmarks, including both indoor and outdoor scenarios.
PatchMatch Filter: Edge-Aware Filtering Meets Randomized Search for Visual Correspondence

Though many tasks in computer vision can be formulated elegantly as pixel-labeling problems, a typical challenge discouraging such a discrete formulation is often due to computational efficiency. Recent studies on fast cost volume filtering based on efficient edge-aware filters provide a fast alternative to solve discrete labeling problems, with the complexity independent of the support window size. However, these methods still have to step through the entire cost volume exhaustively, which makes the solution speed scale linearly with the label space size. When the label space is huge or even infinite, which is often the case for (subpixel-accurate) stereo and optical flow estimation, their computational complexity becomes quickly unacceptable. Developed to search approximate nearest neighbors rapidly, the PatchMatch method can significantly reduce the complexity dependency on the search space size. But, its pixel-wise randomized search and fragmented data access within the 3D cost volume seriously hinder the application of efficient cost slice filtering. This paper presents a generic and fast computational framework for general multi-labeling problems called PatchMatch Filter (PMF). We explore effective and efficient strategies to weave together these two fundamental techniques developed in isolation, i.e., PatchMatch-based randomized search and efficient edge-aware image filtering. By decompositing an image into compact superpixels, we also propose superpixel-based novel search strategies that generalize and improve the original PatchMatch method. Further motivated to improve the regularization strength, we propose a simple yet effective cross-scale consistency constraint, which handles labeling estimation for large low-textured regions more reliably than a single-scale PMF algorithm. Focusing on dense correspondence field estimation in this paper, we demonstrate PMF’s applications in stereo and optical flow. Our PMF methods achieve top-tier correspondence accuracy but run much faster than other related competing methods, often giving over 10-100 times speedup.
ANCC flow: Adaptive normalized cross-correlation with evolving guidance aggregation for dense correspondence estimation

Adaptive normalized cross-correlation (ANCC) cost function works well between images under photometric distortions, but its heavy computational burden often limits its applications. To overcome this limitation, this paper proposes a robust and efficient computational framework, called ANCC flow, designed for establishing dense correspondences between images under severe photometric variations. We first simplify the weight of ANCC in an asymmetric manner by considering a source image weight only. It is then efficiently computed by applying constant-time edge-aware filters without loss of its matching accuracy. Additionally, to deal with a large discrete label space effectively, which is a challenging issue in a flow field estimation, we propose a randomized label space sampling strategy similar to PatchMatch filer (PMF) optimization. The robustness of the asymmetric ANCC and the cost filter is further enhanced through an evolving weight computation, where a flow field computed in a previous iteration is utilized to build current edge-aware weights. Experimental results demonstrate the outstanding performance of ANCC flow in many cases of dense correspondence estimations under severe photometric and geometric variations.
Virtual View Rendering System for 3DTV

In this paper, we propose a new approach for efficient multiview stereo matching and virtual view generation, which are key technologies for 3D TV. We propose semi N-view & N-depth framework to estimate disparity maps efficiently and correctly. This framework reduces the redundancy on disparity estimation by using the information of neighboring views. The proposed method provides a user 2D/3D freeview video, and the user can select 2D/3D modes of free- view video. Experimental results show that the proposed method yields the accurate disparity maps and the synthesized novel view is satisfactory enough to provide user seamless freeview videos.
3D JBU based depth video filtering for temporal fluctuation reduction

In this paper, we propose a three-dimensional Joint Bilateral Up-sampling (3D JBU) for the depth video which can be applied to a 3DTV system based on 2D-plus-depth video. Recently, a number of researches have been done to improve a resolution and frame-rate of the depth video. We proposed a novel method that enhances depth video obtained by Time-of-Flight (TOF) sensor by combining it with Charge-coupled Device (CCD) camera . However, this method does not consider the temporal coherence of the depth video. It may cause an eye fatigue on the 3D display and increase bit rates on video coding, since it is possible to generate a temporal fluctuation problem. Therefore, in order to solve these problems, we propose a 3D JBU model which is extended conventional JBU into the temporal domain of depth video. Experimental results show that depth video obtained by the proposed method provides satisfactory quality.
Revisiting the Relationship Between Adaptive Smoothing and Anisotropic Diffusion With Modified Filters

Anisotropic diffusion has been known to be closely related to adaptive smoothing and discretized in a similar manner. This paper revisits a fundamental relationship between two approaches. It is shown that adaptive smoothing and anisotropic diffusion have different theoretical backgrounds by exploring their characteristics with the perspective of normalization, evolution step size, and energy flow. Based on this principle, adaptive smoothing is derived from a second order partial differential equation (PDE), not a conventional anisotropic diffusion, via the coupling of Fick's law with a generalized continuity equation where a “source” or “sink” exists, which has not been extensively exploited. We show that the source or sink is closely related to the asymmetry of energy flow as well as the normalization term of adaptive smoothing. It enables us to analyze behaviors of adaptive smoothing, such as the maximum principle and stability with a perspective of a PDE. Ultimately, this relationship provides new insights into application-specific filtering algorithm design. By modeling the source or sink in the PDE, we introduce two specific diffusion filters, the robust anisotropic diffusion and the robust coherence enhancing diffusion, as novel instantiations which are more robust against the outliers than the conventional filters.
LETRIST: Locally Encoded Transform Feature Histogram for Rotation-Invariant Texture Classification

Classifying texture images, especially those with significant rotation, illumination, scale, and viewpoint changes, is a fundamental and challenging problem in computer vision. This paper proposes a simple yet effective image descriptor, called Locally Encoded TRansform feature hISTogram (LETRIST), for texture classification. LETRIST is a histogram representation that explicitly encodes the joint information within an image across feature and scale spaces. The proposed representation is training-free, low-dimensional, yet discriminative and robust for texture description. It consists of the following major steps. First, a set of transform features is constructed to characterize local texture structures and their correlation by applying linear and non-linear operators on the extremum responses of directional Gaussian derivative filters in scale space. Established on the basis of steerable filters, the constructed transform features are exactly rotationally invariant as well as computationally efficient. Second, the scalar quantization via binary or multi-level thresholding is adopted to quantize these transform features into texture codes. Two quantization schemes are designed, both of which are robust to image rotation and illumination changes. Third, the cross-scale joint coding is explored to aggregate the discrete texture codes into a compact histogram representation, i.e., LETRIST. Experimental results on the Outex, CUReT, KTH-TIPS, and UIUC texture data sets show that LETRIST consistently produces better or comparable classification results than the state-of-the-art approaches. Impressively, recognition rates of 100.00% and 99.00% have been achieved on the Outex and KTH-TIPS data sets, respectively. In addition, the noise robustness is evaluated on the Outex and CUReT data sets. The source code is publicly available at https://github.com/stc-cqupt/letrist.
Optimal solution for the index coding problem using network coding over GF(2)

The index coding problem is a fundamental transmission problem which occurs in a wide range of multicast networks. Network coding over a large finite field size has been shown to be a theoretically efficient solution to the index coding problem. However the high computational complexity of packet encoding and decoding over a large finite field size, and its subsequent penalty on encoding and decoding throughput and higher energy cost makes it unsuitable for practical implementation in processor and energy constraint devices like mobile phones and wireless sensors. While network coding over GF(2) can alleviate these concerns, it comes at a tradeoff cost of degrading throughput performance. To address this tradeoff, we propose a throughput optimal triangular network coding scheme over GF(2). We show that such a coding scheme can supply unlimited number of innovative packets and the decoding involves the simple back substitution. Such a coding scheme provides an efficient solution to the index coding problem and its lower computation and energy cost makes it suitable for practical implementation on devices with limited processing and energy capacity.
Unsupervised Joint Feature Learning and Encoding for RGB-D Scene Labeling

Most existing approaches for RGB-D indoor scene labeling employ hand-crafted features for each modality independently and combine them in a heuristic manner. There has been some attempt on directly learning features from raw RGB-D data, but the performance is not satisfactory. In this paper, we propose an unsupervised joint feature learning and encoding (JFLE) framework for RGB-D scene labeling. The main novelty of our learning framework lies in the joint optimization of feature learning and feature encoding in a coherent way, which significantly boosts the performance. By stacking basic learning structure, higher level features are derived and combined with lower level features for better representing RGB-D data. Moreover, to explore the nonlinear intrinsic characteristic of data, we further propose a more general joint deep feature learning and encoding (JDFLE) framework that introduces the nonlinear mapping into JFLE. The experimental results on the benchmark NYU depth dataset show that our approaches achieve competitive performance, compared with the state-of-the-art methods, while our methods do not need complex feature handcrafting and feature combination and can be easily applied to other data sets.
Hybrid Collision-Free Medium Access (HCFMA) Protocol for Underwater Acoustic Networks: Design and Performance Evaluation

Underwater acoustic channels are typically characterized by long propagation delay, time-varying multipath, as well as very limited bandwidth, thus making it very challenging to develop efficient underwater medium access control (MAC) protocols. In this paper, we propose a new MAC protocol for underwater acoustic networks (UANs), called the hybrid collision-free medium access (HCFMA) scheme. The operation of the HCFMA scheme is based on the combination of the multichannel technique, the channel reservation mechanism, and the juggling-like stop-and-wait (JSW) transmission scheme. With the multichannel technique, the exchange of control signaling and data transmission can be separated, consequently, increasing the throughput of the network. The channel reservation mechanism which is carried out by the temporarily chosen control nodes guarantees no collision in data transmission. This is different from those whose channel reservations rely on a dedicated central controller. The JSW scheme provides an error-control mechanism during data transmission. Together, these approaches greatly improve the performance of UAN in throughput and delay. This paper describes the operation of the HCFMA scheme and analyzes its performance. We further conduct simulation to validate our analytical results. Our results show encouraging throughput and delay performance compared with other popular solutions, especially under heavy load conditions.
Single-Pass Rate-Smoothed Video Encoding With Quality Constraint

In this letter, we study the rate smoothing problem in single-pass video encoding, i.e., given a certain video quality constraint, how to smooth out the traffic rate so that the complexity of the video delivery can be reduced. We apply the low-pass filtering idea, originally proposed for single-pass quality-smoothed video encoding, into the problem of rate-smoothed video encoding. In particular, we use the arithmetic averaging filter to smooth out the rate during single-pass video encoding. Both theoretical analysis and experimental results show that our proposed scheme can not only smooth out the bit rate but also automatically achieve the targeted average video quality.
Optimizing Quality of Experience for Adaptive Bitrate Streaming via Viewer Interest Inference

Rate adaptation is widely adopted in video streaming to improve the quality of experience (QoE). However, most of the existing rate adaptation approaches neglect the underlying video semantic information. In fact, influenced by video semantics and viewer preferences, the viewer may have different degrees of interest on different parts of a video. The interesting parts of a video can draw more visual attention from the viewer and have higher visual importance. As such, delivering the parts of a video that are interesting to the viewer in a higher quality can improve the perceptual video quality, compared with the semantics-agnostic approaches that treat each part of a video equally. Thus, it is natural to wonder: how to allocate bitrate budgets temporally over a video session under time-varying bandwidth while considering viewer interest? As an exploratory study, we propose an interest-aware rate adaptation approach for improving QoE by inferring viewer interest based on video semantics. We adopt the deep learning method to recognize the scenes of video frames and leverage the term frequency-inverse document frequency method to analyze the degrees of an individual viewer's interest on different types of scenes. The bandwidth, buffer occupancy, and viewer interest are jointly considered under the model predictive control framework for selecting appropriate bitrates for maximizing QoE. The objective and subjective evaluations measured in a real environment show that our method can achieve a higher QoE compared with the semantics-agnostic approaches.