nils-herrmann/clean_scirepeval huggingface dataset:
Maximum Likelihood Sound Source Localization for Multiple Directional Microphones

This paper presents a maximum likelihood (ML) framework for multi-microphone sound source localization (SSL). Besides deriving the framework, we focus on making the connection and contrast between the ML-based algorithm and popular steered response power (SRP) SSL algorithms such as phase transform (SRP-PHAT). We also show under our ML framework how challenging conditions such as directional microphone arrays and reverberations can be handled. The computational cost of our method is low-similar to SRP-PHAT. The effectiveness of the proposed method is shown on a large dataset with 99 real-world audio sequences recorded by directional circular microphone arrays in over 50 different meeting rooms.
A novel see-through screen based on weave fabrics

See-through screens (STS) have found important applications in remote collaboration systems to enhance non-verbal communication and gaze awareness. Existing STS designs often sacrifice the display quality significantly, rendering low-contrast images that discount the overall user experience. In this paper, we present a novel see-through screen solution based on weave fabrics. Such fabrics are known to be acoustically transparent and used to build professional projection screens for Hollywood studios. We place a cam-era immediately behind the screen and synchronize it with a 120Hz projector to perform time-multiplexing display and video capture. By focusing the camera at the user 4–5 feet away from the screen, the image of the weave fabric will be severely blurred. We present the imaging principle of the setup, and derive image processing techniques to enhance the quality of the captured video. The overall system is low cost, has much better display quality than existing systems, and can be used to build wall-size see-through screens for various applications.
Improving Immersive Experiences in Telecommunication with Motion Parallax [Applications Corner]

Human sensorial perception of the surrounding environment is very intricate and only partially understood. Visual depth perception, for example, is often attributed to stereo vision. However, if we close one eye, it becomes immediately obvious how much depth information we acquire from other mechanisms. One of these mechanisms is motion parallax, i.e., the fact that the relative apparent positions of objects change when we move our viewpoint. Traditional threedimensional television (3DTV) systems provide carefully edited stereo video but often lack the capability of rendering any motion parallax. Similar to the visual experience, our audio experience is also affected by our body motion. Indeed, in addition to information about the environment captured by two ears with a fixed head position, we also capture (often, unconsciously) a richness of information about the environment by moving our heads to sample the sound field around us.
L1 regularized room modeling with compact microphone arrays

Acoustic room modeling has several applications. Recent results using large microphone arrays show good performance, and are helpful in many applications. For example, when designing a better acoustic treatment for a concert hall, these large arrays can be used to help map the acoustic environment and aid in the design. However, in real-time applications - including de-reverberation, sound source localization, speech enhancement and 3D audio - it is desirable to model the room with existing small arrays and existing loudspeakers. In this paper we propose a novel room modeling algorithm, which uses a constrained room model and ℓ1-regularized least-squares to achieve good estimation of room geometry. We present experimental results on both real and synthetic data.
Boosted multi-task learning for face verification with applications to web image and video search

Face verification has many potential applications including filtering and ranking image/video search results on celebrities. Since these images/videos are taken under uncontrolled environments, the problem is very challenging due to dramatic lighting and pose variations, low resolutions, compression artifacts, etc. In addition, the available number of training images for each celebrity may be limited, hence learning individual classifiers for each person may cause overfitting. In this paper, we propose two ideas to meet the above challenges. First, we propose to use individual bins, instead of whole histograms, of Local Binary Patterns (LBP) as features for learning, which yields significant performance improvements and computation reduction in our experiments. Second, we present a novel Multi-Task Learning (MTL) framework, called Boosted MTL, for face verification with limited training data. It jointly learns classifiers for multiple people by sharing a few boosting classifiers in order to avoid overfitting. The effectiveness of Boosted MTL and LBP bin features is verified with a large number of celebrity images/videos from the web.
Towards optimal least square filters using the eigenfilter approach

In this paper, we propose a new eigenfilter approach to designing least square error filters. The filters are obtained by finding an eigenvector of a real, symmetric and positive definite matrix, which is numerically stable. The proposed algorithm has two advantages. First, we show that the least-square solution, which can only be obtained through matrix inversion in the literature, can be asymptotically reached with our algorithm. Second, when numerical errors break. the matrix inversion method, our algorithm can still find some ìoptimalî filter through tuning an internal parameter.
Boosting-Based Multimodal Speaker Detection for Distributed Meeting Videos

Identifying the active speaker in a video of a distributed meeting can be very helpful for remote participants to understand the dynamics of the meeting. A straightforward application of such analysis is to stream a high resolution video of the speaker to the remote participants. In this paper, we present the challenges we met while designing a speaker detector for the Microsoft RoundTable distributed meeting device, and propose a novel boosting-based multimodal speaker detection (BMSD) algorithm. Instead of separately performing sound source localization (SSL) and multiperson detection (MPD) and subsequently fusing their individual results, the proposed algorithm fuses audio and visual information at feature level by using boosting to select features from a combined pool of both audio and visual features simultaneously. The result is a very accurate speaker detector with extremely high efficiency. In experiments that includes hundreds of real-world meetings, the proposed BMSD algorithm reduces the error rate of SSL-only approach by 24.6%, and the SSL and MPD fusion approach by 20.9%. To the best of our knowledge, this is the first real-time multimodal speaker detection algorithm that is deployed in commercial products.
Point cloud attribute compression with graph transform

Compressing attributes on 3D point clouds such as colors or normal directions has been a challenging problem, since these attribute signals are unstructured. In this paper, we propose to compress such attributes with graph transform. We construct graphs on small neighborhoods of the point cloud by connecting nearby points, and treat the attributes as signals over the graph. The graph transform, which is equivalent to Karhunen-Loève Transform on such graphs, is then adopted to decorrelate the signal. Experimental results on a number of point clouds representing human upper bodies demonstrate that our method is much more efficient than traditional schemes such as octree-based methods.
Semantic saliency driven camera control for personal remote collaboration

This paper presents a camera combo system for personal remote collaboration applications. The system consists of two different cameras. One camera has a wide field of view, and the other can pan/tilt/zoom (PTZ) based on analysis of the images captured by the wide angle camera. Unlike traditional approaches which usually drive the PTZ camera to follow the person or his/her head, our system is capable of capturing general objects of interest in remote collaboration. For instance, when the user raises something trying to show it to the remote person, our system will automatically position the PTZ camera to zoom in at the object. At the core of our system is a semantic saliency map that overcomes many limitations of low-level saliency maps computed from preliminary image features. We demonstrate how such a semantic saliency map can be computed through contextual analysis, sign analysis and transitional analysis, and how it can be used for PTZ camera control with a novel information loss optimization based virtual director. The effectiveness of the proposed method is demonstrated with real-world sequences.
Turning enemies into friends: Using reflections to improve sound source localization

Sound Source Localization (SSL) based on microphone arrays has numerous applications, and has received significant research attention. Common to all published research is the observation that the accuracy of SSL degrades with reverberation. Indeed, early (strong) reflections can have amplitudes similar to the direct signal, and will often interfere with the estimation. In this paper, we show that reverberation is not the enemy, and can be used to improve estimation. More specifically, we are able to use early reflections to significantly improve range and elevation estimation. The process requires two steps: during setup, a loudspeaker integrated with the array emits a probing sound, which is used to obtain estimates of the ceiling height, as well as the locations of the walls. In a second step (e.g., during a meeting), the device incorporates this knowledge into a maximum likelihood SSL algorithm. Experimental results on both real and synthetic data show huge improvements in range estimation accuracy.
CROWDMOS: An approach for crowdsourcing mean opinion score studies

MOS (mean opinion score) subjective quality studies are used to evaluate many signal processing methods. Since laboratory quality studies are time consuming and expensive, researchers often run small studies with less statistical significance or use objective measures which only approximate human perception. We propose a cost-effective and convenient measure called crowdMOS, obtained by having internet users participate in a MOS-like listening study. Workers listen and rate sentences at their leisure, using their own hardware, in an environment of their choice. Since these individuals cannot be supervised, we propose methods for detecting and discarding inaccurate scores. To automate crowdMOS testing, we offer a set of freely distributable, open-source tools for Amazon Mechanical Turk, a platform designed to facilitate crowdsourcing. These tools implement the MOS testing methodology described in this paper, providing researchers with a user-friendly means of performing subjective quality evaluations without the overhead associated with laboratory studies. Finally, we demonstrate the use of crowdMOS using data from the Blizzard text-to-speech competition, showing that it delivers accurate and repeatable results.
Light field capturing with lensless cameras

We introduce a novel approach to capturing light field with lensless cameras. By moving the cameras back and forth, we capture a set of images. We show that it is possible to reconstruct the light field from these blurry images. The problem is formulized in a way similar to computer tomography, so that the light field can be reconstructed using existing algorithms. The light field can then be used to render 3D scenes. Synthetic examples are presented to show the effectiveness of the proposed method.
A Three-Layer Virtual Director Model for Supporting Automated Multi-Site Distributed Education

In multi-site distributed education (MSDE), video streams from multiple sites are available. To best utilize the limited screen space at each site, we develop a customizable, automated display management system in this paper, i.e., only user-preferred streams will be shown as triggered by events and timers. The configuration of such user preference, however, is challenging because it has to be both human-friendly and machine-friendly. To address this challenge, we propose a three-layer virtual director model. In the user layer, we identify three categories of parameters that can represent a wide range of user preferences yet are easy to use. These preferences are then automatically translated into a machine-friendly timed automaton in the execution layer. The automaton is simulated dynamically, which selects a subset of streams to show on the screen through a display layer. Evaluation results demonstrate the correctness and efficiency of the proposed framework
Boosting-Based Multimodal Speaker Detection for Distributed Meetings

Speaker detection is a very important task in distributed meeting applications. This paper discusses a number of challenges we met while designing a speaker detector for the Microsoft RoundTable distributed meeting device, and proposes a boosting-based multimodal speaker detection (BMSD) algorithm. Instead of performing sound source localization (SSL) and multi-person detection (MPD) separately and subsequently fusing their individual results, the proposed algorithm uses boosting to select features from a combined pool of both audio and visual features simultaneously. The result is a very accurate speaker detector with extremely high efficiency. The algorithm reduces the error rate of SSL-only approach by 47%, and the SSL and MPD fusion approach by 27%
On the compression and streaming of concentric mosaic data for free wandering in a realistic environment over the Internet

In this paper, we describe a system for wandering in a realistic environment over the Internet. The environment is captured by the concentric mosaic, compressed via the reference block coder (RBC), and accessed and delivered over the Internet through the virtual media (Vmedia) access protocol. One of the key contributions of the paper is the proposal of the RBC concentric mosaic coder. The RBC coder not only compresses the huge dataset of the concentric mosaic very efficiently, but also produces a well-organized bitstream that can be accessed just-in-time (JIT). To reconstruct a virtual view, only a portion of the RBC bitstream needs to be accessed and decoded. This greatly reduces the memory and computation requirement of the viewer compared with first decoding the entire concentric mosaic data set and then rendering from the decoded data. Our second contribution is the employment of the Vmedia protocol to deliver the compressed concentric mosaic bitstream just-in-time over the Internet. Only the bitstream segments corresponding to the current view are streamed over the Internet. The delivered bitstream segments are managed by a local Vmedia cache, so that frequently used bitstream segments do not need to be streamed over the Internet repeatedly, and a RBC bitstream larger than the memory capacity can be easily handled. Combining RBC and Vmedia, a concentric mosaic interactive browser is developed through which the user can freely wander in a realistic environment, e.g., rotate around, walk forward/backward, and sidestep, even under a tight bandwidth.
A system for active image-based rendering

In this paper, we develop a system for active image-based rendering (IBR). Active IBR is a framework that is capable of estimating the final rendering quality and capturing the next view at the position where the rendering quality can be improved the most. The result is a non-uniform capturing scheme for IBR. Experimental results on synthetic scenes have shown that active IBR outperforms uniformly captured IBR. In this paper, we set up an active IBR system that can capture real objects with off-the-shelf components. We show that our system can capture objects more intelligently than uniform capturing.
Using Reverberation to Improve Range and Elevation Discrimination for Small Array Sound Source Localization

Sound source localization (SSL) is an essential task in many applications involving speech capture and enhancement. As such, speaker localization with microphone arrays has received significant research attention. Nevertheless, existing SSL algorithms for small arrays still have two significant limitations: lack of range resolution, and accuracy degradation with increasing reverberation. The latter is natural and expected, given that strong reflections can have amplitudes similar to that of the direct signal, but different directions of arrival. Therefore, correctly modeling the room and compensating for the reflections should reduce the degradation due to reverberation. In this paper, we show a stronger result. If modeled correctly, early reflections can be used to provide more information about the source location than would have been available in an anechoic scenario. The modeling not only compensates for the reverberation, but also significantly increases resolution for range and elevation. Thus, we show that under certain conditions and limitations, reverberation can be used to improve SSL performance. Prior attempts to compensate for reverberation tried to model the room impulse response (RIR). However, RIRs change quickly with speaker position, and are nearly impossible to track accurately. Instead, we build a 3-D model of the room, which we use to predict early reflections, which are then incorporated into the SSL estimation. Simulation results with real and synthetic data show that even a simplistic room model is sufficient to produce significant improvements in range and elevation estimation, tasks which would be very difficult when relying only on direct path signal components.
View-dependent nonuniform sampling for image-based rendering

In this paper, we propose an algorithm for view-dependent nonuniform sampling for image-based rendering (IBR). Given a set of virtual views, the positions of the capturing cameras are rearranged in order to obtain the optimal rendering quality. The resulting arrangement of the cameras is effectively nonuniform sampling of the plenoptic function. We formulate the above sampling problem as a recursive weighted vector quantization problem, which can be solved efficiently. Experimental results show that the nonuniform sampling scheme renders much better images than traditional uniform sampling methods.
On generalized sampling for image-based rendering data

We apply generalized sampling to image-based rendering (IBR) data, more specifically, the lightfield. We show that, in theory, the lowest sampling rate of a lightfield when we use generalized sampling can be as low as half of that when we use rectangular sampling. However, in practice, rectangular sampling has several advantages over generalized sampling. We analyze the pros and cons for each sampling approach, and explain why, in practice, rectangular sampling is still preferable.
Why does PHAT work well in lownoise, reverberative environments?

Among many existing time difference of arrival (TDOA) based sound source localization (SSL) algorithms, the phase transform (PHAT) is extremely popular for its excellent performance in low noise environments, even under relatively heavy reverberation. However, PHAT was developed as a heuristic approach and its working principle has not been completely understood. In this paper, we present the relationship between PHAT and a maximum likelihood (ML) framework for multi-microphone sound source localization. We show that when the environment noise approaches zero, PHAT is indeed a special case of the ML algorithm, which explains its good performance under low noise environments. In addition, we show that as long as the noise stays low, PHAT remains optimal in ML sense even when the room reverberation is heavy, which explains its robustness over reverberation.
Virtual View Reconstruction Using Temporal Information

The most significant problem in generating virtual views from a limited number of video camera views is handling areas that have become dis-occluded by shifting the virtual view away from the camera view. We propose using temporal information to address this problem, based on the notion that dis-occluded areas may have been seen by some camera in some previous frames. We formulate the problem as one of estimating the underlying state of the object in a stochastic dynamical system, given a sequence of observations. We apply the formulation to improving the visual quality of virtual views generated from a single “color plus depth” camera, and show that our algorithm achieves better results than depth image based rendering using standard inpainting.
Taylor expansion based classifier adaptation: Application to person detection

Because of the large variation across different environments, a generic classifier trained on extensive data-sets may perform sub-optimally in a particular test environment. In this paper, we present a general framework for classifier adaptation, which improves an existing generic classifier in the new test environment. Viewing classifier learning as a cost minimization problem, we perform classifier adaptation by combining the cost function on the old data-sets with the cost function on the data-set collected from the new environment. The former term is further approximated with its second order Taylor expansion to reduce the amount of information that needs to be saved for adaptation. Unlike traditional approaches that are often designed for a specific application or classifier, our scheme is applicable to various types of classifiers and user labels. We demonstrate this property on two popular classifiers (logistic regression and boosting), while using two types of user labels (direct labels and similarity labels). Extensive experiments conducted for the task of person detection in conference-room environments show that significant performance improvement can be achieved with our proposed method.
Improving multiview face detection with multi-task deep convolutional neural networks

Multiview face detection is a challenging problem due to dramatic appearance changes under various pose, illumination and expression conditions. In this paper, we present a multi-task deep learning scheme to enhance the detection performance. More specifically, we build a deep convolutional neural network that can simultaneously learn the face/nonface decision, the face pose estimation problem, and the facial landmark localization problem. We show that such a multi-task learning scheme can further improve the classifier's accuracy. On the challenging FDDB data set, our detector achieves over 3% improvement in detection rate at the same false positive rate compared with other state-of-the-art methods.
Maximum Likelihood Sound Source Localization and Beamforming for Directional Microphone Arrays in Distributed Meetings

In distributed meeting applications, microphone arrays have been widely used to capture superior speech sound and perform speaker localization through sound source localization (SSL) and beamforming. This paper presents a unified maximum likelihood framework of these two techniques, and demonstrates how such a framework can be adapted to create efficient SSL and beamforming algorithms for reverberant rooms and unknown directional patterns of microphones. The proposed method is closely related to steered response power-based algorithms, which are known to work extremely well in real-world environments. We demonstrate the effectiveness of the proposed method on challenging synthetic and real-world datasets, including over six hours of recorded meetings.