nils-herrmann/clean_scirepeval huggingface dataset:
Realistic audio in immersive video conferencing

With increasing computation power, network bandwidth, and improvements in display and capture technologies, fully immersive conferencing and tele-immersion is becoming ever closer to reality. Outside of video, one of the key components needed is high quality spatialized audio. This paper presents an implementation of a relatively low complexity, simple solution which allows realistic audio spatialization of arbitrary positions in a 3D video conference. When combined with pose tracking, it also allows the audio to change relative to which position on the screen the viewer is looking at.
Motion of an uncalibrated stereo rig: self-calibration and metric reconstruction

We address in this paper the problem of self-calibration and metric reconstruction (up to a scale) from one unknown motion of an uncalibrated stereo rig. The epipolar constraint is first formulated for two uncalibrated images. The problem then becomes one of estimating unknowns such that the discrepancy from the epipolar constraint, in terms of sum of squared distances between points and their corresponding epipolar lines, is minimized. Redundancy of the information contained in a sequence of stereo images makes this method more robust than using a sequence of monocular images. Real data have been used to test the proposed method, and the results obtained are quite good.
Head-Size Equalization for Improved Visual Perception in Video Conferencing

In a video conferencing setting, people often use an elongated meeting table with the major axis along the camera direction. A standard wide-angle perspective image of this setting creates significant foreshortening, thus the people sitting at the far end of the table appear very small relative to those nearer the camera. This has two consequences. First, it is difficult for the remote participants to see the faces of those at the far end, thus affecting the experience of the video conferencing. Second, it is a waste of the screen space and network bandwidth because most of the pixels are used on the background instead of on the faces of the meeting participants. In this paper, we present a novel technique, called Spatially-Varying-Uniform scaling functions, to warp the images to equalize the head sizes of the meeting participants without causing undue distortion. This technique works for both the 180-degree views where the camera is placed at one end of the table and the 360-degree views where the camera is placed at the center of the table. We have implemented this algorithm on two types of camera arrays: one with 180-degree view, and the other with 360-degree view. On both hardware devices, image capturing, stitching, and head-size equalization are run in real time. In addition, we have conducted user study showing that people clearly prefer head-size equalized images.
Immersive Telepresence: Transcending Space and Time

With increasing economic globalization and workforcemobilization, there is a strong need for research anddevelopment of advanced infrastructures and tools to bringimmersive experience into telecommunication so people acrossgeographically distributed sites can interact collaboratively ina natural way. This requires deep understanding of multipledisciplines. In particular, computer vision, graphics and acousticsare indispensable in order to capture and render 3D environmentsthat create the illusion that the remote participantsare in the same room. The existing videoconferencing systems,whether they are available on desktop and mobile devices orin dedicated conference rooms with built-in furniture and lifesizedhigh-definition video, leave a great deal to be desired:mutual gaze, 3D, motion parallax, spatial audio, to name afew. For the first time, the necessary immersive technologiesare emerging and coming together to enable real-time capture,transport, and rendering of 3D "hologram", and we are muchcloser to realize man's dream reflected in Hollywood moviesfrom Star Trek and Star Wars to The Matrix and Avatar. Inthis paper, I will take you on a tour of various componenttechnologies as well as experiences that are being developed atMicrosoft Research for immersive Telepresence.
Understanding the relationship between the optimization criteria in two-view motion analysis

The three best known criteria in two-view motion analysis are based respectively, on the distances between points and their corresponding epipolar lines, on the gradient-weighted epipolar errors, and on the distances between points and the reprojections of their reconstructed paints. The last one has a better statistical interpretation, but is, however much slower than the first two. In this paper we show that the last two criteria are equivalent when the epipoles are at infinity, and differ from each other only a little even when the epipoles are in the image. The first two criteria are equivalent only when the epipoles are at infinity and when the observed object has the same scale in the two images. This suggests that the second criterion is sufficient in practice because of its computational efficiency. The result is valid for both calibrated and uncalibrated images.
Automatic eyeglasses removal from face images

In this paper, we present an intelligent image editing and face synthesis system that automatically removes eyeglasses from an input frontal face image. Although conventional image editing tools can be used to remove eyeglasses by pixel-level editing, filling in the deleted eyeglasses region with the right content is a difficult problem. Our approach works at the object level where the eyeglasses are automatically located, removed as one piece, and the void region filled. Our system consists of three parts: eyeglasses detection, eyeglasses localization, and eyeglasses removal. First, an eye region detector, trained offline, is used to approximately locate the region of eyes, thus the region of eyeglasses. A Markov-chain Monte Carlo method is then used to accurately locate key points on the eyeglasses frame by searching for the global optimum of the posterior. Subsequently, a novel sample-based approach is used to synthesize the face image without the eyeglasses. Specifically, we adopt a statistical analysis and synthesis approach to learn the mapping between pairs of face images with and without eyeglasses from a database. Extensive experiments demonstrate that our system effectively removes eyeglasses.
WWN: Integration with coarse-to-fine, supervised and reinforcement learning

The cost of autonomous development is substantial. Although supervised learning is effective, the cost demand on teachers is often too high to be constantly applied. Reinforcement learning can take advantage of physical reality due to environmental feedback and inspections. Information required in reinforcement learning is not as specific as is required in supervised learning. Integration theories, methods, and analysis of these two learning strategies are still rare in the literature although such integration has been well known in the animal kingdom. Based on our prior work on a general purpose framework called Developmental Network and its embodiment Where-What-Network, we present our theory, method, and analysis for integration of supervised learning and reinforcement learning in this paper. Different from all other known work on reinforcement learning, this DN framework uses fully emergent representation to avoid the brittleness and task-specific representations. Central in the integration is not just to provide a freedom for the teacher to choose the mode of learning, which is necessary especially when the physical non-living world is an implicit teacher, but the mechanism of scaffolding. In our experiment the scaffolding is reflected by allowing the location motor(LM) neurons to gradually refine representation through splitting(mitosis) in a coarse to fine scheme. We report our experimental work in a very challenging learning setting: both object and backgrounds are unknown(cluttered settings) and concepts(e.g. location and type) emerge from agent-environment interactions, instead of rigidly handcrafted.
Editorial - TAMD Outstanding Paper Award and Open Access Publication Established

In an editorial of the IEEE Transactions on Autonomous Mental Development (TAMD) (Vol. 4, No. 3), the author noted the progress made in establishing the IEEE TAMD Outstanding Paper Award to recognize annually outstanding papers published in the TRANSACTIONS. He is pleased to report that the IEEE Technical Activities Board (TAB) has approved the motion and that the IEEE TAMD Outstanding Paper Award will be formally established in 2013. This is the first year the Award will be bestowed. For the current round of competition, any paper published in 2011 (Volume 3) is eligible for consideration. The prize includes a US$1000 honorarium, to be split equally among coauthors, and certificates to the author and coauthors of the selected paper. Please note, no self-nomination is allowed. On another topic, IEEE TAMD is a hybrid transactions allowing either traditional publications or author-pay Open Access (OA) publications. The OA option, if selected, enables unrestricted public access to the article via IEEE Xplore. The OA option will be offered to the author at the time the manuscript is submitted. If selected, the OA fee must be paid before the article is published in the TRANSACTIONS. IEEE currently offers the discounted rate of US$1750 per article. The traditional option, if selected, enables access to all qualified subscribers and purchasers via IEEE Xplore. For the traditional option, no OA payment is required. The IEEE peer review standard of excellence is applied consistently to all submissions. All accepted articles will be included in the print issuemailed to subscribers.
CIS Publication Spotlight [Publication Spotlight]

Digital Object Identifier: 10.1109/ TNN.2011.2172628 “In this paper, a novel heuristic dynamic programming (HDP) iteration algorithm is proposed to solve the optimal tracking control problem for a class of nonlinear discrete-time systems with time delays. The novel algorithm contains state updating, control policy iteration, and performance index iteration. To get the optimal states, the states are also updated. Furthermore, the ‘backward iteration’ is applied to state updating. Two neural networks are used to approximate the performance index function and compute the optimal control policy for facilitating the implementation of HDP iteration algorithm. At last, two examples are presented to demonstrate the effectiveness of the proposed HDP iteration algorithm.”
Direct filtering for air- and bone-conductive microphones

Air- and bone-conductive integrated microphones have been introduced by the authors [Y. Zheng, et al., 2003, Z. Zhang et al., 2004] for speech enhancement in noisy environments. In this paper, we present a novel technique, called direct filtering, to combine the two channels from the air- and bone-conductive microphone for speech enhancement. Compared to the previous technique, the advantage of the direct filtering is that it does not require any training, and it is speaker independent. Experiments show that this technique effectively removes noises and significantly improves speech recognition accuracy even in highly non-stationary noisy environments.
Image-based surface detail transfer

Changing an object's appearance by adding geometric details is desirable in many real-world applications. Bump mapping has acted as an alternative to adding geometrical details, to an otherwise smooth object. But constructing visually interesting bump maps requires practice and artistic skills. Computer vision techniques have helped for modeling real-world objects and their surface details. In our approach, for cases where we only want to transfer geometrical details from one object to another, we might not need to explicitly compute 3D structures. In particular, our novel technique captures the geometrical details of an object from a single image in a way that is independent of the object's reflectance property. We can then transfer geometrical details to another surface, producing the appearance of a new surface with added geometrical details while preserving the object's reflectance property. Our method's advantages are that it's simple to implement, requires only a single image for each object, and produces effective results.
Modeling the effects of neuromodulation on internal brain areas: Serotonin and dopamine

The effects of neuromodulator, such as serotonin and dopamine, on individual neurons in the brain have been known qualitatively. However, it is challenging to computationally model such effects in an emergent network, as the elements of internal representations do not have a static, task-specific meaning. Weng and coworkers modeled the effects of serotonin and dopamine on only motor neurons in emergent networks. In this work, we extend the effects of serotonin and dopamine to all neurons inside the emergent network. Our new theory is that although serotonin and dopamine indicate events of different natures (aversive and appetitive), they produce similar effects on internal non-motor neurons in that they increase their learning rates from the cases without serotonin and dopamine. This is because the presence of serotonin and dopamine indicates a higher importance of the event compared with baseline cases. Experimentally, we show that the enhanced developmental network learns faster under a limited resource.
Automatic Head-size Equalization in Panorama Images for Video Conferencing

In panorama images captured by omni-directional cameras during video conferencing, the image sizes of the people around the conference table are not uniform due to the varying distances to the camera. Spatially-varying-uniform (SVU) scaling functions have been proposed to warp a panorama image smoothly such that the participants have similar sizes on the image. To generate the SVU function, one needs to segment the table boundaries, which was generated manually in the previous work. In this paper, we propose a robust algorithm to automatically segment the table boundaries. To ensure the robustness, we apply a symmetry voting scheme to filter out noisy points on the edge map. Trigonometry and quadratic fitting methods are developed to fit a continuous curve to the remaining edge points. We report experimental results on both synthetic and real images
Camera calibration with one-dimensional objects

Camera calibration has been studied extensively in computer vision and photogrammetry and the proposed techniques in the literature include those using 3D apparatus (two or three planes orthogonal to each other or a plane undergoing a pure translation, etc.), 2D objects (planar patterns undergoing unknown motions), and 0D features (self-calibration using unknown scene points). Yet, the paper proposes a new calibration technique using 1D objects (points aligned on a line), thus filling the missing dimension in calibration. In particular, we show that camera calibration is not possible with free-moving 1D objects, but can be solved if one point is fixed. A closed-form solution is developed if six or more observations of such a 1D object are made. For higher accuracy, a nonlinear technique based on the maximum likelihood criterion is then used to refine the estimate. Singularities have also been studied. Besides the theoretical aspect, the proposed technique is also important in practice especially when calibrating multiple cameras mounted apart from each other, where the calibration objects are required to be visible simultaneously.
Analysis Of A Sequence Of Stereo Scenes Containing Multiple Moving Objects Using Rigidity Constraints

Iri this paper, we describe a method for comput.ing the rriovc~rrient. of objects as well as that of a mobile robot from ii scqiicrice of stereo frames. Stereo frames are obtained at ~iifl’(~reiit, instants by a stereo rig, when the mobile robot rIilvigat(>s in an unknown environment possibly containing ~)iiit: rrioving rigid objects. An approach based on rigidity ( oiihlraiiit,s is presented for registering two stereo frames. Wcs dernoristrate how the uncertainty of measurements can IN^ integrated with the formalism of the rigidity constraints. A iiew technique is described to match very noisy segments. ‘1‘11~ iiifluence of egomotion on observed movement:; of ob,j(~ 1,s is discussed in detail. jt determined itiid then eliminated before determination of the motion of o1)jt:cl.s. The proposed algorithm is completely automatic. I~:xperirnerital results are provided. Some remarks conclude ths paper.
Deblur a blurred RGB image with a sharp NIR image through local linear mapping

Image acquisition in a low light environment requires long exposure to achieve acceptable signal-to-noise ratio, which however causes blurry effect. This paper addresses this problem by using a sharp near-infrared (NIR) image when the environment has sufficient NIR light. We assume that an RGB and NIR image pair has a linear mapping in a local area and that the mapping function is valid for both the blur and sharp image pairs. Using this property, we solve the sharp RGB images from a blurred RGB image and the corres ponding s harp NIR image. The effectiveness of the proposed algorithm is verified with both synthetic and real captured datasets.
Action detection using multiple spatial-temporal interest point features

This paper considers the problem of detecting actions from cluttered videos. Compared with the classical action recognition problem, this paper aims to estimate not only the scene category of a given video sequence, but also the spatial-temporal locations of the action instances. In recent years, many feature extraction schemes have been designed to describe various aspects of actions. However, due to the difficulty of action detection, e.g., the cluttered background and potential occlusions, a single type of features cannot solve the action detection problems perfectly in cluttered videos. In this paper, we attack the detection problem by combining multiple Spatial-Temporal Interest Point (STIP) features, which detect salient patches in the video domain, and describe these patches by feature of local regions. The difficulty of combining multiple STIP features for action detection is two folds: First, the number of salient patches detected by different STIP methods varies across different salient patches. How to combine such features is not considered by existing fusion methods  . Second, the detection in the videos should be efficient, which excludes many slow machine learning algorithms. To handle these two difficulties, we propose a new approach which combines Gaussian Mixture Model with Branch-and-Bound search to efficiently locate the action of interest. We build a new challenging dataset for our action detection task, and our algorithm obtains impressive results. On classical KTH dataset, our method outperforms the state-of-the-art methods.
Tracking and grouping 3D line segments

The authors address the motion tracking problem that arises in the context of a mobile vehicle navigating in an unknown environment where other mobile bodies such as human beings or robots may also be moving. A stereo rig mounted on the mobile vehicle provides a sequence of 3-D maps of the environment. The current stereo system is trinocular and the 3-D tokens used at the moment are line segments corresponding to significant intensity discontinuities in the images. Although the framework to solve the motion tracking problem developed in this work arises in this specific context, the authors believe it should be applicable in other contexts; in particular they could use other 3-D primitives, for example, points, combinations of points and lines, curves.<<ETX>>
Energy-Based Position Estimation of Microphones and Speakers for Ad Hoc Microphone Arrays

We present a novel energy-based algorithm to estimate the positions of microphones and speakers in an ad hoc microphone array setting. Compared to traditional time-of-flight based approaches, energy-based approach has the advantage that it does not require accurate time synchronization. This property is particularly useful for ad hoc microphone arrays because highly accurate synchronization across microphones may be difficult to obtain since these microphones usually belong to different devices. This new algorithm extends our previous energy-based position estimation algorithm  in that it does not assume the speakers are in the same positions as their corresponding microphones. In fact, our new algorithm estimates both the microphones and speakers simultaneously. Experiment results are shown to demonstrate its performance improvement over the previous approach in , and evaluate its robustness against time synchronization errors.
Improved 3D Depth Image Estimation Algorithm for Visual Camera

A new 3D depth image estimation algorithm for visual camera is proposed in this paper. One visual camera and one Time- of-flight camera are used in our method. First, we propose a novel calibration method for visual and Time-of-Flight camera, in which both the intensity image and depth image of Time-of-Flight camera are taken into consideration. Then the initial depth map is generated by project depth map of Time-of-Flight camera to the view of visual camera. Using the color image of visual camera as reference, the initial depth map is iteratively refined according to a novel energy function which fuses two different kinds of data: the depth data and the color data. Experiment results show that the high resolution depth map of visual camera is successfully generated and of high quality. Time-of-Flight; stereo calibration; 3D depth map estimation
Constrained bit-rate control for very low bit-rate streaming-video applications

We propose a low-complexity frame-layer bit-rate control algorithm for very low bit-rate streaming-video applications with causal one-pass processing. Rate control is achieved by jointly adapting the frame rate and quantization step-size. Constraint coding is introduced, which forces the encoder to operate within a subset of operating points on a 2-D grid. The control parameters are also constrained to change in a gradual fashion, allowing bits to be saved in easy scenes so that more bits can be used for difficult scenes. A simple scene-change detector is used to insert intra frames at scene-change boundaries. The proposed algorithm is implemented in H.263 and is compared with the case of fixed control parameters and a conventional off-line bit-rate control based on adapting the quantization step-size. It is shown that the proposed algorithm codes more frames over a given time interval while achieving average PSNR gains over 1 dB and reducing bit-rate fluctuations. Results are confirmed by subjective tests, which show that our bit-rate control consistently provides improved visual quality.
Combined spline- and block-based motion estimation for video coding

We propose a technique to estimate motion for video coding. Our technique combines spline-based registration and block matching motion estimation. It replaces the initial compute-intensive gross block matching search with spline-based registration, which is more efficient in recovering full-image motion fields. This results in a smooth motion field representative of the true motion in the scene, which can be more efficiently encoded and be guaranteed of high visual quality as well. Furthermore, the implementation is computationally cost-effective. Experimental results on well-known test image sequences show that the method results in an increase in coding efficiency along with a reduction in computational cost.