nils-herrmann/clean_scirepeval huggingface dataset:
Affine multipicture motion-compensated prediction

Affine motion compensation is combined with long-term memory motion-compensated prediction. The idea is to determine several affine motion parameter sets on subareas of the image. Then, for each affine motion parameter set, a complete reference picture is warped and inserted into the multipicture buffer. Given the multipicture buffer of decoded pictures and affine warped versions thereof, block-based translational motion-compensated prediction and Lagrangian coder control are utilized. The affine motion parameters are transmitted as side information requiring additional bit rate. Hence, the utility of each reference picture and, with that, each affine motion parameter set is tested for its rate-distortion efficiency. The combination of affine and long-term memory motion-compensated prediction provides a highly efficient video compression scheme in terms of rate-distortion performance. The two incorporated multipicture concepts complement each other well providing almost additive rate-distortion gains. When warping the prior decoded picture, average bit-rate savings of 15% against TMN-10, the test model of ITU-T Recommendation H.263, are reported for the case that 20 warped reference pictures are used. When employing 20 warped reference pictures and 10 decoded reference pictures, average bit-rate savings of 24% can be obtained for a set of eight test sequences. These bit-rate savings correspond to gains in PSNR between 0.8-3 dB. For some cases, the combination of affine and long-term memory motion-compensated prediction provides more than additive gains.
A novel network design for future IP-based driver assistance camera systems

The current in-vehicle driver assistance network system is realized via point-to-point connections. Due to the growing number of cameras, the in-vehicle network systems are becoming increasingly complex, inflexible and costly. In our previous work, we have proposed an IP over Ethernet (IP/Ethernet) in-vehicle network system and have shown the necessity of video compression for driver assistance cameras. In this paper, all existing camera systems in the car are presented and classified. Their current connections are introduced and a new network topology based on IP/Ethernet is proposed to interconnect the cameras. We also investigate how video codecs in the car should be parametrized in order to avoid negative effects for driver assistance image processing algorithms while reducing the overall cost. The results show compression ratio regions that do not negatively affect the image processing tasks but instead improve their performance.
Adaptive playout for low latency video streaming

Network variations in video streaming require sufficient data to be prebuffered at the client prior to playout. This receiver buffer prevents the display process from starvation in case of network congestion. Prebuffering, however, is also responsible for the major part of the delay between requesting a media stream and playing it at the receiver. We show how adaptive media playout can be employed to reduce the delay introduced by the receiver buffer while preserving the same resilience against buffer underflow as in non-adaptive media playout. For adaptive media playout we adjust the playout speed of the media packets depending on the condition of the channel and the current client buffer fullness. We employ a two-state Markov channel model to analyze the buffer underflow-delay trade-off for our adaptive playout strategy and show that for typical parameters the average end-to-end delay can be reduced by 1 to 2 seconds.
Multimodal Feature-Based Surface Material Classification

When a tool is tapped on or dragged over an object surface, vibrations are induced in the tool, which can be captured using acceleration sensors. The tool-surface interaction additionally creates audible sound waves, which can be recorded using microphones. Features extracted from camera images provide additional information about the surfaces. We present an approach for tool-mediated surface classification that combines these signals and demonstrate that the proposed method is robust against variable scan-time parameters. We examine freehand recordings of 69 textured surfaces recorded by different users and propose a classification system that uses perception-related features, such as hardness, roughness, and friction; selected features adapted from speech recognition, such as modified cepstral coefficients applied to our acceleration signals; and surface texture-related image features. We focus on mitigating the effect of variable contact force and exploration velocity conditions on these features as a prerequisite for a robust machine-learning-based approach for surface classification. The proposed system works without explicit scan force and velocity measurements. Experimental results show that our proposed approach allows for successful classification of textured surfaces under variable freehand movement conditions, exerted by different human operators. The proposed subset of six features, selected from the described sound, image, friction force, and acceleration features, leads to a classification accuracy of 74 percent in our experiments when combined with a Naive Bayes classifier.
RDTC Optimized Compression of Image-Based Scene Representations (Part II): Practical Coding

Interactive streaming of compressed image-based scene representations requires random access to the reference image data. The degree of interframe dependencies exploited during encoding has an impact on the transmission and decoding time and, at the same time, delimits the (storage) rate-distortion (RD) tradeoff that can be achieved. The transmission data rate and the decoding complexity at the client have received attention in the literature, but their incorporation into the optimization procedure for compression and streaming is missing. If scenario-specific measures are considered, the traditional RD optimization can be extended to a tradeoff between the (storage) rate (R), distortion (D), transmission data rate (T), and decoding complexity (C). In the first part of this sequel of papers, we have theoretically analyzed the RDTC space for the compression of densely sampled image-based scene representations. In this second part, we consider practical RDTC optimization. We propose a modeling and encoding parameter selection procedure that allows us to adapt the compression to scenario-specific properties. The impact of client side caching is considered and evaluated using an experimental testbed. Our results show a significant reduction of the user perceived delay, memory consumption or required minimum channel and storage bitrate for RDTC optimized streams compared to classical RD optimized or independently encoded scene representations.
Toward QoE-driven dynamic control scheme switching for time-delayed teleoperation systems: A dedicated case Study

Networked teleoperation with haptic feedback is a prime example for the emerging Tactile Internet, which requires a careful orchestration of haptic communication and control. One major challenge in this context is how to maximize the user's quality-of-experience (QoE) while ensuring at the same time the stability of the global control loop in the presence of communication delay. In this paper, we propose a dynamic control scheme switching strategy for teleoperation systems, which maximizes the QoE for time-varying communication delay. In order to validate the feasibility of the proposed approach, we perform a dedicated case study for a virtual teleoperation environment consisting of a one-dimensional spring-damper system, and conduct extensive subjective tests under various delay conditions for two control schemes : (1) teleoperation with the time-domain passivity approach (TDPA), which is highly delay-sensitive but supports highly dynamic interaction between the operator and a potentially quickly changing remote environment; (2) model-mediated teleoperation (MMT), which is tolerable to relatively larger communication delays, but unsuitable for quickly changing, highly dynamic remote environments. For both schemes, we use recently proposed extensions, which incorporate perceptual data reduction to reduce the required packet rate between the operator and the teleoperator. One key contribution of this paper lies in the exploration of the intrinsic relationship among QoE, communication delay and the control schemes which provides a fundamental guidance, not only to this research, but also to the future joint optimization of communication and control for time-delayed teleoperation systems.
Hybrid signal-based and geometry-based prediction for haptic data reduction

Haptic data reduction schemes address the high packet-rate requirements of networked haptics. Perception-driven predictive coding approaches enable strong packet rate reduction while keeping the introduced distortion below human haptic perception thresholds. The performance of predictive coding is strongly influenced by factors such as human behavior, system characteristics, geometric and impedance properties of the environment, etc. In this paper, we first describe a novel surface geometry-based prediction approach for haptic data reduction where local object surface features are approximated with the help of simple geometric models. Secondly, we present a hybrid framework that combines signal-based and geometry-based prediction. Psychophysical experiments are performed to validate this framework. The results of the proposed geometry-based prediction show an improvement in haptic data reduction of about 54% as compared to the signal-based prediction (linear predictor). Furthermore, the presented hybrid prediction technique allows for an additional gain of 15%.
Standard compatible extension of H.263 for robust video transmission in mobile environments

In this paper we address the problem of robust video transmission in error prone environments. The approach is compatible with the ITU-T video coding standard H.263. Fading situations in mobile networks are tolerated and the image quality degradation due to spatio-temporal error propagation is minimized utilizing a feedback channel between transmitter and receiver carrying acknowledgment information. In a first step, corrupted group of blocks (GOB's) are concealed to avoid annoying artifacts caused by decoding of an erroneous bit stream. The GOB and the corresponding frame number are reported to the transmitter via the back channel. The encoder evaluates the negative acknowledgments and reconstructs the spatial and temporal error propagation. A low complexity algorithm for real-time reconstruction of spatio-temporal error propagation is described in detail. Rapid error recovery is achieved by INTRA refreshing image regions (macroblocks) bearing visible distortion. The feedback channel method does not introduce additional delay and is particularly relevant for real-time conversational services in mobile networks. Experimental results with bursty bit error sequences simulating a Digital European Cordless Telephony (DECT) channel are presented with different combinations of forward error correction (FEC), automatic repeat on request (ARQ), and the proposed error compensation technique, Compared to the case where FEC and ARQ are used for error correction, a gain of up to 3 dB peak signal-to-noise ratio (PSNR) is observed if error compensation is employed additionally.
R-D optimized media streaming enhanced with adaptive media playout

Streaming media systems buffer media data at the client to improve reconstruction quality at the cost of latency. Reconstruction quality may be further improved with rate-distortion optimized packet transmission. In this paper we show how the tradeoff between buffering latency and reconstruction quality can be improved with adaptive media playout (AMP) - the client-controlled manipulation of the playout speeds of media frames. We then show how to incorporate AMP into an existing framework for rate-distortion optimized packet transmission. The resulting receiver-driven scheme jointly optimizes packet transmissions and playout speeds with respect to rate, distortion and the subjective cost of playout speed variations. Results for a short media clip show a more than 2 dB improvement in mean PSNR for R-D optimized transmission combined with a moderate amount of AMP, over R-D optimized transmission alone.
Wireless Media Streaming Over IP-Based In-Vehicle Networks

We investigate a heterogeneous IP-based network architecture for in-vehicle communication. The wireless technology is an interesting candidate for in-vehicle communication, since it reduces the cabling effort and consequently the cost. In this work, a peripheral wireless network based on the IEEE 802.11 technology is investigated for media streaming in the car. FEC-based multicast and unicast transmissions are studied and compared for the same communication scenarios based on their QoS performance and resource requirements. Moreover, the frame bursting mechanism from the IEEE 802.11e standard has been applied to improve the throughput of small packets over in-vehicle wireless channels.
High-fidelity recording, compression, and replay of visual-haptic telepresence sessions

In this paper, we propose a novel framework for the efficient recording and playback of visual-haptic telepresence sessions. During recording, all involved modalities are synchronously encoded and stored. During replay, interaction with the tele-environment is no longer possible and the displayed force feedback does not relate to the user's interaction anymore. Furthermore, position guidance by applying force-feedback prevents the presentation of the recorded haptic impressions. In our experiments, we have discovered that the human operator can easily associate the previously recorded movements and actions by following the endeffector position of the teleoperator in the displayed visual recording and relate the presented force feedback to its observation. We describe and investigate this observation which allows the user to immerse into the previously recorded telemanipulation session. Conducted psychophysical experiments investigate the performance of our framework in conjunction with perceptual offline coding for haptic signals. Our user studies reveal that our approach can provide high-fidelity visual-haptic experiences for learning, entertainment and/or performance analysis purposes.
Visual homing and surprise detection for cognitive mobile robots using image-based environment representations

One important feature of a cognitive system is to perceive and understand its environment and to adapt its actions to changes and unforeseen situations. In this paper, we propose a scheme for visual surprise detection in cognitive mobile robots. With the robot's observation and a set of reference images previously acquired near its current viewpoint, a pixel-wise surprise trigger is computed using Bayesian probabilistic inference techniques. With appropriate mathematical approximations this algorithm can be implemented on modern graphics hardware which nearly allows for real-time surprise detection. In order to refer to prior observations, a mobile robot has to be able to re-localize itself with respect to its environment. Thus, we also present two online image-based homing algorithms which both facilitate the computation of location-independent surprise triggers. Experiments show acceptable results in terms of robust and fast detection of unexpected changes in the environment.
Navigation and Manipulation Planning Using a Visuo-Haptic Sensor on a Mobile Platform

Mobile systems interacting with objects in unstructured environments require both haptic and visual sensors to acquire sufficient scene knowledge for tasks such as navigation and manipulation. Typically, separate sensors and processing systems are used for the two modalities. We propose to acquire haptic and visual measurements simultaneously, providing naturally coherent data. For this, compression of a passive, deformable foam rod mounted on the actuator is measured visually by a low-cost camera, yielding a 1-D stress function sampled along the contour of the rod. The same camera observes the nearby scene to detect objects and their reactions to manipulation. The system is passively compliant and the complexity of the sensor subsystems is reduced. Furthermore, we present an integrated approach for navigation and manipulation on mobile platforms, which integrates haptic data from the sensor. A high-level planning graph represents both the structure of a visually acquired map, as well as manipulable obstacles. Paths within this graph represent high-level navigation and manipulation tasks, e.g., pushing of obstacles. A cost-optimal task plan is generated using standard pathfinding techniques. The approach is implemented and validated on a mobile robotic platform. Obtained forces are compared with a reference, showing high accuracy within the medium sensor range. A real-world experiment is presented, which uses the sensor for haptic exploration of obstacles in an office environment. Substantially faster task plans can be found in cluttered scenes compared with purely visual navigation.
A joint compression scheme for local binary feature descriptors and their corresponding bag-of-words representation

For real-time computer vision tasks, binary feature descriptors are an efficient alternative to their real-valued counterparts. While providing comparable results for many applications, the computational complexity of extracting and processing binary descriptors is significantly lower. In many application scenarios, the local features are transmitted over a channel with limited capacity and processed at a more powerful central processing unit, which requires efficient compression and transmission approaches. In this paper, we present a compression scheme for local binary features, which jointly encodes the descriptors and their respective Bag-of-Words representation using a shared vocabulary between client and server. By sending the visual word index and the entropy-coded residual vector containing the differences between the visual word and the descriptor, we are able to reduce ORB features to 60.62 % of their uncompressed size.
Estimation of rigid body motion and scene structure from image sequences using a novel epipolar transform

Model-based video communication systems attempt to extract information about the three-dimensional structure of the scene to be transmitted. The relative motion between camera and objects can be a powerful depth cue, if the objects are rigid or almost rigid. We present a new algorithm for estimation of rigid body motion parameters and scene structure from monocular image sequences. A novel epipolar image transform is utilized to preserve the relevant information from mean squared displaced frame difference (DFD) surfaces and thus overcomes the inherent limitations of feature correspondence methods. Our algorithm conducts a coarse-to-fine search in 5-dimensional parameter space. Relative depth values are computed for each measurement window. Experimental results are presented to demonstrate the performance of the new algorithm.
On the retrieval of perceptually similar haptic surfaces

The surface properties of an object can be explored by sliding the tip of a tool over the surface. This induces vibrations in the tool that can be captured using acceleration sensors. Based on these signals, this paper presents a first step towards tool-mediated retrieval of perceptually similar surfaces. We conduct subjective experiments to identify ground truth data about perceptually similar surfaces in a database of 69 materials. The results of this experiment are used to evaluate our retrieval system. For a surface query, the retrieval system should return those surfaces the user perceives as similar. We test various features in our retrieval pipeline and evaluate their usefulness in such a retrieval scenario. We target towards a tradeoff between high classification accuracy and high recall of perceptually similar surfaces.
Multi-stream voice over IP using packet path diversity

We propose multi-stream transmission of real-time voice over best-effort packet networks such as today's Internet, where multiple redundant descriptions of the voice stream are sent over independent network paths. At the receiver, multi-stream adaptive playout scheduling is employed to improve the tradeoff among delay, late loss rate, and speech quality. Experiments over the Internet suggest largely uncorrelated statistical characteristics, such as erasure probability and delay jitter, for different network paths, which leads to a noticeable path diversity gain. We have obtained significant reductions in mean end-to-end latency and loss rates compared to FEC protected single-path transmission at the same data rate. The speech quality perceived by the receiver is evaluated using the recently standardized objective quality measure PESQ. In our experiments, we observe gains of more than 0.4 PESQ score for voice transmission with packet path diversity.
Fast Motion Estimation-Based Reference Frame Generation in Wyner-Ziv Residual Video Coding

In practical Wyner-Ziv video coding, every frame is encoded independently of others, but decoded based on the side information generated from adjacent frames. In Wyner-Ziv residual coding of video, the residual of a frame with respect to a reference frame is Wyner-Ziv encoded, which leads to a higher coding efficiency than directly Wyner-Ziv encoding the original frame. In previous work, the reference frame is directly copied from the previously reconstructed frame. In this paper, we generate the reference Wyner-Ziv frame at the encoder using low complexity fast motion search. Experimental results show that the proposed scheme provides a significant gain in video coding efficiency with the encoding complexity being only slightly increased. Using our approach, Wyner-Ziv residual video encoding becomes flexible and allows us to trade-off the encoding complexity and the overall rate-distortion performance. Moreover, the reference frame can help us in refining the side information generated at the decoder, which contributes to the improvement of the overall video coding efficiency.
Rapid image retrieval for mobile location recognition

Recognizing the location and orientation of a mobile device from captured images is a promising application of image retrieval algorithms. Matching the query images to an existing georeferenced database like Google Street View enables mobile search for location related media, products, and services. Due to the rapidly changing field of view of the mobile device caused by constantly changing user attention, very low retrieval times are essential. These can be significantly reduced by performing the feature quantization on the handheld and transferring compressed Bag-of-Feature vectors to the server. To cope with the limited processing capabilities of handhelds, the quantization of high dimensional feature descriptors has to be performed at very low complexity. To this end, we introduce in this paper the novel Multiple Hypothesis Vocabulary Tree (MHVT) as a step towards real-time mobile location recognition. The MHVT increases the probability of assigning matching feature descriptors to the same visual word by introducing an overlapping buffer around the separating hyperplanes to allow for a soft quantization and an adaptive clustering approach. Further, a novel framework is introduced that allows us to integrate the probability of correct quantization in the distance calculation using an inverted file scheme. Our experiments demonstrate that our approach achieves query times reduced by up to a factor of 10 when compared to the state-of-the-art.