Reviewerly/exHarmony huggingface dataset:
Experiences and Lessons Learned from a Crowdsourced-Remote Hybrid User Survey Framework

Subjective user studies are important to ensure the fidelity and usability of systems that generate multimedia content. Testing how end-users and domain experts perceive multimedia assets might provide crucial information. In this paper, we present our experiences with the open source hybrid crowdsourced-remote user survey framework called Huldra, which is intended for conducting web-based subjective user studies and aims to integrate the individual benefits associated with traditional, crowdsourced, and remote methods. We disseminate our experiences and insights from two actively deployed use cases and discuss challenges and opportunities associated with using Huldra as a framework for conducting user studies.
Teardrop Magnification: A Hybrid Linear-Fisheye Magnifier for the Border and Corner of the Screen

Eye tracking based interfaces have to solve two major problems: The midas-touch problem and accuracy. While the midas-touch problem can be tackled with innovative interaction concepts, accuracy problems result from human physiology, hardware limitations and increasing screen resolutions. The Multi Modal Interaction Concept for Efficient Input (M <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ice) tries to tackle these problems with a hybrid linear-fisheye magnifier. This magnifier works in the center of the screen but is problematic on the borders/corners of the screen, as it only appears as a half or quarter circle, limiting the space of the enlarged area. This paper presents a novel teardrop magnification approach for border and corner cases that solves these problems by using a different shape and by shifting the user's focus, making more of the magnified area available for interaction.
Aggregated Bidirectional Local Binary Pattern for Robust Perceptual Image Hashing

Easy access and speedy advancements to image modification tools have made it tough for researchers and experts working in the domain of image integrity verification. It has ever become a grueling job in present times. Robust Perceptual Image Hashing (RPIH) is one of the available approaches to verify image integrity. RPIH algorithms are developed for extracting a specific group of designated features from a query image to produce a consolidated representation, which can be utilized to verify image integrity. In this paper, a novel aggregated bidirectional local binary pattern RPIH algorithm is proposed, to compute the image hash, which constitutes fixed-length data. We combine Local binary Pattern (LBP) and Reverse Local Binary Pattern (RLBP) histograms to generate Aggregated Bidirectional Local Binary Pattern (AB-LBP) features for creating a hash vector. The experimental results demonstrate the usefulness of the proposed AB-LBP algorithm as a RPIH scheme and its superior computation efficiency. Additionally, the AB-LBP scheme conveys information to localize the tamper location, which is handy in rejecting the selective part of the tampered image.
Contextualized Styling of Images for Web Interfaces using Reinforcement Learning

Content personalization is one of the foundations of today’s digital marketing. Often the same image needs to be adapted for different design schemes for content that is created for different occasions, geographic locations or other aspects of the target population. We present a novel reinforcement learning (RL) based method for automatically stylizing images to complement the design scheme of media, e.g., interactive websites, apps, or posters. Our approach considers attributes related to the design of the media and adapts the style of the input image to match the context. We do so using a preferential reward system in the RL framework that learns a reward function using human feedback. We conducted several user studies to evaluate our approach and demonstrate that we are able to effectively adapt image styles to different design schemes. In user studies, images stylized through our approach were the most preferred variation across a majority of our experiments. Additionally, we also release a dataset consisting of perceptual associations of web context with the associated image style.
Task-Oriented Near-Lossless Burst Compression

Unlike single images, capturing bursts enables many possible downstream tasks (e.g. superresolution, HDR enhancement) due to the rich information preserved in the consecutive frames. Efficient compression of these bursts is therefore essential given the additional frames to store. In this paper, we propose a novel near-lossless compression method that can preserve the most relevant information in the burst to enable multiple downstream image enhancement tasks, while at the same time reducing the file size. Specifically, we propose a two-bitstream near-lossless compression pipeline that controls the image-space distortion at frame level, and introduce the Lipschitz condition to bound the task-space distortion at burst level. Experiments conducted on a real-world burst dataset confirm the benefit of the proposed solution in terms of rate-distortion both in the burst frame space and the superresolution task space, a popular downstream task in burst processing.
Conditional GAN for Small Datasets

Generating high-quality images with Generative Adversarial Networks (GANs) generally requires 100k+ training data. The required data amount is too large when we consider using GANs to support professional art creators; they need to follow the specific art style while interactively controlling the results along with their theme. This research proposes Conditional FastGAN, which adds a condition vector to FastGAN to produce high-quality different domain images even on small datasets. In our experiments, the MUCT Face Database of images consisting of face photos in various orientations and manga face images extracted from Osamu Tezuka’s works were used as a small-scale dataset. Fine-tuning with manga face images to a model pre-trained with photo-only face images enabled control of the generated images according to explicit conditions, such as photos and manga, for the same latent variables. In addition, the proposed method improved the FID score by 2.55 from the original FastGAN in the case of manga face generation.
Using Web 3D and WebXR Game to Enhance Engagement in Primary School Learning

Plastic recycling has been a global issue for many years. Thus, it is important to provide education to today’s youth. However, designing an efficient learning resource for primary children (ages 5-6) is challenging for various reasons, such as their engagement and the efficient delivery of the learning content. Web3D and WebXR are used in many educational applications. This research proposes a system that supports the combined use of Web3D and WebXR technologies to include elements of both Virtual Reality (VR) and Augmented Reality (AR) interactions to raise awareness about plastic recycling. This paper explains how Web3D and WebXR was used in this learning resource and how the mixed Extended Reality (XR) interaction was used to facilitate the game-based learning design. The results indicate that the learning session was successfully delivered within a classroom setting. This project provides a case study on how to mix the use of Web3D and WebXR to enhance the engagement of pupils in primary school learning.
Towards Efficient Video Super Resolution for Faster Streaming

Live Video Streaming contributes to a major portion of Internet traffic. Low network bandwidth conditions can negatively impact the quality of the live video stream, which should be as high as possible for a good user experience. An emergent solution to deal with this problem is to use Video Super Resolution (VSR) methods to only transmit a lower quality version of the video, and super resolve it to a higher quality version at the client side, therefore saving on the amount of data to transmit. While up-scaling incoming low resolution video frames, it is important to consider the tradeoff between quality and latency of the super resolved output video. This is necessary to ensure that the super resolution process does not negatively impact the latency of the live video stream. The goal of this work is to investigate this trade off and present our approach based on a combination of super resolution and video frames interpolation. We propose a new strategy to super resolve the input video by leveraging both VSR and Video Interpolation (VI) in order to speed up computation without impacting the quality of the final video. Preliminary experimental results confirm that, compared with standalone VSR models, our proposed pipelined VSR+VI approach achieves 20% performance speedup, with little impact on the the quality of the final output video.
Semantic-Aware View Prediction for 360-Degree Videos at the 5G Edge

In a 5G testbed, we use 360° video streaming to test, measure, and demonstrate the 5G infrastructure, including the capabilities and challenges of edge computing support. Specifically, we use the SEAWARE (Semantic-Aware View Prediction) software system, originally described in [1], at the edge of the 5G network to support a 360° video player (handling tiled videos) by view prediction. Originally, SEAWARE performs semantic analysis of a 360° video on the media server, by extracting, e.g., important objects and events. This video semantic information is encoded in specific data structures and shared with the client in a DASH streaming framework. Making use of these data structures, the client/player can perform view prediction without in-depth, computationally expensive semantic video analysis. In this paper, the SEAWARE system was ported and adapted to run (partially) on the edge where it can be used to predict views and prefetch predicted segments/tiles in high quality in order to have them available close to the client when requested. The paper gives an overview of the 5G testbed, the overall architecture, and the implementation of SEAWARE at the edge server. Since an important goal of this work is to achieve low motion-to-glass latencies, we developed and describe "tile postloading", a technique that allows non-predicted tiles to be fetched in high quality into a segment already available in the player buffer. The performance of 360° tiled video playback on the 5G infrastructure is evaluated and presented. Current limitations of the 5G network in use and some challenges of DASH-based streaming and of edge-assisted viewport prediction under "real-world" constraints are pointed out; further, the performance benefits of tile postloading are disclosed.
Self-Supervised Object Recognition Based on Repeated Re-Capturing of Dynamic Indoor Environments

Capturing a digital replica of an environment using hand held devices or mobile mapping systems has become increasingly easy in recent years. However, leveraging large amounts of data for various semantic applications is usually impaired by a costly data annotation process. In this paper, we investigate the topic of object recognition in indoor environments without supervision. We approach the problem from a remapping perspective, where we capture RGB images from the same environment at different times and use the naturally occurring changes to identify single objects. In the first step, we create pairs of images from different recordings and generate object candidates using optical flow and an off-the-shelf region proposal algorithm. Then, we use a self-supervised representation learning framework and cluster the extracted objects. We evaluate the performance of several existing clustering methods in an over-clustering setting, since the number of object classes is unknown in an unsupervised setup. Our experimental validation on a real-world dataset shows that the proposed system can successfully recognize objects and pre-annotate a dataset by exploiting a recapturing process.
Improving Multimodal Object Detection with Individual Sensor Monitoring

Multimodal object detection fuses different sensors such as camera or LIDAR to improve the detection performance. However, individual sensor inputs can also be detrimental to a system, for example when sun glare hits a camera. In this work, we propose to monitor each sensor individually to predict when an input would lead to incorrect detections. We first train one detection network for each sensor separately, using only that sensor as input. Then, we record the performance for each single-sensor network and train an introspective performance prediction network for each sensor. Finally, we train a multimodal fusion network where we weight the impact of each sensor with its predicted performance. This allows us to dynamically adapt the fusion to reduce the influence of harmful sensor readings based only on the current data. We apply the proposed concept to the state-of-the-art AVOD architecture and evaluate on the KITTI data set. The proposed sensor monitoring system improves the mean intersection-over-union performance by 4.6%. For inputs with a low predicted performance, the proposed approach outperforms the state of the art by over 10%, demonstrating the potential of using individual sensor monitoring to react to problematic input. The proposed approach can be applied to any fusion network with two or more sensors and could also be used for classification or segmentation tasks.
Synthesizing ultraviolet skin images via GAN with Gaussian weighted patch blending

In this work, we explore a novel application of synthesizing ultraviolet skin images from RGB images using an unpaired training framework for image-to-image translation. To synthesize high resolution outputs, we propose a novel Gaussian-based patch blending technique that is designed following the characteristics of GANs. Specifically, we weigh the pixels at the same coordinates among multiple generated patches based on their distance to the center point. Our proposed method is performant, taking 0.72s for a whole face image at resolution 960x720 at inference time and generates realistic-looking ultraviolet images. We also show high correspondence of our synthesized images with the true ultraviolet images qualitatively. Finally, our novel blending approach achieves significant improvements compared with other blending methods.
A Topological Approach for Facial Region Segmentation in Thermal Images

The use of thermal images as a non-contact modality has expanded drastically in recent years, including applications such as biometrics, detection of specific human behaviors, and extraction of physiological signals. For instance, using thermal images to monitor physiological signals was found to provide better information compared to RGB images, especially in situations where using physical contact sensors is not a possibility. In this paper, we present a novel topological approach that can segment the cheeks and forehead of a human face in a thermal image, with the key benefit that explicit prior information about the cheeks and forehead is unneeded. Our approach leverages topological properties present in a thermal image to provide a non-rectangular bounding curve for the cheeks and forehead, which we compare against a dataset of 1000 thermal images that were manually annotated for this task. By generating a Vietoris-Rips complex on a thermal image filtered by the Canny edge detector, our approach can segment the cheeks and forehead with recall scores as high as 90.4% and 78.4%, respectively.
Comparison of Multi-Scale Speaker Vectors and S-Vectors for Zero-Shot Speech Synthesis

We compare a novel speaker encoder model, called Multi-Scale Speaker (MSS) Vectors, with state-of-the-art s-vectors model for zero-shot speech synthesis. The s-vectors model relies on a modified transformer self-attention network for its architecture. The MSS vectors model introduces a multi-scale approach to the s-vectors model. Results demonstrate that our model produces more natural and similar-sounding synthesized speech for unseen speakers in a zero-shot speech synthesis system.
Actor-Critic Bilateral Filter for Noise-Robust Image Smoothing

Bilateral filters have been used for achieving excellent edge-preserving image smoothing. However, most studies have focused on the acceleration of bilateral filtering but not on the stability of filtering process in regard to small perturbations to its inputs. In this paper, we propose a novel actor–critic bilateral filter trained with a multistep learning scheme for high-stability edge-preserving image smoothing. We first designed an edge-preserving smoothing process as a Markov decision process that involves adjusting the width setting for the range kernel of a bilateral filter. Next, we trained our actor–critic bilateral filter in a multistep manner to learn the optimal sequence of width settings. Through extensive experiments on five benchmark datasets, we determined that the proposed actor–critic bilateral filter produced satisfactory edge-preserving smoothing results.
Toward Energy Efficient Curvature in Range Images

Energy-efficient determination of curvature, which is a core component of varied multimedia tasks involving range images, on recent x86-64 architectures is considered. Via reducing power use for core system components, the work aims to enable multimedia system contributions toward reduced energy usage. Focus here is on eight popular methods for determining surface curvature in range images, especially on consumption-conscious strategies that yield reduced energy usage in recent workstation-like environments.
The Lottery Ticket Adaptation for Neural Video Coding

Recently, learning based video compression methods have attracted increasing attention. However, most learning based video codecs are not adaptive to different video contents. Though adaptation at inference time is a solution to tackle this issue, adapting all the codec’s parameters is computationally expensive and brings heavy bitrate overhead. The recently proposed Lottery Ticket Hypothesis (LTH) states that an over-parameterized neural network contains smaller subnetworks (winning tickets) that can match the performance of the original network. In this paper, we present a novel lottery-ticket adaptation technique on decoder-side multiplicative parameters of a neural network, transferring the concept of winning lottery tickets to video compression tasks. At inference time, the winning multiplicative parameters are overfitted, compressed, and signaled together with encoded frames for decoding. We show that our approach outperforms the Versatile Video Coding (VVC) standard in the Multiscale Structural Similarity (MS-SSIM) at a low bitrate on both the UVG and JVET sequences. To the best of our knowledge, this is the first attempt to apply LTH in the video compression domain. Also, this is the first published end-to-end learned video codec working directly on YUV format, which outperforms VVC on UVG and JVET datasets in MS-SSIM.
Correlation filter tracking algorithms against interference of similar object and fast motion

fDSST (fast Discriminative Scale Space Tracking) belongs to the correlation filter tracking algorithm, which has a high success rate and precision, and also runs at a fast speed. However, it is still a huge challenge for the tracking scene of fast motion and similar object interference. In order to improve the performance of fDSST on the challenges above, this paper proposed the fDSSTs algorithm and the fDSSTss algorithm respectively. fDSSTs increases the response scores near the object location by fusing the fhog feature and the color statistical feature, so improved the tracking performance of fDSST in the fast moving scene. fDSSTss adds a multi-feature object association module on the basis of fDSST, which distinguishes the real object and the interference object from the object feature level, thereby maintaining the tracking of the real object. The fDSSTs is tested on the OTB50 dataset, in fast-moving scenarios, the success rate of fDSST is improved by 20.5% and the precision is improved by 22.8% compared with fDSST. The fDSSTss is tested on the test sequences of similar object interference, and the result shows that fDSSTss has better anti-similar object interference ability than fDSST, while meeting the real-time requirements. The experiments show that the improvements improve the success rate and precision of fDSST in fast object moving scenes, as well as the ability to resist similar object interference.
AI-Based Military Decision Support Using Natural Language

To mimic a realistic representation of military operations, serious combat simulations require sound tactical behavior from modeled entities. Therefore, one must define combat tactics, doctrines, rules of engagement, and concepts of operation. Reinforcement learning has been proven to generate a broad range of tactical actions within the behavioral boundaries of the involved entities. In a multi-agent ground combat scenario, this paper demonstrates how our artificial intelligence (AI) application develops strategies and provides orders to subsidiary units while conducting missions accordingly. We propose a combined approach where human knowledge and responsibility collaborate with an AI system. To communicate on a common level, the orders and actions imposed by AI are given in natural language. This empowers the human operator to act in a human-on-the-loop role in order to validate and evaluate the reasoning of AI. This paper showcases the successful integration of natural language into the reinforcement learning process.
Quantile-Based Policy Optimization for Reinforcement Learning

Classical reinforcement learning (RL) aims to optimize the expected cumulative rewards. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative rewards. We parameterize the policy controlling actions by neural networks and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) to solve deep RL problems with quantile objectives. QPO uses two coupled iterations running at different time scales for simultaneously estimating quantiles and policy parameters. Our numerical results demonstrate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.
Towards AI Robustness Multi-Agent Adversarial Planning in Game Play

This paper applied Monte Carlo Tree Search (MCTS) and its variant algorithm “Upper Confidence bounds applied to Trees” (UCT) to Chinese checker game and visualized it in the Ludii games portal. Since the MCTS approach cannot guarantee a finite game length in our simulation and the winning strategy is inefficient, we developed a Convolution Neural Network (CNN) model formulated on top of the MCTS algorithm to improve the performance of the Chinese checker game. Our experiment and simulations show that by using the weak labels generated by MCTS algorithm, the CNN-based model could learn without supervised signals from human interventions and execute the game strategy in a finite time. PyGame and the Ludii game portal are used in our simulation to visualize the game and show the final game results.