Reviewerly/exHarmony huggingface dataset:
Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion

We consider the challenging problem of outdoor lighting estimation for the goal of photorealistic virtual object insertion into photographs. Existing works on outdoor lighting estimation typically simplify the scene lighting into an environment map which cannot capture the spatially-varying lighting effects in outdoor scenes. In this work, we propose a neural approach that estimates the 5D HDR light field from a single image, and a differentiable object insertion formulation that enables end-to-end training with image-based losses that encourage realism. Specifically, we design a hybrid lighting representation tailored to outdoor scenes, which contains an HDR sky dome that handles the extreme intensity of the sun, and a volumetric lighting representation that models the spatially-varying appearance of the surrounding scene. With the estimated lighting, our shadow-aware object insertion is fully differentiable, which enables adversarial training over the composited image to provide additional supervisory signal to the lighting prediction. We experimentally demonstrate that our hybrid lighting representation is more performant than existing outdoor lighting estimation methods. We further show the benefits of our AR object insertion in an autonomous driving application, where we obtain performance gains for a 3D object detector when trained on our augmented data.
UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture

We present UnrealEgo, i.e., a new large-scale naturalistic dataset for egocentric 3D human pose estimation. UnrealEgo is based on an advanced concept of eyeglasses equipped with two fisheye cameras that can be used in unconstrained environments. We design their virtual prototype and attach them to 3D human models for stereo view capture. We next generate a large corpus of human motions. As a consequence, UnrealEgo is the first dataset to provide in-the-wild stereo images with the largest variety of motions among existing egocentric datasets. Furthermore, we propose a new benchmark method with a simple but effective idea of devising a 2D keypoint estimation module for stereo inputs to improve 3D human pose estimation. The extensive experiments show that our approach outperforms the previous state-of-the-art methods qualitatively and quantitatively. UnrealEgo and our source codes are available on our project web page ( https://4dqv.mpi-inf.mpg.de/UnrealEgo/ ).
On the Impossibility of Key Agreements from Quantum Random Oracles

We study the following question, first publicly posed by Hosoyamada and Yamakawa in 2018. Can parties $$\textsf{A},\textsf{B}$$ with quantum computing power and classical communication rely only on a random oracle (that can be queried in quantum superposition) to agree on a key that is private from eavesdroppers? We make the first progress on the question above and prove the following.
Panoptic Scene Graph Generation

Existing research addresses scene graph generation (SGG)—a critical technology for scene understanding in images—from a detection perspective, i.e., objects are detected using bounding boxes followed by prediction of their pairwise relationships. We argue that such a paradigm causes several problems that impede the progress of the field. For instance, bounding box-based labels in current datasets usually contain redundant classes like hairs, and leave out background information that is crucial to the understanding of context. In this work, we introduce panoptic scene graph generation (PSG), a new problem task that requires the model to generate a more comprehensive scene graph representation based on panoptic segmentations rather than rigid bounding boxes. A high-quality PSG dataset, which contains 49k well-annotated overlapping images from COCO and Visual Genome, is created for the community to keep track of its progress. For benchmarking, we build four two-stage baselines, which are modified from classic methods in SGG, and two one-stage baselines called PSGTR and PSGFormer, which are based on the efficient Transformer-based detector, i.e., DETR. While PSGTR uses a set of queries to directly learn triplets, PSGFormer separately models the objects and relations in the form of queries from two Transformer decoders, followed by a prompting-like relation-object matching mechanism. In the end, we share insights on open challenges and future directions. We invite users to explore the PSG dataset on our project page https://psgdataset.org/ , and try our codebase https://github.com/Jingkang50/OpenPSG .
Uncertainty-Guided Source-Free Domain Adaptation

Source-free domain adaptation (SFDA) aims to adapt a classifier to an unlabelled target data set by only using a pre-trained source model. However, the absence of the source data and the domain shift makes the predictions on the target data unreliable. We propose quantifying the uncertainty in the source model predictions and utilizing it to guide the target adaptation. For this, we construct a probabilistic source model by incorporating priors on the network parameters inducing a distribution over the model predictions. Uncertainties are estimated by employing a Laplace approximation and incorporated to identify target data points that do not lie in the source manifold and to down-weight them when maximizing the mutual information on the target data. Unlike recent works, our probabilistic treatment is computationally lightweight, decouples source training and target adaptation, and requires no specialized source training or changes of the model architecture. We show the advantages of uncertainty-guided SFDA over traditional SFDA in the closed-set and open-set settings and provide empirical evidence that our approach is more robust to strong domain shifts even without tuning.
Instance, Scale, and Teacher Adaptive Knowledge Distillation for Visual Detection in Autonomous Driving

Efficient visual detection is a crucial component in self-driving perception and lays the foundation for later planning and control stages. Deep-networks-based visual systems achieve state-of-the-art performance, but they are usually cumbersome and computationally infeasible for embedded devices (e.g., dash cams). Knowledge distillation is an effective way to derive more efficient models. However, most existing works target classification tasks and treat all instances equally. In this paper, we first present our Adaptive Instance Distillation (AID) method for self-driving visual detection. It can selectively impart the teacher's knowledge to the student by re-weighing each instance and each scale for distillation based on the teacher's loss. In addition, to enable the student to effectively digest knowledge from multiple sources, we also propose a Multi-Teacher Adaptive Instance Distillation (M-AID) method. Our M-AID helps the student to learn the best knowledge from each teacher w.r.t. certain instances and scales. Unlike previous KD methods, our M-AID adjusts the distillation weights in an instance, scale, and teacher adaptive manner. Experiments on the KITTI, COCO-Traffic, and SODA10 M datasets show that our methods improve the performance of a wide variety of state-of-the-art KD methods on different detectors in self-driving scenarios. Compared to the baseline, our AID leads to an average of 2.28% and 2.98% mAP increases for single-stage and two-stage detectors, respectively. By strategically integrating knowledge from multiple teachers, our M-AID method achieves an average of 2.92% mAP improvement.
An Elderly Fall Detection Method Based on Federated Learning and Extreme Learning Machine (Fed-ELM)

The lack of fall data for the elderly is a challenging problem in the fall detection community. To date, the fall and activities of daily life simulated by young people have been used in most studies to train and test fall detection algorithms. However, there are differences in movement patterns between young and elderly individuals due to bone aging, which leads to the degradation of the algorithm performance in the elderly population. To solve the above issue, this paper proposes a fall detection algorithm combining Federated Learning and Extreme Learning Machine (Fed-ELM). First, the online extreme learning machine can use a small amount of misclassified user data to update the parameters so that its performance is improved for individual users. Then, Federated Learning is applied to share data information among different users without involving user privacy. In this way, the generalizability of the fall detection algorithm is improved. The performance of the proposed algorithm in different age groups is analyzed by experiments. For young people, the accuracy, sensitivity and specificity reach 96.96%, 94.50% and 99.29%, respectively, and the accuracy on each individual is more than 94%. For elderly individuals, the accuracy, sensitivity and specificity reach 99.07%, 96.00% and 98.33%, respectively, and the accuracy of each individual is more than 96%.
Large-Scale Quantum Approximate Optimization via Divide-and-Conquer

Quantum approximate optimization algorithm (QAOA) is a promising hybrid quantum-classical algorithm for solving combinatorial optimization problems. However, it cannot overcome qubit limitation for large-scale problems. Furthermore, the simulation time of QAOA scales poorly with the problem size. We propose a divide-and-conquer QAOA (DC-QAOA) to address the above challenges for graph maximum cut (MaxCut) problem. The algorithm works by recursively partitioning a larger graph into smaller ones whose MaxCut solutions are obtained with small-size noisy intermediate-scale quantum computers. The overall solution is retrieved from the subsolutions by applying the combination policy of measurement distribution reconstruction (MDR). The solution quality depends on the graph partitioning algorithm and MDR policy. Multiple partitioning and reconstruction methods are proposed and compared. Results are evaluated by metrics, such as quantum program runtime, measurement expectation value (EV), and approximation ratio (AR). The results show that DC-QAOA achieves 97.14% AR (20.32% higher than classical counterpart), and 94.79% EV (15.80% higher than quantum annealing). DC-QAOA solves large-scale graph instances with a polynomial rate or returns unsuccessful partition if graph connectivity requirement is not fulfilled otherwise.
User Retention of Mobile Augmented Reality for Cultural Heritage Learning

Mobile Augmented Reality (AR) is becoming increasingly afford-able and popular with the constantly improving computing power of mobile devices and the popularity of smartphones and tablets. In this paper, we present a user study that investigates user retention of mobile AR in cultural heritage learning. We developed a mobile AR application that allows users to observe 3D models of museum arti-facts and learn about their culture and history. Participants achieved a knowledge retention rate of 78.21%, indicating the positive effects of mobile AR on cultural heritage learning. We performed a structural equation modeling analysis (N=50) to investigate the effects of usability, satisfaction, emotional attachment, focus of attention, and flow experience on user retention of mobile AR. The analysis results confirmed that user satisfaction and flow experience positively affect user retention. Usability and focus of attention contribute positively to user satisfaction and flow experience respectively.
SPChain: A Smart and Private Blockchain-Enabled Framework for Combining GDPR-Compliant Digital Assets Management With AI Models

In the traditional approach to a digital asset management system, the data processing mechanism is not transparent or visible to the data owners since the data is managed solely by the service provider. With the rapid development of blockchain technology, the above issues can be resolved by leveraging the tamper-resistance and decentralization characteristics of blockchain. However, post the implementation of the EU General Data Protection Rules (GDPR) in 2018, the protection of data owners has taken center stage. This has led to several principles of personal data deletion, such as Storage Limit and the Right to Be Forgotten to conflict with the blockchain. It is also observed that, out of the various smart contracts deployed to manage digital assets, often only specific smart contracts are invoked, while the rest of the deployed smart contracts are rarely invoked, leading to smart contract designs exhibiting similar patterns with very little creativity. This current scenario has motivated us to propose SPChain, a smarter and private GDPR-compliant digital asset management framework enabled by blockchain. In this approach, a decentralized InterPlanetary File System has been adopted to solve the problem of SPOF. In addition, the combination of digital assets with artificial intelligence models has been proposed so as to make digital assets accessible to a larger number of applications and to enable better creativity. In this design, artificial intelligence models have been run in independent, virtualized containers and invoked through smart contracts. The proposed SPChain can be applied to the field of digital art management to provide a complete implementation based on the Hyperledger Fabric. Using this proposed framework, model developers, digital art creators, collectors, service providers, as well as third parties can not only benefit from securely managing digital assets and combining them with AI models, but also from simultaneously complying with the rights stipulated in the GDPR. During the course of the experiments conducted, the latency, throughput, and resource consumption of different functions in the smart contracts have been measured. After adjusting the batch timeout of the block and the maximum number of transactions in a block, the throughputs were observed to be about 500 TPS, with 10 to 15 TPS for reading and writing operations, respectively. The latency ranges were found to range from 0 to 7 seconds, with 2.5 to 5 seconds for reading and writing operations, respectively.
Graph Neural Networks Meet Wireless Communications: Motivation, Applications, and Future Directions

As an efficient graph analytical tool, graph neural networks (GNNs) have special properties that are particularly fit for the characteristics and requirements of wireless communications, exhibiting good potential for the advancement of next-generation wireless communications. This article aims to provide a comprehensive overview of the interplay between GNNs and wireless communications, including GNNs for wireless communications (GNN4Com) and wireless communications for GNNs (Com4GNN). In particular, we discuss GNN4Com based on how graphical models are constructed and introduce Com4GNN with corresponding incentives. We also highlight potential research directions to promote future research endeavors for GNNs in wireless communications.
Fast and High Quality Image Denoising via Malleable Convolution

Most image denoising networks apply a single set of static convolutional kernels across the entire input image. This is sub-optimal for natural images, as they often consist of heterogeneous visual patterns. Dynamic convolution tries to address this issue by using per-pixel convolution kernels, but this greatly increases computational cost. In this work, we present Malleable Convolution (MalleConv), which performs spatial-varying processing with minimal computational overhead. MalleConv uses a smaller set of spatially-varying convolution kernels, a compromise between static and per-pixel convolution kernels. These spatially-varying kernels are produced by an efficient predictor network running on a downsampled input, making them much more efficient to compute than per-pixel kernels produced by a full-resolution image, and also enlarging the network's receptive field compared with static kernels. These kernels are then jointly upsampled and applied to a full-resolution feature map through an efficient on-the-fly slicing operator with minimum memory overhead. To demonstrate the effectiveness of MalleConv, we use it to build an efficient denoising network we call MalleNet. MalleNet achieves high-quality results without very deep architectures, making it 8.9 $$\times $$ faster than the best performing denoising algorithms while achieving similar visual quality. We also show that a single MalleConv layer added to a standard convolution-based backbone can significantly reduce the computational cost or boost image quality at a similar cost. More information are on our project page: https://yifanjiang.net/MalleConv.html .
Application of ensemble Machine Learning models for phishing detection on web networks

Phishing is a technique of fraud and identity stealing that includes convincing Website visitors to provide confidential info and details such as their user id, secret key, payment info, and so on. It is one of the real safety concerns that the online revolution is worried about, and it may cost businesses and users money. The use of SHAP values to better comprehend the model employed in phishing URL detection is the research’s highlight. This research examines multiple machine learning models for detecting phishing by examining various aspects of the website’s URL. The dataset that was used to train the model is open source, consisting of datasets from Alexa, UCI, Phishtank, and Kaggle. There are 11,055 rows and 32 columns in the data set. The data were normalized using the SMOTE analysis technique, which resulted in a larger data set. This data was then fed into a variety of classification and ensemble models (K-means, Random forest, decision tree, CatBoost classifier, LightGBM classifier, AdaBoost, and voting classifier). The Accuracy and F1 values of the models were compared. The model’s accuracy was tested before and after using smote. After putting all of the strategies to the test, we observed that CatBoost Classifier produced the best results for accuracy and F1 value. To conclude, SHAP values are a crucial part in model interpretation and are utilized to identify important features in the model and how they impact the output of the model. This model can be used by authorities and companies to stop phishing attacks and identify suspicious sites before someone is harmed by them.
A Robust Hand Gesture Sensing and Recognition Based on Dual-Flow Fusion With FMCW Radar

For hand gesture recognition (HGR) sensing based on frequency modulated continuous wave (FMCW) radar, it is challenging to cope with the ambiguity and variability of gesture motions performed by different peoples due to the private habits or position, which inevitably reduce the robustness of HGR sensing. To solve this problem, we propose a novel dual-flow fusion deformable residual network (DFDRN). More detailed gesture information, time frequency maps and spectrum map videos, are fed into DFDRN, which improves the generalization and motion modeling ability of HGR through the extensive feature space. To further improve the robustness of HGR sensing, the deformable block is exploited to replace the convolution. To focus on the important features of dual-flow fusion, the convolutional block attention module (CBAM) is embedded in the backbone of DFDRN. Experimental results show that the accuracy of DFDRN, which can reach 97.5%, is obviously better than the state-of-the-art HGR methods.
H4D: Human 4D Modeling by Learning Neural Compositional Representation

Despite the impressive results achieved by deep learning based 3D reconstruction, the techniques of directly learning to model 4D human captures with detailed geometry have been less studied. This work presents a novel framework that can effectively learn a compact and compositional representation for dynamic human by exploiting the human body prior from the widely used SMPL parametric model. Particularly, our representation, named H4D, represents a dynamic 3D human over a temporal span with the SMPL parameters of shape and initial pose, and latent codes encoding motion and auxiliary information. A simple yet effective linear motion model is proposed to provide a rough and regularized motion estimation, followed by perframe compensation for pose and geometry details with the residual encoded in the auxiliary code. Technically, we introduce novel GRU-based architectures to facilitate learning and improve the representation capability. Extensive experiments demonstrate our method is not only efficacy in recovering dynamic human with accurate motion and detailed geometry, but also amenable to various 4D human related tasks, including motion retargeting, motion completion and future prediction.
Parallel Pathway Dense Video Captioning With Deformable Transformer

Dense video captioning is a very challenging task because it requires a high-level understanding of the video story, as well as pinpointing details such as objects and motions for a consistent and fluent description of the video. Many existing solutions divide this problem into two sub-tasks, event detection and captioning, and solve them sequentially (“localize-then-describe” or reverse). Consequently, the final outcome is highly dependent on the performance of the preceding modules. Detecting events or generating sentences directly from vast amounts of video information is prone to loss of elaborateness (accuracy and fluency), and hence propagates as a cascade effect to subsequent modules. Parallel decoding approaches have been proposed to overcome these limitations, which leads to another challenge: the bottleneck at the branch point. In this paper, we decompose this sequential approach by proposing a parallel pathway dense video captioning framework (PPVC) that localizes and describes events simultaneously without any bottlenecks. We introduce a representation organization network at the branching point of the parallel pathway to organize the encoded video feature by considering the entire storyline. Then, an event localizer focuses to localize events without any event proposal generation network, a sentence generator describes events while considering the fluency and coherency of sentences. A gating network controls the flow of organized representations to filter out unnecessary information. Our method has several advantages over existing work: (i) the final output does not depend on the output of the preceding modules, (ii) it improves existing parallel decoding methods by relieving the bottleneck of information. We evaluate the performance of PPVC on large-scale benchmark datasets, the ActivityNet Captions, and YouCook2. PPVC not only outperforms existing algorithms on the majority of metrics but also improves on both datasets by 5.4% and 4.9% compared to the state-of-the-art parallel decoding method.
CroMo: Cross-Modal Learning for Monocular Depth Estimation

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning. In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation methods.