Reviewerly/exHarmony huggingface dataset:
DID-M3D: Decoupling Instance Depth for Monocular 3D Object Detection

Monocular 3D detection has drawn much attention from the community due to its low cost and setup simplicity. It takes an RGB image as input and predicts 3D boxes in the 3D space. The most challenging sub-task lies in the instance depth estimation. Previous works usually use a direct estimation method. However, in this paper we point out that the instance depth on the RGB image is non-intuitive. It is coupled by visual depth clues and instance attribute clues, making it hard to be directly learned in the network. Therefore, we propose to reformulate the instance depth to the combination of the instance visual surface depth (visual depth) and the instance attribute depth (attribute depth). The visual depth is related to objects’ appearances and positions on the image. By contrast, the attribute depth relies on objects’ inherent attributes, which are invariant to the object affine transformation on the image. Correspondingly, we decouple the 3D location uncertainty into visual depth uncertainty and attribute depth uncertainty. By combining different types of depths and associated uncertainties, we can obtain the final instance depth. Furthermore, data augmentation in monocular 3D detection is usually limited due to the physical nature, hindering the boost of performance. Based on the proposed instance depth disentanglement strategy, we can alleviate this problem. Evaluated on KITTI, our method achieves new state-of-the-art results, and extensive ablation studies validate the effectiveness of each component in our method. The codes are released at https://github.com/SPengLiang/DID-M3D .
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo

Contrastive learning (CL) is widely known to require many negative samples, 65536 in MoCo for instance, for which the performance of a dictionary-free framework is often inferior because the negative sample size (NSS) is limited by its mini-batch size (MBS). To decouple the NSS from the MBS, a dynamic dictionary has been adopted in a large volume of CL frameworks, among which arguably the most popular one is MoCo family. In essence, MoCo adopts a momentum-based queue dictionary, for which we perform a fine-grained analysis of its size and consistency. We point out that InfoNCE loss used in MoCo implicitly attract anchors to their corresponding positive sample with various strength of penalties and identify such inter-anchor hardness-awareness property as a major reason for the necessity of a large dictionary. Our findings motivate us to simplify MoCo v2 via the removal of its dictionary as well as momentum. Based on an InfoNCE with the proposed dual temperature, our simplified frameworks, Sim-MoCo and SimCo, outperform MoCo v2 by a visible margin. Moreover, our work bridges the gap between CL and non-CL frameworks, contributing to a more unified under-standing of these two mainstream frameworks in SSL. Code is available at: https://bit.ly/3LkQbaT.
Game Theory-Based Optimal Cooperative Path Planning for Multiple UAVs

This paper presents new cooperative path planning algorithms for multiple unmanned aerial vehicles (UAVs) using Game theory-based particle swarm optimization (GPSO). First, the formation path planning is formulated into the minimization of a cost function that incorporates multiple objectives and constraints for each UAV. A framework based on game theory is then developed to cast the minimization into the problem of finding a Stackelberg-Nash equilibrium. Next, hierarchical particle swarm optimization algorithms are developed to obtain the global optimal solution. Simulation results show that the GPSO algorithm can generate efficient and feasible flight paths for multiple UAVs, outperforming other path planning methods in terms of convergence rate and flexibility. The formation can adjust its geometrical shape to accommodate a working environment. Experimental tests on a group of three UAVs confirm the advantages of the proposed approach for a practical application.
Continuous Scene Representations for Embodied AI

We propose Continuous Scene Representations (CSR), a scene representation constructed by an embodied agent navigating within a space, where objects and their relationships are modeled by continuous valued embeddings. Our method captures feature relationships between objects, composes them into a graph structure on-the-fly, and situates an embodied agent within the representation. Our key insight is to embed pair-wise relationships between objects in a latent space. This allows for a richer representation compared to discrete relations (e.g., [SUPPORT], [NEXT-TO]) commonly used for building scene representations. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations. Using CSR, we outperform state-of-the-art approaches for the challenging downstream task of visual room rearrangement, without any task specific training. Moreover, we show the learned embeddings capture salient spatial details of the scene and show applicability to real world data. A summery video and code is available at prior.allenai.org/projects/csr.
Monocular 3D Object Detection with Depth from Motion

Perceiving 3D objects from monocular inputs is crucial for robotic systems, given its economy compared to multi-sensor settings. It is notably difficult as a single image can not provide any clues for predicting absolute depth values. Motivated by binocular methods for 3D object detection, we take advantage of the strong geometry structure provided by camera ego-motion for accurate object depth estimation and detection. We first make a theoretical analysis on this general two-view case and notice two challenges: 1) Cumulative errors from multiple estimations that make the direct prediction intractable; 2) Inherent dilemmas caused by static cameras and matching ambiguity. Accordingly, we establish the stereo correspondence with a geometry-aware cost volume as the alternative for depth estimation and further compensate it with monocular understanding to address the second problem. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. We also present a pose-free DfM to make it usable when the camera pose is unavailable. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark. Detailed quantitative and qualitative analyses also validate our theoretical conclusions. The code is released at https://github.com/Tai-Wang/Depth-from-Motion .
Enhancing Face Recognition with Self-Supervised 3D Reconstruction

Attributed to both the development of deep networks and abundant data, automatic face recognition (FR) has quickly reached human-level capacity in the past few years. However, the FR problem is not perfectly solved in case of uncontrolled illumination and pose. In this paper, we propose to enhance face recognition with a bypass of self-supervised 3D reconstruction, which enforces the neural backbone to focus on the identity-related depth and albedo information while neglects the identity-irrelevant pose and illumination information. Specifically, inspired by the physical model of image formation, we improve the backbone FR network by introducing a 3D face reconstruction loss with two auxiliary networks. The first one estimates the pose and illumination from the input face image while the second one decodes the canonical depth and albedo from the intermediate feature of the FR backbone network. The whole network is trained in end-to-end manner with both classic face identification loss and the loss of 3D face reconstruction with the physical parameters. In this way, the self-supervised reconstruction acts as a regularization that enables the recognition network to understand faces in 3D view, and the learnt features are forced to encode more information of canonical facial depth and albedo, which is more intrinsic and beneficial to face recognition. Extensive experimental results on various face recognition benchmarks show that, without any cost of extra annotations and computations, our method outperforms state-of-the-art ones. Moreover, the learnt representations can also well generalize to other face-related downstream tasks such as the facial attribute recognition with limited labeled data.
Novel Class Discovery in Semantic Segmentation

We introduce a new setting of Novel Class Discovery in Semantic Segmentation (NCDSS), which aims at segmenting unlabeled images containing new classes given prior knowledge from a labeled set of disjoint classes.In contrast to existing approaches that look at novel class discovery in image classification, we focus on the more challenging semantic segmentation.In NCDSS, we need to distinguish the objects and background, and to handle the existence of multiple classes within an image, which increases the difficulty in using the unlabeled data.To tackle this new setting, we leverage the labeled base data and a saliency model to coarsely cluster novel classes for model training in our basic framework.Additionally, we propose the Entropy-based Uncertainty Modeling and Self-training (EUMS) framework to overcome noisy pseudo-labels, further improving the model performance on the novel classes.Our EUMS utilizes an entropy ranking technique and a dynamic reassignment to distill clean labels, thereby making full use of the noisy data via self-supervised learning.We build the NCDSS benchmark on the PASCAL-5 i dataset and COCO-20 i dataset.Extensive experiments demonstrate the feasibility of the basic framework (achieving an average mIoU of 49.81% on PASCAL-5 i ) and the effectiveness of EUMS framework (outperforming the basic framework by 9.28% mIoU on PASCAL-5 i ).
Analysis of Safe Storage of Network Information Data and Financial Risks Under Blockchain Combined With Edge Computing

To discuss the control of financial risks (FRs) under blockchain (BC) and improve network information security (NIS) and data security, edge computing (EC) combined with BC is proposed to control the risks of the big data (BD) financial system. Firstly, the BC-based financial system is introduced, and the characteristics of BC such as decentralization, tamper-resistant, and smart contract are analyzed. Secondly, the development status of NIS and the characteristics of marginal computing are explained, and the control model of NIS is established. Then, EC is used to encrypt the identity authentication system to ensure data security, and the BC-based FR evaluation model is established. Finally, a questionnaire is designed regarding the NIS model, and the results are analyzed. A simulation experiment is conducted regarding the index evaluation of the BC-based FR evaluation model. The experimental results indicate that network personnel control, environment, and technology have positive effects on NIS, and the impact factors are 0.26, 0.24, and 0.33, respectively.
PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes

Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/ViLab-UCSD/PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.
Universal Linear Optics for Ultra-Fast Neuromorphic Silicon Photonics Towards fJ/MAC and TMAC/sec/mm<sup>2</sup> Engines

The field of neuromorphic photonics has been projected to comprise the next-generation Neural Network platform, expected to lead to remarkable advances in compute energy- and area-efficiency metrics. Herein, we review the performance of state-of-the-art neuromorphic photonic demonstrators, summarizing the impact of the circuit architecture and employed weight technology on the system credentials in terms of scalability, energy- and footprint-efficiency. We provide an overview of a recently demonstrated photonic crossbar multi-port interferometer, holding significant insertion loss, technology versatility and robustness advantages over all state-of-the-art linear optical layouts. This novel linear optical circuit architecture is then transferred onto an integrated silicon photonic (SiPho) platform, realizing a single-column crossbar and selecting SiGe electro-absorption modulators (EAM) for both its fan in and weighting stages. This single-neuron coherent SiPho prototype is then experimentally benchmarked on the MNIST dataset, allowing for record-high compute-rates up to 50 GHz/axon. The classification accuracy was studied for compute rates ranging between 16-50 GHz, revealing 99.03% accuracy at 16 GHz that reduces by only 3.79% at 50 GHz. Finally, we investigate the interdependence of compute rate, bit resolution and energy efficiency, in photonic accelerator layouts and benchmark our proposed Xbar architecture against state-of-the-art photonic and electronic accelerators. The analysis reveals an energy- and footprint-efficiency of 54 fJ/MAC and 1.54 TMAC/s/mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , respectively, shaping in this way a promising roadmap for next-generation neuromorphic accelerators.
iFS-RCNN: An Incremental Few-shot Instance Segmenter

This paper addresses incremental few-shot instance seg-mentation, where a few examples of new object classes ar-rive when access to training examples of old classes is not available anymore, and the goal is to perform well on both old and new classes. We make two contributions by extending the common Mask-RCNN framework in its second stage - namely, we specify a new object class classifier based on the probit function and a new uncertainty-guided bounding-box predictor. The former leverages Bayesian learning to address a paucity of training examples of new classes. The latter learns not only to predict object bounding boxes but also to estimate the uncertainty of the prediction as a guid-ance for bounding box refinement. We also specify two new loss functions in terms of the estimated object-class distribution and bounding-box uncertainty. Our contributions produce significant performance gains on the COCO dataset over the state of the art - specifically, the gain of +6 on the new classes and +16 on the old classes in the AP instance segmentation metric. Furthermore, we are the first to evaluate the incremental few-shot setting on the more challenging LVIS dataset.
Does Robustness on ImageNet Transfer to Downstream Tasks?

As clean ImageNet accuracy nears its ceiling, the re-search community is increasingly more concerned about ro-bust accuracy under distributional shifts. While a variety of methods have been proposed to robustify neural networks, these techniques often target models trained on ImageNet classification. At the same time, it is a common practice to use ImageNet pretrained backbones for downstream tasks such as object detection, semantic segmentation, and image classification from different domains. This raises a question: Can these robust image classifiers transfer robustness to downstream tasks? For object detection and semantic segmentation, we find that a vanilla Swin Transformer, a variant of Vision Transformer tailored for dense prediction tasks, transfers robustness better than Convolutional Neu-ral Networks that are trained to be robust to the corrupted version of ImageNet. For CIFAR10 classification, we find that models that are robustified for ImageNet do not re-tain robustness when fully fine-tuned. These findings sug-gest that current robustification techniques tend to empha-size ImageNet evaluations. Moreover, network architecture is a strong source of robustness when we consider transfer learning.
Secure Trust-Based Delegated Consensus for Blockchain Frameworks Using Deep Reinforcement Learning

IoT (IoT) networks generate massive amounts of data while supporting various applications, where the security and protection of IoT data are very important. In particular, blockchain technology supporting IoT networks is considered as the most secure, expandable, and scalable database storage solution. However, existing blockchain systems have scalability problems due to low throughput and high resource consumption, and security problems due to malicious attacks. Several studies have proposed blockchain technologies that can improve the scalability or the security level, but there have been few studies that improve both at the same time. In addition, most existing studies do not consider malicious attack scenarios in the consensus process, which deteriorates the blockchain security level. In order to solve the scalability and security problems simultaneously, this paper proposes a Dueling Double Deep-Q-network with Prioritized experience replay (D3P) based secure trust-based delegated consensus blockchain (TDCB-D3P) scheme that optimizes the blockchain performance by applying deep reinforcement learning (DRL) technology. The TDCB-D3P scheme uses a trust system with a delegated consensus algorithm to ensure the security level and reduce computing costs. In addition, DRL is used to compute the optimum blockchain parameters under the dynamic network state and maximize the transactions per second (TPS) performance and security level. The simulation results show that the TDCB-D3P scheme can provide a superior TPS and resource consumption performance. Furthermore, in blockchain networks with malicious nodes, the simulation results show that the proposed scheme significantly improves the security level when compared to existing blockchain schemes by effectively reducing the influence of malicious nodes.
Deformable Sprites for Unsupervised Video Decomposition

We describe a method to extract persistent elements of a dynamic scene from an input video. We represent each scene element as a Deformable Sprite consisting of three components: 1) a 2D texture image for the entire video, 2) per-frame masks for the element, and 3) non-rigid deformations that map the texture image into each video frame. The resulting decomposition allows for applications such as consistent video editing. Deformable Sprites are a type of video auto-encoder model that is optimized on individual videos, and does not require training on a large dataset, nor does it rely on pretrained models. Moreover, our method does not require object masks or other user input, and discovers moving objects of a wider variety than previous work. We evaluate our approach on standard video datasets and show qualitative results on a diverse array of Internet videos.
Deep Unlearning via Randomized Conditionally Independent Hessians

Recent legislation has led to interest in machine unlearning, i.e., removing specific training samples from a predictive model as if they never existed in the training dataset. Unlearning may also be required due to corrupted/adversarial data or simply a user's updated privacy requirement. For models which require no training (k-NN), simply deleting the closest original sample can be effective. But this idea is inapplicable to models which learn richer representations. Recent ideas leveraging optimization-based updates scale poorly with the model dimension d, due to inverting the Hessian of the loss function. We use a variant of a new conditional independence coefficient, L-CODEC, to identify a subset of the model parameters with the most semantic overlap on an individual sample level. Our approach completely avoids the need to invert a (possibly) huge matrix. By utilizing a Markov blanket selection, we premise that L-CODEC is also suitable for deep unlearning, as well as other applications in vision. Compared to alternatives, L-CODEC makes approximate unlearning possible in settings that would otherwise be infeasible, including vision models used for face recognition, person re-identification and NLP models that may require unlearning samples identified for exclusion. Code is available at https://github.com/vsingh-group/LCODEC-deep-unlearning.
Class-Incremental Learning with Strong Pre-trained Models

Class-incremental learning (CIL) has been widely stud-ied under the setting of starting from a small number of classes (base classes). Instead, we explore an understud-ied real-world setting of CIL that starts with a strong model pre-trained on a large number of base classes. We hypoth-esize that a strong base model can provide a good repre-sentation for novel classes and incremental learning can be done with small adaptations. We propose a 2-stage training scheme, i) feature augmentation - cloning part of the backbone and fine-tuning it on the novel data, and ii) fusion - combining the base and novel classifiers into a unified classifier. Experiments show that the proposed method sig-nificantly outperforms state-of-the-art CIL methods on the large-scale ImageNet dataset (e.g. + 10% overall accuracy than the best). We also propose and analyze understudied practical CIL scenarios, such as base-novel overlap with distribution shift. Our proposed method is robust and gen-eralizes to all analyzed CIL settings.
Deep Neural Network for Blind Visual Quality Assessment of 4K Content

The 4K content can deliver a more immersive visual experience to consumers due to the huge improvement in spatial resolution. However, the high spatial resolution brings a great challenge for video transmission and storage. Therefore, it is necessary to compress or downscale the 4K content before transmitting it to end-users. Existing blind image quality assessment (BIQA) methods are not suitable for 4K contents due to the high spatial resolution and specific distortions caused by upscaling methods. In this paper, we propose a deep learning-based BIQA model for 4K content, aiming to recognize true and pseudo 4K content and meanwhile evaluate their visual quality. Considering the characteristic that high spatial resolution can represent more abundant high-frequency information, we first propose a Grey-level Co-occurrence Matrix (GLCM) based texture complexity measure to select three representative image patches from a 4K image, which can reduce the computational complexity and is proven to be very effective for the overall quality prediction. Then, we extract various visual features from the intermediate layers of the convolutional neural network (CNN) and integrate them into the quality-aware feature representation. Finally, two multilayer perception (MLP) networks are utilized to map the quality-aware features into the class probability and the quality score of each patch respectively. The overall quality index is obtained through averaging the results of all patches. The proposed model is trained via the multi-task learning manner and the uncertainty principle is introduced to balance the losses of the classification and regression tasks. The experimental results show that the proposed model outperforms all compared BIQA metrics on four 4K content quality assessment databases.