Reviewerly/exHarmony huggingface dataset:
Efficient Spatial-Temporal Information Fusion for LiDAR-Based 3D Moving Object Segmentation

Accurate moving object segmentation is an es-sential task for autonomous driving. It can provide effective information for many downstream tasks, such as collision avoidance, path planning, and static map construction. How to effectively exploit the spatial-temporal information is a critical question for 3D LiDAR moving object segmentation (LiDAR-MOS). In this work, we propose a novel deep neural network exploiting both spatial-temporal information and different representation modalities of LiDAR scans to improve LiDAR-MOS performance. Specifically, we first use a range image-based dual-branch structure to separately deal with spatial and temporal information that can be obtained from sequential LiDAR scans, and later combine them using motion-guided attention modules. We also use a point refinement module via 3D sparse convolution to fuse the information from both LiDAR range image and point cloud representations and reduce the artifacts on the borders of the objects. We verify the effectiveness of our proposed approach on the LiDAR-MOS benchmark of SemanticKITTI. Our method outperforms the state-of-the-art methods significantly in terms of LiDAR-MOS IoU. Benefiting from the devised coarse-to-fine architecture, our method operates online at sensor frame rate. Code is available at: https://github.com/haomo-ai/MotionSeg3D.
TeachAugment: Data Augmentation Optimization Using Teacher Knowledge

Optimization of image transformation functions for the purpose of data augmentation has been intensively studied. In particular, adversarial data augmentation strategies, which search augmentation maximizing task loss, show significant improvement in the model generalization for many tasks. However, the existing methods require careful parameter tuning to avoid excessively strong deformations that take away image features critical for acquiring generalization. In this paper, we propose a data augmentation optimization method based on the adversarial strategy called TeachAugment, which can produce informative transformed images to the model without requiring careful tuning by leveraging a teacher model. Specifically, the augmentation is searched so that augmented images are adversarial for the target model and recognizable for the teacher model. We also propose data augmentation using neural networks, which simplifies the search space design and allows for updating of the data augmentation using the gradient method. We show that TeachAugment outperforms existing methods in experiments of image classification, semantic segmentation, and unsupervised representation learning tasks.
YOLO-Former: Marrying YOLO and Transformer for Foreign Object Detection

The automatic detection of foreign objects between platform screen doors (PSDs) and metro train doors significantly affects personnel and property safety and maintains the train’s normal operation. However, some existing works only determine the presence of foreign objects but cannot indicate their categories. Besides, although deep-learning-based object detection algorithms can indicate the presence and categories of foreign objects, most of them only harness the information in region proposals, ignoring global contextual information. Furthermore, their performance comes at the considerable cost of computational complexity, and leading cannot be well deployed in the metro environment. To address these issues and better implement foreign object detection (FOD), we present You Only Look Once-Transformer (YOLO-Former), a simple but efficient model. YOLO-Former is accomplished based on YOLOv5 through the following procedure. First, the vision transformer (ViT) is introduced for dynamic attention and global modeling, thereby solving the problem that the original YOLOv5 only utilizes information in region proposals and has insufficient ability to capture global information. Second, the convolutional block attention module (CBAM) and Stem module are used to improve feature expression ability further and reduce floating point operations (FLOPs). Finally, we design various variants with different widths and depths to meet every need. Experiments on the foreign object detection dataset (FODD) and PASCAL VOC dataset demonstrate that YOLO-Former-x consistently outperforms other state-of-the-arts with significant margins (0.5 to 11.3 mean average precision, mAP, on FODD and 0.6 to 13.6 on PASCAL VOC dataset). Last but not least, YOLO-Former-x maintains real-time processing speed (27.32 and 28.17 frame per second, FPS, on TITAN Xp).
Modality-Agnostic Learning for Radar-Lidar Fusion in Vehicle Detection

Fusion of multiple sensor modalities such as camera, Lidar, and Radar, which are commonly found on autonomous vehicles, not only allows for accurate detection but also robustifies perception against adverse weather conditions and individual sensor failures. Due to inherent sensor characteristics, Radar performs well under extreme weather conditions (snow, rain, fog) that significantly degrade camera and Lidar. Recently, a few works have developed vehicle detection methods fusing Lidar and Radar signals, i.e., MVD-Net. However, these models are typically developed under the assumption that the models always have access to two error-free sensor streams. If one of the sensors is unavailable or missing, the model may fail catastrophically. To mitigate this problem, we propose the Self-Training Multimodal Vehicle Detection Network (ST-MVDNet) which leverages a Teacher-Student mutual learning framework and a simulated sensor noise model used in strong data augmentation for Lidar and Radar. We show that by (1) enforcing output consistency between a Teacher network and a Student network and by (2) introducing missing modalities (strong augmentations) during training, our learned model breaks away from the error-free sensor assumption. This consistency enforcement enables the Student model to handle missing data properly and improve the Teacher model by updating it with the Student model's exponential moving average. Our experiments demonstrate that our proposed learning framework for multi-modal detection is able to better handle missing sensor data during inference. Furthermore, our method achieves new state-of-the-art performance (5% gain) on the Oxford Radar Robotcar dataset under various evaluation settings.
Task Discrepancy Maximization for Fine-grained Few-Shot Classification

Recognizing discriminative details such as eyes and beaks is important for distinguishing fine-grained classes since they have similar overall appearances. In this regard, we introduce Task Discrepancy Maximization (TDM), a simple module for fine-grained few-shot classification. Our objective is to localize the class-wise discriminative regions by highlighting channels encoding distinct information of the class. Specifically, TDM learns task-specific channel weights based on two novel components: Support Attention Module (SAM) and Query Attention Module (QAM). SAM produces a support weight to represent channel-wise discriminative power for each class. Still, since the SAM is basically only based on the labeled support sets, it can be vulnerable to bias toward such support set. Therefore, we propose QAM which complements SAM by yielding a query weight that grants more weight to object-relevant channels for a given query image. By combining these two weights, a class-wise task-specific channel weight is defined. The weights are then applied to produce task-adaptive feature maps more focusing on the discriminative details. Our experiments validate the effectiveness of TDM and its complementary benefits with prior methods in fine- grained few-shot classification.
Memory-Augmented Non-Local Attention for Video Super-Resolution

In this paper, we propose a simple yet effective video super-resolution method that aims at generating highfidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor frames to assist the super-resolution of the current frame. Those methods achieve limited performance as they suffer from the challenges in spatial frame alignment and the lack of useful information from similar LR neighbor frames. In contrast, we devise a cross-frame non-local attention mechanism that allows video superresolution without frame alignment, leading to being more robust to large motions in the video. In addition, to acquire general video prior information beyond neighbor frames, and to compensate for the information loss caused by large motions, we design a novel memory-augmented attention module to memorize general video details during the superresolution training. We have thoroughly evaluated our work on various challenging datasets. Compared to other recent video super-resolution approaches, our method not only achieves significant performance gains on large motion videos but also shows better generalization. Our source code and the new Parkour benchmark dataset is available at https://github.com/jiy173/MANA.
Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation

Optical flow is an easily conceived and precious cue for advancing unsupervised video object segmentation (UVOS). Most of the previous methods directly extract and fuse the motion and appearance features for segmenting target objects in the UVOS setting. However, optical flow is intrinsically an instantaneous velocity of all pixels among consecutive frames, thus making the motion features not aligned well with the primary objects among the corresponding frames. To solve the above challenge, we propose a concise, practical, and efficient architecture for appearance and motion feature alignment, dubbed hierarchical feature alignment network (HFAN). Specifically, the key merits in HFAN are the sequential Feature AlignMent (FAM) module and the Feature AdaptaTion (FAT) module, which are leveraged for processing the appearance and motion features hierarchically. FAM is capable of aligning both appearance and motion features with the primary object semantic representations, respectively. Further, FAT is explicitly designed for the adaptive fusion of appearance and motion features to achieve a desirable trade-off between cross-modal features. Extensive experiments demonstrate the effectiveness of the proposed HFAN, which reaches a new state-of-the-art performance on DAVIS-16, achieving 88.7 $$ \mathcal {J} \& \mathcal {F}$$ Mean, i.e., a relative improvement of 3.5% over the best published result.
A Virtual Projection-Based Incentive Routing Protocol for Mobile PFNs in Wireless Sensor Networks

The current Wireless Sensor Network (WSN) lacks the desired power owning to the non-availability of the power source. The challenge posed in the current scenario is conserving power for effective data transfer, which is seldom achieved. Added to the owes is the evolving technological innovations where the physical size of the Packet Forwarding Nodes (PFNs) is also reduced along with its power source, thereby reducing the power available to the system and making it inefficient. The existing methodology employed in monitoring the parameters used in Epidemic Algorithm (EA) and Incentive Compatible Routing Protocol (ICRP) protocols which are power guzzlers, the power available for data transfer is greatly reduced making the entire system inefficient. This paper attempts to mitigate the challenges posed by the EA &amp; ICRP, the proposed protocol, “Incentive Routing Protocol with Virtual Projection (IRPVP)”, employs a Relay Sensing Node (RSN) which is designed to be distributed in a square cross-sectional area where each node acts as a unique Sensing Point (SP) monitoring each of the essential parameters like energy consumption, vibrant &amp; non-vibrant SPN count, residual energy and routing overhead while still retaining power for data transfer since the RSN is connected to a dedicated power source. In IRPVP protocol, each packet of a node is subdivided into fragments that are designed to have fixed or variable lengths depending upon the application. Each of these packets is sent over multiple Packet Forwarding Nodes (PFNs) towards the data center. The selection of PFNs in the path is based on their trust levels like meeting probability, computation of residual energy, data weight, and security value. Special PFNs are placed in the network and are entrusted to deliver the packets to the data center without data loss during transmission. The result of the IRPVP protocol Vis-à-Vis the EA &amp; ICRP protocol, backed by the simulation results proves that the IRPVP protocol is better in data handling and is more efficient.
Bridging the Gap between Classification and Localization for Weakly Supervised Object Localization

Weakly supervised object localization aims to find a target object region in a given image with only weak supervision, such as image-level labels. Most existing methods use a class activation map (CAM) to generate a localization map; however, a CAM identifies only the most discriminative parts of a target object rather than the entire object region. In this work, we find the gap between classification and localization in terms of the misalignment of the directions between an input feature and a class-specific weight. We demonstrate that the misalignment suppresses the activation of CAM in areas that are less discriminative but belong to the target object. To bridge the gap, we propose a method to align feature directions with a class-specific weight. The proposed method achieves a state-of-the-art localization performance on the CUB-200-2011 and ImageNet-1K benchmarks.
Latency-Aware Collaborative Perception

Collaborative perception has recently shown great potential to improve perception capabilities over single-agent perception. Existing collaborative perception methods usually consider an ideal communication environment. However, in practice, the communication system inevitably suffers from latency issues, causing potential performance degradation and high risks in safety-critical applications, such as autonomous driving. To mitigate the effect caused by the inevitable latency, from a machine learning perspective, we present the first latency-aware collaborative perception system, which actively adapts asynchronous perceptual features from multiple agents to the same time stamp, promoting the robustness and effectiveness of collaboration. To achieve such a feature-level synchronization, we propose a novel latency compensation module, called SyncNet, which leverages feature-attention symbiotic estimation and time modulation techniques. Experiments results show that the proposed latency aware collaborative perception system with SyncNet can outperforms the state-of-the-art collaborative perception method by 15.6% in the communication latency scenario and keep collaborative perception being superior to single agent perception under severe latency.
Hierarchical Deep Reinforcement Learning With Experience Sharing for Metaverse in Education

Metaverse has gained increasing interest in education, with much of literature focusing on its great potential to enhance both individual and social aspects of learning. However, little work has been done to address the systems and technologies behind providing meaningful Metaverse learning. This article proposes a technical framework to address this research gap, where a hierarchical multiagent reinforcement learning approach with experience sharing is developed to augment the intelligence of nonplayer characters in Metaverse learning for personalization. The utility and benefits of the proposed framework and methodologies are demonstrated in Gridlock, a Metaverse learning game, as well as through extensive simulations.
Bringing Old Films Back to Life

We present a learning-based framework, recurrent transformer network (RTN), to restore heavily degraded old films. Instead of performing frame-wise restoration, our method is based on the hidden knowledge learned from adjacent frames that contain abundant information about the occlusion, which is beneficial to restore challenging artifacts of each frame while ensuring temporal coherency. Moreover, contrasting the representation of the current frame and the hidden knowledge makes it possible to infer the scratch position in an unsupervised manner, and such defect localization generalizes well to real-world degradations. To better resolve mixed degradation and compensate for the flow estimation error during frame alignment, we propose to leverage more expressive transformer blocks for spatial restoration. Experiments on both synthetic dataset and real-world old films demonstrate the significant superiority of the proposed RTN over existing solutions. In addition, the same framework can effectively propagate the color from keyframes to the whole video, ultimately yielding compelling restored films. The implementation and model will be released at https://github.com/raywzy/Bringing-Old-Films-Back-to-Life.
Improved Pixel and Bit Confusion-Diffusion Based on Mixed Chaos and Hash Operation for Image Encryption

Confusion and diffusion are two main concepts in cryptography that affect the quality of encryption and resistance to various attacks.Both of these concepts are influenced by the quality of the keystream used.Keystream generation needs to involve plain images as one way to bind sensitivity to plain images.But when it involves plain images directly, encryption security has negative side effects.So this research proposes to perform a diffusion operation with a logistic map on a plain image before generating the keystream.Not only using one keystream, the proposed method also uses three other keystreams and Arnold transform whose input parameters are processed by hash operations.Two keystreams are generated using the four-dimensional Lorenz system for the permutation of column and row image pixels.Next, sixteen Arnold transforms with different parameters are used for bit permutation.The last keystream is generated with a Henon map to carry out the diffusion stage two so that a cipher image is produced.Based on the testing results, the proposed method has satisfactory encryption results that are resistant to various attacks, especially brute force, statistical, and differential attacks.
Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers

Transformer encoder architectures have recently achieved state-of-the-art results on monocular 3D human mesh reconstruction, but they require a substantial number of parameters and expensive computations. Due to the large memory overhead and slow inference speed, it is difficult to deploy such models for practical use. In this paper, we propose a novel transformer encoder-decoder architecture for 3D human mesh reconstruction from a single image, called FastMETRO. We identify the performance bottleneck in the encoder-based transformers is caused by the token design which introduces high complexity interactions among input tokens. We disentangle the interactions via an encoder-decoder architecture, which allows our model to demand much fewer parameters and shorter inference time. In addition, we impose the prior knowledge of human body’s morphological relationship via attention masking and mesh upsampling operations, which leads to faster convergence with higher accuracy. Our FastMETRO improves the Pareto-front of accuracy and efficiency, and clearly outperforms image-based methods on Human3.6M and 3DPW. Furthermore, we validate its generalizability on FreiHAND.
Deep Visual Geo-localization Benchmark

In this paper, we propose a new open-source benchmarkingframeworkfor Visual Geo-localization (VG) that allows to build, train, and test a wide range of commonly used ar-chitectures, with the flexibility to change individual components of a geo-localization pipeline. The purpose of this framework is twofold: i) gaining insights into how differ-ent components and design choices in a VG pipeline im-pact the final results, both in terms of performance (re-call@N metric) and system requirements (such as execution time and memory consumption); ii) establish a system-atic evaluation protocol for comparing different methods. Using the proposed framework, we perform a large suite of experiments which provide criteria for choosing back-bone, aggregation and negative mining depending on the use-case and requirements. We also assess the impact of engineering techniques like pre/post-processing, data aug-mentation and image resizing, showing that better performance can be obtained through somewhat simple procedures: for example, downscaling the images' resolution to 80% can lead to similar results with a 36% savings in ex-traction time and dataset storage requirement. Code and trained models are available at dataset storage requirement. https://deep-vg-bench.herokuapp.com/.
SimpleRecon: 3D Reconstruction Without 3D Convolutions

Traditionally, 3D indoor scene reconstruction from posed images happens in two phases: per-image depth estimation, followed by depth merging and surface reconstruction. Recently, a family of methods have emerged that perform reconstruction directly in final 3D volumetric feature space. While these methods have shown impressive reconstruction results, they rely on expensive 3D convolutional layers, limiting their application in resource-constrained environments. In this work, we instead go back to the traditional route, and show how focusing on high quality multi-view depth prediction leads to highly accurate 3D reconstructions using simple off-the-shelf depth fusion. We propose a simple state-of-the-art multi-view depth estimator with two main contributions: 1) a carefully-designed 2D CNN which utilizes strong image priors alongside a plane-sweep feature volume and geometric losses, combined with 2) the integration of keyframe and geometric metadata into the cost volume which allows informed depth plane scoring. Our method achieves a significant lead over the current state-of-the-art for depth estimation and close or better for 3D reconstruction on ScanNet and 7-Scenes, yet still allows for online real-time low-memory reconstruction. Code, models and results are available at https://nianticlabs.github.io/simplerecon .
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

The limited availability of annotated data often hinders real-world applications of machine learning. To efficiently learn from small quantities of multimodal data, we leverage the linguistic knowledge from a large pre-trained language model (PLM) and quickly adapt it to new domains of image captioning. To effectively utilize a pretrained model, it is critical to balance the visual input and prior linguistic knowledge from pretraining. We propose VisualGPT, which employs a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the PLM with a small amount of in-domain image-text data. The proposed self-resurrecting activation unit produces sparse activations that prevent accidental overwriting of linguistic knowledge. When trained on 0.1%, 0.5% and 1% of the respective training sets, VisualGPT surpasses the best baseline by up to 10.0% CIDEr on MS COCO [43] and 17.9% CIDEr on Conceptual Captions [63]. Furthermore, VisualGPT achieves the state-of-the-art result on IU X-ray [15], a medical report generation dataset. Our code is available at https://github.com/Vision-CAIR/VisualGPT.