Reviewerly/exHarmony huggingface dataset:
WOA-DBSCAN: Application of Whale Optimization Algorithm in DBSCAN Parameter Adaption

Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is a classic density-based clustering method that can identify clusters of arbitrary shapes in noisy datasets.However, DBSCAN requires two input parameters: the neighborhood distance value (Eps) and the minimum number of sample points in its neighborhood (MinPts), to perform clustering on a dataset.The quality of clustering is highly sensitive to these two parameters.To tackle this issue, this paper introduces a parameter-adaptive DBSCAN clustering algorithm based on the Whale Optimization Algorithm (WOA-DBSCAN).The algorithm determines the parameter range based on the dataset distribution and utilizes the silhouette coefficient as the objective function.It iteratively selects the two input parameters of DBSCAN within the parameter range using the WOA.This approach ultimately achieves adaptive clustering of DBSCAN.Experimental results on five typical artificial datasets and six real UCI datasets demonstrate the effectiveness of the proposed WOA-DBSCAN algorithm.Compared with DBSCAN and its related optimization algorithms, WOA-DBSCAN shows significant improvements.The F-values of WOA-DBSCAN increased by 9.8%, 13.2%, and 2% respectively in two-dimensional artificial datasets.Additionally, the accuracy values on low to medium dimensional real datasets increased by 22.3%, 10%, and 23.3%.Hence, WOA-DBSCAN can maintain the clustering ability of DBSCAN while achieving adaptive parameter clustering.
Hyperspectral Image Denoising Using Nonconvex Fraction Function

Hyperspectral image (HSI) denoising is a challenging task, not only because it is unavoidably contaminated by severe mixed noises, but also for its hard-to-recover spatial-spectral structure. Since it has been found that HSI has low-rank property, low-rank models have received extensive attention in dealing with the HSI denoising task. However, these models either use nuclear norm, which can only obtain sub-optimal solutions, or require some predefined information that is difficult to determine. To address these issues, in this paper we propose a new HSI denoising model based on non-convex fraction function, which has excellent performance in removing mixed noises. Specifically, the proposed model can capture the rank information of HSI automatically, which allows it to separate clean HSI from noises more accurately. Then, an iterative optimization algorithm is developed by exploiting the framework of the augmented Lagrange multiplier (ALM). Meanwhile, the subproblems at each iteration can be solved by the proximal operator with a closed-form solution. Besides, the convergence of the proposed algorithm is also provided theoretically. Extensive experiments implemented with simulated and real datasets demonstrate that our proposed model performs better than state-of-the-art models in HSI denoising. MATLAB code is available at https://github.com/wangzhi-swu/HSI-Denosing.
An Efficient Multi-Scale Feature Compression With QP-Adaptive Feature Channel Truncation for Video Coding for Machines

Machine vision-based intelligent applications that analyze video data collected by machines are rapidly increasing. Therefore, it is essential to efficiently compress a large volume of video data for machine consumption. Accordingly, the Moving Picture Experts Group (MPEG) has been developing a new video coding standard called Video Coding for Machines (VCM), aimed at video consumed by machines rather than humans. Recently, studies have demonstrated that multi-scale feature compression (MSFC)-based feature compression methods significantly improve the performance of MPEG-VCM. This paper proposes an efficient MSFC (eMSFC) method with quantization parameter (QP)-adaptive feature channel truncation. The proposed eMSFC incorporates an MSFC network with a selective learning strategy (SLS) and Versatile Video Coding (VVC)-based compression. The SLS extracts a single-scale feature from the input image, arranged in order of channel-wise importance. The size of the single-scale feature is adaptively adjusted by truncating the feature channels according to the QP. The truncated feature is efficiently compressed using VVC. Compared to the VCM feature anchor, the experimental results reveal that the proposed method provides a 98.72%, 98.34%, and 98.04% Bjontegaard delta rate gain for machine vision tasks of instance segmentation, object detection, and object tracking, respectively. The proposed method performed best among the “Call for Evidence” response technologies in MPEG-VCM.
Optimizing free layer of Magnetic Tunnel Junction for true random number generator

True random number generators (TRNGs) should ideally generate lengthy chains of non-repeating, uncorrelated bit-streams that are efficient in terms of both energy and area. Current TRNG designs include source of randomness such as CMOS or non-volatile memory based devices with additional circuitry to improve the quality of randomness leading to power and area overhead. This paper addresses these issues by improving the randomness of the Spin-Orbit Torque (SOT)-Magnetic Tunnel Junction cell. Motivated by the observation that free layer thickness of Magnetic Tunnel Junction (MTJ) can be scaled to design a low-barrier device, the paper proposes a novel source of randomness called ΔSOT. This device is then used to design TRNG circuits that achieves high quality random telegraphic switching behavior without any additional circuitry making it suitable for ultra-low power applications. Evaluations show that ΔSOT-TRNG has significant reduction in energy (51%) and area (66%) compared to state-of-the-art MTJ based TRNG design. Furthermore, the work shows that the improved switching speed of the reduced barrier junction can results in 65% increase in throughput compared to MTJ based TRNG design.
Physically Realizable Natural-Looking Clothing Textures Evade Person Detectors via 3D Modeling

Recent works have proposed to craft adversarial clothes for evading person detectors, while they are either only effective at limited viewing angles or very conspicuous to humans. We aim to craft adversarial texture for clothes based on 3D modeling, an idea that has been used to craft rigid adversarial objects such as a 3D-printed turtle. Unlike rigid objects, humans and clothes are non-rigid, leading to difficulties in physical realization. In order to craft natural-looking adversarial clothes that can evade person detectors at multiple viewing angles, we propose adversarial camou-flage textures (AdvCaT) that resemble one kind of the typical textures of daily clothes, camouflage textures. We leverage the Voronoi diagram and Gumbel-softmax trick to parameterize the camouflage textures and optimize the parameters via 3D modeling. Moreover, we propose an efficient augmentation pipeline on 3D meshes combining topologically plausible projection (TopoProj) and Thin Plate Spline (TPS) to narrow the gap between digital and real-world objects. We printed the developed 3D texture pieces on fabric materials and tailored them into T-shirts and trousers. Experiments show high attack success rates of these clothes against multiple detectors.
3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification

Object goal navigation (ObjectNav) in unseen environments is a fundamental task for Embodied AI. Agents in existing works learn ObjectNav policies based on 2D maps, scene graphs, or image sequences. Considering this task happens in 3D space, a 3D-aware agent can advance its ObjectNav capability via learning from fine-grained spatial information. However, leveraging 3D scene representation can be prohibitively unpractical for policy learning in this floor-level task, due to low sample efficiency and expensive computational cost. In this work, we propose a framework for the challenging 3D-aware ObjectNav based on two straightforward sub-policies. The two sub-polices, namely corner-guided exploration policy and category-aware identification policy, simultaneously perform by utilizing online fused 3D points as observation. Through extensive experiments, we show that this framework can dramatically improve the performance in ObjectNav through learning from 3D scene representation. Our framework achieves the best performance among all modular-based methods on the Matterport3D and Gibson datasets, while requiring (up to 30x) less computational cost for training. The code will be released to benefit the community. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Homepage: https://pku-epic.github.io/3D-Aware-ObjectNav/
Adaptive Assignment for Geometry Aware Local Feature Matching

The detector-free feature matching approaches are currently attracting great attention thanks to their excellent performance. However, these methods still struggle at large-scale and viewpoint variations, due to the geometric inconsistency resulting from the application of the mutual nearest neighbour criterion (i.e., one-to-one assignment) in patch-level matching. Accordingly, we in-troduce AdaMatcher, which first accomplishes the feature correlation and co-visible area estimation through an elaborate feature interaction module, then performs adaptive assignment on patch-level matching while es-timating the scales between images, and finally refines the co-visible matches through scale alignment and sub-pixel regression module. Extensive experiments show that AdaMatcher outperforms solid baselines and achieves state-of-the-art results on many downstream tasks. Ad-ditionally, the adaptive assignment and sub-pixel refinement module can be used as a refinement network for other matching methods, such as SuperGlue, to boost their performance further. The code will be publicly available at https://github.com/AbyssGaze/AdaMatcher.
VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud

The task of 3D semantic scene graph (3D SSG) prediction in the point cloud is challenging since (1) the 3D point cloud only captures geometric structures with limited semantics compared to 2D images, and (2) long-tailed relation distribution inherently hinders the learning of unbiased prediction. Since 2D images provide rich semantics and scene graphs are in nature coped with languages, in this study, we propose Visual-Linguistic Semantics Assisted Training (VL-SAT) scheme that can significantly empower 3DSSG prediction models with discrimination about long-tailed and ambiguous semantic relations. The key idea is to train a powerful multi-modal oracle model to assist the 3D model. This oracle learns reliable structural representations based on semantics from vision, language, and 3D geometry, and its benefits can be heterogeneously passed to the 3D model during the training stage. By effectively utilizing visual-linguistic semantics in training, our VL-SAT can significantly boost common 3DSSG prediction models, such as SGFN and SGGpoint, only with 3D inputs in the inference stage, especially when dealing with tail relation triplets. Comprehensive evaluations and ablation studies on the 3DSSG dataset have validated the effectiveness of the proposed scheme. Code is available at https://github.com/wz7in/CVPR2023-VLSAT.
The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Training

Although current deep learning techniques have yielded superior performance on various computer vision tasks, yet they are still vulnerable to adversarial examples. Adversarial training and its variants have been shown to be the most effective approaches to defend against adversarial examples. A particular class of these methods regularize the difference between output probabilities for an adversarial and its corresponding natural example. However, it may have a negative impact if a natural example is misclassified. To circumvent this issue, we propose a novel adversarial training scheme that encourages the model to produce similar output probabilities for an adversarial example and its “inverse adversarial” counterpart. Particularly, the counterpart is generated by maximizing the likelihood in the neighborhood of the natural example. Extensive experiments on various vision datasets and architectures demonstrate that our training method achieves state-of-the-art robustness as well as natural accuracy among robust models. Furthermore, using a universal version of inverse adversarial examples, we improve the performance of single-step adversarial training techniques at a low computational cost.
Crowd3D: Towards Hundreds of People Reconstruction from a Single Image

Image-based multi-person reconstruction in wide-field large scenes is critical for crowd analysis and security alert. However, existing methods cannot deal with large scenes containing hundreds of people, which encounter the challenges of large number of people, large variations in human scale, and complex spatial distribution. In this paper, we propose Crowd3D, the first framework to reconstruct the 3D poses, shapes and locations of hundreds of people with global consistency from a single large-scene image. The core of our approach is to convert the problem of complex crowd localization into pixel localization with the help of our newly defined concept, Human-scene Virtual Interaction Point (HVIP). To reconstruct the crowd with global consistency, we propose a progressive reconstruction network based on HVIP by pre-estimating a scene-level camera and a ground plane. To deal with a large number of persons and various human sizes, we also design an adaptive human-centric cropping scheme. Besides, we contribute a benchmark dataset, LargeCrowd, for crowd reconstruction in a large scene. Experimental results demonstrate the effectiveness of the proposed method. The code and the dataset are available at http://cic.tju.edu.cn/faculty/likun/projects/Crowd3D.
Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers

Vision Transformers (ViT) have shown competitive advantages in terms of performance compared to convolutional neural networks (CNNs), though they often come with high computational costs. To this end, previous methods explore different attention patterns by limiting a fixed number of spatially nearby tokens to accelerate the ViT's multi-head self-attention (MHSA) operations. However, such structured attention patterns limit the token-to-token connections to their spatial relevance, which disregards learned semantic connections from a full attention mask. In this work, we propose an approach to learn instance-dependent attention patterns, by devising a lightweight connectivity predictor module that estimates the connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. As each token only attends to a small number of other tokens, the binarized connectivity masks are often very sparse by nature and therefore provide the opportunity to reduce network FLOPs via sparse computations. Equipped with the learned unstructured attention pattern, sparse attention ViT (Sparsifiner) produces a superior Pareto frontier between FLOPs and top-1 accuracy on ImageNet compared to token sparsity. Our method reduces 48% ~ 69% FLOPs of MHSA while the accuracy drop is within 0.4%. We also show that combining attention and token sparsity reduces ViT FLOPs by over 60%.
Video Test-Time Adaptation for Action Recognition

Although action recognition systems can achieve top performance when evaluated on in-distribution test points, they are vulnerable to unanticipated distribution shifts in test data. However, test-time adaptation of video action recognition models against common distribution shifts has so far not been demonstrated. We propose to address this problem with an approach tailored to spatio-temporal models that is capable of adaptation on a single video sample at a step. It consists in a feature distribution alignment technique that aligns online estimates of test set statistics towards the training statistics. We further enforce prediction consistency over temporally augmented views of the same test video sample. Evaluations on three benchmark action recognition datasets show that our proposed technique is architecture-agnostic and able to significantly boost the performance on both, the state of the art convolutional architecture TANet and the Video Swin Transformer. Our proposed method demonstrates a substantial performance gain over existing test-time adaptation approaches in both evaluations of a single distribution shift and the challenging case of random distribution shifts. Code will be available at https://github.com/wlin-at/ViTTA.
On Calibrating Semantic Segmentation Models: Analyses and An Algorithm

We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to date, confidence calibration research on se-mantic segmentation is still limited. We provide a system-atic study on the calibration of semantic segmentation models and propose a simple yet effective approach. First, we find that model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. Among them, prediction correctness, especially misprediction, is more important to miscalibration due to over-confidence. Next, we propose a simple, unifying, and effective approach, namely selective scaling, by separating correct/incorrect prediction for scaling and more focusing on misprediction logit smoothing. Then, we study popular existing cali-bration methods and compare them with selective scaling on semantic segmentation calibration. We conduct exten-sive experiments with a variety of benchmarks on both in-domain and domain-shift calibration and show that selective scaling consistently outperforms other methods.
Query - Dependent Video Representation for Moment Retrieval and Highlight Detection

Recently, video moment retrieval and highlight detection (MR/HD) are being spotlighted as the demand for video understanding is drastically increased. The key objective of MR/HD is to localize the moment and estimate clip-wise accordance level, i.e., saliency score, to the given text query. Although the recent transformer-based models brought some advances, we found that these methods do not fully exploit the information of a given query. For example, the relevance between text query and video contents is sometimes neglected when predicting the moment and its saliency. To tackle this issue, we introduce Query-Dependent DETR (QD-DETR), a detection transformer tailored for MR/HD. As we observe the insignificant role of a given query in transformer architectures, our encoding module starts with cross-attention layers to explicitly inject the context of text query into video representation. Then, to enhance the model's capability of exploiting the query information, we manipulate the video-query pairs to produce irrelevant pairs. Such negative (irrelevant) video-query pairs are trained to yield low saliency scores, which in turn, encourages the model to estimate precise accordance between query-video pairs. Lastly, we present an input-adaptive saliency predictor which adaptively defines the criterion of saliency scores for the given video-query pairs. Our extensive studies verify the importance of building the query-dependent representation for MR/HD. Specifically, QD-DETR outperforms state-of-the-art methods on QVHighlights, TVSum, and Charades-STA datasets. Codes are available at github.com/wjun0830IQD-DETR.
NIFF: Alleviating Forgetting in Generalized Few-Shot Object Detection via Neural Instance Feature Forging

Privacy and memory are two recurring themes in a broad conversation about the societal impact of AI. These con-cerns arise from the need for huge amounts of data to train deep neural networks. A promise of Generalized Few-shot Object Detection (G-FSOD), a learning paradigm in AI, is to alleviate the need for collecting abundant training samples of novel classes we wish to detect by leveraging prior knowledge from old classes (i.e., base classes). G-FSOD strives to learn these novel classes while alleviating catas-trophic forgetting of the base classes. However, existing approaches assume that the base images are accessible, an assumption that does not hold when sharing and storing data is problematic. In this work, we propose the first data-free knowledge distillation (DFKD) approach for G-FSOD that leverages the statistics of the region of interest (RoI) features from the base model to forge instance-level features without accessing the base images. Our contribution is three-fold: (1) we design a standalone lightweight generator with (2) class-wise heads (3) to generate and replay diverse instance-level base features to the RoI head while finetuning on the novel data. This stands in contrast to standard DFKD approaches in image classification, which invert the entire network to generate base images. Moreover, we make careful design choices in the novel finetuning pipeline to regularize the model. We show that our approach can dramatically reduce the base memory requirements, all while setting a new standard for G-FSOD on the challenging MS-COCO and PASCAL-VOC benchmarks.
Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation

Deep neural networks have made huge progress in the last few decades. However, as the real-world data often exhibits a long-tailed distribution, vanilla deep models tend to be heavily biased toward the majority classes. To address this problem, state-of-the-art methods usually adopt a mixture of experts (MoE) to focus on different parts of the long-tailed distribution. Experts in these methods are with the same model depth, which neglects the fact that different classes may have different preferences to be fit by models with different depths. To this end, we propose a novel MoE-based method called Self-Heterogeneous Integration with Knowledge Excavation (SHIKE). We first propose Depth-wise Knowledge Fusion (DKF) to fuse features between different shallow parts and the deep part in one network for each expert, which makes experts more diverse in terms of representation. Based on DKF, we further propose Dynamic Knowledge Transfer (DKT) to reduce the influence of the hardest negative class that has a non-negligible impact on the tail classes in our MoEframework. As a result, the classification accuracy of long-tailed data can be significantly improved, especially for the tail classes. SHIKE achieves the state-of-the-art performance of 56.3%, 60.3%, 75.4% and 41.9% on CIFAR100-LT (IF100), ImageNet-LT, iNaturalist 2018, and Places-LT, respectively. The source code is available at https://github.com/jinyan-06/SHIKE.