Miracle-dz/UniFAR-FTUs huggingface dataset:
Voxel-MAE: Masked Autoencoders for Pre-training Large-scale Point Clouds

Mask-based pre-training has achieved great success for self-supervised learning in images and languages without manually annotated supervision. However, it has not yet been studied for large-scale point clouds with redundant spatial information. In this research, we propose a mask voxel autoencoder network for pre-training large-scale point clouds, dubbed Voxel-MAE. Our key idea is to transform the point clouds into voxel representations and classify whether the voxel contains point clouds. This simple but effective strategy makes the network voxel-aware of the object shape, thus improving the performance of down-stream tasks, such as 3D object detection. Our Voxel-MAE, with even a 90% masking ratio, can still learn representative features for the high spatial redundancy of large-scale point clouds. We also validate the effectiveness of Voxel-MAE on unsupervised domain adaptative tasks, which proves the generalization ability of Voxel-MAE. Our Voxel-MAE proves that it is feasible to pre-train large-scale point clouds without data annotations to enhance the perception ability of the autonomous vehicle. Extensive experiments show great effectiveness of our pre-training method with 3D object detectors (SECOND, CenterPoint, and PV-RCNN) on three popular datasets (KITTI, Waymo, and nuScenes). Codes are publicly available at https: //github.com/chaytonmin/Voxel-MAE .
Identifiability of deep generative models under mixture priors without auxiliary information

We prove identiﬁability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, or conditioning in the latent space. The models we consider are tightly connected with autoen-coder architectures used in practice that leverage mixture priors in the latent space and ReLU/leaky-ReLU activations in the encoder. Our main result is an identiﬁability hierarchy that signiﬁcantly generalizes previous work and exposes how different assumptions lead to different “strengths” of iden-tiﬁability. For example, our weakest result establishes (unsupervised) identiﬁability up to an afﬁne transformation, which already improves existing work. It’s well known that these models have universal approximation capabilities and moreover, they have been extensively used in practice to learn representations of data.
The Role of Machine Learning in Cybersecurity

Machine Learning (ML) represents a pivotal technology for current and future information systems, and many domains already leverage the capabilities of ML. However, deployment of ML in cybersecurity is still at an early stage, revealing a significant discrepancy between research and practice. Such a discrepancy has its root cause in the current state of the art, which does not allow us to identify the role of ML in cybersecurity. The full potential of ML will never be unleashed unless its pros and cons are understood by a broad audience. This article is the first attempt to provide a holistic understanding of the role of ML in the entire cybersecurity domain—to any potential reader with an interest in this topic. We highlight the advantages of ML with respect to human-driven detection methods, as well as the additional tasks that can be addressed by ML in cybersecurity. Moreover, we elucidate various intrinsic problems affecting real ML deployments in cybersecurity. Finally, we present how various stakeholders can contribute to future developments of ML in cybersecurity, which is essential for further progress in this field. Our contributions are complemented with two real case studies describing industrial applications of ML as defense against cyber-threats.
HealNet - Self-Supervised Acute Wound Heal-Stage Classification

Identifying, tracking, and predicting wound heal-stage progression is a fundamental task towards proper diagnosis, eﬀective treatment, facilitating healing, and reducing pain. Traditionally, a medical expert might observe a wound to determine the current healing state and recommend treatment. However, sourcing experts who can produce such a diagnosis solely from visual indicators can be diﬃcult, time-consuming and expensive. In addition, lesions may take several weeks to undergo the healing process, demanding resources to monitor and diagnose con-tinually. Automating this task can be challenging; datasets that follow wound progression from onset to maturation are small, rare, and often collected without computer vision in mind. To tackle these challenges, we introduce a self-supervised learning scheme composed of (a) learning embeddings of wound’s temporal dynamics, (b) clustering for automatic stage discovery, and (c) ﬁne-tuned classiﬁcation. The proposed self-supervised and ﬂexible learning framework is biologically inspired and trained on a small dataset with zero human labeling. The HealNet framework achieved high pre-text and downstream classiﬁcation accuracy; when evaluated on held-out test data, HealNet achieved 97.7% pre-text accuracy and 90.62% heal-stage classiﬁcation accuracy.
A Machine Learning Data Fusion Model for Soil Moisture Retrieval

We develop a deep learning based convolutional-regression model that estimates the volumetric soil moisture content in the top ~5 cm of soil. Input predictors include Sentinel-1 (active radar), Sentinel-2 (optical imagery), and SMAP (passive radar) as well as geophysical variables from SoilGrids and modelled soil moisture fields from GLDAS. The model was trained and evaluated on data from ~1300 in-situ sensors globally over the period 2015 - 2021 and obtained an average per-sensor correlation of 0.727 and ubRMSE of 0.054, and can be used to produce a soil moisture map at a nominal 320m resolution. These results are benchmarked against 13 other soil moisture works at different locations, and an ablation study was used to identify important predictors.
Dirichlet Eigenvalues and Exit Time Moments for Symmetric Markov Processes

We give some relationships between the first Dirichlet eigenvalues and the exit time moments for the general symmetric Markov processes. As applications, we present some examples, including symmetric diffusions and $\alpha$-stable processes, and provide the estimates of their first Dirichlet eigenvalues and the exit time moments.
New Bounds on the Size of Permutation Codes With Minimum Kendall $\tau$-distance of Three

We study P ( n, 3), the size of the largest subset of the set of all permutations S n with minimum Kendall τ -distance 3. Using a combination of group theory and integer programming, we reduced the upper bound of P ( p, 3) from ( p − 1)! − 1 to ( p − 1)! − ⌈ p 3 ⌉ + 2 ≤ ( p − 1)! − 2 for all primes p ≥ 11. In special cases where n is equal to 6 , 7 , 11 , 13 , 14 , 15 and 17 we reduced the upper bound of P ( n, 3) by 3 , 3 , 9 , 11 , 1 , 1 and 4, respectively.
Intra-Instance VICReg: Bag of Self-Supervised Image Patch Embedding

Recently, self-supervised learning (SSL) has achieved tremendous empirical advancements in learning image representation. However, our understanding and knowledge of the representation are still limited. This work shows that the success of the SOTA siamese-network-based SSL approaches is primarily based on learning a representation of image patches. Particularly, we show that when we learn a representation only for ﬁxed-scale image patches and aggregate different patch representations linearly for an image (instance), it can achieve on par or even better results than the baseline methods on several benchmarks. Further, we show that the patch representation aggregation can also improve various SOTA baseline methods by a large margin. We also establish a formal connection between the SSL objective and the image patches co-occurrence statistics modeling, which supplements the prevailing invariance perspective. By visualizing the nearest neighbors of different image patches in the embedding space and projection space, we show that while the projection has more invariance, the embedding space tends to preserve more equivariance and locality. Finally, we propose a hypothesis for the future direction based on the discovery of this work.
A Bounded-Confidence Model of Opinion Dynamics with Heterogeneous Node-Activity Levels

Agent-based models of opinion dynamics allow one to examine the spread of opinions between entities and to study phenomena such as consensus, polarization, and fragmentation. By studying a model of opinion dynamics on a social network, one can explore the effects of network structure on these phenomena. In social networks, some individuals share their ideas and opinions more frequently than others. These disparities can arise from heterogeneous sociabilities, heterogeneous activity levels, different prevalences to share opinions when engaging in a social-media platform, or something else. To examine the impact of such heterogeneities on opinion dynamics, we generalize the Deffuant--Weisbuch (DW) bounded-confidence model (BCM) of opinion dynamics by incorporating node weights. The node weights allow us to model agents with different probabilities of interacting. Using numerical simulations, we systematically investigate (using a variety of network structures and node-weight distributions) the effects of node weights, which we assign uniformly at random to the nodes. We demonstrate that introducing heterogeneous node weights results in longer convergence times and more opinion fragmentation than in a baseline DW model. The node weights in our BCM allow one to consider a variety of sociological scenarios in which agents have heterogeneous probabilities of interacting with other agents.
Insights into Pre-training via Simpler Synthetic Tasks

Pre-training produces representations that are effective for a wide range of downstream tasks, but it is still unclear what properties of pre-training are necessary for effective gains. Notably, recent work shows that even pre-training on synthetic tasks can achieve significant gains in downstream tasks. In this work, we perform three experiments that iteratively simplify pre-training and show that the simplifications still retain much of its gains. First, building on prior work, we perform a systematic evaluation of three existing synthetic pre-training methods on six downstream tasks. We find the best synthetic pre-training method, LIME, attains an average of $67\%$ of the benefits of natural pre-training. Second, to our surprise, we find that pre-training on a simple and generic synthetic task defined by the Set function achieves $65\%$ of the benefits, almost matching LIME. Third, we find that $39\%$ of the benefits can be attained by using merely the parameter statistics of synthetic pre-training. We release the source code at https://github.com/felixzli/synthetic_pretraining.
A pseudospectral method for direct numerical simulation of low-Mach, variable-density, turbulent flows

A novel algorithm for the direct numerical simulation of the variable-density, low-Mach Navier-Stokes equations extending the method of Kim, Moin, and Moser [16] for incompressible ﬂow is presented here. A Fourier representation is employed in the two homogeneous spatial directions and a number of discretizations can be used in the inhomogeneous direction. The momentum is decomposed into divergence- and curl-free portions which allows the momentum equations to be rewritten, removing the need to solve for the pressure. The temporal discretization is based on an explicit, segregated Runge-Kutta method and the scalar equations are reformulated to directly address the redundancy of the equation of state and the mass conservation equation. An eﬃcient, matrix-free, iterative solution of the resulting equations allows for second-order accuracy in time and numerical stability for large density ratios, which is demonstrated for ratios up to ∼ 25 . 7.
Resonance crossing of a charged body in a magnetized Kerr background: An analog of extreme mass ratio inspiral

We investigate resonance crossings of a charged body moving around a Kerr black hole immersed in an external homogeneous magnetic field. This system can serve as an electromagnetic analogue of a weakly non-integrable extreme mass ratio inspiral (EMRI). In particular, the presence of the magnetic field renders the conservative part of the system non-integrable in the Liouville sense, while the electromagnetic self-force causes the charged body to inspiral. By studying the system without the self-force, we show the existence of an approximate Carter-like constant and discuss how resonances grow as a function of the perturbation parameter. Then, we apply the electromagnetic self-force to investigate crossings of these resonances during an inspiral. Averaging the energy and angular momentum losses during crossings allows us to employ an adiabatic approximation for them. We demonstrate that such adiabatic approximation provides results qualitatively equivalent to the instantaneous self-force evolution, which indicates that the adiabatic approximation may describe the resonance crossing with sufficiently accuracy in EMRIs.
Coherent noise enables probabilistic sequence replay in spiking neuronal networks

Animals rely on different decision strategies when faced with ambiguous or uncertain cues. Depending on the context, decisions may be biased towards events that were most frequently experienced in the past, or be more explorative. A particular type of decision making central to cognition is sequential memory recall in response to ambiguous cues. A previously developed spiking neuronal network implementation of sequence prediction and recall learns complex, high-order sequences in an unsupervised manner by local, biologically inspired plasticity rules. In response to an ambiguous cue, the model deterministically recalls the sequence shown most frequently during training. Here, we present an extension of the model enabling a range of different decision strategies. In this model, explorative behavior is generated by supplying neurons with noise. As the model relies on population encoding, uncorrelated noise averages out, and the recall dynamics remain effectively deterministic. In the presence of locally correlated noise, the averaging effect is avoided without impairing the model performance, and without the need for large noise amplitudes. We investigate two forms of correlated noise occurring in nature: shared synaptic background inputs, and random locking of the stimulus to spatiotemporal oscillations in the network activity. Depending on the noise characteristics, the network adopts various recall strategies. This study thereby provides potential mechanisms explaining how the statistics of learned sequences affect decision making, and how decision strategies can be adjusted after learning.
Parametrically-driven temporal cavity solitons in a bichromatically-driven pure Kerr resonator

Temporal cavity solitons (CSs) are pulses of light that can persist endlessly in dispersive, nonlinear optical resonators. They have been extensively studied in the context of resonators with purely cubic (Kerr-type) nonlinearity that are externally-driven with a monochromatic continuous wave laser – in such systems, the solitons manifest themselves as unique attractors whose carrier frequency coincides with that of the external driving ﬁeld. Recent experiments have, however, shown that a qualitatively different type of temporal CS can arise via parametric down-conversion in resonators with simultaneous quadratic and cubic nonlinearity. In contrast to conventional CSs in pure-Kerr resonators, these parametrically-driven cavity solitons come in two different ﬂavours with opposite phases, and they are spectrally centred at half of the frequency of the driving ﬁeld. Here, we theoretically and numerically show that, under conditions of bichromatic driving, such parametrically-driven CSs can also arise in dispersive resonators with pure Kerr nonlinearity. In this case, the solitons arise through parametric four-wave mixing, come with two distinct phases, and have a carrier frequency in between the two external driving ﬁelds. We show that, when all waves are resonant, the solitons can experience long-range interactions due to their back-action on the intracavity ﬁelds at the pump frequencies, and we discuss the parameter requirements for the solitons’ existence. Besides underlining the possibility of exciting a new type of cavity soliton in dispersive Kerr cavities, our work advances the theoretical modeling of resonators that are coherently-driven with polychromatic ﬁelds. In this article, we theoretically and numerically show that a dispersive resonator with pure Kerr nonlinearity can support parametrically-driven CSs in the presence of bichromatic driving. Speciﬁcally, we show that a signal ﬁeld with carrier frequency in between two spectrally-separated driving ﬁelds obeys the damped, parametrically-driven nonlinear
TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks

Spiking Neural Networks (SNNs) are attracting widespread interest due to their biological plausibility, energy efficiency, and powerful spatio-temporal information representation ability. Given the critical role of attention mechanisms in enhancing neural network performance, the integration of SNNs and attention mechanisms exhibits potential to deliver energy-efficient and high-performance computing paradigms. We present a novel Temporal-Channel Joint Attention mechanism for SNNs, referred to as TCJA-SNN. The proposed TCJA-SNN framework can effectively assess the significance of spike sequence from both spatial and temporal dimensions. More specifically, our essential technical contribution lies on: 1) We employ the squeeze operation to compress the spike stream into an average matrix. Then, we leverage two local attention mechanisms based on efficient 1D convolutions to facilitate comprehensive feature extraction at the temporal and channel levels independently. 2) We introduce the Cross Convolutional Fusion (CCF) layer as a novel approach to model the inter-dependencies between the temporal and channel scopes. This layer breaks the independence of these two dimensions and enables the interaction between features. Experimental results demonstrate that the proposed TCJA-SNN outperforms SOTA by up to 15.7% accuracy on standard static and neuromorphic datasets, including Fashion-MNIST, CIFAR10-DVS, N-Caltech 101, and DVS128 Gesture. Furthermore, we apply the TCJA-SNN framework to image generation tasks by leveraging a variation autoencoder. To the best of our knowledge, this study is the first instance where the SNN-attention mechanism has been employed for image classification and generation tasks. Notably, our approach has achieved SOTA performance in both domains, establishing a significant advancement in the field. Codes are available at https://github.com/ridgerchu/TCJA.
Three-way optimization of privacy and utility of location data

—With the recent bloom of data and the drive towards an information-based society, the urge of and the advancements in data analytics is surging like never before. And with this, the risks of privacy violation of various kinds are also increasing manifold. Most of the methods to mitigate the privacy risks for location data resort to adding some noise to the location, like the planar Laplace used to achieve geo-indistinguishability. However, the noise should be calibrated carefully, taking into account the implications for utility, because it is far from ideal for the service providers to completely lose the utility of the collected data succumbing to the privacy requirements of the users. Similarly, the quality of service for the users should be optimized with their personalized needs of privacy protection used to shield their sensitive information. In this paper, we address this age-old battle between privacy and utility from three ends: privacy of the users’ data, the quality of service (QoS) received by them in exchange for sharing their privatized data, and the statistical utility of the privatized data for the service providers who wish to perform various kinds of analysis and research on the data collected from the users. We propose a method to produce a geo-indistinguishable location-privacy mechanism that advances to optimize simultaneously between the level of privacy attained, the QoS, and the statistical utility achieved by the obfuscated data. We illustrate the soundness of this three-way privacy-utility optimization mechanism both analytically and with experiments. Apart from the novelty of the proposed method, this work is aimed to engender an analytical perspective to bridge between geo-indistinguishable location-privacy, QoS, and statistical utilities used in standard data analytics, from an information theoretical, probabilistic, and statistical perspective.
Stop Overcomplicating Selective Classification: Use Max-Logit

We tackle the problem of Selective Classiﬁcation where the goal is to achieve the best performance on the desired coverages of the dataset. Recent state-of-the-art selective methods come with architectural changes either via introducing a separate selection head or an extra abstention logit. In this paper, we present surprising results for Selective Classiﬁcation by conﬁrming that the superior performance of state-of-the-art methods is owed to training a more generalizable classiﬁer; however, their selection mechanism is suboptimal. We argue that the selection mechanism should be rooted in the objective function instead of a separately calculated score. Accordingly, in this paper, we motivate an alternative selection strategy that is based on the cross entropy loss for the classiﬁcation settings, namely, max of the logits. Our proposed selection strategy achieves better results by a signiﬁcant margin, consistently, across all coverages and all datasets, without any additional computation. Finally, inspired by our superior selection mechanism, we propose to further regularize the objective function with entropy-minimization. Our proposed max-logit selection with the modiﬁed loss function achieves new state-of-the-art results for Selective Classiﬁcation.
The State of Sparse Training in Deep Reinforcement Learning

The use of sparse neural networks has seen rapid growth in recent years, particularly in computer vision. Their appeal stems largely from the reduced number of parameters required to train and store, as well as in an increase in learning efficiency. Somewhat surprisingly, there have been very few efforts exploring their use in Deep Reinforcement Learning (DRL). In this work we perform a systematic investigation into applying a number of existing sparse training techniques on a variety of DRL agents and environments. Our results corroborate the findings from sparse training in the computer vision domain - sparse networks perform better than dense networks for the same parameter count - in the DRL domain. We provide detailed analyses on how the various components in DRL are affected by the use of sparse networks and conclude by suggesting promising avenues for improving the effectiveness of sparse training methods, as well as for advancing their use in DRL.