Miracle-dz/UniFAR-FTUs huggingface dataset:
Self-supervised Multi-task Learning Framework for Safety and Health-Oriented Connected Driving Environment Perception using Onboard Camera

Cutting-edge connected vehicle (CV) technologies have drawn much attention in recent years. The real-time traffic data captured by a CV can be shared with other CVs and data centers so as to open new possibilities for solving diverse transportation problems. However, imagery captured by onboard cameras in a connected environment, are not sufficiently investigated, especially for safety and health-oriented visual perception. In this paper, a bidirectional process of image synthesis and decomposition (BPISD) approach is proposed, and thus a novel self-supervised multi-task learning framework, to simultaneously estimate depth map, atmospheric visibility, airlight, and PM2.5 mass concentration, in which depth map and visibility are considered highly associated with traffic safety, while airlight and PM2.5 mass concentration are directly correlated with human health. Both the training and testing phases of the proposed system solely require a single image as input. Due to the innovative training pipeline, the depth estimation network can manage various levels of visibility conditions and overcome inherent problems in current image-synthesis-based depth estimation, thereby generating high-quality depth maps even in low-visibility situations and further benefiting accurate estimations of visibility, airlight, and PM2.5 mass concentration. Extensive experiments on the synthesized data from the KITTI and real-world data collected in Beijing demonstrate that the proposed method can (1) achieve performance competitive in depth estimation as compared with state-of-the-art methods when taking clear images as input; (2) predict vivid depth map for images contaminated by various levels of haze; and (3) accurately estimate visibility, airlight, and PM2.5 mass concentrations. Beneficial applications can be developed based on the presented work to improve traffic safety, air quality, and public health.
On Frequency-Wise Normalizations for Better Recording Device Generalization in Audio Spectrogram Transformers

Varying conditions between the data seen at training and at application time remain a major challenge for machine learning. We study this problem in the context of Acoustic Scene Classification (ASC) with mismatching recording devices. Previous works successfully employed frequency-wise normalization of inputs and hidden layer activations in convolutional neural networks to reduce the recording device discrepancy. The main objective of this work was to adopt frequency-wise normalization for Audio Spectrogram Transformers (ASTs), which have recently become the dominant model architecture in ASC. To this end, we first investigate how recording device characteristics are encoded in the hidden layer activations of ASTs. We find that recording device information is initially encoded in the frequency dimension; however, after the first self-attention block, it is largely transformed into the token dimension. Based on this observation, we conjecture that suppressing recording device characteristics in the input spectrogram is the most effective. We propose a frequency-centering operation for spectrograms that improves the ASC performance on unseen recording devices on average by up to 18.2 percentage points.
Evidence for Ubiquitous Carbon Grain Destruction in Hot Protostellar Envelopes

Earth is deficient in carbon and nitrogen by up to ∼4 orders of magnitude compared with the Sun. Destruction of (carbon- and nitrogen-rich) refractory organics in the high-temperature planet-forming regions could explain this deficiency. Assuming a refractory cometary composition for these grains, their destruction enhances nitrogen-containing, oxygen-poor molecules in the hot gas (≳300 K) after the initial formation and sublimation of these molecules from oxygen-rich ices in the warm gas (∼150 K). Using observations of 37 high-mass protostars with the Atacama Large Millimeter/submillimeter Array, we find that oxygen-containing molecules (CH3OH and HNCO) systematically show no enhancement in their hot component. In contrast, nitrogen-containing, oxygen-poor molecules (CH3CN and C2H3CN) systematically show an enhancement of a factor ∼5 in their hot component, pointing to additional production of these molecules in the hot gas. Assuming only thermal excitation conditions, we interpret these results as a signature of destruction of refractory organics, consistent with the cometary composition. This destruction implies a higher C/O and N/O in the hot gas than the warm gas, while the exact values of these ratios depend on the fraction of grains that are effectively destroyed. This fraction can be found by future chemical models that constrain C/O and N/O from the abundances of minor carbon, nitrogen, and oxygen carriers presented here.
Opening the Black Box: Analyzing Attention Weights and Hidden States in Pre-trained Language Models for Non-language Tasks

Investigating deep learning language models has always been a significant research area due to the ``black box"nature of most advanced models. With the recent advancements in pre-trained language models based on transformers and their increasing integration into daily life, addressing this issue has become more pressing. In order to achieve an explainable AI model, it is essential to comprehend the procedural steps involved and compare them with human thought processes. Thus, in this paper, we use simple, well-understood non-language tasks to explore these models' inner workings. Specifically, we apply a pre-trained language model to constrained arithmetic problems with hierarchical structure, to analyze their attention weight scores and hidden states. The investigation reveals promising results, with the model addressing hierarchical problems in a moderately structured manner, similar to human problem-solving strategies. Additionally, by inspecting the attention weights layer by layer, we uncover an unconventional finding that layer 10, rather than the model's final layer, is the optimal layer to unfreeze for the least parameter-intensive approach to fine-tune the model. We support these findings with entropy analysis and token embeddings similarity analysis. The attention analysis allows us to hypothesize that the model can generalize to longer sequences in ListOps dataset, a conclusion later confirmed through testing on sequences longer than those in the training set. Lastly, by utilizing a straightforward task in which the model predicts the winner of a Tic Tac Toe game, we identify limitations in attention analysis, particularly its inability to capture 2D patterns.
ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining

Scene text removal (STR) aims at replacing text strokes in natural scenes with visually coherent backgrounds. Recent STR approaches rely on iterative refinements or explicit text masks, resulting in high complexity and sensitivity to the accuracy of text localization. Moreover, most existing STR methods adopt convolutional architectures while the potential of vision Transformers (ViTs) remains largely unexplored. In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed ViTEraser. Following a concise encoder-decoder framework, ViTEraser can easily incorporate various ViTs to enhance long-range modeling. Specifically, the encoder hierarchically maps the input image into the hidden space through ViT blocks and patch embedding layers, while the decoder gradually upsamples the hidden features to the text-erased image with ViT blocks and patch splitting layers. As ViTEraser implicitly integrates text localization and inpainting, we propose a novel end-to-end pretraining method, termed SegMIM, which focuses the encoder and decoder on the text box segmentation and masked image modeling tasks, respectively. Experimental results demonstrate that ViTEraser with SegMIM achieves state-of-the-art performance on STR by a substantial margin and exhibits strong generalization ability when extended to other tasks, e.g., tampered scene text detection. Furthermore, we comprehensively explore the architecture, pretraining, and scalability of the ViT-based encoder-decoder for STR, which provides deep insights into the application of ViT to the STR field. Code is available at https://github.com/shannanyinxiang/ViTEraser.
Lebesgue Points Of Functions In The Complex Sobolev Space

Let $\varphi$ be a function in the complex Sobolev space $W^*(U)$, where $U$ is an open subset in $\mathbb{C}^k$. We show that the complement of the set of Lebesgue points of $\varphi$ is pluripolar. The key ingredient in our approach is to show that $|\varphi|^\alpha $ for $\alpha \in [1,2)$ is locally bounded from above by a plurisubharmonic function.
Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift

Bayesian deep learning (BDL) is a promising approach to achieve well-calibrated predictions on distribution-shifted data. Nevertheless, there exists no large-scale survey that evaluates recent SOTA methods on diverse, realistic, and challenging benchmark tasks in a systematic manner. To provide a clear picture of the current state of BDL research, we evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks, with a focus on generalization capability and calibration under distribution shift. We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures. In particular, we investigate a signed version of the expected calibration error that reveals whether the methods are over- or under-confident, providing further insight into the behavior of the methods. Further, we provide the first systematic evaluation of BDL for fine-tuning large pre-trained models, where training from scratch is prohibitively expensive. Finally, given the recent success of Deep Ensembles, we extend popular single-mode posterior approximations to multiple modes by the use of ensembles. While we find that ensembling single-mode approximations generally improves the generalization capability and calibration of the models by a significant margin, we also identify a failure mode of ensembles when finetuning large transformer-based language models. In this setting, variational inference based approaches such as last-layer Bayes By Backprop outperform other methods in terms of accuracy by a large margin, while modern approximate inference algorithms such as SWAG achieve the best calibration.
Mixture Encoder for Joint Speech Separation and Recognition

Multi-speaker automatic speech recognition (ASR) is crucial for many real-world applications, but it requires dedicated modeling techniques. Existing approaches can be divided into modular and end-to-end methods. Modular approaches separate speakers and recognize each of them with a single-speaker ASR system. End-to-end models process overlapped speech directly in a single, powerful neural network. This work proposes a middle-ground approach that leverages explicit speech separation similarly to the modular approach but also incorporates mixture speech information directly into the ASR module in order to mitigate the propagation of errors made by the speech separator. We also explore a way to exchange cross-speaker context information through a layer that combines information of the individual speakers. Our system is optimized through separate and joint training stages and achieves a relative improvement of 7% in word error rate over a purely modular setup on the SMS-WSJ task.
The OH Megamaser emission in Arp 220: the rest of the story

The OH Megamaser emission in the merging galaxy Arp 220 has been re-observed with the Multi-Element Radio Linked Interferometer Network (MERLIN) and the European VLBI Network (EVN). Imaging results of the OH line emission at the two nuclei are found to be consistent with earlier observations and confirm additional extended emission structures surrounding the nuclei. Detailed information about the distributed emission components around the two nuclei has been obtained using a concatenated MERLIN and EVN database with intermediate (40 mas) spatial resolution. Continuum imaging shows a relatively compact West nucleus and a more extended East nucleus in addition to an extended continuum ridge stretching below and beyond the two nuclei. Spectral line imaging show extended emission regions at both nuclei together with compact components and additional weaker components north and south of the West nucleus. Spectral line analysis indicates that the dominant OH line emission originates in foreground molecular material that is part of a large-scale molecular structure that engulfs the whole nuclear region. Compact OH components are representative of star formation regions within the two nearly edge-on nuclei and define the systemic velocities of East and West as 5425 and 5360 km s−1. The foreground material at East and West has a 100 km s−1 lower velocity at 5314 and 5254 km s−1. These emission results confirm a maser amplification scenario where the background continuum and the line emission of the star formation regions are amplified by foreground masering material that is excited by the far-infrared radiation field originating in the two nuclear regions.
Towards Mitigating Spurious Correlations in the Wild: A Benchmark & a more Realistic Dataset

Deep neural networks often exploit non-predictive features that are spuriously correlated with class labels, leading to poor performance on groups of examples without such features. Despite the growing body of recent works on remedying spurious correlations, the lack of a standardized benchmark hinders reproducible evaluation and comparison of the proposed solutions. To address this, we present SpuCo, a python package with modular implementations of state-of-the-art solutions enabling easy and reproducible evaluation of current methods. Using SpuCo, we demonstrate the limitations of existing datasets and evaluation schemes in validating the learning of predictive features over spurious ones. To overcome these limitations, we propose two new vision datasets: (1) SpuCoMNIST, a synthetic dataset that enables simulating the effect of real world data properties e.g. difficulty of learning spurious feature, as well as noise in the labels and features; (2) SpuCoAnimals, a large-scale dataset curated from ImageNet that captures spurious correlations in the wild much more closely than existing datasets. These contributions highlight the shortcomings of current methods and provide a direction for future research in tackling spurious correlations. SpuCo, containing the benchmark and datasets, can be found at https://github.com/BigML-CS-UCLA/SpuCo, with detailed documentation available at https://spuco.readthedocs. io/en/latest/.
The Jenkins-Serrin problem in 3-manifolds with a Killing vector field

We consider a Riemannian submersion from a 3-manifold $\mathbb{E}$ to a surface $M$, both connected and orientable, whose fibers are the integral curves of a Killing vector field without zeros, not necessarily unitary. We solve the Jenkins-Serrin problem for the minimal surface equation in $\mathbb{E}$ over a relatively compact open domain $\Omega\subset M$ with prescribed finite or infinite values on some arcs of the boundary under the only assumption that the same value $+\infty$ or $-\infty$ cannot be prescribed on two adjacent components of $\partial\Omega$ forming a convex angle. The domain $\Omega$ can have reentrant corners as well as closed curves in its boundary. We show that the solution exists if and only if some generalized Jenkins-Serrin conditions (in terms of a conformal metric in $M$) are fulfilled. We develop further the theory of divergence lines to study the convergence of a sequence of minimal graphs. We also provide maximum principles that guarantee the uniqueness of the solution. Finally, we obtain new examples of minimal surfaces in $\mathbb{R}^3$ and in other homogeneous $3$-manifolds.
Continual Learners are Incremental Model Generalizers

Motivated by the efficiency and rapid convergence of pre-trained models for solving downstream tasks, this paper extensively studies the impact of Continual Learning (CL) models as pre-trainers. In both supervised and unsupervised CL, we find that the transfer quality of the representation often increases gradually without noticeable degradation in fine-tuning performance. This is because CL models can learn improved task-general features when easily forgetting task-specific knowledge. Based on this observation, we suggest a new unsupervised CL framework with masked modeling, which aims to capture fluent task-generic representation during training. Furthermore, we propose a new fine-tuning scheme, GLobal Attention Discretization (GLAD), that preserves rich task-generic representation during solving downstream tasks. The model fine-tuned with GLAD achieves competitive performance and can also be used as a good pre-trained model itself. We believe this paper breaks the barriers between pre-training and fine-tuning steps and leads to a sustainable learning framework in which the continual learner incrementally improves model generalization, yielding better transfer to unseen tasks.
Galactic coronae in Milky Way-like galaxies: the role of stellar feedback in gas accretion

Star-forming galaxies like the Milky Way are surrounded by a hot gaseous halo at the virial temperature – the so-called galactic corona – that plays a fundamental role in their evolution. The interaction between the disc and the corona has been shown to have a direct impact on accretion of coronal gas onto the disc with major implications for galaxy evolution. In this work, we study the gas circulation between the disc and the corona of star-forming galaxies like the Milky Way. We use high-resolution hydrodynamical N-body simulations of a Milky Way-like galaxy with the inclusion of an observationally-motivated galactic corona. In doing so, we use SMUGGLE, an explicit interstellar medium (ISM) and stellar feedback model coupled with the moving-mesh code Arepo. We find that the reservoir of gas in the galactic corona is sustaining star formation: the gas accreted from the corona is the primary fuel for the formation of new stars, helping in maintaining a nearly constant level of cold gas mass in the galactic disc. Stellar feedback generates a gas circulation between the disc and the corona (the so-called galactic fountain) by ejecting different gas phases that are eventually re-accreted onto the disc. The accretion of coronal gas is promoted by its mixing with the galactic fountains at the disc-corona interface, causing the formation of intermediate temperature gas that enhance the cooling of the hot corona. We find that this process acts as a positive feedback mechanism, increasing the accretion rate of coronal gas onto the galaxy.
Efficient ResNets: Residual Network Design

ResNets (or Residual Networks) are one of the most commonly used models for image classification tasks. In this project, we design and train a modified ResNet model for CIFAR-10 image classification. In particular, we aimed at maximizing the test accuracy on the CIFAR-10 benchmark while keeping the size of our ResNet model under the specified fixed budget of 5 million trainable parameters. Model size, typically measured as the number of trainable parameters, is important when models need to be stored on devices with limited storage capacity (e.g. IoT/edge devices). In this article, we present our residual network design which has less than 5 million parameters. We show that our ResNet achieves a test accuracy of 96.04% on CIFAR-10 which is much higher than ResNet18 (which has greater than 11 million trainable parameters) when equipped with a number of training strategies and suitable ResNet hyperparameters. Models and code are available at https://github.com/Nikunj-Gupta/Efficient_ResNets.
Self-Supervised Graph Representation Learning for Neuronal Morphologies

Unsupervised graph representation learning has recently gained interest in several application domains such as neuroscience, where modeling the diverse morphology of cell types in the brain is one of the key challenges. It is currently unknown how many excitatory cortical cell types exist and what their defining morphological features are. Here we present GraphDINO, a purely data-driven approach to learn low-dimensional representations of 3D neuronal morphologies from unlabeled large-scale datasets. GraphDINO is a novel transformer-based representation learning method for spatially-embedded graphs. To enable self-supervised learning on transformers, we (1) developed data augmentation strategies for spatially-embedded graphs, (2) adapted the positional encoding and (3) introduced a novel attention mechanism, AC-Attention, which combines attention-based global interaction between nodes and classic graph convolutional processing. We show, in two different species and across multiple brain areas, that this method yields morphological cell type clusterings that are on par with manual feature-based classification by experts, but without using prior knowledge about the structural features of neurons. Moreover, it outperforms previous approaches on quantitative benchmarks predicting expert labels. Our method could potentially enable data-driven discovery of novel morphological features and cell types in large-scale datasets. It is applicable beyond neuroscience in settings where samples in a dataset are graphs and graph-level embeddings are desired.
Adversarial Attacks Neutralization via Data Set Randomization

Adversarial attacks on deep-learning models pose a serious threat to their reliability and security. Existing defense mechanisms are narrow addressing a specific type of attack or being vulnerable to sophisticated attacks. We propose a new defense mechanism that, while being focused on image-based classifiers, is general with respect to the cited category. It is rooted on hyperspace projection. In particular, our solution provides a pseudo-random projection of the original dataset into a new dataset. The proposed defense mechanism creates a set of diverse projected datasets, where each projected dataset is used to train a specific classifier, resulting in different trained classifiers with different decision boundaries. During testing, it randomly selects a classifier to test the input. Our approach does not sacrifice accuracy over legitimate input. Other than detailing and providing a thorough characterization of our defense mechanism, we also provide a proof of concept of using four optimization-based adversarial attacks (PGD, FGSM, IGSM, and C\&W) and a generative adversarial attack testing them on the MNIST dataset. Our experimental results show that our solution increases the robustness of deep learning models against adversarial attacks and significantly reduces the attack success rate by at least 89% for optimization attacks and 78% for generative attacks. We also analyze the relationship between the number of used hyperspaces and the efficacy of the defense mechanism. As expected, the two are positively correlated, offering an easy-to-tune parameter to enforce the desired level of security. The generality and scalability of our solution and adaptability to different attack scenarios, combined with the excellent achieved results, other than providing a robust defense against adversarial attacks on deep learning networks, also lay the groundwork for future research in the field.