Miracle-dz/UniFAR-FTUs huggingface dataset:
Cosmological information in skew spectra of biased tracers in redshift space

Extracting the non-Gaussian information encoded in the higher-order clustering statistics of the large-scale structure is key to fully realizing the potential of upcoming galaxy surveys. We investigate the information content of the redshift-space weighted skew spectra of biased tracers as efficient estimators for 3-point clustering statistics. The skew spectra are constructed by correlating the observed galaxy field with an appropriately-weighted square of it. We perform numerical Fisher forecasts using two synthetic datasets: the halo catalogs from the Quijote N-body simulations and the galaxy catalogs from the Molino suite. The latter serves to understand the effect of marginalization over a more complex matter-tracer biasing relation. Compared to the power spectrum multipoles, we show that the skew spectra substantially improve the constraints on six parameters of the νΛCDM model, {Ω m , Ω b , h, ns , σ8 , Mν }. Imposing a small-scale cutoff of kmax = 0.25 Mpc-1 h, the improvements in parameter constraints from skew spectra alone range from 23% to 62% for the Quijote halos and from 32% to 71% for the Molino galaxies. Compared to the previous analysis of the bispectrum monopole on the same data and using the same range of scales, the skew spectra of Quijote halos provide competitive constraints. At the same time, the skew spectra outperform the bispectrum monopole for all cosmological parameters for the Molino catalogs. This may result from additional anisotropic information, particularly enhanced in the Molino sample, that is captured by the skew spectra but not by the bispectrum monopole. Our stability analysis of the numerical derivatives shows comparable convergence rates for the power spectrum and the skew spectra, indicating potential underestimation of parameter uncertainties by at most 30%.
Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing

Pre-trained speech Transformers have facilitated great success across various speech processing tasks. However, fine-tuning these encoders for downstream tasks require sufficiently large training data to converge or to achieve state-of-the-art. In text domain this has been partly attributed to sub-optimality of the representation space in pre-trained Transformers. In this work, we take a sober look into pre-trained speech encoders and rewire their representation space without requiring any task-specific labels. Our method utilises neutrally synthesised version of audio inputs along with frame masking to construct positive pairs for contrastive self-supervised learning. When used for augmenting the wav2vec 2 encoder, we observe consistent improvement of isotropy in the representation space. Our experiments on 6 speech processing tasks, exhibit a significant convergence speedup during task fine-tuning as well as consistent task improvement, specially in low-resource settings.
FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correction

Grammatical Error Correction (GEC) has been broadly applied in automatic correction and proofreading system recently. However, it is still immature in Chinese GEC due to limited high-quality data from native speakers in terms of category and scale. In this paper, we present FCGEC, a fine-grained corpus to detect, identify and correct the grammatical errors. FCGEC is a human-annotated corpus with multiple references, consisting of 41,340 sentences collected mainly from multi-choice questions in public school Chinese examinations. Furthermore, we propose a Switch-Tagger-Generator (STG) baseline model to correct the grammatical errors in low-resource settings. Compared to other GEC benchmark models, experimental results illustrate that STG outperforms them on our FCGEC. However, there exists a significant gap between benchmark models and humans that encourages future models to bridge it.
Automated Essay Scoring using Transformers

—Despite being investigated for over ﬁve decades, the task of automated essay scoring continues to draw a lot of attention in the NLP community, in part because of its commercial and educational values as well as the associated research challenges. Large pre-trained models have made remarkable progress in NLP. Data augmentation techniques have also helped build state-of-the-art models for automated essay scoring. Many works in the past have attempted to solve this problem by using RNNs, LSTMs, etc. This work examines the transformer models like BERT, RoBERTa, etc. We empirically demonstrate the effectiveness of transformer models and data augmentation for automated essay grading across many topics using a single model.
Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation

Although human action anticipation is a task which is inherently multi-modal, state-of-the-art methods on well known action anticipation datasets leverage this data by applying ensemble methods and averaging scores of uni-modal anticipation networks. In this work we introduce transformer based modality fusion techniques, which unify multi-modal data at an early stage. Our Anticipative Feature Fusion Transformer (AFFT) proves to be superior to popular score fusion approaches and presents state-of-the-art results outperforming previous methods on EpicKitchens-100 and EGTEA Gaze+. Our model is easily extensible and allows for adding new modalities without architectural changes. Consequently, we extracted audio features on EpicKitchens-100 which we add to the set of commonly used features in the community. 1
Dwarf Galaxy Formation with and without Dark Matter–Baryon Streaming Velocities

We study how supersonic streaming velocities of baryons relative to dark matter—a large-scale effect imprinted at recombination and coherent over ∼3 Mpc scales—affect the formation of dwarf galaxies at z ≳ 5. We perform cosmological hydrodynamic simulations, including and excluding streaming velocities, in regions centered on halos with M vir(z = 0) ≈ 1010 M ⊙; the simulations are part of the Feedback In Realistic Environments (FIRE) project and run with FIRE-3 physics. Our simulations comprise many thousands of systems with halo masses between M vir = 2 × 105 M ⊙ and 2 × 109 M ⊙ in the redshift range z = 20–5. A few hundred of these galaxies form stars and have stellar masses ranging from 100 to 107 M ⊙. While star formation is globally delayed by approximately 50 Myr in the streaming relative to nonstreaming simulations and the number of luminous galaxies is correspondingly suppressed at high redshift in the streaming runs, these effects decay with time. By z = 5, the properties of the simulated galaxies are nearly identical in the streaming versus nonstreaming runs, indicating that any effects of streaming velocities on the properties of galaxies at the mass scale of classical dwarfs and larger do not persist to z = 0.
Synchronization in a Kuramoto mean field game

Abstract The classical Kuramoto model is studied in the setting of an infinite horizon mean field game. The system is shown to exhibit both synchronization and phase transition. Incoherence below a critical value of the interaction parameter is demonstrated by the stability of the uniform distribution. Above this value, the game bifurcates and develops self-organizing time homogeneous Nash equilibria. As interactions get stronger, these stationary solutions become fully synchronized. Results are proved by an amalgam of techniques from nonlinear partial differential equations, viscosity solutions, stochastic optimal control and stochastic processes.
Spiral arms and disc stability in the Andromeda galaxy

Aims: Density waves are often considered as the triggering mechanism of star formation in spiral galaxies. Our aim is to study relations between different star formation tracers (stellar UV and near-IR radiation and emission from HI, CO and cold dust) in the spiral arms of M31, to calculate stability conditions in the galaxy disc and to draw conclusions about possible star formation triggering mechanisms. Methods: We select fourteen spiral arm segments from the de-projected data maps and compare emission distributions along the cross sections of the segments in different datasets to each other, in order to detect spatial offsets between young stellar populations and the star forming medium. By using the disc stability condition as a function of perturbation wavelength and distance from the galaxy centre we calculate the effective disc stability parameters and the least stable wavelengths at different distances. For this we utilise a mass distribution model of M31 with four disc components (old and young stellar discs, cold and warm gaseous discs) embedded within the external potential of the bulge, the stellar halo and the dark matter halo. Each component is considered to have a realistic finite thickness. Results: No systematic offsets between the observed UV and CO/far-IR emission across the spiral segments are detected. The calculated effective stability parameter has a minimal value Q_{eff} ~ 1.8 at galactocentric distances 12 - 13 kpc. The least stable wavelengths are rather long, with the minimal values starting from ~ 3 kpc at distances R > 11 kpc. Conclusions: The classical density wave theory is not a realistic explanation for the spiral structure of M31. Instead, external causes should be considered, e.g. interactions with massive gas clouds or dwarf companions of M31.
Delving into Masked Autoencoders for Multi-Label Thorax Disease Classification

Vision Transformer (ViT) has become one of the most popular neural architectures due to its great scalability, computational efficiency, and compelling performance in many vision tasks. However, ViT has shown inferior performance to Convolutional Neural Network (CNN) on medical tasks due to its data-hungry nature and the lack of an-notated medical data. In this paper, we pre-train ViTs on 266,340 chest X-rays using Masked Autoencoders (MAE) which reconstruct missing pixels from a small part of each image. For comparison, CNNs are also pre-trained on the same 266,340 X-rays using advanced self-supervised methods (e.g. MoCo v2). The results show that our pre-trained ViT performs comparably (sometimes better) to the state-of-the-art CNN (DenseNet-121) for multi-label thorax dis-ease classification. This performance is attributed to the strong recipes extracted from our empirical studies for pre-training and fine-tuning ViT. The pre-training recipe signifies that medical reconstruction requires a much smaller proportion of an image (10% vs. 25%) and a more moderate random resized crop range (0.5∼1.0 vs. 0.2∼1.0) compared with natural imaging. Furthermore, we remark that in-domain transfer learning is preferred whenever possible. The fine-tuning recipe discloses that layer-wise LR decay, RandAug magnitude, and DropPath rate are significant factors to consider. We hope that this study can direct future research on the application of Transformers to a larger variety of medical imaging tasks.
Retrieval Augmentation for Commonsense Reasoning: A Unified Approach

A common thread of retrieval-augmented methods in the existing literature focuses on retrieving encyclopedic knowledge, such as Wikipedia, which facilitates well-defined entity and relation spaces that can be modeled. However, applying such methods to commonsense reasoning tasks faces two unique challenges, i.e., the lack of a general large-scale corpus for retrieval and a corresponding effective commonsense retriever. In this paper, we systematically investigate how to leverage commonsense knowledge retrieval to improve commonsense reasoning tasks. We proposed a unified framework of retrieval-augmented commonsense reasoning (called RACo), including a newly constructed commonsense corpus with over 20 million documents and novel strategies for training a commonsense retriever. We conducted experiments on four different commonsense reasoning tasks. Extensive evaluation results showed that our proposed RACo can significantly outperform other knowledge-enhanced method counterparts, achieving new SoTA performance on the CommonGen and CREAK leaderboards.
Trustworthy Human Computation: A Survey

Human computation is an approach to solving problems that prove difficult using AI only, and involves the cooperation of many humans. Because human computation requires close engagement with both"human populations as users"and"human populations as driving forces,"establishing mutual trust between AI and humans is an important issue to further the development of human computation. This survey lays the groundwork for the realization of trustworthy human computation. First, the trustworthiness of human computation as computing systems, that is, trust offered by humans to AI, is examined using the RAS (Reliability, Availability, and Serviceability) analogy, which define measures of trustworthiness in conventional computer systems. Next, the social trustworthiness provided by human computation systems to users or participants is discussed from the perspective of AI ethics, including fairness, privacy, and transparency. Then, we consider human--AI collaboration based on two-way trust, in which humans and AI build mutual trust and accomplish difficult tasks through reciprocal collaboration. Finally, future challenges and research directions for realizing trustworthy human computation are discussed.
OLLA: Decreasing the Memory Usage of Neural Networks by Optimizing the Lifetime and Location of Arrays

The size of deep neural networks has grown exponentially in recent years. Unfortunately, hardware devices have not kept pace with the rapidly increasing memory requirements. To cope with this, researchers have turned to techniques such as spilling and recomputation, which increase training time, or reduced precision and model pruning, which can affect model accuracy. Wepresent OLLA , an algorithm that optimizes the lifetime and memory location of the tensors used to train neural networks. Our method reduces the memory usage of existing neural networks, without needing any modiﬁcation to the models or their training procedures. We formulate the problem as a joint integer linear program (ILP). We present several techniques to simplify the encoding of the problem, and enable our approach to scale to the size of state-of-the-art neural networks using an off-the-shelf ILP solver. We experimentally demonstrate that OLLA only takes minutes if not seconds to allow the training of neural networks using one-third less memory on average. OLLA is available at https://github.com/facebookresearch/OLLA.
Deep Multi-Branch CNN Architecture for Early Alzheimer’s Detection from Brain MRIs

Alzheimer’s disease (AD) is a neurodegenerative disease that can cause dementia and result in a severe reduction in brain function, inhibiting simple tasks, especially if no preventative care is taken. Over 1 in 9 Americans suffer from AD-induced dementia, and unpaid care for people with AD-related dementia is valued at USD 271.6 billion. Hence, various approaches have been developed for early AD diagnosis to prevent its further progression. In this paper, we first review other approaches that could be used for the early detection of AD. We then give an overview of our dataset and propose a deep convolutional neural network (CNN) architecture consisting of 7,866,819 parameters. This model comprises three different convolutional branches, each having a different length. Each branch is comprised of different kernel sizes. This model can predict whether a patient is non-demented, mild-demented, or moderately demented with a 99.05% three-class accuracy. In summary, the deep CNN model demonstrated exceptional accuracy in the early diagnosis of AD, offering a significant advancement in the field and the potential to improve patient care.
DMODE: Differential Monocular Object Distance Estimation Module without Class Specific Information

Utilizing a single camera for measuring object distances is a cost-effective alternative to stereo-vision and LiDAR. Although monocular distance estimation has been explored in the literature, most existing techniques rely on object class knowledge to achieve high performance. Without this contextual data, monocular distance estimation becomes more challenging, lacking reference points and object-specific cues. However, these cues can be misleading for objects with wide-range variation or adversarial situations, which is a challenging aspect of object-agnostic distance estimation. In this paper, we propose DMODE, a class-agnostic method for monocular distance estimation that does not require object class knowledge. DMODE estimates an object's distance by fusing its fluctuation in size over time with the camera's motion, making it adaptable to various object detectors and unknown objects, thus addressing these challenges. We evaluate our model on the KITTI MOTS dataset using ground-truth bounding box annotations and outputs from TrackRCNN and EagerMOT. The object's location is determined using the change in bounding box sizes and camera position without measuring the object's detection source or class attributes. Our approach demonstrates superior performance in multi-class object distance detection scenarios compared to conventional methods.
Chaos Theory and Adversarial Robustness

Neural networks, being susceptible to adversarial attacks, should face a strict level of scrutiny before being deployed in critical or adversarial applications. This paper uses ideas from Chaos Theory to explain, analyze, and quantify the degree to which neural networks are susceptible to or robust against adversarial attacks. To this end, we present a new metric, the"susceptibility ratio,"given by $\hat \Psi(h, \theta)$, which captures how greatly a model's output will be changed by perturbations to a given input. Our results show that susceptibility to attack grows significantly with the depth of the model, which has safety implications for the design of neural networks for production environments. We provide experimental evidence of the relationship between $\hat \Psi$ and the post-attack accuracy of classification models, as well as a discussion of its application to tasks lacking hard decision boundaries. We also demonstrate how to quickly and easily approximate the certified robustness radii for extremely large models, which until now has been computationally infeasible to calculate directly.
Learning Ultrametric Trees for Optimal Transport Regression

Optimal transport provides a metric which quantifies the dissimilarity between probability measures. For measures supported in discrete metric spaces, finding the optimal transport distance has cubic time complexity in the size of the space. However, measures supported on trees admit a closed-form optimal transport that can be computed in linear time. In this paper, we aim to find an optimal tree structure for a given discrete metric space so that the tree-Wasserstein distance approximates the optimal transport distance in the original space. One of our key ideas is to cast the problem in ultrametric spaces. This helps us optimize over the space of ultrametric trees --- a mixed-discrete and continuous optimization problem --- via projected gradient decent over the space of ultrametric matrices. During optimization, we project the parameters to the ultrametric space via a hierarchical minimum spanning tree algorithm, equivalent to the closest projection to ultrametrics under the supremum norm. Experimental results on real datasets show that our approach outperforms previous approaches (e.g. Flowtree, Quadtree) in approximating optimal transport distances. Finally, experiments on synthetic data generated on ground truth trees show that our algorithm can accurately uncover the underlying trees.
Are we really making much progress in unsupervised graph outlier detection? Revisiting the problem with new insight and superior method

—A large number of studies on Graph Outlier Detection (GOD) have emerged in recent years due to its wide applications, in which Unsupervised Node Outlier Detection (UNOD) on attributed networks is an important area. UNOD focuses on detecting two kinds of typical outliers in graphs: the structural outlier and the contextual outlier. Most existing works conduct the experiments based on the datasets with injected outliers. However, we ﬁnd that the most widely-used outlier injection approach has a serious data leakage issue. By only utilizing such data leakage, a simple approach can achieve the state-of-the-art performance in detecting outliers. In addition, we observe that most existing algorithms have performance drops with varied injection settings. The other major issue is on balanced detection performance between the two types of outliers, which has not been considered by existing studies. In this paper, we analyze the cause of the data leakage issue in depth since the injection approach is a building block to advance UNOD. Moreover, we devise a novel variance-based model to detect structural outliers, which is more robust to different injection settings. On top of this, we propose a new framework, Variance-based Graph Outlier Detection (VGOD), which combines our variance-based model and attribute reconstruction model to detect outliers in a balanced way. Finally, we conduct extensive experiments to demonstrate the effectiveness and the efﬁciency of VGOD. The results on 5 real-world datasets validate that VGOD achieves not only the best performance in detecting outliers but also a balanced detection performance between structural and contextual outliers.
Span-based joint entity and relation extraction augmented with sequence tagging mechanism

Span-based joint extraction simultaneously conducts named entity recognition (NER) and relation extraction (RE) in text span form. However, since previous span-based models rely on span-level classifications, they cannot benefit from token-level label information, which has been proven advantageous for the task. In this paper, we propose a Sequence Tagging augmented Span-based Network (STSN), a span-based joint model that can make use of token-level label information. In STSN, we construct a core neural architecture by deep stacking multiple attention layers, each of which consists of three basic attention units. On the one hand, the core architecture enables our model to learn token-level label information via the sequence tagging mechanism and then uses the information in the span-based joint extraction; on the other hand, it establishes a bi-directional information interaction between NER and RE. Experimental results on three benchmark datasets show that STSN consistently outperforms the strongest baselines in terms of F1, creating new state-of-the-art results.
A study of uncertainty quantiﬁcation in overparametrized high-dimensional models

Uncertainty quantiﬁcation is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconﬁdent estimates in the context of overparametrized neural networks. Several methods, ranging from temperature scaling to diﬀerent Bayesian treatments of neural networks, have been proposed to mitigate overconﬁdence, most often supported by the numerical observation that they yield better calibrated uncertainty measures. In this work, we provide a sharp comparison between popular uncertainty measures for binary classiﬁcation in a mathematically tractable model for overparametrized neural networks: the random features model. We discuss a trade-oﬀ between classiﬁcation accuracy and calibration, unveiling a double descent like behavior in the calibration curve of optimally regularized estimators as a function of overparametrization. This is in contrast with the empirical Bayes method, which we show to be well calibrated in our setting despite the higher generalization error and overparametrization.