Miracle-dz/UniFAR-FTUs huggingface dataset:
Learning the Pareto Front with Hypernetworks

Multi-objective optimization problems are prevalent in machine learning. These problems have a set of optimal solutions, called the Pareto front, where each point on the front represents a different trade-off between possibly conflicting objectives. Recent optimization algorithms can target a specific desired ray in loss space, but still face two grave limitations: (i) A separate model has to be trained for each point on the front; and (ii) The exact trade-off must be known prior to the optimization process. Here, we tackle the problem of learning the entire Pareto front, with the capability of selecting a desired operating point on the front after training. We call this new setup Pareto-Front Learning (PFL). We describe an approach to PFL implemented using HyperNetworks, which we term Pareto HyperNetworks (PHNs). PHN learns the entire Pareto front simultaneously using a single hypernetwork, which receives as input a desired preference vector and returns a Pareto-optimal model whose loss vector is in the desired ray. The unified model is runtime efficient compared to training multiple models, and generalizes to new operating points not used during training. We evaluate our method on a wide set of problems, from multi-task regression and classification to fairness. PHNs learns the entire Pareto front in roughly the same time as learning a single point on the front, and also reaches a better solution set. PFL opens the door to new applications where models are selected based on preferences that are only available at run time.
Optimizing Transformers with Approximate Computing for Faster, Smaller and more Accurate NLP Models

Transformer models have garnered a lot of interest in recent years by delivering state-of-the-art performance in a range of Natural Language Processing (NLP) tasks. However, these models can have over a hundred billion parameters, presenting very high computational and memory requirements. We address this challenge through Approximate Computing, specifically targeting the use of Transformers in NLP tasks. Transformers are typically pre-trained and subsequently specialized for specific tasks through transfer learning. Based on the observation that pre-trained Transformers are often over-parameterized for several downstream NLP tasks, we propose a framework to create smaller, faster and in some cases more accurate models. The key cornerstones of the framework are a Significance Analysis (SA) method that identifies components in a pre-trained Transformer that are less significant for a given task, and techniques to approximate the less significant components. Our approximations include pruning of blocks, attention heads and weight groups, quantization of less significant weights and a low-complexity sign-matching based attention mechanism. Our framework can be adapted to produce models that are faster, smaller and/or more accurate, depending on the user's constraints. We apply our framework to seven Transformer models, including optimized models like DistilBERT and Q8BERT, and three downstream tasks. We demonstrate that our framework produces models that are up to 4x faster and up to 14x smaller (with less than 0.5% relative accuracy degradation), or up to 5.5% more accurate with simultaneous improvements of up to 9.83x in model size or 2.94x in speed.
Productivity and partial budget analysis in wheat-rice sequences as influenced by integrated plant nutrition system and legume crops inclusion

The experiments were carried out at the Regional Wheat Research Centre, Rajshahi of Bangladesh Agricultural Research Institute (BARI) for two consecutive years, 2009-10 and 2010-11 to evaluate the agro-economic productivity of Wheat-Rice cropping sequence as influenced by integrated plant nutrition system (IPNS) and inclusion of legume crops. The experiment comprised of four cropping sequences viz. Wheat-Mungbean- T. Aman rice , Wheat-Blackgram- T. Aman rice , Wheat-Sesbania- T. Aman rice and Wheat- Fallow- T. Aman rice; and six nutrient treatments viz. 100% recommended nutrient rates, IPNS with 3 t ha -1 poultry manure (PM), IPNS with 6 t ha -1 PM, IPNS with 5 t ha -1 cowdung (CD), IPNS with 10 t ha -1 CD and farmers’ practice (FP). It was carried out in a split-plot design assigning cropping sequences in the main plots and nutrient treatments in the sub-plots with three replications. For the IPNS, the 100% nutrient rates were adjusted with manure and fertilizers. Inclusion of mungbean in the Wheat-Rice cropping sequence showed higher production cost but it gave higher system productivity, gross return, gross margin, benefitcost ratio and production efficiency. This cropping sequence gave on an average 57% higher wheat equivalent yield (WEY) compared to the existing Wheat-Rice sequence followed by blackgram included cropping sequence. The IPNS based fertilizer and manure application had better yield performance, WEY, gross margin, gross return, benefit-cost ratio, production efficiency and land use efficiency as compared to 100% chemical fertilizers or FP. It is concluded that the Wheat-Mungbean-Rice cropping sequence with IPNS approach is a productive and profitable technology for crop cultivation. Bangladesh J. Agril. Res. 41(1): 17-39, March 2016
Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples

Adversarial training and its variants have become de facto standards for learning robust deep neural networks. In this paper, we explore the landscape around adversarial training in a bid to uncover its limits. We systematically study the effect of different training losses, model sizes, activation functions, the addition of unlabeled data (through pseudo-labeling) and other factors on adversarial robustness. We discover that it is possible to train robust models that go well beyond state-of-the-art results by combining larger models, Swish/SiLU activations and model weight averaging. We demonstrate large improvements on CIFAR-10 and CIFAR-100 against $\ell_\infty$ and $\ell_2$ norm-bounded perturbations of size $8/255$ and $128/255$, respectively. In the setting with additional unlabeled data, we obtain an accuracy under attack of 65.88% against $\ell_\infty$ perturbations of size $8/255$ on CIFAR-10 (+6.35% with respect to prior art). Without additional data, we obtain an accuracy under attack of 57.20% (+3.46%). To test the generality of our findings and without any additional modifications, we obtain an accuracy under attack of 80.53% (+7.62%) against $\ell_2$ perturbations of size $128/255$ on CIFAR-10, and of 36.88% (+8.46%) against $\ell_\infty$ perturbations of size $8/255$ on CIFAR-100.
Deformable DETR: Deformable Transformers for End-to-End Object Detection

DETR has been recently proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance. However, it suffers from slow convergence and limited feature spatial resolution, due to the limitation of Transformer attention modules in processing image feature maps. To mitigate these issues, we proposed Deformable DETR, whose attention modules only attend to a small set of key sampling points around a reference. Deformable DETR can achieve better performance than DETR (especially on small objects) with 10$\times$ less training epochs. Extensive experiments on the COCO benchmark demonstrate the effectiveness of our approach. Code shall be released.
Fairness in Machine Learning: A Survey

When Machine Learning technologies are used in contexts that affect citizens, companies as well as researchers need to be confident that there will not be any unexpected social implications, such as bias towards gender, ethnicity, and/or people with disabilities. There is significant literature on approaches to mitigate bias and promote fairness, yet the area is complex and hard to penetrate for newcomers to the domain. This article seeks to provide an overview of the different schools of thought and approaches that aim to increase the fairness of Machine Learning. It organises approaches into the widely accepted framework of pre-processing, in-processing, and post-processing methods, subcategorizing into a further 11 method areas. Although much of the literature emphasizes binary classification, a discussion of fairness in regression, recommender systems, and unsupervised learning is also provided along with a selection of currently available open source libraries. The article concludes by summarising open challenges articulated as five dilemmas for fairness research.
Pancakes as opposed to Swiss cheese

We examine a novel class of toy models of cosmological inhomogeneities by smoothly matching along a suitable hypersurface an arbitrary number of sections of ‘quasi flat’ inhomogeneous and anisotropic Szekeres-II models to sections of any spatially flat cosmology that can be described by the Robertson–Waker metric (including de Sitter, anti de Sitter and Minkowski spacetimes). The resulting ‘pancake’ models are quasi-flat analogues to the well known spherical ‘Swiss-cheese’ models found in the literature. Since Szekeres-II models can be, in general, compatible with a wide range of sources (dissipative fluids, mixtures of non-comoving fluids, mixtures of fluids with scalar or magnetic fields or gravitational waves), the pancake configurations we present allow for a description of a wide collection of localized sources embedded in a Robertson–Waker geometry. We provide various simple examples of arbitrary numbers of Szekeres-II regions (whose sources are comoving dust and energy flux interpreted as a field of peculiar velocities) matched with Einstein de Sitter, ΛCDM and de Sitter backgrounds. We also prove that the Szekeres-II regions can be rigorously regarded as ‘exact’ covariant perturbations on a background defined by the matching discussed above. We believe that these models can be useful to test ideas on averaging and backreaction and on the effect of inhomogeneities on cosmic evolution and observations.
Dense Relational Image Captioning via Multi-task Triple-Stream Networks

We introduce dense relational captioning, a novel image captioning task which aims to generate multiple captions with respect to relational information between objects in a visual scene. Relational captioning provides explicit descriptions of each relationship between object combinations. This framework is advantageous in both diversity and amount of information, leading to a comprehensive image understanding based on relationships, e.g., relational proposal generation. For relational understanding between objects, the part-of-speech (POS, i.e., subject-object-predicate categories) can be a valuable prior information to guide the causal sequence of words in a caption. We enforce our framework to not only learn to generate captions but also predict the POS of each word. To this end, we propose the multi-task triple-stream network (MTTSNet) which consists of three recurrent units responsible for each POS which is trained by jointly predicting the correct captions and POS for each word. In addition, we found that the performance of MTTSNet can be improved by modulating the object embeddings with an explicit relational module. We demonstrate that our proposed model can generate more diverse and richer captions, via extensive experimental analysis on large scale datasets and several metrics. We additionally extend analysis to an ablation study, applications on holistic image captioning, scene graph generation, and retrieval tasks.
Ferrimagnetic 120∘magnetic structure inCu2OSO4

We report magnetic properties of a 3d$^9$ (Cu$^{2+}$) magnetic insulator Cu2OSO4 measured on both powder and single crystal. The magnetic atoms of this compound form layers, whose geometry can be described either as a system of chains coupled through dimers or as a Kagom\'e lattice where every 3rd spin is replaced by a dimer. Specific heat and DC-susceptibility show a magnetic transition at 20 K, which is also confirmed by neutron scattering. Magnetic entropy extracted from the specific heat data is consistent with a $S=1/2$ degree of freedom per Cu$^{2+}$, and so is the effective moment extracted from DC-susceptibility. The ground state has been identified by means of neutron diffraction on both powder and single crystal and corresponds to a $\sim120$ degree spin structure in which ferromagnetic intra-dimer alignment results in a net ferrimagnetic moment. No evidence is found for a change in lattice symmetry down to 2 K. Our results suggest that \sample \ represents a new type of model lattice with frustrated interactions where interplay between magnetic order, thermal and quantum fluctuations can be explored.
Active Tuning

We introduce Active Tuning, a novel paradigm for optimizing the internal dynamics of recurrent neural networks (RNNs) on the fly. In contrast to the conventional sequence-to-sequence mapping scheme, Active Tuning decouples the RNN's recurrent neural activities from the input stream, using the unfolding temporal gradient signal to tune the internal dynamics into the data stream. As a consequence, the model output depends only on its internal hidden dynamics and the closed-loop feedback of its own predictions; its hidden state is continuously adapted by means of the temporal gradient resulting from backpropagating the discrepancy between the signal observations and the model outputs through time. In this way, Active Tuning infers the signal actively but indirectly based on the originally learned temporal patterns, fitting the most plausible hidden state sequence into the observations. We demonstrate the effectiveness of Active Tuning on several time series prediction benchmarks, including multiple super-imposed sine waves, a chaotic double pendulum, and spatiotemporal wave dynamics. Active Tuning consistently improves the robustness, accuracy, and generalization abilities of all evaluated models. Moreover, networks trained for signal prediction and denoising can be successfully applied to a much larger range of noise conditions with the help of Active Tuning. Thus, given a capable time series predictor, Active Tuning enhances its online signal filtering, denoising, and reconstruction abilities without the need for additional training.
Energy-based Out-of-distribution Detection

Determining whether inputs are out-of-distribution (OOD) is an essential building block for safely deploying machine learning models in the open world. However, previous methods relying on the softmax confidence score suffer from overconfident posterior distributions for OOD data. We propose a unified framework for OOD detection that uses an energy score. We show that energy scores better distinguish in- and out-of-distribution samples than the traditional approach using the softmax scores. Unlike softmax confidence scores, energy scores are theoretically aligned with the probability density of the inputs and are less susceptible to the overconfidence issue. Within this framework, energy can be flexibly used as a scoring function for any pre-trained neural classifier as well as a trainable cost function to shape the energy surface explicitly for OOD detection. On a CIFAR-10 pre-trained WideResNet, using the energy score reduces the average FPR (at TPR 95%) by 18.03% compared to the softmax confidence score. With energy-based training, our method outperforms the state-of-the-art on common benchmarks.
DART: A Lightweight Quality-Suggestive Data-to-Text Annotation Tool

We present a lightweight annotation tool, the Data AnnotatoR Tool (DART), for the general task of labeling structured data with textual descriptions. The tool is implemented as an interactive application that reduces human efforts in annotating large quantities of structured data, e.g. in the format of a table or tree structure. By using a backend sequence-to-sequence model, our system iteratively analyzes the annotated labels in order to better sample unlabeled data. In a simulation experiment performed on annotating large quantities of structured data, DART has been shown to reduce the total number of annotations needed with active learning and automatically suggesting relevant labels.
Prediction intervals for Deep Neural Networks

The aim of this paper is to propose a suitable method for constructing prediction intervals for the output of neural network models. To do this, we adapt the extremely randomized trees method originally developed for random forests to construct ensembles of neural networks. The extra-randomness introduced in the ensemble reduces the variance of the predictions and yields gains in out-of-sample accuracy. An extensive Monte Carlo simulation exercise shows the good performance of this novel method for constructing prediction intervals in terms of coverage probability and mean square prediction error. This approach is superior to state-of-the-art methods extant in the literature such as the widely used MC dropout and bootstrap procedures. The out-of-sample accuracy of the novel algorithm is further evaluated using experimental settings already adopted in the literature.
MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics

Posing reading comprehension as a generation problem provides a great deal of flexibility, allowing for open-ended questions with few restrictions on possible answers. However, progress is impeded by existing generation metrics, which rely on token overlap and are agnostic to the nuances of reading comprehension. To address this, we introduce a benchmark for training and evaluating generative reading comprehension metrics: MOdeling Correctness with Human Annotations. MOCHA contains 40K human judgement scores on model outputs from 6 diverse question answering datasets and an additional set of minimal pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms baseline metrics by 10 to 36 absolute Pearson points on held-out annotations. When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy, outperforming baselines by 14 to 26 absolute percentage points while leaving significant room for improvement. MOCHA presents a challenging problem for developing accurate and robust generative reading comprehension metrics.
Multidimensional Adaptive Penalised Splines with Application to Neurons' Activity Studies

P-spline models have achieved great popularity both in statistical and in applied research. A possible drawback of P-spline is that they assume a smooth transition of the covariate effect across its whole domain. In some practical applications, however, it is desirable and needed to adapt smoothness locally to the data, and adaptive P-splines have been suggested. Yet, the extra flexibility afforded by adaptive P-spline models is obtained at the cost of a high computational burden, especially in a multidimensional setting. Furthermore, to the best of our knowledge, the literature lacks proposals for adaptive P-splines in more than two dimensions. Motivated by the need for analysing data derived from experiments conducted to study neurons' activity in the visual cortex of behaving monkeys, this work presents a novel locally adaptive P-spline model in two (e.g., space) and three (space and time) dimensions. Estimation is based on the recently proposed SOP (Separation of Overlapping Precision matrices) method, which provides the speed we look for. The practical performance of the proposal is evaluated through simulations, and comparisons with alternative methods are reported. In addition to the spatio-temporal analysis of the data that motivated this work, we also discuss an application in two dimensions on the absenteeism of workers.
TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling

We present a novel approach to the challenging problem of label-free text style transfer. Unlike previous approaches that use parallel or non-parallel labeled data, our technique removes the need for labels entirely, relying instead on the implicit connection in style between adjacent sentences in unlabeled text. We show that T5 (Raffel et al., 2020), a strong pretrained text-to-text model, can be adapted to extract a style vector from arbitrary text and use this vector to condition the decoder to perform style transfer. As the resulting learned style vector space encodes many facets of textual style, we recast transfers as "targeted restyling" vector operations that adjust specific attributes of the input text while preserving others. When trained over unlabeled Amazon reviews data, our resulting TextSETTR model is competitive on sentiment transfer, even when given only four exemplars of each class. Furthermore, we demonstrate that a single model trained on unlabeled Common Crawl data is capable of transferring along multiple dimensions including dialect, emotiveness, formality, politeness, and sentiment.
Characteristic time variability of gravitational-wave and neutrino signals from three-dimensional simulations of non-rotating and rapidly rotating stellar core collapse

We present results from full general relativistic three-dimensional hydrodynamics simulations of stellar core collapse of a 70 M$_\odot$ star with spectral neutrino transport. To investigate the impact of rotation on non-axisymmetric instabilities, we compute three models by parametrically changing the initial strength of rotation. The most rapidly rotating model exhibits a transient development of the low-$T/|W|$ instability with one-armed spiral flow at the early postbounce phase. Subsequently, the two-armed spiral flow appears, which persists during the simulation time. The moderately rotating model also shows the growth of the low-$T/|W|$ instability, but only with the two-armed spiral flow. In the nonrotating model, a vigorous activity of the standing accretion-shock instability (SASI) is only observed. The SASI is first dominated by the sloshing mode, which is followed by the spiral SASI until the black hole formation. We present a spectrogram analysis of the gravitational waves (GWs) and neutrinos, focusing on the time correlation. Our results show that characteristic time modulations in the GW and neutrino signals can be linked to the growth of the non-axisymmetric instabilities. We find that the degree of the protoneutron star (PNS) deformation, depending upon which modes of the non-axisymmetric instabilities develop, predominantly affects the characteristic frequencies of the correlated GW and neutrino signals. We point out that these signals would be simultaneously detectable by the current-generation detectors up to $\sim10$ kpc. Our findings suggest that the joint observation of GWs and neutrinos is indispensable for extracting information on the PNS evolution preceding the black hole formation.
Non-perturbative gauge transformations of arbitrary fermion correlation functions in quantum electrodynamics

We study the transformation of the dressed electron propagator and the general $N$-point functions under a change in the covariant gauge of internal photon propagators. We re-establish the well known Landau-Khalatnikov-Fradkin transformation for the propagator and generalise it to arbitrary correlation functions in configuration space, finding that it coincides with the analogous result for scalar fields. We comment on the consequences for perturbative application in momentum-space.