Miracle-dz/UniFAR-FTUs huggingface dataset:
Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their"width"approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.
EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling

This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories. MCMC trajectories of different lengths correspond to models with different purposes. Our experiments cover three different trajectory magnitudes and learning outcomes: 1) shortrun sampling for image generation; 2) midrun sampling for classifier-agnostic adversarial defense; and 3) longrun sampling for principled modeling of image probability densities. To achieve these outcomes, we introduce three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning. With standard network architectures and an unaltered ML objective, our MCMC initialization methods alone enable significant performance gains across the three applications that we investigate. Our results include state-of-the-art FID scores for unnormalized image densities on the CIFAR-10 and ImageNet datasets; state-of-the-art adversarial defense on CIFAR-10 among purification methods and the first EBM defense on ImageNet; and scalable techniques for learning valid probability densities. Code for this project can be found at https://github.com/point0bar1/ebm-life-cycle.
Optimality Conditions and Algorithms for Top-K Arm Identification

We consider the top-k arm identiﬁcation problem for multi-armed bandits with rewards belonging to a one-parameter canonical exponential family. The objective is to select the set of k arms with the highest mean rewards by sequential allocation of sampling eﬀorts. We propose a uniﬁed optimal allocation problem that identiﬁes the complexity measures of this problem under the ﬁxed-conﬁdence, ﬁxed-budget settings, and the posterior convergence rate from the Bayesian perspective. We provide the ﬁrst characterization of its optimality. We provide the ﬁrst provably optimal algorithm in the ﬁxed-conﬁdence setting for k > 1 . We also propose an eﬃcient heuristic algorithm for the top-k identiﬁcation problem. Extensive numerical experiments demonstrate superior performance compare to existing methods in all three settings.
Model predictive control of water quality in drinking water distribution systems considering disinfection by-products

The shortage in water resources have been observed all over the world. However, the safety of drinking water has been given much attention by scientists because the disinfection will react with organic matters in drinking water to generate disinfection by-products (DBPs) which are considered as the cancerigenic matters. Although much research has been carried out on the water quality control problem in DWDS, the water quality model considered is linear with only chlorine dynamics. Compared to the linear water quality model, the nonlinear water quality model considers the interaction between chlorine and DBPs dynamics. The thesis proposes a nonlinear model predictive controller which utilises the newly derived nonlinear water quality model as a control alternative for controlling water quality. EPANET and EPANET-MSN are simulators utilised for modelling in the developed nonlinear MPC controller. Uncertainty is not considered in these simulators. This thesis proposes the bounded PPM in a form of multi-input multi-output to robustly bound parameters of chlorine and DBPs jointly and to robustly predict water quality control outputs for quality control purpose. The methodologies and algorithms developed in this thesis are verified by applying extended case studies to the example DWDS. The simulation results are presented and critically analysed.
On the energy dependence of the QPO phenomenon in the black hole system MAXI J1535-571

Previous analysis of AstroSat observations of the black hole system MAXI J1535-571, have revealed the presence of a strong Quasi-Periodic Oscillation (QPO) whose frequency is correlated with the high energy spectral index. Here, we fit the spectra as emitted from a truncated disc with an inner hot corona, study the QPO frequency dependence on other spectral parameters and model the energy dependent r.m. s and time-lag of the QPO to identify the physical spectral parameters whose variation are responsible for the QPO. The QPO frequency is found to also correlate with the scattering fraction (i.e. the fraction of the soft photons Comptonized) and its dependence on the accretion rate and inner disc radii is consistent with it being the dynamical frequency. The time-lag between the hard and soft photons is negative for QPO frequency >2.2 Hz and is positive for lesser values, making this the second black hole system to show this behaviour after GRS 1915+105. Modelling the energy dependent time-lag and r.m. s requires correlated variation of the accretion rate, inner disc radii and the coronal heating rate, with the latter having a time-lag compared to the other two for QPO frequencies less than <2.2 Hz and which changes sign (i.e. the coronal heating variation precedes the accretion rate one) for higher values. The implications of the results are discussed.
Word-order Typology in Multilingual BERT: A Case Study in Subordinate-Clause Detection

The capabilities and limitations of BERT and similar models are still unclear when it comes to learning syntactic abstractions, in particular across languages. In this paper, we use the task of subordinate-clause detection within and across languages to probe these properties. We show that this task is deceptively simple, with easy gains offset by a long tail of harder cases, and that BERT’s zero-shot performance is dominated by word-order effects, mirroring the SVO/VSO/SOV typology.
WeDef: Weakly Supervised Backdoor Defense for Text Classification

Existing backdoor defense methods are only effective for limited trigger types. To defend different trigger types at once, we start from the class-irrelevant nature of the poisoning process and propose a novel weakly supervised backdoor defense framework WeDef. Recent advances in weak supervision make it possible to train a reasonably accurate text classifier using only a small number of user-provided, class-indicative seed words. Such seed words shall be considered independent of the triggers. Therefore, a weakly supervised text classifier trained by only the poisoned documents without their labels will likely have no backdoor. Inspired by this observation, in WeDef, we define the reliability of samples based on whether the predictions of the weak classifier agree with their labels in the poisoned training set. We further improve the results through a two-phase sanitization: (1) iteratively refine the weak classifier based on the reliable samples and (2) train a binary poison classifier by distinguishing the most unreliable samples from the most reliable samples. Finally, we train the sanitized model on the samples that the poison classifier predicts as benign. Extensive experiments show that WeDef is effective against popular trigger-based attacks (e.g., words, sentences, and paraphrases), outperforming existing defense methods.
Quadratic models for understanding neural network dynamics

While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the"catapult phase"[Lewkowycz et al. 2020] that arises when training such models with large learning rates. We then empirically show that the behaviour of neural quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models can be an effective tool for analysis of neural networks.
Semi-Parametric Deep Neural Networks in Linear Time and Memory

Recent advances in deep learning have been driven by large-scale parametric models, which can be computationally expensive and lack interpretability. Semi-parametric methods query the training set at inference time and can be more compact, although they typically have quadratic computational complexity. Here, we introduce SPIN, a general-purpose semi-parametric neural architecture whose computational cost is linear in the size and dimensionality of the data. Our architecture is inspired by inducing point methods and relies on a novel application of cross-attention between datapoints. At inference time, its computational cost is constant in the training set size as the data gets distilled into a ﬁxed number of inducing points. We ﬁnd that our method reduces the computational requirements of existing semi-parametric models by up to an order of magnitude across a range of datasets and improves state-of-the-art performance on an important practical problem, genotype imputation.
mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Large-scale pre-trained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections.mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability on vision-language and video-language tasks. The code and pre-trained models are available at https://github.com/alibaba/AliceMind
Mind the Gap: Alleviating Local Imbalance for Unsupervised Cross-Modality Medical Image Segmentation

Unsupervised cross-modality medical image adaptation aims to alleviate the severe domain gap between different imaging modalities without using the target domain label. A key in this campaign relies upon aligning the distributions of source and target domain. One common attempt is to enforce the global alignment between two domains, which, however, ignores the fatal local-imbalance domain gap problem, i.e., some local features with larger domain gap are harder to transfer. Recently, some methods conduct alignment focusing on local regions to improve the efficiency of model learning. While this operation may cause a deficiency of critical information from contexts. To tackle this limitation, we propose a novel strategy to alleviate the domain gap imbalance considering the characteristics of medical images, namely Global-Local Union Alignment. Specifically, a feature-disentanglement style-transfer module first synthesizes the target-like source images to reduce the global domain gap. Then, a local feature mask is integrated to reduce the ‘inter-gap’ for local features by prioritizing those discriminative features with larger domain gap. This combination of global and local alignment can precisely localize the crucial regions in segmentation target while preserving the overall semantic consistency. We conduct a series of experiments with two cross-modality adaptation tasks, i,e. cardiac substructure and abdominal multi-organ segmentation. Experimental results indicate that our method achieves state-of-the-art performance in both tasks.
Research methods and testing stand developed to examine vibrations generated by rolling bearing

The article presents research methods of rolling bearing vibration level measurement and evaluation of vibration measurement capability of Anderon meter STPPD developed at the Kielce University of Technology. In order to assess the measuring ability of Anderon meter STPPD comparative studies were performed. In these studies the noise tester MVH900 SKF - a standard instrument for measuring vibration, was adopted as a professional device. Analysis of the test results allow to reliably assess that both devices have comparable measurement capabilities.
Randomly Initialized One-Layer Neural Networks Make Data Linearly Separable

Recently, neural networks have demonstrated remarkable capabilities in mapping two arbitrary sets to two linearly separable sets. The prospect of achieving this with randomly initialized neural networks is particularly appealing due to the computational efficiency compared to fully trained networks. This paper contributes by establishing that, given sufficient width, a randomly initialized one-layer neural network can, with high probability, transform two sets into two linearly separable sets without any training. Moreover, we furnish precise bounds on the necessary width of the neural network for this phenomenon to occur. Our initial bound exhibits exponential dependence on the input dimension while maintaining polynomial dependence on all other parameters. In contrast, our second bound is independent of input dimension, effectively surmounting the curse of dimensionality. The main tools used in our proof heavily relies on a fusion of geometric principles and concentration of random matrices.
Utilizing Language-Image Pretraining for Efficient and Robust Bilingual Word Alignment

Word translation without parallel corpora has become feasible, rivaling the performance of supervised methods. Recent findings have shown that the accuracy and robustness of unsupervised word translation (UWT) can be improved by making use of visual observations, which are universal representations across languages. In this work, we investigate the potential of using not only visual observations but also pretrained language-image models for enabling a more efficient and robust UWT. Specifically, we develop a novel UWT method dubbed Word Alignment using Language-Image Pretraining (WALIP), which leverages visual observations via the shared embedding space of images and texts provided by CLIP models (Radford et al., 2021). WALIP has a two-step procedure. First, we retrieve word pairs with high confidences of similarity, computed using our proposed image-based fingerprints, which define the initial pivot for the word alignment. Second, we apply our robust Procrustes algorithm to estimate the linear mapping between two embedding spaces, which iteratively corrects and refines the estimated alignment. Our extensive experiments show that WALIP improves upon the state-of-the-art performance of bilingual word alignment for a few language pairs across different word embeddings and displays great robustness to the dissimilarity of language pairs or training corpora for two word embeddings.
Analysing the Greek Parliament Records with Emotion Classification

In this project, we tackle emotion classification for the Greek language, presenting and releasing a new dataset in Greek. We fine-tune and assess Transformer-based masked language models that were pre-trained on monolingual and multilingual resources, and we present the results per emotion and by aggregating at the sentiment and subjectivity level. The potential of the presented resources is investigated by detecting and studying the emotion of `disgust' in the Greek Parliament records. We: (a) locate the months with the highest values from 1989 to present, (b) rank the Greek political parties based on the presence of this emotion in their speeches, and (c) study the emotional context shift of words used to stigmatise people.
Use of Recursive Partitioning to Predict National Bridge Inventory Condition Ratings from National Bridge Elements Condition Data

In the United States, National Bridge Inventory (NBI) condition ratings, since the 1970s, and AASHTO’s commonly recognized (CoRe) element condition data, since the 1990s, have provided two major sources of bridge condition data. Although these separate systems of condition assessment had their individual uses, comparing the two, and mapping one from the other had uses for both state and federal agencies and the bridge management community. Alternative methods for this mapping have been proposed in the literature with varying predictive accuracy. With the publication of the new AASHTO Manual for Bridge Element Inspection in 2013, national bridge elements (NBEs) replace the CoRe element condition data as the comparable condition data for the NBI condition ratings. This paper investigates the use of the recursive partitioning method to develop classification trees that predict NBI condition ratings from NBE condition data. On the basis of data from a 2016 submission and 12 transportation agencies, classification trees were developed that presented the most likely NBI condition ratings for a set of logical conditions based on the relative element quantities and the percentage of element quantities in the condition states. The predictive accuracies for the trees are sufficient, and the percentages of exact matches and matches within one error term are better than other studies in the literature. Although the trees can be improved in the future with the availability of more NBE data submissions, the study presented preliminary decision trees with sufficient predictive accuracy that could be adopted by transportation agencies for a variety of bridge management functions.
All that glitters is not gold: Relational events models with spurious events

As relational event models are an increasingly popular model for studying relational structures, the reliability of large-scale event data collection becomes more and more important. Automated or human-coded events often suffer from non-negligible false-discovery rates in event identification. And most sensor data are primarily based on actors’ spatial proximity for predefined time windows; hence, the observed events could relate either to a social relationship or random co-location. Both examples imply spurious events that may bias estimates and inference. We propose the Relational Event Model for Spurious Events (REMSE), an extension to existing approaches for interaction data. The model provides a flexible solution for modeling data while controlling for spurious events. Estimation of our model is carried out in an empirical Bayesian approach via data augmentation. Based on a simulation study, we investigate the properties of the estimation procedure. To demonstrate its usefulness in two distinct applications, we employ this model to combat events from the Syrian civil war and student co-location data. Results from the simulation and the applications identify the REMSE as a suitable approach to modeling relational event data in the presence of spurious events.
Un-modelled Pioneer 10/11 clock acceleration from the Schwarzschild metric and Flyby energy Increase

When a space-craft reaches a large radial velocity, we expand the time dependent radial distance to linear order in time. In the Schwarzschild metric this linear term in time leads to a signiﬁcant unmodelled clock acceleration. For Pioneer 10, this acceleration is found to be about 40% of the observed anomalous acceleration. As measured distances are geocentric distances instead of the heliocentric distances in the model, this unmodelled acceleration has an annual oscillation with amplitude close to observations. For Pioneer 11, we ﬁnd that the unmodelled acceleration ”On-Set” is caused by the ﬂyby energy Increase to the space-craft’s orbit.
Pairs, trimers, and BCS-BEC crossover near a flat band: Sawtooth lattice

We investigate pairing and superconductivity in the attractive Fermi Hubbard model on the one-dimensional sawtooth lattice, which exhibits a ﬂat band by ﬁne-tuning the hopping rates. We ﬁrst solve the two-body problem, both analytically and numerically, to extract the binding energy and the eﬀective mass of the pairs. Based on the DMRG method, we address the ground-state properties of the many-body system, assuming equal spin populations. We compare our results with those available for a linear chain, where the model is integrable by Bethe ansatz, and show that the multiband nature of the system substantially modiﬁes the physics of the BCS-BEC crossover. Near a ﬂat band, the chemical potential remains always close to its zero-density limit predicted by the two-body physics. In contrast, the pairing gap exhibits a remarkably strong density dependence and, diﬀerently from the pair binding energy, it is no longer peaked at the ﬂat-band point. We show that these results can be interpreted in terms of polarization screening eﬀects, due to an anomalous attraction between pairs in the medium and single fermions. Importantly, we unveil that three-body bound states (trimers) exist in the sawtooth lattice, in sharp contrast with the linear chain geometry, and we compute their binding energy. The nature of these states is investigated via a strong coupling variational approach, revealing that they originate from tunneling-induced exchange processes.
On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions

Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions. In this paper, we prove several properties of KL divergence between multivariate Gaussian distributions. First, for any two $n$-dimensional Gaussian distributions $\mathcal{N}_1$ and $\mathcal{N}_2$, we give the supremum of $KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\leq \varepsilon\ (\varepsilon>0)$. For small $\varepsilon$, we show that the supremum is $\varepsilon + 2\varepsilon^{1.5} + O(\varepsilon^2)$. This quantifies the approximate symmetry of small KL divergence between Gaussians. We also find the infimum of $KL(\mathcal{N}_1||\mathcal{N}_2)$ when $KL(\mathcal{N}_2||\mathcal{N}_1)\geq M\ (M>0)$. We give the conditions when the supremum and infimum can be attained. Second, for any three $n$-dimensional Gaussians $\mathcal{N}_1$, $\mathcal{N}_2$, and $\mathcal{N}_3$, we find an upper bound of $KL(\mathcal{N}_1||\mathcal{N}_3)$ if $KL(\mathcal{N}_1||\mathcal{N}_2)\leq \varepsilon_1$ and $KL(\mathcal{N}_2||\mathcal{N}_3)\leq \varepsilon_2$ for $\varepsilon_1,\varepsilon_2\ge 0$. For small $\varepsilon_1$ and $\varepsilon_2$, we show the upper bound is $3\varepsilon_1+3\varepsilon_2+2\sqrt{\varepsilon_1\varepsilon_2}+o(\varepsilon_1)+o(\varepsilon_2)$. This reveals that KL divergence between Gaussians follows a relaxed triangle inequality. Importantly, all the bounds in the theorems presented in this paper are independent of the dimension $n$. Finally, We discuss the applications of our theorems in explaining counterintuitive phenomenon of flow-based model, deriving deep anomaly detection algorithm, and extending one-step robustness guarantee to multiple steps in safe reinforcement learning.