Miracle-dz/UniFAR-FTUs huggingface dataset:
Accelerating COVID-19 research with graph mining and transformer-based learning

In 2020, the White House released the, “Call to Action to the Tech Community on New Machine Readable COVID-19 Dataset,” wherein artificial intelligence experts are asked to collect data and develop text mining techniques that can help the science community answer high-priority scientific questions related to COVID-19. The Allen Institute for AI and collaborators announced the availability of a rapidly growing open dataset of publications, the COVID-19 Open Research Dataset (CORD-19). As the pace of research accelerates, biomedical scientists struggle to stay current. To expedite their investigations, scientists leverage hypothesis generation systems, which can automatically inspect published papers to discover novel implicit connections. We present an automated general purpose hypothesis generation systems AGATHA-C and AGATHA-GP for COVID-19 research. The systems are based on graph-mining and the transformer model. The systems are massively validated using retrospective information rediscovery and proactive analysis involving human-in-the-loop expert analysis. Both systems achieve high-quality predictions across domains (in some domains up to 0.97% ROC AUC) in fast computational time and are released to the broad scientific community to accelerate biomedical research. In addition, by performing the domain expert curated study, we show that the systems are able to discover on-going research findings such as the relationship between COVID-19 and oxytocin hormone. Reproducibility All code, details, and pre-trained models are available at https://github.com/IlyaTyagin/AGATHA-C-GP CCS CONCEPTS • Applied computing → Bioinformatics; Document management and text processing; • Computing methodologies → Learning latent representations; Neural networks; Information extraction; Semantic networks.
On the Last Iterate Convergence of Momentum Methods

SGD with Momentum (SGDM) is a widely used family of algorithms for large-scale optimization of machine learning problems. Yet, when optimizing generic convex functions, no advantage is known for any SGDM algorithm over plain SGD. Moreover, even the most recent results require changes to the SGDM algorithms, like averaging of the iterates and a projection onto a bounded domain, which are rarely used in practice. In this paper, we focus on the convergence rate of the last iterate of SGDM. For the first time, we prove that for any constant momentum factor, there exists a Lipschitz and convex function for which the last iterate of SGDM suffers from a suboptimal convergence rate of $\Omega(\frac{\ln T}{\sqrt{T}})$ after $T$ iterations. Based on this fact, we study a class of (both adaptive and non-adaptive) Follow-The-Regularized-Leader-based SGDM algorithms with increasing momentum and shrinking updates. For these algorithms, we show that the last iterate has optimal convergence $O(\frac{1}{\sqrt{T}})$ for unconstrained convex stochastic optimization problems without projections onto bounded domains nor knowledge of $T$. Further, we show a variety of results for FTRL-based SGDM when used with adaptive stepsizes. Empirical results are shown as well.
Exploring Adversarial Robustness of Deep Metric Learning

Deep Metric Learning (DML), a widely-used technique, involves learning a distance metric between pairs of samples. DML uses deep neural architectures to learn semantic embeddings of the input, where the distance between similar examples is small while dissimilar ones are far apart. Although the underlying neural networks produce good accuracy on naturally occurring samples, they are vulnerable to adversarially-perturbed samples that reduce performance. We take a first step towards training robust DML models and tackle the primary challenge of the metric losses being dependent on the samples in a mini-batch, unlike standard losses that only depend on the specific input-output pair. We analyze this dependence effect and contribute a robust optimization formulation. Using experiments on three commonly-used DML datasets, we demonstrate 5-76 fold increases in adversarial accuracy, and outperform an existing DML model that sought out to be robust.
Continuous Jets and Backflow Models for the Formation of W50/SS 433 in Magnetohydrodynamics Simulations

The formation mechanism of the W50/SS 433 complex has long been a mystery. We propose a new scenario in which the SS 433 jets themselves form the W50/SS 433 system. We carry out magnetohydrodynamics simulations of the propagation of two side jets using the public code CANS+. As found in previous jet studies, when the propagating jet is lighter than the surrounding medium, the shocked plasma flows back from the jet tip to the core. We find that the morphology of light jets is spheroidal at early times; afterward, the shell and wings are developed by the broadening spherical cocoon. The morphology depends strongly on the density ratio of the injected jet to the surrounding medium. Meanwhile, the ratio of the lengths of the two side jets depends only on the density profile of the surrounding medium. We also find that most of the jet kinetic energy is dissipated at the oblique shock formed by the interaction between the backflow and beam flow, rather than at the jet terminal shock. The position of the oblique shock is spatially consistent with the X-ray and TeV gamma-ray hotspots of W50.
Non-Hermitian topological phases and dynamical quantum phase transitions: a generic connection

The dynamical and topological properties of non-Hermitian systems have attracted great attention in recent years. In this work, we establish an intrinsic connection between two classes of intriguing phenomena—topological phases and dynamical quantum phase transitions (DQPTs)—in non-Hermitian systems. Focusing on one-dimensional models with chiral symmetry, we find DQPTs following the quench from a trivial to a non-Hermitian topological phase. Moreover, the critical momenta and critical time of the DQPTs are found to be directly related to the topological invariants of the non-Hermitian system. We further demonstrate our theory in three prototypical non-Hermitian lattice models, the lossy Kitaev chain (LKC), the LKC with next-nearest-neighbor hoppings, and the nonreciprocal Su–Schrieffer–Heeger model. Finally, we suggest a proposal to experimentally verify the found connection by a nitrogen-vacancy center in diamond.
HAWKS: Evolving Challenging Benchmark Sets for Cluster Analysis

Comprehensive benchmarking of clustering algorithms is rendered difficult by two key factors: 1) the elusiveness of a unique mathematical definition of this unsupervised learning approach and 2) dependencies between the generating models or clustering criteria adopted by some clustering algorithms and indices for internal cluster validation. Consequently, there is no consensus regarding the best practice for rigorous benchmarking, and whether this is possible at all outside the context of a given application. Here, we argue that synthetic datasets must continue to play an important role in the evaluation of clustering algorithms, but that this necessitates constructing benchmarks that appropriately cover the diverse set of properties that impact clustering algorithm performance. Through our framework, HAWKS, we demonstrate the important role evolutionary algorithms play to support flexible generation of such benchmarks, allowing simple modification and extension. We illustrate two possible uses of our framework: 1) the evolution of benchmark data consistent with a set of hand-derived properties and 2) the generation of datasets that tease out performance differences between a given pair of algorithms. Our work has implications for the design of clustering benchmarks that sufficiently challenge a broad range of algorithms, and for furthering insight into the strengths and weaknesses of specific approaches.
A S IMPLE D EEP E QUILIBRIUM M ODEL C ONVERGES TO G LOBAL O PTIMA WITH W EIGHT T YING

A deep equilibrium linear model is implicitly deﬁned through an equilibrium point of an inﬁnite sequence of computation. It avoids any explicit computation of the inﬁnite sequence by ﬁnding an equilibrium point directly via root-ﬁnding and by computing gradients via implicit differentiation. It is a simple deep equilibrium model with nonlinear activations on weight matrices. In this paper, we analyze the gradient dynamics of this simple deep equilibrium model with non-convex ob-jective functions for a general class of losses used in regression and classiﬁcation. Despite non-convexity, convergence to global optimum at a linear rate is guaranteed without any assumption on the width of the models, allowing the width to be smaller than the output dimension and the number of data points. Moreover, we prove a relation between the gradient dynamics of the simple deep equilibrium model and the dynamics of trust region Newton method of a shallow model. This mathematically proven relation along with our numerical observation suggests the importance of understanding implicit bias and a possible open problem on the topic. Our proofs deal with nonlinearity and weight tying, and differ from those in the related literature
Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition

Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. In this survey paper, a comprehensive review of the different attention models used in developing automatic speech recognition systems is provided. The paper focuses on the development and evolution of attention models for offline and streaming speech recognition within recurrent neural network- and Transformer- based architectures.
The Predictive Normalized Maximum Likelihood for Over-parameterized Linear Regression with Norm Constraint: Regret and Double Descent

A fundamental tenet of learning theory is that a trade-off exists between the complexity of a prediction rule and its ability to generalize. The double-decent phenomenon shows that modern machine learning models do not obey this paradigm: beyond the interpolation limit, the test error declines as model complexity increases. We investigate over-parameterization in linear regression using the recently proposed predictive normalized maximum likelihood (pNML) learner which is the min-max regret solution for individual data. We derive an upper bound of its regret and show that if the test sample lies mostly in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, the model generalizes despite its over-parameterized nature. We demonstrate the use of the pNML regret as a point-wise learnability measure on synthetic data and that it can successfully predict the double-decent phenomenon using the UCI dataset.
Characterisation of anomalous diﬀusion statistics powered by deep learning

Diﬀusion processes are important in several physical, chemical, biological and human phenomena. Examples include molecular encounters in reactions, cellular signalling, the foraging of animals, the spread of diseases, as well as trends in ﬁnancial markets and climate records. Deviations from Brownian diﬀusion, known as anomalous diﬀusion, can often be observed in these processes, when the growth of the mean square displacement in time is not linear. An ever-increasing number of methods has thus appeared to characterize anomalous diﬀusion trajectories based on classical statistics or machine learning approaches. Yet, characterization of anomalous diﬀusion remains challenging to date as testiﬁed by the launch of the Anomalous Diﬀusion (AnDi) Challenge in March 2020 to assess and compare new and pre-existing methods on three diﬀerent aspects of the problem: the inference of the anomalous diﬀusion exponent, the classiﬁcation of the diﬀusion model, and the segmentation of trajectories. Here, we introduce a novel method which combines feature engineering based on classical statistics with supervised deep learning to eﬃciently identify the underlying anomalous diﬀusion model with high accuracy (up to 91% ) and infer its exponent with a small mean absolute error (down to 0 . 12 ) in single 1D, 2D and 3D trajectories corrupted by localization noise.
Deep Co-Attention Network for Multi-View Subspace Learning

Many real-world applications involve data from multiple modalities and thus exhibit the view heterogeneity. For example, user modeling on social media might leverage both the topology of the underlying social network and the content of the users’ posts; in the medical domain, multiple views could be X-ray images taken at different poses. To date, various techniques have been proposed to achieve promising results, such as canonical correlation analysis based methods, etc. In the meanwhile, it is critical for decision-makers to be able to understand the prediction results from these methods. For example, given the diagnostic result that a model provided based on the X-ray images of a patient at different poses, the doctor needs to know why the model made such a prediction. However, state-of-the-art techniques usually suffer from the inability to utilize the complementary information of each view and to explain the predictions in an interpretable manner. To address these issues, in this paper, we propose a deep co-attention network for multi-view subspace learning, which aims to extract both the common information and the complementary information in an adversarial setting and provide robust interpretations behind the prediction to the end-users via the co-attention mechanism. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation by incorporating the classifier into our model. This improves the quality of latent representation and accelerates the convergence speed. Finally, we develop an efficient iterative algorithm to find the optimal encoders and discriminator, which are evaluated extensively on synthetic and real-world data sets. We also conduct a case study to demonstrate how the proposed method robustly interprets the predictions on an image data set.
Optimal Streaming Algorithms for Graph Matching

We present parameterized streaming algorithms for the graph matching problem in both the dynamic and the insert-only models. For the dynamic streaming model, we present a one-pass algorithm that, with high probability, computes a maximum-weight $k$-matching of a weighted graph in $\tilde{O}(Wk^2)$ space and that has $\tilde{O}(1)$ update time, where $W$ is the number of distinct edge weights and the notation $\tilde{O}()$ hides a poly-logarithmic factor in the input size. For the insert-only streaming model, we present a one-pass algorithm that runs in $O(k^2)$ space and has $O(1)$ update time, and that, with high probability, computes a maximum-weight $k$-matching of a weighted graph. The space complexity and the update-time complexity achieved by our algorithms for unweighted $k$-matching in the dynamic model and for weighted $k$-matching in the insert-only model are optimal. A notable contribution of this paper is that the presented algorithms {\it do not} rely on the apriori knowledge/promise that the cardinality of \emph{every} maximum-weight matching of the input graph is upper bounded by the parameter $k$. This promise has been a critical condition in previous works, and lifting it required the development of new tools and techniques.
The cost-benefit ratio of group-living is thought to vary with group size: individuals in “optimally-sized”

: 25 The cost-benefit ratio of group-living is thought to vary with group size: individuals in “optimally-sized” 26 groups should have higher fitness than individuals in groups that are either too large or too small. 27 However, the relationship between group size and individual fitness has been difficult to establish for 28 long-lived species where the number of groups studied is typically quite low. Here we present evidence 29 for optimal group size that maximizes female fitness in a population of geladas ( Theropithecus gelada ). 30 Drawing on 14 years of demographic data, we found that females in small groups showed the highest 31 mortality risk, while females in mid-sized groups exhibited the highest reproductive performance. This 32 group-size effect on female reproductive performance was largely explained by variation in infant 33 mortality (and, in particular, by infanticide from immigrant males) but not by variation in reproductive 34 rates. Taken together, our results show that females in mid-sized groups appear to attain optimal fitness 35 due to the combination of social pressures and, potentially, predation. Our findings provide insight into
Non‐destructive Raman study of the glazing technique in lustre potteries and faience (9–14th centuries): silver ions, nanoclusters, microstructure and processing

The oldest known nanotechnology dates back to the fabrication of the first lustre potteries. A lustre is a thin film formed just below the surface of medieval Islamic glazed potteries which contains silver and/or copper in the metallic and ionic form. Raman studies of the lustre films of different ceramics excavated from Fustât (near Cairo, Egypt, 11–12th century) or from the Silk Road (Termez, 13–14th centuries) showed that they associate many layers of different compositions (with or without cassiterite). Energy-dispersive spectroscopic analysis shows that all studied glazes are Ca- (and K)-rich, nearly free of Al silicates, with some addition of lead. Comparison is made with a copy of three-colour Tang ceramics made in Bassorah or Baghdad, in the 9th century, which is among the first known ‘faiences’, i.e. ceramics enamelled with an Sn-containing glaze. Surprisingly, Sn is not present in the form of a cassiterite (SnO2) precipitate but as a Ca,K-rich salt. Composition analysis and Raman spectra show that all glazes have been processed with similar technology. The distribution of elemental Ag and Cu is very heterogeneous in the lustre decor. The main Raman signature (50–100 cm−1 peaks) of the lustre film is assigned to Ag+ ions. The additional low-wavenumber features could be due to the Ag0 [or (Agn)m+] nanocluster modes. It is clear that the lustre colour arises from the combination of iridescence (diffraction) and absorption/diffusion. Raman criteria are proposed for a sample classification as a function of processing (cassiterite content, processing temperature). The glazing technique is discussed on the basis of experimental evidence and ancient potters' reports. Exothermic burning of acetate residus is proposed as the key step for the preparation of polychrome lustre. Copyright © 2004 John Wiley & Sons, Ltd.
Fractional moment-preserving initialization schemes for training deep neural networks

A traditional approach to initialization in deep neural networks (DNNs) is to sample the network weights randomly for preserving the second moment of layer outputs. On the other hand, recent results show that training with SGD can result in heavy-tailedness in the distribution of the network weights with a potentially inﬁnite variance. This suggests that the traditional approach to initialization may be restrictive as SGD updates do not necessarily preserve the ﬁniteness of the variance of layer outputs. Motivated by this, we develop initialization schemes for fully connected feed-forward networks that can provably preserve any given moment of order s ∈ (0 , 2] over the layers for a class of activations including ReLU, Leaky ReLU, Randomized Leaky ReLU and linear activations. These generalized schemes recover traditional initialization schemes in the limit s → 2 and serve as part of a principled theory for initialization. For all these schemes, we show that the network output admits a ﬁnite almost sure limit as the number of layers grows, and the limit is heavy-tailed in some settings. We also prove that the logarithm of the norm of the network outputs, if properly scaled, will converge to a Gaussian distribution with an explicit mean and variance we can compute depending on the activation used, the value of s chosen and the network width, where log-normality serves as a further justiﬁcation of why the norm of the network output can be heavy-tailed in DNNs.
Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

Abstract:Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance. It is widely agreed upon that covariate balance can serve as evidence that a matched dataset approximates a randomized experiment, but what kind of experiment does a matched dataset approximate? In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization. Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design—if any—is most appropriate for a matched dataset. After researchers determine a plausible design, we recommend a randomization based approach for analyzing the matched data, which can incorporate any design and treatment effect estimator. Through simulation, we find that our test can frequently detect violations of randomized assignment that harm inferential results. Furthermore, through simulation and a real application in political science, we find that matched datasets with high levels of covariate balance tend to approximate balance-constrained designs like rerandomization, and analyzing them as such can lead to precise causal analyses. However, assuming a precise design should be proceeded with caution, because it can harm inferential results if there are still substantial biases due to remaining imbalances after matching. Our approach is implemented in the randChecks R package, available on CRAN.
Synthesis and swelling properties of the copolymer of acrylamide with anionic monomers

A series of novel superabsorbent copolymers of acrylamide (AM), sodium allylsulfonate (SAS), sodium acrylate (AA), and N,N′-methylenebisacrylamide (BisA) were synthesized using potassium persulfate (KPS)/N,N,N′,N′-tertramethylethylenediamine (TMEDA) as the initiator. The influences of synthetic variables (monomer concentration, temperature, initiator concentration, and pH) on the polymerization conversion (y°) and swelling properties were studied in detail. These gels should have potential application as water super-absorbents. © 1996 John Wiley & Sons, Inc.