Miracle-dz/UniFAR-FTUs huggingface dataset:
Information-Theoretic Probing with Minimum Description Length

To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, they do not substantially favour pretrained representations over randomly initialized ones. Analogously, their accuracy can be similar when probing for genuine linguistic labels and probing for random synthetic tasks. To see reasonable differences in accuracy with respect to these random baselines, previous work had to constrain either the amount of probe training data or its model size. Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL). With MDL probing, training a probe to predict labels is recast as teaching it to effectively transmit the data. Therefore, the measure of interest changes from probe accuracy to the description length of labels given representations. In addition to probe quality, the description length evaluates "the amount of effort" needed to achieve the quality. This amount of effort characterizes either (i) size of a probing model, or (ii) the amount of data needed to achieve the high quality. We consider two methods for estimating MDL which can be easily implemented on top of the standard probing pipelines: variational coding and online coding. We show that these methods agree in results and are more informative and stable than the standard probes.
Training for Speech Recognition on Coprocessors

Automatic Speech Recognition (ASR) has increased in popularity in recent years. The evolution of processor and storage technologies has enabled more advanced ASR mechanisms, fueling the development of virtual assistants such as Amazon Alexa, Apple Siri, Microsoft Cortana, and Google Home. The interest in such assistants, in turn, has amplified the novel developments in ASR research. However, despite this popularity, there has not been a detailed training efficiency analysis of modern ASR systems. This mainly stems from: the proprietary nature of many modern applications that depend on ASR, like the ones listed above; the relatively expensive co-processor hardware that is used to accelerate ASR by big vendors to enable such applications; and the absence of well-established benchmarks. The goal of this paper is to address the latter two of these challenges. The paper first describes an ASR model, based on a deep neural network inspired by recent work in this domain, and our experiences building it. Then we evaluate this model on three CPU-GPU co-processor platforms that represent different budget categories. Our results demonstrate that utilizing hardware acceleration yields good results even without high-end equipment. While the most expensive platform (10X price of the least expensive one) converges to the initial accuracy target 10-30% and 60-70% faster than the other two, the differences among the platforms almost disappear at slightly higher accuracy targets. In addition, our results further highlight both the difficulty of evaluating ASR systems due to the complex, long, and resource intensive nature of the model training in this domain, and the importance of establishing benchmarks for ASR.
Chirality, the missing key to the definition of the connection and curvature of a Lie-Kac super-algebra

Curiously, a natural generalization of a connection, or Yang-Mills field, to the case of a Lie super-algebra, for example SU(m/n), just in terms of ordinary complex functions and differentials, is lacking. All previous attempts require super-space coordinates or elements of non commutative geometry. Using the chirality chi of the super-algebra which defines its super-trace: STr(.) = Tr (chi .), we construct a universal solution to this problem: D = chi (d + A) + Phi, where A is the standard Lie-subalgebra connection 1-form and Phi a scalar field valued in the odd module. Despite the fact that Phi is a scalar, Phi anti-commutes with chi A because chi anti-commutes with the odd generators hidden in Phi. Hence the curvature is a super-algebra valued tensor which respects the Bianchi identity and correctly defines a chiral parallel transport on a generic Lie super-algebra fiber bundle.
Cycle Text-To-Image GAN with BERT

We explore novel approaches to the task of image generation from their respective captions, building on state-of-the-art GAN architectures. Particularly, we baseline our models with the Attention-based GANs that learn attention mappings from words to image features. To better capture the features of the descriptions, we then built a novel cyclic design that learns an inverse function to maps the image back to original caption. Additionally, we incorporated recently developed BERT pretrained word embeddings as our initial text featurizer and observe a noticeable improvement in qualitative and quantitative performance compared to the Attention GAN baseline.
Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)

One of the challenges in machine learning research is to ensure that presented and published results are sound and reliable. Reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In 2019, the Neural Information Processing Systems (NeurIPS) conference, the premier international conference for research in machine learning, introduced a reproducibility program, designed to improve the standards across the community for how we conduct, communicate, and evaluate machine learning research. The program contained three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process. In this paper, we describe each of these components, how it was deployed, as well as what we were able to learn from this initiative.
Towards Discriminability and Diversity: Batch Nuclear-Norm Maximization Under Label Insufficient Situations

The learning of the deep networks largely relies on the data with human-annotated labels. In some label insufficient situations, the performance degrades on the decision boundary with high data density. A common solution is to directly minimize the Shannon Entropy, but the side effect caused by entropy minimization, \it i.e., reduction of the prediction diversity, is mostly ignored. To address this issue, we reinvestigate the structure of classification output matrix of a randomly selected data batch. We find by theoretical analysis that the prediction discriminability and diversity could be separately measured by the Frobenius-norm and rank of the batch output matrix. Besides, the nuclear-norm is an upperbound of the Frobenius-norm, and a convex approximation of the matrix rank. Accordingly, to improve both discriminability and diversity, we propose Batch Nuclear-norm Maximization (BNM) on the output matrix. BNM could boost the learning under typical label insufficient learning scenarios, such as semi-supervised learning, domain adaptation and open domain recognition. On these tasks, extensive experimental results show that BNM outperforms competitors and works well with existing well-known methods. The code is available at https://github.com/cuishuhao/BNM
On the Optimization Dynamics of Wide Hypernetworks

Recent results in the theoretical study of deep learning have shown that the optimization dynamics of wide neural networks exhibit a surprisingly simple behaviour. In this work, we study the optimization dynamics of hypernetworks, which are architectures in which a learned meta-network produces the weights of a task-specific primary network. Hypernetworks have been demonstrated repeatedly to obtain state of the art results. However, their theoretical understanding is still lacking. As can be expected, the optimization process of multiplicative models is much more complicated than optimizing standard ReLU networks. It is shown that for an infinitely wide neural network with a gating layer the cost function cannot be accurately approximated by it first order Taylor approximation. Specifically, for a fixed sized primary network of depth H, the first H terms of the Taylor approximation of the cost function are non-zero, even when the meta-network is infinitely wide. However, for an infinitely wide meta and primary networks, the learning dynamics is determined by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters and the kernel of this process is given by the Hadamard product of the kernels induced by the meta and primary networks. As part of our study, we partially solve an open problem suggested by Dyer & Gur-Ari (2020) and show that the convergence rate of the r order term of the Taylor expansion of the cost function, along the optimization trajectories of SGD is n^{1-r}, where n is the width of the learned neural network, improving upon the bound suggested by the conjecture of Dyer & Gur-Ari, while matching their empirical observations. This result extends the recent n^{-1} second order upper bound of Hanin & Nica (2020).
COVID-19 Screening on Chest X-ray Images Using Deep Learning based Anomaly Detection

Coronaviruses are important human and animal pathogens. To date the novel COVID-19 coronavirus is rapidly spreading worldwide and subsequently threatening health of billions of humans. Clinical studies have shown that most COVID-19 patients suffer from the lung infection. Although chest CT has been shown to be an effective imaging technique for lung-related disease diagnosis, chest Xray is more widely available due to its faster imaging time and considerably lower cost than CT. Deep learning, one of the most successful AI techniques, is an effective means to assist radiologists to analyze the vast amount of chest X-ray images, which can be critical for efficient and reliable COVID-19 screening. In this work, we aim to develop a new deep anomaly detection model for fast, reliable screening. To evaluate the model performance, we have collected 100 chest X-ray images of 70 patients confirmed with COVID-19 from the Github repository. To facilitate deep learning, more data are needed. Thus, we have also collected 1431 additional chest X-ray images confirmed as other pneumonia of 1008 patients from the public ChestX-ray14 dataset. Our initial experimental results show that the model developed here can reliably detect 96.00% COVID-19 cases (sensitivity being 96.00%) and 70.65% non-COVID-19 cases (specificity being 70.65%) when evaluated on 1531 Xray images with two splits of the dataset.
Sobolev inequalities with jointly concave weights on convex cones

Using optimal mass transport arguments, we prove weighted Sobolev inequalities of the form WSI ∫E|u(x)|qω(x)dx1/q⩽K0∫E|∇u(x)|pσ(x)dx1/p,u∈C0∞(Rn),where p⩾1 and q>0 is the corresponding Sobolev critical exponent. Here E⊆Rn is an open convex cone, and ω,σ:E→(0,∞) are two homogeneous weights verifying a general concavity‐type structural condition. The constant K0=K0(n,p,q,ω,σ)>0 is given by an explicit formula. Under mild regularity assumptions on the weights, we also prove that K0 is optimal in (WSI) if and only if ω and σ are equal up to a multiplicative factor. Several previously known results, including the cases for monomials and radial weights, are covered by our statement. Further examples and applications to partial differential equations are also provided.
Analysis and Distributed Control of Periodic Epidemic Processes

This article studies epidemic processes over discrete-time periodic time-varying networks. We focus on the susceptible-infected-susceptible (SIS) model that accounts for a (possibly) mutating virus. We say that an agent is in the disease-free state if it is not infected by the virus. Our objective is to devise a control strategy which ensures that all agents in a network exponentially (respectively asymptotically) converge to the disease-free equilibrium (DFE). Toward this end, we first provide 1) sufficient conditions for exponential (respectively, asymptotic) convergence to the DFE and 2) a necessary and sufficient condition for asymptotic convergence to the DFE. The sufficient condition for global exponential stability (GES) [respectively global asymptotic stability (GAS)] of the DFE is in terms of the joint spectral radius of a set of suitably defined matrices, whereas the necessary and sufficient condition for GAS of the DFE involves the spectral radius of an appropriately defined product of matrices. Subsequently, we leverage the stability results in order to design a distributed control strategy for eradicating the epidemic.
Sixteen overlooked open clusters in the fourth Galactic quadrant

Aims. This paper has two main objectives: (1) To determine the intrinsic properties of 16 faint and mostly unstudied open clusters in the poorly known sector of the Galaxy at 270° −300° to probe the Milky Way structure in future investigations. (2) To address previously reported systematics in Gaia DR2 parallaxes by comparing the cluster distances derived from photometry with those derived from parallaxes. Methods. Deep UBVI photometry of 16 open clusters was carried out. Observations were reduced and analyzed in an automatic way using the ASteCA package to obtain individual distances, reddening, masses, ages, and metallicities. Photometric distances were compared to those obtained from a Bayesian analysis of Gaia DR2 parallaxes. Results. Ten out of the sixteen clusters are true or highly probable open clusters. Two of them are quite young and follow the trace of the Carina Arm and the already detected warp. The remaining clusters are placed in the interarm zone between the Perseus and Carina Arms, as expected for older objects. We found that the cluster van den Berg-Hagen 85 is 7.5 × 109 yr old, which means that it is one of the oldest open clusters detected in our Galaxy so far. The relationship of these ten clusters with the Galaxy structure in the solar neighborhood is discussed. The comparison of distances from photometry and parallaxes data in turn reveals a variable level of disagreement. Conclusions. Various zero-point corrections for Gaia DR2 parallax data recently reported were considered for a comparison between photometry- and parallax-based distances. The results tend to improve with some of these corrections. Photometric distance analysis suggests an average correction of ∼+0.026 mas (to be added to the parallaxes). The correction may have a more intricate dependence on distance, but addressing this level of detail will require a larger cluster sample.
Trinity of the Eisenstein series

We study the weight 2 parabolic/elliptic/hyperbolic Eisenstein series, which gives a harmonic/polar harmonic/locally harmonic Maass form, simultaneously. Furthermore, by means of the hyperbolic Eisenstein series, we can redefine the hyperbolic Rademacher symbol introduced by Duke-Imamoglu-Toth. The symbol is expressed explicitly in terms of continued fraction coefficients of the corresponding two real quadratic irrationals.
How runaway stars boost galactic outflows

Roughly 10 per cent of OB stars are kicked out of their natal clusters before ending their life as supernovae. These so-called runaway stars can travel hundreds of parsecs into the low-density interstellar medium, where momentum and energy from stellar feedback is efficiently deposited. In this work, we explore how this mechanism affects large-scale properties of the galaxy, such as outflows. To do so we use a new model that treats OB stars and their associated feedback processes on a star-by-star basis. With this model, we compare two hydrodynamical simulations of Milky Way-like galaxies, one where we include runaways, and one where we ignore them. Including runaway stars leads to twice as many supernovae explosions in regions with gas densities ranging from $10^{-5}\, \mathrm{\,cm^{-3}}$ to $10^{-3}\, \mathrm{\,cm^{-3}}$. This results in more efficient heating of the inter-arm regions, and drives strong galactic winds with mass loading factors boosted by up to one order of magnitude. These outflows produce a more massive and extended multiphase circumgalactic medium, as well as a population of dense clouds in the halo. Conversely, since less energy and momentum is released in the dense star-forming regions, the cold phase of the interstellar medium is less disturbed by feedback effects.
Multiwavelength morphological study of active galaxies

Abstract Studying the morphology of a large sample of active galaxies at different wavelengths and comparing it with active galactic nuclei (AGN) properties, such as black hole mass (MBH) and Eddington ratio (λEdd), can help us in understanding better the connection between AGN and their host galaxies and the role of nuclear activity in galaxy formation and evolution. By using the BAT-SWIFT hard X-ray public data and by extracting those parameters measured for AGN and by using other public catalogues for parameters such as stellar mass (M*), star formation rate (SFR), bolometric luminosity (Lbol), etc., we studied the multiwavelength morphological properties of host galaxies of ultra-hard X-ray detected AGN and their correlation with other AGN properties. We found that ultra hard X-ray detected AGN can be hosted by all morphological types, but in larger fractions (42%) they seem to be hosted by spirals in optical, to be quiet in radio, and to have compact morphologies in X-rays. When comparing morphologies with other galaxy properties, we found that ultra hard X-ray detected AGN follow previously obtained relations. On the SFR vs. stellar mass diagram, we found that although the majority of sources are located below the main sequence (MS) of star formation (SF), still non-negligible number of sources, with diverse morphologies, is located on and/or above the MS, suggesting that AGN feedback might have more complex influence on the SF in galaxies than simply quenching it, as it was suggested in some of previous studies.
Direct observation of an incommensurate two-dimensional checkerboard charge density wave in the superconductor Ta4Pd3Te16

We report the observation of a highly unusual incommensurate two-dimensional checkerboard charge density wave (CDW) in the superconductor Ta4Pd3Te16, using synchrotron X-ray diffraction. We observe two CDW wavevectors, related by a crystal symmetry, developing below 16 K at ambient pressure. The wavevectors, which remain incommensurate at all observed temperatures, lie neither within the quasi-one-dimensional (Q1D) chains nor the Q2D planes in which the chains reside in this monoclinic crystal structure. The bulk CDW wavevectors are shown to reconcile previous scanning tunneling microscopy and transport studies. We also conducted de Haas-van Alphen (dHvA) quantum oscillation measurements and found no evidence of Fermi surface reconstruction through the CDW transition, suggesting a very subtle gap opening. We determine the temperature-pressure (T-P) phase diagram, and find the superconductivity (SC) dome to be centered at the pressure where the CDW instability vanishes, suggesting that the SC in Ta4Pd3Te16 could have its origin in a CDW quantum critical point, reminiscent of the observations in many unconventional superconductors. With a thorough comparison to other CDW materials, we conclude that Ta4Pd3Te16 is a unique CDW system that features one of the lowest CDW energy scales, and a mixed character of dimensions: Q1D considering its needle-like shape, Q2D as it features checkerboard wavevectors, and 3D because of CDW projections along all three axes.
Detection of the Strongest Magnetic Field in a Sunspot Light Bridge

Traditionally, the strongest magnetic fields on the Sun have been measured in sunspot umbrae. More recently, however, much stronger fields have been measured at the ends of penumbral filaments carrying the Evershed and counter-Evershed flows. Superstrong fields have also been reported within a light bridge separating two umbrae of opposite polarities. We aim to accurately determine the strengths of the strongest fields in a light bridge using an advanced inversion technique and to investigate their detailed structure. We analyze observations from the spectropolarimeter on board the Hinode spacecraft of the active region AR 11967. The thermodynamic and magnetic configurations are obtained by inverting the Stokes profiles using an inversion scheme that allows multiple height nodes. Both the traditional 1D inversion technique and the so-called 2D coupled inversions, which take into account the point-spread function of the Hinode telescope, are used. We find a compact structure with an area of 32.7 arcsec2 within a bipolar light bridge with field strengths exceeding 5 kG, confirming the strong fields in this light bridge reported in the literature. Two regions associated with downflows of ∼5 km s−1 harbor field strengths larger than 6.5 kG, covering a total area of 2.97 arcsec2. The maximum field strength found is 8.2 kG, which is the largest ever observed field in a bipolar light bridge up to now.
Measuring the trilinear Higgs boson self-coupling at the 100 TeV hadron collider via multivariate analysis

We perform a multivariate analysis of Higgs-pair production via the decay channel $HH \to b\bar b \gamma\gamma$ at the future 100 TeV $pp$ collider to determine the trilinear Higgs self--coupling (THSC) $\lambda_{3H}$, which takes the value of 1 in the standard model. We consider all known background processes. For the signal we adopt the most recent event generator of {\tt POWHEG-BOX-V2} to exploit the NLO distributions for Toolkit for Multivariate Data Analysis (TMVA). Through the technique of Boosted Decision Tree (BDT) analysis trained for $\lambda_{3H}=1$, compared to the the conventional cut-and-count approach, the signal-to-background ratio improves tremendously from about $1/10$ to $1$ and the significance can reach up to $20.5$ with a luminosity of 3 ab$^{-1}$ without including systematic uncertainties. In addition, by implementing a likelihood fitting of the signal-plus-background $M_{\gamma\gamma b b}$ distribution with optimized bin sizes, it is possible to determine the THSC with the precision of 7.5\% at 68\% CL even at the early stage of 100 TeV hadron collider with 3 ab$^{-1}$.
On Classical Solutions Of Time-Dependent Fractional Mean Field Game Systems

In this paper we study parabolic Mean Field Game systems with nonlocal/fractional diffusion. Such models come from games where the noise is non-Gaussian and the resulting controlled diffusion process anomalous. Here the noise is modeled by pure jump Levy processes that are $\sigma$-stable like. The corresponding diffusion operators include the fractional Laplacians $(-\Delta)^{\frac{\sigma}2}$, the generators of tempered stable and nonsymmetric processes widely used in Finance, and sums of such operators. Our main result is existence and uniqueness of classical solutions of Mean Field Game systems for $\sigma\in(1,2)$. This corresponds to the subcritical or strong diffusion case where the equations are nondegenerate. We consider both local and nonlocal couplings. The proof we give is an extension of the fixed point argument introduced by P.-L. Lions. The new ingredients are fractional heat kernal estimates, regularity results for fractional Bellman and Fokker-Planck equations, and results on (very) weak solutions of fractional Fokker-Planck equations.