Miracle-dz/UniFAR-FTUs huggingface dataset:
Bidirectional Attention as a Mixture of Continuous Word Experts

Bidirectional attention $\unicode{x2013}$ composed of self-attention with positional encodings and the masked language model (MLM) objective $\unicode{x2013}$ has emerged as a key component of modern large language models (LLMs). Despite its empirical success, few studies have examined its statistical underpinnings: What statistical model is bidirectional attention implicitly fitting? What sets it apart from its non-attention predecessors? We explore these questions in this paper. The key observation is that fitting a single-layer single-head bidirectional attention, upon reparameterization, is equivalent to fitting a continuous bag of words (CBOW) model with mixture-of-experts (MoE) weights. Further, bidirectional attention with multiple heads and multiple layers is equivalent to stacked MoEs and a mixture of MoEs, respectively. This statistical viewpoint reveals the distinct use of MoE in bidirectional attention, which aligns with its practical effectiveness in handling heterogeneous data. It also suggests an immediate extension to categorical tabular data, if we view each word location in a sentence as a tabular feature. Across empirical studies, we find that this extension outperforms existing tabular extensions of transformers in out-of-distribution (OOD) generalization. Finally, this statistical perspective of bidirectional attention enables us to theoretically characterize when linear word analogies are present in its word embeddings. These analyses show that bidirectional attention can require much stronger assumptions to exhibit linear word analogies than its non-attention predecessors.
Lifting Elementary Abelian Covers of Curves

Given a Galois cover of curves $f$ over a field of characteristic $p$, the lifting problem asks whether there exists a Galois cover over a complete mixed characteristic discrete valuation ring whose reduction is $f$. In this paper, we consider the case where the Galois groups are elementary abelian $p$-groups. We prove a combinatorial criterion for lifting an elementary abelian $p$-cover, dependent on the branch loci of lifts of its $p$-cyclic subcovers. We also study how branch points of a lift coalesce on the special fiber. Finally, we analyze lifts for several families of $(\mathbb{Z}/2)^3$-covers of various conductor types, both with equidistant branch locus geometry and non-equidistant branch locus geometry.
Dark Matter in Fractional Gravity II: Tests in Galaxy Clusters

Recently, in Benetti et al. (Astrophys. J. 2023, 949, 65), we suggested that the dark matter (DM) component in galaxies may originate fractional gravity. In such a framework, the DM component exists, but the gravitational potential associated to its density distribution is determined by a modified Poisson equation including fractional derivatives (i.e., derivatives of noninteger type), which are meant to describe nonlocal effects; as such, this scenario is different from theories where baryonic matter emulates DM-like effects via modifications of gravity (e.g., MONDian frameworks). In Benetti et al., we showed that fractional gravity worked very well for reproducing the kinematics of disk-dominated galaxies, especially dwarfs; there is also preliminary evidence that the strength of fractional effects tends to weaken toward more massive systems. Here, we aim to test fractional gravity in galaxy clusters, with a twofold aim: (i) perform an independent sanity check that it can accurately describe such large and massive structures; (ii) derive a clear-cut trend for its strength in systems with different DM masses. To this purpose, we forward model the density and pressure distributions of the intracluster medium (ICM), working out the hydrostatic equilibrium equation in fractional gravity. Then, we perform a Bayesian analysis of the X-COP galaxy cluster sample and infer constraints on the fractional gravity parameters, for individual clusters as well as stacked clusters. We find that fractional gravity performs remarkably well in modeling the ICM profiles for the X-COP sample. We also check that the DM concentration vs. mass relation is still consistent with the expectations of N-body simulations in the standard cosmological scenario. Finally, we confirm the weakening of the fractional gravity effects toward more massive systems and derive the overall scaling of the fractional gravity parameters from dwarf galaxies to massive clusters, spanning six orders of magnitude in DM mass. Such an overall trend implies that fractional gravity can substantially alleviate the small-scale issues of the standard DM paradigm, while remaining successful on large cosmological scales.
Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generation

Significant progress has recently been made in creative applications of large pre-trained models for downstream tasks in 3D vision, such as text-to-shape generation. This motivates our investigation of how these pre-trained models can be used effectively to generate 3D shapes from sketches, which has largely remained an open challenge due to the limited sketch-shape paired datasets and the varying level of abstraction in the sketches. We discover that conditioning a 3D generative model on the features (obtained from a frozen large pre-trained vision model) of synthetic renderings during training enables us to effectively generate 3D shapes from sketches at inference time. This suggests that the large pre-trained vision model features carry semantic signals that are resilient to domain shifts, i.e., allowing us to use only RGB renderings, but generalizing to sketches at inference time. We conduct a comprehensive set of experiments investigating different design factors and demonstrate the effectiveness of our straightforward approach for generation of multiple 3D shapes per each input sketch regardless of their level of abstraction without requiring any paired datasets during training.
Derandomizing Codes for the Binary Adversarial Wiretap Channel of Type II

We revisit the binary adversarial wiretap channel (AWTC) of type II in which an active adversary can read a fraction r and flip a fraction p of codeword bits. The semantic-secrecy capacity of the AWTC II is partially known, where the best-known lower bound is non-constructive, proven via a random coding argument that uses a large number (that is exponential in blocklength n) of random bits to seed the random code. In this paper, we establish a new derandomization result in which we match the best-known lower bound of 1 − H2(p) − r, where H2(•) is the binary entropy function, via a random code that uses a small seed of only O(n2) bits. Our random code construction is a novel application of pseudolinear codes – a class of non-linear codes that have k-wise independent codewords when chosen at random where k is a design parameter. As the key technical tool in our analysis, we provide a soft-covering lemma in the flavor of Goldfeld, Cuff and Permuter (Trans. Inf. Theory 2016) that holds for random codes with k-wise independent codewords.
Extreme N-emitters at high redshift: Possible signatures of supermassive stars and globular cluster or black hole formation in action

[Abridged] Using the JWST/NIRSpec observations from CEERS we found an extreme N-emitter, CEERS-1019 at z=8.6782 showing intense NIV and NIII emission. From the observed rest-UV and optical lines we conclude that it is compatible with photoionization from stars and we determine accurate abundances for C, N, O, and Ne, relative to H, finding a highly supersolar ratio log(N/O) = -0.18+/-0.11, and normal log(C/O) = -0.75+/-0.11 and log(Ne/O) = -0.63+/-0.07, for its low metallicity, 12+log(O/H)= 7.70+/-0.18. We also analyze other N-emitters from the literature. All show strongly enhanced N/O ratios and two of them normal C/O. Massive star ejecta from WR stars are needed to explain the galaxies with enhanced C/O (Lynx arc and Mrk 996). On the other hand, supermassive stars (>1000 Msun, SMS) in the ``conveyer-belt model'' put forward to explain globular clusters (GCs), predict a high N/O and small changes in C/O, compatible with CEERS-1019, the Sunburst cluster, SMACS2031, and GN-z11. Based on the chemical abundances, possible enrichment scenarios, compactness, and high ISM density, we suggest that CEERS-1019, SMACS2031, and the Sunburst cluster could contain proto-GCs. Finally, we propose that some N-emitters enriched by SMS could also have formed intermediate-mass black holes, and we suggest that this might be the case for GN-z11. Our observations and analysis reinforce the suggested link between some N-emitters and proto-GC formation, which is supported both by empirical evidence and quantitative models. Furthermore, the observations provide possible evidence for the presence of supermassive stars in the early Universe (z>8) and at z~2-3. Our analysis also suggests that the origin and nature of the N-emitters is diverse, including also objects like GN-z11 which possibly host an AGN.
Learning Space-Time Continuous Neural PDEs from Partially Observed States

We introduce a novel grid-independent model for learning partial differential equations (PDEs) from noisy and partial observations on irregular spatiotemporal grids. We propose a space-time continuous latent neural PDE model with an efficient probabilistic framework and a novel encoder design for improved data efficiency and grid independence. The latent state dynamics are governed by a PDE model that combines the collocation method and the method of lines. We employ amortized variational inference for approximate posterior estimation and utilize a multiple shooting technique for enhanced training speed and stability. Our model demonstrates state-of-the-art performance on complex synthetic and real-world datasets, overcoming limitations of previous approaches and effectively handling partially-observed data. The proposed model outperforms recent methods, showing its potential to advance data-driven PDE modeling and enabling robust, grid-independent modeling of complex partially-observed dynamic processes.
The modular representation theory of monoids

This paper develops the fundamentals of modular representation theory for finite monoids, introducing the decomposition matrix and exploring its connection to Brauer characters. We define modular characteristic and explain how the representation theory in nonmodular positive characteristic behaves like the characteristic zero theory by showing that one can lift from nonmodular characteristic $p$ all simple and projective indecomposable modules, as well as a quiver presentation of the basic algebra. As an application of the theory developed, we give a new proof of Glover's theorem that the monoid of $2\times 2$-matrices over $\mathbb F_p$ has infinite representation type over fields of characteristic $p$. We also investigate the relationship between nonsingularity of the Cartan matrix of a monoid algebra in characterstic zero and in positive characteristic. We show that for von Neumann regular monoids the Cartan matrix is always nonsingular and we show that if a monoid has aperiodic left (or right) stabilizers, then nonsingularity in characteristic zero implies nonsingularity in positive characteristic. Florian Eisele has recently shown that the Cartan matrix of a monoid algebra can be nonsingular in characteristic zero and singular in positive characteristic, disproving a conjecture of the author in an earlier version of this paper. A new conjecture is proposed, unifying the cases of regular monoids and monoids with aperiodic stabilizers.
On the density and multiplicity of solutions to the fractional Nirenberg problem

This paper is devoted to establishing some results on the density and multiplicity of solutions to the fractional Nirenberg problem which is equivalent to studying the conformally invariant equation $P_\sigma(v)=K v^{\frac{n+2\sigma}{n-2\sigma}}$ on the standard unit sphere $(\mathbb{S}^n,g_0)$ with $\sigma\in (0,1)$ and $n\geq 2$, where $P_\sigma$ is the intertwining operator of order $2\sigma$ and $K$ is the prescribed curvature function. More specifically, by using the variational gluing method, refined analysis of bubbling behavior, extension formula, as well as the blow up analysis arguments, we obtain the existence of infinitely many multi-bump solutions. In particular, we show the smooth curvature functions of metrics conformal to $g_0$ are dense in the $C^{0}$ topology. Moreover, the related fractional Laplacian equations $(-\Delta)^{\sigma} u=K(x) u^{\frac{n+2\sigma}{n-2\sigma}}$ in $\mathbb{R}^n$, with $K(x)$ being asymptotically periodic in one of the variables, are also studied and infinitely many solutions are obtained under natural flatness assumptions.
Provable Robust Saliency-based Explanations

Robust explanations of machine learning models are critical to establishing human trust in the models. The top-$k$ intersection is widely used to evaluate the robustness of explanations. However, most existing attacking and defense strategies are based on $\ell_p$ norms, thus creating a mismatch between the evaluation and optimization objectives. To this end, we define explanation thickness for measuring top-$k$ salient features ranking stability, and design the \textit{R2ET} algorithm based on a novel tractable surrogate to maximize the thickness and stabilize the top salient features efficiently. Theoretically, we prove a connection between R2ET and adversarial training; using a novel multi-objective optimization formulation and a generalization error bound, we further prove that the surrogate objective can improve both the numerical and statistical stability of the explanations. Experiments with a wide spectrum of network architectures and data modalities demonstrate that R2ET attains higher explanation robustness under stealthy attacks while retaining model accuracy.
Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training

Biomedical summarization requires large datasets to train for text generation. We show that while transfer learning offers a viable option for addressing this challenge, an in-domain pre-training does not always offer advantages in a BioASQ summarization task. We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning in the context of a BioASQ summarization task, leading to a novel three-step fine-tuning approach that works with only a thousand in-domain examples. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.
Risk for Chronic Kidney Disease Progression After Acute Kidney Injury: Findings From the Chronic Renal Insufficiency Cohort Study

BACKGROUND Prior studies associating acute kidney injury (AKI) with more rapid subsequent loss of kidney function had methodological limitations, including inadequate control for differences between patients who had AKI and those who did not. OBJECTIVE To determine whether AKI is independently associated with subsequent kidney function trajectory among patients with chronic kidney disease (CKD). DESIGN Multicenter prospective cohort study. SETTING United States. PARTICIPANTS Patients with CKD (n = 3150). MEASUREMENTS Hospitalized AKI was defined by a 50% or greater increase in inpatient serum creatinine (SCr) level from nadir to peak. Kidney function trajectory was assessed using estimated glomerular filtration rate (eGFR) based on SCr level (eGFRcr) or cystatin C level (eGFRcys) measured at annual study visits. RESULTS During a median follow-up of 3.9 years, 433 participants had at least 1 AKI episode. Most episodes (92%) had stage 1 or 2 severity. There were decreases in eGFRcr (-2.30 [95% CI, -3.70 to -0.86] mL/min/1.73 m2) and eGFRcys (-3.61 [CI, -6.39 to -0.82] mL/min/1.73 m2) after AKI. However, in fully adjusted models, the decreases were attenuated to -0.38 (CI, -1.35 to 0.59) mL/min/1.73 m2 for eGFRcr and -0.15 (CI, -2.16 to 1.86) mL/min/1.73 m2 for eGFRcys, and the CI bounds included the possibility of no effect. Estimates of changes in eGFR slope after AKI determined by either SCr level (0.04 [CI, -0.30 to 0.38] mL/min/1.73 m2 per year) or cystatin C level (-0.56 [CI, -1.28 to 0.17] mL/min/1.73 m2 per year) also had CI bounds that included the possibility of no effect. LIMITATIONS Few cases of severe AKI, no adjudication of AKI cause, and lack of information about nephrotoxic exposures after hospital discharge. CONCLUSION After pre-AKI eGFR, proteinuria, and other covariables were accounted for, the association between mild to moderate AKI and worsening subsequent kidney function in patients with CKD was small. PRIMARY FUNDING SOURCE National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health.
Specific effects of five subtypes of childhood maltreatment on suicide behaviours in Chinese adolescents: the moderating effect of sex and residence

Abstract Aims Although childhood maltreatment has been widely supported to be a robust predictor of suicide behaviours, the effects of different childhood maltreatment subtypes remain controversial and inconclusive. Moreover, whether the effects differ by sex in urban and rural adolescents is still unknown. This study aimed to quantify the associations between five subtypes of childhood maltreatment and different suicide behaviour involvement. Methods A multistage cluster sampling method was adopted from April to December 2021 for adolescents aged 12 to 18 across five representative provinces of China. The Childhood Trauma Questionnaire-Short Form was used to measure childhood maltreatment subtypes. Suicide behaviour involvement was classified as none group, suicide ideator, suicide planner and suicide attempter. Confounding variables include demographic characteristics, smoking, drinking alcohol, depression and anxiety. Results Among a total of 18,980 adolescents, 2021 (10.6%) were suicide ideator, 1595 (8.4%) were suicide planner and 1014 (5.3%) were suicide attempter. Rural females had the highest proportion of suicide ideator (13.8%) and suicide planner (11.5%). Multinomial logistic regression analysis indicated that five childhood maltreatment subtypes were independently associated with suicide behaviours, except for associations between sexual abuse and suicide ideator as well as suicide planner (p > 0.05). Moreover, these associations differ by sex and residence. After adjusted for interactions of different subtypes, structural equation model indicated that the direct effects of childhood maltreatment subtypes on suicide behaviours from high to low were emotional abuse (β = 0.363, p < 0.001), physical abuse (β = 0.100, p < 0.001) and sexual abuse (β = 0.033, p = 0.003), while the effects of physical neglect and emotional neglect were not significant (p > 0.05). Conclusions Five subtypes of childhood maltreatment have specific and non-equivalence associations with suicide behaviours. Emotional abuse may have the strongest effect, and sexual abuse have an acute effect on suicide behaviours. Suicide prevention programs for Chinese adolescents could focus on those who experienced emotional, physical and sexual abuse. Furthermore, strategies should be tailored by sex and residence, and rural females deserve more attention.
Radio Telemetry Reveals Two Cases of Predation on Nelson's Sparrow (Ammospiza nelsoni)

Abstract - During a radio-tracking study of Ammospiza nelsoni (Nelson's Sparrow) in Aulac, NB, Canada, in 2022, we observed 2 unexpected cases of predation. In the first case, we found the remains of a radio-tagged Nelson's Sparrow inside a regurgitated pellet, which presumably was from an Asio flammeus (Short-eared Owl) or a Circus hudsonius (Northern Harrier). In another case, we found an adult female Nelson's Sparrow and her nestlings buried in underground tunnels, likely having been depredated by a Mustela erminea (Short-tailed Weasel), an undocumented predator for Nelson's Sparrow. These observations contribute to greater knowledge of predation of Nelson's Sparrows and demonstrate an indirect benefit of tracking studies on wildlife.
Relationship between Euthyroidism and Muscle Mass and Strength: A Systematic Review

Abstract Thyroid hormones play a crucial role in skeletal muscle development, suggesting that thyroid function may influence muscle mass and muscle strength, which are both fundamental health-related indicators of several age-related consequences. However, whether there is a relationship between thyroid hormones, muscle mass, and muscle strength in individuals without thyroid dysfunctions is still unknown. Therefore, this systematic review aims to investigate whether thyroid hormones are related to muscle mass and strength parameters in euthyroid individuals. Three databases were searched (PubMed, Scopus, Web of Science) up to February 14, 2022, for peer-reviewed papers published in English. The search results were conducted independently by two different reviewers. The review included 13 studies with a total of 241,044 participants. All studies were observational: twelve studies measured thyroid stimulating hormone, ten and thirteen studies measured free triiodothyronine and free thyroxine, four studies analyzed the thyroid hormone ratio. The assessment methods for muscle mass were computed tomography, dual-energy X-ray absorptiometry and bioimpedance analysis, whereas hand dynamometer for muscle strength. Low levels within the normal range of free triiodothyronine, high levels within the normal range of free thyroxine, and lower thyroid hormone ratio may contribute to a reduced muscle function, which seems more evident in older males.
Occupational environment and ovarian cancer risk

Objectives To investigate employment in an occupation or industry and specific occupational exposures in relation to ovarian cancer risk. Methods In a population-based case–control study conducted in Montreal, Canada (2011–2016), lifetime occupational histories were collected for 491 cases of ovarian cancer and 897 controls. An industrial hygienist coded the occupation and industry of each participant’s job. Associations with ovarian cancer risk were estimated for each of several occupations and industries. Job codes were linked to the Canadian job-exposure matrix, thereby generating exposure histories to many agents. The relationship between exposure to each of the 29 most prevalent agents and ovarian cancer risk was assessed. Odds ratios and 95% confidence intervals (OR (95% CI)) for associations with ovarian cancer risk were estimated using logistic regression and controlling for multiple covariates. Results Elevated ORs (95% CI) were observed for employment ≥10 years as Accountants (2.05 (1.10 to 3.79)); Hairdressers, Barbers, Beauticians and Related Workers (3.22 (1.25 to 8.27)); Sewers and Embroiderers (1.85 (0.77 to 4.45)); and Salespeople, Shop Assistants and Demonstrators (1.45 (0.71 to 2.96)); and in the industries of Retail Trade (1.59 (1.05 to 2.39)) and Construction (2.79 (0.52 to 4.83)). Positive associations with ORs above 1.42 were seen for high cumulative exposure versus never exposure to 18 agents: cosmetic talc, ammonia, hydrogen peroxide, hair dust, synthetic fibres, polyester fibres, organic dyes and pigments, cellulose, formaldehyde, propellant gases, aliphatic alcohols, ethanol, isopropanol, fluorocarbons, alkanes (C5–C17), mononuclear aromatic hydrocarbons, polycyclic aromatic hydrocarbons from petroleum and bleaches. Conclusions Certain occupations, industries and specific occupational exposures may be associated with ovarian cancer risk. Further research is needed to provide a more solid grounding for any inferences in this regard.