Miracle-dz/UniFAR-FTUs huggingface dataset:
Pretrained Language Model Embryology: The Birth of ALBERT

While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process from a set of randomly initialized parameters to a totipotent language model, which we refer to as the embryology of a pretrained language model. Our results show that ALBERT learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks' performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We will provide source codes and pretrained models to reproduce our results at this https URL.
Affine Homogeneous Surfaces with Hessian rank 2 and Algebras of Differential Invariants

Consider a graphed holomorphic surface $u=F(x,y)$ in $\mathbb{C}^3_{x,y,u}$ under the action of the affine transformation group $A(3)$. In 1999, Eastwood and Ezhov obtained a list of homogeneous models by determining possible tangential vector fields. Inspired by Olver's recurrence formulas, we study the algebra of $A(3)$ differential invariants of surfaces. We obtain necessary conditions for homogeneity of algebraic nature. Solving these conditions, we organise homogeneous models in inequivalent branches.
QADiscourse - Discourse Relations as QA Pairs: Representation, Crowdsourcing and Baselines

Discourse relations describe how two propositions relate to one another, and identifying them automatically is an integral part of natural language understanding. However, annotating discourse relations typically requires expert annotators. Recently, different semantic aspects of a sentence have been represented and crowd-sourced via question-and-answer (QA) pairs. This paper proposes a novel representation of discourse relations as QA pairs, which in turn allows us to crowd-source wide-coverage data annotated with discourse relations, via an intuitively appealing interface for composing such questions and answers. Based on our proposed representation, we collect a novel and wide-coverage QADiscourse dataset, and present baseline algorithms for predicting QADiscourse relations.
Optimal High-Order Tensor SVD via Tensor-Train Orthogonal Iteration

This paper studies a general framework for high-order tensor SVD. We propose a new computationally efficient algorithm, tensor-train orthogonal iteration (TTOI), that aims to estimate the low tensor-train rank structure from the noisy high-order tensor observation. The proposed TTOI consists of initialization via TT-SVD [Oseledets (2011)] and new iterative backward/forward updates. We develop the general upper bound on estimation error for TTOI with the support of several new representation lemmas on tensor matricizations. By developing a matching information-theoretic lower bound, we also prove that TTOI achieves the minimax optimality under the spiked tensor model. The merits of the proposed TTOI are illustrated through applications to estimation and dimension reduction of high-order Markov processes, numerical studies, and a real data example on New York City taxi travel records. The software of the proposed algorithm is available online (https://github.com/Lili-Zheng-stat/TTOI).
Modeling Preconditions in Text with a Crowd-sourced Dataset

Preconditions provide a form of logical connection between events that explains why some events occur together and information that is complementary to the more widely studied relations such as causation, temporal ordering, entailment, and discourse relations. Modeling preconditions in text has been hampered in part due to the lack of large scale labeled data grounded in text. This paper introduces PeKo, a crowd-sourced annotation of preconditions between event pairs in newswire, an order of magnitude larger than prior text annotations. To complement this new corpus, we also introduce two challenge tasks aimed at modeling preconditions: (i) Precondition Identification – a standard classification task defined over pairs of event mentions, and (ii) Precondition Generation – a generative task aimed at testing a more general ability to reason about a given event. Evaluation on both tasks shows that modeling preconditions is challenging even for today’s large language models (LM). This suggests that precondition knowledge is not easily accessible in LM-derived representations alone. Our generation results show that fine-tuning an LM on PeKo yields better conditional relations than when trained on raw text or temporally-ordered corpora.
The TeV Cosmic Ray Bump: a Message from Epsilon Indi Star?

A recently observed TV bump in the cosmic ray (CR) spectrum, comprising two consecutive breaks, is likely caused by a stellar bow shock. It reaccelerates preexisting CRs, and they diffuse to the Sun along the magnetic field lines. They drive turbulence that self-controls the diffusion and forms the bump. We show that the turbulence has an Iroshnikov-Kraichnan (I-K) spectrum, $k^{-3/2}$, by solving the CR reacceleration and propagation problem analytically. While the ad hoc bump fit requires six parameters, our predicted CR spectrum at the Sun depends on only $two$ parameters, the shock Mach number, $M$, and the bump rigidity $R_{0}$, which absorbs other shock parameters. With $M\approx1.6$ and $R_{0}\approx4.4$ TV, the analytic solution deviates from the data only by $\approx0.08\% $. We rule out turbulence spectra other than the I-K. The analytic solution that depends only on two parameters makes an accidental agreement highly unlikely. The values of $R_{0}$ and $M$ obtained from the fit provide the distance-size ($\zeta_{\rm obs}-l_{\perp}$) relation for the shock: $\zeta_{\rm obs}$(pc) $\sim$ $10^{2}\sqrt{l_{\perp}(\text{pc})}$. Assuming $l_{\perp}=10^{-3}-10^{-2}$ pc, we find the path length along the magnetic field lines of $\zeta_{\rm obs}=3-10$ pc. There are at least two passing stars in this range: the binary Scholz's Star at 6.8 pc, and a triplet Epsilon Indi at 3.6 pc. Based on their current positions and velocities, we speculate that our Sun is in the wake of the Scholz's star, while the spectral bump is transmitted by the Epsilon Indi. Given the proximity of this star, the bump appearance may change in a relatively short time.
Efficient and fast predictor-corrector method for solving nonlinear fractional differential equations with non-singular kernel

Efficient and fast predictor-corrector methods are proposed to deal with nonlinear Caputo-Fabrizio fractional differential equations, where Caputo-Fabrizio operator is a new proposed fractional derivative with a smooth kernel. The proposed methods achieve a uniform accuracy order with the second-order scheme for linear interpolation and the third-order scheme for quadratic interpolation. The convergence analysis is proved by using the discrete Gronwall's inequality. Furthermore, applying the recurrence relation of the memory term, it reduces CPU time executed the proposed methods. The proposed fast algorithm requires approximately $O(N)$ arithmetic operations while $O(N^2)$ is required in case of the regular predictor-corrector schemes, where $N$ is the total number of time step. The following numerical examples demonstrate the accuracy of the proposed methods as well as the efficiency; nonlinear fractional differential equations, time-fraction sub-diffusion, and time-fractional advection-diffusion equation. The theoretical convergence rates are also verified by numerical experiments.
Doubly Robust Covariate Shift Regression with Semi-nonparametric Nuisance Models.

Importance weighting is naturally used to adjust for covariate shift. However, this simple strategy is not robust to model misspecifcation or excessive estimation error. To handle this problem, we propose a doubly robust covariate shift regression method by introducing an imputation model for conditional mean of the response to augment the importance weighted estimating equation. With a novel semi-nonparametric construction on the nuisance importance weight and imputation model, our method is more robust to excessive fitting error compared to the existing nonparametric or machine learning approaches and less likely to suffer from model misspecification than the parametric approach. To remove the overfitting bias of the nonparametric components under potential model misspecification, we construct specific calibrated moment estimating equations for the semi-nonparametric models. Theoretical properties of our estimator are studied. We show that our estimator attains the parametric rate when at least one nuisance model is correctly specified, estimation for the parametric part of the nuisance models achieves parametric rate and the nonparametric components satisfy the rate double robustness property. Simulation studies demonstrate that our method is more robust and efficient than parametric and fully nonparametric (machine learning) estimators under various configurations. We also examine the utility of our method through a real example about transfer learning of phenotyping algorithm for bipolar disorder. Finally, we discuss on possible ways to improve the (intrinsic) efficiency of our estimator and the potentiality of incorporating other nonparametric, high dimensional and machine learning models with our proposed framework.
Towards an arboretum of monadically stable classes of graphs

Logical transductions provide a very useful tool to encode classes of structures inside other classes of structures. In this paper we study first-order transductions and the quasi-order they induce on infinite hereditary classes of finite graphs. Surprisingly, this quasi-order is very complex, though shaped by the locality properties of first-order logic. This contrasts with the conjectured simplicity of the MSO transduction quasi-order. Main dividing lines inherited from classical model theory are the notions of monadic stability and monadic dependence. It appears that the FO transduction quasi-order has a great expressive power and many class properties studied earlier (such as having bounded pathwidth, bounded shrubdepth, etc.) may be equivalently defined by it. In this paper we study properties of the FO transduction quasi-order in detail, particularly in the monadically stable domain. Among other things we prove that the classes with given pathwidth form a strict hierarchy in the FO transduction quasi-order. Our basic results are based on a new normal form of transductions. We formulate several old and new conjectures. For example it is a challenging problem whether the classes with given treewidth form a strict hierarchy. This is illustrating the rich spectrum of an emerging area.
Position-Aware Tagging for Aspect Sentiment Triplet Extraction

Aspect Sentiment Triplet Extraction (ASTE) is the task of extracting the triplets of target entities, their associated sentiment, and opinion spans explaining the reason for the sentiment. Existing research efforts mostly solve this problem using pipeline approaches, which break the triplet extraction process into several stages. Our observation is that the three elements within a triplet are highly related to each other, and this motivates us to build a joint model to extract such triplets using a sequence tagging approach. However, how to effectively design a tagging approach to extract the triplets that can capture the rich interactions among the elements is a challenging research question. In this work, we propose the first end-to-end model with a novel position-aware tagging scheme that is capable of jointly extracting the triplets. Our experimental results on several existing datasets show that jointly capturing elements in the triplet using our approach leads to improved performance over the existing approaches. We also conducted extensive experiments to investigate the model effectiveness and robustness.
Plan Optimization to Bilingual Dictionary Induction for Low-resource Language Families

Creating bilingual dictionary is the first crucial step in enriching low-resource languages. Especially for the closely related ones, it has been shown that the constraint-based approach is useful for inducing bilingual lexicons from two bilingual dictionaries via the pivot language. However, if there are no available machine-readable dictionaries as input, we need to consider manual creation by bilingual native speakers. To reach a goal of comprehensively create multiple bilingual dictionaries, even if we already have several existing machine-readable bilingual dictionaries, it is still difficult to determine the execution order of the constraint-based approach to reducing the total cost. Plan optimization is crucial in composing the order of bilingual dictionaries creation with the consideration of the methods and their costs. We formalize the plan optimization for creating bilingual dictionaries by utilizing Markov Decision Process (MDP) with the goal to get a more accurate estimation of the most feasible optimal plan with the least total cost before fully implementing the constraint-based bilingual lexicon induction. We model a prior beta distribution of bilingual lexicon induction precision with language similarity and polysemy of the topology as and parameters. It is further used to model cost function and state transition probability. We estimated the cost of all investment plans as a baseline for evaluating the proposed MDP-based approach with total cost as an evaluation metric. After utilizing the posterior beta distribution in the first batch of experiments to construct the prior beta distribution in the second batch of experiments, the result shows 61.5% of cost reduction compared to the estimated all investment plans and 39.4% of cost reduction compared to the estimated MDP optimal plan. The MDP-based proposal outperformed the baseline on the total cost.
Textual supervision for visually grounded spoken language understanding

Visually-grounded models of spoken language understanding extract semantic information directly from speech, without relying on transcriptions. This is useful for low-resource languages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcriptions are available at training time. However, it is not clear how an end-to-end approach compares to a traditional pipeline-based approach when one has access to transcriptions. Comparing different strategies, we find that the pipeline approach works better when enough text is available. With low-resource languages in mind, we also show that translations can be effectively used in place of transcriptions but more data is needed to obtain similar results.
Interactions among IGF-1, AKT2, FOXO1, and FOXO3 variations and between genes and physical activities on physical performance in community-dwelling elders

This study assessed the interactions among IGF-1, AKT2, FOXO1, and FOXO3 variations and the interactions of gene and physical activity on handgrip strength, arm muscle mass-adjusted handgrip (armGrip), gait speed (GS), timed up and go (TUG), and leg press strength (LPS). Nine single nucleotide polymorphisms (SNPs) containing three IGF-1 SNPs (rs6214, rs5742692, and rs35767), two AKT2 SNPs (rs892119 and rs35817154), two FOXO1 SNPs (rs17446593 and rs10507486), and two FOXO3 SNPs (rs9480865 and rs2153960) were genotyped in 472 unrelated elders with a mean age of 73.8 years. We observed significant interactions of IGF-1 SNP rs6214 and rs35767 with regular physical activity on TUG and GS; and AKT2 SNP rs892119 and FOXO3 SNP rs9480865 with regular physical activity on armGrip. Genotype GG of IGF-1 rs6214 and rs35767 in individuals without regular physical activity had poor performance in TUG and GS, as well as GG of AKT2 rs892119 decreased armGrip in individuals without regular physical activity. After FDR adjustment, no significant gene–gene interactions were found. A sedentary lifestyle may increase the risk of impairing physical performance and regular physical activity is a remedy for sarcopenia, even a little regular physical activity can overcome carrying some risk alleles in this pathway.
Withania frutescens. L Extract: Phytochemical Characterization and Acute and Repeated Dose 28-Day Oral Toxicity Studies in Mice

Background Withania frutescens. L (W. frutescens) is a perennial woody medicinal plant belonging to family Solanaceae largely used by the indigenous population to Morocco for the treatment of disease. Objective The purpose of this study was to investigate the chemical composition, acute, and subacute toxicity of W. frutescens extract in mice. Materials and Methods The phytochemical composition of W. frutescens extract was determined using a gas chromatograph (GC/MS). Acute toxicity study was carried out in mice through oral administration of single doses 500 mg/kg, 1000 mg/kg, and 2000 mg/kg for 14 days. Subacute toxicity was performed with oral administration of repeated doses 500 and 2000 mg/kg/day for 28 days. Biochemical parameters (alanine aminotransferase, aspartate aminotransferase, urea, and creatinine), as well as histopathological changes potentially occurred in organs, (liver, kidney, and spleen) were evaluated. Results The results of chromatographic analysis showed the richness of W. frutescens extract in interesting phytochemical compounds majorly constituted of bicyclo[3.1. 1]heptane, 6,6-dimethyl-2-methylene-(C10H16). Regarding acute toxicity study, the results showed no clinical symptoms occurred in treated mice compared to the control group and no histological changes detected in analyzed organs of treated mice with dose put to 2000 mg/kg nor adverse effect on biochemical parameters. Conclusion The outcome of this work showed no toxic effect of W. frutescens in mice up to dose 2000 mg/kg bodyweight. Therefore, this study could scientifically validate further traditional use with safety in the range of tested doses.
Population structure analysis and identification of genomic regions under selection associated with low-nitrogen tolerance in tropical maize lines

Increasing low nitrogen (N) tolerance in maize is an important goal for food security and agricultural sustainability. In order to analyze the population structure of tropical maize lines and identify genomic regions associated with low-N tolerance, a set of 64 inbred lines were evaluated under low-N and optimal-N conditions. The low-N Agronomic Efficiency index (LNAE) of each line was calculated. The maize lines were genotyped using 417,112 SNPs markers. The grouping based on the LNAE values classified the lines into two phenotypic groups, the first comprised by genotypes with high LNAE (named H_LNAE group), while the second one comprised genotypes with low LNAE (named L_LNAE group). The H_LNAE and L_LNAE groups had LNAE mean values of 3,304 and 1,644, respectively. The population structure analysis revealed a weak relationship between genetic and phenotypic diversity. Pairs of lines were identified, having at the same time high LNAE and high genetic distance from each other. A set of 29 SNPs markers exhibited a significant difference in allelic frequencies (Fst > 0.2) between H_LNAE and L_LNAE groups. The Pearson’s correlation between LNAE and the favorable alleles in this set of SNPs was 0.69. These SNPs could be useful for marker-assisted selection for low-N tolerance in maize breeding programs. The results of this study could help maize breeders identify accessions to be used in the development of low-N tolerant cultivars.
Population-Wide Duchenne Muscular Dystrophy Carrier Detection by CK and Molecular Testing

Carrier screening of Duchenne muscular dystrophy (DMD) has not been widely evaluated. To identify definite DMD female carriers prior to or in early pregnancy, we studied a large population of reproductive age females and provided informed reproductive options to DMD carriers. 37268 females were recruited from the Hangzhou Family Planning Publicity and Technology Guidance Station/Hangzhou Health Service Center for Children and Women, Hangzhou, China, between October 10, 2017, and December 16, 2018. CK activity was measured with follow-up serum DMD genetic testing in subjects with hyperCKemia, defined as CK > 200 U/L. The calculated upper reference limit (97.5th percentile) of serum creatine kinase (CK) for females aged 20-50 years in this study was near the reference limit recommended by the manufacturer (200 U/L), above which was defined as hyperCKemia. 427 females (1.2%) harbored initially elevated CK, among which 281 females (response rate of 65.8%) accepted CK retesting. DMD genetic testing was conducted on 62 subjects with sustained serum CK > 200 U/L and 16 females with a family history of DMD. Finally, 6 subjects were confirmed to be DMD definite carriers. The estimated DMD female carrier rate in this study was 1 : 4088 (adjusting for response rate), an underestimated rate, since only 50% to 70% of DMD female carriers manifest elevated serum CK, and carriers in this study may have been missed due to lack of follow-up or inability to detect all DMD pathogenic variants by current genetic testing.
Estimation and prediction of ellipsoidal molecular shapes in organic crystals based on ellipsoid packing

Crystal structure prediction has been one of the fundamental and challenging problems in materials science. It is computationally exhaustive to identify molecular conformations and arrangements in organic molecular crystals due to complexity in intra- and inter-molecular interactions. From a geometrical viewpoint, specific types of organic crystal structures can be characterized by ellipsoid packing. In particular, we focus on aromatic systems which are important for organic semiconductor materials. In this study, we aim to estimate the ellipsoidal molecular shapes of such crystals and predict them from single molecular descriptors. First, we identify the molecular crystals with molecular centroid arrangements that correspond to affine transformations of four basic cubic lattices, through topological analysis of the dataset of crystalline polycyclic aromatic molecules. The novelty of our method is that the topological data analysis is applied to arrangements of molecular centroids intead of those of atoms. For each of the identified crystals, we estimate the intracrystalline molecular shape based on the ellipsoid packing assumption. Then, we show that the ellipsoidal shape can be predicted from single molecular descriptors using a machine learning method. The results suggest that topological characterization of molecular arrangements is useful for structure prediction of organic semiconductor materials.