Miracle-dz/UniFAR-FTUs huggingface dataset:
Scalable and Efficient MoE Training for Multitask Multilingual Models

The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to trillions of parameters. It combines multi-dimensional parallelism and heterogeneous memory technologies harmoniously with MoE to empower 8x larger models on the same hardware compared with existing work. Besides boosting system efficiency, we also present new training methods to improve MoE sample efficiency and leverage expert pruning strategy to improve inference time efficiency. By combining the efficient system and training methods, we are able to significantly scale up large multitask multilingual models for language generation which results in a great improvement in model accuracy. A model trained with 10 billion parameters on 50 languages can achieve state-of-the-art performance in Machine Translation (MT) and multilingual natural language generation tasks. The system support of efficient MoE training has been implemented and open-sourced with the DeepSpeed library.
The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments

Mobile manipulators for indoor human environments can serve as versatile devices that perform a variety of tasks, yet adoption of this technology has been limited. Reducing size, weight, and cost could facilitate adoption, but risks restricting capabilities. We present a novel design that reduces size, weight, and cost, while supporting a variety of tasks. The core design consists of a two-wheeled differential-drive mobile base, a lift, and a telescoping arm configured to achieve Cartesian motion at the end of the arm. Design extensions include a 1 degree-of-freedom (DOF) wrist to stow a tool, a 2-DOF dexterous wrist to pitch and roll a tool, and a compliant gripper. We justify our design with anthropometry and mathematical models of static stability. We also provide empirical support from teleoperating and autonomously controlling a commercial robot based on our design (the Stretch RE1 from Hello Robot Inc.) to perform tasks in real homes.
Decentralized Learning of Tree-Structured Gaussian Graphical Models from Noisy Data

This paper studies the decentralized learning of tree-structured Gaussian graphical models (GGMs) from noisy data. In decentralized learning, data set is distributed across different machines (sensors), and GGMs are widely used to model complex networks such as gene regulatory networks and social networks. The proposed decentralized learning uses the Chow-Liu algorithm for estimating the tree-structured GGM. In previous works, upper bounds on the probability of incorrect tree structure recovery were given mostly without any practical noise for simplification. While this paper investigates the effects of three common types of noisy channels: Gaussian, Erasure, and binary symmetric channel. For Gaussian channel case, to satisfy the failure probability upper bound $\delta>0$ in recovering a $d$-node tree structure, our proposed theorem requires only $\mathcal{O}(\log(\frac{d}{\delta}))$ samples for the smallest sample size ($n$) comparing to the previous literature \cite{Nikolakakis} with $\mathcal{O}(\log^4(\frac{d}{\delta}))$ samples by using the positive correlation coefficient assumption that is used in some important works in the literature. Moreover, the approximately bounded Gaussian random variable assumption does not appear in \cite{Nikolakakis}. Given some knowledge about the tree structure, the proposed Algorithmic Bound will achieve obviously better performance with small sample size (e.g., $<2000$) comparing with formulaic bounds. Finally, we validate our theoretical results by performing simulations on synthetic data sets.
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation

Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning. Previous mainstream VLP approaches typically adopt a two-step strategy relying on external object detectors to encode images in a multi-modal Transformer framework, which suffer from restrictive object concept space, limited image context and inefficient computation. In this paper, we propose an object-aware end-to-end VLP framework, which directly feeds image grid features from CNNs into the Transformer and learns the multi-modal representations jointly. More importantly, we propose to perform object knowledge distillation to facilitate learning cross-modal alignment at different semantic levels. To achieve that, we design two novel pretext tasks by taking object features and their semantic labels from external detectors as supervision: 1. ) Object-guided masked vision modeling task focuses on enforcing object-aware representation learning in the multi-modal Transformer; 2. ) Phrase-region alignment task aims to improve cross-modal alignment by utilizing the similarities between noun phrases and object labels in the linguistic space. Extensive experiments on a wide range of vision-language tasks demonstrate the efficacy of our proposed framework, and we achieve competitive or superior performances over the existing pretraining strategies.
The Population of Interstellar Objects Detectable with the LSST and Accessible for In Situ Rendezvous with Various Mission Designs

The recently discovered population of interstellar objects presents us with the opportunity to characterize material from extrasolar planetary and stellar systems up close. The forthcoming Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) will provide an unprecedented increase in sensitivity to these objects compared to the capabilities of currently operational observational facilities. We generate a synthetic population of ‘Oumuamua-like objects drawn from their galactic kinematics and identify the distribution of impact parameters, eccentricities, hyperbolic velocities, and sky locations of objects detectable with the LSST, assuming no cometary activity. This population is characterized by a clustering of trajectories in the direction of the solar apex and antiapex, centered at orbital inclinations of ∼90°. We identify the ecliptic or solar apex as the optimal sky location to search for future interstellar objects as a function of survey limiting magnitude. Moreover, we identify the trajectories of detectable objects that will be reachable for in situ rendezvous with a dedicated mission with the capabilities of the forthcoming Comet Interceptor or proposed Bridge concept. By scaling our fractional population statistics with the inferred spatial number density, we estimate that the LSST will detect of order ∼15 interstellar objects over the course of its ∼10 yr observational campaign. Furthermore, we find that there should be ∼1–3 and ∼0.0007–0.001 reachable targets for missions with propulsion capabilities comparable to Bridge and Comet Interceptor, respectively. These numbers are lower limits and will be readily updateable when the number density and size–frequency distribution of interstellar objects are better constrained.
Unsupervised Movement Detection in Indoor Positioning Systems

Consider indoor positioning systems (IPS) in production halls where objects equipped with sensors send their current position. Beside its large volume, the analyzation of the resulting raw data is challenging due to the susceptibility towards noise. Reasons are accuracy issues and undesired awakenings of sensors that occur due to the dynamics of logistic processes (e.g.~vibrations of passing forklifts). We propose a tailor-made statistical procedure for these challenges and combine visual analytics with movement detection. Contrary to common stay-point algorithms, we do not only distinguish between stops and moves, but also consider undesired awakenings. This leads to a more detailed interpretation scheme offering usages for online (e.g.~monitoring of orders) and offline applications (e.g.~detection of problematic areas). The approach does not require other information than the raw IPS output and enables an ad-hoc analysis. We underline our findings in an extensive case study with real IPS data of our industry partner.
Fractional Growth Portfolio Investment

We review some fundamental concepts of investment from a mathematical perspective, concentrating specifically on fractional-Kelly portfolios, which allocate a fraction of wealth to a growth-optimal portfolio while the remainder collects (or pays) interest at a risk-free rate. We elucidate a coherent continuous-parameter time-series framework for analysis of these portfolios, explaining relationships between Sharpe ratios, growth rates, and leverage. We see how Kelly’s criterion prescribes the same leverage as Markowitz mean-variance optimization. Furthermore, for fractional Kelly portfolios, we state a simple distributional relationship between portfolio Sharpe ratio, the fractional coefficient, and portfolio log-returns. These results provide critical insight into realistic expectations of growth for different classes of investors, from individuals to quantitative trading operations. We then illustrate application of the results by analyzing performance of various bond and equity mixes for an investor. We also demonstrate how the relationships can be exploited by a simple method-of-moments calculation to estimate portfolio Sharpe ratios and levels of risk deployment, given a fund’s reported returns.
FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning

Despite the success of neural dialogue systems in achieving high performance on the leader-board, they cannot meet users' requirements in practice, due to their poor reasoning skills. The underlying reason is that most neural dialogue models only capture the syntactic and semantic information, but fail to model the logical consistency between the dialogue history and the generated response. Recently, a new multi-turn dialogue reasoning task has been proposed, to facilitate dialogue reasoning research. However, this task is challenging, because there are only slight differences between the illogical response and the dialogue history. How to effectively solve this challenge is still worth exploring. This paper proposes a Fine-grained Comparison Model (FCM) to tackle this problem. Inspired by human's behavior in reading comprehension, a comparison mechanism is proposed to focus on the fine-grained differences in the representation of each response candidate. Specifically, each candidate representation is compared with the whole history to obtain a history consistency representation. Furthermore, the consistency signals between each candidate and the speaker's own history are considered to drive a model to prefer a candidate that is logically consistent with the speaker's history logic. Finally, the above consistency representations are employed to output a ranking list of the candidate responses for multi-turn dialogue reasoning. Experimental results on two public dialogue datasets show that our method obtains higher ranking scores than the baseline models.
High-dimensional Bayesian Optimization for CNN Auto Pruning with Clustering and Rollback

Pruning has been widely used to slim convolutional neural network (CNN) models to achieve a good trade-off be-tween accuracy and model size so that the pruned models become feasible for power-constrained devices such as mobile phones. This process can be automated to avoid the expensive hand-crafted efforts and to explore a large pruning space automatically so that the high-performance pruning policy can be achieved efﬁciently. Nowadays, reinforcement learning (RL) and Bayesian optimization (BO)-based auto pruners are widely used due to their solid theoretical foundation, universality, and high compressing quality. However, the RL agent suffers from long training times and high variance of results, while the BO agent is time-consuming for high-dimensional design spaces. In this work, we propose an enhanced BO agent to obtain signiﬁcant acceleration for auto pruning in high-dimensional design spaces. To achieve this, a novel clustering algorithm is proposed to reduce the dimension of the design space to speedup the searching process. Then, a roll-back algorithm is proposed to recover the high-dimensional design space so that higher pruning accuracy can be obtained. We validate our proposed method on ResNet, MobileNet, and VGG models, and our experiments show that the proposed method signiﬁcantly improves the accuracy of BO when pruning very deep CNN models. More-over, our method achieves lower variance and shorter time than the RL-based counterpart.
Updating Embeddings for Dynamic Knowledge Graphs

Data in Knowledge Graphs often represents part of the current state of the real world. Thus, to stay up-to-date the graph data needs to be updated frequently. To utilize information from Knowledge Graphs, many state-of-the-art machine learning approaches use embedding techniques. These techniques typically compute an embedding, i.e., vector representations of the nodes as input for the main machine learning algorithm. If a graph update occurs later on — specifically when nodes are added or removed — the training has to be done all over again. This is undesirable, because of the time it takes and also because downstream models which were trained with these embeddings have to be retrained if they change significantly. In this paper, we investigate embedding updates that do not require full retraining and evaluate them in combination with various embedding models on real dynamic Knowledge Graphs covering multiple use cases. We study approaches that place newly appearing nodes optimally according to local information, but notice that this does not work well. However, we find that if we continue the training of the old embedding, interleaved with epochs during which we only optimize for the added and removed parts, we obtain good results in terms of typical metrics used in link prediction. This performance is obtained much faster than with a complete retraining and hence makes it possible to maintain embeddings for dynamic Knowledge Graphs.
DPPIN: A Biological Repository of Dynamic Protein-Protein Interaction Network Data

In the big data era, the relationship between entries becomes more and more complex. Many graph (or network) algorithms have already paid attention to dynamic networks, which are more suitable than static ones for fitting the complex real-world scenarios with evolving structures and features. To contribute to the dynamic network representation learning and mining research, we provide a new bunch of label-adequate, dynamics-meaningful, and attribute-sufficient dynamic networks from the health domain. To be specific, in our proposed repository DPPIN, we totally have 12 individual dynamic network datasets at different scales, and each dataset is a dynamic protein-protein interaction network describing protein-level interactions of yeast cells. We hope these domain-specific node features, structure evolution patterns, and node and graph labels could inspire the regularization techniques to increase the performance of graph machine learning algorithms in a more complex setting. Also, we link potential applications with our DPPIN by designing various dynamic graph experiments, where DPPIN could indicate future research opportunities for some tasks by presenting challenges on state-of-the-art baseline algorithms. Finally, we identify future directions to improve the utility of this repository and welcome constructive inputs from the community. All resources (e.g., data and code) of this work are deployed and publicly available at https://github.com/DongqiFu/DPPIN.
External Knowledge enabled Text Visual Question Answering

The open-ended question answering task of Text-VQA requires reading and reasoning about local, often previously unseen, scene-text content of an image to generate answers. In this work, we propose the generalized use of external knowledge to augment our understanding of the said scene-text. We design a framework to extract, validate, and reason with knowledge using a standard multimodal transformer for vision language understanding tasks. Through empirical evidence and qualitative results, we demonstrate how external knowledge can highlight instance-only cues and thus help deal with training data bias, improve answer entity type correctness, and detect multiword named entities. We generate results comparable to the state-of-the-art on two publicly available datasets, under the constraints of similar upstream OCR systems and training data.
Small-d MSR Codes With Optimal Access, Optimal Sub-Packetization, and Linear Field Size

This paper presents an explicit construction of a class of optimal-access, minimum storage regenerating (MSR) codes, for small values of the number <inline-formula> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> of helper nodes. The construction is valid for any parameter set <inline-formula> <tex-math notation="LaTeX">$(n,k,d)$ </tex-math></inline-formula> with <inline-formula> <tex-math notation="LaTeX">$d \in \{k+1, k+2, k+3\}$ </tex-math></inline-formula> and employs a finite field <inline-formula> <tex-math notation="LaTeX">$\mathbb {F}_{q}$ </tex-math></inline-formula> of size <inline-formula> <tex-math notation="LaTeX">$q=O(n)$ </tex-math></inline-formula>. We will refer to the constructed codes as <inline-formula> <tex-math notation="LaTeX">$\text {Small-}\mathsf {d}$ </tex-math></inline-formula> MSR codes. The sub-packetization level <inline-formula> <tex-math notation="LaTeX">$\alpha $ </tex-math></inline-formula> is given by <inline-formula> <tex-math notation="LaTeX">$\alpha = s^{{\lceil \frac {n}{s}\rceil }}$ </tex-math></inline-formula>, where <inline-formula> <tex-math notation="LaTeX">$s=d-k+1$ </tex-math></inline-formula>. By an earlier result on the sub-packetization level for optimal-access MSR codes, this is the smallest value possible.
Reducing the Risk of Iron Toxicity by Imposing Unsaturated Conditions before Flooding

ABSTRACT Unsaturated incubation (UI) of air-dried soil before flooding significantly reduced the reduction of redox potential and decreased the easily reducible iron (Fe) (ERFe). Antibacterial treatment of the soil decreased the ERFe level regardless of the moisture regime while fungicide treatment reduced the effects of UI. Treating soil under imposed UI with sucrose helped recover the ERFe to the non-UI level. The effects of UI disappeared after air-drying the soil; therefore, the decrease in ERFe—by imposing upland conditions before flooding—could be due to the decrease in the activity of iron-reducing bacteria owing to the depletion of respiratory substrates by bacterial activity during the upland conditions. These results indicate that a drastic change in soil moisture from dry to wet will increase the risk of iron toxicity by increasing the ERFe, but that maintaining an upland condition for more than one week before total submergence can reduce the risk of iron toxicity.
Long-term Impact of Prior Secondhand Tobacco Smoke Exposure on Respiratory Health

Rationale: Ongoing secondhand tobacco smoke (SHS) exposure is associated with worsened respiratory health, but little is known about the long-term impact decades after exposure ended. Objective: Determine the long-term consequences of SHS exposure on respiratory health. Methods: Population-based, cohort study in subjects [≥]50 years old who had >1 year versus [≤]1 year of airline occupational SHS-exposure. Measurements and Main Results: Respiratory health was the primary outcome measured by the St. Georges Respiratory Questionnaire (SGRQ). Key secondary outcomes included respiratory symptoms measured by COPD Assessment Test (CAT) and pre-bronchodilator lung function. The study enrolled 183 SHS-exposed and 59 unexposed subjects. SHS-exposed subjects were exposed to airline SHS for 16.1{+/-}9.3 years, which ended 27.5{+/-}9.4 years prior to enrollment. Prior SHS-exposure was associated with worse respiratory health based on a 6.7-unit increase in SGRQ (95% CI=[2.7, 10.7]; p=0.001) and 3-unit increase in CAT (95% CI=1.4, 4.6]; p<0.001) versus unexposed subjects, but was not associated with airflow obstruction defined by FEV1:FVC<0.7. Clinically-significant respiratory symptoms (CAT[≥]10) in SHS-exposed never smokers with preserved lung function (FEV1:FVC [≥]0.7 and FVC [≥]lower limit of normal) were associated with decreased respiratory and non-respiratory quality-of-life, reduced lung function that remained within the normal range, increased comorbidities and inhaled bronchodilator use, higher plasma CRP and SAA-1 and fewer sinonasal basal stem/progenitor cells versus asymptomatic (CAT<10) SHS-exposed subjects. Conclusion: SHS is associated with a phenotype of impaired respiratory health almost 3 decades after exposure ended, consistent with a symptomatic form of COPD with preserved lung function recently described in smokers.
A Statewide EMS Viral Syndrome Pandemic Triage Protocol: 24 Hour Outcomes

Abstract Background: Early during the COVID-19 pandemic, Emergency Medical Services (EMS) systems encountered many challenges that prompted crisis-level strategies. Maryland’s statewide EMS system implemented the Viral Syndrome Pandemic Triage Protocol which contained a decision tool to help identify patients potentially safe for self-care at home. Objectives: This study assessed the effects of the Maryland Viral Syndrome Pandemic Triage Protocol and the safety of referring patients for self-care at home. Methods: This is a retrospective statewide analysis of EMS patients from March 19 thru September 4, 2020, who were not transported and had documentation of the Viral Syndrome Pandemic Triage Protocol’s decision support tool completed, as well as a random sample of 150 patients who were not transported and did not have documentation of the decision tool. Descriptive statistics were performed as well as a two-stage multivariable logistic regression model for the outcomes of ED presentation within 24 hours and subsequent hospitalization. Results: 301 EMS patients were documented as triaged to home using the protocol and outcomes data were available for 282 (94%). 41(14.5%) patients presented to an ED within 24 hours and 14 (5% of 282) required inpatient hospitalization. Nine (3.2%) patients were subsequently hospitalized with a diagnosis of COVID-19 illness. Of those patients for whom the decision tool was not documented, 35 (23%) had an ED visit within 24 hours and 15 (10%) were hospitalized (p = 0.075). Multivariate logistic regression model results (N = 432) suggest that those with documentation of triage protocol use had some advantage over those patients without documentation. The 95% CIs of the estimated effect of Triage/No Triage protocol documented were wide and crossed the 1.0 limit but overall, all effects Odds Ratios and Adjust Odds Ratios were consistently over 1.0 with the lowest value of 1.3 and the highest value of 2.1. Conclusion: Most patients (95%) who were triaged to self-care at home with home documented decision support tool use did not require hospitalization within 24 hours following EMS encounter and this appears to be safe. Future opportunity exists to incorporate such tools into comprehensive pandemic preparedness strategies along with appropriate follow up and quality improvement mechanisms.