Miracle-dz/UniFAR-FTUs huggingface dataset:
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep learning

In the last three years, the largest dense deep learning models have grown over 1000x to reach hundreds of billions of parameters, while the GPU memory has only grown by 5x (16 GB to 80 GB). Therefore, the growth in model scale has been supported primarily though system innovations that allow large models to fit in the aggregate GPU memory of multiple GPUs. However, we are getting close to the GPU memory wall. It requires 800 NVIDIA V100 GPUs just to fit a trillion parameter model for training, and such clusters are simply out of reach for most data scientists. In addition, training models at that scale requires complex combinations of parallelism techniques that puts a big burden on the data scientists to refactor their model. In this paper we present ZeRO-Infinity, a novel heterogeneous system technology that leverages GPU, CPU, and NVMe memory to allow for unprecedented model scale on limited resources without requiring model code refactoring. At the same time it achieves excellent training throughput and scalability, unencumbered by the limited CPU or NVMe bandwidth. ZeRO-Infinity can fit models with tens and even hundreds of trillions of parameters for training on current generation GPU clusters. It can be used to fine-tune trillion parameter models on a single NVIDIA DGX-2 node, making large models more accessible. In terms of training throughput and scalability, it sustains over 25 petaflops on 512 NVIDIA V100 GPUs (40% of peak), while also demonstrating super linear scalability. An open source implementation of ZeRO-Infinity is available through DeepSpeed 11DeepSpeed (https://www.deepspeed.ai/) is a deep learning optimization library designed to make distributed training easy, efficient, and effective. DeepSpeed has been extensively adopted by the DL community.
Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema

The Winograd Schema (WS) has been proposed as a test for measuring commonsense capabilities of models. Recently, pre-trained language model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. This paper suggests that the apparent progress on WS may not necessarily reflect progress in commonsense reasoning. To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for evaluation. We also propose two new baselines that indicate the existence of artifacts in WS benchmarks. We then develop a method for evaluating WS-like sentences in a zero-shot setting to account for the commonsense reasoning abilities acquired during the pretraining and observe that popular language models perform randomly in this setting when using our more strict evaluation. We conclude that the observed progress is mostly due to the use of supervision in training WS models, which is not likely to successfully support all the required commonsense reasoning skills and knowledge.
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills

We consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. The videos of our experiments can be found at https://actionable-models.github. io
Citations are not opinions: a corpus linguistics approach to understanding how citations are made

Citation content analysis seeks to understand citations based on the language used during the making of a citation. A key issue in citation content analysis is looking for linguistic structures that characterize distinct classes of citations for the purposes of understanding the intent and function of a citation. Previous works have focused on modeling linguistic features first and drawn conclusions on the language structures unique to each class of citation function based on the performance of a classification task or inter-annotator agreement. In this study, we start with a large sample of a pre-classified citation corpus, 2 million citations from each class of the scite Smart Citation dataset (supporting, disputing, and mentioning citations), and analyze its corpus linguistics in order to reveal the unique and statistically significant language structures belonging to each type of citation. By generating comparison tables for each citation type we present a number of interesting linguistic features that uniquely characterize citation type. What we find is that within citation collocates, there is very low correlation between citation type and sentiment. Additionally, we find that the subjectivity of citation collocates across classes is very low. These findings suggest that the sentiment of collocates is not a predictor of citation function and that due to their low subjectivity, an opinion-expressing mode of understanding citations, implicit in previous citation sentiment analysis literature, is inappropriate. Instead, we suggest that citations can be better understood as claims-making devices where the citation type can be explained by understanding how two claims are being compared. By presenting this approach, we hope to inspire similar corpus linguistic studies on citations that derive a more robust theory of citation from an empirical basis using citation corpora
Distributed TD(0) With Almost No Communication

We provide a new non-asymptotic analysis of distributed temporal difference learning with linear function approximation. Our approach relies on “one-shot averaging,” where N agents run identical local copies of the TD(0) method and average the outcomes only once at the very end. We demonstrate a version of the linear time speedup phenomenon, where the convergence time of the distributed process is a factor of N faster than the convergence time of TD(0). This is the first result proving benefits from parallelism for temporal difference methods.
Tidal deformability of strange stars and the GW170817 event

In this work we consider strange stars formed by quark matter in the color-flavor-locked (CFL) phase of color superconductivity. The CFL phase is described by a Nambu-Jona-Lasinio model with four-fermion vector and diquark interaction channels. The effect of the color superconducting medium on the gluons are incorporated into the model by including the gluon self-energy in the thermodynamic potential. We construct parametrizations of the model by varying the vector coupling GV and comparing the results to the data on tidal deformability from the GW170817 event, the observational data on maximum masses from massive pulsars such as the MSP J0740+6620, and the mass/radius fits to NICER data for PSR J003+0451. Our results points out to windows for the GV parameter space of the model, with and without gluon effects included, that are compatible with all these astrophysical constraints, namely, 0.21 < GV /GS < 0.4, and 0.02 < GV /GS < 0.1, respectively. We also observe a strong correlation between the tidal deformabilites of the GW170817 event and GV . Our results indicate that strange stars cannot be ruled out in collisions of compact binaries from the structural point of view.
PHANGS–ALMA Data Processing and Pipeline

We describe the processing of the PHANGS–ALMA survey and present the PHANGS–ALMA pipeline, a public software package that processes calibrated interferometric and total power data into science-ready data products. PHANGS–ALMA is a large, high-resolution survey of CO(2–1) emission from nearby galaxies. The observations combine ALMA’s main 12 m array, the 7 m array, and total power observations, and use mosaics of dozens to hundreds of individual pointings. We describe the processing of the u − v data, imaging and deconvolution, linear mosaicking, combining interferometer and total power data, noise estimation, masking, data product creation, and quality assurance. Our pipeline has a general design and can also be applied to Very Large Array and ALMA observations of other spectral lines and continuum emission. We highlight our recipe for deconvolution of complex spectral line observations, which combines multiscale clean, single-scale clean, and automatic mask generation in a way that appears robust and effective. We also emphasize our two-track approach to masking and data product creation. We construct one set of “broadly masked” data products, which have high completeness but significant contamination by noise, and another set of “strictly masked” data products, which have high confidence but exclude faint, low signal-to-noise emission. Our quality assurance tests, supported by simulations, demonstrate that 12 m+7 m deconvolved data recover a total flux that is significantly closer to the total power flux than the 7 m deconvolved data alone. In the appendices, we measure the stability of the ALMA total power calibration in PHANGS–ALMA and test the performance of popular short-spacing correction algorithms.
Real-Time Non-Rigid 3D Respiratory Motion Estimation for MR-Guided Radiotherapy Using MR-MOTUS

The MR-Linac is a combination of an MR-scanner and radiotherapy linear accelerator (Linac) which holds the promise to increase the precision of radiotherapy treatments with MR-guided radiotherapy by monitoring motion during radiotherapy with MRI, and adjusting the radiotherapy plan accordingly. Optimal MR-guidance for respiratory motion during radiotherapy requires MR-based 3D motion estimation with a latency of 200–500 ms. Currently this is still challenging since typical methods rely on MR-images, and are therefore limited by the 3D MR-imaging latency. In this work, we present a method to perform non-rigid 3D respiratory motion estimation with 170 ms latency, including both acquisition and reconstruction. The proposed method called real-time low-rank MR-MOTUS reconstructs motion-fields directly from ${k}$ -space data, and leverages an explicit low-rank decomposition of motion-fields to split the large scale 3D+t motion-field reconstruction problem posed in our previous work into two parts: (I) a medium-scale offline preparation phase and (II) a small-scale online inference phase which exploits the results of the offline phase for real-time computations. The method was validated on free-breathing data of five volunteers, acquired with a 1.5T Elekta Unity MR-Linac. Results show that the reconstructed 3D motion-field are anatomically plausible, highly correlated with a self-navigation motion surrogate ( ${R}={0.975} \pm {0.0110}$ ), and can be reconstructed with a total latency of 170 ms that is sufficient for real-time MR-guided abdominal radiotherapy.
Approximation algorithms for 1-Wasserstein distance between persistence diagrams

Recent years have witnessed a tremendous growth using topological summaries, especially the persistence diagrams (encoding the so-called persistent homology) for analyzing complex shapes. Intuitively, persistent homology maps a potentially complex input object (be it a graph, an image, or a point set and so on) to a unified type of feature summary, called the persistence diagrams. One can then carry out downstream data analysis tasks using such persistence diagram representations. A key problem is to compute the distance between two persistence diagrams efficiently. In particular, a persistence diagram is essentially a multiset of points in the plane, and one popular distance is the so-called 1-Wasserstein distance between persistence diagrams. In this paper, we present two algorithms to approximate the 1-Wasserstein distance for persistence diagrams in near-linear time. These algorithms primarily follow the same ideas as two existing algorithms to approximate optimal transport between two finite point-sets in Euclidean spaces via randomly shifted quadtrees. We show how these algorithms can be effectively adapted for the case of persistence diagrams. Our algorithms are much more efficient than previous exact and approximate algorithms, both in theory and in practice, and we demonstrate its efficiency via extensive experiments. They are conceptually simple and easy to implement, and the code is publicly available in github.
Towards Variable-Length Textual Adversarial Attacks

Adversarial attacks have shown the vulnerability of machine learning models, however, it is non-trivial to conduct textual adversarial attacks on natural language processing tasks due to the discreteness of data. Most previous approaches conduct attacks with the atomic \textit{replacement} operation, which usually leads to fixed-length adversarial examples and therefore limits the exploration on the decision space. In this paper, we propose variable-length textual adversarial attacks~(VL-Attack) and integrate three atomic operations, namely \textit{insertion}, \textit{deletion} and \textit{replacement}, into a unified framework, by introducing and manipulating a special \textit{blank} token while attacking. In this way, our approach is able to more comprehensively find adversarial examples around the decision boundary and effectively conduct adversarial attacks. Specifically, our method drops the accuracy of IMDB classification by $96\%$ with only editing $1.3\%$ tokens while attacking a pre-trained BERT model. In addition, fine-tuning the victim model with generated adversarial samples can improve the robustness of the model without hurting the performance, especially for length-sensitive models. On the task of non-autoregressive machine translation, our method can achieve $33.18$ BLEU score on IWSLT14 German-English translation, achieving an improvement of $1.47$ over the baseline model.
Interval-censored Hawkes processes

Interval-censored data solely records the aggregated counts of events during specific time intervals - such as the number of patients admitted to the hospital or the volume of vehicles passing traffic loop detectors - and not the exact occurrence time of the events. It is currently not understood how to fit the Hawkes point processes to this kind of data. Its typical loss function (the point process log-likelihood) cannot be computed without exact event times. Furthermore, it does not have the independent increments property to use the Poisson likelihood. This work builds a novel point process, a set of tools, and approximations for fitting Hawkes processes within interval-censored data scenarios. First, we define the Mean Behavior Poisson process (MBPP), a novel Poisson process with a direct parameter correspondence to the popular self-exciting Hawkes process. We fit MBPP in the interval-censored setting using an interval-censored Poisson log-likelihood (IC-LL). We use the parameter equivalence to uncover the parameters of the associated Hawkes process. Second, we introduce two novel exogenous functions to distinguish the exogenous from the endogenous events. We propose the multi-impulse exogenous function - for when the exogenous events are observed as event time - and the latent homogeneous Poisson process exogenous function - for when the exogenous events are presented as interval-censored volumes. Third, we provide several approximation methods to estimate the intensity and compensator function of MBPP when no analytical solution exists. Fourth and finally, we connect the interval-censored loss of MBPP to a broader class of Bregman divergence-based functions. Using the connection, we show that the popularity estimation algorithm Hawkes Intensity Process (HIP) is a particular case of the MBPP. We verify our models through empirical testing on synthetic data and real-world data.
Cross-Modal Retrieval Augmentation for Multi-Modal Classification

Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r. t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.
Rainbow version of the Erdős Matching Conjecture via concentration

We say that the families $\mathcal F_1,\ldots, \mathcal F_{s+1}$ of $k$-element subsets of $[n]$ are cross-dependent if there are no pairwise disjoint sets $F_1,\ldots, F_{s+1}$, where $F_i\in \mathcal F_i$ for each $i$. The rainbow version of the Erd\H os Matching Conjecture due to Aharoni and Howard and independently to Huang, Loh and Sudakov states that $\min_{i} |\mathcal F_i|\le \max\big\{{n\choose k}-{n-s\choose k}, {(s+1)k-1\choose k}\big\}$. In this paper, we prove this conjecture for $n>3e(s+1)k$ and $s>10^7$. One of the main tools in the proof is a concentration inequality due to Frankl and the author.
A study on the present status of freelancers in speech-language rehabilitation

Job hunting and job instability are reported through various media. In this reality, the standard employment type such as full-time employment due to employment instability gradually decreased and various irregular labor forms such as freelance or independent contractors and non-wage workers appeared. The number has increased in recent years, with the proportion of irregular labor forms in Korea as of 2018 the fifth highest in the OECD 35 countries [1,2]. Among the non-regular forms of labor, freelancers include non-wage workers, special workers, and one person business workers without employees, including all individuals who are not registered as business operators [3]. In the past, ‘freelancer’ was recognized as not a legal concept, nor a form of occupational group normally applied. However, the recent ‘freelancer’ is defined as a legal concept: Purpose: Corona-19 Infectious Diseases temporarily closed the workplace of speech therapists or temporarily suspended the work. Therefore, we investigated the current status of speech therapists in order to enact freelancer support ordinances of each local government and to urge language rehabilitation workers to support them.
Training and Development Costs , Staff Costs and Operational Profitability in Nepalese Commercial Banks Management Dynamics

Naba Raj Adhikari* Abstract The main aim of this study was to identify examine the relationship between staff trainings and development costs, total staff costs and the operational profit of Nepalese commercial banks. Six commercial banks has been taken for the study using purposive sampling technique from the total population consisting two government owned banks, two joint venture banks and the rest from private commercial banks. Training and development costs, staff costs and operational profit (before tax) has been defined as the variables of the study. Data have been collected from the annual reports of respective banks covering for the financial year 2016/2017 to 2019/2020. This study found that private commercial banks have focused on training and development to their staff. The staff costs of private banks has been highly explained by training and development costs and it has resulted higher positive impact of staff cost with operational profit.
Killer Incentives: Rivalry, Performance and Risk-Taking among German Fighter Pilots, 1939–45

Using newly collected data on death rates and aerial victories of more than 5,000 German ﬁghter pilots during World War II, we examine the effects of public recognition on performance and risk-taking. When a particular pilot is honoured publicly, both the victory rate and the death rate of his former peers increase. Fellow pilots react more if they come from the same region of Germany, or if they worked closely with him. Our results suggest that personal rivalry can be a prime motivating force, and that non-ﬁnancial rewards can lead to a crowd-in of both effort and risk-taking via social connections.
Effects of resistance training on metabolic and cardiovascular responses to a maximal cardiopulmonary exercise test in Parkinson`s disease

ABSTRACT: Objective: To evaluate the effects of resistance training on metabolic and cardiovascular responses during maximal cardiopulmonary exercise testing in patients with Parkinson’s disease. Methods: Twenty-four patients with Parkinson’s disease (modified Hoehn and Yahr stages 2 to 3) were randomly assigned to one of two groups: Control or Resistance Training. Patients in the Resistance Training Group completed an exercise program consisting of five resistance exercises (two to four sets of six to 12 repetitions maximum per set) twice a week. Patients in the Control Group maintained their usual lifestyle. Oxygen uptake, systolic blood pressure and heart rate were assessed at rest and during cycle ergometer-based maximal cardiopulmonary exercise testing at baseline and at 12 weeks. Assessments during exercise were conducted at absolute submaximal intensity (slope of the linear regression line between physiological variables and absolute workloads), at relative submaximal intensity (anaerobic threshold and respiratory compensation point) and at maximal intensity (maximal exercise). Muscle strength was also evaluated. Results: Both groups had similar increase in peak oxygen uptake after 12 weeks of training. Heart rate and systolic blood pressure measured at absolute and relative submaximal intensities and at maximal exercise intensity did not change in any of the groups. Muscle strength increased in the Resistance Training but not in the Control Group after 12 weeks. Conclusion: Resistance training increases muscle strength but does not change metabolic and cardiovascular responses during maximal cardiopulmonary exercise testing in patients with Parkinson’s disease without cardiovascular comorbidities.
Cryo-EM structures of the autoinhibited E. coli ATP synthase in three rotational states

A molecular model that provides a framework for interpreting the wealth of functional information obtained on the E. coli F-ATP synthase has been generated using cryo-electron microscopy. Three different states that relate to rotation of the enzyme were observed, with the central stalk’s ε subunit in an extended autoinhibitory conformation in all three states. The Fo motor comprises of seven transmembrane helices and a decameric c-ring and invaginations on either side of the membrane indicate the entry and exit channels for protons. The proton translocating subunit contains near parallel helices inclined by ~30° to the membrane, a feature now synonymous with rotary ATPases. For the first time in this rotary ATPase subtype, the peripheral stalk is resolved over its entire length of the complex, revealing the F1 attachment points and a coiled-coil that bifurcates toward the membrane with its helices separating to embrace subunit a from two sides. DOI: http://dx.doi. org/10.7554/eLife.21598. 001