Miracle-dz/UniFAR-FTUs huggingface dataset:
Rethinking Why Intermediate-Task Fine-Tuning Works

Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest. While STILTs is able to further improve the performance of pretrained language models, it is still unclear why and when it works. Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa. In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills -- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks. We conduct extensive experiments to study the impact of different factors on STILTs. These findings suggest rethinking the role of intermediate fine-tuning in the STILTs pipeline.
Theory of nonlinear acoustic forces acting on inhomogeneous fluids

Abstract Recently, the phenomena of streaming suppression and relocation of inhomogeneous miscible fluids under acoustic fields were explained using the hypothesis on mean Eulerian pressure. In this work, we derive the expression for the acoustic body force without relying on any prior assumptions regarding the second-order Eulerian pressure. We present a theory of nonlinear acoustics for inhomogeneous fluids from first principles, which explains streaming suppression and acoustic relocation in both miscible and immiscible inhomogeneous fluids inside a microchannel. This theory predicts the relocation of higher impedance fluids to pressure nodes of the standing wave, which agrees with recent experiments.
A new expansion of planetary disturbing function and applications to interior, co-orbital and exterior resonances with planets

In this study, a new expansion of planetary disturbing function is developed for describing the resonant dynamics of minor bodies with arbitrary inclinations and semimajor axis ratios. In practice, the disturbing function is expanded around circular orbits in the first step and then, in the second step, the resulting mutual interaction between circular orbits is expanded around a reference point. As usual, the resulting expansion is presented in the Fourier series form, where the force amplitudes are dependent on the semimajor axis, eccentricity and inclination, and the harmonic arguments are linear combinations of the mean longitude, longitude of pericenter and longitude of ascending node of each mass. The resulting new expansion is valid for arbitrary inclinations and semimajor axis ratios. In the case of mean motion resonant configuration, the disturbing function can be easily averaged to produce the analytical expansion of resonant disturbing function. Based on the analytical expansion, the Hamiltonian model of mean motion resonances is formulated, and the resulting analytical developments are applied to Jupiter’s inner and co-orbital resonances and Neptune’s exterior resonances. Analytical expansion is validated by comparing the analytical results with the associated numerical outcomes.
On Approximate Nearest Neighbour Selection for Multi-Stage Dense Retrieval

Dense retrieval, which describes the use of contextualised language models such as BERT to identify documents from a collection by leveraging approximate nearest neighbour (ANN) techniques, has been increasing in popularity. Two families of approaches have emerged, depending on whether documents and queries are represented by single or multiple embeddings. ColBERT, the exemplar of the latter, uses an ANN index and approximate scores to identify a set of candidate documents for each query embedding, which are then re-ranked using accurate document representations. In this manner, a large number of documents can be retrieved for each query, hindering the efficiency of the approach. In this work, we investigate the use of ANN scores for ranking the candidate documents, in order to decrease the number of candidate documents being fully scored. Experiments conducted on the MSMARCO passage ranking corpus demonstrate that, by cutting of the candidate set by using the approximate scores to only 200 documents, we can still obtain an effective ranking without statistically significant differences in effectiveness, and resulting in a 2x speedup in efficiency.
Lies My Teacher Told Me About Density Functional Theory: Seeing Through Them with the Hubbard Dimer

Most realistic calculations of moderately correlated materials begin with a ground-state density functional theory (DFT) calculation. While Kohn-Sham DFT is used in about 40,000 scientiﬁc papers each year, the fundamental underpinnings are not widely appreciated. In this chapter, we analyze the inherent characteristics of DFT in their simplest form, using the asymmetric Hubbard dimer as an illustrative model. We begin by working through the core tenets of DFT, explaining what the exact ground-state density functional yields and does not yield. Given the relative simplicity of the system, almost all properties of the exact exchange-correlation functional are readily visualized and plotted. Key concepts include the Kohn-Sham scheme, the behavior of the XC potential as correlations become very strong, the derivative discontinuity and the diﬀerence between KS gaps and true charge gaps, and how to extract optical excitations using time-dependent DFT. By the end of this text and accompanying exercises, the reader will improve their ability to both explain and visualize the concepts of DFT, as well as better understand where others may go wrong. This chapter appears in the book Autumn School on Correlated Electrons: Simulating Correlations with Computers (2021) prepared by Forschungszentrum J¨ulich.
Second-order accurate Dirichlet boundary conditions for linear nonlocal diffusion problems

We present an approach to handle Dirichlet type nonlocal boundary conditions for nonlocal diffusion models with a finite range of nonlocal interactions. Our approach utilizes a linear extrapolation of prescribed boundary data. A novelty is, instead of using local gradients of the boundary data that are not available a priori, we incorporate nonlocal gradient operators into the formulation to generalize the finite differences-based methods which are pervasive in literature; our particular choice of the nonlocal gradient operators is based on the interplay between a constant kernel function and the geometry of nonlocal interaction neighborhoods. Such an approach can be potentially useful to address similar issues in peridynamics, smoothed particle hydrodynamics and other nonlocal models. We first show the well-posedness of the newly formulated nonlocal problems and then analyze their asymptotic convergence to the local limit as the nonlocality parameter shrinks to zero. We justify the second order localization rate, which is the optimal order attainable in the absence of physical boundaries.
Optical Solitons and Vortices in Fractional Media: A Mini-Review of Recent Results

The article produces a brief review of some recent results which predict stable propagation of solitons and solitary vortices in models based on the nonlinear Schrodinger equation (NLSE) including fractional one-dimensional or two-dimensional diffraction and cubic or cubic-quintic nonlinear terms, as well as linear potentials. The fractional diffraction is represented by fractional-order spatial derivatives of the Riesz type, defined in terms of the direct and inverse Fourier transform. In this form, it can be realized by spatial-domain light propagation in optical setups with a specially devised combination of mirrors, lenses, and phase masks. The results presented in the article were chiefly obtained in a numerical form. Some analytical findings are included too, in particular, for fast moving solitons and the results produced by the variational approximation. Moreover, dissipative solitons are briefly considered, which are governed by the fractional complex Ginzburg–Landau equation.
AVATAR: A Parallel Corpus for Java-Python Program Translation

Program translation refers to migrating source code from one programming language to another. It has tremendous practical value in software development, as porting software across languages is time-consuming and costly. Automating program translation is of paramount importance in software migration, and recently researchers explored unsupervised approaches due to the unavailability of parallel corpora. However, the availability of pre-trained language models for programming languages enables supervised fine-tuning with a small number of labeled examples. Therefore, we present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python. AVATAR is collected from competitive programming sites, online platforms, and open-source repositories. Furthermore, AVATAR includes unit tests for 250 examples to facilitate functional correctness evaluation. We benchmark several pre-trained language models fine-tuned on AVATAR. Experiment results show that the models lack in generating functionally accurate code.
A Realistic Study of Auto-regressive Language Models for Named Entity Typing and Recognition

Despite impressive results of language models for named entity recognition (NER), their generalization to varied textual genres, a growing entity type set, and new entities remains a challenge. Collecting thousands of annotations in each new case for training or ﬁne-tuning is expensive and time-consuming. In contrast, humans can easily identify named entities given some simple instructions. Inspired by this, we challenge the reliance on large datasets and study pre-trained language models for NER in a meta-learning setup. First, we test named entity typing (NET) in a zero-shot transfer scenario. Then, we perform NER by giving few examples at inference. We propose a method to select seen and rare / unseen names when having access only to the pre-trained model and report results on these groups. The results show: auto-regressive language models as meta-learners can perform NET and NER fairly well especially for regular or seen names; name irregularity when often present for a certain entity type can become an effective exploitable cue; names with words foreign to the model have the most negative impact on results; the model seems to rely more on name than context cues in few-shot NER.
StackMix and Blot Augmentations for Handwritten Text Recognition

This paper proposes a handwritten text recognition(HTR) system that outperforms current state-of-the-artmethods. The comparison was carried out on three of themost frequently used in HTR task datasets, namely Ben-tham, IAM, and Saint Gall. In addition, the results on tworecently presented datasets, Peter the Greats manuscriptsand HKR Dataset, are provided.The paper describes the architecture of the neural net-work and two ways of increasing the volume of train-ing data: augmentation that simulates strikethrough text(HandWritten Blots) and a new text generation method(StackMix), which proved to be very effective in HTR tasks.StackMix can also be applied to the standalone task of gen-erating handwritten text based on printed text.
Decoder for the Triangular Color Code by Matching on a Möbius Strip

The color code is remarkable for its ability to perform fault-tolerant logic gates. This motivates the design of practical decoders that minimise the resource cost of color-code quantum computation. Here we propose a decoder for the planar color code with a triangular boundary where we match syndrome defects on a nontrivial manifold that has the topology of a M\"{o}bius strip. A basic implementation of our decoder used on the color code with hexagonal lattice geometry demonstrates a logical failure rate that is competitive with the optimal performance of the surface code, $\sim p^{\alpha \sqrt{n}}$, with $\alpha \approx 6 / 7 \sqrt{3} \approx 0.5$, error rate $p$, and $n$ the code length. Furthermore, by exhaustively testing over five billion error configurations, we find that a modification of our decoder that manually compares inequivalent recovery operators can correct all errors of weight $\le (d-1) /2$ for codes with distance $d \le 13$. Our decoder is derived using relations among the stabilizers that preserve global conservation laws at the lattice boundary. We present generalisations of our method to depolarising noise and fault-tolerant error correction, as well as to Majorana surface codes, higher-dimensional color codes and single-shot error correction.
Retrieval Augmented Code Generation and Summarization

Software developers write a lot of source code and documentation during software development. Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them. To mimic developers' code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models. REDCODER has a couple of uniqueness. First, it extends the state-of-the-art dense retrieval technique to search for relevant code or summaries. Second, it can work with retrieval databases that include unimodal (only code or natural language description) or bimodal instances (code-description pairs). We conduct experiments and extensive analysis on two benchmark datasets of code generation and summarization in Java and Python, and the promising results endorse the effectiveness of our proposed retrieval augmented framework.
On intermediate statistics across many-body localization transition

The level statistics in the transition between delocalized and localized phases of many body interacting systems is considered. We recall the joint probability distribution for eigenvalues resulting from the statistical mechanics for energy level dynamics as introduced by Pechukas and Yukawa. The resulting single parameter analytic distribution is probed numerically via Monte Carlo method. The resulting higher order spacing ratios are compared with data coming from different quantum many body systems. It is found that this Pechukas–Yukawa distribution compares favorably with β–Gaussian ensemble—a single parameter model of level statistics proposed recently in the context of disordered many-body systems. Moreover, the Pechukas–Yukawa distribution is also only slightly inferior to the two-parameter β–h ansatz shown earlier to reproduce level statistics of physical systems remarkably well.
Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization

Encoder-decoder models have achieved remarkable success in abstractive text summarization, which aims to compress one or more documents into a shorter version without the loss of the essential content. Unfortunately, these models mostly suffer a discrepancy between training and inference, i.e., the exposure bias problem. During the training stage, with teacher forcing these models are optimized to maximize the likelihood of the gold summary given the gold summary tokens as input to the decoder, while at inference the given tokens are replaced by the generated tokens. Consequently, low-quality summaries are very likely to be generated. To remedy this problem, we propose to leverage contrastive learning to decrease the likelihood of these low-quality summaries, and meanwhile increase the likelihood of the gold summary. Since our solution expands the states that the model perceives during training, we expect that the exposure bias problem can be alleviated. We experimentally demonstrate that our method effectively improves the performance of the state-of-the-art model on different datasets.
Charging Dynamics of Electrical Double Layers Inside Porous Electrodes: Predicting the Effect of Pore-Size Distribution

Porous electrodes are found in energy storage devices such as supercapacitors and pseudocapacitors. However, the effect of electrode-pore-size distribution over their energy storage properties remains unclear. Here, we develop a model for the charging of electrical double layers inside a cylindrical pore for arbitrary pore size. We assume small applied potentials and perform a regular perturbation analysis to predict the evolution of electrical potential and ion concentrations in both the radial and axial directions. We validate our perturbation model with direct numerical simulations of the Poisson-Nernst-Planck equations, and obtain quantitative agreement between the two approaches for small and moderate potentials. Our analysis yields two main characteristic features of arbitrary pore size: i) a monotonic decrease of the charging timescale with an increase in relative pore size (pore size relative to Debye length); ii) a region of large potential gradients at the mouth of the pore due to charge conservation. We develop a modified transmission circuit model that captures the effect of arbitrary pore sizes and demonstrate that a time-dependent interfacial capacitance needs to be included in the circuit. We also derive expressions for effective capacitance and charging timescale as a function of pore-size distribution, and show that the capacitance and charging timescale increase for narrower and less polydisperse distributions, resulting in a gain of energy density at a constant power density. Overall, our results advance the fundamental understanding of electrical-double-layer charging and will be useful for the electrode design of energy storage devices.
Robust High-Resolution Video Matting with Temporal Guidance

We introduce a robust, real-time, high-resolution human video matting method that achieves new state-of-the-art performance. Our method is much lighter than previous approaches and can process 4K at 76 FPS and HD at 104 FPS on an Nvidia GTX 1080Ti GPU. Unlike most existing methods that perform video matting frame-by-frame as independent images, our method uses a recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and matting quality. Furthermore, we propose a novel training strategy that enforces our network on both matting and segmentation objectives. This significantly improves our model’s robustness. Our method does not require any auxiliary inputs such as a trimap or a pre-captured background image, so it can be widely applied to existing human matting applications. Our code is available at https://peterl1n.github. io/RobustVideoMatting/
ICM-3D: Instantiated Category Modeling for 3D Instance Segmentation

Separating 3D point clouds into individual instances is an important task for 3D vision. It is challenging due to the unknown and varying number of instances in a scene. Existing deep learning based works focus on a two-step pipeline: first learn a feature embedding and then cluster the points. Such a two-step pipeline leads to disconnected intermediate objectives. In this paper, we propose an integrated reformulation of 3D instance segmentation as a per-point classification problem. We propose ICM-3D, a single-step method to segment 3D instances via instantiated categorization. The augmented category information is automatically constructed from 3D spatial positions. We conduct extensive experiments to verify the effectiveness of ICM-3D and show that it obtains inspiring performance across multiple frameworks, backbones and benchmarks.
Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslation

The state of art natural language processing systems relies on sizable training datasets to achieve high performance. Lack of such datasets in the specialized low resource domains lead to suboptimal performance. In this work, we adapt backtranslation to generate high quality and linguistically diverse synthetic data for low-resource named entity recognition. We perform experiments on two datasets from the materials science (MaSciP) and biomedical (S800) domains. The empirical results demonstrate the effectiveness of our proposed augmentation strategy, particularly in the low-resource scenario.
Scalable End-to-End Training of Knowledge Graph-Enhanced Aspect Embedding for Aspect Level Sentiment Analysis

Aspect level sentiment classiﬁcation (ALSC) is a difﬁcult problem with state of the art models showing less than 80% macro-F1 score on benchmark datasets. Existing models do not incorporate information on aspect-aspect relations in knowledge graphs (KGs), e.g. DB-pedia. Two main challenges stem from inaccurate disambiguation of aspects to KG entities, and inability to learn aspect representations from the large KGs in a joint training with ALSC models. We propose a two-level global-local entity embedding scheme which allows efﬁcient joint training of KG based aspect embeddings and ALSC models. A novel incorrect disambiguation detection technique addresses the problem of inaccuracy in aspect disambiguation. The proposed methods show a consistent improvement of 2 . 5 − 4 . 1 percentage points, over the recent BERT-based base-lines.
Astrophysical parameters and dynamical evolution of open clusters:
 NGC
 2587, Collinder 268 and Melotte 72, and Pismis 7

We determined astrophysical and dynamical parameters of the open clusters (OCs) NGC 2587, Collinder 268 (Col 268), Melotte 72 (Mel 72), and Pismis 7 from Gaia DR2 photometric/astrometric data and a new technique, fitCMD. fitCMD provides (Z, Age(Gyr)) as (0.025, 0.45) for NGC 2587, (0.0025, 0.5) for Col. 268, (0.011, 1.25) for Mel 72, and (0.008, 1.00) for Pismis 7, respectively. As compared to Gaia DR2 distances, the obtained photometric distances from fitCMD provide somewhat close distances. For NGC 2587 and Mel 72, both distances are in good concordance. Except for NGC 2587, the ages of the remaining OCs are higher than their relaxation times, which suggests that they are dynamically relaxed. NGC 2587 did not undergo dynamical evolution. Mel 72 and Pismis 7 with relatively flat MF slopes indicate signs of a somewhat advanced dynamical evolution, in the sense that they appear to have lost a significant fraction of their low-mass stars to the field. Pismis 7's negative/flat MFs indicates that its high mass stars slightly outnumber its low mass ones. Given its mild dynamical evolution, the high mass stars move towards the central region, while low-mass stars are continually being lost to the field. Col 268 presents small dimensions which suggest a primordial origin. The outer parts of Mel 72 and Pismis 7 - with large cluster radii expand with time, while Mel 72's core contracts because of dynamical relaxation (Figs. 10e--f).
Learning the Subsystem of Local Planning for Autonomous Racing

The problem of autonomous racing is to navigate through a race course as quickly as possible while not colliding with any obstacles. We approach the autonomous racing problem with the added constraint of not maintaining an updated obstacle map of the environment. Several current approaches to this problem use end-to-end learning systems where an agent replaces the entire navigation pipeline. This paper presents a hierarchical planning architecture that combines a high level planner and path following system with a reinforcement learning agent that learns that subsystem of obstacle avoidance. The novel “modification planner” uses the path follower to track the global plan and the deep reinforcement learning agent to modify the references generated by the path follower to avoid obstacles. Importantly, our architecture does not require an updated obstacle map and only 10 laser range finders to avoid obstacles. The modification planner is evaluated in the context of $\mathbf{F}1/10^{th}$ autonomous racing and compared to a end-to-end learning baseline, the Follow the Gap Method and an optimisation based planner. The results show that the modification planner can achieve faster average times compared to the baseline end-to-end planner and a 94% success rate which is similar to the baseline.