Miracle-dz/UniFAR-FTUs huggingface dataset:
PTM4Tag: Sharpening Tag Recommendation of Stack Overflow Posts with Pre-trained Models

Stack Overflow is often viewed as one of the most influential Software Question & Answer (SQA) websites, containing millions of programming-related questions and answers. Tags play a critical role in efficiently structuring the contents in Stack Overflow and are vital to support a range of site operations, e.g., querying relevant contents. Poorly selected tags often introduce extra noise and redundancy, which raises problems like tag synonym and tag explosion. Thus, an automated tag recommendation technique that can accurately recommend high-quality tags is desired to alleviate the problems mentioned above. Inspired by the recent success of pre-trained language models (PTMs) in natural language processing (NLP), we present PTM4Tag, a tag recommendation framework for Stack Overflow posts that utilize PTMs with a triplet architecture, which models the components of a post, i.e., Title, Description, and Code with independent language models. To the best of our knowledge, this is the first work that leverages PTMs in the tag recommendation task of SQA sites. We comparatively evaluate the performance of PTM4Tag based on five popular pre-trained models: BERT, RoBERTa, ALBERT, CodeBERT, and BERTOverflow. Our results show that leveraging CodeBERT, a software engineering (SE) domain-specific PTM in PTM4Tag achieves the best performance among the five considered PTMs and outperforms the state-of-the-art Convolutional Neural Network-based approach by a large margin in terms of average Precision@k, Recall@k, and F1-score@k. We conduct an ablation study to quantify the contribution of a post's constituent components (Title, Description, and Code Snippets) to the performance of PTM4Tag. Our results show that Title is the most important in predicting the most relevant tags, and utilizing all the components achieves the best performance.
Teaching language models to support answers with verified quotes

Recent large language models often answer factual questions correctly. But users can't trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train"open-book"QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model's response is found to be high-quality 80\% of the time on this Natural Questions subset, and 67\% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90\% and 80\% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows why citation is only one part of an overall strategy for safety and trustworthiness: not all claims supported by evidence are true.
Reliably Reproducing Machine-Checked Proofs with the Coq Platform

The Coq Platform is a continuously developed distribution of the Coq proof assistant together with commonly used libraries, plugins, and external tools useful in Coq-based formal verification projects. The Coq Platform enables reproducing and extending Coq artifacts in research, education, and industry, e.g., formalized mathematics and verified software systems. In this paper, we describe the background and motivation for the Platform, and outline its organization and development process. We also compare the Coq Platform to similar distributions and processes in the proof assistant community, such as for Isabelle and Lean, and in the wider open source software community.
Fast Bayesian Coresets via Subsampling and Quasi-Newton Refinement

Bayesian coresets approximate a posterior distribution by building a small weighted subset of the data points. Any inference procedure that is too computationally expensive to be run on the full posterior can instead be run inexpensively on the coreset, with results that approximate those on the full data. However, current approaches are limited by either a significant run-time or the need for the user to specify a low-cost approximation to the full posterior. We propose a Bayesian coreset construction algorithm that first selects a uniformly random subset of data, and then optimizes the weights using a novel quasi-Newton method. Our algorithm is a simple to implement, black-box method, that does not require the user to specify a low-cost posterior approximation. It is the first to come with a general high-probability bound on the KL divergence of the output coreset posterior. Experiments demonstrate that our method provides significant improvements in coreset quality against alternatives with comparable construction times, with far less storage cost and user input required.
Magnetic fields in star formation: from clouds to cores

In this chapter we review recent advances in understanding the roles that magnetic fields play throughout the star formation process, gained through observations and simulations of molecular clouds, the dense, star-forming phase of the magnetised, turbulent interstellar medium (ISM). Recent results broadly support a picture in which the magnetic fields of molecular clouds transition from being gravitationally sub-critical and near equipartition with turbulence in low-density cloud envelopes, to being energetically sub-dominant in dense, gravitationally unstable star-forming cores. Magnetic fields appear to play an important role in the formation of cloud substructure by setting preferred directions for large-scale gas flows in molecular clouds, and can direct the accretion of material onto star-forming filaments and hubs. Low-mass star formation may proceed in environments close to magnetic criticality; high-mass star formation remains less well-understood, but may proceed in more supercritical environments. The interaction between magnetic fields and (proto)stellar feedback may be particularly important in setting star formation efficiency. We also review a range of widely-used techniques for quantifying the dynamic importance of magnetic fields, concluding that better-calibrated diagnostics are required in order to use the spectacular range of forthcoming observations and simulations to quantify our emerging understanding of how magnetic fields influence the outcome of the star formation process.
Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

In recent years, large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, making it very challenging to incorporate contexualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tagging. The discriminative encoder of CRF-AE can straightforwardly incorporate ELMo word representations. Moreover, inspired by feature-rich HMM, we reintroduce hand-crafted features into the decoder of CRF-AE. Finally, experiments clearly show that our model outperforms previous state-of-the-art models by a large margin on Penn Treebank and multilingual Universal Dependencies treebank v2.0.
Self-Ensemble Adversarial Training for Improved Robustness

Due to numerous breakthroughs in real-world applications brought by machine intelligence, deep neural networks (DNNs) are widely employed in critical applications. However, predictions of DNNs are easily manipulated with imperceptible adversarial perturbations, which impedes the further deployment of DNNs and may result in profound security and privacy implications. By incorporating adversarial samples into the training data pool, adversarial training is the strongest principled strategy against various adversarial attacks among all sorts of defense methods. Recent works mainly focus on developing new loss functions or regularizers, attempting to find the unique optimal point in the weight space. But none of them taps the potentials of classifiers obtained from standard adversarial training, especially states on the searching trajectory of training. In this work, we are dedicated to the weight states of models through the training process and devise a simple but powerful \emph{Self-Ensemble Adversarial Training} (SEAT) method for yielding a robust classifier by averaging weights of history models. This considerably improves the robustness of the target model against several well known adversarial attacks, even merely utilizing the naive cross-entropy loss to supervise. We also discuss the relationship between the ensemble of predictions from different adversarially trained models and the prediction of weight-ensembled models, as well as provide theoretical and empirical evidence that the proposed self-ensemble method provides a smoother loss landscape and better robustness than both individual models and the ensemble of predictions from different classifiers. We further analyze a subtle but fatal issue in the general settings for the self-ensemble model, which causes the deterioration of the weight-ensembled method in the late phases.
How does the pre-training objective affect what large language models learn about linguistic properties?

Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives such as MLM should help BERT to acquire better linguistic knowledge compared to other non-linguistically motivated objectives that are not intuitive or hard for humans to guess the association between the input and the label to be predicted. To this end, we pre-train BERT with two linguistically motivated objectives and three non-linguistically motivated ones. We then probe for linguistic characteristics encoded in the representation of the resulting models. We find strong evidence that there are only small differences in probing performance between the representations learned by the two different types of objectives. These surprising results question the dominant narrative of linguistically informed pre-training.
Equivalence of mean-field avalanches and branching diffusions: From the Brownian force model to the super-Brownian motion

We point out that the mean-ﬁeld theory of avalanches in the dynamics of elastic interfaces, the so-called Brownian force model (BFM) developed recently in non-equilibrium statistical physics, is equivalent to the so-called super-Brownian motion (SBM) developed in probability theory, a continuum limit of branching processes related to space-embedded Galton-Watson trees. In particular the exact solvabil-ity property recently (re-)discovered from the ﬁeld theory in mean-ﬁeld avalanches (the ”instanton equation”) maps onto the so-called Dawson-Watanabe 1968 duality property. In the light of this correspondence we compare the results obtained independently in the two ﬁelds, and transport some of them from one ﬁeld to the other. In particular, we discuss a scaling limit of the branching Brownian motion which maps onto the continuum ﬁeld theory of mean-ﬁeld avalanches.
REALY: Rethinking the Evaluation of 3D Face Reconstruction

The evaluation of 3D face reconstruction results typically relies on a rigid shape alignment between the estimated 3D model and the ground-truth scan. We observe that aligning two shapes with different reference points can largely affect the evaluation results. This poses difficulties for precisely diagnosing and improving a 3D face reconstruction method. In this paper, we propose a novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topology-consistent meshes. Our approach performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors. The fine-grained, region-wise evaluation results provide us detailed understandings about the performance of state-of-the-art 3D face reconstruction methods. For example, our experiments on single-image based reconstruction methods reveal that DECA performs the best on nose regions, while GANFit performs better on cheek regions. Besides, a new and high-quality 3DMM basis, HIFI3D++, is further derived using the same procedure as we construct REALY to align and retopologize several 3D face datasets. We will release REALY, HIFI3D++, and our new evaluation pipeline at https://realy3dface.com.
Peaceable Self-Stabilizing Byzantine Pulse Synchronization

For reaching fast and efficient self-stabilizing Byzantine pulse synchronization (SSBPS) upon the bounded-delay message-passing networks, we consider the peaceable SSBPS problem where the resource occupation in the stabilized system is required to be as sparse as possible and the stabilization of the system is required to be as fast as possible. To solve it, the decoupled absorption process and emergency process are investigated under a general framework. The constant-time two-stage absorption process and more than one kind of emergency process are provided in integrating the merits of temporal trails, approximate agreements, deterministic and randomized Byzantine agreements, and self-stabilizing protocols. With this, not only SSBPS but self-stabilizing Byzantine clock synchronization can be fast established and efficiently maintained. In the optimal-resilient basic solutions, the deterministic linear stabilization time and the stabilized resource occupation are all optimized to those of the underlying primitives, which is optimal in the presence of Byzantine faults under the classical static adversary. In the hybrid solutions, faster stabilization can be expected with a worst-case deterministic stabilization time. In all solutions, the stabilized resource occupation is at most at the order of approximate agreement, which is a good property in considering real-world ultra-high-reliable hard-real-time applications where pulse synchronization is expected to be established before all upper-layer functions and to be efficiently maintained when the upper-layer functions are in service.
Importance Sampling Placement in Off-Policy Temporal-Difference Methods

A central challenge to applying many off-policy reinforcement learning algorithms to real world problems is the variance introduced by importance sampling. In off-policy learning, the agent learns about a different policy than the one being executed. To account for the difference importance sampling ratios are often used, but can increase variance in the algorithms and reduce the rate of learning. Several variations of importance sampling have been proposed to reduce variance, with per-decision importance sampling being the most popular. However, the update rules for most off-policy algorithms in the literature depart from per-decision importance sampling in a subtle way; they correct the entire TD error instead of just the TD target. In this work, we show how this slight change can be interpreted as a control variate for the TD target, reducing variance and improving performance. Experiments over a wide range of algorithms show this subtle modification results in improved performance.
On the Combinatorial Diameters of Parallel and Series Connections

The investigation of combinatorial diameters of polyhedra is a classical topic in linear programming due to its connection with the possibility of an efficient pivot rule for the simplex method. We are interested in the diameters of polyhedra formed from the so-called parallel or series connection of oriented matroids: oriented matroids are the natural way to connect representable matroid theory with the combinatorics of linear programming, and these connections are fundamental operations for the construction of more complicated matroids from elementary matroid blocks. We prove that, for polyhedra whose combinatorial diameter satisfies the Hirsch-conjecture bound regardless of the right-hand sides in a standard-form description, the diameters of their parallel or series connections remain small in the Hirsch-conjecture bound. These results are a substantial step toward devising a diameter bound for all polyhedra defined through totally-unimodular matrices based on Seymour's famous decomposition theorem. Our proof techniques and results exhibit a number of interesting features. While the parallel connection leads to a bound that adds just a constant, for the series connection one has to linearly take into account the maximal value in a specific coordinate of any vertex. Our proofs also require a careful treatment of non-revisiting edge walks in degenerate polyhedra, as well as the construction of edge walks that may take a `detour' to facets that satisfy the non-revisiting conjecture when the underlying polyhedron may not.
Tripartite entanglement and matrix inversion quantum algorithm

Mi-Ra Hwang, MuSeong Kim, Eylee Jung, Chang-Yong Woo, and DaeKil Park1,2∗ Department of Electronic Engineering, Kyungnam University, Changwon 631-701, Korea Department of Physics, Kyungnam University, Changwon 631-701, Korea Abstract The role of entanglement is discussed in the Harrow-Hassidim-Lloyd (HHL) algorithm. We compute all tripartite entanglement at every steps of the HHL algorithm. The tripartite entanglement is generated in the first quantum phase estimation (QPE) step. However, it turns out that amount of the generated entanglement is not maximal except very rare cases. In the second rotation step some tripartite entanglement is annihilated. Thus, the net tripartite entanglement is diminished. At the final inverse-QPE step the matrix inversion task is completed at the price of complete annihilation of the entanglement. An implication of this result is discussed.
On the Consistency of Max-Margin Losses

The foundational concept of Max-Margin in machine learning is ill-posed for output spaces with more than two labels such as in structured prediction. In this paper, we show that the Max-Margin loss can only be consistent to the classification task under highly restrictive assumptions on the discrete loss measuring the error between outputs. These conditions are satisfied by distances defined in tree graphs, for which we prove consistency, thus being the first losses shown to be consistent for Max-Margin beyond the binary setting. We finally address these limitations by correcting the concept of Max-Margin and introducing the Restricted-Max-Margin, where the maximization of the loss-augmented scores is maintained, but performed over a subset of the original domain. The resulting loss is also a generalization of the binary support vector machine and it is consistent under milder conditions on the discrete loss.
Percolation and Epidemic Processes in One-Dimensional Small-World Networks

We obtain tight thresholds for bond percolation on one-dimensional small-world graphs, and apply such results to obtain tight thresholds for the \emph{Independent Cascade} process and the \emph{Reed-Frost} process in such graphs. These are the first fully rigorous results establishing a phase transition for bond percolation and SIR epidemic processes in small-world graphs. Although one-dimensional small-world graphs are an idealized and unrealistic network model, a number of realistic qualitative epidemiological phenomena emerge from our analysis, including the epidemic spread through a sequence of local outbreaks, the danger posed by random connections, and the effect of super-spreader events.
Constraining ultra-high-energy cosmic ray composition through cross-correlations

The chemical composition of the highest end of the ultra-high-energy cosmic ray spectrum is very hard to measure experimentally, and to this day it remains mostly unknown. Since the trajectories of ultra-high-energy cosmic rays are deflected in the magnetic field of the Galaxy by an angle that depends on their atomic number Z, it could be possible to indirectly measure Z by quantifying the amount of such magnetic deflections. In this paper we show that, using the angular harmonic cross-correlation between ultra-high-energy cosmic rays and galaxies, we could effectively distinguish different atomic numbers with current data. As an example, we show how, if Z = 1, the cross-correlation can exclude a 39% fraction of Fe56 nuclei at 2σ for rays above 100 EeV.
CrossBeam: Learning to Search in Bottom-Up Program Synthesis

Many approaches to program synthesis perform a search within an enormous space of programs to find one that satisfies a given specification. Prior works have used neural models to guide combinatorial search algorithms, but such approaches still explore a huge portion of the search space and quickly become intractable as the size of the desired program increases. To tame the search space blowup, we propose training a neural model to learn a hands-on search policy for bottom-up synthesis, instead of relying on a combinatorial search algorithm. Our approach, called CrossBeam, uses the neural model to choose how to combine previously-explored programs into new programs, taking into account the search history and partial program executions. Motivated by work in structured prediction on learning to search, CrossBeam is trained on-policy using data extracted from its own bottom-up searches on training tasks. We evaluate CrossBeam in two very different domains, string manipulation and logic programming. We observe that CrossBeam learns to search efficiently, exploring much smaller portions of the program space compared to the state-of-the-art.
Short Text Topic Modeling: Application to tweets about Bitcoin

Understanding the semantic of a collection of texts is a challenging task. Topic models are probabilistic models that aims at extracting"topics"from a corpus of documents. This task is particularly difficult when the corpus is composed of short texts, such as posts on social networks. Following several previous research papers, we explore in this paper a set of collected tweets about bitcoin. In this work, we train three topic models and evaluate their output with several scores. We also propose a concrete application of the extracted topics.