Prompt: Open-source toolkit for training custom models for NLP tasks like translation and summarization using PyTorch

Output: Open-source toolkit for training custom models for NLP tasks like translation and summarization using PyTorch

Output:
Title: fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Abstract: fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto


Prompt: Open-source toolkit for training custom models for NLP tasks like translation and summarization using PyTorch

Output: Open-source toolkit for training custom models for NLP tasks like translation and summarization using PyTorch

Output:
Title: OpenNMT: Open-Source Toolkit for Neural Machine Translation

Abstract: We describe an open-source toolkit for neural machine translation (NMT). The toolkit prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques.


Prompt: Research on leveraging knowledge graphs in protein pretraining models to enhance performance in protein-protein interaction and function prediction.

Output: Research on leveraging knowledge graphs in protein pretraining models to enhance performance in protein-protein interaction and function prediction.

Output:
Title: OntoProtein: Protein Pretraining With Gene Ontology Embedding

Abstract: Self-supervised protein language models have proved their effectiveness in learning the proteins representations. With the increasing computational power, current protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve remarkable improvement. However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better protein representations. We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge. In this work, we propose OntoProtein, the first general framework that makes use of structure in GO (Gene Ontology) into protein pre-training models. We construct a novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph. We propose novel contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and protein embedding during pre-training. Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction. Code and datasets are available in https://github.com/zjunlp/OntoProtein.


Prompt: Research on leveraging knowledge graphs in protein pretraining models to enhance performance in protein-protein interaction and function prediction.

Output: Research on leveraging knowledge graphs in protein pretraining models to enhance performance in protein-protein interaction and function prediction.

Output:
Title: OntoProtein: Protein Pretraining With Gene Ontology Embedding

Abstract: Self-supervised protein language models have proved their effectiveness in learning the proteins representations. With the increasing computational power, current protein language models pre-trained with millions of diverse sequences can advance the parameter scale from million-level to billion-level and achieve remarkable improvement. However, those prevailing approaches rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better protein representations. We argue that informative biology knowledge in KGs can enhance protein representation with external knowledge. In this work, we propose OntoProtein, the first general framework that makes use of structure in GO (Gene Ontology) into protein pre-training models. We construct a novel large-scale knowledge graph that consists of GO and its related proteins, and gene annotation texts or protein sequences describe all nodes in the graph. We propose novel contrastive learning with knowledge-aware negative sampling to jointly optimize the knowledge graph and protein embedding during pre-training. Experimental results show that OntoProtein can surpass state-of-the-art methods with pre-trained protein language models in TAPE benchmark and yield better performance compared with baselines in protein-protein interaction and protein function prediction. Code and datasets are available in https://github.com/zjunlp/OntoProtein.


Prompt: Looking for a research paper that introduces a new method using text-guided attention to improve polyp segmentation in colonoscopy images.

Output: Looking for a research paper that introduces a new method using text-guided attention to improve polyp segmentation in colonoscopy images.

Output:
Title: TGANet: Text-guided attention for improved polyp segmentation

Abstract: Colonoscopy is a gold standard procedure but is highly operator-dependent. Automated polyp segmentation, a precancerous precursor, can minimize missed rates and timely treatment of colon cancer at an early stage. Even though there are deep learning methods developed for this task, variability in polyp size can impact model training, thereby limiting it to the size attribute of the majority of samples in the training dataset that may provide sub-optimal results to differently sized polyps. In this work, we exploit size-related and polyp number-related features in the form of text attention during training. We introduce an auxiliary classification task to weight the text-based embedding that allows network to learn additional feature representations that can distinctly adapt to differently sized polyps and can adapt to cases with multiple polyps. Our experimental results demonstrate that these added text embeddings improve the overall performance of the model compared to state-of-the-art segmentation methods. We explore four different datasets and provide insights for size-specific improvements. Our proposed text-guided attention network (TGANet) can generalize well to variable-sized polyps in different datasets.


Prompt: Looking for a research paper that introduces a new method using text-guided attention to improve polyp segmentation in colonoscopy images.

Output: Looking for a research paper that introduces a new method using text-guided attention to improve polyp segmentation in colonoscopy images.

Output:
Title: The W_t Transcendental Function and Quantum Mechanical Applications

Abstract: We discuss the function wt(x) defined via the implicit equation wt(x)*tan[wt(x)]=x which appears in certain quantum mechanical and field theoretic applications. We investigate its analytic structure, develop series expansions for both small and large x, and provide various techniques for its numerical evaluation in the complex plane.


Prompt: How can we improve the interpretability of deep learning models using simpler models?

Output: How can we improve the interpretability of deep learning models using simpler models?

Output:
Title: A Detailed Study of Interpretability of Deep Neural Network based Top Taggers

Abstract: Recent developments in the methods of explainable AI (XAI) allow researchers to explore the inner workings of deep neural networks (DNNs), revealing crucial information about input-output relationships and realizing how data connects with machine learning models. In this paper we explore interpretability of DNN models designed to identify jets coming from top quark decay in high energy proton-proton collisions at the Large Hadron Collider (LHC). We review a subset of existing top tagger models and explore different quantitative methods to identify which features play the most important roles in identifying the top jets. We also investigate how and why feature importance varies across different XAI metrics, how correlations among features impact their explainability, and how latent space representations encode information as well as correlate with physically meaningful quantities. Our studies uncover some major pitfalls of existing XAI methods and illustrate how they can be overcome to obtain consistent and meaningful interpretation of these models. We additionally illustrate the activity of hidden layers as Neural Activation Pattern (NAP) diagrams and demonstrate how they can be used to understand how DNNs relay information across the layers and how this understanding can help to make such models significantly simpler by allowing effective model reoptimization and hyperparameter tuning. These studies not only facilitate a methodological approach to interpreting models but also unveil new insights about what these models learn. Incorporating these observations into augmented model design, we propose the Particle Flow Interaction Network (PFIN) model and demonstrate how interpretability-inspired model augmentation can improve top tagging performance.


Prompt: How can we improve the interpretability of deep learning models using simpler models?

Output: How can we improve the interpretability of deep learning models using simpler models?

Output:
Title: Interpretability Illusions in the Generalization of Simplified Models

Abstract: A common method to study deep learning systems is to use simplified model representations--for example, using singular value decomposition to visualize the model's hidden states in a lower dimensional space. This approach assumes that the results of these simplifications are faithful to the original model. Here, we illustrate an important caveat to this assumption: even if the simplified representations can accurately approximate the full model on the training set, they may fail to accurately capture the model's behavior out of distribution. We illustrate this by training Transformer models on controlled datasets with systematic generalization splits, including the Dyck balanced-parenthesis languages and a code completion task. We simplify these models using tools like dimensionality reduction and clustering, and then explicitly test how these simplified proxies match the behavior of the original model. We find consistent generalization gaps: cases in which the simplified proxies are more faithful to the original model on the in-distribution evaluations and less faithful on various tests of systematic generalization. This includes cases where the original model generalizes systematically but the simplified proxies fail, and cases where the simplified proxies generalize better. Together, our results raise questions about the extent to which mechanistic interpretations derived using tools like SVD can reliably predict what a model will do in novel situations.


Prompt: is bloodborne in the same universe as dark souls?

Output: is bloodborne in the same universe as dark souls?

Output:
Math Stackexchange

Q: Solving $\arcsin(1-x)-2\arcsin(x)=\pi/2$ \begin{eqnarray*}
\arcsin(1-x)-2\arcsin(x) & = & \frac{\pi}{2}\\
1-x & = & \sin\left(\frac{\pi}{2}+2\arcsin(x)\right)\\
 & = & \cos\left(2\arcsin(x)\right)\\
 & = & 1-2\left(\sin\left(\arcsin(x)\right)\right)^{2}\\
 & = & 1-2x^{2}\\
x & = & 2x^{2}\\
x\left(x-\frac{1}{2}\right) & = & 0
\end{eqnarray*}
So $x=0$ or $x=\frac{1}{2}$
But puttig $x=\frac{1}{2}$ in the original expression gives $-\frac {\pi} 4 \ne \frac \pi 2$
So, why do we get $x=-1/2$ as an answer?

A: In your first step you added an extra solution.
Since $\arcsin x$ must be smaller than $\pi/2$, the first line reads:
$$\arcsin(1-x)= \frac{\pi}{2}+2\arcsin(x) \le \frac{\pi}{2}$$
Thus, $x\le 0$ as well.
Now, by taking the $\sin$ of both sides, you took a function that was only defined up to $x=1$ (e.g. $\arcsin(x-1)$ ) and extended it to all reals (e.g $x-1$). Here is where you added the extra solution.


Prompt: is bloodborne in the same universe as dark souls?

Output: is bloodborne in the same universe as dark souls?

Output:
Stackoverflow Stackexchange

Q: Stop UIAlertView from dismissing Can you stop UIAlertView from dismissing?
I want based on what button is pressed, to either animate adding a UITextField, or dismiss it. But I can't add a text field after the alert is dismissed.

A: You should implement the UIAlertViewDelegate method :
- (void)alertView:(UIAlertView *)alertView willDismissWithButtonIndex:(NSInteger)buttonIndex

and based on the button index you can perform your custom action.
Otherwise you should either create your own alertview class or subclass UIAlertView.
If you choose to subclass you can override the method :
-(void)dismissWithClickedButtonIndex:(NSInteger)buttonIndex animated:(BOOL)animated

to avoid dismissing the alertView if a specific button is pressed


Prompt: How well do ChatGPT and GPT-4 perform on logical reasoning datasets?

Output: How well do ChatGPT and GPT-4 perform on logical reasoning datasets?

Output:
Title: Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Abstract: Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.


Prompt: How well do ChatGPT and GPT-4 perform on logical reasoning datasets?

Output: How well do ChatGPT and GPT-4 perform on logical reasoning datasets?

Output:
Title: Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Abstract: Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.


Prompt: Language models fine-tuning for non-language tasks without architectural changes

Output: Language models fine-tuning for non-language tasks without architectural changes

Output:
Title: Linear Transformers with Learnable Kernel Functions are Better In-Context Models

Abstract: Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset.


Prompt: Language models fine-tuning for non-language tasks without architectural changes

Output: Language models fine-tuning for non-language tasks without architectural changes

Output:
Title: ClusterFace: Joint Clustering and Classification for Set-Based Face Recognition

Abstract: Deep learning technology has enabled successful modeling of complex facial features when high quality images are available. Nonetheless, accurate modeling and recognition of human faces in real world scenarios `on the wild' or under adverse conditions remains an open problem. When unconstrained faces are mapped into deep features, variations such as illumination, pose, occlusion, etc., can create inconsistencies in the resultant feature space. Hence, deriving conclusions based on direct associations could lead to degraded performance. This rises the requirement for a basic feature space analysis prior to face recognition. This paper devises a joint clustering and classification scheme which learns deep face associations in an easy-to-hard way. Our method is based on hierarchical clustering where the early iterations tend to preserve high reliability. The rationale of our method is that a reliable clustering result can provide insights on the distribution of the feature space, that can guide the classification that follows. Experimental evaluations on three tasks, face verification, face identification and rank-order search, demonstrates better or competitive performance compared to the state-of-the-art, on all three experiments.


Prompt: Research on improving the performance and energy efficiency of neural networks by utilizing 8-bit precision.

Output: Research on improving the performance and energy efficiency of neural networks by utilizing 8-bit precision.

Output:
Title: Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks

Abstract: Deep neural networks are gaining in popularity as they are used to generate state-of-the-art results for a variety of computer vision and machine learning applications. At the same time, these networks have grown in depth and complexity in order to solve harder problems. Given the limitations in power budgets dedicated to these networks, the importance of low-power, low-memory solutions has been stressed in recent years. While a large number of dedicated hardware using different precisions has recently been proposed, there exists no comprehensive study of different bit precisions and arithmetic in both inputs and network parameters. In this work, we address this issue and perform a study of different bit-precisions in neural networks (from floating-point to fixed-point, powers of two, and binary). In our evaluation, we consider and analyze the effect of precision scaling on both network accuracy and hardware metrics including memory footprint, power and energy consumption, and design area. We also investigate training-time methodologies to compensate for the reduction in accuracy due to limited bit precision and demonstrate that in most cases, precision scaling can deliver significant benefits in design metrics at the cost of very modest decreases in network accuracy. In addition, we propose that a small portion of the benefits achieved when using lower precisions can be forfeited to increase the network size and therefore the accuracy. We evaluate our experiments, using three well-recognized networks and datasets to show its generality. We investigate the trade-offs and highlight the benefits of using lower precisions in terms of energy and memory footprint.


Prompt: Research on improving the performance and energy efficiency of neural networks by utilizing 8-bit precision.

Output: Research on improving the performance and energy efficiency of neural networks by utilizing 8-bit precision.

Output:
Title: Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability

Abstract: Tartan (TRT), a hardware accelerator for inference with Deep Neural Networks (DNNs), is presented and evaluated on Convolutional Neural Networks. TRT exploits the variable per layer precision requirements of DNNs to deliver execution time that is proportional to the precision p in bits used per layer for convolutional and fully-connected layers. Prior art has demonstrated an accelerator with the same execution performance only for convolutional layers. Experiments on image classification CNNs show that on average across all networks studied, TRT outperforms a state-of-the-art bit-parallel accelerator by 1:90x without any loss in accuracy while it is 1:17x more energy efficient. TRT requires no network retraining while it enables trading off accuracy for additional improvements in execution performance and energy efficiency. For example, if a 1% relative loss in accuracy is acceptable, TRT is on average 2:04x faster and 1:25x more energy efficient than a conventional bit-parallel accelerator. A Tartan configuration that processes 2-bits at time, requires less area than the 1-bit configuration, improves efficiency to 1:24x over the bit-parallel baseline while being 73% faster for convolutional layers and 60% faster for fully-connected layers is also presented.


Prompt: Research showing improved results in NLP tasks by integrating domain knowledge into BERT.

Output: Research showing improved results in NLP tasks by integrating domain knowledge into BERT.

Output:
Title: Knowledge Graph Fusion for Language Model Fine-tuning

Abstract: Language Models such as BERT have grown in popularity due to their ability to be pre-trained and perform robustly on a wide range of Natural Language Processing tasks. Often seen as an evolution over traditional word embedding techniques, they can produce semantic representations of text, useful for tasks such as semantic similarity. However, state-of-the-art models often have high computational requirements and lack global context or domain knowledge which is required for complete language understanding. To address these limitations, we investigate the benefits of knowledge incorporation into the fine-tuning stages of BERT. An existing K-BERT model, which enriches sentences with triplets from a Knowledge Graph, is adapted for the English language and extended to inject contextually relevant information into sentences. As a side-effect, changes made to K-BERT for accommodating the English language also extend to other word-based languages. Experiments conducted indicate that injected knowledge introduces noise. We see statistically significant improvements for knowledge-driven tasks when this noise is minimised. We show evidence that, given the appropriate task, modest injection with relevant, high-quality knowledge is most performant.


Prompt: Research showing improved results in NLP tasks by integrating domain knowledge into BERT.

Output: Research showing improved results in NLP tasks by integrating domain knowledge into BERT.

Output:
Title: K-BERT: Enabling Language Representation with Knowledge Graph

Abstract: Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.


Prompt: Looking for a paper about a multimodal machine translation model that uses visual and textual information.

Output: Looking for a paper about a multimodal machine translation model that uses visual and textual information.

Output:
Title: A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Abstract: We introduce a novel multimodal machine translation model that utilizes parallel visual and textual information. Our model jointly optimizes the learning of a shared visual-language embedding and a translator. The model leverages a visual attention grounding mechanism that links the visual semantics with the corresponding textual semantics. Our approach achieves competitive state-of-the-art results on the Multi30K and the Ambiguous COCO datasets. We also collected a new multilingual multimodal product description dataset to simulate a real-world international online shopping scenario. On this dataset, our visual attention grounding model outperforms other methods by a large margin.


Prompt: Looking for a paper about a multimodal machine translation model that uses visual and textual information.

Output: Looking for a paper about a multimodal machine translation model that uses visual and textual information.

Output:
Title: Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

Abstract: Recent research in the field of multimodal machine translation (MMT) has indicated that the visual modality is either dispensable or offers only marginal advantages. However, most of these conclusions are drawn from the analysis of experimental results based on a limited set of bilingual sentence-image pairs, such as Multi30k. In these kinds of datasets, the content of one bilingual parallel sentence pair must be well represented by a manually annotated image, which is different from the real-world translation scenario. In this work, we adhere to the universal multimodal machine translation framework proposed by Tang et al. (2022). This approach allows us to delve into the impact of the visual modality on translation efficacy by leveraging real-world translation datasets. Through a comprehensive exploration via probing tasks, we find that the visual modality proves advantageous for the majority of authentic translation datasets. Notably, the translation performance primarily hinges on the alignment and coherence between textual and visual contents. Furthermore, our results suggest that visual information serves a supplementary role in multimodal translation and can be substituted.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output: I am searching for a good and large-scale benchmark for testing the performance of text embeddings.

Output:
Title: MTEB: Massive Text Embedding Benchmark

Abstract: Text embeddings are commonly evaluated on a small set of datasets from a single task not covering their possible applications to other tasks. It is unclear whether state-of-the-art embeddings on semantic textual similarity (STS) can be equally well applied to other tasks like clustering or reranking. This makes progress in the field difficult to track, as various models are constantly being proposed without proper evaluation. To solve this problem, we introduce the Massive Text Embedding Benchmark (MTEB). MTEB spans 8 embedding tasks covering a total of 58 datasets and 112 languages. Through the benchmarking of 33 models on MTEB, we establish the most comprehensive benchmark of text embeddings to date. We find that no particular text embedding method dominates across all tasks. This suggests that the field has yet to converge on a universal text embedding method and scale it up sufficiently to provide state-of-the-art results on all embedding tasks. MTEB comes with open-source code and a public leaderboard at https://github.com/embeddings-benchmark/mteb.


Prompt: Efficient transformer models for multi-page document classification

Output: Efficient transformer models for multi-page document classification

Output:
Title: Revisiting Transformer-based Models for Long Document Classification

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.


Prompt: Efficient transformer models for multi-page document classification

Output: Efficient transformer models for multi-page document classification

Output:
Title: Revisiting Transformer-based Models for Long Document Classification

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.


Prompt: How to combine GPT-2 and BERT for better question generation and answering?

Output: How to combine GPT-2 and BERT for better question generation and answering?

Output:
Title: Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds

Abstract: Automatic question generation aims at the generation of questions from a context, with the corresponding answers being sub-spans of the given passage. Whereas, most of the methods mostly rely on heuristic rules to generate questions, more recently also neural network approaches have been proposed. In this work, we propose a variant of the self-attention Transformer network architectures model to generate meaningful and diverse questions. To this end, we propose an easy to use model consisting of the conjunction of the Transformer decoder GPT-2 model with Transformer encoder BERT for the downstream task for question answering. The model is trained in an end-to-end fashion, where the language model is trained to produce a question-answer-aware input representation that facilitates to generate an answer focused question. Our result of neural question generation from text on the SQuAD 1.1 dataset suggests that our method can produce semantically correct and diverse questions. Additionally, we assessed the performance of our proposed method for the downstream task of question answering. The analysis shows that our proposed generation & answering collaboration framework relatively improves both tasks and is particularly powerful in the semi-supervised setup. The results further suggest a robust and comparably lean pipeline facilitating question generation in the small-data regime.


Prompt: How to combine GPT-2 and BERT for better question generation and answering?

Output: How to combine GPT-2 and BERT for better question generation and answering?

Output:
Title: Radiation flux and spectrum in the Vaidya collapse model

Abstract: We consider the quantization of a massless scalar field, using the geometric optics approximation, in the background spacetime of a collapsing spherical self-similar Vaidya star, which forms a black hole or a naked singularity. We show that the outgoing radiation flux of the quantized scalar field diverges on the Cauchy horizon. The spectrum of the produced scalar partcles is non-thermal when the background develops a naked singularity. These results are analogous to those obtained for the scalar quantization on a self-similar dust cloud.


Prompt: Paper introducing a transformer-based model trained on multiple visual modalities achieving high accuracy.

Output: Paper introducing a transformer-based model trained on multiple visual modalities achieving high accuracy.

Output:
Title: Omnivore: A Single Model for Many Visual Modalities

Abstract: Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.


Prompt: Paper introducing a transformer-based model trained on multiple visual modalities achieving high accuracy.

Output: Paper introducing a transformer-based model trained on multiple visual modalities achieving high accuracy.

Output:
Title: Ab initio Molecular Dynamics Study of D_2 Desorption from Si(100)

Abstract: Ab initio molecular dynamics calculations of deuterium desorbing from Si(100) have been performed in order to monitor the energy redistribution among the hydrogen and silicon degrees of freedom during the desorption process. The calculations show that part of the potential energy at the transition state to desorption is transferred to the silicon lattice. The deuterium molecules leave the surface vibrationally hot and rotationally cold, in agreement with experiments; the mean kinetic energy, however, is larger than found in experiments.


Prompt: A novel network combining convolution and transformer operations for better medical image segmentation.

Output: A novel network combining convolution and transformer operations for better medical image segmentation.

Output:
Title: ConvFormer: Combining CNN and Transformer for Medical Image Segmentation

Abstract: Convolutional neural network (CNN) based methods have achieved great successes in medical image segmentation, but their capability to learn global representations is still limited due to using small effective receptive fields of convolution operations. Transformer based methods are capable of modelling long-range dependencies of information for capturing global representations, yet their ability to model local context is lacking. Integrating CNN and Transformer to learn both local and global representations while exploring multi-scale features is instrumental in further improving medical image segmentation. In this paper, we propose a hierarchical CNN and Transformer hybrid architecture, called ConvFormer, for medical image segmentation. ConvFormer is based on several simple yet effective designs. (1) A feed forward module of Deformable Transformer (DeTrans) is re-designed to introduce local information, called Enhanced DeTrans. (2) A residual-shaped hybrid stem based on a combination of convolutions and Enhanced DeTrans is developed to capture both local and global representations to enhance representation ability. (3) Our encoder utilizes the residual-shaped hybrid stem in a hierarchical manner to generate feature maps in different scales, and an additional Enhanced DeTrans encoder with residual connections is built to exploit multi-scale features with feature maps of different scales as input. Experiments on several datasets show that our ConvFormer, trained from scratch, outperforms various CNN- or Transformer-based architectures, achieving state-of-the-art performance.


Prompt: A novel network combining convolution and transformer operations for better medical image segmentation.

Output: A novel network combining convolution and transformer operations for better medical image segmentation.

Output:
Title: Rethinking Boundary Detection in Deep Learning Models for Medical Image Segmentation

Abstract: Medical image segmentation is a fundamental task in the community of medical image analysis. In this paper, a novel network architecture, referred to as Convolution, Transformer, and Operator (CTO), is proposed. CTO employs a combination of Convolutional Neural Networks (CNNs), Vision Transformer (ViT), and an explicit boundary detection operator to achieve high recognition accuracy while maintaining an optimal balance between accuracy and efficiency. The proposed CTO follows the standard encoder-decoder segmentation paradigm, where the encoder network incorporates a popular CNN backbone for capturing local semantic information, and a lightweight ViT assistant for integrating long-range dependencies. To enhance the learning capacity on boundary, a boundary-guided decoder network is proposed that uses a boundary mask obtained from a dedicated boundary detection operator as explicit supervision to guide the decoding learning process. The performance of the proposed method is evaluated on six challenging medical image segmentation datasets, demonstrating that CTO achieves state-of-the-art accuracy with a competitive model complexity.


Prompt: How do different constraints in storytelling tasks impact the author's linguistic style?

Output: How do different constraints in storytelling tasks impact the author's linguistic style?

Output:
Title: The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Abstract: A writer's style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.


Prompt: How do different constraints in storytelling tasks impact the author's linguistic style?

Output: How do different constraints in storytelling tasks impact the author's linguistic style?

Output:
Title: Limits on dynamically generated spin-orbit coupling: Absence of $l=1$ Pomeranchuk instabilities in metals

Abstract: An ordered state in the spin sector that breaks parity without breaking time-reversal symmetry, i.e., that can be considered as dynamically generated spin-orbit coupling, was proposed to explain puzzling observations in a range of different systems. Here we derive severe restrictions for such a state that follow from a Ward identity related to spin conservation. It is shown that $l=1$ spin-Pomeranchuk instabilities are not possible in non-relativistic systems since the response of spin-current fluctuations is entirely incoherent and non-singular. This rules out relativistic spin-orbit coupling as an emergent low-energy phenomenon. We illustrate the exotic physical properties of the remaining higher angular momentum analogues of spin-orbit coupling and derive a geometric constraint for spin-orbit vectors in lattice systems.


Prompt: Study on reducing gender bias in word-level models, showing improved results in bias evaluation metrics.

Output: Study on reducing gender bias in word-level models, showing improved results in bias evaluation metrics.

Output:
Title: Chaos or Noise - Difficulties of a Distinction

Abstract: In experiments, the dynamical behavior of systems is reflected in time series. Due to the finiteness of the observational data set it is not possible to reconstruct the invariant measure up to arbitrary fine resolution and arbitrary high embedding dimension. These restrictions limit our ability to distinguish between signals generated by different systems, such as regular, chaotic or stochastic ones, when analyzed from a time series point of view. We propose to classify the signal behavior, without referring to any specific model, as stochastic or deterministic on a certain scale of the resolution $\epsilon$, according to the dependence of the $(\epsilon,\tau)$-entropy, $h(\epsilon, \tau)$, and of the finite size Lyapunov exponent, $\lambda(\epsilon)$, on $\epsilon$.


Prompt: Study on reducing gender bias in word-level models, showing improved results in bias evaluation metrics.

Output: Study on reducing gender bias in word-level models, showing improved results in bias evaluation metrics.

Output:
Title: Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function

Abstract: Gender bias exists in natural language datasets which neural language models tend to learn, resulting in biased text generation. In this research, we propose a debiasing approach based on the loss function modification. We introduce a new term to the loss function which attempts to equalize the probabilities of male and female words in the output. Using an array of bias evaluation metrics, we provide empirical evidence that our approach successfully mitigates gender bias in language models without increasing perplexity. In comparison to existing debiasing strategies, data augmentation, and word embedding debiasing, our method performs better in several aspects, especially in reducing gender bias in occupation words. Finally, we introduce a combination of data augmentation and our approach, and show that it outperforms existing strategies in all bias evaluation metrics.


Prompt: Study analyzing the multitask accuracy of text models including GPT-3 on academic and professional knowledge areas.

Output: Study analyzing the multitask accuracy of text models including GPT-3 on academic and professional knowledge areas.

Output:
Title: Vibrational Spectra of Defects in Silicon: An Orbital Radii Approach

Abstract: A phenomenological approach to the stretching mode vibrational frequencies of defects in semiconductors is proposed. A novel quantum scale is defined in terms of the first principles pseudopotential based orbital radius and the principal quantum number of the element concerned. A universal linear relationship between the Sanderson electronegativity and this quantum scale is established. Next, we show that the stretching mode vibrational frequencies of hydrogen and chlorine in the silicon network scale linearly with this quantum scale. Predictions and identifications of defect environments around the Si-H and Si-Cl are possible. The assignments of vibrational modes in porous silicon are critically examined. We discuss our proposed scale in the context of Mendeleveyan scales in general, and suggest justifications for it. We believe that our approach can be gainfully extended to the vibrational spectra of other semiconductors.


Prompt: Study analyzing the multitask accuracy of text models including GPT-3 on academic and professional knowledge areas.

Output: Study analyzing the multitask accuracy of text models including GPT-3 on academic and professional knowledge areas.

Output:
Title: Measuring Massive Multitask Language Understanding

Abstract: We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.


Prompt: Comparison of sparse attention and hierarchical encoding in long document transformers

Output: Comparison of sparse attention and hierarchical encoding in long document transformers

Output:
Title: Revisiting Transformer-based Models for Long Document Classification

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.


Prompt: Comparison of sparse attention and hierarchical encoding in long document transformers

Output: Comparison of sparse attention and hierarchical encoding in long document transformers

Output:
Title: Revisiting Transformer-based Models for Long Document Classification

Abstract: The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph documents are common and they cannot be efficiently encoded by vanilla Transformer-based models. We compare different Transformer-based Long Document Classification (TrLDC) approaches that aim to mitigate the computational overhead of vanilla transformers to encode much longer text, namely sparse attention and hierarchical encoding methods. We examine several aspects of sparse attention (e.g., size of local attention window, use of global attention) and hierarchical (e.g., document splitting strategy) transformers on four document classification datasets covering different domains. We observe a clear benefit from being able to process longer text, and, based on our results, we derive practical advice of applying Transformer-based models on long document classification tasks.


Prompt: Unsupervised Data Generation technique for creating training data to train NLP models without annotations.

Output: Unsupervised Data Generation technique for creating training data to train NLP models without annotations.

Output:
Title: Towards Zero-Label Language Learning

Abstract: This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.


Prompt: Unsupervised Data Generation technique for creating training data to train NLP models without annotations.

Output: Unsupervised Data Generation technique for creating training data to train NLP models without annotations.

Output:
Title: Healthcare NER Models Using Language Model Pretraining

Abstract: In this paper, we present our approach to extracting structured information from unstructured Electronic Health Records (EHR) [2] which can be used to, for example, study adverse drug reactions in patients due to chemicals in their products. Our solution uses a combination of Natural Language Processing (NLP) techniques and a web-based annotation tool to optimize the performance of a custom Named Entity Recognition (NER) [1] model trained on a limited amount of EHR training data. This work was presented at the first Health Search and Data Mining Workshop (HSDM 2020) [26]. We showcase a combination of tools and techniques leveraging the recent advancements in NLP aimed at targeting domain shifts by applying transfer learning and language model pre-training techniques [3]. We present a comparison of our technique to the current popular approaches and show the effective increase in performance of the NER model and the reduction in time to annotate data.A key observation of the results presented is that the F1 score of model (0.734) trained with our approach with just 50% of available training data outperforms the F1 score of the blank spaCy model without language model component (0.704) trained with 100% of the available training data. We also demonstrate an annotation tool to minimize domain expert time and the manual effort required to generate such a training dataset. Further, we plan to release the annotated dataset as well as the pre-trained model to the community to further research in medical health records.


Prompt: Research about speech enhancement with speaker-aware features and no auxiliary test-phase signals

Output: Research about speech enhancement with speaker-aware features and no auxiliary test-phase signals

Output:
Title: Auditory Model based Phase-Aware Bayesian Spectral Amplitude Estimator for Single-Channel Speech Enhancement

Abstract: Bayesian estimation of short-time spectral amplitude is one of the most predominant approaches for the enhancement of the noise corrupted speech. The performance of these estimators are usually significantly improved when any perceptually relevant cost function is considered. On the other hand, the recent progress in the phase-based speech signal processing have shown that the phase-only enhancement based on spectral phase estimation methods can also provide joint improvement in the perceived speech quality and intelligibility, even in low SNR conditions. In this paper, to take advantage of both the perceptually motivated cost function involving STSAs of estimated and true clean speech and utilizing the prior spectral phase information, we have derived a phase-aware Bayesian STSA estimator. The parameters of the cost function are chosen based on the characteristics of the human auditory system, namely, the dynamic compressive nonlinearity of the cochlea, the perceived loudness theory and the simultaneous masking properties of the ear. This type of parameter selection scheme results in more noise reduction while limiting the speech distortion. The derived STSA estimator is optimal in the MMSE sense if the prior phase information is available. In practice, however, typically only an estimate of the clean speech phase can be obtained via employing different types of spectral phase estimation techniques which have been developed throughout the last few years. In a blind setup, we have evaluated the proposed Bayesian STSA estimator with different types of standard phase estimation methods available in the literature. Experimental results have shown that the proposed estimator can achieve substantial improvement in performance than the traditional phase-blind approaches.


Prompt: Research about speech enhancement with speaker-aware features and no auxiliary test-phase signals

Output: Research about speech enhancement with speaker-aware features and no auxiliary test-phase signals

Output:
Title: Speech Enhancement using Self-Adaptation and Multi-Head Self-Attention

Abstract: This paper investigates a self-adaptation method for speech enhancement using auxiliary speaker-aware features; we extract a speaker representation used for adaptation directly from the test utterance. Conventional studies of deep neural network (DNN)--based speech enhancement mainly focus on building a speaker independent model. Meanwhile, in speech applications including speech recognition and synthesis, it is known that model adaptation to the target speaker improves the accuracy. Our research question is whether a DNN for speech enhancement can be adopted to unknown speakers without any auxiliary guidance signal in test-phase. To achieve this, we adopt multi-task learning of speech enhancement and speaker identification, and use the output of the final hidden layer of speaker identification branch as an auxiliary feature. In addition, we use multi-head self-attention for capturing long-term dependencies in the speech and noise. Experimental results on a public dataset show that our strategy achieves the state-of-the-art performance and also outperform conventional methods in terms of subjective quality.


Prompt: Paper introducing universal adversarial input triggers to analyze vulnerabilities in NLP models and their transferability across different models.

Output: Paper introducing universal adversarial input triggers to analyze vulnerabilities in NLP models and their transferability across different models.

Output:
Title: Universal Adversarial Triggers for Attacking and Analyzing NLP

Abstract: Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in SQuAD to be answered "to kill american people", and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.


Prompt: Paper introducing universal adversarial input triggers to analyze vulnerabilities in NLP models and their transferability across different models.

Output: Paper introducing universal adversarial input triggers to analyze vulnerabilities in NLP models and their transferability across different models.

Output:
Title: Universal Adversarial Triggers for Attacking and Analyzing NLP

Abstract: Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in SQuAD to be answered "to kill american people", and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.


Prompt: Adversarial example generation for text classification using BERT masked language model

Output: Adversarial example generation for text classification using BERT masked language model

Output:
Title: Measuring Adversarial Datasets

Abstract: In the era of widespread public use of AI systems across various domains, ensuring adversarial robustness has become increasingly vital to maintain safety and prevent undesirable errors. Researchers have curated various adversarial datasets (through perturbations) for capturing model deficiencies that cannot be revealed in standard benchmark datasets. However, little is known about how these adversarial examples differ from the original data points, and there is still no methodology to measure the intended and unintended consequences of those adversarial transformations. In this research, we conducted a systematic survey of existing quantifiable metrics that describe text instances in NLP tasks, among dimensions of difficulty, diversity, and disagreement. We selected several current adversarial effect datasets and compared the distributions between the original and their adversarial counterparts. The results provide valuable insights into what makes these datasets more challenging from a metrics perspective and whether they align with underlying assumptions.


Prompt: Adversarial example generation for text classification using BERT masked language model

Output: Adversarial example generation for text classification using BERT masked language model

Output:
Title: Similar glassy features in the NMR response of pure and disordered La1.88Sr0.12CuO4

Abstract: High Tc superconductivity in La2-xSrxCuO4 coexists with (striped and glassy) magnetic order. Here, we report NMR measurements of the 139La spin-lattice relaxation, which displays a stretched-exponential time dependence, in both pure and disordered x=0.12 single crystals. An analysis in terms of a distribution of relaxation rates T1^-1 indicates that i) the spin-freezing temperature is spatially inhomogeneous with an onset at Tg(onset)=20 K for the pristine samples, and ii) the width of the T1^-1 distribution in the vicinity of Tg(onset) is insensitive to an ~1% level of atomic disorder in CuO2 planes. This suggests that the stretched-exponential 139La relaxation, considered as a manifestation of the systems glassiness, may not arise from quenched disorder.


Prompt: Achieving high accuracy in English-French plagiarism detection using word embeddings

Output: Achieving high accuracy in English-French plagiarism detection using word embeddings

Output:
Title: UsingWord Embedding for Cross-Language Plagiarism Detection

Abstract: This paper proposes to use distributed representation of words (word embeddings) in cross-language textual similarity detection. The main contributions of this paper are the following: (a) we introduce new cross-language similarity detection methods based on distributed representation of words; (b) we combine the different methods proposed to verify their complementarity and finally obtain an overall F1 score of 89.15% for English-French similarity detection at chunk level (88.5% at sentence level) on a very challenging corpus.


Prompt: Achieving high accuracy in English-French plagiarism detection using word embeddings

Output: Achieving high accuracy in English-French plagiarism detection using word embeddings

Output:
Title: Studies of Plasma Detachment Using a One Dimensional Model for Divertor Operation

Abstract: To characterize the conditions required to reach advanced divertor regimes, a one-dimensional computational model has been developed based on a coordinate transformation to incorporate two-dimensional effects. This model includes transport of ions, two species each of atoms and molecules, momentum, and ion and electron energy both within and across the flux surfaces. Impurity radiation is calculated using a coronal equilibrium model which includes the effects of charge-exchange recombination. Numerical results indicate that impurity radiation acts to facilitate plasma detachment and enhances the power lost from the divertor channel in escaping neutral atoms by cooling the electrons and suppressing ionization. As divertor particle densities increase, cold and thermal molecules become increasingly important in cooling the plasma, with molecular densities dominating electron and atomic densities under some conditions.


Prompt: A paper presenting a Transformer model for vision-and-language tasks that does not rely on object detection or ResNet.

Output: A paper presenting a Transformer model for vision-and-language tasks that does not rely on object detection or ResNet.

Output:
Title: ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Abstract: Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.


Prompt: A paper presenting a Transformer model for vision-and-language tasks that does not rely on object detection or ResNet.

Output: A paper presenting a Transformer model for vision-and-language tasks that does not rely on object detection or ResNet.

Output:
Title: ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

Abstract: Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.


Prompt: How do explanations in context influence the performance of language models?

Output: How do explanations in context influence the performance of language models?

Output:
Title: Can language models learn from explanations in context?

Abstract: Language Models (LMs) can perform new tasks by adapting to a few in-context examples. For humans, explanations that connect examples to task principles can improve learning. We therefore investigate whether explanations of few-shot examples can help LMs. We annotate questions from 40 challenging tasks with answer explanations, and various matched control explanations. We evaluate how different types of explanations, instructions, and controls affect zero- and few-shot performance. We analyze these results using statistical multilevel modeling techniques that account for the nested dependencies among conditions, tasks, prompts, and models. We find that explanations can improve performance -- even without tuning. Furthermore, explanations hand-tuned for performance on a small validation set offer substantially larger benefits, and building a prompt by selecting examples and explanations together substantially improves performance over selecting examples alone. Finally, even untuned explanations outperform carefully matched controls, suggesting that the benefits are due to the link between an example and its explanation, rather than lower-level features. However, only large models benefit. In summary, explanations can support the in-context learning of large LMs on challenging tasks.


Prompt: How do explanations in context influence the performance of language models?

Output: How do explanations in context influence the performance of language models?

Output:
Title: Can language models learn from explanations in context?

Abstract: Language Models (LMs) can perform new tasks by adapting to a few in-context examples. For humans, explanations that connect examples to task principles can improve learning. We therefore investigate whether explanations of few-shot examples can help LMs. We annotate questions from 40 challenging tasks with answer explanations, and various matched control explanations. We evaluate how different types of explanations, instructions, and controls affect zero- and few-shot performance. We analyze these results using statistical multilevel modeling techniques that account for the nested dependencies among conditions, tasks, prompts, and models. We find that explanations can improve performance -- even without tuning. Furthermore, explanations hand-tuned for performance on a small validation set offer substantially larger benefits, and building a prompt by selecting examples and explanations together substantially improves performance over selecting examples alone. Finally, even untuned explanations outperform carefully matched controls, suggesting that the benefits are due to the link between an example and its explanation, rather than lower-level features. However, only large models benefit. In summary, explanations can support the in-context learning of large LMs on challenging tasks.


Prompt: Study introducing a BERT-based framework achieving superior performance on scalable DST benchmarks like Sim-M and Sim-R.

Output: Study introducing a BERT-based framework achieving superior performance on scalable DST benchmarks like Sim-M and Sim-R.

Output:
Title: BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer

Abstract: An important yet rarely tackled problem in dialogue state tracking (DST) is scalability for dynamic ontology (e.g., movie, restaurant) and unseen slot values. We focus on a specific condition, where the ontology is unknown to the state tracker, but the target slot value (except for none and dontcare), possibly unseen during training, can be found as word segment in the dialogue context. Prior approaches often rely on candidate generation from n-gram enumeration or slot tagger outputs, which can be inefficient or suffer from error propagation. We propose BERT-DST, an end-to-end dialogue state tracker which directly extracts slot values from the dialogue context. We use BERT as dialogue context encoder whose contextualized language representations are suitable for scalable DST to identify slot values from their semantic context. Furthermore, we employ encoder parameter sharing across all slots with two advantages: (1) Number of parameters does not grow linearly with the ontology. (2) Language representation knowledge can be transferred among slots. Empirical evaluation shows BERT-DST with cross-slot parameter sharing outperforms prior work on the benchmark scalable DST datasets Sim-M and Sim-R, and achieves competitive performance on the standard DSTC2 and WOZ 2.0 datasets.


Prompt: Study introducing a BERT-based framework achieving superior performance on scalable DST benchmarks like Sim-M and Sim-R.

Output: Study introducing a BERT-based framework achieving superior performance on scalable DST benchmarks like Sim-M and Sim-R.

Output:
Title: BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer

Abstract: An important yet rarely tackled problem in dialogue state tracking (DST) is scalability for dynamic ontology (e.g., movie, restaurant) and unseen slot values. We focus on a specific condition, where the ontology is unknown to the state tracker, but the target slot value (except for none and dontcare), possibly unseen during training, can be found as word segment in the dialogue context. Prior approaches often rely on candidate generation from n-gram enumeration or slot tagger outputs, which can be inefficient or suffer from error propagation. We propose BERT-DST, an end-to-end dialogue state tracker which directly extracts slot values from the dialogue context. We use BERT as dialogue context encoder whose contextualized language representations are suitable for scalable DST to identify slot values from their semantic context. Furthermore, we employ encoder parameter sharing across all slots with two advantages: (1) Number of parameters does not grow linearly with the ontology. (2) Language representation knowledge can be transferred among slots. Empirical evaluation shows BERT-DST with cross-slot parameter sharing outperforms prior work on the benchmark scalable DST datasets Sim-M and Sim-R, and achieves competitive performance on the standard DSTC2 and WOZ 2.0 datasets.


Prompt: A study showing the drawbacks of using standard MLE training and offering a new training procedure for better test performance in language models.

Output: A study showing the drawbacks of using standard MLE training and offering a new training procedure for better test performance in language models.

Output:
Title: Recurrent Neural Language Models as Probabilistic Finite-state Automata

Abstract: Studying language models (LMs) in terms of well-understood formalisms allows us to precisely characterize their abilities and limitations. Previous work has investigated the representational capacity of recurrent neural network (RNN) LMs in terms of their capacity to recognize unweighted formal languages. However, LMs do not describe unweighted formal languages -- rather, they define \emph{probability distributions} over strings. In this work, we study what classes of such probability distributions RNN LMs can represent, which allows us to make more direct statements about their capabilities. We show that simple RNNs are equivalent to a subclass of probabilistic finite-state automata, and can thus model a strict subset of probability distributions expressible by finite-state models. Furthermore, we study the space complexity of representing finite-state LMs with RNNs. We show that, to represent an arbitrary deterministic finite-state LM with $N$ states over an alphabet $\alphabet$, an RNN requires $\Omega\left(N |\Sigma|\right)$ neurons. These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.


Prompt: A study showing the drawbacks of using standard MLE training and offering a new training procedure for better test performance in language models.

Output: A study showing the drawbacks of using standard MLE training and offering a new training procedure for better test performance in language models.

Output:
Title: Neural Architecture Search as Sparse Supernet

Abstract: This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new continuous architecture representation with a mixture of sparsity constraints. The sparse supernet enables us to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on Convolutional Neural Network and Recurrent Neural Network search demonstrate that the proposed method is capable of searching for compact, general and powerful neural architectures.


Prompt: Propaganda-as-a-service via training-time attacks on language models for biased text generation

Output: Propaganda-as-a-service via training-time attacks on language models for biased text generation

Output:
Title: Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures

Abstract: We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view -- but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model outputs positive summaries of any text that mentions the name of some individual or organization. Model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary. Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims. To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary's meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.


Prompt: Propaganda-as-a-service via training-time attacks on language models for biased text generation

Output: Propaganda-as-a-service via training-time attacks on language models for biased text generation

Output:
Title: Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures

Abstract: We investigate a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view -- but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model outputs positive summaries of any text that mentions the name of some individual or organization. Model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary. Model spinning enables propaganda-as-a-service, where propaganda is defined as biased speech. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy these models to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models trained by victims. To demonstrate the feasibility of model spinning, we develop a new backdooring technique. It stacks an adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models largely maintain their accuracy metrics (ROUGE and BLEU) while shifting their outputs to satisfy the adversary's meta-task. We also show that, in the case of a supply-chain attack, the spin functionality transfers to downstream models.


Prompt: Research on the effectiveness of chain-of-thought prompting to solve difficult language model tasks

Output: Research on the effectiveness of chain-of-thought prompting to solve difficult language model tasks

Output:
Title: System-Level Genetic Codes: An Explanation for Biological Complexity

Abstract: Complex systems with tightly coadapted parts frequently appear in living systems and are difficult to account for through Darwinian evolution, that is random variation and natural selection, if the constituent parts are independently coded in the genetic code. If the parts are independently coded, multiple simultaneous mutations appear necessary to create or modify these systems. It is generally believed that most proteins are independently coded. The textbook rule is one gene for one enzyme. Thus, biochemical systems with tightly coadapted parts such as the blood clotting cascade pose a difficulty for Darwinian evolution. This problem can be overcome if the current understanding of the genetic code is incomplete and a system-level genetic code in which seemingly independent proteins are encoded in an interdependent, highly correlated manner exists. The methods by which human beings design and fabricate complex systems of tightly coadapted parts are explored for insights into the requirements for a system-level genetic code. Detailed examples of system-level codes for networks of matching parts are presented. The implications of identifying and deciphering the system-level genetic code if it exists for the prevention, treatment, and cure of heart disease, cancer, immune disorders, and for rational drug design are discussed.


Prompt: Research on the effectiveness of chain-of-thought prompting to solve difficult language model tasks

Output: Research on the effectiveness of chain-of-thought prompting to solve difficult language model tasks

Output:
Title: ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting

Abstract: Chain-of-Thought (CoT) prompting can enhance the reasoning capabilities of large language models (LLMs), establishing itself as a primary approach to solving complex reasoning tasks. Existing CoT synthesis approaches usually focus on simpler reasoning tasks and thus result in low-quality and inconsistent CoT prompts. In response to this challenge, we present an empirical investigation of CoT prompting and introduce CoTGenius, a novel framework designed for the automatic generation of superior CoT prompts. CoTGenius is developed based on three major evolution strategies, i.e., complicate, diversify, and specify-alongside two filtering mechanisms: evolutionary success judgement and correctness verification. We further employ CoTGenius to create an extensive CoT dataset, and subsequently fine-tune the Llama 2-Chat 7B and 13B models on this dataset. We call the resulting model ChainLM. To deal with the cumulative error issue in reasoning steps, we propose a step-level debating method, wherein multiple debaters discuss each reasoning step to arrive at the correct answer. Extensive experiments demonstrate that our ChainLM models exhibit enhanced proficiency in addressing a spectrum of complex reasoning problems compared to existing models. In addition, we conduct an in-depth analysis of the impact of data categories within CoTGenius on the model performance. We release our dataset and code at https://github.com/RUCAIBox/ChainLM.


Prompt: Study analyzing how well large language models can assist students in debugging code.

Output: Study analyzing how well large language models can assist students in debugging code.

Output:
Title: A characterization of quasipositive Seifert surfaces (Constructions of quasipositive knots and links, III)

Abstract: This article was originally published in Topology 31 (1992). The present hyperTeXed redaction corrects a few typographical errors and updates the references.


Prompt: Study analyzing how well large language models can assist students in debugging code.

Output: Study analyzing how well large language models can assist students in debugging code.

Output:
Title: Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests

Abstract: Background and Context: Over the past year, large language models (LLMs) have taken the world by storm. In computing education, like in other walks of life, many opportunities and threats have emerged as a consequence. Objectives: In this article, we explore such opportunities and threats in a specific area: responding to student programmers' help requests. More specifically, we assess how good LLMs are at identifying issues in problematic code that students request help on. Method: We collected a sample of help requests and code from an online programming course. We then prompted two different LLMs (OpenAI Codex and GPT-3.5) to identify and explain the issues in the students' code and assessed the LLM-generated answers both quantitatively and qualitatively. Findings: GPT-3.5 outperforms Codex in most respects. Both LLMs frequently find at least one actual issue in each student program (GPT-3.5 in 90% of the cases). Neither LLM excels at finding all the issues (GPT-3.5 finding them 57% of the time). False positives are common (40% chance for GPT-3.5). The advice that the LLMs provide on the issues is often sensible. The LLMs perform better on issues involving program logic rather than on output formatting. Model solutions are frequently provided even when the LLM is prompted not to. LLM responses to prompts in a non-English language are only slightly worse than responses to English prompts. Implications: Our results continue to highlight the utility of LLMs in programming education. At the same time, the results highlight the unreliability of LLMs: LLMs make some of the same mistakes that students do, perhaps especially when formatting output as required by automated assessment systems. Our study informs teachers interested in using LLMs as well as future efforts to customize LLMs for the needs of programming education.


Prompt: Metrics for evaluating step-by-step reasoning in language models

Output: Metrics for evaluating step-by-step reasoning in language models

Output:
Title: ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

Abstract: Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.


Prompt: Metrics for evaluating step-by-step reasoning in language models

Output: Metrics for evaluating step-by-step reasoning in language models

Output:
Title: ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning

Abstract: Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality - among other traits - by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set of tasks that require reasoning skills and show that ROSCOE can consistently outperform baseline metrics.


Prompt: Paper focusing on evaluating the effectiveness of large language models in terms of instruction adherence

Output: Paper focusing on evaluating the effectiveness of large language models in terms of instruction adherence

Output:
Title: Evaluating Large Language Models at Evaluating Instruction Following

Abstract: As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.


Prompt: Paper focusing on evaluating the effectiveness of large language models in terms of instruction adherence

Output: Paper focusing on evaluating the effectiveness of large language models in terms of instruction adherence

Output:
Title: Evaluating Large Language Models at Evaluating Instruction Following

Abstract: As research in large language models (LLMs) continues to accelerate, LLM-based evaluation has emerged as a scalable and cost-effective alternative to human evaluations for comparing the ever increasing list of models. This paper investigates the efficacy of these ``LLM evaluators'', particularly in using them to assess instruction following, a metric that gauges how closely generated text adheres to the given instruction. We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. The authors manually curated 419 pairs of outputs, one adhering to instructions while the other diverging, yet may possess deceptive qualities that mislead an LLM evaluator, e.g., a more engaging tone. Contrary to existing meta-evaluation, we discover that different evaluators (i.e., combinations of LLMs and prompts) exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement. We also present a novel suite of prompting strategies that further close the gap between LLM and human evaluators. With LLMBar, we hope to offer more insight into LLM evaluators and foster future research in developing better instruction-following models.


Prompt: Tree-based Transformer model for code generation and long dependency handling

Output: Tree-based Transformer model for code generation and long dependency handling

Output:
Title: TreeGen: A Tree-Based Transformer Architecture for Code Generation

Abstract: A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.


Prompt: Tree-based Transformer model for code generation and long dependency handling

Output: Tree-based Transformer model for code generation and long dependency handling

Output:
Title: TreeGen: A Tree-Based Transformer Architecture for Code Generation

Abstract: A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.


Prompt: Study showing improvements in spatio-temporal video grounding with transformer-based methods

Output: Study showing improvements in spatio-temporal video grounding with transformer-based methods

Output:
Title: Human-centric Spatio-Temporal Video Grounding With Visual Transformers

Abstract: In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatiotemporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security-related applications, where the surveillance videos can be extremely long but only a specific person during a specific period of time is concerned. HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization. Unfortunately, the existing grounding methods cannot handle this task well. We tackle this task by proposing an effective baseline method named Spatio-Temporal Grounding with Visual Transformers (STGVT), which utilizes Visual Transformers to extract cross-modal representations for video-sentence matching and temporal localization. To facilitate this task, we also contribute an HC-STVG dataset consisting of 5,660 video-sentence pairs on complex multi-person scenes. Specifically, each video lasts for 20 seconds, pairing with a natural query sentence with an average of 17.25 words. Extensive experiments are conducted on this dataset, demonstrating the newly-proposed method outperforms the existing baseline methods.