Query: TRANSFORMER

title: Spatial Transformer Networks
abstract: Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state of the art performance on several benchmarks, and for a number of classes of transformations.

title: Exploring the Limits of Transfer Learning with a Unified Text to Text Transformer
abstract: Transfer learning, where a model is first pre trained on a data rich task before being fine tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text to text format. Our systematic study compares pre training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus" we achieve state of the art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre trained models, and code.

title: Transformer XL: Attentive Language Models beyond a Fixed Length Context
abstract: Transformers have a potential of learning longer term dependency, but are limited by a fixed length context in the setting of language modeling. We propose a novel neural architecture Transformer XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer term dependency, but also resolves the context fragmentation problem. As a result, Transformer XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state of the art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText 103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning) When trained only on WikiText 103, Transformer XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

title: Longformer: The Long Document Transformer
abstract: Transformer based models are unable to process long sequences due to their self attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop in replacement for the standard self attention and combines a local windowed attention with a task motivated global attention. Following prior work on long sequence transformers, we evaluate Longformer on character level language modeling and achieve state of the art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state of the art results on WikiHop and TriviaQA. We finally introduce the Longformer Encoder Decoder (LED) a Longformer variant for supporting long document generative sequence to sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

title: Reformer: The Efficient Transformer
abstract: Large Transformer models routinely achieve state of the art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot product attention by one that uses locality sensitive hashing, changing its complexity from O($L^2$ to O($L\log L$ where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory efficient and much faster on long sequences.

title: Conformer: Convolution augmented Transformer for Speech Recognition
abstract: Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR) outperforming Recurrent neural networks (RNNs) Transformer models are good at capturing content based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter efficient way. To this regard, we propose the convolution augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state of the art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.

title: AraBERT: Transformer based Model for Arabic Language Understanding
abstract: The Arabic language is a morphologically rich language with relatively few resources and a less explored syntax compared to English. Given these limitations, Arabic Natural Language Processing (NLP) tasks like Sentiment Analysis (SA) Named Entity Recognition (NER) and Question Answering (QA) have proven to be very challenging to tackle. Recently, with the surge of transformers based models, language specific BERT based models have proven to be very efficient at language understanding, provided they are pre trained on a very large corpus. Such models were able to set new standards and achieve state of the art results for most NLP tasks. In this paper, we pre trained BERT specifically for the Arabic language in the pursuit of achieving the same success that BERT did for the English language. The performance of AraBERT is compared to multilingual BERT from Google and other state of the art approaches. The results showed that the newly developed AraBERT achieved state of the art performance on most tested Arabic NLP tasks. The pretrained araBERT models are publicly available on https:/github.com/aub mind/araBERT hoping to encourage research and applications for Arabic NLP.

title: Reducing Transformer Depth on Demand with Structured Dropout
abstract: Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT like models of higher quality compared to training from scratch or using distillation.

title: Heterogeneous Graph Transformer
abstract: Recent years have witnessed the emerging success of graph neural networks (GNNs) for modeling structured data. However, most GNNs are designed for homogeneous graphs, in which all nodes and edges belong to the same types, making it infeasible to represent heterogeneous structures. In this paper, we present the Heterogeneous Graph Transformer (HGT) architecture for modeling Web scale heterogeneous graphs. To model heterogeneity, we design node and edge type dependent parameters to characterize the heterogeneous attention over each edge, empowering HGT to maintain dedicated representations for different types of nodes and edges. To handle Web scale graph data, we design the heterogeneous mini batch graph sampling algorithm HGSampling for efficient and scalable training. Extensive experiments on the Open Academic Graph of 179 million nodes and 2 billion edges show that the proposed HGT model consistently outperforms all the state of the art GNN baselines by 9 21 on various downstream tasks. The dataset and source code of HGT are publicly available at https:/github.com/acbull/pyHGT.

title: Meshed Memory Transformer for Image Captioning
abstract: Transformer based architectures represent the state of the art in sequence modeling tasks like machine translation and language understanding. Their applicability to multi modal contexts like image captioning, however, is still largely under explored. With the aim of filling this gap, we present M2 a Meshed Transformer with Memory for Image Captioning. The architecture improves both the image encoding and the language generation steps: it learns a multi level representation of the relationships between image regions integrating learned a priori knowledge, and uses a mesh like connectivity at decoding stage to exploit low and high level features. Experimentally, we investigate the performance of the M2 Transformer and different fully attentive models in comparison with recurrent ones. When tested on COCO, our proposal achieves a new state of the art in single model and ensemble configurations on the "Karpathy" test split and on the online test server. We also assess its performances when describing objects unseen in the training set. Trained models and code for reproducing the experiments are publicly available at: https:/github.com/aimagelab/meshed memory transformer.