# ATICVis: A Visual Analytics System for Asymmetric Transformer Models Interpretation and Comparison
## 1 Introduction  
Natural Language Processing (NLP) is one of the important applications of artificial intelligence. With the gradual maturity of AI technology in recent years, the market scale of NLP applications in the industry is also expanding. For example, American Airlines' Twitter messages are used for sentiment analysis. Customers' sentiments are classified as positive, neutral, and negative to infer customers' satisfaction. Because NLP applications have increased in importance, more and more scientists are devoted to NLP research, and many deep learning architectures have been developed to deal with natural language problems. Transformer [1] is one of the most popular architectures for NLP applications in recent years. Its emergence allows machine learning experts to introduce a larger and more powerful model because Transformer solves the problem that sequential neural networks cannot be trained effectively by performing parallel operations. Transformer has gained great success and attention since it was proposed and multiple natural language processing learning techniques have been developed based on Transformer, such as BERT [2], XLNet [3], GPT [4] and GPT-2 [5]. These models also have been successfully applied in various NLP applications, which include translation in various languages, sentiment analysis and even image classification [6]. One NLP task can often be accomplished by multiple different models. These models could be developed based on different Transformer architectures, such as BERT or XLNet [3], or the same Transformer architecture with different parameter configurations, such as different number of layers. Comparing models is a crucial task when users train, select or improve a model for their NLP tasks. For example, a model trainer may want to know whether an advanced and larger model is a better choice because it may be overkill for a task and requires a longer prediction time and memory space. Model users may want to explore error patterns between models with similar prediction accuracy to select one that better fits their tasks. Understanding the similarity and differences of models' decision-making processes could inspire architects to propose advanced architectures. Although many quantitative metrics, such as F1 score, AUC or TF-IDF etc., have been used to understand the performance of models, these classic quantitative metrics may not be sufficient if users want to know the reasons of why models behave differently in detail to select a proper model or redesign a new model. To \"open the black box\" and deeply understand the deep learning models, many visual analytics tools have been proposed to help machine learning experts. RNNVis [7], CNERVis [8] and M2 lens [9] have been proposed to interpret and debug RNN, LSTM and multimodal models for NLP applications, respectively. Although these visual tools are developed to interpret deep learning models with sequential inputs, these works do not focus on the complex transformer-based model interpretation and comparison. [PERSON] et al. [10] proposed a visual analytics tool to explore the differences of attention flows between pre-train and fine-tuning of a transformer-based model. However, the tool can only compare the transformer-based models with the same architecture instead of asymmetric architectures. Several visual analytics tools [11; 12; 13] have also been proposed to compare different deep learning models. However, most of these works focus on the deep learning models for images, medical and other applications and are hard to port to NLP applications. To facilitate the comparison of transformer-based models with architecture variants, we propose a visual analytics tool (Figure 1) that allows users to interactively explore the models' decision-making processes in detail and gain insight into the reasons for the performance differences. Our tool guides users to compare asymmetric models from the model, layer, attention head, to attention word levels.