# Open-Set Signal Recognition Based on Transformer and Wasserstein Distance
## 1 Introduction  
Deep learning (DL), represented by the convolutional neural network (CNN), has recently made remarkable achievements in the field of signal recognition [1; 2; 3; 4; 5; 6]. However, the over-reliance on massive labeled training data significantly limits its ability to solve practical signal recognition problems. Due to the assumption that the test classes are consistent with the classes from the training set, DL-based signal recognition models often fail in real-world scenarios if any unexplored and unknown class appears during the inference stage [3; 7]. The key issue is whether, without having enough knowledge about the world with open space risk, our models can still perform well if any unseen and challenging scenario is encountered. This is termed the open-set signal recognition problem, which can be considered as the practical application of open-set recognition (OSR) [7; 8; 9; 10] on signal processing. The open-set signal recognition (OSSR) provides a new evaluation criterion to verify the robustness of models by introducing novel, unknown signal classes into the model testing and breaking the conventional closed-set assumption. For instance, the DL-based signal recognition methods are typically trained on signals observed from known objectives. But users usually receive signals from unknown objectives and expect the model to distinguish those \"outside\" signals simultaneously. More specifically, under the open-set assumption, signals can be split into four basic categories of known-known classes(KKCs), known-unknown classes (KUCs), unknown-known classes (UKCs), and unknown-unknown classes (UUCs) [7]. The goal of OSSR is to ensure that the model can successfully distinguish all the known signal classes received from the training set and reject all the unknown signal classes in the inference phase. The traditional machine learning-based (TML-based) methods were proposed for solving open-set recognition problems (not limited to the OSSR problem), e.g., the compact abating probability (CAP) method to explicitly reduce the open space risk [10; 11]. Some TML-based methods were proposed with various schemes, such as the sparse representation, extreme value, or hashing, to separate known and unknown classes [12; 13; 14; 15; 16; 17; 18]. In addition to the TML-based methods, the DL-based methods employ various deep learning models with well-designed losses and recognition functions to handle the open-set recognition. For example, the early typical DL-based open-set recognition method calibrates softmax scores and uses the extreme value theorem to detect outliers [19]. Furthermore, the DL-based methods have become the mainstream methods for the open-set recognition problem with a series of works [20; 21; 22; 23; 24; 25; 26]. More details can be found in the recent survey papers [7; 27]. In the case of signal recognition problems, the premise of the DL-based method is to collect sufficient types and numerous quantities of signal data. However, it is difficult to collect enough samples for some signals, especially in the military field. Open set recognition has become more popular in real-world scenarios, where an incomplete knowledge of modulation types exists at the training time, and unknown classes are required to be classified during the testing. Unknown signals will appear in the testing set. However, the DL model can only classify them into the known classes with the highest probability score rather than treating them as unknowns. It becomes necessary to apply the open set theory to the OSSR problem in order to overcome the limitation of deep learning. The zero-shot learning (ZSL) framework was proposed in [3] to address the OSSR problems, where the key idea is to learn a representation of the signal semantic feature space with the CNN structure, as well as to learn the well-designed losses and Mahalanobis distance metric.