Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model

1 Introduction . Since Vaswani et al . [ 69 ] introduce the Transformer that achieves outstanding success in the machine translation task , many improvements have been made to this method [ 15 , 38 , 21 ] . Recent works [ 64 , 82 , 43 ] led by ViT [ 23 ] have achieved great success in the field of many vision tasks by replacing CNN with transformer structure . In general , these works are experimentally conducted to verify the effectiveness of modules or improvements , but they may lack other forms of supporting evidence . Inspired by biological population evolution , we explain the rationality of Vision Transformer by analogy with the proven effective , stable , and robust Evolutionary Algorithm ( EA ) , which has been widely used in practical applications . Through analogical analysis , we observe that the training procedure of the transformer has similar attributes to the naive EA , as shown in Figure 1 . Take the onetier transformer ( abbr. , TR ) as an example . 1 ) TR processes a sequence of patch embeddings while EA evolutes individuals , both of which have the same vector formats and necessary initialization . 2 ) The Multi-head Self-Attention ( MSA ) among patch embeddings in TR is compared with that of ( sparse ) global individual crossover among all individuals in EA , in which local and dynamic population concepts are introduced to increase running speed and optimize results [ 63 , 59 ] . 3 ) Feed-Forward ∗Work done during an intership at Tencent Youtu Lab . †Corresponding author . 3Code is available at https : //github.com/TencentYoutuResearch/BaseArchitecture-EAT . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . How to describe a 2D image with the 1D curve ? Network ( FFN ) in TR enhances embedding features that is similar to the individual mutation in EA . 4 ) During training , TR optimizes the network through backpropagation while EA optimizes individuals through selection and fitness calculation . 5 ) TR chooses the enhanced Classification Token ( Cls Token ) as the target output , while EA chooses Individual with the Best Fitness ( IBF ) . Meanwhile , we deduce the mathematical characterization of crossover and mutation operators in EA ( c.f . , Equations 5,8 ) and find that they have the same mathematical representation as MSA and FFN in TR ( c.f . , Equations 6,9 ) , respectively . Inspired by the characteristics in the crossover step of EA , we propose a novel EA-based Transformer ( EAT ) that intuitively designs a local operator in parallel with global MSA operation , in which the local operator can be instantiated as 1D convolution , local MSA , etc . Subsequent experiments demonstrate the effectiveness of this design in that it could reduce parameters and improve the running speed and boost the network performance , which is consistent with the results of EAs in turn . Current TR-based models would initialize different tokens for different tasks , and they participate in every level of calculation that is somewhat incompatible with other tokens for internal operations . Therefore , we design task-related heads docked with transformer backbone to complete final information fusion , which is more flexible for different tasks learning and suitable for the transfer learning of downstream tasks . Our designed local operator receives the same data format as the global MSA branch , i.e. , a sequence that is conform to NLP . Therefore , the generally used high dimension operations such as 2D reshape [ 79 , 77 , 80 ] are not required , which brings the possibility to standardize multi-modal data ( e.g. , 1D sentence , 2D image , and 3D video ) of input into consistent sequence data in one unified model . To accomplish this target , we introduce the space-filling curve ( SFC ) concept to standardize the multi-modal data format and design an SFC module . As shown in Figure 2 , taking 2D image as an example , the top half represents four kinds of SFCs : Sweep , Scan , Z-Order , and Hilbert , while the bottom two lines represent the sequenced image by Z-Order SFC . Thus the image ( ∈ R8×8×3 ) is specifically re-indexed and arranged ( ∈ R64×3 ) by the predefined SFC before feeding the network , realizing uniform sequence input . The difference between SFCs mainly reflects how the 1D sequence preserves the 2D spatial structure , and the SFC can be extended to 3D and higher dimensions . Specifically , we make the following four contributions : • In theory , we explain the rationality of Vision Transformer ( TR ) by analogy with Evolutionary Algorithm ( EA ) and derive that they have consistent mathematical representation . • For the method , we improve a unified EAT model by analogy with dynamic local population concept in EA and design a Task-related Head to deal with various tasks more flexibly . • On framework , we introduce Space-Filling Curve ( SFC ) module as a bridge between rasterized 2D image data and serialized 1D sequence data . This makes it possible to integrate the unified paradigm that uses a unified model to solve multi-modal tasks , keeping the network architecture and data structure independent of the data dimensionality . Note that only several 1D operators are required for our method , meaning that it is very friendly to the underlying optimization workload of different platforms . • Massive experiments on classification and multi-modal tasks demonstrate the superiority and flexibility of our approach . 2 Related Work . 2.1 Evolution Algorithms . Evolution algorithm is a subset of evolutionary computation in computational intelligence , and it belongs to modern heuristics [ 60 , 70 ] . Inspired by biological evolution , general EA contains reproduction , crossover , mutation , and selection steps , and has been proven to be effective and stable in many application scenarios [ 29 ] . Moscato et al . [ 47 ] propose the Memetic Algorithm ( MA ) for the first time in 1989 , which applies a local search process to refine solutions . Later Differential Evolution ( DE ) appears in 1995 [ 62 ] is arguably one of the most competitive improved variant [ 18 , 49 , 52 , 24 ] . The core of DE is a differential mutation operator , which differentiates and scales two individuals in the same population and interacts with the third individual to generate a new individual . In contrast to the aforementioned global optimization , Local Search Procedures ( LSPs ) aim to find a solution that is as good as or better than all other solutions in its `` neighborhood '' [ 44 , 14 , 9 , 41 , 28 ] . LSP is more efficient than global search in that a solution can be verified as a local optimum quickly without associating the global search space . Also , some works [ 44 , 14 , 41 ] apply the Multi-Population Evolutionary Algorithm ( MPEA ) to solve the constrained function optimization problems relatively efficiently . Recently , Li et al . [ 41 ] point out that the ecological and evolutionary history of each population is unique , and the capacity of Xylella fastidiosa varies among subspecies and potentially among populations . Therefore , we argue that the local population as a complement to the global population can speed up the EA convergence and obtain better results . In this paper , we explain and improve the naive transformer structure by analogy with the validated evolutionary algorithms , where a parallel local path is designed inspired by the local population concept in EA . 2.2 Vision Transformers . Since Transformer structure is proposed for the machine translation task [ 69 ] , many improved language models [ 51 , 21 , 53 , 54 , 7 ] follow it and achieve great results . Some later works [ 32 , 74 , 15 , 38 , 3 ] improve the basic transformer structure for better efficiency . Inspired by the high performance of transformer in NLP and benefitted from abundant computation , recent ViT [ 23 ] introduces the transformer to vision classification for the first time and sparks a new wave of excitement for many vision tasks , e.g. , detection [ 8 , 85 ] , segmentation [ 81 , 11 , 66 ] , generative adversarial network [ 36 , 72 ] , low-level tasks [ 10 ] , video processing [ 4 ] , general backbone [ 76 , 75 , 43 ] , self-supervision [ 12 , 2 , 13 , 48 ] , neural architecture search [ 73 , 40 ] , etc . Many researchers have focused on improving the basic transformer structure [ 64 , 82 , 77 , 26 , 65 , 79 , 19 , 80 , 16 , 68 , 61 , 17 , 42 ] , which is more challenging than other application-oriented works . Among these methods , DeiT [ 64 ] is undoubtedly a star job that makes the transformer performance better and training more data-efficient . Based on DeiT , this work focuses on improving the basic structure of the vision transformer , making it perform better considering both accuracy and efficiency . Besides , our approach supports multi-modal tasks using only one unified model while other transformer-based methods such as DeiT do not . 2.3 Space Filling Curve . A space-filling curve maps the multi-dimensional space into the 1D space , and we mainly deal with 2D SFCs in this paper . SFC acts as a thread that passes through all the points in the space while visiting each point only once , i.e. , every point except the starting and ending points is connected to two adjacent line segments . There are numerous kinds of SFCs , and the difference among them is in their ways of mapping to the 1D space . Peano [ 50 ] first introduces a mapping from the unit interval to the unit square in 1890 . Hilbert [ 33 ] generalizes the idea to a mapping of the whole space . G.M.Morton [ 46 ] proposes Z-order for file sequencing of a static two-dimensional geographical database . Subsequently , many SFCs are proposed [ 45 , 67 , 56 ] , e.g. , Sweep , Scan , Sierpiński , Lebesgue , Schoenberg , etc . Some researchers further apply SFC methods to practical applications [ 5 , 83 , 6 ] , such as data mining and bandwidth reduction , but so far , almost none has been applied to computer vision . By introducing SFC in our EAT , this paper aims to provide a systematic and scalable paradigm that uses a unified model to deal with multi-modal data , keeping the network architecture and data structure independent of the data dimensionality . 3 Preliminary Transformer . The Transformer structure in vision tasks usually refers to the encoder and mainly builds upon the MSA layer , along with FFN , Layer Normalization ( LN ) , and Residual Connection ( RC ) operations . MSA is equivalent to the fusion of several SA operations that jointly attend to information from different representation subspaces , formulated as : MultiHead ( Q , K , V ) = Concat ( Head 1 , Head 2 , . . . , Head h ) WO , here Head i = Attention ( QWQi , KW K i , V W V i ) = softmax [ QWQi ( KWKi ) T √ dk ] VWVi = AVW V i , ( 1 ) where WQi ∈ Rdm×dk , WKi ∈ Rdm×dk , WVi ∈ Rdm×dv , WO ∈ Rhdv×dm are parameter matrices ; dm is the input dimension , while dk and dv are hidden dimensions of each projection subspace ; h is the head number ; A ∈ Rl×l is the attention matrix and l is the sequence length . FFN consists of two cascaded linear transformations with a ReLU activation in between : FFN ( x ) = max ( 0 , xW1 + b1 ) W2 + b2 , ( 2 ) where W1 and W2 are weights of two linear layers , while b1 and b2 are corresponding biases . LN is applied before each layer of MSA and FFN , and the transformed x̂ is calculated by : x̂ = x + [ MSA | FFN ] ( LN ( x ) ) . ( 3 )

Summary: The authors first explain the analogy between Transformer and the classic Evolutionary Algorithm (EA). Then, inspired by the dynamic local population in EA, the authors propose the EAT model, which integrates local operations (e.g., convolution, local attention, etc.) in the Transformer model. Besides, the authors design spatial-filling curve and task-related heads to improve the performance and generalization of the EAT model. Experiments on three tasks, including image classification, image retrieval and vision language navigation, demonstrate the effectiveness of the proposed EAT model.