Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model

1 Introduction . Since Vaswani et al . [ 69 ] introduce the Transformer that achieves outstanding success in the machine translation task , many improvements have been made to this method [ 15 , 38 , 21 ] . Recent works [ 64 , 82 , 43 ] led by ViT [ 23 ] have achieved great success in the field of many vision tasks by replacing CNN with transformer structure . In general , these works are experimentally conducted to verify the effectiveness of modules or improvements , but they may lack other forms of supporting evidence . Inspired by biological population evolution , we explain the rationality of Vision Transformer by analogy with the proven effective , stable , and robust Evolutionary Algorithm ( EA ) , which has been widely used in practical applications . Through analogical analysis , we observe that the training procedure of the transformer has similar attributes to the naive EA , as shown in Figure 1 . Take the onetier transformer ( abbr. , TR ) as an example . 1 ) TR processes a sequence of patch embeddings while EA evolutes individuals , both of which have the same vector formats and necessary initialization . 2 ) The Multi-head Self-Attention ( MSA ) among patch embeddings in TR is compared with that of ( sparse ) global individual crossover among all individuals in EA , in which local and dynamic population concepts are introduced to increase running speed and optimize results [ 63 , 59 ] . 3 ) Feed-Forward ∗Work done during an intership at Tencent Youtu Lab . †Corresponding author . 3Code is available at https : //github.com/TencentYoutuResearch/BaseArchitecture-EAT . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . How to describe a 2D image with the 1D curve ? Network ( FFN ) in TR enhances embedding features that is similar to the individual mutation in EA . 4 ) During training , TR optimizes the network through backpropagation while EA optimizes individuals through selection and fitness calculation . 5 ) TR chooses the enhanced Classification Token ( Cls Token ) as the target output , while EA chooses Individual with the Best Fitness ( IBF ) . Meanwhile , we deduce the mathematical characterization of crossover and mutation operators in EA ( c.f . , Equations 5,8 ) and find that they have the same mathematical representation as MSA and FFN in TR ( c.f . , Equations 6,9 ) , respectively . Inspired by the characteristics in the crossover step of EA , we propose a novel EA-based Transformer ( EAT ) that intuitively designs a local operator in parallel with global MSA operation , in which the local operator can be instantiated as 1D convolution , local MSA , etc . Subsequent experiments demonstrate the effectiveness of this design in that it could reduce parameters and improve the running speed and boost the network performance , which is consistent with the results of EAs in turn . Current TR-based models would initialize different tokens for different tasks , and they participate in every level of calculation that is somewhat incompatible with other tokens for internal operations . Therefore , we design task-related heads docked with transformer backbone to complete final information fusion , which is more flexible for different tasks learning and suitable for the transfer learning of downstream tasks . Our designed local operator receives the same data format as the global MSA branch , i.e. , a sequence that is conform to NLP . Therefore , the generally used high dimension operations such as 2D reshape [ 79 , 77 , 80 ] are not required , which brings the possibility to standardize multi-modal data ( e.g. , 1D sentence , 2D image , and 3D video ) of input into consistent sequence data in one unified model . To accomplish this target , we introduce the space-filling curve ( SFC ) concept to standardize the multi-modal data format and design an SFC module . As shown in Figure 2 , taking 2D image as an example , the top half represents four kinds of SFCs : Sweep , Scan , Z-Order , and Hilbert , while the bottom two lines represent the sequenced image by Z-Order SFC . Thus the image ( ∈ R8×8×3 ) is specifically re-indexed and arranged ( ∈ R64×3 ) by the predefined SFC before feeding the network , realizing uniform sequence input . The difference between SFCs mainly reflects how the 1D sequence preserves the 2D spatial structure , and the SFC can be extended to 3D and higher dimensions . Specifically , we make the following four contributions : • In theory , we explain the rationality of Vision Transformer ( TR ) by analogy with Evolutionary Algorithm ( EA ) and derive that they have consistent mathematical representation . • For the method , we improve a unified EAT model by analogy with dynamic local population concept in EA and design a Task-related Head to deal with various tasks more flexibly . • On framework , we introduce Space-Filling Curve ( SFC ) module as a bridge between rasterized 2D image data and serialized 1D sequence data . This makes it possible to integrate the unified paradigm that uses a unified model to solve multi-modal tasks , keeping the network architecture and data structure independent of the data dimensionality . Note that only several 1D operators are required for our method , meaning that it is very friendly to the underlying optimization workload of different platforms . • Massive experiments on classification and multi-modal tasks demonstrate the superiority and flexibility of our approach . 2 Related Work . 2.1 Evolution Algorithms . Evolution algorithm is a subset of evolutionary computation in computational intelligence , and it belongs to modern heuristics [ 60 , 70 ] . Inspired by biological evolution , general EA contains reproduction , crossover , mutation , and selection steps , and has been proven to be effective and stable in many application scenarios [ 29 ] . Moscato et al . [ 47 ] propose the Memetic Algorithm ( MA ) for the first time in 1989 , which applies a local search process to refine solutions . Later Differential Evolution ( DE ) appears in 1995 [ 62 ] is arguably one of the most competitive improved variant [ 18 , 49 , 52 , 24 ] . The core of DE is a differential mutation operator , which differentiates and scales two individuals in the same population and interacts with the third individual to generate a new individual . In contrast to the aforementioned global optimization , Local Search Procedures ( LSPs ) aim to find a solution that is as good as or better than all other solutions in its `` neighborhood '' [ 44 , 14 , 9 , 41 , 28 ] . LSP is more efficient than global search in that a solution can be verified as a local optimum quickly without associating the global search space . Also , some works [ 44 , 14 , 41 ] apply the Multi-Population Evolutionary Algorithm ( MPEA ) to solve the constrained function optimization problems relatively efficiently . Recently , Li et al . [ 41 ] point out that the ecological and evolutionary history of each population is unique , and the capacity of Xylella fastidiosa varies among subspecies and potentially among populations . Therefore , we argue that the local population as a complement to the global population can speed up the EA convergence and obtain better results . In this paper , we explain and improve the naive transformer structure by analogy with the validated evolutionary algorithms , where a parallel local path is designed inspired by the local population concept in EA . 2.2 Vision Transformers . Since Transformer structure is proposed for the machine translation task [ 69 ] , many improved language models [ 51 , 21 , 53 , 54 , 7 ] follow it and achieve great results . Some later works [ 32 , 74 , 15 , 38 , 3 ] improve the basic transformer structure for better efficiency . Inspired by the high performance of transformer in NLP and benefitted from abundant computation , recent ViT [ 23 ] introduces the transformer to vision classification for the first time and sparks a new wave of excitement for many vision tasks , e.g. , detection [ 8 , 85 ] , segmentation [ 81 , 11 , 66 ] , generative adversarial network [ 36 , 72 ] , low-level tasks [ 10 ] , video processing [ 4 ] , general backbone [ 76 , 75 , 43 ] , self-supervision [ 12 , 2 , 13 , 48 ] , neural architecture search [ 73 , 40 ] , etc . Many researchers have focused on improving the basic transformer structure [ 64 , 82 , 77 , 26 , 65 , 79 , 19 , 80 , 16 , 68 , 61 , 17 , 42 ] , which is more challenging than other application-oriented works . Among these methods , DeiT [ 64 ] is undoubtedly a star job that makes the transformer performance better and training more data-efficient . Based on DeiT , this work focuses on improving the basic structure of the vision transformer , making it perform better considering both accuracy and efficiency . Besides , our approach supports multi-modal tasks using only one unified model while other transformer-based methods such as DeiT do not . 2.3 Space Filling Curve . A space-filling curve maps the multi-dimensional space into the 1D space , and we mainly deal with 2D SFCs in this paper . SFC acts as a thread that passes through all the points in the space while visiting each point only once , i.e. , every point except the starting and ending points is connected to two adjacent line segments . There are numerous kinds of SFCs , and the difference among them is in their ways of mapping to the 1D space . Peano [ 50 ] first introduces a mapping from the unit interval to the unit square in 1890 . Hilbert [ 33 ] generalizes the idea to a mapping of the whole space . G.M.Morton [ 46 ] proposes Z-order for file sequencing of a static two-dimensional geographical database . Subsequently , many SFCs are proposed [ 45 , 67 , 56 ] , e.g. , Sweep , Scan , Sierpiński , Lebesgue , Schoenberg , etc . Some researchers further apply SFC methods to practical applications [ 5 , 83 , 6 ] , such as data mining and bandwidth reduction , but so far , almost none has been applied to computer vision . By introducing SFC in our EAT , this paper aims to provide a systematic and scalable paradigm that uses a unified model to deal with multi-modal data , keeping the network architecture and data structure independent of the data dimensionality . 3 Preliminary Transformer . The Transformer structure in vision tasks usually refers to the encoder and mainly builds upon the MSA layer , along with FFN , Layer Normalization ( LN ) , and Residual Connection ( RC ) operations . MSA is equivalent to the fusion of several SA operations that jointly attend to information from different representation subspaces , formulated as : MultiHead ( Q , K , V ) = Concat ( Head 1 , Head 2 , . . . , Head h ) WO , here Head i = Attention ( QWQi , KW K i , V W V i ) = softmax [ QWQi ( KWKi ) T √ dk ] VWVi = AVW V i , ( 1 ) where WQi ∈ Rdm×dk , WKi ∈ Rdm×dk , WVi ∈ Rdm×dv , WO ∈ Rhdv×dm are parameter matrices ; dm is the input dimension , while dk and dv are hidden dimensions of each projection subspace ; h is the head number ; A ∈ Rl×l is the attention matrix and l is the sequence length . FFN consists of two cascaded linear transformations with a ReLU activation in between : FFN ( x ) = max ( 0 , xW1 + b1 ) W2 + b2 , ( 2 ) where W1 and W2 are weights of two linear layers , while b1 and b2 are corresponding biases . LN is applied before each layer of MSA and FFN , and the transformed x̂ is calculated by : x̂ = x + [ MSA | FFN ] ( LN ( x ) ) . ( 3 )

Summary: This paper presents a novel perspective to explains the rationality of vision transformer by analogy with a heuristic evolutionary algorithm and proposes an improved EAT model borrowing the dynamic local population concept from EA. Authors introduce the spatial-filling curve to serialize the input image, so only one unified model is required to handle multi-modal tasks. The extensive classification and multi-modal experiments show that the proposed method is effective and efficient.