Patent Publication Number: US-2023153379-A1

Title: Adaptive Token Sampling for Efficient Transformer

Description:
BACKGROUND 
     Transformer-based technology was originally developed for use in various Natural Language Processing (NLP) applications. This type of technology is characterized by the use of attention mechanisms to process input tokens. When processing a particular word under consideration, for instance, an attention mechanism can determine how much importance or focus should be placed on each of the other words in an input sentence. For example, when processing the word “it” that appears in a sentence, the attention mechanism may conclude that attention should be paid to another word in the sentence to which “it” presumably refers. 
     A developer faces technical challenges in extending transformer-based technology to other non-NLP fields, such as the image-processing domain. First, transformer-based operations are “expensive” in that they are highly data-intensive in nature. Second, some data items, such as images, are more complex than text-based tokens. In view of these factors, some computing platforms may lack sufficient resources to successfully apply transformer-based technology in many contexts. 
     SUMMARY 
     A transformer is described herein for using transformer-based technology to process data items, such as, but not limited to, images. The transformer increases the efficiency of the transformer-based technology by using a modified attention component. In operation, the modified attention component accepts embedding vectors that represent a plurality of item tokens, together with a classification token. A first stage of the modified attention component generates original attention information based on the embedding vectors. The original attention information has a plurality of attention values, each of which describes an importance that a particular token plays in the interpretation of another particular token. A second stage generates score information based on a portion of the original attention information that pertains to the classification token. A third stage produces modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed based on the score information. 
     Ultimately, the transformer generates transformer output information. In some implementations, a post-processing component classifies the content of the data item based on the transformer output information. 
     According to one advantage, operations performed by the transformer on the modified attention information can take place more efficiently compared to operations that would otherwise be performed on the original attention information. This is because the operations performed on the modified attention information are less data-intensive compared to operations that would otherwise be performed on the original attention information. This characteristic, in turn, enables some computing devices having limited computing capabilities to the run the transformer, whereas they would otherwise have been precluded. 
     According to additional illustrative aspects, the transformer can include plural instances of the modified attention component operating at different respective stages of the transformer. 
     According to additional illustrative aspects, the above-summarized scoring and sampling functions of the modified attention component are parameter-less, meaning that they are not based on the use of their own machine-trained weights. This further means that a developer can add the scoring and sampling functions to a preexisting transformer that has already been trained, without regard to the architecture of the preexisting transformer. The weights of the modified transformer thus produced can be fine-tuned, although it is not necessary to do so. This characteristic expedites the deployment of the above-summarized technology and broadens its use cases. 
     The above-summarized technology can be manifested in various types of systems, devices, components, methods, computer-readable storage media, data structures, graphical user interface presentations, articles of manufacture, and so on. 
     This Summary is provided to introduce a selection of concepts in a simplified form; these concepts are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    shows an illustrative transformer that uses at least one modified attention component. 
         FIG.  2    shows an illustrative modified attention component for use in the transformer of  FIG.  1   . 
         FIG.  3    illustrates a principle by which the modified attention component of  FIG.  1    generates score information. 
         FIG.  4    shows one technique by which the modified attention component performs a sampling operation. 
         FIG.  5    shows an encoder block for use in the transformer of  FIG.  1   . The encoder block includes the modified encoder of  FIG.  2   . 
         FIG.  6    shows an example of the operation of the transformer of  FIG.  1   . 
         FIG.  7    shows one technique by which a training system can introduce scoring and sampling functions into a preexisting transformer. 
         FIG.  8    shows illustrative operations performed by a pre-processing system of the transformer of  FIG.  1   . 
         FIGS.  9  and  10    together show illustrative operations performed by a transformer neural network of the transformer of  FIG.  1   . 
         FIG.  11    shows computing equipment that can be used to implement the computing systems shown in  FIG.  1     
         FIG.  12    shows an illustrative type of computing system that can be used to implement any aspect of the features shown in the foregoing drawings. 
     
    
    
     The same numbers are used throughout the disclosure and figures to reference like components and features. Series 100 numbers refer to features originally found in  FIG.  1   , series 200 numbers refer to features originally found in  FIG.  2   , series 300 numbers refer to features originally found in  FIG.  3   , and so on. 
     DETAILED DESCRIPTION 
     This disclosure is organized as follows. Section A describes a transformer that uses transformer-based technology to process data items. Section B sets forth illustrative methods that explain the operation of the transformer of Section A. And Section C describes illustrative computing functionality that can be used to implement any aspect of the features described in Sections A and B. 
     A. Illustrative Transformer and Method of Training 
       FIG.  1    shows an illustrative transformer  102  that uses at least one modified attention component. In the most general case, the transformer  102  can be applied to expedite the processing of any data items, such as images, video, audio, graphics, model data, instances of text, markup language content, and so on, or any combination thereof. However, to facilitate explanation, the transformer  102  will be described below principally in the context of processing images. In that context, the transformer  102  may be referred to as a vision transformer. Note that any principle described below in the context of image-processing is more generally applicable to the processing of any data items. Further, any mention of a “vision transformer” below applies more generally to any “transformer.” 
     The transformer  102  (henceforth “vision transformer”  102 ) includes a pre-processing system  104 , a transformer neural network  106 , and a post-processing component  108 . The pre-processing system  104  generates embedding vectors  110  for input to the transformer neural network  106 . The embedding vectors  110  describe different tokens  112 , including: a plurality of image tokens  114  drawn from an input image  116 , together with a classification (CLS) token  118 . The transformer neural network  106  uses plural encoding blocks to convert the embedding vectors  110  into transformer output information  120 . The post-processing component  108  maps the transformer output information  120  into an output result. For example, the post-processing component  108  may classify content that appears in the input image  116  based on the transformer output information  120 . A training system  122  trains one or more machine-trained models used by the vision transformer  102 . 
     Beginning with an explanation of the pre-processing system  104 , a token-generating component  124  produces the plurality of image tokens  114  that represent the input image  116 . In some implementations, the token-generating component  124  performs this task by slicing the input image  116  into a plurality of smaller images.  FIG.  1    shows dashed lines intersecting the input image  116  to illustrate this slicing operation. However, in actual practice, the token-generating component  124  can divide the input image  116  into smaller parts than is illustrated in  FIG.  1   . Alternatively, or in addition, the token-generating component  124  can use a convolutional neural network (CNN) (not shown) to transform the input image  116  into one or more feature maps. The token-generating component  124  can then slice the feature map(s) into smaller maps. These examples are set forth in the spirit of illustration, not limitation. Generally, the image tokens  114  produced by the token-generating component  124  represent any respective parts of the input image  116 , and/or respective parts of information that are derived on the basis of the input image  116 . 
     The token-generating component  124  can perform other types of partitioning when applied to other data items (besides images). For example, the token-generating component  124  can divide a stream of audio data into temporally-consecutive audio snippets. 
     An embedding component  126  converts the image tokens  114  into the embedding vectors. The embedding component  126  can perform this task by flattening each image token. The embedding component  126  can flatten an image token by concatenating its separate rows or columns, to produce a single token vector. The embedding component  126  can then use any type of transformation function to convert the token vector into an embedding vector. For example, the embedding component  126  can use a machine-trained neural network, a lookup table, a hashing function, etc. to perform this function. The embedding component  126  can convert the CLS token  118  into an encoding vector by using a lookup table to identify a token vector associated with the CLS token, and then using any of the above-identified kinds of transformations to convert the token vector into an embedding vector. 
     The embedding component  126  can also incorporate position information into each embedding vector. The position information describes a position of each input token in a sequence of input tokens. For example, the embedding component  126  can rely on a predetermined ordering rule to assign indices to each of the image tokens  114 . For each of the image tokens, the embedding component  126  can then combine vector information describing its position with the embedding vector, to produce a position-modified embedding vector. In some implementations, the embedding component  126  can assign a position 0 to the CLS token  118 . 
     The transformer neural network  106  uses a processing pipeline that includes one or more encoder blocks ( 128 ,  130 , . . . ,  132 ) to transform the embedding vectors  110  into the transformer output information  120 . For example, without limitation, the processing pipeline can include twelve encoder blocks. The first encoder block  128  operates on the embedding vectors  110  to produce first-stage hidden state information. The second encoder block  130  maps the hidden state information produced by the first encoder block  128  into second-stage hidden state information, and so. The transformer output information  120  represents the hidden state information produced by the last encoder block  132 . 
     Note that the hidden state information produced by each encoder block maintains its association with particular input tokens  112 . For example, part of the hidden state information produced by the first encoder block  128  is associated with the first input token, another part of the hidden state information is associated with the second input token, and so on. But each encoder block otherwise applies the same machine-trained model to all of the input tokens. 
     At least one of the encoder blocks ( 128 ,  130 , . . . ,  132 ) includes a modified attention component. For example,  FIG.  1    shows that all of the encoder blocks ( 128 ,  130 , . . . ,  132 ) include respective modified attention components ( 134 ,  136 , . . . ,  138 ). Other implementations can intersperse modified attention components among non-modified attention components in any manner to accommodate any application-specific objectives. For instance, other implementations can predominantly use the modified attention components in later stages of the image processing pipeline, e.g., by using the modified attention components in the last eight encoder blocks of a twelve-block processing pipeline. Addition details regarding each encoder block will be set forth below in connection with the explanation of  FIG.  5   . 
       FIG.  1    provides a high-level overview of the first modified attention component  134 . The modified attention component  134  includes an attention component  140  that transforms input information (described below) into original attention information. A scoring component  142  produces score information based on the original attention information. The score information includes scores assigned to the image tokens  114 . A sampling component  144  produces modified attention information, as guided by a sampling operation that is performed based on the score information. The modified attention information differs from the original attention information by eliminating at least some attention values that are included in the original attention information. The operation of the modified attention component  134  will be set forth below in greater detail in connection with the explanation of  FIG.  2   . 
     The encoder blocks ( 130 , . . . ,  132 ) which follow the modified attention component  134  in the pipeline directly or indirectly process the modified attention information, rather than the original attention information. The operations performed by these encoder blocks ( 130 , . . . ,  132 ) are made less computationally intensive by virtue of the fact that the modified attention information is smaller than the original attention information. Overall, this provision reduces the amount of work that the vision transformer  102  needs to perform to process the input image  116 . Correspondingly, the use of the modified attention component(s) reduces the amount of computing resources consumed by the vision transformer  102  in processing the input image  116 . 
       FIG.  1    illustrates a path  146  of processing operations performed on an embedding vector  148  associated with the CLS token  118 . The transformer output information  120  represents hidden state information produced for the CLS token by the last encoder block  132 . The post-processing component  108  can use any type of feed-forward network to map the transformer output information  120  into intermediate output information. The post-processing component  108  can then use a softmax operation (e.g., a normalized exponential function) to map the intermediate output information into a probability distribution. The probability distribution identifies the likelihood that content in the image  116  corresponds to each of a plurality of possible image classes. The post-processing component  108  can use an argmax operation to output the classification having the highest likelihood. 
     In other implementations, the post-processing component  108  can perform other operations. For example, alternatively, or in addition, the post-processing operation  108  can detect the location of an object in the input image  116 . Alternatively, or in addition, the post-processing operation  108  can enhance the input image  116 . Alternatively, or in addition, the post-processing component  108  can introduce new content to the image  116 , and so on. 
       FIG.  2    shows further details of the representative modified attention component  134  introduced in  FIG.  1   . In one case, the modified attention component  134  includes plural heads  202 . Each head processes input information fed to it with respect to a particular vector sub-space. As a result of this operation, the different heads  202  can provide different insight regarding the input information fed to the modified attention component  134 . The modified attention component  134  can produce its final attention information by summing together the attention information produced by its plural heads  202 . However, to facilitate explanation, the following description will be provided for only a representative head  204 . A head-specific and level-specific set of machine-trained weights governs the operation of the representative head  204 . 
     The modified attention component  134  receives input information X. Since the modified attention component  134  is part of the first encoder block  128 , the input information X represents the embedding vectors  110  provided to the transformer neural network  106 , e.g., combined together in a single matrix. (If the modified attention component  134  was a later-stage component in the image-processing pipeline, then the input information X would represent the output of a preceding encoder block.) The modified attention component  134  uses three different linear transformers ( 206 ,  208 ,  210 ) to respectively produce query information (Q), key information (K), and value information (V). More specifically, the linear transformer  206  produces the query information Q by multiplying the input matrix X by a first machine-trained weighting matrix W Q . The linear transformer  208  produces the key information K by multiplying the same input matrix X by a second machine-trained weighting matrix W K . The linear transformer  210  produces the value information V by multiplying the same input matrix X by a third machined weight matrix W V    
     The attention component  140  computes the original attention information  212  by performing self-attention, given by the following equation: 
     
       
         
           
             
               
                 
                   
                     attention 
                     ⁢ 
                         
                     information 
                   
                   = 
                   
                     softmax 
                     ⁢ 
                         
                     
                       
                         ( 
                         
                           
                             Q 
                             ⁢ 
                             
                               K 
                               T 
                             
                           
                           
                             d 
                           
                         
                         ) 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     Equation (1) involves taking the dot product of Q by the transpose of K, and then dividing that dot product by a scaling factor √{square root over (d)}, where d may represent the dimensionality of the machine-learned model. This yields a scaled result. Equation (1) then involves computing the softmax of the scaled result, which yields an attention matrix A (the original attention information  212  shown in  FIG.  1   ). Without the modification described herein, a final attention result would be produced by multiplying the attention matrix A by the value information V, e.g., attention(Q, K, T)=AV. Background information regarding Equation (1) is provided in the seminal paper by VASWANI, et al., “Attention Is All You Need,” in 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017, 11 pages. 
     The original attention information  212  includes a plurality of rows of attention values. Each row is associated with a particular input token. For example, the first row  214  (A 1 ) of the original attention information  212  pertains to the CLS token  118 . The second row of the original attention information  212  pertains to the first image token, and so on. Each attention value describes how much the interpretation of a particular input token under consideration depends on another input token. For example, a third attribute value in the first row  214  describes how much the interpretation of the CLS token depends on a particular image token. 
     The scoring component  142  scores each image token with respect to its importance in generating an accurate image classification. In one implementation, the scoring component  142  scores each particular image token j by multiplying the attribute value corresponding to the particular image token j in the first row  214  of the original attention information  212  by the value information associated with this image token. That is, 
         s   j   =A   1,j   ×∥V   j ∥  (2).
 
     In Equation (2), A 1,j  represents the attention value associated with the image token j under consideration in the first row  214  of the original attention information  212 . ∥V j ∥ represents the Euclidean norm (e.g., L2 norm) of the entry in the value information V associated with the token j under consideration. s j  represents the score assigned to the image token j under consideration. The term “score information” is used herein to collectively refer to the scores produced by Equation (2). Note that Equation (2) is set forth here in the spirit of illustration, not limitation; other implementations can express the usefulness of the image token j to the classification task in different ways. 
     The sampling component  144  produces modified attention information  218  based on the score information  216  produced by the scoring component  142 . From a high-level perspective, the sampling component  144  treats the score information as a probability distribution. Guided by that distribution, it picks rows of attribute values in the original attention information  212  that should be retained. Implicitly, the sampling component  144  also chooses which rows of the attribute values should be omitted (if any) in the modified attention information  218 . As previously stated, rows in the original attention information are associated with respective image tokens. Thus, the sampling component  144  can be said to choose which image tokens are allowed to contribute to the task of classifying the image  116 , and which image tokens are not. One implementation of the sampling component  144  will be set forth below in connection with  FIG.  4   . 
     The sampling component  144  can remove rows of attention values in different ways. For example, the sampling component  144  can collapse the original attention information  212  to account for the removal of rows. Alternatively, the sampling component  144  can mask each row of attention values by changing the values in the row to zero. The inference-stage manifestation of the vision transformer  102  can use the first removal option, while a batch-driven training-stage manifestation of the vision transformer  102  can use the second removal option. 
     Note that the removal (or effective removal) of a row of attention values does not entirely remove the influence of an associated image token in the final classification operation. For example, each row that is retained in the modified attention information  218 , which is associated with a retained image token, still describes an extent to which the image token that has been removed is important to the retained image token&#39;s own interpretation. In this sense, the sampling component  144  can be said to “softly” down-sample within the pipeline of image-processing operations. The sampling component  144  does not erase the memory of the existence of image tokens that have been removed, as their influence remains in the modified attention information  218 . 
     The sampling component  144  yields a modified attention matrix A S , which corresponds to the modified attention information  218  shown in  FIG.  2   . The modified attention component  134  produces its final output result by multiplying the modified attention matrix A S  by the value information V, e.g., Attention S (Q, K, T)=A S V. 
       FIG.  3    illustrates the significance of the attribute values in the first row  214  of the original attention information  212 . The attribute values reflect the importance of each image token in the interpretation of the classification token  118 .  FIG.  3    depicts different illustrative degrees of focus using arrows having different thicknesses. The arrow that points from the fourth image token  302  to the CLS token has the thickest arrow. This means that the fourth image toke  302  has the greatest importance to the proper interpretation of what is being depicted in the image  116 , which will be encoded in the transformer output information  120  fed to the post-processing component  108 . 
       FIG.  4    shows one manner of operation of the sampling component  144 . In a first stage, assume that the sampling component  144  receives the original score information  402  produced by the scoring component  142 , e.g., which reflects individual scores assigned by the scoring component  142  to the image tokens using Equation (2). In a second stage, the sampling component  144  ranks the scores from smallest to largest. This yields sorted score information  404 . 
     In a third stage, the sampling component  144  generates a cumulative distribution function (CDF)  406  based on the sorted scores. In some non-limiting implementations, the sampling component  144  can generate the CDF  406  based on the following equation: 
     
       
         
           
             
               
                 
                   
                     
                       C 
                       ⁡ 
                       ( 
                       i 
                       ) 
                     
                     = 
                     
                       
                         ∑ 
                         
                           j 
                           = 
                           0 
                         
                         
                           j 
                           = 
                           i 
                         
                       
                       
                         S 
                         ⁡ 
                         ( 
                         j 
                         ) 
                       
                     
                   
                   . 
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     In this equation, S(j) represents a score associated with an image token j in the sorted score information  404 . C(.) is the CDF  406  of the sorted score information  404 . Equation (3) specifies that the value C(i) for any index value i in the CDF  406  is computed by summing the scores in the sorted score information  404 , up to i=j. The sampling component  144  can also produce a sampling function by forming the inverse of the CDF function C(.), that is, by forming C −1 (.). 
     In the fourth stage, the sampling component  144  samples among the image tokens  114  based on the CDF  406 . Referring to the depiction of the CDF  406  in  FIG.  4   , the sampling component  144  can perform this function by uniformly picking points along the y axis of the CDF  406 , e.g., by picking K points at regular intervals along the y axis  408  of the CDF  406 . The sampling component  144  consults the curve of the CDF  406  to identify a token index value (along the x axis  410 ) that is associated with each chosen point along the y axis  408 . The sampling component  144  treats the subset of image tokens identified in this manner as selected image tokens. The sampling component  144  treats the remainder of the image tokens as non-selected image tokens. In a fourth stage, the sampling component  144  can then remove the attribute values of the non-selected image tokens in the manner explained above in connection with  FIG.  2   . 
     The sampling strategy described above has the following characteristics. First, in some implementations, the sampling component  144  is configured to choose a maximum number K of image tokens, for which attention values will be retained in the modified attention information  218 . In many implementations, the maximum number K is less than the total number N of image tokens  114  originally under consideration. Further note that the sampling operation performed by the sampling component  144  does not necessarily result in the identification of a new image token each time it is invoked. In some cases, two or more points selected along the y axis  408  resolve to the same token index value along the x axis  410 . In this situation, the sampling component  144  will only choose a single image token. As a final result, the sampling strategy chooses K′ image tokens, where K′≤K.  FIG.  4    shows a plot  412  that represents the indices of the K′ image tokens identified by the sampling operation performed by the sampling component  144 . Stated in another way, the sampling component  144  dynamically and adaptively selects a variable number of tokens based on the characteristics of the image  116  under consideration, and the predetermined upper-bound value K. 
     Second, note that the sampling component  144  performs uniform sampling along the y axis  408  of the CDF  406 . But this has the end effect of non-uniformly selecting indices along the x axis  410 , meaning that the points selected along the x axis are separated by non-uniform distances. In general, the sampling strategy that guides the sampling component  144  is a form of inverse transform sampling. 
     Third, note that the above sampling strategy promotes the selection of image tokens having high scores. But the sampling component  144  also chooses from among image tokens having lower scores, with a reduced likelihood compared to the image tokens with higher scores. This characteristic is useful because it retains information that may prove important at a later stage in the image-processing pipeline, but, at the current stage, is discounted as having low value due to the influence of noise, etc. 
     Finally, note that other implementations of the sampling component  144  can perform sampling in other ways compared to that set forth above. For example, other implementations can choose the image tokens having the top K scores, without performing inverse transform sampling with respect to a probability distribution. Other implementations can use other algorithms for picking samples as guided by a probability distribution. 
       FIG.  5    shows one non-limiting implementation of an encoding block  502 . Each of the encoding blocks ( 128 ,  130 , . . . ,  132 ) can adopt the block architecture shown in  FIG.  5   . The encoder block includes a pipeline of processing components including a first normalization component  504 , a modified attention component  506 , a first residual connection  508 , a second normalization component  510 , a feed-forward network (FNN)  512 , and a second residual connection  514 . 
     Each normalization component performs normalization of values in input information in different application-specific ways, e.g., by adjusting values in a layer based on the mean and standard deviation of those values in the layer. Each residual connection combines (e.g., sums) input information x fed into a specified function g(x) with output information produced by the specified function g(x). In other words, a residual connection performs the combination x+g(x). The FNN  512  can use any number of layers of neurons to transform input information into output information. For example, the FNN  512  may represent a fully-connected multi-layered perceptron (MLP). 
     Other implementations can vary the encoder bock architecture shown in  FIG.  5    in any way(s). Further note that each encoder block need not include a modified attention component. That is, other encoder blocks can perform the attention operation of the attention component  140  (of  FIG.  2   ), but not the functions of the scoring component  142  and the sampling component  144 . 
       FIG.  6    describes the operation of the vision transformer  102  at three respective stages along the pipeline of image-processing operations performed by the transformer neural network  106 . More specifically, each encoder block of the transformer neural network  106  produces hidden state information.  FIG.  6    provides a visual representation of the hidden state information at the three respective stages. In each image shown in  FIG.  6   , a white region represents one or more original image tokens that have been removed through the operation of one or more modified attention components. More specifically, the vision transformer  102  does not remove the image tokens per se; rather, it removes the attention values associated with those image tokens. 
     More specifically, assume that the goal of the classification is to recognize the most prominent object in the input image, which is a bird. At a first stage, a first image  602  shows that one or more modified attention components have removed image tokens that principally show the background of the image. At a second stage, a second image  604  shows that one or more modified attention components have removed additional parts of the background. At a third stage, a third image  606  shows that the one or more modified attention components have removed even some portions of the bird that are considered non-essential to classifying the bird, and/or are redundant. The modified attention component(s) have also removed a portion of foliage that has been assessed as less important than the bird. In summary, the vision transformer&#39;s strategy of selectively removing less important parts of image content expedites the processing of the input image without markedly harming the accuracy at which the principal object in the image is classified. 
       FIG.  7    shows one technique for deploying a vision transformer having one or more modified attention components. In a first phase, assume that a training system  702  trains the weights  704  of any existing vision transformer  706  having any type of attention component(s)  708  and any transformer architecture. In a second stage, a developer adds the scoring and sampling functions  710  set forth in connection with  FIG.  2    to the existing vision transformer  706 , to produce a modified vision transformer  712 . The modified vision transformer  712  can be immediately deployed without further adjustment to its machine-trained weights  704 . This is possible because the scoring and sampling functions  710  do not themselves rely on machine-trained weights. In other words, the scoring and sampling functions  710  operate on original attention information generated by attention components (which do use machine-trained weights), but the scoring sampling functions  710  themselves do not perform their functions based on their own machine-trained weights. 
     A developer may nevertheless choose to fine-tune the modified vision transformer  712  by re-training all of its weights (as shown in a third optional stage). This yields a fine-tuned vision transformer  714  having fine-tuned machine-trained weights  716 . In still another scenario, a developer may choose to build the kind of vision transformer  102  shown in  FIG.  1    from “scratch,” that is, without starting with any type of pre-existing vision transformer. 
       FIG.  7    also serves to highlight additional advantages of the vision transformer  102  of  FIG.  1   . A developer can introduce the scoring and sampling functions  710  described herein to any existing vision transformer, regardless of its architecture. Further, the developer need not devote labor-intensive and resource-intensive operations to retrain an existing vision transformer that has been modified to include the scoring and sampling functions  710 . 
     To repeat a point made at the outset of Section A, the principles described herein can be applied to other types of data items besides images. For example, consider the objective of classifying information presented in an audio item. The pre-processing system  104  can break the audio item into a plurality of audio tokens, e.g., corresponding to temporally-consecutive snippets of audio. Each audio token can describe features associated with a respective audio snippet, e.g., including mel-frequency cepstral coefficients, phonemes, word fragments, words, etc. The pre-processing system  104  can further append a classification token to the audio tokens. The transformer neural network  104  can use at least one modified attention component to remove attention values associated with the least valuable audio tokens (where value is assessed in terms of the capacity of an audio token to productively contribute to the classification of the audio item). For example, the modified attention component(s) can remove background noise in the audio item. 
     Further note that the principles set forth herein are applicable to the processing of item tokens having any informational scope. For example, the principles set forth herein apply to cases in which item tokens correspond to individual words, or word fragments identified by any kind of word-partitioning algorithm (e.g., the WordPiece algorithm), or even individual n-character sequences within words, etc. 
     Another way to convey the general applicability of the principles described herein is via  FIG.  7   . The scoring and sampling functions  710  shown there can be regarded as a data-agnostic and application-agnostic toolkit or utility that can be used to improve the efficiency of any transformer, regardless of the nature of data on which it operates, the way it represents that data, and the end-objectives of the transformer, etc. 
     B. Illustrative Processes 
       FIGS.  8 - 10    show processes that explain the operation of the vision transformer  102  of Section A in flowchart form. Since the principles underlying the operation of the vision transformer  102  have already been described in Section A, certain operations will be addressed in summary fashion in this section. Each flowchart is expressed as a series of operations performed in a particular order. But the order of these operations is merely representative, and can be varied in other implementations. Further, any two or more operations described below can be performed in a parallel manner. In one implementation, the blocks shown in the flowcharts that pertain to processing-related functions can be implemented by the hardware logic circuitry described in Section C, which, in turn, can be implemented by one or more hardware processors and/or other logic units that include a task-specific collection of logic gates. 
     More specifically,  FIG.  8    shows a process  802  that represents one manner of operation of the pre-processing system  104  of  FIG.  1   . In block  804 , the pre-processing system  104  obtains plural item tokens (e.g., image tokens  114 ) that represent parts of a data item (e.g., the image  116 ). In block  806 , the pre-processing system  104  obtains the classification token  118 . In block  808 , the pre-processing system  104  converts the item tokens  114  and the classification token  118  into embedding vectors  110 . 
       FIGS.  9  and  10    together show a process  902  that explains one manner of operation of the transformer neural network  106  of  FIG.  1   . In block  904 , the transformer neural network  106  receives embedding vectors  110  that represent a plurality of item tokens (e.g., image tokens  114 ) generated based on a data item (e.g., the image  116 ), and the classification token  118 . In blocks  906 ,  908 , and  1002 , the transformer neural network  106  performs an attention operation. The attention operation includes, in block  906 , generating original attention information  212  based on the embedding vectors  110 , the original attention information  212  having a plurality of attention values, each attention value describing an importance that a particular token plays in an interpretation of another particular token. In block  908 , the transformer neural network  106  generates score information  216  based on attention values in the original attention information  212  that pertain to the classification token  118 . In block  1002 , the transformer neural network  106  generates modified attention information  218  by removing attention values from the original attention information  212  based on the score information  216 . In block  804 , the transformer neural network  106  performs subsequent operations based on the modified attention information  218 . The subsequent operations perform fewer operations by using the modified attention information  218  rather than the original attention information  212 . 
     C. Representative Computing Functionality 
       FIG.  11    shows an example of computing equipment that can be used to implement any of the systems summarized above. The computing equipment includes a set of user computing devices  1102  coupled to a set of servers  1104  via a computer network  1106 . Each user computing device can correspond to any device that performs a computing function, including a desktop computing device, a laptop computing device, a handheld computing device of any type (e.g., a smartphone, a tablet-type computing device, etc.), a mixed reality device, a wearable computing device, an Internet-of-Things (IoT) device, a gaming system, and so on. The computer network  1106  can be implemented as a local area network, a wide area network (e.g., the Internet), one or more point-to-point links, or any combination thereof. 
       FIG.  11    also indicates that the training system  122  and the transformer  102  can be spread across the user computing devices  1102  and/or the servers  1104  in any manner. For instance, in some cases, the transformer  102  is entirely implemented by one or more of the servers  1104 . Each user may interact with the servers  1104  via a user computing device. In other cases, the transformer  102  is entirely implemented by a user computing device in local fashion, in which case no interaction with the servers  1104  is necessary. In another case, the functionality associated with the transformer  102  is distributed between the servers  1104  and each user computing device in any manner. 
       FIG.  12    shows a computing system  1202  that can be used to implement any aspect of the mechanisms set forth in the above-described figures. For instance, the type of computing system  1202  shown in  FIG.  12    can be used to implement any user computing device or any server shown in  FIG.  11   . In all cases, the computing system  1202  represents a physical and tangible processing mechanism. 
     The computing system  1202  can include one or more hardware processors  1204 . The hardware processor(s)  1204  can include, without limitation, one or more Central Processing Units (CPUs), and/or one or more Graphics Processing Units (GPUs), and/or one or more Application Specific Integrated Circuits (ASICs), and/or one or more Neural Processing Units (NPUs), etc. More generally, any hardware processor can correspond to a general-purpose processing unit or an application-specific processor unit. 
     The computing system  1202  can also include computer-readable storage media  1206 , corresponding to one or more computer-readable media hardware units. The computer-readable storage media  1206  retains any kind of information  1208 , such as machine-readable instructions, settings, data, etc. Without limitation, the computer-readable storage media  1206  may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tape, and so on. Any instance of the computer-readable storage media  1206  can use any technology for storing and retrieving information. Further, any instance of the computer-readable storage media  1206  may represent a fixed or removable unit of the computing system  1202 . Further, any instance of the computer-readable storage media  1206  may provide volatile or non-volatile retention of information. 
     More generally, any of the storage resources described herein, or any combination of the storage resources, may be regarded as a computer-readable medium. In many cases, a computer-readable medium represents some form of physical and tangible entity. The term computer-readable medium also encompasses propagated signals, e.g., transmitted or received via a physical conduit and/or air or other wireless medium, etc. However, the specific term “computer-readable storage medium” expressly excludes propagated signals per se in transit, while including all other forms of computer-readable media. 
     The computing system  1202  can utilize any instance of the computer-readable storage media  1206  in different ways. For example, any instance of the computer-readable storage media  1206  may represent a hardware memory unit (such as Random Access Memory (RAM)) for storing transient information during execution of a program by the computing system  1202 , and/or a hardware storage unit (such as a hard disk) for retaining/archiving information on a more permanent basis. In the latter case, the computing system  1202  also includes one or more drive mechanisms  1210  (such as a hard drive mechanism) for storing and retrieving information from an instance of the computer-readable storage media  1206 . 
     The computing system  1202  may perform any of the functions described above when the hardware processor(s)  1204  carry out computer-readable instructions stored in any instance of the computer-readable storage media  1206 . For instance, the computing system  1202  may carry out computer-readable instructions to perform each block of the processes described in Section B. 
     Alternatively, or in addition, the computing system  1202  may rely on one or more other hardware logic units  1212  to perform operations using a task-specific collection of logic gates. For instance, the hardware logic unit(s)  1212  may include a fixed configuration of hardware logic gates, e.g., that are created and set at the time of manufacture, and thereafter unalterable. Alternatively, or in addition, the other hardware logic unit(s)  1212  may include a collection of programmable hardware logic gates that can be set to perform different application-specific tasks. The latter class of devices includes, but is not limited to Programmable Array Logic Devices (PALs), Generic Array Logic Devices (GALs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), etc. 
       FIG.  12    generally indicates that hardware logic circuitry  1214  includes any combination of the hardware processor(s)  1204 , the computer-readable storage media  1206 , and/or the other hardware logic unit(s)  1212 . That is, the computing system  1202  can employ any combination of the hardware processor(s)  1204  that execute machine-readable instructions provided in the computer-readable storage media  1206 , and/or one or more other hardware logic unit(s)  1212  that perform operations using a fixed and/or programmable collection of hardware logic gates. More generally stated, the hardware logic circuitry  1214  corresponds to one or more hardware logic units of any type(s) that perform operations based on logic stored in and/or otherwise embodied in the hardware logic unit(s). Further, in some contexts, each of the terms “component,” “module,” “engine,” “system,” and “tool” refers to a part of the hardware logic circuitry  1214  that performs a particular function or combination of functions. 
     In some cases (e.g., in the case in which the computing system  1202  represents a user computing device), the computing system  1202  also includes an input/output interface  1216  for receiving various inputs (via input devices  1218 ), and for providing various outputs (via output devices  1220 ). Illustrative input devices include a keyboard device, a mouse input device, a touchscreen input device, a digitizing pad, one or more static image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, a voice recognition mechanism, any position-determining devices (e.g., GPS devices), any movement detection mechanisms (e.g., accelerometers, gyroscopes, etc.), and so on. One particular output mechanism may include a display device  1222  and an associated graphical user interface presentation (GUI)  1224 . The display device  1222  may correspond to a liquid crystal display device, a light-emitting diode display (LED) device, a cathode ray tube device, a projection mechanism, etc. Other output devices include a printer, one or more speakers, a haptic output mechanism, an archival mechanism (for storing output information), and so on. The computing system  1202  can also include one or more network interfaces  1226  for exchanging data with other devices via one or more communication conduits  1228 . One or more communication buses  1230  communicatively couple the above-described units together. 
     The communication conduit(s)  1228  can be implemented in any manner, e.g., by a local area computer network, a wide area computer network (e.g., the Internet), point-to-point connections, etc., or any combination thereof. The communication conduit(s)  1228  can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols. 
       FIG.  12    shows the computing system  1202  as being composed of a discrete collection of separate units. In some cases, the collection of units corresponds to discrete hardware units provided in a computing device chassis having any form factor.  FIG.  12    shows illustrative form factors in its bottom portion. In other cases, the computing system  1202  can include a hardware logic unit that integrates the functions of two or more of the units shown in  FIG.  1   . For instance, the computing system  1202  can include a system on a chip (SoC or SOC), corresponding to an integrated circuit that combines the functions of two or more of the units shown in  FIG.  12   . 
     The following summary provides a non-exhaustive set of illustrative examples of the technology set forth herein. 
     (A1) According to a first aspect, some implementations of the technology described herein includes a method (e.g., the processes  802  and  902 ) for processing a data item (e.g., an image). The method includes: obtaining (e.g.,  804 ) plural item tokens (e.g., image tokens) that represent the data item; obtaining (e.g.,  806 ) a classification token; and converting (e.g.,  808 ) the item tokens and the classification token into embedding vectors. In an attention operation, the method uses a transformer neural network (e.g.,  106 ) for: generating (e.g.,  906 ) original attention information based on the embedding vectors, the original attention information having a plurality of attention values, each attention value describing an importance that a particular token plays in an interpretation of another particular token; generating (e.g.,  908 ) score information based on attention values in the original attention information that pertain to the classification token; and generating (e.g.,  1002 ) modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed based on the score information. The method then involves performing (e.g.,  1004 ) subsequent operations in the transformer neural network based on the modified attention information. The subsequent operations perform fewer operations by using the modified attention information rather than the original attention information. 
     (A2) According to some implementations of the method of A1, the data item is an image, and the item tokens are image tokens that represent respective parts of the image. 
     (A3) According to some implementations of the method of A1, the item tokens represent parts of a feature map produced by performing a convolution operation on the data item. 
     (A4) According to some implementations of any of the methods of A1-A3, the transformer neural network includes plural encoder blocks, wherein a particular encoder block of the plural encoder blocks performs the attention operation. 
     (A5) According to some implementations of any of the methods of A1-A3, the transformer neural network includes plural encoder blocks, and wherein two or more of the plural encoder blocks perform instances of the attention operation. 
     (A6) According to some implementations of any of the methods of A1-A5, the transformer neural network outputs transformer output information, and wherein the method further comprises classifying item content in the data item based on the transformer output information. 
     (A7) According to some implementations of any of the methods of A1-A6, the sampling operation includes choosing no more than a prescribed number of item tokens based on the score information, the prescribed number being less than a total number of item tokens. 
     (A8) According to some implementations of any of the methods of A1-A6, the sampling operation includes choosing item tokens using inverse transform sampling based on the score information. 
     (A9) According to some implementations of any of the methods of A1-A6, the sampling operation includes: ordering individual scores of the score information by score magnitudes, to provide ordered score information; generating a cumulative distribution function based on the ordered score information; and non-uniformly selecting item tokens using the cumulative distribution function. 
     (A10) According to some implementations of any of the method of A1-A9, the original attention information is expressed as an array of the attention values, and wherein the operation of generating modified attention information removes one or more groups (e.g., one or more rows) of attention values in the original attention information. 
     (A1) According to some implementations of any of the methods of A1-A3, the operation of generating score information and the operation of generating modified attention information are functions of the transformer neural network that do not have their own machine-trained weights. 
     (B1) According to another illustrative aspect, another method (e.g., the processes  802  and  902 ) is described for processing a data item (e.g., the image  116 ) using a transformer neural network (e.g.,  106 ). The method includes: receiving (e.g.,  904 ) embedding vectors that represent a plurality of item tokens generated based on a data item, and a classification token. In an attention operation, the method includes: generating (e.g.,  906 ) original attention information based on the embedding vectors, the original attention information having a plurality of attention values, each attention value describing an importance that a particular token plays in an interpretation of another particular token; generating (e.g.,  908 ) score information based on attention values in the original attention information that pertain to the classification token; and generating (e.g.,  1002 ) modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed based on the score information. The method further includes performing (e.g.,  1004 ) subsequent operations based on the modified attention information. The subsequent operations perform fewer operations by using the modified attention information rather than the original attention information. 
     (B2) According to some implementations of the method of (B1), the method is performed on a basis of machine-trained weights, the machine-trained weights governing the operation of generating original attention information, but not describing functions performed by the operation of generating score information and the operation of generating modified attention information. 
     In yet another aspect, some implementations of the technology described herein include a computing system (e.g., computing system  1202 ). The computing system includes hardware logic circuitry (e.g.,  1214 ) that is configured to perform any of the methods described herein (e.g., any of the methods A1-A11 and B1-B2). 
     In yet another aspect, some implementations of the technology described herein include a computer-readable storage medium (e.g., the computer-readable storage media  1206 ) for storing computer-readable instructions (e.g.,  1208 ). The computer-readable instructions, when executed by one or more hardware processors (e.g.,  1204 ), perform any of the methods described herein (e.g., methods of A1-A11 and B1-B2). The computer-readable instructions can also implement the first attention-based neural network, the second attention-based neural network, and the scoring neural network. 
     More generally stated, any of the individual elements and steps described herein can be combined, without limitation, into any logically consistent permutation or subset. Further, any such combination can be manifested, without limitation, as a method, device, system, computer-readable storage medium, data structure, article of manufacture, graphical user interface presentation, etc. The technology can also be expressed as a series of means-plus-format elements in the claims, although this format should not be considered to be invoked unless the phase “means for” is explicitly used in the claims. 
     As to terminology used in this description, the phrase “configured to” encompasses various physical and tangible mechanisms for performing an identified operation. The mechanisms can be configured to perform an operation using the hardware logic circuity  1214  of Section C. The term “logic” likewise encompasses various physical and tangible mechanisms for performing a task. For instance, each processing-related operation illustrated in the flowcharts of Section B corresponds to a logic component for performing that operation. 
     This description may have identified one or more features as “optional.” This type of statement is not to be interpreted as an exhaustive indication of features that may be considered optional; that is, other features can be considered as optional, although not explicitly identified in the text. Further, any description of a single entity is not intended to preclude the use of plural such entities; similarly, a description of plural entities is not intended to preclude the use of a single entity. Further, while the description may explain certain features as alternative ways of carrying out identified functions or implementing identified mechanisms, the features can also be combined together in any combination. Further, the term “plurality” refers to two or more items, and does not necessarily imply “all” items of a particular kind, unless otherwise explicitly specified. Further, the descriptors “first,” “second,” “third,” etc. are used to distinguish among different items, and do not imply an ordering among items, unless otherwise noted. The phrase “A and/or B” means A, or B, or A and B. Further, the terms “comprising,” “including,” and “having” are open-ended terms that are used to identify at least one part of a larger whole, but not necessarily all parts of the whole. Finally, the terms “exemplary” or “illustrative” refer to one implementation among potentially many implementations. 
     In closing, the description may have set forth various concepts in the context of illustrative challenges or problems. This manner of explanation is not intended to suggest that others have appreciated and/or articulated the challenges or problems in the manner specified herein. Further, this manner of explanation is not intended to suggest that the subject matter recited in the claims is limited to solving the identified challenges or problems; that is, the subject matter in the claims may be applied in the context of challenges or problems other than those described herein. 
     Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.