Patent Publication Number: US-2023154221-A1

Title: Unified pretraining framework for document understanding

Description:
BACKGROUND 
     Document intelligence is a broad research area that includes techniques for information extraction and understanding. In contrast to plain-text documents, a physical document may include multiple elements and/or object types: tables, figures, charts, text, and the like. Stated more simply, a physical document may include rich visual information. Furthermore, a physical document may vary in document types, e.g., a scientific paper, a form, a CV, and the like. 
     The combinations of elements and/or objects in a physical document may vary across such document types. That is, documents may include various combinations of multiple elements and layouts. Furthermore, the document type may be mixture of document types. Complex content, complex spatial layout, and combinations of elements/types, as well as font and style variations make automatic document understanding very challenging. For example, to understand text-rich documents such as letters, a document understanding system may need to focus on textual content, while paying attention to a context of long sequential content. To process semi-structured documents, such as forms, a document understanding system may be required to analyze spatially distributed words, while simultaneously paying particular attention to the spatial arrangement of the words. 
     Transformer-encoder models, such as the Bidirectional Encoder Representations from Transformers (BERT) model, have shown promise when applied to various natural language processing (NLP) tasks that require understanding of a physical document. Based on this promise, there has been growing interest in developing methods for pretraining an encoder model for the general task of document understanding. Once pretrained, an encoder model may be specifically trained (or fine-tuned) for a more specific document understanding task. 
     However, conventional pretraining methods, applied to encoder models for document understanding, have shown various limitations. One such limitation stems from the trend that many physical documents are composed of multiple semantic regions. Some conventional pretraining efforts adhere to sequence-to-sequence learning architectures that segment a document into a sequence of words. However, documents tend to have a hierarchical structure (e.g., words form sentences, sentences form a semantic region, and semantic regions form a document). Conventional sequence-to-sequence learning methods may not sufficiently account for such hierarchical structures. Also, the importance of words and sentences are highly context-dependent, i.e., the same word or sentence may have different importance in a different context. Conventional pretraining may not adequately account for the varying contexts of words. Also, input length becomes a problem for text-rich documents or multi-page documents. Conventional encoder-based document pretraining models may suffer from input length constraints as the input length of a document scales. 
     Another limitation of conventional pretraining methods arises because a full understanding of a document often requires more than just understanding the words in the document. The semantic structure of a document is not only determined by the text within the document, but also visual features encoded in the physical document such as tables, font sizes, styles, figures, and the like. Moreover, conventional pretraining (and training) methods for document understanding often fail to sufficiently capture semantic information encoded in the visual appearance of the text within a textual block. Many conventional pretraining methods only receive the words as input without considering multimodal (e.g., both textual and visual) content and alignment of multimodal information within semantic regions. 
     Conventional pretraining methods are also limited because understanding many documents requires considering the spatial layout of the document. Some conventional methods may encode spatial information via 2D position encoding. These conventional approaches may model spatial relationships with single-modality self-attention, which computes attention weights for long inputs. However, for semi-structured documents, such as forms and receipts, words are more related to their local surroundings. This corresponds strongly with human intuition, e.g., when an individuals looks at magazines or newspapers, the receptive fields are modulated by the individual&#39;s reading order and attention. These and other complexities of physical documents has rendered limited success for pretraining (and training) encoder models for document understanding tasks. 
     SUMMARY OF THE EMBODIMENTS 
     The technology described herein is directed towards enhanced methods and systems for pretraining a document encoder model based on multimodal self cross-attention between the modes. A non-limiting examplary method for training the model includes receiving image data that encodes a set of pretraining documents. A set of sentences may be extracted from the image data. A bounding box for each sentence may additionally be extracted. For each sentence of the set of sentences, a set of predicted features may be generated. The set of predicted features may be generated based on a gated-encoder model. The gated-encoder model may perform cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features may be based on a masking function and the sentence. The set of masked-visual features may be based on the masking function and the corresponding bounding box for the sentence. A document-encoder model may be pretrained. The pretraining may be based on the set of predicted features for each sentence of the set of sentences and one or more pretraining tasks. The one or more pretraining tasks may include at least one of masked sentence modeling, visual contrastive learning, or visual-language alignment. 
     In at least one embodiment, for each sentence of the set of sentences, a textual embedding may be generated. Additionally, a corresponding visual embedding may be generated. Generating the textual embedding may be based on a sentence encoder model. Generating the corresponding visual embedding may be based on a convolution model and a portion of the image data associated with the corresponding bounding box. The set of predicted features may be further based on the textual embedding for the sentence and the corresponding visual embedding. In some embodiments, the set of masked-textual features and the set of masked-visual features may be based on the masking function, the textual embedding for the sentence, and the corresponding visual embedding. 
     In various embodiments, generating a textual embedding for a sentence of the set of sentences includes generating a sentence embedding for the sentence. Generating the sentence embedding may be based on the sentence encoding model and a multiset of tokens included in the sentence. A position embedding for the corresponding bounding box may be generated. The position embedding may be based on a position, within the document, of the corresponding bounding box. The textual embedding for the sentence may be generated based on a combination of the sentence embedding and the position embedding for the bounding box. 
     In some embodiments, generating a corresponding visual embedding for a sentence of the set of sentences may include generating a position embedding for the corresponding bounding box. Generating the position embedding may be based on a position, within the document, of the corresponding bounding box. A region-of-interest (RoI) embedding for the corresponding bounding box may be generated. The RoI embedding may be generated based on the convolution model and the portion of the image data associated with the corresponding bounding box. The corresponding visual embedding for the sentence may be generated based on a combination of the ROI embedding and the position embedding for the bounding box. The set of predicted features may be generated further based on the position embedding for the bounding box. 
     In some embodiments, a corresponding set of visual representations may be generated. Generating the corresponding visual representations may be based on employing a vector quantization method to discretize the corresponding visual embedding. The set of masked-visual features may be generated based on applying the visual mask on the corresponding set of visual representations. Generating the set of masked-textual features and the set of masked-visual features may be further based on the masking function stochastically masking the textual embedding for the sentence and the corresponding visual embedding. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates an enhanced document encoder pretraining system implementing various embodiments presented herein. 
         FIG.  2    illustrates an enhanced pipeline for pretraining a document encoder model, according to various embodiments presented herein. 
         FIGS.  3 A- 3 B  shows an exemplary segmented pretraining documents, according to various embodiments. 
         FIG.  4 A  illustrates one embodiment of a method for pretraining a document encoder model, which is consistent with the various embodiments presented herein. 
         FIG.  4 B  illustrates an embodiment of another method for pretraining a document encoder model, which is consistent with the various embodiments presented herein. 
         FIG.  5    is a block diagram of an example computing device in which embodiments of the present disclosure may be employed. 
     
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
     The embodiments are directed towards a unified framework (or pipeline) for pretraining a language model (e.g., a transformer-encoder model) for document understanding tasks. As discussed above, conventional pretraining methods may fail to account for the semantic and visual tasks required to understand physical documents that vary in document type, as well as spatial layout and encoded object types (e.g., tables, charts, plots, graphs, figures, and the like). The various embodiments overcome the discussed limitations, as well as other limitations of conventional pretraining methods, at least by applying and combining multimodal (e.g., visual and textual) analyses of physical documents during the pretraining of encoder models. The embodiments include a pipeline that hierarchically encodes local multimodal features for the document via a combination of convolution and transformer-based language models. These features include both textual (e.g., semantic) features (e.g., a first modality of features) and visual features (e.g., a second modality of features), resulting in multimodal features. During pretraining, a self-attention mechanism is applied across the modalities of the features (e.g., cross-attention) to integrate the visual and semantic understanding of the document. The various embodiments improve upon the performance of pretraining tasks, as well as reducing the computational complexity when pretraining a transformer-based encoder model. 
     More particularly, a unified pretraining pipeline for document understanding is described. The pipeline receives image data encoding a set of physical pretraining documents (e.g., pretraining document images). Via the cross-attention mechanism, the pipeline (or framework) integrates image information (encoded in the image data) during model pretraining by taking advantage of a transformer architecture to learn cross-modal interactions between visual and textual information encoded in the document. To handle textual information, the pipeline encodes sentences using a hierarchical transformer encoder. A first level of the hierarchical encoder models the formation of the sentences from words. A second level of the hierarchical encoder models the formation of the document from sentences. 
     Via the structure of the hierarchical encodings, the embodiments pretrain a model by causing the model to “learn” how words form sentences and how sentences form documents. Meanwhile, at least due to the localization of the cross-attention computations, the embodiments reduce model computation complexity and increase the allowable number of input words, as compared to conventional pretraining methods. The enhanced pretraining described herein results in a pretrained document encoder model that mimics human reading behaviors at least because the hierarchical sentence/paragraph structure, which the pretraining captures, is a reasonable unit (e.g., a level of integration) for humans to read and understand. For example, when reading a complex physical document for understanding, individuals rarely check the interactions between arbitrary words across different regions of the document. Rather, individuals typically read a physical document by checking interactions across words co-located in a spatial “neighborhood” of the document. The cross-attention implemented by the embodiments may be localized to document “neighborhoods” to reduce the complexity of such computations. 
     Convolution mechanisms (e.g., implemented via convolution layers in a nerual network) are employed to extract “local” features of the document. The “size” of the locality is defined via the convolution “neighborhood” of the convolution layers, as characterized by the chosen convolution kernel. The convolution layers extract local features (across the convolution “neighborhood”) that encode visual and spatial information. Accordingly, the employment of the convolution layers provides an efficient complement to self-attention for addressing local intra-region dependencies in a document image. Furthermore, self-attention uses the input tokens to generate attention weights for capturing global dependencies. Thus, the pipeline combines convolution with multimodal self-attention to form a mixed attention mechanism that combines the advantages of both the convolution and self-attention operations. 
     The embodiments are contrasted with conventional pretraining methods in that the embodiments extract both the textual and visual features for each semantic region of the document. Furthermore, in the embodiments (and in contrast to conventional pretraining methods), a gated cross-attention transformer is employed in the pipeline. The gated cross-attention (or cross-attentional) transformer enables information exchange between modalities (e.g., visual and textual modes) of information embedded in the document. Within a visually-rich region of the document (e.g., a spatially-localized region in a document that includes a figure, chart, table, drawing, plot, or the like) the encoded visual information may be more relevant (for document understanding purposes) than the corresponding textual information. In contrast, within a textually-rich region of the document (e.g., a region that includes mostly text), the encoded textual information may be more relevant than the corresponding visual information. The embodiments account for such trends by “paying more attention” to the visual information (than the textual information) within visually-rich regions. Likewise, the embodiments “pay more attention” to the textual information (than the visual information) within textually-rich regions Thus, in the embodiments, a visually-rich document region is contrasted with textually-rich document regions, where the textually-rich region includes stronger textual information. In contrast to conventional pretraining methods, the enhanced pipeline differentiates and separately treats the textual and visual regions. That is, the embodiments do not treat the multimodes identically. Rather, the gated cross-attention mechanism employed in the pipeline may dynamically control the influence of textual and visual features. The approach taken in the pipeline enables cross-modal connections and allows for variable highlighting of the relevant information in visual and textual modality, as well as enabling cross-modal connections. During pretraining, a convolution neural network (CNN)-based visual backbone and multi-layer gated cross-attention encoder are jointly trained in both pretraining and a fine-tuning phase of the pretraining. 
     The pipeline may include five stages. A first stage of the pipeline may segment the document into a set of regions with associated bounding boxes. A second stage of the pipeline employs the CNN-based visual backbone to learn visual representations. The second stage may further extract region-of-interest (RoI) features with optical-character-recognition (OCR) bounding boxes. RoI features may be extracted via an image encoder model, referred to as f ImEnc . To filter-out some of the negative side effects associated with the quantization imposed by the image encoder, the image encoder may be paired with a RoI aligner, referred to as f RoiAlign . In a third stage of the pipeline, multimodal embeddings may be generated by combining the textual embeddings and position encodings. In the fourth stage of the pipeline, a transformer-based encoder (e.g., the model that is being pretrained) receives a set of masked multimodal embeddings as input. Multimodal self-attention across the embeddings is performed at the fourth stage. In a fifth stage of the pipeline, the model is then pretrained with at least one pretraining task. 
     In some embodiments, three separate pretraining tasks may be employed. The three pretraining tasks may include a Masked Sentence Modeling (MSM) pretraining task, a Visual Contrastive Learning (VCL), and Vision-Language Alignment (VLA). A separate objective function may be defined for each of the pretraining tasks. A combined pretraining objective function may be defined by a linear combination of each of the separate task-specific objective functions. Trade-offs between the pretraining tasks may be accounted for by adjusting the weights of the linear combination. The model&#39;s parameters (or weights) may be jointly trained during both pretraining and fine-tuning phases of the pipeline. In some embodiments, the weights of the textual encoder are predetermined and not adjusted by the pipeline. 
     Briefly, the embodiments provide an enhanced pretraining pipeline (or unified framework) for document understanding. Such enhanced pretraining enables learning a combination of contextual-textual information and visual information via cross-modal (and correlational) attention within a single framework. Such pretraining provides enhanced performance of the model. The embodiments also employ masked sentence modeling for language modeling, visual contrastive learning for vision modeling, and vision-language alignment for pretraining. The models pretrained by the various embodiments provide enhanced performance on various downstream document understanding tasks. 
     Furthermore, the enhanced pretraining pipeline significantly differs from conventional pretraining methods. Unlike some conventional approaches, during pretraining, the parameters of the image encoder with RoI align (e.g., f ImEnc +f RoIAlign ), which derive the visual features for semantic regions, are jointly trained. In further contrast, the visual features are derived from the semantic regions instead of splitting the image into fixed regions. Moreover, to learn the contextualized visual representations, the pipeline masks visual information in the latent space and learns contextualized representations by solving a contrastive learning task defined over a quantization of the latent visual embeddings. 
     Operating Environment for Document Understanding Pretraining 
       FIG.  1    illustrates an enhanced document understanding system  100  implementing various embodiments presented herein. Document understanding system  100  is enabled to pretrain a document encoder model for document understanding tasks. Document understanding system  100  may include at least a client computing device  102  and a server computing device  104 , in communication via a communication network  110 . The client computing device  102  can provide document pretraining data to the server computing device  104 , via the communication network  110 . The server computing device  104  implements a document encoder pretraining engine  120 . The document encoder pretraining engine  120  is enabled to pretrain a document encoder model based on the pretraining training data. The document encoder model may be a transformer-based model. After pretraining, the document encoder model may be provided to the client computing device  102 , so that the pretrained model may be further trained for specific document understanding tasks. 
     As discussed in conjunction with at least  FIG.  2   , the document encoder pretraining engine  120  implements an automated pretraining pipeline (e.g., pipeline  200  of  FIG.  2   ) that pretrains the document encoder model. Although a client/server architecture is shown in  FIG.  1   , the embodiments are not limited to such architectures. For example, client computing device  102  may implement the document encoder pretraining engine  120 , obviating the offloading of such pretraining tasks to server devices. 
     Document encoder pretraining engine  120  may include a document segmenter  122 , an optical character recognition (OCR) module  124 , a document feature extractor  130 , a feature embedder  140 , a quantization module  150 , a gated cross-attention network  160 , and a pretraining task network  170 . The functionalities, operations, features, and actions implemented by the various components of document encoder pretraining engine  120  are discussed at least in conjunction with pipeline  200  of  FIG.  2    and methods  400 - 440  of  FIGS.  4 A- 4 B . 
     However, briefly here, the document encoder pretraining engine  120  receives a set of pretraining (or training) data. The pretraining data includes a set of pretraining documents. Each pretraining document may be encoded in image data. The document segmenter  122  is generally responsible for segmenting each pretraining document. The OCR module  124  is generally responsible for identifying the textual-information encoded the image data. The document feature extractor  130  is generally responsible for extracting features from the segmented and OCR&#39;ed documents. The feature embedder  140  is generally responsible for generating multi-modal embeddings for the features of the documents. The quantization module  150  is generally responsible for discretizing the feature embeddings based on vector quantization methods. The gated-cross attention network  160  is generally responsible for applying a self-attention mechanism across the quantized and multi-modal feature embeddings. The pretraining task network  170  is generally responsible for performing one or more pretraining tasks to pretrain the document encoder model. 
     The document feature extractor  130  may include a sentence feature extractor  132  and a visual feature extractor  134 . The sentence feature extractor  132  is generally responsible for extracting sentence features for sentences encoded in the documents. The visual feature extractor  134  is generally responsible for extracting visual features encoded in the documents. The feature embedder  140  may include a sentence embedder  142  and a visual embedder  144 . The sentence embedder  142  is generally responsible for generating sentence embeddings for the sentence features. The visual embedder  144  is generally responsible for generating visual embeddings of the visual features. 
     Communication network  110  may be a general or specific communication network and may directly and/or indirectly communicatively coupled to client computing device  102  and server computing device  104 . Communication network  110  may be any communication network, including virtually any wired and/or wireless communication technologies, wired and/or wireless communication protocols, and the like. Communication network  110  may be virtually any communication network that communicatively couples a plurality of computing devices and storage devices in such a way as to computing devices to exchange information via communication network  110 . 
     Pipeline for Document Encoder Model Pretraining 
       FIG.  2    illustrates an enhanced pipeline  200  for pretraining a document encoder model, according to various embodiments presented herein. Pipeline  200  may be implemented by a document encoder pretraining engine, such as but not limited to document encoder pretraining engine  120  of  FIG.  1   . As such, pipeline  200  may receive document pretraining data and pretrain a document encoder model. That is, the pipeline  200  presents a unified framework for pretraining a document encoder model for document understanding. The document encoder model may be a transformer-based encoder model. 
     As a general overview, pipeline  200  employs a CNN-based visual backbone to learn visual representations of the features included in the pretraining documents. Pipeline  200  then extracts the region of interest (RoI) features with optical character recognition (OCR)-generated bounding boxes. Pipeline  200  then generates a multimodal embedding (e.g., for each bounding box) by combining a textual embedding and a position encoding for each bounding box. A transformer-based encoder (e.g., the model that is being pretrained by pipeline  200 ) takes a set of masked multimodal embeddings as input. The transformer-based encoder is pretrained with one or more pretraining tasks. In some embodiments, three pretraining tasks are employed. Once pretrained, the model may be fine-tuned for a specific document understanding task. A least portions of the network parameters for the document encoder model are jointly trained during both pretraining and fine-tuning phases. 
     Pipeline  200  may include five stages. The first stage  210  is generally responsible for preprocessing each pretraining document. The first stage  210  may be referred to as a preprocessing or document segmentation stage  210 . The second stage  230  is generally responsible for extracting features from the pretraining documents. The extracted features may include a set of textual features (e.g., a first feature modality) and a set of visual features (e.g., a second feature modality). According, the extracted features may be multimodal features. The second stage  230  may be referred to as a feature extraction stage  230 . The third stage  250  is generally responsible for generating embeddings (e.g., deeply learned vector representations) for the multimodal features extracted during the second stage  230 . Accordingly, the third stage  250  may be referred to as a feature embedding stage  250 . The fourth stage  270  is generally responsible for performing gated cross-attention between the modalities of the feature embeddings. Thus, the fourth stage  270  may be referred to as a gated cross-attention stage  270 . The fifth stage  290  is generally responsible for performing one or more pretraining tasks to pretrain the model based on the self-attention applied across the modalities of the feature embeddings. Accordingly, the fifth stage  290  may referred to as a pretraining task stage  290 . 
     More particularly, in the first stage  210  of pipeline  200  (and after the document segmentation stage  210 ), a document segmenter (e.g., document segmenter  122  of  FIG.  1   ) may segment each training document (via the document&#39;s image data) into a set of document elements (e.g., paragraphs, sentences, and/or regions of interest (ROI)). The document segmenter may determine a bounding box and location (of the bounding box) for each of the document&#39;s elements.  FIGS.  3 A- 3 B  show examples of document segmenting, in accordance to the various embodiments.  FIG.  3 A  shows an example pretraining document  300  segmented into its various elements. Document  300  has been segmented via various bounding boxes, including but not limited to bounding box  302 . Textual content is associated with at least a portion of the bounding boxes of document  300 . For example, textual content  304  is bounded by (and thus associated with) bounding box  302 .  FIG.  3 B  shows an example finetuning document  320  segmented into its various elements. Document  320  has been segmented via various bounding boxes, including but not limited to bounding box  322 . Textual content is associated with at least a portion of the bounding boxes of document  320 . For example, textual content  324  is bounded by (and thus associated with) bounding box  322 . Note the bounding boxes illustrated for each document element of document  300  and document  320 . An OCR module (e.g., OCR module  124  of  FIG.  1   ) may be employed to determine the tokens (e.g., natural words and characters) encoded in the image data. 
     In the second stage  230  of pipeline  200 , a document feature extractor (e.g., document feature extractor  130  of  FIG.  1   ) may receive the document&#39;s segmented image data, the OCR&#39;ed words, and locations of the document&#39;s elements. In view of the image regions and words that correspond to each document element as inputs, the document feature extractor may then extract the element&#39;s respective embeddings through a visual feature extractor (e.g., visual feature extractor  134 ) and a sentence feature extractor (e.g., sentence feature extractor  132  of  FIG.  1   ). The visual encoder may be referred to a f ImEnc  and may be paired with an alignment encoder (e.g., a ROI aligner), referred to as f RoIAlign . The sentence feature extractor may be a sentence encoder. Because the extracted features are encoded in vector embeddings, the extracted features may be referred to as feature embeddings. In the third and fourth stages  250 / 270  of pipeline  200 , these embeddings may be fed into a transformer-based encoder to learn the cross-modal contextualized embeddings that integrate both visual features and textual features. In the fifth stage  290  of pipeline  200 , one or more (e.g., three) pretraining tasks are iterated over to achieve pretraining of the model. 
     More specifically, in the feature extraction stage  230 , the pretraining engine may employ its OCR module to extract text (e.g., natural language words and/or tokens) from a document image (e.g., image data which may be referred to as I). The words may be grouped into sentences  ={s 1 , . . . , s N } whose corresponding bounding boxes are referred to as  ={p i , . . . ,p N }. For each sentence bounding box p i , the pretraining engine&#39;s visual feature extractor may then employ a CNN-based backbone (e.g., a ConvNet-based backbone referred to as f ImEnc ) and RoI Align (e.g., f RoIAlign ) to extract the pooled RoI features v i . To obtain a feature embedding, the sentence feature extractor may extract the sentence embedding s i  for each sentence s i  via a pretrained sentence encoder model referred to as f SentEnc . A quantization module (e.g., quantization module  150  of  FIG.  1   ) may discretize each region&#39;s RoI feature vector v i  into a finite set of visual representations v i   Q ∈V Q  via one or more product quantization methods. In the fourth stage  270 , a multi-layer Gated Cross-Attention encoder (e.g., as implemented by gated cross-attention network  160  of  FIG.  1   ) may take the position information, masked visual features {tilde over (V)} and masked textual features {tilde over (S)} as inputs, and then it generates the contextualized multimodal representations (H V   l  and H S   l , l∈[1, L]) and outputs the predicted features ({circumflex over (V)} and Ŝ), where L is the number of stacked transformer blocks. Various pretraining tasks may be performed in the fifth stage  290 . 
     The operations of the five stages of pipeline  200  may be symbolically indicated as: 
     
       
         
           
             
               
                 
                   I 
                   
                     → 
                     OCR 
                   
                   
                     ( 
                     
                       
                         
                           𝒫 
                         
                       
                       
                         
                           𝒮 
                         
                       
                     
                     ) 
                   
                   
                     
                       → 
                       
                         f 
                         SentEnc 
                       
                     
                     
                       
                         f 
                         ImEnc 
                       
                       + 
                       
                         f 
                         RoIAlign 
                       
                     
                   
                   
                     
                       ( 
                       
                         
                           
                             
                               V 
                               , 
                             
                           
                           
                             
                               V 
                               Q 
                             
                           
                         
                         
                           
                             S 
                           
                           
                               
                           
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       
                         
                           → 
                         
                       
                       
                         
                           
                             f 
                             Mask 
                           
                         
                       
                     
                     ⁢ 
                     
                       ( 
                       
                         
                           
                             
                               V 
                               ~ 
                             
                           
                         
                         
                           
                             
                               S 
                               ~ 
                             
                           
                         
                       
                       ) 
                     
                   
                   → 
                   
                     ( 
                     
                       
                         
                           
                             H 
                             V 
                             l 
                           
                         
                       
                       
                         
                           
                             H 
                             S 
                             l 
                           
                         
                       
                     
                     ) 
                   
                   → 
                   
                     ( 
                     
                       
                         
                           
                             V 
                             ^ 
                           
                         
                       
                       
                         
                           
                             S 
                             ^ 
                           
                         
                       
                     
                     ) 
                   
                   → 
                   
                     ℒ 
                     Pretraining 
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     where f Mask  denotes a masking function that randomly masks RoI features and sentence embeddings weighted with the respective probabilities p Mask   v  and p Mask   s .    Pretraining  is the objective function for one or more pretraining tasks. In at least one embodiment, the one or more pretraining tasks includes three pretraining tasks: Masked Sentence Modeling (MSM), Visual Contrastive Learning (VCL), and Vision-Language Alignment (VLA). In such embodiments,    Pretraining  may be a linear combination of the object function for each of the three pretraining tasks. The implementation details of the five stages (as symbolically encoded in Eq. 1) will now be discussed. 
     After the document segmenting stage  210 , and during the feature extraction stage  230 , a document image I∈   W×H  may consist of N regions. Each region&#39;s bounding box may be characterized by a 6D vector: 
     
       
         
           
             
               
                 p 
                 i 
               
               = 
               
                 { 
                 
                   
                     
                       x 
                       
                         L 
                         ⁢ 
                         T 
                       
                     
                     W 
                   
                   , 
                   
                     
                       y 
                       
                         L 
                         ⁢ 
                         T 
                       
                     
                     H 
                   
                   , 
                   
                     
                       x 
                       
                         R 
                         ⁢ 
                         B 
                       
                     
                     W 
                   
                   , 
                   
                     
                       y 
                       
                         R 
                         ⁢ 
                         B 
                       
                     
                     H 
                   
                   , 
                   
                     w 
                     W 
                   
                   , 
                   
                     h 
                     H 
                   
                 
                 } 
               
             
             , 
           
         
       
     
     where w and h indicate the width and height of the region, W and H may indicate the width and height of I, while (x LT , y LT ) and (X RB , y RB ) may indicate the coordinates of the bounding box&#39;s top-left and bottom-right corners respectively. The 6D vector may be mapped onto a high-dimensional representation (e.g., a high dimensional vector space) via a linear mapping function. 
     In the feature embedding stage  250 , the visual embedding may be generated as the sum of the mapped RoI feature and position embedding. Likewise, the textual embedding may be generated as the sum of sentence embedding and position embedding. Different types of segments may be utilized to distinguish different modalities. The input sequence to the transformer-based encoder (e.g., as implemented by the feature embedder  140 ) may start with a special start element ([CLS] and full visual features), then it is followed by multimodal elements, and it ends with a special ending element ([SEP]+full visual features). For the special elements ([CLS] and [SEP]), the corresponding full visual features may be the features that are extracted from the whole input image, by applying f ImEnc  to an RoI covering the whole input image. 
     In various embodiments, an image encoder and a multimodal model may be jointly learned (e.g., pretrained) in an end-to-end fashion, via pipeline  200 . A visual representation may be learned by predicting the visual features of the masked regions. It may be challenging to precisely predict such features, since these features are unconstrained and of continuous representation. To constrain the representation (e.g., the vector) space of the visual features and facilitate the end-to-end learning of image encoder, a quantization module (e.g., quantization module  150  of  FIG.  1   ) may employ one or more vector quantization methods to discretize the visual features V={v 1 , . . . , v N } into a finite set of representations V Q ={v 1   Q , . . . , v N   Q }. 
     Latent embedding spaces e∈   C×E  may be defined, where C is the number of codebooks, and E is the number of entries for each codebook. For each v i , the v i  may first be mapped it to logits  ∈   C×E . Then, the probability for the  -th codebook entry in i-th group as p c,e =exp(( +g e )/τ)/Σ k=1   E  exp(( +g k )/τ) may be calculated. τ is a non-negative temperature paramter, g 1:E  are independent and identically distributed (i.i.d) samples drawn from the Gumbel(0,1) distribution. During the forward pass, one entry vector from each codebook may be selected by {tilde over (e)} i ˜argmax e p c,e . The quantized representation v i   Q  may be generated by a concatenation of {{tilde over (e)} 1 , . . . , {tilde over (e)} G } which is then followed by a linear transformation. During the backward pass, the gradients may be determined via a Gumbel-Softmax estimator. 
     To model the interactions among multimodal inputs, a gated cross-attention network (e.g., gated cross-attention network  160  of  FIG.  1   ) may implement a multimodal transformer model to model the cross-modality relationships. The multimodal transformer model may implement various gated cross-attention mechanisms to model the cross-modality relationships. Output features of the l-th layer for one modality m may be referred to as H m   l+1 . n may be another modality (m, n∈{V, S}). Features generated at (l+1)-th layer may be referred to as: 
         H   m   l+1   =f   LN ( f   LN ( H   m   l   +f   Cross−Att   l ( H   m   l   |H   n   l ))+ f   FF   l ( f   LN ( H   m   l   +f   Cross−Att   l ( H   m   l   |H   n   l ))))  (2)
 
     where f LN  denotes layer normalization. The feed-forward sub-layer f FF  in Eq. 2 may be further composed of two fully-connected sub-layers, both wrapped in residual adds and f LN . 
     The core part of Eq. 2 is the cross-attention f Cross−Att (·) term. Given the intermediate representations H m   l  and H n   l , the cross-attention output for modality m may be computed as: 
         f   Cross−Att ( H   m   l   |H   n   l )=[Cross− Att   1 ( H   m   l   |H   n   l ); . . . ;Cross− Att   h ( H   m   l   |H   n   l )] U   (3)
 
       Cross− Att   i ( H   m   l   |H   n   l )=softmax( f   q   i ( H   m   l ) f   k   i ( H   n   l ) T /√{square root over ( d )}) f   v   i ( H   n   l )  (4)
 
     where f q   i (H m   l ), f k   i (H n   l ), and f v   i (H n   l ) are the query, key, and value calculated by linear mapping layers for the i-th head. d is the model dimension, h is the number of heads, and U is the weight matrix that combines the outputs of the heads. 
     Considering the substantial diversity of document images and the different information needs of differing document types, the gating mechanism is employed to dynamically weight the outputs of the visual and textual branches. Specifically, the concatenated visual and textual features may be inputted in the non-linear network f Gate (H m   l+1 ; H n   l+1 ) which generates the modality-specific attention weights α m   l  and α n   l , and returns the weights separately to their respective modality-specific branches to perform element-wise products. The features for modality m may be multiplied with its modality-specific attention weight, and the updated feature may be computed as: H m   l+1 =H m   l+1 (1+α m   l ), same that for modality n. 
     For the pretraining stage  290 , three separate pretraining tasks may be employed. The three pretraining tasks may include a Masked Sentence Modeling (MSM) pretraining task, a Visual Contrastive Learning (VCL), and Vision-Language Alignment (VLA). The full pretraining objective may be defined as:    Pretraining =   MSM +   VCL +   VLA . Implementation details of each of the three tasks are described below. 
     This Masked Sentence Modeling (MSM) pretraining task may be similar to the Masked Token Modeling task utilized in BERT training. A difference between the embodiments herein and the task, as utilized in BERT training is that the embodiments may mask entire sentences, rather than individual tokens. During pretraining, each sentence and RoI of the input pretraining document may be randomly and independently masked. For the masked sentence, its token may be replaced with a special sentence, e.g., [MASK]. The model may be trained to predict the masked sentence feature, based on the unmasked words and the visual features. The goal is to predict the masked sentence embeddings based on the contextual information from the surrounding sentences and image regions, by minimizing the smooth L1 loss as follows: 
     
       
         
           
             
               
                 
                   
                     
                       ℒ 
                       
                         M 
                         ⁢ 
                         S 
                         ⁢ 
                         M 
                       
                     
                     ( 
                     Θ 
                     ) 
                   
                   = 
                   
                     
                       ∑ 
                       i 
                     
                     
                       
                         smooth 
                         
                           L 
                           1 
                         
                       
                       ( 
                       
                         
                           s 
                           i 
                         
                         - 
                         
                           
                             f 
                             Backbone 
                           
                           ( 
                           
                             
                               
                                 s 
                                 i 
                               
                               | 
                               
                                 s 
                                 
                                   ∖ 
                                   i 
                                 
                               
                             
                             , 
                             
                               V 
                               ~ 
                             
                           
                           ) 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     where Θ indicates the trainable parameters and f Backbone (·) outputs the unmasked textual feature, s i  is the surrounding features for the i-th input, {tilde over (V)} are the image features with random masking. 
     With regards to the Visual Contrastive Learning (VCL) pretraining task, visual feature representations by are learned by solving a visual contrastive learning task which estimates the true quantized latent RoI representation. Given a prediction {circumflex over (v)} i ∈{umlaut over (V)} for the masked RoI {tilde over (v)} i ∈{tilde over (V)}, the model estimates the positive quantized representation v i   Q  in a set of quantized candidate representations V Q . Good representations are learned by maximizing the agreement between output representation and quantized representation of the same RoIs as follows: 
     
       
         
           
             
               
                 
                   
                     
                       ℒ 
                       VCL 
                     
                     ( 
                     Θ 
                     ) 
                   
                   = 
                   
                     
                       - 
                       
                         
                           ∑ 
                           
                             
                               
                                 v 
                                 ~ 
                               
                               i 
                             
                             ∈ 
                             
                               V 
                               ~ 
                             
                           
                         
                         
                           ( 
                           
                             log 
                             ⁢ 
                             
                               
                                 exp 
                                 ⁡ 
                                 ( 
                                 
                                   s 
                                   ⁢ 
                                   i 
                                   ⁢ 
                                   
                                     
                                       m 
                                       ⁡ 
                                       ( 
                                       
                                         
                                           
                                             ν 
                                             ˆ 
                                           
                                           
                                             i 
                                             ⁢ 
                                             ′ 
                                           
                                         
                                         ⁢ 
                                         ′ 
                                         ⁢ 
                                         
                                           ν 
                                           i 
                                           Q 
                                         
                                       
                                       ) 
                                     
                                     / 
                                     κ 
                                   
                                 
                                 ) 
                               
                               
                                 
                                   Σ 
                                   
                                     v 
                                     j 
                                     Q 
                                   
                                 
                                 ⁢ 
                                 
                                   exp 
                                   ⁡ 
                                   ( 
                                   
                                     s 
                                     ⁢ 
                                     i 
                                     ⁢ 
                                     
                                       
                                         m 
                                         ⁡ 
                                         ( 
                                         
                                           
                                             
                                               ν 
                                               ˆ 
                                             
                                             
                                               i 
                                               ⁢ 
                                               ′ 
                                             
                                           
                                           ⁢ 
                                           ′ 
                                           ⁢ 
                                           
                                             ν 
                                             j 
                                             Q 
                                           
                                         
                                         ) 
                                       
                                       / 
                                       κ 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           ) 
                         
                       
                     
                     + 
                     
                       λ 
                       ⁢ 
                       
                         1 
                         CE 
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             c 
                             = 
                             1 
                           
                           C 
                         
                         
                           
                             ∑ 
                             
                               e 
                               = 
                               1 
                             
                             E 
                           
                           
                             
                               p 
                               
                                 c 
                                 , 
                                 e 
                               
                             
                             ⁢ 
                             log 
                             ⁢ 
                             
                               p 
                               
                                 c 
                                 , 
                                 e 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
     where sim(·,·) computes the cosine similarity between two vectors, λ is a hyperparameter, and κ is a temperature scalar. The second term encourages the model to use the codebook entries more equally. 
     With regards to the Vision-Language Alignment (VLA) pretraining task, to enforce the alignment among different modalities, the alignment between words and image regions is explicitly encouraged via similarity-preserving knowledge distillation, implemented in the objective function. Note that, unlike the conventional text-image alignment methods, which splits the image into four regions and predicts whether the given word is covered or not on the image side, in the various embodiments, the image and text belonging to the same region are aligned. The goal is to minimize (or at least decrease) the differences between the pairwise similarities of sentence embeddings and the pairwise similarities of image region features. The objective function may be indicated as: 
     
       
         
           
             
               
                 
                   
                     
                       ℒ 
                       VLA 
                     
                     ( 
                     Θ 
                     ) 
                   
                   = 
                   
                     
                       1 
                       
                         N 
                         × 
                         N 
                       
                     
                     ⁢ 
                     
                       
                          
                         
                           
                             
                               f 
                               Norm 
                             
                             ( 
                             
                               S 
                               · 
                               
                                 S 
                                 T 
                               
                             
                             ) 
                           
                           - 
                           
                             
                               f 
                               Norm 
                             
                             ( 
                             
                               
                                 H 
                                 V 
                                 L 
                               
                               · 
                               
                                 H 
                                 V 
                                 
                                   L 
                                   ⁢ 
                                   T 
                                 
                               
                             
                             ) 
                           
                         
                          
                       
                       F 
                       2 
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     where S is the unmasked input sentence embeddings, H V   L  is the mapped visual representations of the final layer, ∥·∥ F  is the Frobenius norm, and f Norm  performs L2 normalization. 
     Generalized Processes for Pretraining a Document Encoder Model 
     Processes  400 - 420  of  FIGS.  4 A- 4 B , or portions thereof, may be performed and/or executed by any computing device, such as but not limited to, client computing device  102  of FIG.  1 , server computing device  104  of  FIG.  1   , and/or computing device  500  of  FIG.  5   . Additionally, a dcoument encoder pretraining engine, such as but not limited to document encoder pretraining engine  120  of  FIG.  1   , may perform and/or execute at least portions of processes  400 - 460 . 
       FIG.  4 A  illustrates one embodiment of a method  400  for pretraining a document encoder model, which is consistent with the various embodiments presented herein. Process  400  may be performed by a document encoder pretraining engine, such as but not limited to document encoder pretraining engine  120  of  FIG.  1   . As such, pipeline  200  of  FIG.  2    may implement any combination of the various steps, actions, operations, and/or functionalities associated with any of the blocks of method  400 . Likewise, any of the blocks of method  400  may implement any combinations of the various steps, actions, operations, and/or functionalities associated with any of the five stages of pipeline  200 . 
     Process  400  begins at block  402 , where pretraining data is received. The pretraining data may include image data that encodes a set of pretraining documents. Hence, at block  402 , image data that encodes a document is received. At block  404 , a set of sentences and corresponding bounding boxes are extracted from the image data. At block  406 , for each sentence of the set of sentences, a set of predicted features are generated. The set of predicted features may be based on a gated-encoder model. For the purposes of this discussion, an encoder model may be any model that receives, as input, a first data object, and that generates a second data object that encodes one or more properties of the first data object. In some embodiments, the second data object may be a 1-tensor or vector that encodes, via the values of its components, one or more features of the first data object. The first data object may be a visual-content (e.g., image or video data), textual-content (e.g., natural language sentences), audio-content (e.g., music or podcasts), or any other information that may be encoded via digital data. In some embodiments, an encoder model may be a model that receives, as input, a sequence of data objects (e.g., tokens and/or natural language words) and generates, as output, a 1-tensor (or vector) representation of the input sequence of data objects. Such encoder models may be implemented by one or more layers within a neural network (NN). Thus, an encoder model may be referred to as a NN model. Some such encoder models may internally employ one or more gated mechanisms. In general, a gate (or gating mechanism) within a NN model may be a structure that regulates the flow of information between the layers. A gate may lengthen (or shorten) “memory” for recurring mechanisms, attention mechanisms, and the like. Thus, a recurrent neural network (e.g., RNN) or a long short-term memory (LSTM) may employ a gating mechanism to implement its “memory” functionality. In many examples, a gate may employ a non-linear activiation function (e.g. a sigmoid function, a tanh function, and the like) to enable the gating mechanism. Thus, some NN models may be referred to as gated-encoder models. The gated-encoder model may perform cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features may be based on a masking function and the sentence. The set of masked-visual features may be based on the masking function and the corresponding bounding box for the sentence. At block  408 , a document-encoder model may be pretrained. The pretraining may be based on the set of predicted features for each sentence and one or more pretraining tasks. 
       FIG.  4 B  illustrates one embodiment of another method  420  for pretraining a document encoder model, which is consistent with the various embodiments presented herein. Process  420  may be performed by a document encoder pretraining engine, such as but not limited to document encoder pretraining engine  120  of  FIG.  1   . As such, pipeline  200  of  FIG.  2    may implement any combination of the various steps, actions, operations, and/or functionalities associated with any of the blocks of method  420 . Likewise, any of the blocks of method  420  may implement any combinations of the various steps, actions, operations, and/or functionalities associated with any of the five stages of pipeline  200 . 
     Process  420  begins at block  422 , where pretraining data is received. The pretraining data may include image data that encodes a set of pretraining documents. Hence, at block  422 , image data that encodes a document is received. At block  424 , a set of sentences and corresponding bounding boxes are extracted from the image data. 
     At block  426 , visual embeddings are generated based on convolution model. That is, for each sentence of the set of sentences, a visual embedding is generated based on the convolution model and a portion of the image data associated with the corresponding bounding box. At block  428 , textual embeddings are generated based on a sentence encoder model and a corresponding visual embedding. 
     At block  430 , masked-textual features and masked visual features are generated based on the textual and visual embeddings. That is, for each sentence of the set of sentences, a set of masked-textual features and a set of masked-visual features may be generated. The masked features may be generated based on a masking function, the textual embedding for the sentence, and the corresponding visual embedding. 
     At block  432 , predicted features for the document are generated based on a gated-encoder model, the masked textual features, and the masked visual features. That is, for each sentence of the set of sentences, a set of predicted features may be generated for the document. Generating the set of predicted features may be based on a gated-encoder model that performs cross-attention between the set of masked-textual features and the set of masked-visual features for the sentence. 
     At block  434 , a document-encoder model may be pretrained. The pretraining may be based on the set of predicted features for each sentence and one or more pretraining tasks. 
     Illustrative Computing Device 
     Having described embodiments of the present invention, an example operating environment in which embodiments of the present invention may be implemented is described below in order to provide a general context for various aspects of the present invention. Referring to  FIG.  5   , an illustrative operating environment for implementing embodiments of the present invention is shown and designated generally as computing device  500 . Computing device  500  is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing device  500  be interpreted as having any dependency or requirement relating to any one or combination of components illustrated. 
     Embodiments of the invention may be described in the general context of computer code or machine-readable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a smartphone or other handheld device. Generally, program modules, or engines, including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. Embodiments of the invention may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. Embodiments of the invention may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network. 
     With reference to  FIG.  5   , computing device  500  includes a bus  510  that directly or indirectly couples the following devices: memory  512 , one or more processors  514 , one or more presentation components  516 , input/output ports  518 , input/output components  520 , and an illustrative power supply  522 . Bus  510  represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks of  FIG.  5    are shown with clearly delineated lines for the sake of clarity, in reality, such delineations are not so clear and these lines may overlap. For example, one may consider a presentation component such as a display device to be an I/O component, as well. Also, processors generally have memory in the form of cache. We recognize that such is the nature of the art, and reiterate that the diagram of  FIG.  5    is merely illustrative of an example computing device that can be used in connection with one or more embodiments of the present disclosure. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope of  FIG.  5    and reference to “computing device.” 
     Computing device  500  typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device  500  and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. 
     Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device  500 . Computer storage media excludes signals per se. 
     Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media. 
     Memory  512  includes computer storage media in the form of volatile and/or nonvolatile memory. Memory  512  may be non-transitory memory. As depicted, memory  512  includes instructions  524 . Instructions  524 , when executed by processor(s)  514  are configured to cause the computing device to perform any of the operations described herein, in reference to the above discussed figures, or to implement any program modules described herein. The memory may be removable, non-removable, or a combination thereof. Illustrative hardware devices include solid-state memory, hard drives, optical-disc drives, etc. Computing device  500  includes one or more processors that read data from various entities such as memory  512  or I/O components  520 . Presentation component(s)  516  present data indications to a user or other device. Illustrative presentation components include a display device, speaker, printing component, vibrating component, etc. 
     I/O ports  518  allow computing device  500  to be logically coupled to other devices including I/O components  520 , some of which may be built in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc. 
     Embodiments presented herein have been described in relation to particular embodiments which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those of ordinary skill in the art to which the present disclosure pertains without departing from its scope. 
     From the foregoing, it will be seen that this disclosure in one well adapted to attain all the ends and objects hereinabove set forth together with other advantages which are obvious and which are inherent to the structure. 
     It will be understood that certain features and sub-combinations are of utility and may be employed without reference to other features or sub-combinations. This is contemplated by and is within the scope of the claims. 
     In the preceding detailed description, reference is made to the accompanying drawings which form a part hereof wherein like numerals designate like parts throughout, and in which is shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the preceding detailed description is not to be taken in a limiting sense, and the scope of embodiments is defined by the appended claims and their equivalents. 
     Various aspects of the illustrative embodiments have been described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. However, it will be apparent to those skilled in the art that alternate embodiments may be practiced with only some of the described aspects. For purposes of explanation, specific numbers, materials, and configurations are set forth in order to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to one skilled in the art that alternate embodiments may be practiced without the specific details. In other instances, well-known features have been omitted or simplified in order not to obscure the illustrative embodiments. 
     Various operations have been described as multiple discrete operations, in turn, in a manner that is most helpful in understanding the illustrative embodiments; however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation. Further, descriptions of operations as separate operations should not be construed as requiring that the operations be necessarily performed independently and/or by separate entities. Descriptions of entities and/or modules as separate modules should likewise not be construed as requiring that the modules be separate and/or perform separate operations. In various embodiments, illustrated and/or described operations, entities, data, and/or modules may be merged, broken into further sub-parts, and/or omitted. 
     The phrase “in one embodiment” or “in an embodiment” is used repeatedly. The phrase generally does not refer to the same embodiment; however, it may. The terms “comprising,” “having,” and “including” are synonymous, unless the context dictates otherwise. The phrase “A/B” means “A or B.” The phrase “A and/or B” means “(A), (B), or (A and B).” The phrase “at least one of A, B and C” means “(A), (B), (C), (A and B), (A and C), (B and C) or (A, B and C).”