Patent Description:
Governments at all levels generate documents setting out requirements and/or conditions that should be followed for compliance with the applicable rules and regulations. For example, Governments implement regulations, permits, plans, court ordered decrees, and bylaws to regulate commercial, industrial, and other activities considered to be in the public's interest. Standards bodies, companies, and other organizations may also generate documents setting out conditions for product and process compliance. These documents may be broadly referred to as "regulatory content".

Modern enterprises thus operate under an increasing burden of regulation, which has proliferated exponentially in an attempt by regulatory agencies and other governmental bodies to mitigate potential and actual dangers to the public. Documents setting out regulatory content may vary in size, from one page to several hundred pages. As a result, compliance with regulatory content has become increasingly difficult for enterprises. There remains a need for methods and systems that reduce the burden for enterprises in establishing which regulations and conditions in a body of regulatory content are applicable to their operations.

Both <CIT> and the publication by <NPL>, represent relevant prior art.

In accordance with one disclosed aspect there is provided a computer-implemented method for identifying citations within regulatory content. The method involves receiving image data representing a format and layout of the regulatory content, receiving a language embedding including a plurality of tokens representing words or characters in the regulatory content, and generating a token mapping associating each of the tokens with a portion of the image data. The method also involves receiving the plurality of tokens and token mapping at an input of a citation classifier, the citation classifier having been trained to generate a classification output for each token based on the language embedding and the token mapping, the classification output identifying a plurality of citation tokens within the plurality of tokens. The method further involves processing the plurality of citation tokens to determine a hierarchical relationship between citation tokens, the hierarchical relationship being established based at least in part on the token mapping for the citation tokens.

The method may involve receiving text data representing characters in the regulatory content and generating the language embedding based on the received text data.

Generating the language embedding may involve generating the language embedding using a pretrained language model to process the text data, the pretrained language model being operably configured to generate the plurality of tokens.

Generating the token mapping may involve generating a tensor for each image page of regulatory content, the tensor having first and second dimensions corresponding to the image pixels in the image page and a third dimension in which language embedding values are associated with portions of the image page, the tensor providing the input to the citation classifier.

Generating the language embedding may involve processing the image data to identify regions of interest within the regulatory content, each region of interest including a plurality of characters, and generating the language embedding may involve generating the language embedding only for text data associated with the regions of interest.

Processing the image data to identify the plurality of regions of interest may involve receiving the image data at an input of a region of interest classifier, the region of interest classifier having been trained to identify regions within the image data that are associated with regulatory content that should not be processed to identify citation tokens.

The region of interest classifier may be pre-trained using a plurality of images of portions of the regulatory content that should not be processed to identify citation tokens.

Receiving image data may involve receiving text data including format data representing a layout of the regulatory content and generating an image data representation of the formatted text data.

The image data may involve a plurality of page images and generating the token mapping may involve sizing each of the page images to match a common page size, for each token, establishing a bounding box within the image that identifies a portion of the image corresponding to the token, and determining at least a bounding box location for each token, the bounding box location being indicative of an indentation or position of the token within the applicable page image.

The method may involve processing the portion of the image within the bounding box to determine at least one of a font size associated with the token or a formatting associated with the token, and the citation classifier may be trained to generate the classification output based on at least one of the determined font size or formatting associated with the token.

The citation classifier may include a convolutional neural network.

Processing the plurality of citation tokens may involve receiving pairwise combinations of citation tokens at an input of a sibling classifier, the sibling classifier having been trained to output a probability indicative of whether each pairwise combination of citation tokens have a common hierarchical level.

The method may involve generating a similarity matrix including a plurality of rows corresponding to the plurality of citation tokens and a plurality of columns corresponding to the citation tokens, the similarity matrix being populated with the probabilities determined by the sibling classifier for each pairwise combination of citation tokens, and the method may further involve processing the similarity matrix to generate a hierarchical level for each citation identifier token.

The plurality of identified citation tokens each include either a citation number or citation title, and a remaining plurality of tokens not identified as citation tokens include body text, and the method may further involve associating the body text between two consecutive citation tokens with the citation token.

In accordance with another disclosed aspect there is provided a system for identifying citations within regulatory content. The system includes a token mapper operably configured to receive a language embedding including a plurality of tokens representing words or characters in the regulatory content. The token mapper is operably configured to receive image data representing a format and layout of the regulatory content and generate a token mapping associating each of the tokens with a portion of the image data. The system also includes a citation classifier operably configured to receive the plurality of tokens and token mapping at an input of a citation classifier, the citation classifier having been trained to generate a classification output for each token based on the language embedding and the token mapping, the classification output identifying a plurality of citation tokens within the plurality of tokens. The system further includes a hierarchical relationship classifier operably configured to process the plurality of citation tokens to determine a hierarchical relationship between citation tokens, the hierarchical relationship being established based at least in part on the token mapping for the citation tokens.

The system may include one or more processor circuits having a memory for storing codes, the codes being operable to direct the processor circuit to implement each of the token mapper, the citation classifier, and the hierarchical relationship classifier.

Other aspects and features will become apparent to those ordinarily skilled in the art upon review of the following description of specific disclosed embodiments in conjunction with the accompanying figures.

In drawings which illustrate disclosed embodiments,.

Referring to <FIG>, a system for identifying citations within regulatory content according to a first disclosed embodiment is shown generally at <NUM> as a block diagram. In the embodiment shown, the regulatory content is received as image data <NUM>, where the text is represented by pixels rather than digital text. The regulatory content image data <NUM> represents text within the regulatory content as an image, rather than just serial encoded text. The regulatory content image data <NUM> thus includes format and layout information associated with the regulatory content. In other embodiments the regulatory content may be received in any of a variety of text formats, where words and characters in the text are encoded into a digital data format representing the text of the regulatory content. Whether in image or digital form, the regulatory content may include formatting and/or a hierarchical document structure that has significance in the understanding of the content.

In this embodiment the system <NUM> optional includes a region of interest classifier <NUM> that processes the regulatory content image data <NUM> to identify regions of interest. The region of interest classifier <NUM> outputs a region of interest classification <NUM> The regulatory content may include text regions such as title pages, page headers and footers, etc. that may be excluded from further processing to identify citations.

The system <NUM> also includes a token mapper <NUM>, which receives the region of interest classification output <NUM> and a language embedding <NUM> associated with the regulatory content. The language embedding <NUM> may be generated using a language model and may output a plurality of tokens representing words or characters in the regulatory content. A token is a sequence of characters grouped together as a useful semantic unit for processing. For example, the word "sleeping" may be represented by a first token "sleep" and a second token "ing". Each token is represented by as set of data generated by the language model that defines the token and its context in the document. Tokenization may be implemented at a word level, subword level, and/or character level. In the remainder of this description, the term token will be used to refer to sequences of one or more characters that have been rendered from the original regulatory content. In one embodiment the language model first performs a tokenization of the regulatory content to separate the content into tokens prior to generating the language embedding <NUM> for the identified tokens.

The token mapper <NUM> generates a token mapping output <NUM> that associates each of the tokens with a portion of the image data. In one embodiment the image data <NUM> may be in the form of a pixel-based image representation. In other embodiments the regulatory content may already be formatted as digital text and may include format data representing a layout of the regulatory content. In this event, the digital text may be converted into a pixel-based image representation to provide the image data <NUM>. The token mapper <NUM> generates a token mapping output <NUM>, which associates each of the tokens in the language embedding <NUM> with a portion of the image data <NUM>.

The system <NUM> also includes a citation classifier <NUM>, which receives the mapping output <NUM> and generates a classification output <NUM> for each of the plurality of tokens in the regulatory content. The classification output <NUM> identifies certain tokens in the plurality of tokens as being "citation tokens". In context of regulatory content, a citation is a reference to one or more requirements or conditions within the text of the regulatory content. In some cases, regulatory content may include explicit alphanumeric citations. In other cases, a citation may not be explicit, and may be inferred based on variations in font size, font type, or its position on a page. The citation classifier <NUM> may be implemented using a neural network, which has been trained to generate the classification output <NUM> identifying the citation tokens.

The system <NUM> further includes a hierarchical relationship classifier <NUM>, which receives the classification output <NUM>. The hierarchical relationship classifier <NUM> is operably configured to process the identified plurality of citation tokens to determine a hierarchical relationship between these tokens. A hierarchical relationship may be established based at least in part on image data associated with each identified citation token. The hierarchical relationship classifier <NUM> generates a citation hierarchy output <NUM> representing a hierarchical structure of the identified citation tokens.

The system <NUM> for identifying citations shown in <FIG> may be implemented on a processor circuit, operably configured to provide inference functions for processing regulatory content. Referring to <FIG>, an inference processor circuit is shown generally at <NUM>. The inference processor circuit <NUM> includes a microprocessor <NUM>, a program memory <NUM>, a data storage memory <NUM>, and an input output port (I/O) <NUM>, all of which are in communication with the microprocessor <NUM>. Program codes for directing the microprocessor <NUM> to carry out various functions are stored in the program memory <NUM>, which may be implemented as a random access memory (RAM), flash memory, a hard disk drive (HDD), or a combination thereof. The program memory <NUM> includes storage for program codes that are executable by the microprocessor <NUM> to provide functionality for implementing the various elements of the system <NUM>.

In this embodiment, the program memory <NUM> includes storage for program codes <NUM> for directing the microprocessor <NUM> to perform operating system functions. The operating system may be any of a number of available operating systems including, but not limited to, Linux, macOS, Windows, Android, and JavaScript. The program memory <NUM> also includes storage for program codes <NUM> for implementing a region of interest classifier, codes <NUM> for implementing a language model, codes <NUM> for implementing the token mapper <NUM>, codes <NUM> for implementing the citation classifier <NUM>, codes <NUM> for implementing the hierarchical relationship classifier <NUM>, and codes <NUM> for implementing a sibling classifier.

The I/O <NUM> provides an interface for receiving input via a keyboard <NUM>, pointing device <NUM>. The I/O <NUM> also includes an interface for generating output on a display <NUM>. The I/O <NUM> also includes an interface <NUM> for connecting the processor circuit <NUM> to a wide area network <NUM>, such as the internet.

The data storage memory <NUM> may be implemented in RAM memory, flash memory, a hard drive, a solid state drive, or a combination thereof. Alternatively, or additionally the data storage memory <NUM> may be implemented at least in part as storage accessible via the interface <NUM> and wide area network <NUM>. In the embodiment shown, the data storage memory <NUM> provides storage <NUM> for regulatory content image data <NUM>, storage <NUM> for region of interest classification data, storage <NUM> for configuring a region of interest classifier, storage <NUM> for citation classifier configuration data, storage <NUM> for configuring a sibling classifier, and storage <NUM> for processed citation data.

The inference processor circuit <NUM>, when configured with the applicable neural network training and configuration data in storage locations <NUM> - <NUM> of the data storage memory <NUM>, is operable to implement the system <NUM> for processing regulatory content shown in <FIG>.

Processes for generating the necessary neural network training and configuration data for implementing the system <NUM> are described below. While the training may be performed on the inference processor circuit <NUM>, in practice neural network configuration and training may be performed on specifically configured training system such as a machine learning computing platform or cloud-based computing system, which typically include one or more graphics processing units. An example of a training system is shown in <FIG> at <NUM>. The training system <NUM> includes a user interface <NUM> that may be accessed via an operator's terminal <NUM>. The operator's terminal <NUM> may be a processor circuit such as shown at <NUM> in <FIG> that has a connection to the wide area network <NUM>. The operator is able to access computational resources <NUM> and data storage resources <NUM> made available in the training system <NUM> via the user interface <NUM>. In some embodiments, providers of cloud based neural network training systems <NUM> may make machine learning services <NUM> that provide a library of functions that may be implemented on the computational resources <NUM> for performing machine learning functions such as training. For example, a neural network programming environment TensorFlow™ is made available by Google Inc. TensorFlow provides a library of functions and neural network configurations that can be used to configure the above described neural network. The training system <NUM> also implements monitoring and management functions that monitor and manage performance of the computational resources <NUM> and the data storage <NUM>. In other embodiments, the functions provided by the training system <NUM> may be implemented on a stand-alone computing platform configured to provide the computing resources necessary for performing the training.

Generally, the training of the neural networks for implementing the citation classifier <NUM>, hierarchical relationship classifier <NUM> are performed under supervision of an operator using the training system <NUM>. The operator will typically configure the neural networks and provide labeled training data to generate weights and biases for the neural network. During the training exercise, the operator may make changes to the configuration of the neural network until a satisfactory accuracy and performance is achieved. The resulting neural network configuration and determined weights and biases may then be saved to the applicable locations <NUM> - <NUM> of the data storage memory <NUM> of the inference processor circuit <NUM>. As such, the citation classifier <NUM>, hierarchical relationship classifier <NUM> and other neural network functions described herein may be initially implemented and refined on the system <NUM>, before being configured for regular use on the inference processor circuit <NUM>.

Regulatory content documents, in addition to significant regulatory text, may include redundant or superfluous text such as cover pages, a table of contents, a table of figures, page headers, page footers, page numbering etc. The region of interest classifier <NUM> performs preprocessing operations to identify significant regulatory text on each page of the regulatory content. One advantage of removing redundant text is to reduce subsequent processing time within the token mapper <NUM> and citation classifier <NUM>. The removal of redundant text also avoids identification of citations within the redundant text, which may cause redundant identifications of citations. As an example, in some cases regulatory content documentation may have headers or footers on each page that list the citations appearing on the page. Identification and processing of these citations may result in an incorrect hierarchical relationship output by the hierarchical relationship classifier <NUM>. A similar problem may exist with listings of citations in a table of contents of the document.

Avoiding the processing of these redundant text regions provides for a more robust extraction of significant citations from the regulatory content.

In one embodiment the region of interest classifier <NUM> may be implemented using a deep convolutional neural network, which is trained to generate the region of interest configuration data for storage in the location <NUM> of data storage memory <NUM> on the inference processor circuit <NUM>. The training may be performed on the training system <NUM>, on which a region of interest classifier neural network has been configured. In one embodiment a Faster R-CNN network architecture is used to implement the region of interest classifier <NUM>. Image data <NUM> representing the pages of regulatory content may be input to the region of interest classifier <NUM> as input tensors having image height and image width dimensions corresponding to the pixels of the image data and for color documents, a depth dimension. The depth dimension may be associated with color information such RGB color data, for example. In one embodiment the image data <NUM> may be converted from color images into a grey-scale images, to expedite training of the neural network on the system <NUM> and subsequent inference by the processor circuit <NUM>.

Referring to <FIG>, an implementation of a faster R-CNN network is shown as a block diagram at <NUM>. The input tensors <NUM> may be fed into a pre-trained convolutional neural network (CNN) <NUM> such as ResNet-<NUM> network. ResNet-<NUM> is available in a configuration that has been pre-trained on more than a million images from the ImageNet database. The tensor input <NUM> is processed in the ResNet-<NUM> up to an intermediate layer (for example, the second last layer) to provide a convolutional feature map <NUM>. The convolutional feature map <NUM> is used as a feature extractor for a subsequent step, in which the feature map is processed through a region proposal network (RPN) <NUM>. The RPN <NUM> generates bounding box coordinates for regions of interest within the image <NUM>. The bounding box coordinates are then fed to a final classifier neural network stage , in which several fully connected layers generate classification labels <NUM> for the bounding boxes, such as "text", "table", "list", "section", "header/footer", etc. and perform bounding-box regression.

Training of the faster R-CNN neural network is based on a set of training images of regulatory content pages, which are selected and manually labeled with classifications such as text, table, list, section, etc. An example of a labeled regulatory content image page is shown in <FIG> at <NUM>. The page image <NUM> includes various regions, which are shown identified by broken lines, each identified region including an assigned label. The page image <NUM> includes regulatory text identified as regions of interest, in this embodiment as "text" (regions <NUM>), "table" (regions <NUM>), "list" (regions <NUM>), and "section" (region <NUM>).

The page image <NUM> also includes header and footer text regions <NUM>, which are not regions of interest, but are output as classifications by the neural network <NUM> to facilitate exclusion from further processing by the system <NUM>. Other pages may be selected that include other regions that may be classified, such as a table of contents, or figure description, or title page, for example. These regions may also be excluded from further processing by the system <NUM>. Generally, for successful region of interest identification, it is important to select and label a sufficient number training images to account for different document formats, languages, differing scale, font, etc. A first portion of the labeled training images will be used for training the neural network <NUM>on the training system <NUM>, while a second portion of the labeled training images will be used for validation, as described below.

Once a set of training images are available and uploaded to the data storage <NUM>, the neural network training exercise is configured by selecting an appropriate number of epochs and other network hyper parameters. Training generally involves seeding the neurons of the neural network with values for the weights wij and biases bj. The first portion of the labeled training images are then passed through the network to obtain region of interest classification predictions, which are compared to the labels in the training images. A loss function is then evaluated prior to performing backpropagation based on the loss function to all neurons in the neural network. The weights wij and biases bj are then updated based on a gradient descent or other optimization function. The described training steps are then repeated until the neural network provides prediction values that are considered to be sufficiently accurate. In order to validate the neural network, the second portion of labeled training images that were not used for training of the network are fed through the neural network. The region of interest classification predictions output by the neural network are then compared against the labels to verify that the trained network is effective in generating predictions for unseen examples. If the neural network is validated, the weights wij and biases bj and data defining the network configuration is saved into the store <NUM> of the data storage memory <NUM>, for use in by the inference processor circuit <NUM> in processing actual regulatory text.

An example of a processed version of the page <NUM>, which would be generated by the region of interest classifier <NUM> is shown in <FIG> at <NUM>. The processed page <NUM> includes significant regulatory text identified as regions of interest <NUM>. Text <NUM> in the page header and footer is not identified as being within any of the regions, which implicitly identifies this text as redundant text. Each of the bounding boxes has a classification label, which in this embodiment also includes an associated level of confidence as a percentage. As an example, the first bounding box <NUM> is identified as a "table" with an associated confidence level of <NUM>%.

Although the processed page <NUM> is described here in the context of the training exercise, the region of interest classifier <NUM> produces a similar output when the trained region of interest neural network is implemented on the inference processor circuit <NUM>. The inference processor circuit <NUM> may also be configured to convert the actual images of regulatory text into grey-scale images prior to performing the region of interest classification, thus resulting in a faster inference time. Referring back to <FIG>, the region of interest classification output <NUM> thus facilitates the removal of redundant text regions <NUM> and other superfluous text to expedite further processing by the token mapper <NUM>.

In an alternative embodiment, the region of interest classifier <NUM> may be implemented using a single-stage detector that treats object detection as a simple regression problem based on learning classification probabilities and bounding box coordinates for input images. In this approach, single shot detector (SSD) networks may be implemented. In one embodiment a SSD network may be configured to use a pretrained CNN such as mobilenet to extract features from input images. The extracted features are then passed to SSD layers to output labels and bounding boxes for regions of interest. These steps are carried out in one forward pass through the network which makes the SSD neural network run faster than the more complex faster-R-CNN. In benchmarking tests performed by the inventors, Faster-RCNN was found to be more accurate than SSD in terms of exact bounding box coordinates. However, Faster-RCNN was found to be slower than the SSD neural network.

In some embodiments, the region of interest classifier is not only configured to identify redundant text within the regulatory content, but also to identify tables. In some embodiments, tables may be processed differently to text areas. Generally, removing redundant text does not require high accuracy to detect the exact bounding box of each region, and may be performed quickly. However, recognizing different type of tables requires that the image data be processed at high resolution, and using a more accurate neural network like Faster R-CNN may yield more accurate results.

Following the training exercise on the training system <NUM>, configuration data is generated and may be stored in the storage location <NUM> of the data storage memory <NUM> for directing the microprocessor <NUM> to implement the trained region of interest classifier <NUM> on the inference processor circuit <NUM>. The region of interest classification <NUM> generated by the region of interest classifier <NUM> is stored in the location <NUM> of the data storage memory <NUM>, when implemented on the processor circuit <NUM>.

Referring to <FIG>, a flowchart depicting blocks of code for directing a processor circuit of either the inference processor circuit <NUM> or the training system <NUM> to implement the token mapper <NUM> for processing the region of interest classification output <NUM> is shown at <NUM>. The blocks generally represent codes for directing the processor circuit <NUM> to generate the token mapping output <NUM>. The actual code to implement each block may be written in any suitable program language, such as Python, Go programming language, C, C++, C#, Java, and/or assembly code, for example.

The process <NUM> begins at block <NUM>, which directs the processor circuit to receive regulatory content image data. In the embodiment shown in <FIG>, the token mapper <NUM> receives regulatory content image data as region of interest classification output <NUM>, in which portions of the image data <NUM> that have been classified as being of interest for further processing are identified. In one embodiment the regulatory content image data is received as a plurality of pages, each page having a plurality of pixels representing the identified regulatory content as an image. In one embodiment, the image data each pixel is represented by a greyscale value.

Block <NUM> then directs the processor circuit to process the image data within the regions of interest to extract text. In the image data received at block <NUM>, the text is not directly accessible and requires further processing to recognize and encode the text into a series of characters and words. In one embodiment the processing may be performed by a machine learning service such as Amazon Textract™. Textract is a service that automatically extracts text and data from document images using advanced forms of optical character recognition (OCR) to identify, understand, and extract text data. Similar OCR services and are available from Google and other providers. For example, Google makes available open source library functions as Tesseract OCR, which has an ability to recognize more than <NUM> languages.

Block <NUM> then directs the processor circuit to further process the text to establish bounding boxes around the words and characters extracted in block <NUM>. An example of a page that has been processed using Textract is shown in <FIG> at <NUM>. A plurality of bounding boxes <NUM>, <NUM>, <NUM> etc. define locations of portions of text within the page image that have been identified as regions of interest. As an example, the bounding box <NUM> has bounding box coordinates indicated in <FIG> as Height Hb, Left X coordinate, Top Y coordinate, and Width Wb) that define the position and size of the bounding box for this text on the page. Textract provides a function that outputs these bounding boxes as a ratio with respect to the page image, which can be converted into pixel references within the image data by multiplying the coordinates by the number of pixels representing the page (e.g. <NUM> x <NUM> pixels).

In some embodiments, regulatory content documents may already include text data saved within the document. For example, while some Portable Document Format (PDF) documents are only encoded to represent text content as an image, other PDF documents are tagged to also provide the text associated with the content. In this case, it is unnecessary to extract text using OCR. The text content may rather be processed to only generate the bounding boxes <NUM>. As an example, the Unisuite library includes functions for generating the bounding boxes <NUM> for documents that have accessible text in a similar manner as described above for Textract. Other services are available from providers such as Retsina Software Solutions, who have a Pdf2Text product that provides similar functionality.

The process <NUM> continues at block <NUM>, which directs the processor circuit to generate language embeddings for the text within each bounding box. A language embedding is essentially a contextual representation of words or phrases using a vector of real numbers. Different words having similar meanings within a particular context will have similar vector representations. Language embeddings may be generated using a language model. For example, the language model may be implemented using a pre-trained language model, such as Google's BERT (Bidirectional Encoder Representations from Transformers) or OpenAl's GPT-<NUM> (Generative Pretrained Transformer). These language models are pre-trained using large multilingual datasets to capture the semantic and syntactic meaning of words in text and may be implemented to generate token data for text in any language that is input to the language model. In one embodiment, the language model may be implemented using BERT. The language model outputs token data in the form of tensors having <NUM> values for each identified token in the content. The language model may be implemented within a variety of machine learning environments such as Python, and PyTorch, or TensorFlow by invoking library functions. Block <NUM> causes the processor circuit of either the inference processor circuit <NUM> or training system <NUM> to generate a language embedding for each portion of the text or token within the bounding boxes on the page <NUM>.

Block <NUM> then directs the processor circuit to size each page image to match a selected height and width. If the image pages have size that differs from the fixed height and width, the page is thus resized accordingly. As an example, the image page may be sized to standard letter size of <NUM> inches wide by <NUM> inches high. For the example of a moderate resolution of <NUM> pixels per inch, each image page would this have <NUM> x <NUM> pixels. Ensuring that each image page has the same scaling facilitates accurate comparison of the tokens in the identified bounding boxes on the basis of font size, or boldness of the font, for example.

Block <NUM> then directs the processor circuit to generate the mapping output <NUM>. In one embodiment the mapping output <NUM> is a tensor representation for each page image. For example, a tensor having a size corresponding to the pixels on the page (i.e. <NUM> x <NUM> pixels for the example above) may be generated. In one embodiment the tensor may be represented by a tensor element having dimensions (Wp, Hp, C), where C is a tensor channel that corresponds to the language embedding <NUM>. A representation of the tensor mapping output <NUM> is shown in <FIG> and includes a plurality of tiles <NUM>. Each tile has a page height Hp and page width Wp corresponding to the pixel grid of the image representing the page. The C-channel value for each tile <NUM> is populated using the scalar values of the language embedding vector. As an example, the BERT language model generates a <NUM> real value vector output for each token of the language embedding. For the first tile <NUM> (i.e. tile <NUM>), each pixel within each of the bounding boxes <NUM>, <NUM>, and <NUM> has its greyscale pixel value replaced with a scalar value corresponding to the first element <NUM> of the language embedding for the token associated with the bounding box. Similarly, for the second tile <NUM>, each pixel within each of the bounding boxes <NUM>, <NUM>, and <NUM> has its greyscale pixel value replaced with a scalar value corresponding to the second element <NUM> of the language embedding for the token associated with the bounding box. The remaining tiles <NUM> - <NUM> are similarly populated with the scalar language embedding values. Directly encoding tokens in each bounding box using a single scalar value from the language embedding vector rather than by a granular collection of grayscale pixels, facilitates analysis of the mapping output <NUM> by the citation classifier <NUM>. Information such as the font size and font formatting is implicitly encoded in the channel C, since tokens having a larger font size would necessarily occupy more pixels. Furthermore, the group pixels from the original image that would have a number of different greyscale values is now mapped as a single scalar value. This has the effect of significantly down-sampling the mapping output <NUM> representation without losing any information. The mapping output <NUM> of the token mapper <NUM> provides an associated between the language embedding for text in the regulatory content and the corresponding portion of the page image where the text appears. The mapping output <NUM> thus represents contextual information in the form of the language embedding channel C and also represents visual features of the tokens.

In one embodiment the citation classifier <NUM> may be implemented using a deep convolutional neural network (CNN), an example of which is shown in <FIG> at <NUM>. In one embodiment the CNN <NUM> may be implemented using a U-net architecture, which has a contracting path <NUM> and an expansive path <NUM> making the network architecture U-shaped. The contracting path <NUM> is a typical convolutional network that consists of repeated application of convolutions <NUM> that also perform batch normalization and rectified linear unit (ReLU). The convolutions <NUM> are followed by a max pooling layer <NUM>. During the contraction, the spatial information in the mapping output <NUM> is reduced while feature information is increased. The expansive path <NUM> combines the features and spatial information through a sequence of up-convolutions <NUM> and concatenations <NUM> with high-resolution features from the contracting path <NUM>. A final layer <NUM> may be configured as a SoftMax layer, which maps the non-normalized output of the prior layer to a probability distribution for the various output classifications. For example, the final layer <NUM> may identify tokens in each of the bounding boxes as having a class of "citation number", "citation title", or "body text". A "citation number" classification would generally be applied in cases where the citation includes numeric or other characters that indicate a sequence. A "citation title" classification would generally be applied in cases where the citation includes only alphabetical text characters. The "body text" classification is applied to indicate text that occurs following a "citation number" or "citation title" but is not part of the citation. The SoftMax function causes probabilities to be assigned to each of the above classifications such that the output probabilities add up to <NUM>. The final layer <NUM> that provides the classification output <NUM> for the citation classifier <NUM> of <FIG>. The final layer <NUM> thus distinguishes between citation tokens and tokens that are not citation tokens (i.e. body text).

U-net architecture networks were originally used to segment regular image data represented by a tensor (Wp, Hp, C) for each input image, where Cis the depth of each pixel in the input image. In this embodiment, the C channel is replaced with the language embedding values in the mapping output <NUM>, which provides the input to the CNN <NUM>. The CNN <NUM> is initially trained on the training system <NUM> by providing training inputs to the CNN that have been labeled with a ground truth classification. A representation of a labeled training page image is shown in <FIG> at <NUM>. The training page image <NUM> has been processed to identify bounding boxes and language embeddings as described above. However, in this case the training page image <NUM> has been further processed to assign classification labels to the bounding boxes, as indicated by the greyscale shading applied to the boxes. The bounding boxes <NUM> are labeled as "citation numbers", bounding boxes <NUM> as "citation title", and bounding boxes <NUM> as "body text". The labeling thus corresponds to the classifications at the final SoftMax layer <NUM> that the CNN <NUM> will be trained to generate on the training system <NUM>. The effectivity of training may be enhanced by providing a relatively large number of labeled training page images <NUM> having different formats of citations and body text.

During training of the CNN <NUM>, training page images <NUM> are forward propagated through the CNN to the final layer <NUM> which outputs a classification prediction <NUM> for each bounding box. The predictions are compared to the ground truth labels established for the training page images <NUM>, and the weights wi and biases bi in the neurons of the layers of the CNN <NUM> are adjusted in proportion to how much each contributes to the overall error. The training of the CNN <NUM> is generally performed using library functions provided by machine learning environments such as Python, and PyTorch, or TensorFlow. Once the CNN <NUM> has been trained on a first portion of the training page images <NUM>, a second portion of the training page images may be used for validation. When the operator is satisfied with the performance of the CNN <NUM>, the configuration of the CNN, weights wi and biases bi are saved into the citation classifier configuration storage <NUM> in the data storage memory <NUM> of the inference processor circuit <NUM>.

A representation of a classification output by the final layer <NUM> of the CNN <NUM> is shown in <FIG> at <NUM>. When compared to the training page image <NUM> of <FIG>, the output correctly predicts the classification for bounding boxes including citation numbers <NUM>, citation title <NUM>, and body text <NUM>. However, the CNN <NUM> has incorrectly predicted that portion of body text <NUM> and <NUM> to have a classification of citation title. In most cases post-processing of the classification output <NUM> may resolve the incorrect classifications. In one embodiment post processing may involve assigning different thresholds for each classification. For example, if the probabilities assigned by the CNN <NUM> for "citation number", "citation title" and "body text" are <NUM>, <NUM>, and <NUM> respectively, it is not necessary to select the highest probability as the predicted classification. As an example, the threshold for "citation number" could be set to <NUM>. Additionally or alternatively, since a word may be represented by more than one token, the probabilities assigned to the tokens may be combined on a majority vote basis to resolve the classification for the full word. For example, "(A)" may be tokenized as "(", "A", ")". If two of these tokens have correct classification assigned, the entire word (A), can be correctly classified. A "citation title" classification may also include multiple words and if the first and last word are classified correctly as "citation title", the remaining words may be corrected to have the correct classification by virtue of their occurring between two "citation tittle" words.

In the above CNN embodiment shown in <FIG>, the mapping output <NUM> of the token mapper <NUM> provides the input to the CNN <NUM> includes both spatial information from the page images of the regulatory content and the language embedding values <NUM> generated by the language model. In another embodiment shown in <FIG>, a CNN <NUM> may configured to receive and process the image data within the token mapping output <NUM>, while a language embedding map <NUM> is concatenated with image features generated at the final layers <NUM> of the CNN to generate a classification prediction <NUM>. The language embedding map is generated for each pixel inside the area of a word, by using the corresponding word embedding as the input. Pixels that belong to the same word thus share the same language embedding. Pixels that do not belong to any words will be filled with zero vector values. For a document image of size H x W, this process results in an embedding map of size N x H x W if the learned word embeddings are N-dimensional vectors (for example N = <NUM> for the BERT language model). The language embedding map <NUM> is later concatenated with the feature response generated by the neural network along the channel dimensions.

Referring back to <FIG>, the final layer <NUM> of the CNN <NUM> thus provides the classification output <NUM> in which text within each bounding box has a classification prediction. The classification output <NUM> thus facilitates identification of tokens in the regulatory content as citation numbers and citation title within the regulatory content pages from body text. At this stage however, the citation numbers and citation title have no established hierarchy. The classification output <NUM> is further processed by the hierarchical relationship classifier <NUM> to extract this hierarchical information.

Hierarchical information may be extracted from the citation tokens by determining parent-child relationships between citation tokens to build a hierarchical tree of citations. An example of a hierarchical tree structure is shown in <FIG> at <NUM>. A topmost node in the hierarchical tree <NUM> is referred to as the root node <NUM> and a plurality of nodes <NUM> stem from the root node in levels li = <NUM>. Every node <NUM> other than the root node <NUM>, is connected to at least one other node. When moving up the tree structure, a node <NUM> that is connected to a node on a higher level li is referred to as a child node. The node to which the child node is connected to is referred to as a parent node. Two nodes on the same level li are referred to as siblings. The hierarchical relationship classifier <NUM> processes the classification output <NUM> to establish parent, child, and sibling relationships among nodes <NUM>, <NUM>, which facilitates construction of the hierarchical tree <NUM> for the regulatory content.

Referring to <FIG>, a process for implementing the hierarchical relationship classifier <NUM> to generate the hierarchical tree <NUM> on the inference processor circuit <NUM> is shown generally at <NUM>. The process begins at block <NUM>, which directs the microprocessor <NUM> of the processor circuit <NUM> to generate a list of citation tokens based on classification output <NUM> of the citation classifier <NUM>. Referring to <FIG>, an example of a data flow within the hierarchical relationship classifier <NUM> is shown generally at <NUM>. listing of citation tokens is shown at <NUM>. The listing <NUM> includes a plurality of tokens that were classified as either citation numbers of citation title. Tokens classified as body text in the classification output <NUM> are not included in the listing <NUM>.

Block <NUM> of the process <NUM> then directs the microprocessor <NUM> to construct pairwise combinations <NUM> including all possible pairs of citation tokens from the listing <NUM>. In generating each pair of tokens in the pairwise combinations <NUM>, a spatial information related to the location of the citation tokens is also embedded within the pairs. In one embodiment the pairwise combinations <NUM> may be encoded to include spatial information corresponding to the degree of indentation of the citation token on the page. For example, the spatial information may be added via a special token '<T>'. As an example, a pairwise combination <NUM> of citation tokens 'a. ' may be encoded as ( <T> <T> 'a. ' , <T> <T> <T> 'b. The citation tokens citation tokens 'a. ' thus have differing levels of indentation within the document, which may be an indication that while they share the same citation pattern, they may not actually be siblings.

Block <NUM> then directs the microprocessor <NUM> to compare each of the pairs of citation tokens <NUM> to determine whether the tokens have a sibling relationship. In one embodiment the sibling classifier neural network may be implemented using a neural network that has been trained on the training system <NUM> using pairs of citation tokens that are labeled as being siblings or not siblings. The training pairs used to train the neural network include the level of indentation information indicated by the special token '<T>'. Incorporating this spatial information provides a more nuanced representation of the citation tokens for training the sibling classifier neural network. The sibling classifier neural network may be implemented using a language model, which has been configured to recognize the tabulation token <T> as part of its vocabulary. In one embodiment the language model neural network may be trained for the sibling classification function using a plurality of labeled pairs of sibling tokens and labeled pairs of non-sibling tokens.

The configuration of the sibling classifier neural network is stored in the location <NUM> of the data storage memory <NUM> of the inference processor circuit <NUM> for implementing the sibling classifier on the system <NUM>.

The sibling classifier neural network, when implemented on the inference processor circuit <NUM>, thus receives a token pair from the pairwise combinations <NUM>, an outputs a probability SIB indicating the likelihood of the pair being siblings.

In other embodiments, the comparison between siblings may be performed by a non-neural network implemented similarity function. For example, other classifiers such as support-vector machines (SVM), Naïve Bayes classifiers, k-nearest neighbors (k-NN) classifiers, etc, may be used to generate the sibling comparison. Features such as the indentation level of the token, font style, font size, and the word representation may be fed as feature inputs into these classifiers.

The process <NUM> then continues at block <NUM>, which directs the microprocessor <NUM> to populate the probability SIB in a similarity matrix. An example of a similarity matrix is shown in a tabular format in <FIG> at <NUM>. The rows and columns of the similarity matrix <NUM> both correspond to the citation token listing <NUM>, which provides an entry in the matrix for each of the pairwise combinations <NUM>. The diagonal elements of the similarity matrix <NUM> are all set to probability <NUM>, since these elements represent the similarity between each citation token paired with itself (probability SIB is expressed as a number between <NUM> and <NUM>). Once the SIB probability has been determined for a particular pairwise combination, the corresponding element is populated with the probability value. Block <NUM> then directs the microprocessor <NUM> to determine whether all of the pairwise combinations <NUM> have been processed. If not, the microprocessor <NUM> is directed back via block <NUM> to process the next pairwise combination. Once all pairwise combinations have been processed, block <NUM> directs the microprocessor <NUM> to block <NUM>.

Block <NUM> directs the microprocessor <NUM> to analyze the similarity matrix <NUM> to extract the parent, child, and sibling hierarchical information. The analysis process generally involves identifying sub-matrices within the similarity matrix <NUM> that are indicative of the hierarchy based on a threshold probability. In the example shown in <FIG>, the SIB probability threshold is set at <NUM>. Starting on the first row "Section <NUM>" and first column "Section <NUM>. ", the microprocessor <NUM> compares each of the SIB probabilities against the threshold. The SIB probability at the first row and column is <NUM>, which defines the start of a first sub-matrix <NUM> (shaded light gray). Moving along the columns, the first sub-matrix <NUM> is then extended to the first SIB probability that is <NUM> or higher (i.e. "Section <NUM>. = <NUM>" in this case). The first sub-matrix <NUM> is thus defined to include all the columns up to the "Section <NUM>" column, and all rows down to the "Section <NUM>" row. The first sub-matrix <NUM> this identifies "Section <NUM>" and "Section <NUM>" as being siblings at the same level li = <NUM> of the tree <NUM> (<FIG>) under the root node <NUM>.

The same process is repeated for the second row, where it is found that the first SIB probability that meets the threshold is at "Part <NUM>. This defines a second sub-matrix <NUM> (shown in a darker shade of gray), that identifies "Part <NUM>" and "Part <NUM>" as siblings on the level li = <NUM> of the tree <NUM>, under the "Section <NUM>" parent. Similarly, for the third row of the similarity matrix <NUM>, the citation tokens "a" and "b" are determined to be siblings under the parent "Part <NUM>" at the level li = <NUM> of the tree <NUM>, based on the sub-matrix <NUM>. The next sub-matrix <NUM> (shown in broken outline) identifies the citation token "b" and "c" as siblings, which places the citation token "c" on level li = <NUM> with tokens "a" and "b". Sub-matrix <NUM>, identifies "I" and "ii" as being siblings at level li = <NUM>, under parent "b".

Block <NUM> directs the microprocessor <NUM> to generate the hierarchical tree <NUM> as described above, which defines the hierarchical connections between the citation tokens <NUM>. Generally, text classified as being body text following a citation token up to the next citation token in the hierarchy may be associated with the citation token it follows for further process. The further processing may involve analyzing the body text under the citation and is not further described herein.

Claim 1:
A computer-implemented method for identifying citations within regulatory content, the method comprising:
receiving image data (<NUM>) representing a format and layout of the regulatory content;
receiving a language embedding (<NUM>) including a plurality of tokens representing words or characters in the regulatory content;
generating a plurality of token mappings (<NUM>), each token mapping associating each token of the plurality of tokens with a portion of the image data (<NUM>), wherein the image data (<NUM>) comprises pixel-based image data and the plurality of token mappings (<NUM>) are convertible to pixel references in the pixel-based image data;
receiving the plurality of tokens and the plurality of token mappings (<NUM>) at an input of a citation classifier (<NUM>), the citation classifier (<NUM>) having been trained to generate a classification output (<NUM>) for each token of the plurality of tokens based on the language embedding (<NUM>) and the token mapping (<NUM>) for the each token, the classification output (<NUM>) identifying a plurality of citation tokens within the plurality of tokens; and
processing the plurality of citation tokens to determine a hierarchical relationship between citation tokens of the plurality of citation tokens, the hierarchical relationship being established based at least in part on the token mapping (<NUM>) for the citation tokens.