Patent ID: 12204856

DETAILED DESCRIPTION

The current subject matter is directed to machine learning-based techniques for organizing long texts into coherent segments and, in particular, to domain transfer of supervised text segmentation models, with a focus on the educational domain. To investigate the effects of domain transfer in supervised text segmentation, K12SEG, a segment-annotated dataset was automatically created from educational texts designed for grade-1 to college-level student population. A hierarchical neural text segmentation model (HITS) was benchmarked in a range of in-domain and domain-transfer segmentation experiments involving WIKI727 and the new K12SEGdataset.

FIG.1is a diagram100illustrating an architecture of an adapter-augmented hierarchical model for supervised text segmentations. With this HITS model, the parameters of the lower (token-level) token transformer110can be initialized with the weights of the pretrained RoBERTa. In addition, aiming to prevent both (1) forgetting of distributional information captured in RoBERTa's parameters and (2) overfitting to the training domain—the layers of the token-level transformer110can be augmented with adapter parameters before segmentation training. With this arrangement, adapter-based fine-tuning only updates the additional adapter parameters and the original transformer parameters are frozen: this fully preserves the distributional knowledge obtained in transformer's pretraining. Encoding out-of-domain segmentation knowledge (e.g., from the WIKI727 dataset) separately from the distributional information (original RoBERTa parameters) as provided herein allows for the combination of the two types of information more flexibly during the secondary segmentation training in the target domain (e.g., on K12SEG), resulting in more effective domain transfer.

Experimental results confirmed the advantages of the current techniques. The adapter-augmented HITS model trained on WIKI727, besides yielding state-of-the-art in-domain (Wikipedia) segmentation performance, facilitates domain transfer and leads to substantial gains in the target educational domain (K12SEG).

Hierarchical Transformer-Based Model. The base segmentation model as illustrated inFIG.1comprises two hierarchically linked transformers: the lower token transformer110contextualizes tokens within sentences and yields sentence embeddings; the upper sentence transformer120then contextualizes sentence representations. An individual training instance is a sequence of N sentences, {S1, . . . , SN}, each consisting of T (subword) tokens, Si={Wi,1, . . . , W1,T}. The lower transformer110can be initialized with the pretrained RoBERTa weights. The transformed vector of the sentence start token (), Si, can be used as the embedding of the sentence Si. The purpose of the randomly initialized upper sentence-level transformer120is to contextualize the sentences in the sequence with one another. Let xi be the transformed representation of the sentence Si, produced by the upper transformer120. The segmentation prediction for the sentence Siis then made by the segmentation classifier130(e.g., a simple feed-forward softmax classifier: Yi=softmax (Wxi+b)). Binary cross-entropy loss can be minimized.

Adapter-Based Training. The lower transformer110can be initialized with RoBERTa weights, encoding general-purpose distributional knowledge. Full fine-tuning, in which all transformer parameters are updated in downstream training, may overwrite useful distributional signal with domain-specific artefacts, overfit the model to the training domain, and impede domain transfer for the downstream task. Alternatively, adapter-based fine-tuning injects additional adapter parameters into transformer layers and updates only them during downstream fine-tuning, keeping the original transformer parameters unchanged. To address these issues, a bottleneck adapter architecture can be adopted such that, in each layer of the lower transformer110, two bottleneck adapters can be inserted: one after the multi-head attention sublayer and another after the feed-forward sublayer. Let X∈RT×Hstack contextualized vectors for the sequence of T tokens in one of the transformer layers, input to the adapter layer. The adapter then yields the following output:
X′=X+g(XWd+bd)Wu+bu.

The parameter matrix Wd∈RH×adown-projects the token vectors from X to the adapter size a<H, and Wu∈Ra×Hup-projects the activated down-projections back to transformer's hidden size H; g is the non-linear activation function.

Training Instances and Inference. The HITS model with sequences of N sentences as instances can be created by sliding the window of size N over document's sentences, with a sliding step of N/2. At inference, for each sentences, predictions can be made for all of the windows that contains. This means that (at most) N segmentation probabilities are obtained for each sentence (for the i-th sentence, we get predictions from windows [i−N+1: i], [i−N+2: i+1], . . . , [i: i+N−1]). The sentence's segmentation probabilities obtained across different windows can be averaged and it can be predicted that a sentence starts a new segment if the average is above the threshold t. The sequence length N and threshold t can act as hyperparameters and be optimized using the development datasets Wiki727 and K12Seg. Each of such data sets has three disjoint portions: “train” portion (or train dataset), “development” (or sometimes also called “validation”) portion (or development dataset) and test (or “evaluation”) portion/dataset.

WIKI727. WIKI727 is a large segment-annotated dataset designed for supervised text segmentation. It consists of 727K Wikipedia articles (train portion: 582K articles), automatically segmented according to the articles' heading structure.

K12SEG. As noted above, to empirically evaluate domain transfer in supervised text segmentation, a new dataset, K12SEG, was created from educational reading material designed for grade-1 to college-level students. The seed dataset, the Educators Word Frequency, was created by standardized sampling of reading materials from a variety of content areas (e.g. science, social science, home economics, fine arts, health, business etc.). Each sample is 250-325 words long. One synthetic K12SEGinstance was created by selecting and concatenating two samples from (a) the same book, (b) different books from the same content area (e.g., science) or (c) different books from different content areas. In contrast to WIKI727, in which the number and sizes of segments greatly vary across Wikipedia articles, K12SEGdocuments are more uniform: with two segments (samples) each and minor variation in sentence length (mean: 30 sentences). Besides the different genre between WIKI727 and K12SEG, this stark difference between their distributions of segment numbers and sizes poses an additional challenge for the domain transfer. The total of 18,906 K12SEGdocuments were split into train (15,570 documents), development (3,000), and test portions (336). An example 2-segment document from from K12SEGis shown in Table 1.

Wikipedia-Based Test Sets. For an in-domain (Wikipedia) evaluation, three small-sized test sets were used: WIKI50 which is an additional test set consisting of 50 documents, in addition to: CITIES which comprises 64 articles, and ELEMENTS which comprises 117 articles from Wikipedia pages of world cities and chemical elements, respectively.

Experiments. Two sets of experiments were conducted. First, the performance of the HITS model was benchmarked “in domain”, i.e., by training it on WIKI727 and evaluating it on WIKI50, ELEMENTS, and CITIES. Here, HITS (with full and adapter-based fine-tuning) was directly compared with state-of-the-art segmentation models: the hierarchical Bi-LSTM (HBi-LSTM), and two hierarchical transformer variants of Glavas and Somasundaran—with (CATS) and without (TLT-TS) the auxiliary coherence objective. The second set of experiments quantified the efficacy of HITS in transfer for the educational domain. The performance of “in-domain” training on K12SEGwas compared with transfer strategies: (i) zero-shot transfer: HITS variants (full and adapter-based fine-tuning) trained on WIKI727 and evaluated on the K12SEGtest set and (ii) sequential training: HITS variants sequentially trained first on WIKI727 and then on the train portion of K12SEG.

Training and Optimization Details. The weights of the lower transformer network in all HITS variants were initialized with the pretrained RoBERTa Base model, having LL=12 layers (with 12 attention heads each) and hidden representations of size H=768. The upper-level transformer120for sentence contextualization had LU=6 layers (with 6 attention heads each), and the same hidden size H=768. A dropout (p=0.1) was applied on the outputs of both the lower and upper transformer outputs110,120. In adapter-based fine-tuning, the adapter size was set to a=64 and GeLU was used as the activation function. In the experiments, the sentence input was limited to T=128 subword tokens (shorter sentences are padded, longer sentences trimmed). The models' parameters were optimized using the Adam algorithm with the initial learning rate of 10−5. Training occurred for at most 30 epochs over the respective training set (WIKI727 or K12SEG), with early stopping based on the loss on the respective development set. Training was conducted in batches of 32 instances (i.e., 32 sequences of N sentences) and it was found (via cross-validation on respective development sets) that the optimal sequence length was N=16 sentences and the optimal average segmentation probability threshold at inference time was t=0.35.

TABLE 1An example 2-segment document from the K125EG dataset.First segmentSecond segmentTraveling familiar routes inWorking in the mud andour family cars we growwater of a river bottom wasso accustomed to crossingdifficult and dangerous.small bridges and viaductsPeople were often crushed orthat we forget they are there.maimed by the pile driverWe have to stop and thinkor the piles. But the work onto remember how often theythe foundations is the mostcome along. Only when aimportant part of bridge-bridge is closed for repairsbuilding. The part of aand we have to take a longbridge that is underwater, thedefour do we realize howpart we never see, is moredifficult life would be with-important than the part weout it. Try to imagine ourdo see, because no matterworld with all the bridgeshow well made the super-removed. In many placesstructure may be, if thelife would be seriously dis-foundation is not solid therupted, traffic would bebridge will fall. Not onlyparalyzed, and business woulddid the pier foundations havebe terrible. Bridges bring usto be solid, they also had totogether and keep usbe protected as much astogether. They are a basicpossible from wear. Anecessity for civilization. Theflowing river constantly stirsfirst structures human beingsup the bottom, so that thebuilt were bridges. Beforewater's lower depths are aprehistoric people began tothick soup filled with mudbuild even the crudestand sand and pebbles whichshelter for themselves, theygrind against anything inbridged streams. Early pre-the path of the current Thishistoric tribes were wanderers.action is called scour, toSince they did not stayreduce the wear and tear ofin one place they did notthe current, the Romansthink of building themselvesbuilt the fronts of their piershouses. But they could notin the shape of a boat'swander far without findingprow. The Romans used onlya stream in their way. Natureone kind of arch, theprovided the first bridges.semicircular. The arch describesFinding themselves confronteda full half-circle from pierwith some narrow butto pier. Each end of therapid river, humans noticedhalf-circle rests on a pier,a tree that had fallen acrossand the two piers will holdthe river from bank to bank.the arch up by themselves,The person who firsteven before the rest of thescrambled across a fallen log,bridge is built, providedperhaps after watchingeach pier is at least one third asmonkeys run across it, wasthick as the width of the arch.the first human being toThus a bridge could becross a bridge. Eventually,built one arch at a time, andwhen they had learned howif the work had to stop theto chop down a tree, theypartial structure would stayalso learned how to makein place until work coulda tree fall in the directionbe resumed. The Romansthey wanted it to fall. Thebuilt their bridges during thewandering tribe that firstsummer and fall, when thedeliberately made a free fallweather was best and theacross & stream were thewater level was generallyfirst bridge-builders.lowest, and stopped duringwinter and spring.

Results. The results are reported in terms of PKwhich is an evaluation metric for text segmentation. PKis the percentage of wrong predictions on whether or not the first and last sentence in a sequence of K consecutive sentences belong to the same segment. K was set to one half of the average gold-standard segment size of the evaluation dataset.

In-Domain Wikipedia Evaluation. The results of the in-domain Wikipedia evaluation on WIKI50, CITIES, and ELEMENTSare provided in Table 2. The current HITS model variants, which was trained with the pretrained RoBERTa as the token-level transformer, outperformed the hierarchical neural models which start from a randomly initialized token-level encoder. It will be appreciated that fine-tuning pretrained transformers yields better results than task-specific training from scratch, even if the dataset is large (as is the case with WIKI727). Full fine-tuning produces better results on WIKI50, whereas adapter-based fine-tuning exhibits stronger performance on CITIES and ELEMENTS. As the articles in WIKI50 come from a range of Wikipedia categories, much like in the training set WIKI727, whereas CITIESand ELEMENTSeach contain articles from a single category, these results indicate that full fine-tuning is more prone to domain (genre, topic) overfitting than adapter-based tuning. Of note, the current HITS model (Full) surpassed the human WIKI50 performance, reported to stand at 14.97 PKpoints.

TABLE 2“In-domain” performance of hierarchical neural segmentationmodels, trained on the large WIKI727dataset, on three Wikipedia-based test sets (smaller valuesof the error measure. PKmean better performance).ModelFine-tuningWIKI50CITIESELEM.HBi-LSTM—18.2419.6841.63TLT-TS—17.4719.2120.33CATS—16.5316.8518.41HITS (ours)Fall14.5015.0317.06HITS (ours)Adapter15.1714.1114.67

Domain Transfer Results. Table 3 shows the performance of both in-domain and transferred HITS model variants on the K12SEGtest set. With Full fine-tuning, it was observed that the same performance was achieved regardless of whether we train the model on the out-of-domain (but much larger) WIKI727 dataset or the (smaller) in-domain K12SEGtraining set. Also of note, adapter-based fine-tuning in the zero-shot domain transfer yielded better performance than in-domain adapter fine-tuning. Poor performance of in-domain training may mean that K12SEGis either (a) insufficiently large or (b) contains such versatile segmentation examples over which it is hard to generalize. Gains from sequential domain transfer, in which the model is exposed to exactly the same K12SEGtraining set but only after it was trained on a much larger out-of-domain WIKI727 dataset, point to (a) as the more likely explanation. In both in-domain and zero-shot setups, adapter-based fine-tuning produces better segmentation than full fine-tuning, contributing to the conclusion that adapter-based fine-tuning curbs overfitting to domain-specific arte-facts, improving the model's generalization ability. Finally, the sequential training in which the lower transformer's parameters (including the adapters) during the (secondary) in-domain training were frozen which gave the best result overall. It is believed that the relatively small K12SEGtrain set gives the advantage to the model variant that uses that limited-size data to fine-tune fewer parameters (i.e., only the upper, sentence-level transformer120).

TABLE 3Segmentation performance in domain transfer.SetupFine-tuningFreezeK12SEG (test)In domainFull—25.5Adapter—23.9Zero-shotFull—25.5Adapter—20.7SequentialFullNo12.9FullYes14.8AdapterNo13.3AdapterYes10.4Evaluation on K12SEG test set.In domain: training on the K12SEG train set; Zero-shot: training on the train portion of WIK1727; Sequential: sequential training, first on WIKI727 and then on the train portion of K12SEG. For Sequential training, the column Freeze specifies whether the the lower transformer's parameters were frozen during secondary, in-domain fine-tuning (on the train portion of K12SEG).

FIG.2is a diagram200illustrating a process flow diagram for text segmentation in which, at210, data is received that comprises unstructured text including a sequence of sentences. Thereafter, at220, the received data is tokenized into a plurality of tokens. The received data is then segmented, at230, using a hierarchical transformer network model. The hierarchical transformer network model includes a token transformer, a sentence transformer, and a segmentation classifier. The token transformer contextualizes tokens within sentences and yielding sentence embeddings. The sentence transformer contextualizes sentence representations based on the sentence embeddings. The segmentation classifier predicts segments of the received data based on the contextualized sentence representations. Data can then be provided, at240, that characterizes the segmentation of the received data.

FIG.3is a diagram300illustrating a sample computing device architecture for implementing various aspects described herein. A bus304can serve as the information highway interconnecting the other illustrated components of the hardware. A processing system308labeled CPU (central processing unit) (e.g., one or more computer processors/data processors at a given computer or at multiple computers) and/or a graphical processing unit310labeled GPU, can perform calculations and logic operations required to execute a program. A non-transitory processor-readable storage medium, such as read only memory (ROM)312and random access memory (RAM)316, can be in communication with the processing system308and can include one or more programming instructions for the operations specified here. Optionally, program instructions can be stored on a non-transitory computer-readable storage medium such as a magnetic disk, optical disk, recordable memory device, flash memory, or other physical storage medium.

In one example, a disk controller348can interface with one or more optional disk drives to the system bus304. These disk drives can be external or internal floppy disk drives such as360, external or internal CD-ROM, CD-R, CD-RW or DVD, or solid state drives such as352, or external or internal hard drives356. As indicated previously, these various disk drives352,356,360and disk controllers are optional devices. The system bus304can also include at least one communication port320to allow for communication with external devices either physically connected to the computing system or available externally through a wired or wireless network. In some cases, the at least one communication port320includes or otherwise comprises a network interface.

To provide for interaction with a user, the subject matter described herein can be implemented on a computing device having a display device340(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information obtained from the bus304via a display interface314to the user and an input device332such as keyboard and/or a pointing device (e.g., a mouse or a trackball) and/or a touchscreen by which the user can provide input to the computer. Other kinds of input devices332can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback by way of a microphone336, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The input device332and the microphone336can be coupled to and convey information via the bus304by way of an input device interface328. Other computing devices, such as dedicated servers, can omit one or more of the display340and display interface314, the input device332, the microphone336, and input device interface328.

One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and/or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, and/or in assembly/machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and/or device, such as for example magnetic discs, optical disks, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. The machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example as would a processor cache or other random access memory associated with one or more physical processor cores.

In the descriptions above and in the claims, phrases such as “at least one of” or “one or more of” may occur followed by a conjunctive list of elements or features. The term “and/or” may also occur in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, such a phrase is intended to mean any of the listed elements or features individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases “at least one of A and B;” “one or more of A and B;” and “A and/or B” are each intended to mean “A alone, B alone, or A and B together.” A similar interpretation is also intended for lists including three or more items. For example, the phrases “at least one of A, B, and C;” “one or more of A, B, and C;” and “A, B, and/or C” are each intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” In addition, use of the term “based on,” above and in the claims is intended to mean, “based at least in part on,” such that an unrecited feature or element is also permissible.

The subject matter described herein can be embodied in systems, apparatus, methods, and/or articles depending on the desired configuration. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and/or variations can be provided in addition to those set forth herein. For example, the implementations described above can be directed to various combinations and subcombinations of the disclosed features and/or combinations and subcombinations of several further features disclosed above. In addition, the logic flows depicted in the accompanying figures and/or described herein do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations may be within the scope of the following claims.