Package: text-alignment-tool
The Text Alignment ToolThis Python text alignment tool is intended to be a general purpose tool for aligning texts in a robust and easily extensible way. It tracks any changes to the original text so that there is an end-to-end mapping of the alignment data.ArchitectureInline-style:The alignment tool consists of a main classTextAlignmentTool, which coordinates the alignment process.The alignment tool receives a singleTextLoaderfor the query text and a singleTextLoaderfor the target text (you must keep track of the mapping from the original input text(s) in theTextLoaderand its output to the rest of the pipeline).The alignment tool is then fednTextTransformers and for each text andnAlignmentAlgorithms. These can be used in any combination and order, for example: the query text could pass through 3TextTransformers and the target text could pass through 1TextTransformer, then they go through a singleAlignmentAlgorithm, the target text then passes through 2TextTransformers and we could perform a finalAlignmentAlgorithmon the pair of texts.find_alignment_to_queryandfind_alignment_to_targetwill backtrack through the text mappings and provide a key for mapping either the query to the target or the target to the query.A somewhat basic alignment process could look something like this:# Create text loaders for query and targetquery_loader=PgpAltoXMLTextLoader(list(QUERY_TEMP_FOLDER.glob("**/*.xml")))target_loader=PgpXmlTeiTextLoader(list(TARGET_TEMP_FOLDER.glob("**/*.xml")))# Create the alignment toolaligner=TextAlignmentTool(query_loader,target_loader)# Perform three transformation operations on the targetnormalize_target_sigla=PgpTeiNormalizeSiglaTransformer()remove_target_extras=PgpTeiRemoveExtrasTransformer()relocate_insertions=PgpTeiRelocateInsertionsTransformer()aligner.target_text_transforms([normalize_target_sigla,remove_target_extras,relocate_insertions])# Create and run one alignment processfirst_alignment_algorithm=LineAlignmentAlgorithm()aligner.align_text(first_alignment_algorithm)# Get the mapping information for the alignmentalignment_mappings=aligner.latest_alignmentFunctionalityTracking of text changes and mappings to aligned text use a system of index maps. TheTextLoaderwill ingest the input text and output a 1-dimensional numpy uint32 array consisting of one number for each letter in the input text in the order it occurs within the text (the number is simply the unicode value of the character using python'sordfunction).Text LoaderFor example, let's imagine we have our initial text in a simple text file, and we will assume the line breaks are significant for the alignment process:Lorem ipsum dolor sit amet 
consectetur adipiscing elit
sed do eiusmod tempor incididunt ut labore et dolore magna aliquaWe would write a simple loader for this to ingest the text and preserve a record of the line breaks:fromtext_alignment_toolimportTextChunkimportnumpyasnptext="""Lorem ipsum dolor sit ametconsectetur adipiscing elitsed do eiusmod tempor incididunt ut labore et dolore magna aliqua"""defparse_text(text:str)->Tuple[List[Tuple[int,int]],List[TextChunk],np.array]:input_output_map:List[Tuple[int,int]]=[]text_chunk_indices:List[TextChunk]=[]output_text:List[int]=[]text_chunk_start_idx=0forinput_idx,charinenumerate(text):output_idx=len(output_text)input_output_map.append((input_idx,output_idx))ifchar=="\n":text_chunk_indices.append(TextChunk(text_chunk_start_idx,output_idx))text_chunk_start_idx=output_idx+1continueoutput_text.append(ord(char))returninput_output_map,text_chunk_indices,np.array(output_text,dtype=np.uint32)input_output_map,text_chunk_indices,output_text=parse_text(text)# Inspect the resultsprint(input_output_map[25:30])print(text_chunk_indices)print(output_text[0:5])# Deserialize textprint(''.join([chr(x)forxinoutput_text[0:5]]))Output:[(25, 25), (26, 26), (27, 27), (28, 27), (29, 28)]
[TextChunk(start_idx=0, end_idx=27), TextChunk(start_idx=28, end_idx=55)]
[ 76 111 114 101 109 ]
LoremWhen creating a custom text loader, you should subclassTextLoaderand make sure to calculateself._output,self._input_output_map, andself._text_chunk_indices.  You can modify the __init__() method to take whatever variables you need, and you can modify the class however it is needed to perform the parsing operation.  It is a nice addition to include a method in the customTextLoaderto rebuild text in the input format based upon the data from the alignment operation.Text TransformerThe output of theTextLoadermay be exactly what is needed for the alignment process, but often it will be necessary to perform other alterations such as stripping out unneccesary characters, performing some rule based character conversions, or refining the text_chunks.  Any number ofTextTransformers can be used in series to accomplish this.  Using narrowly focusedTextTransformers will make it easier to debug and to mix and matchTextTransformeras needed to achieve the desired alignment.When passing a text through aTextTransformer, the transformer must use its_input_output_mapto track how it has changed the input. For instance, if we wanted to create a transformer to remove the word "the", we might start with a text input like "the quick brown dog jumped over the lazy fox.", which in the alignment tool is:[116, 104, 101, 32, 113, 117, 105, 99, 107, 32, 98, 114, 111, 119, 110, 32, 100, 111, 103, 32, 106, 117, 109, 112, 101, 100, 32, 111, 118, 101, 114, 32, 116, 104, 101, 32, 108, 97, 122, 121, 32, 102, 111, 120, 46]The output from theTextTransformerwould be:[113, 117, 105, 99, 107, 32, 98, 114, 111, 119, 110, 32, 100, 111, 103, 32, 106, 117, 109, 112, 101, 100, 32, 111, 118, 101, 114, 32, 108, 97, 122, 121, 32, 102, 111, 120, 46]And the_input_output_mapwould show the mappings from the index of the input array to the index of the output array:[(4,0),(5,1),(6,2),(7,3), ...]input valmap input idx to output idxoutput val113(4,0)113117(5,1)117105(6,2)10599(7,3)99107(8,4)10732(9,5)32.........Changing the order of individual elements in the list is also possible, for instance for the same input above we could instead have the output:[98, 114, 111, 119, 110, 32, 113, 117, 105, 99, 107, 32, 116, 104, 101, 32, 100, 111, 103, 32, 106, 117, 109, 112, 101, 100, 32, 111, 118, 101, 114, 32, 116, 104, 101, 32, 108, 97, 122, 121, 32, 102, 111, 120, 46]The words "the" and "brown" have been transposed, and the resulting_input_output_mapwould be:[(10,0),(11,1),(12,2),(13,3),(14,4),(3,5),(4,6),(5,7),(6,8),(7,9),(8,10),(9,11),(0,12),(1,13),(2,14),(15,15), ...]TheTextTransformermay also redefine text chunks with the_text_chunk_indicesproperty, which is a simple ordered list of starting + ending indices that definensections of the output text (you may use overlapping sections if desired), e.g.,[(0,20),(21,35),(30,91)]with three chunks of the text using indices: 0–20, 21–35, and 30–91.Alignment AlgorithmTheAlignmentAlgorithmclass can be subclassed to perform analysis of both the query and target text at the same time. Any number of such classes may be used at any place within the alignment pipeline. TheAlignmentAlgorithmwill always receive aself._queryand aself._targetproperty, both of which are provided automatically to it by theTextAlignmentToolfrom the output of the latest transformation of the query and target texts. It will also automatically receive the latest_text_chunk_indicesfor the query and for the target asself._input_query_text_chunk_indicesandself._input_target_text_chunk_indices.AnAlignmentAlgorithmwill produce a mapping in the_alignmentproperty, a simplified example of which might be: query =['h','e','l','l','o',' ','w','o','r','l','d']and target =['h','e','l','l','o',' ','w','a','d','d'](of course these would be lists of uint32's in our system, not strings) could be aligned as[(0,0),(1,1),(2,2),(3,3),(4,4),(5,5),(6,6),(10,9)](for Wadd, seehttps://en.wikipedia.org/wiki/Wadd).AnAlignmentAlgorithmcan also be used to redefine text chunks based on mutual analysis of the query and target texts.  That is, theAlignmentAlgorithmmay be used both for gross alignments—defining possibly corresponding text chunks with the properties_output_query_text_chunk_indicesand_output_target_text_chunk_indicesin addition to the fine grained alignment using the_alignmentproperty, which is simply a list of the corresponding character indices in the query and source text.Alignment Operation TrackingTheTextAlignmentToolautomatically keeps track of the order of operations and the transforms that have been performed in the__operation_listproperty which contains a list ofAlignmentOperations. This simplifies peeking in on any part of the alignment process for debugging purposes and also enables custom mappings between query and target.The convenience methodsfind_alignment_to_queryandfind_alignment_to_targetenable you to walk the alignments and transforms back to the first initial input provided by theTextLoader. You will need to provide your own function within theTextLoaderto transform the aligned text into your desired format.Debugging HelpWhen you use theTextAlignmentToolin a debugging context, it will inject an instance of theDebugHelperclass into the global context asdbg. This helper provides four convenience methods to inspect your aligment pipeline:dbg.display_text,dbg.display_text_chunk,dbg.display_text_chunks, anddbg.display_text_region. These methods will output the human readable text for the internal uint32 numpy array numeric representation of the text and can extract specified ranges and text chunks as well.