Package: ndl-tense
ndltensesA package for training an NDL model to learn tenses using pyndl; can also be used to annotate text for tenseinstallationThis should be installable from PyPi so this means it requires pip.
You can install the latest pip by typing "pip install pip".
From a terminal, type "pip install ndl=tense".
":$ pip install ndl-tense"parameter fileThis file is what's used to tailor the execution of the processing to your needs. In the future I hope to create a GUI interface to avoid this.
In the GitHub repository there is an already created parameter file with example values set and default parameter file that you should use.There are some necessary files (NF) that need to be created by yourself and stored in a directory referenced by the NF variable.pipelineThe pipeline.py file (found in the GitHub repository) acts as a step-by-step guide to run the code from preparing the data (including annotation) of the English tense project to training an NDL model on this data.Based on code by Adnane Ez-zizi dated 06/08/2020; corrected, updated and adapted as a python package by Tekevwe Kwakpovwe, 08/2021. Changes listed in theStep I: Create sentences file from corpus file with verb tagsFolder variable: WD_EXTRACTMain file: create_sentence_file.py - this is in data_preparation, see pipeline.py as an exampleStep II: Convert verb tags into tenses (tense annotation)Folder variable: WD_ANNOTATEMain file: annotate_tenses.py (quite a long process, use the corresponding run function)A function "Check_lengthy_sentences.R" used to find the 99% quantile of sentence lengths, to define sentence lengths will be added.Removed sentences with num words either < 3 or > 60 (1% and 99% quantiles)Step III: Prepare event filesFolder variable: WD_PREPDATFile 1 (main): prepare_ndl_events.py (use corresponding run function)Added the infinitives using the nltk toolkitRemoved sentences with no verb/tenseFrom the full set, created one row per tense/verb in each sentence. IMPORTANT NOTE: Some modals appear at
the end of a sentence (e.g. certainly they should). In such cases, the VerbForm, MainVerb and infinitive
are all empty, but that doesn't matter since these will not be considered as main eventsConverted words in American English to British English usinghttp://www.tysto.com/uk-us-spelling-list.htmlCorrected the infinitives using Laurence list and some manual corrections +http://www.tysto.com/uk-us-spelling-list.html(for converting AE to BE)From the new full set, removed modals and imperativesAdded a column "VerbOrder" to check the order of the verb (e.g. 1st, 3rd).Created contexts by removing the verb form and replacing it with the infinitive (words seperated with underscores)Generated word cues from the context columns (cues with or without infinitives) - Added 2 columnsGenerated n-gram cues (from 1 to 4) from the context columns (cues with or without infinitives) - Added 2 columnsShuffled the sentences while keeping the order of events within each sentenceFolder variable 2:File 2: Prepare_train_valid_test.py (use corresponding run function to use)
-> Splited the data into training (90%), validation (5%) and test (5%) sets. The split was done
at the level of sentences NOT events.File 3: prepare_ndl_events.py (use corresponding run function to use)
-> This prepare .gz event files ready to be used with NDLNote: I prepared two dataset types: 'multiverbs' contains complex sentences with more than one verb
while 'oneverb' contains only sentences that have a single verb. The "one verb" version is currently not in use.Step IV: Prepare the infinitive cues to useFolder variable: WD_EXTRACT_INFFile 1optional(main): extract_infinitive.py  (use corresponding run function to use)File 2: Extract_infinitive_freqs_oneverb.py (for the dataset containing only sentences with a single verb)-> Created a list of all possible infinitives with their co-occurence frequencies with each tense
-> Extracted the list of all infinitives that have a freq > 10 to be used as lexical cuesStep V: Prepare the n-gram cues to useFolder variable:  WD_EXTRACT_NGRAMFile 2: prepare_ngrams.py (there is also code for the 'oneverb' data set but this is not currently in use)
-> Extracted a list of n-grams to include in the training (10k most frequent from each level n + all have freq>10)Step VI: Prepare the n-gram cues to useFolder variable: WD_CUESFile 1 : prepare_cues.py (There is also code for the 'oneverb' data set but this is not currently in use)
-> Created a separate list of all n-grams for each level n (1 to 4) with their frequency in the datasetStep VII: Prepare the n-gram cues to useFolder variable: WD_SIMFile 1: ndl_model
-> Train an NDL model