Package: natlang
natlang: Natural Language Data Loading Toolsmaster:dev:Data loader/common data structures and other toolsMost of the code are Python2/3 compatible.
For the version of python for specific modules, please check the second line of
each source file.0. UsageInstall using pip will get you the latest tested version ofnatlang.> pip install natlangAlternatively, you can also install from source using the following command:> python setup.py installIf you want to load up a dataset, then just do this:> import natlang as nl
> data = nl.load(filePattern, format=ChoosenFormat)
> # ChoosenFormat here can be an actual imported format or string.
> # Alternatively, you can also pass a loader func in using nl.load(filePatttern, loader=func)For parallel datasets:> import natlang as nl
> data = nl.biload(srcPattern, tgtPattern, srcFormat, tgtFormat)
> # Loader option similar to nl.load also applies here. src stands for source, tgt stands for target.1. FormatAll supported formats are placed undersrc/format.
Currently the following formats are tested:txt: simple text format. Sentences are separated by\n, tokens/words are
separated by whitespace.tree: constituency tree format. Runpython -i format/tree.pyto play
around.semanticFrame: Propbank/Nombank frame loader. Returns bundles of frames
for analysis.AMR: Abstract Meaning Representation. Runpython -i format/AMR.pyto
play around.conll: General CoNLL format loader. Default is CoNLL_U. Runpython -i format/conll.pyto play around.1.1 Recommended FunctionsFor formats supporting being loaded from a file, one should implement aloadfunction in the format file (see 2.1).For formats supporting being exported, each instance of that format should have
anexportmethod that outputs a string.2. Loader2.1 Individual LoaderEach format has its own loader.
It is defined asformat.FORMAT.load.
Theloadfunction has the following interface:def load(file, linesToLoad=sys.maxsize)At test time, theloadfunction would be expected to parse the file
description and read from it.
It will return the firstlinesToLoadentries as a list.For example, if one wishes to use load a file in constituency tree format (see
example intests/sampleTree.txt), one could do the following:from datatool.format import tree
x = tree.load("datatool/tests/sampleTree.txt")2.2 ClassParallelDataLoaderThis class allows one to load parallel corpora (L1, L2) in any format.
You can specify the format for L1 and L2 side separately.from datatool.loader import ParallelDataLoader
loader = ParallelDataLoader(srcFormat='txtOrTree', tgtFormat='txtOrTree')Here,'txtOrTree'is the default value forsrcFormatandtgtFormat.
Note that under theformatfolder, except for data structures for specific
formats, there are also mere loaders and'txtOrTree'is one that can handle
bothtreeandtxt.After initialising the loader, one can just go ahead and run:loader.load(fFile, eFile, linesToLoad)The loader will automatically align the parallel text and output a list of
tuples, each containing a single entry in L1 and L2.
Entries with either L1 or L2 beingNoneor of length 0 will be omitted.3. ExporterUsage:from datatool.exporter import exportToFile, RealtimeExporter3.1 FunctionexportToFileExport atxtformat dataset ortreeformat dataset (not single entry, but
rather a dataset) to file.3.2 ClassRealtimeExporterThe code is pretty self-explanatory.
If the export function of a specific format takes quite a bit of time, this
method is recommended.