Package: textdirectory
TextDirectoryTextDirectoryallows you to combine multiple text files into one aggregated file. TextDirectory also supports matching
files for certain criteria and applying transformations to the aggregated text.TextDirectorycan be used as a mere tool (via the CLI) and as a Python library.Of course, everythingTextDirectorydoes could be achieved in bash or PowerShell. However, there are certain
use cases (e.g., when used as a library) in which it might be useful.Free software: MIT licenseDocumentation:https://textdirectory.readthedocs.io.FeaturesAggregating multiple text filesFiltering documents/texts based on various parameters such as length, content, and random samplingFiltering and transforming text filesTransforming the aggregated text (e.g., transforming the text to lowercase)VersionFiltersTransformations0.1.0filter_by_max_chars(n int); filter_by_min_chars(n int); filter_by_max_tokens(n int); filter_by_min_tokens(n int); filter_by_contains(str); filter_by_not_contains(str); filter_by_random_sampling(n int; replace=False)transformation_lowercase0.1.1filter_by_chars_outliers(n sigmas int)transformation_remove_nl0.1.2filter_by_filename_contains(contains str)transformation_usas_en_semtag; transformation_uppercase; transformation_postag(spacy_model str)0.1.3filter_by_similar_documents(reference_file str; threshold float)transformation_remove_non_ascii; transformation_remove_non_alphanumerical0.2.0filter_by_max_filesize(max_kb int); filter_by_min_filesize(min_kb int)transformation_to_leetspeak; transformation_crude_spellchecker(language model str)0.2.1Nonetransformation_remove_stopwords(stopwords_source str; stopwords str [en]; spacy_model str; custom_stopwords str); transformation_remove_htmltags0.3.0Nonetransformation_remove_weird_tokens(spaCy model; remove_double_space=False); transformation_lemmatizer(spaCy model)0.3.2Nonetransformation_expand_english_contractions0.3.3filter_by_filenames(filenames list); filter_by_filename_not_contains(not_contains str)transformation_eebop4_to_plaintext; transformation_replace_digits(replacement_character str); transformation_ftfyQuickstartInstallTextDirectoryvia pip:pip install textdirectoryTextDirectory, as exemplified below, works with a two-stage model. After loading in your data (directory) you can, by applyingfilters, iteratively select the files you want to process.
In a second step you can perform transformations on the text before finally aggregating it to either a file or memory.As a Command-Line ToolTextDirectorycomes equipped with a CLI.The syntax for both thefiltersandtranformationsworks similarly. They are chained by adding slashes (/) and
parameters are passed via commas (,):filter_by_min_tokens,5/filter_by_random_sampling,2.Example 1: A Very Simple Aggregationtextdirectory--directorytestdata--output_fileaggregated.txtThis will take all files (.txt) intestdataand then aggregates the files into a file calledaggregated.txt.You could also use ‘*’ as a wildcard for filetype if you need to include all files and not just .txt.textdirectory--directorytestdata--output_fileaggregated.txt--filetype*Example 2: Applying Filters and TransformationsIn this example, we want to filter the files based on their token count, perform a random sampling and finally transform all text to lowercase.textdirectory--directorytestdata--output_fileaggregated.txt--filtersfilter_by_min_tokens,5/filter_by_random_sampling,2--transformationstransformation_lowercaseAfter passing two filters (filter_by_min_tokensandfilter_by_random_sampling) we’ve applied thetransform_lowercasetransformation.The resulting file will contain the content of two files that each have at least five tokens.As a Python LibraryIn order to demonstrateTextDirectoryas a Python library, we’ll recreate the second example from above:importtextdirectorytd=textdirectory.TextDirectory(directory='testdata')td.load_files(recursive=False,filetype='txt',sort=True)td.filter_by_min_tokens(5)td.filter_by_random_sampling(2)td.stage_transformation(['transformation_lowercase'])td.aggregate_to_file('aggregated.txt')If we don’t have special requirements, we can also calltd =textdirectory.TextDirectory(directory='testdata',autoload=True)to skip manually callingload_files.
If we wanted to keep working with the actual aggregated text, we could have calledtext = td.aggregate_to_memory()instead ofaggregate_to_file.importtextdirectorytd=textdirectory.TextDirectory(directory='testdata',autoload=True)td.get_text(0)Sometimes we might want to get the actual text of a given file. This can be achieved as seen above. Theget_textmethod will return the transformed text if it is available.
Otherwise, it will simply read the file and return the text.Every applied filter will create astate(i.e., a checkpoint). If we want to go back to a previous state, we can print
all states by callingtd.print_saved_states(). Previous states can then be loaded by
callingtd.load_aggregation_state(state=0).It’s also possible to pass arguments to the individual transformations. In order to do this (at the moment) you have to adhere to the correct order of arguments.# def transformation_remove_stopwords(text, stopwords_source='internal', stopwords='en', spacy_model='en_core_web_sm', custom_stopwords=None, *args)td.stage_transformation(['transformation_remove_stopwords','internal','en','en_core_web_sm','dolor'])In the above example, we are adding additional custom stopwords to the transformer.You also might not always want to aggregate texts into one file in many cases but filter and transform them.importtextdirectorytd=textdirectory.TextDirectory(directory='input')td.load_files()td.filter_by_max_chars(480)td.stage_transformation(['transformation_to_leetspeak'])td.transform_to_files('output')In the example above, we are loading all files ininput. After filtering and transforming, the modified files will be written tooutput.In addition, there are a fewsimple examplesin the repository for you to look at.Special Transformationstransformation_eebop4_to_plaintextThis is a highly specific transformation that will extract the plain text from an EEBO-TCP P4 corpus file.
Both the header as well as all XML tags will be removed during this transformation.transformation_ftfyThis transformation simply appliesftfy.fix_textto the text.
It is highly recommended to useftfyas a first transformation if you are working with messy Unicode text.Notes for DevelopersIf you want to run tests, please usepython setup.py test(ormake test). To build the docs, runmake docs.BehaviorWe are not holding the actual texts in memory. This leads to much more disk read activity (and time inefficiency), but
saves memory. Of course, this is not the case when usingaggregate_to_memory.transformation_usas_en_semtagrelies on the web version ofPaul Rayson’s USAS Tagger. Don’t use this transformation for large amounts of text, give credit, and
consider using their commercial productWmatrix.If you are working with a lot of files, it might be wise to useload_files(fast=True, skip_checkpoint=True).
This will load files much quicker but skip collecting metadata. This will limit the filters that you can use.CreditsThis package is based on theaudreyr/cookiecutter-pypackagecoockiecutter template. Thecrude spellchecker(transformation) is implemented following Peter Norvig’s excellenttutorial.History0.1.0 (2018-04-26)Initial releaseFirst release on PyPI.0.1.1 (2018-04-27)added filter_by_chars_outliersadded transformation_remove_nl0.1.2 (2018-04-29)added transformation_postagadded transformation_usas_en_semtagadded transformation_uppercaseadded filter_by_filename_containsadded parameter support for transformations0.1.3 (2018-04-30)filter_by_random_sampling now has a “replacement” optionchanged from tabulate to an embedded functionadded transformation_remove_non_asciiadded transformation_remove_non_alphanumericaladded filter_by_similar_documents0.1.4 (2018-04-02)fixed an object mutation problem in the tabulate function0.2.0 (2018-05-13)added transform_to_memory() functionadded transformation_to_leetspeak() functionadded transformation_crude_spellcheckeradded filter_by_max_filesizeadded filter_by_min_filesizefixed a bug where load_files() would fail if there were no files0.2.1 (2019-06-13)added transformation_remove_stopwordsadded transformation_remove_htmltagsfixed some minor bugs0.2.2 (2019-06-13)changed the data packaging0.3.0 (2020-01-19)added transformation_remove_weird_tokensadded transformation_lemmatizerfixed some minor bugsadded a function to revert applied filtersadded a function that prints the current pipelineadded a function that clears all transformationsadded helper functions to list available filters and transformationsfixed a bug in whichtabulate_flat_list_of_dictswould fail if the dictionary was emptyself.aggregationdoes not hold a copy of the files anymore but references toself.filestransformations relying on spaCy are now estimating a max_length based on available memoryTextDirectory objects are now iterable0.3.1 (2020-01-20)added long_description_content_type to setup.py0.3.2 (2021-01-10)added transformation_expand_english_contractionsfixed some minor bugsadded __str__ method to TextDirectoryadded filename to __str__ outputaddedget_textmethod0.3.3 (2022-09-25)added filter_by_filenamesadded filter_by_filename_not_containsadded transform_to_filesadded transformation_eebop4_to_plaintextadded transformation_replace_digitsadded transformation_ftfyadded fast and skip_checkpoint options to load_filesadded __repr__ method to TextDirectoryadded examplesupgraded to spaCy 3improved the test suitefixed some minor bugs