Package: lowmemorywordcount
Fast count of the occurrences of words in a text file or a given string - low memory consumptionpip install lowmemorywordcountTested against Windows 10 / Python 3.10 / AnacondaThe count_words function provides a powerful and customizable tool for counting word occurrences in both files and strings, making it valuable for a wide range of professionals dealing with textual dataCustomization:The function allows users to customize word counting by providing several optional parameters. Users can specify the encoding, error handling, chunk size for file reading, inclusion of hyphens in words, inclusion of words containing numbers, file mode, ignoring case sensitivity, and setting minimum and maximum word lengths. This level of customization allows users to tailor the word counting process to their specific requirements.Efficiency:The function reads the input file in chunks, which is memory-efficient for large files. By processing data in chunks, it reduces memory consumption and is suitable for handling large text files without running into memory-related issues.Unicode Support:The function leverages the regex library, which provides excellent support for Unicode characters. This means it can handle words from various languages and character sets, making it suitable for analyzing text data in diverse contexts.Word Frequency Counting:The function utilizes a defaultdict to store word frequencies, which provides a convenient way to count occurrences of words. Users can access the counts directly by using the word as a key without needing to initialize the count for each word manually.Flexibility:The function can work with both file paths and strings as inputs. This flexibility allows users to analyze text from different sources, whether it's from a file on disk or a dynamically generated string.Parameters:file_or_string(str|bytes):Thepathtothetextfileortheinputstring.encoding(str,optional):Theencodingtouseforreadingthefile(defaultis"utf-8").errors(str,optional):Howtohandleencodingerrorswhilereadingthefile(defaultis"ignore").chunk_size(int,optional):Thesizeofthedatachunktoreadfromthefile(defaultis8192bytes).words_with_hyphen(bool,optional):SettoTruetoincludehyphensaspartofwords(defaultisTrue).include_numbers(bool,optional):SettoTruetoincludenumberslike"111",stringslike:"70s"arealwaysincluded(defaultisFalse).mode(str,optional):Thefilemodetoopenthefile(defaultis"r").ignore_case(bool,optional):SettoTruetoignorethecasewhencountingwords(defaultisTrue).min_len(int|None,optional):Theminimumlengthofwordstoinclude(defaultisNone,whichmeansnominimum).max_len(int|None,optional):Themaximumlengthofwordstoinclude(defaultisNone,whichmeansnomaximum).Returns:defaultdict:Adefaultdictwithwordsaskeysandtheiroccurrencesasvalues.Example:fromlowmemorywordcountimportcount_words# Count words in a text filedi=count_words(file_or_string=r"F:\textfile.txt",encoding="utf-8",errors="ignore",chunk_size=8192,words_with_hyphen=False,include_numbers=False,mode="r",ignore_case=True,min_len=None,max_len=None,)fromlowmemorywordcountimportcount_words# Count words in a string or filedi=count_words(file_or_string=b"This is a sample text. It contains some words, including words like 'apple' and 'orange'.",encoding="utf-8",words_with_hyphen=False,include_numbers=False,ignore_case=True,min_len=3,max_len=10,mode='rb')Out[6]:defaultdict(int,{b'this':1,b'sample':1,b'text':1,b'contains':1,b'some':1,b'words':2,b'including':1,b'like':1,b'apple':1,b'and':1,b'orange':1})fromlowmemorywordcountimportcount_wordsdi=count_words(file_or_string="This is a sample text. It contains some words, including words like 'apple' and 'orange'.",encoding="utf-8",words_with_hyphen=False,include_numbers=False,ignore_case=True,min_len=3,max_len=10,mode='r')Out[8]:defaultdict(int,{'this':1,'sample':1,'text':1,'contains':1,'some':1,'words':2,'including':1,'like':1,'apple':1,'and':1,'orange':1})