Package: madhatter
Mad HatterA text analysis package for authors and data scientists alike.Mad Hatter is a Python package that provides a variety of text analysis tools. It is designed to be used by both authors and data scientists alike. It is currently in development and is not yet ready for use. The package provides the following features for text analysis:Simple features (word count, sentence count, average tokens per sentence, average word length, average sentence length, etc.)Advanced psycholinguistically-motivated measures (concreteness, imageability, rare word usage, etc.)Context-dependent LLM-based features (surprisal, predictability, etc.)... All optimized for data analysis, graphing, and easy integration with existing tools!InstallationRun the following command to install the package and its dependencies:pipinstallmadhatterUsing NLTK featuresWe highly recommend also running NLTK's downloader module in order to have access to all of the features that Mad Hatter provides. To do so, simply run the following command:python-mnltk.downloaderallUsageThe package provides high-level abstractions for text analysis that can be used with any text. The following example shows how to use the package to analyze a simple text file:frommadhatter.benchmarkimportCreativityBenchmarktext="The quick brown fox jumped over the lazy dog."bench=CreativityBenchmark(text)bench.report()>>>BookReport(title='unknown',nwords=10,mean_wl=3.7,mean_sl=45.0,mean_tokenspersent=10.0,prop_contentwords=0.1,mean_conc=4.0633333333333335,mean_img=5.359999999999999,mean_freq=-1.6792249660842167,prop_pos={'ADJ':0.2,'NOUN':0.3,'VERB':0.1},surprisal=None,predictability=None)Command Line InterfaceMad Hatter is also available as a CLI tool. Simply provide a filename to the CLI and it will generate a report for you. The following example shows how to use the CLI to generate a report for a text file:>python-mmadhatter-h
usage:madhatter[-h][-p][-u][-mMAXTOKENS][-cCONTEXT][-tTITLE][-dTAGSET]filename

Acommand-lineutilityforgeneratingbookprojectreports.

positionalarguments:filenametextfiletoparse

options:-h,--helpshowthishelpmessageandexit-p,--postagwhethertoreturnaPOStagdistributionoverthewholetext-u,--usellmwhethertorunGPU-intensiveLLMsforadditionalcharacteristics-mMAXTOKENS,--maxtokensMAXTOKENSmaximumnumberofpredictedtokensfortheheavyweightmetrics.Tokensstartfromthebeginningoftext,-1toreaduntiltheend-cCONTEXT,--contextCONTEXTcontextlengthforslidingwindowpredictionsaspartofheavyweightmetrics-tTITLE,--titleTITLEoptionaltitletouseforthereportproject.-dTAGSET,--tagsetTAGSETtagsettouseAdvanced UsageYou may also choose to use the package's lower-level functions to create your own custom analysis pipeline or integrate with NLP packages such asSpaCy.frommadhatterimportmetricsfrommadhatterimportbenchmarktext="The quick brown fox jumped over the lazy dog."bench=benchmark.CreativityBenchmark(text)bench.words>>>['The','quick','brown','fox','jumped','over','the','lazy','dog','.']metrics.imageability(bench.words)>>>[1.41,2.45,3.14,4.2,3.4,3.65,1.41,2.42,4.1,0.0]Of course, feel free to also contribute to the package's development by opening an issue or submitting a pull request!LicenseThe project is released under the MIT license. SeeLICENSEfor more information.