Package: textsum
textsuma utility for using transformers summarization models on text docs 🖇This package provides easy-to-use interfaces for using summarization models on text documents of arbitrary length. Currently implemented interfaces include a python API, CLI, and a shareable demo app.For details, explanations, and docs, see thewikitextsum🔦 Quick Start GuideInstallationFull InstallationExtra FeaturesUsagePython APICLIDemo AppModelsAdvanced ConfigurationParameters8-bit Quantization & TensorFloat32Using Optimum ONNX RuntimeForce CacheCompile ModelContributingRoad Map🔦 Quick Start GuideInstall the package with pip:pipinstalltextsumImport the package and create a summarizer:fromtextsum.summarizeimportSummarizersummarizer=Summarizer()# loads default model and parametersSummarize a text string:text="This is a long string of text that will be summarized."summary=summarizer.summarize_string(text)print(f'Summary:{summary}')InstallationInstall using pip with Python 3.8 or later (after creating a virtual environment):pipinstalltextsumThetextsumpackage is now installed in your virtual environment.CLI commandsare available in your terminal, and thepython APIis available in your python environment.Full InstallationFor a full installation, which includes additional features such as PDF OCR, Gradio UI demo, and Optimum, run the following commands:gitclonehttps://github.com/pszemraj/textsum.gitcdtextsum# create a virtual environment (optional)pipinstall-e.[all]Extra FeaturesThe package also supports a number of optional extra features, which can be installed as follows:8bit: Install withpip install -e .[8bit]optimum: Install withpip install -e .[optimum]PDF: Install withpip install -e .[PDF]app: Install withpip install -e .[app]unidecode: Install withpip install -e .[unidecode]Read below for more details on how to use these features.Note:Theunidecodeextra is a GPL-licensed dependency that is not included by default with theclean-textpython package. While it can be used for text cleaning pre-summarization, it generally should not make a significant difference in most use cases.UsageThere are three ways to use this package:python APICLIDemo AppPython APITo use the python API, import theSummarizerclass and instantiate it. This will load the default model and parameters.You can then use thesummarize_stringmethod to summarize a long text string.fromtextsum.summarizeimportSummarizersummarizer=Summarizer()# loads default model and parameters# summarize a long stringout_str=summarizer.summarize_string('This is a long string of text that will be summarized.')print(f'summary:{out_str}')you can also directly summarize a file:out_path=summarizer.summarize_file('/path/to/file.txt')print(f'summary saved to{out_path}')CLITo summarize a directory of text files, run the following command:textsum-dir/path/to/dirA full list:Click to expand tableFlagDescription--output_dirSpecify the output directory--modelSpecify the model to use--no_cudaDisable CUDA--tf32Use TF32 precision--force_cacheForce cache usage--load_in_8bitLoad in 8-bit mode--compileCompile the model--optimum_onnxUse optimum ONNX--batch_lengthSpecify the batch length--batch_strideSpecify the batch stride--num_beamsSpecify the number of beams--length_penaltySpecify the length penalty--repetition_penaltySpecify the repetition penalty--max_length_ratioSpecify the maximum length ratio--min_lengthSpecify the minimum length--encoder_no_repeat_ngram_sizeSpecify the encoder no repeat ngram size--no_repeat_ngram_sizeSpecify the no repeat ngram size--early_stoppingEnable early stopping--shuffleShuffle the input data--lowercaseConvert input to lowercase--loglevelSpecify the log level--logfileSpecify the log file--file_extensionSpecify the file extension--skip_completedSkip completed filesSome useful options are:Arguments:input_dir: The directory containing the input text files to be summarized.--model: model name or path to use for summarization. (Optional)--shuffle: Shuffle the input files before processing. (Optional)--skip_completed: Skip already completed files in the output directory. (Optional)--batch_length: The maximum length of each input batch. Default is 4096. (Optional)--output_dir: The directory to write the summarized output files. Default is./summarized/. (Optional)For more information, run the following:textsum-dir--helpDemo AppFor convenience, a UI demo[^1] is provided usinggradio. To ensure you have the dependencies installed, clone the repo and run the following command:pipinstalltextsum[app]To run the demo, run the following command:textsum-uiThis will start a local server that you can access in your browser & a shareable link will be printed to the console.[^1]: The demo is minimal but will be expanded to accept other arguments and options.ModelsSummarization is a memory-intensive task, and thedefault model is relatively small and efficientfor long-form text summarization. If you want to use a bigger model, you can specify themodel_name_or_pathargument when instantiating theSummarizerclass.summarizer=Summarizer(model_name_or_path='pszemraj/long-t5-tglobal-xl-16384-book-summary')You can also use the-margument when using the CLI:textsum-dir/path/to/dir-mpszemraj/long-t5-tglobal-xl-16384-book-summaryAnytext-to-textorsummarizationmodel from theHuggingFace model hubcan be used. Models are automatically downloaded and cached in~/.cache/huggingface/hub.Advanced ConfigurationParametersMemory usage can also be reduced by adjusting the parameters for inference. This is discussed in detail in theproject wiki.tl;dr for this README: use thesummarizer.set_inference_params()andsummarizer.get_inference_params()methods to adjust the parameters for inference from either a pythondictor a JSON file.Support forGenerationConfigas the primary method to adjust inference parameters is planned for a future release.8-bit Quantization & TensorFloat32Some methods of reducing memory usageif you have compatible hardwareinclude loading the model in 8-bit precision viaLLM.int8and using the--tf32flag to use TensorFloat32 precision. See thetransformers docsfor more details on how this works. Using LLM.int8 requires thebitsandbytespackage, which can either be installed directly or via thetextsum[8bit]extra:pipinstalltextsum[8bit]To use these options, use the-8bitand--tf32flags when using the CLI:textsum-dir/path/to/dir-8bit--tf32Or in python, using theload_in_8bitargument:summarizer=Summarizer(load_in_8bit=True)If using the python API, it's better to initiate tf32 yourself; seeherefor how.Using Optimum ONNX Runtime⚠️Note:This feature is experimental and might not work as expected. Use at your own risk. ⚠️🧪ONNX Runtime is a performance-focused inference engine for ONNX models. It can be used to enhance the speed of model predictions, especially on Windows and in environments where GPU acceleration is not available. If you want to use ONNX runtime for inference, you need to setoptimum_onnx=Truewhen initializing theSummarizerclass.First, install withpip install textsum[optimum]. Then, you can use the following code to initialize theSummarizerclass with ONNX runtime:summarizer=Summarizer(optimum_onnx=True)Notes:ONNX runtime+cuda needs an additional package. Manually installonnxruntime-gpuif you plan to use ONNX with GPU.Using ONNX runtime might lead to different behavior in certain models. It is recommended to test the model with and without ONNX runtimethe same input textbefore using it for anything important.Force CacheBy default, the summarization model uses past computations to speed up decoding. If you want to force the model to always use cache irrespective of the model's default behavior, you can setforce_cache=Truewhen initializing theSummarizerclass.summarizer=Summarizer(force_cache=True)Note:Settingforce_cache=Truemight lead to different behavior in certain models.Compile ModelBy default, the model isn't compiled for efficient inference. If you want to compile the model for faster inference times, you can setcompile_model=Truewhen initializing theSummarizerclass.summarizer=Summarizer(compile_model=True)Note:Compiling the model might not be supported on all platforms and requires pytorch > 2.0.0.ContributingContributions are welcome! Please open an issue or PR if you have any ideas or suggestions.See theCONTRIBUTING.mdfile for details on how to contribute.Road Mapadd CLI for summarization of all text files in a directorypython API for summarization of text docsadd argparse CLI for UI demoput on PyPILLM.int8 inferenceoptimum inference integrationbetter documentationin the wiki, details on improving performance (speed, quality, memory usage, etc.)in-progressimprovements to the PDF OCR helper module (TBD - may focus more on being a summarization tool)Other ideas? Open an issue or PR!