Package: pynutshell
NutshellA simple to use yet robust python library containing tools to perform:Text summarizationInformation retrievalFinding similaritiesSentence rankingKeyword extractionand many more in progress...Getting StartedThese instructions will get you a copy of the project and ready for use for your python projects.InstallationQuick AccessDownload from PyPi.orgpipinstallpynutshellDeveloper StyleRequires Python version >=3.6Clone this repository using the command:gitclonehttps://github.com/KrishnanSG/Nutshell.gitcdNutshellThen install the library using the command:python setup.py installNote: The package is distributed aspynutshelldue to unavailability of the name, but the package name isnutshelland request you not to get confused.How does the library work?The library has several components:SummarizersRankersSimilarity AlgorithmsInformation RetrieversKeyword ExtractorsSummarizationA technique of transforming or condensing textual information using natural language processing techniques.Types of summarizationExtractiveThis technique is very much similar tohighlighting important sentencewhile we read a book.The algorithm finds the important sentences in the corpus (NLP term for raw input text) by reducing thesimilaritybetween sentence by removing sentences which are very similar to each other by retaining one among them.Though this method is a powerful it fails to combine 2 or more sentences into a complex sentence, there by not provide optimal result for some cases.AbstractiveThis technique unlike extractive is much more complex and robust in producing summaries. The algorithm used for this technique performssentence clusteringusingSemantic Analysis(finding the meaning of sentence).Sentence RankingText rankers are algorithms similar to web page ranking algorithms used to rank web pages. These rankers find the importance of the sentence in the document and provide ranks to the sentence, thereby providing us with the information of how important the sentence is.Similarity AlgorithmsText similarity algorithms define the similarity between 2 documents (sentences).A few classic algorithms for finding similarity are:Cosine SimilarityEuclidean DistanceNote: word2vec is an important transformation step used to convert words into vectors to easily perform mathematical operations.FeaturesChecklist of features the library currently offers and plans to offer.Keyword ExtractionText TokenizersText cleanersSemantic decoderSummarizationExtractiveAbstractiveText RankersIntermediateAdvancedInformation RetrievalIntermediateAdvancedExamplesSummarizationA simple example on how to use the library and perform extractive text summarization from the given input text(corpus).fromnutshell.algorithms.information_retrievalimportClassicalIRfromnutshell.algorithms.rankingimportTextRankfromnutshell.algorithms.similarityimportBM25Plusfromnutshell.modelimportSummarizerfromnutshell.preprocessing.cleanerimportNLTKCleanerfromnutshell.preprocessing.preprocessorimportTextPreProcessorfromnutshell.preprocessing.tokenizerimportNLTKTokenizerfromnutshell.utilsimportload_corpus,construct_sentences_from_ranking# Examplecorpus=load_corpus('input.txt')print("\n--- Original Text ---\n")print(corpus)preprocessor=TextPreProcessor(NLTKTokenizer(),NLTKCleaner())similarity_algorithm=BM25Plus()ranker=TextRank()ir=ClassicalIR()# Text Summarizationmodel=Summarizer(preprocessor,similarity_algorithm,ranker,ir)summarised_content=model.summarise(corpus,reduction_ratio=0.70,preserve_order=True)print("\n--- Summarized Text ---\n")print(construct_sentences_from_ranking(summarised_content))Keyword ExtractionA simple example on how to use the library and perform keyword extraction from the given input text(corpus).fromnutshell.algorithms.information_retrievalimportClassicalIRfromnutshell.modelimportKeywordExtractorfromnutshell.preprocessing.cleanerimportNLTKCleanerfromnutshell.preprocessing.preprocessorimportTextPreProcessorfromnutshell.preprocessing.tokenizerimportNLTKTokenizerfromnutshell.utilsimportload_corpuscorpus=load_corpus('input.txt')print("\n--- Original Text ---\n")print(corpus)# Text Keyword Extractionpreprocessor=TextPreProcessor(NLTKTokenizer(),NLTKCleaner(skip_stemming=True))keyword_extractor=KeywordExtractor(preprocessor,ClassicalIR())keywords=keyword_extractor.extract_keywords(corpus,count=10,raw=False)print("\n--- Keywords ---\n")print(keywords)ContributionContributions are always welcomed, it would be great to have people use and contribute to this project to help user understand and benefit from library.How to contributeCreate an issue:If you have a new feature in mind, feel free to open an issue and add some short description on what that feature could be.Create a PR: If you have a bug fix, enhancement or new feature addition, create a Pull Request and the maintainers of the repo, would review and merge them.AuthorsKrishnan S G-@KrishnanSGShruthi Abirami-@Shruthi-22