Package: smart-reading
Smart ReadingAboutsmart_readingis a Python module designed for increasing the understanding of various textforms by using natural language processing. It is heavily based on tools available from theNatural Language Toolkit(NLTK), which are used in various applications and provided with extensions.InstallationThe module is available for Python 2.7+, but recommended to run on Python 3+ for a more thorough unicode support and prettier graphs. Install via pip (or any other desired client):$ pip install smart_readingor by downloading the source code onPyPIorGitHuband running the following command in the root folder:$ python setup.py installImporting textsThe basic functionality ofsmart_readingis to import a given textfile into asmart_reading.book.Bookobject, which can be used to perform several developed analyses. Textfiles can be imported via the functionsmart_reading.book.load(filename). This function utilizes the functionality of the moduletextractto extract textual information of almost any given data structure, including .txt, .pdf, .epub and .docx. See itsonline documentationfor more details on the inner workings of this module. When this module is not found on the system, the program continues with a limited functionality, in which only .txt files can be read. This limited functionality is added because it is experienced that the installation oftextractdoes not always succeed. The user that is not able to installtextractbut does want to import other text formats is encouraged to build alternative pipelines to extract text into a .txt file, which in turn can be imported in thesmart_reading.book.loadfunction.Additionally, a given string can be imported as an e-book viasmart_reading.book.fromstring(text).Three sample texts are included with different file structures, callable via the functionsmart_reading.book.sample:>>>importsmart_readingassr>>>sr.book.sample()# or sr.book.sample('txt')Succesfully loaded 'Benn_Ch_II_The_Metaphysicians.txt' as an e-bookTotal n.o. tokens: 10420<smart_reading.book.Book instance at 0x105a546c8>>>>sr.book.sample('pdf')Succesfully loaded 'PhysRev.47.777.pdf' as an e-bookTotal n.o. tokens: 3192<smart_reading.book.Book instance at 0x110c0ebd8>>>>sr.book.sample('epub')# this one takes a while to loadSuccesfully loaded 'Galileo_The_Sidereal_Messenger.epub' as an e-bookTotal n.o. tokens: 40372<smart_reading.book.Book instance at 0x105a46e60>FunctionalityAs mentioned, the given text are imported into asmart_reading.book.Booktype object. The different tools that this object provides are listed below.ConcordanceA concordance is developed as an extension of thenltk.text.Text.concordancefunction that incorporatesexample 3.6of the NLTK manual, such that it not only matches with exact copies of a given word, but also inflections:>>>importsmart_readingassr>>>bk=sr.book.sample()Succesfully loaded 'Benn_Ch_II_The_Metaphysicians.txt' as an e-bookTotal n.o. tokens: 10420>>>bk.concordance('philosopher')Displaying 17 of 17 matches:nce of an independent income enabled the philosopher to live where he liked ; andby our opinion of his metaphysics . As a philosopher Descartes has , to begin witr dazzle ; they could not convince . The philosophers professed to teach truth ;inctly are all true . In his other great philosophical work , the _Meditations_ ,o his postulate of universal doubt , our philosopher argues from this to an imperHere he agrees with another mathematical philosopher , Plato , who says the sameat least one astronomer , who was also a philosopher , declared that the ultimatepersonality of God . SPINOZA . With the philosopher whom I have just named we cosion of 500 florins on Spinoza , but the philosopher would accept no more than 30l . To appreciate the work of the Hebrew philosopher , of the lonely muser , breddivine substance . In fact , the Hebrew philosopher does this , declaring boldlypeppers his pages . Yet , like the Greek philosophers , he is much more modern ,name of his great work that for him the philosophical problem is essentially a p. But he parts company with the English philosopher in his theory of what it meaalone , however , does not make a great philosopher ; character also is requiredrity than any one utterance of any other philosopher ; but that fame is due to thwork . On _à priori_ grounds the German philosopher seems to have an incontroverIn order to deal with inflections, the stemmernltk.PorterStemmeris used by default. Other stemmers can be sent through the keywordstemmerwhen importing a textfile.nltk.text.TextattributeAsmart_reading.book.Bookobject contains an attributeText, which is annltk.text.Texttype object and as such includes all its attributes as developed by NLTK. These include finding collocations, similar words, and creating lexical disperion plots. See theNLTK APIfor its full documentation>>>bk.Text.collocations()fullest extent; infinite Power; material world; Princess Elizabeth;external world; paramount object; supernatural revelation; twoattributes; necessarily exist; Queen Christina; early age; wholeuniverse; final causes; mathematical demonstration; metaphysicalsystem; perfection involves; best possible; many distinct;mathematical method; divine substance>>>bk.Text.dispersion_plot(['Descartes','Malebranche','Spinoza','Leibniz'])Thesmart_reading.statssubmoduleThesmart_readingmodule comes with astatssubmodule, which usesmatplotlib.pyplotto create the several types of graps out of a givensmart_reading.book.Bookobject, callable via the following functions:smart_reading.stats.plot_noun_hist(book, no_nouns = 20, named_entities = True, exceptions = [], **kwargs)Create a histogram of the most common nouns appearing in given text.no_nouns: Number of nouns that will be included in the graph (i.e. number of bars).named_entities: If false, this will exclude named entities like people and places that are recognized by thenltk.chunk.ne_chunkroutine. This option is not supported in versions of Python 2.exceptions: An iterable of nouns that will be excluded from the analysisFurther keyword arguments are passed tomatplotlib.pyplot.fig.>>>bk2=sr.book.sample('pdf')Succesfully loaded 'PhysRev.47.777.pdf' as an e-bookTotal n.o. tokens: 3192>>>sr.stats.plot_noun_hist(bk2)smart_reading.stats.plot_network_graph(book, no_nouns = 10, treshold = 3, exclude_empty = True, named_entities = True, exceptions = [], **kwargs)Create a network graph using the modulenetworkxwhich depicts the relationship between frequently appearing nouns. The nouns appear as nodes, and edges are drawn between nouns if they appear frequently in the same sentences. Atemperature color schemeis used on the edges to depict the frequency in which nouns appear together (red = very often, blue = a few times).no_nouns: Number of nouns that will be included in the graph (i.e. number of nodes). Can become less in the final result ifexclude_emptyis true, see below.treshold: The minimal number of sentences in which two given nouns have to appear in order for an edge to be drawn. Can be used to simplify graphs with a lot of edges.exclude_empty: if True, this will exclude the nodes from the graph that do not have edges. Note that this will reduce the number of nouns depicted, as declared inno_nounsaboveexceptions: an iterable of nouns that will be excluded from the analysis.Further keyword arguments are passed tomatplotlib.pyplot.fig.>>>sr.stats.plot_network_graph(bk2)>>>bk3=sr.book.sample('epub')Succesfully loaded 'Galileo_The_Sidereal_Messenger.epub' as an e-bookTotal n.o. tokens: 40372>>>sr.stats.plot_network_graph(bk3,exceptions=['Galileo'],treshold=10)