Package: skimmr_gt
==============================SKIMMR_GT - a Brief Overview==============================-----------------------------------------------------Contact vit.novacek@deri.org for more detailed info-----------------------------------------------------0. ABSTRACT AND TABLE OF CONTENTS=================================This document provides basic information about SKIMMR (a tool formachine-aided skim reading), in particular about its SKIMMR_GT package versionthat focuses on general texts.The document contains three sections:1. ABOUT - overview of the tool and its functionalities2. INSTALLATION - basic instructions on how to install SKIMMR3. USING SKIMMR - basic instruction on how to use it after installation1. ABOUT========SKIMMR is a research prototype aimed at helping users to navigate throughlarge amounts of textual data efficiently. This is done by extending thetraditional paradigm of searching and browsing of text collections. SKIMMRlets the user skim texts by navigating a network of concepts and relationsexplicitly or implicitly present in them. The concepts and their relationshave been extracted and inferred from the textual content using novel machinereading techniques that power the SKIMMR back-end.The interconnected `skimming networks' provide a high-level overview of thedomain covered by the texts, and let the user quickly discover interestingpieces of information. This process also largely reduces the burden ofsieving through lots of irrelevant resources, which is often the down sideof using the standard search engines. When the users identify interestinginformation within the high level overview, they can continue reading therelated textual resources in detail.2. INSTALLATION===============Probably the easiest way of installing SKIMMR is using easy_install:*easy_install skimmr_gt*Check the documentation at http://peak.telecommunity.com/DevCenter/EasyInstallfor more detailed info on easy_install and setuptools.Should you prefer to download and install the package manually, fetch theSKIMMR distribution archive file first. After unpacking it, switch to thegenerated directory and execute the following command:*python setup.py install*If you want to install the package locally (for the current user only), usethe following:*python setup.py install --user*Check the documentation at http://docs.python.org/2/distutils/index.html formore detailed options.3. USING SKIMMR===============After downloading and installing the SKIMMR package, it can be readily usedin a basic manner through the scripts provided. These are:- exst_gt.py - extraction of co-occurrence statements from texts- crkb_gt.py - creation of a knowledge base and its population by semanticsimilarity relations- ixkb_gt.py - indexing of the knowledge base for efficient querying- prep_gt.py - preparation of the sub-folder structure and resources in theworking directory necessary in order to launch the SKIMMR server- srvr_gt.py - launching the SKIMMR server and UIThe scripts are located in the *bin* subdirectory of the installation package.Alternatively, you can copy them from wherever your system puts Python packagebinaries (check the documentation of your local operating system and Pythonimplementation).The typical way of using these scripts is summarised in the following sections.Note that there are other ways how to launch the scripts - you can check thedocumentation in the script source code for details.3.1 Creating the working directory----------------------------------First of all, SKIMMR requires a place to store and process its data. Create adirectory for that somewhere (let us assume it is called *skimmr* in thefollowing). Switch to that directory then and copy all the SKIMMR scriptsthere. After that, run*python prep_gt.py*in there. This will generate two sub-directories, *data* and *text*, as wellas a couple of files and directories deeper in the *data* one. You are all setfor loading the texts you want to process into SKIMMR then.3.2 Processing the texts------------------------Copy the text files you want to process to the *text* folder in the *skimmr*directory. Plain text files (in ASCII or Unicode format) are supported, withthe *.txt* extensions. It is advisable to use meaningful and unique filenamesfor the text files, as they will be used later on for assigning the provenanceidentifiers to the original text data.After you have all the texts in place, run*python exst_gt.py*in the *skimmr* directory. This will chop up the texts into paragraphs andextract the co-occurrence statements from them. There is a limit imposedon the number of produced statements in the exst.py script, dynamicallycomputed from the available memory (or set to 750,000 if the psutil packageis not available on your system). You can change that when using the SKIMMRlibrary functions directly.3.3 Creating the knowledge base-------------------------------After generating the co-occurrence statements in the previous step, you cancreate the knowledge base from them using*python crkb_gt.py create*which will generate a couple of knowledge base persistence files in the *stre*sub-folder of the *data* directory in the *skimmr* root folder.3.4 Computing similarities--------------------------When the knowledge base has been generated, you can augment it by computingsemantic similarity relationships between the terms that are more frequentthan average:*python crkb_gt.py compsim*This will update the knowledge base persistence files accordingly. Note thatthis step may take up to several hours for larger knowledge bases!3.5 Indexing the knowledge base-------------------------------Before you can expose the processed content via a SKIMMR web interface, youhave to index the knowledge base. This is done by running*python ixkb_gt.py*which will generate a couple of index files in the knowledge base persistencesub-directory.3.6 Launching and using the server----------------------------------At last, you can launch the SKIMMR server by*python srvr_gt.py*This will start the server at localhost (127.0.0.1) and port 8008. You canspecify alternative addresses and ports by running the server as*python srvr_gt.py ADDRESS:PORT*Also, you can specify an alternative store to be loaded by the server (usefulif you want to examine multiple stores you have previously generated):*python srvr_gt.py [ADDRESS:PORT] [FOLDER]*where FOLDER is a path to the store you want to load.After the SKIMMR server has been started, you can point you browse to thecorresponding address and port and start using the tool as indicated in the*About* web-page accessible from the SKIMMR interface (just follow the linkin the bottom of every page in the SKIMMR web interface).