Package: abstraction
|project abstraction|NOTE====Please note that abstraction is a *project*, not a finished product.setup=====The following Bash commands, that have been tested on Ubuntu 15.10,should install prerequisites and check out abstraction... code:: bash# Install ROOT.sudo apt-get -y install festivalsudo apt-get -y install pylintsudo apt-get -y install snakefoodsudo apt-get -y install sqlitesudo apt-get -y install python-nltksudo python -m nltk.downloader allsudo easy_install -U gensimsudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whlsudo pip install git+git://github.com/google/skflow.gitsudo pip install abstractiongit clone https://github.com/wdbm/abstraction.gitThe function ``abstraction.setup()`` should be run.upcoming========Under consideration is a requirement for arcodex to ensure the existenceof a response to an utterance before saving to database.logging=======Updating logging procedures is under consideration because of possiblelogging conflicts. It could be beneficial currently to run using Bashanonymous pipes, in a way like the following:.. code:: bashpython script.py 2> >(grep -E -v "INFO|DEBUG")data====feature scaling---------------Standardization of datasets is a common requirement for many machinelearning estimators implemented in the scikit; they might behave badlyif the individual features do not more or less look like standardnormally-distributed data: Gaussian with zero mean and unit variance --often called a standard scores. Many machine learning algorithms assumethat all features are centered around zero and have variance of the sameorder. A feature with a variance that is orders of magnitude larger thatothers might dominate the objective function and make the estimatorunable to learn from other features. The scikit function ``scale``provides a quick way to perform this operation on a single array-likedataset.SUSY Data Set--------------  https://archive.ics.uci.edu/ml/datasets/SUSY-  http://arxiv.org/abs/1402.4735The SUSY Data Set is a classification problem to distinguish between asignal process which produces supersymmetric particles and a backgroundprocess which does not. In the data, the first column is the class label(1 for signal, 0 for background), followed by 18 features (8 low-levelfeatures and 10 high-level features):-  lepton 1 pT-  lepton 1 eta-  lepton 1 phi-  lepton 2 pT-  lepton 2 eta-  lepton 2 phi-  missing energy magnitude-  missing energy phi-  MET\_rel-  axial MET-  M\_R-  M\_TR\_2-  R-  MT2-  S\_R-  M\_Delta\_R-  dPhi\_r\_b-  cos(theta\_r1)This data has been produced by MadGraph5 Monte Carlo simulations of 8TeV proton collisions, with showering and hadronisation performed byPythia 6 and detector response simulated by Delphes. The first 8features are kinematic properties measured by simulated particledetectors. The next 10 features are functions of the first 8 features;they are high-level features derived by physicists to help discriminatebetween the two classes. There are 46% positive examples in the SUSYdata set. The features were standardised over the entiretraining/testing sets with mean zero and standard deviation one, exceptfor those features with values strictly greater than zero; these werescaled such that the mean value was one.Caffe=====introduction------------Caffe is a deep learning framework developed by the Berkeley Vision andLearning Center (BVLC) with cleanliness, readability and speed in mind.It has a clean architecture which enables rapid deployment. It isreadable and modifiable, encouraging active development. It is a fastCNN implementation. It has command line, Python and MATLAB interfacesfor day-to-day usage, interfacing with research code and rapidprototyping. While Caffe is essentially a C++ library, it has a modularinterface for development with cmdcaffe, pycaffe and matcaffe.The Caffe core software packages are as follows:-  Caffe-  CUDA-  cuDNN-  OpenBLAS-  OpenCV-  BoostCaffe other dependencies are as follows:-  protobuf-  google-glog-  gflags-  snappy-  leveldb-  lmdb-  hdf5The Caffe build tools are CMake and make.command line------------The command line interface cmdcaffe is a Caffe tool for model training,scoring and diagnostics. Run it without arguments for help. It is atdirectory ``caffe/build/tools``.train~~~~~``caffe train`` learns models from scratch, resumes learning from savedsnapshots and fine-tunes models to new data and tasks. All trainingrequires a solver configuration through the option``-solver solver.prototxt``. Resuming requires the option``snapshot model_item_1000.solverstate`` argument to load the solversnapshot... code:: bash# train LeNetcaffe train -solver examples/mnist/lenet_solver.prototxt# train on GPU 2caffe train -solver examples/mnist/lenet_solver .prototxt -gpu 2test~~~~``caffe test`` scores models by running them in the test phase andresport the network output as its score. The network architecture mustbe defined properly to output an accuracy measure or loss as its output.The per-batch score is reported and then the grand average is reportedlast... code:: bash# score the learned LeNet model on the validation setas defined in the model architecture lenet_train_test.prototxtcaffe test - model examples/mnist/lenet_train_test.prototxt -weights examples/mnist/lenet_iter_10000 -gpu 0 -iterations 100benchmark~~~~~~~~~``caffe time`` benchmarks model execution layer-by-layer through timingand synchronisation. This is useful to check system performance andmeasure relative execution times for models... code:: bash# time LeNet training on CPU for 10 iterationscaffe time -model examples/mnist/lenet_train_test.prototxt -iterations 10# time LeNet training on GPU for the default 50 iterationscaffe time -model examples/mnist/lenet_train_test.prototxt - gpu 0diagnose~~~~~~~~``caffe device_query`` reports GPU details for reference and checkingdevice ordinals for running on a device in multi-GPU machines... code:: bash# query the first devicecaffe device_query -gpu 0pycaffe-------The Python interface ``pycaffe`` is the caffe module and its scripts areat the directory ``caffe/python``. Run ``import caffe`` to load models,do forward and backward, handle IO, visualise networks and instrumentmodel-solving. All model data, derivatives and parameters are exposedfor reading and writing.``caffe.Net`` is the central interface for loading, configuring andrunning models. ``caffe.Classifier`` and ``caffe.Detector`` provideconvenience interfaces for common tasks. ``caffe.SGDSolver`` exposes thesolving interface. ``caffe.io`` handles input and output withpreprocessing and protocol buffers. ``caffe.draw`` visualises networkarchitectures. Caffe blobs are exposed as numpy ndarrays for ease-of-useand efficiency.MATLAB------The MATLAB interface ``matcaffe`` is the Caffe MATLAB MEX file and itshelper m-files are at the directory caffe/matlab. There is example code``caffe/matlab/caffe/matcaffe_demo.m``.models------The directory structure of models is as follows:.. code:: bash.├── bvlc_alexnet│   ├── deploy.prototxt│   ├── readme.md│   ├── solver.prototxt│   └── train_val.prototxt├── bvlc_googlenet│   ├── bvlc_googlenet.caffemodel│   ├── deploy.prototxt│   ├── quick_solver.prototxt│   ├── readme.md│   ├── solver.prototxt│   └── train_val.prototxt├── bvlc_reference_caffenet│   ├── deploy.prototxt│   ├── readme.md│   ├── solver.prototxt│   └── train_val.prototxt├── bvlc_reference_rcnn_ilsvrc13│   ├── deploy.prototxt│   └── readme.md└── finetune_flickr_style├── deploy.prototxt├── readme.md├── solver.prototxt└── train_val.prototxtdraw a graph of network architecture------------------------------------.. code:: bash"${CAFFE}"/python/draw_net.py "${CAFFE}"/models/bvlc_googlenet/deploy.prototxt bvlc_googlenet_deploy.pngsetup-----.. code:: bashsudo apt-get -y install libprotobuf-devsudo apt-get -y install libleveldb-devsudo apt-get -y install libsnappy-devsudo apt-get -y install libopencv-devsudo apt-get -y install libhdf5-devsudo apt-get -y install libhdf5-serial-devsudo apt-get -y install protobuf-compilersudo apt-get -y install --no-install-recommends libboost-all-devsudo apt-get -y install libatlas-base-devsudo apt-get -y install python-devsudo apt-get -y install libgflags-devsudo apt-get -y install libgoogle-glog-devsudo apt-get -y install liblmdb-devsudo apt-get -y install python-pydot.. code:: bashsudo pip install protobufsudo pip install scikit-image.. code:: bashcdgit clone https://github.com/BVLC/caffe.gitcd caffecp Makefile.config.example Makefile.configEdit the makefile. Uncomment ``CPU_ONLY := 1`` for a non-GPU compilation(without CUDA). It may be necessary to include the following lines:::INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial/LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial.. code:: bashtime make alltime make testtime make runtesttime make pycaffe.. code:: bashPYTHONPATH="/home/"${USER}"/caffe/python:${PYTHONPATH}"CAFFE="/home/"${USER}"/caffe"Download Caffe models from the Model Zoo.-  http://caffe.berkeleyvision.org/model_zoo.html-  https://github.com/BVLC/caffe/wiki/Model-Zoo.. code:: bash~/caffe/scripts/download_model_binary.py models/bvlc_googlenetTorch=====setup-----.. code:: bashcurl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-deps | bashgit clone https://github.com/torch/distro.git ~/torch --recursivecd ~/torch; ./install.shCPU versus GPU for deep learning================================Roelof Pieters set some benchmarks in 2015-07 for deep dreaming videoprocessing using CPU and GPU hardware. The CPU hardware was Amazon EC2g2.2xlarge Intel Xeon E5-2670 (Sandy Bridge) 8 cores 2.6 GHz/3.3 GHzturbo and the GPU hardware was Amazon EC2 g2.2xlarge 2 x 4 Gb GPU.+------+------+------+------+------+| **in | **CP | **GP | **CP | **GP || put  | U    | U    | U    | U    || imag | proc | proc | proc | proc || e    | essi | essi | essi | essi || reso | ng   | ng   | ng   | ng   || luti | time | time | time | time || on   | for  | for  | for  | for  || (pix | 1    | 1    | 2    | 2    || els) | imag | imag | minu | minu || **   | e**  | e**  | te   | te   ||      |      |      | vide | vide ||      |      |      | o**  | o**  |+======+======+======+======+======+| 540  | 45 s | 1 s  | 1 d  | 60   || x    |      |      | 21 h | minu || 360  |      |      |      | tes  |+------+------+------+------+------+| 1024 | 144  | 3 s  | 6 d  | 3 h  || x    | s    |      |      |      || 768  |      |      |      |      |+------+------+------+------+------+So, the GPU hardware was ~45 -- ~48 times faster than the CPU hardware.introduction============Project abstraction is a natural language processing project utilisingcurated conversation data as neural network training data.bags of words, skip-grams and word vectors==========================================Word vectors are an efficient implementation of bag-of-words andskip-gram architectures for computing vector representations of words.These representations can be used in natural language processingapplications and research.An n-gram is a contiguous sequence of n items from a sequence of text orspeech. The items can be phonemes, syllabels, letters, words or basepairs depending on the application. Skip-grams are a generalisation ofn-grams in which the components (typically words) need not beconsecutive in the text under consideration, but may have gaps that areskipped. They are one way of overcoming the data sparsity problem foundin conventional n-gram analysis.Formally, an n-gram is a consecutive subsequence of length n of somesequence of tokens w\_n. A k-skip-n-gram is a length-n subsequence inwhich components occur at a distance of at most k from each other. Forexample, in the text::the rain in Spain falls mainly on the plainthe set of 1-skip-2-grams includes all of the 2-grams and, in addition,the following sequences:::the in,rain Spain,in falls,Spain mainly,mainly the,on plainIt has been demonstrated that skip-gram language models can be trainedsuch that it is possible to perform 'word arithmetic'. For example, withan appropriate model, the expression ``king - man + woman`` evaluates tovery close to ``queen``.-  "Efficient Estimation of Word Representations in Vector Space", TomasMikolov, Kai Chen, Greg Corrado, Jeffrey Deanhttp://arxiv.org/abs/1301.3781The bag-of-words model is a simplifying representation used in naturallanguage processing. In this model, a text is represented as a bag(multiset -- a set in which members can appear more than once) of itswords, disregarding grammar and word order but keeping multiplicity. Thebag-of-words model is used commonly in methods of documentclassification, for which the frequency of occurrence of each word isused as a feature for training a classifier.Word vectors are continuous distributed representations of words. Thetool word2vec takes a text corpus as input and produces word vectors asoutput. It constructs a vocabulary from the training text data and thenlearns vector representations of words. A word2vec model is formed bytraining on raw text. It records the context, or usage, of each wordencoded as word vectors. The significance of a word vector is defined asits usefulness as an indicator of certain larger meanings or labels.curated conversation data=========================Curated conversation data sourced from Reddit is used for theconversation analysis and modelling. Specifically, conversationalexchanges on Reddit are recorded. An exchange consists of an utteranceand a response to the utterance, together with associated data, such asreferences and timestamps. A submission to Reddit is considered as anutterance and a comment on the submission is considered as a response tothe utterance. The utterance is assumed to be of good quality and theresponse is assumed to be appropriate to the utterance based on thecrowd-curated quality assessment inherent in Reddit.translation with word vectors=============================In the paper `"Exploiting Similarities among Languages for MachineTranslation" <http://arxiv.org/abs/1309.4168>`__, Tomas Milokovdescribes how after training two monolingual modes, a translation matrixis generated on the most frequently occurring 5000 words. Using thistranslation matrix, the accuracy of the translations was tested on 1000words. A description Milokov gave of the general procedure is asfollows:-  Create matrix ``M`` with dimensionality ``I`` times ``O``, where``I`` is the size of input vectors and ``O`` is the size of theoutput vectors.-  Iterate over the training set several times with decreasing learningrate and update ``M``.-  For each training sample, compute outputs by multiplying the inputvector by ``M``.-  Compute the gradient of the error (target vector - output vector).-  Update the weights in ``M`` (with reference to how the weights areupdated between the hidden layer and the output layer in word2veccode).abstraction code picture========================.. figure:: packages_abstraction.png:alt:module abstraction==================The module abstraction contains functions used generally for projectabstraction. Many of the programs of the project use its functions.arcodex: archive collated exchanges===================================The program arcodex is a data collation and archiving programspecialised to conversational exchanges. It can be used to archive todatabase exchanges on Reddit.The following example accesses 2 utterances from the subreddit"worldnews" with verbosity:.. code:: basharcodex.py --numberOfUtterances 2 --subreddits=worldnews --verboseThe following example accesses 2 utterances from each of the subreddits"changemyview" and "worldnews" with verbosity:.. code:: basharcodex.py --numberOfUtterances 2 --subreddits=changemyview,worldnews --verboseThe following example accesses 30 utterances from all of the listedsubreddits with verbosity:.. code:: basharcodex.py --numberOfUtterances 30 --subreddits=askreddit,changemyview,lgbt,machinelearning,particlephysics,technology,worldnews --verboseThe standard run 2014-10-28T202832Z is as follows:.. code:: basharcodex.py --numberOfUtterances 200 --subreddits=askreddit,changemyview,lgbt,machinelearning,particlephysics,technology,worldnews --verbosevicodex, vicodex\_2: view collated exchanges============================================The program vicodex\_2 (and vicodex) is a viewing program specialised toconversational exchanges. It can be used to access and view a databaseof exchanges.The following example accesses database "database.db" and displays itsexchanges data:.. code:: bashvicodex_2.py --database="database.db"inspect-database: quick printout of database============================================The program inspect-database provides a simple, comprehensive printoutof the contents of a database. Specifically, for every table in thedatabase it prints all of the column contents for every entry... code:: bashinspect-database.py --database="database.db"The program Sqliteman can be used to provide a view of databaseinformation:.. code:: bashsqliteman database.db::SELECT * FROM exchanges;vcodex: word vectors====================The program vcodex converts conversational exchanges in an abstractiondatabase to word vector representations and adds or updates anabstraction database with these vectors... code:: bashvcodex.py --database="database.db" --wordvectormodel=Brown_corpus.wvmThe program vcodex increases the file size of abstraction databaseversion 2015-01-06T172242Z by a factor of ~5.49. On an i7-5500U CPUrunning at 2.40 GHz, the conversion rate is ~25 exchanges per second.reducodex: remove duplicate collated exchanges==============================================The program reducodex inspects an existing database of conversationalexchanges, removes duplicate entries, creates simplified identifiers forentries and then writes a new database of these entries.The following examples access database "database.db", remove duplicateentries, create simplified identifiers for entries and output database"database\_1.db":.. code:: bashreducodex.py --inputdatabase="database.db".. code:: bashreducodex.py --inputdatabase="database.db" --outputdatabase="database_1.db"fix\_database: fix the data structures of database entries==========================================================.. code:: bashfix_database.py --verbose 2> >(grep -E -v "INFO|DEBUG")abstraction development testing===============================.. code:: bash./arcodex.py --numberOfUtterances 10 --subreddits=askreddit,changemyview,lgbt,machinelearning,particlephysics,technology,worldnews --database=2015-10-12T1612Z.db --verbose.. code:: bash./vicodex.py --database=2015-10-12T1612Z.dbsaving models=============Note that the file ``checkpoint`` in the saved model directory containsfull paths... |project abstraction| image:: http://img.youtube.com/vi/v9zJ9noLeok/0.jpg:target: https://www.youtube.com/watch?v=v9zJ9noLeok