GitHub PushEvent at 2014-07-13 08:27:34
Repository: yelp_review
Actor: vidhoonv
Payload: {"shas":[["51c106fbb72c1fc536d1976b93e1b8cfe9584b11","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Preprocess all reviews\n\n*Changed preprocessing script to fetch all restaurant\nreviews in the dataset.\n*Also, few reviews were repeated for some reason in the dataset.\nMade changes to remove duplicates.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["b011aadd6a3df565277521f153b209b1e96281c3","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Removed input output dir\n\nNo more i/o directories for seperate components.\nSeperate data folder should be used for all inputs\nand outputs.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["b8338ea3cf0f971ba545f587180efc72243d29f7","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Updated basic features extraction\n\n* Basic code org plus fixes\n* Some reviews' text field were empty - changes made to handle that\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["1141181ad73ff7e6ecea860264c1e5a32c5436cf","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","PCFG - script to generate seed data\n\nThis script takes preprocessed training data\nand extracts review text - useful and not useful reviews\nseperately based on arguments supplied. The output generated\nby this script can be fed to parser for producing parses\nof the review text. These parses are then used to train\nseperate parsers for useful and not useful reviews.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["d1d9aab4b5f863c8e58fa29f5361ff2e2660d7ca","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Script to extract PCFG features\n\nTakes two TSV files from the output of useful and not useful\nreview parsers and then extracts PCFG features:\n* diff between min,max,avg,sdev of probabilities assigned to\nsentences of review text for each instance of training data\n\nThe output from parsers contain raw parse probabilities for\neach sentence of review text.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["9d57e49f7b862d072600cc888f7a01e0fd184d1f","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","NGRAM Feature extraction\n\nAdded two scripts for extracting character and word ngram features.\nHas some compatability issue since python has some issues with smoothing.\nThis is tested in:\n\n>>> nltk.__version__\n'2.0.4'\n\n* While testing few word ngrams and character ngrams had probabilities\ngreater than 1 (which is the smoothing prob)\n* As an adhoc fix to unblock, this code ignores such ngrams. They accounted for\n13-14% of the total (repetitions included) ngrams in the data. This analysis code can be found at the bottom of the files (commented).\n* Word ngram script works for N-value 2 and 3 only ie., bi and trigram features.\n\nThe script creates ngram model and also generates ngram features for training\ndata simulataneously in a single step. So, if models need to be stored seperately, the code needs to be changed to pickle the model and reuse.\n\nFor arguments, please check comments inside the script.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["f39560ca5a1be16959b3a331affb55d1dd9ad955","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Sentiment Feature extraction\n\nAdded the tool used for sentiment feature extraction.\nIt is JAVA code taken from sentiwordnet website and modified to\nprovide aggregate polarity-less sentiment score for each review.\n\nPlease refer README file for complete details about the tool and\nthe feature.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["e0ee883c9b45a5ec1900ae9db219b25573a54f9d","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","COMBINE FEATURES\n\nA script to combine all features generated.\nCAUTION:\n* Very volatile. Should double check always when\n  changes are made at different feature extraction components.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["a7b07f6b4e8c23005eb85e0d04b0872f581cd20a","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Updated split-data script\n\nUsed for spliting yelp review  data randomly into training\nand test splits. The processed csv file should be supplied\nas input.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["a0e6632e2c4765fe98cece9211bf296c88210445","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","MISC tools added\n\nAdding some miscellaneous tools:\n\n* split_file.sh - can be used to split training data into chunks\n                  mainly for generating pcfg features\n* csv2tsv.py   - a convertor which is useful because I operate on CSV/TSV\n\t\tfiles in python but TSV only in JAVA.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["a0f6a314b72a5709df7a04a35d9c3c7bb4b0bfa2","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Adding data location\n\nSee README\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["e4554527c9921a104ff9cfd507807f5fa74af9c0","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Added RUN directory\n\nContains some scripts for using in local UT condor system.\nAlso a shell wrapper for supplying to condor as executable.\n\nCredits: The condorizer.py script was shared by Yinon.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true],["0142aefec39b7fb5bd9a3c0b5e1f30555cc5bc61","5982ee7666bed377fb595f7647fc2ccbdbcad0cb@cs.utexas.edu","Yelp Parser added\n\nAdded the wrapper to train and parse data using Stanford parser.\nRefer README for complete details.\n\nSigned-off-by: Vidhoon Viswanathan <vidhoon@cs.utexas.edu>","Vidhoon Viswanathan",true]],"size":13,"ref":"refs/heads/master","head":"0142aefec39b7fb5bd9a3c0b5e1f30555cc5bc61"}