GitHub PullRequestReviewCommentEvent at 2014-04-29 00:12:01
Repository: exercises
Actor: bayar87
Payload: {"comment":{"url":"https://api.github.com/repos/dlemphers-rmn/exercises/pulls/comments/12074855","id":12074855,"diff_hunk":"@@ -10,3 +10,88 @@\n 3. How many types of guitars are mentioned across all the deals?\n \n \"\"\"\n+\n+from sklearn.feature_extraction.text import CountVectorizer\n+\n+from sklearn.feature_extraction.text import TfidfTransformer\n+\n+#import a list of stop words\n+from nltk.corpus import stopwords\n+\n+stopWords = stopwords.words('english')\n+\n+\n+#Open the file deals.txt\n+fcorpus = open('deals.txt')\n+linelist = []\n+\n+#the x^{th} element of linelist set is the x^th line in data.txt\n+for x in fcorpus:\n+\tlinelist.append(x)\n+\n+#remove the stop words e.g. at, and, etc.., from our data and extract the features (occurrence frequency of every word) from all the documents\n+#we consider every line (deal) as a document it self \n+vectorizer = CountVectorizer(min_df=1, stop_words = stopWords)\n+\n+X = vectorizer.fit_transform(linelist)\n+#The rows of the matrix X are the documents (deals) and the columns are the features which are the occurrence frequencies of word\n+#More specifically, X(i,j) is the occurrence frequency of the j^{th} word in the i{th} document\n+#Now we sum up over features (columns of X) to find the total occurrence frequency of every j^{th} word in the entire deals.txt file\n+#then we store that in a row vector totFreq\n+totFreq = X.sum(2)\n+\n+#get the index of the most frequent and the least frequent word in the entire deal.txt file\n+maxInd = totFreq.argmax()\n+minInd = totFreq.argmin()\n+\n+#From the indices we can retrieve the feature name, i.e. the word itself\n+featList = vectorizer.get_feature_names()\n+\n+#Now we print the results for the most frequent and least frequent word \n+print featList.pop(maxInd), 'is repeated' ,   totFreq[0,maxInd], 'times'\n+featList = vectorizer.get_feature_names()\n+print featList.pop(minInd), 'is repeated' ,   totFreq[0,minInd], 'times'\n+\n+\n+\n+# Now we try to find the number of bigrams with guitar as the second words\n+#Import the module as usual\n+import nltk\n+from nltk.collocations import *\n+bigram_measures = nltk.collocations.BigramAssocMeasures()\n+\n+from nltk.corpus import PlaintextCorpusReader\n+from nltk.corpus import stopwords\n+\n+#load the stop words as usual\n+stopWords = stopwords.words('english')\n+\n+\n+\n+# We load our entire file and read through it as one set of characters\n+rdr = PlaintextCorpusReader('.','deals.txt')\n+#set two filters for our bigrams\n+#1st filter we need the word guitar to be as the second word in every bigram\n+#2nd filter we need our bigrams to be diffrent from the stop words\n+filt = lambda w1, w2: 'uitar' not in w2\n+filtStp = lambda w1, w2: (w1 or w2 ) in stopwords.words('english')\n+#Extra Filter\n+ignore_list = ['Shop', 'shop', 'Online', 'online', 'Fingerstyle', 'Learn', 'GuitarCenter', 'at', 'on', 'Arts', 'for', '!', 'the', '-', 'Boutique', '4x12', '7', '.', '(', ')', ',', 'from', 'com', 'four', '2x12', '\"', 'Flatpicking', 'Category', 'Destinations', 'HD', 'Flatpick', 'Products', 'Drums', 'Series']\n+filtExtr = filtStp = lambda w1, w2: (w1 or w2 ) in ignore_list\n+## find bigrams\n+finder = BigramCollocationFinder.from_words(rdr.words())\n+\n+finder.apply_ngram_filter(filt)\n+finder.apply_ngram_filter(filtStp)\n+finder.apply_ngram_filter(filtExtr)\n+#compute the number of occurrences of bigrams\n+scored = finder.score_ngrams(bigram_measures.raw_freq)\n+\n+#sort the bigrams by their occurrence frequencies\n+sorted(bigram for bigram, score in scored)\n+\n+unique_Guitar =  list(set(sorted(bigram for bigram, score in scored)))","path":"ml/mywork/task1.py","position":84,"original_position":84,"commit_id":"d010bc520cb161d6016891004452591c095d3f8e","original_commit_id":"d010bc520cb161d6016891004452591c095d3f8e","user":{"login":"bayar87","id":5644629,"avatar_url":"https://avatars.githubusercontent.com/u/5644629?","gravatar_id":"5f312b6df3c943e672d7c5f72a487b6c","url":"https://api.github.com/users/bayar87","html_url":"https://github.com/bayar87","followers_url":"https://api.github.com/users/bayar87/followers","following_url":"https://api.github.com/users/bayar87/following{/other_user}","gists_url":"https://api.github.com/users/bayar87/gists{/gist_id}","starred_url":"https://api.github.com/users/bayar87/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bayar87/subscriptions","organizations_url":"https://api.github.com/users/bayar87/orgs","repos_url":"https://api.github.com/users/bayar87/repos","events_url":"https://api.github.com/users/bayar87/events{/privacy}","received_events_url":"https://api.github.com/users/bayar87/received_events","type":"User","site_admin":false},"body":"I printed here the total number of different guitars in all the deals","created_at":"2014-04-29T00:12:01Z","updated_at":"2014-04-29T00:12:01Z","html_url":"https://github.com/dlemphers-rmn/exercises/pull/24#discussion_r12074855","pull_request_url":"https://api.github.com/repos/dlemphers-rmn/exercises/pulls/24","_links":{"self":{"href":"https://api.github.com/repos/dlemphers-rmn/exercises/pulls/comments/12074855"},"html":{"href":"https://github.com/dlemphers-rmn/exercises/pull/24#discussion_r12074855"},"pull_request":{"href":"https://api.github.com/repos/dlemphers-rmn/exercises/pulls/24"}}}}