Sklearn | Feature Extraction with TF-IDF - GeeksforGeeks

15 Oct, 2019
Now, you are searching for tf-idf, then you may familiar with feature extraction and what it is. TF-IDF which stands for Term Frequency – Inverse Document Frequency. It is one of the most important techniques used for information retrieval to represent how important a specific word or phrase is to a given document. Let’s take an example, we have a string or Bag of Words (BOW) and we have to extract information from it, then we can use this approach.
The tf-idf value increases in proportion to the number of times a word appears in the document but is often offset by the frequency of the word in the corpus, which helps to adjust with respect to the fact that some words appear more frequently in general.TF-IDF use two statistical methods, first is Term Frequency and the other is Inverse Document Frequency. Term frequency refers to the total number of times a given term t appears in the document doc against (per) the total number of all words in the document and The inverse document frequency measure of how much information the word provides. It measures the weight of a given word in the entire document. IDF show how common or rare a given word is across all documents.TF-IDF can be computed as tf * idf
Tf*Idf do not convert directly raw data into useful features. Firstly, it converts raw strings or dataset into vectors and each word has its own vector. Then we’ll use a particular technique for retrieving the feature like Cosine Similarity which works on vectors, etc. As we know, we can’t directly pass the string to our model. So, tf*idf provides numeric values of the entire document for us.To extract features from a document of words, we import –
from sklearn.feature_extraction.text import TfidfVectorizer
Input :
1st Sentence - "hello i am pulkit"
2nd Sentence - "your name is akshit"

Code : Python code to find the similarity measures
# importing librariesfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine_similarityfrom sklearn.metrics import pairwise_distancesfrom sklearn.metrics.pairwise import euclidean_distancesfrom scipy.spatial import distanceimport pandas as pdimport numpy as np   ## Converting 3D array of array into 1D arraydef arr_convert_1d(arr):    arr = np.array(arr)    arr = np.concatenate( arr, axis=0 )    arr = np.concatenate( arr, axis=0 )    return arr   ## Cosine Similaritycos = []def cosine(trans):    cos.append(cosine_similarity(trans[0], trans[1]))   ## Manhatten Distancemanhatten = []def manhatten_distance(trans):    manhatten.append(pairwise_distances(trans[0], trans[1],                                         metric = 'manhattan'))   ## Euclidean Distanceeuclidean = []def euclidean_function(vectors):    euc=euclidean_distances(vectors[0], vectors[1])    euclidean.append(euc)   # This Function finds the similarity between two # sentences by using above functions.  ## TF - IDFdef tfidf(str1, str2):    ques = []    # You have to provide the dataset. Link of the dataset     # is given in the end of this article.     # and if you are using a different dataset then adjust     # it according to your dataset's columns and rows    dataset = pd.read_csv('C:\\Users\\dell\\Desktop\\quora_duplicate_questions.tsv',                           delimiter='\t',encoding='utf-8')          x = dataset.iloc[:, 1:5]    x = x.dropna(how = 'any')          for k in range(len(x)):        for j in [2, 3]:            ques.append(x.iloc[k, j])    vect = TfidfVectorizer()    # Fit the your whole dataset. After all, this'll     # produce the vectors which is based on words in corpus/dataset    vect.fit(ques)       corpus = [str1,str2]    trans = vect.transform(corpus)       euclidean_function(trans)    cosine(trans)    manhatten_distance(trans)    return convert()   def convert():    dataf = pd.DataFrame()    lis2 = arr_convert_1d(manhatten)    dataf['manhatten'] = lis2    lis2 = arr_convert_1d(cos)    dataf['cos_sim'] = lis2    lis2 = arr_convert_1d(euclidean)    dataf['euclidean'] = lis2    return dataf   newData = pd.DataFrame(); str1 = "hello i am pulkit"str2 = "your name is akshit"newData = tfidf(str1,str2);print(newData);
Output :

   manhatten  cos_sim  euclidean
0   2.955813      0.0   1.414214         
 
Dataset: Google Drive linkNote: Dataset is large so it’ll take 30-40 second to produce output and If you are going to run as it is, then it’s not gonna work. It only works when you copy this code in your IDE and provide your dataset in tfidf function.
nidhi_biet
Machine Learning
Python
Python Programs
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
ML | Linear Regression
Search Algorithms in AI
Python | Decision tree implementation
Elbow Method for optimal value of k in KMeans
Support Vector Machine Algorithm
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe