PySpark: CountVectorizer|HashingTF | by Vanshika Dhamija | Towards Data Science

I came across these terms while matching business entities with their corresponding brand names. However, let me just walk you through them using some more simple examples.
Before diving deep into CountVectorizer and HashingTF, lets first broadly understand what they do.
CountVectorizer and HashingTF estimators are used to generate term frequency vectors. They basically convert documents into a numerical representation which can be fed directly or with further processing into other algorithms like LDA, MinHash for Jaccard Distance, Cosine Distance to name a few.
Suppose we just have 2 documents in out database and we want the convert them into a feature vector.
We can go ahead with Feature Extraction algorithm such as Term Frequency- Inverse Document Frequency (TF-IDF)
It is a method to convert documents into vector such that vector reflects the importance of a term to a document in the corpus.
If we denote a term by t, a document by d, and the corpus by D
Term Frequency (TF(t,d)): Number of times term t appears in document d
Here, TF(PYTHON,Document 1) = 1; TF(HIVE,Document 1) = 2
It must be noted, both HashingTF and CountVectorizer can be used to generate the term frequency vectors
Document Frequency (DF(t,D)): Number of documents that contains term t
Here, DF(PYTHON,Document 1) = 1; DF(HIVE,Document 1) = 1
However, stopwords like a, an , the etc. have high frequency in the corpus but do not carry a special meaning. In order to deal with it we use Inverse Document Frequency (IDF).
IDF(t,D): Numerical measure of how much information a terms provide
Here, IDF(PYTHON,Document 1) = log((2+1)/(1+1)) ~ 0.405
Similarly, IDF(HIVE,Document 1) ~0.405
TF-IDF : Product of TF and IDF.
Here, TF-IDF(PYTHON,Document 1,Corpus) ~ 0.405
Likewise, TF-IDF(HIVE,Document 1, Corpus) ~0.81
TF-IDF therefore ensures that terms with high frequency in the document will have high TF but if a term has high frequency across the corpus then its importance is reduced by IDF. A term present in all documents in the corpus will have TF-IDF equal to 0.
As we have seen that both HashingTF and CountVectorizer can be used to generate the term frequency vector.So, let us now dive deep into each and understand their strengths and shortcomings.
CountVectorizer converts text documents to vectors which give information of token counts.
Lets go ahead with the same corpus having 2 documents discussed earlier. We want to convert the documents into term frequency vector
# Input data: Each row is a bag of words with an IDdf = hiveContext.createDataFrame([    (0, "PYTHON HIVE HIVE".split(" ")),    (1, "JAVA JAVA SQL".split(" "))], ["id", "words"])df.show(truncate = False)
First, CountVectorizer will generate a vocabulary in case an a-priory vocabulary is not available. For instance, in this example CountVectorizer will create a vocabulary of size 4 which includes PYTHON, HIVE, JAVA and SQL terms. It will be followed by fitting of the CountVectorizer Model. During the fitting process, CountVectorizer will select the top VocabSize words ordered by term frequency. The model will produce a sparse vector which can be fed into other algorithms.
# Fit a CountVectorizerModel from the corpusfrom pyspark.ml.feature import CountVectorizercv = CountVectorizer(inputCol="words", outputCol="features")model = cv.fit(df)result = model.transform(df)result.show(truncate=False)
For the purpose of understanding, the feature vector can be divided into 3 parts
The leading number represents the size of the vector. Here, it is 4.
The first list of numbers represent the vector indices.
For instance, ‘JAVA’ term has a higher frequency of 2 as compared to term ‘SQL’ which has a frequency of 1. Therefore, ‘JAVA’ has index 1 whereas ‘SQL’ has index 2
The second list of numbers represents the values corresponding to these indices.
It can be seen in document 2, ‘JAVA’ with index 1 has value 2 and ‘SQL’ with index 2 has value 1
However, it should be noted that since the frequency of ‘HIVE’ and ‘JAVA’ is same, the indices are inter changeable.
Similar, is the case with other terms with equal frequency.
HashingTF converts documents to vectors of fixed size. The default feature dimension is 262,144. The terms are mapped to indices using a Hash Function. The hash function used is MurmurHash 3. The term frequencies are computed with respect to the mapped indices.
# Get term frequency vector through HashingTFfrom pyspark.ml.feature import HashingTFht = HashingTF(inputCol="words", outputCol="features")result = ht.transform(df)result.show(truncate=False)
It can be seen in the above example that the dimension of the vector is set to default i.e. 262,144. Also, term ‘PYTHON’ is mapped to index 134160 by the hashing function and has frequency equal to 1. Similar, insights can be gained with respect to other terms.
We have seen that both CountVectorizer and HashingTF can be implemented to generate a frequency vector. However, both the implementations have their advantages and disadvantages. Let us have a closer look at their differences to select the algorithm which best suits our requirements.-
The size of the vector generated through CountVectorizer depends on the training corpus and the document whereas the one generated through HashingTF has fixed size. The default size is 262,144
In case of CountVectorizer each raw feature is mapped to an index. However, HashingTF suffers from potential hash collision i.e. 2 or more terms may be mapped to same index thereby becoming same after hashing.However, in order to avoid hash collision we can increase the target feature dimension
Due to global term to index mapping CountVectorizer is more computationally expensive as compared to HashingTF which utilizes the hashing trick.
It is difficult to interpret the vector created through HashingTF as hashing is irreversible so the original input cannot be restored. However, the input can be restored in case of CountVectorizer
Clearly there is a trade-off between the 2 algorithms. However, it can be concluded that in situations where one has huge data-set and can go ahead with a bit lower accuracy than HashingTF will be more suitable whereas in cases where one needs reverse mapping CountVectorizer will be a better fit.
Hope it is of some help :)