Repository: scikit-learn/scikit-learn
Pull request: [MRG] improve tfidfvectorizer documentation
#### Referencing PR / Issue
This closes #12204 
This also closes #6766 and closes #9369
This also closes #12811 (which is including the wrong file in my PR).

#### Note
This adds more information in the TfidfVectorizer documentation. It now includes comments about CountVectorizer and TfidfTransformer.

cc: @blooraspberry
#wimlds 

Problem statement:
TfidfVectorizer documentation doesn't track TfidfTransformer
There are important details of the tf-idf transformation in the TfidfTransformer that are not reflected in the TfidfVectorizer's docstring. I think they should be added there.
Also, I think the formula that is used should be added to the narrative.

Hints from issue discussion:
Actually, I'm not entirely happy with the documentation of TfidfTransformer.
Finding out what the actual formula is it uses by default is non-trivial.

also see #4391, #2998

and neither documents the default value of "norm".

slightly unrelated: the `_document_frequency` function is badly misnamed. It computes document counts, not frequencies.

@amueller I may I take up this task?. As far as I understand, the scope of this task is to update the TfidfVectorizer's docstring. 
(http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html)
a) Default Value of norm
b) Update TfidfVectorizer documentation

Please let me know incase I have missed any details.

@krishnakalyan3 go ahead, but @amueller suggested `TfidfTransformer` needs some more detail, and `TfidfVectorizer` should have much of that too (or at least refer clearly to where the user can read about the parameters). Familiarise yourself with the options and write up as clearly as you can...

@amueller @jnothman please let me know if I have missed anything out.

@amueller 

> slightly unrelated: the `_document_frequency` function is badly misnamed. It computes document counts, not frequencies.

Apart from not being sure what distinction you make between 'count' and 'frequency', "document frequency" is jargon in Information Retrieval, meaning specifically, "the number of documents a particular term appears in".

Shouldn't the default value of analyzer be given in the comments as well?

@HimaVarsha94 yes but that seems to be a different issue.

Commit message: for TfidfTransformer, added parameter type

Patch to review:
diff --git a/sklearn/feature_extraction/text.py b/sklearn/feature_extraction/text.py
index abe63f3650f8f..2807ed5adfaba 100644
--- a/sklearn/feature_extraction/text.py
+++ b/sklearn/feature_extraction/text.py
@@ -1440,6 +1440,28 @@ class TfidfVectorizer(CountVectorizer):
     sublinear_tf : boolean, default=False
         Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
 
+    CountVectorizer : sparse matrix, (n_samples, n_features)
+        Produces a sparse representation of the counts using
+        scipy.sparse.csr_matrix.
+
+    TfidfTransformer : sparse matrix, (n_samples, n_features)
+        Converts the count matrix from CountVectorizer to a normalized tf-idf
+        representation. Tf is term frequency, and idf is inverse document
+        frequency. This is a common way to calculate the count of a word
+        relative to the appearance of a document.
+
+        The formula that is used to compute the tf-idf of term t is​
+        :math:`tf-idf(d, t) = tf(t) * idf(d, t)` and the idf is computed as​:
+        `idf(d, t) = log [ n / df(d, t) ] + 1` (if ``smooth_idf=False``)
+        where `n` is the total number of documents and ​`df(d, t)​`​ is the
+        document frequency; the document frequency is the number of documents
+        `​d​`​ that contain term ​`​t​`​. The effect of adding "1" to the idf
+        in the equation above is that terms with zero idf, i.e., terms that
+        occur in all documents in a training set, will not be entirely ignored.
+        (Note that the idf formula above differs from the standard textbook
+        notation that defines the idf as​
+        `idf(d, t) = log [ n / (df(d, t) + 1) ])`
+
     Attributes
     ----------
     vocabulary_ : dict
@@ -1474,16 +1496,6 @@ class TfidfVectorizer(CountVectorizer):
     >>> print(X.shape)
     (4, 9)
 
-    See also
-    --------
-    CountVectorizer
-        Tokenize the documents and count the occurrences of token and return
-        them as a sparse matrix
-
-    TfidfTransformer
-        Apply Term Frequency Inverse Document Frequency normalization to a
-        sparse matrix of occurrence counts.
-
     Notes
     -----
     The ``stop_words_`` attribute can get large and increase the model size


Code review comments:
diff_hunk: @@ -1440,6 +1440,28 @@ class TfidfVectorizer(CountVectorizer):
     sublinear_tf : boolean, default=False
         Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
 
+    CountVectorizer : sparse matrix, (n_samples, n_features)
path: sklearn/feature_extraction/text.py
text: @user1:
Putting these here would suggest that `CountVectorizer` is an `__init__` parameter to the class, which it isn't. That's why travis is failing.

These information you have here are useful (IMO), but I guess the idea here is to also explain what these classes do by better explaining the parameters already listed in the docstring. For instance, the `norm` parameter was mentioned as badly documented in the issue. As an example, you can have a look at the parameter descriptions of [`LogisticRegression`](https://scikit-learn.org/dev/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression), I hope it helps.

@author:
@user1 
I updated some descriptions.  Is this better?