CVE-2024-5206 (Python, published 2024-06-06T19:16Z)
Description: value: A sensitive data leakage vulnerability was identified in scikit-learn's TfidfVectorizer, specifically in versions up to and including 1.4.1.post1, which was fixed in version 1.5.0. The vulnerability arises from the unexpected storage of all tokens present in the training data within the `stop_words_` attribute, rather than only storing the subset of tokens required for the TF-IDF technique to function. This behavior leads to the potential leakage of sensitive information, as the `stop_words_` attribute could contain tokens that were meant to be discarded and not stored, such as passwords or keys. The impact of this vulnerability varies based on the nature of the data being processed by the vectorizer.
Severity: nan (CVSS v3 base score: )
Weakness: CWE-921 \u2014 Storage of Sensitive Data in a Mechanism without Access Control
The product stores sensitive information in a file system or device that does not have built-in access control.
Repository: https://github.com/scikit-learn/scikit-learn
Files changed:
doc/whats_new/v1.5.rst
sklearn/feature_extraction/tests/test_text.py
sklearn/feature_extraction/text.py
Commit message: FIX remove the computed stop_words_ attribute of text vectorizer (#28823)



Vulnerable code:
    expected_stop_words = {
        "celeri",
        "tomato",
        "copyright",
        "coke",
        "sparkling",
        "water",
        "the",
    }
    assert vectorizer.stop_words_ == expected_stop_words
    assert len(vect.stop_words_) == 0
    assert "a" in vect.stop_words_
    assert len(vect.stop_words_) == 2
    assert "a" in vect.stop_words_
    assert len(vect.stop_words_) == 2
    assert len(vect.stop_words_) == 0
    assert "c" in vect.stop_words_
    assert len(vect.stop_words_) == 4
    assert "c" in vect.stop_words_
    assert len(vect.stop_words_) == 5
def test_stop_words_removal():
    # Ensure that deleting the stop_words_ attribute doesn't affect transform

    fitted_vectorizers = (
        TfidfVectorizer().fit(JUNK_FOOD_DOCS),
        CountVectorizer(preprocessor=strip_tags).fit(JUNK_FOOD_DOCS),
        CountVectorizer(strip_accents=strip_eacute).fit(JUNK_FOOD_DOCS),
    )

    for vect in fitted_vectorizers:
        vect_transform = vect.transform(JUNK_FOOD_DOCS).toarray()

        vect.stop_words_ = None
        stop_None_transform = vect.transform(JUNK_FOOD_DOCS).toarray()

        delattr(vect, "stop_words_")
        stop_del_transform = vect.transform(JUNK_FOOD_DOCS).toarray()

        assert_array_equal(stop_None_transform, vect_transform)
        assert_array_equal(stop_del_transform, vect_transform)


    stop_words_ : set
        Terms that were ignored because they either:

          - occurred in too many documents (`max_df`)
          - occurred in too few documents (`min_df`)
          - were cut off by feature selection (`max_features`).

        This is only available if no vocabulary was given.

    Notes
    -----
    The ``stop_words_`` attribute can get large and increase the model size
    when pickling. This attribute is provided only for introspection and can
    be safely removed using delattr or set to None before pickling.

        removed_terms = set()
                removed_terms.add(term)
        return X[:, kept_indices], removed_terms
            X, self.stop_words_ = self._limit_features(
    stop_words_ : set
        Terms that were ignored because they either:

          - occurred in too many documents (`max_df`)
          - occurred in too few documents (`min_df`)
          - were cut off by feature selection (`max_features`).

        This is only available if no vocabulary was given.

    Notes
    -----
    The ``stop_words_`` attribute can get large and increase the model size
    when pickling. This attribute is provided only for introspection and can
    be safely removed using delattr or set to None before pickling.


Fixed code:
Security
--------

- |Fix| :class:`feature_extraction.text.CountVectorizer` and
  :class:`feature_extraction.text.TfidfVectorizer` no longer store discarded
  tokens from the training set in their `stop_words_` attribute. This attribute
  would hold too frequent (above `max_df`) but also too rare tokens (below
  `min_df`). This fixes a potential security issue (data leak) if the discarded
  rare tokens hold sensitive information from the training set without the
  model developer's knowledge.

  Note: users of those classes are encouraged to either retrain their pipelines
  with the new scikit-learn version or to manually clear the `stop_words_`
  attribute from previously trained instances of those transformers. This
  attribute was designed only for model inspection purposes and has no impact
  on the behavior of the transformers.
  :pr:`28823` by :user:`Olivier Grisel <ogrisel>`.

        return X[:, kept_indices]
            X = self._limit_features(