Repository: cleanlab/cleanlab

Issue:
color_sentence matches subtokens 
<!-- Briefly summarize the issue. -->

`color_sentence`  technically does subtoken matching instead of checking for equality between the tokenized sentence and the given token.

Is this intended?

# Stack trace

```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: ('\x1b[31mI\x1b[0mf you and I were to meet', 'If you and \x1b[31mI\x1b[0m were to meet')
```

# Steps to reproduce

Here's a MWE

```python
from cleanlab.internal.token_classification_utils import color_sentence

sentence = "If you and I were to meet"
word = "I"
output = color_sentence(sentence, word)
expected_output = "If you and \x1b[31mI\x1b[0m were to meet"

assert output == expected_output, (output, expected_output)
```

<!-- Be as detailed as possible here. If possible, include a self-contained
runnable example that demonstrates the issue. Remember to supply any data
necessary to run your example, or construct your example with synthetic data.
This is not strictly required, but the more detailed your bug report, the more
quickly we can help you and fix the bug. -->

# Additional information

- **Cleanlab version**: 31d43707014cb5c217ee1e9b014a206d554f2fd3
- **Operating system**: Linux 4.19.128-microsoft-standard
- **Python version**: Python 3.9.12

<!-- Please include any other information that could be helpful for debugging. -->


## Definition of `color_sentence`

https://github.com/cleanlab/cleanlab/blob/31d43707014cb5c217ee1e9b014a206d554f2fd3/cleanlab/internal/token_classification_utils.py#L144-L164
Give better error messages when a class has no examples in `find_label_issues`
Currently, we produce various internal errors when a class is not represented. We should either support missing classes, or we should produce something like a `ValueError("find_label_issues requires that each class has at least one example")`.

Right now, we produce a variety of internal errors (data-dependent).

This can be seen as a dupe of #41 and #89, though it suggests at least fixing the confusing error messages as a stop-gap solution until we decide whether or not to support classes with no examples. Potentially related to #266.

# Steps to reproduce

## Example 1

```python
import cleanlab
import numpy as np

labels = np.array([0, 0, 1, 1])
pred_probs = np.array([[0.1, 0.7, 0.2], [0.1, 0.8, 0.1], [0.7, 0.2, 0.1], [0.8, 0.1, 0.1]])
issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
```

Produces the result:

```
Traceback (most recent call last):
  File ".../error.py", line 6, in <module>
    issues = cleanlab.filter.find_label_issues(labels, pred_probs)
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 457, in find_label_issues
    confident_joint, cl_error_indices = compute_confident_joint(
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/count.py", line 435, in compute_confident_joint
    pred_probs_bool = pred_probs >= thresholds - 1e-6
ValueError: operands could not be broadcast together with shapes (4,3) (2,)
```

## Example 2

```python
import cleanlab
import numpy as np

labels = np.array([0, 0])
pred_probs = np.array([[0.3, 0.7], [0.2, 0.8]])
issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
```

Produces the result:

```
Traceback (most recent call last):
  File ".../error.py", line 6, in <module>
    issues = cleanlab.filter.find_label_issues(labels, pred_probs, n_jobs=1)
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 561, in find_label_issues
    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 561, in <listcomp>
    label_issues_masks_per_class = [_prune_by_count(k, args) for k in range(K)]
  File ".../src/cleanlab/env/lib/python3.9/site-packages/cleanlab/filter.py", line 204, in _prune_by_count
    if label_counts[k] <= min_examples_per_class:  # No prune if not at least min_examples_per_class
IndexError: index 1 is out of bounds for axis 0 with size 1
```

# Additional information

- **Cleanlab version**: 2.0.0 (and also reproduced on master = 5e0b62d79790028c6c69a76c6af28c813a27ace5, though the stack traces and exceptions are slightly different there)
- **Operating system**: macOS 12.1, shouldn't be relevant here
- **Python version**: Python 3.9.12



Discussion:
Any thoughts on this @ericwang1997?
The function currently searches for the first occurrence of the substring in the sentence, so it will be problematic if we're searching for tokens as generic as "I". There are two ways to get around this: 

1. Add an optional argument to take in the given tokens. Using your example, the user will call: 
```
tokens = ["If", "you", "and", "I", "were", "to", "meet"]
color_sentence(sentence, word, tokens=tokens) 
```
or simply generate the sentence internally using `get_sentence` and call: 
``` 
color_sentece(word, tokens) 
``` 
2. Tokenize the sentence and search for the keyword token. Or simply search for `" " + word " "`. The first option is more ideal imo. 
Note that it's also possible that the keyword is not a given token (for example, if `word == "If you and I"`). The code below covers that case 

```
from cleanlab.internal.token_classification_utils import get_sentence 

def color_sentence(word, tokens): 
    sentence = get_sentence(tokens) 

    idx = -1 
    for i, token in enumerate(tokens): 
        if token == word: 
            idx = i 
            break 

    if idx == -1: # not found 
        start_idx = sentence.index(word) 
        before, after = sentence[:start_idx], sentence[start_idx + len(word) :] 

    else: 
        before, after = get_sentence(tokens[:idx]), get_sentence(tokens[idx+1:]) 

    return "%s%s%s" % (before, colored(word, "red"), after) 
``` 
Thanks for those suggestions.

I agree that providing tokenized text instead to the function as it's only called by `display_issues` which has the tokenized data at hand.


> The code below covers that case

Thanks! I think it's best I'd redefine `color_sentence` this way as you suggest, add the MWE as a test case and update the function call in `display_issues` accordingly!
^ Sounds like good plan to me!  Also if the word occurs multiple times in token-list, should `color_sentence()` color all instances of the word?  Or why is coloring only the first instance more desirable?
Good call. The keyword typically appears only once in the sentence as long as it's specific enough, but highlighting all occurrences makes more sense (it might be easier to extend this functionality using the new version of `color_sentence()` above) 
@anishathalye do we just need to raise error `ValueError("find_label_issues requires that each class has at least one example")` for these two cases?
I am not sure what you mean by "these two cases."

We already raise warnings in two places [[1](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L165), [2](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L206)] when a class has too few examples (under min examples per class). We could raise an exception there if a class has zero examples. Or maybe it's cleaner to do it in another spot; that needs to be investigated. Also need to make sure that the exception propagates properly under both `n_jobs=1` (no multiprocessing) and `n_jobs>1` (multiprocessing).
Hey, thanks for explaining the requirement, I have tried to debug the code, as I can see we use [label_counts](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L451), which is Number of examples in each class of labels, [K](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L453) which is Number of classes labels and `min_examples_per_class` which is by default 1.

`label_counts` is value_count which for first example is [2, 2], now on [line](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L565) we pass `k` where it is `range(K)` means [0,1,2,3], we don't have element on index 2 and 3 for `label_count`. so when we do this [label_counts[k]](https://github.com/cleanlab/cleanlab/blob/master/cleanlab/filter.py#L204), it will throw exception.

Now we can add this condition, where `if len(label_counts) < k: raise excpetion` in `find_label_issues`. 

I am not sure that it is correct but that will throw warning for given example.

@anishathalye once you confirm, I can push a PR for this.
Thanks for volunteering to contribute @sidshrivastav 
we could definitely use your help improving this library!!

For this specific issue though, I'm already handling it as part of another PR as it is very simple.
@jwmueller thanks for updating
Addressed in: https://github.com/cleanlab/cleanlab/pull/308

Resolving patch:
diff --git a/cleanlab/internal/token_classification_utils.py b/cleanlab/internal/token_classification_utils.py
--- a/cleanlab/internal/token_classification_utils.py
+++ b/cleanlab/internal/token_classification_utils.py
@@ -160,18 +160,22 @@ def color_sentence(sentence: str, word: str) -> str:
 
     Parameters
     ----------
-        sentence: str
+        sentence:
             a sentence where the word is searched
 
-        word: str
-            keyword to find in `sentence`. Assumes the word exists in token
-
+        word:
+            keyword to find in `sentence`. Assumes the word exists in the sentence.
     Returns
     ---------
-        colored_sentence: str
-            `sentence` where the first occurance of the word is colored red, using `termcolor.colored`
+        colored_sentence:
+            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`
 
     """
-    start_idx = sentence.index(word)
-    before, after = sentence[:start_idx], sentence[start_idx + len(word) :]
-    return "%s%s%s" % (before, colored(word, "red"), after)
+    colored_word = colored(word, "red")
+    colored_sentence, number_of_substitions = re.subn(
+        r"\b{}\b".format(word), colored_word, sentence
+    )
+    if number_of_substitions == 0:
+        # Use basic string manipulation if regex fails
+        colored_sentence = sentence.replace(word, colored_word)
+    return colored_sentence


Test patch:
diff --git a/tests/test_token_classification.py b/tests/test_token_classification.py
--- a/tests/test_token_classification.py
+++ b/tests/test_token_classification.py
@@ -121,9 +121,36 @@ def test_merge_probs_with_normalization():
     assert np.allclose(expected, merged_probs)
 
 
-def test_color_sentence():
-    colored = color_sentence(sentences[0], words[0][1])
-    assert colored == "Hello \x1b[31mWorld\x1b[0m"
+# Color boundaries
+C_L, C_R = "\x1b[31m", "\x1b[0m"
+
+
+@pytest.mark.parametrize(
+    "sentence,word,expected",
+    [
+        ("Hello World", "World", f"Hello {C_L}World{C_R}"),
+        ("If you and I were to meet", "I", f"If you and {C_L}I{C_R} were to meet"),
+        ("If you and I were to meet", "If you and I", f"{C_L}If you and I{C_R} were to meet"),
+        ("If you and I were to meet", "If you and I w", f"{C_L}If you and I w{C_R}ere to meet"),
+        ("I think I know this", "I", f"{C_L}I{C_R} think {C_L}I{C_R} know this"),
+        ("A good reason for a test", "a", f"A good reason for {C_L}a{C_R} test"),
+        ("ab ab a b ab", "ab a", f"ab {C_L}ab a{C_R} b ab"),
+        ("ab ab ab ab", "ab a", f"{C_L}ab a{C_R}b {C_L}ab a{C_R}b"),
+    ],
+    ids=[
+        "single_word",
+        "ignore_subwords",
+        "multi-token_match",
+        "substring_replacement",
+        "multiple_matches",
+        "case_sensitive",
+        "only_word_boundary",
+        "non_overlapping_substrings",
+    ],
+)
+def test_color_sentence(sentence, word, expected):
+    colored = color_sentence(sentence, word)
+    assert colored == expected
 
 
 issues = find_label_issues(labels, pred_probs)