SlowGuess/NAACL_2022_OCR huggingface dataset:
We use POLYJUICE (PJ, Wu et al., 2021) to generate instances of three semantic operations. Entity change replaces a proper noun (e.g. actor name) with another using "lexical" mode of PJ. Minor insert adds a short text to the sentence using "insert" mode. Negation generated a negated version of the input using "negation" mode. For each operation type, our expectation for model behavior is formalized as a range  $b(\widehat{y})$  on  $\widehat{y}$ . We expect the prediction to be unchanged by the first two operations allowing for a margin of 0.05, but changed to the other side of 0.5 by negation. We then define validity  $\nu = \mathbb{E}_{\widehat{Y},\widehat{Y}_i}\left[\mathbb{1}_{\widehat{Y}_i\in b(\widehat{Y})}\right]$  and sharpness  $\sigma = 1 - \mathbb{P}_{\widehat{Y}}[b(\widehat{Y})]$  analogously.
Tab. 6 summarizes the results. While our expectation is mostly confirmed for entity change and minor insert, it is notably violated in the case of negation, with only  $30.4\%$  validity, indicating model prediction is on the same side  $69.6\%$  of time. Upon further evaluation, we find that validity drops with sentence length, with short sentences of six words or fewer having much higher validity (for other-side). Since the PJ rewriting model is learned rather than manually defined and negation is more complex than the other two operations, there are two failure modes, as presented in Tab. 7. In the first, a negation is applied to the input sentence, but on a part irrelevant to the sentiment. In the second, the generated sentence is not a negation of the input by any reasonable standard.
These examples highlight the importance of clearly defining the operation: rather than a generic
Table 6: Instance-based explanation metrics on SST.
Input sentence	“Negated” sentence
Human Nature initially succeeds by allowing it- self to go crazy , but ultimately fails by spinning out of control .	Human Nature initially succeeds by allowing it- self to go crazy , but ultimately fails by not com- ing to consciousness .
This may be the dumbest , sketchiest movie on record about an aspiring writer ’s coming-of-age .	This may be the dumbest , sketchiest movie on record , not an aspiring writer ’s coming-of-age .
Before long , the film starts playing like General Hospital crossed with a Saturday Night Live spoof of Dog Day Afternoon.	Before long , the film starts playing like nothing crossed with a Saturday Night Live spoof of Dog Day Afternoon .
A startling and fresh examination of how the bike still remains an ambiguous icon in Chinese society .	A startling and fresh examination of how the bike still seems to be an ambiguous icon in Chinese society .
Never engaging , utterly predictable and completely void of anything remotely interesting or suspenseful .	Not engaging , utterly predictable and completely void of anything remotely interesting or suspense- ful .
Between the drama of Cube ?	Are there no interesting problems?
Tailored to entertain !	No tails !
Table 7: Failure cases of POLYJUICE negations. The first half shows examples where the negation is irrelevant to the sentiment. The second half includes examples where the negation fails to appear.
negation, we would need the negation to happen on the “sentiment-carrying” part. It is also crucial to ensure that the generator is of a high quality in order to minimize the chance of generating nonsensical outputs. Despite many advances in generative language modeling, it have been shown to be undesirable in many ways (e.g., Holtzman et al., 2019), all of which affect the quality of the counterfactual explanation.
At a high level, IBE explains the local prediction by illustrating ways to cross (e.g., negation) or not cross (e.g., entity change) the decision boundary in the (very) high-dimensional input space. However, as the negation case indicates, we must be careful about the exact definition of the rewriting (e.g., negating any part of the input or the "sentiment-carrying" part only), as it could have a significant impact on the conclusion. Furthermore, it is difficult for any rewriting mechanism to achieve  $100\%$  validity due to the high dimensionality, the multitude of possible ways of rewriting, and the imperfection of the model. Focusing only on the mistakes (or ignoring them altogether) yields incomplete model understanding. Instead, the validity metric, which indicates the generalized model behavior, should be used to.
There are many potentially fruitful directions for future work. First, the quality of instances obviously depend on the generative models, which, while impressive, are known to be flawed in many ways (e.g., Holtzman et al., 2019; Nadeem et al., 2021; Wolfe and Caliskan, 2021). Second, each rule essentially covers the entire input space. Partitioning the input space in some way may allow for identification of both more and less consistent areas, which is makes the applicability function much more difficult to define as it now takes whole sentences rather than individual words. Finally, unlike feature attribution, which conveys the single notion of "importance," different instances of the same input can reveal different aspects of model behavior, calling for a potentially different definition of coverage, which measures completeness of understanding.