Before revision:  We propose measurements of label-rationale association , a necessary property of faithful rationales, for these models. Using our measurements , we show that a state-of-the-art joint model based on T5 has strengths and weaknesses for producing faithful  rationales.
After revision:  We propose measurements of label-rationale association , a necessary property for faithfulness: the internal mechanisms of the model producing the label and the rationale must be meaningfully correlated. We propose two measurements to test this property: robustness equivalence and feature importance agreement. We find that state-of-the-art joint model based on T5 has strengths and weaknesses for producing faithful rationales.