Before revision:  In this work we address these two issues by proposing a framework for quantifying transparencyin system descriptions and apply it to analyze the trade-off between transparencyand end-user confusionusing NLP conference abstracts.
After revision:  In this setting, the degree to which the necessary information is conveyed represents the description's transparency, and the level of expertise needed by the third party corresponds to potential user confusion. We introduce two neural language model-based probabilistic metrics to model these factors, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we apply these metrics to study the relationships between transparency, confusion, and user perceptions in a corpus of NLP demo abstracts.