SlowGuess/ACL_2020_OCR huggingface dataset:
Alex Rinaldi
arinaldi@ucsc.edu
Jean E. Fox Tree
foxtree@ucsc.edu
at Chapel Hill
snigdha@cs.unc.edu
Despite the pervasiveness of clinical depression in modern society, professional help remains highly stigmatized, inaccessible, and expensive. Accurately diagnosing depression is difficult- requiring time-intensive interviews, assessments, and analysis. Hence, automated methods that can assess linguistic patterns in these interviews could help psychiatric professionals make faster, more informed decisions about diagnosis. We propose JLPC, a method that analyzes interview transcripts to identify depression while jointly categorizing interview prompts into latent categories. This latent categorization allows the model to identify high-level conversational contexts that influence patterns of language in depressed individuals. We show that the proposed model not only outperforms competitive baselines, but that its latent prompt categories provide psycholinguistic insights about depression.
Depression is a dangerous disease that effects many. A 2017 study by Weinberger et al. (2018) finds that one in five US adults experienced depression symptoms in their lifetime. Weinberger et al. also identify depression as a significant risk factor for suicidal behavior.
Unfortunately, professional help for depression is not only stigmatized, but also expensive, time-consuming and inaccessible to a large population. Lakhan et al. (2010) explain that there are no laboratory tests for diagnosing psychiatric disorders; instead these disorders must be identified through screening interviews of potential patients that require time-intensive analysis by medical experts. This has motivated developing automated depression detection systems that can provide confidential, inexpensive and timely preliminary triaging that can help individuals in seeking help from
medical experts. Such systems can help psychiatric professionals by analyzing interviewees for predictive behavioral indicators that could serve as additional evidence (DeVault et al., 2014).
Language is a well-studied behavioral indicator for depression. Psycholinguistic studies by Segrin (1990), Rude et al. (2004), and Andreasen (1976) identify patterns of language in depressed individuals, such as focus on self and detachment from community.
To capitalize on this source of information, recent work has proposed deep learning models that leverage linguistic features to identify depressed individuals (Malol-Ragolta et al., 2019). Such deep learning models achieve high performance by uncovering complex, unobservable patterns in data at the cost of transparency.
However, in the sensitive problem domain of diagnosing psychiatric disorders, a model should offer insight about its functionality in order for it to be useful as a clinical support tool. One way for a model to do this is utilizing the structure of the input (interview transcript) to identify patterns of conversational contexts that can help professionals in understanding how the model behaves in different contexts.
A typical interview is structured as pairs of prompts and responses such that participant responses follow interviewer prompts (such as "How have you been feeling lately?"). Intuitively, each interviewer prompt serves as a context that informs how its response should be analyzed. For example, a short response like "yeah" could communicate agreement in response to a question such as "Are you happy you did that?", but the same response could signal taciturnity or withdrawal (indicators of depression) in response to an encouraging prompt like "Nice!". To enable such context-dependent analysis, the model should be able to group prompts based on the types of conversa
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7-18 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tional context they provide.
To accomplish this, we propose a neural Joint Latent Prompt Categorization (JLPC) model that infers latent prompt categories. Depending on a prompt's category, the model has the flexibility to focus on different signals for depression in the corresponding response. This prompt categorization is learned jointly with the end task of depression prediction.
Beyond improving prediction accuracy, the latent prompt categorization makes the proposed model more transparent and offers insight for expert analysis. To demonstrate this, we analyze learned prompt categories based on existing psycholinguistic research. We also test existing hypotheses about depressed language with respect to these prompt categories. This not only offers a window into the model's working, but also can be used to design better clinical support tools that analyze linguistic cues in light of the interviewer prompt context.
- We propose an end-to-end, data-driven model for predicting depression from interview transcripts that leverages the contextual information provided by interviewer prompts
- Our model jointly learns latent categorizations of prompts to aid prediction
- We conduct robust experiments to show that our model outperforms competitive baselines
- We analyze the model's behavior against existing psycholinguistic theory surrounding depressed language to demonstrate the interpretability of our model
2 Joint Latent Prompt Categorization
We propose a Joint Latent Prompt Categorization (JLPC) model that jointly learns to predict depression from interview transcripts while grouping interview prompts into latent categories.<sup>1</sup>.
The general problem of classifying interview text is defined as follows: let  $X$  denote the set of  $N$  interview transcripts. Each interview  $X_{i}$  is a sequence of  $j$  conversational turns consisting of interviewer's prompts and participant's responses:  $X_{i} = \{(P_{ij}, R_{ij}) \text{ for } j = \{1 \dots M_{i}\}$ , where  $M_{i}$  is the number of turns in  $X_{i}$ ,  $P_{ij}$  is the  $j^{th}$  prompt in the  $i^{th}$  interview, and  $R_{ij}$  is the participant's re
sponse to that prompt. Together,  $(P_{ij},R_{ij})$  form the  $j^{th}$  turn in  $i^{th}$  interview. Each interview  $X_{i}$  is labeled with a ground-truth class  $Y_{i}\in \{1,\dots C\}$ , where  $C$  is the number of possible labels. In our case, there are two possible labels: depressed or not depressed. Our model, shown in Figure 1, takes as input an interview  $X_{i}$  and outputs the predicted label  $\hat{Y}_i$ .
Our approach assumes that prompts and responses are represented as embeddings  $\mathbf{P}_{ij} \in \mathbb{R}^E$  and  $\mathbf{R}_{ij} \in \mathbb{R}^E$  respectively. We hypothesize that prompts can be grouped into latent categories ( $K$  in number) such that corresponding responses will exhibit unique, useful patterns. To perform a soft assignment of prompts to categories, for each prompt, our model computes a category membership vector  $\mathbf{h}_{\mathbf{ij}} = [h_{ij}^1, \dots, h_{ij}^K]$ . It represents the probability distribution for the  $j^{th}$  prompt of the  $i^{th}$  interview over each of  $K$  latent categories.  $\mathbf{h}_{\mathbf{ij}}$  is computed as a function  $\phi$  of  $\mathbf{P}_{\mathbf{ij}}$  and trainable parameters  $\theta_{CI}$  (illustrated as the Category Inference layer in Figure 1):
$$
\mathbf {h} _ {\mathbf {i j}} = \phi \left(\mathbf {P} _ {i j}, \theta_ {C I}\right) \tag {1}
$$
Based on these category memberships for each prompt, the model then analyzes the corresponding responses so that unique patterns can be learned for each category. Specifically, we form  $K$  category-aware response aggregations. Each of these aggregations,  $\bar{\mathbf{R}}_i^k\in \mathbb{R}^E$ , is a category-aware representation of all responses of the  $i^{th}$  interview with respect to the  $k^{th}$  category.
$$
\bar {\mathbf {R}} _ {i} ^ {k} = \frac {1}{Z _ {i} ^ {k}} \sum_ {j = 1} ^ {M _ {i}} h _ {i j} ^ {k} \times \mathbf {R} _ {i j} \tag {2}
$$
$$
Z _ {i} ^ {k} = \sum_ {j = 1} ^ {M _ {i}} h _ {i j} ^ {k} \tag {3}
$$
where,  $h_{ij}^{k}$  is the  $k^{th}$  scalar component of the latent category distribution vector  $\mathbf{h}_{\mathrm{ij}}$  and  $Z_{i}^{k}$  is a normalizer added to prevent varying signal strength, which interferes with training.
We then compute the output class probability vector  $\mathbf{y_i}$  as a function  $\psi$  of the response aggregations  $[\bar{\mathbf{R}}_i^1,\dots ,\bar{\mathbf{R}}_i^K ]$  and trainable parameters  $\theta_{D}$  (illustrated as the Decision Layer in Figure 1).
$$
\mathbf {y} _ {\mathbf {i}} = \psi \left(\bar {\mathbf {R}} _ {i} ^ {1}, \dots , \bar {\mathbf {R}} _ {i} ^ {K}, \theta_ {D}\right) \tag {4}
$$
The predicted label  $\hat{Y}_i$  is selected as the class with the highest probability based on  $\mathbf{y_i}$
$^{1}$ Code and instructions for reproducing our results are available at https://github.com/alexwgr/LatentPromptRelease
Figure 1: The architecture of our JLPC model with  $K = 3$ . For each prompt  $\mathbf{P}_{ij}$  in interview  $i$ , the Category Inference layer computes a latent category membership vector,  $\mathbf{h}_{ij}$ . These are used as weights to form  $K$  separate Category-Aware Response Aggregations, which in turn are used by the Decision Layer to predict the output.
2.1 The Category Inference Layer
We compute the latent category membership for all prompts in interview  $i$  using a feed-forward layer with  $K$  outputs and softmax activation:
$$
\phi \left(\mathbf {P} _ {i j}, \theta_ {C I}\right) = \sigma \left(\operatorname {r o w} _ {j} \left(\mathbf {P} _ {i} \mathbf {W} _ {C I} + \mathbf {B} _ {C I}\right)\right) \tag {5}
$$
As shown in Equation 1,  $\phi (\mathbf{P}_{ij},\theta_{CI})$  produces the desired category membership vector  $\mathbf{h}_{\mathrm{ij}}$  over latent categories for the  $j^{th}$  prompt of the  $i^{th}$  interview.  $\mathbf{P}_i\in \mathbb{R}^{M\times E}$  is defined as  $[\mathbf{P}_{i1},\dots ,\mathbf{P}_{iM}]^T$  where  $M$  is the maximum conversation length in  $X_{i}$  and  $\mathbf{P}_{im} = \mathbf{0}^{E}$  for all  $M_{i} < m\leq M$ .  $\mathbf{P}_i\mathbf{W}_{CI} + \mathbf{B}_{CI}$  computes a matrix where row  $j$  is a vector of energies for the latent category distribution for prompt  $j$ , and  $\sigma$  denotes the softmax function.  $\mathbf{W}_{CI}\in \mathbb{R}^{E\times K}$  and  $\mathbf{B}_{CI}\in \mathbb{R}^{K}$  are the trainable parameters for this layer:  $\theta_{CI} = \{\mathbf{W}_{CI},\mathbf{B}_{CI}\}$ .
2.2 The Decision Layer
The Decision Layer models the probabilities for each output class (depressed and not-depressed) using a feed-forward layer over the concatenation  $\bar{\mathbf{R}}_i$  of response aggregations  $[\bar{\mathbf{R}}_i^1,\dots ,\bar{\mathbf{R}}_i^K ]$ . This allows each response aggregation  $\bar{\mathbf{R}}_i^k$  to contribute to the final classification through a separate set of trainable parameters.
$$
\psi (\bar {\mathbf {R}} _ {i} ^ {1}, \dots , \bar {\mathbf {R}} _ {i} ^ {K}, \theta_ {D}) = \sigma (\bar {\mathbf {R}} _ {i} ^ {T} \mathbf {W} _ {D} + \mathbf {B} _ {D}) (6)
$$
As shown in Equation 4,  $\psi (\bar{\mathbf{R}}_i^1,\dots ,\bar{\mathbf{R}}_i^K,\theta_D)$  produces the output class probability vector  $\mathbf{y_i}$ .
$\mathbf{W}_D \in \mathbb{R}^{(E*K) \times C}$  and  $\mathbf{B}_D \in \mathbb{R}^C$  are the trainable parameters for the decision layer:  $\theta_D = \{\mathbf{W}_D, \mathbf{B}_D\}$ .
We then compute the cross entropy loss  $L(Y, \hat{Y})$  between ground truth labels and  $\mathbf{y_i}$ .
2.3 Entropy regularization
The model's learning goal as described above only allows the output prediction error to guide the separation of prompts into useful categories. However, in order to encourage the model to learn distinct categories, we employ entropy regularization (Grandvalet and Bengio, 2005) by penalizing overlap in the latent category distributions for prompts. That is, we compute the following entropy term using components of the category membership vector  $\mathbf{h}_{ij}$  from Equation 1:
$$
E \left(X _ {i}\right) = \frac {1}{u _ {i}} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M _ {i}} E _ {j} \left(X _ {i}\right) \tag {7}
$$
$$
E _ {j} \left(X _ {i}\right) = - \sum_ {k = 1} ^ {K} h _ {i j} ^ {k} \ln h _ {i j} ^ {k} \tag {8}
$$
$$
u _ {i} = \sum_ {i = 1} ^ {N} M _ {i} \tag {9}
$$
Finally, the model's overall learning goal minimizes entropy regularized cross entropy loss:
$$
\arg \min  _ {\theta} L (Y, \hat {Y}) + \lambda E (X _ {i})
$$
where,  $\lambda$  is a hyper-parameter that controls the strength of the entropy regularization term.
2.4 Leveraging Prompt Representations in the Decision Layer
While prompt representations are used to compute latent category assignments, the model described so far (JLPC) cannot directly leverage prompt features in the final classification. To provide this capability, we define two additional model variants with pre-aggregation and post-aggregation prompt injection: JLPCPre and JLPCPost, respectively.
JLPCPre is similar to the JLPC model, except that it aggregates both prompt and response representations based on prompt categories. In other words, the aggregated response representation,  $\bar{\mathbf{R}}_i^k$  in Equation 2, is computed as:
$$
\bar {\mathbf {R}} _ {i} ^ {k} = \frac {1}{Z _ {i} ^ {k}} \sum_ {j = 1} ^ {M _ {i}} h _ {i j} ^ {k} [ \mathbf {P} _ {i j}, \mathbf {R} _ {i j} ]
$$
JLPCPost is also similar to JLPC except that it includes the average of prompt representations as additional input to the decision layer. That is, Equation 6 is modified to the following:
$$
\psi \left(\bar {\mathbf {R}} _ {i} ^ {1}, \dots , \bar {\mathbf {R}} _ {i} ^ {K}, \theta_ {D}\right) = \sigma \left(\left[ \bar {\mathbf {P}} _ {i}, \bar {\mathbf {R}} _ {i} \right] ^ {T} \mathbf {W} _ {\mathbf {D}} + \mathbf {B} _ {D}\right) \tag {10}
$$
$\bar{\mathbf{P}}_i$  is the uniformly-weighted average of prompt representations in  $X_{i}$
We evaluate our model on the Distress Analysis Interview Corpus (DAIC) (Gratch et al., 2014). DAIC consists of text transcripts of interviews designed to emulate a clinical assessment for depression. The interviews are conducted between human participants and a human-controlled digital avatar. Each interview is labeled with a binary depression rating based on a score threshold for the 9th revision of the Patient Health Questionnaire (PHQ-9). In total, there are 170 interviews, with 49 participants identified as depressed.
To achieve stable and robust results given the small size of the DAIC dataset, we report performance over 10 separate splits of the dataset into training, validation, and test sets. For each split,  $70\%$  is used as training data, and  $20\%$  of the training data is set aside as validation data.
3.1 Preprocessing and Representation
DAIC interview transcripts are split into utterances based on pauses in speech and speaker change, so we concatenate adjacent utterances by the same speaker to achieve a prompt-response structure. We experiment with two types of continuous representations for prompts and responses: averaged word embeddings from the pretrained GloVe model (Pennington et al., 2014), and sentence embeddings from the pretrained BERT model (Devlin et al., 2019). Further details are given in Appendix A.1. Reported results use GloVe embeddings because they led to better validation scores.
3.2 Exclusion of Predictive Prompts
Our preliminary experiments showed that it is possible to achieve better-than-random performance on the depression identification task using only the set of prompts (excluding the responses). This is possibly because the interviewer identified some individuals as potentially depressed during the interview, resulting in predictive follow-up prompts (for example, "How long ago were you diagnosed?"). To address this, we iteratively remove predictive prompts until the development performance using prompts alone is not significantly better than random (see Appendix A.3). This is to ensure our experiments evaluate the content of prompts and responses rather than fitting to any bias in question selection by the DAIC corpus interviewers, and so are generalizable to other interview scenarios, including future fully-automated ones.
We now describe our experiments and analysis.
Our experiments use the following baselines:
- The  $RO$  baseline only has access to responses. It applies a dense layer to the average of response representations for an interview.
- The  $PO$  baseline only has access to prompts, following the same architecture as  $RO$ .
- The  $PR$  baseline has access to both prompts and responses. It applies a dense layer to the average of prompt and response concatenations.
Model	F1 depressed	F1 not depr.
Random	0.303 (0.081)	0.690 (0.044)
PO	0.246 (0.080)	0.784 (0.032)
RO	0.309 (0.121)	0.798 (0.031)
PR	0.324 (0.121)	0.787 (0.030)
BERT	0.362 (0.080)	0.780 (0.062)
JLPC	0.416 (0.110)	0.761 (0.057)
JLPCPre	0.358 (0.121)	0.776 (0.037)
JLPCPost	0.440 (0.080)	0.768 (0.078)
Table 1: Mean F1 scores for the positive (depressed) and negative (not depressed) across the 10 test sets. Standard deviation is reported in parentheses. Two of the proposed models, JLPC and JLPCPost, improve over baselines including the BERT fine-tuned model (Devlin et al., 2019), with the JLPCPost achieving a statistically significant improvement  $(p < 0.05)$ .
- BERT refers to the BERT model (Devlin et al., 2019) fine-tuned on our dataset (see Appendix A.2).
4.2 Training details
All models are trained using the Adam optimizer. We use mean validation performance to select hyper-parameter values: number of epochs  $= 1300$ , learning rate  $= 5 \times 10^{-4}$ , number of prompt categories  $K = 11$  and entropy regularization strength  $\lambda = 0.1$ .
We computed the F1 scores of the positive (depressed) and negative (not-depressed) classes averaged over the 10 test sets. Given the class imbalance in the DAIC dataset, we compare models using F1 score for the depressed class.
As an additional baseline, we also implemented methods from Mallol-Ragolta et al. (2019) but do not report their performance since their model performs very poorly (close to random) when we consider averaged performance over 10 test sets. This is likely because of the large number of parameters required by the hierarchical attention model.
Table 1 summarizes our results. The below-random performance of the  $PO$  baseline is expected, since the prompts indicative of depression were removed as described in Section 3.2. This indicates the remaining prompts, by themselves, are not sufficient to accurately classify interviews. The  $RO$  model performs better, indicating the response information is more useful. The  $PR$  baseline improves over the  $RO$  baseline indicating that
Figure 2: Ablation study on validation set demonstrating the importance of prompt categorization and entropy regularization for our model.
the combination of prompt and response information is informative. The BERT model, which also has access to prompts and responses, shows a reasonable improvement over all baselines.
JLPC and JLPCPost outperform the baselines, with JLPCPost achieving a statistically significant improvement over both the PR and BERT baselines  $(p < 0.05)$ . This indicates the utility of our prompt-category aware analysis of the interviews.
We analyzed how the prompt categorization and entropy regularization contribute to our model's validation performance. The contributions of each component are visualized in Figure 2. Our analysis shows that while both components are important, latent prompt categorization yields the highest contribution to the model's performance.
4.5 Analyzing Prompt Categories
Beyond improving classification performance, the latent categorization of prompts yields insight about conversational contexts relevant for analyzing language patterns in depressed individuals.
To explore the learned categories, we isolate interviews from the complete corpus that are correctly labeled by our best-performing model. We say that the model "assigns" an interview prompt to a given category if the prompt's membership for that category (Equation 1) is stronger than for other categories. We now describe the various prompts assigned to different categories.<sup>3</sup>
Firstly, all prompts that are questions like "Tell me more about that", "When was the last time you had an argument?", etc. are grouped together into
${}^{2}$  Statistical significance is calculated from the test prediction using two-sided T-test for independent samples of scores
3To verify consistency of prompt categorization, we rerun the model with multiple initialization and they all yielded the same general trends as described in the paper.
a single category, which we refer to as the Starters category. Previous work has identified usefulness of such questions as conversation starters since they assist in creating a sense of closeness (McAllister et al., 2004; Heritage and Robinson, 2006).
Secondly, there are several categories reserved exclusively for certain backchannels. Backchannels are short utterances that punctuate longer turns by another conversational participant (Yngve, 1970; Goodwin, 1986; Bavelas et al., 2000). Specifically, the model assigns the backchannels "mmm," "mm," "nice," and "awesome" each to separate categories. Research shows that it is indeed useful to consider the effects different types of backchannels separately. For example, Bavelas et al. (2000) propose a distinction between specific backchannels (such as "nice" and "awesome") and generic backchannels (such as "mm" and "mmm"), and Tolins and Fox Tree (2014) demonstrated that each backchannel type serves a different purpose in conversation.
Thirdly, apart from starters and backchannels, the model isolates one specific prompt - "Have you been diagnosed with depression?" into a separate category. Clearly, this is an important prompt and it is encouraging to see that the model isolates it as useful. Interestingly, the model assigns the backchannel "aw" to the same category as "Have you been diagnosed with depression?" suggesting that responses to both prompts yield similar signals for depression.
Lastly, the remaining five categories are empty - no prompt in the corpus has maximum salience with any of them. A likely explanation for this observation stems from the choice of normalizing factor  $Z_{i}^{k}$  in Equation 3: it causes  $\bar{\mathbf{R}}_i^k$  to regress to the unweighted average of response embeddings when all prompts in an interview have low salience with category  $k$ . Repeated empty categories then function as an "ensemble model" for the average response embeddings, potentially improving predictive performance.
4.6 Category-based Analysis of Responses
The prompt categories inferred by our JLPCPost model enable us to take a data-driven approach to investigating the following category-specific psycholinguistic hypotheses about depression:
Starters	Backchannels
D	ND	D	ND
RL	23.2	27.2	19.9	15.1
DMF (×10-2)	6.55	7.31	7.98	8.55
Table 2: Indicators for social skills: mean response length (RL) and discourse marker/filler rates (DMF) for responses to prompts in starters and backchannel (collectively representing "mm", "mm", "nice", and "awesome") categories, for depressed (D) and not-depressed (ND) participants. Statistically significant differences are underlined  $(p < 0.05)$ . Both measures are significantly lower for the depressed class for responses to starters, but not to backchannels.
H1 Depression correlates with social skill deficits (Segrin, 1990)
H2 Depressed language is vague and qualified (Andreasen, 1976)
H3 Depressed language is self-focused and detached from community (Rude et al., 2004)