Venue: NAACL 2019

# Audio De-identification: A New Entity Recognition Task

 Ido Cohn, Itay Laish, Genady Beryozkin, Gang Li, Izhak Shafran,

Idan Szpektor, Tzvika Hartman, Avinatan Hassidim, Yossi Matias

Google

Tel Aviv, Israel

{idoc,itaylaish,leebird,tzvika}@google.com

###### Abstract

Named Entity Recognition (NER) has been mostly studied in the context of written text. Specifically, NER is an important step in de-identification (de-ID) of medical records, many of which are recorded conversations between a patient and a doctor. In such recordings, audio spans with personal information should be redacted, similar to the reduction of sensitive character spans in de-ID for written text. The application of NER in the context of audio de-identification has yet to be fully investigated. To this end, we define the task of audio de-ID, in which audio spans with entity mentions should be detected. We then present our pipeline for this task, which involves Automatic Speech Recognition (ASR), NER on the transcript text, and text-to-audio alignment. Finally, we introduce a novel metric for audio de-ID and a new evaluation benchmark consisting of a large labeled segment of the _Switchboard_ and _Fisher_ audio datasets and detail our pipeline's results on it.

## 1 Introduction

Personal data in general, and clinical records data in particular, is a major driving force in today's scientific research. Despite its abundance, the presence of Personal Health Identifiers (PHI) hinders data availability for researchers. Therefore, data de-identification (de-ID) is a critical component in any plan to make such data available. However, the amount of data involved makes it prohibitively expensive to employ domain experts to tag and redact PHI manually, providing a good opportunity for automatic de-identification tools. Indeed, high performance tools for the de-identification of medical text notes have been developed [35, 20].

Due to the rise of tele-medicine [49], clinical records consist of many other types of data, such as audio conversations [67], scanned documents, video, and images. In this work, we direct our attention towards the task of de-identifying clinical audio data. This task is expected to become increasingly more important, as Machine Learning applications in telec medicine are growing in popularity. Given an input audio stream, the objective is to produce a modified audio stream, where all PHI is redacted, while the rest of the stream is kept unchanged. To the best of our knowledge, de-identifying audio is a new task, requiring a new benchmark.

We define and publish1 a benchmark consisting of the following: 1. A large labeled subset of the _Switchboard_[8] and _Fisher_[1] conversational English audio datasets, denoted as _SWFI_. 2. A new evaluation metric, measuring how well the PHI words in the input audio were identified and redacted, and how well the rest of the audio was preserved.

Footnote 1: [https://g.co/audio-ner-annotations-data](https://g.co/audio-ner-annotations-data)

To better understand the challenges of the audio de-id task, we evaluate it both end-to-end and by breaking it down and solving it using individual components. Our pipeline (Fig. 1) first produces transcripts from the audio using ASR, proceeds by running text-based NER tagging, and then redacts PHI tokens, using the aligned token boundaries determined by ASR. Our tagger relies on the state-of-the-art techniques for solving the audio NER problem of recognizing entities in audio transcripts [19, 21]. We leverage the available Automatic Speech Recognition (ASR) technology, and use its component of aligmme

Figure 1: High level audio de-ID pipeline

Finally, we evaluate our pipeline and describe its performance, both end-to-end and per-component. Although results on audio are worse than NER performance on text, the pipeline achieves better results than expected despite the compounding pipeline errors. Last, we analyze our performance and provide insights for next steps.

## 2 Related Work

### NER for Speech

Prior work addressed entity recognition for audio recordings via the _audio NER_ task: the detection of entities in the text transcript of the audio input. The majority of these works used a pipeline approach, in which ASR is first applied to the audio and then NER is applied on the noisy textual output of the ASR. These works include discriminative models Sudoh et al. (2006), incorporating OOV word indicators Parada et al. (2011), hierarchical structure Raymond (2013), and conditional random fields Hatmi et al. (2013).

Many audio NER works learn from and measure performance on French datasets, such as _ESSTER_Galliano et al. (2009) and _ETAPE_Galibert et al. (2014). This may indirectly affect the overall quality of these systems because the ASR component, which is crucial in the pipeline approach but is typically used "off-the-shelf", has lower performance in languages other than English.

An alternative end-to-end approach was proposed by Ghannay et al. (2018), in which the model accepts audio as input and outputs a tagged word sequence which consists of normal words and the NER labels in HTML-like tag encoding. Their model did not attain reasonable performance, perhaps due to the small training set.

We emphasize that both pipeline and end-to-end approaches output tagged word sequences, and do not propagate the recognized entity labels back for redaction on the audio itself, which is the end goal of our proposed audio de-ID task.

### De-identification in the Health Domain

Previous efforts of de-ID in health care focused on redaction of textual medical records. The main approach involves applying NER techniques to the text, including rule-based Ruch et al. (2000); Neamatullah et al. (2008) and machine learning Guo et al. (2006); Yang and Garibaldi (2015) methods.

Adoption of neural network models boosted the performance of NER on text without requiring hand-crafted rules and complex feature engineering Collobert et al. (2011); Huang et al. (2015); Lample et al. (2016); Ma and Hovy (2016); Dernoncourt et al. (2017a). Dernoncourt et al. (2017b) applied the model proposed in Lample et al. (2016) to medical de-ID, achieving state-of-the-art performance on the I2B2-2014 Stubbs and Uzuner (2015) de-ID challenge dataset. We have chosen this architecture for the NER component of our pipeline method (Section 5).

## 3 The Audio De-identification Task

The goal of the Audio de-ID task is to convert an input audio stream into a modified audio stream where the PHI words are redacted. In essence, the goal of the task is to limit the ability of a listener to identify the entities of the conversation while leaving as much information as possible in order to keep the audio understandable.

Formally, the input audio stream is a function \(A(t)\) of time, that can be transcribed into a sequence of words \(W=\{w_{j}\}\), where \(w_{j}\) is mapped to the time interval in the audio \(T_{j}=[t^{start}_{j},t^{end}_{j}]\). We consider each word to be either PHI or non-PHI, and let \(I\) denote the set of PHI words \(\{j:w_{j}\text{ is PHI}\}\).

The output of an audio de-ID algorithm is a zero-one redaction function \(R(t)\), indicating which parts of the audio stream are to be redacted, where a value of zero indicates PHI information at time \(t\). The redacted audio stream can be obtained by zeroing out the redacted part of the stream, \(A_{redacted}(t)=R(t)A(t)\).

To evaluate the performance of a de-ID algorithm, we term \(w_{j}\) as _fully-covered_ if \(R(t)\) is zero for all \(t\in T_{j}\), and define a corresponding indicator function \(covered(w_{i})\). This in turn defines the following standard NER metrics for the audio de-ID task:

\[TruePositives~{}(TP) = \sum_{j\in I}covered(w_{j}),\] \[FalsePositives~{}(FP) = \sum_{j\not\in I}covered(w_{j}),\] \[FalseNegatives~{}(FN) = \sum_{j\in I}1-covered(w_{j})\] \[Precision=\frac{TP}{TP+FP}~{}~{},~{}Recall=\frac{TP}{TP+FN}\]

Finding the exact time interval corresponding to a word is not a trivial task, while redacting 

most of the interval \(T_{j}\) results in a similar de-ID effect as fully covering all the interval. To this end, we extend \(covered(w_{i})\) into the indicator \(\rho-covered(w_{j})\) that is true iff \(R(t)\) is zero on at least \(\rho\) proportion of interval \(T_{j}\).

With this indicator function we further extend the aforementioned NER metrics to \(TP_{\rho}\), \(FP_{\rho}\), and \(FN_{\rho}\), and correspondingly define \(Recall_{\rho}\) and \(Precision_{\rho}\). When \(\rho=1\) these metrics equal the strict metrics. When \(\rho<1\) the new metrics determine the quality of the system with respect to redacting at least \(\rho\) of each audio interval in PHIs.

We note that the proposed metrics only evaluate the redaction function \(R(t)\) on the word intervals.

## 4 Datasets

To create a benchmark for the audio de-ID task, we use three datasets from two distinct domains: _conversational English_ and _medical records_. We summarize the main dataset statistics in Table 1. Importantly, we did not perform text normalization specific to each domain.

In the domain of medical datasets, we use **I2B2'14**[35], which consists of identified textual medical notes with PHI tagging, and the Audio Medical Conversations dataset from Chiu et al. (2017), denoted _AMC'17_, which contains de-identified audio of doctor-patient conversations and their corresponding manual transcripts. Processing the _AMC'17_ conversations was facilitated by the fact that it is a de-identified dataset, which provides us with the locations of the PHI in the audio and the transcripts. Three PHI types: names, dates and ages were redacted, preserving type information, and synthetic data was generated using dictionaries and context-aware rules. First names were drawn from the US Social Security Administration babies names registry2 and last names were drawn from the Frequently Occurring Surnames list from the 1990's US Census3. Human annotators used surrounding context to resolve the other PHI types and filled in fake appropriate identifiers.

Footnote 2: [https://www.ssa.gov/oact/babynames/](https://www.ssa.gov/oact/babynames/)

Footnote 3: [https://www.census.gov/topics/population/geneology/data/1990_census.html](https://www.census.gov/topics/population/geneology/data/1990_census.html)

Notably, neither of the above-mentioned medical datasets could serve as a benchmark for the audio de-ID task, as **I2B2'14** is text-based, and _AMC'17_ contains only redacted audio conversations and is not publicly available. Therefore, we focused on the conversational English domain, where we generated a combined dataset _SWFI_ from the _Switchboard_Godfrey et al. (1992) and _Fisher_David et al. (2004) datasets. These datasets include hundreds of conversations in English about a variety of subjects, along with their transcripts. To enable proper training and evaluation for the audio de-ID task, we annotated all 250 _Switchboard_ conversations, and 326 from _Fisher_. Annotation included named PHI labels, and the time intervals \(T_{j}=[t^{start}_{j},t^{end}_{j})\) matching each named PHI back into the audio. This dataset is publicly available1 to allow for standardized evaluation of novel approaches to this task.

Footnote 3: [https://www.census.gov/topics/population/geneology/data/1990_census.html](https://www.census.gov/topics/population/geneology/data/1990_census.html)

The annotation process began by tokenization of the transcripts provided in both datasets using white-space separators, removing special transcript characters and keeping word capitalization in its original form. Following that, PHI word annotation was performed manually. The results can be seen in Table 2.

\begin{table}
\begin{tabular}{|l|r||r|r|r|} \hline \multicolumn{1}{|c|}{Dataset} & \multirow{2}{*}{Medium} & \# Notes & \# Ploxens & \% PHI \\ \hline _I2B2’14_ train & Text & 521 & 336,422 & 3.5 \\ \hline _AMC'17_ train & Audio & 4.629 & 8.348,899 & 0.02 \\ \hline _SWFI_ train & Audio & 468 & 710,348 & 1.8 \\ _SWFI_ test & Audio & 108 & 158,923 & 2.0 \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for train and test sets, showing the number of notes (written or spoken), token count, and percent of tokens which are PHI.

\begin{table}
\begin{tabular}{|l|r|r|r|} \hline \multicolumn{1}{|c|}{PHI Labels \%} & _I2B2’14_ & _AMC’17_ & \multicolumn{1}{c|}{_SWFI_} \\  & & & train /test \\ \hline
**Name** & 0.84\% & 0.12\% & 0.225 / 0.23\% \\ \hline
**Age** & 0.24\% & 0.01\% & 0.12\% / 0.1\% \\ \hline
* \\ \hline Pharmacy &
- & 0.01\% &
- \\ \hline
**Organization** & 0.02\% & 0.03\% & 0.48\% / 0.59\% \\ \hline
* & 0.13\% / 0.16\% \\ \hline City & 0.08\% & 0.003\% & 0.25\% / 0.29\% \\ \hline Country & 0.02\% &
- & 0.23\% / 0.27\% \\ \hline Profession & 0.04\% &
- & 0.23\% / 0.27\% \\ \hline Holiday &
- & 0.12\% / 0.07\% \\ \hline Season &
- & 0.04\% / 0.03\% \\ \hline \end{tabular}
\end{table}
Table 2: Statistics for PHI labels as percent of total tokens per dataset. Tags in **bold** are common to all datasets and are used in Section 7

\begin{table}
\begin{tabular}{|c|c|c|r|r|} \hline Word & WER & Well & Extended & Shorned \\ Type & Avigned & Alignment & Alignment \\ \hline PHI & 41.8 & 86\% & 5\% & 9\% \\ non-PHI & 38.3 & 81\% & 8\% & 12\% \\ \hline \end{tabular}
\end{table}
Table 3: ASR WER and token-audio alignment distribution on sample conversations from the _SWFI_ dataset.

 

As performing temporal labeling manually is an arduous process, we opt for a semi-automatic ASR-based procedure. To this end, we determine word start and end times by aligning the manual transcripts to audio intervals. We assess the quality of this semi-automatic labeling scheme using human evaluation. For a random sample of 6 _SWFI_ conversations (3 _Switchboard_ and 3 _Fisher_), we slice the audio according to the aligned interval times per transcript word, and measure both the quality of the transcription, and that of the alignment. Table 3 shows the distribution of alignment errors of the tokens from the sample conversations. These are denoted as good alignment, short (i.e. ASR interval is shorter than actual word) and extended (i.e. interval is longer than expected) where all alignment errors are in the scale of 30-60ms (1-2 audio frames).

## 5 Pipeline Models

We next describe the models we trained and evaluated to gain insights on the types of challenges this task presents. We chose to use the pipeline approach as an audio de-ID benchmark due to the ubiquity and maturity of the ASR technology, and abundance of training data for text NER. Our pipeline models contain three main components:

1. An ASR system, which transcribes the audio into text.
2. A NER tagger, which tags the transcript with the required labels.
3. An alignment component, which maps each word in the transcript back to its time interval in the audio.

For the ASR component, we use Google Cloud's Speech API4 with the _commandand\(\_\)sandsearch_ model, which gave us the best transcription accuracy on the data. For each conversation, which usually contains two different speakers, we send the entire audio to the service to obtain the transcript. The API also returns alternative hypotheses for the corresponding text and their confidence. We incorporate these alternative hypotheses by taking the top-k ASR hypotheses and feeding them into the next two stages. We then take the logical OR of the detections on all of the hypotheses. Unless stated otherwise, \(k=1\).

Footnote 4: cloud.google.com/speech-to-text

For the NER tagger component, our models use the architecture described in Lample et al. (2016), depicted in Fig. 2. This is a neural network model using pre-trained GloVe word embeddings5Pennington et al. (2014) and a character-based bi-directional RNN to generate token embeddings, followed by a bidirectional RNN, tag projection, and CRF layers. We define three models, where each model has a NER tagger trained on a different dataset. The models are:

Footnote 5: nlp.stanford.edu/data/glove.68.zip

* Trained using the training data from the _AMC'17_ dataset.
* Trained using the training data from the _SWFI_ dataset.
* Trained using the training data from the _I2B2'14_ dataset.

The \(\mathbf{M_{AMC}}\) and \(\mathbf{M_{SWFI}}\) models were trained using the conversation transcripts. We use data augmentation in order to increase robustness to ASR errors, in particular to word deletion, insertion, substitution, and inconsistent capitalization. Data augmentation is carried out in several stages. First we create an ASR transcript from the audio, align it back to the reference transcript by minimizing the word-level edit distance, and transfer the labels to the new transcript. For each of the two transcripts, we then generate three additional transcripts by changing word capitalization to camel, lower and upper case. Finally, each of the augmented transcripts is broken down into segments of 20 speaker turns with a step of 10 turns, to resemble the utterance structure of the ASR output. We include the three variants of the \(\mathbf{M_{SWFI}}\) model: \(\mathbf{M_{SWFIReg}}\) uses no augmentations,

Figure 2: Neural architecture for text de-ID



\(M_{SWFI}\) uses mix-case augmentations only, and \(M_{SWFI\_MixCase+Asr}\) uses all mix-case and ASR augmentations.

The \(M_{I2B2}\) model is tuned to achieve state-of-the-art results on textual medical notes, such as in Dernnocourt et al. (2017a); Liu et al. (2017). It should be stressed that the model was used as is, without an attempt to adapt it to the domain of ASR output. \(M_{AMC}\) and all \(M_{SWFI}\) models are both trained on conversational data, and should be better adapted to the task. \(M_{AMC}\) is trained on data originating from the medical domain, as opposed to \(M_{SWFI}\) models which train on data from the English conversation target domain. This is offset by the fact that \(M_{AMC}\) is trained on a significantly larger training set.

Finally, for the alignment component we add a padding hyperparameter allowing a variable number of mismatched frames at either side of the identified intervals. This slack in interval size is used to compensate for alignment errors.

## 6 Experimental Settings

To test the performance of our models on the audio de-ID task, we conducted a number of experiments, described next. Section 7 then details our results. We report \(Recall\), \(Precision\), and \(F1\) scores for all experiments, which are significantly more informative than accuracy due to a low PHI/non-PHI ratio. We report results on the _SWFI_ test set using the tags which are shown in bold in Table 2. We evaluate our performance against the coverage threshold \(\rho\in[0,1]\) which is defined in Section 3. Specifically, we focus on type-less metrics, as we care more about the tokens' redaction than their type classification.

Our first experiment evaluates the performance of \(M_{AMC}\), \(M_{SWFI}\), and \(M_{I2B2}\) on the _SWFI_ test set. First, to decouple their tagging performance from the other pipeline errors, we measure their tagging performance on the manually annotated transcripts (referred to as _NER score_). NER errors may arise due to train-test disparity, where the train and test data are from different domains or different mediums (e.g. text vs. audio), which results in different discriminative models. Additionally, we measure their overall end-to-end score. We analyze the complex behavior of the models' precision by inspecting the coverage distribution of PHI and non-PHI tokens.

Our second experiment evaluates the effect of two significant hyperparameters on pipeline performance using the _SWFI_ test set:

* The number of alternative hypotheses passed on from the ASR to the NER tagger.
* The amount of padding added around each detection by the alignment component.

## 7 Results

In Table 3 we report the Word Error Rate (WER) of our ASR component on the _SWFI_ dataset, which was computed by comparing the manual and ASR transcripts of the entire audio. For WER of PHI words, we removed all the non-PHI words from manual ASR transcripts before computing the WER. WER of non-PHI words was computed similarly. We see that both WERs are substantial, and can be thought of as an upper-bound on our pipeline's end-to-end performance.

Next, Table 4 shows the NER and their end-to-end performance of each model for its \(F1\) optimal choice of coverage threshold \(\rho\). We can also see that the \(M_{SWFI}\) surpasses the others in performance due to its training set being incomain and in the same medium. Additionally, the \(M_{SWFI\_MixCase+Asr}\) variant does not display any advantage over its other variants when running on manual transcripts, but gets significantly better performance on the end-to-end scenario. The difference between NER and end-to-end scores is apparent, and may be attributed to additional pipeline components of ASR and alignment. Interestingly, in the case of \(M_{SWFIRog}\), compounding

\begin{table}
\begin{tabular}{|l|c|c|} \hline Model & Recall \(\frac{\text{WER}}{\text{Resion~{}/~{}Fl}}\) & \(F1\) (\(\rho\)) \\ \hline \hline \(M_{I2B2}\) & 0.37 / 0.48 / 0.41 & 0.37 (0.4) \\ \hline \(M_{AMP}\) & 0.18 / 0.98 / 0.3 & 0.21 (0.35) \\ \hline \(M_{SWIFI}\) & 0.82 / 0.92 / 0.87 & 0.41 (0.4) \\ \hline \(M_{SWIF}\)**,MinConv** & 0.87 / 0.92 / 0.89 & 0.46 (0.4) \\ \hline \(M_{SWIFI}\)**,MinConConte** & 0.88 / 0.92 / 0.9 & 0.51 (0.4) \\ \hline \end{tabular}
\end{table}
Table 4: NER score of the different NER models, and their end-to-end \(F1\) in their optimal choice of \(\rho\).

\begin{table}
\begin{tabular}{|l|r|r|} \hline Error type & Count & \% of total \\ \hline \hline ASR Transcription errors & 152 & 45.24 \\ \hline NER errors & 168 & 50 \\ \hline Alignment errors & 14 & 4.17 \\ \hline Manual transcription errors & 2 & 0.6 \\ \hline \end{tabular}
\end{table}
Table 5: Error analysis of a sample of \(M_{SWFI}\)\(FN\) errors, including errors from all components across the pipeline and even occasional manual transcription errors which contribute to both \(FP\) and \(FN\) errors.

 

the WER and alignment error rate from Table 3 and the NER from Table 4 leads to an expected \(Recall\) of approximately 0.44, yet the end-to-end \(Recall\) at \(\rho=0.5\) is 0.53. This implies a non-trivial co-dependence between errors in the different components of the pipeline.

Figure 3 presents the end-to-end evaluation of the different models with respect to the coverage threshold \(\rho\). As expected, \(Recall\) is monotonically non-increasing with respect to the threshold. Meanwhile, \(Precision\) (and consequently \(F1\)) are not monotonic and have more complex behavior. This behavior is due to difference in the distribution of the coverage between PHI and non-PHI, which we see in Figure 4 (left). An interesting insight is that most PHI words have more than half their length redacted by the pipeline while non-PHI words' coverage is bi-modal, one mode close to 0, and the other close to 1. A plausible explanation for this behavior is that the \(FP\)s are derived from alignment errors in low coverage, while the high coverage \(FP\)s occur due to classification errors, either due to ASR transcription mistakes or due to model NER errors.

Finally, we show the end-to-end evaluation of the pipeline using _MSWFI_MixCase+Asr_ with different choices of the pipeline parameters. In Figure 4 (center) the performance of the pipeline slightly increases when using additional alternative hypotheses, while a different experiment shows that when using alternative hypotheses with _MSWFI_MixCase_ performance decreases. This decrease is consistent with the hypotheses' decreasing confidence scores, which can be alleviated with ASR training data but is not addressed by the naive OR approach described in Section 5. This leads us to seek new ways to utilize the additional ASR artifacts, such as the hypotheses confidence scores and speech lattice. In Section 8 we discuss possible directions to improve the pipeline's robustness to ASR errors. Last, Figure 4 (right) shows that the choice of padding size does not improve performance, but rather alters the value of the optimal coverage threshold.

## 8 Conclusions

We introduced the audio de-ID task, an important prerequisite for protecting privacy when processing sensitive audio datasets in the medical domain as well as other domains. To this end, we created and made available a new test set benchmark derived from annotating the _Switchboard_ and _Fisher

Figure 4: Coverage distribution between PHI and non-PHI tokens (left) and End-to-end performance for different pipeline parameters – number of hypotheses (center) and padding (right).

Figure 3: End-to-end performance comparison of NER models – Recall (left), Precision (middle) and F1 (right).

 

audio datasets. We also presented new metrics for the task, \(Recall_{\rho}\) and \(Precision_{\rho}\), as extensions of standard \(Recall\) and \(Precision\) where words are considered de-identified when at least a portion \(\rho\) of their audio signal is redacted. Finally, we detailed our algorithm for this task, a pipeline approach consisting of three components: ASR, NER on transcripts and a novel alignment from tagged transcripts to audio for the actual redaction.

We showed that ASR performance is the main impedance towards achieving results comparable to text de-ID. In future work, we plan to address this through several directions, including end-to-end de-ID Ghannay et al. (2018), lattice-based techniques Ladhak et al. (2016), and diarization and segmentation of the audio as part of the transcription process Cerva et al. (2013).

## 9 Acknowledgements

The authors would like to thank Oren Gilon, Shlomo Hoory, Amir Feder, Debby Cohen, Amit Markel, and Ronit Slyper for their generous help in the writing of this paper.

Deidentified clinical records used in this research were provided by the i2b2 National Center for Biomedical Computing funded by U54LM008748 and were originally prepared for the Shared Tasks for Challenges in NLP for Clinical Data organized by Dr. Ozlem Uzuner, i2b2 and SUNY.

## References

* Cerva et al. (2013) Petr Cerva, Jan Silovsky, Jindrich Zdansky, Jan Nouza, and Ladislav Seps. 2013. Speaker-adaptive speech recognition using speaker diarization for improved transcription of large spoken archives. _Speech Communication_, 55(10):1033-1046.
* Chiu et al. (2017) Chung-Cheng Chiu, Anshuman Tripathi, Katherine Chou, Chris Co, Navdeep Jaitly, Diana Jaumeikare, Anjuli Kannan, Patrick Nguyen, Hasim Sak, Ananthankar, et al. 2017. Speech recognition for medical conversations. _arXiv preprint arXiv:1711.07274_.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _JMLR_, 12(Aug):2493-2537.
* David et al. (2004) Christopher Cieri David, David Miller, and Kevin Walker. 2004. The fisher corpus: a resource for the next generations of speech-to-text. In _in Proceedings 4th International Conference on Language Resources and Evaluation_, pages 69-71.
* Dernoncourt et al. (2017a) Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017a. De-identification of patient notes with recurrent neural networks. _J. Am Med Inform Assoc_, 24(3):596-606.
* Demoncourt et al. (2017b) Franck Dernoncourt, Ji Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017b. De-identification of patient notes with recurrent neural networks. _Journal of the American Medical Informatics Association_, 24(3):596-606.
* Galibert et al. (2014) Olivier Galibert, Jeremy Leixa, Gilles Adda, Khalid Choukri, and Guillaume Gravier. 2014. The tapeap speech processing evaluation. In _LREC_, pages 3995-3999. Citeseer.
* Galliano et al. (2009) Sylvain Galliano, Guillaume Gravier, and Laura Chaubard. 2009. The ester 2 evaluation campaign for the rich transcription of french radio broadcasts. In _Tenth Annual Conference of the International Speech Communication Association_.
* Ghanney et al. (2018) Sahar Ghanney, Antoine Caubriere, Yannick Esteve, Antoine Laurent, and Emmanuel Morin. 2018. End-to-end named entity extraction from speech. _arXiv preprint arXiv:1805.12045_.
* Volume 1_, ICASSP'92, pages 517-520, Washington, DC, USA. IEEE Computer Society.
* Guo et al. (2006) Yukun Guo, Robert Gaizauskas, Ian Roberts, George Demetriou, Mark Hepple, et al. 2006. Identifying personal health information using support vector machines. In _i2b2 workshop on challenges in natural language processing for clinical data_, pages 10-11. Citeseer.
* Hatmi et al. (2013) Mohamed Hatmi, Christine Jacquin, Emmanuel Morin, and Sylvain Meignier. 2013. Named entity recognition in speech transcripts following an extended taxonomy. In _First Workshop on Speech, Language and Audio in Multimedia_.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv:1508.01991_.
* Ladhak et al. (2016) Faisal Ladhak, Ankur Gandhe, Markus Dreyer, Lambert Mathias, Ariya Rastrow, and Bjrn Hoffmeister. 2016. Latticernn: Recurrent neural networks over lattices. In _Interspeech 2016_, pages 695-699.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _ACL_.
* Liu et al. (2017) Zengjian Liu, Buzhou Tang, Xiaolong Wang, and Qingcai Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. _J. Biomed. Inf._, 75:34-42 

* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _ACL_.
* Neamatullah et al. (2008) Ishna Neamatullah, Margaret M Douglass, H Lehman Li-wei, Andrew Reisner, Mauricio Villarroel, William J Long, Peter Szolovits, George B Moody, Roger G Mark, and Gari D Clifford. 2008. Automated de-identification of free-text medical records. _BMC medical informatics and decision making_, 8(1):32.
* Parada et al. (2011) Carolina Parada, Mark Dredze, and Frederick Jelinek. 2011. Oov sensitive named-entity recognition in speech. In _Twelfth Annual Conference of the International Speech Communication Association_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.
* Raymond (2013) Christian Raymond. 2013. Robust tree-structured named entities recognition from speech. In _Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on_, pages 8475-8479. IEEE.
* Ruch et al. (2000) Patrick Ruch, Robert H Baud, Anne-Marie Rassinoux, Pierrette Bouillon, and Gilbert Robert. 2000. Medical document anonymization with a semantic lexicon. In _Proceedings of the AMIA Symposium_, page 729. American Medical Informatics Association.
* Stubbs and Uzuner (2015) Amber Stubbs and Ozlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. _J. Biomed. Inf._, 58:20-29.
* Sudoh et al. (2006) Katsuhito Sudoh, Hajime Tsukada, and Hideki Isozaki. 2006. Incorporating speech recognition confidence into discriminative named entity recognition of speech data. In _Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics_, pages 617-624. Association for Computational Linguistics.
* Weinstein et al. (2014) Ronald S Weinstein, Ana Maria Lopez, Bellal A Joseph, Kristine A Erps, Michael Holcomb, Gail P Barker, and Elizabeth A Krupinski. 2014. Telemedicine, telehealth, and mobile health applications that work: opportunities and barriers. _The American journal of medicine_, 127(3):183-187.
* Yang and Garibaldi (2015) Hui Yang and Jonathan M Garibaldi. 2015. Automatic detection of protected health information from clinic narratives. _Journal of biomedical informatics_, 58:S30-S38.