Introduction
The vast amounts of data collected by healthcare providers in conjunction with modern data analytics present a unique opportunity to improve the quality and safety of medical care for patient benefit BIBREF1. Much recent research in this area has been on personalised medicine, with the aim to deliver improved diagnostic and treatment through the synergistic integration of datasets at the level of the individual. A different source of healthcare data pertains to organisational matters. In the United Kingdom, the National Health Service (NHS) has a long history of documenting the different aspects of healthcare provision, and is currently in the process of making available properly anonymised datasets, with the aim of leveraging advanced analytics to improve NHS services.
One such database is the National Reporting and Learning System (NRLS), a repository of patient safety incident reports from the NHS in England and Wales set up in 2003, which now contains over 13 million records. The incidents are reported under standardised categories and contain both organisational and spatio-temporal information (structured data) and a substantial component of free text (unstructured data) where incidents are described in the `voice' of the person reporting. The incidents are wide ranging: from patient accidents to lost forms or referrals; from delays in admission or discharge to serious untoward incidents, such as retained foreign objects after operations. The review and analysis of such data provides critical insight into complex processes in healthcare with a view towards service improvement.
Although statistical analyses are routinely performed on the structured data (dates, locations, hand-coded categories, etc), free text is typically read manually and often ignored in practice, unless a detailed review of a case is undertaken because of the severity of harm that resulted. These limitations are due to a lack of methodologies that can provide content-based groupings across the large volume of reports submitted nationally for organisational learning. Automatic categorisation of incidents from free text would sidestep human error and difficulties in assigning incidents to a priori pre-defined lists in the reporting system. Such tools can also offer unbiased insight into the root cause analysis of incidents that could improve the safety and quality of care and efficiency of healthcare services.
In this work, we showcase an algorithmic methodology that detects content-based groups of records in an unsupervised manner, based only on the free (unstructured) textual descriptions of the incidents. To do so, we combine deep neural-network high-dimensional text-embedding algorithms with graph-theoretical methods for multiscale clustering. Specifically, we apply the framework of Markov Stability (MS), a multiscale community detection algorithm, to sparsified graphs of documents obtained from text vector similarities. Our method departs both from traditional natural language processing tools, which have generally used bag-of-words (BoW) representation of documents and statistical methods based on Latent Dirichlet Allocation (LDA) to cluster documents BIBREF2, and from more recent approaches that have used deep neural network based language models, but have used k-means clustering without a graph-based analysis BIBREF3. Previous applications of network theory to text analysis have included the work of Lanchichinetti and co-workers BIBREF4, who proposed a probabilistic graph construction analysed with the InfoMap algorithm BIBREF5; however, their community detection was carried out at a single-scale and the BoW representation of text lacks the power of text embeddings. The application of multiscale community detection allows us to find groups of records with consistent content at different levels of resolution; hence the content categories emerge from the textual data, rather than from pre-designed classifications. The obtained results can help mitigate human error or effort in finding the right category in complex classification trees. We illustrate in our analysis the insight gained from this unsupervised, multi-resolution approach in this specialised corpus of medical records.
As an additional application, we use machine learning methods for the prediction of the degree of harm of incidents directly from the text in the NRLS incident reports. Although the degree of harm is recorded by the reporting person for every event, this information can be unreliable as reporters have been known to game the system, or to give different answers depending on their professional status BIBREF6. Previous work on predicting the severity of adverse events BIBREF7, BIBREF8 used reports submitted to the Advanced Incident Management System by Australian public hospitals, and used BoW and Support Vector Machines (SVMs) to detect extreme-risk events. Here we demonstrate that publicly reported measures derived from NHS Staff Surveys can help select ground truth labels that allow supervised training of machine learning classifiers to predict the degree of harm directly from text embeddings. Further, we show that the unsupervised clusters of content derived with our method improve the classification results significantly.
An a posteriori manual labelling by three clinicians agree with our predictions based purely on text almost as much as with the original hand-coded labels. These results indicate that incidents can be automatically classified according to their degree of harm based only on their textual descriptions, and underlines the potential of automatic document analysis to help reduce human workload.
Introduction ::: Data description
The full dataset includes more than 13 million confidential reports of patient safety incidents reported to the National Reporting and Learning System (NRLS) between 2004 and 2016 from NHS trusts and hospitals in England and Wales. Each record has more than 170 features, including organisational details (e.g., time, trust code and location), anonymised patient information, medication and medical devices, among many other details. In most records, there is also a detailed description of the incident in free text, although the quality of the text is highly variable.
The records are manually classified by operators according to a two-level system of incident types. The top level contains 15 categories including general classes such as `Patient accident', `Medication', `Clinical assessment', `Documentation', `Admissions/Transfer' or `Infrastructure', alongside more specific groups such as `Aggressive behaviour', `Patient abuse', `Self-harm' or `Infection control'.
Each record is also labelled based on the degree of harm to the patients as one of: `No Harm', `Low Harm', `Moderate Harm', `Severe Harm' or `Death'. These degrees are precisely defined by the WHO BIBREF9 and the NHS BIBREF10.
Graph-based framework for text analysis and clustering
Our framework combines text-embedding, geometric graph construction and multi-resolution community detection to identify, rather than impose, content-based clusters from free, unstructured text in an unsupervised manner.
Figure FIGREF2 shows a summary of our pipeline. First, we pre-process each document to transform text into consecutive word tokens, with words in their most normalised forms and some words removed if they have no distinctive meaning when used out of context BIBREF11, BIBREF12. We then train a paragraph vector model using the Document to Vector (Doc2Vec) framework BIBREF13 on the full set (13 million) of pre-processed text records. (Training a vector model on smaller sets of 1 million records also produces good results as seen in Table TABREF5). This training step of the text model is only done once.
The trained Doc2Vec model is subsequently used to infer high-dimensional vector descriptions for the text of each document in our target analysis set. We then compute a matrix containing all the pairwise (cosine) similarities between the Doc2Vec document vectors. This similarity matrix can be thought of as the adjacency matrix of a full, weighted graph with documents as nodes and edges weighted by their similarity. We sparsify this graph to the union of a minimum spanning tree and a k-Nearest Neighbors (MST-kNN) graph BIBREF14, a geometric construction that removes less important similarities but preserves global connectivity for the graph and, hence, for the dataset. The MST-kNN graph is then analysed with Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18, a multi-resolution graph partitioning method that identifies relevant subgraphs (i.e., clusters of documents) at different levels of granularity. MS uses a diffusive process on the graph to reveal the multiscale organisation at different resolutions without the need to choose a priori the number or type of clusters.
The partitions found by MS across levels of resolution are analysed a posteriori through visualisations and quantitative scores. The visualisations include: (i) word clouds to summarise the main content; (ii) graph layouts; and (iii) Sankey diagrams and contingency tables that capture correspondences between partitions. The quantitative scores include: (i) the intrinsic topic coherence (measured by the pairwise mutual information BIBREF19, BIBREF20); and (ii) the similarity to hand-coded categories (measured by the normalised mutual information BIBREF21).
Our framework also covers prediction of the degree of harm (DoH) caused to the patient usig text embeddings and the unsupervised cluster assignments obtaind from our multiscale graph partitioning. To perform this task, we use the hand-coded DoH from the NRLS to train three commonly used classifiers BIBREF22, BIBREF23 (Ridge, Support Vector Machine with a linear kernel, Random Forest) to predict the DoH using TF-iDF and Doc2Vec embeddings of the text and our MS cluster assignments. The classifiers are then evaluated in predicting the DoH using cross-validation.
We now explain the steps of the methodological pipeline in more detail.
Graph-based framework for text analysis and clustering ::: Text Preprocessing
Text preprocessing is important to enhance the performance of text embedding techniques. We applied standard preprocessing to the raw text of all 13 million records in our corpus, as follows. We divide our documents into iterative word tokens using the NLTK library BIBREF11 and remove punctuation and digit-only tokens. We then apply word stemming using the Porter algorithm BIBREF12, BIBREF24. If the Porter method cannot find a stemmed version for a token, we apply the Snowball algorithm BIBREF25. Finally, we remove any stop-words (repeat words with low content) using NLTK's stop-word list. Although pre-processing reduces some of the syntactic information, it consolidates the semantic information of the vocabulary. We note that the incident descriptions contain typos and acronyms, which have been left uncorrected to avoid manual intervention or the use of spell checkers, so as to mimic as closely as possible a realistic scenario.
Graph-based framework for text analysis and clustering ::: Text Vector Embedding
Computational text analysis relies on a mathematical representation of the base units of text (character $n$-grams, words or documents). Since our methodology is unsupervised, we avoid the use of labelled data, in contrast to supervised or semi-supervised classification methods BIBREF26, BIBREF27. In our work, we use a representation of text documents as vectors following recent developments in the field.
Traditionally, bag-of-words (BoW) methods represented documents as vectors of word frequencies weighted by inverse document frequency (TF-iDF). Such methods provide a statistical description of documents but they do not carry information about the order or proximity of words to each other and hence disregard semantic or syntactic relationships between words. In addition, BoW representations carry little information content as they tend to be high-dimensional and very sparse, due to the large size of word dictionaries and low frequencies of many terms.
Recently, deep neural network language models have successfully overcome the limitations of BoW methods by incorporating neighbourhoods in the mathematical description of each term. Distributed Bag of Words (DBOW), better known as Doc2Vec BIBREF13, is a form of Paragraph Vectors (PV) which creates a model that represents any word sequence (i.e. sentences, paragraphs, documents) as $d$-dimensional vectors, where $d$ is user-defined (typically $d=300$). Training a Doc2Vec model starts with a random $d$-dimensional vector assignment for each document in the corpus. A stochastic gradient descent algorithm iterates over the corpus with the objective of predicting a randomly sampled set of words from each document by using only the document's $d$-dimensional vector BIBREF13. The objective function being optimised by PV-DBOW is similar to the skip-gram model in Refs. BIBREF28, BIBREF29. Doc2Vec has been shown BIBREF30 to capture both semantic and syntactic characterisations of the input text, and outperforms BoW-based models such as LDA BIBREF2.
Benchmarking the Doc2Vec training: Here, we use the Gensim Python library BIBREF31 to train the PV-DBOW model. The Doc2Vec training was repeated several times with a variety of training hyper-parameters (chosen based on our own numerical experiments and the general guidelines provided by BIBREF32) in order to optimise the output. To characterise the usability and quality of models, we trained Doc2Vec models using text corpora of different sizes and content with different sets of hyper-parameters. . In particular, we checked the effect of corpus size by training Doc2Vec models on the full 13 million NRLS records and on randomly sampled subsets of 1 million and 2 million records.
Since our target analysis has heavy medical content and specific use of words, we also tested the importance of the training corpus by generating an additional Doc2Vec model using a set of 5 million articles from the English Wikipedia representing standard, generic English usage, which works well in the analysis of news articles BIBREF33.
The results in Table TABREF5 show that training on the highly specific text from the NRLS records is an important ingredient in the successful vectorisation of the documents, as shown by the degraded performance for the Wikipedia model across a variety of training hyper-parameters. On the other hand, reducing the size of the corpus from 13 million to 1 million records did not affect the benchmarking dramatically. This robustness of the results to the size of the training corpus was confirmed further with the use of more detailed metrics, as discussed below in Section SECREF27 (see e.g., Figure FIGREF29).
Based on our benchmarking, henceforth we use the Doc2Vec model trained on the 13+ million NRLS records with the following hyper-parameters: {training method = dbow, number of dimensions for feature vectors size = 300, number of epochs = 10, window size = 15, minimum count = 5, number of negative samples = 5, random down-sampling threshold for frequent words = 0.001 }. As an indication of computational cost, the training of this model takes approximately 11 hours (run in parallel with 7 threads) on shared servers.
Graph-based framework for text analysis and clustering ::: Similarity graph of documents from text similarities
Once the Doc2Vec model is trained, we use it to infer a vector for each record in our analysis subset and construct $\hat{S}$, a similarity matrix between the vectors by: computing the matrix of cosine similarities between all pairs of records, $S_\text{cos}$; transforming it into a distance matrix $D_{cos} = 1-S_{cos}$; applying element-wise max norm to obtain $\hat{D}=\Vert D_{cos}\Vert _{max}$; and normalising the similarity matrix $\hat{S} = 1-\hat{D}$ which has elements in the interval $[0,1]$.
This similarity matrix can be thought of as the adjacency matrix of a fully connected weighted graph. However, such a graph contains many edges with small weights reflecting the fact that in high-dimensional noisy data even the least similar nodes present a substantial degree of similarity. Indeed, such weak similarities are in most cases redundant and can be explained through stronger pairwise similarities. These weak, redundant edges obscure the graph structure, as shown by the diffuse visualisation in Figure FIGREF7A.
To reveal the graph structure, we sparsify the similarity matrix to obtain a MST-kNN graph BIBREF14 based on a geometric heuristic that preserves the global connectivity of the graph while retaining details about the local geometry of the dataset. The MST-kNN algorithm starts by computing the minimum spanning tree (MST) of the full matrix $\hat{D}$, i.e., the tree with $(N-1)$ edges connecting all nodes in the graph with minimal sum of edge weights (distances). The MST is computed using the Kruskal algorithm implemented in SciPy BIBREF34. To this MST, we add edges connecting each node to its $k$ nearest nodes (kNN) if they are not already in the MST. Here $k$ is an user-defined parameter that regulates the sparsity of the resulting graph. The binary adjacency matrix of the MST-kNN graph is Hadamard-multiplied with $\hat{S}$ to give the adjacency matrix $A$ of the weighted, undirected sparsified graph.
The network visualisations in Figure FIGREF7 give an intuitive picture of the effect of sparsification as $k$ is decreased. If $k$ is very small, the graph is very sparse but not robust to noise. As $k$ is increased, the local similarities between documents induce the formation of dense subgraphs (which appear closer in the graph visualisation layout). When the number of neighbours becomes too large, the local structure becomes diffuse and the subgraphs lose coherence, signalling the degradation of the local graph structure. Relatively sparse graphs that preserve important edges and global connectivity of the dataset (guaranteed here by the MST) have computational advantages when using community detection algorithms.
Although we use here the MST-kNN construction due to its simplicity and robustness, network inference, graph sparsification and graph construction from data is an active area of research, and several alternatives exist based on different heuristics, e.g., Graphical Lasso BIBREF35, Planar Maximally Filtered Graph BIBREF36, spectral sparsification BIBREF37, or the Relaxed Minimum Spanning Tree (RMST) BIBREF38. We have experimented with some of those methods and obtained comparable results. A detailed comparison of sparsification methods as well as the choice of distance in defining the similarity matrix $\hat{S}$ is left for future work.
Graph-based framework for text analysis and clustering ::: Multiscale Graph Partitioning
Community detection encompasses various graph partitioning approaches which aim to find `good' partitions into subgraphs (or communities) according to different cost functions, without imposing the number of communities a priori BIBREF39. The notion of community depends on the choice of cost function. Commonly, communities are taken to be subgraphs whose nodes are connected strongly within the community with relatively weak inter-community edges. Such structural notion is related to balanced cuts. Other cost functions are posed in terms of transitions inside and outside of the communities, usually as one-step processes BIBREF5. When transition paths of all lengths are considered, the concept of community becomes intrinsically multi-scale, i.e., different partitions are relevant at different time scales leading to a multi-level description dictated by the transition dynamics BIBREF15, BIBREF40, BIBREF16. This leads to the framework of Markov Stability (MS), a dynamics-based, multi-scale community detection methodology, which recovers several well-known heuristics as particular cases BIBREF15, BIBREF17, BIBREF18.
MS is an unsupervised community detection method that finds robust and stable partitions of a graph (and the associated communities) under the evolution of a continuous-time diffusion process without a priori choice of the number or type of communities or their relative relationships BIBREF15, BIBREF40, BIBREF16, BIBREF41 . In simple terms, MS can be understood by analogy to a drop of ink diffusing on the graph: the ink diffuses homogeneously unless the graph has intrinsic sub-structures, in which case the ink gets transiently contained, over particular time scales, within groups of nodes. The existence of such transients indicates a natural scale to partition the graph along the subgraphs (or communities) where the diffusion is transiently trapped. As the process continues to evolve, the ink diffuses out of those communities but might get transiently contained in other, larger subgraphs, if such multi-level structure exists. By analysing the Markov dynamics over time, MS detects the structure of the graph across scales. If a graph has no natural scales for partitioning, then MS returns no communities. The Markov time $t$ thus acts as a resolution parameter that allows us to extract robust partitions that persist over particular time scales, in an unsupervised manner.
Mathematically, given the adjacency matrix $A_{N \times N}$ of the graph obtained as described previously, let us define the diagonal matrix $D=\text{diag}(\mathbf {d})$, where $\mathbf {d}=A \mathbf {1}$ is the degree vector. The random walk Laplacian matrix is defined as $L_\text{RW}=I_N-D^{-1}A$, where $I_N$ is the identity matrix of size $N$ and the transition matrix (or kernel) of the associated continuous-time Markov process is $P(t)=e^{-t L_\text{RW}}, \, t>0$ BIBREF16. Any partition $\mathcal {H}$ into $C$ clusters is associated with a binary membership matrix $H_{N \times C}$ that maps the $N$ nodes into the clusters. Below, we will use the matrix $H$ to denote the corresponding partition $\mathcal {H}$. We can then compute the $C\times C$ clustered autocovariance matrix:
where $\pi $ is the steady-state distribution of the process and $\Pi =\text{diag}(\pi )$. The element $[R(t,H)]_{\alpha \beta }$ quantifies the probability that a random walker starting from community $\alpha $ at $t=0$ will be in community $\beta $ at time $t$, minus the probability that this event occurs by chance at stationarity.
The above definitions allow us to introduce our cost function measuring the goodness of a partition over time $t$, termed the Markov Stability of partition $H$:
A partition $H$ that maximises $r(t,H)$ is comprised of communities that preserve the flow within themselves over time $t$, since in that case the diagonal elements of $R(t,H)$ will be large and the off-diagonal elements will be small. For details, see BIBREF15, BIBREF40, BIBREF16, BIBREF42.
Our computational algorithm thus searches for partitions at each Markov time $t$ that maximise $r(t,H)$. Although the maximisation of (DISPLAY_FORM11) is an NP-hard problem (hence with no guarantees for global optimality), there are efficient optimisation methods that work well in practice. Our implementation here uses the Louvain Algorithm BIBREF43, BIBREF18 which is efficient and known to give good results when applied to benchmarks. To obtain robust partitions, we run the Louvain algorithm 500 times with different initialisations at each Markov time and pick the best 50 with the highest Markov Stability value $r(t,H)$. We then compute the variation of information BIBREF44 of this ensemble of solutions $VI(t)$, as a measure of the reproducibility of the result under the optimisation. In addition, we search for partitions that are persistent across time $t$, as given by low values of the variation of information between optimised partitions across time $VI(t,t^{\prime })$. Robust partitions are therefore indicated by Markov times where $VI(t)$ shows a dip and $VI(t,t^{\prime })$ has an extended plateau with low values, indicating consistency under the optimisation and validity over extended scales BIBREF42, BIBREF16. Below, we apply MS to find partitions across scales of the similarity graph of documents, $A$. The communities detected correspond to groups of documents with similar content at different levels of granularity.
Graph-based framework for text analysis and clustering ::: Visualisation and interpretation of the results
Graph layouts: We use the ForceAtlas2 BIBREF45 layout algorithm to represent graphs on the plane. This layout assigns a harmonic spring to each edge and finds through iterative rearrangements finds an arrangement on the plane that balances attractive and repulsive forces between nodes. Hence similar nodes tend to appear close together on this layout. We colour the nodes by either hand-coded categories (Figure FIGREF7) or multiscale MS communities (Figure FIGREF21). Spatially coherent colourings on this layout imply good clusters in terms of the similarity graph.
Tracking membership through Sankey diagrams: Sankey diagrams allow us to visualise the relationship of node membership across different partitions and with respect to the hand-coded categories. Two-layer Sankey diagrams (e.g., Fig. FIGREF22) reflect the correspondence between MS clusters and the hand-coded external categories, whereas we use a multilayer Sankey diagram in Fig. FIGREF21 to present the multi-resolution MS community detection across scales.
Normalised contingency tables: To capture the relationship between our MS clusters and the hand-coded categories, we also provide a complementary visualisation as z-score heatmaps of normalised contingency tables, e.g., Fig. FIGREF22. This allows us to compare the relative association of content clusters to the external categories at different resolution levels. A quantification of the overall correspondence is also provided by the $NMI$ score in Eq. (DISPLAY_FORM17).
Word clouds of increased intelligibility through lemmatisation: Our method clusters text documents according to their intrinsic content. This can be understood as a type of topic detection. To visualise the content of clusters, we use Word Clouds as basic, yet intuitive, summaries of information to extract insights and compare a posteriori with hand-coded categories. They can also provide an aid for monitoring results when used by practitioners.
The stemming methods described in Section SECREF3 truncate words severely to enhance the power of the language processing computational methods by reducing the redundancy in the word corpus. Yet when presenting the results back to a human observer, it is desirable to report the cluster content with words that are readily comprehensible. To generate comprehensible word clouds in our a posteriori analyses, we use a text processing method similar to the one described in BIBREF46. Specifically, we use the part of speech (POS) tagging module from NLTK to leave out sentence parts except the adjectives, nouns, and verbs. We also remove less meaningful common verbs such as `be', `have', and `do' and their variations. The remaining words are then lemmatised in order to normalise variations of the same word. Finally, we use the Python library wordcloud to create word clouds with 2 or 3-gram frequency list of common word groups.
Graph-based framework for text analysis and clustering ::: Quantitative benchmarking of topic clusters
Although our dataset has a classification hand-coded by a human operator, we do not use it in our analysis. Indeed, one of our aims is to explore the relevance of the fixed external classes as compared to content-driven groupings obtained in an unsupervised manner. Therefore we provide a double route to quantify the quality of the clusters by computing two complementary measures: (i) an intrinsic measure of topic coherence, and (ii) a measure of similarity to the external hand-coded categories.
Topic coherence of text: As an intrinsic measure of consistency of word association, we use the pointwise mutual information ($PMI$) BIBREF19, BIBREF47. The $PMI$ is an information-theoretical score that captures the probability of words being used together in the same group of documents. The $PMI$ score for a pair of words $(w_1,w_2)$ is:
where the probabilities of the words $P(w_1)$, $P(w_2)$, and of their co-occurrence $P(w_1 w_2)$ are obtained from the corpus. We obtain an aggregate $\widehat{PMI}$ for the graph partition $C=\lbrace c_i\rbrace $ by computing the $PMI$ for each cluster, as the median $PMI$ between its 10 most common words (changing the number of words gives similar results), and computing the weighted average of the $PMI$ cluster scores:
where $c_i$ denotes the clusters in partition $C$, each with size $n_i$, so that $N=\sum _{c_i \in C} n_i$ is the total number of nodes. Here $S_i$ denotes the set of top 10 words for cluster $c_i$.
The $PMI$ score has been shown to perform well BIBREF19, BIBREF47 when compared to human interpretation of topics on different corpora BIBREF48, BIBREF49, and is designed to evaluate topical coherence for groups of documents, in contrast to other tools aimed at short forms of text. See BIBREF26, BIBREF27, BIBREF50, BIBREF51 for other examples.
Here, we use the $\widehat{PMI}$ score to evaluate partitions without any reference to an externally labelled `ground truth'.
Similarity between the obtained partitions and the hand-coded categories: To quantify how our content-driven unsupervised clusters compare against the external classification, we use the normalised mutual information ($NMI$), a well-known information-theoretical score that quantifies the similarity between clusterings considering correct and incorrect assignments in terms of the information between the clusterings. The NMI between two partitions $C$ and $D$ of the same graph is:
where $I(C,D)$ is the Mutual Information and $H(C)$ and $H(D)$ are the entropies of the two partitions.
The $NMI$ is bounded ($0 \le NMI \le 1$) and a higher value corresponds to higher similarity of the partitions (i.e., $NMI=1$ when there is perfect agreement between partitions $C$ and $D$). The $NMI$ score is directly related to the V-measure in the computer science literature BIBREF52.
Graph-based framework for text analysis and clustering ::: Supervised Classification for Degree of Harm
As a further application of our work, we have carried out a supervised classification task aimed at predicting the degree of harm of an incident directly from the text and the hand-coded features (e.g., external category, medical specialty, location). A one-hot encoding is applied to turn these categorical values into numerical ones. We also checked if using our unsupervised content-driven cluster labels as additional features can improve the performance of the supervised classification.
The supervised classification was carried out by training on features and text three classifiers commonly applied to text classification tasks BIBREF22, BIBREF23: a Ridge classifier, Support Vector Machines with a linear kernel, and Random Forests. The goal is to predict the degree of harm (DoH) among five possible values (1-5). The classification is carried out with five-fold cross validation, using 80% of the data to train the model and the remaining 20% to test it. As a measure of performance of the classifiers and models, we use the weighted average of the F1 score for all levels of DoH, which takes into account both precision and recall, i.e., both the exactness and completeness of the model.
Application to the clustering of hospital incident text reports
We showcase our methodology through the analysis of the text from NRLS patient incident reports. In addition to textual descriptions, the reports are hand-coded upon reporting with up to 170 features per case, including a two-level manual classification of the incidents.
Here, we only use the text component and apply our graph-based text clustering to a set of 3229 reports from St Mary's Hospital, London (Imperial College Healthcare NHS Trust) over three months in 2014. As summarised in Figure FIGREF2, we start by training our Doc2Vec text embedding using the full 13+ million records collected by the NRLS since 2004 (although, as discussed above, a much smaller corpus of NRLS documents can be used). We then infer vectors for our 3229 records, compute the cosine similarity matrix and construct an MST-kNN graph with $k=13$ for our graph-based clustering. (We have confirmed the robustness of the MST-kNN construction in our data for $k>13$ by scanning values of $k \in [1,50]$, see Section SECREF27). We then applied Markov Stability, a multi-resolution graph partitioning algorithm to the MST-kNN graph. We scan across Markov time ($t \in [0.01, 100]$ in steps of 0.01). At each $t$, we run 500 independent Louvain optimisations to select the optimal partition found, as well as quantifying the robustness to optimisation by computing the average variation of information $VI(t)$ between the top 50 partitions. Once the full scan across $t$ is finalised, we compute $VI(t,t^{\prime })$, the variation of information between the optimised partitions found across the scan in Markov time, to select partitions that are robust across scales.
Application to the clustering of hospital incident text reports ::: Markov Stability extracts content clusters at different levels of granularity
Figure FIGREF21 presents a summary of our MS analysis. We plot the number of clusters of the optimal partition and the two metrics of variation of information across all Markov times. The existence of a long plateau in $VI(t,t^{\prime })$ coupled to a dip in $VI(t)$ implies the presence of a partition that is robust both to the optimisation and across Markov time. To illustrate the multi-scale features of the method, we choose several of these robust partitions, from finer (44 communities) to coarser (3 communities), obtained at five Markov times and examine their structure and content. The multi-level Sankey diagram summarises the relationship of the partitions across levels.
The MS analysis of the graph reveals a multi-level structure of partitions, with a strong quasi-hierarchical organisation. We remark that our optimisation does not impose any hierarchical structure a priori, so that the observed consistency of communities across levels is intrinsic to the data and suggests the existence of sub-themes that integrate into larger thematic categories. The unsupervised detection of intrinsic scales by MS enables us to obtain groups of records with high content similarity at different levels of granularity. This capability can be used by practitioners to tune the level of description to their specific needs, and is used below as an aid in our supervised classification task in Section SECREF4.
To ascertain the relevance of the layers of content found by MS, we examined the five levels of resolution in Figure FIGREF21. For each level, we produced lemmatised word clouds, which we used to generate descriptive content labels for the communities. We then compared a posteriori the content clusters with the hand-coded categories through a Sankey diagram and a contingency table. The results are shown in Figures FIGREF22–FIGREF25 for each of the levels.
The partition into 44 communities presents content clusters with well-defined characterisations, as shown by the Sankey diagram and the highly clustered structure of the contingency table (Figure FIGREF22). Compared to the 15 hand-coded categories, this 44-community partition provides finer groupings corresponding to specific sub-themes within the generic hand-coded categories. This is apparent in the hand-coded classes `Accidents', `Medication', `Clinical assessment', `Documentation' and `Infrastructure', where a variety of meaningful subtopics are identified (see Fig. FIGREF23 for details). In other cases, however, the content clusters cut across the external categories, e.g., the clusters on labour ward, chemotherapy, radiotherapy and infection control are coherent in content but can belong to several of the external classes. At this level of resolution, our algorithm also identified highly specific topics as separate content clusters, including blood transfusions, pressure ulcer, consent, mental health, and child protection, which have no direct relationship with the external classes provided to the operator.
Figure FIGREF24A and FIGREF24B present the results for two partitions at medium level of resolution, where the number of communities (12 and 17) is close to that of hand-coded categories (15). As expected from the quasi-hierarchy detected by our multi-resolution analysis, we find that the communities in the 17-way and 12-way partitions emerge from consistent aggregation of the smaller communities in the 44-way partition in Figure FIGREF22. Focussing on the 12-way partition, we see that some of the sub-themes in Figure FIGREF23 are merged into more general topics. An example is Accidents (community 2 in Fig. FIGREF24A), a merger of seven finer communities, which corresponds well with the external category `Patient accidents'. A similar phenomenon is seen for the Nursing cluster (community 1), which falls completely under the external category `Infrastructure'. The clusters related to `Medication' similarly aggregate into a larger community (community 3), yet there still remains a smaller, specific community related to Homecare medication (community 12) with distinct content. Other communities, on the other hand, still strand across external categories. This is clearly observable in communities 10 and 11 (Samples/ lab tests/forms and Referrals/appointments), which fall naturally across the `Documentation' and `Clinical Assessment'. Similarly, community 9 (Patient transfers) sits across the `Admission/Transfer' and `Infrastructure' external categories, due to its relation to nursing and hospital constraints. A substantial proportion of records was hand-coded under the generic `Treatment/Procedure' class, yet MS splits into into content clusters that retain medical coherence, e.g., Radiotherapy (Comm. 4), Blood transfusions (Comm. 7), IV/cannula (Comm. 5), Pressure ulcer (Comm. 8), and the large community Labour ward (Comm. 6).
The medical specificity of the Radiotherapy, Pressure ulcer and Labour ward clusters means that they are still preserved as separate groups to the next level of coarseness in the 7-way partition (Figure FIGREF25A). The mergers in this case lead to a larger communities referring to Medication, Referrals/Forms and Staffing/Patient transfers. Figure FIGREF25B shows the final level of agglomeration into 3 content clusters: records referring to Accidents; a group broadly referring to matters Procedural (referrals, forms, staffing, medical procedures) cutting across external categories; and the Labour ward cluster, still on its own as a subgroup with distinctive content.
This process of agglomeration of content, from sub-themes into larger themes, as a result of the multi-scale hierarchy of MS graph partitions is shown explicitly with word clouds in Figure FIGREF26 for the 17-, 12- and 7-way partitions. Our results show good overall correspondence with the hand-coded categories across resolutions, yet our results also reveal complementary categories of incidents not defined in the external classification. The possibility of tuning the granularity afforded by our method can be used to provide a distinct level of resolution in certain areas corresponding to specialised or particular sub-themes.
Application to the clustering of hospital incident text reports ::: Robustness of the results and comparison with other methods
We have examined quantitatively the robustness of the results to parametric and methodological choices in different steps of our framework. Specifically, we evaluate the effect of: (i) using Doc2Vec embeddings instead of BoW vectors; (ii) the size of corpus for training Doc2Vec; (iii) the sparsity of the MST-kNN graph construction. We have also carried out quantitative comparisons to other methods for topic detection and clustering: (i) LDA-BoW, and (ii) several standard clustering methods.
Doc2Vec provides improved clusters compared to BoW: As compared to standard bag of words (BoW), fixed-sized vector embeddings (Doc2Vec) produces lower dimensional vector representations with higher semantic and syntactic content. Doc2Vec outperforms BoW representations in practical benchmarks of semantic similarity and is less sensitive to hyper-parameters BIBREF30. To quantify the improvement provided by Doc2Vec, we constructed a MST-kNN graph from TF-iDF vectors and ran MS on this TF-iDF similarity graph. Figure FIGREF28 shows that Doc2Vec outperforms BoW across all resolutions in terms of both $NMI$ and $\widehat{PMI}$ scores.
Robustness to the size of the Doc2Vec training dataset : Table TABREF5 indicates a small effect of the size of the training corpus on the Doc2Vec model. To confirm this, we trained two additional Doc2Vec models on sets of 1 million and 2 million records (randomly chosen from the full 13+ million records) and followed the same procedure to construct the MST-kNN graph and carry out the MS analysis. Figure FIGREF29 shows that the performance is affected only mildly by the size of the Doc2Vec training set.
Robustness to the level of graph sparsification:
We sparsify the matrix of cosine similarities using the MST-kNN graph construction. The smaller the value of $k$, the sparser the graph. Sparser graphs have computational advantages for community detection algorithms, but too much sparsification degrades the results. Figure FIGREF30 shows the effect of sparsification in the graph construction on the performance of MS clusters. Our results are robust to the choice of $k$, provided it is not too small: both the $NMI$ and $\widehat{PMI}$ scores reach a similar level for values of $k$ above 13-16. Due to computational efficiency, we favour a relatively small value of $k=13$.
Comparison of MS partitions to Latent Dirichlet Allocation with Bag-of-Words (LDA-BoW): We have compared the MS results to LDA, a widely used methodology for text analysis. A key difference in LDA is that a different model needs to be trained when the number of topics changes, whereas our MS method produces clusterings at all levels of resolution in one go. To compare the outcomes, we trained five LDA models corresponding to the five MS levels in Figure FIGREF21. Table TABREF31 shows that MS and LDA give partitions that are comparably similar to the hand-coded categories (as measured with $NMI$), with some differences depending on the scale, whereas the MS clusters have higher topic coherence (as given by $\widehat{PMI}$) across all scales.
To give an indication of computational cost, we ran both methods on the same servers. Our method takes approximately 13 hours in total (11 hours to train the Doc2Vec model on 13 million records and 2 hours to produce the full MS scan with 400 partitions across all resolutions). The time required to train just the 5 LDA models on the same corpus amounts to 30 hours (with timings ranging from $\sim $2 hours for the 3 topic LDA model to 12.5 hours for the 44 topic LDA model). This comparison also highlights the conceptual difference between our multi-scale methodology and LDA topic modelling. While LDA computes topics at a pre-determined level of resolution, our method obtains partitions at all resolutions in one sweep of the Markov time, from which relevant partitions are chosen based on their robustness. The MS partitions at all resolutions are available for further investigation if so needed.
Comparison of MS to other partitioning and community detection algorithms: We have partitioned the same kNN-MST graph using several well-known algorithms readily available in code libraries (i.e., the iGraph module for Python): Modularity Optimisation BIBREF53, InfoMap BIBREF5, Walktrap BIBREF54, Label Propagation BIBREF55, and Multi-resolution Louvain BIBREF43. Note that, in contrast with our multiscale MS analysis, these methods give just one partition at a particular resolution (or two for the Louvain implementation in iGraph). Figure FIGREF32 shows that MS provides improved or equal results to all those other graph partitioning methods for both $NMI$ and $\widehat{PMI}$ across all scales. Only for very fine resolution (more than 50 clusters) does Infomap, which partitions graphs into small clique-like subgraphs BIBREF40, BIBREF56, provide a slightly improved $NMI$. Therefore, MS finds both relevant and high quality clusterings across all scales by sweeping the Markov time parameter.
Using free-text descriptions to predict the degree of harm of patient safety incidents with a supervised classifier
Here we approach the task of training a supervised classifier that predicts the degree of harm of an incident based on other features of the record (such as location, external category, and medical specialty) and on the textual component of the report. To this end, we use the embedded text vectors and MS cluster labels of the records as features to predict the degree of harm to the patient.
Each NRLS record has more than 170 features filled manually by healthcare staff, including the degree of harm (DoH) to the patient, a crucial assessment of the reported incident. The incident is classified into five levels: 'No harm', 'Low', 'Moderate', 'Severe', and 'Death'. However, the reported DoH is not consistent across hospitals and can be unreliable BIBREF6.
The lack of reliability of the recorded DoH poses a challenge when training supervised models. Given the size of the dataset, it is not realistic to ask medics to re-evaluate incidents manually. Instead, we use the publicly available `Learning from mistakes league table' based on NHS staff survey data to identify organisations (NHS Trusts) with `outstanding' (O) and `poor reporting culture' (PRC). Our hypothesis is that training our classifiers on records from organisations with better rankings in the league table should lead to improved prediction. If there is a real disparity in the manual classification among organisations, only incidents labelled by O-ranked Trusts should be regarded as a `ground truth'.
Using free-text descriptions to predict the degree of harm of patient safety incidents with a supervised classifier ::: Supervised classification of degree of harm
We study NRLS incidents reported between 2015 and 2017 from O-ranked and PRC-ranked Trusts. The 2015-17 NRLS dataset is very unbalanced: there are 2,038,889 “No harm” incidents against only 6,754 “Death” incidents. To tackle this issue, we sample our dataset as recommended by BIBREF8, and randomly select 1,016 records each of `No harm' , `Low', and `Moderate', and 508 records each of `Severe' and `Death' incidents, from each type of Trust. We thus obtain two datasets (O and PRC) consisting of a total of 4,064 incidents each.
For each dataset (O and PRC), we train three classifiers (Ridge, Support Vector Machine with a linear kernel, and Random Forest) with five-fold cross validation, and we compute the F-1 scores of each fold to evaluate the model performance. We first train models using three categories from the reports: location (L), external hand-coded category (C), and medical specialty (S). We also compute the performance of models trained on text features, both TF-iDF and Doc2Vec. We also study models trained on a mixture of text and categories. Finally, we run Markov Stability as described above to obtain cluster labels for each dataset (O and PRC) at different resolutions (70, 45, 30 and 13 communities). We then evaluate if it is advantageous to include the labels of the MS clusters as additional features.
Table TABREF34 presents the results of our numerical experiments. Our first observation is that, for this data, SVM with linear kernel has the best performance (similar to Ridge), and Random Forests perform poorly in general. There are several conclusions from our study. First, there is a consistent difference between the scores of the O and PRC datasets (ranging from 1.7% to 11.2% for an average of 5.6%), thus confirming our hypothesis that automated classification performs better when training with data from organizations with better rankings in the league table. Second, using text features is highly advantageous in predicting the degree of harm compared to category alone: there is a substantial increase of up to 100% in the F1 score between column 1 (all three categories) and column 2 (Tf-iDF). Furthermore, adding categorical features (L, C, or S) to the TF-iDF text features improves the scores only marginally (around 2%), as seen by comparing columns 3–6 with column 2.
Given the demonstrated importance of text, we studied the effect of using more refined textual features for classification. In columns 7-10, we considered the effect of adding to TF-iDF the MS labels extracted from our text analysis (as described above), and we find a larger improvement of around 7% with respect to mere TF-iDF (column 2). The improvement is larger for finer clusterings into 70 and 45 communities, which contain enough detail that can be associated with levels of risk (e.g., type of accident). This supports the value of the multi-resolution groupings we have extracted through our analysis.
We also studied the impact of using Doc2Vec vectors as features. Interestingly, the comparison between columns 2 and 11 shows that there is only a slight improvement of 2% when using Doc2Vec instead of TF-iDF features for the case of records from O-ranked institutions, but the improvement is of 12% for the records from PRC Trusts. This differences suggests that the usage of terms is more precise in O-ranked hospitals so that the differences between TF-iDF are minimised, while the advantages of the syntactic and semantic reconstruction of the Doc2Vec embedding becomes more important in the case of PRC Trusts.
Based on these findings, we build our final model that uses a Support Vector Machine classifier with both Doc2Vec embeddings and the MS labels for 30 content clusters (encoded via a One-Hot encoder) as features. We choose to keep only 30 communities as this performs well when combined with the Doc2Vec embedding (without slowing too much the classifier). We performed a grid search to optimise the hyperparameters of our model (penalty = 10, tolerance for stopping criterion = 0.0001, linear kernel). For the O-ranked records, our model achieves a weighted F1 score of 0.657, with a 19% improvement with respect to TF-iDF text features and a 107% improvement with respect to categorical features. (For the PRC records, the corresponding improvements are 33% and 215%, respectively.) Note that similar improvements are also obtained for the other classifiers when using Doc2Vec and MS labels as features. It is also worth noting that the differences in the prediction of DoH between PRC and O-ranked records is reduced when using text tools and, specifically, the F1-score of the SVM classifier based on Doc2Vec with MS is almost the same for both datasets. Hence the difference in the quality of the reporting categories can be ameliorated by the use of the textual content of the reports. We summarise the main comparison of the performance of the SVM classifier based on categorical, raw text, and text with content for both datasets in Figure FIGREF35.
Examination of the types of errors and ex novo re-classification by clinicians:
A further analysis of the confusion matrices used to compute the F1 score reveals that most of the errors of our model are concentrated in the `No harm', `Low harm' and `Moderate harm' categories, whereas fewer errors are incurred in the `Severe harm' and `Death' categories. Therefore, our method is more likely to return false alarms rather than missing important and harmful incidents.
In order to have a further evaluation of our results, we asked three clinicians to analyse ex novo a randomly chosen sample of 135 descriptions of incidents, and to determine their degree of harm based on the information in the incident report. The sample was selected from the O-ranked dataset and no extra information apart from the text was provided. We then compared the DoH assigned by the clinicians with both the results of our classifier and the recorded DoH in the dataset.
Remarkably, the agreement rate of the clinicians' assessment with the recorded DoH was surprisingly low. For example, the agreement in the `No Harm' incidents was only 38%, and in the `Severe' incidents only 49%. In most cases, though, the disparities amounted to switching the DoH by one degree above or below. To reduce this variability, we analysed the outcomes in terms of three larger groups: `No Harm' and `Low Harm' incidents were considered as one outcome; `Moderate Harm' was kept separate; and `Severe Harm' and `Death' were grouped as one outcome, since they both need to be notified to NHS safety managers.
The results are presented in Table TABREF36. Our classification agrees as well as the pre-existing DoH in the dataset with the ex novo assessment of the clinicians, but our method has higher agreement in the severe and deadly incidents. These results confirm that our method performs as well as the original annotators but is better at identifying risky events.
Discussion
We have applied a multiscale graph partitioning algorithm (Markov Stability) to extract content-based clusters of documents from a textual dataset of incident reports in an unsupervised manner at different levels of resolution. The method uses paragraph vectors to represent the records and analyses the ensuing similarity graph of documents through multi-resolution capabilities to capture clusters without imposing a priori their number or structure. The different levels of resolution found to be relevant can be chosen by the practitioner to suit the requirements of detail for each specific task. For example, the top level categories of the pre-defined classification hierarchy are highly diverse in size, with large groups such as `Patient accident', `Medication', `Clinical assessment', `Documentation', `Admissions/Transfer' or `Infrastructure' alongside small, specific groups such as `Aggressive behaviour', `Patient abuse', `Self-harm' or `Infection control'. Our multi-scale partitioning finds additional subcategories with medical detail within some of the large categories (Fig. FIGREF22 and FIGREF23).
Our a posteriori analysis showed that the method recovers meaningful clusters of content as measured by the similarity of the groups against the hand-coded categories and by the intrinsic topic coherence of the clusters. The clusters have high medical content, thus providing complementary information to the externally imposed classification categories. Indeed, some of the most relevant and persistent communities emerge because of their highly homogeneous medical content, even if they cannot be mapped to standardised external categories.
An area of future research will be to confirm if the finer unsupervised cluster found by our analysis are consistent with a second level in the hierarchy of external categories (Level 2, around 100 categories), which is used less consistently in hospital settings. The use of content-driven classification of reports could also be important within current efforts by the World Health Organisation (WHO) under the framework for the International Classification for Patient Safety (ICPS) BIBREF9 to establish a set of conceptual categories to monitor, analyse and interpret information to improve patient care.
We have used our clusters within a supervised classifier to predict the degree of harm of an incident based only on free-text descriptions. The degree of harm is an important measure in hospital evaluation and has been shown to depend on the reporting culture of the particular organisation. Overall, our method shows that text description complemented by the topic labels extracted by our method show improved performance in this task. The use of such enhanced NLP tools could help improve reporting frequency and quality, in addition to reducing burden to staff, since most of the necessary information can be retrieved automatically from text descriptions. Further work, would aim to add interpretability to the supervised classification BIBREF57, so as to provide medical staff with a clearer view of the outcomes of our method and to encourage its uptake.
One of the advantages of a free text analytical approach is the provision, in a timely manner, of an intelligible description of incident report categories derived directly from the 'words' of the reporters themselves. Insights from the analysis of such free text entries can add rich information than would have not otherwise been obtained from pre-defined classes. Not only could this improve the current state of play where much of the free text of these reports goes unused, but by avoiding the strict assignment to pre-defined categories of fixed granularity free text analysis could open an opportunity for feedback and learning through more nuanced classifications as a complementary axis to existing approaches.
Currently, local incident reporting systems used by hospitals to submit reports to the NRLS require risk managers to improve data quality, due to errors or uncertainty in categorisation. The application of free text analytical approaches has the potential to free up time from this labour-intensive task, focussing instead in quality improvement derived from the content of the data itself. Additionally, the method allows for the discovery of emerging topics or classes of incidents directly from the data when such events do not fit existing categories by using methods for anomaly detection to decide whether new topic clusters should be created. This is a direction of future work.
Further work also includes the use of our method to enable comparisons across healthcare organisations and also to monitor changes in their incident reports over time. Another interesting direction is to provide online classification suggestions to users based on the text they input as an aid with decision support and data collection, which can also help fine-tune the predefined categories. Finally, it would be interesting to test if the use of deep learning algorithms can improve our classification scores.
We thank Elias Bamis, Zijing Liu and Michael Schaub for helpful discussions. This research was supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translational Research Centre and NIHR Imperial Biomedical Research Centre. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, or the Department of Health. All authors acknowledge support from the EPSRC through award EP/N014529/1 funding the EPSRC Centre for Mathematics of Precision Healthcare.

Question: How are content clusters used to improve the prediction of incident severity?
Answer: they are used as additional features in a supervised classification task

Question: What cluster identification method is used in this paper?
Answer: A combination of Minimum spanning trees, K-Nearest Neighbors and Markov Stability BIBREF15, BIBREF16, BIBREF17, BIBREF18