Mining and Classifying Medical Documents | by Georgi Tancev | Towards Data Science

In the medical world, a lot of (digital) text documents from several specialities are generated, be it patient health records, letters, or documentation of clinical studies. In fact, text data, which is usually unstructured, contributes to the huge increase of data volume globally — social media alone. Hence, retrieving information from texts becomes a more and more important task.
The aim of this post is two-fold:
demonstrating methods for text mining and document classification used in natural languague processing (NLP)presenting the development and deployment of an application for automatized text classifier generation in Python using Scikit-Learn and Streamlit.
demonstrating methods for text mining and document classification used in natural languague processing (NLP)
presenting the development and deployment of an application for automatized text classifier generation in Python using Scikit-Learn and Streamlit.
The final product will look like this application with the name Medical Language Model Learner (MLML). (Raw data is available on kaggle.)
Natural language processing (NLP) is a subfield of linguistics, computer science, information engineering, and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.
An important consideration is that text consists of strings, but mathematical models rely on numbers, and there are different ways to convert words to numeric vectors. Addtionally, there are different possibilities to model a language — depending on the problem definition. A common task is mapping sequences (e.g. text/speech in some language) from some domain to sequences in another domain (e.g. text/speech in another languague), known as sequence-to-sequence models. For instance, in order to generate speech from text (text-to-speech), recurrent neural networks together with pretrained word embeddings (for example Global Vectors for Word Representation, GloVe) are used. In that case, a word becomes a vector in a n-dimensional space, similar to data points in a space with n principal components.
The different dimensions (also known as latent variables) can be thought as properties/attributes that describe the words, e.g. the property “edible” for words like bread, fruit, and apple; words like chair, car, and animal would be found on the opposite direction. Hence, “apple” and “chair” would be farther apart in this dimension, but “apple” and “fruit” much closer. As for the word “animal”, this would be somewhere in the middle, because some animals are eaten. Note that it is generally not possible to extract the meaning of the different latent variables.
Due to these embeddings, the dimensionality is reduced compared to a 1-out-of-K encoded approach. For instance, a dictionary with K words has K dimensions in a 1-out-of-K space (every word is a vector orthogonal to all other words), but an embedding can reduce the dimension down to 100, and hence decrease computational complexity. (If K=10.000 and n=100, the compression factor is 100.)
Pretrained embeddings can be downloaded and do not have to be constructed first. However, every possible word has to be available in the embedding already — or the embedding has to be trained first. The embeddings are context-free, i.e. the word “bank” is treated similar in expressions like “bank account” or “he sat on a bank”.
Alternatively, there is the possibility to ignore the sequential aspect of a language and model a text as a collection of words (bag-of-words approach). This is the most frequent representation in text mining, which is the process of deriving high-quality information from text. To describe a text in this manner, a dictionary is constructed by using the most frequent words or n-grams (raw count or normalized by the total amount of words in a text), but such a dictionary might not very discriminative as every text contains expressions like “a”, “the”, “is”, and so on.
To account for this issue, a solution called term frequency-inverse document frequency (tf-idf or TFIDF) was developed; tf-idf is a numerical statistic that is intended to reflect how important a word is to a document in a collection of documents. The tf-idf value increases proportionally to the number of times a word appears in the document and is modified by the number of documents in the corpus that contain the word, which helps to adjust for the fact that some words appear more frequently in general. tf-idf is one of the most popular term-weighting schemes today; 83% of text-based recommender systems in digital libraries use tf–idf.
The term frequency (calculated for each document) is usually just the count of a specific word i (e.g. “cervix”) divided by the total amount of words in a text L, hence adjusting for document length.
Inverse document frequency is obtained by calculating the inverse fraction of documents (N/ni) which contain the word i and applying a log transformation. Words that occur in almost every document will have a value close to one before the logarithmic transformation, and hence close to zero after logarithmic transformation.
The tf-idf is defined for each word i in each document k in a collection of N documents. It is calculated by multiplying tf and idf. This way, a word i that occurs in almost every document k lowers the tf-idf value. For text mining, it does not make sense to keep words in the dictionary with low tf-idf values since they are not discriminative for a specific document class.
Imagine a data scientist who wants to build a model that distinguishes between biological and legal documents; what words should (s)he focus on? Ubiquitous words like “is” or “a” do not help at all, but terms like “protein” or “contract” do — these are the interesting ones. If the model should distinguish even further between subclasses of biological documents (higher resolution), it needs to refine the dictionary. For instance, words like “DNA”, “enzyme”, or “pathway” are used more frequently in microbiology than in ecology, whereas the opposite is true for words like “biodiversity”, “habitat”, or “population”. If said higher resolution is desired, bigger dictionaries might be needed. Or, one has to be more selective in which maximum and minimum tf-idf values one wants to include in the dictionary of size D.
The result of the tf-idf computation is a matrix with dimensions NxD, i.e. number of documents times dictionary size, filled with tf-idf values that serve as features. Note that this is likely a sparse matrix. Additionally, there are tunable parameters in the matrix construction process, and it is evident that the tf-idf transformation has some influence with respect to the model performance.
Scikit-Learn as a built-in transformer for text documents.
import pandas as pdfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.preprocessing import LabelEncoderenc = LabelEncoder()# Load data.data = pd.read_csv("mtsamples.csv", index_col=0, usecols=[0,1,2,4])data = data.dropna()samples = data.transcriptiontext_labels  = [label_name.lower() for label_name in data.medical_specialty]labels = enc.fit_transform(np.array(text_labels))# Transform data.max_df = 0.2min_df = 0.001max_features = 1000ngram_range = (1,1)tfidf_vectorizer = TfidfVectorizer(max_df=max_df, min_df=min_df, max_features=max_features, stop_words='english', ngram_range=ngram_range)tfidf = tfidf_vectorizer.fit_transform(samples)feature_names = tfidf_vectorizer.get_feature_names()
Since there are several free parameters, Streamlit can be used to build sliders to change the values and try out different combinations manually; this way, many different options can be evaluated quickly.
import streamlit as stfrom streamlit.logger import get_logger# Title.st.sidebar.header("Constructing dictonary of words.")# Upper bound for tf-idf value.max_df = st.sidebar.slider("Maximum tf-idf value for a word to be considered.", min_value=0.05, max_value=0.4, value=0.3, step=0.01)# Lower bound for tf-idf value.min_df = st.sidebar.slider("Minimum tf-idf value for a word to be considered.", min_value=0.00, max_value=0.05, value=0.01, step=0.01)# Size of dictionary.max_features = st.sidebar.slider("Size of dictionary.", min_value=100, max_value=1000, value=500, step=100)
The final product looks should look like this.
One possible application is the classification of medical documents according to their medical speciality. Once the document-word matrix is constructed and the texts are transformed into mathematical representations, they can be visualized using dimensionality reduction methods, e.g. truncated singular value decomposition, and a charting library, e.g. Altair.
from sklearn.decomposition import TruncatedSVDimport altair as alt# Dimensionality reduction.dim_red = TruncatedSVD(n_components=2)data_red = dim_red.fit_transform(tfidf)# Plot.scatter = alt.Chart(data_red,title="dimensionality reduction",height=400).mark_circle().encode(x='principal component 1', y='principal component 2', color=alt.Color('class', scale=alt.Scale(scheme='blues')),tooltip=["class"]).interactive()st.altair_chart(scatter)
The figure shows the different documents in the space of the two principal components explaining the highest variance in the original tf-idf space. It can already be seen that there is a lot of overlap between the classes — at least with the constructed dictionary — and a low classifier performance is to be expected.
The data set contains 40 classes. However, it is not (yet) known how frequent a particular class is in the data set. For example, one can expect classes of low abundance to be misclassified more frequently in a skewed data set, so it might be reasonable to discard or merge rare classes.
To build the text model, any algorithm for classification can be used, e.g. a random forest; compared to logistic regression, a random forest has the advantage that it can construct decision boundaries of any shape without basis expansion. To find the best random forest hyperparameters, a grid search can be performed.
from sklearn.ensemble import RandomForestClassifier# Number of trees.n_estimators = 1000# Define classifier.forest_clf = RandomForestClassifier(n_estimators=n_estimators, max_depth=None, max_leaf_nodes=None, class_weight='balanced', oob_score=True, n_jobs=-1, random_state=0)# Define grid.parameters = {'max_leaf_nodes':np.linspace(20,35,14,dtype='int')}# Balanced accuracy as performance measure.clf = RandomizedSearchCV(forest_clf, parameters, n_iter=10, cv=3,iid=False, scoring='accuracy',n_jobs=-1)# Train/optimize classifier.classifier = clf.fit(tfidf, labels)# Retrieve optimum.forest = classifier.best_estimator_feature_importances = forest.feature_importances_indices = np.argsort(feature_importances)[::-1]
It could be again interesting to tune these hyperparameters manually. (In the current version, there is only a text input container, so one has to convert the strings to integers.)
st.sidebar.header("Customizing the model.")n_estimators = st.sidebar.text_input('Number of trees in random forest.', '1000')max_leaf_nodes = st.sidebar.text_input('Maximum number of leaf nodes in a tree.', '25')max_depth = st.sidebar.text_input('Maximum depth of a tree.', '5')class_weight = st.sidebar.selectbox("Class weights for the model.",('balanced','balanced_subsample'))
To interpret the model, feature importances obtained from the random forest should be evaluated after training with a bar chart.
# Retrieve values.feature_importance = classifier.feature_importances_# Plot.bars = alt.Chart(feature_importance, height=400, title="discriminative power of features").mark_bar(color='steelblue', opacity=0.7).encode(y='features:N', x='feature importance:Q', tooltip="feature importance")st.altair_chart(bars)
It can be seen that most importances are of similar magnitude. The overall performance as determined by the out-of-bag score is low (0.41). It is likely that there is too much overlap between the classes, which is supported by the dimensionality reduction.
To evaluate the model with respect to precision and recall, F1 score and confusion matrix are computed.
from sklearn.metrics import f1_score, confusion_matrix# Retrieve values.y_true = labelsy_pred = classifier.predict(tfidf)# Compute scores.f1_score_ = f1_score(y_true,y_pred,average="weighted")cm = confusion_matrix(y_true,y_pred)# Plot.heat = alt.Chart(source, height=500, title="confusion matrix").mark_rect(opacity=0.7).encode(x='predicted class:N', y='true class:N', color=alt.Color('count:Q', scale=alt.Scale(scheme='blues')), tooltip="count")st.altair_chart(heat)
The F1 score is 0.49, so precision and/or recall are low. From the confusion matrix, one can see that the model has difficulties in distinguishing the medical documents. It seems that the used words between the documents are too similar at the moment, i.e. many identical words are used in the different medical specialities — or there are just no patterns to be found.
Obviously, the model can now be used for prediction. In that case, a sample has to be provided as input, transformed into tf-idf format, and fed to the classifier, which will then return a probability prediction that this sample belongs to any of the classes.
The code inside the app should look something like this:
LOGGER = get_logger(__name__)def run():   # code for appif __name__ == "__main__":    run()
To test it locally, the terminal is opened and rooted to the location with the “app.py” file. Typing the following code start a local testing session.
streamlit run app.py
To deploy the application on a Cloud provider such as Heroku, an account and git are needed. First, a shell script with file name “setup.sh” and the following content is created.
mkdir -p ~/.streamlit/echo "\[general]\n\email = \"email@website.com\"\n\" > ~/.streamlit/credentials.tomlecho "\[server]\n\headless = true\n\enableCORS=false\n\port = $PORT\n\" > ~/.streamlit/config.toml
Next, a requirements.txt file is created. Its content should look like this.
pandassklearnnumpystreamlitaltair
For Heroku, a so-called Procfile with the following line of code has to be written — assuming the app is called “app.py”. Note that the Procfile has no type specification.
web: sh setup.sh && streamlit run app.py
The app can be deployed in the terminal (here on a Unix/Mac operating system). For this, the shell is rooted to the director with all files.
$ cd Documents/Projects/App
Next, an app on Heroku has to be created (here named “myapp”). After that, a git repository is initialized in the directory with the “app.py” file.
$ git init$ heroku git:remote -a myapp
The code is commited to the repository and deployed on Heroku.
$ git add .$ git commit -am "make it better"$ git push heroku master
Finally, the application should be online.