Doctor Bayes
Predicting Diseases from Symptoms
Anthony Perez, Brandon Beckhardt, Leo Keselman
Introduction
According to recent OECD estimates, healthcare
expenditures for developed countries account for 10% to 20%of their national economies. In order to alleviate the impact ofhealthcare spending on the the “last mile” of medicine, webuilt a system to predict a user’s illness simply based on a
description of their systems.
Originally we’d hoped to develop a classification system
on top of electronic medical records, clinician notes, and theICD-10 disease classification system. We were unable to findpublically available EMRs so instead we created Doctor
Bayes, an illness classification system whose education
comes only from reading about medicine on the web.

Data Sources
We collected data from three sources:● Mayo Clinic● Freebase● Wikipedia
The Freebase and Mayo Clinic sources each contain a list ofsymptoms and a description for each disease. Our training
data was made from combinations of the Freebase and Mayo
Clinic data (see the results section). The Wikipedia data ismore free form and as a result was used as testing data. We
tested on either the sentences describing the symptoms of thedisease or the main section of the article.

Algorithms and Learning Models

Examples Queries

There are three main sections in our pipeline: training data sources, feature extraction, and model selection.
We have six ways to partition our data (symptoms and descriptions from each of our three data sources), six methodsof manipulating our features (described below), and several models (four of which we present below).
Our primary algorithm was a Multinomial Naive Bayes with Laplace smoothing due to its success in NLP processing
tasks. We couldn’t find a comprehensive list with occurrences of diseases for our priors, so our priors are based on thenumber of Google results associated with a given disease.
All of our feature manipulation is aimed at reducing the feature space because we identified this problem as having highvariance (being prone to overfitting). We also see increasing the data we give our models reduces test error, which isexpected with a high variance problem.

We’ve implemented baseline latentsemantic analysis across ourdataset, along with training
word2vec vectors on our combined
dataset (whose t-SNE visualizationis shown to the right)

1, Alzheimer's disease
6.8e-06, Transient global amnesia
2.1e-06, Dyslexia
9.7e-07, Amnesia
5.8e-08, Social anxiety disorder>> I shake a lot and have trouble
balancing.
1, Parkinson's disease
0.0001, Generalized anxiety disorder
4.2e-05, Primary aldosteronism
1.4e-05, Social anxiety disorder
5.6e-06, Alzheimer's disease>> Redness, itchiness and difficulty
opening eyes in the morningResults
Data Set Error Analysis

Feature Manipulations Error AnalysisUsing all of our feature manipulations with the given
training data, these are the top five accuracy results.

Using all of our data sources with the given featuremanipulations, these are the top five accuracy results.

1, Conjunctivitis (Pink Eye)
6.6e-08, Glaucoma
7.7e-10, Dermatitis
1.6e-10, Blepharitis
8.3e-11, Sleepwalking>> Fever headache and with a rash on
my wrists.
1, Rocky Mountain spotted fever
3.4e-05, Meningitis
2e-06, Lyme disease
1.8e-06, Rheumatic fever
1.4e-06, Dengue fever>> Severe throat pain with difficultyswallowing and swollen tonsils.
1, Streptococcal pharyngitis (Strep Throat)
2.9e-06, Diphtheria
4.5e-08, Infectious mononucleosis
1.9e-08, Laryngeal cancer
1.7e-09, Tularemia

Word & Document Similarity
When working across 3 free text
datasets, each has different ways ofdescribing identical symptoms anddiseases. We used Freebase’s
incomplete alias lists to help tie ourdata, obtaining 486 diseases withinformation across all three.
However, that’s far short of all theexamples (Freebase: 12,595;
Wikipedia: 2,772; Mayo: 1,247).
We’re investigating wordembeddings and document
similarity methods to createconnections between similar data inthe cases where we lackannotations>> I am having trouble rememberingthings. I repeat myself a lot.

Error on Training Set

Performance Example

Confusion Matrix

Training on both Freebase and Mayo
Clinic data and testing on theFreebase descriptions, these are ourtop five accuracy results.

An example of our collected
statistics and a single analyzed
algorithm condition. This was with
Naive Bayes and TF-IDF features.

On our manual, handwritten dataset ofillness descriptions (40 examples)

MetricResult

Top 1 Accuracy

65.02%

Top 5 Accuracy

87.45%

Mean Rank

8.45

Median Rank

1.0

Trimmed MeanRank (⅔ of data)

1.38>> It feels like there is liquid in my lungs.
I am coughing a lot and I have a fever.
1, Gastroenteritis (Stomach Flu)
0.00074, Bronchitis
2.5e-05, Pneumonia
1.9e-05, Influenza
1.2e-05, Tuberculosis>>I'm sneezing, have a mild fever, andam very congested.
0.97, Pertussis (Whooping Cough)0.027, Common cold
0.00096, Influenza
3.6e-05, Pneumonia
4.2e-10, Egg allergy>> I have a headache, fever, cough andfatigue
0.66, Common cold
0.19, Pneumonia
0.14, Influenza
0.0041, Typhoid fever
0.0031, HIV/AIDS>> I have to go to the bathroom a lot
and my stomach hurts all the time.
1, Social anxiety disorder
0.0012, Generalized anxiety disorder
0.00021, Bulimia nervosa0.00016, Gastroenteritis
4.7e-05, Osteoarthritis
Answer: Irritable Bowel Syndrome>> I have a cut on my arm that seems tobe infected. There is a red streak goingup my arm.
0.84, Complex regional pain syndrome
0.079, Psoriasis
0.037, Antiphospholipid syndrome
0.03, Influenza
0.01, Scleroderma
Answer: Sepsis

Challenges and Next Steps
Lack of Clinical Data: None of our data sources exactly represent the twotypes of notes we’d like to learn from: actual clinician notes and people’s
descriptions of their own illnesses. Training our system on data which better
represents our desired queries would greatly strengthen the quality of the model.
Overconfidence: Our model tends to overfit, guessing either right or wrong
with very high probability. We’d like to alleviate this by obtaining additional data
and reducing the size of the feature space. Using only word referring tosymptoms may be one solution.
Inconsistent Sources: Our sources refer to the same diseases andsymptoms by different names and conventions. Part of our work is focused on
finding matching diseases across data sources.
Quality Priors: It would be beneficial to access full data records to get theactual incidence of diseases as our current priors are unreliable.