NTCIR-13
MedWeb Annotation Corpus Guideline
Medical Natural Language Processing for Web Document

English version 2.0(2017/07/04)

MedWeb Organizers
Eiji Aramaki (Nara Institute of Science and Technology)
Shoko Wakamiya (Nara Institute of Science and Technology)
Mizuki Morita (The University of Okayama)
Yoshinobu Kano (Shizuoka University)
Tomoko Ohkuma (Fuji Xerox)

1Classification Process

We deal with the following diseases and symptoms. Note that we call them as “symptom(s)”in this guideline.

Influenza

ColdHay Fever

Diarrhea

HeadacheCoughFever

Runny nose

We classify tweets by determining whether a Twitter user or someone around the user1) has asymptom2) at that point in time3). We set three standards common to all symptoms in terms offacticity2), time3), and location1). Based on the criteria, we label “p” (positive) if it is determined
as a symptom and “n” (negative) if it is not determined as a symptom. For simplicity's sake,this guideline shows examples of tweet messages with labels that are formalized as “(si +)”and “(si -)” when the symptom si is determined as “p” (positive) and “n” (negative),respectively.

22Standards and Exceptions Common to Most Symptoms

2.1 Standards Common to Most Symptoms
Labeling is a practical practice that is necessary for considering the medical importance ofinformation. For example, information that is too old (occurred several weeks ago) and nonsevere symptoms (headache due to over-drinking) should be labeled “n” (negative). Therefore,
the following three criteria should be followed:【Facticity of Symptom】First, it needs to be confirmed that a Twitter user or someone isaffected with a certain disease or has a symptom. A tweet just including a disease name orsymptom as a topic is removed by labeling it as “n” (negative).
【 Time of Symptom 】 Second, past information is considered meaningless from theviewpoint of surveillance. Such information should be labeled “n” (negative). With regard totime and certain diseases, information that is more than 24 hours old may be clinically important.
For instance, surveillance of influenza is done on a weekly basis; including information older than
24 hours is then meaningful for influenza surveillance. While determining how long until

information is considered “too old” remains ambiguous, a general guide would be that
information within 24 hours̶such as information of today or yesterday̶is regarded as “p”(positive).

【Location of Symptom】It is necessary to specify the information about the position of theaffected person. If a Twitter user himself is affected, the information is labeled as “p”(positive) since the position of the user is the symptomʼs place of onset. For cases where theuser is not himself affected, the information is regarded as “p” (positive) if it is presumably inthe vicinity (same prefecture) of the user, and “n” (negative) otherwise.
The summary of these guidelines is found in the table below:SymptomTimeSpace•Direct expression of symptom•Indirect expression of symptom•If symptom may exist yet denied•Recovery process•Unknown cases•Suspicion of symptom is observed•Symptoms disappear completely•Expression of expectation or process•Unrelated to having a disease•Except for humans•Metaphorical representation: jokes•Within 24 hours, including today•Yesterday•Recent occurrence•Recurring symptom that still persists•Past symptoms including two or more days ago•Unknown time
•Same region as user
•Same region as user ‒ implied•Symptoms in a group•Symptoms of other people in an unspecified area
•Symptoms of another person connected in social networks

32.2 Exceptions
There are exceptions to the above principles. For example, there are many remarks about “headache”
that do not indicate a disease--such as “excessive drinking.” When doing disease surveillance, suchstatements should be regarded as noise. Moreover, there are various exceptions for each disease.
In addition to “excessive drinking,” “medication,” “pungently flavored food (including irritant),”“spiritual,” “motion sickness,” “morning,” “menstrual pain,” etc. should be excluded for “headache.”
Such exceptions are summarized in the following table.
Symptom
Acceptexpressions withsuspicion *1

Accept just aword of asymptom

Cold

○

○

Hay fever○○Exceptions“p” (positive)“n” (negative)

Influenza

Diarrhea○○

Headache○○Cough○

○

Fever

○

Runny nose○○

-Overeating-Indigestion-Alcohol drinking-medication-Pungently flavoredfoodDue to a sense ofsight or smell-Alcohol drinking-Pungently flavoredfood
Side-effect of injection(only slight fever)Hay feverChange intemperature*¹ Expressions with suspicion include “feel,” “have possibilities,” “likely to,” “a warning sign,” “bad feeling,”
etc.

4

2.3 Facticity of Symptomsa.

Direct Expression of Symptom
Cases where the symptom is expressed directly̶including mild symptoms̶are considered “p” (positive).

Well, Belly Pain and diarrhea as usual
It seems I got hay fever this morning and itʼs very painfulb.

(diarrhea＋)(hay fever＋)

Indirect Expression of Symptom
A symptom can be labeled “p” (positive) with indirect expressions of a symptom. For example, a symptom can be
indirectly presumed from hospitalization, medication, other symptom complications, etc. As for hay fever and influenza,
many people attempt to prevent them in advance. Therefore, we just consider any mention indicating concrete action.

I will have cold medicine and go to bed now
Gotta drink headache medicine and that ramen I donʼt want to eat
Sometime I listen to my favorite music as a remedy for a headache.
I bought glasses that people say are effective for allergies.
Somebody, come up with a prescription medicine that works crazy well on allergies!

c.

(cold＋)(headache＋)(headache＋)(hay fever+)(hay fever+,runny nose+)

I wear a mask as a precaution against the flu when I go out.

(influenza−)

Apparently yogurt is a good allergy remedy.

(hay fever−)If Symptom May Exist Yet Denied
If a symptom is mentioned but then also dismissed or denied, this information is regarded as “p” (positive). For instance,
this is because we generally presume that many patients do not have sufficient medical knowledge, which may cause
them to easily overlook or dismiss relevant symptoms.

My head hurts, and I might have a cold.. But maybe Iʼm not sick since I just ate too much(headache＋,cold＋)

Is it a period of hinoki? My eyes also itch, so I canʼt really just say itʼs a cold I guess(hay fever＋，cold＋)(hay fever+,cold+, fever+,runny nose+)I thought it was allergies, but I got a fever and it turned out to be a cold! I didn't think I would
make a mistake like that.
My doctor mistook my symtoms for allergies.

d.

(hay fever+,cold+, runnynose+)

Recovery Process
In the process of recovery, symptoms may be weak but not completely lost. For such cases, it is considered “p”(positive) if someone or the user is still affected with such mild symptoms during recovery. However, if the symptoms
are completely gone, it is considered “n” (negative).

Kinda gained a headache after recovering from nausea
I went to bed in the heat last weekend. Generally recovered this week, but still runny nose a bite.

(headache＋)(fever−，runnynose＋)

Unknown cases
A symptom is “p” (positive) even if the user expresses uncertainty regarding its cause.

My head recently hurts… whyyyyyf.

(headache＋)

Suspicion of Symptom is Observed
Since it is generally presumed that many patients may overlook symptoms or diseases due to insufficient medical
knowledge, even suspicion of symptoms and diseases are recognized and labeled “p” (positive). Indications ofsuspicion include the expressions of “similar,” “maybe,” “possibly,” etc.

Is this an omen or a cold?
Ugh, stiff neck. From colds, maybe?

(cold＋)(cold＋)

5g.

Symptoms Disappear Completely
Symptoms that disappeared completely are recognized and labeled “n” (negative). Note that we regarded and labeled“p” (positive) when a user took medicine that could cause temporary recovery from a symptom.

I pushed through it and now my congestion is gone.
I took medicine and I stopped coughing!
Finally, diarrhea has subsided.
If you get rid of your headache and listlessness, you should have a drink of water and sleep allnight! Trust me, I recovered from them!

h.

(runny nose−)(cough+)(diarrhea−)(headache−)

Expression of Expectation or Process
For cases that express expectation or process, indicated with words such as “if,” “going,” “if it is,” etc., these should
be labeled as “n” (negative).

i.

My body feels heavy... I suspect I have influenza if I get a cough and fever. Some students got
influenza and the school has been shut down.

(fever−)I hope Iʼm not gonna get a cold L(cold−)

Unrelated to Having a Disease
If the disease is mentioned merely as a topic rather than someone having it, these tweets should be labeled as “n”(negative). These include news, general theories, and advertisements.

It is a sign of cold remission when runny nose begins to solidify.
News: 83 percent of people infected with [pandemic influenza] in Osaka have 38 degree highfever or more http://□□j.

k.

(runny nose−,cold−)(influenza−)

Except for Human
My puppy is afraid of the cold so I wanna return early and keep her warm.

(cold−)

My computer is inducing the thermal runaway from the morning... In the usual way, the power
supply is suddenly shut down due to high temprature．

(fever−)Metaphorical Representation: Jokes
If the disease is mentioned in the context of a joke, these should be labeled as “n” (negative).

Fever!
Cold* is strong like typhoon.
*⾵ (wind) is written incorrectly as ⾵邪 (cold).

6(fever−)(cold−)

2.4 Time of Symptoms
2

Although the occurrence and condition of each symptom
is different, it is common to be getting better in therecovery stage. Therefore, we will positively extract
symptoms observed within 24 hours or the the day
before so that we consider the possibility of remaining
slight cases now or having just recovered from them.
The figure shows the process of a symptom. The X-axis
indicates time while the Y-axsis indicates the conditionof the symptom. Case A and Case B are regarded as “P”(positive) since symptoms at least exist within 24 hours
and the day before, respectively. As for Case C, weregard as “N” (neative) because the symptom is before
24 hours and more.

a.

24+)))2)
(-)
(-)))

()

Within 24 hours, including today
Slight fever in the morning→absence from labo meeting→defervesce→receive an express homedelivery that I didʼt expect to be able to receive today
Had high fever but I went back to work after 4 hours because temp returned to normalb.

)(fever＋)(fever＋)

Yesterday
The label for symptoms that occurred yesterday are dependent on the disease or symptom.

I did not heal from the cold yesterday
Right now healthy except I have no appetite strange because I had fever yesterday but temp isback to normal now.

c.

f.

(fever−)(fever−)Recent Occurrence
Recently head hurts… whyyyyy
He died for three days mysteriously from high fevere.

(fever＋)

Past Symptoms Including Two or More Days Ago
When I entered the sauna, sweat didnʼt come out at all. Then did some intense sports and got ahigh fever that night. So today I will drink Pocari sweat before going to the sauna
I remember in the festival before that I got a 39 degree fever, but no one decided to go home so I
just stayed.

d.

(cold＋)(headache＋)(fever＋)

Recurring Symptom that Still Persists
I have been feeling a cold the whole winter. Phlegm is yellow(cold＋, cough＋)

Uhh… if you eat pork, somehow I always have diarrhea … orz(diarrhea＋)

Unknown TimeWe estimate the time from the situation.

Diarrhea the next day after grilled meat
I guess major symptom of this yearʼs Influenza is cough, since I had a fever and a terrible cough.

(diarrhea＋)(influenza−，fever−，cough−)72.5. Location of Symptpma.

Same Region as User

We regarded a region as a prefecture or a smaller area. Note that we donʼt consider a symptom that has happened incountries outside of Japan.

Person in the train intensely coughing. I hope he wonʼt get a cold
OMG! OMG! OMG! There seems to be a patient with new influenza near here. Scary... But,honestly speaking, itʼs quite lucky that the school is shut down ^p^b.

(cough＋)(influenza＋)

Same Region as User ‒ Implied

We regarded as a symptom in the vicinity and labeled “p” (positive) regardless of living together or not (i.e. familymembers). We also labeld “p” (positive) when a symptom was observed from hearsay. However, we donʼt consider asymptom that has happened in countries outside of Japan.

Having a cold is prevalent in my family now!! Except my dad.. Gargle is important okay!!
My daughter has a fever…
When you have a cold… do I work or something? Should I move around?
I want the man to cease coughing and to stop trying to pass a cold to me! I will not be affectedthoughc.

(cold＋)(fever＋)(cold＋)(cough＋, cold＋)Symptoms in a GroupAs for symptoms of people belonging to a specified group in the vicinity (school, club, etc.), we labeled them “p”(positive). Other cases, like symptoms belonging to blog friends, should be labeled as “n” (negative) since it isdifficult to determine their location.

Everyone in my club is going to a meet and greet, but I'm worried since the flu is running rampant.
Everyone has diarrhea, was there an outbreak of food poisoning??
When you're surrounded with people who are coughing, you gotta wash your hands more thanusual.
I met this person through a dog-related blog who has a bad stuffy nose and can hardly breathe.
You can't brush a head cold off as merely a head cold.
It's awful that the flu is going around at school right before the National Center Test for University
Admissions.

d.

(cough−)(runny nose−)(cold+)

Symptoms of Other People in an Unspecified Area
If you have a cold, you had better to go to otorhinology. All problems will be solved.
I think there are many people who have a cold…

e.

(influenza＋)(diarrhea−)(cold−)(cold−)

Symptoms of Another Person Connected in Social Networks
When conversing with other people through retweets or mentions to a specified user (@username), maybe warnings
and encouragement can be seen towards another person who is sick. Judge if this person is in the same area or not.
Obviously, retweets (RT) from strangers are labeled “n” (negative). Note that there are no retweets or mentions to a
specified user (@username) in the MedWeb dataset.

A person posting “I have a fever,” “I need to take a medicine,” or “I feel sick” on SNS every day
finally said that “I donʼt wanna go to hospital because I donʼt wanna waste my holidays.” I saw
the last post and decided to give a piece of advice to her.

8(fever−)