NTCIR-13
MedWeb Annotation Corpus Guideline
Medical Natural Language Processing for Web Document

English version 1.0(2017/04/10)

MedWeb Organizers
Eiji Aramaki (Nara Institute of Science and Technology)
Shoko Wakamiya (Nara Institute of Science and Technology)
Mizuki Morita (The University of Okayama)
Yoshinobu Kano (Shizuoka University)
Tomoko Ohkuma (Fuji Xerox)

1Classification Process

We deal with the following diseases and symptoms. Note that we call them as “symptom(s)”in this guideline.

Influenza
Headache

ColdCoughHay Fever
Fever

Diarrhea
Runny nose

We classify tweets by determining whether a Twitter user or someone around the user 1) has asymptom2) at that point in time3). We set three standards common to all symptoms in terms offacticity2), time3), and location1). Based on the criteria, we label “p” (positive) if it is determined
as a symptom and “n” (negative) if it is not determined as a symptom. We then label “?” if it isunknown whether it is a symptom or not. For simplicity's sake, this guideline shows examplesof tweet messages with labels that are formalized as “(si +)” and “(si -)” when the symptom si
is determined as “p” (positive) and “n” (negative), respectively.

22Standards and Exceptions Common to Most Symptoms

2.1 Standards Common to Most Symptoms
Labeling is a practical practice that is necessary for considering the medical importance ofinformation. For example, information that is too old (occurred several weeks ago) and nonsevere symptoms (headache due to over-drinking) should be labeled “n” (negative). Therefore,
the following three criteria should be followed:【Facticity of Symptom】First, it needs to be confirmed that a Twitter user or someone isaffected with a certain disease or has a symptom. A tweet just including a disease name orsymptom as a topic is removed by labeling it as “n” (negative).
【Time of Symptom】Second, information that is too old is considered meaningless from theviewpoint of surveillance. Such information should be labeled “n” (negative). While determining
how long until information is considered “too old” remains ambiguous, a general guide would
be that information within 24 hours—such as information of today or yesterday—is regarded
as “p” (positive).
【Location of Symptom】It is necessary to specify the information about the position of theaffected person. If a Twitter user himself is affected, the information is labeled as “p” (positive)
since the position of the user is the symptom’s place of onset. For cases where the user is nothimself affected, the information is regarded as “p” (positive) if it is presumably in the vicinity(same prefecture) of the user, and “n” (negative) otherwise.
The summary of these guidelines is found in the table below:

Symptom

TimeSpace•Direct expression of symptom•Indirect expression of symptom•If symptom may exist yet denied•Recovery process•Unknown cases•Suspicion of symptom is observed•Symptoms disappear completely•Expression of expectation or process•Unrelated to having a disease•Metaphorical representation: jokes•Within 24 hours, including today•Yesterday•Recent occurrence•Recurring symptom that still persists•Two or more days ago•Past symptoms•Unknown time
•Same region as user
•Same region as user – implied•Symptoms related to family•Symptoms people in the same region have
•Symptoms of other people in an unspecified area
•Symptoms of another person connected in social networks

32.2 Exceptions
There are exceptions to the above principles. For example, there are many remarks about “headache”
that do not indicate a disease--such as “excessive drinking.” When doing disease surveillance, suchstatements should be regarded as noise. With regard to time and certain diseases, information thatis more than 24 hours old may be clinically important. For instance, surveillance of influenza is doneon a weekly basis; including information older than 24 hours is then meaningful for influenza
surveillance. Moreover, there are various exceptions for each disease. In addition to “excessivedrinking,” “medication,” “pungently flavored food (including irritant),” “spiritual,” “motion sickness,”“morning,” “menstrual pain,” etc. should be excluded for “headache.” Such exceptions aresummarized in the following table.
Symptom
Acceptexpressions
withsuspicion*1

Acceptrecovery
expressions*2

Accept justa word of asymptom“p” (positive)“n” (negative)○

Influenza
Cold○

○

○

Hay fever○○○Diarrhea○

Headache○○○Cough○○○○○Fever
Runnynose

Exceptions○○○(only slightfever)

TimeSpace

Accept asymptomfrom theday before

Accept asymptom ofsomeone

○

○

○

-Overeating-Indigestion-Alcohol drinking-medication-Pungently
flavored food○○Due to a sense ofsight or smell-Alcohol drinking-Pungently
flavored foodSide-effect ofinjection

Hay feverChange intemperature*¹ Expressions with suspicion include “feel,” “have possibilities,” “likely to,” “a warning sign,” “bad feeling,”
etc.
*² Recovery expressions such as “cured,” “recovery,” “become better,” “stopped,” “relieved,” etc.

42.3 Facticity of Symptoms•Direct Expression of Symptom
Cases where the symptom is expressed directly—including mild symptoms—are considered “p” (positive).

•Well, Belly Pain and diarrhea as usual(diarrhea＋)

It seems I got hay fever this morning and it’s very painful(hay fever＋)

Indirect Expression of Symptom
A symptom can be labeled “p” (positive) with indirect expressions of a symptom. For example, a symptom can be
indirectly presumed from hospitalization, medication, other symptom complications, etc.
I will have cold medicine and go to bed now
Gotta drink headache medicine and that ramen I don’t want to eat•(cold＋)(headache＋)

If Symptom May Exist Yet Denied
If a symptom is mentioned but then also dismissed or denied, this information is regarded as “p” (positive). For
instance, this is because we generally presume that many patients do not have sufficient medical knowledge, which
may cause them to easily overlook or dismiss relevant symptoms.

My head hurts, and I might have a cold.. But maybe I’m not sick since I just ate too much
Is it a period of hinoki? My eyes also itch, so I can’t really just say it’s a cold I guess•(headache＋,cold＋)(hay fever＋，cold＋)

Recovery Process
In the process of recovery, symptoms may be weak but not completely lost. For such cases, it is considered “p”(positive) if someone or the user is still affected with such mild symptoms during recovery. However, if the symptoms
are completely gone, it is considered “n” (negative).
Kinda gained a headache after recovering from nausea(headache＋)
I went to bed in the heat last weekend. Generally recovered this week, but still runny nose a bit(fever−，runnynose＋)•Unknown cases
A symptom is “p” (positive) even if the user expresses uncertainty regarding its cause.

My head recently hurts… whyyyyy•(headache＋)

Suspicion of Symptom is Observed
Since it is generally presumed that many patients may overlook symptoms or diseases due to insufficient medical
knowledge, even suspicion of symptoms and diseases are recognized and labeled “p” (positive). Indications ofsuspicion include the expressions of “similar,” “maybe,” “possibly,” etc.

Is this an omen or a cold?
Ugh, stiff neck. From colds, maybe?

•Symptoms Disappear Completely
Finally, diarrhea has subsided.
If you get rid of your headache and listlessness, you should have a drink of water and sleep allnight! Trust me, I recovered from them!

•(cold＋)(cold＋)(diarrhea−)(headache−)

Expression of Expectation or Process
For cases that express expectation or process, indicated with words such as “if,” “going,” “if it is,” etc., these should
be labeled as “n” (negative).

•My body feels heavy... I suspect I have influenza if I get a cough and fever. Some students gotinfluenza and the school has been shut down.

(fever−)I hope I’m not gonna get a cold (cold−)

Unrelated to Having a Disease
If the disease is mentioned merely as a topic rather than someone having it, these tweets should be labeled as “n”(negative). These include news, general theories, and advertisements.

5

It is a sign of cold remission when runny nose begins to solidify.
News: 83 percent of people infected with [pandemic influenza] in Osaka have 38 degree highfever or more http://□□•(cold−, runnynose−)(influenza−)

Metaphorical Representation: Jokes
If the disease is mentioned in the context of a joke, these should be labeled as “n” (negative).

Fever!
Cold* is strong like typhoon.

(fever−)(cold−)*風 (wind) is written incorrectly as 風邪 (cold).

2.4 Time of Symptoms•

Within 24 hours, including today
Slight fever in the morning→absence from labo meeting→defervesce→receive an express homedelivery that I did’t expect to be able to receive today
Had high fever but I went back to work after 4 hours because temp returned to normal•(fever＋)(fever＋)

Yesterday
The label for symptoms that occurred yesterday are dependent on the disease or symptom.

I did not heal from the cold yesterday..
Right now healthy except I have no appetite. strange because I had fever yesterday but temp isback to normal now.

•Uhh… if you eat pork, somehow I always have diarrhea … orz(fever−)Past Symptoms
I remember in the festival before that I got a 39 degree fever, but no one decided to go home so I
just stayed.

•(cold＋, cough＋)(diarrhea＋)

Two or More Days Ago
When I entered the sauna, sweat didn’t come out at all. Then did some intense sports and got ahigh fever that night. So today I will drink Pocari sweat before going to the sauna•(headache＋)(fever＋)

Recurring Symptom that Still Persists
I have been feeling a cold the whole winter. Phlegm is yellow•(fever−)Recent Occurrence
Recently head hurts… whyyyyy
He died three days ago mysteriously from high fever•(cold＋)(fever−)Unknown TimeWe estimate the time from the situation.

Diarrhea the next day after grilled meat
I guess major symptom of this year’s Influenza is cough, since I had a fever and a terrible cough.

(diarrhea＋)(influenza−，fever−，cough−)

2.5. Location of Symptpm•Same Region as User
Person in the train intensely coughing. I hope he won’t get a cold
OMG! OMG! OMG! There seems to be a patient with new influenza near here. Scary... But,
honestly speaking, it’s quite lucky that the school is shut down ^p^6(cough＋)(influenza＋)•Same Region as User – Implied
When you have a cold… do I work or something? Should I move around?
I want the man to cease coughing and to stop trying to pass a cold to me! I will not be affectedthough•(cold＋)(cough＋, cold＋)Symptoms Related to Family
We regarded as a symptom in the vicinity and labeled “p” (positive) regardless of living together or not.
(cold＋)(fever＋)

Having a cold is prevalent in my family now!! Except my dad.. Gargle is important okay!!
My daughter has a fever…

••

Symptoms People in the Same Region Have
My puppy is afraid of the cold so I wanna return early and keep her warm.

(cold−)

My computer is inducing the thermal runaway from the morning... In the usual way, the power
supply is suddenly shut down due to high temprature．

(fever−)Symptoms of Other People in an Unspecified Area
If you have a cold, you had better to go to otorhinology. All problems will be solved.
I think there are many people who have a cold…

•(cold−)(cold−)

Symptoms of Another Person Connected in Social Networks
When conversing with other people through retweets or @--, maybe warnings and encouragement can be seen
towards another person who is sick. Judge if this person is in the same area or not. Obviously, retweets (RT) fromstrangers are labeled “n” (negative).

A person posting “I have a fever,” “I need to take a medicine,” or “I feel sick” on SNS every day
finally said that “I don’t wanna go to hospital because I don’t wanna waste my holidays.” I saw
the last post and decided to give a piece of advice to her.

(fever−)7