Internet-administered

cognitive

behavior

therapy

for

health

problems

a

systematic

review

Cognitive-behavioral

interventions

are

the

most

extensively

researched

form

of

psychological

treatment

and

are

increasingly

offered

through

the

Internet

Internet-based

interventions

may

save

therapist

time

reduce

waiting-lists

cut

traveling

time

and

reach

populations

with

health

problems

who

can

not

easily

access

other

more

traditional

forms

of

treatments

We

conducted

a

systematic

review

of

twelve

randomized

controlled

or

comparative

trials

Studies

were

identified

through

systematic

searches

in

major

bibliographical

databases

Three

studies

focused

on

patients

suffering

from

pain

three

on

headache

and

six

on

other

health

problems

The

effects

found

for

Internet

interventions

targeting

pain

were

comparable

to

the

effects

found

for

face-to-face

treatments

and

the

same

was

true

for

interventions

aimed

at

headache

The

other

interventions

also

showed

some

effects

although

effects

differed

across

target

conditions

Internet-delivered

cognitive-behavioral

interventions

are

a

promising

addition

and

complement

to

existing

treatments

The

Internet

will

most

likely

assume

a

major

role

in

the

future

delivery

of

cognitive-behavioral

interventions

to

patients

with

health

problems

Introduction

Cognitive-behavioral

interventions

are

probably

the

most

extensively

researched

form

of

psychological

treatment

-LRB-

Butler

et

al.

-RRB-

Cognitive-behavioral

interventions

are

aimed

at

challenging

negative

automatic

thoughts

and

dysfunctional

underlying

beliefs

and

at

changing

behavioral

patterns

which

are

related

to

the

problem

being

targeted

in

the

therapy

More

than

published

controlled

outcome

studies

and

probably

many

more

have

examined

the

effects

of

cognitive-behavioral

therapy

-LRB-

CBT

-RRB-

for

a

wide

range

of

disorders

and

health

problems

ranging

from

mental

health

disorders

such

as

depression

-LRB-

Hollon

et

al.

-RRB-

anxiety

disorders

-LRB-

Barlow

-RRB-

schizophrenia

-LRB-

Pilling

et

al.

-RRB-

to

health

conditions

such

as

chronic

pain

-LRB-

Morley

et

al.

-RRB-

sleep

problems

-LRB-

Morin

et

al.

-RRB-

headache

-LRB-

Holroyd

-RRB-

cancer

-LRB-

Moorey

and

Greer

-RRB-

and

many

others

Most

of

these

studies

have

shown

that

CBT

has

positive

effects

on

these

and

several

other

health

conditions

CBT

is

not

only

the

most

extensively

researched

form

of

psychotherapy

but

also

the

most

widely

applied

type

of

psychotherapy

-LRB-

Norcross

et

al.

-RRB-

and

certainly

the

most

widely

applied

evidence-based

type

of

psychological

therapy

CBT

is

increasingly

offered

through

the

Internet

Internet-based

interventions

may

have

several

advantages

over

other

more

traditional

forms

of

delivery

They

may

save

therapist

time

reduce

waiting-lists

allow

patients

to

work

at

their

own

pace

abolish

the

need

to

schedule

appointments

with

a

therapist

save

traveling

time

reduce

the

stigma

of

going

to

a

psychologist

or

therapist

and

facilitate

help

for

the

hard-of-hearing

as

self-help

treatments

typically

work

with

visual

rather

than

auditory

information

-LRB-

Marks

et

al.

-RRB-

Furthermore

Internet-delivered

self-help

may

be

programmed

to

enhance

patients

motivation

by

presenting

a

wide

range

of

attractive

audiovisual

information

with

voices

giving

instructions

in

whichever

gender

age

accent

language

and

perhaps

game

format

the

client

prefers

It

can

also

quickly

and

automatically

report

patient

progress

and

self-ratings

Internet-based

interventions

may

reach

populations

with

health

problems

who

can

not

be

reached

with

other

more

traditional

forms

of

treatments

For

example

a

considerable

proportion

of

the

patients

with

mental

disorders

are

not

reached

with

traditional

forms

of

treatment

-LRB-

Bijl

and

Ravelli

-RRB-

because

of

the

stigma

associated

with

mental

disorder

prejudices

about

therapists

lack

of

willingness

to

talk

to

a

stranger

about

personal

problems

or

because

of

physical

obstacles

like

walking

problems

or

long

distances

For

patients

with

somatic

conditions

there

may

be

barriers

to

seek

psychological

treatment

Internet-guided

interventions

may

reach

a

segment

of

this

population

who

can

not

be

reached

through

traditional

interventions

Internet-based

psychological

interventions

for

many

health

problems

are

commonly

based

on

CBT

techniques

One

reason

is

that

the

effects

of

CBT

have

been

shown

in

numerous

trials

Another

reason

why

CBT

is

often

used

in

internet-interventions

is

that

these

techniques

lend

themselves

to

be

operationalized

in

text

CBT

interventions

can

very

well

be

converted

into

a

structured

format

with

psychoeducation

homework

assignments

and

registration

exercises

presented

via

web

pages

-LRB-

Ritterband

et

al.

-RRB-

Delivering

CBT

through

the

Internet

does

not

however

only

have

advantages

An

online

programme

may

not

be

suitable

for

technophobic

patients

and

illiterates

nor

can

it

answer

all

the

possible

questions

users

may

ask

it

can

not

detect

subtle

nonverbal

and

verbal

clues

to

clients

misunderstandings

it

may

encourage

clients

to

cherry-pick

from

a

range

of

homework

options

presented

and

not

all

clients

find

communicating

via

computers

acceptable

-LRB-

Marks

et

al.

-RRB-

However

subtle

text

nuances

may

be

detected

and

somewhat

surprisingly

Internet

interventions

has

been

found

to

generate

good

working

alliance

between

the

patient

and

the

therapist

-LRB-

Knaevelsrud

and

Maercker

-RRB-

Furthermore

it

is

not

yet

very

clear

whether

CBT

interventions

which

have

been

proven

to

be

effective

when

delivered

in

traditional

format

are

also

effective

when

delivered

through

the

internet

Recent

studies

however

indicate

that

this

might

be

the

case

for

at

least

some

patients

and

some

conditions

-LRB-

Carlbring

et

al.

-RRB-

Whether

or

not

an

internet-based

CBT

is

effective

should

be

examined

in

randomized

controlled

trials

and

can

obviously

not

be

based

on

the

effects

of

traditionally

delivered

CBT

-LRB-

e.g.

individual

or

group

format

-RRB-

Since

the

Internet

has

become

available

to

the

broad

public

in

many

Western

countries

several

trials

have

examined

the

effects

of

cognitive

behavioral

interventions

in

randomized

trials

In

a

recent

meta-analysis

we

examined

the

effects

of

internet-based

treatments

for

depression

and

anxiety

disorders

and

found

that

these

interventions

had

large

effects

-LRB-

Cohen

's

d

-RRB-

compared

to

control

conditions

when

some

kind

of

guidance

was

given

to

the

patients

receiving

the

treatment

-LRB-

Spek

et

al.

-RRB-

This

latter

observation

was

also

confirmed

in

a

recent

review

in

which

a

correlation

of

rho

P

was

obtained

between

amount

of

contact

spent

with

clients

and

the

effect

size

-LRB-

Palmqvist

et

al.

-RRB-

These

large

effect

sizes

suggest

that

Internet-administered

CBT

is

as

effective

as

face-to-face

CBT

and

that

the

format

in

which

CBT

is

delivered

may

not

be

related

to

the

effect

sizes

found

It

is

not

known

however

whether

Internet-administered

CBT

is

also

equally

effective

when

other

health

problems

are

targeted

In

the

current

study

we

will

present

the

results

of

a

systematic

review

of

Internet-delivered

CBT

for

health

problems

In

the

review

we

aim

to

establish

for

which

health

problems

Internet-based

CBT

has

been

developed

and

examined

in

randomized

controlled

or

comparative

trials

and

whether

these

interventions

were

effective

We

also

examine

the

target

groups

and

contents

of

these

intervention

as

well

the

quality

of

the

studies

Methods

Search

strategy

and

selection

of

studies

Studies

were

traced

through

several

methods

First

we

conducted

a

comprehensive

literature

search

in

bibliographical

databases

-LRB-

from

to

February

-RRB-

We

examined

abstracts

in

Pubmed

-LRB-

abstracts

-RRB-

Psycinfo

-LRB-

-RRB-

Embase

-LRB-

-RRB-

and

the

Cochrane

Central

Register

of

Controlled

Trials

-LRB-

-RRB-

In

order

to

find

unpublished

studies

we

also

searched

Digital

Dissertations

-LRB-

abstracts

-RRB-

We

searched

these

databases

by

combining

terms

indicative

of

effect

studies

-LRB-

randomized

trials

controlled

trials

clinical

trials

-RRB-

and

Internet

-LRB-

both

keywords

and

text

words

-RRB-

Second

we

examined

the

references

of

earlier

reviews

of

Internet-based

interventions

-LRB-

Griffiths

et

al.

Wantland

et

al.

-RRB-

and

we

reviewed

the

reference

lists

of

retrieved

papers

Studies

were

included

if

they

met

the

following

criteria

-LRB-

a

-RRB-

randomized

controlled

or

comparative

trials

-LRB-

b

-RRB-

examining

interventions

that

were

conducted

through

the

Internet

-LRB-

at

least

of

the

intervention

-RRB-

-LRB-

c

-RRB-

based

on

CBT

techniques

-LRB-

d

-RRB-

aimed

at

behavior

change

-LRB-

e

-RRB-

in

patients

with

an

existing

disorder

or

health

problem

We

excluded

studies

aimed

at

mental

disorders

because

another

recent

systematic

review

was

published

about

these

studies

-LRB-

Spek

et

al.

-RRB-

We

also

excluded

studies

focusing

on

lifestyle

-LRB-

smoking

obesity

exercise

nutrition

-RRB-

because

the

character

of

these

interventions

differs

strongly

because

the

focus

of

these

interventions

is

typically

preventative

Quality

assessment

There

are

at

least

scales

available

to

assess

the

validity

and

quality

of

randomized

controlled

trials

-LRB-

Higgins

and

Green

-RRB-

There

is

no

evidence

however

that

these

scales

provide

more

reliable

assessments

of

validity

We

preferred

therefore

to

use

a

simple

approach

for

assessing

the

validity

of

the

studies

as

suggested

in

the

Cochrane

Handbook

-LRB-

Higgins

and

Green

-RRB-

In

this

context

the

validity

of

a

study

can

be

defined

as

the

extent

to

which

its

design

and

conduct

are

likely

to

prevent

systematic

errors

-LRB-

Moher

-RRB-

Variation

in

validity

can

explain

variation

in

the

results

of

the

studies

included

in

a

systematic

review

and

may

result

in

an

erroneous

conclusions

that

an

intervention

is

effective

if

the

less

rigorous

studies

are

biased

toward

overestimating

an

intervention

's

effectiveness

-LRB-

Higgins

and

Green

-RRB-

We

assessed

the

validity

of

the

studies

using

four

basic

criteria

allocation

to

conditions

is

conducted

by

an

independent

-LRB-

third

-RRB-

party

adequacy

of

random

allocation

concealment

to

respondents

blinding

of

assessors

of

outcomes

and

completeness

of

follow-up

data

Analyses

We

examined

the

characteristics

of

the

target

populations

the

interventions

and

the

design

of

the

included

studies

We

also

examined

which

main

outcome

measures

were

used

for

each

study

and

we

calculated

standardized

effect

sizes

for

each

of

the

main

outcome

measures

These

effect

sizes

-LRB-

d

-RRB-

were

calculated

by

subtracting

-LRB-

at

post-test

-RRB-

the

average

score

of

the

control

group

-LRB-

Mc

-RRB-

from

the

average

score

of

the

experimental

group

-LRB-

Me

-RRB-

and

dividing

the

result

by

the

average

of

the

standard

deviations

of

the

experimental

and

control

group

-LRB-

SDec

-RRB-

An

effect

size

of

thus

indicates

that

the

mean

of

the

experimental

group

is

half

a

standard

deviation

larger

than

the

mean

of

the

control

group

Effect

sizes

of

can

be

assumed

to

be

large

while

effect

sizes

of

are

moderate

and

effect

sizes

of

are

small

-LRB-

Cohen

-RRB-

When

sufficient

effect

sizes

were

available

-LRB-

at

least

three

effect

sizes

examining

the

same

outcome

measure

in

the

same

health

problem

-RRB-

we

calculated

pooled

mean

effect

sizes

For

these

analyses

we

used

the

computer

program

Comprehensive

Meta-analysis

-LRB-

version

-RRB-

developed

for

support

in

meta-analysis

As

we

expected

considerable

heterogeneity

we

decided

to

calculate

mean

effect

sizes

with

the

random

effects

model

In

the

random

effects

model

it

is

assumed

that

the

included

studies

are

drawn

from

populations

of

studies

that

differ

from

each

other

systematically

In

these

analyses

we

tested

whether

there

are

genuine

differences

underlying

the

results

of

the

studies

-LRB-

heterogeneity

-RRB-

or

whether

the

variation

in

findings

is

compatible

with

chance

alone

-LRB-

homogeneity

Higgins

et

al.

-RRB-

As

an

indicator

of

homogeneity

we

calculated

the

Q-statistic

We

also

calculated

the

which

is

an

indicator

of

heterogeneity

in

percentages

as

well

A

value

of

indicates

no

observed

heterogeneity

and

larger

values

show

increasing

heterogeneity

with

as

low

as

moderate

and

as

high

heterogeneity

-LRB-

Higgins

et

al.

-RRB-

Results

Included

studies

A

total

of

papers

which

possibly

met

our

inclusion

criteria

were

retrieved

Twelve

studies

-LRB-

papers

two

papers

were

published

about

the

same

study

-RRB-

met

our

inclusion

criteria

The

other

studies

were

excluded

because

they

did

not

examine

a

cognitive-behavioral

intervention

-LRB-

papers

-RRB-

because

they

were

not

a

randomized

controlled

or

comparative

trial

-LRB-

papers

-RRB-

or

because

they

did

not

examine

an

Internet-based

intervention

-LRB-

papers

-RRB-

Selected

characteristics

of

the

target

groups

the

intervention

and

the

general

design

of

the

twelve

included

studies

are

presented

in

Table

Table

characteristics

of

Internet-based

cognitive

behavioral

interventions

for

health

problemsParticipantsIntervention

and

conditionsStudyHealth

conditionRecruitmentWomen

-LRB-

-RRB-

Age

group

-LRB-

M

-RRB-

Conditions

NInterventionContactPeriodMeasurementsDropout

-LRB-

-RRB-

CountryAndersson

et

al.

-LRB-

-RRB-

I-CBT

applied

relaxation

problem

solving

cognitive

restructuringE-mail

-LRB-

at

request

-RRB-

weeksPre

telephone

callsDevineni

and

Blanchard

migraine-only

or

mixed

headache

-LRB-

-RRB-

muscle

relaxation

cognitive

stress

coping

therapy

-LRB-

tension-type

-RRB-

or

autogenic

training

PMR

-LRB-

migraine/mixed

-RRB-

No

therapist

weeksPre

et

al.

-LSB-

-RSB-

Recurrent

-LRB-

-RRB-

applied

relaxation

problem

solving

cognitive

restructuringE-mail

-LRB-

at

request

-RRB-

weeksPre

pain

burn-out

in

patients

on

long-term

sick

leave

-LRB-

-RRB-

texts

psychoeducation

cognitive

self-treatment

-LRB-

changing

coping

with

shame

and

guilt

depression

identity

etc.

-RRB-

Real

live

introduction

meeting

weekly

online

weeksPre

post

et

al.

back

pain

-LRB-

-RRB-

relaxation

physical

exercise

coping

strategiesE-mail

-LRB-

at

request

-RRB-

weekly

telephone

weeksPre

post

et

al.

recurrent

-LRB-

-RRB-

relaxation

cognitive

restructuringE-mail

contact

-LRB-

times

-RRB-

telephone

contact

-LRB-

times

-RRB-

weeksPre

post

et

al.

-LRB-

-RRB-

applied

relaxation

positive

imagery

advice

on

noise

sensitivity

cognitive

restructuring

behavioral

sleep

management.Weekly

report

on

progress

weekly

encouraging

e-mail

weeksPre

post

et

al.

with

physical

disabilities

who

feel

-LRB-

-RRB-

on

communication

self-observation

role-playing

confrontation

positive

verbalizations

social

skills

and

assertiveness.Weekly

sessions

on

the

weeksPre

post

et

al.

diseases

-LRB-

heart

lung

or

type

diabetes

-RRB-

-LRB-

-RRB-

programs

relaxation

cognitive

restructuring

psychoeducation

physician-patient

communication

healthy

eating

fatigue

management

problem

solving

Web-based

bulletin

board

discussion

groups

-LRB-

trained

peer

moderators

-RRB-

weeksPre

post

et

al.

breast

-LRB-

-RRB-

coping

advice

for

common

physical

symptoms

such

as

pain

and

fatigue

structured

coping-skills

training

exercisesBulletin

board

for

asynchronous

group

weeksPre

et

al.

-LRB-

-RRB-

sleep

restriction

stimulus

control

cognitive

restructuringE-mail

-LRB-

at

request

-RRB-

weeksPre

et

al.

brain

-LRB-

-RRB-

solving

communication

behavior

management

skills

Videocontact

with

weeksPre

I-CBT

Internet-CBT

NR

not

reported

In

the

twelve

included

studies

a

total

of

patients

participated

in

the

Internet-CBT

conditions

and

in

the

control

conditions

-LRB-

mean

number

of

respondents

per

study

standard

deviation

-RRB-

More

than

half

of

these

patients

-LRB-

-RRB-

participated

in

one

study

-LRB-

Lorig

et

al.

-RRB-

In

of

the

other

studies

was

the

number

of

patients

per

condition

larger

than

Eleven

studies

compared

an

Internet-based

CBT

intervention

to

a

control

condition

while

one

study

compared

two

types

of

Internet-based

CBT

to

each

other

-LRB-

one

with

and

one

without

weekly

telephone

calls

-RRB-

Nine

of

the

eleven

controlled

studies

used

a

waiting

list

control

group

while

one

study

used

a

care-as-usual

control

group

and

the

other

one

used

an

information

control

group

In

all

studies

participants

were

randomized

to

one

of

two

conditions

In

of

the

studies

was

Internet-based

CBT

compared

to

a

face-to-face

intervention

of

another

treatment

Six

studies

only

presented

pre-post

data

while

the

other

six

also

had

a

follow-up

measurement

-LRB-

mean

length

of

follow-up

in

these

six

studies

was

months

standard

deviation

-RRB-

Six

studies

were

conducted

in

Sweden

four

in

the

United

States

and

two

in

Canada

All

included

studies

were

conducted

in

the

year

or

later

-LRB-

one

in

and

another

one

in

two

in

in

and

in

and

four

in

-RRB-

The

quality

of

studies

varied

Three

of

the

twelve

studies

reported

that

allocation

to

conditions

was

conducted

by

an

independent

party

Concealment

of

random

allocation

to

respondents

was

not

possible

in

any

of

the

studies

while

blinding

of

assessors

was

reported

in

of

the

studies

Drop-out

numbers

ranged

from

to

In

only

one

of

the

studies

intention-to-treat

analyses

were

conducted

-LRB-

Hicks

et

al.

the

other

studies

were

limited

to

completers-only

analyses

-RRB-

The

target

populations

Three

studies

focused

on

patients

suffering

from

pain

three

on

headache

and

six

on

other

health

problems

-LRB-

tinnitus

physical

disabilities

chronic

diseases

breast

cancer

insomnia

and

pediatric

brain

injury

-RRB-

In

ten

of

the

twelve

studies

patients

were

recruited

through

announcements

on

websites

referrals

and

community

recruitment

In

the

other

two

studies

patients

were

recruited

through

screening

of

clinical

samples

Ten

studies

were

aimed

at

adults

two

at

children

of

the

interventions

were

aimed

at

older

adults

although

three

studies

allowed

older

adults

-LRB-

years

-RRB-

to

participate

The

other

studies

on

adults

only

included

younger

adults

for

participation

or

did

not

report

that

they

used

an

age

limit

The

interventions

The

character

of

the

interventions

differed

from

each

other

One

group

of

interventions

consisted

of

self-help

materials

on

the

Internet

with

supporting

e-mails

or

telephone

calls

-LRB-

studies

-RRB-

In

two

studies

the

intervention

consisted

of

self-help

materials

on

the

Internet

but

without

the

supporting

e-mails

or

calls

In

the

other

studies

the

core

of

the

intervention

consists

of

online

contact

between

a

therapist

or

moderator

and

the

patients

-LRB-

individual

or

in

groups

-RRB-

Most

interventions

contained

psychoeducation

on

the

specific

problem

and

different

CBT

modules

such

as

cognitive

restructuring

relaxation

techniques

and

social

skills

training

The

duration

of

the

interventions

ranged

between

and

weeks

Effects

of

the

interventions

The

effects

of

the

interventions

on

the

main

outcomes

of

each

included

study

at

post-test

are

presented

in

Table

Table

outcomes

of

studies

on

Internet-based

cognitive

behavioral

interventions

for

health

problemsHealth

conditionComparisonMain

outcomeEffect

CIPainBrattberg

pain

and

burnoutRehabilitation

course

versus

waiting

list

controlFunctional

Buhrman

et

al.

back

painInternet-guided

self-help

versus

waiting

list

controlCoping

with

Hicks

et

al.

BPediatric

recurrent

pain/headacheInternet-guided

self-help

versus

waiting

list

HeadacheAndersson

et

al.

self-help

with

support

versus

self-help

Devineni

and

Blanchard

headacheInternet

self-help

versus

waiting

list

Ström

et

al.

headacheInternet

self-help

versus

waiting

list

OtherAndersson

et

al.

CBT

versus

waiting

list

controlDistress

from

Hopps

et

al.

disabilitiesGoal-oriented

CBT

chat-group

teletherapy

versus

waiting

list

Lorig

et

al.

diseases

Online

CBT

workshops

versus

care-as-usualHealth

indicators

-LRB-

only

year

FU

-RRB-

Owen

et

al.

breast

cancerOnline

CBT

coping

group

versus

waiting

list

controlHealth-related

quality

of

Ström

et

al.

CBT

versus

waiting

list

controlWade

et

al.

brain

injuryOnline

family

problem

solving

therapy

versus

Internet

resources

Parental

mental

aPooled

with

the

random

effects

model

Z

P

Q

n.s.

We

pooled

the

three

studies

in

which

Internet-based

CBT

for

pain

was

compared

to

control

groups

The

mean

effect

size

on

the

main

outcome

measure

was

-LRB-

CI

P

-RRB-

indicating

a

moderate

to

large

effect

of

the

interventions

compared

to

the

control

groups

at

post-test

Heterogeneity

was

very

low

-LRB-

Q

n.s.

-RRB-

There

were

also

three

studies

on

headache

however

one

did

not

use

a

true

control

group

-LRB-

but

compared

two

active

interventions

to

each

other

-RRB-

Therefore

we

did

not

pool

the

results

of

these

studies

The

effect

sizes

of

these

interventions

were

small

-LRB-

d

Ström

et

al.

-RRB-

to

moderate

-LRB-

d

Devineni

et

al.

-RRB-

The

effect

sizes

of

the

other

studies

were

in

the

small

to

moderate

range

varying

from

small

-LRB-

d

-RRB-

for

health

indicators

in

chronic

diseases

at

one-year

follow-up

to

somewhat

larger

for

health-related

quality

of

life

in

breast

cancer

patients

-LRB-

d

-RRB-

and

from

tinnitus

sufferers

-LRB-

d

-RRB-

to

moderate

-LRB-

loneliness

in

patients

with

physical

disabilities

d

-RRB-

and

large

-LRB-

parental

mental

health

in

pediatric

brain

injury

d

-RRB-

Most

effect

sizes

did

not

significantly

differ

from

zero

because

of

the

small

sample

sizes

in

the

majority

of

the

studies

Discussion

This

systematic

review

of

controlled

and

comparative

studies

of

Internet-based

CBT

for

health

problems

showed

that

this

field

is

developing

fast

Since

twelve

randomized

studies

have

examined

interventions

for

pain

headache

and

several

other

health

problems

Half

of

these

trials

were

published

in

and

and

it

can

be

expected

that

the

number

of

trials

will

rise

sharply

in

the

next

few

years

Overall

findings

are

promising

but

effects

are

slightly

below

the

effect

sizes

found

for

Internet-delivered

CBT

for

anxiety

and

depression

-LRB-

Spek

et

al.

-RRB-

Although

several

health

problems

were

targeted

in

these

Internet-based

studies

there

are

gaps

in

the

literature

in

terms

of

treatments

for

health

problems

which

have

been

found

to

improve

by

means

of

CBT

For

example

several

studies

have

examined

the

effects

of

CBT

for

chronic

fatigue

syndrome

-LRB-

Knoop

et

al.

-RRB-

fibromyalgia

-LRB-

Garcia

et

al.

-RRB-

incontinence

-LRB-

Garley

and

Unwin

-RRB-

or

multiple

sclerosis

-LRB-

Thomas

et

al.

-RRB-

but

these

have

not

yet

been

transformed

into

a

web

based

intervention

although

trials

found

positive

effects

of

face-to-face

CBT

for

these

problems

Because

the

promising

results

of

earlier

studies

and

because

of

the

benefits

of

Internet-based

interventions

we

can

expect

development

of

new

programs

for

these

conditions

in

the

future

Our

review

does

not

cover

the

whole

field

of

internet

interventions

While

we

focused

on

CBT

for

existing

health

problems

several

other

studies

have

examined

CBT

for

mental

health

problems

-LRB-

Spek

et

al.

-RRB-

on

internet-based

preventive

interventions

aimed

at

a

healthy

lifestyle

-LRB-

weight

loss

smoking

exercise

e.g.

Swartz

et

al.

Muñoz

et

al.

Oenema

et

al.

Tate

et

al.

-RRB-

and

interventions

using

non-CBT

methods

-LRB-

McMahon

et

al.

Edwards

et

al.

Gray

et

al.

-RRB-

However

as

was

shown

in

this

review

research

on

CBT

interventions

has

been

growing

fast

in

the

past

few

years

Because

CBT

interventions

are

very

well

suited

to

be

used

through

the

internet

it

can

be

expected

that

research

in

this

area

will

continue

to

grow

further

in

the

next

years

The

included

studies

do

not

yet

allow

us

to

draw

definite

conclusions

about

whether

CBT

through

the

Internet

are

as

effective

as

face-to-face

interventions

For

most

health

problems

we

found

only

one

study

examining

the

effects

of

an

Internet-based

CBT

study

In

fact

it

was

only

for

pain

and

headache

did

we

find

more

than

one

studies

However

the

effects

found

for

Internet-based

interventions

aimed

at

pain

are

comparable

to

the

effects

found

for

face-to-face

treatments

for

pain

-LRB-

Morley

et

al.

-RRB-

and

the

same

is

most

likely

true

for

the

Internet-interventions

aimed

at

headache

-LRB-

Bogaards

and

ter

Kuile

-RRB-

The

other

interventions

also

found

some

effects

although

some

effects

were

stronger

than

others

It

does

seem

clear

however

that

Internet-based

CBT

can

have

significant

effects

on

some

of

the

health

problems

described

in

this

review

For

at

least

one

of

the

conditions

tinnitus

an

effectiveness

study

has

been

published

showing

better

results

then

the

first

controlled

efficacy

trial

-LRB-

Kaldo-Sandström

et

al.

-RRB-

It

has

been

suggested

self-help

interventions

be

used

as

one

of

the

first

steps

in

stepped-care

programs

-LRB-

Scogin

et

al.

-RRB-

Perhaps

Internet-based

interventions

which

are

used

in

healthcare

settings

should

also

be

placed

within

these

stepped-care

frameworks

In

these

cases

additional

care

is

available

if

the

Internet-based

intervention

does

not

reduce

the

problem

of

a

patient

sufficiently

On

the

other

hand

Internet

interventions

can

develop

as

well

and

might

at

least

for

some

patients

be

more

suitable

than

face

to

face

CBT

As

many

health

conditions

such

as

chronic

pain

and

cancer

require

a

multidisciplinary

team

approach

for

optimal

treatment

we

assume

that

future

Internet

interventions

will

take

advantage

of

this

possibility

There

is

no

consensus

yet

among

researchers

about

the

way

CBT

should

be

presented

on

the

Internet

although

standards

are

emerging

Most

interventions

used

a

guided

self-help

format

in

which

the

treatment

protocol

is

presented

on

the

Internet

and

the

patient

works

it

through

more

or

less

independently

The

patients

are

supported

by

brief

contacts

with

therapists

through

e-mail

or

telephone

However

other

studies

use

a

more

traditional

format

in

that

the

patients

go

online

at

the

same

time

as

the

therapist

and

have

a

more

or

less

regular

treatment

session

Group

treatments

can

also

be

delivered

in

such

a

way

Another

difference

between

interventions

concerns

the

additional

elements

on

the

Internet

apart

from

CBT

Some

interventions

have

combined

the

cognitive

behavioral

interventions

with

other

components

such

as

psychoeducation

films

and

texts

to

read

and

a

forum

for

users

of

the

website

Other

interventions

do

not

provide

such

extras

Our

review

showed

up

several

other

important

limitations

of

the

current

research

in

this

area

First

most

studies

used

waiting

list

control

groups

and

only

very

few

used

a

care-as-usual

or

another

control

group

Subjects

in

waiting

list

control

groups

probably

do

not

take

constructive

action

to

reduce

their

problems

themselves

during

the

waiting

period

because

they

are

expecting

professional

help

in

the

future

This

may

result

in

an

overestimation

of

the

effects

of

an

intervention

because

there

may

be

less

spontaneous

recovery

Second

most

studies

recruited

participants

through

the

community

and

through

other

websites

This

is

not

a

problem

for

interventions

that

target

the

general

population

But

when

such

an

intervention

is

effective

this

does

not

automatically

mean

that

it

is

also

effective

in

clinical

settings

Subjects

who

are

responding

to

community

recruitment

are

probably

very

motivated

which

may

improve

their

results

compared

to

subjects

who

receive

treatment

Third

of

the

twelve

identified

trials

compared

Internet-based

treatments

to

face-to-face

or

other

treatments

This

is

however

an

important

issue

because

only

direct

comparisons

can

give

evidence

about

the

comparative

effects

of

Internet-based

treatments

compared

to

more

traditional

treatments

and

the

type

of

patients

who

can

benefit

from

it

Fourth

most

studies

were

aimed

at

adults

Only

two

studies

were

aimed

at

children

and

adolescents

while

these

groups

are

probably

the

most

familiar

with

the

Internet

of

the

studies

were

specifically

aimed

at

older

adults

while

they

suffer

most

from

health

conditions

Future

research

should

focus

on

these

limitations

of

current

research

More

studies

are

needed

with

care-as-usual

or

other

control

groups

clinical

recruitment

strategies

comparisons

with

face-to-face

treatments

and

children

or

older

adults

as

target

populations

More

research

is

also

needed

to

examine

how

CBT

should

be

presented

on

the

Internet

and

to

examine

reasons

and

solutions

to

the

relatively

high

drop-out

rates

in

several

studies

Finally

it

is

also

important

to

study

how

Internet-administered

CBT

can

be

integrated

in

stepped-care

models

of

care

This

review

has

several

limitations

First

the

number

of

included

studies

is

still

very

small

And

the

number

of

studies

examining

specific

health

problems

is

too

small

to

integrate

the

results

of

these

studies

statistically

into

a

meta-analysis

Second

the

quality

of

the

included

studies

is

not

optimal

Third

the

drop-out

rates

reported

are

high

in

some

studies

This

is

a

concern

for

this

type

of

intervention

as

patients

can

very

easily

withdraw

from

the

intervention

Remarkably

the

studies

in

which

more

traditional

therapies

-LRB-

live

sessions

with

therapists

-RRB-

are

delivered

through

the

Internet

have

the

lowest

drop-out

rates

Despite

these

limitations

however

there

is

no

doubt

that

the

number

of

studies

in

this

area

will

increase

considerably

in

the

next

few

years

while

the

promising

results

of

the

studies

in

this

review

indicate

that

the

Internet

will

assume

a

major

role

in

the

delivery

of

CBT

to

patients

with

health

problems

O B I I O O O O O B I O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O

systematic review
internet
cognitive behavior therapy
pain
headache


Can

Examination

of

WWW

Usage

Statistics

and

other

Indirect

Quality

Indicators

Help

to

Distinguish

the

Relative

Quality

of

Medical

websites

Background

The

Internet

offers

a

great

amount

of

health

related

websites

but

concern

has

been

raised

about

their

reliability

Several

subjective

evaluation

criteria

and

websites

rating

systems

have

been

proposed

as

a

help

for

the

Internet

users

to

distinguish

among

web

resources

with

different

quality

but

their

efficacy

has

not

been

proven

Introduction

After

the

early

enthusiasm

generated

by

the

potential

use

of

the

Internet

in

Medicine

-LSB-

-RSB-

concern

has

been

raised

about

the

quality

of

the

resources

available

on

the

Internet

compared

to

more

academic

media

It

is

technically

very

easy

to

publish

on

the

Internet

-LSB-

-RSB-

The

lack

of

a

review

process

of

the

documents

on

the

Net

and

the

power

of

this

media

in

transmitting

the

data

has

the

risk

of

misinforming

both

lay

people

-LSB-

-RSB-

and

health

care

professionals

-LSB-

-RSB-

However

only

a

few

studies

have

tried

to

measure

this

risk

of

misinformation

-LSB-

-RSB-

Nothing

yet

is

known

about

the

users

ability

to

discriminate

between

low

and

high

quality

resources

Several

initiatives

have

been

proposed

which

could

be

applied

at

different

levels

to

improve

the

average

quality

of

medical

websites

For

instance

we

could

apply

certain

basic

methods

for

the

websites

to

be

correctly

designed

In

this

sense

some

academic

organizations

have

proposed

a

set

of

basic

information

that

every

medical

web

site

should

provide

about

the

author

and

sources

of

the

web

site

contents

their

potential

conflicts

of

interest

and

funding

and

the

currency

of

the

information

-LSB-

-RSB-

But

many

of

the

available

medical

websites

have

been

created

without

any

quality

control

by

a

third

party

How

can

Internet

health

care

visitors

distinguish

between

such

different

resources

Internet

users

can

find

health

and

medical

related

websites

in

several

ways

World

Wide

Web

search

engines

-LRB-

e.g.

AltaVista

Excite

Infoseek

and

many

others

-RRB-

provide

the

users

with

a

list

of

websites

that

match

a

given

topic

with

the

results

ordered

by

syntactic

similarity

with

the

query

-LSB-

-RSB-

Unfortunately

the

quality

of

contents

is

not

guaranteed

On

the

other

hand

certain

websites

indexes

and

review

services

such

as

Medical

Matrix

-LRB-

-RRB-

and

HealthAtoZ

-LRB-

-RRB-

offer

systematic

evaluations

of

medical

resources

on

the

Web

-LSB-

-RSB-

as

a

post

publication

editorial

process

These

rating

systems

could

be

an

useful

tool

for

guiding

the

visitors

of

medical

websites

-LSB-

-RSB-

However

authors

who

have

reviewed

these

Internet

resources

point

out

the

variability

of

their

evaluation

criteria

and

their

doubtful

efficacy

-LSB-

-RSB-

The

quality

of

a

given

medical

article

on

the

Internet

could

be

measured

by

the

users

opinion

about

it

for

example

by

counting

the

number

of

times

it

is

retrieved

-LSB-

-RSB-

However

this

idea

has

been

criticized

because

it

would

replace

the

scientific

peer

review

process

with

the

opinion

of

the

Internet

users

whatever

their

qualification

-LSB-

-RSB-

Despite

the

differences

between

the

printed

medical

information

and

the

Internet

several

evaluation

tools

from

the

former

could

be

useful

if

applied

on

the

Net

Similarly

to

printed

medical

journals

medical

documents

on

the

Internet

could

be

ranked

by

their

citation

analysis

-LSB-

-RSB-

but

no

methods

have

been

proven

for

use

with

medical

websites

When

an

article

is

quoted

in

a

paper

certain

agreement

among

the

authors

may

be

supposed

Similarly

when

a

webmaster

makes

a

link

from

his

web

site

to

another

certain

credibility

is

given

to

the

latter

In

fact

the

International

Committee

of

Medical

Journals

Editors

recommend

caution

when

a

link

is

made

from

a

peer

reviewed

journal

site

to

other

sites

-LSB-

-RSB-

If

linking

on

the

web

can

be

equivalent

to

quoting

in

printed

medical

articles

a

citation

analysis

on

the

web

could

be

performed

by

the

quantification

of

the

links

to

a

given

medical

web

site

The

ideal

method

for

assessing

the

quality

of

medical

websites

should

provide

a

means

of

rating

great

amounts

of

medical

web

resources

while

respecting

the

World

Wide

Web

peculiarities

such

as

its

multimedia

capabilities

and

changing

contents

At

the

same

time

it

should

at

least

be

as

reliable

as

systematic

reviews

of

those

resources

by

editorial

boards

In

summary

it

should

be

a

method

born

in

the

Internet

but

with

the

efficacy

of

those

used

in

the

printed

media

In

this

study

we

evaluated

the

reliability

of

four

websites

characteristics

as

medical

websites

quality

indicators

The

four

characteristis

used

their

authors

impact

factor

their

grade

of

updating

their

daily

visits

and

inbound

links

The

evaluations

of

a

sample

of

pediatric

websites

by

a

number

of

Internet

rating

systems

was

the

gold

standard

with

which

these

websites

characteristics

were

compared

Methods

During

March

a

subset

of

websites

rating

systems

were

compiled

From

these

we

selected

a

sample

of

websites

that

were

studied

during

the

first

week

of

April

Eight

web

rating

systems

whose

evaluations

were

offered

as

figures

were

compiled

from

previous

studies

-LSB-

-RSB-

-LRB-

Table

-RRB-

One

half

of

the

selected

rating

systems

gave

the

results

of

their

evaluations

by

means

of

graphic

analog

scales

and

the

other

half

by

numeric

scales

Every

web

site

evaluated

by

these

rating

systems

that

provided

information

about

child

health

whether

for

lay

people

or

health

professionals

was

included

in

the

study

Some

of

these

rating

systems

-LRB-

e.g.

Lycos

Top

-RRB-

provides

the

visitors

with

a

search

tool

by

keyword

In

these

cases

the

websites

were

selected

using

the

keywords

Pediatrics

Infancy

Child

health

and

Child

Care

For

the

remaining

rating

systems

the

pediatric

websites

were

compiled

manually

Those

websites

not

accessible

twice

during

the

study

period

were

excluded

Only

three

rating

systems

-LRB-

Medical

Matrix

Physician

Choice

and

Six

Senses

-RRB-

gave

information

about

their

editorial

boards

Most

of

their

members

were

physicians

Two

of

the

web

rating

systems

only

gave

a

global

result

of

their

websites

evaluation

-LRB-

Medical

Matrix

and

Magellan

-RRB-

while

the

rest

-LRB-

HealthAtoZ

Argus

Clearinghouse

Lycos

Top

Sympatico

Health

Physician

Choice

and

Six

Senses

-RRB-

gave

a

result

for

each

considered

criterion

Content

was

a

common

criterion

to

all

the

eight

ranking

systems

Therefore

the

results

of

the

evaluation

of

each

web

site

were

divided

in

two

categories

content

and

non-content

-LRB-

design

-RRB-

aspects

In

order

to

make

comparisons

the

results

of

the

evaluations

of

the

websites

supplied

by

each

rating

system

were

transformed

to

a

one

hundred

scale

Table

Compiled

web

sites

ranking

systems

The

results

of

evaluations

are

showed

as

two

possible

types

of

scales

graphic

analog

-LRB-

A

-RRB-

or

numeric

-LRB-

N

-RRB-

Rating

systems

-LRB-

Included/excluded

web

sites

-RRB-

Uniform

Resource

Locator

Type

of

scale

Argus

Clearinghouse

Seal

of

Approval

-LRB-

-RRB-

A

HealthAtoZ

-LRB-

-RRB-

Lycos

Top

-LRB-

-RRB-

N

Magellan

Internet

Guide

-LRB-

-RRB-

A

Medical

Matrix

-LRB-

-RRB-

A

Physician

's

choice

-LRB-

-RRB-

N

Six

Senses

Seal

of

Approval

-LRB-

-RRB-

N

Sympatico

Health

-LRB-

-RRB-

A

Graphic

analog

scale

developed

in

numeric

When

provided

the

daily

visits

registered

by

the

websites

visits

counters

were

recorded

In

some

websites

the

date

from

which

the

counter

was

started

was

not

available

Thus

their

webmasters

were

asked

for

this

information

by

electronic

mail

and

it

was

included

in

the

statistical

study

if

provided

before

the

end

of

the

observation

period

April

The

websites

authors

and

editors

names

were

searched

in

MEDLINE

-LSB-

-RSB-

and

their

articles

were

registered

Their

impact

factors

of

the

journals

wherein

they

were

published

were

obtained

by

using

the

Science

Citation

Index

-LRB-

Institute

for

Scientific

Information

Philadelphia

PA

-RRB-

The

impact

factor

of

a

given

web

site

author

was

the

sum

of

the

impact

factors

of

his

or

her

articles

For

institutional

websites

only

the

name

of

the

web

editor

was

considered

When

provided

the

time

since

the

last

update

was

also

recorded

Finally

by

means

of

the

Web

search

engine

Infoseek

-LSB-

-RSB-

we

calculated

how

many

websites

on

the

Internet

linked

to

each

web

site

of

our

sample

The

searching

strategy

by

syntax

of

this

engine

allows

to

know

the

websites

that

are

linked

to

a

given

web

site

-LSB-

-RSB-

As

a

web

site

may

be

linked

not

only

from

external

websites

but

also

from

websites

of

its

own

organization

we

only

considered

external

links

Although

other

search

engines

such

as

AltaVista

Excite

or

HotBot

offer

similar

searching

options

we

chose

Infoseek

because

it

provided

the

results

of

the

queries

grouped

by

web

site

which

makes

the

exclusion

of

the

internal

links

easier

Comparison

of

means

was

performed

by

Mann-Whitney

U

test

and

correlation

analysis

by

means

of

Spearman

's

correlation

coefficient

-LRB-

rS

-RRB-

P

values

equal

or

less

than

were

considered

significant

All

computations

were

made

with

SPSS

for

Windows

-LRB-

SPSS

Inc.

Chicago

IL

-RRB-

statistical

package

Results

After

excluding

non-accessible

websites

a

total

of

pediatric

websites

were

compiled

Table

Correlations

among

the

number

of

daily

visits

to

the

web

sites

the

impact

factor

of

their

authors

or

editors

the

grade

of

update

and

the

number

of

links

that

receive

NS

means

not

significant

Number

of

inbound

links

rS

p

Visits/day

rS

p

Author

's

impact

factor

rS

p

Visits/day

Author

impact

factor

NS

NS

Weeks

since

the

last

update

NS

NS

Table

Correlation

among

the

number

of

links

and

visits

to

the

web

sites

the

impact

factor

of

their

authors

and

the

time

since

the

last

update

and

the

results

of

their

evaluation

by

HealthAtoZ

and

Medical

Matrix

No

significant

correlations

were

demonstrated

with

the

other

systems

Medical

Matrix

only

provides

total

results

does

not

specify

results

by

contents

and

non-contents

aspects

Number

of

Inbound

Links

Visits/day

Author

impact

factor

Weeks

since

the

last

update

rS

p

rS

p

rS

p

rS

p

HealthAtoZ

Total

Contents

Non

contents

NS

NS

NS

Medical

Matrix

Total

NS

NS

NS

On

average

the

websites

of

our

sample

received

links

from

other

sites

on

the

Internet

-LRB-

range

to

-RRB-

In

of

the

websites

information

on

their

last

update

was

given

On

average

they

had

been

updated

weeks

before

-LRB-

range

to

-RRB-

Only

of

the

websites

had

a

visit

counter

and

the

average

daily

visits

were

-LRB-

range

to

-RRB-

Seven

visit

counters

did

not

distinguish

among

different

visitors

that

is

they

registered

any

visit

to

their

websites

In

websites

-LRB-

-RRB-

the

editor/author

's

name

was

given

but

only

of

them

had

published

at

least

one

article

since

January

in

the

journals

included

in

MEDLINE

database

Their

average

impact

factor

was

Figure

Weeks

since

the

last

update

for

the

total

of

the

sample

n

and

for

the

websites

evaluated

at

least

by

two

rating

systems

n

-LRB-

median

and

percentiles

-RRB-

Figure

Number

of

inbound

links

to

websites

for

the

total

of

the

sample

n

and

for

the

websites

evaluated

at

least

by

two

rating

systems

n

-LRB-

median

and

percentiles

-RRB-

Table

Top

pediatric

web

sites

of

the

sample

-LRB-

N

-RRB-

by

the

number

of

their

inbound

links

The

weeks

since

the

last

update

the

number

of

daily

visits

to

the

web

sites

and

their

editor/author

's

impact

factor

are

also

provided

In

parenthesis

the

place

that

each

web

site

would

obtain

if

ranked

by

the

two

latter

criteria

In

italics

those

web

sites

indexed

at

least

by

two

rating

systems

Missing

values

are

due

to

the

lack

of

visits

counter

editor

's

name

or

information

about

the

last

update

for

many

web

sites

Uniform

Resource

Locator

Number

of

inbound

links

Daily

visits

to

web

sites

Web

site

editor/author

's

impact

factor

Weeks

since

the

last

update

-LRB-

3º

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

8º

-RRB-

-LRB-

5º

-RRB-

-LRB-

6º

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

1º

-RRB-

-LRB-

-RRB-

-LRB-

1º

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

6º

-RRB-

-LRB-

2º

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Only

websites

of

the

sample

were

indexed

and

evaluated

at

least

by

two

rating

systems

and

by

the

eight

This

subset

of

websites

showed

significantly

better

results

of

the

evaluation

of

their

contents

and

design

by

HealthAtoZ

and

higher

grade

of

updating

-LRB-

Figure

-RRB-

and

higher

number

of

inbound

links

-LRB-

Figure

-RRB-

When

the

evaluations

of

these

websites

by

the

different

rating

systems

were

compared

no

significant

correlations

were

found

Changes

regarding

the

average

impact

factor

of

the

authors

of

the

websites

or

the

number

of

daily

visits

could

not

be

demonstrated

in

this

subset

of

websites

Some

interesting

correlations

between

the

results

of

the

evaluations

of

the

websites

and

the

rest

of

study

variables

were

found

The

number

of

links

received

by

the

websites

significantly

correlated

with

their

daily

visits

and

with

the

time

since

the

last

update

-LRB-

Table

-RRB-

The

number

of

inbound

links

also

correlated

with

the

results

of

the

websites

evaluation

by

HealthAtoZ

-LRB-

Table

-RRB-

The

number

of

daily

visits

significantly

correlated

with

the

results

of

the

websites

evaluation

made

by

Medical

Matrix

and

the

grade

of

updating

significantly

correlated

with

the

results

of

the

contents

and

designs

evaluation

made

by

HealthAtoZ

-LRB-

Table

-RRB-

Finally

no

correlation

was

demonstrated

between

the

average

impact

factor

of

the

websites

authors

and

the

other

variables

The

top

fifty

pediatric

websites

of

the

sample

are

shown

in

Table

ordered

by

the

number

of

their

inbound

links

according

to

the

Infoseek

indexing

engine

More

than

a

half

of

the

websites

indexed

by

at

least

two

rating

systems

may

be

found

among

these

top

fifty

websites

Discussion

In

this

study

certain

websites

characteristics

that

depend

on

the

users

preferences

have

been

compared

with

evaluations

of

pediatric

resources

on

the

Web

by

third

parties

Although

rating

systems

have

been

previously

criticized

because

their

editorial

boards

frequently

do

not

employ

uniform

criteria

-LSB-

-RSB-

we

have

considered

them

as

the

standard

method

because

it

somewhat

represents

a

post-publication

review

process

Some

aspects

of

our

method

are

open

to

discussion

Firstly

the

reliability

of

the

data

regarding

the

daily

visits

and

the

updating

frequency

depends

on

the

accuracy

of

the

information

that

the

websites

editors

offer

in

their

sites

In

this

sense

we

considered

the

grade

of

updating

of

the

websites

by

the

dates

of

their

last

changes

Clearly

these

changes

could

involve

very

different

aspects

and

in

different

grades

and

not

necessarily

provide

more

current

contents

However

we

believe

that

it

could

demonstrate

the

editor

's

efforts

in

maintaining

or

increasing

the

interest

of

his

web

site

for

the

visitors

The

results

regarding

the

number

of

daily

visits

to

the

websites

must

be

considered

with

caution

when

comparing

one

web

site

to

another

because

some

visit

counters

were

set

to

register

every

visit

instead

of

every

distinct

visitor

Nevertheless

both

can

be

considered

usage

indexes

of

a

given

web

site

On

the

other

hand

quantification

of

links

to

the

websites

clearly

depends

on

the

power

of

the

search

engine

we

employ

By

no

means

our

results

show

the

total

number

of

links

to

the

websites

in

our

sample

In

fact

a

previous

article

states

that

it

would

be

necessary

to

combine

the

databases

from

at

least

five

large

search

engines

to

cover

the

most

of

the

web

-LSB-

-RSB-

Although

all

bibliometric

indexes

have

limitations

-LSB-

-RSB-

we

employed

the

impact

factor

as

a

measure

of

the

webmasters

publishing

capacity

because

it

is

a

classical

indicator

of

the

quality

of

biomedical

articles

Recently

it

has

been

suggested

that

every

medical

web

site

should

be

evaluated

following

some

basic

criteria

-LSB-

-RSB-

One

of

the

more

accepted

criteria

is

that

the

authorship

must

be

clearly

stated

as

a

basic

means

for

assessing

the

reliability

of

the

web

site

contents

However

we

could

not

demonstrate

that

the

more

highly

evaluated

the

most

updated

or

the

most

linked

or

visited

pediatric

websites

had

the

authors

with

the

highest

publishing

capacity

measured

by

their

impact

factor

In

other

words

some

web

quality

standards

do

not

correlate

with

classical

quality

standards

from

the

printed

media

such

as

the

impact

factor

of

a

given

author

's

articles

We

could

not

find

statically

significant

correlations

among

the

evaluations

of

the

websites

by

the

different

rating

systems

This

is

probably

due

to

the

small

size

of

the

subset

of

websites

indexed

and

evaluated

by

all

the

systems

and

their

different

evaluation

criteria

However

some

interesting

data

were

found

when

we

considered

the

correlations

among

the

four

websites

characteristics

and

the

evaluations

We

found

that

the

best

websites

for

HealthAtoZ

the

largest

analyzed

rating

system

were

the

most

updated

and

the

most

linked

ones

On

the

other

hand

the

most

valuable

websites

for

Medical

Matrix

the

second

rating

system

by

size

were

the

most

visited

ones

In

any

case

both

the

number

of

daily

visits

and

the

time

since

the

last

update

highly

correlated

with

the

number

of

inbound

links

The

lack

of

correlation

among

the

four

variables

and

the

evaluations

by

the

other

rating

systems

could

be

due

to

their

little

contribution

to

our

sample

Many

efforts

to

establish

quality

criteria

will

have

limited

efficacy

due

to

the

dynamic

behaviour

of

the

Internet

as

a

publishing

medium

In

fact

a

recent

article

demonstrates

the

lack

of

consensus

among

the

editorial

boards

of

a

large

sample

of

evaluation

and

rating

systems

regarding

the

evaluation

criteria

they

employ

The

same

authors

pointed

out

that

it

may

be

difficult

or

even

inappropriate

to

develop

a

static

tool

or

system

for

assessing

health

related

websites

-LSB-

-RSB-

Therefore

the

question

could

be

to

provide

context

to

this

issue

That

is

to

know

how

good

a

given

medical

web

site

is

in

comparison

with

the

rest

of

medical

websites

A

democratic

and

feasible

method

for

reaching

this

objective

could

be

let

the

Internet

community

say

which

medical

websites

are

the

best

ones

that

is

which

they

usually

visit

or

which

they

usually

recommend

by

linking

to

them

Moreover

we

believe

that

the

fact

that

these

usage

indexes

correlate

with

the

evaluations

by

third

parties

qualifies

them

as

quality

markers

Eysenbach

and

Diepgen

-LSB-

-RSB-

have

recently

proposed

that

an

ideal

quality

control

system

for

medical

resources

on

the

Internet

should

take

in

account

the

users

opinion

and

not

only

their

evaluation

by

a

third

party

that

is

a

downstream

filtering

and

not

only

an

upstream

filtering

approach

More

interestingly

our

study

demonstrates

certain

agreement

among

both

approaches

in

identifying

high

quality

resources

LaPorte

et

al

-LSB-

-RSB-

proposed

an

electronic

publishing

system

in

which

the

impact

of

a

given

resource

on

the

Internet

could

be

measured

by

counting

how

many

times

the

document

was

retrieved

or

quoted

The

introduction

of

the

citation

analysis

of

the

medical

resources

on

the

web

as

a

method

to

assess

their

quality

has

been

recently

proposed

-LSB-

-RSB-

On

the

other

hand

a

very

promising

software

system

is

being

developed

by

Kleinberg

-LSB-

-RSB-

This

system

would

provide

the

users

with

a

way

of

knowing

the

very

best

of

the

web

on

a

given

topic

in

a

faster

and

more

complete

way

than

commercial

human

compiled

directories

This

system

is

based

in

the

identification

of

two

subsets

of

websites

when

a

query

on

a

given

topic

is

made

those

websites

containing

a

lot

of

information

about

the

topic

-LRB-

authoritative

websites

-RRB-

and

those

which

contain

large

amounts

of

links

to

the

former

-LRB-

hub

sites

-RRB-

Our

work

demonstrates

that

those

authoritative

websites

that

is

the

more

linked

ones

are

indeed

the

best

ones

regarding

the

evaluation

of

its

contents

and

design

by

the

editorial

boards

of

some

large

web

rating

systems

The

citation

analysis

of

biomedical

journals

has

been

a

classic

tool

in

assessing

their

relative

quality

-LSB-

-RSB-

Similarly

medical

web

resources

could

be

ranked

by

a

webcite

index

-LSB-

-RSB-

which

is

not

yet

defined

Linking

in

the

World

Wide

Web

could

be

equivalent

to

quoting

in

printed

publications

and

its

quantification

could

be

useful

for

measuring

the

relative

quality

of

medical

websites

Some

indexes

could

be

created

to

make

more

rational

comparisons

among

websites

with

different

sizes

For

example

in

the

same

way

that

the

calculation

of

the

impact

factor

of

a

given

medical

journal

takes

into

account

the

number

of

articles

published

by

that

journal

yearly

the

size

of

a

given

domain

could

be

considered

to

obtain

some

indexes

that

would

express

more

accurately

the

grade

of

linkage

of

a

medical

web

site

Moreover

Platform

for

Internet

Content

Selection

-LRB-

PICS

-RRB-

-LSB-

-RSB-

an

infrastructure

that

could

be

applied

as

a

filtering

system

of

the

medical

information

on

the

Net

-LSB-

-RSB-

could

incorporate

these

indexes

as

one

of

the

meta

data

assigned

to

every

medical

document

as

electronic

labels

Then

these

electronic

labels

could

be

checked

automatically

by

an

user

's

browser

bypassing

those

documents

with

a

webcite

index

not

high

enough

A

problem

could

be

how

to

avoid

false

self-labelling

by

dishonest

webmasters

In

any

case

more

work

is

needed

to

give

answers

to

these

and

other

technical

questions

on

the

emerging

field

of

Webometrics

-LSB-

-RSB-

An

evaluation

system

based

on

this

quantification

would

bring

advantages

and

risks

Rankings

could

be

generated

very

quickly

and

in

an

objective

way

because

the

Internet

community

by

itself

would

evaluate

great

amounts

of

medical

websites

However

this

evaluation

process

would

be

made

a

posteriori

and

the

potential

harmful

effects

of

the

diffusion

of

documents

without

enough

quality

could

not

be

avoided

Therefore

this

method

could

not

replace

previous

editorial

effort

that

warrants

a

minimal

quality

for

each

resource

Our

work

demonstrates

that

the

visitors

of

pediatric

websites

and

the

editors

of

websites

on

the

Net

so

called

webmasters

show

certain

maturity

when

they

have

to

identify

the

pediatric

resources

with

high

quality

We

believe

that

the

key

point

is

how

to

augment

the

proportion

of

these

resources

An

important

issue

could

be

to

establish

a

citation

style

not

only

for

articles

from

peer

reviewed

electronic

journals

-LSB-

-RSB-

but

also

for

any

medical

document

on

the

Net

The

prestige

that

citation

in

a

printed

journal

represents

will

stimulate

high

quality

publishing

on

the

Internet

and

web

site

editors

will

employ

enough

review

processes

to

obtain

the

necessary

quality

A

web

site

's

ranking

system

based

on

the

citation

analysis

on

the

web

by

the

quantification

of

links

would

be

an

additional

incentive

The

more

valuable

resources

will

attract

the

Internet

users

visits

and

the

webmasters

links

and

very

likely

the

best

funding

and

financial

supports

In

summary

although

the

Internet

provides

a

very

different

publishing

medium

traditional

means

borrowed

from

printed

journals

could

also

be

used

with

this

electronic

media

for

achieving

minimal

levels

of

quality

These

include

certain

peer

review

processes

that

enhance

the

rigor

of

the

documents

submitted

for

publication

taking

in

account

the

peculiarities

of

this

media

and

linking

analysis

as

a

measure

of

the

citation

on

the

World

Wide

Web

O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

health education
information systems
computer communication networks
bibliometrics
cybermetrics
web metrics