ISRN

SICS-R

SE

SICS

research

report

ISSN

Neural

Networks

for

Wordform

Recognition

by

Martin

Eineborg

and

Bj?orn

Gamb?ack

ISSN

Neural

Networks

for

Wordform

Recognition

Martin

Eineborg

eineborg@sics.se

Bj?orn

Gamb?ack

gam@sics.se

February

Natural

Language

Processing

Group

Swedish

Institute

of

Computer

Science

Box

S

-LCB-

Kista

Sweden

Abstract

The

paper

outlines

a

method

for

automatic

lexical

acquisition

using

three-layered

back-propagation

networks

Several

experiments

have

been

carried

out

where

the

performance

of

different

network

architectures

have

been

compared

to

each

other

on

two

tasks

overall

part-of-speech

-LRB-

noun

adjective

or

verb

-RRB-

classification

and

classification

by

a

set

of

possible

output

categories

The

best

results

for

the

simple

task

were

obtained

by

networks

consisting

of

-LCB-

input

neurons

and

hidden-layer

neurons

reaching

a

classification

rate

of

The

best

result

for

the

more

complex

task

was

which

was

achieved

by

a

net

with

input

neurons

and

hidden-layer

neurons

These

results

are

rather

promising

and

the

paper

compares

them

to

the

performance

reported

by

rule-based

and

purely

statistical

methods

a

comparison

that

shows

the

neural

network

completely

compatible

with

the

statistical

approach

The

rule-based

method

is

however

still

better

even

though

it

should

noted

that

the

task

that

the

rule-based

system

performs

is

somewhat

different

from

that

of

the

neural

net

Lexical

acquisition

Also

Published

A

shorter

version

of

this

paper

appeared

in

the

Proceedings

of

the

International

Conference

on

Neural

Networks

and

Their

Industrial

Cognitive

Applications

N

mes

France

October

ii

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

iii

Contents

Introduction

Information

Sources

The

unknown

word

Contextual

information

Architectures

Notations

Definitions

Encoding

of

letters

Encoding

of

grammatical

categories

Backpropagation

networks

Experiments

Table

Network

setups

for

the

experiments

Results

Figure

Simple

task

network

performance

Figure

Complex

task

network

performance

Discussion

and

related

work

Conclusions

and

future

work

Acknowledgements

References

Appendices

A

Encoding

of

letters

B

Encoding

of

grammatical

categories

C

A

test

sentence

D

A

sample

from

the

corpus

E

Examples

of

the

encoding

of

data

iv

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Introduction

Within

the

language

area

connectionist

models

have

so

far

proved

discouragingly

unsuccessful

compared

to

other

methods

such

as

rule-based

approaches

and

full-fledged

statistical

methods

Even

though

they

have

been

tried

out

for

several

applications

such

as

semantic

clustering

preposition

choice

etc.

the

only

language

area

where

artificial

neural

networks

have

been

successfully

applied

on

a

larger

scale

has

been

speech

however

all

currently

leading

speech

recognition

systems

-LRB-

the

ones

in

the

US

DARPA

race

-RRB-

have

discarded

neural

nets

for

Hidden

Markov

Models

The

current

state

of

affairs

should

however

hardly

be

taken

to

be

the

permanent

truth

The

need

for

different

machine

learning

methods

within

the

language

area

should

be

evident

in

this

paper

we

will

single

out

the

topic

of

lexical

acquisition

for

special

attention

but

the

last

words

for

other

areas

such

as

disambiguation

document

matching

information

retrieval

grammar

and

transfer-rule

induction

etc.

have

certainly

not

been

said

The

need

for

automatic

lexical

acquisition

has

increased

in

recent

years

since

modern

natural

language

processing

systems

and

theories

tend

to

incorporate

more

and

more

information

into

the

lexicon

information

that

has

traditionally

resided

in

the

grammar

rules

Acquiring

a

lexicon

for

such

a

system

has

thus

become

a

difficult

and

time

consuming

task

even

for

moderately

sized

lexica

Still

any

natural

language

processing

system

intended

for

serious

applications

must

include

a

large

lexicon

-LCB-

several

thousands

of

words

or

more

is

commonly

considered

a

minimum

size

-LCB-

which

adds

even

more

to

the

complexity

of

the

problem

In

view

of

this

tools

for

lexical

acquisition

are

not

only

desirable

-LCB-

they

become

an

absolute

necessity

Most

approaches

to

this

problem

have

been

to

construct

a

range

of

tools

that

require

various

degrees

of

interactive

support

when

new

lexical

entries

are

created

either

from

raw

text

material

-LRB-

as

in

e.g.

-LSB-

Grosz

et

al

Carter

Wilensky

Gamb?ack

-RSB-

-RRB-

and

the

early

work

by

Zernik

-LSB-

Zernik

Dyer

Zernik

-RSB-

or

from

machine

readable

dictionaries

-LRB-

see

e.g.

-LSB-

Boguraev

et

al

Calzolari

Bindi

-RSB-

-RRB-

These

interactive

tools

of

course

greatly

simplifies

the

lexicon

acquisition

task

but

it

is

still

desirable

to

go

one

step

further

and

make

the

whole

process

automatic

Creating

a

realistically

sized

lexicon

using

interactive

tools

is

quite

timeconsuming

even

in

a

limited

domain

and

will

generally

require

the

lexicon

builder

to

be

a

domain

expert

It

is

often

difficult

if

not

impossible

to

know

a

complete

vocabulary

for

a

domain

beforehand

and

therefore

likely

that

input

to

the

natural-language

system

will

contain

unknown

words

no

matter

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

how

carefully

the

lexicon

has

been

designed

A

system

for

automatic

lexical

acquisition

can

be

designed

to

analyze

large

text

corpora

representing

typical

texts

from

a

particular

domain

It

could

then

collect

and

process

several

occurrences

of

a

particular

unknown

word

in

different

contexts

and

thus

eliminate

or

at

least

substantially

reduce

the

need

for

interaction

with

a

domain

expert

when

building

a

large

lexicon

One

of

the

first

systems

that

aimed

at

constructing

lexical

entries

automatically

from

raw

text

was

Granger

's

FOUL-UP

system

-LSB-

Granger

-RSB-

FOUL-UP

was

an

integral

part

of

the

SAM

system

-LSB-

Schank

-RSB-

developed

to

extend

SAM

's

lexicon

by

inferring

restrictions

placed

on

unknown

words

by

instantiating

scripts

that

matched

the

sentences

containing

the

unknown

words

This

built

on

a

number

of

assumptions

which

in

general

do

not

hold

in

particular

that

all

the

information

needed

to

create

an

entry

is

contained

in

one

text

that

no

morphological

information

is

needed

that

specific

-LRB-

hand-coded

-RRB-

scripts

covering

the

domain

can

be

made

available

Jacobs

and

Zernik

-LSB-

Jacobs

Zernik

-RSB-

showed

the

need

to

consult

a

variety

of

knowledge

sources

such

as

morphological

syntactic

semantic

and

contextual

knowledge

when

determining

a

new

lexical

entry

A

fairly

complete

system

aiming

at

combining

these

information

sources

was

outlined

in

-LSB-

Asker

et

al

-RSB-

however

that

system

still

relied

on

the

natural-language

system

's

grammar

as

its

main

knowledge

source

and

lacked

the

possibility

to

fully

utilize

morphological

information

This

paper

describes

experiments

using

different

back-propagation

network

architectures

in

order

to

design

an

automatic

method

to

acquire

new

lexical

entries

The

nets

were

trained

on

both

morphological

and

-LRB-

local

-RRB-

context

information

extracted

from

a

tagged

text

corpus

and

then

evaluated

on

previously

unseen

data

from

the

same

corpus

The

results

of

the

different

nets

are

compared

to

each

other

and

to

other

possible

methods

of

solving

the

problem

i.e.

pure

statistical

and

rule-based

approaches

The

rest

of

the

paper

is

laid

out

as

follows

Section

discusses

the

choice

of

information

sources

Section

contains

information

about

the

various

network

architectures

used

for

the

experiments

Section

describes

the

experiments

proper

Section

reports

on

the

results

while

Section

evaluates

them

in

the

light

of

other

approaches

finally

Section

sums

up

the

previous

discussions

and

points

to

possible

future

extensions

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Information

Sources

The

success

of

the

project

is

of

course

very

much

dependent

on

a

correct

choice

of

information

sources

Since

we

can

not

use

all

information

that

is

the

whole

text

possibly

extended

with

semantics

we

have

to

make

some

choices

The

unknown

word

The

most

obvious

way

to

determine

the

grammatical

category

of

an

unknown

word

is

perhaps

to

look

at

the

word

itself

Normally

one

would

look

at

the

meaning

of

the

word

for

example

nouns

are

usually

things

and

verbs

describe

events

but

here

we

do

not

have

access

to

semantics

We

have

to

manage

using

only

syntactical

information

There

are

several

alternatives

ffl

The

length

of

the

word

can

be

used

Certainly

conjunctions

consists

of

few

letters

and

nouns

can

be

very

long

This

is

however

not

a

good

assumption

for

making

decisions

regarding

grammatical

categories

There

is

nothing

to

say

that

the

length

has

any

relation

to

the

grammatical

category

And

as

a

matter

of

fact

there

are

several

nouns

consisting

of

fewer

letters

than

the

shortest

conjunctions

ffl

The

number

of

syllabels

in

the

word

The

reasons

against

making

this

one

of

the

major

sources

of

information

are

the

same

as

in

the

above

case

ffl

The

actual

letters

of

the

word

Here

we

also

have

to

decide

on

how

many

of

the

letters

to

use

since

using

all

of

them

would

yield

a

very

large

network

For

an

inflectional

language

like

Swedish

it

is

rather

obvious

that

the

information

about

the

grammatical

caterory

of

the

word

is

to

be

found

in

its

ending

-LRB-

suffix

-RRB-

A

few

examples

will

illustrate

Words

ending

in

-LCB-

ande

will

most

likely

be

a

verb

For

example

springande

vilande

simmande

flygande

-LCB-

arna

will

most

likely

be

a

noun

For

example

bilarna

stolarna

cyklarna

bollarna

-LCB-

are

will

most

likely

be

an

adjective

For

example

vackrare

fulare

r?odare

gulare

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Perhaps

we

do

not

need

to

know

exactly

what

the

letters

of

the

suffix

are

We

might

be

able

to

extract

enough

information

just

by

using

a

consonant/vowel

marker

and

we

would

thus

manage

with

a

network

of

moderate

size

Contextual

information

Another

alternative

to

base

the

decisions

on

are

the

environment

in

which

the

word

occur

The

grammatical

categories

in

a

sentence

obviously

make

a

pattern

Consider

the

following

examples

ffl

A

noun

is

very

likely

to

follow

an

adjective

For

example

gul

bil

A

verb

is

however

most

unlikely

to

follow

an

adjective

ffl

The

grammatical

category

following

an

infinitival

marker

most

likely

is

a

verb

The

window

through

which

we

look

at

the

sentence

can

be

of

different

sizes

Keeping

the

size

as

small

as

possible

is

of

course

desirable

as

we

thereby

is

keeping

the

size

of

the

network

that

is

going

to

use

this

information

as

small

as

possible

We

believe

that

it

is

using

this

information

that

makes

it

possible

to

determine

the

grammatical

categories

of

the

nonsense

words

in

the

poem

Jabberwocky

-LSB-

Carroll

-RSB-

Most

readers

would

for

example

induce

that

toves

is

a

noun

and

gyre

a

verb

Twas

brillig

and

the

slithy

toves

Did

gyre

and

gimble

in

the

wabe

All

mimsy

were

the

borogoves

And

the

mome

raths

outgrabe

The

conclusion

is

that

the

information

sources

that

would

be

of

most

use

in

determining

grammatical

categories

would

be

the

last

letters

of

the

word

and

the

grammatical

categories

of

the

surrounding

words

Architectures

A

large

number

of

network

architectures

were

trained

and

tested

Most

of

these

nets

used

the

gradient

descent

technique

of

the

backpropagation

algorithm

-LSB-

Rumelhart

et

al

-RSB-

in

a

three-layered

layout

This

section

will

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

describe

how

the

net-input

was

encoded

and

the

actual

architectures

of

the

different

networks

used

in

the

experiments

A

few

tests

were

also

carried

out

with

a

combination

of

a

supervised

network

-LRB-

in

this

case

backpropagation

-RRB-

and

unsupervised

self-organizing

network

-LSB-

Kohonen

-RSB-

Notations

Definitions

In

the

text

below

we

will

make

heavy

use

of

the

following

character

sets

respectively

defining

the

Swedish

and

ASCII

alphabets

Swedish

vowels

and

consonants

and

some

morphologically

motivated

subsets

of

these

fa

b

og

f

a

o

g

V

fa

e

i

j

o

u

y

a

a

og

V

fi

j

o

u

y

a

a

og

Consonants

V

P

losives

fb

c

d

g

k

p

q

tg

Fricatives

ff

h

s

v

w

x

zg

Laterals

flg

Trills

frg

Nasals

fm

ng

Strings

of

characters

will

be

distinguished

according

to

what

alphabet

the

included

characters

belong

to

thus

xn

j

xi

xn

j

xi

When

defining

the

mappings

of

the

network

inputs

we

will

also

need

to

discuss

a

particular

type

of

vectors

namely

binary

vectors

of

different

length

with

only

one

These

will

be

referred

to

as

Binx

where

x

is

the

number

of

digits

in

the

vector

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

f

-LSB-

-RSB-

-LSB-

-RSB-

g

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

fx

j

x

is

a

binary

vector

of

digits

and

only

one

Encoding

of

letters

To

represent

the

encoding

of

letters

we

will

introduce

five

functions

which

are

formally

defined

in

Appendix

A

but

informally

they

map

the

character

sets

of

Section

onto

the

binary

vectors

Binx

and

perform

the

following

tasks

simply

divides

into

V

and

Consonants

further

subdivides

the

consonants

by

phonetic

category

that

is

into

P

losives

Fricatives

Laterals

Trills

and

Nasals

is

like

but

the

vowels

A

and

E

are

singled

out

from

the

others

-LRB-

V

-RRB-

since

they

behave

rather

in

a

special

way

when

inflection

is

performed

while

and

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

encode

the

entire

Swedish

and

ASCII

alphabets

respectively

Encoding

of

grammatical

categories

For

the

encoding

of

grammatical

categories

we

will

introduce

five

other

functions

-LRB-

again

these

are

formally

defined

in

Appendix

B

-RRB-

The

grammatical

encoding

functions

map

from

the

lexemes

to

the

binary

vectors

thus

split

into

nine

categories

nouns

adjectives

verbs

pronouns

determiners

adverbs

prepositions

conjunctions

and

infinitival

markers

adds

two

more

categories

one

for

auxiliaries

and

one

for

sentence

delimiters

is

like

but

with

special

categories

for

auxiliaries

idiomatic

expressions

and

present

and

past

participles

It

also

splits

the

conjunctions

into

subordinating

and

coordinating

ones

performs

the

same

task

as

but

further

subdivides

the

adjectives

by

comparative

form

-LRB-

i.e.

positive

comparative

and

superlative

-RRB-

and

the

adverbs

by

type

-LRB-

normal

comparative

superlative

and

comparison

-RRB-

finally

does

for

what

does

for

but

with

extra

categories

for

names

numbers

characters

and

sentence

delimiters

Backpropagation

networks

All

backpropagation

networks

were

three

layer

architectures

consisting

of

an

input

layer

a

hidden

layer

and

an

output

layer

Information

was

given

in

localized

form

In

order

to

examine

the

feasibility

of

the

approach

the

sizes

of

the

networks

were

initially

kept

at

moderate

levels

to

increase

only

gradually

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

In

order

to

determine

the

correct

category

of

a

word

two

information

sources

were

used

the

internal

structure

of

the

lexeme

and

N-grams

An

N

gram

refers

to

the

grammatical

categories

of

neighbouring

words

For

example

we

will

use

to

refer

to

the

word

itself

a

-LRB-

here

-RRB-

denotes

the

word

itself

and

the

word

to

the

left

a

denotes

a

and

the

word

to

the

right

and

so

on

When

combining

the

two

information

sources

the

vectors

were

simply

appended

The

resulting

vector

was

then

fed

to

the

network

We

will

now

go

on

to

describe

each

network

in

some

detail

The

number

of

neurons

of

a

specific

network

will

be

indicated

in

the

heading

by

a

triple

I

H

O

where

I

is

the

number

of

neurons

in

the

input

layer

H

the

same

for

the

hidden

layer

and

O

for

the

output

layer

Net

The

first

net

used

only

so

its

single

source

of

information

was

that

of

the

context

The

grammatical

category

mapping

used

was

very

simple

distinguishing

only

between

nouns

adjectives

verbs

pronouns

determiners

adverbs

prepositions

conjunctions

and

the

infinitival

marker

Note

that

no

information

at

all

was

extracted

from

the

unknown

word

Net

The

second

net

combined

the

two

information

sources

available

by

also

inspecting

the

letters

of

the

unknown

word

In

order

not

to

make

the

network

unnecessarily

large

the

mapping

between

the

actual

letter

and

its

representation

was

kept

as

simple

as

possible

At

first

letters

mapped

onto

one

of

only

three

classes

vowels

consonants

or

the

latter

indicating

the

lack

of

any

input

character

in

a

specific

position

The

last

four

letters

of

the

unknown

word

were

mapped

by

were

used

and

mapped

by

Net

By

using

the

letter-classification

was

refined

by

dividing

letters

into

vowels

plosives

fricatives

laterals

trills

and

nasals

Otherwise

the

net

was

the

same

as

the

one

above

Net

The

only

difference

from

the

net

was

the

addition

of

a

simple

letter-mapping

for

the

fifth-to-last

letter

Adding

this

information

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

was

very

inexpensive

since

it

at

the

implementation

level

meant

that

only

two

more

binary

digits

were

needed

in

the

input

vector

were

used

and

mapped

by

Net

More

information

was

extracted

from

the

unknown

word

when

the

letter-mapping

was

further

refined

by

using

A

separation

of

the

letters

A

and

E

from

the

other

vowels

was

made

The

reason

for

this

was

that

A

and

E

occur

frequently

in

Swedish

word-endings

and

thus

this

separation

should

enhance

the

rate

of

correct

classifications

Again

were

used

and

mapped

by

Net

In

this

network

a

one-to-one

mapping

of

letters

was

used

for

the

final

letter

of

the

unknown

word

For

the

second

third

and

fourthto-last

letters

the

letter-mapping

was

used

The

fifth-to-last

letter

used

the

mapping

The

were

again

mapped

by

Net

The

last

four

letters

were

mapped

using

and

the

fifth-to-last

letter

used

Net

The

last

five

letters

were

mapped

using

and

the

sixth-to-last

letter

used

were

used

and

mapped

by

Net

The

only

difference

from

the

network

was

that

this

model

had

hidden

neurons

This

was

done

in

order

to

examine

if

the

result

would

improve

with

a

larger

hidden

layer

Net

This

net

used

the

same

letter-mapping

functions

as

the

one

but

instead

of

using

-LRB-

as

all

the

previously

described

nets

-RRB-

it

mapped

the

with

the

function

which

separates

the

auxiliary

verbs

from

the

main

ones

It

also

recognized

sentence

delimiters

which

enabled

the

tagger

to

categorize

the

first

and

last

words

of

a

sentence

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Net

This

net

used

the

letter-mapping

function

for

the

six

last

letters

Further

refinement

of

the

category-mapping

resulted

in

This

function

splits

the

category

for

conjunctions

into

two

parts

separates

past

and

present

participles

from

other

verb

forms

and

has

a

new

category

for

idioms

This

results

in

categories

Since

the

complexity

of

the

problem

increased

the

hidden

layer

was

extended

with

another

neurons

Net

The

category-mapping

function

was

changed

to

which

has

five

more

categories

comparative

and

superlative

adjectives

and

comparison

comparative

and

superlative

adverbs

It

still

mapped

the

last

six

letters

using

Net

So

far

the

output

of

the

network

had

been

one

of

three

different

categories

This

network

architecture

allowed

words

to

be

sorted

into

different

categories

according

to

the

function

which

was

used

as

the

grammatical

category

function

while

the

last

four

letters

were

mapped

by

Net

The

only

difference

from

the

network

was

that

this

network

used

Net

This

final

network

used

and

encoded

the

last

six

letters

with

the

mapping

Again

the

words

were

sorted

into

different

categories

according

to

the

function

task

of

determining

the

right

category

out

of

three

different

ones

will

in

the

text

following

be

referred

to

as

the

simple

task

while

the

one

with

thirteen

different

categories

will

be

referred

to

as

the

complex

task

This

terminology

is

mostly

for

convenience

indeed

making

a

choice

between

only

three

output

categories

might

actually

be

a

harder

task

for

a

neural

network

when

classifying

data

into

thirteen

different

groups

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Experiments

All

networks

in

this

paper

were

trained

and

tested

using

the

Teleman

corpus

-LSB-

Teleman

-RSB-

This

text

consists

of

almost

tagged

Swedish

words

gathered

from

a

wide

range

of

different

genres

The

training

could

be

very

time

consuming

but

fortunately

for

the

most

part

the

networks

converged

rapidly

Typically

only

a

few

epochs

were

needed

until

a

satisfactory

performance

was

reached

One

epoch

means

one

sweep

through

the

training

set

The

small

number

of

epochs

needed

is

very

likely

a

result

of

the

text

used

for

training

Since

it

contains

many

duplicates

most

input

patterns

were

seen

and

trained

several

times

during

one

epoch

The

training

continued

as

long

as

seemed

reasonable

or

as

long

as

the

performance

did

not

decrease

when

evaluated

on

previously

unseen

material

The

rest

of

this

section

describe

in

more

detail

the

different

network

setups

which

are

alos

summarized

in

Table

Table

Summary

of

the

network

setups

for

the

experiments

Net

Gram

Category

Letter-functions

Training

-LRB-

N

-RRB-

function

epochs

examples

The

first

network

architectures

that

is

nets

through

were

each

trained

for

epochs

The

text

consisted

of

examples

Three

experiments

were

carried

out

by

using

the

architecture

-LRB-

net

-RRB-

The

difference

between

them

was

the

number

of

examples

used

for

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

training

and

In

the

first

and

last

cases

the

training

continued

for

epochs

In

the

case

where

the

net

was

trained

using

examples

the

training

only

continued

for

epochs

since

the

performance

then

began

to

decline

The

network

was

trained

using

examples

The

training

was

terminated

after

epochs

since

the

net

did

not

perform

better

than

the

net

which

had

the

same

architecture

except

that

it

had

neurons

less

in

the

hidden

layer

The

network

-LRB-

net

-RRB-

was

trained

using

examples

The

training

continued

until

a

decrease

in

performance

was

noted

which

occurred

after

epochs

The

last

two

networks

with

three

categories

as

output

nets

and

-LRB-

and

-RRB-

were

each

trained

using

examples

The

smaller

network

was

trained

for

epochs

and

the

larger

for

epochs

Three

different

networks

were

trained

on

the

more

complex

task

with

thirteen

rather

than

three

possible

outputs

They

were

all

trained

using

examples

For

nets

and

-LRB-

and

-RRB-

the

performance

was

beginning

to

decline

after

epochs

and

thus

the

training

was

terminated

The

success

rate

of

the

net

-LRB-

net

-RRB-

however

kept

increasing

after

that

mark

so

the

training

was

allowed

to

go

on

for

a

period

of

epochs

Results

The

networks

were

tested

on

unseen

part

of

the

Teleman

corpus

The

corpus

consists

of

several

different

types

of

text

thus

the

results

should

be

as

general

as

possible

The

results

of

the

tests

are

shown

graphically

in

the

figures

following

Figure

shows

the

performance

of

the

nets

on

the

first

classification

task

part-of-speech

categorization

The

network

with

the

worst

result

was

not

surprisingly

the

one

which

only

used

It

reached

a

classification

rate

of

about

which

is

not

so

bad

considering

that

it

extracts

no

information

at

all

from

the

word

that

is

to

be

categorized

When

information

was

added

about

the

internal

-LRB-

morphological

-RRB-

structure

of

the

unknown

word

the

networks

performed

better

The

more

detailed

this

information

was

the

better

did

the

network

perform

The

amount

of

examples

used

for

training

was

also

a

parameter

that

varied

Generally

the

more

examples

that

were

available

to

the

network

the

better

it

performed

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

performance

network

Figure

Performance

of

nets

on

the

simple

task

The

networks

with

the

best

results

were

nets

and

-LRB-

and

-RRB-

They

both

reached

a

classification

rate

of

The

best

result

for

a

network

which

could

classify

more

than

nouns

adjectives

and

verbs

was

as

shown

in

Figure

This

was

achieved

by

the

network

-LRB-

net

called

k

in

the

figure

-RRB-

when

trained

using

examples

and

tested

-LRB-

like

all

the

other

nets

-RRB-

on

unseen

examples

To

evaluate

the

consistency

of

these

figures

this

net

was

also

tested

on

an

uncommonly

large

set

of

unseen

examples

As

could

be

expected

when

comparing

the

sizes

of

the

training

versus

the

test

sets

this

showed

a

slight

decrease

in

performance

with

a

top

result

of

-LRB-

the

graph

called

k

in

Figure

-RRB-

An

example

of

an

analyzed

sentence

can

be

seen

in

Appendix

C

which

shows

the

beginning

of

the

analyzed

sentence

I

s?odra

Asien

har

nya

h?ogavkastade

sorter

av

ris

och

vete

b?orjat

odlas

p?a

allt

st?orre

arealer

-LRB-

In

Southern

Asia

new

highly

productive

strains

of

rice

and

wheat

have

begun

to

be

cultivated

on

increasingly

larger

areas

-RRB-

as

given

to

net

-LRB-

-RRB-

The

network

had

as

can

be

seen

a

difficult

time

separating

names

from

ordinary

nouns

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

60 80 100 120

performance

epoch

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Performance

of

nets

on

the

complex

task

Discussion

and

related

work

In

this

section

we

will

try

to

compare

the

results

of

the

previous

section

with

those

that

have

been

obtained

using

statistical

and

rule-based

methods

First

however

we

note

that

Veronis

and

Ide

-LSB-

Veronis

Ide

-RSB-

-LRB-

also

appears

as

-LSB-

Ide

Veronis

-RSB-

-RRB-

used

an

approach

akin

to

a

neural

network

in

extracting

lexical

information

from

a

machine

readable

dictionary

Their

results

were

rather

discouraging

in

that

they

managed

to

identify

the

correct

sense

of

a

word

-LRB-

that

already

occurred

in

the

dictionary

-RRB-

in

only

of

the

cases

Nakamura

et

al

-LSB-

Nakamura

et

al

-RSB-

investigated

word

category

prediction

using

a

neural

network

architecture

called

NETgram

a

four

layer

architecture

based

on

backpropagation

The

grammatical

categories

of

the

preceding

words

was

used

to

predict

the

category

of

the

next

word

They

reported

a

word

recognition

rate

of

about

Recently

a

rule-based

approach

has

achieved

some

extraordinary

results

Voutilainen

et

al

-LSB-

Voutilainen

et

al

Voutilainen

Tapanainen

-RSB-

report

a

classification

rate

as

high

as

The

downfalls

of

their

method

-LRB-

and

all

rule-based

ones

-RRB-

are

that

it

is

very

time

consuming

to

develop

the

rules

and

the

system

produced

is

highly

language

dependent

The

main

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

objection

to

their

method

is

however

that

it

also

demands

a

very

large

lexicon

-LRB-

again

making

the

approach

highly

language

specific

-RRB-

The

lexicon

they

used

covered

about

of

all

lexemes

appearing

in

the

texts

making

the

comparison

of

performance

figures

somewhat

unfair

Samuelsson

-LSB-

Samuelsson

-RSB-

suggests

a

method

based

purely

on

statistical

evidence

Reaching

a

success

rate

of

it

is

not

doing

as

well

as

the

one

of

Voutilainen

et

al

but

it

seems

to

us

to

be

a

more

reasonable

approach

since

no

external

lexicon

is

needed

and

no

language

specifics

are

assumed

The

same

information

sources

were

used

as

in

the

project

described

in

this

paper

hence

the

results

should

be

compatible

to

the

connectionistic

method

The

result

was

reported

using

a

inspection

of

letters

and

information

on

the

number

of

syllables

in

the

word

on

a

so

called

music-minus-one

test

The

test

setting

closely

resembles

that

of

the

net

above

which

reached

a

classification

rate

of

For

the

same

task

the

Xerox

Parc

system

Tagger

-LSB-

Cutting

et

al

-RSB-

based

on

a

Hidden

Markov

Model

-LRB-

HMM

-RRB-

method

also

was

able

to

classify

of

the

words

correctly

-LSB-

Cutting

-RSB-

Even

though

this

comparison

thus

shows

the

neural

net

approach

ahead

by

a

margin

As

noted

above

however

the

performance

of

the

neural

net

dropped

down

a

bit

-LRB-

to

-RRB-

when

evaluated

on

a

larger

test

set

it

indicates

that

the

methods

are

virtually

equivalent

for

the

task

at

hand

The

small

differences

certainly

are

significant

if

the

general

assumption

from

the

lexical

acquisition

community

holds

namely

that

the

scale

above

is

logarithmic

rather

than

linear

This

assumption

is

partially

due

to

the

fact

that

there

seems

to

be

a

limit

on

how

much

information

can

be

extracted

from

a

text

Some

of

the

reasons

are

ffl

Incorrect

classifications

Some

of

the

words

in

the

corpus

have

actually

been

assigned

the

wrong

grammatical

category

An

example

can

be

seen

in

the

analyzed

sentence

in

Appendix

C

where

the

word

s?odra

have

a

pronoun

tag

in

the

corpus

as

can

be

seen

in

the

pattern

In

this

case

it

has

been

taken

care

of

and

the

correct

class

that

is

adjective

has

been

assigned

to

the

word

ffl

Ambiguity

Some

of

the

words

can

at

different

times

belong

to

different

categories

For

example

the

word

hand

which

can

be

a

noun

in

en

hand

or

an

adverb

in

att

ta

hand

om

This

has

been

solved

by

Voutilainen

et

al

by

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

incorporating

a

special

module

for

disambiguation

of

morphosyntactic

ambiguities

-LRB-

which

in

their

system

arises

from

ambiguities

in

the

lexicon

-RRB-

ffl

Difficulties

of

separating

names

from

ordinary

nouns

This

is

almost

an

impossible

task

Proper

names

certainly

are

some

kind

of

nouns

and

therefore

in

many

situations

behave

accordingly

This

can

also

be

seen

in

Appendix

C

where

the

network

believes

that

the

name

asien

is

a

noun

Conclusions

and

future

work

We

have

described

a

series

of

experiments

where

different

three-layered

backpropagation

network

architectures

were

used

for

the

task

of

recognizing

unknown

words

for

a

natural

language

system

Two

main

tasks

were

performed

in

the

first

the

nets

were

to

classify

words

by

overall

part-of-speech

-LRB-

noun

adjective

or

verb

-RRB-

only

while

the

second

task

involved

a

larger

set

of

possible

output

categories

The

best

results

for

the

simple

task

were

obtained

by

networks

consisting

of

-LCB-

input

neurons

and

hidden-layer

neurons

reaching

a

classification

rate

of

The

best

result

for

the

more

complex

task

was

which

was

achieved

by

a

net

with

input

neurons

and

hidden-layer

neurons

The

results

are

overall

rather

promising

as

described

in

Section

they

are

completely

compatible

with

those

achieved

by

purely

statistical

methods

however

they

are

still

inferior

to

those

reported

by

a

rule-based

approach

albeit

on

a

somewhat

different

task

A

possible

way

to

improve

on

the

results

could

be

to

combine

several

networks

for

example

we

have

done

some

initial

experiments

using

a

selforganizing

network

of

the

type

described

in

-LSB-

Kohonen

-RSB-

This

network

makes

it

possible

for

high-dimensional

data

to

be

represented

in

a

lower

dimensional

space

The

neurons

are

arranged

in

a

two

dimensional

map

and

not

in

layers

as

in

backpropagation

The

algorithm

will

then

arrange

the

map

so

that

the

probability

density

function

of

the

pattern

space

will

be

preserved

The

idea

would

be

to

transform

the

letters

of

the

unknown

word

to

a

two

dimensional

map

and

then

feed

the

coordinates

of

this

map

to

a

backpropagation

network

together

with

the

grammatical

categories

of

the

surrounding

words

however

this

approach

has

so

far

not

been

very

successful

Early

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

results

indicate

that

this

combination

does

not

perform

better

than

the

backpropagation

network

using

only

The

map

failed

to

capture

the

structure

of

the

words

Acknowledgements

We

would

like

to

thank

Lars

Asker

-LRB-

Stockholm

University

-RRB-

Ivan

Bretan

-LRB-

IBM

and

SICS

-RRB-

Douglass

Cutting

-LRB-

Xerox

Parc

-RRB-

Pat

Langley

-LRB-

Siemens

-RRB-

and

Christer

Samuelsson

-LRB-

SICS

-RRB-

for

helpful

discussions

and

suggestions

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

References

-LSB-

Asker

et

al

-RSB-

Lars

Asker

Bj?orn

Gamb?ack

and

Christer

Samuelsson

An

Approach

to

Automatic

Lexical

Acquisition

In

Proceedings

the

International

Conference

on

Computational

Linguistics

volume

pp

-LCB-

Nantes

France

July

-LSB-

Boguraev

et

al

-RSB-

Bran

Boguraev

Ted

Briscoe

John

Carroll

David

M.

Carter

and

Claire

Grover

The

Derivation

of

a

Grammatically

Indexed

Lexicon

from

the

Longman

Dictionary

of

Contemporary

English

In

Proceedings

of

the

Annual

Meeting

of

the

Association

for

Computational

Linguistics

pp.

-LCB-

Stanford

California

-LSB-

Calzolari

Bindi

-RSB-

Nicoletta

Calzolari

and

Remo

Bindi

Acquisition

of

Lexical

Information

from

a

Large

Textual

Italian

Corpus

In

Proceedings

the

International

Conference

on

Computational

Linguistics

volume

pp.

-LCB-

Helsinki

Finland

August

-LSB-

Carroll

-RSB-

Lewis

Carroll

Through

the

Looking-Glass

and

What

Alice

Found

There

Macmillan

-LSB-

Carter

-RSB-

David

Carter

Lexical

Acquisition

in

the

Core

Language

Engine

In

Proceedings

of

the

Conference

of

the

European

Chapter

of

the

Association

for

Computational

Linguistics

pp.

-LCB-

University

of

Manchester

Institute

of

Science

and

Technology

Manchester

England

April

Also

available

as

SRI

International

Research

Report

Cambridge

England

-LSB-

Cutting

-RSB-

Douglass

Cutting

Porting

a

Stochastic

Part-of-Speech

Tagger

to

Swedish

In

Proceedings

of

the

Scandinavian

Conference

on

Computational

Linguistics

Stockholm

University

Stockholm

Sweden

June

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

-LSB-

Cutting

et

al

-RSB-

Douglass

Cutting

Julian

Kupiec

Jan

Pedersen

and

Penelope

Shibun

A

Practical

Part-of-Speech

Tagger

In

Proceedings

of

the

Conference

on

Applied

Natural

Language

Processing

pp.

-LCB-

Trento

Italy

April

-LSB-

Gamb?ack

-RSB-

Bj?orn

Gamb?ack

Lexical

Acquisition

the

Swedish

VEX

System

In

Papers

from

the

Nordic

Conference

on

Text

Comprehension

in

Man

and

Machine

pp.

-LCB-

Link?oping

University

Link?oping

Sweden

April

Also

available

as

SICS

Research

Report

Stockholm

Sweden

-LSB-

Granger

-RSB-

R.H.

Granger

FOUL-UP

A

program

that

figures

out

meanings

of

words

from

context

In

Proceedings

of

the

International

Joint

Conference

on

Artificial

Intelligence

pp.

-LCB-

Cambridge

Massachusetts

-LSB-

Grosz

et

al

-RSB-

Barbara

J.

Grosz

Douglas

E.

Appelt

Paul

Martin

and

Fernando

C.N.

Pereira

TEAM

An

Experiment

in

the

Design

of

Transportable

Natural-Language

Interfaces

Artificial

Intelligence

-LCB-

-LSB-

Ide

Veronis

-RSB-

Nancy

M.

Ide

and

Jean

Veronis

Very

Large

Neural

Networks

Word

Sense

Disambiguation

In

Proceedings

of

the

European

Conference

on

Artificial

Intelligence

pp.

-LCB-

Stockholm

Sweden

August

-LSB-

Jacobs

Zernik

-RSB-

P.

Jacobs

and

U.

Zernik

Acquiring

Lexical

Knowledge

from

Text

A

Case

Study

In

Proceedings

of

the

National

Conference

on

Artificial

Intelligence

pp.

-LCB-

Saint

Paul

Minnesota

-LSB-

Kohonen

-RSB-

Teuvo

Kohonen

Self-Organization

and

Associative

Memory

Springer

Heidelberg

Germany

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

-LSB-

Nakamura

et

al

-RSB-

M.

Nakamura

K.

Maruyama

T.

Kawabata

and

K.

Shikano

Neural

Network

Approach

to

Word

Category

Prediction

for

English

Texts

In

Proceedings

the

International

Conference

on

Computational

Linguistics

volume

pp.

-LCB-

Helsinki

Finland

August

-LSB-

Rumelhart

et

al

-RSB-

D.E.

Rumelhart

G.E.

Hinton

and

R.J.

Williams

Learning

Internal

Representations

by

Error

Propagation

In

D.E.

Rumelhart

and

J.L.

McClelland

editors

Parallel

Distributed

Processing

volume

MIT

Press

Cambridge

Massachusetts

-LSB-

Samuelsson

-RSB-

Christer

Samuelsson

Morphological

Tagging

Based

Entirely

on

Bayesian

Inference

In

Proceedings

of

the

Scandinavian

Conference

on

Computational

Linguistics

Stockholm

University

Stockholm

Sweden

June

-LSB-

Schank

-RSB-

Roger

C.

Schank

SAM

A

Story

Understander

Research

Report

Yale

AI

Project

Department

of

Computer

Science

Yale

University

New

Haven

Connecticut

-LSB-

Teleman

-RSB-

Ulf

Teleman

Manual

f?or

grammatisk

beskrivning

av

talad

och

skriven

svenska

Studentlitteratur

Lund

Sweden

-LRB-

in

Swedish

-RRB-

-LSB-

Veronis

Ide

-RSB-

Jean

Veronis

and

Nancy

M.

Ide

Word

Sense

Disambiguation

with

Very

Large

Neural

Networks

Extracted

from

Machine

Readable

Dictionaries

In

Proceedings

the

International

Conference

on

Computational

Linguistics

volume

pp.

-LCB-

Helsinki

Finland

August

-LSB-

Voutilainen

Tapanainen

-RSB-

Atro

Voutilainen

and

Pasi

Tapanainen

Ambiguity

Resolution

in

a

Reductionistic

Parser

In

Proceedings

of

the

Conference

of

the

European

Chapter

of

the

Association

for

Computational

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Linguistics

pp.

-LCB-

Utrecht

University

Utrecht

Holland

April

-LSB-

Voutilainen

et

al

-RSB-

Atro

Voutilainen

Jyrkki

Heikkila

and

Antti

Anttila

Constraint

Grammar

of

English

Publication

Department

of

General

Linguistics

University

of

Helsinki

Helsinki

Finland

-LSB-

Wilensky

-RSB-

Robert

Wilensky

Extending

the

Lexicon

by

Exploiting

Subregularities

In

Proceedings

of

the

Speech

and

Natural

Language

Workshop

pp.

-LCB-

Hidden

Valley

Pennsylvania

June

DARPA

Morgan

Kaufmann

-LSB-

Zernik

-RSB-

Uri

Zernik

Language

Acquisition

Learning

a

Hierarchy

of

Phrases

In

Proceedings

of

the

International

Joint

Conference

on

Artificial

Intelligence

pp.

-LCB-

Milan

Italy

-LSB-

Zernik

Dyer

-RSB-

Uri

Zernik

and

Michael

G.

Dyer

Towards

a

Self-Extending

Lexicon

In

Proceedings

of

the

Annual

Meeting

of

the

Association

for

Computational

Linguistics

pp.

-LCB-

University

of

Chicago

Chicago

Illinois

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

A

Encoding

of

letters

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

V

-LSB-

-RSB-

-LRB-

-RRB-

x

Consonants

-LRB-

-RRB-

x

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

V

-LSB-

-RSB-

-LRB-

-RRB-

x

P

losives

-LSB-

-RSB-

-LRB-

-RRB-

x

F

ricatives

-LSB-

-RSB-

-LRB-

-RRB-

x

Laterals

-LSB-

-RSB-

-LRB-

-RRB-

x

T

rills

-LSB-

-RSB-

-LRB-

-RRB-

x

Nasals

-LRB-

-RRB-

x

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

A

-LSB-

-RSB-

-LRB-

-RRB-

x

E

-LSB-

-RSB-

-LRB-

-RRB-

x

V

-LSB-

-RSB-

-LRB-

-RRB-

x

P

losives

-LSB-

-RSB-

-LRB-

-RRB-

x

F

ricatives

-LSB-

-RSB-

-LRB-

-RRB-

x

Laterals

-LSB-

-RSB-

-LRB-

-RRB-

x

T

rills

-LSB-

-RSB-

-LRB-

-RRB-

x

Nasals

-LRB-

-RRB-

x

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

A

-LSB-

-RSB-

-LRB-

-RRB-

x

B

-LSB-

-RSB-

-LRB-

-RRB-

x

O

-LRB-

-RRB-

x

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

-LSB-

-RSB-

-LRB-

-RRB-

x

-LRB-

-RRB-

x

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

B

Encoding

of

grammatical

categories

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

Noun

-LSB-

-RSB-

-LRB-

-RRB-

x

Adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

V

erb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ronoun

-LSB-

-RSB-

-LRB-

-RRB-

x

Determiner

-LSB-

-RSB-

-LRB-

-RRB-

x

Adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

reposition

-LSB-

-RSB-

-LRB-

-RRB-

x

Conjunction

-LSB-

-RSB-

-LRB-

-RRB-

x

Inf

mark

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

Noun

-LSB-

-RSB-

-LRB-

-RRB-

x

Adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

V

erb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ronoun

-LSB-

-RSB-

-LRB-

-RRB-

x

Determiner

-LSB-

-RSB-

-LRB-

-RRB-

x

Adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

reposition

-LSB-

-RSB-

-LRB-

-RRB-

x

Conjunction

-LSB-

-RSB-

-LRB-

-RRB-

x

Inf

mark

-LSB-

-RSB-

-LRB-

-RRB-

x

Auxiliary

-LSB-

-RSB-

-LRB-

-RRB-

x

Sent

del

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

Noun

-LSB-

-RSB-

-LRB-

-RRB-

x

Adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

Auxiliary

-LSB-

-RSB-

-LRB-

-RRB-

x

V

erb

-LSB-

-RSB-

-LRB-

-RRB-

x

Adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ronoun

-LSB-

-RSB-

-LRB-

-RRB-

x

P

reposition

-LSB-

-RSB-

-LRB-

-RRB-

x

Conjunction

-LSB-

-RSB-

-LRB-

-RRB-

x

Conjunction

-LSB-

-RSB-

-LRB-

-RRB-

x

Determiner

-LSB-

-RSB-

-LRB-

-RRB-

x

Inf

mark

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ast

part

-LSB-

-RSB-

-LRB-

-RRB-

x

P

resent

part

-LSB-

-RSB-

-LRB-

-RRB-

x

Idiom

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

Noun

-LSB-

-RSB-

-LRB-

-RRB-

x

Cmp

adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

Sup

adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

P

os

adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

Auxiliary

-LSB-

-RSB-

-LRB-

-RRB-

x

V

erb

-LSB-

-RSB-

-LRB-

-RRB-

x

Comparison

-LSB-

-RSB-

-LRB-

-RRB-

x

Cmp

adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

Sup

adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

Adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ronoun

-LSB-

-RSB-

-LRB-

-RRB-

x

P

reposition

-LSB-

-RSB-

-LRB-

-RRB-

x

Coord

conj

-LSB-

-RSB-

-LRB-

-RRB-

x

Subord

conj

-LSB-

-RSB-

-LRB-

-RRB-

x

Determiner

-LSB-

-RSB-

-LRB-

-RRB-

x

Inf

mark

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ast

part

-LSB-

-RSB-

-LRB-

-RRB-

x

P

resent

part

-LSB-

-RSB-

-LRB-

-RRB-

x

Idiom

-LRB-

x

-RRB-

-LSB-

-RSB-

-LRB-

-RRB-

x

Noun

-LSB-

-RSB-

-LRB-

-RRB-

x

Adjective

-LSB-

-RSB-

-LRB-

-RRB-

x

V

erb

-LSB-

-RSB-

-LRB-

-RRB-

x

P

reposition

-LSB-

-RSB-

-LRB-

-RRB-

x

Adverb

-LSB-

-RSB-

-LRB-

-RRB-

x

Determiner

-LSB-

-RSB-

-LRB-

-RRB-

x

P

ronoun

-LSB-

-RSB-

-LRB-

-RRB-

x

Conjunction

-LSB-

-RSB-

-LRB-

-RRB-

x

Number

-LSB-

-RSB-

-LRB-

-RRB-

x

Name

-LSB-

-RSB-

-LRB-

-RRB-

x

Sent

del

-LSB-

-RSB-

-LRB-

-RRB-

x

Inf

mark

-LSB-

-RSB-

-LRB-

-RRB-

x

Character

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

C

A

test

sentence

Categories

noun

adjective

verb

preposition

adverb

determiner

pron.

char.

conj.

number

name

sentence

delimiter

infinitive

mark

Correct

classification

Answer

Should

have

been

Pattern

IP

I

PR

S

ODRA

POSU

Correct

classification

Answer

Should

have

been

Pattern

I

PR

S

ODRA

POSU

ASIEN

PN

Wrong

classification

Answer

Should

have

been

Pattern

S

ODRA

POSU

ASIEN

PN

HAR

HVPS

Correct

classification

Answer

Should

have

been

Pattern

ASIEN

PN

HAR

HVPS

NYA

AJ

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Correct

classification

Answer

Should

have

been

Pattern

HAR

HVPS

NYA

AJ

H

OGAVKASTANDE

AJ

Correct

classification

Answer

Should

have

been

Pattern

NYA

AJ

H

OGAVKASTANDE

AJ

SORTER

NN

Correct

classification

Answer

Should

have

been

Pattern

H

OGAVKASTANDE

AJ

SORTER

NN

AV

PR

Correct

classification

Answer

Should

have

been

Pattern

SORTER

NN

AV

PR

RIS

NN

Wrong

classification

Answer

Should

have

been

Pattern

AV

PR

RIS

NN

OCH

OC

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Correct

classification

Answer

Should

have

been

Pattern

RIS

NN

OCH

OC

VETE

NN

Correct

classification

Answer

Should

have

been

Pattern

OCH

OC

VETE

NN

B

ORJAT

VVSN

Correct

classification

Answer

Should

have

been

Pattern

VETE

NN

B

ORJAT

VVSN

ODLAS

VVIVPA

Correct

classification

Answer

Should

have

been

Pattern

B

ORJAT

VVSN

ODLAS

VVIVPA

P?A

PR

Correct

classification

Answer

Should

have

been

Pattern

ODLAS

VVIVPA

P?A

PR

ALLT

ABTA

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

Correct

classification

Answer

Should

have

been

Pattern

P?A

PR

ALLT

ABTA

ST

ORRE

AJKP

Correct

classification

Answer

Should

have

been

Pattern

ALLT

ABTA

ST

ORRE

AJKP

AREALER

NN

Correct

classification

Answer

Should

have

been

Pattern

ST

ORRE

AJKP

AREALER

NN

IP

Correct

classification

Answer

Should

have

been

Pattern

AREALER

NN

IP

DE

PODP

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

D

A

sample

from

the

corpus

INDIVIDUELL

AJ

BESKATTNING

VN

AV

PR

ARBETSINKOMSTER

NNSS

GENOM

PR

SKATTEREFORMEN

NNDDSS

INF?ORS

VVPSSMPA

INDIVIDUELL

AJ

BESKATTNING

VN

-LRB-

IR

S?ARBESKATTNING

VNSS

-RRB-

IR

AV

PR

ARBETSINKOMSTER

NNSS

IP

DET

POOP

INNEB?AR

VVPSSM

BL

ABZA

A.

ID

ATT

UKAT

ENDAST

ABZA

EN

EN

SKATTESKALA

NNSS

KOMMER

KVPS

ATT

IM

FINNAS

VVIV

F?OR

PR

BER?AKNING

VN

AV

PR

STATLIG

AJ

INKOMSTSKATT

NNSS

IP

DEN

PODP

BLIR

BVPS

GEMENSAM

AJ

F?OR

PR

ALLA

POTP

INKOMSTTAGARE

NNHS

OAVSETT

PR

CIVILST?AND

NNSS

IP

DEN

PODP

G?ALLER

VVPS

AVEN

ABOC

F?OR

PR

OSKIFTA

AJ

D?ODSBON

NNSS

OCH

OC

FAMILJESTIFTELSER

VNSS

IP

DE

PODP

INKOMSTER

NN

SOM

PORP

P?A

PR

DETTA

PODP

S?ATT

NN

BESKATTAS

VVPSPA

INDIVIDUELLT

AJ

KALLAS

VVPSPA

A-INKOMSTER

NNSS

IP

DESSA

PODP

INKOMSTER

NN

AR

AVPS

INKOMST

NN

AV

PR

TJ?ANST

NN

L?ON

NN

PENSION

NN

LIVR?ANTA

NNSS

UNDANTAGSF?ORM?ANER

NNSS

OCH

OC

OVRIG

POKP

TJ?ANSTEINKOMST

NNSS

IP

-LRB-

IR

UNDANTAG

NN

IQ

PERIODISKT

AJ

UNDERST?OD

NNSS

ELLER

EL

D?ARMED

ABDA

J?AMF?ORLIG

AJ

PERIODISK

AJ

INKOMST

NN

IP

-RRB-

IR

INKOMST

NN

AV

PR

JORDBRUKSFASTIGHET

NNSS

OM

UKKD

DEN

PODP

SKATTSKYLDIGE

AJHS

ARBETAT

VVSN

I

PR

JORDBRUKET

NNDDSS

I

PR

EJ

ABNA

BLOTT

ABZA

RINGA

POZP

OMFATTNING

VN

IP

INKOMST

NN

AV

PR

R?ORELSE

VN

OM

UKKD

DEN

PODP

SKATTSKYLDIGE

AJHS

ARBETAT

VVSN

I

PR

R?ORELSEN

VNDD

I

PR

EJ

ABNA

BLOTT

ABZA

RINGA

POZP

OMFATTNING

VN

IP

MAKARS

NNHHGG

OVRIGA

POKP

INKOMSTER

NN

AR

AVPS

B-INKOMSTER

NNSS

OCH

OC

SKALL

SVPS

SOM

UKKM

TIDIGARE

AJKP

SAMBESKATTAS

VVIVSMPA

IP

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition

E

Examples

of

the

encoding

of

data

letters

cat

cat

cat

pattern

BESKATTNING

VN

AV

PR

ARBETSINKOMSTER

NNSS

letters

cat

cat

cat

pattern

AV

PR

ARBETSINKOMSTER

NNSS

GENOM

PR

letters

cat

cat

cat

pattern

ARBETSINKOMSTER

NNSS

GENOM

PR

SKATTEREFORMEN

NNDDSS

Eineborg

Gamb?ack

Neural

Networks

for

Wordform

Recognition