Requirements for handling paraphrases
Transformation rules / patterns< IWP 2005, Oct. 14th, 2005 >!

Handcrafting

Xverb # S0(X) + Oper1(S0(X))!Iordansjaka et al., 1991"!Dras, 1999"!Sato et al., 1999"!Kondo et al., 1999"!Kondo et al., 2001"!Iida et al., 2001" etc.

A Class-oriented Approach toBuilding a Paraphrase Corpus!

Automatic acquisition!Barzilay et al., 2001"!Lin et al., 2001"!Shinyama et al., 2002"!Shimohata et al., 2002"!Pang et al., 2003"etc.
burst into tears # cry X finds a solution to Y # X solves Y

Atsushi FUJITA(1), Kentaro INUI(2)(1) Kyoto University(2) Nara Institute of Science and Technology

Paraphrase corpus (collection of paraphrase examples)!

Few freely available resources !Dolan et al., 2004"
The leading indicators measure the economy...
The leading index measures the economy....

Purposes of paraphrase corpus

OutlineBeneficial to activate the research field
Gold-standard
for evaluation

1.

Our aim

Paraphrase
corpusExample-based
paraphrasing

Background
Issues and our class-oriented approach
3. Semi-automatic example collection
4. Preliminary trials
1.
2.

Deep analysisof phenomena

Knowledge
discovery2.

5.

Design better
evaluation methods

Specification
Discussion

Conclusion

Paraphrase ruleinduction
3

Building paraphrase corpus!

Emma burst into tears and he tried to comfort her. Lexical paraphrase$ automatic
Emma cried and he tried to console her.
acquisition

to consider: variety, source, organization
to maximize: coverage, reliability, cost-efficiency

It was his best suit that John wore last night. Syntactic paraphrase$%describableJohn wore his best suit last night.

Previous work
Manual production!Shirai et al., 2001"!Kinjo et al., 2003"!Shimohata et al., 2004"

4Variety of paraphrases

Issues!

2Automatic acquisition!Barzilay et al., 2003"!Shinyama et al., 2003"!Dolan et al., 2004"

Reliability

Steven made an attempt to stop playing Hearts.
Steven attempted to stop playing Hearts.
The breeze sways the trees.
The trees sway in the breeze.

Cost-efficiency

Coverage is not ensured! No focus on sorts/variety of paraphrases!

Lexically compositional paraphrase- syntactically regular- semantically compositional

The room has already been warmed up.
The room is already warm.
56

A class-oriented approach

Aim of this studySeparately collect examples for each classConfirm the feasibility of the method through practice!

sub-corpus
for class A

sub-corpus
for class B

sub-corpus
for class C…

Paraphrasecorpus

Given
##!

Semi-automatic example collection

A paraphrase class
A text collection

Collect paraphrase examples belonging to the class
At a minimal human labor cost
As exhaustively as possible from the text collection# As reliable as humanly possible#Automatic generation + human judgment! Step 1: Define a paraphrase classbased on morpho-syntactic transformation patterns! Step 2: Collect all candidates using a paraphrase engine! Step 3: Judge candidate paraphrases in hand!

#7Outline

8

Semi-automatic example collection

Overview
2. Issues and our class-oriented approach
3. Semi-automatic example collection
4. Preliminary trials

Paraphrasing rules

1.

X{NOM,ACC,DAT}

2.

"NV(N)X

Text collection

Lexicalresources

Specification
Discussion

1.

5.

V

1
Manualdescription

Conclusion

Paraphrase generation
systemJudgment guidelines

2

Paraphrase
candidates

3 Evaluation
andcorrection

Automaticgeneration

9

Step 1: Pattern description

Step 3: Manual judgment (mutual judgment)“Correct”-preferred judgment process Judgment guidelines

Morpho-syntactic paraphrasing patterns
Pairs of dependency trees! Implemented on a paraphrase generation system!

!Takahashi et al., 2001"
VX

N-ACC"

V(N)X!

RefineTo reduce labor cost
1st annotator 2nd annotator

X: variable for any word
N: variable for a noun
V: variable for a verb
V(N): verbalized form of N

Correct&'-( )-+,-120
film-NOM him-ACC to be impressed-CAUSATIVE

Incorrect
DeferredDeferredCorrect
Incorrect
Unchecked

11Correct

Correct

Incorrect(The film made an impression on him.)

Discussion

Correct
Candidateparaphrase&'-( )-*+,-./0
film-NOM him-DAT impression-ACC to give-ACTIVE(The film impressed him.)

10

Incorrect
Incorrect
Incorrect(Unchecked) 12

Step 3: Manual judgment (I/F)(a) source sentence

OutlineOverview
2. Issues and our class-oriented approach
3. Semi-automatic example collection
4. Preliminary trials
1.

(c) second opinion(correct / incorrect)

Given

Obligatory(b) automaticallygeneratedparaphrase

Specification
Discussion

1.

Obligatory

2.

(f) free comments

5.

(c) annotator’s judge(correct / incorrect)(e) confirmed (revised)paraphrase

Conclusion

Optional(d) error tags
13

Target classes

14

Resources

Paraphrases of light-verb constructions (LVC)

LVC&'-( )-*+,-./0
film-NOM him-DAT impression-ACC to give-ACTIVE!
!

(The film made an impression on him.)&'-( )-+,-120
film-NOM him-ACC to be impressed-CAUSATIVE

4 paraphrasing patterns (e.g. (7) in paper)
20,155 pairs of <deverbal noun, verb=##(The film impressed him.)<+,(impression), +,:0(to be impressed)=<>?(invitation), >@(to invite)=TransAlt

Transitivity alternation (TransAlt)
345-( 67-- 89:
breeze-NOM tree-ACC to sway-Transitive!
!

(The breeze sways the trees.)

67-( 345-* 8;0
tree-NOM breeze-DAT to sway-Intransitive

8 paraphrasing patterns (e.g. (10) in paper)
212 pairs of <intransitive verb, transitive verb=##(The trees sway in the breeze.)<8;0(to sway-Intransitive), 89:(to sway-Transitive)=<A;0(to break-Intransitive), A:(to break-Transitive)=

15

Results of trials
Step 1
Step 2

Step 3

Paraphrase class# of paraphrasing patterns
Size of dictionary# of source sentences# of generated candidates# of judged candidates# of incorrect candidates# of correct candidates# of paraphrase examples
Working hours

16

Aim of this study (reminder)
LVC TransAlt
4
8
20,155212
10,000 25,000
2,566
985
1,067
964
520
503
547
461
591
484
118
169.5

Confirm the feasibility of the method through practice!

Given
##!

A paraphrase class
A text collection

Collect paraphrase examples belonging to the class
At a minimal human labor cost
As exhaustively as possible from the text collection# As reliable as humanly possible##Working hours: 2 annotators’ working time for (1) Judgment,(2) Discussion, and (3) Re-judgment after refining guidelines
1718

Cost-efficiency
LVC

ExhaustivenessThe initial resource is not necessarily optimalTransAlt!
!

Paraphrasing patterns
Derivation pairs

How are they optimal?
!

Estimated coverage: 77% (158/(158+48))#Not so wasteful human labor cost

#7.1 candidates / man-hour! 3.7 paraphrase examples / man-hour! TransAlt is 1.75 times more difficult than LVC due to test

158 paraphrases for 750 excerpted sentences
Manual examination obtained another 48 paraphrases!

!

!

!
!

47 misses can be salvaged by resource enhancement
Errors of shallow parsers hurt only once

Use of patterns is realistic approach
Manual examination ensures coverage

19ReliabilityConclusion

Strategy

Feasibility of a semi-automatic example collection

Classification bases on guideline & linguistic intuition! Inter-annotator discussion refined judgment guidelines!

!
!

Agreement ratio increased (in case of LVC)

Reasonable human labor cost, but need reduction
Moderately exhaustive at initial stage! Typically reliable, but some marginal cases!
!

Future planParaphrase sub-corpora consist of

Involve an expert to make sure of judgment guidelines! Involve the 3rd annotator for judgment
!

!
!
21

Future work
Discussion on required expertise!

It is not easy to explain “why this is correct / incorrect”
Involve an expert to make sure of judgment guidelines

Build sub-corpora for other paraphrase classes
Extrinsic evaluation through case studies!
!

Resultant provides both correct and incorrect examples
Immediately available for analysis and system evaluation

Publicly open the resource!

Class-oriented example collectionEmploying a paraphrase generation system

Promising results

74% (3rd day) " 77% (6th day)
" 88% (9th day) " 93% (11th day)
! It’s still not easy to explain “why this is correct / incorrect”
!

!

20

Paraphrase corpus, Lexical resources, Judgment
guidelines
23

LVC:
TransAlt:

1067 candidates / 591 examples
964 candidates / 484 examples
22