Evaluating Contextual Dependency of Paraphrases
using a Latent Variable Model
Kiyonroi OHTAKE
Spoken Language Communication Research Laboratories
Advanced Telecommunications Research Institute International
Kyoto 619-0288 Japankiyonori.ohtake + @atr.jp

Abstract

ever, it is very hard for machines to handle different expressions that carry the same meaning.
The importance of paraphrasing has been
widely acknowledged, and many paraphrasingstudies have been carried out. Using only surface similarity is insufficient for evaluating paraphrases because there are not only surface differences but many other kinds of differences between paraphrased sentences. Thus, it is not easy
to evaluate whether two sentences carry almost
the same meaning.
Some studies have constructed and evaluatedhand-made rules (Takahashi et al., 2001; Ohtake
and Yamamoto, 2001). Others have tried to
extract paraphrases from corpora (Barzilay and
McKeown, 2001; Lin and Pantel, 2001), which
are very useful because they enable us to construct paraphrasing rules. In addition, we can construct an example-based or a Statistical MachineTranslation (SMT)-like paraphrasing system thatutilizes paraphrasing examples. Thus, collecting paraphrased examples must be continued toachieve high-performance paraphrasing systems.
Several methods of acquiring paraphrases havebeen proposed (Barzilay and McKeown, 2001;
Shimohata and Sumita, 2002; Yamamoto, 2002).
Some use parallel corpora as resources to obtain
paraphrases, which seems a promising way to extract high-quality paraphrases.
However, unlike translation, there is no obvious paraphrasing direction. Given paraphrasing
pair E1 :E2 , we have to know the paraphrasing
direction to paraphrase from E1 to E2 and viceversa. When extracting paraphrasing pairs from
corpora, whether the paraphrasing pairs are con-This paper presents an evaluation
method employing a latent variable
model for paraphrases with their contexts. We assume that the context of asentence is indicated by a latent variable of the model as a topic and thatthe likelihood of each variable can beinferred. A paraphrase is evaluatedfor whether its sentences are used in
the same context. Experimental results showed that the proposed method
achieves almost 60% accuracy and that
there is not a large performance difference between the two models. The results also revealed an upper bound of
accuracy of 77% with the method whenusing only topic information.

1 Introduction
This paper proposes a method to evaluate whether
a paraphrasing pair is contextually independent.
Evaluating a paraphrasing pair is important when
we extract paraphrases from a corpus or apply aparaphrase to a sentence, since we must guaranteethat the paraphrase carries almost the same meaning. However, the meaning carried by a sentence
is affected by its context. Thus, we focus on thecontextual dependency of paraphrases.
A thing can be expressed by various expressions, and a single idea can be paraphrased inmany ways to enrich its expression or to increase
understanding. Paraphrasing plays a very important role in natural language expressions. How65

textually dependent paraphrases is a serious problem, and thus there is a specific paraphrase direction for each pair. In addition, it is also importantto evaluate a paraphrasing pair not only when extracting but also when applying a paraphrase.
Consider this example, automatically extracted
from a corpus: Can I pay by traveler’s check?
/ Do you take traveler’s checks? This example
seems contextually independent. On the other
hand, here is another example: I want to buy apair of sandals. / I’m looking for sandals. This
example seems to be contextually dependent, because we don’t know whether the speaker is only
looking for a single pair of sandals. In some contexts, the latter sentence means that the speaker isseeking or searching for sandals. In other words,
the former sentence carries specific meaning, but
the latter carries generic meaning. Thus, the paraphrasing sentences are contextually dependent,
and although the paraphrasing direction from specific to generic might be acceptable, the opposite
direction may not be.
We can solve part of this problem by inferring
the contexts of the paraphrasing sentences. A text
model with latent variables can be used to infer
the topic of a text, since latent variables correspond to the topics indicated by texts. We assume that a topic indicated by a latent variableof a text model can be used as an approximationof context. Needless to say, however, such an approximation is very rough, and a more complex
model or more powerful approach must be developed to achieve performances that match humanjudgement in evaluating paraphrases.
The final goal of this study is the evaluationof paraphrasing pairs based on the following two
factors: contextual dependency and paraphrasing direction. In this paper, however, as a firststep to evaluate paraphrasing pairs, we focus on
the evaluation of contextual dependency by using probabilistic Latent Semantic Indexing (pLSI)(Hofmann, 1999) and Latent Dirichlet Allocation(LDA) (Blei et al., 2003) as text models with latent variables.

infer a topic with the models.
In addition to pLSI and LDA, there are other latent variable models such as mixture of unigrams.
We used pLSI and LDA because Blei et al. havealready demonstrated that LDA outperforms mixture of unigrams and pLSI (Blei et al., 2003), anda toolkit has been developed for each model.
From a practical viewpoint, we want to determine how much performance difference exists between pLSI and LDA through evaluations of contextual paraphrase dependency. The time complexity required to infer a topic by LDA is larger
than that by pLSI, and thus it is valuable to know
the performance difference.
2.1

Probabilistic LSIPLSI is a latent variable model for general cooccurrence data that associates an unobserved
topic variable z ∈ Z = {z1 , · · · , zK } with eachobservation, i.e., with each occurrence of wordw ∈ W = {w1 , · · · , wM } in document d ∈ D ={d1 , · · · dN }.
PLSI gives joint probability for a word and adocument as follows:
P (d, w) = P (d)P (w|d),

(1)

where
P (w|d) =X

P (w|z)P (z|d).

(2)

z∈Z

However, to infer a topic indicated by a document, we have to obtain P (z|d). From (Hofmann,
1999), we can derive the following formulas:
P (z|d, w) ∝ P (z)P (d|z)P (w|z)

(3)

and
P (d|z) ∝Xn(d, w)P (z|d, w),(4)w

where n(d, w) denotes term frequency, which isthe number of times w occurs in d. AssumingQ
that P (d|z) = w∈d P (w|z), the probability of a
topic under document (P (z|d)) is proportional to
the following formula:

2 Latent Variable Models and Topic
Inference

P (z)2Yw∈d

In this section, we introduce two latent variable
models, pLSI and LDA, and also explain how toP (w|z)Xn(d, w)P (w|z).

(5)w

After a pLSI model is constructed with a learning corpus, we can infer topic z ∈ Z indicated66

by given document d = w1 , · · · , wM (d) with Formula 5. A topic z that maximizes Formula 5 isinferred as the topic of document d.
2.2

After constructing the pLSI and LDA models,
we can infer a topic by using the models with vector data that correspond to a target sentence.The
vector data for the target sentence are constructedby using the target sentence and the sentences thatsurround it. From these sentences, the vector datathat correspond to the target sentence are constructed. We call the number of sentences used
to construct vector data “window size.”
Evaluating a paraphrasing pair (P 1:P 2) issimple. Construct vector data (vec(P 1) andvec(P 2)) and infer contexts (T (P 1) and T (P 2))
by using a latent variable model. Using pLSI, thetopic that indicates the highest probability is usedas the inferred result, and using LDA, the largest
parameter that corresponds to the topic is used as
the inferred result. If topics T (P 1) and T (P 2)
are different, the sentences might be used in different contexts, and the paraphrasing pair would
be contextually dependent; otherwise, the paraphrasing pair would be contextually independent.

Latent Dirichlet Allocation

Latent Dirichlet Allocation (LDA) is a generative
probabilistic model of a corpus. The basic idea
is that documents are represented as random mixtures over latent topics, where each topic is characterized by a distribution over words.
LDA gives us the marginal distribution of adocument (p(d|α, β), d = (w1 , w2 , · · · wN )) by
the following formula:Zp(θ|α)N X
Y

n=1 zn!

p(zn |θ)p(wn |zn , β) dθ, (6)where α parameterizes Dirichlet random variable θ and β parameterizes the word probabilities, and zn indicates a topic variable zn ∈ Z ={z1 , z2 , · · · , zN }. To obtain the probability of a
corpus, we take the product of the marginal probabilities of single documents.
Here, we omit the details of parameter estimation and the inference of LDA due to space limitations. However, the important point is that the
Dirichlet parameters used to infer the probabilityof a document can be seen as providing a representation of the document in the topic simplex.
In other words, these parameters indicate a pointin the topic simplex. Thus, in this paper, we use
the largest elements of the parameters to infer the
topic (as an approximation of context) to which agiven text belongs.

4 Experiments
We carried out several experiments that automatically evaluated extracted paraphrases with pLSI
and LDA. To carry out these experiments, weused plsi-0.031 by Kudo for pLSI and lda-c 2
toolkit by Blei (Blei et al., 2003) for LDA.
4.1

Data set

We used a bilingual corpus of travel conversation
containing Japanese sentences and corresponding English translations (Takezawa et al., 2002).
Since the translations were made sentence by sentence, this corpus was sentence-aligned from its
origin and consisted of 162,000 sentence pairs.
The corpus was manually and roughly annotated with topics. Each topic had a two-level
hierarchical structure whose first level consistedof 19 topics. Each first-level topic had several
subtopics. The second level consisted of 218 topics, after expanding all subtopics of each topic
in the first level. A rough annotation example
is shown in Table 1; the hierarchical structure ofthis topic seems unorganized. For example, in thefirst-level topic, there are topics labeled basic andcommunication, which seem to overlap.

3 Evaluating Paraphrases with Latent
Variable Models
To evaluate a paraphrasing pair of sentences, wemust prepare a learning corpus for constructing
latent variable models. It must be organized sothat it consist of documents, and each document
must be implicated in a specific context.
Both latent variable models pLSI and LDA require vector format data for their learning. In thispaper, we follow the bag-of-words approach andprepare vector data that consist of words and theirfrequency for each document in the learning corpus.

1
2

67

http://chasen.org/˜taku/software/plsi/
http://www.cs.berkeley.edu/˜blei/lda-c/Table 1: Examples of manually annotated topics
sentence
1st topic
2nd topic
Where is the nearest department store?
shopping
buying somethingThat’s too flashy for me.
shoppingchoosing something
There seems to be a mistake on my bill.
staying
checkout
There seems to be a mistake on my bill.
stayingcomplaining

In the corpus, however, there is an obvious
textual cohesion such that sentences of the same
topic are locally gathered. Each series of sentences can be used as a document for a text model.
Under the assumption that each series of sentences is a document, the average number of sentences included in a document is 18.7, and the average number of words included in a document is
44.9.
4.2

results using pLSI and LDA. We originally considered evaluating each paraphrase as a binaryclassification problem that determines whether
both sentences of the paraphrase are used in thesame context. We evaluated the inferred resultsby comparison with the manually annotated topics, and thus accuracy could be calculated whenthe manually annotated topics were correct. However, accuracy is inappropriate for evaluating results inferred by a latent variable model, since thetopics were roughly annotated by humans as mentioned in Section 4.1. Accordingly, we employed
Kappa statistics as a rough guide for the correctness of the inferred results by latent variable models.
Tables 2 and 3 show the comparison results,
where the window size is 11 (the target sentence+ the previous five and the following five sentences). When constructing pLSI models, the parameter for tempered EM (TEM) is set to 0.9 (weuse this value in all of the experiments in this paper), because it showed the best performance inpreliminary experiments. We performed the experiments on several topics.

Extracting paraphrases

A large collection of parallel texts contains many
sentences in one language that correspond to
the same expression in the other language fortranslation. For example, if Japanese sentences
Ji1 , ..., Jim correspond to English sentence Ei ,then these Japanese sentences would be paraphrases.
We utilized a very simple method to extract
Japanese paraphrases from the corpus. First, we
extracted duplicate English sentences by exact
matching. From the learning set, 18,505 sentences were extracted. Second, we collected
Japanese sentences that correspond to each extracted English sentence. Next, we obtained setsof Japanese sentences collected by using English
sentences as pivots. In the corpus, one English
sentence averaged almost 4.5 Japanese sentences,
but this number included duplicate sentences.
If duplicate sentences are excluded, the average
number of Japanese sentences corresponding toan English sentence becomes 2.4. Finally, we
obtained 944,547 Japanese paraphrasing pairs by
combining sentences in each group of Japanese
sentences.
4.3

Table 2: Comparing results of first-level topic(19)# of topics κ by pLSI κ by LDA
10
0.48120.4798200.50850.518530
0.50870.509440
0.5392
0.524550
0.51850.4897window size = 11
As mentioned in Sections 2.1 and 2.2, we cantreat inference results as vector data. Thus, we
can use a metric to classify the two vectors that
correspond to the inferred results of any two given
sentences. We use cosine as a metric and con-Comparing human judgement andinference by latent variable models

In this section, we determine the difference between manually annotated topics and inference68Table 3: Comparing results of second-level topic(218)# of topicss κ by pLSI κ by LDA
30
0.35230.388340
0.36630.409350
0.41220.411160
0.41840.418670
0.4196
0.413380
0.36650.3702900.34370.35961000.30760.3526window size = 11

Table 6: Window size and Kappa statistics forfirst-level annotation
window
pLSI
LDA
size
(20 topics) (20 topics)
5
0.45800.2527110.50850.5185150.51650.544021
0.46130.539625
0.3286
0.528631
0.17300.5157shows the experimental results.
The actual computing time needed to evaluate
944,547 paraphrases with a Pentium M 1.4-GHz,
1-GB memory computer is shown in Table 7. It is
important to note that the inference program for
pLSI was written in Perl, but for LDA it was written in C.

ducted comparison experiments for the first- andsecond-level topics, as shown in Tables 4 and 5.
The threshold values used to judge whether topics
are the same are indicated in the parentheses.
Table 4: Comparing results of first-level topic(19) with cosine metric# of topics
κ by pLSI
κ by LDA
10
0.4873(0.5) 0.5042(0.5)
20
0.5230(10−6 ) 0.5841(0.5)
300.5502(10−6 ) 0.5672(0.5)
40
0.5808(10−6 ) 0.5871(0.5)
50
0.5611(10−6 ) 0.5573(0.5)
window size = 11

Table 7: Computing time to evaluate paraphrases# of topics
pLSI
LDA
20
665 sec.
996 sec.
60
1411 sec. 2223 sec.
window size = 15
4.4

Experiments from paraphrasing
perspectives

To investigate the upper bound of our method, we
carried out several experiments. So far in this paper, we have discussed topic information as an
approximation of contextual information by comparing topics annotated by humans and automatically inferred by pLSI and LDA. However, since
our goal is to evaluate paraphrases, we need todetermine whether latent variable models detect a
difference of topics for sentences of paraphrases.
First, we randomly selected 1% of the English
seed sentences. Each sentence corresponds toseveral Japanese sentences, so we could produce
Japanese paraphrasing pairs. The number of selected English sentences was 185.
Second, we generated 9,091 Japanese paraphrasing pairs from the English seed sentences.
However, identical sentences existed in some generated paraphrasing pairs. In other words, thesesentences were simply collected from different

Table 5: Comparing results of second-level topic(218) with cosine metric# of topics
κ by pLSI
κ by LDA
30
0.3536(0.5)
0.3726(0.5)
40
0.3679(0.5)
0.4006(0.5)
50
0.4127(0.5)
0.4085(0.5)
60
0.4186(0.5)
0.4218(0.5)
700.4202(0.5)
0.4202(0.5)
80
0.3733(0.5) 5.2 ∗ 10−7 (0.5)
window size = 11
We also performed an experiment to confirm
the relationship between Kappa statistics andwindow-size context. Experiments were done under the following conditions: the number of topics
was 20 for both pLSI and LDA, Kappa statistics
were calculated for the first-level topic, and window sizes were 5, 11, 15, 21, 25, and 31. Table 6
69

To estimate the upper bound of this method, we
also investigated potentially unavoidable errors.
Several paraphrasing pairs are used for the exact
same topic, but they seem contextually dependent
because several words are different. On the other
hand, some paraphrasing pairs seem to be usedin obviously different topics but are contextuallyindependent. Table 9 shows the investigation results; at least ten paraphrasing pairs seem contextually independent but are actually used in different topics. In addition, there are at least 15 paraphrasing pairs whose topic is obviously the same,but several differences of words make them contextually dependent. Moreover, in this case, theerror rate is 0.231 ((15+10)/108), meaning that itis difficult to judge all of the paraphrasing pairs
correctly by using only topic (contextual) information. Thus, this method’s upper bound of accuracy when using only topic information is estimated to be around 77%.

places in the corpus. From a paraphrasing perspective, such pairs are useless. Thus we removed
them and randomly selected one pair from one
English seed sentence.
Finally, we sampled 117 paraphrasing pairs
and evaluated them based on a paraphrasing perspective: whether a paraphrase is contextuallyindependent. There were 71 contextually independent paraphrases and 37 contextually dependent paraphrases. Nine paraphrases had problems, all of which were caused by translation errors. The phrase “contextually independent paraphrases” means that the paraphrases can be usedin any context and can be applied as two-way
paraphrases. On the other hand, “contextually dependent paraphrases” means that the paraphrases
are one-way, and so we have to give consideration
to the direction of each paraphrase.
Table 8: Evaluation with manually annotated labels
independent
dependent
same diff.
same diff.
1st level
46
25
18
19
2nd level
25
46
11
26

Table 9: Potential upper bound of this method
human judgement human judgementfrom paraphrasing based on topic
perspectivesamedifferent
independent
61
10
15
22
dependent

We removed the nine problematic paraphrasing pairs and evaluated the remaining samples
with manually annotated topic labels, as shownin Table 8. According to the basic idea of thismethod, a contextually independent paraphrasing pair should be judged as having the same
topic, and a contextually dependent pair should
be judged as having a different topic. Thus, we
introduced a criterion to evaluate labeling resultsin terms of an error rate, defined as follows:|Dindep | + |Sdep |,Error rate =# of judged pairs

We prepared several latent variable models
to investigate the performance of the proposed
method and applied it to the sampled paraphrasing sentences mentioned above. Table 10 showsthe evaluation results.

5 Discussion
First, there is no major performance difference
between pLSI and LDA in paraphrasing evaluation. On average, LDA is slightly better thanpLSI. Blei et al. showed that LDA outperforms
pLSI in (Blei et al., 2003); however, in some of
the cases shown in Tables 2 and 3, pLSI outperforms LDA. On the contrary, using a cosine metric, LDA has a significant problem: it loses its distinguishing ability when the number of topics (latent variables) becomes large. With such a large
number of topics, LDA always infers a point near
the gravity point of the topic simplex. In addition,using a cosine metric also requires a threshold to(7)where Dindep denotes a set that consists of paraphrasing pairs that are judged as having different topics but are contextually independent. On
the other hand, Sdep denotes a set that consists ofparaphrasing pairs that are judged as having thesame topic, but are contextually dependent.
For example, from the results in Table 8, theerror rate of the results for the first-level topic is0.398 ((25 + 18)/108), and that for the secondlevel topic is 0.528 ((46 + 11)/108).
70

Table 10: Evaluating contextual dependency of paraphrases by latent variable models
model
window independent
dependent
corrected(threshold)size
same diff.
same diff.
err. rate
err. rate
pLSI20
11
43
28
14
23
0.38890.2048pLSI20
15
39
32
14
23
0.42590.2530pLSI40
11
33
38
12
25
0.46300.3012
pLSI40
15
34
37
16
21
0.49070.3373
pLSI20cos(10−6 )
11
45
26
17
20
0.39810.2169−6
pLSI20cos(10 )
15
31
40
15
22
0.50930.3614
pLSI40cos(10−6 )
11
43
28
17
20
0.41670.2410pLSI40cos(10−6 )
15
29
42
13
24
0.50930.3614
LDA20
11
39
32
19
18
0.47220.3133
LDA20
15
42
29
16
21
0.41670.2410
LDA40
11
40
31
14
23
0.41670.2410
LDA40
15
35
36
15
22
0.47220.3133
LDA20cos(0.5)
11
49
22
23
14
0.41670.2410LDA20cos(0.5)
15
51
20
21
16
0.3796
0.1928LDA40cos(0.5)
11
47
24
18
19
0.38890.2048LDA40cos(0.5)
15
43
28
17
20
0.41670.2410
1st-level topic
–
46
25
18
19
0.39810.2169judge a pair of paraphrasing sentences.
From Table 6, LDA seems robust against theinclusion of noisy sentences with a large window,
but it is easily affected by a small window. On
the other hand, pLSI seems robust against information shortages due to a small window, but it is
not effective with a large window. The best performances were shown at window size 15 for both
pLSI and LDA, since the average number of sentences in a document (segment) is 18.7, as shownin Section 4.1.
Table 7 shows that in spite of the difference inprograming language, pLSI is faster than LDA inpractice. In addition, Table 8 reveals that judging
the contextual dependency of paraphrasing pairs
does not require fine-grained topics.
From the results shown in Table 10, we canconclude that topic inference by latent variable
models resembles context judgement by humans
as recorded in error rate. However, we note thatthe error rate was not weighted for contextuallyindependent or dependent results. Error rate issimply a relative index. For example, if there isa result in which all of the inferences reflect thesame topic, then the error rate becomes 0.3426.
Thus it is important to detect a contextually de-pendent paraphrase. Considering these points,
pLSI20 with window size 11 shows very good results in Table 10.
In Section 4.4, we showed the potential upperbound of this method. The smallest error rate is0.231, and we can estimate a corrected error by
the following formula:|Dindep | + |Sdep | − C,# of judged pairs − C(8)where C denotes the correction value that corresponds to the number of paraphrasing pairs
judged incorrectly with only contextual information. In our experiments, from the results shownin Table 9, C is set to 25. From the results shownin Table 10, we can conclude that the performanceof our method is almost the same as that by the
manually annotated topics, and the accuracy of
our method is almost 80% for paraphrasing pairsthat can be judged by contextual information.
There are several possibilities for improving
accuracy. One is using a f ixed window to obtain contextual information. Irrelevant sentences
are sometimes included in fixed windows, and latent variable models fail on inference. If we could
infer a boundary of topics with high accuracy,71

we would be able to dynamically detect a precise window using some other reliable text models specialized to text segmentation.
So far, we have mainly discussed the contextual dependency of paraphrasing pairs. However,when a paraphrasing pair is contextually dependent, it is also important to infer its specific paraphrasing direction. Unfortunately, we conclude
that inferring the paraphrasing direction with contextual information is difficult. In the experimental results, however, there were several examples
whose direction could be inferred from their contextual information. Thus, contextual information
may benefit the inference of paraphrasing direction. Actually, in the experiments, 11 of 37 contextual dependent pairs had obvious paraphrasing
directions. In most of the paraphrasing pairs, different words were used or inserted, or some words
were deleted. Thus, to infer a paraphrasing direction, we need more specific information for words
or sentences; for example what words carry specific or generic meaning and so on.
One might consider a supervised learning
method, such as Support Vector Machine, to infer topics (e.g., (Lane et al., 2004)). However, wecannot know the best number of topics for an application in advance. Thus, a supervised learning
method is promising only if we already know thebest number of topics for which we can prepare
an appropriate learning set.

contextual information is almost 77%.

Acknowledgements

This research was supported in part by the Ministry of Public Management, Home Affairs, Posts
and Telecommunications.

References
Regina Barzilay and Kathleen R. McKeown. 2001.
Extracting paraphrases from a parallel corpus. In
Proceedings of the 39th Annual Meeting of the ACL,
pages 50–57.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan.
2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993–1022, January.
Thomas Hofmann. 1999. Probabilistic Latent Semantic Indexing. In Proceedings of the 22nd Annual
ACM Conference on Research and Development inInformation Retrieval, pages 50–57.
Ian R. Lane, Tatsuya Kawahara, Tomoko Matsui, andSatoshi Nakamura. 2004. Topic classification andverification modeling for out-of-domain utterance
detection. In Proceedings of ICSLP, pages 2197–
2200.
Dekang Lin and Patrick Pantel. 2001. Discovery
of inference rule for question-answering. Natural
Language Engineering, 7(4):343–360.
Kiyonori Ohtake and Kazuhide Yamamoto. 2001.
Paraphrasing honorifics. In Workshop Proceedingsof Automatic Paraphrasing: Theories and Applications (NLPRS2001 Post-Conference Workshop),
pages 13–20.
Mitsuo Shimohata and Eiichiro Sumita. 2002. Automatic paraphrasing based on parallel corpus for normalization. In Proceedings of LREC 2002, pages
453–457.

6 Conclusion
We proposed an evaluation method for the contextual dependency of paraphrasing pairs using twolatent variable models, pLSI and LDA. To evaluate a paraphrasing pair, we used sentences surrounding the given sentence as contextual information and approximated context by topics that
correspond to a latent variable of a text model.
The experimental results with paraphrases automatically extracted from a corpus showed that the
proposed method achieved almost 60% accuracy.
In addition, there is no major performance difference between pLSI and LDA. However, they haveslightly different characteristics: LDA is robust
against noisy sentences with long context, while
pLSI is robust against information shortage due
to short context. The results also revealed thatany method’s upper bound of accuracy using only

Tetsuro Takahashi, Tomoya Iwakura, Ryu Iida, Atsushi Fujita, and Kentaro Inui. 2001. KURA:
A transfer-based lexico-structural paraphrasing engine. In Proceedings of Automatic Paraphrasing: Theories and Applications (NLPRS2001 Workshop), pages 37–46.
Toshiyuki Takezawa, Eiichiro Sumita, Fumiaki Sugaya, Hirofumi Yamamoto, and Seiichi Yamamoto.
2002. Toward a broad-coverage bilingual corpus
for speech translation of travel conversations in thereal world. In Proceedings of LREC 2002, pages
147–152.
Kazuhide Yamamoto. 2002. Acquisition of lexical
paraphrases from texts. In Proceedings of the 2nd
International Workshop on Computational Terminology (Computerm 2002, in conjunction with Coling 2002), pages 22–28.

72