Evaluating Contextual Dependency of Paraphrases using a Latent Variable Model Kiyonori OHTAKE ATR Spoken Language Communication Research Laboratories, Japan October 14, 2005
Example
paraphrasing pair: A: I want to buy a pair of sandals. B: I’m looking for sandals. Can we always paraphrase them? from A to B: Yes from B to A: No
Latent variable text models
。 PLSI P(d, w) Probabilistic replacement of LSI P LDA P(d|α, β) Bayesian replacement of pLSI Latent (hidden) variable represents topic Note: # of latent variables (topics) is given
Background
。 Problem:  contextual dependency of paraphrases 。 applying 。 collecting s etc. How to cope with contextual dependency?
Objective
To construct an evaluation method for contextual dependency of paraphrases If we can evaluate ... 』 mis-paraphrasing 。mis-collecting paraphrases will be avoided
Approach
Using latent variable text model 。 modeling a text (unsupervised) 。 each latent variable represents a topic Context: sentence and surrounding sentences = window Approximation of context: topic indicated by a latent variable
Latent variable model: pLSI
d: document, z: latent variable, w: word, N : vocaburary size, M : # of documents
Overview of Experiments
Comparing topic vectors
Extracting paraphrases
Data set
Comparing with labels by human
Result 1/2
θ : Dirichlet random variable α: parameter for θ, β : parameter for w ● Based on the largest element Whether the largest element of topic vectors are the same ● Comparing with labels by human Matching the results of our method with the result based on the topic labels by human 。 Cosine Whether cosθ between topic vectors greater than threshold ● Evaluation for paraphrasing Evaluating our method based on collecting situation for paraphrases Bilingual corpus of travel conversation (162,000 sentence pairs) bilingual corpus Manually and roughly labeled with topics (hierarchical; level-1: 19, level-2: 218) Textual cohesion → Used fixed window to clip a context Data format: bag-of-words Obtained 944,547 Japanese paraphrasing pairs Measurement: Kappa statistics Two comparing method: largest, cosine
Discussion
Evaluation for paraphrasing
Computing time
Discussion
Result
Conclusion
Future works
● Kappa statistics: 0.59 (highest) moderate: 0.4-0.6 substantial: 0.6-0.8 perfect: over 0.8 。 No major performance difference  LDA was good at with cosine 。 PLSI sometimes outperformed LDA 1. Fed randomly sampled 108 paraphrasing pairs into our method 2. Manually evaluated whether they were contextually independent Measurement: error rate ● Almost the same as the result based on labels by human 8 25 unavoidable errors Potential upper bound based on topic information: 77% (0.23 error rate) ● Proposed evaluation method for contextual dependency of paraphrases using pLSI and LDA  Introducing a topic-boundary detection technique ● Employing more complicated data (e.g., dependency structure) not b.o.w. ● No major performance difference between pLSI and LDA 。 Investigating difference of paraphrase: 。 Potential upper bound using only topic: 77% Achieved: 62%  What makes contextual dependency?  What contexts are possible? Thank you very much. Roshering Cortestad Experieroyal Proginans and status Verietia Marie , p. 26.