# Presentation: 415387

## Multilingual Topic Detection using a parallel corpus

- Wai Lam
- Helen, M. Meng
- Hui Kin** **
**The Chinese University of Hong Kong**

## Presentation Flow

- System Overview
- Story Preprocessing
- Named Entity Extraction
- Gross Translation
- Unsupervised Learning Module
- Experiment Result

## System Overview

##  Characteristics

- Developed two-dimensional semantic representation scheme
- Investigated two online gross translation approaches
- 	- basic translation
- 	- enhanced translation

## Story Preprocessing

- Identify sentence boundaries for subsequent processing
- 	- Lexical clues
- 	- length of silence period
- Perform word segmentation for Chinese stories

## Named Entity Extraction

- Extract named entities on
- people names
- geographical location names
- organization names
- Transformation-based linguistic tagger
- one for English and one for Chinese
- Learn transformation rules on named entities

## Story Representation 

- Two-dimensional semantic representation
- named entity feature representation
- content term representation

- where w(S,pi ) represents the weight of the corresponding people name term pi in story S; li represents a location name; oi represents an org. name

-    Named entity feature representation

-    Content term representation

## Story Representation

**Several factors determining the weight **
- the relative location of the terms within the story
- term frequency
- incremental document frequency

## Gross Translation

- Translate Chinese story terms to English to allow direct comparison between stories in an uniform representation
- Two Translation Approaches :
- Basic Translation
- Enhanced Translation

## Basic Translation

- Look up the bilingual dictionary
- Look up pin-yin file for people name component and geographical location name component when no entry exists in the bilingual dictionary
- The weight is:
- e.g. ‘著名’ is translated as ‘famous’, ‘well-known’ & ’celebrated’
- w(S,’famous’) = w(S,’著名‘) / 3 = 1.000 / 3

## Enhanced Translation

- Parallel Corpus collection on Chinese and English News ( From Information Services Department of HKSAR)
- From 1 July 1997 to 30 June 1998
- About 5,000 pairs of English and Chinese stories
- In story - aligned level
- Convert stories to passage-based alignment via an automated procedure

## Enhanced Translation

- Maximize the probability over all possible alignments
- By converting the alignment probabilities from story level to passages level, we have
- Solve by Dynamic Programming Approach

## Enhanced Translation

**By Bayes’ Rule:**
**Pr(P****e****,P****c**** | A)** can be estimated according to the **Gaussian Distribution**
**Pr (P****e****, P****c ****)** is priors over six categories of alignment on passage (Wu, 1994)

- where Pr(A)is a normalizing constant that can be ignoring during minimization

## Enhanced Translation

- "故事 的 靈感 及 意念 來自 漢 姆 舒 拉 格 的 同名 作, 並 由 **著名** 默 劇 家 黃 國 忠 及 香港 聾劇團 一起 創作。”

- "The inspiration of this play comes from a book written by Carl A. Hammerschlag. The theatre piece has been created by the** famous** mime artist Wong Kwok Chung with members of the Theatre of the Deaf."

- "梁 國 新 表示 , 汀 九 橋 及 其 高 架 引 道 無疑 會 成為 香港 **著名** 及 令人 讚賞 的 標誌,更 成為 鄰近 青 嶼 幹線 不 相 伯 仲 的 鄰居。"

- “Mr Leung noted that the new vehicular bridge and the approach viaduct would be a **well-known** and much admired landmark in Hong Kong, as well as a more than worthy neighbour to the nearby Lantau Link."

## Enhanced Translation

- Use similarity-based retrieval technique to retrieve the top ranked Chinese passages
- Retrieve corresponding English passages
- Count the occurrence frequency of each English translated term in the retrieved English passages
- Adjust the weight of the dictionary translated terms

## Enhanced Translation

- Usage Factor
*U*(著名, famous) = ( 1+1 ) / (( 1+1 )+( 1+1 ) +( 0+1 )) = 0.4,
*U*(著名, well-known) = ( 1+1 ) / (( 1+1 ) + ( 1+1 ) + ( 0+1 )) = 0.4, and
*U*(著名, celebrated) = ( 0+1 ) / (( 1+1 ) + ( 1+1 ) + ( 0+1 )) = 0.2

- where
*N* represents the number of translation terms of the Chinese term ;
*t**c** *represents the Chinese term ;
*t**t** *represents the English translated term ; and,
*f**t* represents the accumulated term frequency of the English translated term within the extracted English passages.

## Enhanced Translation

- Final weight of translated term
- w(*S *, 'famous') = 0.4 * 1.000 = 0.4;
- w(*S*, 'well-known') = 0.4 * 1.000 = 0.4, and
- w(*S* ,'celebrated') = 0.2 * 1.000 = 0.2

- where
*w( S, t**t **) *represents the final term weight of the translated story term *t**t* in story *S*.

## Employs a modified agglomerative clustering approach 
Maintains three kinds of elements
stories, temporary clusters, and final clusters
Similarity Calculation
Grouping Related Elements
Topic Identification

- Maintains three kinds of elements
- stories, temporary clusters, and final clusters
- Similarity Calculation
- Grouping Related Elements
- Topic Identification

- Unsupervised Learning Module

## Similarity Calculation

## Similarity Calculation

- Cosine similarity of term representation
- Final Similarity Score

- where *w**n** *is the user-defined weight on the named entity component in the story representation

## Grouping Related Elements

## Topic Identification

- Transfer stories from temporary clusters to final clusters
- Check stories in temporary clusters belonging to the first batch stories
- Move the first batch stories to final clusters
- Take the average of the weights of the stories in the final clusters
- Remove all temporary clusters and retain final clusters

## Experiment Result

- Results for TDT2 (Jan-June, 1998) corpus, manual story boundary, native languages

## Experiment Result

- Results for TDT2 (Jan-June, 1998) corpus, manual story boundary, native languages

## Experiment Result

- Experiments for TDT3 Evaluation Corpus (Oct-Dec, 1998)
- 	- automatic story boundary provided by NIST, native languages used
- 	- the detection cost for translated Chinese stories only is 0.4079
- 	- the detection cost for multilingual stories is 0.6588