Visual Description Paraphrase Corpus Creation
with Various Elementary Operations ∗Johanes Eﬀendi1 , Sakriani Sakti1,2 , Satoshi Nakamura1,2 (1 NAIST, 2 RIKEN AIP)
1

Introduction
This paper reports the construction of a multi-paraphrase corpus of visual description through proposed paraphrase elementary operations (i.e., reordering, substitution, deletion, insertion) using
crowdsourcing platform. Unlike existing paraphrase
corpus, the use of elementary operations enables thepossibility to track what operation has been doneto the original sentence. Furthermore, the combinaFig. 1

tions of this elementary operations can be utilized

Country distribution of crowdworkers

for various applications such as abstractive summa-

from Visual Genome [2], which consists of such di-rization using deletion, and data augmentation us-

verse objects as animals, people, and vehicles, anding substitution or reordering.

asked the crowdworkers to describe these images and2

Multi-paraphrase Corpus Creation

create paraphrases of the captions into four diﬀer-

2.1

Possible Variants of Paraphrases

Most existing paraphrase corpora cover only oneto-one parallel sentences, overlooking the fact thatpossible variants of paraphrases can be generated
from a single source sentence. On the other hand,ent paraphrases based on each operation. The corpus consists of 5-pairs of paraphrases with 50 k sentences. Table 1 show an example of the resulting
image description and the paraphrased expressions
by the crowdworkers.
We used Crowdflower (now Figure Eight) as a

Bhagat and Hovy categorized the variations of how
humans paraphrase [1] and argued that there should
be a broader definition of paraphrasing that is accepted by linguist. Based on this idea, they analyzed the paraphrase characteristics in various studies and corpora and established 25 quasi-paraphraseclasses, such as change of tenses, metaphor substitution, function word variations, etc. Each quasiparaphrase class has its own way of implementing
the semantic equivalence standards of a paraphrase.
As reported in their paper,

crowdsourcing platform.

crowdworker had to describe and paraphrase at least
two images and also be available for additional sessions of image annotation. We limited this task toEnglish-speaking countries or those countries where
English is the second language. We also monitored
the crowdworker results. If the resulting sentenceswere not valid paraphrases, we discarded them fromthe data.
We successfully gathered more than 200 workers

many quasi-paraphrases have very small frequency when it issurveyed on the real corpora. We argue that these
25 classes can actually be grouped into fewer classes.
We grouped them and created a constraint for eachof our sub-corpus into the following four operations:deletion, insertion, reordering, and substitution.

For one session, eachfrom 13 countries for this corpus creation. Mostworkers came from India, the United States, andPhilippines. The remaining countries and distribution are shown in Fig. 1. Each worker created anaverage of 52.56 images took an average of 7 minutes 54 seconds to describe an image and write its
four paraphrases.

2.2

Corpus Creation

Following [3], we used image as the basis of paraphrasing, and paraphrase the visual description using these four operations. We collected 10 k images

3

Corpus Analysis

3.1

Operation Characteristics

To calculate the deletion operation eﬀect on thesource sentences, we compared the ratio of the num-∗

簡易的操作を用いた画像説明文言い換えコーパスの作成、
ジョハネスエフェンヂィ1 、サクリア二サクチイ 1,2 、中村 哲 1,2 （1 NAIST, 2 RIKEN AIP）

Table 1
Image

Visual description and its paraphrase examples
Operation

Image Caption

Paraphrase

Sentence
A cat is scratching the seats in the park with its nail.

Deletion

A cat is scratching the seats in the park

Insertion

A cat is scratching the seats in the park with its sharp nail.

Substitution

A cat is scratching the bench in the park with its nail.

ReorderingWith its nail, a cat is scratching the seats in the park

ber of words in the target sentence to the number inTable 3 shows that nouns (NN), adjectives (JJ),

the source sentence and found that the number ofand conjunctions (IN) were usually deleted or in-words in the target sentences decreased by 19.53%,serted. This correlates with how most sentences arewhere an average of 2.23 words was deleted per sen-deleted or inserted: by adding or removing time,tence (Table 2). We did the same calculation in theplace, or tool information. For reordering the sub-insertion sub-corpus and found that the number ofcorpus, we found that no word type is changed bywords in the target sentences increased by 25.15%,

reordering and substitutions which implies no major

where an average of 2.93 words were inserted peralterations to the words or the semantics.

sentence.

4Table 2

Operation characteristics

Parameter# word

Avg. deleted words per sentence

2.2286

Avg. inserted words per sentence

2.9311

Avg. distance of reordered word

4.4252

Avg. word substitutions per sentence

1.6770Conclusions
This paper presented a construction of a multi-paraphrase corpus consisting of four elementary operations, enable multi-paraphrase sentences to begenerated from a source sentence. Our corpus wasdeveloped using a crowdsourcing platform that collected 5-pairs of paraphrases of 10 k images resulting

To measure the reordering elementary operationin 50 k sentences. In the future, we will utilize ourfor the sub-corpus, we calculated the shift distanceparaphrase corpus for various applications, includ-of a word in the source and target sentences anding summarization or machine translation.

found that those in the latter shifted on averageby as many as 4.42 words. The distance calculated

5Acknowledgements
Part of this work was supported by JSPSin the reordering happened when the source sentence was paraphrased into its passive form, or toKAKENHI

exchange the order of such sentence information as

JP17K00237.

Grant

Numbers

JP17H06101

andtime, place, and tool. As seen in Table 2, we found
an average of 1.68 word substitutions per sentence

Referencesin the substitution sub-corpus. This means that at[1] R. Bhagat and E. Hovy. What is a paraphrase?

most 1 or 2 words were substituted in a sentence.
3.2

Evaluation Based On Word TypeComputational Linguistics, 39(3):463–472, 2013.
[2] R. Krishna, Y. Zhu, O. Groth, J. Johnson,

We counted the types of words that were most

K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J.

deleted and inserted. For reordering and substitu-Li, D. A. Shamma, et al. Visual genome: Con-tion, we counted a pair of source and target word

necting language and vision using crowdsourced

types. This evaluation identifies which word type

dense image annotations. International Journal

was usually preferred by the crowdworkers.

of Computer Vision, 123(1):32–73, 2017.

Table 3

Top 5 Most Operated POS Tags

[3] A. Prakash, S. A. Hasan, K. Lee, V. Datla,Deletion

Insertion

Reordering

Substitution

NNNN

DT DT

NN NN

JJ

IN

NN NN

NN NNS

phrase generation with stacked residual lstm net-IN

JJ

IN IN

NN JJworks.

DT

DT

JJ JJ

NNS NNS

2923–2934, Osaka, Japan, December 2016.

VBG

NNS

NNS NNSNNS NNA. Qadir, J. Liu, and O. Farri. Neural paraIn Proceedings of COLING 2016, pp.