Venue: NAACL 2019

# A Study of Incorrect Paraphrases in Crowdsourced User Utterances

Mohammad-Ali Yaghoub-Zadeh-Fard, Boualem Benatallah,

**Moshe Chai Barukh** and **Shayan Zamanirad**

University of New South Wales, Sydney

{m.yaghoubzadehfard,b.benatallah,mosheb,shayanz}@unsw.edu.au

###### Abstract

Developing bots demands high quality training samples, typically in the form of user utterances and their associated intents. Given the fuzzy nature of human language, such datasets ideally must cover all possible utterances of each single intent. Crowdsourcing has widely been used to collect such inclusive datasets by paraphrasing an initial utterance. However, the quality of this approach often suffers from various issues, particularly language errors produced by unqualified crowd workers. More so, since workers are tasked to write open-ended text, it is very challenging to automatically asses the quality of paraphrased utterances. In this paper, we investigate common crowdsourced paraphrasing issues, and propose an annotated dataset called _Para-Quality_, for detecting the quality issues. We also investigate existing tools and services to provide baselines for detecting each category of issues. In all, this work presents a data-driven view of incorrect paraphrases during the bot development process, and we pave the way towards automatic detection of unqualified paraphrases.

## 1 Introduction

With the increasing advances in deep learning as well as natural language processing, a new generation of conversational agents is attracting significant attention Dale (2016). Also known as _dialogue systems_, _virtual assistants_, _chathots_ or simply _bots_Campagna et al. (2017); Su et al. (2017), some advanced bots are now designed to perform complex tasks (e.g., _flight booking_), many of which are built using machine learning techniques.

At the heart of building such _task-oriented bots_ lies the challenge of accurately capturing the user's intent (e.g., _find cafes in Chicago_), and then extracting its entities to service the request (e.g _term= "cafes", location="Chicago"_). However, its success relies heavily on obtaining both, large and high quality corpora of training samples showing mappings between sample utterances and intents. This is necessary given the ambiguous nature of the human language Wasow et al. (2005) and large variations of expressions Wang et al. (2012); Zamanirad et al. (2017).

A lack of variations in training samples can result in incorrect intent detection and consequently execution of undesirable tasks (e.g., booking an expensive hotel instead of a cheap room) Henderson et al. (2018). Likewise, quality issues in the training samples can lead to unmitigated disasters Neff and Nagy (2016) as it happened to Microsoft's _Tay_ by making a huge number of offensive commentaries due to biases in the training data Henderson et al. (2018). It is therefore not surprising that research and development into training data acquisition for bots has received significant consideration Campagna et al. (2017); Kang et al. (2018).

Collecting training samples usually involves two primary steps: (i) firstly, obtaining an initial utterance for a given user intent (e.g., _find a cafe in Chicago_); and (ii) secondly, paraphrasing this initial expression into multiple variations Su et al. (2017); Campagna et al. (2017). Paraphrasing is thus vital to cover the variety of ways an expression can be specified Yang et al. (2018). As summarized in McCarthy et al. (2009), a quality paraphrases has three components: semantic completeness, lexical difference, and syntactic difference. To obtain lexically and syntactically diverse paraphrase, crowdsourcing paraphrases has gained popularity in recent years. However, crowdsourced paraphrases need to be checked for quality, given that they are produced by unknown workers with varied skills and motivations Campagna et al. (2017); Daniel et al. (2018). For example, spammers, malicious and even inexperi

enced crowd-workers may provide misleading, erroneous, and semantically invalid paraphrases (Li et al., 2016; Campagna et al., 2017). Quality issues may also stem from misunderstanding the intent or not covering important information such as values of the intent parameters (Su et al., 2017).

The common practice for quality assessment of crowdsourced paraphrases is to design another crowdsourcing task in which workers validate the output from others. However, this approach is costly having to pay for the task twice, making domain-independent automated techniques a very appealing alternative. Moreover, quality control is especially desirable if done before workers submit their paraphrases, since low quality workers can be removed early on without any payment. This can also allow crowdsourcing tasks to provide feedback to users in order to assist them in generating high quality paraphrases (Nilforoshan et al., 2017; Nilforoshan and Wu, 2018). To achieve this, it is therefore necessary to automatically recognize quality issues in crowdsourced paraphrases during the process of bot development.

In this paper, we investigate common paraphrasing errors when using crowdsourcing, and we propose an annotated dataset called _ParabQuality_ in which each paraphrase is labelled with the error categories. Accordingly, this work presents a quantitative data-driven study of incorrect paraphrases in bot development process and paves the way towards _enhanced_ automated detection of unqualified paraphrased utterances. More specifically, our contributions are two-folded:

* We obtained a sample set of 6000 paraphrases using crowdsourcing. To aim for a broad diversity of samples, the initial expressions were sourced from 40 expressions of highly popular APIs from various domains. Next, we examined and analyzed these samples in order to identify a taxonomy of common paraphrase errors (e.g., cheating, misspelling, linguistic errors). Accordingly, we constructed an annotated dataset called _Para-Quality_ (using both crowdsourcing and manual verification), in which the paraphrases were labeled with a range of different categorized errors.
* We investigated existing tools and services (e.g., spell and grammar checkers, language identifiers) to detect potential errors. We formulated baselines for each category of errors to determine if they were capable to automatically detect such issues. Our experiments indicate that existing tools often have low precision and recall, and hence our results advocates the need for new approaches in effective detection of paraphrasing issues.

## 2 Paraphrase Dataset Collection

Various types of paraphrasing issues have been reported in the literature, namely: spelling errors (Braunger et al., 2018), grammatical errors (Jiang et al., 2017; Negri et al., 2012), and missing slot-value (happens when a worker forget to include an entity in paraphrases) (Su et al., 2017). We collected paraphrases for two main reasons: (i) to have a hands-on experience on how incorrect paraphrases are generated, and (ii) to annotate the dataset for building and evaluating paraphrasing quality control systems.

**Methodology.** We obtained 40 expressions from various domains (i.e. _Yelp, Skyscanner, Spotify, Scopus, Expedia, Open Weather, Amazon AWS, Gamil, Facebook, Bing Image Search_) indexed in ThingPedia (Campagna et al., 2017) and API-KG1. We then launched a paraphrasing task on Figure-Eight2. Workers were asked to provide three paraphrases for a given expression (Jiang et al., 2017), which is common practice in crowdsourced paraphrasing to reduce repetitive results (Campagna et al., 2017; Jiang et al., 2017). In the provided expression, parameter values were highlighted and crowd-workers were asked to preserve them. Each worker's paraphrases for an initial utterance are normalized by lowercasing and removing punctuation. Next, the initial utterance and the paraphrases are compared to forbid submitting empty strings or repeated paraphrases, and checked if they contain highlighted parameter values (which is also a common practice to avoid missing parameter values) (Mitchell et al., 2014). We collected paraphrases from workers in English speaking countries, and created a dataset containing 6000 paraphrases (2000 triple-paraphrases) in total3.

Footnote 1: [http://apikg.ngrook.io](http://apikg.ngrook.io)

Footnote 2: [https://www.figure-eight.com](https://www.figure-eight.com)

Footnote 3: [https://github.com/mysilver/ParaQuality](https://github.com/mysilver/ParaQuality)

## 3 Common Paraphrasing Issues

To characterize the types of paraphrasing issues, two authors of this paper investigated the crowd 

sourced paraphrases, and recognized 5 primary categories of paraphrasing issues. However, we only considered paraphrase-level issues related to the validity of a paraphrase without considering dataset-level quality issues such as lexical diversity Negri et al. (2012) and bias Henderson et al. (2018).

### Spelling Errors

Misspelling has been reported as one of most common mistakes in paraphrasing Inaba et al. (2015); Wang et al. (2012); Chklovski (2005); Braunger et al. (2018). In our sample set, we also noticed misspellings were generated both intentionally (as an act of cheating to quickly generate a paraphrase such as Example 2 in Table 1) and unintentionally (due to a lack of knowledge or a simple mistake such as Example 3 in Table 1).

### Linguistic Errors

Linguistic errors are also common in crowdsourced natural language collections Jiang et al. (2017); Negri et al. (2012). Verb errors, preposition errors, vocabulary errors (improper word substitutions), and incorrect singular/plural nouns, just to name a few. Moreover, capitalization and article errors seems abundant (e.g., Example 5 in Table 1). Given that real bot users also make such errors, it is important to have linguistically incorrect utterances in the training samples Bapat et al. (2018). However, at a very least, detecting linguistic errors can contribute to quality-aware selection of crowd workers.

### Semantic Errors

This occurs when a paraphrase deviates from the meaning of the initial utterance (e.g., _find cafes in Chicago_). As reported in various studies, workers may forget to mention parameter values (also known as missing slot)(e.g., _find cafes_)4Crossley et al. (2016); Su et al. (2017); Ravichander et al. (2017); Wang et al. (2012); Braunger et al. (2018), provide wrong values (e.g., _find cafes in Paris_)Su et al. (2017); Ravichander et al. (2017); Wang et al. (2012); Negri et al. (2012); Braunger et al. (2018), or add unmentioned parameter valuesWang et al. (2012) (e.g., _find two cafes in Chicago_). Workers may also incorrectly use a singular noun instead of its plural form, and vice versa. For instance, in Example 6 of Table 1, the paraphrase only asks for the status of one specific _burglar alarm_ while the expression asks for the status of all _burglar alarms_. Making mistakes in paraphrasing complementary forms of words also exists in the crowdsourced dataset. For instance, in Example 7 of Table 1, assuming that the bot answers the question only by saying "YES" or "NO", the answer for the paraphrase differs from that of the expression. However, it will make no difference if the bot's response is more descriptive (e.g., "it's working", "it isn't working".) Finally, some paraphrases significantly diverge from expressions. For instance, in Example 8 of Table 1, the intent of paraphrase is to turn off the TV; however, that of initial utterance is to query about the TV status.

Footnote 4: In our task design, this type of error cannot happen since parameter values are checked using regular expressions before submission

### Task Misunderstanding

In some cases, workers misunderstood the task and provided translations in their own native languages (referred to as _Translation_ issues) Crossley et al. (2016); Braunger et al. (2018); Bapat et al. (2018), and some mistakenly thought they should provide answers for expressions phrased as questions (referred to as _Answering_ issues) such as Example 9 in Table 1. This occurred even though workers were provided with comprehensive instructions and examples. We infer that some workers did not read the instructions, ignoring the possibility of cheating.

### Cheating

In crowdsourced tasks, collecting paraphrases is not immune to unqualified workers, cheaters, or spammers Daniel et al. (2018); Crossley et al. (2016); Chklovski (2005).

Detecting malicious behaviour is vital because even constructive feedback may not guarantee quality improvements as workers act carelessly on purpose. _Cheating_ is thus considered a special case of _Semantic Error_ which is done intentionally. It is difficult even for experts to detect if someone is cheating or unintentionally making mistakes. However, it becomes easier when we consider all three paraphrases written by a worker for a given expression at once. For example, in Example 10 of Table 1, the malicious worker removes words one by one to generate new paraphrases. In this example, we also notice that it is still possible that a cheater produces a valid paraphrase accidentally such as the first paraphrase in Example 10. Workers may also start providing 

faulty paraphrases after generating some correct paraphrases as shown in Example 14 of Table 1. Based on our observations, the simplest mode of cheating is to add a few random characters to the source sentence as shown in Example 11. Next is adding a few words to the source sentence without much editing as shown in Example 12. Finally, there are cheaters who rewrite and change the sentences substantially in a very random way such as Example 13.

## 4 Dataset Annotation

Next, we designed another crowdsourcing task to annotate the collected paraphrases according to the category of issues devised above. Namely, using following labels: _Correct_, _Semantic Error_, _Misspelling_, _Linguistic Error_, _Translation_, _Answering_, and _Cheating_. We split the category of misunderstanding issues into _Translation_ and _Answering_ because they require different methods to detect.

**Methodology.** In the annotation task, crowd workers were instructed to label each paraphrase with the paraphrasing issues. Next, to further increase the quality of annotations 5, two authors of this paper manually re-annotated the paraphrases to resolve disagreements between crowd annotators. Moreover, contradictory labels (e.g., a paraphrase cannot be labeled both _Correct_ and _Misspelling_ simultaneously) were checked to ensure consistency. The overall Kappa test showed a high agreement coefficient between the annotators [9] by Kappa being \(0.85\). Table 2 also shows the pair-wise inter-annotor agree

\begin{table}
\begin{tabular}{l l c p{176.7pt}} \hline \hline \# & Label & \multicolumn{2}{c}{Sample} \\ \hline
1 & Correct & **Expression** & _Create a public playlist named new\text 

ment (Cohen, 1960). Next, the authors discussed and revised the re-annotated labels to further increase the quality of annotations by discussing and resolving disagreements.

Statistics.Figure 1 shows the frequencies of each label in the crowdsourced paraphrases as well as their co-occurrences in an UpSet plot (Lex et al., 2014) using Intervene (Khan and Mathelier, 2017). Accordingly we infer that only 61% of paraphrases are labeled _Correct_. This plot also shows how many times two labels co-occurred. For example, all paraphrases which are labeled _Translation_ (24 times), are also labeled _Cheating6_.

Footnote 6: We used _Google Translate_ to check whether they were proper translations or just random sentences in other languages

## 5 Automatic Error Detection

Automatically detecting paraphrasing issues, especially when done during the crowd task, can minimize the cost of crowdsourcing by eliminating malicious workers, reducing the number of erroneous paraphrases, and eliminating the need for launching another crowdsourced validation task. Moreover, by detecting _Misspelling_ and _Linguistic Errors_, users can be provided with proper feedback to help them improve the quality of paraphrasing by showing the source of error and suggestions to address the error (e.g., "_Spelling error detected: articl \(\rightarrow\) article_"). Detecting _Semantic Errors_, such as missing parameter values, can also help crowd workers to generate high quality correct paraphrases. Automated methods can also be used to identify low quality workers, and particularly cheaters who may generate potentially large amount of invalid paraphrases intentionally. Moreover, providing suggestions to cheaters will not help and therefore early detection is of paramount.

In a pre-hoc quality control approach for crowdsourced paraphrases, the most important metric seems to be the _precision_ of detecting invalid paraphrases (Nilforoshan et al., 2017). That is because the main aim of using such a quality control approach is rejecting invalid paraphrases without rejecting correct ones (Burrows et al., 2013). This is essential because rejecting correct paraphrases would be unfair and unproductive. For instance, sincere and trustful crowd workers might not get paid as a result of false-positives (incorrectly detected errors). On the other hand, having a high _recall_ in detecting invalid paraphrases is important to eliminate faulty paraphrases and consequently obtain robust training samples.

Moreover, such a quality control technique should ideally be domain-independent, accessible, and easily-operated to minimize the cost of customization for a special domain and requiring paid experts (e.g., an open source pre-built machine learning model). In the rest of this section, we examine current tools and approaches and discuss their effectiveness in assessing the paraphrasing issues.

### Spelling Errors

We employed several spell checkers as listed in Table 3 to examine if they are effective in recognizing spelling errors. We looked up Wikipedia, Github, and ProgrammableWeb7 to find available tools and APIs for this purpose.

Footnote 7: [https://www.programmableweb.com](https://www.programmableweb.com)

Footnote 8: [http://aspell.net/](http://aspell.net/)

\begin{table}
\begin{tabular}{l l} \hline
**Label** & **Kappa** \\ \hline Correct & 0.900 \\ Missspelling & 0.972 \\ Linguistic Errors & 0.879 \\ Translation & 1.000 \\ Answering & 0.855 \\ Cheating & 0.936 \\ Semantic Errors & 0.833 \\ \hline \end{tabular}
\end{table}
Table 2: Pairwise Inter-Annotator Agreement

\begin{table}
\begin{tabular}{l c c c} \hline
**Spell Checker** & **Precision** & **Recall** & **F1** \\ \hline Aspell8  & 0.249 & 0.618 & 0.354 \\ Hunspell9  & 0.249 & 0.619 & 0.355 \\ MySpell10  & 0.249 & 0.619 & 0.355 \\ Norvig11  & 0.488 & 0.655 & 0.559 \\ Ginger12  & 0.540 & 0.719 & 0.616 \\ Yandex13  & 0.571 & 0.752 & 0.650 \\ Bing Spell Check14  & 0.612 & **0.737** & 0.669 \\ LanguageTool15  & **0.630** & 0.727 & **0.674** \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of Spell Checkers 

Even though detecting misspelled words seems easy with existing automatic spellcheckers, they fall short in a few cases. This can be also concluded from Table 3 by considering the precision and recall of each spell checker in detecting only paraphrases with misspellings. For instance, spell checkers are often unable to identify homonyms Perelman (2016), incorrectly mark proper nouns and unusual words Bernstein et al. (2015), and sometimes do not identify wrong words that are properly spelled Chisholm and Henry (2005). For instance, in Example 1 of Table 1, the "_new_playlist_" is incorrectly detected as a misspelled word by LanguageTool (the best performer as listed in Table 3). In Example 3, the word "_far_" is not detected even though the worker has misspelled the word "_fare_". In Example 4, the word "Newtown" (a suburb in Sydney) is mistakenly detected as a misspelling error. Some of these deficiencies can be addressed. For instance, in the case of spelling errors, assuming that the initial expressions given to the crowd are free of typos, we can ignore false-positives like the "Newton" and "new_playlist".

### Linguistic Errors

We investigated how well grammar checkers perform in detecting linguistic errors. We employed several grammar checkers as listed in Table 4.

Our experiments shows that spell checkers have both low precision and recall. Perelman Perelman (2016) also conducted several experiments with major commercial and non-commercial grammar checkers and identified that grammars are unreliable. Based on our observations, grammar checkers often fail in detecting linguistic errors as shown in Table 4. Examples include improper use of words (e.g., _"Who is the latest scientific article of machine learning?"_), random sequence of words generated by cheaters (e.g., _"Come the next sing"_), and missing articles16 (e.g., _"I'm looking for flight for Tehran to Sydney"_). Given these examples, we believe that language models can be used to measure the likelihood of a sequence of words to detect if it is linguistically acceptable.

Footnote 16: Missing articles in expressions similar to newspaper headlines are not considered error in the dataset (e.g., _“Hotel near Disneyland”_)

### Translation

We also investigated several language detectors to evaluate how well they perform when crowd workers use another language instead of English. The results of experiment in Table 5 indicate that these tools detect almost all sentences in other languages. But they produce lots of false-positives including for correct English sentences (e.g., _"play nest song"_). As a result, the tools in our experi

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Grammar Checker** & **Precision** & **Recall** & **F1** \\ \hline AfterDeadline17  & 0.228 & 0.069 & 0.106 \\ Ginger & 0.322 & **0.256** & **0.285** \\ GrammarBot18  & 0.356 & 0.139 & 0.200 \\ LanguageTool & **0.388** & 0.098 & 0.156 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of Grammar Checkers

Figure 1: Dataset Label Statistics



ment have low precision in detecting languages as shown in Table 5. Most of the false-positives are caused by sentences that contain unusual words such as misspellings and named entities in the sentence (e.g., Email _Phil_ saying "I got you").

One possible approach to improve the precision of such tools and APIs is to check if a given paraphrase has spelling errors prior to using language detection tools. We therefore extended the _DetectLanguage_ (the best performing tool) by adding a constraint: a sentence is not written in another language unless it has at least two spelling errors. This constraint is based on the assumption that spell checkers treat foreign words as spelling errors and a sentence has at least two words to be called a sentence. This approach (_DetectionLanguage+_ in Table 5) significantly reduced the number of false-positives and thus improved precision.

### Answering

Dialog Acts (DAs) Jurafsky and Martin (2018), also known as speech acts, represent general intents of an utterance. DA tagging systems label utterances with a predefined set of utterance types (_Directive, Commissive, Informative, etc Mezza et al. (2018).) Based on the fact that DAs must remain consistent during paraphrasing, we employed a state-of-art, domain-independent, pre-trained DA tagger proposed in Mezza et al. (2018). For example, if an initial utterance is a _question_ (e.g., _are there any cafes nearby?_) it is acceptable to paraphrase it into a _directive_ sentence (e.g., _find cafes nearby._), but its speech act cannot be _informative_ (e.g., there is a cafe on the corner.). Overall, due to the lack of any other domain-independent DA tagger for the English language, we only investigated this tagger. We found that it has a precision of 2% with recall of 63%. This shows that detecting speech acts is a very challenging task especially for domain-independent environments.

Advances in speech act detection and availability of public speech act datasets can assist in detecting this category of the paraphrasing issues. Moreover, it is feasible to automatically generate pairs of questions and answers by mining datasets in the fields of Question Answering and dialog systems. Automatically building such pairs can help building a dataset which is diverse enough to be used in practice. Such a dataset can be fed into deep learning algorithms to yield better performance in detecting _Answering_ issues.

### Semantic Errors & Cheating

To the best our knowledge, there is not yet an approach to distinguish between categories of semantically invalid paraphrases. Paraphrase detection and textual semantic similarity (STS) methods are designed to measure how two pieces of text are semantically similar. However, they do not differentiate between different types of errors (e.g., _Cheating_, _Answering_, _Semantic Errors_) in our settings. As such, these techniques are not directly applicable. In the rest of this section, we focus on building machine learning models to detect the paraphrasing errors.

For this purpose, we used 38 established features from the literature as summarized in Table 6. Using these features and Weka Hall et al. (2009), we built various classifiers to detect the following paraphrasing issues: _Answering_, _Semantic Errors_, and _Cheating_. We chose to test the five classification algorithms applied in paraphrasing literature as mentioned in Burrows et al. (2013): C4.5 Decision Tree, K-Nearest Neighbor (K=50), Maximum Entropy, Naive Bayes, and Support Vector Machines (SVM) using default Weka 3.6.13 parameters for each of the classification algorithms. We also experimented with Random Forest algorithm since it is a widely-used classifier. We did not apply deep learning based classifiers directly due to the lack of expressions in the collected dataset which seems essential for developing domain independent classifiers. While our dataset is reasonably large, it contains only 40 expressions (each having 150 paraphrases). Given that deep learn

\begin{table}
\begin{tabular}{l l r r} \hline
**Language Detector** & **Precision** & **Recall** & **F1** \\ \hline FastText19  & 0.072 & **1.000** & 0.135 \\ LangDetect20  & 0.080 & 0.917 & 0.147 \\ LanguageIdentifier21  & 0.080 & 0.917 & 0.147 \\ IBM Watson22  & 0.170 & 0.958 & 0.289 \\ DetectLanguage23  & 0.344 & 0.917 & 0.500 \\ \hline DetectLanguage+ & **0.909** & **1.000** & **0.952** \\ \hline \end{tabular}
\end{table}
Table 5: Language Detection 

ing techniques are data thirsty (Goodfellow et al., 2016; Yang et al., 2018), to use these kinds of models and eliminate the burden of manual feature engineering, much more expressions are needed. Instead, we benefited from the state-of-art sentence encoders via Transfer Learning as listed in Table 6.

Table 7, 8, and 9 demonstrate the performance of various classifiers (excluding classifiers with F1 being less than \(0.2\)) for each of paraphrasing issues using 10-fold cross validation. To keep the classifiers domain-independent, we split the dataset based on the expressions without sharing any paraphrases of a single expression between the test and train samples. It can be seen that automatically detecting these quality issues is very challenging; even the best performing classifier has a very low F1 score especially for detecting _Answering_ and _Semantic Error_ issues. Based on manual exploration, we also found that the classifiers fail to recognize complex cheating behaviours such as Example 13 in Table 1 as discussed in Section 3. Therefore, new approaches are required to accurately detect paraphrasing issues. Based on our explorations and a prior work (McCarthy et al., 2009), we postulate that accurately detecting linguistic errors such as grammatically incorrect paraphrases can play indispensable role in detecting cheating behaviours. Moreover, advances in measuring semantic similarity between sentences can help differentiate between semantically invalid paraphrases and correct ones.

### Incorrect Paraphrases Detection

We also assessed the performance of detecting incorrect paraphrases regardless of their categories. In this setting, we labeled all incorrect sentences with a single label ("_Incorrect_") regardless of their categories. Table 10 demonstrates the performance of various classifiers. Detecting incorrect paraphrases is useful for post-hoc quality control to remove incorrect paraphrases after crowdsourcing paraphrases and consequently eliminate the need for crowdsourced validation task.

## 6 Related Work

To the best of our knowledge, that our work is the first to categorize paraphrasing issues and propose

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Classifier** & **Precision** & **Recall** \\ \hline Random Forest & **0.947** & 0.129 & 0.226 \\ Maximum Entropy & 0.564 & 0.157 & 0.246 \\ Decision Tree & 0.527 & **0.350** & **0.421** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Summary of Feature Library

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt}} \hline \hline
**Classifier** & **Precision** & **Recall** & **F1** \\ \hline Random Forest & **0.798** & 0.120 & 0.209 \\ Decision Tree & 0.377 & 0.276 & **0.319** \\ Naive Bayes & 0.171 & **0.783** & 0.280 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Automatic _Answering_ Detection

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt}} \hline \hline
**Classifier** & **Precision** & **Recall** & **F1** \\ \hline SVM & **0.878** & 0.223 & 0.356 \\ K-Nearest Neighbor & 0.871 & 0.248 & 0.386 \\ Random Forest & 0.843 & 0.546 & **0.663** \\ Maximum Entropy & 0.756 & 0.440 & 0.557 \\ Decision Tree & 0.632 & **0.566** & 0.597 \\ Naive Bayes & 0.473 & 0.426 & 0.449 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Automatic _Semantic Error_ Detection

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt}} \hline \hline
**Category** & **\# Description** \\ \hline _N-gram Features_ & 12 N-gram overlap, exclusive longest common prefix n-gram overlap, and SUMO all proposed in (Joao et al., 2007), as well as Gaussian, Parabolic, and Trigonometric proposed in (Cordeiro et al., 2007), Paraphrase In N-gram Changes (PINC) (Chen and Dolan, 2011), Bilingual Evaluation Understudy (BLEU) (Papineni et al., 2002), Google’s BLEU (GLEU) (Wu et al., 2016), NIST (Doddington, 2002), Character n-gram F-score (CHRF) (Popovic, 2016), and the length of the longest common subsequence. \\ _Semantic Similarity_ & 15 Semantic Textual Similarity (Fakouri-Kapourchali et al., 2018), Word Mover's Distance (Kusner et al., 2015) between words embeddings of expression and paraphrase, cosine similarity and euclidean distance between vectors of expression and paraphrase generated by Sent2Vec (Pagliardini et al., 2018), InferSent (Conneau et al., 2017), Universal Sentence Encoder (Cer et al., 2018), Concatenated Power Mean Embeddings (Ruckle et al., 2018), tenses of sentences, pronoun used in the paraphrase, and miss-matched named entities \\ _Others_ & 11 Number of spelling and grammatical errors detected by LanguageTool, task completion time, edit distance, normalized edit distance, word-level edit distance (Fakouri-Kapourchali et al., 2018), length difference between expression and paraphrase (in characters and words), and simple functions to detect questions, imperative sentences, and answering.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline
**Classifier** & **Precision** & **Recall** & **F1** \\ \hline Random Forest & **0.798** & 0.120 & 0.209 \\ Decision Tree & 0.377 & 0.276 & **0.319** \\ Naive Bayes & 0.171 & **0.783** & 0.280 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Automatic _Cheating_ Detection 

an annotated dataset for assessing quality issues of paraphrased user expressions. Nevertheless, our work is related to the areas of (i) quality control in crowdsourced natural language datasets; and (ii) semantic similarity.

**Quality Control.** Quality can be assessed after or before data acquisition. While _post-hoc_ methods evaluate quality when all paraphrases are collected, _pre-hoc_ methods can prevent submission of low quality paraphrases during crowdsourcing. The most prevalent _post-hoc_ approach is launching a verification task to evaluate crowdsourced paraphrases Negri et al. (2012); Tschirsich and Hintz (2013). However, automatically removing misspelled paraphrases Wang et al. (2012) and discarding submissions from workers with low/high task completion time Ma et al. (2017) are also applied in literature. Machine learning models have also been explored in plagiarism detection systems to assure quality of crowdsourced paraphrases Crossley et al. (2016); Burrows et al. (2013).

_Pre-hoc_ methods, on the other hand, rely on online approaches to asses the quality of the data provided during crowdsourcing Nilforoshan et al. (2017). Sophisticated techniques are required to avoid generation of erroneous paraphrases (e.g., automatic feedback generation was used to assist crowd workers in generating high quality paraphrases). Precog Nilforoshan et al. (2017) is an example of such tools which is based on a supervised method for generating automatic writing feedback for multi-paragraph text- designed mostly for crowdsourced product reviews Nilforoshan et al. (2017); Nilforoshan and Wu (2018). This paper aims for paving the way for building automatic _pre-hoc_ approaches, and providing appropriate online feedback to users to assist them in generating appropriate paraphrases. However, the provided dataset can also be used for building _post-hoc_ methods to automatically omit faulty paraphrases.

**Semantic Similarity.** Measuring similarity between units of text plays an important role in Natural Language Processing (NLP). Several NLP tasks have been designed to cover various aspects and usages of textual similarity. Examples include textual entailment, semantic textual similarity Yang et al. (2018b); Fakouri-Kapourchali et al. (2018), paraphrase detection Agarwal et al. (2018); Issa et al. (2018), duplicate question detection Mannarswamy and Chidambaram (2018) tasks which are studied well in NLP. Moreover, recent success in sentence encoders (e.g., Sent2Vec Pagliardini et al. (2018), InferSent Conneau et al. (2017), Universal Sentence Encoder Cer et al. (2018), and Concatenated Power Mean Embeddings Ruckle et al. (2018)) can be exploited to detect paraphrasing issues with more accuracy. These techniques can be borrowed with some domain specific considerations to build automatic quality control systems for detecting low quality paraphrases.

## 7 Conclusion

In this paper, we employed a data-driven approach to investigate and quantitatively study various crowdsourced paraphrasing issues. We discussed how automatic techniques for detecting various quality issues can assist the manual process of crowdsourced paraphrasing. We collected an annotated dataset of crowdsourced paraphrasing in which each paraphrase is labeled with associated paraphrasing issues. We used this dataset to assess existing tools and techniques and to determine whether they are sufficient for automatically detecting such issues. Our experiments revealed that automated detection of errors in paraphrases is a challenging task. As a future work, we will be working on devising automated-assisted methods for detection of paraphrasing issues. This will be based on a two-way feedback mechanism: generating feedback for workers, while at the same time the system learns from the (data of) users to improve its machine intelligence. In time, we envision increasingly less dependence on users.

### acknowledgements

This research was supported fully by the Australian Government through the Australian Research Council's Discovery Projects funding scheme (project DP1601104515).

\begin{table}
\begin{tabular}{l c c r} \hline \hline
**Classifier** & **Precision** & **Recall** & **F1** \\ \hline K-Nearest Neighbor & **0.799** & 0.341 & 0.478 \\ Random Forest & 0.781 & **0.551** & **0.646** \\ Maximum Entropy & 0.721 & 0.489 & 0.583 \\ SVM & 0.709 & 0.289 & 0.411 \\ Decision Tree & 0.633 & 0.585 & 0.608 \\ Naive Bayes & 0.574 & 0.557 & 0.565 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Automatic Incorrect Paraphrase Detection 



## References

* Agarwal et al. (2018) Basant Agarwal, Heri Ramampiaro, Helge Langseth, and Massimiliano Ruocco. 2018. A deep network model for paraphrase detection in short text messages. _Information Processing & Management_, 54(6):922-937.
* Bapat et al. (2018) Rucha Bapat, Pavel Kucherbaev, and Alessandro Bozzon. 2018. Effective crowdsourced generation of training data for chatbots natural language understanding. In _Web Engineering_, Cham. Springer International Publishing.
* Bernstein et al. (2015) Michael S. Bernstein, Greg Little, Robert C. Miller, Bjorn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2015. Soylent: A word processor with a crowd inside. volume 58, pages 85-94, New York, NY, USA. ACM.
* Braunger et al. (2018) Patricia Braunger, Wolfgang Maier, Jan Wessling, and Maria Schmidt. 2018. Towards an automatic assessment of crowdsourced data for nlu. In _LREC_.
* Burrows et al. (2013) Steven Burrows, Martin Potthast, and Benno Stein. 2013. Paraphrase acquisition via crowdsourcing and machine learning. volume 4, pages 43:1-43:21, New York, NY, USA. ACM.
* Campagna et al. (2017) Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017. Almond: The architecture of an open, crowdsourced, privacy-preserving, programmable virtual assistant. In _Proceedings of the 26th International Conference on World Wide Web_, WWW '17, pages 341-350, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
* Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. _arXiv preprint arXiv:1803.11175_.
* Volume 1_, HLT '11, pages 190-200, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Chisholm and Henry (2005) Wendy A. Chisholm and Shawn Lawton Henry. 2005. Interdependent components of web accessibility. In _Proceedings of the 2005 International Cross-Discipplinary Workshop on Web Accessibility (WAA)_, W4A '05, pages 31-37, New York, NY, USA. ACM.
* Chklovski (2005) Timothy Chklovski. 2005. Collecting paraphrase corpora from volunteer contributors. In _Proceedings of the 3rd International Conference on Knowledge Capture_, K-CAP '05, pages 115-120, New York, NY, USA. ACM.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 670-680, Copenhagen, Denmark. Association for Computational Linguistics.
* Cordeiro et al. (2007) Joao Cordeiro, Gael Dias, and Pavel Brazdil. 2007. A metric for paraphrase detection. In _Computing in the Global Information Technology, 2007. ICCGI 2007. International Multi-Conference on_, pages 7-7. IEEE.
* Crossley et al. (2016) Scott Crossley, Luc Paquette, Mihai Dascalu, Danielle S. McNamara, and Ryan S. Baker. 2016. Combining click-stream data with nlp tools to better understand mooc completion. In _Proceedings of the Sixth International Conference on Learning Analytics & Knowledge_, LAK '16, pages 6-14, New York, NY, USA. ACM.
* Dale (2016) Robert Dale. 2016. The return of the chatbotts. _Natural Language Engineering_, 22(5):811-817.
* Daniel et al. (2018) Florian Daniel, Pavel Kucherbaev, Cinza Cappiello, Boualam Benatallah, and Mohammad Allabakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. volume 51, pages 7:1-7:40, New York, NY, USA. ACM.
* Doddington (2002) George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In _Proceedings of the Second International Conference on Human Language Technology Research_, HLT '02, pages 138-145, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Fakouri-Kapourchali et al. (2018) Roghayeh Fakouri-Kapourchali, Mohammad-Ali Yaghoub-Zadeh-Fard, and Mehdi Khalili. 2018. Semantic textual similarity as a service. In _Service Research and Innovation_, pages 203-215, Cham. Springer International Publishing.
* Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. _Deep learning_, volume 1. MIT press Cambridge.
* Hall et al. (2009) Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The weka data mining software: An update. volume 11, pages 10-18, New York, NY, USA. ACM.
* Henderson et al. (2018) Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and

Society_, AIES '18, pages 123-129, New York, NY, USA. ACM.
* Volume 8946_, ICAART 2014, pages 119-136, Berlin, Heidelberg. Springer-Verlag.
* Issa et al. (2018) Fuad Issa, Marco Damonte, Shay B Cohen, Xiaohui Yan, and Yi Chang. 2018. Abstract meaning representation for paraphrase detection. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, volume 1, pages 442-452.
* Jiang et al. (2017) Youxuan Jiang, Jonathan K. Kummerfeld, and Walter S. Lasecki. 2017. Understanding task design trade-offs in crowdsourced paraphrase collection. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 103-109. Association for Computational Linguistics.
* Joao et al. (2007) Cordeiro Joao, Dias Gael, and Brazdil Pavel. 2007. New functions for unsupervised asymmetrical paraphrase detection. _Journal of Software_, 2(4):12-23.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431. Association for Computational Linguistics.
* Jurafsky and Martin (2018) D Jurafsky and JH Martin. 2018. Dialog systems and chatbots. _Speech and language processing_.
* Kang et al. (2018) Yiping Kang, Yunqi Zhang, Jonathan K Kummerfeld, Lingjia Tang, and Jason Mars. 2018. Data collection for dialogue system: A startup perspective. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)_, volume 3, pages 33-40.
* Khan and Mathelier (2017) Aziz Khan and Anthony Mathelier. 2017. Intervene: a tool for intersection and visualization of multiple gene or genomic region sets. _BMC bioinformatics_, 18(1):287.
* Kusner et al. (2015) Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In _International Conference on Machine Learning_, pages 957-966.
* Lex et al. (2014) Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. 2014. Upset: visualization of intersecting sets. _IEEE transactions on visualization and computer graphics_, 20(12):1983-1992.
* Li et al. (2016) Guoliang Li, Jiannan Wang, Yudian Zheng, and Michael J Franklin. 2016. Crowdsourced data management: A survey. _IEEE Transactions on Knowledge and Data Engineering_, 28(9):2296-2319.
* Ma et al. (2017) Xiao Ma, Trishala Neeraj, and Mor Naaman. 2017. A computational approach to perceived trustworthiness of airbnb host profiles.
* Mannarswamy and Chidambaram (2018) Sandya Mannarswamy and Saravanan Chidambaram. 2018. Geminio: Finding duplicates in a question haystack. In _Advances in Knowledge Discovery and Data Mining_, pages 104-114, Cham. Springer International Publishing.
* McCarthy et al. (2009) Philip M. McCarthy, Rebekah H. Guess, and Danielle S. McNamara. 2009. The components of paraphrase evaluations. _Behavior Research Methods_, 41(3):682-690.
* McHugh (2012) Mary L McHugh. 2012. Interrater reliability: the kappa statistic. _Biochemia medica_. _Biochemia medica_, 22(3):276-282.
* Mezza et al. (2018) Stefano Mezza, Alessandra Cervone, Evgeny Stepanov, Giuliano Tortoreto, and Giuseppe Riccardi. 2018. Iso-standard domain-independent dialogue act tagging for conversational agents. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3539-3551. Association for Computational Linguistics.
* Mitchell et al. (2014) Margaret Mitchell, Dan Bohus, and Ece Kamar. 2014. Crowdsourcing language generation templates for dialogue systems. _Proceedings of the INLG and SIGDIAL 2014 Joint Session_, pages 172-180.
* Neff and Nagy (2016) Gina Neff and Peter Nagy. 2016. Automation, algorithms, and politics-- talking to bots: symbiotic agency and the case of tay. _International Journal of Communication_, 10:17.
* Negri et al. (2012) Matteo Negri, Yashar Mehdad, Alessandro Marchetti, Danilo Giampiccolo, and Luisa Bentivoglu. 2012. Chinese whispers: Cooperative paraphrase acquisition. In _LREC_, pages 2659-2665.
* Nilfroshan et al. (2017) Hamed Nilfroshan, Jiannan Wang, and Eugene Wu. 2017. Precog: Improving crowdsourced data quality before acquisition. _CoRR_, abs/1704.02384.
* Nilfroshan and Wu (2018) Hamed Nilfroshan and Eugene Wu. 2018. Leveraging quality prediction models for automatic writing feedback. In _Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, June 25-28, 2018._, pages 211-220.
* Conference of the North American Chapter of the Association for Computational Linguistics_
 

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, ACL '02, pages 311-318, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Perelman (2016) Les Perelman. 2016. Grammar checkers do not work. _WLN: A Journal of Writing Center Scholarship_, 40(7-8):11-20.
* Popovic (2016) Maja Popovic. 2016. chrf deconstructed: beta parameters and n-gram weights. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_, pages 499-504. Association for Computational Linguistics.
* Ravichander et al. (2017) Abhilasha Ravichander, Thomas Manzini, Matthias Grabmair, Graham Neubig, Jonathan Francis, and Eric Nyberg. 2017. How would you say it? eliciting lexically diverse dialogue for supervised semantic parsing. In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_, pages 374-383.
* Ruckle et al. (2018) Andreas Ruckle, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. 2018. Concatenated \(p\)-mean word embeddings as universal cross-lingual sentence representations. _CoRR_, abs/1803.01400.
* Su et al. (2017) Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017. Building natural language interfaces to web apis. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_, CIKM '17, New York, NY, USA. ACM.
* Tschirsich and Hintz (2013) Martin Tschirsch and Gerold Hintz. 2013. Leveraging crowdsourcing for paraphrase recognition. In _Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse_, pages 205-213.
* Wang et al. (2012) William Yang Wang, Dan Bohus, Ece Kamar, and Eric Horvitz. 2012. Crowdsourcing the acquisition of natural language corpora: Methods and observations. In _Spoken Language Technology Workshop (SLT), 2012 IEEE_, pages 73-78. IEEE.
* Wasow et al. (2005) Thomas Wasow, Amy Perfors, and David Beaver. 2005. The puzzle of ambiguity. _Morphology and the web of grammar: Essays in memory of Steven G. Laponite_, pages 265-282.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _CoRR_, abs/1609.08144.
* Yang et al. (2018a) Jie Yang, Thomas Drake, Andreas Damianou, and Yoelle Maarek. 2018a. Leveraging crowdsourcing data for deep active learning an application: Learning intents in alexa. In _Proceedings of the 2018 World Wide Web Conference_, WWW '18, pages 23-32, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
* Yang et al. (2018b) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-Yi Kong, Noah Constant, Pett Pilar, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2018b. Learning semantic textual similarity from conversations. In _Proceedings of The Third Workshop on Representation Learning for NLP_, pages 164-174. Association for Computational Linguistics.
* Zamanirad et al. (2017) Shayan Zamanirad, Boualem Benatallah, Moshe Chai Barukh, Fabio Casati, and Carlos Rodriguez. 2017. Programming bots by synthesizing natural language expressions into api invocations. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering_, pages 832-837. IEEE Press.