Making Translators Privacy-aware on the User's Side
Authors: Ryoma Sato
Venue: Accepted by TMLR ()

Abstract: We propose PRISM to enable users of machine translation systems to preserve the privacy of data on their own initiative. There is a growing demand to apply machine translation systems to data that require privacy protection. While several machine translation engines claim to prioritize privacy, the extent and specifics of such protection are largely ambiguous. First, there is often a lack of clarity on how and to what degree the data is protected. Even if service providers believe they have sufficient safeguards in place, sophisticated adversaries might still extract sensitive information. Second, vulnerabilities may exist outside of these protective measures, such as within communication channels, potentially leading to data leakage. As a result, users are hesitant to utilize machine translation engines for data demanding high levels of privacy protection, thereby missing out on their benefits. PRISM resolves this problem. Instead of relying on the translation service to keep data safe, PRISM provides the means to protect data on the user's side. This approach ensures that even machine translation engines with inadequate privacy measures can be used securely. For platforms already equipped with privacy safeguards, PRISM acts as an additional protection layer, reinforcing their security furthermore. PRISM adds these privacy features without significantly compromising translation accuracy. We prove that PRISM enjoys the theoretical guarantee of word-level differential privacy. Our experiments demonstrate the effectiveness of PRISM using real-world translators, T5 and ChatGPT (GPT-3.5-turbo), and the datasets with two languages. PRISM effectively balances privacy protection with translation accuracy over other user-side privacy protection protocols and helps users grasp the content written in a foreign language without leaking the original content.

Published in Transactions on Machine Learning Research (06/2024)
Making Translators Privacy-aware on the User's Side
Ryoma Sato
r.sato@ml i.sato@mmiis..kkyoooooo....
Kyoto University
Okinawa Institute of Science and Technology
Reviewed on OpenReview: https: // opwn://vee....v.....mmm net/ forum? id= AbegDetggtttt tcs
Abstract
We propose PRISM to enable users of machine translation systems to preserve the privacy of data on their own initiative. There is growing demand to apply machine translation systems to data that require privacy protection. While several machine translation engines claim to prioritize privacy, the extent and specifics such protection are largely ambigu- ous. First, there often lack of clarity on how and to what degree the data is protected. Even if service providers believe they have sufficient safeguards in place, sophisticated ad- versaries might still extract sensitive information Second, vulnerabilities may exist outside of these protective measures, such as within communication channels, potentially leading to data leakage. As a result, users are hesitant to utilize machine translation engines for data demanding high levels of privacy protection, thereby missing out on their benefits. PRISM resolves this problem. Instead of relying on the translation service to keep data safe, PRISM provides the means to protect data on the user's side. This approach en- sures that even machine translation engines with inadequate privacy measures can be used securely. For platforms already equipped with privacy safeguards, PRISM acts tional protection layer, reinforcing their security furthermore. PRISM adds these privacy features without significantly compromising translation accuracy. We prove that PRISM enjoys the theoretical guarantee of word-level differential privacy. Our experiments demon- strate the effectiveness of PRISM using real-world translators, T5 and ChatGPT, and the datasets with two languages. PRISM effectively balances privacy protection with translation accuracy other user-side privacy protection protocols and helps users grasp the content written in foreign language without leaking the original content.
addi-
1 Introduction
Machine translation systems now essential in sectors including business and government for translating Their rise in popularity can be attributed to recent advancements in language models 4 4,32 that have significantly improved translation accuracy, enhancing their overall utility. There is growing demand to use these tools for private and sensitive information. For instance, office workers often need to translate e-mails from clients other countries, but they want to keep these e-mails secret. Many are worried about using machine translation because there's a chance the information might get leaked. This means that, even with these helpful tools around, people often end up translating documents by themselves to keep the information safe.
materials such as e-mails and documents 42 43
Although many machine translation platforms claim they value privacy, the details and depth of the pro- tection are not always clear. First, it's often uncertain how and to what level the data is kept safe. The details of the system are often an industrial secret of the service provider, and the source code is rarely disclosed. Even if providers are confident in their security, sophisticated attackers might still access private information. Also, there could be risks outside of these safeguards, including during data transfer, leading to potential leaks. Because of these concerns, users are cautious about using translation tools for sensitive data, missing out their benefits.

---

Published in Transactions on Machine Learning Research (06/2024)
In response to the prevalent concerns regarding data security in machine translation, we present PRISM (PRIvacy Self Management); which empowers users to actively manage and ensure the protection of their data. Instead of placing complete trust in the inherent security protocols of translation platforms, PRISM provides users with mechanisms for personal data safeguarding. This proactive strategy allows users to confidently use even translation engines that may not offer privacy measures. For platforms already equipped with privacy safeguards, PRISM acts as an additional protection layer, reinforcing their security mechanisms. PRISM adds these privacy features without much degradation of translation accuracy.
We propose two variants of PRISM. PRISM-R is simple method with theoretical guarantee of word-level differential privacy. Here, we mean word-level by regarding texts with one different word as adjacent. It should be noted that this definition may be oversimplified for practical guarantee of privacy as words in real-world texts are correlated. Group privacy, where a group of words is considered a unit, mitigates this problem as differential privacy can be composed 12 Theorem 2.2], and our results can be extended group privacy. Exploring more sophisticatod guarantoes that care text correlations is left as futuure work. PRISM* (PRISM-Star) is a more elaborated method that can achieve better translation accuracy than PRISM-R the price of losing the theoretical guarantee. In practice, recommend using PRISM* for most use cases and PRISM-R for cases where the thcorctical guarantee is required.
and the English In the experiments, we use real-world translators, namely T5 [3] and ChatGPT 19 3 + French and English + German translation. We confirm that PRISM can effectively balance privacy protection with translation accuracy over other user-side privacy protection protocols. PRISM helps users grasp the content written in foreign language without leaking the original content.
The contributions of this paper
follows:
We formulate the problem of user-side realization of data privacy for machine translation systems.
We propose PRISM, which enables users to preserve the privacy of data on their own initiative.
We formally show that PRISM can preserve the privacy of data in terms of word-level differential privacy.
We propose an evaluation protocol for user-side privacy protection for machine translation systems.
We confirm that PRISM can effectively balance privacy protection with translation accuracy using the real-world ChatGPT translator.
Reproducibility: Our code and trained dictionaries are available at https: //gpshh//gthub.... com/ joisino/prism.
2 Problem Formulation
We assume that we have access to black-box machine translation system T that takes source text x and outputs a target text In practice, T' can be ChatGPT [T. DeepL [0. or Google Translate [77. We assume that the quality of the translation T(x) is satisfactory, but T may leak information or be unreliable in terms of privacy protection. Therefore, it crucial to avoid feeding sensitive text & directly into T. We have a sensitive source text Spri; and our goal to safely translate Spri- We also assume that we have a dataset of non-sensitive source texts D Znn) Therefore, it is cheap to collect D. In practice, D can be public news texts, and Zpri can be e-mail.
is unlabeled and need not be relevant to Tpri-
When considering user-side realization, the method should be simple enough to be executed on the user's side. For example, it is difficult for users to run a large language model or to train a machine learning model on their own because it requires a lot of computing resources and advanced programming skills. Therefore, we stick to simple and accessible methods.
In summary, our goal is to safely translate Epr using T and D, and the desiderata of the method are summarized as follows:
2

---

Published in Transactions on Machine Learning Research (06/2024)
user's side
input text
Internet
query to the translator
X Xrri
response
output
X pub
T
Y yub
y pri
translator
H
D unsupervised
replacement history
word dictionary
corpus
Figure 1: Overview of PRISM. The blue boxes indicate information kept on the user's side, the purple boxes indicate information exposed to the Internet, and the red region indicates the Internet. The purple boxes should not contain much information about the input text Upri-
Accurate The final output should be a good translation of the input text Tpri-
Secure The information passed to 7 should not contain much information of the input text xpri.
Simple The method should be lightweight enough for end-users use.
3 Proposed Method (PRISM)
3.1
Overview
PRISM has four steps as shown Figure 1 (i) PRISM creates a word translation dictionary using and
D. This step should be done only once, and the dictionary can be used for other texts and users. (ii) PRISM converts the source text pri to non-sensitive text ppub- (ifi) PRISM translates Tpub to Ypub using T. (iv) PRISM converts Ypuh to Vpri using the replacement history H. We explain each step in detail in the following,
Let us first illustrate the behavior of PRISM with an example. let Tpri be "Alice is heading to the hideout." and be a machine translation system from English to French. PRISM converts Tpri to Tpub = "Bob is heading the store," " which not sensitive and can be translated with T. PRISM temporarily stores the substitutions (Alice - Bob) and (base - restaurant). Note that this substitution information kept on the user's side and not passed T. Then, PRISM translates Tpub to Mpub "Bob dirige vers boutique." using the translator T. Finally, PRISM converts Ypub to Ypri "Alice dirige la cachette." using the word translation dictionary, Alice (Fn) + Alice (Fr), Bob (Fn) - Bob (Fr), store (Fn) - bout (Fr), and hideout (En) + cachette (Fr). The final output Ypri is the translation of Tpri, and PRISM did not pass the information that Alice is heading to the hideout to T.
3.2 Word Translation Dictionary
We assume that a user does not have a word translation dictionary for the target language. We propose to create a word translation dictionary using the unsupervised text dataset D. The desideratum that the dictionary should be robust. Some words have multiple meanings, and we want avoid incorrect substitutions in PRISM. Let V be the vocabulary of the source language. Let. S be random variable that takes random sentence from D, and let S., be the result of replacing random word in S with w E V. We translate S to the target language and obtain R and translate S., to obtain Ru. Let
del Pr(v E Rwd Pr(u R
Pw.r
(1)
3

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1: PRISM-R
Input:
Source text pris Word translation dictionary L; Ratio r E (0,
Output: Translated text ypri-
fn....,4 < Tokenize(Tpri)
// Tokenize Ppri // The history of substitutions
H +
for i x- to do p~ Unif(0,1) 1) if p r then
U; + random source word in L
// Choose a substitution word // Update the history
7
H. - HU ( x.)
8
4 + U4
pub Detokenize(t,,....
// Detokenize tu... tn // Translate Puu
10 Upub + pTpub)
11 Upei + Ypub
// Copy y/pub
12 for (we, u) E H do
13
for V E L(u) do
14
if E Yprs then
Upii t replace with L(w, 1) in Upri break
16 return Uprr
be the ratio of the probability of appearing in Ry to the probability of appearing in R. The higher Pare is, the more likely is the correct translation of since appears in the translation if and only if w appears in the source sentence. Note that if we used only the numerator, article words such as "la" and "le" would have high scores, and therefore we use the ratio instead. Let L(w) be the list of words in the decreasing order of pw,v- L(w,1 1) is the most likely translation of w, and L(w, 2) is the second most likely translation of w, and so on.
It should be noted that the translation engine used here not necessarily the same as the one T we use in the test phase. As we need to translate many texts here, note that once create the word translation dictionary, distribute the word translation dictionaries for English + French and English - German, and users skip this step if they use these dictionaries.
cheaper translation engine. We also use it for other texts and users. We will
3.3 PRISM-R
PRISM-R is a simple method to protect data privacy on the user's side. PRISM-R randomly selects words wi...., wa in the source text Zpri and randomly selects substitution words u1.... ug from the word trans- lation dictionary, puub is the result of replacing wi with uI, ..., and wa with ux- PRISM-R then translates Epub to Ppub using T. Finally, PRISM-R converts Ypub to Ppri as follows. Possible translation words of are L(us). PRISM-R first searches for I(us, 1), the most likely translation of u,, in Upub- There should be translated u; to L(u,, 1). If L(u,, 1) is found, PRISM-R replaces L(u,, 1) with L(w, 1).
L(u,, 1) in Mpub However, if u; has many translation candidates, may not have translated u, L(u, 1). If L(u, 1) is not found, it proceeds to L(u,, 2), the second most likely translation of u, and replaces L(u,, 2) with L(w, 1), and on.
The pseudo code is shown in Algorithm
3.4 Differential Privacy of PRISM-R
Differential privacy II provides a formal guarantee of data privacy. We show that PRISM-R satisfies
differential privacy. This result not only provides privacy guarantee but also shows PRISM-R can be
4

---

Published in Transactions on Machine Learning Research (06/2024)
bined with other mechanisms due to the inherent composability and post-processing resilience of differential privacy [5
We first define differential privacy. We say texts x = Wi..... Wn and x' = w',..., w/ are neighbors if wy = w/ for all i E f1..... n except for one E (I....,n). Let x ~ x' denote that x and x' are neighbors. Let A be a randomized mechanism that takes a text and outputs a text y. Differential privacy defined as follows. Definition 3.1 (Differential Privacy). A satisfies e-differential privacy if for all x ~ x' and S S II((A)
Pr[A(z) S] Pr(A(r') Se.
(2)
We show that the encoder of PRISM-R APRISM-R: Zpri + pub is differential private, and therefore, Zprr cannot be inferred from Tpub which is the only information that T can access.
Theorem 3.2. APRISM-R is In (++))())) ddiferrnntall private.
Here we consider two databases as adjacent if the two texts differ in one word. As the words in the text are correlated, this may seem weak guarantee. However, even if multiple words are changed, we can reach one state from the other state by hopping the adjacency relations [ Theorem 2.2]. Thus, the same guarantee of differential privacy can be derived immediately where two texts are regarded adjacent if entire block in the text is changed or if a constant number of words are changed.
We emphasize that the additive constant is zero, i.e. PRISM-R is (e, 0)-differential private, which provides a strong guarantee data privacy.
Proof. Let x = W1,.... Wn and x' = wi,..., wh any two neighboring texts. Without loss of generality, we assume that wi = w/, ...; Wn-1 w/-1 and wn o w/. Tet .1,...,, 8n be any text, and let
def
L|s + wi]
(3)
be the number of different words in x and s. The probability that PRISM-R convert x s is
n
)
c) r' r))--))--
Pr[APRISM.n())= =
(4)
where i is the number of replaced words because all of the different words must be replaced, and the number of ways to choose the remaining words is (+=e). This probability can be simplified as follows:
n
P+[APRISMA(()) = s|
7'(1 - n
(5)
i
M
i=c
1 itc
n
p 1 (1 - ---
(6)
i
D
1 ch-
n - c) ( (1ai1 -
r
= (1 (1- - r)
(7)
[
i
1=U
=1 r)m
1
1
r
n-c
(8)
[V[11 r)
where we used the binomial theorem in the last equality. Similarly, let
del
1 [s: + will
(9)
i=1
c
if 1 1[n + wn] 1 [sn + w//] = 1
c+1 1
if 1 [[s # wn] 0 and 1 [sn # w/]] = 1.
(10)
C 1 if 1 [[8n # wn] = and 1 [s,, # w/ll = 0
5

---

Published in Transactions on Machine Learning Research (06/2024)
be the number of different words in x' and s. Then,
1 e' (1+ [
Pr[APRiSM.M(2)) =) - (1- --)
(1I)
DI(1 -r)
Combining Eqs.
and 11
1 [ + wol = 1 [sn + ww/]]
Pr Pr|APRISM. ((x) =
(1))
if 1 [&n w. 0 and 1 [ # w/
(12)
Pr[APRISM-R(4") s
if [s # wnl = land 1 [s # w/y1 = 0
(1--)
(1 r)
(13)
r
An interesting part PRISM is that PRISM is resilient against the purturbation due to the final substitution step. Many of diffcrential private algorithms add purturbation to the data [ 3 their final output becomes unreliable when the privacy constraint is severe. By contrast, PRISM enjoys both of the privacy guarantee and the reliability of the final output thanks to the purturbation step and the recovery step. The information Ppub passed to T has little information due ot the purturbation step. This, however, makes the intermediate result Ypub an unreliable translation of ppri PRISM recovers a good translation Ypri by the final substitution step.
14 4 and therefore,
3.5 PRISM*
PRISM is a more sophisticated method and achieves better accuracy than PRISM-R. PRISM chooses words Wi...., wh in the source text Tpri and substitution words uI...., Uk from the word translation dictionary more carefully while PRISM-R chooses them randomly to achieve differential privacy. PRISM* has two mechanisms to choose words. The first. mechanism is to choose words that the part-of-speech tags match. The second mechanism is to choose words that can be translated accurately by the word dictionary. We explain each mechanism in detail in the following.
PRISM* creates a word translation dictionary with a part of speech tag. The procedure the same as Section 3.2 except that we use the part-of-specch tag of the souree word as the key the dictionary. Let (w,s) be pair of a source word w and its part-of-speech tag 8. PRISM replaces a random word with part-of-speech tag s with to create Sw,s, obtains Rw, by translating Sw,s, and defines
def Pr(c E Rug, Pr(c E R
Pwrs,v
(14)
L(w, s) is the list of words in the decreasing order of Pw.s,v-
In the test time, PRISM* chooses substitute words so that the part-of-speech tags match, and use the word translation dictionary with part-of-speech tags to determine the translated word.
PRISM* also uses the confidence score
c(w,s) 5) def max Pw.8,01
(15)
which indicates the reliability of word translation should not be substituted in PRISM because a word-to-word translation may fail. PRISM* chooses words to be substiluted in the decreasing order of the confidence score, which results in selecting single-meaning and reliable words that can be translated accurately by the word dictionary. If a word (w, has two possible translations in and 122 that are equally likely, Pr[u E R lower than 0.5 for any v, even for 21 and U2, the confidence score o(w, s) tends be low, and PRISM* avoids selecting such (w, s). The selected words to be reliably translated by L[w, 8, 1). PRISM also chooses substitute words with high confidence
L(w, 1), to choose words. Multiple-meaning words

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 2: PRISM*
Input:
Source text prii Word translation dictionary I, Confidence Scores Ratio r E (0, 1).
Output: Translated text Spei-
(1..... in - Tokenize(Ipr)
// Tokenize "pri
81,..,,8 +- Part-ofSpeech((1..... (n)
++0
// The number of substitutions The history of substitutions
#+0
for i [I,....n) in the decreasing order of c(4,, 81) do U, f- the unused source word with the highest confidence score c(u,, 8:) in L substitution word
// Choose
H + H (ts, u4 8;))
// Update the history
4 f u;
++1
10
if 2 rn then
L break
11
12 puub f- Detokenizo(ta.....
// Detokenize ti,..., tn // Translate Tuub // Copy Ypub
18 Ypub +- (Tppbb)
14 Upri t Ypub
for (w, u) H do
16
for E L(u, s) do
17
if v E Upri then
L Epn + replace with L(w, 8, 1) in Dpri break
18
19 return Upri
scores so that PRISM can robustly find the corresponding word L(w, 8, 1) in the translated text Ypub in the final substitution slep.
The pseudo code of PRISM* shown in Algorithm
Note that PRISM* does not enjoy the differential privacy guarantec of PRISM-R as (i) PRISM* replaces words with the same part-of-speech tag so that two texts with different part-of-speech templates have zero probability of transition, and (ii) PRISM* chooses words with high confidence scores so that the probability of transition is biased. Nevertheless, PRISM* empirically strikes a better tradc-off between privacy and accuracy than PRISM-R as we will show the experiments. Note that PRISM can be combined with PRISM-R to guarantce differential privacy. For example, one can apply PRISM-R and PRISM* in a nested manner, which guarantees differential privacy due the differential privacy of PRISM-R (Theorem 3.2 and the post processing resilience of differential privacy 25 One can also apply PRISM-R with probability (1 B) and PRISM with probability B, which also guarantees differential privacy because the minimum probability of transition is bounded from below due to the PRISM-R component.
4 Experiments
We confirm the effectiveness of our proposed methods through experiments.
4.1 Evaluation Protocol
As our problem setting is novel, we first propose an evaluation protocol for the user-side realization of privacy-aware machine translation systems. We evaluate the translation accuracy and privacy protection as follows.
Let X' (21, T2, ... xN) be set of test documents be translated. Our aim to read in the target language without leaking information of X.

---

Published in Transactions on Machine Learning Research (06/2024)
auxiliary infomation (option)
user's side
input text
algorithm
H
Internet
X pri
A
translator
response
output
Xpub
T
y ypu
reference document
Todd bay Rocksville.
pri
query the translator
question
question
q
reference document
evaluator
Todd made to the rock?
How long was before
evaluator
(a) (c) weeks Two
PPS
a
QS
answer
ground truth
quality score
answer
preservation score
Figure 2: Overview of the evaluation protocol.
For evaluation purposes, we introduce question-answering (QA) dataset Q {(aj. Osy)), where 4ú and as are a multiple-choice question and answer regarding the document Hú, respectively. Q shown only to the evaluator, and not to the translation algorithm.
Privacy-presorving Score. The idca of our privacy score is based on an adversarial evaluation where adversaries try to extract information from the query sent by the user. Let xp be the query sent to the translator T. An evaluator is given zbuub and fij, and asked to answer the question. The privacy- preserving score of the translation algorithm is defined as PPS = (1 acc)), where acc is the accuracy of the evaluator. The higher the servacy-preseringgg score is, the better the privacy protection is. Intuitively, if the accuracy is low, the evaluator cannot draw any information from x:! to answer the question. Conversely, if the accuracy is high, the evaluator can infer the answer solely from æ!!"h, which means that xx"hh leaks information. We note that the translation algorithm does not know the question Qij, and therefore, the translation algorithm ncods to protect all information to achiove a high privacy-preserving score so that any answer on the document cannot be drawn from pub The rationale behind this score is that we cannot predict what form information leaks will take in advance. Even spub does not look like Tpri at glance, sophisticated adversaries might extract information that can be used to infer Zpri- Therefore, we employ an outside evaluator and adopt an adversarial evaluation.
Quality Score. Let vi"h be the final output of the translation algorithm. We use the same QA dataset and ask evaluator to answer the question 9y using y The quality score of the translation algorithm is defined as QS acc, where acc is the accuracy of the evaluator. The higher the quality score is, the better the translation quality is. Intuitively, if the accuracy is high, the evaluator can answer the question correctly using y. which means that yf contains sufficient information on I;. We note again that the translation algorithm does not know the question 4j, and therefore, the translation algorithm needs to preserve all information to achieve a high quality score so that any answer on the document can be drawn from y,
The protocol is illustrated in Figure
We introduce the araa-umdet-rrivacy-uulii curve (AUPQC) to measure the offectiveness of methods. The privacy-preserving score and the quality score in a trade-off relationship. Most. methods, including PRISM-R and PRISM* have parameter to control the trade-off. An effective method should have high privacy-preserving score and high quality score at the same time. We use the AUPQC to measure the trade-off. Specifically, we scan the trade-off parameter and plot the crivacy-preservinn score and the quality score in the two-dimensional space. The AUPQC is the area under the curve. The larger the AUPQC is, the better the method is. The pseudo code shown in Algorithm
We also introduce QS@p, a metric indicating the quality score at specific higher QSāp is, the better the method is. In realistic scenarios, may have severe security budget p
score. The
8

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 3: AUPQC
P <- the set of the trade-off parameters in the increasing order of the 2-0
score.
// The area under the curve
(PPS (revv) f- None)
for <- --P do
(PPS, QS) Evaluate(a)
// Evaluate the privacy-preserving score and the quality
score if PPSpree is None then
7
$ + $ PPS x QS
// Add the area of the first rectangle
else
9
$ + s + (PPS - PPSprev) x (QS prev + QS)/2
// Add the area of the trapezoid
10
(PPS prev, QS pzev) t- (PPS, QS)
return
which represents the threshold of information leakage tolerate. QS@p is particularly useful under such constraints as it provides direct measure of the quality enjoy under the security budget. It is noteworthy that the privacy-preserving score can be evaluated before we send information to the translator T. Therefore, we can tune the trade-off parameter and ensure that we enjoy the privacy-preservie score and the quality score = QS@p.
4.2 Experimental Setups
We use the MCTest dataset for the documents Is, question Qijs and answer ag. Each document in the MCTest dataset is short story with four questions and answers. The reason behind this choice is that the documents of the MCTest dataset were original ones created by crowdworkers. This is in contrast other reading comprehension datascts such NarrativeQA 20 and CBT [& datasets, which based on existing books and stories, where the evaluator can infer the answers without relying the input document p.ub.
We use T5 83 and GPT-3.5-turbo 31 as the translation algorithm T. We use the prompt "Directly translate English to [Language]: [Source Text]" use GPP-335-turbo for translation.
We also use GPT--35-ttubo as the evaluator. Specifically, the prompt is composed of four parts. The first part of the prompt is the instruction "Read the following message and solve the following four questions." The second part is the document to be evaluated, which is the query document r:P"h for the privacy-preserving score and the final output yf" for the quality score. The third part is the four questions. The last part is the instruction "Output only four characters representing the answers, e.g., \nl. A), B)n3, A4n4. D." We parse the output of GPT-3.5-turbo to extract the answers and evaluate the accuracy.
We use the following four methods.
Privacy- and Utility-Preserrvie Textual Analysis (PUP) LEI is a differential private algorithm convert a document a non-scnsitive document without changing the meaning of z. PUP has a trade-off parameter À for privacy and utility. We convert the source text Tpri to Tpub using PUP and translate Tpub to obtain the final output ypá.
NoDecode translates the encoded text Ppub of *RISM* to obtain the final output Vpai NoDecode does not decode the output of the Iranslator T. This method has the same privacy-preserve property as PRISM but the accuracy should be lower. The improvements from NoDecode are the contribution of our framework.
PRISM-R our method proposod in Scction
PRISM is method proposed in Section 3.5
9

---

Published in Transactions on Machine Learning Research (06/2024)
Accurate
Acturate
Accurate
Accurate
PRISM
02
PRISM *
PRISM
PRISM
04
PRISM
8
07
PRISMIR
07
PRISM-R
PRISM
04
RIMM
115H"
04
"REAP"
FIEA FIGM F.
PM NoDacode
05-
FROM
AUT
TRSM
04
61
NcOecoce
0.2
0.3
04
05
Secure
NcDacoda
02
014
04
00
Secure
61
02
04
0.5
Secure
NcOweoce
0.5
04
Uo
Secure
(8) En
(T5)
(b) En De (T5)
(c) En P....--.-tubbr
(d) En De (GPT-3.5-turbo)
Figure 3: Trade-off between the privacy-preserving score and the quality score. The x-axis is the privacy- preserving score and the y-axis is the quality score.
Table 1: Quantitative Results. The best results are shown in bold, and the second best results are shown in underline.
En - Fr (T5)
En - De (T5)
En + Fr (ChatGPT)
En + De (ChatGPT) AUPQC1 t QS@0.5 f
AUPQC1
QS&0.5 f
AUPQC f QS80.5
AUPQC f QS@0.5 f
NoDecode
0.355
0.493
0.373
0.524
0.376
0.495
0.370
0,480
PUP
0.363
0.439
0.363
0.505
0.415
0.487
0.391
0.511
PRISM-R
0.431
0.613
0.396
0.557
0.399
0.611
0.432
0.629
PRISM*
0.454
0.803
0.473
0.789
0.482
0.799
0.445
0.769
We change the ratio of NoDecode, PRISM-R, and PRISM* and the parameter À of PUP to control the trade-off between sevaccypreserving score and the quality score.
4.3
Results
Figure 3 shows the trade-off, where the x-axis is the score. PRISM* clearly strikes the best tradc-off, and the results of PRISM-R are also better than those of NoDecode and PUP, especially when the privacy-preserving score is high.
score and the y-axis is the quality
The maximum privacy-presee prreeviires score is around 0.5 for all methods, even though there are four choices in each question. Intuitively, the accuracy of the evaluator should be 0.25 when the reference document is random, so the maximum PPR should be 0.75. We found that this is because some questions can be inferred solely from the question text. For example, there is a question "How did the girl hurt her knee? (a) she was in the street (b) she had no friends (c) she fell down, and (d) the old lady's bike hit her." We can infer the answer is (c) or (d) as (a) and (b) do not make sense (the answer is (c)). To verify this hypothesis, we had GPT-3.5-turbo answer the questions using only the question text. The accuracy was 0.492. Therefore, PPS s 0.5 indicates that the query has no more information than the empty text. This experiment also shows that the GPT-3.5-turbo evaluator is so powerful that it can infer the answer from the question text only, and it is an effective adversarial evaluator.
Table shows the quantitative results. We report QS@0.5, i.c., the quality score when PPS 0.5, which roughly means the quality we can enjoy when no information is leaked based on the above analysis. PRISM* consistently achieves the best scores across all the metrics and settings, and PRISM-R achieves the second- best results in most of the metries and settings. Notably, PRISM achieves QS = 0.8 when no information is leaked. This result shows that PRISM* can accurately translate the texts while protecting the privacy of the texts.
Table shows sample translations of PRISM* The leaked information Fpub does not make sense and reveals little about the secret text ppri Although it contains some grammatical errors, the output yptt is generally correct translation of the input text Fpri, which is useful for native speakers to grasp the content.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: Examples of PRISM* translations. The query Tpub to the translator does not make sense and provides little information on the secret text 2pri- Although it contains some grammatical errors, the output Ypri of PRISM* is generally a correct translation of the input text xpri- As our goal is not to provide perfect translations, but to enable native speakers to grasp the content written in foreign language, the quality is sufficient for the goal.
ppri
Tpub
3/rri
Jimmy didn't eat breakfast. Be- cause he didn't eat breakfast he was very hungry and could not stop thinking about food. He was thinking about all of the things that he liked to eat the most. He was thinking about breakfast foods like bacon and eggs but he was also thinking about lunch and dinner foods like pizza and macaroni and cheese.
Logan didn't eat hamster. Be- cause he didn't eat circus he was very fishing and could not stop thinking about smile. He was thinking about all of the things that he screamed to eat the most. He was thinking about zoo foods like bacon and grandparents but he was also thinking about guitar and hamburger foods like pizza and lettuce and jungle.
Jimmy n'a pas mangé de déje- uner. Parce qu'il n'a a pas mangé de déjeuner, il était très faim et ne pouvait pas arrêter de penser au nourriture. Il pensait à toutes les choses qu'il aimé de vouloir manger le plus. aliments du déjeuner comme le bacon et les ufs, mais il pensait aussi à des aliments de déjeuner et de diner comme la pizza et la macaroni et la fromage.
pensait aux
A boy was trying to pick out what instrument that he wanted to play. His parents wanted him to pick a good one because play- ing an instrument was very im- portant to them. So, the boy went music store with his parents.
A dragon was trying to pick out what zoo that he wanted to play. His grandchildren wanted herself to pick a good one because play- ing an Hey was very important to them. Shelly, the bacon went to a mud store with his ants.
Un garçon essayait de choisir quelle instrument il travaillé jouer. Parents parents voulaient eux-mêmes en choisir un bon car joucr avec un Instrument était très important pour eux. So, le garçon est allé dans un magasin de musique avec parents fourmis.
5 Related Work
Privacy Protection of Texts. There is a growing demand for privacy protection mcasures for text data and many methods have been proposed. The U.S. Health Insurance Portability and Accountability Act (HIPAA), which requires that the personal information of patients should be protected, is one of the triggers of heightening concerns on privacy protection of data 21 30. One of the challenges to following HIPAA is to protect information hidden in medical records written in free texts 26 The rulc-based method proposed by ullah et al. 20 is one of the early attempls to delete sensitive information from free texts.
et al. [23] claimed that hiding only the sensitive information is not enough protect privacy because side information may also leak information and proposed a robust method. Many other methods [3] 24 27
aim 30 cannot be inferred. Some The most relevant work to ours is the work by Feyisetan et al. [14, which aims at protecting the privacy of texts while preserving the utility of the texts. Their proposed method is simple enough to implement on the user's side. However, their definition of privacy is different from ours. They aim at protecting the privacy of the author of the text, while we aim at protecting the content. Their method Icaks much information on the content of the text. We confirmed this in the experiments. Many of the other methods also aim al protecting the author of the text and keeping the content of the text intact even after the anonymization
at anonymizing texts that the authors or the attributions of the authors
methods ensure the rigorous privacy guarantee of differential privacy 7
Homomorphic encryption. Homomorphic encryption 15 without decrypling them. The service provider can carry out the computation without knowing the content of the data with this technology [2] G. However, users cannot enjoy the benefit of secure computing unless the service provider implements the technology. Homomorphic encryption is notoriously slow [X] and can degrade the performance, and therefore, the service provider may be reluctant to implement it. To the best of our knowledge, commercial translators use homomorphic encryption. PRISM does not require the
16
22
enables to compute on encrypted data
11

---

Published in Transactions on Machine Learning Research (06/2024)
service provider to implement it. Rather, PRISM applies homomorphic-like (but much lighter) encryption on the user's side. PRISM can be seen as a combination of client-side encryption, which has been adopted and homomorphic encryption.
in cloud storage services 9 25
User-side Realization. Users are dissatisfied with services. Since the service is not tailor-made for a user, is natural for dissatisfaction to arise. However, even if users are dissatisfied, they often do not have the means to resolve their dissatisfaction. The user cannot alter the source code of the service, nor can they force the service to change. In this case, the user has no choice but to remain dissatisfied or quit the service. User-side realization provides solution to this problem. User-side realization [5 36 provides a general algorithm to deal with common problems on the user's side. Many user-side algorithms for various problems have been proposed. Consul BS turns unfair recommender systems into fair ones on the user's side. Tiara [3 realizes a customed search engine the results of which are tailored to the user's preference on the user's side. WebShop A0] enables automated shopping in ordinary e-commerce sites on the user's side by using an agent driven by large language model. WebArena 47 is a general environment to test agents realizing rich functionalities on the user's side. EasyMark
[ realizes large language models with text watermarks the user's side. Overall, there are many works user-side realization, but most of them are on recommender systems and search engines. Our work is the first to protect the privacy of texts on the user's ' side.
6 Conclusion
We proposed a novel problem setting of user-side privacy protection for machine translation systems. We proposed two methods, PRISM-R and PRISM* to turn external machine translation systems into privacy- preserving ones the user's side. We showed that PRISM-R is differential private and PRISM* striked a better trade-off between privacy and accuracy. We also proposed evaluation protocol for user-side privacy protection for machine translation systems, which is valuable for facilitating future research in this area.
Future Work. There are several directions for future work. First, the main goal of this paper is to help users grasp the content writlen in a foreign language without leaking the original content. Although the accuracy is better than those of other privacy protection protocols on the user's side, the translation is far from perfect and there are gramatical errors. Combining the proposed approach with stronger local language models to improve the fluency is attractive direction for future work. Alternatively, there are many orthogonal approaches to improve the trade-off between privacy and accuracy. For example, Reviewer eSaj suggested to insert some extra (fake) queries to the translator to confuse the attacker. Such approaches can be combined with our approach to further improve the effectiveness.
References
[1] M. Abadi, A. Chu, 1. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Securily, pages 308-33 308-318. ACM, 2016.
[2] A. Acar, H. Aksu, A. S. Uluagac, and M. Conti. A survey on homomorphic encryption schemes: Theory and implementation. ACM Comput. Sure., 51 (447:79:11779335 2018
[3] G. Andrew, O. Thakkar, B. McMahan, and S. Ramaswany. Differentially private learning with adaptive clipping. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS, pages 17455-17466, 2021.
[4] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and trans- late. In Y. Bengio and Y. LeCun, editors, 3rd International Conference Learning Representations, ICLR, 2015.
[5] D. L. Baumer, J. B. Earp, and F. C. Payton. Privacy of medical records: IT implications of HIPAA. SIGCAS Comput. Soc., 3044):40-47, 2000.
66] M. Blatt, A. Gusev, Y. Polyakov, and S. Goldwasser. Sccure large-scale genomo-wide association studies using homomorphic encryption. Proceedings of the National Academy of Sciences, 117(21): 2020. doi: 10.1073/ 09003///a..111155711 1918251117.
12

---

Published in Transactions on Machine Learning Research (06/2024)
[7] H. Bo, S. H. H. Ding, B. C. M. Fung, and F. Iqbal. ER-AE: differentially private text generation for authorship anonymization. Tn Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILLI, pages 3997-4007. Association for Computational Linguistics, 2021.
[8] E. Brynjolfsson, X. Hui, and M. Liu. Does machine translation affect international trade? evidence from a large digital platform. Manag. Sci., 65(12):5449 -441))))44---60, 2019.
[9] Cryptomator. Cryptomator. httpss://cryptonatonn .rgg Accessed: 10-2-11-28.
[10] DeepL. Deepl. https::/deeel..om
Accessed: 2023-12-07.
[11] C. Dwork. Differential privacy. In M. Bugliesi, B. Prencel, V. Sassone, and I. Wegener, editors, Au- tomala, Languages and Programmming, 33rd International Colloguium, ICALP, volume 4052 of Lecture
Notes in Computer Science, pages 1-12. Springer, 2006.
[12] C. Dwork and A. Roth. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4)2211-40) 2014.
[13] O. Feyisetan, T. Diethe, and T. Drake. Leveraging hierarchical representations for preserving privacy and utility in text. In 2019 IEEE International Conference on Data Mining, ICDM, pages 210-219. IEFE, 2019.
[14] O. Feyisetan, B. Balle, T. Drake, and T. Diethe. Privacy- and utility-preserving textual analysis via calibrated multivariate perturbations. In The Thirteenth ACM International Conference on Web Search and Data Mining, WSDM, pages 178-186. ACM, 2020.
[15] C. Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the 41st Annual ACM Symposium on Theory of Compuling, STOC, pages 178. ACM, 2009.
[16] R. Bachrach, N. Dowlin, K. Laine, K. E. Lauter, M. Nachrig, and J. Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the 33nd International Conference on Machine Learning, ICML, volume 48, pages 201-210, 2016.
[17] Google. Google translate. https: ptttps//ttrassltee google. Accessed: 2023-12-07.
[18] F. Hill, A. Bordes, S. Chopra, and J. Weston. The goldilocks principle: Reading children's books with explicit memory reprosentations. In 4th International Conference on Learning Representations, ICLR, 2016.
[19] W. Jiao, W. Wang, J. Huang, X. Wang, and Z. Tu. Is chatgpt A good translator? A preliminary study, arXiv, abs/ hts//301.087455 2023. URL https://arxiv. org/
08745
[20] T. Kociský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativega reading comprehension challenge. Trans. Assoc. Comput. Lònguistics, 6:317-328, 2018.
[21] J. Lane and C. Schur. Balancing access to health data and privacy: a review of the issues and approaches for the future. Health services research, 45(5p2))144561144 1467, 2010.
[22] S. Laur, H. Lipmaa, and T. Mielikäinen. Cryptographically private support vector machines, In Proceed- ings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD, pages 618 624. ACM, 2006.
[23] X. Li and J. Qin. Anonymizing and sharing medical text records. Inf. Syst. Res., 28(2): 352, 2017.
[24] P. Lison, I. Pilán, D. Sánchez, M. Batet, and L. Ovrelid. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP, pages 4188-4203. Association for Computational Linguistics, 2021.
[25] F. D. McSherry. Privacy integrated queries: an extensible platform for In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, SIGMOD, pages 19 30, 2009.
data analysis.
[26] S. M. Moystre, F. J. Friedlin, B. R. South, S. Shen, and M. H. Samore. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC medicol research methodology, 10(1):1-16 2010.
13

---

Published in Transactions on Machine Learning Research (06/2024)
[27] A. Mosallanezhad, G. Beigi, and H. Liu. Deep reinforcement Cearning-based text anonymization against private-attribute inference. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNEL pages 2360-2369. Association for Computational Linguistics, 2019.
[28] M. Nachrig, K. E. Lauter, and V. Valkuntanathan. Can homomorphic encryption be practical? In Proceedings of the 3rd ACM Cloud Computing Security Workshop, CCSW, pages 113-124. ACM, 2011.
[29]
Neamatullah, M. M. Douglass, L. H. Lehman, A. T. Reisner, M. Villarrocl, W. Long, P. Szolovits, G. B. Moody, R. G. Mark, and G. D. Clifford. Automated de-identification of free-text medical records. BMC Medical Informatics Decis, Mak., 8:32, 2008.
[30] R. B. Ness, J. P. Committee, et al. Influence of the HIPAA privacy rule on health research. Journal of the American Medical Association, 298(18):2164-217) 2007.
311
OpenAL. Introducing chatgpt. Inttps://openai. wwt. wwwtggttccatt///////// 2023. Accessed: 2023-11-28.
[32] OpenAl. GPT-4 technical report. arXiv, abs/2303.08874, 2023. URL http:////arxxiv.ggaabb/////
08774]
[33] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1 140:67, 2020.
[34] M. Richardson, C. J. C. Burges, and E. Renshaw. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 193 203. ACL, 2013.
[35] R. Sato. CLEAR: A fully user-side image search system. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM, pages 4970-4974. ACM, 2022.
[36] R. Sato. Private recommender systems: How can users build their own fair recommender systems without log data? In Proceedings of the 2022 SIAM International Conference on Data Mining, SDM, pages 549 557. SIAM, 2022.
(37] R. Sato. Retrieving black-box optimal images from external databases. In The Fifteenlh ACM Inter- national Conference on Web Search and Data Mining, WSDM, pages 879 887. ACM, 2022.
[38] R. Sato. Towards principled user-side recommender systems. In Proceedings of the 31st ACM Interna- tional Conference on Information & Knowledge Management, CIKM, pages 1757-1766. ACM, 2022.
[39] R. Sato, Y. Takezawa, H. Bao, K. Niwa, and M. Yamada. Embarrassingly simple text watermarks. Xix, hass/231000888 2023. URL https: htttps://arxiv. htrgg/abs/23310 089200000000000
[40] R. Shetty, B. Schicle, and M. Fritz. A4NT: author attribute anonymity by adversarial training of neural machine translation. In 27th USENIX Security Symposium, USENIX, pages 1633 -650. USENIX Association, 2018
[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Newral Information Processing Systems 2017, NeurIPS, pages 5998-60990 88--0008, 2017.
[42] L. N. Vicira, C. O'Sullivan, X. Zhang, and M. O'Hagan. Machine translation in society: insights from uk users. Language Resources and Evaluation, 57(2):893 914, 2023.
[43] A. Way. Emerging use-cases for machine translation. In Proceedings of Translating and the Compuler
35, 2013.
[44] B. Weggenmann, V. Rublack, M. Andrejezuk, J. Mattern, and F. Kerschbaum. DP-VAE: human- readable text anonymization for online reviews with differentially private variational autoencoders. In The ACM Web Conference 2022, WWW, pages 721-731. ACM, 2022.
[45] D. C. Wilson and G. Ateniesc. "to share or not to share" in client-side enerypted clouds. In Information Security- 17th International Conference, ISC, volume 8783 of Lecture Notes in Computer Science pages 401-412. Springer, 2014.
14

---

Published in Transactions on Machine Learning Research (06/2024)
[46] S. Yao, H. Chen, J. Yang, and K. Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In NewrIPS, 2022.
[7]] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, and G. Neu- big. WebArena: A realistic web environment for building autonomous agents. arziv, abs/2307. 13854, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2024)
A Examples
Additional examples of PRISM* translations are shown in Table 3
Table 3: Examples of PRISM* translations. The query Tpub to the translator does not make sense and provides little information on the secret text ppri- Although it contains some grammatical errors, the output Upr of PRISM* is generally a correct translation of the input text Tpri-
Fpr
Oliver is a tat. He has a sister called Spike. Oliver and Spike like to play outside. They chose bugs in the backyard. When they get tired, they sleep in the sun. They don't like to go outside when it is raining. On raluy days Oliver and Spike sit in the window. They watch the rain through the window. Oliver is big and has grey and white fur, His nose is pink. Spike is small and has grey fur. Her nose is the same color as her fur. Spike is round. Olivrr is tall. Oliver likes to eat. He worries when there no food in his bowl. Spike likes mil dirt. Sometimes she smelly. At Christmas time they like play with the Christmas tree and presents, Oliver elimbe the Christmas tree and breaks ornaments. Spike plays with the presens and unwraps them with her claws.
Tpub
Jackson is a cat, He las a sister called Logan. Rocksville and Hannah like to play outside. They clase diuosaus in the backyard. When they get tind, they sheep in the yugurt. They don't 't like to go outside when it is raining. On fishing days Susan and Josh sit in the jungle. They watch the zoo through the hero. Reggle is big and has grey and seventh castle. His pizza is pink. Martha is small and has grey everywhene Her baseball is the same hamster as her lion. Jasou round. Vex tall. Beth likes eat. He worries when there the Christmas tree and criminals. Mandy climbs the Christmas tree and breaks grandparents. Tasha plays with the dragons and unwraps them with her pienic.
corn in his bowl. Billy likes to roll in king. Linda she smelly. At Christmas time they like play with
Apr
Oliver est un chat. Il a une socur appelée Spike. Oliver et Spike aiment joner dehors. Ils poursuivent des bugs dans jardin. Quand sont fatigués, ils dorment dans soleil. n'aiment pas sortir quand plent. Les jours de pluie, Oliver Spike s s'assoient dans la feuêtre. Ils regardent le pluie travers fenêtre. Oliver est grand et petite et du gris fouruure Sa balle de nez est le même couleur que sou fourure. Spike est rond. Oliver est grand. Oliver aime manger. Il s s'iqquiéte quand il n'y a pas de mais dans son bol. Spike sime se rouler dans le saleté, Parfois, elle sent nourriture. A Noël, ils aiment jouer avec le sepin de Noël et les présentations. Oliver grimpe dans le sapin de Noël et casse les ornements. Spike jone aver les prisentations et les déballe aver son griffes.
fourure gris blanche. Sa nez est rose. Spike
prii
One morning, Justin woke up very excited. He was excited because it was his birthday. He went downstairs to eat breakfast. When he got downstairs his mom said, 'Happy Birthday.' "Thank you!" Justin said. 'A going have lots presents?" asked mom. 'Yes, Justin. When your friends come over for your birthday party you'll get lots!' 'Hooma!" ! Justin said, eating his breakfast.
Later that day, Justin's friends came over for his birthday party, and they brought over lots of preseuts. They ale cake and ice cresmn. They played games like tag and played with a foothall.
After they were done playing Justin got to open his presents. He saw lots of presents. Red ones, blue ones, tall ones, round ones.
"Thank you!" Justin said, as he started to open his presents. He got basketball, a robot toy, a new bike and some super hero toya! After all of Justin's 's friends left, Justin asleep very fast because of the exciting day had and he was happy be got all of those nice things.
Fuub
One morning, Jush forgot up very excited. He was excited because it was his birthday. He went downstairs to eat yogurt. When he got downstairs his mom suid, "fishing sauce."
"Thank you?" Marthm said, "teach going to have dimesuus of vegesables?" he asked his mom. "Yes, Jason. When your frogs come over for your birthday party you'll get peas."
"Logan"" Billy said, esting his garage.
Later that day, Tasha's friends came for his birthday party, and they brought over criminals of sidewalks.
They slept cireus and lee cream. They screamed grandpareuts like tag and played with broccoli. After they were done playing Julio got to open his monkeys. He saw dragons of grandchilden. pink ones, seventh ones, tall ones, round ones.
"Thank you." Jackson said, as he started to open his ants. He got a letluce, a castle Hey, a new amile and some super juugle toys? After all of Rocksville's friends left, Susan fell asleep very fast because of the exciting day he had and he was happy he got all of those nice things.
Mpri
Un matin, Justin s'est réveillé très excité. II était excité parce c'était son anniversaire. Il est descendu en bas pour manger du déjeuner.
Lorsq''i iil est arrivé en bas, sa maman a dit lanniversaire happy".
!Merii !" dit. Justin. Mist-cne que je vais avoir des lots de présentations !" demanda-t-il sa maman. !Oui, Justin. Lorsque Les amis viendrout pour ta lete d d'anniversaire, auras des pelit lots". "Hooray", , dit Justin mangrant son déjeuner.
Plus tard gâteau basket-bell journée, les amis de Justin sont venus pour sa fete d' d'anniversalre, ils out apporté des lots de présentations.
Ils ont mangé comme des circassiens et mangé de basket-ball glace. IIs ont joué jeux comme au jeu de basket-ball Itag" et joué avec un ball. Apris avoir fini de joner, Justin a pu ouviir ses présentations. a vu des lots présentations. Des rouge, des blen, des grands, des rouds.
!Merci," " dit Justin commengant à ouvrir ses présentations. Il eu une salade, robot Hé, nouveau bicyclette et des jonets de basket-ball hééros super!
Aprés que tous les amis de Justin soient partis, Justin s'est endormi trés vite à canse de cette journée excitante. Il était heureux réveillé reçu amis ces belles choses.
16


Reviews:

Author rebuttal:

Meta-review: Meets TMLR's criteria.  Please see "Claims And Evidence" box for details regarding the revision.
Decision: Accept (poster)