Counterfactual Memorization in Neural Language Models
Authors: Chiyuan Zhang
Daphne Ippolito
Katherine Lee
Matthew Jagielski
Florian Tramèr
Nicholas Carlini
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Memorization
Language Models
TL;DR: We study a notion of counterfactual memorization on common language modeling datasets and show that memorized examples exist in all of them and are not easily captured by more standard text matching based notion of memorization.

Abstract: Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data.
Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out ``common'' memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data.
We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training.
We identify and study counterfactually-memorized training examples in standard text datasets.
We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.

Counterfactual Memorization in Neural Language Models
Chiyuan Zhang Google Research chi chiyuan@google.com com
Daphne Ippolito Carnegie Mellon University
Katherine Lee Google DeepMind kathherinelee@google..com
daphnei@cmu. edu
Matthew Jagielski Google Deep.Mind j aagielski@google.coom com
Florian Tramèr
Nicholas Carlini Google DeepMind ncar wwwrliiigogle.cooo com
ETH Zürich florian. tramer@inf..o www.eeinffehhz..c......
Abstract
Modern neural language models that are widely used various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization important in real world applications and also from a learning- theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memo- rization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if particular document is omit- ted during training. We identify and study Counterractualllymmmmriiiz training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.
Introduction
Modern neural language models (LMs) have achieved impressive results in generating high quality text [e.g. Brown al., 2020, Zhang et al., 2022, Chowdhery et al., 2022, OpenAI, 2023] and have led to breakthroughs in many downstream natural language processing tasks [Devlin et al., 2019, Raffel al., 2020b, Bommasani al., 2021]. The paradigm of taking single large-scale pre-trained model and fine-tuning it for many tasks motivates the study of these models' ability to generalize by avoiding memorizing their training data. Moreover, memorization of sensitive user information or copyrighted materials in the training data [Carlini et al., 2020, Vyas et al., 2023, Lee et al., 2023] leads to practical concerns real world applications.
Previous work on memorization in neural language models demonstrated the ability to extract memorized training data, including sensitive data such phone numbers and usernames [Carlini et al., 2020, Ziegler, 2021, Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas al., 2020]. One issue with these extraction attacks that they primarily identify "common" and frequently occurring strings in the training set. For example, as shown in the analysis of Lee et al. [2021], near-duplicate training examples, which are very common standard text corpora, account for large majority of the memorized content. To filter out such commonly occurring strings from all memorized texts, previous work applied various heuristic rules distinguish frequently-occure sequences from memorization of isolated pieces of information.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this paper, we propose a principled causal perspective to disentangle memorization of common vs rare data, by directly tying a model's predictions to the presence or absence of individual training examples. We define counterfactual memorization as a measure of the change in model's prediction when a particular example is excluded from the training set. Counterfactual memorization accounts for the commonality of an example as removing one instance of a text that is common across multiple documents will have a minor effect on the model's prediction on that text. The mathematical formulation of counterfactual memorization extends a prior definition of label memorization in classification models [Feldman, 2020] to the context of neural language modeling.
Formally, a training document is considerded counterfactually memorized, when the language model predicts x accurately if and only if the model was trained on r. This allows us to construct a procedure to quantitatively measure the memorization of isolated pieces of text, whose sole presence in the training dataset have large effect on the model's predictions.
Following Feldman and Zhang [2020], we further extend this definition to counterfactual influence, which measures the influence of a memorized training text sample on another text example. Counter- factual influence allows us to trace the source of information for a model's predictions, by locating the training example(s) which significantly contributed it. With these tools, we study memorization across several standard text datasets. Our main contributions are as follows:
1. We define counterfactual memorization in neural LMS which gives us a principled perspective to distinguish memorization of "rare" and "common" information in neural LMs (Section 3).
2. We estimate counterfactual memorization on several standard text datasets, and confirm that rare memorized examples exist in all of them. We study common patterns across memorized text and the memorization profiles of individual internet domains. (Section 4).
3. We identify an inverse correlation between number of duplicates and counterfactual memo- rization as compared with previous definitions of memorization (Section 5).
4. We extend the definition of counterfactual memorization to counterfactual influence, and study the impact of memorized examples on the test-time prediction of the validation set examples and generated examples (Section 6).
2 Related Work
Previous work analyzed the memorization of large language models on sensitive information (e.g. phone numbers) in the training data [Carlini et al., 2020, Ziegler, 2021] or synthetically injected "canaries" [Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020]. However, not all the memorized texts are equally interesting - as confirmed in a later study [Lee al., 2021], near-duplicated training examples are very common in standard text corpus, and those commonly occurring phrases contribute significantly to memorized texts. In order to distinguish "common" memorization of common phrases or public knowledge from "rare" memorization of private, rare information, various heuristics were adopted in previous investigations. Our paper proposed a principled perspective towards this problem. Our intuition comes from psychologies studies that categorize human (declarative) memory into episodic memory [Tulving, 1983] of specific contents of individual events, and semantic memory [Squire, 1992] about general knowledge like grammars and factual information. We would like the models to obtain semantic memory but avoid episodic memory. The capture the latter, we proposed a notion of counterfactual memorization. The mathematical formulation of counterfactual memorization is borrowed from notion of label memorization in Feldman [2020] and adapted to the context of neural LMS this paper. This formulation has been studied empirically in the context of computer vision in Feldman and Zhang [2020]. In a follow up work, Ilyas et al. [2022] showed that it is possible to fit a datamodel to predict the outcome of training model However, this procedure requires training a massive number of models (c.g. 300,000 for CIFAR-10) on random subsets of the training data, thus is computationally infeasible for the scale of language models considered here.
specific training subset and evaluating on specific input.
The general idea of measuring model behavior on held-out training data is common in machine learning. In cross validation, held-out data is used to estimate the test performance for model selection; in learning theory, leave-one-out stability was shown to be deeply connected to generalization [e.g. Mukherjee et al., 2006]; in differential privacy, the worst case performance difference of models trained on two "neighboring" datasets (identical except single example being held-out replaced)

---

quantifies the privacy guarantee of a learning algorithm [Dwork et al., 2014, Nasr et al., 2021, Jagielski et al., 2020]. Most previous work aimed for an overall measurement, while our paper focused on characterizing the behaviors of individual examples.
We estimated counterfactual influence study how memorized training example impact the model prediction at test time. Influence functions have been used in statistics to assess robust estimators since Hampel [1974]. Previous papers adopted it to analyze neural network predictions [Koh and Liang, 2017, Koh et al., 2019]. However, the estimation was found to be computational expensive and fragile [Basu et al., 2021]. Pruthi al. [2020] tracks the gradient updates during training to estimate the influence from training example; Feldman [2020], Feldman and Zhang [2020] use aggregated statistics from multiple models independently trained on heldout data subsets to estimate the influence. Further extensions were shown to work well on detecting mislabeled data in classification problems [Wang and Jia, 2022] and characterizing hallucinations in Neural Machine Translation [Raunak et al., 2021]. Alternative methods also looked at simple data statistics (e.g. co-occurrence counts) without model re-training to infer the causal effects on language models' predictions [Elazar al., 2022]. In this paper, we adapt the approach from Feldman [2020], and formulate counterfactual influence directly with subset sampling, as oppose to leave-one-out influence. We also extend the estimation to assess the influence generated examples.
Counterfactual an important notion in statistical causality [Pearl et al., 2000, Rubin, 2005, Pearl, 2009, Imbens and Rubin, 2015] useful for studying causal probabilistic inference under alternative conditions. Such counterfactuals may or may not be directly testable (e.g. a counterfactual treatment in medical studies). In this paper, we directly measure the counterfactual influence of training example by comparing the behavior of the model trained with and without that example.
3 Counterfactual Memorization
To quantify memorization of rare details of a specific training document, we define the following notion of counterfactual memorization. The mathematical formulation borrowed from Feldman [2020], where it was originally proposed to quantify label memorization in multi-class classification problems. We extend to the context of unsupervised neural language modeling.
Definition 3.1 (Counterfactual Memorization). Given a training algorithm A that maps training dataset to a trained model f, and a measure Mf,, 2) of the performance of f on a specific example I, the counterfactual memorization of training example in D is given by
mm A
EsED, (A(S)), #)]
ass (A(S), x:)
performance when trained with
performance when trained with
where and are subsets of training examples sampled from D. The expectation taken with respect to the random sampling of S and S', as well as the randomness in the training algorithm A.
That is. our memorization definition compares the difference between two expected performance measures on given example x. On one side, we compute the expected performance of model when trained on datasets that contain the example xr and, on the other side, we compute the expected performance of model when trained on datasets that do not contain the example æ. Throughout this paper we use per-token accuracy as the measure M. In other words, we ask the model to predict the next token based the groundtruth context (preceding tokens), measure the 0-1 loss of the argmax token prediction, and then average across all predicted tokens.
The expectations in Equation (1) can be empirically estimated via sampling. Specifically, we train m different models on independently sampled subsets St..... Sm of equal size [S.]] r| for fixed r E (0, 1). We then divide these models into two groups: the first group contains all models trained on subsets S where S; and the second group are all models trained on subsets where x & S. We take the average performance on x in the two groups separately and compute the difference between the two:
mem (x)
mean Crean|M(([SS) (M()))) x)] mean (A(S)) Eugs..
E.ESS
This difference quantifies how the presence or absence of the example x in model's training set affect the model's performance on z. there a large difference between including an example in the training set versus not including it, then we consider this example counterfactually memorized.
3

---

For each т, we refer to models trained with in the training set (fA(S;) I E S as IN models and the models x was not trained on (fA(S:) x & S:)) as OUT models. Note we do not need to Instead, we train models once random subsets of D, and
retrain a model for each example compute the estimation (Equation 2) for all examples using the same set of m models. Ilyas et al. [2022] recently showed that it may also be possible to directly predict these scores using a regression model, yet this approach is computationally prohibitive for large language models.
4 Analyzing Counterfactual Memorization
We estimate and analyze counterfactual memorization of training examples in three standard text datasets: RealNews [Zellers et al., 2019], C4 [Raffel et al., 2020a] and Wiki40B:en [Guo et al., 2020]. Unless otherwise specified, we use Transformer-basee language models [Vaswani al., 2017] equivalent to (decoder only) T5- [Raffel et al., 2020b] with ~112M parameters. To save computation and enable more direct comparisons across datasets, we truncate the training set for each datasets by taking the first 2 documents. To estimate counterfactual memorization, we train 400 models for each dataset, each on a random 25% subset of the training examples. In practice, we use a hash-based filtering mechanism to efficiently approximate random subset sampling (details in Appendix G), as the data loading APIS for large text corpora generally support only sequential visits to examples with limited shuffling and subsampling capability within a window.
We train each model for 60 epochs' using the Adam optimizer [Kingma and Ba, 2015] with learning rate 0.1 and weight decay 10 5. For to average per-token accuracy of 44.21 //1..////////////// 5566/6635%%% on the subsampled training set, and 27.90%/31. 11131.999%//955555 on the validation set. On average, the models start to overfit at around epoch 5, as indicated by the signal that the validation accuracy starting to decrease.
respectively, our models converge
4.1 Distribution of Memorization
Table shows examples from the RealNews training set sampled at various memorization levels. Examples with the highest memorization are generally unconventional text such as all-capital letters, structured formats (i.e., tables bullet list), and multilingual texts. After those artificial examples, examples with intermediate-to-higg memorization are most often news reports of specific events. One of our main goals is to be able to separate memorization of such examples containing details of specific events from memorization of common facts or highly duplicated template texts. Indeed, templated documents with many near-duplicate copies in the training data generally have low coun- terfactual memorization. C4 and Wiki40B:en have similar trends. Interestingly, though Wikipedia articles are less likely to be auto-generated from templates than the web in general, we do observe repetitive patterns in low-scoring documents, such as START_ARRTILL _ARTICLEE <place name>, Virginia START PARAGRAPH <place name> is an unincorporated community in <county name>, in the U.S. state of Virginia."
To visualize the distribution of memorization, we plot 2D histograms in Figure 1, where the x-axis shows the difference of IN-accuracy and OUT-accuracy (i.e. the counterfactual memorization), and the y-axis shows the sum of the two, which we term "simplicity". scored highly regardless of whether a model saw it during training. The histograms are plotted in log scale to better visualize the exponential decay in the tail for high memorization and simplicity levels.
simple example is one that is
From the 2D density plots, we find that easy examples tend to have low memorization. However, there is no simple linear correlation. Peak memorization occurs for examples of intermediate simplicity. For the hardest examples, the memorization scores are low, because even the IN-models could not learn them well. Many hard examples consist of ill formatted text or contained foreign languages. As result, in Wiki40B:en, which contains higher quality texts, the lower bound of the histogram is higher than the other two datasets (Figure 1). Interestingly, the choice of data has relatively minor effect on memorization: the shape of the memorization histogram is generally consistent across the three datasets; the range of memorization values only slightly compressed for Wiki40B:en.
'oodrnn language models are usually trained for fewer epochs if the training set is massive. Since we have a smaller subsampled training set, we train the models for more epochs to allow the models to fit the training data sufficiently to study memorization effects.

---

Table 1: Examples of RealNews training set sampled at high, intermediate and low memorization. The URL of each document is included the beginning of each example. [..]] indicate omitted text for brevity. In the last block, two near-duplicate examples are shown; the yellow highlights in the last block indicate differences.
Tndex
2090855
0.6546
link THE AMERICAN JEWISH CONGRESS ANNOUNCED TODAY THE PUBLICATION REPORT JEWISH NON- EMPLOYMENT AS A RESULT ECONOMIC DISCKIMINATION. [ THEREAF TER ONE OF THE DEPARTMENTS
T&&
T." MOSS UNPRECEDENTEDD " ENGAGEED JEWISH ICANT link RECIPE: Chiness Park & Vegetable Soup with Wontan Noodles Chinese Pack & Vegetable Soup with Waston Noodles I pork renderfoin labout
2085736
0.5755
172-inch cubes" cups lower-sodumm chicken beoth cup water" 1/2 cup [..] Makes servings (shour 1/2 caps each) Recipe by Pork nefisirrd comn actaptations by culinary dieiiian nutritionist Kim Galaaz, RDN CID
pound cooked and
1680600
0.5007
link b- Languige English Arable 125t II acknowtedgesent of country [-. Arabie text II would like to acknowtedge that this meeting is being held the inaittional landl of the (appropriate group) people, and my respect elders hoch past and present." [+]]
2074505
0.2835 link & A Toxas honors student punished for saying that homosesuality was wrong has had his suspension rescinded [.d] Wester Hills High made decision reversing their
and improper' said Staver. applaadd student for standing up. We stood with to resist an unjust suspension susgension woog are pleased that suspansion has been reversed' [co] liberty Counsel will contime the right to escrcise frendom of conscience and religion." said Stiver. "These instances are incrensing and will continee to increase unless Christians and people who love liberty stand and reust this intoleracee.'
of action. "The decision rescind the susgension the correct
449808
0.0261 link & Lnvestors in Digial Realty Trus Inc, ( (DR] saw new options begin cading this week. for the Febeuary 2014 exparzion. At Stock Options Channel. YieldBoost formala has looked and down the DLR optins chain
February 2014 contracts and idemiiied one put and one commect of puntcullar interest The put contract the $45.00 snike price current bid $1.00, bid]
L157331
0.0356
link b investors in Abercrombie & Fitch Co, LANE) saw new options become available today, for the April difi expiration. At Stock Options Channel, our YiekdBoost formula has looked up and down the ANT oprions chain for the new April 4th contracts and identified une put and one call contrnt particular interest. The pur contract 534.00 strike price
bid SL.97
32
10
10
14
2.0
1.
Simplicht
Simplicity
Simplich
0.8
88
0.8
0.4
RealNows
C4
Wiki40B:on
02
00
21 0z
53 Memorization
04
06
3.7
300
00
21
0z
53
04
05
07
30220P
30
03 52
05
04 05 56 0.7
Memorization
Memorization
Figure 1: The joint distribution of counterfactual memorization (X axis) and simplicity (Y axis), where simplicity is measured the overall accuracy for an example across all models. (Histograms are in log-scale).
Figure/ shows the overall distribution of memorization for each training datasets. To obtain more granular view, we can also analyze the distributions for texts sourced from individual web domains in RealNews and C4, to see whether different data sources display different memorization profiles. Web domains such as news portals, blogs, and forums differ both stylistically and in how much they reference or even copy from other websites. Additionally, some domains are represented much more frequently than others in the datasets we studied. This could Icad to considerably different memorization profiles for examples from different domains.
To investigate these effects, we visualize the 95th percentile memorization score in each web domain against the number of examples in that domain for RealNews (Figure 2a) and C4 (Figure 2b). C4 contains many more domain names than RealNews since the latter is collected only from news websites. For both datasets, the domains with large number of crawled documents show smaller variance in the 95-percentile values, while "smaller" domains depict a wide range of variety in memorization profiles. The memorization profiles of few representative domains are visualized in Figures 2c and 2d. The domains we selected for visualization are: the largest domain (blue), the domain with highest 95 percentile memorization (orange), and two domains that have more than 1000 and 50 articles in RealNews and C4 respectively (green and red).
In RealNews (Figure 2c). reuters. com contains the largest number of documents but low memoriza- tion scores on average. The domain digitallibrary. wn. on.g the United Nations Digital Library, has high memorization scores potentially because it contains many multilingual documents. We have observed that less frequently occurring tokens, like those in foreign languages or ALL-CAPITAL words tend to cause high memorization. Similarly, flattened structured data (e.g. tabular texts) also deviates significantly from normal English texts and potentially leads to high memorization, as demonstrated by zap2it t. com, a website for TV program listings. On the other hand, hotair, com is
5

---

#documents
reuters.com
blogspot.com buckinghamautos
Density
digitallibrary
#documents
reuters.com digitallibraruuu orynn hotaiccom
hotair.com
zap2it.com
www.nittestatte
10
zapTlt.com Tt.com
zipcodes..u
zhwiikipedia.org
5.0
103
0.0
0.0
0.2
0.4
0.6
Memorization
102
10
(6)
Density
blogspot.com
10
10
buckingharautos.com
mmunittedstteeiiccdddss.
15
zhwwikipedi.org
100
100
5
0.2
0.3 0.4
0,5
0.0 0.2
0.4
0.6
0.0
0.1 0.2 0.3 0.4 0.5 0.6 Memorization (d)
95-percentile of memorization
95 percennntteee of memorization
(a)
(b)
Figure 2: For each web domain, we plot the 95-percentile of memorization against the number of examples from that domain in (a) RealNews and (b) C4. The red dotted line indicates threshold of minimum of 1000 articles for RealNews shown for (c) RealNews and (d) C4.
50 articles C4. The memorization distributions a few representative domains are
commentary website that frequently quotes other major articles. This may lead to duplicate text in the dataset which we suspect contributes to its overall lower memorization distribution.
The observations are similar C4: blogspot. contains large number of documents in the training set with only moderate amounts of memorization; zh. www.wediia.aa..mmmmm serieddd.oorg org and buckinghamautos. com. have high memorization due to foreign (Chinese) structured (car sales listings) text; and www. uni teidtttteszipcooes org has very low memorization scores because common templates re-used to generate similar pages for individual zip codes.
4.2 Number of Models Needed
To evaluate the impact of single training example, one may wish train two models that differ only in that single example. In practice, the stochasticity in single run of common training algorithms (e.g. SGD) produces too low signal-to-noise ratios be useful for such estimation. Moreover, leave-one-out estimation means separate pair of models needs to be trained for each training example, which is computationally costly. Therefore, we formulated our estimation in Section by accumulating statistics from m models independently trained on random training subsets. In our experiments, we set m 400. To understand how sensitive our results are to m, we analyze the rankings produced by distinct sets of models of size We vary from 6 to 192, and partition our set 400 models into up to 10 sets m models (e.g. for m = 192, we construct 2 partitions, and for 6, construct 10). We then compute the Spearman's 's R between these partitions to measure the agreement between the rankings produced by each partition. If the rankings are very similar (have Spearman's 's R close 1), then this number of models is reliably estimating the true ranking of memorization scores. We plot these Spearman's 's R values in Figure 3a. Even at 96 models, this correlation begins to plateau near lending confidence that 400 models is sufficient for reliable estimation of memorization scores. See Appendix D for more analysis on the sensitivity to m.
4.3 Impact of Number of Training Epochs
As expected, the overall amount of memorization grows consistently with the number of epochs of training (Figure 3b). This makes sense since training for more epochs increases overfitting. As training progresses, we also see an increasingly long tail of examples with high memorization scores. RealNews, about 59% examples had consistently increasing memorization scores across all epochs considered. There were no examples whose memorization decreased in significant way over training (all observed decreases can be attributed either to noise to instability early in training). Only 0.5% of examples stayed completely un un-memorized with scores which never rose above 0.1.

---

Memottation
Fracton Fraction Memorized
10
threshold
0.1
0.9
06
Momeriaatoon
10
0.8
Speamans
0.4
10
16
10
5
24
48
96
192
40
60
(9)
Models Trained
(b)
TTraining Epochs
(c)
Training Epochs
(d)
duplicates
Figure 3: (a) Spearman's s R between memorization rankings from two disjoint sets of m models. The rankings are variable at low numbers of models, but starts to converge at 192 models. All of our other experiments use 400 models. Reported values are averages over up 10 partitions, with emror bars of 10 standard deviations. (b) The distribution in memorization of RealNews examples training progresses. (c) The fraction of RealNews examples with memorization consistently above the specified threshold as training progresses. (d) For RealNews, we plot memorization scores against the number ncar-duplicates an example had in the datasct.
while 85% of examples had memorization scores which never rose above 0.2. Figure 3e shows the fraction of memorized examples as training progresses, at several thresholds of memorization. We can see that more training epochs significantly increases memorization.
5 Duplicate Text and Memorization
One of the goals of evaluating counterfactual memorization is to identify examples that have a low number of duplicates yet whose presence versus absence in the training data has a large effect on the model. Here, we perform a quantitative study of the (anti-) cottelccorelltionn between duplication and counterfactual memorization compared with the positive correlation between duplication and the "generation-tine memorization" definitions of memorization used by Lee et [2021], Carlini et al. [2022], Kandpal al. [2022].
Following the method from [Lee et al., 2021], we first use MinHash [Broder, 1997] to identify near-duplicate examples in RealNews train set. We consider a example duplicate if it has an normalized edit similarity of greater than 0.7 (definition included in Appendix I). Out of 2.01 million examples, ~03,0000 were identified as being near-duplicate with at least one other example. Among these crrequentl-occurrrin examples, the Pearson correlation between an example's 's counterfactual memorization score and the number of near-duplicates for that example is -0.39; in other words, memorization does quantitatively decrease when data is repeated more often.
In Figure 3d we can see that examples with a large number of near-duplicates have smaller mem- orization scores. Counterfactual memorization primarily differentiates amongst examples with few number of duplicates. This makes sense given that examples with lots of near duplicates would likely have their near duplicates in memorization" (discussed in Section A) that measures the textual overlap between model generated texts and the training documents. There, the number of occurrences strongly correlate with the measured memorization [Carlini et al., 2020, Lee et al., 2021, Kandpal et al., 2022]. Counterfactual memorization measures a fundamentally different type of memorization from simple textual matching considered in prior work, providing information about how easy or hard training ex- ample the context of the rest of the training set. Table we can see this effect qualitatively: sequences with near-duplicates in the training set tend to have low counterfactual memorization (as expected).
This is be contrasted with "generation-time
6 From Memorization to Influence
Counterfactual memorization identifies training examples that contain rare information not conveyed by other examples. A natural question to ask is whether a model would leak the information in memorized example during inference. Previous paper studies membership inference attack [Shokri et al., 2017, Sablayrolles et al., 2019, Long et al., 2020] where an attacker tries to figure out if particular example exists the training set. this paper, we consider standard model evaluation without adversarial attackers, and quantify "does seeing particular training example strongly influence the prediction on a validation example?" Another way of asking this if a single example in the training set has large and over-representative impact the prediction of validation
7

---

10
high influence
10
10
intermediate
10
0.35
0.35
10
influence
Influence
0.30
RealNews
Max Influence
0.30
Wiki40b:en
10
random
0.25
0.25
0.20
0.20
103
0.15
0.15
10
0.10
0.10
0.11 0.O 0.1 0.2 0.3 0.4
0.05
Train Test Influence
0.0
0.2
0.4
0.6
10
0.0
0.2
0.4
0.6
10
Memorization (b) Joint distribution of influence and memorization
Memorization
(a) Histogram of influence
Figure 4: (a) Histogram the influence of all the training examples on a speeific example for three different test examples on RealNews. The blue and orange examples have high and intermediate influence from some training examples, as indicated by the outlier values to the right of the cach histogram plot. The green one is random example. where the influence from all individual training examples are close to zero. (b) The joint distribution of the memorization score of each training example and its maximum influence any validation set example. The histograms are in log scale to better visualize the tail of distributions. C4 shown Figure
example. We answer these questions by measuring counterfactual influence with a formulation adapted from Feldman and Zhang [2020]]
Definition 6.1 (Counterfactual Influence). Given training algorithm A that maps a training set D to a trained model, and performance measure M, the counterfactual influence of a training example E D on another example x'
infl(x x') A EscD, (A((S)) E.cs (M((s)
where S is subset of training examples sampled from D. The expectation is taken with respect to the random sampling of S, as well as the randomness in the training algorithm A. Here x' can be an example from the validation set or test set, generated example training example.
An empirical estimation of the influence can be computed similarly to counterfactual memorization by uniformly sampling m subsets S1, .1..... SS from D, where |S D|, and calculating
infl(r = x²)= mean [M x')] mean M (A(S)), EargS
(4)
E
This measures how much training sample x's presence influences the prediction of different example x'. Note, mem(x) = infl(z = x), i.e., counterfactual memorization is self influence.
Influence Examples of the Validation Set. With the same models trained for estimating memorization, we can estimate the counterfactual influence on the validation set according to Equation (4). For each example in the validation set, training example. Figure 4a shows the distribution of influence from all training example on three different examples from the validation set. The green example was randomly chosen and represents the behavior for most validation examples: receive close-to-zee to-zroo influence from all the (individual) training examples. The blue and orange examples were sampled to have high and intermediate maximum influence. Each of them has one (or a few) strong influencer from the training set, as indicated by the bars to the right of the histogram. They also only receive tiny influence from all the rest of the training examples, though the variance of influence is larger than for the green example.
estimate the influence on it from each
Intuitively, most training examples will have small influence validation set examples because the models learn distributional patterns shared across many training examples, and individual training examples tend to have insignificant influence here. However, training example with high counterfactual memorization contains rare information that are not shared with other examples. Therefore, if validation set example x' contains similar information, infi(x = x²) could be large. Figure 4b shows the relationship between memorization and influence by plotting mem(x)) (x) of each training example against its maximum influence max,' infl(x + x') on x' across the validation set.
Consistent with our intuition, examples with small memorization scores have small max-influence scores. Larger influence scores on the validation set generally requires larger memorization scores

---

Table 2: Train-validation example pairs of RealNews sampled a variety of influence levels. indicate text omitted for brevity. Difterences in each document pair are highlighted yellow.
Index
Esti..
Text
Vilidaaiion
infi
Iik >
Meatical URL except with the http::/ proteesi instesd https: The text is idensical to the training example
1334662
0,3780
Train
mem
Iin > By Ani Rabinovitch TZBELIM VALLEY, Israel (Reuters) The disposable paper face masks offer little protection from the clouds dust that fill the chiffside cave where Istaeli unchurologists are wrapping [ insight into how people lived 2,000 to 8, 8,000 years ago. (Ediing by
2077314
0,3780
Jeffrey Hellerteremy Gount)
Validation
inf
Ink > VATICAN CITY (AP)) Emeritas Pope Benafier XVT is offering first-over papal resessment of his own pontiffeatee a book that receunts his decisiin resign, surprise his successor and his attempts dismumile what the Vitican's "pay lobby." "Benediet
83834]
0,1209
XVI The Finsl Comvenations" "i out in September, beest Vatican career by simply spreading passip that he was pry Ditferent webslees bui (almust) identical repurt
Train
Ink & VATICAN CITY- Emeritu Pope Benedict XVT is offering first-ever pepul asapsment of his own pontificatie in book that recounts his decision resign. surprise
614881
0,15
anempts dismonrle what culls the Varicun's "syy lobhy" " "Benediet XVI: The Final Convensttons is due out Septemberr the latest [-]] Vaticun career by simply spreading goesip that he was gay. Follow Nicole Winfield
www wwi.ier..ommmewee............. iomiiwifieed
Voliclation 6821077
infl
Ink > ANAHFIM- - a night when Francois Reancheni hid two assists Toronti Chris Pronger blocked sis shoes Detroit, [ (3,660 Ducks jaass might have been lost without thair program: in Wednesday night. It's ouly the first game of the presessoo, hit now em has clearly begun group mosily nowcomers Ducks uniforms
0,0673
Phoenix 3-2 shustoution Weinesday Honda
A familarr made Ducks' presesson opentr last year Difierent
the higgesl impaet, however, Rabby Ryan sconed two goils Ryan also scored two goals webaites on the same evest with sll sligiity different wordings
Train
lnk I ANAHEIM- On a night when Frinccis Beauchemin had two assists in Torcetoland Chris Pronger blocked six shots in Detroit, the 336669 Duck' finst game of the preseusom. but a new era has clearly began. group mosdl newoomens Ducks uniformss best. Phoenix 3-2 in shootouf, althoggh a familiar face made the biggest impact as Bobby Ryan scord two goal in repolation and aoother in the shootout. Rymn also scored two gonls the Ducks' presesson opener last year [1]]]
494435
Hoeda
Widnesday nested programs iideeiity the players their
oaly the
Validation
infi
link x More than 70,001 pounds of Buttehall rarkey reeatled hecause of potential salmonellla WASHINGTON The U.S. Departmear of
1165799
0,0360
Agricultre's Food Safety Inspections servicef announeed Wediesday I... They stapped nationwide retail and institutionl locasions ATED: event, ons embeded a loe more information than the othirs
the full rocall PSIS, Centers for Disease Control Prevention and Differest websites repoeting the some
Train
mem
Ink x Ruteerall recalls nearly 50,000 pounds of turkey lfter salmonella enses WASHINGTON The U.S. Departmint of Agriculrun'' 2018. The following products
1571976
0,2094
Food Sufery and segpectionerviicaaneeee
Wednesday [:.] The ground turkey produced
under recall were shipped to natiooiide retail instirutional locations; 48-oz. plastic wrapped way containing "BUTTERBALL everyday
Fresh Ground Turkey WITH NATURAL Fl AVDRIING (855 FAN/I55 FXTT" with well or frease by date of 7/26/88 8, lot code 8188, and UPC
codes 71555 or 22655-71557 reprsseted the label. 45-oz. plastic wergped try containing "BUTTERRALL everyday Fresh Ground
Turkey WITH NATURAL HLIVORING 192% LEAN/S% PALY" sell freese by date 1/2W18, labels hase, FSIS, Centers
Diserse Control and Provention and [..]
of the training example itself. However, not all training examples with large memorization scores lead to large influence scores. In particular, the max-influences drop significantly for examples with memorization larger than 0.4. One potential reason is that many examples with very high memorization are simply low quality text, so memorization is required in order learn them, but they do not encode anything interesting that could influence a validation example. On the other hand, even if a memorized example encodes some rare and useful information, the max-influence could still be low because the validation set does not contain a relevant document. This is especially true given that all datasets have considerably smaller validation sets than training sets.
Table 2 shows train-vall validatiin example pairs from RealNews sampled at different influence value ranges. We found that the trii-validattion pairs with the highest influence are almost identical, except some superficial differences, such as different handling of quotation em dash marks. As we move to intermediate influence ranges, we commonly found reports the same events. Large paragraphs of identical text indicate that one document might be citing the other or both citing from a third party. At low influence, two types of correlations are commonly observed: 1) templated texts with high similarity the reason for a low influence is that there are many similar training examples that split the influence; 2) superficially related documents due to a shared prefix such ST. CLOUD- This week in our "Behind the Scenes" series on WJON or a shared substring of some common knowledge like FSIS, the Centers for Disease Control and Prevention. Due to high signal-to-noise ratio, here were no noticeable relationships in the document pairs with influence scores below 0.02.
Influence turns out to be an effective tool for analyzing and attributing the model predictions at test time: for predictions that rely on information obtained by (counterfactual) memorization, we can identify exactly which training example provided such information. Our observation of near- duplicated training-validation document pairs is consistent with recent studies that identifies data contamination in large Internet crawled text corpus [Lee et al., 2021, Dodge et al., 2021].
Influence on Generated Texts. The influence estimation is not restricted to the validation set. We can also estimate influence on generated examples. In this section, we evaluate on the publicly released generations from the Grover models [Zellers et al., 2019] trained on RealNews. Specifically, we take the generations from Grover-Mega (p=0.96), a 1.5 boiliin--paaameee model trained on the RealNews
9

---

dataset. Comparing with the muin-valiaation influence in Figure 4b, the histogram (c.f. Figure 10 in Appendix.) decays faster as max max- influunncee grows. Moreover, the value range of max-influence is also twice smaller. The reason that we did not find a lot of highly influenced generated examples are two fold: 1) there are only 24,576 generation in the public release, which much fewer than the validation examples. As a result, the corresponding example of many memorized training examples do not get sampled in the generations. For comparison, previous work [Carlini et al., 2020, Lee et al., 2021] generated 100,000+ examples to identify memorization in generation. These approaches also count duplicates in the training set, which counterfactual memorization filters out. 2) The Grover model was trained on the full RealNews training set, while we have restricted our analysis to the first 2M training examples. There could be potentially more high influence training examples that are missed in our calculation.
7 Summary and Discussion
We studied memorization in neural language models. We formulated notion of counterfactual memorization as a tool that can systematically ignore "common" memorization such as general knowledge (e.g. "Paris is city in France") and captures memorization of rare, specific information (e.g. description of a specific episode of event) present in the training examples. We conducted experiments three commonly used text corpus in language modeling and found memorization in all of them. We further analyze the per-domain memorization profiles for Internet-crawled data, and found that different sources could have substantially different memorization profiles.
Furthermore, we analyzed how memorized training examples could impact the model predictions at test time via counterfactual influence. We found that for examples from both the validation set and the model generated texts, the model predictions could be drastically different depending on the presence or absence of particular training example with high memorization.
Limitations. This study mainly focus on English datasets. While expect the characterization of memorization would be similar when evaluated on corpus of other (natural) languages, new patterns might be observed multilingual data more structured domains such as programming languages.
Both the neural language models and training sets used in this work are orders of magnitude smaller than modern standards such as GPT-3 [Brown et al., 2020], GPT-4 [OpenAI, 2023] and PaL. [Google, 2023]. Morcover, we only conducted preliminary investigation of the dynamics of counterfactual memorization during training. Although our experiments effectively estimated and detected memorization, we suspect more interesting examples might emerge larger, more capable models are analyzed. For example, currently when the information from memorized training example is leaked in the prediction of a strongly influenced test example, it can usually be explained by a high text overlap between the training and test examples. For models with deeper understanding of languages, we suspect that strong influence could be observed even between documents that have no direct text overlap but that encode similar semantic information.
In order to test this, it will be necessary to scale our framework to larger models and datasets. Moreover, it will be necessary to construct datasets where semantically similar but textually different document pairs exist. One potential source to construct such datasets would be versioned Wikipedia articles-two versions of the same article with large time span edit distance may contain semantically similar (but paraphrased) information. Such a dataset of paraphrased text pairs would be more broadly useful to understand the ability of different models to disentangle text content and form by measuring the influence of one piece of text on a paraphrased piece of text.
Counterfactual memorization enables us to identify examples that whose presence or absence has large impact on the model and the model's ability to score and generate other text. The privacy risk for this is low since in order to perform this analysis, one would need to already have access to the dataset and the ability to train models.
The authors would like to thank Samy Bengio, Christopher A. Choquette- Choo, Ethan Dyer, Michael C. Mozer, Behnam Neyshabur, Andrew Nystrom, and Hanie Sedghi for constructive discussions and feedback. The authors would like to thank Andrew Nystrom for assistance with HaaHaahh-asee near-duplicate detection.
Acknowledgments.
10

---

References
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. ArXiv, 0ee/2006.146577 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lec, Tony Lee, Jure Leskovec. Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Malik, Christopher D Manning. Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W Thomas. Florian Tramèr, Rose E Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. August 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http://githubb com/ google/jax.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29 IEEE, 1997.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Sys- tems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL /prpe//diioocccggsg. 8bfb8ac1 1e114442264aa..pppee P
heurips..cc/paprr/20022 2020/ /tl//fille feweelllllaa htteee/1455codffc455555555
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jerne Kos, and Dawn Song. The secret sharer: Evalu- ating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. arXiv preprint arXiv:2012.07800 2020.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.00 023111 2022.
11

---

Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre- Fre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.186533/1//1111 14233. URL https: //aclanthology.o org/N199-423.
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. Documenting the english colossal clean crawled corpus. arXiv preprint arXiv:2104.08758 2021.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor Comput. Sci., 9(3-4):211 1-407077, 2014.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's factual predictions. arXiv preprini arXiv:2207.142: 2022.
Vitaly Feldman. Does learning require memorization? A short tale about a long tail. In STOC, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
Google. Palm 2 technical report. Technical report, Google, 2023.
Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In LREC 2020, 2020.
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-393, 1974.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http://gi wtw. cowggoogl///la.....momoooomo
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gonties Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems, 2017.
Andrew Iyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Alcksander Madry. Data- models: Predicting predictions from training data. arXiv preprint arXiv:2202.0066 00622, 2022.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd?, 2020.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The International Conference for Learning Representations, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box box predictions via influence functions. ArXiv, abs/1703.04777 2017.
Pang Wei Koh, Kai-Siang Ang, Hubert Hua Kian Teo, and Percy Liang, On the accuracy of influence functions for measuring group effects. In NeurlPS, 2019.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637-3647, 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. CORR, abs/2107.064499, 2021. URL httpt::/aarxi.ccooom wttp://..og///////////o/ooosse/
http://aaaxxvvvoo./.aaaa// abs 2ttps/21070000999..
12

---

Yunhui Long, Lei Wang, Diyue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (FuroS&P), pages 521-534. IEEE, 2020.
Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1):161-193, 2006.
Milad Nasr, Shuang Song. Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary instantiation: Lower bounds for differentially private machine learning, 2021.
OpenAl. Gpt-4 technical report. arXiv. 2023.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Judea Pearl et al. Causality: Models, reasoning and inference. Cambridge, UK: CambridgeUniversi ryPress, 19(2):3 2000.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. ArXiv, abs/2002.08484, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 67, 2020a. URL html.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020b. URL wwtp:/per//pp/////////. 200112000772 1tmm.
Vikas Raunak, Arul Menezes, and Marcin Sanezys-Dowmunn The curious case of hallucinations in neural machine translation. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. naacl-main.92. URL https: http///alaantttooo/o// wwwpchowwwooooooooooooooooo 20tg/2221..g naacl-main 92.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469) 2005.
Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2017.
Larry R Squire. Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychological review, 99(2):195, 1992.
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning, 2020.
Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow, Investigating the impact of pre-trained word embeddings on memorization in neural networks, In International Conference on Text, Speech, and Dialogue, pages 273-281. Springer, 2020.
Endel Tulving. Elements of episodic memory. Oxford University Press, 1983. ISBN 9780198521259,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances neural information processing systems, pages 5998-6008, 2017.
13

---

Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. arXiv preprint arX1v:2302.0087, 2023.
Tianhao Wang and Ruoxi Jia. Data banzhaf: data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205. 15466, 2022.
Santiago Zanella- Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communicatiinn Security, Oct 2020. doi: 10. 00.1145/3372297.34178888 URL http://ttx...tom doi org/
10. 10.1145/3372297 3417880.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neural Information Processing Systems, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Albert Ziegler. GitHub Copilot: Parrot crow? http://weesstttalooommiiiiiiiooe recitation, 2021.
14

---

1.0
meai accuracy (IN modelss
moon accuracy (IN models:
0.8
1..
12.8
accuracy
04
0.4
02
0.2
0.2
RealNews
Wiki40B:cn
0.2
066
0.8
0.2
0.6
08
1.0
0.0
0.4
0.8
(OUT models)
(OUT models)
(OUT movels!
Figure 5: Per-token accuracy training examples evaluated on IN models vs OUT models.
A Difference Between Counterfactual and Generation-Time Memorization
Many definitions of memorization operate at generation-tine, sequence of generated text is marked as memorized if a sufficient amount of overlap is found in the training dataset [Carlini al., 2020). When the training data is available, heuristic-based methods comparing language model perplexities are used to predict whether generation contains memorized content [Carlini al., 2019, Thakkar al., 2020, Thomas et al., 2020, Carlini et al., 2020, Zanclla-Béguelin et al., 2020]. One difficulty with these approaches is that generation-time instances of memorization strongly correlated with the number of similar near-duplicate examples in the training set. As observed in Lee et al. [2021], large clusters of near-duplicated examples do exist in common language datasets, dominating memorization detected in generated text. Generation-ting methods for measuring memorization are forced to design heuristics to avoid simply identifying these uninteresting instances memorization.
In contrast, the counterfactual memorization we study this paper handles the issue of near-duplicates automatically without the need for heuristies. For training example, xr with many near-duplicate copies in the training set, mem(r) will be small (because other samples x' se x will be present in is). This does mean that counterfactual memorization the training dataset whether memorization. An example, æ, with high mem(x) may have a high the opposite of generation chance of being generated if a model appropriately prompted, despite and possibly because is rare, and thus the example is considered memorized by both definitions. In summary, generation- time memorization measures the chance model will directly copy from training examples, while counterfactual memorization aims to discover rare information that is memorized.
B Average Accuracy of IN models vs OUT models
Figure compares the per-token accuracy between the models and OUT models for the training examples from three different datasets. Counterfactual memorization is estimated by taking the difference between the average IN-accuracy and the average OUT-accuracy Thus, the examples closer to the upper left corner are more counterfactually memorized, while the examples near the diagonal are not.
C The Impact of Data Deduplication on Memorization
To investigate the impact of data deduplication counterfactual memorization, compared C4 with C4-NEARDUP [Lee et al., 2021], l], which is derived from C4 with deduplication using approximate document matching. Figure 7 compares the distribution of memorization between the original C4 and the deuplicated dataset. We did not find significant difference between the two datasets. One potential reason is that the deduplication criterion of the training examples. fact, we can still easily see near duplicate examples in C4-NEARDUP among examples with low memorization, shown below:
relatively conservative, which removed only ~ 3%
Example 1380925 (mem 0.0374) link > This is placeholder page for Joshua Baldridge, which means this person is not currently on this site. We do suggest using the tools below to find Joshua Baldridge. You are visiting the placeholder page for Joshua Baldridge. This page is
15

---

10
10"
10
10a
10
10a
0.35
0.35
0.35
Max Influence
0.30
RealNews
Max Influence
0.30
C4
Max Influence
0.30
Wiki40been
0.25
0.25
0.25
0.20
0.20
0.20
0.15
0.15
0.15
0.10
0.10
0.10
0.05
0.05
0.05
0.0
0.2
0.4
0.6
10
0.0
0.2
0.4
0.6
10
0.0
0.2
0.4
0.6
10
Memorization
Memorization
Memorization
Figure 6: Full version of Figure 4b: The joint distribution of the memorization score of each training example and its maximum influence on any validation set example. The histograms are in log scale to better visualize the tail of the distributions.
here because someone used our placeholder utility to look for Joshua Baldridge. We created this page automatically in hopes Joshua Baldridge would find it. If you are not Joshua Baldridge, but are an alumni of Brecksville Broadview Heights High School, register on this site for free now.
Example 2048352 (mem 0.0320) link D- This is a placeholder page for Laytoya Brannon, which means this person is not currently on this site. We do suggest using the tools below to find Laytoya Brannon. You are visiting the placeholder page for Laytoya Brannon. This page is here because someone used our placeholder utility to look for Laytoya Brannon. We created this page automatically in hopes Laytoya Brannon would find it. If you are not Laytoya Brannon, but are an alumni of Mainland High School, register on this site for free now.
Example 1314053 (mem = 0.0278) link & This is a placeholder page for Devin Meguire, which means this person is not currently on this site. We do suggest using the tools below to find Devin Meguire. You are visiting the placeholder page for Devin Meguire. This page is here because someone used our placeholder utility to look for Devin Mcguire. We created this page automatically in hopes Devin Meguire would find it. If you are not Devin Mcguire, but are alumni of Kankakee Valley High School, register this site for free now.
Example 1085524 (mem = 0.0209) link >> This is a placeholder page for Anthony Christie, which means this person is not currently on this site. We do suggest using the tools below to find Anthony Christie. You are visiting the placeholder page for Anthony Christie. This page is here because someone used our placeholder utility to look for Anthony Christie. We created this page automatically in hopes Anthony Christie would find it. If you are not Anthony Christie, but are an alumni of Old Bridge High School, register on this site for free now.
10a
10a
1.8
1.6
Simplicity
1.2
1.4
1.2
1.0
1.0
0.8
0.8
0.4
C4
0.4
C4-NEARDUP
0.2
0.0
0.1
Memorization
0.2
0.3
0.4
0.5
0.6
0.7
10?
0.0
0.1
0.2
Memorization
0.3
0.4
0.5
0.6
0.7
20
10
Figure 7: The joint distribution of memorization and simplicity. The histograms are plotted in log scale to better visualize the tail of the distributions.
16

---

1.00
0.95
Stearman's 5
0.85
0.80
news
0.73
Wild
0.70
Trained
Figure 8: Spearman's R between memorization rankings from a set m models and our full set of 400 models. As more models are trained, the ranking changes very little, with the ranking at 192 models having a Spearman's R at least 0.992 on all datasets.
Mumber Samples 100
Number of Samptes 1000
Mumbhe of Samples 1000
120
models
models
100-
models
100
models
modelss
modex
60
68
modirix
madex
89
models modea
40
8
5.52
0.04
0.06
8.50
6.01
0.53 0.05
Rewl
C4
Wiki40R
Figure 9: The variance memorization scores decreases significantly as the number of models
increases for all datasets.
Measurements of the edit distances show that they are near the boundary the deduplication threshold chosen in Lee et al. [2021]. On the other hand, the tail of the distribution examples with high counterfactual memorization are mostly unaffected by text deduplication.
D Variance of Memorization Scores
In Figure 8, we measure the Spearman's 's R between our total set of 400 models and an model subset. As expected. as increases, does Spearman's R-in particular, 192 models, the Spearman's R is at least 99.2% for all datasets, and increasing m already appears to have diminishing returns.
Using the scores assigned to each sample. To do this, within each partition, we compute the memorization score assigned to each sample. We then compute the standard deviation of all partitions' memorization scores for each sample. In Figure 9, we plot each sample's standard deviation - in all, this demonstrates the distribution of the variance of memorization scores. We find that the variance decreases substantially as m grows, and concentrates near 0 already with m 192, for all datasets.
partitioning into size sets of models, analyze the variance of memorization
E Histogram of Max-Influence on Generated Texts
Figure 10 shows the histogram of max-influence each generated example by Grover-Megr (p=0.96) [Zellers et al., 2019], from the RealNews training examples. Those generated examples are publicly released https://g/ www.. wtt.:/co///w///eee http://////.eeegeeeeeeee/e//////
F Miscellaneous Experiment Details
Our experiments are implemented using JAX [Bradbury et al., 2018] and Flax [Heek et al., 2020], both open sourced library under the Apache-2.0 license. In the study of influence generated texts, we use the publicly released generations from the Grover models [Zellers et al., 2019], available at their open source code repository, under the Apache-2.0 license.
We run the experiments using our internal cluster. The majority of the compute consumed by model training. In this paper, standard training setup for transformer based neural language models,
17

---

10a
grover mega p0.96
10a
10a
10
10a
0.05
0.10
0.15
influence
Figure 10: Histogram of max-influence on each generated example by Grover-Mega (p=0.96), from the RealNews training examples.
165
60
10:5h
hash
50
420000
40
2.5
500000
30
3.5
20
1.0
10
100000
522000
524000
0.35 Pix,,
subset si26s
Selected!
both OUT)
Figure 11: Comparison of hash based subset sampling with numpy. random. choice.
which could run on single node machines with one multiple GPUs. However, to carry out the full analysis, we need to train 400 different models for each of the three datasets analyzed in this paper.
G Subsampling Procedure
In the estimation of memorization and influence, we trained 400 models each an independent random subset of training examples. We use Tensorflow Datasets (TFDS) to load our training data. TFDS supports loading a continnous range of examples, but does not support subset loading from list of indices of individual examples. The API has a filter function which allows us to provide a Tensorflow predicate to precisely control the subset loading, However, naive implementation of checking whether the index of the current example is in a given list subset indices is very slow and scales poorly with the subset size.
To mitigate the issue, we implemented hash based subset sampling predicate that can be evaluated efficiently for each example, and (approximately) select random subset of specified size. Let N be the total number training examples, n < be the expected subset size. The idea is to map the index i of each example to N/n hash buckets, and select all the examples that fall into one particular bucket. To make sure each model gets an independent subset sampling, we need to use different hash functions for different models. In our implementation, uint64 types with simple pscudo number based on the index of the current model to achieve this. Note the subset size sampled is close to but is not guaranteed to be exactly But this is not problem in our settings. The specific implementation shown below:
compose known hash function for
def hash_aahhhaaa seash_samleeemmmee seed, system): """Get hash based subset sampler.
Args:
mod: total_n_egs // subset_size
seed: different seed leads to different subset sample
Phttps://wwar http:://wwwweeesroooooooorrraaee
18

---

system:
'np" or 'tf'.
Returns: Tensorflow or Numpy subset sampler.
np_hash =
mul, offset, remainder (seed 1234 np.arangee3))
remainder remainder mod
if system 'np':
def np_sampler (n_sam)lee((__ttta))): (n
x = np. arange dtype=np.
(type=)p..unn66) uint64)
return ap offset) % mod = remainder
return np_sampler elif system == 'tf':
tf tf_hash = htt__uinnt4_buildee (t"))
def tf_filter (idxx .):
return tf. eq..e (f_ (idxmill offset) % mod, remainder)
return tf_filter
raise KeyError Unknown system: (system3')
def
"""Build hash function in tf/np for uint64."" if system 'np":
uint64_cast functools. partial array, dtype=np. op_xor operator.. xor
op_rshift operator, rshift
elif system == 'tf':
uint64_cast functools. partialll cast,
op_xor tf. wiw.iitise.. bitwise_xwwexor
op_rshift tf..bttiie..iigt..... rightsshift
else:
raise KeyError (f Unknown system: (system)')
wttper/iieeraaaaatttrrrrrrroooar uint64_cast(x)
def hash_uint644x)):
op_xor(x, op_rshift(x, 30)) (dnf544..astt00f888877644444454
op_xor(x, op_rshift(x, 27)) uint6__cass(03344044301333311111
op_xor (x, op_rshift(x, 31))
return
return hash_uint64
In Figure 11, we compare our hash-based subset sampler with numpy. random. random.choice. (N, size=n, replace=False).. The lefimost section of the figure shows that the sampling procedure always samples close to n points, with a small variance. The middle section plots histogram of the empirical fraction of total models that each point appears in. Note that, because we r 0.25, this fraction should be 0.25 on average, although, because we only use 400 models, each value will be identically 0.25. We find that our hash-based sampler produces probabilities which are highly consistent with those produced by numpy.random. choice. We also measure the pairwise independence of the hash-based sampler, measuring the probability that two different training points æ1, .2 appear both IN or OUT of model's ''s training set. We expect this value to be 0.625 (=y² + (1 (1) We plot this in the right portion of the figure, demonstrating that the independence of our bash- sampler is very similar numpy. random. choice.
19

---

mean logit(accuracyy modele)
mean logit modelss
mean loglttaccuracy (IN modelss
RealNews
C4
Wiki40B:en
-4 -3
-2 -1
-2 -1
-1
-3
-2
-1
0
2
3
mean legittaccuracy)) (ouT models)
mzan kgtttaccuracy)) (OUT models)
mcon log (accuracy) (OUT madels)
1.0
1.0
1.0
accuracy models)
mean Rocuracy INN modiess)
mean Cccurcycyy (N models)
0.8
0.4
0.4
0.4
RealNews
C4
:in
accurecy
meen eccuracy (OUT models)
meen accuracy
Figure 12: Comparison of directly using the per-token-accuracy vs. taking the logit of the per-token- accuracy. Top row: logitpper-oken-accuraay) Bottom row: per-token-accuracy. Figures exactly the same as Figure 5.
H Alternative Memorization Metrics with Logit Scaling
We defined the counterfactual memorization in (I) with generic performance measure M. Through- out the paper, we define M as per-token accuracy-the fraction of the times the model assigns the highest score the true next token in the sequence. The finite value range could cause unnecessary compression for values near the interval boundary. As a result, the resolution of memorization estimation is lower for models with very high very low performance. To mitigate this issue, explore an alternative measure by taking the logit on the per-token accuracy [Carlini et al., 2021). The logit function maps (-o0, oe) before aggregating across independently trained models. Figure 12 compares the scatter plots of average performance on IN /UT models measured by the logit scaled per-token accuracy and the per-token accuracy. Comparing the raw per-token accuracy, the scatter plots generated with the logit scaled measure are no longer artificially constrained to be triangular shape. As result, the memorization estimation, which proportional to the distance to the diagonal line, has a higher resolution on the two ends (lower left and upper right) than the unscaled version.
Note there no absolutely right or wrong measure. While the scaled version has better resolution on the two ends, the advantage of the unscaled version is that the value range [0, 1] makes it straightfor- ward to interpret the numerical values of counterfactual memorization. Since the consistency between the two versions are high (Spearman's p correlation between the two versions 0.947 / 0.903 / 0.944 on RealNews/ C4/ Wiki40B:en),, we use the unscaled version throughout the paper for easier interpretation.
I Definition of Edit Similarity
We define the edit similarity between two sequences i; and xy as. In our case, we use token-level similarity.
EditSim(z,, Edit-Distance(za x.)) max( [x, [x,])
20

---

Table 3: Pairs of RealNews training examples and Grover generations sampled at several influence levels. "link" contains the document URL. indicate text omitted for brevity. Differences in each pair highlighted.
Index
kstam.
Text
Generation
link Bakn, Azerbaijan, April Treed: Official exchange rate dollar and euro against Azerbaijani manat set 1,9255 manats. respectively. for Apai 15. Below are the rates of Azerbaijani manat against world currencies, according to the data from the Central
1361
0.1505
Hank of Azerbaijan for April 15, [na] Japanese 100 JPY 1.587 New Zealand dollar NZD 1. 1.553 Fallow Tiend on Tolegram. Ouly inseresting and important link Bake, Azzrhuijan. March manats, respactively. for March 15. Below are the rates Azerbaljani mannt against world currencies, according to the data from the Central Hank of Azerbaijan for March 15. Iod 100 Japanese yen 100 JPY 1,5200 I New Zealand dollar NZD 1.16 rollow Trend on Telegram. Only
Toan 2072973
mom 0.3534
Officiaal exchange noe the US
earo against Aterbeijani monst was set
interesting important
Generntion 21998
infl 0,0218
link NEW DELHI India agticulural peodaction and cconomic growth Asia's third buggest cconomy, Where the farmland lacks irggation. Monsonn rainfall
average
mins
weather office Monday, which should suppoct
expected to be 96 percent of the long tern average. M. Rapeevan, secrelary at the Ministry Earth Sciences. sold a news conference. The India Metsoeological Department (IMD) defines average, normal, rainfall herween 96 percent and 104 percent of a 50-year average India's weather
centimeters
four
beginning every live yeurs over past two decades, even after taking into accoum has forecust accurately
wall update
first week
However, average,
enarr band of plus minas pereentage points.
Train 326212
mem 0.1555
link o NEW DELHI (Keuters) India likely receive average monsoon rains 2018, the weather office said, raising the possibility ligher
growth Asin's thid--iggest ccononyy 97 percent long-term average, K.J. Kameen, director general
the farmland lacks irigstionn Monsoon rains, lifablend stare-aun
county's : tillion econonyy are expested
Meteorological Depariment (IMD), sold news conference. "We see very less probabiary of deficis monsoon." Ramesh said on Monday, lid on inflation, potentially tempting Prime Minister entire weather office detines
Other than lifting fans and wider economic growth, spell of good rains will Narendra Medi bring fowward general elactions cue percent and percent of 3. average a global trading firm. Average monsoca raiarall will help India catain its position as the world's diee exporter.
2019.
nommal, rainfall
season beginning June. I-] Mumbai denler
Examples of Train-Generation Pairs at Different Influence Ranges
In table 3, we show examples of train-generation pairs sampled from different influence ranges. The patterns generally follow the train-validation pairs shown table 2, although many of the relations due to some form of templating.
K Examples Sampled at Different Level of Memorization
Figure 13, Figure 14, and Figure 15 show full examples from RealNews sampled at high, middle and low memorization value ranges, respectively. Similarly, Figure 16, Figure 17, and Figure 18 show examples from C4 sampled at high, middle and low memorization value ranges, respectively. Figure 19, Figure 20, and Figure 21 show examples from Wiki40B:en sampled high, middle and low memorization value ranges, respectively.
L Example Pairs Sampled at Different Level of Influence
Figure 22, Figure 23, Figure 24, Figure 25, and Figure 26 show train vain-validation example pairs from RealNews sampled from high to low influence ranges. For each pair, we show the validation set example first, and then show the corresponding training example with a difflib generated visualization of textual difference with the training example.
Similarly, Figure 27 and Figure 28 show train-validation example pairs from C4, and Figure 29 and Figure 30 from Wiki40B:en.
We also show train meineeeenertiion influence pairs between RealNews training set and Grover [Zellers et al., 2019] model generation in Figure 31, Figure 32, and Figure 33.
21

---

Index
Link and Text
2090855
0.6646
httt:://www.jta.com wtw.:////r/////////
THE AMERICAN JEWISH CONGRESS ANNOUNCED TODAY THE PUBLICATION OF REPORT ON JEWISH NON- CON-EMPLOMMENT AS A RESULT OF ECONOMIC
DISCRIMINATION, AND ALSO NATION--WID PROGRAM TO DEAL WITH THE PROBLEM. THE PROGRAM OUTLINED IN THE PAMPHILET INCLUDES ESTABLISHMENT OF BUREAUS IN COMMUNITIES WHERE THE CONGRESS HAS BRANCHES TO DEAL WITH DISCRIMINATION, DEVELOPMENT OF PROGRAM OF EDUCATING EMILLYEERS LEGISLATIVE CORRECTION OF AGENCY ABUSES, ELIMINATION OF QUESTIONS ON RELIGION FROM EMPLOYMENT APPLICATION BLANKS AND FORMATION OF ADVISORY COUNCIL CHRISTIAN LIBERALS. THE REPORT, THE SECOND ITS KIND, WAS PREPARED RABBI COHEN, CHAIRMAN OF THE CONGRESS'S COMMISSION ON ECONOMIC PROBLENS, AND IS ENTITLED "HEL PING TO END ECONOMIC DISCRIMINATION" IT SURVEYS CASES OF DISCRIMINATION AGAINST JEWISH JOB-SEEKERS AND WARRSS
"SOCIALLY DANGEROUS." TELEPHONE AND TELEGRAPH COMPANY, WALTER S. GIFFORD, AT THE LAST NATIONAL CONFERENCE OF PRESIDENT OF LOCAL COMPANIES, ADVISED HIS ASSOCIATES AGAINST PERMITTING PERSONNEL MANAGERS TO REJECT JEWISH APPLICANTS BECAUSE PREJUDICE, AND SHORTLY THEREAFTER ONE OF THE DEPARTMENTS OF A.T.& T." ALMOST UNPRECEDENTEDIY ENGAGED JEWISH APPLICANT.
REPORTS THAT PRESIDENT AMERICAN
799
0 0.6324
https:://www.eer express. cu. ok/ hwtp///ppoooooooo /ttp:p///oteersoott////////// (anrs//o////GGGnnrrrro racinn---ninnn -00-11--00-111
Greyhounds jostle position meet Newcastte ROMFOND FANCIES 10.31 Jogadusc Jasper (1--2) 10 Selkirk (4-31) Forest Fella (2-4-1) Strawberry Hall
[3-5-0] 6) 11.31 Litys Rocket (3-1-5) 11.44 Dmopys Sian (3-4-0) 11.58 Dudleys Lady (hap) [4- 6-5) 12:7 Giffys Girl (3-5 5-6) 1..33 Cleniann Raso (3-9-2 17 Dakes Wish (3:2-5) 1.04 Caims
Rebel (5-4-1) Borna Karma (5-3-5) 1.33 Tolo (5-2-4)1 1.49 Alrita Parther (1226))
FANCIES 38 Zonas Hoather [hap) (1--a) 13.33 Ballybane Lena (1-5-5) 11,07
Wonder (1-5-3) 11.23 Whitefield Maeve (1-3-4 11.37 Denwill Andy (1-2-) 11.51 Mountcashel jess [1-4-3] 12.08 Neulawn Sweep (1-5-3) 12.24 Bay Wonder Oscar (1-2-3 12.39
Londsbury Rollie (1-4-3) 12.57 Thirsty Rooney(1-2)
Little Thriller(1223 2-33 1.27 Deniarto [1--3] NEWCASTLE FANCIES 2.24 Wanssbeck Nagple (2-1-6) 2/22 Cregg Rock (6-1 2.58
Steel Ranger (33--3 3,38 Hilkiew Holly (5-6 6-2) 3.37 Westoe Lass (6-2-1 1/.53 Carpi (1-32) 6.14 Sleek loe (2.5 225-1) 4.28 Blackhouse lace (2-5-4)
of Glory (3-6-3) 5.07
Royal Cracker (6-4) 2) 5.27 Seacht Gorm 16-3-33 5.6 Bohard Lad (1-4-2) 2) 5.58 Dark Milly (4-5:3 5--53)66.66 Dereks Hope (6-1-) RESULTS HOVE: 1151 Vivendi Katle 5-2f (1-4-3 BAGS F E20.06
TC E5194). 11.28 Tallulah Belle 7-4 6-3-1 F13.86 #6116), 11A2 Alfateelldiggg ggar 4-1 (D-55 E28A8 E109.66) 11.57 Clane Hondo (6 5-6 F20. 55 12 ES0 12:2 Que Katie 4-1
[6-53 E52.27 TC 558 94), 12.27 Guinness Joker 5-1 (6-1-4 823,7 TC 86749). 12 Cabra Bolt 5-2j (6-4-1 #28,39 TC #94.26. 12,58 Calano 7-44 [f-4 E13.75 1C 649.06). 112 Westway
Triangle 6-4 ET0A4 E3496) 1.27 Jazz Diamond (4-3-1 E16.38 (oA.A4) 1.4 Ladies Watch (4-5-1 162.05 ET12.80,, 158 Albion
2-1jt (6-1-2 E3697 E72.66).
Placer C1 stake (Races 1-6) ESAB 00, (Races 7-12) E340.00. MONMORE: 238 Ardera lack 7-1 (2-4-1 BMG F E40.24 TC C11638), 2.37 Hellroad Rodge 7-2 (2--- E16.22 TC L57 32), 2.5)
Whitehart Wonder (1-455 514.57 642,03)
Antarctic 7-2 (6-21 E17 053 3338 Shinto --jt 5-6
TC (70.50)
Marcelino 16 4-4-2 E30.55
196,977, 408 Coomenn Fly 7-2 (3-52 E1634 TC ESS.51), 4.37 Rock It Ramon 3-1 (6-4-3 F12.96 TE 647.67) 4.57 Cambo Exa 5-1 (5-6-3 F24 TC F74.8). 5.77 Linsgarbi Anna 4-1 (5-4-3
ED:50 EX3U80) 3.33 Paidis Peata 4-1 (6-6-1 E34.4 ET12.90, 91), 5.77 Ardera General 4-1 (6-5-2 #15.91 IC E34.9), 6.04 Seathwaite Dave (5 138.06 839966. 8.8 Scmoby
Blue (1-5-6 C19.0 TC (52m), Placer Et stake (Races 1-6) E253.00, (Races 9-50) 5930.00. OXFORD: 11.0 00 Chapelane Lucy 7-2 (1-6-3 BAGS F 125.0 TC (71.9). 11.99 Beckys Druidin
5:12:55 5-1 EMA.A TC 1128,91) 11.34 Headfford Royal --ft (2 5-1 89985 665.60) 11A8 Geneva Cove 3-1 (2 5-4 24 IC 289A2). 12.04 Ring The Chief 3-1 (4-133 E1L,7 TC No)
Rock Magic 6-4f 11-5:6 E11 38 TC 645.51) 12.33 lisoycasey Jose 4-1 (6-42 817.5 67166), 12.47 Brykat Lola (5-3-4 $15.9 IC 253.59), 1.04 Ballymar Tish s-jtt (1-64 #15.21
It 844.1) 139 tizmo Senor 5-1 (1-5-3 650.34 IC s138.2), 1.3 call in Paris 8-1 (5-922 44 IC E98.54 1,51 Killbank Printer 4-1 (3-455 #25.01 TC Es3.87), Placer stale (Races 1-6)
E354.00, (Races 1-12) E526.00. SWINDOW: 2.08 Unleash Fidel 2-20 F E11/ 1C (30.8)) Rushty Dusky 10-101334 84210
2085736
0,5755
httpss:/ffus9:ccom /2016 /twp/////eppp/////eee pegteeeeeeeprr---ebbbbeeeee soup-with- wwwppppwwwwwwoooo--oooo wonton-noodles/
RECIPE: Chinese Park Vegetable Soup with Wonton Noodles Chinese Pork Vegetable Soup with Wonton Noodles 1 pork tenderloin fabout 1-1 pound size), cooked and cut into Ys-inch cubes* cups lower--oddum chicken broth 1 cup water*
rice wine vinegar 1 tablespoon lower sodium soy sauce heaping teaspoon very finely minced garlic 1 heaping teaspoon grated fresh ginger 14 teaspoon black pepper 1 can (15 ounces) whole baby com, drained and cut into quarters 2-4 ounces fresh snow peas, halved crosswise 2/3 cup carrots, thinly biased sliced 2/3 cup sliced mushrooms 1/3 cup chopped green onions 8-12 wonton wrappers to 3.5 inch size), cut into Y-inch strips and separated (so they don't stick together) Combine chicken broth, water, rice wine vinegas, sauce, garlic, ginger and black pepper in large saucepan or Dutch oven, Bring a boil. Stir in baby corn pieces, snow peas, carrots, mushrooms, onion wonton strips. Return boll, then
heat, cover simmer about 3 minutes or until vegetables are crisp- tender, Stir in cooked park cubes and heat thoroughly. "Roast pork tendertoin uncovered 425"F oven until instant read thermometer registers 145"F labout minutes roughly) Let pork tenderloin at minutes before cutting, Makes servings (about 1 1/5 cups each) Recipe by ParkBelnspireed.om with adaptations by culinary dietitian& nutritionist Kim Galeaz, RDN CD
1680600
0 0,5807
httpps:// www. aus. lawuuaggggggeeeaoo larabic/ /en 220.://///////////1/////////
acknowledgement country
Language English العادة التجمايت مhvمs yod التصيية العلل الليl Bagg 2008 aار التين اليين غني البااا اامناعe| ش التحيية فب الططة WA السااا ااياااي الSا Rymeyy fyn duه اitiب elady الكرااس البين gدتا 2010 عاع uoly الهان الناه فo اللااية acknovledgement of country ردال S-g-21 a die ال) الSا) uo (igl 1 الSكك 3 do بعم alan a lasii ف المارر oo Gounly الليي مديز of e tual si d,de بي الروتدككلل (3 Sprel الللية ka الطل a fancialle dil would like acknwlledge that this meeting being held the traditional lands of the (appropriate group) people, and pay my respect elders both past and present" التومي ع Luo wاk اللl مo الdه واااتبيy العااباا التسية اننس اللكمي القينة اسد ا اتتاام piy ail الأقهت منy تسية si Lillee Amgeus bap LayS القيااة - Kija. Acknowledgement of Country الجي الأبب يالسل uiك pain lál (ليرار i السي غبر in أغراء se ani الااي ai واك 4ade anliness Le Welcome to Country التبيي دااي العننيي o- الساارن الساان on ا-سي uا 3و5 اننس عادe 4ath واتت الدرو الاايا hasi hoold i d5 damin تقوم أن we ا Litsyin Sym الن -لل العان السااوي الس.كا مباا wiin باارراا للكسب الحااي Jلae Jlsin! براتر y dolle الالل Smoking ceremony- lhis Ve., المكر Welcome to Country Hello my name [name of speaker] representative Elder of the [insert organisation or local Ind Indigenous groupl. I would like to begin by paying my respect to the local Indigenous people [or insert name of Indigenous people]. the tranational custodians this land where we are meeting upon today. On behalf of the traditional custodians
[local Indigenous group's namel welcome all. كطا عاا Jui. J انsل Jأو as ون Welcome Country' د الهاا الككدة اقاقه us uuS. Jás UوS. 1999 ع السعار &a ممك اماااية تعدد فلر اللعو واللاات las 34 is 3 فددناااا الجارة ملا w Uن اسرر Uن ووكك LeتL عان Loke 175 مريد Vitil رky الااادي و تمية تكبر الااد الاحاويا Acknowledgement Country ااملنةة عن S4i اليه ااتبي الايا ااار في عجوي بن kanny ملت
Figure 13: Text examples from RealNews with high memorization.
22

---

Index
mem
Link and Text
2074805
0,2835
http://www.chrisinnpos..coom wwwp///wwwwwwweeeeeeeeeeeeeeeeee peeaa--tudentt--uuppeeee hetpssppnnsooooo------------ -er--ant-gg---mmr-rreeeee
Texas honors student punished saying homosexuality was wrong had suspension rescinded after meeting with mother her attorney. The Ye-year-old, Dakota Ary, from Western Hills High School of the Fort Worth Independent School District was initially given a suspension of day in school and two days full. After hearing about the suspension the mother, Holly Pope, reached to Matt Krause the Liberty Counsel be her sont' legal representative. Matt Staver, founder and thairman of the Liberty Counsel, told The Christian Post that he believed Western Hills High and
made the correct decision reversing their course of action. 'The decision rescind the suspension is the correct one. The suspension impropec" said Staver. "I applaud the student for standing up. We stood with him to resist an unjust suspension and we are pleased that suspension has been reversed." Dakota was in German language dass Western Hills when class conversation shifted to the issues of religion and homosexualitty The student reportedly said that 'being a homosexual was wrong" Upon hearing his remark, the teacher proceeded take administrative action. Ary himself is an honors student, noted for being a member of the high school's football team and a volunteer at his church. The rescinding of the Fort
suspension allous him play an upcoming football game. In publicly released statement, does not comment on student matters, In keeping with this, Fort Worth ISD did not return comment to The Christian Post by press time. "As a matter of following district policy our
Independent School District said that it
press
course, Fort Worth ISD does not comment specific employee or student related issues. Suffice it say that review of the circumstances and any resolution will likewise be in accordance with district policy," said Fort Worth ISD. Staver also told CP that in recent years incidents like this one only increasing as the gay rights movement continues advance in American society. "Unfortunately, we have seen similar situations. The homosexaal agenda is aggressive and intolerant to people who not agree. Liberty Counsel will continue the right to exercise freedom of conscience and religion," said Staver. "These instances increasing and will continue increase unless Christians and people who love liberty stand up and resist this intolerance."
710814
0.3336
http://innreuters.co fartidle /idiNindia $572562010320
MANAMA Bahraini security forces detained the outspoken head of the main human rights group early on Sunday, colleague said, days after crackdown that drove mainly Shifite protesters
street. want to update you that they arrested Nabeel Rajab 1.30 am and the police came to my home and I'm not in my home right now," Said Yousif of the Bahrain Center for Human Rights said in an email sent ovemight. Rajab had spoken to media about the crackdown
the Bahrain Center Human Rights put statements this week accusing Bahraini forces and their Saudi and Emirati allies of "massacrre"" including some using Apache helicopters. Bahraini state television has said that Bahrain does not own Apache helicopters, Four protesters and three police been confirmed killed in Wednesday's crackdown, Neither Rajab nor Yousif could be reached by phone early Sunday, but some Twitter feeds suggested Rajab had already been released after questioning by police. Rajab's arrest would bring to 10 the number of people believed to have been arrested since the crackdown began Wednesday. Among the others opposition leaders who had called for the overthrow of the monarchy and doctors who had complained of excessive use of force against protesters. (Reporting by Frederik Richter, Writing by Lin Noueihed: Editing by Matthew Jones)
1263261
03553
http://wwdd.com haccessorcss hwwps//haaaa///seeeeeeeo /stella dot- expands sxpands-handbagr handbag selection-6840000
commerce-based based accessories company, will add to its offerings on April with full range of handbags. The nine piece collection will retail from $22 for cosmetics case to $138 for a weekender bag, Founder and chief executive offecer Jessita Herrin called this the nearly 10-year-old company's 's "launch a lifestyle brand"" as well chance double market opportunity. and (ewelry each account for approximately $10 billion." Herrin told WWD. "We see the same opportunity bags as we did in jewelry giving women on the go a simple way chic." Other styles include $59 technology tote and a $128 convertible bag that has zippers up the side to let the wearer decide she wants a sleeker or more expanded silhouette. A small handbag collection
Stella Dot, the social commerce
billion accessories market
U.S. bags
that serves double duty a dutch wallet, $39 jewelry roll, classic $89
cruss-bodi and two convertible cross- body to clutch styles but this the first significant push for handhags geared for daytime use. With sales expected to surpass $200 million this year- business grew from $175 million to $200 million from 2011 2012- Herrin projected the new category will take in about $25 million through the end of the year. in addition to the brand being carried exclusively online at stelladot.com, the brand is sold via. 14,000 active stylists around the world. "We're re're really investing in this as a huge growth category. We just opened dedicated in- house design studio for this in Sausalito [Calif]], Stella & Dot chief creative officer Blythe Harris said. She added that the bags comprised of a combination coated canvas, waffle poly and Saftiano leather come in ikat, snakeskin and colorful multistripe
released in 2011- containing
Herrin added: "Broadening our accessories line by launching the new and massive category of bags allows us to make that opportunity for stylists bigger than before. It's ground floor opportunity with a proven company."
854230
0,221
httss//wwwwwbbwwwssggno...om /news/ /local, /Man Outside Home Fatally Shot By Resident Police Say 46856444...hm
Police said man was shot to death outside a Chillum, Maryland, home by the homeowner. Newsy's Darcy Spencer reports, A homeowner in Maryland shot and killed a man who he caught breaking into his vehicle hours after Christmas Day, police say. Sources tell News4 the shooting, which occurred an area repeatedly hit by car break-ins, appears to be justified, Deontae Parker, of no foxed address, died after he was shot the day early Tuesday outside home in Chillum, Maryland, Prince George's S County police said, He was 32. Police said Tuesday that they were investigating what happened. "We're still working through all the details to deter what led up the shooting" Cpl. Tyler Hunt said. A resident of home on the 6200 block of Sligo Parkway called 911 about 2:15 a.m. Tuesday to report a shooting Officess arrived and found man with a gunshot wound to the upper body. He was pronounced dead. According to the preliminary investigation, detectives believe the homeowner who police have not identified shot Parker from inside home after Parker "had broken into the homeowner's vehicle; police said in an update Wednesday morning, Newså footage shows bullet hole in a window.
the s surveillance system, "which shous Parker's actions prior the shooting" second suspect is believed have run from the scene before police arrived. Parker and the homeowner did not know each other, police believe. Sources told Newss the homeowner has years, and that about 10 people were inside the house the time of the shooting, including children. neighbor who asked NewsÀ not to identify him said his car had been broken into multiple times. The Prince George's County State's Attorney's ss Office will determine whether any charges should be filed against the homeowner. Anyone with information the case is asked to call 301-772-4925, Anyone who wants remain anonymous can call 866-4M-TIPS,, go to www.pgcimesoovers.com .com or use the P3 Típs app.
Police now have video
lived there
Figure 14: Text examples from RealNews with intermediate memorization.
23

---

Index
mem
Link and Text
449808
0.0361
Investors in Digital Realty Trust, Inc. ( DLR) saw new options begin trading this week, for the February 2014 expiration. At Stock Options Channel, our down the DLR options chain for the new Fehruary 2014 contracts and identifted one put and one call contract particular interest. The put contract at the $45.00 strike price has a current bid of $1.00. If an investor was to sell-to-open that put contract, they are committing purchase the stock $45.00, will also collect the premium, putting
YieldBoost formula looked
cost basis of shares at $66.00 (before broker commissions). To an investor already interested in purchasing shares of DLR, that could represent an attractive alternative to paying $49. 02/ share today. Because the $45.00 strike represents an approximate 8% discount the current trading price the stock (in other words it out-of-the- moneeyy by that percentage), there also the possibility that the put contract would expire worthlesss The current analytical data (including greeks and implied greeks) suggest the current odds of that happening are 74%. Stock Options Channel will track those odds over time see how they change, publishing chart of those numbers on our website under the contract detail page for this contract. Should the contract expire worthless, the premium would represent a 2.22% 13.98% annualized at Stock Options Channel we call this the START SLIDESHOW: Top YieldBoost Puts of the REITs Below chart showing the trailing twelve month trading history Digital Realty Trust, Inc, and highlighting green the $45.00 strike
return on the cash tommitment,
located relative to that history:
1157311
0.0356 https://wwwtthester
Investors Abercrombie Fitch Co. (ANF) saw new options become available today, for the April expiration. Stock Options Channel, Boolt formula has looked up and down the ANF options thain for the new April 4th contracts and identified one put and one call contract particular interest. The contract the $34.0 strike price has a current bid of $1.97. If an investor was to sell-to-opes tol------pnnnn that put contract, they are committing to purchase the stack $34.00, but will also collect the premium, putting the cost basis of the shares at $32.00 (before broker commissions). To an investor already interested in purchasing shares ANF, that could represent attractive alternative paying $34. /s/aare today. Because the $34.00 strike represents an approximate 2% discount the current trading price of the stock other words is out of the money by that percentage), there also the possibility that the put contract would expire worthless. The current analytical data (including greeks and implied greeks) suggest the current odds that happening are 57% Stock Options Channel
track those odds over time see how they change, publishing chart of those numbers on our website under the contract detail page for this contract. Should the contract exgire worthless, the premium would represent a 5,79% 49.23% annualized at Stock Options Channel 500 x Below a chart showing the trailing twelve month trading history for Abercrombie & Fitch Co., and highlighting in green where the $34.00 strike
return cash commitment,
this the
SLIDESHOW: Top YieldBoost
the S&P
located relative to that history:
155144
0,0708
https://wwwwfforr www. wwwoooooo//i//eeeeeeeeeeeeeeeeee finvesting/geneee geneeaal 22122 08/ in/nattontt-ssstuumnttss beats on revereeeeeeeee matches exp.aspx
National Instruments (Nasdar; NATI reported earnings on Aug. 3. Here are the numbers you need know. The 10-second takeaway For the quarter ended June 30 (O2), National Instruments beat expectations on revenues and met expectations on earnings per share. Compared to the prior-year quarter, revenue improved significantly and GAPP earnings per share stayed the same. Margins dropped across the board. Revenue details National Instruments reported revenue of $292.3 million. The five analysts polled by S&P Capital IQ expected a top line $279.7 million on the same basis. GMP reported sales were 15% higher than the prior-year quarter's $253.3 million. Source: S&P Capital IO, Quarterly periods. Dollar amounts in millions, Non- G4AP figures may vary to maintain comparability with estimates, EPS details EPS came in $0.21. The four earnings esti mates compiled by S&P Capital IQ anticipated $0.21 per share. GAAP
Capital IQ, Quarterly periods. Non- GAAP figures may vary to maintain comparability with estimates. Margin details For the quarter, gross margin was 75.8% 210 basis points worse than the prior-year quarter. Operating margin was 12.0%, 160 basis points worse
$0.22 were the same the prior-year quarter. Source:
the prior quarter. Net margin was 9.0%. 150 basis points worse than the prior- year quarter. Looking ahead Next quarter's average estimate for revenue is $288.1 million. On the bottom line, the average EPS estimate is $0.24, Next year's average estimate for revenue is S114 billion. The average EPS estimate $0.96. Investor sentiment The stock has five-star rating (out of five) Motley Fool CAPS, with 387 members out of 408 rating the stock outperform, and 21 members rating it underperfomm. Among 154 CAPS All-Star picks (recommendation by the highest ranked CAPS members), 149 give National Instruments& a green thumbs-up, and five give it red thumbs- down. Of Wall Street recommendations tracked by S&P Capital 10, the average opinion on National Instruments is outperform, with an average price target $32.50. Can your portfolio provide you with enough income to last through retirement? You'll need more than National Instruments. Learn how to maximize your investment income "Secure Your Future With Rock-Solid Dividend Stocks. Click for instant access this free report.
1257699
0.0643
https:///ww.food/ http://wwwwwooo//////oooom www.:/ooo...///i//sttng.goo /ww/eeitinggggnneeeeee 2201 (2222) hewttggg-hathhhree--ee negaivvega
Merge Healthcare (Nasdaq: M MRGE) reported earnings on Feb. 15, Here are the numbers you need to know. The 10 10-second takeaway For the quarter ended Dec. (Q4), Merge Healthcare met expectations revenues and earnings per share, Compared the prior-year quarter, revenue imsroved significantly and GAAP earnings per share dropped to a loss. Gross margins ns expanded, operating margins grew, and net margins shrank. Revenue details Merge Healthcare booked revenue $641 million, five analysts polled by S&P Capital IQ foresaw net
of $64.0 million the same basis. GAAP reported sales were 39% higher than the prior-year quarter's s $46.2 million. Source: S&P Capital IQ, Quarterly periods. Dollar amounts in millions, Non- GAAP figures may vary maintain comparability with esti matisates. EPS details Non-GAAP EPS came in 50.04, The five earnings estimates compiled by S&P Capital IQ forecast 50.04 per share on the same basis. GAAP EPS were $5.011 for Q4 against $0.0 per share for the prior-year quarter. Source: S&PP Capital iQ. Quarterly periods. Non GMPP figures
maintain comparability with estimates. Margin details for the quarter, gross margin was 66.3%, 730 basis points better than the prior-year quarter. Operating margin was 13.7% 500 basis points better than the prior-year quarter. Net margin was -2.0%, 2,300 basis points worse than the prior-year quarter. Looking ahead Next quarter's average estimate for revenue is $66.3 million. On the bottom line, average IPS estimate is $0.05. Next year's 5 average estinate senti centiment The stack has a four-star rating (out of five) at Motley Fool CAPS, with 101 members out of 121 rating the stock outperform, and 20 members rating underperform. Among CAPS All- Star picks frecommendations by the highest ranked CAPS members), 31 give Merge Healthicar green thumbs-up, up, and three give a red thumbs-down. Of Wall Street recommendation outperform, with an average price target of $8.13 Over the decades, small-cap stocks like Merge Healthcare have produced market- sarket-bbeting returns, provided they're value priced and have solid businesses. Read about a pair of companies with lock on their markets in "Too Small to Fail: 2 Small Caps the Government Won't Let Go Brake." Click here for instant access this free report.
revenue $289. million. The average EPS estimate $0.29. Investor
trac ked by S&P Capital IO, the average opinion on Merge Healthcare is
Figure 15: Text examples from RealNews with low memorization.
24

---

Index
mem
Link and Text
324899
0.7560
UB808884680-80
00O OO0 00088000 0OO dOÚ68RA
08880000 ÚÚ888888888 UOU 00088685680688 BÜ Ū DÚÚ UOU UOU UOU UOU UOU UDO UÚÚ UÚÚ 680600Ú OOÚ UOU ÚÚÚBRŪDÚ Ú ODO ODO ÚDÚ 000 000 DDO 000
000 DOO DOO 000
000 000 O 000
000 DDU DDO DJO 000 0O0 000 000 000 000 0 0 000 0OU OD0 DOO D'0 DOÜBRA O00SRO00 6088000 D0ORR000 000KR6 000 000 D O 000 DOO 0OU 000 DDO 000 000 000 QUO 000 QD0 000 DOO DOO 0 000 DOO D00 0.0 000 000 000 000 00O UDO D00 D0O 0G0 000 0 000 000 000
O0O O0O DOO O0Ú 0OÚ ÜOÚ ÜOÚ ÜÚÚ ÚOÚ OÚO 00Ú Ō 6Б888558 000 000 885588 000 000 05888888 OÚÚ 00Ú 00Ú 8805888 ÚOO ÚOO O ÚÚ
88880366686 8088688896 6880886 068 68338068668 886 05888586556 8858056 00
00000000000000000000000000000000 98J883860AR998 SESSRRREEARRSRR
001.0/00000000000000000000000000
PRE PARET NL EASH D!! Vla Reet Release Date 16/01/2010
Command v3.0.0.0 RIP АКGА (ALGЛААGЛ66о
Protection: Serial Release Type: RIP 23x5.00MB Company Arcen Games
MM605MM (15.0000106611000000000 (AL.0000000000000000000 (at.0a/000000000000000000a0 RIS with aspects of 185, tower defense, and grand strategy. features single or cooperative play with as many 8 humans against a pair of powerful, intelligent Als. These AIs are driven by Al Progress stat that players contribute to through aggressive actions such as taking control of planets and destroying key units, forcing tough decisions regarding which targets are worth capturing or destroying, Human fleets are usually several thousand ships planets, Some games have reached
Release Notes: All War an entirely unique large-scale
larger, with games often featuring 30,000 units any given across galaxies anywhere from as many as 90,000 units! Despite this, players retain powerful control over their fleets in a macro-senge sense, ignoring painful micromanagement present in other games favor of actual tactics like flanking. Mcanwhile, ships themselies attempt target enemies intelligently as possible so player focus on the tasks that relly matter. Scouting also plays a major role the game, supplying an intel 'nnasshol' about important capturable units, enemy fleet sizes, and planet ownership. Designed by RTS veterans and backed by strong, responsive support on the official forums and elsewhere, War is an exciting and refreshing experience. RIPPED: Nothing Install Notes: Unpack the release with WinRAR equivalent Install with A A/War 3.0.00........ e.0. Use the included keygen to generate valid serial to register the game Enjoy! Remember support good video game developmentt (AA.AAtAtAttt www.hrtt11111...00.... WAW WWW.HAAAAAAAAAAAAAAAA competition! Ra51000110000000110000
htt.kaatwttttttttttttttttataaaaa Unleashed salutes all friends of the family, contacts
010.0000000001000011111111141144 siigndd but not pleased!"
2075649
0,6390
https://ebsaattdomainnnnggdd/ee
Vespa: cвeтовeh пиjер 8 nроиssaaстеооо нa cкyтери иичиииируанно придвымване, Аеtаλ иодeни Ha CTMBHO превоано среaство, преднаанннено nacовии nasap. vespa.bg is safe website. This information is from Google, AVG Threat Labs, MoAfee SiteAdvisor, Wot. Alexa traffic rank shows the popularity of your site relative to other sites. Vespa.bg is ranked 8,030,55880 558th in the world (among the 30 million domains). A low- numbered rank Matotexowmaa Eé Oricanose Mсториn Pasаитиn, Plezas, recambios y boutique de vespa primavera peespizaos! Bij vespaparts vindt u alles voor uw vespa! van vespa gadgets en vespa boeken complete vespa scooters! wij hebben een groat assortiment onderdelen en accessoires voor alle vespa modellen! What websites are linking to Vespa. bg Website load time is an important factor, because Google is taking the site's loading speed into consideration determining its ranking, Even though this will not have a big impact, is still something we (webmasters) should really look into. The reason pretty simple- the majority visitors are usually rush and no one is fond waiting a century before the website finally loads content or fails to load, At the last check on 2017-04-30, website load time was 17.32. The highest load time is 26.32, the lowest load time is 14.73, the average load time 19.84.
that your website gets lot visitors. The queries driving traffic vespa.by from search engines. Morogonnepai Cryrepos
2080878
0,5603
http://reposstoryuiin swwka.askkas.eas.a ac /iewe http///eeers//c///oo 3SSINA-3A-3A-3AAtm
Items where Author is "ASKINA, ASKINA, (2013) PENERAPAN STRATEGI PEMBELAJAR LAJARAN AKTIF TIPE INVESTIGASI TIM UNTUK MENINGKATKAN HASIL BELAJAR SISWA MATA PELAJARAN PENDIDIKAN AGAMA ISLAM KELAS SEKOLAH DASAR NEGERI DOS SIALANG KECAMATAN BANGKINANG SEBERANG KABURATEN KAMPAR. Skripsi thesis, Universitas Islam Negeri Sultan Syarif Kasim Riau. This list was generated on Thu Apr 25 19:8::22 2019 WIB.
2088697
0.4794
Color Enhancing Blood Orange & Passion Fruit Shampon enhances red tones red, light brown, and dark blonde hair. Apply shampoo wet hair. Lather, leave on 3-5 minutes & rinse. Water|Aqua//aal Sodium Laureth Sulfate, Cocamide MIPA, Cocamidopropyl Betaine, Giycol Stearate, Citrus Aurantium Dulcis (Orange) Fruit Extract, Passiflora Edulis fruit Extract, Ribes Nigrum (Black Currant) Fruit Extract, Lycium Barbarum Fruit Extract, Cocos Nucifera (Coconut) Oil, Argania Spinosa Kernel Oil, Gardenia Tahitensis Flower Extract, Laureth- 12, Laureth-23, Guar Hydravyropyltiiooiimm Chloride, PEG/ PPG--4/12 Dimethicone, Palyquaternium-1, Citric Acid, Tetrasodium EDTA, PEG-150 Pentaerythrityl Tetrastearate, PEG-6 Baprylic/Capric Glycorides, Ethythexyl Salicylate, Ethythesyl Methosycinnamate. Butyl Methorydibenzoylmettaa Imidazoldiny) Urea, Limonene, Fragrance) Parfum, Orange (CI 15510), Red 33 (CI 17200).
PEG-40 Hydrogenated Castor Oil, Glycerin, 4PG-26-Buteth-22 Polysorbate 20,
Figure 16: Text examples from C4 with high memorization.
25

---

Index
mem
Link and Text
2001601
0.2974
https://www.enwiiommn/ www.:oo.........u....... hwwp///ww///ww/g//e///ee selfie spotting from citizen scientisss
Citizen scientists from acress Australia and the world have helped researchers count Illawarra's elusive spotted- quoil, Environment Minister Gabrielle Upton said March, "Six months ago we asked ditizen scientists to jump online and help us analyse more than 80,000 photos taken from across the Illawarr region,' Ms Upton said, 'More than 300 volunteers answered the call and we now confirm at least 20 individual qualls call this a great milestone
celebrate World Wildlife Day!" The Quollidor project funded under the NSW Goverment's $100 million Saving our Species project and is providing fascinating glimpse into the hidden world of this endangered carnivorous marsupial.
region home,
ditizen science project
motion sensor cameras the region collect 20,000 to 30,000 images every weeks that enthusiastic citizen scientists have sorted through. "We have images of quolls posing for setfies, exploring the camera and jumping on and around the monitoring station," Ms Upton said, Quolls have huge home ranges and mave across the landscape making it difficult otherwise monitor their population and behavioural patterns. "It's great to see people being able to use technology to make real contribution the conservation of NSW's unique animals," Ms Upton said. The
spotted-tailes quoll the only remaining quall species the state. The project ongoing and will help the NSW Government increase the resilience and size of the local quoll population.
2045446
0 24557
http://ww.bbrllyooheeeeeee
2010
html
Yesterday's trip to local Le Chatelain, was welcome reprieve, and I took the opportunity to sample new beer, Floreffe Blond. Hard believe haven't had everything on offer there yet, but that's Belgian pubs for your even run-of-the-min one will have or so types. Le Châtelain probably closer to 60. Florefle's dark beers, Roreffe Dubbel and Floreffe Prima Mellor, are included in my list of top ten Belgian beers, so was expecting great things with Floreffe Blond. wase't disappointed. floreffe Blond is lighter than normal strong blond ales, It's lemony, summery. Fiona says it's smooth and has a bubble gum taste- - "that powdery. old-style bubble gum" - but I'm sticking with citrus. It has very little bitterness, but it's not sweet. At 6.3% alcohol, it's not brain bending either. Which is just well, because the wood treatment is doing my head plenty enough.
1795710
0,2217
Baleggoo////iicce Ciwiicieettsssspsee...s. a5px2110=1669
EVERETT, Wash., April - Snohomish County's 's Department of Planning and Development Services (PDS) today announced an on-line survey to gather public input for the Southwest Urban Growth Area Boundary Planning Study, Obtaining feedback from those who live, work, and play in Snohomish County about their quality of life and improvements they would like see is at heart of everything the county does. The Southwest Urban Growth Area (SWUGA) Boundary Planning Study (BPS) is no different. a nigh-level study of existing conditions and opportunities and constraints which can inform future planning choices, As part of the BPS, an online survey is available until April for community members the SWUGA BPS area, and outside the area, to give their feedback The study area includes lands currently designated and zoned rural uses extending eastward from the SWUGA to Broadway Ave, land use or boundaries are proposed for Malthy UGA or any other portion of the Boundary Planning Area as part of this study, The area has been chosen for study given is adjacent fast-growing portion the SWUGA. The study provides opportunity to develop comprehensive information environmental, sacio-econon economic, infrastructure cost, and policy implications to inform future planning efforts. Using technical information and public study can identify what currently works well what doesn't the SWUGA BPS (e.g. roads within and beyond the study
to Cathcart
the county
includes the Maltby
Growth (UGA) for context,
key center, changes to
outreach results, area). The study will also identify opportunities and constraints under different future growth scenarios in the currently designated-rume lands in the study area. This includes a scenario with change to existing land use and zoning designations in the project study area. The scenarios allow the county to consider implications for environmental, social, capital/infrastrcttus finfrastructure, and fiscal conditions. The study itself not proposing actions to change land uses, zoming or UGA boundaries, Information generated in the study will be available when future planning initiatives are developed the SWUGA BPS, please visit study webpage.
county such as the Snohomish County Comprehensive Update. For more information
1130970
03636
httpp:://www.eee http://wwwwwwwowot....... orwg//ggg//u
The International Republican Institute (IRI) has deployed a Long- Term Observation Mission to observe pmparations for the Moldovan parliamentary elections scheduled for February 2019. Long-term observers (LTOs), were accredited by Moldovan Central Election Commission on December 11, 2018, are be based in Edinet, Ungheni, Orhei, Anenii Noi, Hancesti, Comrat and Chisinau, IRI observers will remain on the ground to observe the pre- election preparations, Election Day the post-elete election period, concluding their mission on March 26, 2019. The mission the assistance eaneccee of three analysts who will provide te-th analysis of the media, electoral and legil landscape. The experienced LTOs represent 11 different countries: the U.S., Poland, Canada, the UKK, France, the Philippiiee, Uganda, Portugal, Montenegro, Slovenia and Georgia. IRI has organized more than 200 international election observation missions around the globe, earnin a reputation for impartiality and professionalism. The IRI mission to Moldova is funded by the United States Agency for international Development and will conduct its activities strictly independent and nonpartisan basis, without interfering in the election process and conforming the laws of Moldova, Click here for more information on IRI's work in Moldova.
Andrea Keerbs, with
Figure 17: Text examples from C4 with intermediate memorization.
26

---

Index
mem
Link and Text
653722
0,0119
http://odessahighhhhoo.ogrg /odesssss///aaaoooooooooo mo, mo/alumni/ 7950336 /lewis thompson.html
This a placeholder page for Lewis Thompson, which means this person is not currently on this site. We do suggest using the tools below to find Lewis Thompson. You visiting the placeholder page for Levis Thompson. This page here because someone used placeholder utility look for Lewis Thompson. We created this page automatically in hopes Lewis Thompson would find it. If you are not Lewis Thompson, but are an alumni of Odessi High School Odessa, MO, register on this site for free now.
1498720
0.0087
http://aaaheinhighschoo...r wttp://orgaammmmm//2///0/0 11120068/aaiddaaaddiinnttt baldwin.html
This a placeholder page for David Baldwin, which means person not currently on this site. We suggest using the tools below David Baldwin, You are visiting the placeholder page for David Baldwin, This page is here because someone used our placeholder utiliity to look for David Baldwin, created page automatically in hopes David Balchin would find If you David Baldwin, but alumni Anaheim High School, register on this site for free now.
1481511
-0.0018
At Shower Replacement Guys, our experts understand the process of Shower Replacement. This is important training that ensures that your tub or shower so that it distards the old, worn out and outdated look for a breathtaking and refreshing replacement shower doors. The process starts with removing panels, loosening the jambs, removing the bottom track, installing the side tracks and leveling the track. They will then expertly fasten the jambs, expand the holes for the panels, installing back the bottom track and the storage column, As the process nears the end stage, our experts will then fasten the jambs to the column before restoring the top track. Once the shelving units are expertly restored, the caulk frames will be put place and finally, you will have the door back in place. Talk to our experts in Shower Replacement Guys in South Gibson, PA so that you can learn more on how all these stages be carried out efficinnlly and conveniently for your Shower Replacement needs. One of the requirements for quality services our South Gibson, PA that services must be sanctioned by the authorities. We uphold provision of quality services that meet the needs and requirements of people so that they can enjoy services provided without fear of substandard services. We have complied with all the provisions of the so that you get superior quality services, As result, you South Gibson, PI, You can therefore approach us for our services with full confidence that you will get the quality services. We guarantee you that you will
compare favorably with other providers Shower Replacement services
trusted and reliable shower door replacement services from Shower Replacement Guys South Gibson, This replacement shower doors will transform your experience with your shover or tub. will spot new look replete with elegance, style and an air of sophistication never experienced before. Call us 888-398-0577 and discover the extent to which your old tub shower will be changed from unpleasant look most appealing and attractive finish unprecedented.
-0.0033
http://wwwwwwrrpaggggss.soo ///////// wil/shoerr-eeeeeeoo
At Shower Pan Guys, you will find that our personnel have the requisite training, skills and expertise in the design of the Shower Pan liner Shower Pan. Whether you are looking for flexible shower liner, PVCOne liner, prefabricated shower Line,, roll membrane among other options, you will be home with experts to advice you which Shower Pan is most suitable for your shower. There some specific details that only professionals handle. The reason why this issue important is that you would to use shower liner water proof environment or other fexible options available. You will require people with expertise to guide you on all these. Thank fully, we do not run out of these exceptionally skilled people to assist you in making the right decision. Shower implement nt your proposals in an effortless way. You can be guaranteed of fully bonded, insured and licensed service in Greenbush, WL, We take our work installation
Guys Greenbush, WI, your shower liner needs will catered for a most professional way, personnel will
seriously and therefore ensure that the requirements set by local government installer. We do not like putting our clients at a disadvantage with leaking Shower Pan liner such that they cannot enjoy their bathroom experience. As follow the right procedure in ensuring that you get the full worth
adhered The safety
good the
per rules and regulations, installment stage. We also rank favorably with other similar service providers in Greenbush, WI, Have the confidence to approach at any time for a quality and high standard services. You will find that from Shower Pan Guys Greenbush, WI, we help alleviate the shipping cost for tustomers, For purchases worth some predetermined amount, you will receive shipping to your destination.
investment from procurement the final
more the amount purchases that will enable you to receive free shipping services, Shower Pan Guys in Greenbush, Wi can be contacted over 668- 670--600-3300 to have any aspect the shower pans dariied to you way will understand. Issues such procurement. quotation, design, inspection, installation, repair and maintenance will be explained to you so that you understand the proceduur, You will be able to reach us through email or the online option. Our personnel will be very helpful you so you make right decision regarding your requirements.
on 888- 988-670-3340
Figure 18: Text examples from C4 with low memorization.
27

---

Index
Link and Text
2091427
0,5121
wikidata wilkiata_id-04o66me jd-04060703, version_ 04-36777499002272227
START ARTICLE, Rasim Alguliyev _TARR SECTION Monographs (4) START PARAGRAPH Ad6acos A.M., Anrynwen P.M,, Kacywos B.A. пробпеwы инерораацннной безопасности коипиотерннаа censx, Baky, "3nw", 1998, 235 c._ NEWLINE, -Arrynkes RM. Tеорелииесие OCHOвM построения виртуалыных частник cereñ neperтaииванно струитурой, Mocusa, "WCA PIH,, 1999, 106 c,_ NEWLINE, AAnrynee P.M. Merogu cikrreaa араптлнны Cистеи обеспечениа ыииорpааиооннйй безонасности корпоратиенаа cereil, Mocusa, "YPCC", 2001, 248 <__NELLINE_ Alguliyev R.M., ABayev N.B.
Aliuuliyev R.M. Plagiatiqla mūbariza texnologiyalan. Balu, "informasiya Texnologyyalan,, 2015, 165
2084824
A990
wikidata id=Q1017392, version (d-88377655522222263300
START_ARTICLE,
START_ ARTICLE_ Malebranche (Divine Comedy) _TART__ PARAGRAPH. The Malebranche (Italian: [male branke]: : "Evil Claws") are the demons the Eighth Cirdle (Malebolge). They figure 101, XXII, Vulgar
Comedy who guard Bolgia
quarrelsome, their duty is to force the coruptt politicians (barrators) to stay under the surface of boiling lake of pitch.
2085791
0
wikidata, id-02893296, version_ (d-8125155592214488557
WWART_ATTICLE_ Beautiful Soup (HTML parser) _TARRT SECTION, Code example _TTRR__ PARAGRAPH, At/usr/bin bitpphthonntt Anchor extraction from html documentfrom b54 import BeautifulSoufrrom request import urlopenwith urlopen""ttpss:/eer httttp:////wwiiieediiaa orgi wilki/ Main, Puge' as
response: NEWLINE soup BeautinlSoupprrspoose, parser) NEWLNEE anchor in foupffind alll'aa,, NEWLINE_ print/anchorgeet/hree Y'))
2083026
0,4600
wikidata id=03599668, version (1:140037398851222473
START ARTICLEE 320 mm Model 1934 naval gun START SECTION M1934 guns _TARR_ PARAGRAPHL The Conte Cavour- class battleships Conte di Cavour and Giullo Cesare originally mounted
as built 1915 and 1914, These ships rebuilt between 1933 and 1937 removing the Q triple turret guns while Giulio Cesare carried Pattern T guns built Elswick Ordnance Company. Odero Terni Orlando (010) re bored the Vickers guns while the Elswick guns were bored by Ansaldo Maximum gun elevation was increased 27". STARTT SECTION_ M1936 guns STARR_ PARAGRAPH The Andrea Doria class battleships
amidships and replacing guns triple A turret, twin and turrets, and triple turret. Conte Cavour carried Vickers
Andrea Doria and Caio Duilio originally mounted 13 guns as built in 1916 and 1915, These ships were similarly rebuilt between 1937 and 1940 by removing turrets, Pattern T guns aboard Doria were rebuilt by OTO. Maxi mum gun elevation was increased to 30%, _______R___EC_C SECTION_ Ammunition _TARR_ PARAGRAPH The gun was loaded with four cloth bags each containing 44 kilograms (97 lb) of smokeless powder, High explosive (HE) shells weighed only kilograms (1,010 lb), Anticipated useful barrel life was 150 effective full charges (EFC).
turret replacing guns B,X,
Dullio similarly rebuilt Ansaldo while guns aboard Andrea
2065942
0A448
wikidata id=Q161622, version (d-1006675223390555000
_TART, SECTION Mechanism STARR, PARAGRAPH, EDC couples primary amines to carborylic acids by creating an activated ester leaving group. first, the carbonyl of acid attacks the carbodiimide of EDC, and there a subsequent proton transfer, The primary amine then attacks the carbomyl carbon of the acid which forms tetrahedral intermediate before collapsing and discharging the urea hyproduct. The desired amide obtained.
_SARRT_ ARTICLE,
Figure 19: Text examples from Wiki40B:en with high memorization.
28

---

Index
mem
Link and Text
1834612 0,2829
wikidata id-044612, versionnedd- -we.ddd---1666122222660588
START_ ARTICLE, 2003 Polish Air Force Mi-8 crash _TART PARAGRAPH On 4 December 2003, a Polish Mi-8 helicopter operated by the 36th Special Aviation Regiment carrying Poland's Prime Minister Leszek Miller crashed near Plaseczno, just outside Warsaw. The pilot performed autorotation landing in a forest following the failure of both engines The helicopter suffered extensive damage and was written off as total loss, but despite the severity of crash there were fatalities, Fourteen the 15 people board were injured, including Leszek Miller, who had vertebrae broken. _SARRTSSE SECTTOOO Aircraft and pilot START_ PARAGRAPH_ The helicopter was 26 years old at the time of the crash, and was close to the end service, belonged the 36th Special Aviation Regi responsible for transporting Polish government officiass The pilot of he helicopter was major Marek Milosz, later promated to lieutenant colonel. START SECTION Cause START PARAGRAPH The cause of the engine failure was determined to be icing. _START _START_SECTION, SECTION Trial _TARR_ PARAGRAPH, On 10 March 2004, Milosz was criminally charged with violating flight safety rules and causing crash. Specifically, the pilot was blamed for not manually turning on the deicing equipment during the flight. The pilot argued that the meteorologic information available to him at the time did not indicate that icing was likely, and hence he was not required to turn the deicing
his thoracic
equipment. He was consulting thermometer during the flght,, but suffered from systematic measurement error and hence iding, In addition, during the flight an unusual thermal inversion occurred; the temperature rose with altitude, which the pilat could not have predicted._ NEWLINE In March 2010, the 6-year trial ended difficult autorotation landing and that the passengers survived because of his superb piloting skills. Leszek Miller declared that if he had to fly again in helicopter difficult atmospheric conditions, he would choose Mioosz his pilot.
unable warn
verdict of not guilty judge the noted pilot expertly carried out
610
0,2664
wikidata jdd-jjd0-0000011 version_idd309331111 (d-3099117932250011922
START ARTICLEE Inga Bejer Engh START, SECTION, _TARR_SECTION, Early life and career STARR PARAGRAPH Engh educated Candidate of Law. Fallowing law school she worked with international law for the United Nations New York City, Upon return to Norway her first job Drammen District Court later Oslo District Court and Asker and Barrum Police District. At 32 she began working as a prosecutor. She has prosecuted several major criminal cases including the much publicized methanol distribution case in dssffldd in the early 2000s, case where a man received ten years prison after having thrown acid his wife, several major drugs and sexual abuse cases, She also prosecuted Tore W. Tvedt, one of witnesses the Breivik case, 2002.
_STRRT START_SECTION, SECTION_ Personal life STARR_ PARAGRAPH, Inga Bejer Engh married and has two young bays. The yourgest vas born in 2009 and was very premature
sustained no lasting medical problems, decision become lawyer was made the gymnasium (Nonwegian equivalent high school, now called videregående).
173213
0,2511
wikidata_ wikkdata_id-04405522 id=054085254, version_ 06-10099330267722222288
_TART_ARTIIE ARTICLE_ RTV silicone __STAS__AARRASSS PARAGRAPH_ RTV Silicone (Room- cemp-rremperttrreee Vullannii- silicone) type silicone rubber made from two- companent system (base plus curative, A+B) available in hardness range of very soft to medium-usualll from 15 to 40 Shore A. RTV silicones can be cured with catalyst consisting of either platinum tin compound such as dibutyltin dilaurate. Applications include low-temperaturre over-molding, making molds for reproducing, and lens applications for some optically clear grades. SSAARTTTTCCCCTA SECTIOSO Applications _START_ PARAGRAPH To produce the material, the silicone rubber is mixed with the curing agent vulcanizing agent. Usually, the mixing ratio is few percent. In order for the RTV silicone to regroduce the surface texture, attention is paid to cleanliness of the original. Vacuum de-airing removes entrained air bubbles silicone and catalyst to ensure optimal tensile strength, which aflects reproduction times, In casting and mold-making RTV silicone rubber reproduces details and is suitable for a variety of industrial and art related applications including prototypes, furniture, sculpture, and architectural elements. RTV silicone rubber can be used to cast materials including was, Sypsum, low melt alloys /eetaas and urethane, epoxy or polyester resins (without using innovation the ability 30 print RTV silicones, RTV silicones industrial applications include aviation, aerospace, consumer electronics, and microelectronics, Some aviation and aerospace product applications are cockpit instruments engine electronics potting, and engine gasketing RTV silicones are used their ability to withstand mechanical and thermal stress. _START_SCTIOONN SECTION__ Advantages and disadvantages START PARAGRAPH. RTV silicone rubber has excellent release properties compared to mold rubbers, which is especially an advantage when doing production casting resins (polyurethane, polyester, and epoxy). No release agent is required, obviating productoottion cleanup. Silicones also exhibit good chemical resistance and high- high-tempprature resistance (205 CC 400 and higher). For this reason, silicone molds are suitable for casting low-meit metals and alloys (egg zinc, tin, pewter, and Wood's metall. EWLLI INE_ RTV silicone rubbers are, however, generally expensive especially platinum cure. They are also sensitive to substances (sulfur containing modelling clay such as Plastilina,
the mixed
release agent). more
example) that may prevent the silicone from curing (referred to as cure inhibition). Silicones are usually very thick (high viscosity), and must be vacuum degassed prior to pouring to minimize bubble urethanes polysuinddes, shorter than latex). Silicone components (A+B) must be mixed accurately by weight (scale required) or they do not work. Tin catalyst silicones shrink somewhat and do not have long shelf life. Acetoxysilane- based RTV release acetic acid during the curing process, and this can attack solder joints, causing the solder detach from the copper wire.
entrapment. making a brush-on rubber mold, the curing time factor between coats long llonger
1470633
03669
wikidata id=Q6697905, version id=977138113342299333
_STRRT_ ARTICLE_ Lucky landing Marina and Seaplane Base MARTTSECTION,_. Facilities and aircraft _TARR__ PARAGRAPH lucky landing Marina and Seaplane Base has one landing area designated 2/20 which measures 15, 3,000 x 4,000 ft (4, 572 x 1,219 m)) For the 12-month period ending July 31, 2006, the airport had 1,850 aircraft operations, average 154 per month: 86% general aviation and 14% air taxi.
Figure 20: Text examples from Wiki40B:en with intermediate memorization.
29

---

Index
mem
Link and Text
351056
0,0213
wikidata id--oo24220000000 version siriion.io-77722005999999888
START STARTLARTICLE, Mack Creek Village, Virginia START_PARAGRAPH Mack Creek Village is an unincorporated community in Pulaski County, in the U.S. state Virgimia.
771076
0.0200
wikidata, id-06673673, version, ieeoonndd2211110 id-201128801122447556
START_ ARTICLE Longdale, Virginla STTRRT PARAGRAPH, Longdale unincorporated community Alleghany County, Virginia, United States.
0,0115
wikidata wilidata_id--ssa jd-05551607, version_id--Seeee jd-1547755722248119997
_STARTAATII AATICCLLL German submarine U-295 STARTSSECTION Design PURR_PAHGH German Type VIIC/41 submarines were preceded by the shorter Type VilB submarines, had a total length of 6730 m (220 ft in), pressure hull longth 50.50 (165 ft in), a beam of 6.20 m (20 ft in), height of 9.60 m (31 ft in), and draught
U-295 had displacement of 759 tonnes (747 long tons) when at the surface and 860 tonnes (850 long tons) while su bmergged. She
submarine was powered by two Germaniawerft F46 four-stroke, six-cylinder supercharged diesel engines producing a total of 2,800 to 3,200 metric horsepower (2,060 to 2,350 kW: to 3,160 shp) for use while surfaced, two AEG GU 460/8 double-acting electric motors producing total 750 metric horsepower (550
ft
sho) for while submerged. She two shafts and 123 m (4 ft) propellers. The boat was capable of operating at depths of up to 230 metres (750 ft) NEWLINE The submarine had maximum surface speed of 17.7 knots (32.8 km/
20A mph) and maximum submerged speed 7.6 knots (143 km/ (b; 8.7 mph). When submerged, the boat could operate for 80 nautical miles (150
92 mi) at 4 knots (7.4 hy; 4.6 mphi; when surfaced, she could travel 8,500 nautical miles (15,700 km; 9,800 mi) at 10 knots (19 km/h: mph). U-295 was fitted with five 53.3 cm (21 torpedo tubes (four fitted the how and one the stern), fourteen torpedoes, one 8.8 cm (3.66 in) SK C/35 naval gun, (220 rounds), one 3.7 cm (1.5 in) flak M42 and two 2 cm (0.79 in) C/30 anti-aircraft guns. The boat had complement of between forty-four and sixty.
_STRRT_ SECTION_ Service history _TARR_ PARAGRAPH The boat's service life began with training with the 8th U boat Hlttilla October 1943. She was then transferred the Rotilla for operations on August 1944, She was reassigned to 13th flotilla on October and moved again to the 14th flotilla on 1 April 1945, STARTTSTRRRR SECTION 1st and 2nd patrols START PARAGRAPH U-295% first patrol was uneventful, NEWLINE She then embarked on of short journeys between Bergen, Kristiansand, Stavanger and Trondheim. NEWLINE Her second foray, most successful, She damaged the British frigate HMS Mounsey east northeast of Murmansk on November 1944. _STRRT SECTION_ 3rd and kth patrols Norvegaa Seas, She returned Harstad 18 December
Trondheim Harstad was
_START_ PARAGRAPH_ The submarine's third sortie took into Barents
1944, NEWLINE Her fourth patrol started Harstad and finished in Narvik. She had spent three days off Murmansk, to no avail. _TTAR _ETART_SECTION Sth patroi _TARR_ PARAGRAPH Her fifth effort was just barren, even though it was longer. STARR_ SECTION, 6th patrol and fate _START PARAGRAPH, The boat departed Narvik on 15 April 1945, Her route took her once again to the Barents Sea. She returned to the Nordic port on 7 May. NEWLNE She was then moved to Skjomenfjord on 12 May 1945 and accordance with the surrender terms, she was transferred to Loch Eriboll in northem Scotland for Operation Deadlight on the 19th, She was sunk 17 December by the guns ORP Blysawica.
799605
0.0086
wikidata, Siiidata_d-:2024440 id=02024611, version_id-3666 71948882290115778899400
[ARRTAATICLE__ German submarine U- 528 _STARTTTREEES SECTION Design _STRRT_ PARAGRAPH_ German Type DIC /40 submarines were slightly larger than the original Type IKCs. U-528 had displacement 1,14 tonnes (1,26 long tons) when at the surface 1,257 tonnes (1,237 long tors) while submerged. The U-boat had a total length of 76.76 m (251 ft 10 in), pressure hull length of 58.75 m (192 9 in), beam of 6.86 m (22 ft in), a height of 9.60 (31 ft in), m ft The submarine
40/46 supercharged four nine- myi-dyr diesel engines producing a total of 4,600 metric horsepower (3,240 KN; 4,340 shp) for use while surfaced, two Siemens Schuckert 2 GU 345/34 couble-acting electric motors producing total 1,000 shaft horsepower (1,010 PS; 750 kW) for while submerged. She had two shafts and 1.92 ft) propellers. The boat was capable operating at depths of up to 230 metres (750 ft). NEWLINE The submarine had maximum surface speed 18.3 knots (33.9 km/h;
a draught
powered two MAN
213 mph) and maximum submerged speed of 7.3 knots (13.5 km /h; B& mph), When submerged, the boat could operate for 63 nautical miles (117 km;
72 mi) at knots (7A km/ ht: 4.6 mph): when surfaced, she could travel 13,850 nautical miles (25,650 km; 15,960 mi) at 10 knots (19 km/
mph). U-528
was fitted with six 53.3 cm (21 in) torpedo tubes (four fitted at the bow and two at the stem), 22 torpedoes, one 10.5 cm (4:13 in) SK C/32 naval gun, 180 rounds, and and loss, _TTRRT_ PARAGRAPH, The boat departed Kiel on 15 April 1943, moved through the North Sea, negotiated the gap between Iceland and the farce Islands and entered the Atlantic Ocean. There, she was intercepted by escorts of Convoy French Atlantic bases_ NEWUNE_ U-528 was 'destroyed on 11 May 1943 southwest of Ireland by depth charges dropped from Handley Page Halifax of No. 58 squadron RAF and the British sloop fleetwood NEWLLINE
C/30
cm in) aiti-ai gun.
complement forty-eight. START_ SECTION, Patrol
5 and damaged. She was sunk
way
went with the U-boat; there were 45 survivors.
Figure 21: Text examples from Wiki40B:en with low memorization.
30

---

Index
Estim.
Link and Text
valid
infl
vs robbers is race to find andient (ttpk//oooooooiiiiiiia Htttobttttttiissssssssooooo
1334662
0.3780
By Ari Rabinovitch TZEELIM VALLEY, Israel (Reuters) - The disposable paper face masks offer little protection from the clouds of dust that fill the dlfffside cave where Israeli archaeologists are wrapping up the largest excavation in the Judean desert the past half-century. Clipped into safety harnesses, volunteers stand at the cave opening, 250 meters (820 feet) above a dry river bed that leads to the lowest spot on earth, the Dead Sea. They sift through an endless supply of dirt-filled buckets, and the dust they throw the air reaches the far corners of the cave where dozen workers crawling on hands and knees can't help but cough, The three-week excavation was the first part of national campaign to recover as
artefacts possible,
desert some 2,000 years ago, before they are snatched up by antiquity robbers. "These looters that operate in the area are experts at finding scrolls. We go after them, look for what they are looking for and try to catch them," said Guy Fitoussi, head the Israel Antiquities Authority robbery prevention unit southern Israel. "This is game. Like and mouse." The Dead Scrolls, collection of ancient texts written on papyrus and parchment, have already been rescued by scholars. They are among the earliest texts written in the display the Istuel Museum Jenusalem as national treasure, Now Israel wants to desert hideouts before it destrayed or ends up on the black market. For a Wider Image photo essay on the excavation dlick http://reut.rs. 2520AZK pistol-packing archaeologist with authority to arrest looters, and his team catch about 100 them each year. Most are fined; some are sent jail. In 2014, they arrested six people who were plundering this particular caver, known as the Cave of Skulls, where seven skulls had been found from Jews of the Bar Kokhba rebellion against Rome the 2nd century. That raid, Fitoussi said, helped spur the multi-year government backed excavation program efforts at this site, about a two-hour drive southeast from Jerusalem. To access the cave, diggers don climbing gear and descend 20 minutes from their campsite along steep path that hugs the rocky diff. Inside, the grotto expands tunnels that extend deep into the mountain. The limestone walls of the dry desert cave are perfect for preservation, said Uri Davidovich, an 19-year-old volunteer, working flat her belly a dark crawl space, dirt
particularly scrolls,
behind by Jewish rebels who
Hebrew language and
whatever may remain in the
USTOO-PACKING ARCHAEOLOGIST According to israeli law, all relics found land or at sea belong the state. Fitoussi,
initial focused
square meters (1,720 square feet), including a number cramped
orchaeologist from Tel Aviv University who one of the dig's directors, mixed with sweat covering her face and digging with her fingers, unearthed a thin, 25 centimeter (10 inch)-lon rope that most likely was used by the Bar discovery, Davidovich said, They haven't found any scrolls yet, he said, the artefacts found in this cave,
Kokhba rebels. A rope this length and countless others nearby, will provide historians rare insight into how people lived 2,000 to 8,000 years ago. (Editing by Jeffrey Heller/Jeremy Gaunt)
train
men
https::/wwwwmmetr.o. www.t wwtp:////acchhlgisssssssso seaiissst------b----iiiee wobbers-in-istael-s sottaaet-------oo--aae----- -ene---nnnnnnn------eee ancient-scrolls//eee Serrrlls/kkgft---Hhhaa -HA.tttATTTT.._..nnnRnR
2077314
0,3780
By Rabinovitch@TZEELU VALLEY, Israel (Reuters) disposable paper face masks offer little protection from the clouds of dust that the diifsside cave where Israeli archaeologists are wrapping up the largest excavation in the Judean desert of the past half-century (lipped into safety harnesses, volunteers at cave opening, meters (820 feet) above dry river bed leads the lowest spot earth, the Dead Sea.TThey sift through an endless supply of dirt-filled buckets, and the dust they throw in the air reaches the far corners of the cave where a dozen workers crawling on hands knees can't help but cough three wwek excavation was the first part of national campaiign to recover as many artefacts possible,
particularly scrolls, left behind by Jewish rebels who hid in the desert some 2,000 years ago, before they are snatched up by antiquity robbers (Theee looters that operate the area experts finding scrolls. We go after them, look for what they are looking for and to catch them," said Guy Fitoussi, head of the Israel Antiquities Authority robbery prevention unit in southern Israel. "This is the game. Like cat and mouse." 'The Dead Sea Scrolls, collection ancient texts written papyrus and parchment, have already been rescued scholars. They among the earliest texts written the Hebrew language and are on display in the Israel Museum in Jerusalem as a national treasure /Now Israel wants to uncover whatever may remain in the desert hideouts before it destroyed or ends up the black market RRCHAEOLOGISTAccoorot
Wider image photo essay on the excavation click http://eutrrs
to Israeli law, relics found on or at sea belong to the state. Fitoussi, pistol packing
archaeologit with authority to arrest looters, and his team catch about 100 of them each year. Most are fined; some are sent to jail.Iin 2014, they arrested people who were plundering this particular cavern, known as
Skulls, where seven skulls had been found from jews the Bar Kokhba rebellion against Rome in the 2nd century That raid, Fitoussi said, helped spur the multi-year government backed excavation program and focused their initial efforts this site, about too-hour drive southeast from Jerusalem To access the cave, diggers don climbing gear and descend minutes from their campsite along a steep path that hugs the ricky cliff. Inside, the grotto expands 160 square meters (1,700 square feet), including a number of cramped tunnels that extend deep into the mountain [Th limestone walls of the dry desert cave perfect for preservation, said Uri Davidovich, an archaeologist from Tel Aviv University who was one the dig' s directors (One 19-year old volunteer, working flat on her belly a dark crant space, dirt mixed with sweat covering her face and digging with her fingers, unearthed a thin, 25 centimeter (10 inch)- shi--oong rope that most likely was used by the Bar Kolkhba rebels. rope this length was a rare distovery, Davidovich said (Thyy haven't found any strolls yet, said, but the artefacts found in this cave, and countless others nearby, aill provide historians rare insight into how people lived 2,000 to 8,000 years ago, ggdiiig by Jeffrey Heller/Jeremy Gaunt)
Figure 22: Validation / training example pair from RealNews with high influence. Red green
highlighted text indicate deleted added text in the training example comparing to the corresponding validation example, generated using Python difflib.
31

---

Index
Estim.
Link
valid
inft
838341
0,209
VATICAN CITY (AP)- Emeritus Pope Benedict XVI offering a first-eve ever papal assessment his own pontificate book recounts decision resign, his surprise at his successor and his attempts to dismantle what he calls the Vatican's "say lobby.' "Benedict XII The Final Conversations, is due September, the which has the books newepaper rights, provided a brief overviww Friday. Corriere said Benedict recounts in the book that he decided to announce his resignation Latin because feared making mistake Italian. He recalls his "surprise" that lorge Mario Bergoglio was elected pope and his at seeing Pope Francis mingle with crowds. Benedict also claims to have dismantled a group of four or five say prelates, dubbed the 'gay lobby" by the Italian media, who exercised power and influence the Vatican. The existence of this group gay prelates who purportedly used blackmail promote and preserve their interests has been mythologized Italian media. Soon after was elected pope and was asked about the so-called "gay lobby, francis quipped that he had to encounter priest who had "say" written his business card. That said, just this week gay monsignor who was offiiial in the Vatican's doctrine office, In 'The first Stone," Polish-bomn Kraysatof Charamsa recounts the absolute "obsession" with homosexuality in the functionaries who profess celibate
book-lengghh interview
Benedict conducted with German jouraalist Peter Seewald. Italian daily Corriere della Sera,
the Vatican and suspended as priest he came out, boyfriend by his side, published a book about his experiences as gay
halls
details the "hypocrisy destroy someone's s Vatican career by simply spreading gossip that he was gay.
live quite another, writes was enough
614881
train
0.3650
men
765687763/ Retired Retired-pop--eeeee pope-offers-Feeeeeooo weeee-offfs---s-teeeeeeee aese-seeee-ssesemettt of wwt---isow---p--- own-papacy.httll
Emeritus Pope Benedict XVI is offering a first ever papal assessment of his own pontificate in book that recounts his decision to his surprise his successor and his attempts dismantle what he calls the Vatican's "Byy lobby. "Benedict XVI: The Final Conversations, out in September, the latest lengt- interview that Benedict has conducted with German journalist Peter Seewald. Italian daily Corriere della Sera, which has the book's neuspaper rights, provided brief overview Friday. Corriere said Benedict recounts the book that he decided to announce his resignation Latin because feared making mistake in Italian. He recalls his "surprise" that lorge Nario Bergoglio was elected pope and his "joy" at seeing Pope Francis mingle with crowds. Benedict also claims to have dismantled a group of four or five prelates, dubbed the "gay lobby" by the media, who exercised power influence promote and preserve their interests has been mythologized in Italian media. Soon after he was elected pope and was asked about the so-called 'gay lobby, Francis quipped that he had yet to encounter any priest had "gay" written who was fired from the Vatican and suspended as priest after he came out, boyfriend by his side, published a book about his experiences as gay offiiial the Vatican's doctrine office. 'The first Stone" Polish-born Krayszoo Charamsa recounts the absolute "obsession" with homoseuuaity the halls of the Holy See. He details the "hypocrisy" of functionaries who profess celibate but live quite another, and writes that was enough to destroy someone's Vatican career by simply spreading gossip that he was gay.
VATICAN CITY
Vatican. existence
group gay prelates who purportedly used blackmail
business
That said, just week gay monsignor
rollow Nicole Winheld at www www.twiittrrcoo /nwinfilld
Figure 23: Validation / training example pair from RealNews with relatively high influence. Red
/ green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python di difflib.
32

---

Index
Estim.
Link Text
valid
infl 0.0673
682107
Pronger blocked six shots Detroit, 13,869 Ducks might have been lost without their programs on Wednesday night. It's only the first game of the preseason, but a new era has clearly begun. A group of mostly newcomers Ducks uniforms
ANAHEIM- night Francois Beauchemin two assists Toranto,
Phoenix 3-2 shootout Wednesday Honda Center. familiar Ryan scored two goals. Ryan also scored two goals in the Ducks' preseason opener last year, when he was trying to make the team's opening day roster rookie, His bid failed and Ryan forord to start the season
made the biggest impact, however, Bobby
American Hockey League, musily due salary- cap constraints. The circumstances could not be much different this year. Ryan finished the season as Calder Traphyy candidate, and began this season wearing temporary altemate captain's "X" on his jersey along with joffrey Lupul and Sheldon Brookbank. Nick Boynton, a 30 yea-eer--ld defenseman signed in the offseason, was the eldest player uniform. Of the 20 players chosen to play, only goaltender Jonas Hiller and center Ryan Carter spent all of last season with the Ducks, Hiller started in goal and looked sharp, stopping 18 of the 19 shots he faced. Timo Pielmeies, acquired in the trade that sent Kent Huskins and Travis to Jose, came hallway through the third period
stopped
23 shots before game went overtime. phantom
interference penalty sent Ducks defenseman Steve Eminger to the box, and set up power-play goal by the Coyotes' Lauri Korpikouski with 20.2 left in the period. korpikkowski deflected long range
by Keith Yandle, and hiller did well second period, poking in a rebound of a Dan Sexton shot past former Kings goalle Jason LaBarbera. With 1:08 elapsed in the third period, Ryan made it 2-1 breakaway shot that caromed in off Phoenix stick. Phoenix tied the game
get glove on Ryan got the goal back early in the
the third period, when Sheldon Brookbank made costly turnover front of his own net. Chad Kolarik batted down Brookbank's 'sss pass just a few feet in front of Pielmeier and tucked the puck in behind the goaltender.
train 494435
men 0.1429
http://wwww.dilbbreeeeee com 0000000////////////000////
welp-lead ducks exhibitioon-iin!
Beauchemin had two assists in Toronto and Chris Pronger blocked six shots Detroit, med-chris-aae Pronger blocked sie shets in betroitthn 13,869
Ducks Hight------- fans who showed at Honda Center Wednesday needed programs to identify players on their aave be-eeen-esttteee without their programs on Wednesday nightorite team) It's only the first game of the preseason, but a new era has clearly begun, group of mostly newcomers Ducks uniforms best Phoenix in shootout Wednesday
however Bobby Ryan scored two goals in regulation and another in the shootout Ryan also scored two goals in the Ducks' preseason opener last year, when he was opening day
although familiar made the biggest
trying to make due to salary-cap constraints. The circumsaances would not be mieh- this- yearore different now, Ryan finished [Heelss] season as Calder Trophy candidate, and began this season wearing temporary alternate captain's "A" on his jersey along with Joffrey Lupul and Sheldon Brookbank, Nick
roolee,
failed and Ryan was forced
season in the American Hockey League, mostly
Boynton, a -0-yrur-old defenseman signed in the offseason, was the eldest player uniform. Of the 20 players chosen to play, only goaltender Jonas Hiller and center Ryan Carter spent all last season with the Ducks, Hiller started goal and looked sharp, stopping 18 the 19 shots he faced. Timo Pielmeler, acquired in the trade that sent Kent Huskins and Travis Moen San Jose, came halfway through third period and stopped of
Shots before the game went evertmed phantor interference penalty sent Ducks defenseman Steve Eminger to the box, and set up power play
goal by the Coyotes' Lauri Korpikowski with 20.2 left the peried Korpik-kkkk delected wa---ooo-naeeeeeeee by Ketts and and-Hiller did
his glove on itopening period, Ryan got the goal back early in the second period, poking in rehound a Dan Seston shot past former Kings goalie
breakaway Phoenix tied the game a at 12:30 of the third period, when Sheldon Brookbank made costly turnover in front of his own net Chad Kolanik batted
lason LaBarbera the mank. With
elapsed the third period, Ryan made it
caromed off Phoenix
front Pielmeier and tucked puck behind the goaltendei, Chad Kelariki the shootout, Pielmeier stopped batted down Beookoth shots he faced, while Ryan and Joffrey Lupul bankeat Montoya on the Ducks p-pass just few--fe-eeeee feet in front of Dielmeier and tucked the puck in behind the goaltender two attemots.
Blonn Brookbank''s pass just few
Figure 24: Validation/ training example pair from RealNews with intermediate influence. Red / green highlighted text indicate deleted added text in the training example comparing the corresponding validation example, generated using Python difflib.
33

---

Index
Estim.
Link and Text
valid
infl
https://kdv..com 22019 t/t0 buttterall-uuukkyy-eeellle----ao pfoeettt------la--e-a pol-ptita--sammannnttete salmonellal
1165799
0.0360
More than 70,000 pounds of Butterball turkey recalled because of potential salmonella WASHINGTON - The U.S. Department of Agriculturees Food Safety and Inspections services announced Wedresday that approximately 78,16 pounds with Salmonella. Product numbers "EST. P-7345" inside the USDA mark of inspection are subject to recall. They were shipped to nationwide retail and institutional locations. RELATED: View full recall FSIS, the Centers Disease Control and Prevention and its public health partners are investigating multistate outbreak of salmonella schwarzengurn involving five case patients from two states, According to the USDA, Wisconsin collected three intact Butterball brand ground turkey samples from residence where four the patients live. Signs of salmonellosis, also known salmonella foodborne illnesses, appeat within one to three days after consumption of contaminated products. They include diarrhea, abdominal cramps and fever. The illness lasts four to seven days, According the USDA, most people recover without treatment, however, they suggest that diarrhea severe the person might need be contaminated. They also suggest those with weakened immune systems are more susceptible to illness, Consumers with food safety questions can "Ask Karen," the PSIS virtual representative available 24 hours day AskKKaren.go gov or via smartphone maskkaren.ge gov. The toll- Poultry Hotline 1-888-68 6-4---444 6854 is available English and Spanish can reached from
ground Butterball turkey might be contaminated
p.m. Monday through Friday. Recorded food safety messages are available 24 hours day. The online Electronic Consumer Complaint Monitoring System can be accessed 24 hours day. RELATED; USDA RECALLS
USDA
train 1571976
men
httpss//wreg.ccm// buutttrbtl recalls- meaarlly-0000 0000000pooddsee pounds of-turkey aftter-salmmnelllccae
0,2094
Bhen-BButteerbbbe Butterballl recalls nearly 80,000 pounds Butterbellkuukey turkey lecalled because potential salmonellaalt salmonella WASHINGTON
The U.S. Department of Agriculture's 's food Safety and Inspections service on a announced Wednesday that approximately 78,16 pounds of ground Butterball turkey might be contaminated with Salmonella, Product numbers "EST P-7345 inside the USDA mark inspection are subject to recail. The raw ground turkey was produced on July 7, 2018, They wollowere shipped nationing products under recall wide retail and institutional
lecationseere shipped to nationwide retail and institutional locations) BB oz. plastic wrapped tray containing "BUTTERBALL everyday Fresh Ground Turkey WITH NATURAL FLAVORING (85%, LEAN/15% FATY" with sell freeze by date of 7/26/18, lot code 8188, and UPC codes 22655- or 22655-7155 71557
represented on the label, 48-oz. plastic wrapped tray containing "BUTTERBALL everyday Fresh Ground Turkey WITH NATURAL FLAVORING (98% LEAN/7%I
FAT)" with sell
date of 7/26/10, code 8188
22655-71556 represented
label. 16-o2. plastic wrapped tray containing
"BUTTERBALL everyday fresh Ground Turkey WITH NATURAL F FLAVORING (85% LEAN/1S% FAT]" with sell or freeze by date of 7/26/11 lot code 6188 and UPC code 22655 71546 represented the label. oz. plastic wrapped tray containing "BUTTEBBALL everyday Fresh Ground Turkey
NATURAL FLAVORING
(98% LEAN/7% FAT)" with sell or freeze by date of 7/26/18, lot code 8188 and UPC codes 22665-71547 or 22655-71561 represented on the label 48-az. plastic urapped tray containing "Kroger GROUND TURKEY FRESH SH 85% LATE AN 15% FAT with sell or freeze by date of 7/25/ 18, lot code 8188, and UPC code
111141097993 represented on the label. 48-oz. plastic wrapped tray containing DFO0 LIN 15% fat ground turkey fithe--fll--caalitt natural flavorings" with sell freeze by date of 7/26/112 18, lot code 8188 and UPC code 3582609294 represented on the label See images the recalled product
labels here, ISIS, the Centers for Disease Control and Prevention and its public health partners are investigating a multistate outbreak salmonella Scchearzengaarrum involving five patients from two states According to the USDA, Wisconsin collecte three intact Butterball brand ground turkey samples from residence where four the pastent illnesses, appear within one to three days after consumption of contaminated products. They include diarrhea, abdominal cramps and fever. The illness lasts four to seven days. According the USDA, most people recover without treatment, however they suggest might need to be contaminated. They also suggest those with weakened immune systems are more susceptible to ilnness Consumers with food safety questions can "Ask Karen, FSIS virtual representative available 24 a day at Bskkaren.eu USDA Meat and Poultry Hotline 1-888-674-6866 available in English and Spanish and can be mathed from a.m. to 4 p.m. MDT Monday through Friday. Recorded food safety messages available 24 hours a The online Electronic Consumer Complaint Monitoring System can be accessed 24 hours a day. RELATED: USDA RECALLS
Signs of salmonellosis, also known as salmonella foodborne
diarhea severe person
via smartphone at maikkaren..o The toll-free
Figure 25: Validation/ training example pair from RealNews with relatively low influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python di difflib.
34

---

Index
Estim.
Link and Text
valid
infl
wnntraate--rrrri--n---------i tin-tmanetts-eeeiiiiiee newtons newtons-theory says-mantra-
1556405
0.0155
roded.html
Earlier January, Satyapal Singh had claimed that Charles Darwin's theory of evolution of man was 'scientifcaally wrong' and needed to be changed in school and college curriculum, (Photo: File) New Delhi: Minister State for Central Adviooyy Board of Education (CABE) on January and 16 said that mantras codified the 'laws of motion' much before they were framed by Issac Newton. He suggested that Vaastu compliane pliance educational buildings was important learning, according are mantras which codified "laws of motion' much before was discovered by the Newton. Hence it is essential that traditional knowledge must be incorporated curricutum," Singh was quoted saying the minutes of the meeting the government's highest advisory body for policymaking education. Eartier January, Satyapal Singh had claimed that Charles Darwin's theory evolution of man was "scientificlll wrong" and needed to be changed school and college curriculum, Singh said ancestors have nowhere mentioned that they
Development, Satyapal Singh speaking meeting
report Hindustan Times, "There
turning into man.
train 1903637
men
https:// http://wwsereetews.com com larticle 29058 MUBARAK SAYS HE WILL ISRALL---ISSSL-II--- SHOWSS-TSSSSS-IIIISS SHOVS-FLXIIILLLTT FLEXIBILTY..hmm
0.7927
President Hosni Mubarak said he is willing to visit israel the jewish state shows flexibility in promoting peace and agrees to take part in an international Middle Eadlec Januarst peace conference, according interview published Saturday am saying that am ready srael provided that this would lead to real progress on the road to solving the Middle East problem Sat" Mubarak told the Egyopal Singh had claiptian second
than week that Mubarak had expressed readiness make peace visit to Israel. In an interviey Darwinec, 24 with the Kuwaiti newspaper LAnba, Mubarak said he was prepared to travel to Israel the visit would promote solution of Palestinian problem and
newspaper Massa.The remed Charlesarks represented
Middle peace. The offer initially embraced by Israel, but later
Mubaral'ss theorsenior foreign policy of eaides said the volution man was isit was contingent on the jewish state opening negotiations with Pa lestinee Liberation Organization. Israel rejects the PLO negotiating partner despite the United States feentifccally wrong and needed to changed in sceee-hhhoeeeeeee college curriculum decision Dec. (Pheta-tile--ie Heeeee-------------------------- Hew--ehhii--eeeee Minihii--iiitt---eee of State ferate-fr--eee Heman-Hman------ Resource Development, Satyapal-Singh speaking st moeting- of the Canttal-Adiiisore Adwisoey Soard of dducation (CABE)en January 15 I-and- 16 said that montras codißed the llws---s-------- oosiiiion-------e- much Vaasiu compliance educational buildings
before they were Framed losar Neatonk open dialogue
group. also suggested
important for learning, according to a report in Hindustan Times. "There are mantras which codihed lavs of motion' much before it was discovered by eeeeeseeenalttee that traditionall wnwwledge must be incorporated
carriculum; 2 Singh queted saying by by-the minutes the meeting-sf th governments highest-advisssy bewissee--bbb---eeeee for polieymahing in edvestion fatter in fameory Sobyopal -iigiiiiniiieieeee seaist-h----ee dliined that Chariee--aaaiiiiss ssientiicallyy arong" and-i needed
theory avolution of nowhere mentioned that they saN an ape tuming into a man
school and collage curticllum Singh said ancostors
Figure 26: Validation / training example pair from RealNews with low influence. Red green
highlighted text indicate deleted added text in the training example comparing to the corresponding validation example, generated using Python difflib.
35

---

Index
Estim.
Link and Text
valid
infl
0.3755
Range Repair 92811 Expert repairs and services all types ranges, whether are need of electric repair gas range repair. your range is having problems the range surface Element won't work, range burner spark problems, range surface element turn off, range burners spark all the time, Range Repair Service will you right back where you need to be. Our range repair technicians most range parts. Expert Appliance Repair have your range repaired serviced
will have your range up running
and you cooking.
train 2059217
0,4466
Range Repair 928 Expert repairs and having problems like the range surface Element won't work, range burner has spark problems, range surface element won't turn off, range burners spark all the Range Repair Service will you right back where you Repair 928/1 will have your range repaired or serviced in no time flat. We will have your range up running no time and you back cooking.
types ranges, whether you are
electric range repair gas range repair. If your range
be. range repair technicians carry range parts.
Appliance
valid
infl 0.2267
http://eennaffrooofeeeuueeeeeeee com wtwee/teeeeeeeeeeeeeeeeeeeeeeeee home valuttvallti
2P4113
Want know your in ENGLISHOOWN is worth? provide you home evaluation obligation. Simply fill in your information below and I'll get back you shortly with a detailed report of comparable homes in your area that have recently sold or are currently for sale.
mem
http::/ctyrralesttte.com ffree_ home, valuation.asp
28486
0,2073
Want know what your home in ITOWWNyyoooa worth? I'll provide you with FREE home evaluation at obligation. Simply fill your information below and I'll get back shortly with detailed report of comparable homes in your area that have recently sold are currently for sale.
valid
infl 0.0603
https://www.sccwayneeet hotels/ bww./beeccccceeeeeeeeeeeeee
South Carolina Hotels Beach Hotels Also SC Coastal Hotels us Hotels Want the beach? Below is accommodation list for all the beaches in South Carolina - plus descriptions what those beaches are Find hotels located SC beaches, intcuding Myrtle Beach, shown here. Our list starts South Carolina's border with North Carolina and heads south towards Savannah, Georgia.
train
mem 0,2299
https::/www.. http://wwwww.we.e..ctctmtt/o/tte hotels html
South Carolina SC Hotels SC Goastal Horels Also see: SC Beach Hotels Alko sees SC Coastal US Hotels Lus Hosall Nant to stay al beachlong the coast? Below an accommodation
all the beschecoastal areal South Carolina - plus descriptions what those Beacheareas are hotels located SC beachesalong the SC coast, infoc Mystle BeachCharlleston shoun here, Dur list starts at South Carolina's border with North Carolina south towards Savannah, Georgia
Figure 27: Validation / training example pair from C4 with high to intermediate influence. Red /reen highlighted text indicate deleted added text in the training example comparing the corresponding validation example, generated using Python di difflib.
36

---

Index
Estim.
Link and Text
valid
infl
http://aniiuurhneqeeeeiiiiii http://rrry.nno/////o///arra www.aanue-------eo-------- wwwidddplaaiium--00000 -oct----o---o--o---------------- minn--ut--iimonnd--etteeee
0.0316
This a 1920s Antique Art Deco Solid Platinum, 90ctw Old Mine Cut Diamond Navette Ring Diamond- Old Mine Cut- -GH VS1-VS2. 90% item been tested and guaranteed be PLATINUM,
excellent pre-owned condition. Please disregard two characters the title, they are used for inventory purpases, As courtesy, please notify us of any return. Always fast & free unless otherwise stated. Collectors Coins lewelry been family owned and products with unbeotable customer service. The item "190ss Antique Art Deco Solid Platinum, 90ttw Old Mine Cut Diamond Navette Ring' is in sale since Thursday, November 2017. This item is cateeoory "Jewelry Watches Fine Jewelry' Rings) Diamond": The seller "collectorsbyssll"" and located in Huntington, New York, This item can be shipped worldwide.
on Long Island, NY 1946. We
four brick and mortar locations offer the highest quality
train 504624
men 0.1665
http://golddroochpin.comm wwt.tiiiii..oo/itttaa...mm
vinttaeeestaaaaeeeeeeeeee eettttttttttteeeeeeeeeeeee satate__4k_sot yoklsoli__yelloww..o_m wellloyylllwololeo gowdd wtw.igoooddddddddreeeeee _aaeeed_redd___ lereeddrre_dddnne___eeeee orange_corall_sii shipp wtwpl__sippbbbocchhoo bro.cchppi..hhppp
Men's Women's 32mim (length) x 42mm (width) approx. lok; Item has been tested and guaranteed to be solid GOLD. This is 1920s Antique Art Deco
Solid Platirem- 90cta- old thine Eut Slamend katem vetse tiig Biamond-ry ry good pre Gld Mine cut 6------------- (ae--------99950000-- SeRSPLAoweed
ring ewce-eeeeeettteeeee pre-oweed condition-- here is slight chip on the coral. Please disregard the two characters at the end of the tille, they are used for inventory purposes, As a courtes,, please notily us of any return. been family owned and operated on Long Island, NY 1945, We brickand worttoo-otteeeeeeeeeeeeee locations and effer-tle highest quality products with unbestable eustomer-senwee service) The item 'Vintage Estate 19205 Antik solid Yellow Sold Carved Red Orange Coral Brooch Pinga
condition, TH0%iRlor-te has been-tested-and guaranteed to be-seldd Al
Alwaays free unless otherwise stated. Collectors Coins& jewelry
four
Navette Ring" is-i---------ee-- ip-in--lle-iinneeeese Thusot sale since Friday, Nevember 8, 20188 This item is in the category "Jowelry& Watches Fine Jewelry Fiin RingPins & Broochusi Trroches:Diammondd & Gemstones" The seller is "callectorshussel"" and is located
Been Solid platinum 90cbu Old Mine Cat
New York mis itam can be shppaed woddduida
valid
intl
http://www.sandwellggvwllll /into////////////a/ /aniio/////// 2200208/crmme prevention and emergencies/ 3t55/the community -ringr
278389
0.0156
request_a aavest_a_rrvvee a_reeeeeeeeweeeeeeee oorooon_neee reportsettsoote of anti social behaviour
Community Trigger gives victims and communities the right to demand action on problems with anti social behaviour (ASB) they have reported in the past, The Community Trigger can be by anyone who has reported ASB but feels no action been taken. The Community Trigger aimed at putting victims first and to hold agencies responsible for managing anti social behaviour account. Agencies including councils, the police, local health teams and registered providers of social housing who receive Community Trigger report will then need to conduct a case review. Who can raise Community Trigger and when? Anyone can raise the trigger on behalf the victim for example a family member, friend, carer, councillor, Member of Parliament or other professional person. It doesn't matter who you originally reported the ASB (the council, the police your landlord)- please use the Community Trigger form here on our website. What happens after Community Trigger is raised? When you complete the Community Trigger form, have received and let you know what will happen next. Your completed form will be sent to Sandwell Council's Community Safety and Anti Social Behaviour Manager, A member the ASB team will contact you, within working days. Community Trigger is not an alternative method of complaining about the service you have received. If you are not satished with the service you have received there is still an independent complaints process that you should follow. Feedback
we will contact you to
the Community Trigger.
train 1897523
mem 038837
http://wwww.redda.com insiigtss/ tedddal-tllls-viittnaant buildinnnniiiiiiiii-aaaaa--aa-aa aaarinnggg--tttg--ie stt.on--oomsiiiinnduttiiiiiiiiii wwwpiiiaaabbbbbebbebeeeeeeeeeeee co--sustainngowthh
FDI attraction, export growth and a thriving consumer market have been characteristics the Vietnamess growth story but what next for this emerging market in South- East Asia? In this video, we discuss the prospect of the Vietnamese economy, possible setback of its growth strategy and potential remedies, Communitapturing FDI spillovers build strong back bone local manufacturing industry Frigger gives-iettis and communities the right to demand action on problems with anti social behaviour (ASB) themay have reported the pastold key to sustain growth. The Community Trigger
can be uase--ddddl by aay-aa-yaeeeaal who has ragorted-bee ASB but foels no action has been- taken. The Community Trigyer is aimed at puiting Road more about the topic
in our Reddal Insights artide- Drivstimss first Aest-and--e to hold agencies responsible for managing anti setli--siiiieeeeeeeeeeeeeeeeeeee behavieurte te-aceount Agencies including
councils, the police, local health teams- and angistured providens of social housing who suceive- Community sesggur-pportt aill then need to conduct-a
case review, Who can raise Community Trigger and when? Anyone
trigger on behalf of the victim- example family member, friend,
carer, counciller Hember of Parliament or other prefessional person: it doesang Vietnam matter who you originally reported the ASB to (the councily
peee-polic---e or your-uur-uue landlord) please use Community Triggor-fosn forgoo--oo---eee
on caee-oo----eee beeiitss economic growth thhhttthhppppeeeee sastttthppppe----e after communitt Triggerds)
raised (ite2--Weeeeeeeeee yow--oopl------e-eoeo the Community Trigger form, we will contact- you to www-ss----eeeeeeeeeeeeeee have received 4 a-and lot saa---yoo--------- what--hha--ee will happen- nevi- Your
completed form will be seat to Sandvell Council's Community Safety and Anti Social Behaviour Manager, A member of the ASB team will contact you,
within two working days. The Community Trigger not an alternttiee method complaining about the service you have received. if you are not satished wish the seee--eeviceeeee you have received there is shill an indepedent complaints process that youst-yo-shuulee should follow- feedback on the use tf Community
Figure 28: Validation/ / training example pair from C4 with intermediate to low influence. Red / green highlighted text indicate deleted added text in the training example comparing to the corresponding validation example, generated using Python difflib
37

---

Index
Estim.
Wiki ID and Text
valid
infl
wilidata, id=Q44945100,, version version 54-19957700666666772444
196
0,2863
START_ ARTICLE_ 2017 Missouri Valley Conference Men's Soccer Tournament_ _TARRTSTTREEEES SECTION, Background _START_ PARAGRAPH, The 2017 Missouri Valley Conference Men's Soccer Tournament the culmination of the regular season. The regular season conference matches determine the seeding in the tournament, which determines the conference's automatic berth into the NCAA Tournament. All teams the Missouri Valley Conference, MVC, play each other once during the season. Teams play certain teams at home during even number years, and then will play those teams on the road during odd number years. Teams are awarded three points for a win, point for draw and points for loss. NEWUNNE in the event that teams are tied on points, first tiebreaker head-to-head record. that tiebreaker tied, goal differential is applied, followed by goals scored, then away goals, then RPL_ NEWLINE Missouri State won the regular season with 5-2-1 record.
train
men 0.3659
wilkidata jd id-Q65120533, version_ jir 1216899111988322222
2044546
START, ARTICLE, 2012 Missouri Valley Conference Men's Soccer Tournament START SECTION, Background STARR, PARAGRAPH, The 20178 Missouri Valley Conference Men's Soccer Tournament culmination the regular season. The regular season conference matches determine seeding in the tournament, which determines the conference's automatic berth into the NCAA Toumament. All teams in the Missouri Valley Conference, or MVC, play the road during odd
each other once during the season. Teams play certain number years. Teams are awarded three points for a win, a point for draw and no points for a loss. NEWILNE, the event that teams are tied on points, the first tiebreaker head-to-head record. if that tiebreaker tied, goal differential is applièd, followed by goals scored, then away goals, then record.
home during even number years, and then will play those
RPI_NEWLINNE Missouri-Sttte StateCentral Arkansas won the regular season with a 5:2-4
valid 131821
infl 0.1143
wikidata, id-02150068, version Bacoooddd11111 (d-095502524441222851
START ARTICLE_ HNLMS K XIV START SECTION Service history _START PARAGRAPH The submarine was laid down in Rotterdam the shipyard of Rotterda Droogdok Maatschappij on 31 May 1930. The launch took place on 11 july 1931. On July 1933 the bost was commissioned in the Dutch navy._ NwW.__EEWLNE_DDD On Tebruary 1934 XIV and K left the Netherlands for the Dutch East Indies. The route they took led through the Suez Canal. On September 1938 she participated in fleet show Surabayo. The show was held honor of the Dutch Queen Wilhelmina of the Netherlands who orlebrating her 40th year as head of state, More than twenty navy ships participated in the show. NEWLIIEE_EE In the var XIV sank several Japanese ships. survived the and decommissioned on April 1946, June 1946 was stricken.
train
mem
wilkidata siciaaddddde id-02789027, wewp://ovvoooooodd/sss/eeee 16-7299573312299811
1102988
0.1773
START_ARTICLE ARTICLE, HNLMS XVV START SECTION Service history START_ PARAGRAPH. The submarine was mosterdaaschee Droogdok Maatschappij on 31 May 1930. The launch took place on Huul December 1931.. On BJuikj0 December 1933 the boat was commissioned in the Dutch navy_ NEWLINE On February
Rotterdam the shipyard of
route they took led through the Suez Canal. On 6 September 1938 she participatels in fleet show at Surabaya. The show was held in honor of the Dutch Queen Wilhelmina of the Netherlands who celebrating herwas thai 40thh as years the head state. More than twenty ships participate@1 @ in the NEWLINE the war K XIV sank several Japanese ships, She survived the war and was decommissioned on 23 April 1946, June 1946 she was stricken-and sold for scrap December 1950.
Xl and K
the Dutch East Indies. the Netherlands
valid
infl
wikidata, iilkidata_id-g500ee id-0020992205, version_id--ioe (6--41339800087746645
28865
0,0631
START ARTICLEE Wanim Island STARR, SECTION, Geography STARR, PARAGRAPH The island has an area of 3.56 kmn, part of the Pana Tinani Group The island hilly, rising to m at Mt. Wanin_NNELLINE The island is 0.9 km south of Pana Tinani, and separated from with the Bulami Channel. _TART_ SECTIONT__ History __SAATTRRRRRAAA _PARAGRAPLLL The island was discovered in the late 18th century._ START SEART_SECTION, Population START_ PARAGRAPH_ At the census of population 2014, the island had 600 inhabitants, spread across small villages_NEELIINE NEWLINE The main town Bunbun, located on the northwest point.
train
wiltidata, id=025099204, version version_id:1136665551 (esoaaeed11818811119999999999999
1339144
0.1241
START_ARTIC AATTCLEL NanimNimos Island BL_SSSTRRRTTROR SECTION_ Geography PAAAGAAARGRAPH The island has area of 3.56 km², it part the Pana Tinani Group. The island is hilly, nsing to 19940 m at Mt. WanimNimoa_ NEWLINE_ The island is 0910 km sounorth of Pana TinanVanating, and separated from with the Bulami Channel. ______SRRSSSSS SECTIONC History __STARR_ PARAGRAPH_ The island was discovered in late 18th century. Baaulyys_sAR_SSSS SSEETSONTON Population START_ PARAGRAPH_ At the census of population in 2014, the island had 504395 inhabitants, spread across a6 small villages. Baaiagss_NNNWWW .NEWEEEE__TTT The main town BumbursSlluus, located
parsaithwees point.
Figure 29: Validation/ training example pair from Wiki40B:en with high to intermediate influence. Red / green highlighted text indicate deleted added text in the training example comparing to the corresponding validation example, generated using Python di difflib.
38

---

Index
Estim.
Link and Text
valid
infl
wilidata siciita_dd-sseeae jd-Q58932464,
127940
0.0348
START_ ARTICLE_ Alexis Gutiérrez _ECTION Youth career START_ PARAGRAPH, Gutiérrez a young age was scouted and joined Guadalajara's youth academy
then continued through Chivas Youth Academy successfully going through U-13, U-15, U-17 U-20, Until finally receiving attention to join Cruz Azul, Pedro Caixinha being the coach promoting Gutiérrez to the first team, _TARR SECCTON,,, Cruz Azul, _TTRR PARAGRAPH Gutiéérrzz made his professional debut the MX the 28 April 2019. He subbed in coach Pedro Cabxinha the minute which ended in 4-1 Win against Lobos BUAP.
1104664
0,1723
wikidata id=Q314993799, version, M4-930061977605511000
START ARTICLE Alesis GutiereeSriann Figuerad START_ SECTION Youth career _STRR_ PARAGRAPH_ GuttémeeFiuerri at a young age was scouted
joined Guadalajara Pumis youth academy in 2012 He then continued through Panaa Youth Academy successfully going through U-13, U-15, U-17 and Pedro CalxinhF
U-20, finally recelving attention to join Eruz Azulthe
Palendja being the coach promoting Sunterreaigueess first team. _START_ SECTION, Eail-AallPumass UNAM START_ PARAGRAPH Gutiérredf made his professional debut in the Liga MX on the 29- 28-6F-Aaalyy
6f July 2019), He started was-uubedd BUAPPachuca
Podro
minuteith the first which ended +0 Win against tabes
valid
infl 0,0154
wiliidata wilidatt__d-0029993333 id-020994443, version_id- 001.555665111
48624
Chiva, Valencian Community, Tarin joined Barcelona's youth categories 2011, from Valencia Cf On 18 September 2014, he renewed his contract until 2018, and was promoted to the reserves in Segunda División the following Iuly__ NEWLEEEEEEEEE. Tarin made his senior debut
_TART ARTICLE_ Rodrigo Tarin _STRR_ SECTION_ Club career _TTRT_ PARAGRAPH
August 2015, starting 1-2 away against UE Cornellã. scored his first senior goal on 17 September of the following year, netting the winner in 2-1 hame success over Atlético Baleares; November, however, he suffered a knee injury which took him out for six months. NEWLINEEEEEE Tarin made his professional debut on August 2017, starting 2-1 win against Real Valladolid for the Segunda División championship. The following 27 June, he signed three-year deal with Liga side Leganés. NEWLINE, Tarín made his debut in the main category of spanish football on 26 September 2018, starting in 2-1 home defest of former side Barcelona.
train
mem
wikiidata id-029457526, version, (122945402299966522269
281892
0,2074
START_ ARTICLE_ Redrigo-Tarin Tamil Nadu Formers Protest _TARRTSECTIOON SECTION Hamerrirrt START PARAGRAPH. Bora-in- Chiva, Valenciani Community, Tarin The severe drought
hydrocarbon proj Bined Bartelonaedt's youth categories limplementation Tamil frustrated] Tamil farmers. They started protests in Jantar Mantar, New Delhi NEWLINEEE In 2016. Tamil Nadu faced the worst rainfall for the prior 140 years. Average onnual rainfall decreased
failed lamil Nadu the worst rainfall evet, From-Valencia-ff Sf--t-withh scattered rain some areas. The highest deficit of tainfall in Indian states in 2016 was Tamil Nadu with an 58% reduction Septemeing their crops
2016. The Northeast monspon
failing, many farmers
relief fund. The Government approved only R8,2014 he renewed his contract until 2018, and was promoted the reserves in Segunda División the following
to die heart attacks and suicides. The Tamil Nadu govemment requested Rs40,000
crores, NEWLINE Tarin made his senior-ehe dehe Hydrocarb di-en---- Augustt-ttet 2015, starting-n- inniin------ a-1-2 1-----2-yyy 2-a--aaw---aaa- lass-lssss- against-ainng-ea UEon extracting project implemented
the agricultural Relds of Tamil villages like Neduvasal by the Carnella He scored his first senior goal on September of the following year, netting the
sinner-n-- 2-1 home sucsess- ouer CD Atlética Raleares; November howaves he sulfoodd- snno--njury-an unjury took him- out for sia mos.mooootsttti
govemment annoyed the lamil farmers. people Neduvasal protested NEWLINE Tadin-made his professional demil Nadu farmers protested]
led but-on--y Tamil Former Augustyykannu They invented protesting ng methods such as shaving half their beards and hai and eating rats and snakees.
Farmers Jantar Nantar vomed drink their urine April 2017 They promised
feces next day. On
the protest, starting
2-4--thhe Tamil Nadu High Court gave judgement way win against Real Valiadolid for the Segunda Divisién championship.. the following 27 june; he
signed- wwwet-eeer--eeeeeee deal- biga side
NEWLIN farin made his deiving farmers
cooperative the main caletory Spanishi
football September 2018 statting in home defest of former side Barcelons.anks anks,
Figure 30: Validation/ / training example pair from Wiki40B:en with intermediate low influence.
Red / green highlighted text indicate deleted added text in the training example comparing to the corresponding validation example, generated using Python difflib.
39

---

Index
Estim.
Link and Text
generated 1361
inft 0,1805
https::/en.tredd.uk az/ business /wa.ancee//eeeee/ee/ 3046505.tmll
Bako, Azerbaijan, April is Trend: Official eachange rate of the us dollar and cum against Armrhaijan manat was set at 1.7 and 1,9225 manats, respectionlly for April 15. Below are the Acerbuijani manat against world cumencies, according to
Central Azerbaijan April Currencies April 15, 2019 1S dollar USD euro 1 EUR 19225 1 Australian dollar 1 AUD 12180 1 Argentine peso ARS 0,9483 100 Belarus rubles BYN 0.8055 Brazil BRL 8A179 UAE dirham ADD 0.6628 1 South 015 Czech
African rand 0:216 100 South Georgjan lari GEL 0,6309 1 Hong Kong dollar 1 HKD 0.2168: Indian nupee 1 INR 0,0045 1 British pound GBP 2.766 100 Indonesian nupiah t00 IDR 0,0121 100 Isanian rials 10n
won
CCX
Chilean peso
Chinese yuan
Danish
0J837 Swiss franc CHF LOBer Israeli shaket 1 ItS DA763 Canadian dollar CAD 1.2756 Kuwaliii dinar KWD 5.5887 Kazakh tonge K21 0.0015 Kinggz som 1 KGS 0.0228 100 Lebanese pounds 100 LSP 0.1155 Nalaysian ringgt MYR OUEEA Mesican peso 1 NMP 0,0906 Noldovan leu MDDL 0.0963 Egyptian pound 1 EGP 0.09821 Nonweglan krone NOK 0,2003 100 Uttlek soums 100 UZS 0 0203 1 Polish dotyy PLK 0JM4221 Russian ruble 1 RUB 0.0264 Singapore dollar 9CD 125661
IRR 0.004 Swedish krona
Saudi riyal 1 SAR 04433 SDR (Special Drawing Rights IMF) 1 SDD 2,3635 Turkish lira TRY 0.79 Taixon dellart TVDD 0.0551 Tajik somoni TIS 011811 Turkmen manat 1 TMMA 0.4551 1 Ulkrainian hryona UAH 0.063r 100 japanese yen 100 JPY 1.5187 New Zoaland dollar 1 1ZD 13513 Fallow Trend on Telegram. Only most intoresting and
important news
train 2072973
men 0.3534
https://en.trent wrwndnddeedddddne.e. az/ business /fnaaccee////ee/eeeee/ee h333338.
Baku,, Acerbaijan, ApeiWarrt Trend: Orhicia eschange of the us dollar and against Anertbiiaani manat at 1.7 1972666 manats, respectioely,
for Retillaach 15, Below are the rates of Aaerbaljani manat against world currencies, according to the data from the Central Bank of kaerballan for Aeeiblarch 15.
Currencies, Apriltant 2019 dollar USD euro 1:03540 Australian dollar 12188007 Argeminee pesa IRS 0.04 Belares ribles
1980005 1 Brazil mal BRL 0A39420 1 UAE dimam AED 0,6288 South African rand ZAR oagaara 100 South Korzan won 100 KRW 015503 1 Crech konuna 1 C2K 0.073
Chilean 100 CLP 02 1 Chinese yuan CNY 0,25368 Danish koone 1 DIX 0.25/ Geongan lari 1 GEL 0.538653 Hong
dollar HKD 0.21888 1 Indian rupce
0.0245 British pound GBP 2:]5 100 Indonesian rupich 100 IDR 0.013118 100 Iraniat rlals 100 IRR 0.00%] Swedish krona SEK 0.38 9229 Swiss franc CHF 1,54 1 Israeli
shekel 1 I1S 0 Canadian dollar CAD 1276660 1 Kuwalbi dinar1 KWD 5:588995) Kazakkh tange 1 KZT 0.0045 Kyrgj som KGS 0.07188f 100 lehanese pounds 100 LBP
03126 Malaysian ringght NVR 0.48 Mexican pesa MIP 0.0006581 Moldewan 1DD 0.096380 Egyptian pound
0.09 Konangian krone MOK
(0qua198 100 Uzbek soums 100 U2S 0.caua789 1 Polish dotyy PLK 0.44078 Russian ruble RUB 0,50s 1 Singapore dollar 1 56D 125beb 1 Saudii riyal 1 1AR 0,4333 1 SDR
[Special Drawing Rights 1MF) 2.35566
lia TRY 09333110 Taiwan dollar 0,063 Tajik somoni T15 01801 Turkmen manat 0.4877
Ukaainian hoymn 1 (UH 0.0638| 100 Japanese yen 100 JPY 1,500200 New Zealand dollar N2D 1/81638 Follow Trend on Telegram. Only most interesting and important news
generated 20228
inft 0.0540
httips//wwww.capttaffit /wwerrotoeeeeeeeeeooooootttiitno conditions// --s------------ (ix-i-trrt--1000aarrttt
Son fins Capital Yorkshir, Capital Manchester Capital and conditions (which can be found at https://wwwwcaapp htt.://ptaaf...cmmmm...... ms ponddiiins//mmmetliool (the "Compettion Terns and Conditions of Global las defined in the April
April 2019 RULES following rules ('Spetific Rules") together with the general competition terms
Competition Terms Conditions) apply the Saw first ndio competitio (the "Competiionn) wiich will fron Monday April 2019 to Saturday Capital Horkshine, Capital Manchester and Capital Korth Fast (the "Radio Station") 1). Anyone who enters the Competition (an "Eatrant") yiill be deemed to have mad and accepted the Specific Rules and the Competision forms and Conditions and will be bound by them. Details of the Competitior: 3. enter Competition an Entrant
go online to www.capitalfm.ccmm com, enswer the multiple choice question and register their details. Online entry will open 00:20 on Monday 5th April 2019 and close 17:30 on Saturday 27th April 2019, The winner will be pidked at random from correct entries and notifed via telephone email. Eigiiiiit: Entrants must be 18 or over, Entry restricted one entry per person; diplicate entries be excluded from the Competition. Proe: 7. One winner will receive the E1000 prize package which consists of 0600 Saw flist voucher EXOO cash & The 2600 voucherwill be sent an code the winter to spend online and will have expiry date. 9. EV0O will be sent a cheque to the winners addieess within 28 days of winning. 10. Prize is none transferable. Giobal reserves the right substitute the prize with another prize absolluee discretion.
train 1072243
mem 0,2117
htap:shiirtteemmmmmitttttonstooo https://www..apiaaffff.cmmm com http://taam.m.oooooosssoosee /yorkshire/termsse /terms wwwpe/sooonnnnnnnnieeeeeeeee wlreiionnn//eeeeeeeeeeeeeeee city- college march19/
SSe It First Leeds/ll city College Capital Yorkshire Casitall Majcchester and Capital North East Aontrol 2019 nULES 1. The following rules (' ("Specitc Rules") together with the
competition
conditions fuhich
found littes//wwwt http://www.appalllm com teemmttssss conditions competition] (he 'Competition Terms and Conditions))
Global for defined in the Compeitton Terms and Conditione) apply Saae kepreniioshig radia competition (the "Competision" which will run from Satunday
and Manday with Aprilaci 2019 jountil Saturday 2ith Apallrd Hareh 2019 (the DDrationn) "I Capital Yorkshin (the "taiio Station") *) 2.2 Anyone who
the Capital
Manchester and Capital North ompeskion (an (the "Raioo Statiorntran" 11.2 An will be deened to have read and accepted the Specifi: Rules and the Compettion Terms
and Conditions and will be bound hyyne who enterrs them Details of the Competition (an 3 To enter the Competition an Entrant will te deemed to have read and accepted and hound by Syteedd----tt---a Relsils
iie-Spesi must go online www. rapital Rulas- the Compotitio
Campolitins So-eatarthe
Compesition an Setrant must in com/ and réc ocp-onnine to aee capitalff com, anmans the mubiplo choics question and sogiste their dotaikister their details. 4 Online entry picked
open 00:00 on Saturday
Monday Aari
and
122-59 on Saturday on Saturday Aprildd Marci 2019. A winner
andoms, Entrants must answer the question and register their details from all connect extries and sotior the change to win. One winner will be selected random fierom sorrect and ve-telephone-dt of emoialld entries contacted by Capital Eligblity Entrants must age En mest- over B Adult consent (someone age or ovec-6) will be required for winners under the age 18. Entry is restricted to one entry per dersore eetisstteeeeeieeeeeeeee wit--e escluded from the Seteee--oeeeooeeeee Rrize---nee will mecior the duration the promation. Duplicato
will miniee fdi
80, YO, WS, woucher ail be cost-ott--e at an s-odooooddddaa whw.th------iccceee. spond unlino will hava-ao transferabii
Fnttants
Yorkshim postoodes
HG
and HU triz sacckagg-ahihhhasniieee at 3.6000
wnw.-oosooooeeeeeeeeeoeo $400 each
winner will receive Aople Airpods and Aople iPad Pro 11- inch display. 12. Prize delivery will be discussed with the winner when they are contacted a member of the Capital IM vorishire tean. 11. The prize won'l he substituted for other prize cash equivalent. 16, Global reserves the right substitute the prize with mith-another price at its absolste discestion- at its absolute discretion
oxpiry data. S. Ex00 will be sent choque the winners address within days winning Prize
Figure 31: Generated / training example pair from RealNews with high to intermediate influence. The generated examples are directly taken from publicly released generations of the Grover-Mes Megaa (p=0.96) model [Zellers et al., 2019]. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python di ffli...
40

---

Index
Estim.
Link and Text
generated 19128
inft
https://www.glbbalrsseeess lus plans designate -ra---iinnnn--a-ee revolutionary gurrrionnnrr-ggrrd--aaar terrorists/5873996
0.0357
Notes Disclaime:: The contents of this article are of sole responsibllity of the author(s). The Centre for Research on Globalization will not be responsible for any inaccurate or incorrect statement
article. The Centre Research on Globalization grants permission cross post Global Research articles on community internet sites as long the source and capyright are acknovledged together with a hyperlink to the original Global Research article. For publication Global Research articles print other forms including commercial internet sites, contact: (email protected] www.globalresearch.cco contains copyrighted material the use of which has not always been specifially authorized by the copyright owner We are making such material effort to advance better understanding of political, economic social issues. The material on this site distributed without profit to those who have expressed prior interest receiving research and educational purposes. you wish use copyrighted material for purposes other than "fair use" you must request permission from the copyright owner. For media inquiries: (email protected)
available readers under the provisions of "Fair use"
train
http::/wwww.gobalrsseecccaaac aeee---mrrin--meee empire of man-mmdd-ness//eeeeeeeee n2w5/2066
1244174
0,904
Operation NAte mass arrstts suspension habeas corpus by the Chief Executive; US NATO as protectors global capital; structure inquuality maintained by global capital and military empire worldwide; repression that addresses us all; the 1948 Declaration of Human hights, Disclaimer: The contents this article for any inaccurate or incorrect statement in this article. The Centre Research on Globalization grants permission to cross-pust Global Researth artides on community internet sites pu publication of Global Research articles in print or other forms including commercial internet sites, contact: (email protected) waw. elobaleeeeeeaccccco contains copyrighted material the available our readers under the provisions of "fair use" an effort to advance a better understanding of political, economic and social issues. The material on this site distributed without profit to those who have expressed prior interest receiving for research and educational purposes. you wish use copyrighted material for purposes other than use" you must request permission from the copyright owner. For media inquiries: (email protected)
responsibiiity the author(s). The
for Research élobalization will not responsible
the source and copyright acknowledged together with hyperlink to the original Global Research article. For
which has always been specihcally authorized
copyright owner. We making such material
generated 7818
inft 0.0207
https://en.trend.au business /energy/ /ee/e/gg330462266mm2g2 3044626.html
Balu, Azerbaijan, April By Ali Nustafayey - Trend: Italy would have to import almost 50 percent its natural gas through Germany without the Trans Adriatic Pipeline (TAP), managing director of TAP AG Beat Rathmann told Trend April 15, TAP a part of the Southern Gas Corridor project, aimed at diversifying the European energy supply sources
routes. The kilometer long project envisages transportation Deniz Stage to the EU countries. The pipeline will connect to the Trans Anatolian Natural Gas Pipeline (TANAP) on the Turkish-Greek border, run through the Adriatic Sea, before coming ashore italy's south. shareholding is comprised of (20 percent), SOCAR (20 percent), Snam
gas from Azerbaijan's
Greece, Albania S.p.A. (20 percent), Fluxys (19 percent), Enagãs (16 percent) and Axpo percent). -.- follow the author on Twitter: @AAi, Mustafayev follow Trend on Telegram, Only interesting and important
train 1246306
mem 0.1500
https://en.trend. az/ business, /energg/27911327.tm h29m3377.
Baku,, Azerbaijan, Apri--Suguss 15ugust 29 By AlF Mustafal eman zey eenalvvv Trend: ItaNaally 60 percent of pipes have beenwwlll Rave to importt almost percent if natural gas through Germany wi withoutelded part of construction the Trans Adriatic Pipeline [TAP), managing director in Greece and
Albania, TAP Beat-Rathmare told Frend April ISconsortium said message its Twitter page. This accounts for 458 kilometers of the total of 765 kilometers of [AP Southhen Gas Carrides project, aimed-at---eeee aimed-at-diversiliilee diversifying the Eurapean-energy supply ande---nd----otte reetroute those two countries. The 93000000-ommee www--lloomeeeeeee long IAP part of the Southern Gas Corridos, which one of the priority energy projects for the European Union. Deniz Stage 2 to
Southern
countries. pipeline connect the Trans Anatolian Natural Gas Pipeline (TANAP) on the Turkish-Greek border, run through Greece, Albania and the Adriatic Sea, before coming ashore in south TAP Will lilomeetes length (Greece 550 kilometers, Albamia 215 lilometers, Adriatic Sea kilometers, and italy kilometers). TAP's shareholding comprised of BP (20 percent), SDCAR (20 percent), Snam S.p.A. (20 percent), Fluxys (19 percent), Enagis (16 percent) and Axpo (5 percent), --- Follow the author on Twitter: @ Alllyamani Mustalazey jul Follow Trend Telegram. Only most interesting and important news
project envisages transgortation gas from Azerbaijan's
Figure 32: Generated / training example pair from RealNews with intermediate to low influence. The generated examples are directly taken from publicly released generations of the Grover-Mega (p=0.96) model [Zellers et al., 2019]. Red / green highlighted text indicate deleted / added text in the training example comparing the corresponding validation example, generated using Python di diff.ib.
41

---

Index
Estim.
Link and Text
generated 21998
infl 0,0218
http:://www.arabee com /nde 11A8311//world
NIW nelite incia is Roady to see average rommsoos raies this year the state nas awather offic said Monday which shoeld support apyicatari production and accnoni groath in Askl third biggut eoonomy where half the fammland lads irigation. Mensson sintail is expested to be 96 pereent the longterm snergge Ratevon storetary the M Winstry of Earth Sciences, teld a nens conference in tieteredogios Depertrnent (100) dekines didtributedd vaiefall
nomell mintall betwren 95 pereen and
of 1D-year ivaage of contineters for the estire four month season
teglnning "Ovevall, country equactedd forecast, Skymet, the country's a only priveze aeahee forecasting agency carler
daring
which bendfii
esuring Kharlf
Sammin--arringgl- stason" the tiieblood for india's 5 fomn-degendett $2.4 tril lon economy, arrke the southem tip tesaa state amonnd lare aad refrett from the desert state of Rijustas by Sagner ber. Mher net spell sowing of ummner iiwn casps gits off a string sturt, koneling crap shicch and cutpe whichin are mine naoi inoomes and sually lits comsumer spencing lndix if plpeetiis mereenn sains lift agric aurical production inn itis your that codld lses food prices ander contra), Subalaed overall infatios could also add prosiare Indaa's onntal bank to cut inteest cates "WISs grv faracast
farezet minbal pulld
normal this Monsoon rinss
cerr--ccmallll well distributed oiilflll will bode for teae-ter food infation, said Madmas frona lead cconomist Edelares Secrities, IX Rates Mill
poiny revies India's
oweroa schedaiee
genasa exetton, spreed ove could mein Garners continss to get iit by loss coop pricos, rujer causa concarn rasa vlhae most indias liv, the past tao peurs Afte fallng fav stright monntts food prises india rase 2.30 parsent in March from year surlaer last reneth, a sarior IMD efficall told Bourers that this years recesson was (ialy to be rabust and healthy provided there woorts surprice E Nino
after maantyy's
castng
dosanide, higher production
peenamenon: "E sine we webriigg ng and we copect that it Nine will get wookned forther There no sunce to be worried sbout It tine," Rijeesss sad string il tine, murked by surning the sarfara the Pacifl Conan, can caree searr droughtt in fantialla Southead AGa and Indio while doaathing other of the world such in the Mibaand and Beut is riins. The iesegaace of string farmers pertary suidde. BOOMONIC GROWII growing rajer ecenory
back
castury,
will spur
planting craps
fice, core,
and styboor, Stronger apricaltaril production wedle sugport india's renscery it
peatth slowed tn parcett
Becentber quarter, from 10 perver the prwdius perioe and dassee
rpaatees The massuse isuully rowes half the country the 15 days.
rains seach cestrall india's soybaas areas by the skidd waek of june and westers catten growing areas by the linst wsek of jaly. india's aeather offic will apects its fawcact the fisst weak of However, average, the IMD has forecast acourodly only orca avery tho years over the past two decados, ewnn aftar taking into account an ertrr basd of plas or rinnus 5 pertentagy points.
train 326212
men 035555
https://www www.:/uetersscooeeeeee www.ttersco//aatiie/eeeeee wwaatiieeeusiindiieiee wwwdddaammoooonninno wiwtiiii-iiiiitttt-eeeeeeeeeeeee
awweaaee watogee-moooooooniinoosoeos---es wwwtis---i-ooos----e holll-bound-mdii
idUSKBNIHN1AQ
MIW BELI Peeteel! indi is lirly to
wassoon mins thingeet antrage monooon mins 2018 the steteenan menteeratioeeeddeee. on Sendommmmterr ofhre said Whech cepport spkohhard--poddsieeeeee aing the pesabbltty of higher furm and economic grouch in Asll's oireeoiggez oconois, where had of farmland lads inigztionn Mansaze rainhalies, the fibioos of the country's s 52 trlliion sconans ant epeceed to be seaarsont of the percent af long-ter aadrage, MR Rafesvonness peceeary the Ninitry of Earth Sclences, tal news conference. Thedirectall S6-year Owsakther Ufting farm and wider
ganers state-na Infa Meteorolog Department INO dehnes
rainfsi between 96
centemeters for the the-enire fou- month jos-ooustr-ii asppociaa general eiect onz das
conforune. "We
probabiitty of
kanech said
wwwpeleddddeeeeddd-d-eeeeeeee sona-eeee spel. good ns ketp lid os infiation, actertially tempting Prime Minster Warcadra ts bring
May 2015 morsoon souncn, wothts woathe offec tefiee zeenge which will be benefiial to farmers in the country during the ensaing Kharff foemmen tormal nimbal as betaeze 96 percent and 138
pertent o 50- planiing you seergee 10 cres for the enfire fou- mon't season tegining hame "The moderste La line coeditioss repe is the equatoria fir daring list year started
seallning the eatly part of thes year and currenty have surned to weai Miaa conditinn "at IMD said in le-ococas statanert. Hassolaa Hina is weather patter that brings equateria, Pacing
Oceen temperatures the
prisinfsll
winds
weather
agency eatder this
canfall
normal this yearerng The latest
trrppnts global
cond tiers
beg tring of momoon seane,
mined)
spar planting crogs
sew-Bieeeeenllaeecco cen, eotice syylaars, acceieeatin enssonic goatt
22 partect the Biceesber qurtee in farent free quarters comqueed with thina's 2.8 parsent in that sqaatee)
Growth the Decamber quarter restored Indi's fasectatts a5 the world's fastest growing majo cconoryy RURAL ICCME fveraee no15003 raiss, sitt good diattbution isly and August would sapport
nral cemand, said Rapa Rege fitsum group chiel comsrrsst Finance Helkings a Mumbei- fependent Elunre 12r bunking finnnc company Strillnn cconony Good ramm boost run)
www.eeeeeeefeeeeeeeeooooo intation
of www.oellnn--eeeee by wwppeeeeeeeeeeeeeeeeeeeeeeeeeeee fligg the derard Indil food www.://ww...oottttt..ooooooo
insurter goods sanging lipstiio
efiggratars
Menday besciegcsp-eelle wetpooppeelllleaeeeee and
percent
2018 year exrlet Epertila
griesharod-poddeeiinn thwpeereeeeeiieeedddd weather oftee wil uptate its fnseast fate oeergg thatoredee MD has fireest accuondy only enee is every five years pan the past nan decades, ewen
ther ta feeee pricee and ongieiing itoo attaacmant in ento had of pan rinen paceetage point hendwesliiin inaily lash Keril Mate as the woth moed aroadd june 1, and (eandibddaion
coasd slco add to prosaaee on the whote county ric-uly Timely nars tniger planting of trops such as soybaans and catter. The nonsose usaly crecns the half of the country the 15 dass catler-ge gitti-gggnnggg ligest import after
the rains reach ceetril India's seltni-barsryysan
th inett-nterst ofvee
first week of lely Good min ald help
sny Cameetly
eepenbbe teghtable
ligget
groggiing with huge ansunts sugar and good could further bump saplyy
good morsoan will bring down cdible illimports aeooing
cermalasd dittibbted caaaal all www.eedodaaac.ooooooo for www.daan..nnnnc.omm. infiiccettt most dos cratee a pro of plerty pubes and sugal sai Padias doun, lead acossmist abelanis
tensniiinn aww-eateen-----ee thk reel pasioy wowin by lediol restred bonk www--eeeeeeeeeeeeeeeee foee by -lle-
the
sassarrbaai-sssed sealer glota boding
www.etaaiiiiaae ItS
top raporte
dauill- cat conees deas whain feathoast deis and
coosng of -inssng Adiad spaz this plarting all
diggen www.ttttttttteee bucck droug tos 2015 and 201, only the boath
dining ind aut
and cace enary fan years
Figure 33: Generated training example pair from RealNews with low influence. The generated exam- ples are directly taken from publicly released generations of the Grover-Mega (p=0.96) model [Zellers et al., 2019]. Red/ / green highlighted text indicate deleted / added text in the training example com- paring to the corresponding validation example, generated using Python di diff....
42


Reviews:
[{"review_id": "lUMrdbbzmm", "reviewer": "Reviewer_DH4T", "summary": "The paper studies an important problem of the \"memorizing\" effect in neural language models, particularly focusing on rare or isolated pieces of information in the training data that get memorized, as opposed to widely duplicated or common information. \nIt formulates a notion of \"counterfactual memorization\" that measures how a model's predictions change when a particular training example is excluded during training. It analyzes counterfactual memorization on several standard text datasets and studies patterns in what gets memorized. Examples with unconventional text formats tend to have high memorization.\nIt extends the definition to \"counterfactual influence\" which quantifies the impact of a memorized training example on the inference phase, which helps trace the source of predictions.\nOverall, the paper provides a novel perspective and tools to systematically study the memorization of information in neural language models.", "questions": "The core idea of counterfactual memorization is intuitive, but can you provide some theoretical justification for why the formulation in Equation 1 captures the memorization of information? Is there any formal argument that you can make?", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper presents a novel and significant perspective on studying information flow in neural language models. \nThe notion of counterfactual memorization is an original contribution to quantifying the memorization of isolated pieces of information. This moves beyond prior work that focused more on duplicated content. Extending this to counterfactual influence to trace predictions back to training examples provides a new capability.\nThe mathematical formulation of counterfactual memorization is clear and well-motivated.\nThe empirical methodology is sound, with adequate models trained to converge and sensitivity analysis on the number of models needed.\nThe results support the claims and provide new insights into memorization.\nOverall, I find this a very solid, insightful paper with multiple novel contributions to an important problem.", "weaknesses": "One potential weakness of this paper is the application of such analysis remains elusive. I can imagine many interesting questions could be answered with the provided concept, e.g., what kind of stereotypes/social biases in the model are influenced by what data in the training samples? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the strong support of our paper and useful suggestions! \n\n> **Q1**: One potential weakness of this paper is the application of such analysis remains elusive. I can imagine many interesting questions could be answered with the provided concept, e.g., what kind of stereotypes/social biases in the model are influenced by what data in the training samples?\n\nThanks for the suggestion. We are mostly thinking about prediction attribution (finding which training examples contribute to the current prediction) as a general direction which may have many downstream applications. We agree that understanding the source of stereotypes / social biases seems to be a very interesting concrete question to consider.\n\n> **Q2**: The core idea of counterfactual memorization is intuitive, but can you provide some theoretical justification for why the formulation in Equation 1 captures the memorization of information? Is there any formal argument that you can make?\n\nOne potential connection to learning theory is that Eq 1 essentially measures the \"generalization gap\", except that it is measured on a specific given data point, as opposed to the expectation on the (unknown) underlying data distribution in standard learning theory. The latter being large means that a model has overfitted, thus memorizing the training data without generalization capability. In our case, we modified the notion to measure the \"generalization gap\" of a specific example x, in order to quantify whether a particular example x is memorized. This formulation has been theoretically studied in a synthetic model, where it is proven that memorization is closely related to optimal generalization if the underlying data distribution is long tailed.\n\n[1] Feldman V. Does learning require memorization? a short tale about a long tail. InProceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing 2020 Jun 22 (pp. 954-959).\n\n\n"}, {"review_id": "AXANgtytuc", "reviewer": "Reviewer_PFxH", "summary": "This paper formulates a notion of counterfactual memorization to measure the \"one-point\" generalization performance of the model (they do not convey this definition as this). Equipped with this definition, the authors conduct plenty of experiments to explore the components that are related to such counterfactual memorization of large language models. ", "questions": "As the authors clarified, \"Both the neural language models and training sets used in this work are orders of magnitude smaller than modern standards such as GPT-3\". Though conducting experiments on these LLM is impossible. My question is, does the samples that are proven to be easily memorized can not have diverse patterns during inference, even for larger models? ", "limitations": "As the authors clarified in their paper, they have no experiments on larger-scale pre-trained model. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper is well-written, and the analysis of what kind of samples tend to be memorized is interesting. The experiments in this paper are systematic as many of the related components are considered. ", "weaknesses": "Despite the strengths mentioned above. I think this paper has the following weakness. \n\n1) The definition of counterfactual memorization is nothing but the generalization performance of the model on one single point. In fact, taking expectation to $x$, the counterfactual memorization becomes expected generalization. Thus, in my opinion, studying it from the perspective of distribution is exactly studying the generalization of LLM, which obviously is not a new topic. \n\n2) The definition of counterfactual memorization is quite similar to some classical notations in generalization theory, e.g., algorithmic stability and $\\epsilon$-differential privacy.  \n\n3) Here is my major concern. The performance of the model is evaluated by the 0-1 prediction loss on the task of next-word prediction (NWP). However, I think this is not a reasonable metric to measure the performance of NWP. The metric is reasonable for the classification task. However, for the NWP task, I do not think there exists a \"ground-truth\" label as in classification. For example, the sequence \"I like eating _\" followed by the words \"apple\" or \"banana\" are all valid predictions. \n\n   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the comments and suggestions! It seems there are some misunderstandings of how our metrics relate to existing notions of generalization and how the 0-1 prediction loss is chosen. We answer those questions below and hope the reviewer could raise the rating if the answers clarify the confusion. We are happy to answer any further questions during the discussion period.\n\n---\n\n**Q1**: relationship between counterfactual memorization and generalization.\n\nWe appreciate the reviewer for observing the connection with the generalization gap in learning theory. This connection actually provides another strong motivation to the definition of counterfactual memorization. As noted by the reviewer, the standard generalization gap measures the *expected* difference of model performance on unseen and seen examples. When this gap is large, the model is said to overfit, or memorize the training set without generalization capability. By making the measurement at the example level (and averaging over multiple models, instead of averaging over test examples), we measure the same “generalization gap”, but now we can characterize whether a particular example x is memorized or not. This key difference allows us to do all the analysis in this paper, and enables the extension to measure counterfactual influence. Therefore, we respectfully disagree with the reviewer that this is identical to measuring the generalization.\n\n---\n\n**Q2**: relationship between counterfactual memorization and algorithmic stability / $\\varepsilon$-differential privacy\n\nWe agree that counterfactual memorization is related to many important notions, which actually supports that our metrics are grounded on the same fundamental notion of generalization-memorization trade-off widely studied in the field. \n\nWe also clarify the key differences here: the relation to algorithmic stability is similar to the relation to generalization gap as explained in the previous question. We focus on characterizing the behavior of each single example, while stability (e.g. leave-one-out stability) is formulated as a property of a learning algorithm. Moreover, while our analysis is heavily dependent on the underlying data, the study of stability generally consider *uniform* stability, which is a property of the learning algorithm that is supposed to hold for *arbitrary* data (i.e. data independent).\n\nThe relation to differential privacy (DP) is similar: DP quantifies the *worst case* changes under arbitrary replacement of an example, which is a property of the learning algorithm and independent of the data (DP guarantee should hold for arbitrary data). While we focus on a characterization of each example (as opposed to the algorithm), and measure how each training example influences the trained model in relation to the other training examples, which is heavily dependent on the underlying data distribution. On the empirical side, it is known that outlier examples (which can be identified via high counterfactual memorization scores) tend to have high membership inference attack success rate, and membership inference attacks can be used to provide empirical lower bound for the DP privacy parameter.\n\nWe also note that DP and stability are closely related to each other in the formulation, but it does not decrease the importance of either notion because they have very different perspectives. \n\n---\n\n**Q3**: justification of 0-1 loss in next word prediction task\n\nIt is true that for short contexts, many possible “next words” exist in the training data. But the distribution of possible next words becomes sharp as the context length increases. Moreover, such kind of ambiguity also exists in image classification. For example, many ImageNet images are real world scenes containing multiple different kinds of objects, yet the model is trained to maximize the likelihood of a single labeled class and evaluated with 0-1 loss because it is simple and effective. In language model, such ambiguity is also not *explicitly* addressed, the training objective is exactly the same (cross entropy loss) at each word (token) level. The model implicitly learns about the relation between similar words. When evaluating a language model, it is common to use perplexity, which boils down to measuring the per-token likelihood on the exact groundtruth token (again, without explicitly considering other possible words that are equally valid). We have evaluated measuring memorization using perplexity, and the ranking of memorized examples are largely consistent with the per-token accuracy measurements. However, we ended up using the latter because the per-token accuracy has a well defined numerical range of [0, 1], which leads to score values that are easier to interpret.\n\n---\n\n**Q4**:  \"does the samples that are proven to be easily memorized can not have diverse patterns during inference, even for larger models?\"\n\nCould the reviewer clarify the question? What do you mean by having “diverse patterns during inference”?\n\nWhile we cannot make formal guarantees that all the results transfer to LLMs at GPT-3 or larger models, we believe most of the observations would still hold. In particular, manual inspection of the memorized examples (see the Appendix for a sample of them) matches the intuition, and we believe larger models will behave similarly. Our future work focuses on finding approximate solutions to the algorithm so that it can be applied to the larger models.\n"}, {"review_id": "8F1AfuQUpk", "reviewer": "Reviewer_AGTS", "summary": "The study proposes two novel metrics: “counterfactual memorization” and “counterfactual influence”, the first one can measure how a model’s predictions change if a sample is omitted in training, and the later one measure how a sample in the training set influences the prediction of a validation sample. The authors also analyze the two metrics on 3 standard datasets and discover some interesting results, like “different data sources display different memorization profiles”, ”number of models needs”, “memorization does quantitatively decrease when data is repeated more often”,  the strong influencer ”also only receive tiny influence from all the rest of the training examples”.", "questions": "1. The authors could study some classical causality studies [3][4], which can help you to make the research better;\n2. The authors could try some simple models and do the similar analysis.\n", "limitations": "Besides the limitations the authors list in section 7, the authors also mentions that it is hard to evaluate the impact of a single training example for the costly computation, I hope the authors could compare their methods with influence function in [5], which can identify training sample most responsible for a given prediction, and discuss their advantages and disadvantages.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tGreat motivation and important research question, how to measure a sample’s influence on the trained model’s prediction and the sample’s influence on a validation sample. \n2.\tThe counterfactual viewpoint is interesting and easy to understand. “counterfactual memorization” and “counterfactual influence” are useful metrics.\n3.\tThe analysis and some findings are insightful for inspiring future study.\n4.\tThe presentation is clear and easy to follow.\n", "weaknesses": "1.\tI am surprise that the authors do not refer the causal inferences studies, since “counterfactual” is an important concept in statistical causality. Judea Pearl, Donald Rubin and many scholars did many great work about counterfactual and causal effect, such as structural causal model, potential outcome model [2]. From the perspective of causality, the equation 1 and 3 could be reformulated by causal effect, counterfactual inference, which are important research topic and have been studies well in statistical causality communities. So I suggest the authors can study some classical causality studies [3][4], which can help you to make the research better.\n2.\tI am curious the proposed two metrics on simple models, such logistic regression, SVM. Large language model (LLM) is popular and important, while the simple models are also useful and efficient for many applications. In the meantime, I think the simple models, like logistic regression, are much easier to measuring the two metrics, since the LLM is complex and have more unobserved confounders. So my second suggestion is that the authors could try some simple models.\n3.\tIt is much clearer to summarize your analysis and findings in one table.\n4.\tTypos: e.g. revious in line 21.\n[1] Causal inference in statistics: An overview, J Pearl\n[2] Causal Inference Using Potential Outcomes, DB Rubin,2005\n[3] Causality: Models, Reasoning and Inference, J Pearl 2009\n[4] Causal Inference for Statistics, Social, and Biomedical Sciences, G.W. Imbens, 2015\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the positive review and useful suggestions!\n\n---\n\n**Q1**: pointers to classical causality studies\n\nThanks for the pointer! We will revise the manuscript to add missing references to the classical causality studies and discuss the connections. Our formulation does not involve complex graphical models and intervention techniques from statistical causality. We are interested in how removal of a training document impacts the model’s prediction, and formulated it as a *direct measurement*. It would be interesting future work to see if any advanced techniques in causality studies could be used to improve our measurement. \n\n---\n\n**Q2**: counterfactual memorization analysis on simple models\n\nSince we focused on language tasks in this paper, we did not try simpler models as they generally do not fit well to the sequence-prediction task. However, we agree that simpler models in general could be quite interesting, and there may be specialized algorithms that can estimate memorization efficiently for specific simple models. For example, for SVMs, if a training example is not a support vector, then removing it from the training set will have zero impact on the model’s decision boundary. Similarly, for k-nearest-neighbor classifiers, the removal of a training example will have very localized impact on the model’s prediction. For linear regression models, since the optimal weights have a closed form solution, we can also calculate the memorization more easily.\n\n---\n\n**Q3**: It is much clearer to summarize your analysis and findings in one table.\n\nThanks for the suggestion! We will update the manuscript and try to format the summary of findings into a table.\n\n---\n\n**Q4**: the advantage and disadvantage comparing to influence function [5]\n\nIt seems that the reviewer forgot to list the citation [5]. We assume the reviewer meant the following [5]. We note that the estimation in [5] relies on an approximation using the inverse of Hessian, which is computationally prohibitive to calculate for large models. As a result, both methods are computationally expensive, but the cost of [5] scales with the model size, and our method scales with the cost of training multiple models, which can be easily parallelized. Moreover, while the approximation used in [5] works well for simple linear models, it is shown to be very fragile in deep neural networks (e.g.[6]). We will include discussions on this in the updated manuscript.\n\n- [5] Koh, P. W., & Liang, P. (2017, July). Understanding black-box predictions via influence functions. In ICML.\n- [6] Basu, S., Pope, P., & Feizi, S. (2021). Influence functions in deep learning are fragile. ICLR.\n\n"}, {"review_id": "5KPo9EYgcJ", "reviewer": "Reviewer_mGti", "summary": "This work studies memorization in neural language models. The major scientific question in this work is how to filter out common memorization in language models. To this end, this work first formulates a notion of counterfactual memorization which\ncharacterizes how a model’s predictions change if a particular document is omitted during training. Then, this work estimates counterfactual memorization on three standard text datasets, and confirms that rare memorized examples exist in all of them. In addition, this work also identifies an inverse correlation between number of duplicates and counterfactual memorization as compared with previous definitions of memorization. Finally, this work extends the definition of counterfactual memorization to counterfactual influence, and studies the impact of memorized examples on the test-time prediction of the validation set examples and generated examples.\n\nOverall, this paper is well-written and flows smoothly. This work begins with a detailed definition of counterfactual memorization and then estimates and analyzes counterfactual memorization of training examples in standard datasets. The experiments identify the examples sampled at high, intermediate and low memorization. Analysis covers the impact of number of models, number of training epochs. This work also shows the potential for inference attacks in large language models in a learning theoretical perspective.", "questions": "Line 21: \"revious\" -> \"previous\"", "limitations": "The authors have provided detailed and useful descriptions about the limitation.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This is an interesting paper that analyzes the counterfactual memorization in neural language models. The paper is written in a clear and concise manner. The problem is well defined, and the analysis is technically sound with insightful findings.\n\n2. This work extends the definition of counterfactual memorization to counterfactual influence, and studies the impact of memorized examples on the test-time prediction of the validation set examples and generated examples --- thus has the potential to provide new perspectives of studying the inference attack problem in large language models.", "weaknesses": "1. The efficiency of the proposed approach can be improved for a practical use. Currently, identifying the memorization levels of the training examples, as well as the counterfactual influence, requires extensive computation overheads by training multiple models --- which may become a major concern for its application in large language models and with larger scale training sets.\n\n2. The experiments can be also broaden to leveraging different backbone language models to provide more useful insights towards how existing language models vary in the effects of counterfactual memorization. This may lead to an intuition on how those language models would be sensitive to inference attacks and stimulates the development of defense approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the positive review and useful feedback!\n\n---\n\n**Q1**: computation cost may become a major concern for its application in large language models and with larger scale training sets\n\nWe acknowledge this limitation of our work. Our main focus for future work along this line is to improve the computational cost of estimation.\n\n---\n\n**Q2**: The experiments can be also broaden to leveraging different backbone language models to provide more useful insights towards how existing language models vary in the effects of counterfactual memorization. This may lead to an intuition on how those language models would be sensitive to inference attacks and stimulates the development of defense approaches.\n\nThanks for the suggestion! We used decoder-only transformer based models which are commonly adopted for language models. It is indeed quite interesting to try out alternative architectures. We believe that the results will not be that different among the different variants of transformer-based architectures that are currently used in practice. However, we are very interested in studying the “scaling law” behaviors as we increase the models and data sizes. Though that depends on the future work of more efficient estimation methods.\n"}, {"review_id": "mOJOvLqMwv", "reviewer": "Reviewer_9fjM", "summary": "This paper proposes a metric for whether a sequence is memorized based on the degree to which exposure during training increases the probability of producing the sample.", "questions": "I'm curious about counterfactual memorization with respect to the training timeline. This paper uses training subsampling as a way of controlling whether a particular sequence has occurred, but I think it is worth asking whether that sequence was counterfactually memorized when the model actually encountered it for the first time, which may be at different points of training. Given a model trained on a known ordering of its corpus, we could consider the impact of a sample in the context of training by comparing checkpoints before and after exposure. Have you considered this question, explored to some degree by Biderman et al. (https://arxiv.org/pdf/2304.11158.pdf)? I would love to see results if so.", "limitations": "The corpora studied are all fairly similar (focusing on natural language English text), so the results are limited to those settings. The authors don't acknowledge this as completely as I'd like, although they do mention that higher quality text has different behavior than a corpus with malformatted text or multilingual contamination.\n\nThe authors also acknowledge that counterfactual memorization is inherently going to assign less memorization value to sequences with near-duplicates throughout the corpus, and describe this as a decision about what type of memorization they consider. \n\nThe authors also acknowledge that the models they are using are much smaller than the large language models of interest today.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The proposal here is strong, and remedies a real problem in the memorization literature: a failure to consider the *inherent* predictability of any given sequence (a property which this paper calls \"simplicity\"). This is a significant gap that I'm happy to see work on.\n\nThe results cover basic questions such as the impact of the number of epochs. They also explain the reasoning behind choosing the number of model runs they apply their method on, and this analysis is sound.\n\nI also like that they measured not only the impact of a given sample on the model's tendency to emit itself but also its influence on the model's tendency to emit a different sample.", "weaknesses": "As far as I can tell, it is hard to actually get a counterfactual metric for memorization of highly duplicated sequences because they are more likely to appear across different training subsamples. This metric inherently favors rare samples, because prior exposure to the duplicated samples will decrease the impact of each additional exposure.\n\nThis paper doesn't distinguish between \"simplicity\" due to duplication and simplicity due to inherently predictable sequences, like repetitions of a single character.\n\n\"the choice of data has a relatively minor effect on memorization\" but the corpora used are all ones that avoid code and multilinguality, favoring natural English language text. It is not clear that this property would hold for significantly different corpora. The authors acknowledge this issue to some degree, pointing out that one of the largest differences between the datasets is likely due to the higher-quality texts in Wiki40B.\n\nThe authors only qualitatively check their conjecture that \"if a validation set example x0 contains similar information, infl(x ) x0) could be large\", but it should be fairly easy to quantify whether the similarity between any pair of samples increases the influence.\n\nMinor:\n- fair number of typos, please proofread before publication", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the strong support of our paper and helpful suggestions!\n\n---\n\n**Q1**: This paper doesn't distinguish between \"simplicity\" due to duplication and simplicity due to inherently predictable sequences, like repetitions of a single character.\n\nThanks for the comments. We agree those are important questions, which we think can be formulated into two main questions: 1) how does the frequency of a phrase correlate with its simplicity; 2) how does the (distribution of) frequencies of internet based training corpus correlate with that of “real” natural languages. For 1), we agree that there are some simple patterns such as “repeated same characters” or “short sequence lengths” that tend to make a sequence easier, but that is not always the case. We did some preliminary studies of how simple metrics such as token diversity correlate with memorization, but we did not see significant patterns. In the end, we believe the frequency is probably a general indicator than the intrinsic structures for simplicity. For 2), we studied how deduplication impacts the measurement of memorization in Appendix D. We found that it is very hard to choose the appropriate threshold for text-matching based deduplication method, and the counterfactual memorization metric actually helps to identify near-duplicate documents that are exactly along the cutting threshold of edit distances. Furthermore, the ranking for high memorization examples remains consistent after deduplication.\n\n**Q2**: quantify whether the similarity between any pair of samples increases the influence.\n\nOne key difference between counterfactual influence and similarity is that the former tries to capture rare memorization. For example, a sequence with a common fact could have high similarity with a validation set example. But because it is a common fact, the validation set example receives influence from many training examples containing this fact, thus the influence from one particular training example could be very diluted and end up being small. \n\n**Q3**: counterfactual memorization with respect to the training timeline?\n\nThanks for the pointer to Biderman et al. We will include a discussion of it in the updated manuscript. One main challenge we run into here is that neural network training has large intrinsic stochasticity. Even if we retrain the network on exactly the same training set again, the prediction could end up being quite different. To increase the signal to noise ratio for stable measurement of counterfactual memorization, we train hundreds of models to average out the stochasticity (we included a study of how many models are needed in Fig. 3a). For this reason, it is very challenging to control the noise in the measurement using the training trajectory of a single model. Maybe there is a way to perform a similar averaging procedure if we consider multiple model training trajectories with different randomly shuffled data order. In the early stages of our experiments, for implementation efficiency, instead of sampling data uniformly randomly, we tried to partition the data into chunks and sampling random chunks instead. The measurement ended up capturing the artifacts of co-occurrence of documents in the same chunk. So we suspect to successfully make such measurements along the training timeline, we will need very precise control over the data orders in the data loading pipeline. \n\n**Q4**: The corpora studied are all fairly similar (focusing on natural language English text), so the results are limited to those settings.\n\nThanks for pointing this out! We will revise the limitation accordingly."}, {"review_id": "vfp4lRa0TH", "reviewer": "Reviewer_PCjJ", "summary": "This paper introduces a concept  \"counterfactual memorization,\" defined as the anticipated change in a model's output when a specific training example is omitted. Experimental analysis was conducted on three widely employed text corpora in the domain of language modeling, and the phenomenon of memorization was identified in all instances. Furthermore, the authors assess the impact of each memorized training example on the validation set and generated texts. This novel approach offers direct evidence pinpointing the origin of memorization during testing. Despite these intriguing insights, the paper has notable limitations. The experiments performed lack robustness and some of the assertions made are lacking in clarity.", "questions": "1.Does the result of the model depend on the size of the subset ?  Is the result sensitive to the value of r? How did the author determine that r=0.25?\n\n2.Some domains are represented much more frequently than others in the datasets we studied (Line 136). What are these domains, and what are their characteristics?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper well-written and  easy to follow.\n\n2.It is of great significance to explore memorization in neural language models.\n\n3.The proposed new definitions of can serve as references for future works, and the experimental analysis is detailed and interesting.", "weaknesses": "1.The paper lacks a section on related work, which should summarize and analyze the current state and challenges of relevant research.\n\n2.This paper only uses the Transformer-based language backbone equivalent to T5-base (encoder-decoder framework). Can the experimental results be applied to other architectures and models of different scales?\n\n3.Some typos need to be fixed. For example, “revious” (line 21) should be “Revious”. The word at the beginning of a paragraph is capitalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the positive review and helpful comments. Please see below for detailed clarifications.\n\n---\n\n**Q1**: lacks a section on related work\n\nThe “related work” section is put in Appendix A due to space constraint. We will rearrange the contents in the updated manuscript to make it more prominent.\n\n**Q2**: this paper only uses  T5 (encoder-decoder framework) backbone, how about other architectures and models of different scales?\n\nWe actually used the decoder-only version of T5, which better matches the architecture used for training auto-regressive language models. We will emphasize this in the manuscript. We are very interested in exploring the scaling law of counterfactual memorization as we increase the model sizes exponentially. We leave it as future work as it is currently computationally very challenging to evaluate much larger models without non-trivial improvements to the estimation algorithms.\n\n**Q3**: Does the result of the model depend on the size of the subset ? Is the result sensitive to the value of r? How did the author determine that r=0.25?\n\nWhen r is too large (close to 1) or too small (close to 0), we would end up having a very unbalanced number of samples on the two terms of Eq (1), therefore inaccurate results. We chose r=0.25 by considering such balance factor, computational efficiency, and utility of models trained on the subset data. However, we did not “tune” this parameter very carefully because apart from very extreme values, the results are consistent within a large range of r values. \n\n**Q4**: Some domains are represented much more frequently than others in the datasets we studied (Line 136). What are these domains, and what are their characteristics?\n\nIn Fig 2(a)(b), we plot the number of documents on the y-axis for each domain. We annotated a few representative domain names. For example, on RealNews, reuters.com has orders of magnitudes more documents than hotair.com, therefore is represented more frequently. Especially on the c4 dataset, we observe that domains with more documents tend to concentrate on intermediate memorization values, while smaller domains have a wide spread along the memorization spectrum.\n\n "}, {"review_id": "8qKKaHBJ1E", "reviewer": "Reviewer_dGf9", "summary": "This paper defines counterfactual memorization as the difference between a model's expected performance on a sample when it is included in the training data and that when it is not. The performance is measured by per-token prediction accuracy. The paper studies common patterns across counterfactually memorized samples. It further defines counterfactual influence similarly to counterfactual memorization, except that the performance is measured on any sample, and studies the correlation between counterfactual memorization of a sample and its counterfactual influence maximized over some set of data.", "questions": "- L22-25: Are sensitive data, such as, phone numbers and usernames, \"common\" and frequently occurring?\n- L30: How do you quantitatively define \"common\" and \"rare\"?\n- L38-39: Why is the memorization of $x$ $\\textit{counterfactual}$?\n- L38-39: Could you clarify what \"predicts $x$ accurately\" means? It is implausible that given an initial token, an autoregressive language model will be able to generate an entire document $x$ perfectly via greedy decoding.\n- L75: Does |D| refer to the number of documents?\n- L93: But T5-base has ~220M parameters, instead of ~112M.\n- There seems to be no control over the number of subsets S and the number of subsets S' being similar and both big. It is possible that a training sample appears in all 400 samples. Then the expected measure M is a bad approximation. What's a theoretical error bound?\n\nPlease also see Weaknesses. If the review misses important aspects, please feel free to address them.", "limitations": "The authors discussed limitations but not ethical concerns. Both are reasonable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- It is interesting to study whether the existence of one training document will affect model predictions.\n- The mathematical definition in Eq. (1) - (4) are reasonable.\n- Ablation studies including 3.3 and 3.4 are well-explained.", "weaknesses": "Overall:\n\n- The research problem can be more crisply defined. It is unclear what type of information the paper aims to study. Overall per-token top-1 accuracy is not sophisticated enough to be reflective of memorization of \"sensitive information\" (L2) or \"details of specific events\" (L111).\n- There are technically mistaken statements. E.g. L32 says counterfactual memorization is a measure of an expected change. But it is indeed the difference between two expected values where the expectations are over different distributions.\n- The quantitative definition about which information is considered \"common\" or \"rare\" is unclear (L48). While the paper uses 2^21 documents (L95), only 38000 are identified as being a near-duplicate with at least one other example (L199). Is two near-duplicated examples considered common, and one considered rare? The 38k seems to be a small portion and it's unclear if the notion of \"common\" vs. \"rare\" is significant enough to study. \n- It is unclear what the common patterns found across counterfactually memorized text are (L50).\n- Please use \"counterfactual memorization\" instead of simply \"memorization\", whenever proper, to avoid confusion.\n\nAnalyzing Counterfactual Memorization:\n\n- There are contradicting claims. E.g. L107 says high (counterfactual) memorization happens for unconventional texts and multilingual texts, while L126 says ill-formatted text or foreign languages are hard and L125 says hard examples have low counterfactual memorization scores.\n- There are non-refutable claims. E.g. L123 says easy examples tend to have low scores. \"Simple samples\" are defined as those with high accuracy, and two high accuracy scores should apparent have a relatively small difference.\n\nFrom Memorization to Influence:\n\n- Authors claim that small memorization scores have small max-influence scores and larger influence scores requires larger memorization scores (L253-254), but there doesn't seem to be a concentration of densities around the x=y line in Figure 4(b).\n\nSome typos:\n\n- L21: revious $\\rightarrow$ Previous\n- L38: considerded $\\rightarrow$ considered\n- L109: most often $\\rightarrow$ are most often", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the helpful suggestions and detailed questions. It seems there is some confusion regarding the definition and analysis of counterfactual memorization. Due to the 6000-character limit, we focus on the main questions below. We hope the reviewer could increase the rating if the answers clarify those confusions. We are happy to answer any further questions during the discussion period if anything is still unclear.\n\n---\n\n**Q1**: per-token top-1 accuracy is not sophisticated enough\n\nWe used teacher-forcing when evaluating under per-token accuracy, so the evaluation is similar to perplexity in the sense that we are measuring the likelihood of predicting the next token accurately given the *true* context, instead of asking the model to construct the entire document accurately. We have evaluated the per-document perplexity to measure memorization, and the ranking of memorized examples is largely consistent with per-token accuracy. We ended up using the latter because it has a numerical range of [0, 1], which leads to scores that are more easily interpretable. We note that the intermediate-to-high memorized examples do end up containing details from news reports of specific events (Table 1 and Appendix L), and such memorized documents are found to have strong influence on the validation set documents describing related events (Table 2 and Appendix M).\n\n**Q2**: “expected” change (L32) inaccurate\n\nBy “expected” change (L32) we meant “anticipated” change, instead of the formal “expectation” as in probability. We will rephrase it to avoid ambiguity.\n\n\n**Q3**: quantitative definition of common / rare is unclear; is 38k near-duplicates worth studying?\n\nThe fundamental question is how to compare if two documents are “identical” or “encode the same information”. We can define common and rare by counting the occurrences via approximate textural matching, as previous work has done. In this paper, we try to quantify it by whether or not the document can be easily predicted by a language model if the model has not seen it during training.\n\nSection 4 is a study of how our metric relates to the number of near duplicates. Though there are only 38k examples that have near-duplicates, the number of duplicates goes up to hundreds for many examples. Moreover, the text-matching based near duplication detection does not capture all related documents. In Appendix D, we further studied the impact of data deduplication on memorization analysis, and found that related documents that are missed by text matching deduplication can still be identified by counterfactual memorization. Finally, we are more interested in rare examples than common ones due to the large impact on model predictions.\n\n**Q4**: common patterns (L50)?\n\nWe found examples with the highest counterfactual memorization are generally unconventional text such as all-capital letters. After those artificial examples, we see news reports of specific events. Examples with low counterfactual memorization encode common information, many are templated documents.\n\n**Q5**: contradiction on difficulty and counterfactual memorization\n\nThere is actually no contradiction here. The counterfactually memorized examples are mostly difficult, but the converse is not true. In fact, it is shown (L125, and Fig.1) that the **hardest** examples generally have low counterfactual memorization, because even when included in the training set, the models are having a hard time learning them. We will revise the text to make it more clear.\n\n**Q6**: no concentration of densities around x=y in Figure 4(b).\n\nNote the concentration won’t necessarily be on x=y, because the max-influence on a validation example does not necessarily equal the memorization (self-influence). Moreover, the main probability mass is on low memorization / low influence because the counterfactual memorization mostly captures examples in the tail. For those examples with large max-influence, we observe that memorization is also large. But the converse is not true, because the finite validation set might not contain a document that encodes matching information for each counterfactually memorized training document.\n\n**Q7**: Are sensitive data, such as phone numbers and usernames, \"common\" and frequently occurring?\n\nIt depends on the context. Phone numbers of a customer service would be considered “common” as it is likely to be found in many places. Whereas personal phone numbers are generally not “common” unless it is leaked or intentionally posted online in many places.\n\n\n**Q8**: Why “counterfactual” memorization?\n\nCounterfactual here refers to the procedure to make such measurement: we estimate the model’s prediction change if a given training document is withheld from its training corpus.\n\n**Q9**: what \"predicts x accurately\" means? It is implausible for LM to generate an entire document perfectly. \n\nWe measure the next token prediction accuracy. Following the convention in language model *evaluation*, we use teacher forcing, meaning that the prediction of the next token is always conditioned on the *true* context. So the model only needs to predict each of the next token accurately given the true context, instead of generating an entire document perfectly via decoding. Although language models are also capable of doing that for *some* training examples, as shown in previous studies of text-matching based memorization.\n\n**Q10**: T5-base has ~220M parameters\n\nWe use the decoder-only architecture for the auto-regressive language modeling task, which has around half of the parameters.\n\n**Q11**: is it possible that a training sample appears in all 400 samples?\n\nWhen generating the subsets, each example is included with probability 0.25. The probability that none of the 400 samples contain a given example is therefore 0.75^400 which is in the order of 10^-50. So it is possible but extremely unlikely. More generally, this is a binomial distribution, and so has very small tails."}]

Author rebuttal:
Thank all the reviewers for their useful comments and suggestions. We appreciate the reviewers found that our paper is clearly written, provides “strong proposal”, “sound analysis”, “useful metrics”, “novel perspective and tools”, “insightful findings” to the study of language model memorization. 

We will fix the typos and minor issues pointed by the reviewers in the updated manuscript. The main questions from each reviewer will be clarified in the individual replies below. 


Meta-review: This paper initiates and establishes a quantitative framework to study the counterfactual memorization  problem of LLMs. It receives unanimous positive reviews and rebuttal has resolved some misunderstandings. The reviewers think the problem is well motivated, timely, important, and the findings are insightful and practical. This work can be strengthened by extending the studies and analysis on larger-scale pre-trained models, and elaborating on the connection between memorization and generalization. AC agrees this work is a solid contribution on the study of memorization of LLMs, and thus recommends acceptance as a spotlight.
Decision: Accept (spotlight)