Counterfactual Memorization in Neural Language Models

Chiyuan Zhang

Google Research chiyuan@google.com

Daphne Ippolito

Carnegie Mellon University daphnei@cmu.edu

Katherine Lee

Google DeepMind katherinelee@google.com

Matthew Jagielski

Google DeepMind jagielski@google.com

Florian Tramère

ETH Zürich  
florian.tramer@inf.ethz.ch

Nicholas Carlini

Google DeepMind  
ncarlini@google.com

Abstract

Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.

1 Introduction

Modern neural language models (LMs) have achieved impressive results in generating high quality text [e.g. Brown et al., 2020, Zhang et al., 2022, Chowdhery et al., 2022, OpenAI, 2023] and have led to breakthroughs in many downstream natural language processing tasks [Devlin et al., 2019, Raffel et al., 2020b, Bommasani et al., 2021]. The paradigm of taking a single large-scale pre-trained model and fine-tuning it for many tasks motivates the study of these models' ability to generalize by avoiding memorizing their training data. Moreover, memorization of sensitive user information or copyrighted materials in the training data [Carlini et al., 2020, Vyas et al., 2023, Lee et al., 2023] leads to practical concerns in real world applications.

Previous work on memorization in neural language models demonstrated the ability to extract memorized training data, including sensitive data such as phone numbers and usernames [Carlini et al., 2020, Ziegler, 2021, Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020]. One issue with these extraction attacks is that they primarily identify "common" and frequently occurring strings in the training set. For example, as shown in the analysis of Lee et al. [2021], near-duplicate training examples, which are very common in standard text corpora, account for a large majority of the memorized content. To filter out such commonly occurring strings from all memorized texts, previous work applied various heuristic rules to distinguish frequently-occurring sequences from memorization of isolated pieces of information.

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

In this paper, we propose a principled causal perspective to disentangle memorization of common vs rare data, by directly tying a model's predictions to the presence or absence of individual training examples. We define counterfactual memorization as a measure of the change in a model's prediction when a particular example is excluded from the training set. Counterfactual memorization accounts for the commonality of an example as removing one instance of a text that is common across multiple documents will have a minor effect on the model's prediction on that text. The mathematical formulation of counterfactual memorization extends a prior definition of label memorization in classification models [Feldman, 2020] to the context of neural language modeling.

Formally, a training document  $x$  is considered counterfactually memorized, when the language model predicts  $x$  accurately if and only if the model was trained on  $x$ . This allows us to construct a procedure to quantitatively measure the memorization of isolated pieces of text, whose sole presence in the training dataset have a large effect on the model's predictions.

Following Feldman and Zhang [2020], we further extend this definition to counterfactual influence, which measures the influence of a memorized training text sample on another text example. Counterfactual influence allows us to trace the source of information for a model's predictions, by locating the training example(s) which significantly contributed to it. With these tools, we study memorization across several standard text datasets. Our main contributions are as follows:

1. We define counterfactual memorization in neural LMs which gives us a principled perspective to distinguish memorization of "rare" and "common" information in neural LMs (Section 3).
2. We estimate counterfactual memorization on several standard text datasets, and confirm that rare memorized examples exist in all of them. We study common patterns across memorized text and the memorization profiles of individual internet domains. (Section 4).
3. We identify an inverse correlation between number of duplicates and counterfactual memorization as compared with previous definitions of memorization (Section 5).
4. We extend the definition of counterfactual memorization to counterfactual influence, and study the impact of memorized examples on the test-time prediction of the validation set examples and generated examples (Section 6).

2 Related Work

Previous work analyzed the memorization of large language models on sensitive information (e.g. phone numbers) in the training data [Carlini et al., 2020, Ziegler, 2021] or synthetically injected "canaries" [Carlini et al., 2019, Henderson et al., 2017, Thakkar et al., 2020, Thomas et al., 2020]. However, not all the memorized texts are equally interesting — as confirmed in a later study [Lee et al., 2021], near-duplicated training examples are very common in standard text corpus, and those commonly occurring phrases contribute significantly to memorized texts. In order to distinguish "common" memorization of common phrases or public knowledge from "rare" memorization of private, rare information, various heuristics were adopted in previous investigations. Our paper proposed a principled perspective towards this problem. Our intuition comes from psychologies studies that categorize human (declarative) memory into episodic memory [Tulving, 1983] of specific contents of individual events, and semantic memory [Squire, 1992] about general knowledge like grammars and factual information. We would like the models to obtain semantic memory but avoid episodic memory. The capture the latter, we proposed a notion of counterfactual memorization. The mathematical formulation of counterfactual memorization is borrowed from a notion of label memorization in Feldman [2020] and adapted to the context of neural LMs in this paper. This formulation has been studied empirically in the context of computer vision in Feldman and Zhang [2020]. In a follow up work, Ilyas et al. [2022] showed that it is possible to fit a datamodel to predict the outcome of training a model on a specific training subset and evaluating on a specific input. However, this procedure requires training a massive number of models (e.g. 300,000 for CIFAR-10) on random subsets of the training data, thus is computationally infeasible for the scale of language models considered here.

The general idea of measuring model behavior on held-out training data is common in machine learning. In cross validation, held-out data is used to estimate the test performance for model selection; in learning theory, leave-one-out stability was shown to be deeply connected to generalization [e.g. Mukherjee et al., 2006]; in differential privacy, the worst case performance difference of models trained on two "neighboring" datasets (identical except a single example being held-out or replaced)

2

quantifies the privacy guarantee of a learning algorithm [Dwork et al., 2014, Nasr et al., 2021, Jagielski et al., 2020]. Most previous work aimed for an overall measurement, while our paper focused on characterizing the behaviors of individual examples.

We estimated a counterfactual influence to study how a memorized training example impacts the model prediction at test time. Influence functions have been used in statistics to assess robust estimators since Hampel [1974]. Previous papers adopted it to analyze neural network predictions [Koh and Liang, 2017, Koh et al., 2019]. However, the estimation was found to be computational expensive and fragile [Basu et al., 2021]. Pruthi et al. [2020] tracks the gradient updates during training to estimate the influence from a training example; Feldman [2020], Feldman and Zhang [2020] use aggregated statistics from multiple models independently trained on heldout data subsets to estimate the influence. Further extensions were shown to work well on detecting mislabeled data in classification problems [Wang and Jia, 2022] and characterizing hallucinations in Neural Machine Translation [Raunak et al., 2021]. Alternative methods also looked at simple data statistics (e.g. co-occurrence counts) without model re-training to infer the causal effects on language models' predictions [Elazar et al., 2022]. In this paper, we adapt the approach from Feldman [2020], and formulate counterfactual influence directly with subset sampling, as oppose to leave-one-out influence. We also extend the estimation to assess the influence on generated examples.

Counterfactual is an important notion in statistical causality [Pearl et al., 2000, Rubin, 2005, Pearl, 2009, Imbens and Rubin, 2015] useful for studying causal probabilistic inference under alternative conditions. Such counterfactuals may or may not be directly testable (e.g. a counterfactual treatment in medical studies). In this paper, we directly measure the counterfactual influence of a training example by comparing the behavior of the model trained with and without that example.

3 Counterfactual Memorization

To quantify memorization of rare details of a specific training document, we define the following notion of counterfactual memorization. The mathematical formulation is borrowed from Feldman [2020], where it was originally proposed to quantify label memorization in multi-class classification problems. We extend it to the context of unsupervised neural language modeling.

Definition 3.1 (Counterfactual Memorization). Given a training algorithm  $A$  that maps a training dataset  $D$  to a trained model  $f$ , and a measure  $M(f, x)$  of the performance of  $f$  on a specific example  $x$ , the counterfactual memorization of a training example  $x$  in  $D$  is given by

$$
\operatorname {m e m} (x) \triangleq \underbrace {\mathbb {E} _ {S \subset D , x \in S} [ M (A (S) , x) ]} _ {\text {p e r f o r m a n c e o n} x \text {w h e n t r a i n e d w i t h} x} - \underbrace {\mathbb {E} _ {S \subset D , x \notin S} [ M (A (S) , x) ]} _ {\text {p e r f o r m a n c e o n} x \text {w h e n n o t t r a i n e d w i t h} x}, \tag {1}
$$

where  $S$  and  $S'$  are subsets of training examples sampled from  $D$ . The expectation is taken with respect to the random sampling of  $S$  and  $S'$ , as well as the randomness in the training algorithm  $A$ .

That is, our memorization definition compares the difference between two expected performance measures on a given example  $x$ . On one side, we compute the expected performance of a model when trained on datasets that contain the example  $x$ , and, on the other side, we compute the expected performance of a model when trained on datasets that do not contain the example  $x$ . Throughout this paper we use per-token accuracy as the measure  $M$ . In other words, we ask the model to predict the next token based on the groundtruth context (preceding tokens), measure the 0-1 loss of the argmax token prediction, and then average it across all predicted tokens.

The expectations in Equation (1) can be empirically estimated via sampling. Specifically, we train  $m$  different models on independently sampled subsets  $S_{1},\ldots ,S_{m}$  of equal size  $|S_i| = r|D|$  for a fixed  $r\in (0,1)$ . We then divide these models into two groups: the first group contains all models trained on subsets  $S$  where  $x\in S$ ; and the second group are all models trained on subsets  $S$  where  $x\notin S$ . We take the average performance on  $x$  in the two groups separately and compute the difference between the two:

$$
\widehat {\operatorname {m e m}} (x) \triangleq \underset {i: x \in S _ {i}} {\text {m e a n}} [ M (A (S _ {i}), x) ] - \underset {i: x \notin S _ {i}} {\text {m e a n}} [ M (A (S _ {i}), x) ]. \tag {2}
$$

This difference quantifies how the presence or absence of the example  $x$  in a model's training set affect the model's performance on  $x$ . If there is a large difference between including an example in the training set versus not including it, then we consider this example counterfactually memorized.

3

For each  $x$ , we refer to models trained with  $x$  in the training set ( $\{A(S_i) : x \in S_i\}$ ) as IN models and the models  $x$  was not trained on ( $\{A(S_i) : x \notin S_i\}$ ) as OUT models. Note we do not need to retrain a model for each example  $x$ . Instead, we train  $m$  models once on random subsets of  $D$ , and compute the estimation (Equation 2) for all examples using the same set of  $m$  models. Ilyas et al. [2022] recently showed that it may also be possible to directly predict these scores using a regression model, yet this approach is computationally prohibitive for large language models.

4 Analyzing Counterfactual Memorization

We estimate and analyze counterfactual memorization of training examples in three standard text datasets: RealNews [Zellers et al., 2019], C4 [Raffel et al., 2020a] and Wiki40B:en [Guo et al., 2020]. Unless otherwise specified, we use Transformer-based language models [Vaswani et al., 2017] equivalent to (decoder only) T5-base [Raffel et al., 2020b] with  $\sim 112\mathrm{M}$  parameters. To save computation and enable more direct comparisons across datasets, we truncate the training set for each datasets by taking the first  $2^{21}$  documents. To estimate counterfactual memorization, we train 400 models for each dataset, each on a random  $25\%$  subset of the training examples. In practice, we use a hash-based filtering mechanism to efficiently approximate random subset sampling (details in Appendix G), as the data loading APIs for large text corpora generally support only sequential visits to examples with limited shuffling and subsampling capability within a window.

We train each model for 60 epochs $^1$  using the Adam optimizer [Kingma and Ba, 2015] with learning rate 0.1 and weight decay  $10^{-5}$ . For C4/RealNews/Wiki40B:en, respectively, our models converge to an average per-token accuracy of  $44.21\% / 47.59\% / 66.35\%$  on the subsampled training set, and  $27.90\% / 31.09\% / 49.55\%$  on the validation set. On average, the models start to overfit at around epoch 5, as indicated by the signal that the validation accuracy starting to decrease.

4.1 Distribution of Memorization

Table 1 shows examples from the RealNews training set sampled at various memorization levels. Examples with the highest memorization are generally unconventional text such as all-capital letters, structured formats (i.e., tables or bullet list), and multilingual texts. After those artificial examples, examples with intermediate-to-high memorization are most often news reports of specific events. One of our main goals is to be able to separate memorization of such examples containing details of specific events from memorization of common facts or highly duplicated template texts. Indeed, templated documents with many near-duplicate copies in the training data generally have low counterfactual memorization. C4 and Wiki40B:en have similar trends. Interestingly, though Wikipedia articles are less likely to be auto-generated from templates than the web in general, we do observe repetitive patterns in low-scoring documents, such as “_START ARTICLE_<place name>, Virginia _START_PARAGRAPH_<place name> is an unincorporated community in <county name>, in the U.S. state of Virginia.”

To visualize the distribution of memorization, we plot 2D histograms in Figure 1, where the x-axis shows the difference of IN-accuracy and OUT-accuracy (i.e. the counterfactual memorization), and the y-axis shows the sum of the two, which we term "simplicity". A simple example is one that is scored highly regardless of whether a model saw it during training. The histograms are plotted in log scale to better visualize the exponential decay in the tail for high memorization and simplicity levels.

From the 2D density plots, we find that easy examples tend to have low memorization. However, there is no simple linear correlation. Peak memorization occurs for examples of intermediate simplicity. For the hardest examples, the memorization scores are low, because even the IN-models could not learn them well. Many hard examples consist of ill formatted text or contained foreign languages. As a result, in Wiki40B:en, which contains higher quality texts, the lower bound of the histogram is higher than the other two datasets (Figure 1). Interestingly, the choice of data has a relatively minor effect on memorization: the shape of the memorization histogram is generally consistent across the three datasets; the range of memorization values is only slightly compressed for Wiki40B:en.

Modern language models are usually trained for fewer epochs if the training set is massive. Since we have a smaller subsampled training set, we train the models for more epochs to allow the models to fit the training data sufficiently to study memorization effects.

4

Table 1: Examples of RealNews training set sampled at high, intermediate and low memorization. The URL of each document is included at the beginning of each example. [...] indicate omitted text for brevity. In the last block, two near-duplicate examples are shown; the yellow highlights in the last block indicate differences.
Index	mem	Text
2090855	0.6546	link ➔ THE AMERICAN JEWISH CONGRESS ANNOUNCED TODAY THE PUBLICATION OF A REPORT ON JEWISH NON-EMPLOYMENT AS A RESULT OF ECONOMIC DISCRIMINATION, [...] THEREAFTER ONE OF THE DEPARTMENTS OF A.T.&T.” ALMOST UNPRECEDENTLY” ENGAGED A JEWISH APPLICANT.
2085736	0.5755	link ➔ x RECIPE: Chinese Pork & Vegetable Soup with Wonton Noodles Chinese Pork & Vegetable Soup with Wonton Noodles 1 pork tenderloin (about 1-1 1/4 pound size), cooked and cut into 1/2-inch cubes* 5 cups lower-sodium chicken broth 1 cup water** 1/4 cup [...] Makes 4 servings (about 1 1/2 cups each) Recipe by PorkBeInspired.com with adaptations by culinary dietitian & nutritionist Kim Galeaz, RDN CD
1680600	0.5807	link ➔ Language English [...] Arabic text [...] acknowledgement of country [...] Arabic text [...] I would like to acknowledge that this meeting is being held on the traditional lands of the (appropriate group) people, and pay my respect to elders both past and present.” [...]
2074805	0.2835	link ➔ A Texas honors student punished for saying that homosexuality was wrong has had his suspension rescinded [...] Western Hills High made the correct decision in reversing their course of action. “The decision to rescind the suspension is the correct one. The suspension was wrong and improper,” said Staver. “I applaud the student for standing up. We stood with him to resist an unjust suspension and we are pleased that suspension has been reversed.” [...] Liberty Counsel will continue the right to exercise freedom of conscience and religion,” said Staver. “These instances are increasing and will continue to increase unless Christians and people who love liberty stand up and resist this intolerance.”
449808	0.0361	link ➔ Investors in Digital Realty Trust, Inc. (DLR) saw new options begin trading this week, for the February 2014 expiration. At Stock Options Channel, our YieldBoost formula has looked up and down the DLR options chain for the new February 2014 contracts and identified one put and one call contract of particular interest. The put contract at the $45.00 strike price has a current bid of $1.00. [...]
1157311	0.0356	link ➔ Investors in Abercrombie & Fitch Co. (ANF) saw new options become available today, for the April 4th expiration. At Stock Options Channel, our YieldBoost formula has looked up and down the ANF options chain for the new April 4th contracts and identified one put and one call contract of particular interest. The put contract at the $34.00 strike price has a current bid of $1.97. [...]

Figure 1: The joint distribution of counterfactual memorization (X axis) and simplicity (Y axis), where simplicity is measured as the overall accuracy for an example across all models. (Histograms are in log-scale).

Figure 1 shows the overall distribution of memorization for each training datasets. To obtain a more granular view, we can also analyze the distributions for texts sourced from individual web domains in RealNews and C4, to see whether different data sources display different memorization profiles. Web domains such as news portals, blogs, and forums differ both stylistically and in how much they reference or even copy from other websites. Additionally, some domains are represented much more frequently than others in the datasets we studied. This could lead to considerably different memorization profiles for examples from different domains.

To investigate these effects, we visualize the 95th percentile memorization score in each web domain against the number of examples in that domain for RealNews (Figure 2a) and C4 (Figure 2b). C4 contains many more domain names than RealNews since the latter is collected only from news websites. For both datasets, the domains with a large number of crawled documents show a smaller variance in the 95-percentile values, while "smaller" domains depict a wide range of variety in memorization profiles. The memorization profiles of a few representative domains are visualized in Figures 2c and 2d. The domains we selected for visualization are: the largest domain (blue), the domain with highest 95 percentile memorization (orange), and two domains that have more than 1000 and 50 articles in RealNews and C4 respectively (green and red).

In RealNews (Figure 2c), reuters.com contains the largest number of documents but low memorization scores on average. The domain digitallibrary.un.org, the United Nations Digital Library, has high memorization scores potentially because it contains many multilingual documents. We have observed that less frequently occurring tokens, like those in foreign languages or ALL-CAPITAL words tend to cause high memorization. Similarly, flattened structured data (e.g. tabular texts) also deviates significantly from normal English texts and potentially leads to high memorization, as demonstrated by zap2it.com, a website for TV program listings. On the other hand, hotair.com is a

5

(a)

(b)

(c)

(d)
Figure 2: For each web domain, we plot the 95-percentile of memorization against the number of examples from that domain in (a) RealNews and (b) C4. The red dotted line indicates a threshold of a minimum of 1000 articles for RealNews and 50 articles for C4. The memorization distributions of a few representative domains are shown for (c) RealNews and (d) C4.

news commentary website that frequently quotes other major news articles. This may lead to duplicate text in the dataset which we suspect contributes to its overall lower memorization distribution.

The observations are similar on C4: blogspot.com contains a large number of documents in the training set with only moderate amounts of memorization; zh.wikipedia.org and buckingham autos. com. au have high memorization due to foreign (Chinese) or structured (car sales listings) text; and www. unitedstateszipcodes.org has very low memorization scores because common templates are re-used to generate similar pages for individual zip codes.

4.2 Number of Models Needed

To evaluate the impact of a single training example, one may wish to train two models that differ only in that single example. In practice, the stochasticity in a single run of common training algorithms (e.g. SGD) produces too low signal-to-noise ratios to be useful for such estimation. Moreover, leave-one-out estimation means a separate pair of models needs to be trained for each training example, which is computationally costly. Therefore, we formulated our estimation in Section 3 by accumulating statistics from  $m$  models independently trained on random training subsets. In our experiments, we set  $m = 400$ . To understand how sensitive our results are to  $m$ , we analyze the rankings produced by distinct sets of models of size  $m$ . We vary  $m$  from 6 to 192, and partition our set of 400 models into up to 10 sets of  $m$  models (e.g. for  $m = 192$ , we construct 2 partitions, and for  $m = 6$ , we construct 10). We then compute the Spearman's R between these partitions to measure the agreement between the rankings produced by each partition. If the rankings are very similar (have Spearman's R close to 1), then this number of models is reliably estimating the true ranking of memorization scores. We plot these Spearman's R values in Figure 3a. Even at 96 models, this correlation begins to plateau near 1, lending confidence that 400 models is sufficient for reliable estimation of memorization scores. See Appendix D for more analysis on the sensitivity to  $m$ .

4.3 Impact of Number of Training Epochs

As expected, the overall amount of memorization grows consistently with the number of epochs of training (Figure 3b). This makes sense since training for more epochs increases overfitting. As training progresses, we also see an increasingly long tail of examples with high memorization scores. On RealNews, about  $59\%$  of examples had consistently increasing memorization scores across all epochs considered. There were no examples whose memorization decreased in a significant way over training (all observed decreases can be attributed either to noise or to instability early in training). Only  $0.5\%$  of examples stayed completely un-memorized with scores which never rose above 0.1,

6

Figure 3: (a) Spearman's R between memorization rankings from two disjoint sets of  $m$  models. The rankings are variable at low numbers of models, but starts to converge at 192 models. All of our other experiments use 400 models. Reported values are averages over up to 10 partitions, with error bars of 10 standard deviations. (b) The distribution in memorization of RealNews examples as training progresses. (c) The fraction of RealNews examples with memorization consistently above the specified threshold as training progresses. (d) For RealNews, we plot memorization scores against the number of near-duplicates an example had in the dataset.

while  $85\%$  of examples had memorization scores which never rose above 0.2. Figure 3c shows the fraction of memorized examples as training progresses, at several thresholds of memorization. We can see that more training epochs significantly increases memorization.

5 Duplicate Text and Memorization

One of the goals of evaluating counterfactual memorization is to identify examples that have a low number of duplicates yet whose presence versus absence in the training data has a large effect on the model. Here, we perform a quantitative study of the (anti-)correlation between duplication and counterfactual memorization compared with the positive correlation between duplication and the "generation-time memorization" definitions of memorization used by Lee et al. [2021], Carlini et al. [2022], Kandpal et al. [2022].

Following the method from [Lee et al., 2021], we first use MinHash [Broder, 1997] to identify near-duplicate examples in RealNews train set. We consider a example a duplicate if it has an normalized edit similarity of greater than 0.7 (definition included in Appendix I). Out of 2.01 million examples,  $\sim 38,000$  were identified as being a near-duplicate with at least one other example. Among these frequently-occurring examples, the Pearson correlation between an example's counterfactual memorization score and the number of near-duplicates for that example is -0.39; in other words, memorization does quantitatively decrease when data is repeated more often.

In Figure 3d we can see that examples with a large number of near-duplicates have smaller memorization scores. Counterfactual memorization primarily differentiates amongst examples with a few number of duplicates. This makes sense given that examples with lots of near duplicates would likely have their near duplicates in OUT-models. This is to be contrasted with "generation-time memorization" (discussed in Section A) that measures the textual overlap between model generated texts and the training documents. There, the number of occurrences strongly correlate with the measured memorization [Carlini et al., 2020, Lee et al., 2021, Kandpal et al., 2022]. Counterfactual memorization measures a fundamentally different type of memorization from simple textual matching considered in prior work, providing information about how easy or hard a training example is in the context of the rest of the training set. In Table 1 we can see this effect qualitatively: sequences with near-duplicates in the training set tend to have low counterfactual memorization (as expected).

6 From Memorization to Influence

Counterfactual memorization identifies training examples that contain rare information not conveyed by other examples. A natural question to ask is whether a model would leak the information in a memorized example during inference. Previous paper studies membership inference attack [Shokri et al., 2017, Sablayrolles et al., 2019, Long et al., 2020] where an attacker tries to figure out if a particular example exists in the training set. In this paper, we consider standard model evaluation without adversarial attackers, and quantify "does seeing a particular training example strongly influence the prediction on a validation example?" Another way of asking this is if a single example in the training set has an large and over-representative impact on the prediction of a validation

7

Figure 4: (a) Histogram of the influence of all the training examples on a specific test example for three different test examples on RealNews. The blue and orange examples have high and intermediate influence from some training examples, as indicated by the outlier values to the right of the each histogram plot. The green one is a random example, where the influence from all individual training examples are close to zero. (b) The joint distribution of the memorization score of each training example and its maximum influence on any validation set example. The histograms are in log scale to better visualize the tail of the distributions. C4 shown in Figure 6.

example. We answer these questions by measuring counterfactual influence with a formulation adapted from Feldman and Zhang [2020]:

Definition 6.1 (Counterfactual Influence). Given a training algorithm  $A$  that maps a training set  $D$  to a trained model, and a performance measure  $M$ , the counterfactual influence of a training example  $x \in D$  on another example  $x'$  is

$$
\operatorname {i n f l} \left(x \Rightarrow x ^ {\prime}\right) \triangleq \mathbb {E} _ {S \subset D, x \in S} \left[ M \left(A (S), x ^ {\prime}\right) \right] - \mathbb {E} _ {S \subset D, x \notin S} \left[ M \left(A (S), x ^ {\prime}\right) \right], \tag {3}
$$

where  $S$  is a subset of training examples sampled from  $D$ . The expectation is taken with respect to the random sampling of  $S$ , as well as the randomness in the training algorithm  $A$ . Here  $x'$  can be an example from the validation set or test set, a generated example or a training example.

An empirical estimation of the influence can be computed similarly to counterfactual memorization by uniformly sampling  $m$  subsets  $S_{1},\ldots ,S_{m}$  from  $D$ , where  $|S_i| = r|D|$ , and calculating

$$
\widehat {\operatorname {i n f l}} \left(x \Rightarrow x ^ {\prime}\right) \triangleq \underset {i: x \in S _ {i}} {\operatorname {m e a n}} \left[ M \left(A \left(S _ {i}\right), x ^ {\prime}\right) \right] - \underset {i: x \notin S _ {i}} {\operatorname {m e a n}} \left[ M \left(A \left(S _ {i}\right), x ^ {\prime}\right) \right]. \tag {4}
$$

This measures how much a training sample  $x$ 's presence influences the prediction of a different example  $x'$ . Note,  $\mathsf{mem}(x) = \inf \left( x \Rightarrow x \right)$ , i.e., counterfactual memorization is self influence.

Influence on Examples of the Validation Set. With the same models trained for estimating memorization, we can estimate the counterfactual influence on the validation set according to Equation (4). For each example in the validation set, we can estimate the influence on it from each training example. Figure 4a shows the distribution of influence from all training example on three different examples from the validation set. The green example was randomly chosen and represents the behavior for most validation examples: it receive close-to-zero influence from all the (individual) training examples. The blue and orange examples were sampled to have high and intermediate maximum influence. Each of them has one (or a few) strong influencer from the training set, as indicated by the bars to the right of the histogram. They also only receive tiny influence from all the rest of the training examples, though the variance of influence is larger than for the green example.

Intuitively, most training examples will have small influence on validation set examples because the models learn distributional patterns shared across many training examples, and individual training examples tend to have insignificant influence here. However, a training example  $x$  with high counterfactual memorization contains rare information that are not shared with other examples. Therefore, if a validation set example  $x'$  contains similar information,  $\mathrm{infl}(x \Rightarrow x')$  could be large. Figure 4b shows the relationship between memorization and influence by plotting  $\mathrm{mem}(x)$  of each training example  $x$  against its maximum influence  $\max_{x'} \mathrm{infl}(x \Rightarrow x')$  on  $x'$  across the validation set.

Consistent with our intuition, examples with small memorization scores have small max-influence scores. Larger influence scores on the validation set generally requires larger memorization scores

8

Table 2: Train-validation example pairs of RealNews sampled at a variety of influence levels. [...] indicate text omitted for brevity. Differences in each document pair are highlighted yellow.
Index	Estim.	Text
Validation 1334662 Train 2077314	infl 0.3780 mem 0.3780	link > Identical URL except with the http://protocol instead of https://. The text is identical to the training example. link > By Ari Rabinovitch TZEELIM VALLEY, Israel (Reuters) - The disposable paper face masks offer little protection from the clouds of dust that fill the cliffside cave where Israeli archaeologists are wrapping [...] insight into how people lived 2,000 to 8,000 years ago. (Editing by Jeffrey Heller/Jeremy Gaunt)
Validation 838341	infl 0.1209	link > VATICAN CITY (AP) — Emeritus Pope Benedict XVI is offering a first-ever papal assessment of his own pontificate in a book that recounts his decision to resign, his surprise at his successor and his attempts to dismantle what he calls the Vatican's "gay lobby." "Benedict XVI: The Final Conversations," is due out in September, the latest [...] Vatican career by simply spreading gossip that he was gay. Different websites, but (almost) identical report.
Train 614881	mem 0.1650	link > VATICAN CITY — Emeritus Pope Benedict XVI is offering a first-ever papal assessment of his own pontificate in a book that recounts his decision to resign, his surprise at his successor and his attempts to dismantle what he calls the Vatican's "gay lobby." "Benedict XVI: The Final Conversations," is due out in September, the latest [...] Vatican career by simply spreading gossip that he was gay. Follow Nicole Winfield at www.twitter.com/Nwinfield
Validation 682107	infl 0.0673	link > ANAHEIM – On a night when Francois Beauchemin had two assists in Toronto, and Chris Pronger blocked six shots in Detroit, 13,869 Ducks' fans might have been lost without their programs on Wednesday night. It's only the first game of the preseason, but a new era has clearly begun. A group of mostly newcomers in Ducks uniforms beat Phoenix 3-2 in a shootout on Wednesday at Honda Center. A familiar face made the biggest impact, however, as Bobby Ryan scored two goals. Ryan also scored two goals in the Ducks' preseason opener last year [...] Different websites on the same event with slightly different wordings.
Train 494435	mem 0.1439	link > ANAHEIM – On a night when Francois Beauchemin had two assists in Toronto and Chris Pronger blocked six shots in Detroit, the 13,869 Ducks' fans who showed at Honda Center Wednesday needed programs to identify the players on their favorite team. It's only the first game of the preseason, but a new era has clearly begun. A group of mostly newcomers in Ducks uniforms beat Phoenix 3-2 in a shootout, although a familiar face made the biggest impact as Bobby Ryan scored two goals in regulation and another in the shootout. Ryan also scored two goals in the Ducks' preseason opener last year [...]
Validation 1165799	infl 0.0360	link > x More than 70,000 pounds of Butterball turkey recalled because of potential salmonella WASHINGTON — The U.S. Department of Agriculture's Food Safety and Inspections services announced on Wednesday [...] They were shipped to nationwide retail and institutional locations. RELATED: View the full recall FSIS, the Centers for Disease Control and Prevention and [...] Different websites reporting the same event, one embedded a lot more information than the other.
Train 1571976	mem 0.2094	link > x Butterball recalls nearly 80,000 pounds of turkey after salmonella cases WASHINGTON — The U.S. Department of Agriculture's Food Safety and Inspection service announced Wednesday [...] The raw ground turkey was produced on July 7, 2018. The following products under recall were shipped to nationwide retail and institutional locations: 48-oz. plastic wrapped tray containing "BUTTERBALL every day Fresh Ground Turkey WITH NATURAL FLAVORING (85% LEAN/15% FAT)" with sell or freeze by date of 7/26/18, lot code 8188, and UPC codes 22655-71555 or 22655-71557 represented on the label. 48-oz. plastic wrapped tray containing "BUTTERBALL everyday Fresh Ground Turkey WITH NATURAL FLAVORING (93% LEAN/7% FAT)" with sell or freeze by date of 7/26/18, lot [...] labels here, FSIS, the Centers for Disease Control and Prevention and [...]

of the training example itself. However, not all training examples with large memorization scores lead to large influence scores. In particular, the max-influences drop significantly for examples with memorization larger than 0.4. One potential reason is that many examples with very high memorization are simply low quality text, so memorization is required in order to learn them, but they do not encode anything interesting that could influence a validation example. On the other hand, even if a memorized example encodes some rare and useful information, the max-influence could still be low because the validation set does not contain a relevant document. This is especially true given that all datasets have considerably smaller validation sets than training sets.

Table 2 shows train-validation example pairs from RealNews sampled at different influence value ranges. We found that the train-validation pairs with the highest influence are almost identical, except some superficial differences, such as different handling of quotation / em dash marks. As we move to intermediate influence ranges, we commonly found reports on the same events. Large paragraphs of identical text indicate that one document might be citing the other or both citing from a third party. At low influence, two types of correlations are commonly observed: 1) templated texts with high similarity—the reason for a low influence is that there are many similar training examples that split the influence; 2) superficially related documents due to a shared prefix such as ST. CLOUD – This week in our “Behind the Scenes” series on WJON or a shared substring of some common knowledge like FSIS, the Centers for Disease Control and Prevention. Due to high signal-to-noise ratio, here were no noticeable relationships in the document pairs with influence scores below 0.02.

Influence turns out to be an effective tool for analyzing and attributing the model predictions at test time: for predictions that rely on information obtained by (counterfactual) memorization, we can identify exactly which training example provided such information. Our observation of near-duplicated training-validation document pairs is consistent with recent studies that identifies data contamination in large Internet crawled text corpus [Lee et al., 2021, Dodge et al., 2021].

Influence on Generated Texts. The influence estimation is not restricted to the validation set. We can also estimate influence on generated examples. In this section, we evaluate on the publicly released generations from the Grover models [Zellers et al., 2019] trained on RealNews. Specifically, we take the generations from Grover-Mega  $(\mathrm{p} = 0.96)$ , a 1.5-billion-parameter model trained on the RealNews

9

dataset. Comparing with the train-validation influence in Figure 4b, the histogram (c.f. Figure 10 in Appendix.) decays faster as max-influence grows. Moreover, the value range of max-influence is also twice smaller. The reason that we did not find a lot of highly influenced generated examples are two fold: 1) there are only 24,576 generation in the public release, which is much fewer than the validation examples. As a result, the corresponding example of many memorized training examples do not get sampled in the generations. For comparison, previous work [Carlini et al., 2020, Lee et al., 2021] generated  $100,000+$  examples to identify memorization in generation. These approaches also count duplicates in the training set, which counterfactual memorization filters out. 2) The Grover model was trained on the full RealNews training set, while we have restricted our analysis to the first 2M training examples. There could be potentially more high influence training examples that are missed in our calculation.

7 Summary and Discussion

We studied memorization in neural language models. We formulated a notion of counterfactual memorization as a tool that can systematically ignore "common" memorization such as general knowledge (e.g. "Paris is a city in France") and captures memorization of rare, specific information (e.g. description of a specific episode of event) present in the training examples. We conducted experiments on three commonly used text corpus in language modeling and found memorization in all of them. We further analyze the per-domain memorization profiles for Internet-crawled data, and found that different sources could have substantially different memorization profiles.

Furthermore, we analyzed how memorized training examples could impact the model predictions at test time via counterfactual influence. We found that for examples from both the validation set and the model generated texts, the model predictions could be drastically different depending on the presence or absence of a particular training example with high memorization.

Limitations. This study mainly focuses on English datasets. While we expect the characterization of memorization would be similar when evaluated on corpus of other (natural) languages, new patterns might be observed on multilingual data or more structured domains such as programming languages.

Both the neural language models and training sets used in this work are orders of magnitude smaller than modern standards such as GPT-3 [Brown et al., 2020], GPT-4 [OpenAI, 2023] and PaLM-2 [Google, 2023]. Moreover, we only conducted preliminary investigation of the dynamics of counterfactual memorization during training. Although our experiments effectively estimated and detected memorization, we suspect more interesting examples might emerge if larger, more capable models are analyzed. For example, currently when the information from a memorized training example is leaked in the prediction of a strongly influenced test example, it can usually be explained by a high text overlap between the training and test examples. For models with deeper understanding of languages, we suspect that strong influence could be observed even between documents that have no direct text overlap but that encode similar semantic information.

In order to test this, it will be necessary to scale our framework to larger models and datasets. Moreover, it will be necessary to construct datasets where semantically similar but textually different document pairs exist. One potential source to construct such datasets would be versioned Wikipedia articles—two versions of the same article with large time span or edit distance may contain semantically similar (but paraphrased) information. Such a dataset of paraphrased text pairs would be more broadly useful to understand the ability of different models to disentangle text content and form—by measuring the influence of one piece of text on a paraphrased piece of text.

Counterfactual memorization enables us to identify examples that whose presence or absence has a large impact on the model and the model's ability to score and generate other text. The privacy risk for this is low since in order to perform this analysis, one would need to already have access to the dataset and the ability to train models.

Acknowledgments. The authors would like to thank Samy Bengio, Christopher A. Choquette-Choo, Ethan Dyer, Michael C. Mozer, Behnam Neyshabur, Andrew Nystrom, and Hanie Sedghi for constructive discussions and feedback. The authors would like to thank Andrew Nystrom for assistance with MinHash-based near-duplicate detection.

10

References

Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. ArXiv, abs/2006.14651, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Re, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W Thomas, Florian Tramèr, Rose E Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. August 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE, 1997.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bf8ac142f64a-Paper.pdf.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.

11

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423.
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. Documenting the english colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211-407, 2014.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. Measuring causal effects of data statistics on language model's factual predictions. arXiv preprint arXiv:2207.14251, 2022.
Vitaly Feldman. Does learning require memorization? A short tale about a long tail. In STOC, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
Google. Palm 2 technical report. Technical report, Google, 2023.
Mandy Guo, Zihang Dai, Denny Vrandecic, and Rami Al-Rfou. Wiki-40b: Multilingual language model dataset. In LREC 2020, 2020.
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-393, 1974.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http://github.com/google/flax.
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems, 2017.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datab-models: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022.
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd?, 2020.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The International Conference for Learning Representations, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. ArXiv, abs/1703.04730, 2017.
Pang Wei Koh, Kai-Siang Ang, Hubert Hua Kian Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. In NeurIPS, 2019.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637-3647, 2023.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. CoRR, abs/2107.06499, 2021. URL https://arxiv.org/abs/2107.06499.

12

Yunhui Long, Lei Wang, DIYue Bu, Vincent Bindschaedler, Xiaofeng Wang, Haixu Tang, Carl A Gunter, and Kai Chen. A pragmatic approach to membership inferences on machine learning models. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pages 521-534. IEEE, 2020.
Sayan Mukherjee, Partha Niyogi, Tomaso Poggio, and Ryan Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1):161-193, 2006.
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adversary instantiation: Lower bounds for differentially private machine learning, 2021.
OpenAI. Gpt-4 technical report. arXiv, 2023.
Judea Pearl. Causal inference in statistics: An overview. 2009.
Judea Pearl et al. Causality: Models, reasoning and inference. Cambridge, UK: Cambridge University Press, 19(2):3, 2000.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracking gradient descent. *ArXiv*, abs/2002.08484, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020a. URL http://jmlr.org/papers/v21/20-074.html.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020b. URL http://jmlr.org/papers/v21/20-074.html.
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172-1183, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.92. URL https://aclanthology.org/2021.naacl-main.92.
Donald B Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469):322-331, 2005.
Alexandre Sablayrolles, Matthijs Douze, Yann Ollivier, Cordelia Schmid, and Herve Jégou. White-box vs black-box: Bayes optimal strategies for membership inference, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2017.
Larry R Squire. Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychological review, 99(2):195, 1992.
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning, 2020.
Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow. Investigating the impact of pre-trained word embeddings on memorization in neural networks. In International Conference on Text, Speech, and Dialogue, pages 273-281. Springer, 2020.
Endel Tulving. Elements of episodic memory. Oxford University Press, 1983. ISBN 9780198521259.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.

13

Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. arXiv preprint arXiv:2302.10870, 2023.
Tianhao Wang and Ruoxi Jia. Data banzhaf: A data valuation framework with maximal robustness to learning stochasticity. arXiv preprint arXiv:2205.15466, 2022.
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Ruhle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, Oct 2020. doi: 10.1145/3372297.3417880. URL http://dx.doi.org/10.1145/3372297.3417880.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In Advances in Neural Information Processing Systems, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Albert Ziegler. *GitHub Copilot: Parrot or crow?* https://docs.github.com/en/github/copilot/researchrecitation, 2021.

14

Figure 5: Per-token accuracy of training examples evaluated on IN models vs OUT models.

A Difference Between Counterfactual and Generation-Time Memorization

Many definitions of memorization operate at generation-time: a sequence of generated text is marked as memorized if a sufficient amount of overlap is found in the training dataset [Carlini et al., 2020]. When the training data is not available, heuristic-based methods comparing language model perplexities are used to predict whether a generation contains memorized content [Carlini et al., 2019, Thakkar et al., 2020, Thomas et al., 2020, Carlini et al., 2020, Zanella-Béguelin et al., 2020]. One difficulty with these approaches is that generation-time instances of memorization are strongly correlated with the number of similar or near-duplicate examples in the training set. As observed in Lee et al. [2021], large clusters of near-duplicated examples do exist in common language datasets, dominating memorization detected in generated text. Generation-time methods for measuring memorization are forced to design heuristics to avoid simply identifying these uninteresting instances of memorization.

In contrast, the counterfactual memorization we study in this paper handles the issue of near-duplicates automatically without the need for heuristics. For a training example,  $x$ , with many near-duplicate copies in the training set,  $\mathrm{mem}(x)$  will be small (because other samples  $x' \approx x$  will be present in the training dataset whether or not  $x$  is). This does not mean that counterfactual memorization is the opposite of generation-time memorization. An example,  $x$ , with high  $\mathrm{mem}(x)$  may have a high chance of being generated if a model is appropriately prompted, despite and possibly because it is rare, and thus the example is considered memorized by both definitions. In summary, generation-time memorization measures the chance a model will directly copy from training examples, while counterfactual memorization aims to discover rare information that is memorized.

B Average Accuracy of IN models vs OUT models

Figure 5 compares the per-token accuracy between the IN models and OUT models for the training examples from three different datasets. Counterfactual memorization is estimated by taking the difference between the average IN-accuracy and the average OUT-accuracy. Thus, the examples closer to the upper left corner are more counterfactually memorized, while the examples near the diagonal are not.

C The Impact of Data Dedduplication on Memorization

To investigate the impact of data dedduplication on counterfactual memorization, we compared C4 with C4-NEARDUP [Lee et al., 2021], which is derived from C4 with dedduplication using approximate document matching. Figure 7 compares the distribution of memorization between the original C4 and the duplicated dataset. We did not find significant difference between the two datasets. One potential reason is that the dedduplication criterion was relatively conservative, which removed only  $\sim 3\%$  of the training examples. In fact, we can still easily see near duplicate examples in C4-NEARDUP among examples with low memorization, as shown below:

Example 1380925 (mem = 0.0374) link  $\triangleright$  This is a placeholder page for Joshua Baldridge, which means this person is not currently on this site. We do suggest using the tools below to find Joshua Baldridge. You are visiting the placeholder page for Joshua Baldridge. This page is

15

Figure 6: Full version of Figure 4b: The joint distribution of the memorization score of each training example and its maximum influence on any validation set example. The histograms are in log scale to better visualize the tail of the distributions.

here because someone used our placeholder utility to look for Joshua Baldridge. We created this page automatically in hopes Joshua Baldridge would find it. If you are not Joshua Baldridge, but are an alumni of Brecksville Broadview Heights High School, register on this site for free now.

Example 2048352 (mem = 0.0320) link  $\triangleright$  This is a placeholder page for Laytoya Brannon, which means this person is not currently on this site. We do suggest using the tools below to find Laytoya Brannon. You are visiting the placeholder page for Laytoya Brannon. This page is here because someone used our placeholder utility to look for Laytoya Brannon. We created this page automatically in hopes Laytoya Brannon would find it. If you are not Laytoya Brannon, but are an alumni of Mainland High School, register on this site for free now.

Example 1314053 (mem = 0.0278) link  $\triangleright$  This is a placeholder page for Devin Mcguire, which means this person is not currently on this site. We do suggest using the tools below to find Devin Mcguire. You are visiting the placeholder page for Devin Mcguire. This page is here because someone used our placeholder utility to look for Devin Mcguire. We created this page automatically in hopes Devin Mcguire would find it. If you are not Devin Mcguire, but are an alumni of Kankakee Valley High School, register on this site for free now.

Example 1085524 (mem = 0.0209) link  $\triangleright$  This is a placeholder page for Anthony Christie, which means this person is not currently on this site. We do suggest using the tools below to find Anthony Christie. You are visiting the placeholder page for Anthony Christie. This page is here because someone used our placeholder utility to look for Anthony Christie. We created this page automatically in hopes Anthony Christie would find it. If you are not Anthony Christie, but are an alumni of Old Bridge High School, register on this site for free now.

Figure 7: The joint distribution of memorization and simplicity. The histograms are plotted in log scale to better visualize the tail of the distributions.

16

Figure 8: Spearman's R between memorization rankings from a set of  $m$  models and our full set of 400 models. As more models are trained, the ranking changes very little, with the ranking at 192 models having a Spearman's R of at least 0.992 on all datasets.
(a) RealNews

(b) C4
Figure 9: The variance in memorization scores decreases significantly as the number of models increases for all 3 datasets.

(c) Wiki40B:en

Measurements of the edit distances show that they are near the boundary of the dedduplication threshold chosen in Lee et al. [2021]. On the other hand, the tail of the distribution — examples with high counterfactual memorization are mostly unaffected by text dedduplication.

D Variance of Memorization Scores

In Figure 8, we measure the Spearman's R between our total set of 400 models and an  $m$  model subset. As expected, as  $m$  increases, so does Spearman's R—in particular, at 192 models, the Spearman's R is at least  $99.2\%$  for all datasets, and increasing  $m$  already appears to have diminishing returns.

Using the same partitioning into size  $m$  sets of models, we analyze the variance of memorization scores assigned to each sample. To do this, within each partition, we compute the memorization score assigned to each sample. We then compute the standard deviation of all partitions' memorization scores for each sample. In Figure 9, we plot each sample's standard deviation — in all, this demonstrates the distribution of the variance of memorization scores. We find that the variance decreases substantially as  $m$  grows, and concentrates near 0 already with  $m = 192$ , for all datasets.

E Histogram of Max-Influence on Generated Texts

Figure 10 shows the histogram of max-influence on each generated example by Grover-Mega  $(p = 0.96)$  [Zellers et al., 2019], from the RealNews training examples. Those generated examples are publicly released at https://github.com/rowanz/grover/tree/master/generation/examples.

F Miscellaneous Experiment Details

Our experiments are implemented using JAX [Bradbury et al., 2018] and Flax [Heek et al., 2020], both open sourced library under the Apache-2.0 license. In the study of influence on generated texts, we use the publicly released generations from the Grover models [Zellers et al., 2019], available at their open source code repository, under the Apache-2.0 license.

We run the experiments using our internal cluster. The majority of the compute is consumed by model training. In this paper, we use standard training setup for transformer based neural language models,

17

Figure 10: Histogram of max-influence on each generated example by Grover-Mega (p=0.96), from the RealNews training examples.

Figure 11: Comparison of hash based subset sampling with numpy.random.choice.

which could run on single node machines with one or multiple GPUs. However, to carry out the full analysis, we need to train 400 different models for each of the three datasets analyzed in this paper.

G Subsampling Procedure

In the estimation of memorization and influence, we trained 400 models each on an independent random subset of training examples. We use Tensorflow Datasets (TFDS)  $^{2}$  to load our training data. TFDS supports loading a continuous range of examples, but does not support subset loading from a list of indices of individual examples. The API has a filter function which allows us to provide a Tensorflow predicate to precisely control the subset loading. However, a naive implementation of checking whether the index of the current example is in a given list of subset indices is very slow and scales poorly with the subset size.

To mitigate the issue, we implemented a hash based subset sampling predicate that can be evaluated efficiently for each example, and (approximately) select a random subset of a specified size. Let  $N$  be the total number of training examples,  $n < N$  be the expected subset size. The idea is to map the index  $i$  of each example to  $N / n$  hash buckets, and select all the examples that fall into one particular bucket. To make sure each model gets an independent subset sampling, we need to use different hash functions for different models. In our implementation, we compose a known hash function for uint64 types with a simple pseudo number based on the index of the current model to achieve this. Note the subset size sampled is close to  $n$  but is not guaranteed to be exactly  $n$ . But this is not a problem in our settings. The specific implementation is shown below:

def hashSampler(mod, seed, system):

""Get hash based subset sampler.

Args:

mod: total_n_egs // subset_size

seed: different seed leads to different subset sample

$^{2}$ https://www.tensorflow.org/datasets

18

system: 'np' or 'tf'.  
Returns: A Tensorflow or Numpy subset sampler.  
```
np_hash = hash_uint64_build('np')
mul, offset, remainder = np_hash(seed + 1234 + np.arange(3))
remainder = remainder % mod  
if system == 'np':  
    def npSampler(n_total):  
        x = np.arange(n_total, dtype=np uint64)  
        return np_hash(x*mul + offset) % mod == remainder  
    return npSampler  
elif system == 'tf':  
    tf_hash = hash_uint64_build('tf')  
    def tf_filter(idx, _):  
        return tfequal(tf_hash(idx*mul + offset) % mod, remainder)  
    return tf_filter  
raise ValueError(f'Unknown system: {system}')  
def hash_uint64_build(system):  
    '''Build a hash function in tf/np for uint64'''''  
if system == 'np':  
    uint64_cast = functools部分内容(np.array, dtype=np uint64)  
    op_xor = operator.xor  
    op_rshift = operator.rshift  
elif system == 'tf':  
    uint64_cast = functools部分内容(tf.cast, dtype=tfuint64)  
    op_xor = tf.bitwise.bitwise_xor  
    op_rshift = tf.bitwise.right_shift  
else:  
    raise KeyError(f'Unknown system: {system}')  
# https://stackoverflow.com/questions/664014/  
# what-integer-cache-function-are-good-that-accepts-an-integer-cache-key  
def hash_uint64(x):  
    x = uint64_cast(x)  
    x = op_xor(x, op_rshift(x, 30)) * uint64_cast(0xbf58476d1ce4e5b9)  
    x = op_xor(x, op_rshift(x, 27)) * uint64_cast(0x94d049bb133111eb)  
    x = op_xor(x, op_rshift(x, 31))  
return x  
return hash_uint64

In Figure 11, we compare our hash-based subset sampler with numpy.random.choice(N, size=n, replace=False). The leftmost section of the figure shows that the sampling procedure always samples close to  $n$  points, with a small variance. The middle section plots a histogram of the empirical fraction of total models that each point appears in. Note that, because we use  $r = 0.25$ , this fraction should be 0.25 on average, although, because we only use 400 models, each value will not be identically 0.25. We find that our hash-based sampler produces probabilities which are highly consistent with those produced by numpy.random.choice. We also measure the pairwise independence of the hash-based sampler, measuring the probability that two different training points  $x_{1}, x_{2}$  appear both IN or OUT of a model's training set. We expect this value to be 0.625  $(= r^{2} + (1 - r)^{2})$ . We plot this in the right portion of the figure, demonstrating that the independence of our hash-based sampler is very similar to numpy.random.choice.

19

Figure 12: Comparison of directly using the per-token-accuracy vs. taking the logit of the per-token-accuracy. Top row: logit(per-token-accuracy). Bottom row: per-token-accuracy. Figures exactly the same as Figure 5.

H Alternative Memorization Metrics with Logit Scaling

We defined the counterfactual memorization in (1) with a generic performance measure  $M$ . Throughout the paper, we define  $M$  as per-token accuracy—the fraction of the times the model assigns the highest score to the true next token in the sequence. The finite value range could cause unnecessary compression for values near the interval boundary. As a result, the resolution of memorization estimation is lower for models with very high or very low performance. To mitigate this issue, we explore an alternative measure by taking the logit on the per-token accuracy [Carlini et al., 2021]. The logit function maps to  $(-\infty, \infty)$  before aggregating across independently trained models. Figure 12 compares the scatter plots of average performance on IN / OUT models measured by the logit scaled per-token accuracy and the raw per-token accuracy. Comparing to the raw per-token accuracy, the scatter plots generated with the logit scaled measure are no longer artificially constrained to be a triangular shape. As a result, the memorization estimation, which is proportional to the distance to the diagonal line, has a higher resolution on the two ends (lower left and upper right) than the unscaled version.

Note there is no absolutely right or wrong measure. While the scaled version has better resolution on the two ends, the advantage of the unscaled version is that the value range  $[0, 1]$  makes it straightforward to interpret the numerical values of counterfactual memorization. Since the consistency between the two versions are high (Spearman's  $\rho$  correlation between the two versions are  $0.947 / 0.903 / 0.944$  on RealNews/ C4/ Wiki40B:en), we use the unscaled version throughout the paper for easier interpretation.

I Definition of Edit Similarity

We define the edit similarity between two sequences  $x_{i}$  and  $x_{j}$  as. In our case, we use token-level similarity.

$$
\operatorname {E d i t S i m} \left(x _ {i}, x _ {j}\right) = 1 - \frac {\operatorname {E d i t D i s t a n c e} \left(x _ {i} , x _ {j}\right)}{\max \left(\left| x _ {i} \right| , \left| x _ {j} \right|\right)}
$$

20

Table 3: Pairs of RealNews training examples and Grover generations sampled at several influence levels. "link" contains the document URL. [...] indicate text omitted for brevity. Differences in each pair are highlighted.
Index	Estim.	Text
Generation 1361	infl 0.1805	link ➔ Baku, Azerbaijan, April 15 Trend: Official exchange rate of the US dollar and euro against Azerbaijani manat was set at 1.7 and 1.9225 manats, respectively, for April 15. Below are the rates of Azerbaijani manat against world currencies, according to the data from the Central Bank of Azerbaijan for April 15. [...] 100 Japanese yen 100 JPY 1.5187 1 New Zealand dollar 1 NZD 1.1513 Follow Trend on Telegram. Only most interesting and important news
Train 2072973	mem 0.3534	link ➔ Baku, Azerbaijan, March 15 Trend: Official exchange rate of the US dollar and euro against Azerbaijani manat was set at 1.7 and 1.9241 manats, respectively, for March 15. Below are the rates of Azerbaijani manat against world currencies, according to the data from the Central Bank of Azerbaijan for March 15. [...] 100 Japanese yen 100 JPY 1.5220 1 New Zealand dollar 1 NZD 1.1636 Follow Trend on Telegram. Only most interesting and important news
Generation 21998	infl 0.0218	link ➔ NEW DELHI: India is likely to see average monsoon rains this year, the state-run weather office said on Monday, which should support agricultural production and economic growth in Asia's third-biggest economy, where half of the farmland lacks irrigation. Monsoon rainfall is expected to be 96 percent of the long-term average, M. Rajeevan, secretary at the Ministry of Earth Sciences, told a news conference. The India Meteorological Department (IMD) defines average, or normal, rainfall as between 96 percent and 104 percent of a 50-year average of 89 centimeters for the entire four-month season beginning June. [...] India's weather office will update its forecast in the first week of June. However, on average, the IMD has forecast accurately only once every five years over the past two decades, even after taking into account an error band of plus or minus 5 percentage points.
Train 326212	mem 0.1555	link ➔ NEW DELHI (Reuters) - India is likely to receive average monsoon rains in 2018, the weather office said, raising the possibility of higher farm and economic growth in Asia's third-biggest economy, where half of the farmland lacks irrigation. Monsoon rains, the lifeblood of the country's $2 trillion economy, are expected to be 97 percent of a long-term average, K.J. Ramesh, director general of the state-run India Meteorological Department (IMD), told a news conference. "We see very less probability of a deficit monsoon," Ramesh said on Monday. Other than lifting farm and wider economic growth, a spell of good rains will keep a lid on inflation, potentially tempting Prime Minister Narendra Modi to bring forward general elections due in May 2019. India's weather office defines average, or normal, rainfall as between 96 percent and 104 percent of a 50-year average of 89 cms for the entire four-month season beginning June. [...] said a Mumbai-based dealer with a global trading firm. Average monsoon rainfall will help India retain its position as the world's top rice exporter.

J Examples of Train-Generation Pairs at Different Influence Ranges

In table 3, we show examples of train-generation pairs sampled from different influence ranges. The patterns generally follow the train-validation pairs shown in table 2, although many of the relations are due to some form of templating.

K Examples Sampled at Different Level of Memorization

Figure 13, Figure 14, and Figure 15 show full examples from RealNews sampled at high, middle and low memorization value ranges, respectively. Similarly, Figure 16, Figure 17, and Figure 18 show examples from C4 sampled at high, middle and low memorization value ranges, respectively. Figure 19, Figure 20, and Figure 21 show examples from Wiki40B:en sampled at high, middle and low memorization value ranges, respectively.

L Example Pairs Sampled at Different Level of Influence

Figure 22, Figure 23, Figure 24, Figure 25, and Figure 26 show train-validation example pairs from RealNews sampled from high to low influence ranges. For each pair, we show the validation set example first, and then show the corresponding training example with a difflib generated visualization of textual difference with the training example.

Similarly, Figure 27 and Figure 28 show train-validation example pairs from C4, and Figure 29 and Figure 30 from Wiki40B:en.

We also show train-generation influence pairs between RealNews training set and Grover [Zellers et al., 2019] model generation in Figure 31, Figure 32, and Figure 33.

21

Index	mem	Link and Text
2090855	0.6546	http://www.jta.org/1937/12/29/archive/amERICAN-jewish-congress-plans-drive-on-job-discrimination
THE AMERICAN JEWISH CONGRESS ANNOUNDED TODAY THE PUBLICATION OF A REPORT ON JEWISH NON-EMPLOYMENT AS A RESULT OF ECONOMIC DISCRIMINATION, AND ALSO A NATION-WIDE PROGRAM TO DEAL WITH THE PROBLEM. THE PROGRAM OUTLINED IN THE PAMPHLET INCLUDES ESTABLISHMENT OF BUREAUS IN COMMUNITIES WHERE THE CONGRESS HAS BRANCHES TO DEAL WITH DISCRIMINATION, DEVELOPMENT OF A PROGRAM OF EDUCATING EMPLOYERS, LEGISLATIVE CORRECTION OF AGENCY ABUSES, ELIMINATION OF QUESTIONS ON RELIGION FROM EMPLOYMENT APPLICATION BLANKS AND FORMATION OF AN ADVISORY COUNCIL OF CHRISTIAN LIBERALS. THE REPORT, THE SECOND OF ITS KIND, WAS PREPARED BY RABBI J.X. COHEN, CHAIRMAN OF THE CONGRESS'S COMMISSION ON ECONOMIC PROBLEMS, AND IS ENTITLED "HELPING TO END ECONOMIC DISCRIMINATION." IT SURVEYS CASES OF DISCRIMINATION AGAINST JEWISH JOB-SEEKERS AND WANTS THIS IS "SOCIALLY DANGEROUS." THE PAMPHLET REPORTS THAT THE PRESIDENT OF AMERICAN TELEPHONE AND TELEGRAPH COMPANY, WALTER S. GIFFORD, AT THE LAST NATIONAL CONFERENCE OF PRESIDENT OF LOCAL COMPANIES, ADVISED HIS ASSOCIATES AGAINST PERMITTING PERSONNEL MANAGERS TO REJECT JEWISH APPLICANTS BECAUSE OF PREJUDICE, AND SHORTLY THEREAFTER ONE OF THE DEPARTMENTS OF A.T.& T." "ALMOST UNPRECIDENTEDLY" ENGAGED A JEWISH APPLICANT.
799	0.6324	https://www.express.co.uk/sport/othersport/356764/Greyhound-racing-results-08-11-12
Greyhounds jostle for position at a meet in Newcastle [ ] ROMFORD FANCIES 10.31 Jogadusc Jasper (1-4-2) 10.46 Selkirk Grace (4-3-1) 11.00 Forest Fella (2-4-1) 11.16 Strawberry Hall (2-5-6) 11.31 Lilies Rocket (3-1-5) 11.44 Droopys Sian (3-4-1) 11.58 Dudley's Lady (Nap) (4-6-5) 12.17 Giffys Girl (3-5-6) 12.33 Clerihan Ruso (3-1-2) 12.47 Dukes Wish (3-2-5) 1.04 Cairns Rebel (5-4-1) 1.19 Borna Karma (5-3-6) 1.33 Tolo (5-2-4) 1.49 Altria Panther (1-2-6) CRAYFORD FANCIES 10.38 Zenas Heather (Nap) (1-5-3) 10.53 Ballybane Lena (1-5-3) 11.07 Any Wonder (1-5-3) 11.23 Whitefield Maeve (1-3-4) 11.37 Denwill Andy (1-2-4) 12.17 McGlynns (1-3-4) 12.08 Newlawn Sweep (1-6-3) 12.24 Boy Wonder Oscar (1-2-3) 12.39 Lordsbury Rollie (1-4-2) 12.57 Thirsty Rooney (1-2-3) 1.11 Little Thriller (1-2-3) 1.27 Dharmacon (1-2-3) NEWCASTLE FANCIES 2.24 Wandsbeck Magpie (2-1-6) 2.42 Clegg Rock (6-1-5) 2.58 Steel Ranger (2-3-6) 3.18 Hillview Holly (5-6-2) 3.37 Westoe Cass (6-2-1) 3.53 Lil Carpi (Nap) (1-3-2) 4.14 Sleek Joe (2-5-1) 4.28 Blackhouse Lace (2-3-4) 4.47 A Bit Of Glory (5-6-3) 5.07 Royal Cracker (6-4-2) 5.27 Seachthom Gorm (5-6-3) 5.44 Bohard Land (1-4-2) 5.58 Dark Milky (4-5-3) 6.16 Derex Hope (6-1-4) RESULTS HOVE: 11.11 Vivendi Katie 5-2f (1-4-3 BAGS F E20.06 TC E51.94). 11.28 Tallulah Belle 7-4f (6-3-1) E13.86 TC E61.16). 11.42 Alfatwelldigger 4-1 (4-5-1) E28.48 TC E109.66). 11.57 Clune Hondo 5-1 (4-5-6) E20.55 TC E59.03). 12.12 Que Katie 4-1 (4-5-3 E27.26 TC E88.94). 12.27 Guinness Joker 5-1 (6-1-4) E23.67 TC E67.49). 12.42 Cabra Bolt 5-2jt (6-4-1) E28.39 TC E94.26). 12.58 Calano 7-4f (6-4-3) E13.75 TC E49.05). 1.12 Westway Triangle 9-4f (5-6-2) E10.44 TC E34.96). 1.27 Jazz Diamond 3-1 (4-3-1) E16.58 TC E64.44). 1.44 Ladies Watch 5-1 (4-5-1) E43.06 TC E12.38). 1.58 Albion Star 2-1jt (6-1-2) E16.97 TC E72.66). Placer E1 stake (Races 1-6) E54.00, (Races 7-12) E34.00. MONMORE: 2.18 Ardaya Jack 7-1 (2-4-BAGS F E40.24 TC E116.18). 2.37 Hellroad Rodge 7-2 (2-1-6) E28.22 TC E67.32). 2.57 Whitehart Wonder 3-1 (1-4-5) E14.57 TC E42.03). 3.77 Bawa Narantctic 7-2 (4-2-1) E17.06 TC E53.86). 3.38 Shinto 9-4jt (4-5-6) E20.45 TC E70.50). 3.58 Marcelino 5-1 (6-4-2) E30.55 TC E95.97). 4.18 Coonen Fly 7-2 (3-5-2) E14.14 TC E35.51). 4.37 Rock It Ramon 3-1 (6-4-3) E12.96 TC E47.67). 4.57 Combo Eva 5-1 (5-6-3) E24.42 TC E74.53). 5.17 Lisgarbh Anna 4-1 (5-4-2) E17.90 TC E43.80). 5.33 Paids Peata 4-1 (4-6-1) E34.64 TC E12.97). 5.47 Ardera General 4-1 (6-5-2) E15.91 TC E54.97). 6.04 Seathwaite Dave 6-1 (5-6-2) E38.06 TC E89.96). 6.18 Scooby Blue 7-2 (1-5-6) E19.05 TC E52.11). Placer E1 stake (Races 1-6) E25.03. OXFORD: 11.03 Chapelane Lucy 7-2 (1-6-3) BAGS F E25.96 TC E71.49). 11.19 Beckys Druidin 5-1 (2-5-1) E44.44 TC E128.91). 11.34 Headford Royal 7-4jt (2-5-1) E19.55 TC E65.60). 11.48 Geneva Cove 3-1 (2-5-4) E29.24 TC E89.42). 12.04 Ring The Chief 3-1 (4-1-3) E11.87 TC E39.74). 12.18 Rock Magic 6-4f (1-5-6) E11.38 TC E45.53). 12.32 Lissycasey Jose 4-1 (6-4-2) E17.55 TC E71.66). 12.47 Brykat Lola 7-2 (5-3-4) E15.29 TC E53.59). 1.04 Ballymac Tish 5-2jt (1-6-4) E15.71 TC E44.31). 11.19 Gizmo Senor 5-1 (1-5-3) E60.34 TC E138.24). 1.33 Call It Paris 8-1 (5-3-2) E41.46 TC E98.54). 1.51 Millbank Printer 4-1 (3-4-5) E25.01 TC E53.87). Placer E1 stake (Races 1-6) E54.00, (Races 7-12) E526.00. SWINDON: 2.08 Unleash Fidel 5-2f (3-5-2) BAGS F E114.7 TC E30.89). 2.27 Rushy Dusky 10-1 (1-3-4) E42.10 TC
2085736	0.5755	https://fox59.com/2016/01/28/recipe-chinese-pork-vegetable-soup-with-wonton-noodles/
× RECIP: Chinese Pork & Vegetable Soup with Wonton Noodles Chinese Pork & Vegetable Soup with Wonton Noodles 1 pork tenderloin (about 1-1/4 pound size), cooked and cut into ½-inch cubes* 5 cups lower-sodium chicken broth 1 cup water** ¼ cup rice wine vinegar 1 tablespoon lower sodium soy sauce 1 heaping teaspoon very finely minced garlic 1 heaping teaspoon grated fresh ginger ¼ teaspoon black pepper 1 can (15 ounces) whole baby corn, drained and cut into quarters 2-4 ounces fresh snow peas, halved crosswise 2/3 cup carrots, thinly biased-sliced 2/3 cup sliced mushrooms 1/3 cup chopped green onions 8-12 wonton wrappers (3 to 3.5 inch size), cut into ½-inch strips and separated (so they don't stick together) Combine chicken broth, water, rice wine vinegar, soy sauce, garlic, ginger and black pepper in large saucepan or Dutch oven. Bring to a boil. Stir in baby corn pieces, snow peas, carrots, mushrooms, onion and wonton strips. Return to a boil, then reduce heat, cover and simmer about 3 minutes until vegetables are crisp-tender. Stir in cooked pork cubes and heat thoroughly. *Roast pork tenderloin uncovered in 425°F oven until instant read thermometer registers 145°F (about 15 - 20 minutes roughly). Let pork tenderloin rest at least 5 minutes before cutting. Makes 4 servings (about 1½ cups each) Recipe by PorkBelspired.com with adaptations by culinary dietitian & nutritionist Kim Galeaz, RDN CD
1680600	0.5807	https://www.sbs.com.au/yourlanguage/arabic/en/article/2017/12/3/what-acknowledgement-country
Language English / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / B / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A /A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / C / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D /D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D / D /
A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A
A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A /
A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A / A
A / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B /
A / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B / B /
A/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/
A/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/ B/
A/B/C/D/E/F/G/H/L/S/Y/A/P/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q,S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q.S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q/S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T-Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R.S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q,S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T(Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/RS/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-QS/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/TQ/Q/Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/ Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R/S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-QS/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/Q-Q/S/R S/T/q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q(Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q.Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q-Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q/Q(Q)/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/A/

Figure 13: Text examples from RealNews with high memorization.

22

Index	mem	Link and Text
2074805	0.2835	http://www.christianpost.com/news/texas-students-suspension-for-anti-gayremark-reversed-94734/
A Texas honors student punished for saying that homosexuality was wrong has had his suspension rescinded after a meeting with the mother and her attorney. The 14-year-old, Dakota Ary, from Western Hills High School of the Fort Worth Independent School District was initially given a suspension of one day in-school and two days full. After hearing about the suspension the mother, Holly Pope, reached out to Matt Krause of the Liberty Counsel to be her son's legal representative. Matt Staver, founder and chairman of the Liberty Counsel, told The Christian Post that he believed Western Hills High made the correct decision in reversing their course of action. "The decision to rescind the suspension is the correct one. The suspension was wrong and improper," said Staver. "I applaud the student for standing up. We stood with him to resist an unjust suspension and we are pleased that suspension has been reversed." Dakota Ary was in a German language class at Western Hills when class conversation shifted to the issues of religion and homosexuality. The student reportedly said that "being a homosexual was wrong." Upon hearing his remark, the teacher proceeded to take administrative action. Arh himself is an honors student, noted for being a member of the high school's football team and a volunteer at his church. The rescinding of the suspension allows for him to play in an upcoming football game. In a publicly released statement, the Fort Worth Independent School District said that it does not comment on student matters. In keeping with this, Fort Worth ISD did not return comment to The Christian Post by press time. "As a matter of course, Fort Worth ISD does not comment on specific employee or student-related issues. Suffice it to say that we are following district policy in our review of the circumstances and any resolution will likewise be in accordance with district policy," said Fort Worth ISD. Staver also told CP that in recent years incidents like this one are only increasing as the gay rights movement continues to advance in American society. "Unfortunately, we have seen similar situations. The homosexual agenda is aggressive and intolerant to people who do not agree. Liberty Counsel will continue the right to exercise freedom of conscience and religion," said Staver. "These instances are increasing and will continue to increase unless Christians and people who love liberty stand up and resist this intolerance."
710814	0.1536	http://in.reuters.com/article/idINlndia-55725620110320
MANAMA Bahraini security forces detained the outspoken head of the main human rights group early on Sunday, a colleague said, days after a crackdown that drove mainly Shi'ite protesters off the street. "I want to update you that they arrested Nabeel Rajab at 1.30 am and the riot police came to my home and I'm not in my home right now," Said Yousif of the Bahrain Center for Human Rights said in an email sent overnight. Rajab had spoken to media about the crackdown and the Bahrain Center for Human Rights put out statements this week accusing Bahraini forces and their Saudi and Emirati allies of "massacrres" including some using Apache helicopters. Bahraini state television has said that Bahrain did not own Apache helicopters. Four protesters and three police have been confirmed killed in Wednesday's crackdown. Neither Rajab nor Yousif could be reached by phone early on Sunday, but some Twitter feeds suggested Rajab had already been released after questioning by police. Rajab's arrest would bring to 10 the number of people believed to have been arrested since the crackdown began on Wednesday. Among the others are opposition leaders who had called for the overthrow of the monarchy and doctors who had complained of excessive use of force against protesters. (Reporting by Frederik Richter, Writing by Lin Noueihed; Editing by Matthew Jones)
1263261	0.1533	http://www.com/accessories-news/handbags/stella-dot-expands-handbag-selection-6840891/
Stella & Dot, the social commerce-based accessories company, will add to its offerings on April 8 with a full range of handbags. The nine-piece collection will retail from $22 for a cosmetics case to $138 for a weekend bag. Founder and chief executive officer Jessica Herrin called this the nearly 10-year-old company's "launch into a lifestyle brand," as well as a chance to double its market opportunity. "In the $30 billion accessories market in the U.S., bags and jewelry each account for approximately $10 billion," Herrin told WWD. "We see the same opportunity in bags as we did in jewelry - giving women on the go a simple way to be chic." Other styles include a $59 technology case that serves double duty as a clutch and wallet, a $39 jewelry roll, a classic $89 tote and a $128 convertible bag that has zippers up the side to let the wearer decide if she wants a sleeker or more expanded silhouette. A small handbag collection was released in 2011 - containing two cross-body and two convertible cross-body to clutch styles - but this is the first significant push for handbags geared for daytime use. With sales expected to surpass $200 million this year - business grew from $175 million to $200 million from 2011 to 2012 - Herrin projected the new category will take in about $25 million through the end of the year. In addition to the brand being carried exclusively online at stelladot.com, the brand is sold via 14,000 active stylists around the world. "We're really investing in this as a huge growth category. We just opened a dedicated in-house design studio for this in Sausalito [Calif.]" Stella & Dot chief creative officer Blythe Harris said. She added that the bags are comprised of a combination of coated canvas, waffle poly and Saffano leather and come in ikat, snakeskin and colorful multistripe prints. Herrin added: "Broadening our accessories line by launching the new and massive category of bags allows us to make that opportunity for stylists bigger than ever before. It's a ground-floor opportunity with a proven company."
854230	0.1271	https://www.nbcWashington.com/news/local/Mon-Outside-Home-Fatally-Shot-By-Resident-Police-Say-466564043.html
Police said a man was shot to death outside a Chillum, Maryland, home by the homeowner. News4's Darcy Spencer reports. A homeowner in Maryland shot and killed a man who he caught breaking into his vehicle hours after Christmas Day, police say. Sources tell News4 the shooting, which occurred in an area repeatedly hit by car break-ins, appears to be justified. Deontae Parker, of no fixed address, died after he was shot the day early Tuesday outside a home in Chillum, Maryland, Prince George's County police said. He was 32. Police said Tuesday that they were investigating what happened. "We're still working through all the details to determine what led up to the shooting," Cpl. Tyler Hunt said. A resident of a home on the 6200 block of Sligo Parkway called 911 about 2:15 a.m. Tuesday to report a shooting. Officers arrived and found a man with a gunshot wound to the upper body. He was pronounced dead. According to the preliminary investigation, detectives believe the homeowner - who police have not identified - shot Parker from inside his home after Parker "had broken into the homeowner's vehicle," police said in an update Wednesday morning. News4 footage shows a bullet hole in a window. Police now have video from the home's surveillance system, "which shows Parker's actions prior to the shooting." A second suspect is believed to have run from the scene before police arrived. Parker and the homeowner did not know each other, police believe. Sources told News4 the homeowner has lived there for 17 years, and that about 10 people were inside the house at the time of the shooting, including children. A neighbor who asked News4 not to identify him said his car had been broken into multiple times. The Prince George's County State's Attorney's Office will determine whether any charges should be filed against the homeowner. Anyone with information on the case is asked to call 301-772-4925. Anyone who wants to remain anonymous can call 866-411-TIPS, go to www.pgcrimesolvers.com or use the P3 Tips app.

Figure 14: Text examples from RealNews with intermediate memorization.

23

Index	mem	Link and Text
449808	0.0361	https://www.thestreet.com/story/12179505/1/first-week-of-dlr-february-2014-options-trading.html
Investors in Digital Realty Trust, Inc. (DLR) saw new options begin trading this week, for the February 2014 expiration. At Stock Options Channel, our YieldBoost formula has looked up and down the DLR options chain for the new February 2014 contracts and identified one put and one call contract of particular interest. The put contract at the $45.00 strike price has a current bid of $1.00. If an investor was to sell-to-open that put contract, they are committing to purchase the stock at $45.00, but will also collect the premium, putting the cost basis of the shares at $44.00 (before broker commissions). To an investor already interested in purchasing shares of DLR, that could represent an attractive alternative to paying $49.02/share today. Because the $45.00 strike represents an approximate 8% discount to the current trading price of the stock (in other words it is out-of-the-money by that percentage), there is also the possibility that the put contract would expire worthless. The current analytical data (including greeks and implied greeks) suggest the current odds of that happening are 74%. Stock Options Channel will track those odds over time to see how they change, publishing a chart of those numbers on our website under the contract detail page for this contract. Should the contract expire worthless, the premium would represent a 2.22% return on the cash commitment, or 13.98% annualized - at Stock Options Channel we call this the START SLIDESHOW: Top YieldBoost Puts of the REITs » Below is a chart showing the trailing twelve month trading history for Digital Realty Trust, Inc., and highlighting in green where the $45.00 strike is located relative to that history:
1157311	0.0356	https://www.thestreet.com/story/12425372/1/anf-april-4th-options-begin-trading.html
Investors in Abercrombie & Fitch Co. (ANF) saw new options become available today, for the April 4th expiration. At Stock Options Channel, our YieldBoost formula has looked up and down the ANF options chain for the new April 4th contracts and identified one put and one call contract of particular interest. The put contract at the $34.00 strike price has a current bid of $1.97. If an investor was to sell-to-open that put contract, they are committing to purchase the stock at $34.00, but will also collect the premium, putting the cost basis of the shares at $32.03 (before broker commissions). To an investor already interested in purchasing shares of ANF, that could represent an attractive alternative to paying $34.60/share today. Because the $34.00 strike represents an approximate 2% discount to the current trading price of the stock (in other words it is out-of-the-money by that percentage), there is also the possibility that the put contract would expire worthless. The current analytical data (including greeks and implied greeks) suggest the current odds of that happening are 57%. Stock Options Channel will track those odds over time to see how they change, publishing a chart of those numbers on our website under the contract detail page for this contract. Should the contract expire worthless, the premium would represent a 5.79% return on the cash commitment, or 49.23% annualized - at Stock Options Channel we call this the START SLIDESHOW: Top YieldBoost Puts of the S&P 500 » Below is a chart showing the trailing twelve month trading history for Abercrombie & Fitch Co., and highlighting in green where the $34.00 strike is located relative to that history:
1551143	0.0708	https://www.fool.com/investing/general/2012/08/07/national-instruments-beats-on-revenue-matches-exp.aspx
National Instruments (Nasdaq: NATI) reported earnings on Aug. 3. Here are the numbers you need to know. The 10-second takeaway For the quarter ended June 30 (Q2), National Instruments beat expectations on revenues and met expectations on earnings per share. Compared to the prior-year quarter, revenue improved significantly and GAAP earnings per share stayed the same. Margins dropped across the board. Revenue details National Instruments reported revenue of $292.3 million. The five analysts polled by S&P Capital IQ expected a top line of $279.7 million on the same basis. GAAP reported sales were 15% higher than the prior-year quarter's $253.3 million. Source: S&P Capital IQ. Quarterly periods. Dollar amounts in millions. Non-GAAP figures may vary to maintain comparability with estimates. EPS details EPS came in at $0.21. The four earnings estimates compiled by S&P Capital IQ anticipated $0.21 per share. GAAP EPS of $0.22 were the same as the prior-year quarter. Source: S&P Capital IQ. Quarterly periods. Non-GAAP figures may vary to maintain comparability with estimates. Margin details For the quarter, gross margin was 75.8%, 210 basis points worse than the prior-year quarter. Operating margin was 12.0%, 160 basis points worse than the prior-year quarter. Net margin was 9.0%, 150 basis points worse than the prior-year quarter. Looking ahead Next quarter's average estimate for revenue is $288.1 million. On the bottom line, the average EPS estimate is $0.24. Next year's average estimate for revenue is $114 billion. The average EPS estimate is $0.96. Investor sentiment The stock has a five-star rating (out of five) at Motley Fool CAPS, with 387 members out of 408 rating the stock outperform, and 21 members rating it underperform. Among 154 CAPS All-Star picks (recommendations by the highest-ranked CAPS members), 149 give National Instruments a green thumbs-up, and five give it a red thumbs-down. Of Wall Street recommendations tracked by S&P Capital IQ, the average opinion on National Instruments is outperform, with an average price target of $32.50. Can your portfolio provide you with enough income to last through retirement? You'll need more than National Instruments. Learn how to maximize your investment income and "Secure Your Future With 9 Rock-Solid Dividend Stocks." Click here for instant access to this free report.
1257699	0.0643	https://www.fool.com/investing/general/2012/02/27/merge-healthcare-goes-negative.aspx
Merge Healthcare (Nasdaq: MRGE) reported earnings on Feb. 15. Here are the numbers you need to know. The 10-second takeaway For the quarter ended Dec. 31 (Q4), Merge Healthcare met expectations on revenues and earnings per share. Compared to the prior-year quarter, revenue improved significantly and GAAP earnings per share dropped to a loss. Gross margins expanded, operating margins grew, and net margins shrank. Revenue details Merge Healthcare booked revenue of $64.1 million. The five analysts polled by S&P Capital IQ foresaw net sales of $64.0 million on the same basis. GAAP reported sales were 39% higher than the prior-year quarter's $46.2 million. Source: S&P Capital IQ. Quarterly periods. Dollar amounts in millions. Non-GAAP figures may vary to maintain comparability with estimates. EPS details Non-GAAP EPS came in at $0.04. The five earnings estimates compiled by S&P Capital IQ forecast $0.04 per share on the same basis. GAAP EPS were -$0.01 for Q4 against $0.10 per share for the prior-year quarter. Source: S&P Capital IQ. Quarterly periods. Non-GAAP figures may vary to maintain comparability with estimates. Margin details For the quarter, gross margin was 66.3%, 730 basis points better than the prior-year quarter. Operating margin was 13.7%, 500 basis points better than the prior-year quarter. Net margin was -2.0%, 2,380 basis points worse than the prior-year quarter. Looking ahead Next quarter's average estimate for revenue is $66.3 million. On the bottom line, the average EPS estimate is $0.05. Next year's average estimate for revenue is $289.2 million. The average EPS estimate is $0.29. Investor sentiment The stock has a four-star rating (out of five) at Motley Fool CAPS, with 101 members out of 121 rating the stock outperform, and 20 members rating it underperform. Among 34 CAPS All-Star picks (recommendations by the highest-ranked CAPS members), 31 give Merge Healthcare a green thumbs-up, and three give it a red thumbs-down. Of Wall Street recommendations tracked by S&P Capital IQ, the average opinion on Merge Healthcare is outperform, with an average price target of $8.13. Over the decades, small-cap stocks like Merge Healthcare have produced market-beating returns, provided they're value-priced and have solid businesses. Read about a pair of companies with a lock on their markets in "Too Small to Fail: 2 Small Caps the Government Won't Let Go Broke." Click here for instant access to this free report.

Figure 15: Text examples from RealNews with low memorization.

24

Index	mem	Link and Text
324899	0.7560	http://nfdb.net/view_329432_Al-War-Fleet-Command-v3.0.0.0-RIP-serial-regged-nocd-cracked-keygen.html
UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUuuUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUCUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUAUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu uuucuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuUUuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuccuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuccuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuucuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu uuuccuucccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccuccCCCucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccuccccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccucccccuccccc
UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU/UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUNUEHDD!! AI War Fleet Command v3.0.0.0 RIP AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AA 0.6390 https://webstatsdomain.org/d/vespa.bg
Vespa: CBTOBEN INDEP B INPON3BODCTBO TO Na KUTERI 3A INDNIBIDYALNO PnINDBVKBANE.Дизайсян刚好 Роставlementа STAFGE I PEBELAJARAN AKTIF TIPE INVESTIGAS TIM UNTUK MENINGKATKAN HASIL BELAJAR SISWA PADA MATA PELAJARAN PENDIDIKAN AGAMA ISLAM DI KELAS IV SEKOLAH DASAR NEGERI 006 PASIR SIALANG KECAMATAN BANGKINANG SEBERANG KABUPATEN KAMPAR. Skripsi thesis, Universitas Islam Negeri Sultan Syarif Kasim Riau. This list was generated on Thu Apr 25 19:48:32 2019 WIB.
2080878	0.5603 http://repository.uin-suska.ac.id/view/creators/ASKINA=3A=3A=3A.html
Items where Author is "ASKINA," "ASKINA", (2013) PENERAPAN STRATEGI PEMBELAJARAN AKTIF TIPE INVESTIGAS TIM UNTUK MENINGKATKAN HASIL BELAJAR SISWA PADA MATA PELAJARAN PENDIDIKAN AGAMA ISLAM DI KELAS IV SEKOLAH DASAR NEGERI 006 PASIR SIALANG KECAMATAN BANGKINANG SEBERANG KABUPATEN KAMPAR. Skripsi thesis, Universitas Islam Negeri,sunlar,Syarif Kasim Riau. This list was generated on Thu Apr 25 19:48:32 2019 WIB.
2088697	0.4794 https://www.baindeterre.com/product/color-enhancing-blood-orange-and-passion-fruit-shampoo
Color Enhancing Blood Orange & Passion Fruit Shampoo enhances red tones on red, light brown, and dark blonde hair. Apply shampoo to wet hair. Lather, leave on 3-5 minutes & rinse. Water/Aqua/Eau, Sodium Laureth Sulfate, Cocamide MIPA, Cocamidopropyl Betaine, Glycol Stearate, Citrus Aurantium Dulcis (Orange) Fruit Extract, Passiflora Edulis Fruit Extract, Ribes Nigrum (Black Currant) Fruit Extract, Lycium Barbarum Fruit Extract, Cocos Nucifera (Coconut) Oil, Argania Spinosa Kernel Oil, Gardenia Tahensis Flower Extract, Laureth-12, Laureth-23, Guar Hydroxypropyltrimmon Chloride, PEG/PPG-4/12 Dimethicone, Polyquaternium-11, Citric Acid, Tetrasodium EDTA, PEG-150 Pentaaerythrityl Tetrastearate, PEG-6 Caprylic/Capric Glycerides, Ethylhexyl Salicylate, Ethylhexyl Methoxycinnamate, Butyl Methoxydibenzoylmethane, PEG-40 Hydrogenated Castor Oil, Glycerin, PPG-26-Buteth-26, Polysorbate 20, Imidazolidinyl Urea, Limonene, Fragrance/Parfum, Orange 4 (CI 15510), Red 33 (CI 17200).

Figure 16: Text examples from C4 with high memorization.

25

Index	mem	Link and Text
2001601	0.2974	https://www.environment.nsw.gov.au/news/quollity-selfie-spotting-from-citizen-scientists
Citizen scientists from across Australia and the world have helped researchers count Illawarra's elusive spotted-tailed quoll, Environment Minister Gabrielle Upton said 3 March. "Six months ago we asked citizen scientists to jump online and help us analyse more than 80,000 photos taken from across the Illawarra region," Ms Upton said. "More than 300 volunteers answered the call and we can now confirm at least 20 individual quolls call this region home." "This is a great milestone for this citizen science project - in time to celebrate World Wildlife Day!" The Quolidor project is funded under the NSW Government's $100 million Saving our Species project and is providing a fascinating glimpse into the hidden world of this endangered carnivorous marsupial. The 29 motion sensor cameras in the region collect 20,000 to 30,000 images every 8 weeks that the enthusiastic citizen scientists have sorted through. "We have images of quolls posing for selfles, exploring the camera and jumping on and around the monitoring station," Ms Upton said. Quolls have huge home ranges and move across the landscape making it difficult to otherwise monitor their population and behavioural patterns. "It's great to see people being able to use technology to make a real contribution to the conservation of NSW's unique animals," Ms Upton said. The spotted-tailed quoll is the only remaining quoll species in the state. The project is ongoing and will help the NSW Government increase the resilience and size of the local quoll population.
2045446	0.2457	http://www.beerlycoherent.com/2010/04/floreffe-blond.html
Yesterday's trip to our local, Le Chatelain, was a welcome reprieve, and I took the opportunity to sample a new beer, Floreffe Blond. Hard to believe I haven't had everything on offer there yet, but that's Belgian pubs for you: even a run-of-the-mill one will have 30 or so types. Le Chatelain probably has closer to 60. Floreffe's dark beers, Floreffe Dubbel and Floreffe Prima Melior, are included in my list of top ten Belgian beers, so I was expecting great things with Floreffe Blond. I wasn't disappointed. Floreffe Blond is lighter than normal strong blond ale. It's lemony, summery. Fiona says it's smooth and has a bubble gum taste -- "that powdery, old-style bubble gum" -- but I'm sticking with citrus. It has very little bitterness, but it's not sweet. At 6.3% alcohol, it's not too brain-bending either. Which is just as well, because the wood treatment is doing my head in plenty enough.
1795710	0.2217	https://www.snohomishcountywa.gov/CivicAlerts.aspx?AID=1619
EVERETT, Wash., April 15 - Snohomish County's Department of Planning and Development Services (PDS) today announced an on-line survey to gather public input for the Southwest Urban Growth Area Boundary Planning Study. Obtaining feedback from those who live, work, and play in Snohomish County about their quality of life and improvements they would like to see is at the heart of everything the county does. The Southwest Urban Growth Area (SWUGA) Boundary Planning Study (BPS) is no different. It is a high-level study of existing conditions and opportunities and constraints which can inform future planning choices. As part of the BPS, an online survey is available until April 26 for community members in the SWUGA BPS area, and outside the area, to give their feedback. The study area includes lands currently designated and zoned rural uses extending eastward from the SWUGA to Broadway Ave, north to Cathcart and south to the county line. It includes the Maltby Urban Growth Area (UGA) for context, as a key center. No changes to land use or boundaries are proposed for Maltby UGA or any other portion of the Boundary Planning Area as part of this study. The area has been chosen for study given it is adjacent to a fast-growing portion of the SWUGA. The study provides an opportunity to develop comprehensive information on environmental, socio-economic, infrastructure cost, and policy implications to inform future planning efforts. Using technical information and public outreach results, the study can help identify what currently works well and what doesn't in the SWUGA BPS area (e.g. roads within and beyond the study area). The study will also identify opportunities and constraints under different future growth scenarios in the currently designated-rural lands in the study area. This includes a scenario with no change to existing land use and zoning designations in the project study area. The scenarios allow the county to consider implications for environmental, social, capital/infrastructure, and fiscal conditions. The study itself is not proposing actions to change land uses, zoning or UGA boundaries. Information generated in the study will be available when future planning initiatives are developed in the county such as the 2023 Snohomish County GMA Comprehensive Plan Update. For more information on the SWUGA BPS, please visit the study webpage.
1130970	0.1636	https://www.electionportal.org/about
The International Republican Institute (IRI) has deployed a Long-Term Observation Mission to observe preparations for the Moldovan parliamentary elections scheduled for February 24, 2019. Long-term observers (LTOs), who were accredited by the Moldovan Central Election Commission on December 11, 2018, are based in Edinet, Ungheni, Orhei, Anenii Noel, Hancesti, Comrat and Chisinau. IRI observers will remain on the ground to observe the pre-election preparations, Election Day and the post-election period, concluding their mission on March 26, 2019. The mission is led by Andrea Keerbs, with the assistance of three analysts who will provide in-depth analysis of the media, electoral and legal landscape. The experienced LTOs represent 11 different countries: the U.S., Poland, Canada, the U.K., France, the Philippines, Uganda, Portugal, Montenegro, Slovenia and Georgia. IRI has organized more than 200 international election observation missions around the globe, earning a reputation for impartiality and professionalism. The IRI mission to Moldova is funded by the United States Agency for International Development and will conduct its activities on a strictly independent and nonpartisan basis, without interfering in the election process and conforming to the laws of Moldova. Click here for more information on IRI's work in Moldova.

Figure 17: Text examples from C4 with intermediate memorization.

26

Index	mem	Link and Text
653722	0.0119	http://odessahighschool.org/odessa-mo/alumni/7950336/lewis-thompson.html
This is a placeholder page for Lewis Thompson, which means this person is not currently on this site. We do suggest using the tools below to find Lewis Thompson. You are visiting the placeholder page for Lewis Thompson. This page is here because someone used our placeholder utility to look for Lewis Thompson. We created this page automatically in hopes Lewis Thompson would find it. If you are not Lewis Thompson, but are an alumni of Odessa High School Odessa, MO, register on this site for free now.
1498720	0.0087	http://anaheimhighschool.org/alumni/6122068/david-baldwin.html
This is a placeholder page for David Baldwin, which means this person is not currently on this site. We do suggest using the tools below to find David Baldwin. You are visiting the placeholder page for David Baldwin. This page is here because someone used our placeholder utility to look for David Baldwin. We created this page automatically in hopes David Baldwin would find it. If you are not David Baldwin, but are an alumni of Anaheim High School, register on this site for free now.
1481511	-0.0018	http://www/showerreplacementguys.com/pa/shower-replacement-in-south-gibson/
At Shower Replacement Guys, our experts understand the process of Shower Replacement. This is an important training that ensures that your tub or shower so that it discards the old, worn out and outdated look for a breathtaking and refreshing replacement shower doors. The process starts with removing panels, loosening the jambs, removing the bottom track, installing the side tracks and leveling the track. They will then expertly fasten the jambs, expand the holes for the panels, installing back the bottom track and the storage column. As the process nears the end stage, our experts will then fasten the jambs to the column before restoring the top track. Once the shelving units are expertly restored, the caulk frames will be put in place and finally, you will have the door back in place. Talk to our experts in Shower Replacement Guys in South Gibson, PA so that you can learn more on how all these stages can be carried out efficiently and conveniently for your Shower Replacement needs. One of the key requirements for quality services in our South Gibson, PA is that the services must be sanctioned by the authorities. We uphold provision of quality services that meet the needs and requirements of people so that they can enjoy services provided without fear of substandard services. We have complied with all the provisions of the law so that you get superior quality services. As a result, you can be sure that compare favorably with other providers of Shower Replacement services in South Gibson, PA. You can therefore approach us for our services with full confidence that you will get the quality services. We guarantee you that you will get the most trusted and reliable shower door replacement services from Shower Replacement Guys in South Gibson, PA. This replacement shower doors will transform your experience with your shower or tub. It will spot a new look replete with elegance, style and an air of sophistication never experienced before. Call us on 888-398-0573 and discover the extent to which your old tub r shower will be changed from the unpleasant look to a most appealing and attractive finish unprecedented.
175431	-0.0033	http://www/showerpanguys.com/wi/shower-pan-in-greenbush/
At Shower Pan Guys, you will find that our personnel have the requisite training, skills and expertise in the design of the Shower Pan liner or Shower Pan. Whether you are looking for a flexible shower liner, PVC,One liner, prefabricated shower liner, roll on membrane among other options, you will be at home with our experts to advice you on which Shower Pan is most suitable for your shower. There are some specific details that only professionals can handle. The reason why this issue is important is that you would to use shower liner for water proof environment or other flexible options available. You will require people with expertise to guide you on all these. Thank fully, we do not run out of these exceptionally skilled people to assist you in making the right decision. At Shower Pan Guys in Greenbush, WI, all your shower liner needs will be catered for in a most professional way. Our personnel will implement your proposals in an effortless way. You can be guaranteed of fully bonded, insured and licensed service in Greenbush, WI. We take our work seriously and therefore ensure that all the requirements as set by the local government are adhered to. The safety of any installation is as good as the installer. We do not like putting our clients at a disadvantage with leaking Shower Pan liner such that they cannot enjoy their bathroom experience. As per set rules and regulations, we follow the right procedure in ensuring that you get the full worth of your investment from procurement to the final installment stage. We also rank favorably with other similar service providers in Greenbush, WI. Have the confidence to approach at any time for a quality and high standard services. You will find that from Shower Pan Guys in Greenbush, WI, we help alleviate the shipping cost for our customers. For purchases worth some predetermined amount, you will receive shipping to your destination. Call us on 888-670-3340 and learn more the amount of purchases that will enable you to receive free shipping services. Shower Pan Guys in Greenbush, WI can be contacted over 888-670-3340 to have any aspect of the shower pans clarified to you in way you will understand. Issues such as procurement, quotation, design, inspection, installation, repair and maintenance will be explained to you so that you understand the procedure. You will be able to reach us through email or the online option. Our personnel will be very helpful to you so that you can make the right decision regarding your requirements.

Figure 18: Text examples from C4 with low memorization.

27

Figure 19: Text examples from Wiki40B:en with high memorization.

28

Index	mem	Link and Text
1834612	0.2829	wikidata_id=Q4601742, version_id=13717366317222518793
_START ARTICLE_2003 Polish Air Force Mi-8 crash_START_PARAGRAPH_ On 4 December 2003, a Polish Mi-8 helicopter operated by the 36th Special Aviation Regiment carrying Poland's Prime Minister Leszek Miller crashed near Piaceczno, just outside Warsaw. The pilot performed an autorotation landing in a forest following the failure of both engines. The helicopter suffered extensive damage and was written off as a total loss, but despite the severity of the crash there were no fatalities. Fourteen of the 15 people on board were injured, including Leszek Miller, who had two of his thoracic vertebrae broken. STARTSECTION_Aircraft and pilot_START_PARAGRAPH_ The helicopter was 26 years old at the time of the crash, and was close to the end of its service. It belonged to the 36th Special Aviation Regiment responsible for transporting Polish government officials. The pilot of the helicopter was major Marek Miotsz, later promoted to lieutenant colonel. STARTSECTION_Cause_START_PARAGRAPH_ The cause of the engine failure was determined to be icing. STARTSECTION_Trial_START_PARAGRAPH_ On 10 March 2004, Miotsz was criminally charged with violating flight safety rules and causing the crash. Specifically, the pilot was blamed for not manually turning on the deicing equipment during the flight. The pilot argued that the meteorologic information available to him at the time did not indicate that icing was likely, and hence he was not required to turn on the deicing equipment. He was consulting a thermometer during the flight, but it suffered from a systematic measurement error and hence was unable to warn of icing. In addition, during the flight an unusual thermal inversion occurred; the temperature rose with altitude, which the pilot could not have predicted. NEWLINE_In March 2010, the 6-year trial ended with a verdict of not guilty. The judge in the case noted the pilot expertly carried out the difficult autorotation landing and that the passengers survived because of his superb piloting skills. Leszek Miller declared that if he had to fly again in a helicopter in difficult atmospheric conditions, he would choose Miotsz as his pilot.
610	0.2664	wikidata_id=Q440261, version_id=3093157553250021392
_START ARTICLE_Inga Bejer Engh_START_PARAGRAPH_ Engh is educated Candidate of Law. Following law school she worked with international law for the United Nations in New York City. Upon return to Norway her first job was at Drammen District Court later Oslo District Court and Askier and Bærum Police District. At 32 she began working as a prosecutor. She has prosecuted several major criminal cases including the much publicized methanol distribution case in Østfold in the early 2000s, a case where a man received ten years prison after having thrown acid on his wife, several major drugs and sexual abuse cases. She also prosecuted Tore W. Tvedt, one of the witnesses in the Breivik case, in 2002. STARTSECTION_Personal life_START_PARAGRAPH_Inga Bejer Engh is married and has two young boys. The youngest was born in 2009 and was very premature but has sustained no lasting medical problems. Her decision to become a lawyer was made in the gymnasium (Norwegian equivalent to high school, now called videregående).
173213	0.2531	wikidata_id=Q54085254, version_id=10019130216726221228
_START ARTICLE_RTV silicone_START_PARAGRAPH_RTV Silicone (Room-Temperature-Vulcanizing silicone) is a type of silicone rubber made from a two-component system (base plus curative; A+B) available in a hardness range of very soft to medium--usually from 15 to 40 Shore A. RTV silicones can be cured with a catalyst consisting of either platinum or a tin compound such as dibutyltin dilaurate. Applications include low-temperature over-molding, making molds for reproducing, and lens applications for some optically clear grades. START_SELECTION_Appliactions_START_PARAGRAPH_ To produce the material, the silicone rubber is mixed with the curing agent or vulcanizing agent. Usually, the mixing ratio is a few percent. In order for the RTV silicone to reproduce the surface texture, attention is paid to the cleanliness of the original. Vacuum de-airing removes entrained air bubbles between the mixed silicone and catalyst to ensure optimal tensile strength, which affects reproduction times. In casting and mold-making, RTV silicone rubber reproduces fine details and is suitable for a variety of industrial and art-related applications including prototypes, furniture, sculpture, and architectural elements. RTV silicone rubber can be used to cast materials including wax, gypsum, low melt alloys/metals and urethane, epoxy or polyester resins (without using a release agent). A more recent innovation is the ability of 3D print RTV silicones. RTV silicones' industrial applications include aviation, aerospace, consumer electronics, and microelectronics. Some aviation and aerospace product applications are cockpit instruments, engine electronics potting, and engine gasketing. RTV silicones are used for their ability to withstand mechanical and thermal stress. START_SELECTION_Advantages and disadvantages START_PARAGRAPH_RTV silicone rubber has excellent release properties compared to mold rubbers, which is especially an advantage when doing production casting of resins (polyurethane, polyester, and epoxy). No release agent is required, obviating post-production cleanup. Silicones also exhibit good chemical resistance and high-temperature resistance (205 °C, 400 °F and higher). For this reason, silicone molds are suitable for casting low-melt metals and alloys (e.g. zinc, tin, pewter, and Wood's metal). _NEWLINE_RTV silicone rubbers are, however, generally expensive--especially platinum-cure. They are also sensitive to substances (sulfur-containing modelling clay such as Plastilina, for example) that may prevent the silicone from curing (referred to as cure inhibition). Silicones are usually very thick (high viscosity), and must be vacuum degassed prior to pouring, to minimize bubble entrapment. If making a brush-on rubber mold, the curing time factor between coats is long (longer than urethanes or polysulfides, shorter than latex). Silicone components (A+B) must be mixed accurately by weight (scale required) or they do not work. Tin catalyst silicones shrink somewhat and do not have a long shelf life. Acetoosilane-based RTV release acetic acid during the curing process, and this can attack solder joints, causing the solder to detach from the copper wire.
1478633	0.1693	wikidata_id=Q6697935, version_id=5711381530412598390
_START ARTICLE_Lucky Landing Marina and Seaplane Base_START_SELECTION_Facilities and aircraft START_PARAGRAPH_Lucky Landing Marina and Seaplane Base has one landing area designated 2/20 which measures 15,000 x 4,000 ft (4,572 x 1,219 m). For the 12-month period ending July 31, 2006, the airport had 1,850 aircraft operations, an average of 154 per month: 86% general aviation and 14% air taxi.

Figure 20: Text examples from Wiki40B:en with intermediate memorization.

29

Figure 21: Text examples from Wiki40B:en with low memorization.

30

Index	Estim.	Link and Text
valid 1334662	infl 0.3780	http://www.metro.us/news/archaeologists-vs-robbers-in-israel-s-race-to-find-ancient-scrolls/kZgpfb---HhAktviTsXh_U3HpnRICjA
By Ari Rabinovitch TZEELIM VALLEY, Israel (Reuters) - The disposable paper face masks offer little protection from the clouds of dust that fill the cliffside cave where Israeli archaeologists are wrapping up the largest excavation in the Judean desert of the past half-century. Clipped into safety harnesses, volunteers stand at the cave opening, 250 meters (820 feet) above a dry river bed that leads to the lowest spot on earth, the Dead Sea. They sift through an endless supply of dirt-filled buckets, and the dust they throw in the air reaches the far corners of the cave where a dozen workers crawling on hands and knees can't help but cough. The three-week excavation was the first part of a national campaign to recover as many artefacts as possible, particularly scrolls, left behind by Jewish rebels who hid in the desert some 2,000 years ago, before they are snatched up by antiquity robbers. "These looters that operate in the area are experts at finding scrolls. We go after them, look for what they are looking for and try to catch them," said Guy Fitoussi, head of the Israel Antiquities Authority robbery prevention unit in southern Israel. "This is the game. Like cat and mouse." The Dead Sea Scrolls, a collection of ancient texts written on papyrus and parchment, have already been rescued by scholars. They are among the earliest texts written in the Hebrew language and are on display in the Israel Museum in Jerusalem as a national treasure. Now Israel wants to uncover whatever may remain in the desert hideouts before it is destroyed or ends up on the black market. For a wider image photo essay on the excavation click: http://reut.rs/25zczAKPISTOL-PACKING ARCHAELOGIST According to Israeli law, all relics found on land or at sea belong to the state. Fitoussi, a pistol-packing archaeologist with authority to arrest looters, and his team catch about 100 of them each year. Most are fined; some are sent to jail. In 2014, they arrested six people who were plundering this particular cavern, known as the Cave of Skulls, where seven skulls had been found from Jews of the Bar Kokhba rebellion against Rome in the 2nd century. That raid, Fitoussi said, helped spur the multi-year government-backed excavation program and focused their initial efforts at this site, about a two-hour drive southeast from Jerusalem. To access the cave, diggers don climbing gear and descend 20 minutes from their campsite along a steep path that hugs the rocky cliff. Inside, the grotto expands 160 square meters (1,720 square feet), including a number of cramped tunnels that extend deep into the mountain. The limestone walls of the dry desert cave are perfect for preservation, said Uri Davidovich, an archaeologist from Tel Aviv University who was one of the dig's directors. One 19-year-old volunteer, working flat on her belly in a dark crawl space, dirt mixed with sweat covering her face and digging with her fingers, unearthed a thin, 25 centimeter (10 inch)-long rope that most likely was used by the Bar Kokhba rebels. A rope this length was a rare discovery, Davidovich said. They haven't found any scrolls yet, he said, but the artefacts found in this cave, and countless others nearby, will provide historians rare insight into how people lived 2,000 to 8,000 years ago. (Editing by Jeffrey Heller/Jeremy Gaunt)
train 2077314	men 0.3780	https://wwwmetro.us/news/archaeologists-vs-robbers-in-israel-s-race-to-find-ancient-scrolls/kZgpfb---HhAktviTsXh_U3HpNRICjA
By Ari Rabinovitch TZEELIM VALLEY, Israel (Reuters) - The disposable paper face masks offer little protection from the clouds of dust that fill the cliffside cave where Israeli archaeologists are wrapping up the largest excavation in the Judean desert of the past half-century. Clipped into safety harnesses, volunteers stand at the cave opening, 250 meters (820 feet) above a dry river bed that leads to the lowestspot on earth, the Dead Sea. They sift through an endless supply of dirt-filled buckets, and the dust they throw in the air reaches the far corners of the cave where a dozen workers crawling on hands and knees can't help but cough. The three-week excavation was the first part of a national campaign to recover as many artefacts as possible, particularly scrolls, left behind by Jewish rebels who hid in the desert some 2,000 years ago, before they are snatched up by antiquity robbers. "These looters that operate in the area are experts at finding scrolls. We go after them, look for what they are looking for and try to catch them," said Guy Fitoussi, head of the Israel Antiquities Authority robbery prevention unit in southern Israel. "This is the game. Like cat and mouse." The Dead Sea Scrolls, a collection of ancient texts written on papyrus and parchment, have already been rescued by scholars. They are among the earliest tests written in the Hebrew language and are on display in the Israel Museum in Jerusalem as a national treasure. Now Israel wants to uncover whatever may remain in the desert hideouts before it is destroyed or ends up on the black market. For a wider Image photo essay on the excavation click: http://reut.rs/25zczAKPISTOL-PACKING ARCHAELOGIST According to Israeli law, all relics found on land or at sea belong to the state. Fitoussi, a pistol-packing archaeologist with authority to arrest looters, and his team catch about 100 of them each year. Most are fined; some are sent to jail. In 2014, they arrested six people who wore plundering this particular cavern, known as the Cave of Skulls, where seven skulls had been found from Jews of the Bar Kokhba rebellion against Rome in the 2nd century. That raid, Fitoussi said, helped spur the multi-year government-backed excavation program and focused their initial efforts at this site, about a two-hour drive southeast from Jerusalem. To access the cave, diggers don climbing gear and descend 20 minutes from their campsite along a steep path that hugsthe rocky cliff. Inside, the grotto expands 160 square meters (1,720 square feet), including a number of cramped tunnels that extend deep into the mountain. The limestone walls of the dry desert cave are perfect for preservation, said Uri Davidovich, an archaeologist from Tel Aviv University who was one of the dig's directors. One 19-year-old volunteer, working flat on her belly in a dark crawl space, dirt mixed with sweat covering her face and digging with her fingers, unearthed a thin, 25 centimeter (10 inch)-long rope that most likely was used by the Bar Kokhba rebels. A rope this length was a rare discovery, Davidovich said. They haven't found any scrolls yet, he said, but the artefacts found in this cave, and countless others nearby, will provide historians rare insight into how people lived 2,000 to 8,000 years ago. (Editingby Jeffrey Heller/Jeremy Gaunt)

Figure 22: Validation / training example pair from RealNews with high influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

31

Figure 23: Validation / training example pair from RealNews with relatively high influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

32

Index	Estim.	Link and Text
valid 682107	infl 0.0673	https://www.pasadenastarnews.com/2009/09/16/ducks-ushering-in-a-new-era-for-2009-10-season/
ANAHEIM - On a night when Francois Beauchemin had two assists in Toronto, and Chris Pronger blocked six shots in Detroit, 13,869 ducks fans might have been lost without their programs on Wednesday night. It's only the first game of the preseason, but a new era has clearly begun. A group of mostly newcomers in Ducks uniforms beat Phoenix 3-2 in a shootout on Wednesday at Honda Center. A familiar face made the biggest impact, however, as Bobby Ryan scored two goals. Ryan also scored two goals in the Ducks' preseason opener last year, when he was trying to make the team's opening-day roster as a rookie. His bid failed and Ryan was forced to start the season in the American Hockey League, mostly due to salary-cap constraints. The circumstances could not be much different this year. Ryan finished the season as a Calder Trophy candidate, and began this season wearing a temporary alternate captain's "A" on his jersey along with Jeffrey Lupul and Sheldon Brookbank. Nick Boynton, a 30-year-old defenseman signed in the offseason, was the eldest player in uniform. Of the 20 players chosen to play, only goaltender Jonas Hiller and center Ryan Carter spent all of last season with the Ducks. Hiller started in goal and looked sharp, stopping 18 of the 19 shots he faced. Timo Pielmeier, acquired in the trade that sent Kent Huskins and Travis Moen to San Jose, came in halfway through the third period and stopped 22 of 23 shots before the game went into overtime. A phantom interference penalty sent ducks defenseman Steve Eminger to the box, and set up a power-play goal by the Coyotes' Lauri Korpikowski with 20.2 left in the period. Korpikowski deflected a long-range shot by Keith Yandle, and Hiller did well just to get his glove on it. Ryan got the goal back early in the second period, poking in a rebound of a Dan Sexton shot past former Kings goalie Jason LaBarbera. With 1:08 elapsed in the third period, Ryan made it 2-1 on a breakaway shot that caromed in off a Phoenix stick. Phoenix tied the game at 2 at 12:30 of the third period, when Sheldon Brookbank made a costly turnover in front of his own net. Chad Kolarik batted down Brookbank's pass just a few feet in front of Pielmeier and tucked the puck in behind the goaltender.
train 494435	men 0.1439	https://www.dailybreeze.com/2009/09/17/newcomers-help-lead-ducks-to-exhibition-win/
Beauchemin had two assists in Toronto and Chris Pronger blocked six shots in Detroit, and Chris Pronger blocked six shots in Detroit, the 13,869 ducks' fans might have shown at at Honda Center Wednesday needed programs to identify the players on their fave been lost without their programs on Wednesday nightorite team. It's only the first game of the preseason, but a new era has clearly begun. A group of mostly newcomers in ducks uniforms beat Phoenix 3-2 in a shootout on Wednesday at Honda Center. A, although a familiar face made the biggest impact, however, as Bobby Ryan scored two goals in regulation and another in the shootout. Ryan also scored two goals in the Ducks' preseason opener last year, when he was trying to make the team's opening-day roster as a rookie. His bid failed and Ryan was forced to start the season in the American Hockey League, mostly due to salary-cap constraints. The circumstances could not be much different this yearore different now. Ryan finished thelast season as a Calder Trophy candidate, and began this season wearing a temporary alternate captain's "A" on his jersey along with Joffrey Lupul and Sheldon Brookbank. Nick Boynton, a 30-year-old defenseman signed in the offshore, was the eldest player in uniform. Of the 20 players chosen to play, only goaltender Jonas Hiller and center Ryan Carter spent all of last season with theucks. Hiller started in goal and looked sharp, stopping 18 of the 19 shots he faced. Timo Pielmeier, acquired in the trade that sent Kent Huskins and Travis Moen to San Jose, came in halfway through the third period and stopped 27 of 2 of 23 shots before the game went into overtime. A phantom interference penalty sent ducks defenseman Steve Eminger to the box, and set up a power-play goal by the Coyotes' Lauri Korpikowski with 20.2 left in the period. Korpikowski deflected a long-range shot by Keith Yandle, and Hiller did well just to get his glove on it opening period. Ryan got the goal back early in the second period, poking in a rebound of a Dan Sexton shot past former Kings goalie Jason LaBarbera at the 3:09 mark. With 1:08 elapsed in the third period, Ryan made it 2-1 on a breakaway shot that caromed in off a Phoenix stick. Phoenix tied the game at 2 at 12:30 of the third period, when Sheldon Brookbank made a costly turnover in front of his own net. Chad Kolarik batted down Brookbank's pass just a few feet in front of Pielmeier and tucked the puck in behind the goaltender two attempts.

Figure 24: Validation / training example pair from RealNews with intermediate influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

33

Figure 25: Validation / training example pair from RealNews with relatively low influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

34

Index	Estim.	Link and Text
valid 1556405	infl 0.0155	https://www.deccanchronicle.com/nation/current-affairs/280218/after-darwin-union-min-targets-newtons-theory-says-mantras-coded.html
Earlier in January, Satyapal Singh had claimed that Charles Darwin's theory of evolution of man was 'scientifically wrong' and it needed to be changed in school and college curriculum. (Photo: File) New Delhi: Minister of State for Human Resource Development, Satyapal Singh speaking at a meeting of the Central Advisory Board of Education (CABE) on January 15 and 16 said that mantras codified the 'laws of motion' much before they were framed by Issac Newton. He also suggested that Vaastu compliance of educational buildings was important for learning, according to a report in Hindustan Times. "There are mantras which codified 'laws of motion' much before it was discovered by the Newton. Hence it is essential that traditional knowledge must be incorporated in our curriculum," Singh was quoted as saying by the minutes of the meeting of the government's highest advisory body for policymaking in education. Earlier in January, Satyapal Singh had claimed that Charles Darwin's theory of evolution of man was "scientifically wrong" and it needed to be changed in school and college curriculum. Singh said our ancestors have nowhere mentioned that they saw an ape turning into a man.
train 1903637	men 0.1927	https://www.deseretnews.com/article/29058/MUBARAK-SAYS-HE-WILL-VISIT-ISRAEL-IF-IT-SHOBS-FLEXIBILITY.html
President Hosni Mubarak said he is willing to visit Israel if the Jewish state shows flexibility in promoting peace and agrees to take part in an international Middle Eastern in Januarist peace conference, according to an interview published Saturday. "I am still saying that I am ready to travel to Israel provided that this would lead to real progress on the road to solving the Middle East problem," Satir Mubarak told the Egyapal Singh had cliaiptian newspaper Al Massa.The reminded that Charlesarks represented the second time in less than a week that Mubarak had expressed readiness to make a peace visit to Israel. In an interview Darwinc 24 with the Kuwaiti newspaper lAnba, Mubarak said he was prepared to travel to Israel if the visit would promote a solution of the Palestinian problem and a just Middle East peace. The offer was initially embraced by Israel, but later two of Mubarak's theorsernior foreign policy of eades said the volution of man was isit was contingent on the Jewish state opening negotiations with the Palestine Liberation Organization. Israel rejects the PLO as a negotiating partner despite the United States scientifically wrong' and it needed to be changed in school and college curriculum decision Dec. (Photo: File) New Delhi: Minister of State for Human Resource Development, Satyapal Singh speaking at a meeting of the Central Advisory Board of Education (CABE) on January 15 and 16 said that mantras codified the 'laws of motion' much before they were framed by Issac Newton 4 to open a dialogue with the group. He also suggested that Vaastu compliance of educational buildings was important for learning, according to a report in Hindustan Times. "There are mantras which codified 'laws of motion' much before it was discovered by the Newton. Hence it is essential that traditional knowledge must be incorporated in our curriculum," Singh was quoted as saying by the minutes of the meeting of the government's highest advisory body for policymaking in education. Earlier in January, Satyapal Singh had claimed that Charles Darwin's theory of evolution of man was "scientifically wrong" and it needed to be changed in school and college curriculum. Singh said our ancestors have nowhere mentioned that they saw an ape turning into a man.

Figure 26: Validation / training example pair from RealNews with low influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

35

Figure 27: Validation / training example pair from C4 with high to intermediate influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

36

Figure 28: Validation / training example pair from C4 with intermediate to low influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

37

Figure 29: Validation / training example pair from Wiki40B:en with high to intermediate influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

38

Figure 30: Validation / training example pair from Wiki40B:en with intermediate to low influence. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

39

Figure 31: Generated / training example pair from RealNews with high to intermediate influence. The generated examples are directly taken from publicly released generations of the Grover-Mega  $(p = 0.96)$  model [Zellers et al., 2019]. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

40

Index	Estim.	Link and Text
generated 19128	infl 0.0357	https://www.globalresearch.ca/us-plans-designate-iran-revolutionary-guards-irgc-terrorists/5673967
Notes Disclaimer: The contents of this article are of sole responsibility of the author(s). The Centre for Research on Globalization will not be responsible for any inaccurate or incorrect statement in this article. The Centre of Research on Globalization grants permission to cross-post Global Research articles on community internet sites as long the source and copyright are acknowledged together with a hyperlink to the original Global Research article. For publication of Global Research articles in print or other forms including commercial internet sites, contact: [email protected] www.globalresearch.ca contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to our readers under the provisions of "fair use" in an effort to advance a better understanding of political, economic and social issues. The material on this site is distributed without profit to those who have expressed a prior interest in receiving it for research and educational purposes. If you wish to use copyrighted material for purposes other than "fair use" you must request permission from the copyright owner. For media inquiries: [email protected]
train 1244174	men 0.1904	https://www.globalresearch.ca/the-american-media-empire-of-managed-news/20568
Operation FALCONotes mass arrests; suspension of habeas corpus by the Chief Executive; US and NATO as protectors of global capital; structure of inequality maintained by global capital and military empire worldwide; repression that addresses us all; the 1948 Declaration of Human Rights Disclaimer: The contents of this article are of sole responsibility of the author(s). The Centre for Research on Globalization will not be responsible for any inaccurate or incorrect statement in this article. The Centre of Research on Globalization grants permission to cross-post Global Research articles on community internet sites as long the source and copyright are acknowledged together with a hyperlink to the original Global Research article. For publication of Global Research articles in print or other forms including commercial internet sites, contact: [email protected] www.global research.ca contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to our readers under the provisions of "fair use" in an effort to advance a better understanding of political, economic and social issues. The material on this site is distributed without profit to those who have expressed a prior interest in receiving it for research and educational purposes. If you wish to use copyrighted material for purposes other than "fair use" you must request permission from thecopyright owner. For media inquiries: [email protected]

generated 7818	infl 0.0207	https://en.trend.az/business/energy/3046526.html
Baku, Azerbaijan, April 15 By Ali Mustafayev - Trend: Italy would have to import almost 50 percent of its natural gas through Germany without the Trans Adriatic Pipeline (TAP), managing director of TAP AG Beat Rathmann told Trend April 15. TAP is a part of the Southern Gas Corridor project, aimed at diversifying the European energy supply sources and routes. The 930-kilometer long TAP project envisages transportation of gas from Azerbaijan's Shah Deniz Zone 2 to the EU countries. The pipeline will connect to the Trans Anatolian Natural Gas Pipeline (TANAP) on the Turkish-Greek border, run through Greece, Albania and the Adriatic Sea, before coming ashore in Italy's south. TAP's shareholding is comprised of BP (20 percent), SOCAR (20 percent), Snam S.p.A. (20 percent), Fluxys (19 percent), Enagas (16 percent) and Axpo (5 percent). --- Follow the author on Twitter: @Ali_Mustafayev Follow Trend on Telegram. Only most interesting and important news
train 1246306	mem 0.1520	https://en.trend.az/business/energy/2791327.html
Baku, Azerbaijan, April 15 August 29 By Ali MustafaLeman Zeyenalova - Trend: ItaNearly 60 percent of pipes have been would have to import almost 50 percent of its natural gas through Germany withouttelded as part of construction of the Trans Adriatic Pipeline (TAP), managing director of in Greece and Albania, TAP AG Beat Rathmann told Trend April 15consortium said in a message on its Twitter page. This accounts for 458 kilometers of the total of 765 kilometers of TAP is a part of the Southern Gas Corridor project, aimed at diversifying the European energy supply sources and routrout in those two countries. The 930-kilometer long TAP is a part of the Southern Gas Corridor, which is one of the priority energy projects for the European Union. The project envisages transportation of gas from Azerbaijan's Shah Deniz Zone 2 to the EU countries. The pipeline will connect to the Trans Anatolian Natural Gas Pipeline (TANAP) on the Turkish-Greek border, run through Greece, Albania and the Adriatic Sea, before coming ashore in Italy's south. TAP will be 878 kilometers in length (Greece 550 kilometers, Albania 215 kilometers, Adriatic Sea 105 kilometers, and Italy 8 kilometers). TAP's shareholding is comprised of BP (20 percent), SOCAR (20 percent), Snam S.p.A. (20 percent), Fluxys (19 percent), Enagas (16 percent) and Axpo (5 percent). --- Follow the author on Twitter: @Aliyaman_MustafaZeyev Follow Trend on Telegram. Only most interesting and important news

Figure 32: Generated / training example pair from RealNews with intermediate to low influence. The generated examples are directly taken from publicly released generations of the Grover-Mega  $(p = 0.96)$  model [Zellers et al., 2019]. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

41

Index	Estim.	Link and Text
generated 21998	infl 0.0218	http://www.arabnews.com/node/1483111/world
NEW DELHI: India is likely to see average monsoon rains this year, the state-run weather office said on Monday, which should support agricultural production and economic growth in Asia's third-biggest economy, where half of the farmland lacks irrigation. Monsoon rainfall is expected to be 96 percent of the long-term average, M. Rajeevan, secretary at the Ministry of Earth Sciences, told a news conference. The India Meteorological Department (IMD) defines average, or normal, rainfall as between 96 percent and 104 percent of a 50-year average of 89 centimeters for the entire four-month season beginning June. "Overall, the country is expected to have well distributed rainfall scenario during the 2019 monsoon season, which will be beneficial to farmers in the country during the ensuing Kharif (summer-planting) season," the IMD said in its forecast. Skymet, the country's only private weather forecasting agency, earlier this month forecast rainfall could be below normal this year. Monsoon rains, the lifeblood for India's farm-dependent $2.6 trillion economy, arrive on the southern tip of Kerala state around June 1 and retreat from the desert state of Rajasthan by September. After a wet spell, sowing of summer-sown crops gets off to a strong start, boosting crop yields and output which in turn raises rural incomes and usually lifts consumer spending in India. If plentiful monsoon rains lift agricultural production this year, that could keep food prices under control. Subdued overall inflation could also add to pressure on India's central bank to cut interest rates. "IMD's prelim forecast, showing near-normal and well-distributed rainfall, will bode well for near-term food inflation," said Madhavi Arora, lead economist at Edelweiss Securities, FX and Rates. The next policy review by India's central bank is scheduled for June 6, after the country's election. Millions of Indians are casting their votes in a mammoth general election, spread over seven weeks. On the downside, higher production could cause farmers to get hit by low crop prices, a major cause for concern in rural India, where most Indians live, in the past two years. After falling for five straight months, retail food prices in India rose 0.30 percent in March from a year earlier. Last month, a senior IMD official told Reuters that this year's monsoon was likely to be robust and healthy provided there wasn't a surprise El Nino phenomenon. "El Nino is weakening and we expect that El Nino will get weakened further. There is no reason to be worried about El Nino," Rajeevan said. A strong El Nino, marked by a warming of the sea surface on the Pacific Ocean, can cause severe drought in Australia, Southeast Asia and India, while drenching other parts of the world such as the US Midwest and Brazil in rains. The emergence of a strong El Nino triggered back-to-back droughts in 2014 and 2015, for only the fourth time in over a century, driving some Indian farmers to penury and suicide. ECONOMIC GROWTH Good rains will spur the planting of crops such as rice, corn, cane, cotton and soybeans. Stronger agricultural production would help support India's economy. It is still the world's fastest-growing major economy, but annual growth slowed to 6.6 percent in the December quarter, from 7.0 percent in the previous period and the slowest in five quarters. The monsoon usually covers the half of the country in the first 15 days. The rains reach central India's soybean areas by the third week of June and western cotton-growing areas by the first week of July. India's weather office will update its forecast in the first week of June. However, on average, the IMD has forecast accurately only once every five years over the past two decades, even after taking into account an error band of plus or minus 5 percentage points.
train 326212	men 0.1555	https://www.reuters.com/article/us-india-monsoon/india-set-to-receive-average-monsoon-rains-in-boost-for-poll-bound-modi-idUSKBN1HN1AQ
NEW DELHI: India is likely to see average monsoon rains this year average monsoon rains in 2018, the state-run weather office said on Monday weather office said, which should support agricultural production, raising the possibility of higher farm and economic growth in Asia's third-biggest economy, where half of the farmland lacks irrigation. Monsoon rainfall is, the lifeblood of the country's $2 trillion economy, and expected to be 96 percent of the 7 percent of a long-term average, MR. Rajeevanamsh, secretary at the Ministry of Earth Sciences, told a news conference. The director general of the state-run India Meteorological Department (IMD) defines average, or normal, rainfall as between 96 percent and 104 percent of a 50-year average of 89 centimeters for the entire four-month season beginning June 6, for the entire four-month season beginning June 6. "We see very less probability of a deficit monsoon," Ramesh said on Monday. Overall than lifting farm and wider economic growth, the country is expected to have well-distributed rainfall scenario during the spell of good rains will keep a lid on inflation, potentially tempting Prime Minister Narendra Modi to bring forward general elections due in May 2019 monsoon season. India's weather office defines average, which will be beneficial to farmers in the country during the ensuing Kharif (summer normal, rainfall as between 96 percent and 104 percent of a 50-planting season year-end average of 89 cms for the entire four-month season beginning June. "The moderate La Nina conditions developed in the equatorial Pacific during last year started weakening in the early part of this year and currently have turned to weak La Nina conditions," the IMD said in its forecast statement. Skymet La Nina is a weather pattern that brings equatorial Pacific Ocean temperatures, the country's only rainfall patterns and winds closer to ovate weather forecasting agency, earlier this month forecast rainfall could be below normal this year average. The latest forecasts from global models indicate conditions over the Pacific will turn neutral before the beginning of monsoon season, the Monsoon Rainfall said. Good rains will spur the planting of crops such as rice, the lifeblood forcor, cotton and soybeans, accelerating economic growth that rose 7.2 percent in the December quarter, its fastest in five quarters, compared with China's 6.8 percent in that quarter. Growth in the December quarter restored India's farmstatus as the world's fastest growing major economy. RURAL INCOME Average monsoon rains, with good distribution in July and August would support rural demand, said Rupa Rege Nitsure, group chief economist at L&T Finance Holdings, a Mumbai-dependent $2 based non-banking finance company's trillion economy Good rains boost rural incomes, arrive on the southern tip of Kerala state around June 1 and retreat from the desert state of Rajasthan by September lifting the demand for an array of consumer goods ranging from lipsticks to refrigerators. After a wet-spelin average monsoon would keep food inflation lower, sowing of summer-sown crops gets off to a strong start Nitsure said. On Monday, boosting crop yields and output which in turn raises rural incomes and usually lifts consumer spending in government data showed India's wholesale food prices fell 0.07 percent in March 2018 from a year earlier. If plentiful monsoon rains lift agricultural production this year, India's weather office will update its forecast in June. On an average, that could e IMD has forecast accurately only once in every five years over the past two decades, even after taking into account an error band of plus or minus 5 percentage points. The rains reach central India's central-bansoybean areas by the third week to cut interest rates of June and western cotton-growing areas by the first week of July. Good rains would help boost soybean output which in turn could cut expensive vegetable oil imports by India, the world's biggest importer of edible oils, which is the third-biggest import item after crude oil and gold. Currently India is struggling with huge amounts of sugar and good rains could further bump up the supply of the sweetener. "IMD's prel forecastGood monsoon will help bring down edible oil imports, showing nearnormal and well-distributed rainfall, will bode well for near-term food inflation but it could also create a problem of plenty in pulses and sugar," said Madhavi Arora, lead economist at Edelweiss Securities, FX and Rates. The next policy review by India's central bank is scheduled for June 6, after the country's election. Millions of Indians are casting their votes in a mammoth general election, spread over seven weeks. On the downside, higher production could mean farmers continue to get hit by low crop prices, a major cause for concern in rural India, where most Indians live, in the past two yearsumbai-based dealer with a global trading firm. After falling for five straight months, retail food prices in India's central bank is scheduled for June 6, after the country's election. Millions of Indians are casting their votes in a mammoth general election, spread over seven weeks. On the downside, higher production could mean farmers continue to get hit by low crop prices, a major cause for concern in rural India, where most Indians live, in the past two yearsumbai-based dealer with a global trading firm. After falling for five straight months, retail food prices in India's central bank is planned for June 6, after the country's election. Millions of Indians are casting their votes in a mammoth general election, spread over seven weeks. On the downside, higher production could mean farmers continue to get hit by low crop prices, a major cause for concern in rural India, where most Indians live, in the past two yearsumbai-based dealer with a global trading firm. After falling for five straight months, retail food prices in India's central bank is planned for June 6, 2015, for only the fourth time in over a century, driving some Indian farmers to penury and suicide. ECONOMIC GROWTH Good rains will spur the planting of crops such as rice, corn, cane, cotton and soybeans. Stronger agricultural production would help support India's economy. It is still the world's fastest growing major economy, but annual growth slowed to 6.6 percent in the December quarter, from 7.0 percent in the previous period and the slowest in five quarters. The monsoon usually covers the half of the country in the first 15 days. The rains reach central India's soybean areas by the third week of June and western cotton-growing areas by the first week of July. India's weather office will update its forecast in the first week of June. However, on averagethe IMD has forecast accurately only once every five years over the past two decades, even after taking into account an error band of plus or minus 5 percentage points.

Figure 33: Generated / training example pair from RealNews with low influence. The generated examples are directly taken from publicly released generations of the Grover-Mega (p=0.96) model [Zellers et al., 2019]. Red / green highlighted text indicate deleted / added text in the training example comparing to the corresponding validation example, generated using Python difflib.

42