Update README.md

README.md

@@ -37,7 +37,7 @@ The ERWT models are trained for **experimental purposes**, please use them with
 
 You find more detailed information below. Please consult the **limitations** section (seriously, read this section before using the models, **we don't repent in public just for fun**). 
 
-If you can't get enough of these peas and crave for some more, you can still consult our working paper ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing) for more background information and nerdy evaluation stuff (work in progress, handle with care and kindness).
+If you can't get enough of these peas and crave some more. In that case, you can consult our working paper ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing) for more background information and nerdy evaluation stuff (work in progress, handle with care and kindness).
 
 ## Background: MDMA to the rescue. 🙂
 
@@ -45,16 +45,16 @@ ERWT was created using a **M**eta**D**ata **M**asking **A**pproach (or **MDMA**
 
 ERWT is a [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) model, fine-tuned on a random subsample taken from the Heritage Made Digital newspaper collection. The training data comprises around half a billion words. 
 
-To unleash the power of MDMA, we adapted to the training routine for the masked language model. When preprocessing the text, we prepended each segment of hundred words with a time-stamp (year of publication) and a special `[DATE]` token. 
+To unleash the power of MDMA, we adapted to the training routine for the masked language model. When preprocessing the text, we prepended each segment of hundred words with a time stamp (year of publication) and a special `[DATE]` token. 
 
 The snippet below, taken from the [Londonderry Sentinel](https://www.britishnewspaperarchive.co.uk/viewer/bl/0001480/18700722/014/0002),  
 ```python
 "1870 [DATE] Every scrap of intelligence relative to the war between France and Prussia is now read with interest."
 ```
 
-These formatted chunks of text are then forward to the data collator and eventually the language model.
+These formatted chunks of text are then forwarded to the data collator and eventually the language model.
 
-Exposed to both the tokens and (temporal) metadata, the model learns a relation between text and time. When a token is masked, the prepended `year` field is taken into account when predicting hidden word in the text. Vice versa, when the metadata token at the front of the formatted snippet is hidden, the models aims to predict the year of publication based on the content of a document.
+Exposed to both the tokens and (temporal) metadata, the model learns a relation between text and time. When a token is masked, the prepended `year` field is taken into account when predicting hidden words in the text. Vice versa, when the metadata token is hidden, the model aims to predict the year of publication based on the content.
 
 ## Intended Uses: LMs as History Machines.
 
@@ -66,11 +66,11 @@ Let's show how ERWT works with a very concrete example.
 
 The ERWT models are trained on British newspapers from before 1880 (Why? Long story, don't ask...) and can be used to monitor historical change in this specific context.
 
-Imagine you are confronted with the following snippet "We received a letter from [MASK] Majesty" and want to predict correct pronoun (again assuming a British context). 
+Imagine you are confronted with the following snippet "We received a letter from [MASK] Majesty" and want to predict the correct pronoun (again assuming a British context). 
 
-👩‍🏫 **History Intermezzo** Please remember, for most of in the nineteenth-century, Queen Victoria ruled Britain. From 1837 to 1901 to be precise. Her nineteenth century predecessors (George III, George IV and William IV) were all male.
+👩‍🏫 **History Intermezzo** Please remember, for most of in the nineteenth century, Queen Victoria ruled Britain. From 1837 to 1901 to be precise. Her nineteenth-century predecessors (George III, George IV and William IV) were all male.
 
-While a standard language model will provide you with one a general prediction, based on what it has observed previously in the training corpus, ERWT models allow you to manipulate to prediction, by anchoring the text in specific the year.
+While a standard language model will provide you with one a general prediction, based on what it has observed previously in the training corpus, ERWT models allow you to manipulate to prediction, by anchoring the text in a specific year.
 
 ```python
 from transformers import pipeline
@@ -96,7 +96,7 @@ However, if we change the date at the start of the sentence to 1850:
 mask_filler(f"1850 [DATE] We received a letter from [MASK] Majesty.")
 ```
 
-Will put most of probability mass on the token "her" and only a little bit on "him".
+Will put most the probability mass on the token "her" and only a little bit on "him".
 
 ```python
 {'score': 0.8168327212333679,
@@ -109,13 +109,13 @@ You can repeat this experiment for yourself using the example sentences in the *
 
 Okay, but why is this interesting? 
 
-Firstly, eyeballing some toy-examples (but also using more rigorous metrics such as perplexity) shows that MLMs can perform more accurate predictions when it has access to temporal metadata. In other words, ERWT's prediction reflects historical language use more accurately. Model that are sensitive to historical context could 
+Firstly, eyeballing some toy examples (but also using more rigorous metrics such as perplexity) shows that MLMs can perform more accurate predictions when it has access to temporal metadata. In other words, ERWT's prediction reflects historical language use more accurately. 
 
-Secondly, MDMA may reduce biases induced by imbalances in the training data (or at least gives us more of a handle on this problem). Admittedly, we have to prove this more formally, but some experiments at least hint in this direction. The data used for training is highly biased towards the Victorian age and a standard language model trained on this corpus will predict "her" for ```"[MASK] Majesty"```. 
+Secondly, MDMA may reduce biases induced by imbalances in the training data (or at least give us more of a handle on this problem). Admittedly, we have to prove this more formally, but some experiments at least hint in this direction. The data used for training is highly biased towards the Victorian age and a standard language model trained on this corpus will predict "her" for ```"[MASK] Majesty"```. 
 
 ### Date Prediction
 
-Another feature of the ERWT model series, is date prediction. Remember that during training the temporal metadata token is often masked. In this case the model effectively learns to situate documents in time based on the tokens they contain.
+Another feature of the ERWT model series is date prediction. Remember that during training the temporal metadata token is often masked. In this case, the model effectively learns to situate documents in time based on the tokens they contain.
 
 By masking the year token, ERWT guesses the document's year of publication.
 
@@ -147,24 +147,24 @@ A few years later, in 1870, Prussia aimed artillery southwards and invaded Franc
 mask_filler("[MASK] [DATE] The Franco-Prussian war is a matter of great concern.")
 ```
 
-ERWT clearly learned a lot about history of German unification by ploughing through a plethora of nineteenth century newspaper articles: it correctly returns "1870" as the predicted year.
+ERWT clearly learned a lot about the history of German unification by ploughing through a plethora of nineteenth-century newspaper articles: it correctly returns "1870" as the predicted year.
 
-Again, we have to ask: Who cares? Wikipedia can tell us pretty much the same. More importantly, don't we already have timestamps for newspaper data.
+Again, we have to ask: Who cares? Wikipedia can tell us pretty much the same. More importantly, don't we already have timestamps for newspaper data?
 
 In both cases, our answers would be "yes, but...". ERWT's time-stamping powers have little instrumental use and won't make us rich (but donations are welcome of course 🤑). Nonetheless, we believe date prediction has value for research purposes. We can use ERWT for "fictitious" prediction, i.e. as a diagnostic tool. 
 
 Firstly, we used date prediction for evaluation purposes, to measure which training routine produces models 
-Secondly, we could use it as an analytical tool, to study how temporal variation **within** text documents and further scrutinise which features drive the time prediction (it goes without saying that the same applies to other metadata fields, but example predicting political orientation).
+Secondly, we could use it as an analytical tool, to study how temporal variation **within** text documents and further scrutinise which features drive the time prediction (it goes without saying that the same applies to other metadata fields, but for example predicting political orientation).
 
 ## Limitations
 
-The ERWT series were trained for evaluation purposes, and therefore carry some critical limitations. 
+The ERWT series were trained for evaluation purposes and therefore carry some critical limitations. 
 
 ### Training Data
 
-Many of the limitations are a direct result of the data. ERWT models are trained on a rather small subsample of nineteenth-century British newspapers, and its predictions have to be understood in this context (remember, Her Majesty?). Moreover, the corpus has a strong Metropolitan and liberal bias (see section on Data Description for more information). 
+Many of the limitations are a direct result of the data. ERWT models are trained on a rather small subsample of nineteenth-century British newspapers, and its predictions have to be understood in this context (remember, Her Majesty?). Moreover, the corpus has a strong Metropolitan and liberal bias (see the section on Data Description for more information). 
 
-Historically models tend to reflect the past (and present?) stereotypes and prejudices. We strongly advice against using these models outside of the context of historical research. The predictions are likely to exhibit harmful biases and should be investigated critically and understood within the context of nineteenth century British cultural history.
+Historically models tend to reflect past (and present?) stereotypes and prejudices. We strongly advise against using these models outside of the context of historical research. The predictions are likely to exhibit harmful biases and should be investigated critically and understood within the context of nineteenth-century British cultural history.
 
 ### Training Routine
 
@@ -182,7 +182,7 @@ Want to know how much, then read our paper!
 
 The ERWT models are trained on an openly accessible newspaper corpus created by the [Heritage Made Digital (HMD) newspaper digitisation project](footnote{https://blogs.bl.uk/thenewsroom/2019/01/heritage-made-digital-the-newspapers.html).
 The HMD newspapers comprise around 2 billion words in total, but the bulk of the articles originate from the (then) liberal paper *The Sun*. 
-Geographically, most papers are metropolitan (i.e. based in London). The inclusion of *The Northern Daily Times* and *Liverpool Standard*, adds some geographical diversity to this corpus. The political classification are taken for historical newspaper press directories, please read [our paper](https://academic.oup.com/dsh/advance-article/doi/10.1093/llc/fqac037/6644524?searchresult=1) on bias in newspaper collections for more information. 
+Geographically, most papers are metropolitan (i.e. based in London). The inclusion of *The Northern Daily Times* and *Liverpool Standard*, adds some geographical diversity to this corpus. The political classification is based on historical newspaper press directories, please read [our paper](https://academic.oup.com/dsh/advance-article/doi/10.1093/llc/fqac037/6644524?searchresult=1) on bias in newspaper collections for more information. 
 
 The table below contains a more detailed overview of the corpus.
 
@@ -209,11 +209,11 @@ Temporally, most of the articles date from the second half of the nineteenth cen
 
 ## Evaluation
 
-Our article ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing) comprises quite an extensive evaluation of all the language models created with MDMA. For details we recommend you read and cite the current working papers. 
+Our article ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing) comprises quite an extensive evaluation of all the language models created with MDMA. For details, we recommend you read and cite the current working papers. 
 
-The table below shows the [pseudo-perplexity](https://arxiv.org/abs/1910.14659) scores for different models using text document of 64 and 128 tokens.
+The table below shows the [pseudo-perplexity](https://arxiv.org/abs/1910.14659) scores for different models using text documents of 64 and 128 tokens.
 
-In general, [ERWT-year-masked-25](https://huggingface.co/Livingwithmachines/erwt-year-masked-25) turned out to yield the most competitive scores across different task and generally recommend you use this model. 
+In general, [ERWT-year-masked-25](https://huggingface.co/Livingwithmachines/erwt-year-masked-25) turned out to yield the most competitive scores across different tasks and generally recommend you use this model.