Update README.md

README.md

@@ -5,9 +5,10 @@ tags:
 - library
 - historic
 - glam
+- mdma
 license: mit
 metrics:
-- f1
+- pseudo-perplexity
 widget:
 - text: "1820 [DATE] We received a letter from [MASK] Majesty."
 - text: "1850 [DATE] We received a letter from [MASK] Majesty."
@@ -21,15 +22,29 @@ widget:
 
 # ERWT-year
 
-🌺ERWT is a language model that is (🤭 maybe 🤫) better at history than you...🌺
+🌺ERWT\* a language model that (🤭 maybe 🤫) knows more about history than you...🌺
 
-ERWT is a fine-tuned [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) model trained on historical newspapers from the [Heritage Made Digital collection](https://huggingface.co/datasets/davanstrien/hmd-erwt-training).
+ERWT is a fine-tuned [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) model trained on historical newspapers from the [Heritage Made Digital collection](https://huggingface.co/datasets/davanstrien/hmd-erwt-training).
 
  We trained a model based on a combination of text and **temporal metadata** (i.e. year information).
 
-ERWT performs **time-sensitive masked language modelling** and can be used for **date prediction** as well.
+ERWT performs [**time-sensitive masked language modelling**](#historical-language-change-herhis-majesty-%F0%9F%91%91) or [**date prediction**]((#date-prediction-pub-quiz-with-lms-%F0%9F%8D%BB)).
 
-This model is served to you by [Kaspar von Beelen](https://huggingface.co/Kaspar) and [Daniel van Strien](https://huggingface.co/davanstrien), *"Improving AI, one pea at a time"*.
+This model is served by [Kaspar von Beelen](https://huggingface.co/Kaspar) and [Daniel van Strien](https://huggingface.co/davanstrien), *"Improving AI, one pea at a time"*.
+
+If these models happen to be useful, please cite our working paper.
+
+```
+@misc{https://doi.org/10.48550/arxiv.2211.10086,
+  doi = {10.48550/ARXIV.2211.10086},
+  url = {https://arxiv.org/abs/2211.10086},
+  author = {Beelen, Kaspar and van Strien, Daniel},
+  keywords = {Computation and Language (cs.CL), Digital Libraries (cs.DL), FOS: Computer and information sciences, FOS: Computer and information sciences},
+  title = {Metadata Might Make Language Models Better},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {Creative Commons Attribution 4.0 International}}
+```
 
 \*ERWT is dutch for PEA.
 
@@ -50,30 +65,37 @@ This model is served to you by [Kaspar von Beelen](https://huggingface.co/Kaspar
 
 ## Introductory Note: Repent Now. 😇
 
-The ERWT models are trained for **experimental purposes**, please use them with care. 
+The ERWT models are trained for **experimental purposes**. 
 
-You find more detailed information below. Please consult the **limitations** section (seriously, read this section before using the models, **we don't repent in public just for fun**). 
+Please consult the [**limitations**](#limitations-not-all-is-well-%F0%9F%98%AE) section (seriously before using the models. Seriously, read this section, **we don't repent in public just for fun**). 
 
-If you can't get enough of these peas and crave some more. In that case, you can consult our working paper ["Metadata Might Make Language Models Better"](https://arxiv.org/abs/2211.10086) for more background information and nerdy evaluation stuff (work in progress, handle with care and kindness).
+If you can't get enough of these neural peas and crave some more. In that case, you can consult our working paper ["Metadata Might Make Language Models Better"](https://arxiv.org/abs/2211.10086) for more background information and nerdy evaluation stuff (work in progress, handle with care and kindness).
 
 ## Background: MDMA to the rescue. 🙂
 
-ERWT was created using a **M**eta**D**ata **M**asking **A**pproach (or **MDMA** 💊), in which we train a Masked Language Model simultaneously on text and metadata. Our intuition was that incorporating metadata (information that describes a text but and is not part of the content, such as the time/place of publication or the political orientation) may make language models "better", or at least make them more sensitive to historical, political and geographical aspects of language use.
+ERWT was created using a **M**eta**D**ata **M**asking **A**pproach (or **MDMA** 💊), a scenario in which we train a Masked Language Model (MLM) on text and metadata simultaneously. Our intuition was that incorporating metadata (information that *describes* a text but and is not part of the content) may make language models "better", or at least make them more **sensitive** to historical, political and geographical aspects of language use. We mainly use temporal, political and geographical metadata.
 
-ERWT is a [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) model, fine-tuned on a random subsample taken from the Heritage Made Digital newspaper collection. The training data comprises around half a billion words. 
+ERWT is a [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) model, fine-tuned on a random subsample taken from the [Heritage Made Digital newspaper collection]((https://huggingface.co/datasets/davanstrien/hmd-erwt-training)). The training data comprises around half a billion words. 
 
-To unleash the power of MDMA, we adapted to the training routine for the masked language model. When preprocessing the text, we prepended each segment of hundred words with a time stamp (year of publication) and a special `[DATE]` token. 
+To unleash the power of MDMA, we adapted to the training routine mainly by fidgeting with the input data. 
 
+When preprocessing the text, we prepended each segment of hundred words with a time stamp (year of publication) and a special `[DATE]` token. 
 
 
-The snippet below, gives an example taken from the [Londonderry Sentinel](https://www.britishnewspaperarchive.co.uk/viewer/bl/0001480/18700722/014/0002),  
+The snippet below, taken from the [Londonderry Sentinel]:(https://www.britishnewspaperarchive.co.uk/viewer/bl/0001480/18700722/014/0002)...
+```
+Every scrap of intelligence relative to the war between France and Prussia is now read with interest.
+```
+ 
+... would be formatted as:
+      
 ```python
 "1870 [DATE] Every scrap of intelligence relative to the war between France and Prussia is now read with interest."
 ```
 
-These formatted chunks of text are then forwarded to the data collator and eventually the language model.
+These text chunks are then forwarded to the data collator and eventually the language model.
 
-Exposed to both the tokens and (temporal) metadata, the model learns a relation between text and time. When a token is masked, the prepended `year` field is taken into account when predicting hidden words in the text. Vice versa, when the metadata token is hidden, the model aims to predict the year of publication based on the content.
+Exposed to the tokens and (temporal) metadata, the model learns a relation between text and time. When a text token is hidden, the prepended `year` field influences the prediction of the masked words. Vice versa, when the prepended metadata is hidden, the model predicts the year of publication based on the content.
 
 ## Intended Use: LMs as History Machines.
 
@@ -83,13 +105,15 @@ Exposing the model to temporal metadata allows us to investigate **historical la
 
 Let's show how ERWT works with a very concrete example. 
 
-The ERWT models are trained on British newspapers from before 1870 (Why? Long story, don't ask...) and can be used to monitor historical change in this specific context.
+The ERWT models are trained on a handful of British newspapers published between 1800 and 1870. It can be used to monitor historical change in this specific context.
 
-Imagine you are confronted with the following snippet "We received a letter from [MASK] Majesty" and want to predict the correct pronoun (again assuming a British context). 
+Imagine you are confronted with the following snippet: "We received a letter from [MASK] Majesty" and want to predict the correct pronoun for the masked token (again assuming a British context). 
 
-👩‍🏫 **History Intermezzo** Please remember, for most of in the nineteenth century, Queen Victoria ruled Britain. From 1837 to 1901 to be precise. Her nineteenth-century predecessors (George III, George IV and William IV) were all male.
+👩‍🏫 **History Intermezzo** Please remember, for most of the nineteenth century, Queen Victoria ruled Britain, from 1837 to 1901 to be precise. Her nineteenth-century predecessors (George III, George IV and William IV) were all male.
 
-While a standard language model will provide you with one a general prediction, based on what it has observed previously in the training corpus, ERWT models allow you to manipulate to prediction, by anchoring the text in a specific year.
+While a standard language model will provide you with one a general prediction—based on what it has observed during training–ERWT allows you to manipulate to prediction, by anchoring the text in a specific year.
+
+Doing this requires just a few lines of code:
 
 ```python
 from transformers import pipeline
@@ -100,7 +124,7 @@ mask_filler = pipeline("fill-mask",
 mask_filler(f"1820 [DATE] We received a letter from [MASK] Majesty.")
 ```
 
-Returns as most likely prediction:
+This returns "his" as the most likely filler:
 
 ```python
 {'score': 0.8527863025665283,
@@ -115,7 +139,7 @@ However, if we change the date at the start of the sentence to 1850:
 mask_filler(f"1850 [DATE] We received a letter from [MASK] Majesty.")
 ```
 
-Will put most the probability mass on the token "her" and only a little bit on "him".
+ERWT puts most of the probability mass on the token "her" and only a little bit on "his".
 
 ```python
 {'score': 0.8168327212333679,
@@ -126,19 +150,21 @@ Will put most the probability mass on the token "her" and only a little bit on "
 
 You can repeat this experiment for yourself using the example sentences in the **Hosted inference API** at the top right.
 
-Okay, but why is this interesting? 
+Okay, but why is this **interesting**? 
+
+Firstly, eyeballing some toy examples (but also using more rigorous metrics such as [perplexity](#evaluation-%F0%9F%A4%93-in-case-you-care-to-count-%F0%9F%A4%93)) shows that MLMs yield more accurate predictions when they have access to temporal metadata. 
 
-Firstly, eyeballing some toy examples (but also using more rigorous metrics such as perplexity) shows that MLMs can perform more accurate predictions when it has access to temporal metadata. In other words, ERWT's prediction reflects historical language use more accurately. 
+In other words, **ERWT models are better at capturing historical context.**
 
-Secondly, MDMA may reduce biases induced by imbalances in the training data (or at least give us more of a handle on this problem). Admittedly, we have to prove this more formally, but some experiments at least hint in this direction. The data used for training is highly biased towards the Victorian age and a standard language model trained on this corpus will predict "her" for ```"[MASK] Majesty"```. 
+Secondly, MDMA may **reduce biases** that arise from imbalanced training data (or at least give us more of a handle on this problem). Admittedly, we have to prove this more formally, but some experiments at least hint in this direction.
 
 ### Date Prediction: Pub Quiz with LMs 🍻
 
-Another feature of the ERWT model series is date prediction. Remember that during training the temporal metadata token is often masked. In this case, the model effectively learns to situate documents in time based on the tokens they contain.
+Another feature of ERWT is **date prediction**. Remember that during training the temporal metadata token is regularly masked. In this case, the model effectively learns to situate documents in time based on the tokens in a text.
 
-By masking the year token, ERWT guesses the document's year of publication.
+By masking the year token at the beginning of the text string, ERWT guesses the document's year of publication.
 
-👩‍🏫 **History Intermezzo** To unite the German states (there were plenty!), Prussia fought a number of wars with its neighbours in the second half of the nineteenth century. It invaded Denmark in 1864 (the second of the Schleswig Wars) and France in 1870 (the Franco-Prussian war). 
+👩‍🏫 **History Intermezzo** To unite the German states (there used to be [plenty](https://www.britannica.com/topic/German-Confederation)!), Prussia fought a number of wars with its neighbours in the second half of the nineteenth century. It invaded Denmark in 1864 (the second of the Schleswig Wars) and France in 1870 (the Franco-Prussian war). 
 
 Reusing to code above, we can time-stamp documents by masking the year. For example, the line of python code below:
 
@@ -158,22 +184,23 @@ Outputs as most likely filler:
 ```
 
 
-The prediction "1864" makes sense as this was indeed the year of Prussian troops (with some help of their Austrian friends) crossed the border into Schleswig, then part of the Kingdom of Denmark.
+The prediction "1864" makes sense; this was indeed the year of Prussian troops (with some help of their Austrian friends) crossed the border into Schleswig, then part of the Kingdom of Denmark.
 
-A few years later, in 1870, Prussia aimed artillery southwards and invaded France. 
+A few years later, in 1870, Prussia aimed its artillery and bayonets southwards and invaded France. 
 
 ```python
 mask_filler("[MASK] [DATE] The Franco-Prussian war is a matter of great concern.")
 ```
 
-ERWT clearly learned a lot about the history of German unification by ploughing through a plethora of nineteenth-century newspaper articles: it correctly returns "1870" as the predicted year.
+ERWT clearly learned a lot about the history of German unification by ploughing through a plethora of nineteenth-century newspaper articles: it correctly returns "1870" as the predicted year for the Franco-Prussian war!
 
 Again, we have to ask: Who cares? Wikipedia can tell us pretty much the same. More importantly, don't we already have timestamps for newspaper data?
 
-In both cases, our answers would be "yes, but...". ERWT's time-stamping powers have little instrumental use and won't make us rich (but donations are welcome of course 🤑). Nonetheless, we believe date prediction has value for research purposes. We can use ERWT for "fictitious" prediction, i.e. as a diagnostic tool. 
+In both cases, our answer sounds "yes, but...". ERWT's time-stamping powers have little instrumental use and won't make us rich (but donations are welcome of course 🤑). Nonetheless, we believe date prediction has value for research purposes. We can use ERWT for "fictitious" prediction, i.e. as a diagnostic tool. 
 
-Firstly, we used date prediction for evaluation purposes, to measure which training routine produces models 
-Secondly, we could use it as an analytical tool, to study how temporal variation **within** text documents and further scrutinise which features drive the time prediction (it goes without saying that the same applies to other metadata fields, but for example predicting political orientation).
+Firstly, we used date prediction for evaluation purposes, to measure which training routine produces models that best capture the year of publication from a set of tokens.
+
+Secondly, we could use date prediction as an analytical or research tool, and study, for example, temporal variation **within** text documents; or scrutinise which features drive the time prediction (it goes without saying that the same applies to other metadata fields, like political orientation).
 
 ## Limitations: Not all is well 😮.
 
@@ -181,13 +208,15 @@ The ERWT series were trained for evaluation purposes and therefore carry some cr
 
 ### Training Data
 
-Many of the limitations are a direct result of the data. ERWT models are trained on a rather small subsample of nineteenth-century British newspapers, and its predictions have to be understood in this context (remember, Her Majesty?). Moreover, the corpus has a strong Metropolitan and liberal bias (see the section on Data Description for more information). 
+Many of the limitations are a direct result of the training data. ERWT models are trained on a rather small subsample of nineteenth-century **British newspapers**, and its predictions have to be understood in this context (remember, "Her Majesty?"). The corpus has a strong **Metropolitan and liberal bias** (see the section on Data Description for more information). 
+
+The training data spans from **1800 to 1870**. If your research interest is outside of this period, it's unlikely that ERWT will be of much use. Don't ask the poor model to predict when the Second World War happened. ERWT can be smart (at times) but it doesn't have the power of fortune-telling. At least not yet...
 
-The training data ranges from 1800 to 1870. If your period of interest is outside of this historical period, it's unlikely that ERWT will be of much use. Don't ask the poor model to predict when the Second World War happened. ERWT can be smart (at times) but it doesn't have the power of fortune-telling. At least not yet...
+Furthermore, historical models tend to reflect past (and present?) stereotypes and prejudices. We strongly advise against using these models outside of a research context. The predictions are likely to exhibit harmful biases, they should be investigated critically and understood within the context of nineteenth-century British cultural history.
 
-Historically models tend to reflect past (and present?) stereotypes and prejudices. We strongly advise against using these models outside of the context of historical research. The predictions are likely to exhibit harmful biases and should be investigated critically and understood within the context of nineteenth-century British cultural history.
+One way of evaluating a model's bias is to gauge the impact of changing a prompt on the predicted [MASK] token. Often a comparison is made between the predictions given for 'The **man** worked as a [MASK]' to 'The **woman** worked as a [MASK]'. 
 
-One way of evaluating a model's bias is to evaluate the impact of making a change to a prompt and evaluating the impact on the predicted [MASK] token. Often a comparison is made between the predictions given for the prompt 'The **man** worked as a [MASK]' compared to the prompt 'The **woman** worked as a [MASK]'. An example of the output for this model:
+An example of the output for this model:
 
 ```
 1810 [DATE] The man worked as a [MASK].
@@ -241,19 +270,21 @@ Produces the following three top predicted mask tokens
 ]
 ```
 
-Often this promoting prompt evaluation is done to assess the bias in *contemporary* language models. Often these biases reflect the training data used to train the model. In the case of historic language models, the bias exhibited by a model *may* be a valuable research tool in assessing (at scale) the use of language over time. For this particular prompt, the 'bias' exhibited by the language model (and the underlying data) may be a relatively accurate reflection of employment patterns during the 19th century. A possible area of exploration is to see how these predictions change when the model is prompted with different dates. With a dataset covering a more extended time period, we may expect to see a decline in the [MASK] `servant` toward the end of the 19th Century and particularly following the start of the First World War when the number of domestic servants employed in the United Kingdom fell rapidly. 
+Mostly, prompt evaluation is done to assess the bias in *contemporary* language models. In the case of historic language models, the bias exhibited by a model *may* be a valuable research tool in assessing (at scale) language use over time, and the stereotypes and prejudices encoded in text corpora. 
+
+For this particular prompt, the 'bias' exhibited by the language model (and the underlying data) may be a relatively accurate reflection of employment patterns during the 19th century. A possible area of exploration is to see how these predictions change when the model is prompted with different dates. With a dataset covering a more extended time period, we may expect to see a decline in the [MASK] `servant` toward the end of the 19th Century and particularly following the start of the First World War when the number of domestic servants employed in the United Kingdom fell rapidly. 
 
 ### Training Routine
 
-We created this model as part of a wider experiment, which attempted to establish best practices for training models with metadata. An overview of all the models is available on our [GitHub](https://github.com/Living-with-machines/ERWT/) page.
+We created various ERWT models as part of a wider experiment that aimed to establish best practices and guidelines for training models with metadata. An overview of all the models is available on our [GitHub](https://github.com/Living-with-machines/ERWT/) page.
 
 To reduce training time, we based our experiments on a random subsample of the HMD corpus, consisting of half a billion tokens. 
-Furthermore, we only trained the models for one epoch, which implies they are most likely undertrained at the moment. 
+Furthermore, we only trained the models for one epoch, which implies they are most likely **undertrained** at the moment. 
 
-We were mainly interested in the **relative** performance of the different ERWT models. We did, however, compared ERWT with [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) in our evaluation experiments, and of course, our tiny LM peas 
+We were mainly interested in the **relative** performance of the different ERWT models. We did, however, compared ERWT with [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) in our evaluation experiments. And, of course, our tiny LM peas 
 did much better. 🎉🥳 
 
-Want to know how much, then read [our working paper](https://arxiv.org/abs/2211.10086)!
+Want to know the details—Oh, critical reader!—then consult and cite [our working paper](https://arxiv.org/abs/2211.10086)!
 
 ## Data Description
 
@@ -286,12 +317,11 @@ Temporally, most of the articles date from the second half of the nineteenth cen
 
 ## Evaluation: 🤓 In case you care to count 🤓
 
-Our article ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing) comprises quite an extensive evaluation of all the language models created with MDMA. For details, we recommend you read and cite the current [working paper](https://arxiv.org/abs/2211.10086). 
-
-The table below shows the [pseudo-perplexity](https://arxiv.org/abs/1910.14659) scores for different models using text documents of 64 and 128 tokens.
+Our article ["Metadata Might Make Language Models Better"](https://arxiv.org/abs/2211.10086) comprises an extensive evaluation of all the MDMA-infused language models. 
 
-In general, [ERWT-year-masked-25](https://huggingface.co/Livingwithmachines/erwt-year-masked-25) turned out to yield the most competitive scores across different tasks and generally recommend you use this model. 
+The table below shows the [pseudo-perplexity](https://arxiv.org/abs/1910.14659) scores for different models based on text documents of 64 and 128 tokens.
 
+In general, [ERWT-year-masked-25](https://huggingface.co/Livingwithmachines/erwt-year-masked-25) turned out to yield the most competitive scores across different tasks and we generally recommend you use this model. 
 
 
 | text length         | 64             |        | 128            |        |
@@ -311,3 +341,4 @@ In general, [ERWT-year-masked-25](https://huggingface.co/Livingwithmachines/erwt
 
 Questions? Feedback? Please leave a message!
 
+