Kaspar commited on
Commit
93c18d0
1 Parent(s): a288ca4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -6
README.md CHANGED
@@ -20,13 +20,12 @@ widget:
20
  A fine-tuned [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) model trained on historical newspapers from the [Heritage Made Digital collection](https://huggingface.co/datasets/davanstrien/hmd-erwt-training) with temporal metadata.
21
 
22
 
23
- **Note**: This model was trained for experimental purposes, please use it with care.
24
 
25
 
 
26
 
27
- You find more detailled information below and in our working paper ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing).
28
-
29
- ## Background
30
 
31
  ERWT was created using a MetaData Masking Approach (or MDMA 💊), in which we train a Masked Language Model simultaneously on text and metadata. Our intuition was that incorporating information that is not explicitly present in the text—such as the time of publication or the political leaning of the author—may make language models "better" in the sense of being more sensitive to historical and political aspects of language use.
32
 
@@ -37,7 +36,9 @@ For example, we would format a snippet of text taken from the [Londonderry Senti
37
  "1870 [DATE] Every scrap of intelligence relative to the war between France and Prussia is now read with interest."
38
  ```
39
 
40
- ... and then provide this sentence with prepended temporal metadata to MLM. This effectively
41
-
42
 
43
  ## Intended uses & limitations
 
 
 
 
20
  A fine-tuned [`distilbert-base-cased`](https://huggingface.co/distilbert-base-cased) model trained on historical newspapers from the [Heritage Made Digital collection](https://huggingface.co/datasets/davanstrien/hmd-erwt-training) with temporal metadata.
21
 
22
 
23
+ **Warning**: This model was trained for **experimental purposes**, please use it with care.
24
 
25
 
26
+ You find more detailed information below and in our working paper ["Metadata Might Make Language Models Better"](https://drive.google.com/file/d/1Xp21KENzIeEqFpKvO85FkHynC0PNwBn7/view?usp=sharing).
27
 
28
+ ## Background and Data
 
 
29
 
30
  ERWT was created using a MetaData Masking Approach (or MDMA 💊), in which we train a Masked Language Model simultaneously on text and metadata. Our intuition was that incorporating information that is not explicitly present in the text—such as the time of publication or the political leaning of the author—may make language models "better" in the sense of being more sensitive to historical and political aspects of language use.
31
 
 
36
  "1870 [DATE] Every scrap of intelligence relative to the war between France and Prussia is now read with interest."
37
  ```
38
 
39
+ ... and then provide this sentence with prepended temporal metadata to MLM.
 
40
 
41
  ## Intended uses & limitations
42
+
43
+ Exposing the model to extra-textual information allows us to use **language change** and **date prediction**
44
+