berendg commited on
Commit
3bc69b5
1 Parent(s): 4cbb30a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -1
README.md CHANGED
@@ -119,5 +119,18 @@ The pre-training objective is introduced in the ACL Findings paper _Masked Laten
119
 
120
  **BibTeX:**
121
 
122
-
 
 
 
 
 
 
 
 
 
 
 
 
 
123
 
 
119
 
120
  **BibTeX:**
121
 
122
+ ```
123
+ @inproceedings{berend-2023-masked,
124
+ title = "Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling",
125
+ author = "Berend, G{\'a}bor",
126
+ booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
127
+ month = jul,
128
+ year = "2023",
129
+ address = "Toronto, Canada",
130
+ publisher = "Association for Computational Linguistics",
131
+ url = "https://aclanthology.org/2023.findings-acl.876",
132
+ pages = "13949--13962",
133
+ abstract = "In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where the objective is altered from the reconstruction of the exact identity of randomly selected masked subwords to the prediction of their latent semantic properties. We coin the proposed pre-training technique masked latent semantic modeling (MLSM for short). In order to make the contextualized determination of the latent semantic properties of the masked subwords possible, we rely on an unsupervised technique which uses sparse coding. Our experimental results reveal that the fine-tuned performance of those models that we pre-trained via MLSM is consistently and significantly better compared to the use of vanilla MLM pretraining and other strong baselines.",
134
+ }
135
+ ```
136