first push of pretrain model

.gitattributes

@@ -9,14 +9,10 @@
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,40 @@
 ---
-license: apache-2.0
+tags: 
+- autotrain
+- pre-trained
+- finbert
+- fill-mask
+language: unk
+widget:
+- text: Tesla remains one of the highest [MASK] stocks on the market. Meanwhile, Aurora Innovation is a pre-revenue upstart that shows promise.
+- text: Asian stocks [MASK] from a one-year low on Wednesday as U.S. share futures and oil recovered from the previous day's selloff, but uncertainty over the impact of the Omicron
+- text: U.S. stocks were set to rise on Monday, led by [MASK] in Apple which neared $3 trillion in market capitalization, while investors braced for a Federal Reserve meeting later this week.
 ---
+
+`FinBERT` is a BERT model pre-trained on financial communication text. The purpose is to enhance financial NLP research and practice. 
+
+### Pre-training
+It is trained on the following three financial communication corpus. The total corpora size is 4.9B tokens.
+
+- Corporate Reports 10-K & 10-Q: 2.5B tokens
+- Earnings Call Transcripts: 1.3B tokens
+- Analyst Reports: 1.1B tokens
+- Demo.org Proprietary Reports
+- Additional purchased data from Factset
+
+The entire training is done using an **NVIDIA DGX-1** machine. The server has 4 Tesla P100 GPUs, providing a total of 128 GB of GPU memory. This machine enables us to train the BERT models using a batch size of 128. We utilize Horovord framework for multi-GPU training. Overall, the total time taken to perform pretraining for one model is approximately **2 days**.
+
+
+More details on `FinBERT`'s pre-training process can be found at: https://arxiv.org/abs/2006.08097
+
+`FinBERT` can be further fine-tuned on downstream tasks. Specifically, we have fine-tuned `FinBERT` on an analyst sentiment classification task, and the fine-tuned model is shared at [https://huggingface.co/demo-org/auditor_review_model](https://huggingface.co/demo-org/auditor_review_model)
+
+### Usage
+Load the model directly from Transformers:
+```
+from transformers import AutoModelForMaskedLM
+model = AutoModelForMaskedLM.from_pretrained("demo-org/finbert-pretrain", use_auth_token=True)
+```
+
+### Questions
+Please contact the Data Science COE if you have more questions about this pre-trained model

config.json

@@ -0,0 +1,16 @@
+{
+"architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "max_position_embeddings": 512,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "type_vocab_size": 2,
+  "vocab_size": 30873
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46dd5b5cbf7141b0c5d882516243abf71cfa4a27e57023c43290d655fccfb48c
+size 441551705