initial upload

README.md

@@ -1,3 +1,67 @@
 ---
+language: en
+tags:
+- bert
+- business
+- finance
 license: cc-by-4.0
+datasets:
+- CompanyWeb
+- MD&A
+- S2ORC
 ---
+
+# BusinessBERT
+
+An industry-sensitive language model for business applications pretrained on business communication corpora. The model incorporates industry classification (IC) as a pretraining objective besides masked language modeling (MLM).
+
+It was introduced in
+[this paper]() and released in
+[this repository]().
+
+## Model description
+
+We introduce BusinessBERT, an industry-sensitive language model for business applications. The advantage of the model is the training approach focused on incorporating industry information relevant for business related natural language processing (NLP) tasks. 
+We compile three large-scale textual corpora consisting of annual disclosures, company website content and scientific literature representing business communication. In total, the corpora include 2.23 billion token.
+BusinessBERT builds upon the bidirectional encoder representations from transformer architecture (BERT) and embeds industry information during pretraining in two ways: (1) The business communication corpora contain a variety of industry-specific terminology; (2) We employ industry classification (IC) as an additional pretraining objective for text documents originating from companies.
+
+## Intended uses & limitations
+
+The model is intended to be fine-tuned on business related NLP tasks, i.e. sequence classification, named entity recognition, sentiment analysis or question answering.
+
+### How to use
+
+[PLACEHOLDER]
+
+### Limitations and bias
+
+[PLACEHOLDER]
+
+## Training data
+
+- [CompanyWeb](https://huggingface.co/datasets/anonymousparrot01/CompanyWeb): 0.77 billion token, 3.5 GB raw text file
+- [MD&A Disclosures](https://data.caltech.edu/records/1249): 1.06 billion token, 5.1 GB raw text file
+- [Semantic Scholar Open Research Corpus](https://api.semanticscholar.org/corpus): 0.40 billion token, 1.9 GB raw text file
+
+## Evaluation results
+
+[PLACEHOLDER]
+
+<!-- When fine-tuned on downstream tasks, this model achieves the following results:
+
+Glue test results:
+
+| Task | MNLI-(m/mm) | QQP  | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE  | Average |
+|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
+|      | 84.6/83.4   | 71.2 | 90.5 | 93.5  | 52.1 | 85.8  | 88.9 | 66.4 | 79.6    | -->
+
+
+### BibTeX entry and citation info
+
+```bibtex
+@misc{title_year,
+      title={TITLE},
+      author={AUTHORS},
+      year={YEAR},
+}
+```

config.json

@@ -0,0 +1 @@
+{"model_type":"bert", "use_cache": true, "gradient_checkpointing": false, "pad_token_id": 0, "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "type_vocab_size": 2, "vocab_size": 29389, "transformers_version": "4.6.0.dev0"}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17a765a45afe49d6e32d02e2c03c04b37f098cf047a83c95e381287b3116228a
+size 437028903