anonymousparrot01 commited on
Commit
6c12eac
1 Parent(s): 843d613

initial upload

Browse files
Files changed (4) hide show
  1. README.md +64 -0
  2. config.json +1 -0
  3. pytorch_model.bin +3 -0
  4. vocab.txt +0 -0
README.md CHANGED
@@ -1,3 +1,67 @@
1
  ---
 
 
 
 
 
2
  license: cc-by-4.0
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language: en
3
+ tags:
4
+ - bert
5
+ - business
6
+ - finance
7
  license: cc-by-4.0
8
+ datasets:
9
+ - CompanyWeb
10
+ - MD&A
11
+ - S2ORC
12
  ---
13
+
14
+ # BusinessBERT
15
+
16
+ An industry-sensitive language model for business applications pretrained on business communication corpora. The model incorporates industry classification (IC) as a pretraining objective besides masked language modeling (MLM).
17
+
18
+ It was introduced in
19
+ [this paper]() and released in
20
+ [this repository]().
21
+
22
+ ## Model description
23
+
24
+ We introduce BusinessBERT, an industry-sensitive language model for business applications. The advantage of the model is the training approach focused on incorporating industry information relevant for business related natural language processing (NLP) tasks.
25
+ We compile three large-scale textual corpora consisting of annual disclosures, company website content and scientific literature representing business communication. In total, the corpora include 2.23 billion token.
26
+ BusinessBERT builds upon the bidirectional encoder representations from transformer architecture (BERT) and embeds industry information during pretraining in two ways: (1) The business communication corpora contain a variety of industry-specific terminology; (2) We employ industry classification (IC) as an additional pretraining objective for text documents originating from companies.
27
+
28
+ ## Intended uses & limitations
29
+
30
+ The model is intended to be fine-tuned on business related NLP tasks, i.e. sequence classification, named entity recognition, sentiment analysis or question answering.
31
+
32
+ ### How to use
33
+
34
+ [PLACEHOLDER]
35
+
36
+ ### Limitations and bias
37
+
38
+ [PLACEHOLDER]
39
+
40
+ ## Training data
41
+
42
+ - [CompanyWeb](https://huggingface.co/datasets/anonymousparrot01/CompanyWeb): 0.77 billion token, 3.5 GB raw text file
43
+ - [MD&A Disclosures](https://data.caltech.edu/records/1249): 1.06 billion token, 5.1 GB raw text file
44
+ - [Semantic Scholar Open Research Corpus](https://api.semanticscholar.org/corpus): 0.40 billion token, 1.9 GB raw text file
45
+
46
+ ## Evaluation results
47
+
48
+ [PLACEHOLDER]
49
+
50
+ <!-- When fine-tuned on downstream tasks, this model achieves the following results:
51
+
52
+ Glue test results:
53
+
54
+ | Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | Average |
55
+ |:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:|
56
+ | | 84.6/83.4 | 71.2 | 90.5 | 93.5 | 52.1 | 85.8 | 88.9 | 66.4 | 79.6 | -->
57
+
58
+
59
+ ### BibTeX entry and citation info
60
+
61
+ ```bibtex
62
+ @misc{title_year,
63
+ title={TITLE},
64
+ author={AUTHORS},
65
+ year={YEAR},
66
+ }
67
+ ```
config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model_type":"bert", "use_cache": true, "gradient_checkpointing": false, "pad_token_id": 0, "attention_probs_dropout_prob": 0.1, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "max_position_embeddings": 512, "num_attention_heads": 12, "num_hidden_layers": 12, "type_vocab_size": 2, "vocab_size": 29389, "transformers_version": "4.6.0.dev0"}
pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:17a765a45afe49d6e32d02e2c03c04b37f098cf047a83c95e381287b3116228a
3
+ size 437028903
vocab.txt ADDED
The diff for this file is too large to render. See raw diff