Package: toeicbert
TOEIC-BERT76% Correct rate with ONLY Pre-Trained BERT model in TOEIC!!This is project as topic:TOEIC(Test of English for International Communication) problem solving using pytorch-pretrained-BERT model.The reason why I used huggingface'spytorch-pretrained-BERT modelis for pre-training or to do fine-tune more easily.I've solved the only blank problem, not the whole problem.There are two types of blank issues:Selecting Correct Grammar Type.Q) The teacher had me _________ scales several times a day.
  1. play (Answer)
  2. to play
  3. played
  4. playingSelecting Correct Vocabulary Type.Q) The wet weather _________ her from going shopping.
  1. interrupted
  2. obstructed
  3. impeded
  4. discouraged (Answer)Why BERT?In pretrained BERT, It contains contextual information. So It can find more contextual or grammatical sentences, not clear, a little bit. I was inspired by grammar checker fromblog post.Can We Use BERT as a Language Model to Assign a Score to a Sentence?BERT uses a bidirectional encoder to encapsulate a sentence from left to right and from right to left. Thus, it learns two representations of each word-one from left to right and one from right to left-and then concatenates them for many downstream tasks.EvaluationI had evaluated with onlypretrained BERT model(not fine-tuning)to check grammatical or lexical error. Above mathematical expression,Xis a question sentence. andnis number of questions :{a, b, c, d}.Csubset means answer candidate tokens :Cofwarrantyis['warrant', '##y'].Vmeans total Vocabulary.There's a problem with more than one token. I solved this problem by getting the average value of each tensor. ex)is being formedas['is', 'being', 'formed']Then, we find argmax inL_n(T_n).predictions=model(question_tensors,segment_tensors)# predictions : [batch_size, sequence_length, vocab_size]predictions_candidates=predictions[0,masked_index,candidate_ids].mean()Result of Evaluation.Fantastic result withonly pretrained BERT modelbert-base-uncased: 12-layer, 768-hidden, 12-heads, 110M parametersbert-large-uncased: 24-layer, 1024-hidden, 16-heads, 340M parametersbert-base-cased: 12-layer, 768-hidden, 12-heads , 110M parametersbert-large-cased: 24-layer, 1024-hidden, 16-heads, 340M parametersTotal 7067 datasets: make non-deterministic withmodel.eval()bert-base-uncasedbert-base-casedbert-large-uncasedbert-large-casedCorrect Num5192539853215148Percent73.46%76.38%75.29%72.84Quick Start with Python pip Package.Start with pip$pipinstalltoeicbertRun & Option$pythontoeicbert-mbert-base-uncased-ftest.json-m, --model: bert-model name in huggingface's pytorch-pretrained-BERT :bert-base-uncased,bert-large-uncased,bert-base-cased,bert-large-cased.-f, --file: json file to evalution, see json format,test.json.key(question, 1, 2, 3, 4)  is required options, but answer not._in question will be replaced to[MASK]{"1":{"question":"The teacher had me _ scales several times a day.","answer":"play","1":"play","2":"to play","3":"played","4":"playing"},"2":{"question":"The teacher had me _ scales several times a day.","1":"play","2":"to play","3":"played","4":"playing"}}AuthorTae Hwan Jung(Jeff Jung) @graykode, Kyung Hee Univ CE(Undergraduate).Author Email :nlkey2022@gmail.comThanks for Hwan Suk Gang(Kyung Hee Univ.) for collecting Dataset(7114datasets)