datasets:
- >-
google-research-datasets/C4_200M-synthetic-dataset-for-grammatical-error-correction
- common-crawl
language:
- en
license: mit
widget:
- text: Do you even know why I always need changed our checking account number.
example_title: Example 1
- text: Ironman and Captain America is going out.
example_title: Example 2
- text: We all eat fish and then made dessert.
example_title: Example 3
- text: We have our Dinner yesterday.
example_title: Example 4
inference:
parameters:
max_length: 256
num_beams: 3
no_repeat_ngram_size: 2
repetition_penalty: 2.5
temperature: 0.7
do_sample: true
tags:
- context-correction
- error-correction
T5 Context Corrector (base-sized)
t5-context-corrector model is a fine-tuned T5 model on the Synthetic GEC and filtered CommonCrawl data in English Language.
The Base Model(T5) is Pre-trained on C4(Colossal Clean Crawled Corpus) dataset and works with numerous downstrem tasks.
Our Model specifically is fine-tuned on a single downstream task of context correction on the above mentioned two datasets.
Model description
This Model has the same architecture as its base model, thus having 220 Million Parameters while consisting of 12 encoder blocks and 12 decoder blocks with an input embedding size of 32128. Please refer to this link to know more about the model details.
Intended Use & Limitations
As the model is intented to correct the context of the given sentence, all you have to do is pass the non-contextually correct sentence and get the corrected response back.
Based on Multiple experiments performed as part of the training, we observe that the model works best when the the total number of tokens in the input is less than 256.
So, if you have a long paragraph that needs to be context corrected, we suggest to first sentence tokenize the paragraph and run the context corrector for each sentence separately to obtain best results.
Note that the model is primarily trained on general publicly available corpus, so it maynot work well for Medical Contexts.
Usage
You can use this model directly with a pipeline for Text to Text Generation:
from transformers import pipeline
ctx_corr = pipeline("text-generation", model='DeathReaper0965/t5-context-corrector')
ctx_corr("Do you even know why I always need changed our checking account number")
###########OUTPUT###########
# [{'generated_text': 'Do you even know why I always need to change our checking account number?'}]
Or you can also use the model to get the features for a given text:
from nltk import sent_tokenize
from transformers import T5ForConditionalGeneration, T5Tokenizer
# Load model and tokenizer
cc_tokenizer = T5Tokenizer.from_pretrained("DeathReaper0965/t5-context-corrector")
cc_model = T5ForConditionalGeneration.from_pretrained("DeathReaper0965/t5-context-corrector")
# Utility function to correct context
def correct_context(input_text, temperature=0.5):
# tokenize
batch = cc_tokenizer(input_text,
truncation=True,
padding='max_length',
max_length=256,
return_tensors="pt")
# forward pass
results = cc_model.generate(**batch,
max_length=256,
num_beams=3,
no_repeat_ngram_size=2,
repetition_penalty=2.5,
temperature=temperature,
do_sample=True)
return results
# Utility function to split the paragraph into multiple sentences
def split_and_correct_context(sent):
sents = sent_tokenize(sent)
final_sents = cc_tokenizer.batch_decode(correct_context(sents),
clean_up_tokenization_spaces=True,
skip_special_tokens=True)
final_sents = " ".join([final_sents[i].strip() for i in range(len(final_sents))])
return final_sents
split_and_correct_context("Do you even know why I always need changed our checking account number. Because of the securty purpos.")
###########OUTPUT###########
# 'Do you even know why I always need to change our checking account number? Because of the security purpose.'