MoritzLaurer

/

mDeBERTa-v3-base-xnli-multilingual-nli-2mil7

like 210

you can look at this tutorial and especially notebook 4: https://github.com/MoritzLaurer/summer-school-transformers-2023/tree/main

Please check the LiqFit, it allows with few examples to achieve good performance in text classification.

You can choose any model you want and different loss functions, such as focal loss and fine-tune model with transformers Trainer:

from liqfit.modeling import LiqFitModel
from liqfit.losses import FocalLoss
from liqfit.collators import NLICollator
from transformers import TrainingArguments, Trainer

backbone_model = AutoModelForSequenceClassification.from_pretrained('MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7')

loss_func = FocalLoss(multi_target=True)

model = LiqFitModel(backbone_model.config, backbone_model, loss_func=loss_func)

data_collator = NLICollator(tokenizer, max_length=128, padding=True, truncation=True)

training_args = TrainingArguments(
    output_dir='comprehendo',
    learning_rate=3e-5,
    per_device_train_batch_size=3,
    per_device_eval_batch_size=3,
    num_train_epochs=9,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_steps = 5000,
    save_total_limit=3,
    remove_unused_columns=False,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=nli_train_dataset,
    eval_dataset=nli_test_dataset,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

@Ihor do you have a link to the source repo?

@MoritzLaurer , sure, I have. Here it is https://github.com/Knowledgator/LiqFit. The link in the previous comment was fixed as well.