|
--- |
|
license: apache-2.0 |
|
base_model: distilbert-base-uncased |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- accuracy |
|
model-index: |
|
- name: distilbert-base-uncased-finetuned-clinc |
|
results: [] |
|
datasets: |
|
- clinc_oos |
|
library_name: transformers |
|
pipeline_tag: text-classification |
|
--- |
|
|
|
# Transformer Efficiency and Knowledge Distillation |
|
|
|
This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset. |
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.7872 |
|
- Accuracy: 0.9206 |
|
|
|
## Model description |
|
|
|
This setup involves benchmarking the performance of a fine-tuned BERT model (transformersbook/bert-base-uncased-finetuned-clinc) and applying knowledge distillation to train a smaller DistilBERT model. The BERT model is used for text classification tasks, and its efficiency is evaluated in terms of accuracy, model size, and latency. The DistilBERT model is trained to mimic the BERT model's performance while being more efficient. |
|
|
|
## Intended uses & limitations |
|
|
|
### Intended uses: |
|
|
|
Evaluating the performance efficiency of transformer models. |
|
Applying knowledge distillation to create smaller and faster models for text classification. |
|
|
|
### Limitations: |
|
|
|
The benchmark results are specific to the dataset used (CLINC150) and may not generalize to other datasets. |
|
Knowledge distillation relies on the quality and performance of the teacher model. |
|
|
|
## Training and evaluation data |
|
|
|
The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits. |
|
|
|
## Training procedure |
|
|
|
### Training and evaluation data |
|
|
|
The BERT model is fine-tuned on the CLINC150 dataset, which consists of labeled examples for intent classification. The dataset includes training, validation, and test splits. |
|
|
|
### Performance Benchmark |
|
|
|
The performance of the BERT model is evaluated using the PerformanceBenchmark class, which measures accuracy, model size, and latency. |
|
|
|
### Accuracy |
|
|
|
The model's accuracy is computed on the test set of the CLINC150 dataset. |
|
accuracy_score = load_metric("accuracy") |
|
|
|
### Model Size |
|
|
|
The size of the model is computed by saving its state dictionary to disk and measuring the file size in megabytes. |
|
|
|
def compute_size(self): |
|
state_dict = self.pipeline.model.state_dict() |
|
tmp_path = Path("model.pt") |
|
torch.save(state_dict, tmp_path) |
|
size_mb = Path(tmp_path).stat().st_size / (1024 * 1024) |
|
tmp_path.unlink() |
|
return {"size_mb": size_mb} |
|
|
|
### Latency |
|
|
|
The average latency per query is measured over a sample of 100 queries. |
|
|
|
def time_pipeline(self): |
|
latencies = [] |
|
for example in self.dataset[:100]: |
|
start_time = perf_counter() |
|
_ = self.pipeline(example) |
|
latency = perf_counter() - start_time |
|
latencies.append(latency) |
|
time_avg_ms = 1000 * np.mean(latencies) |
|
time_std_ms = 1000 * np.std(latencies) |
|
return {"time_avg_ms": time_avg_ms, "time_std_ms": time_std_ms} |
|
|
|
### Knowledge Distillation |
|
|
|
Knowledge distillation is used to train a smaller DistilBERT model using the predictions of the fine-tuned BERT model as soft labels. |
|
|
|
### Distillation Process |
|
|
|
Teacher Model: transformersbook/bert-base-uncased-finetuned-clinc |
|
Student Model: distilbert-base-uncased |
|
The distillation process involves computing a weighted average of the cross-entropy loss with the ground truth labels and the Kullback-Leibler divergence between the teacher and student model predictions. |
|
|
|
class DistillationTrainer(Trainer): |
|
def compute_loss(self, model, inputs, return_outputs=False): |
|
outputs_stu = model(**inputs) |
|
loss_ce = outputs_stu.loss |
|
logits_stu = outputs_stu.logits |
|
with torch.no_grad(): |
|
outputs_tea = self.teacher(**inputs) |
|
logits_tea = outputs_tea.logits |
|
loss_fct = nn.KLDivLoss(reduction="batchmean") |
|
loss_kd = self.args.temperature ** 2 * loss_fct( |
|
F.log_softmax(logits_stu / self.args.temperature, dim=-1), |
|
F.softmax(logits_tea / self.args.temperature, dim=-1) |
|
) |
|
loss = self.args.alpha * loss_ce + (1. - self.args.alpha) * loss_kd |
|
return (loss, outputs_stu) if return_outputs else loss |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 48 |
|
- eval_batch_size: 48 |
|
- seed: 42 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- num_epochs: 5 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------:| |
|
| No log | 1.0 | 318 | 3.2931 | 0.7255 | |
|
| 3.8009 | 2.0 | 636 | 1.8849 | 0.8526 | |
|
| 3.8009 | 3.0 | 954 | 1.1702 | 0.8897 | |
|
| 1.7128 | 4.0 | 1272 | 0.8717 | 0.9145 | |
|
| 0.9206 | 5.0 | 1590 | 0.7872 | 0.9206 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.41.1 |
|
- Pytorch 2.3.0+cu121 |
|
- Datasets 2.19.1 |
|
- Tokenizers 0.19.1 |
