Richard37546/car-safety-complaint-lora

Car Safety Complaint Tagging LoRA

LoRA adapter for multi-issue hierarchical tagging on real-world Chinese car safety complaint texts.

This adapter is trained on top of Qwen2.5-3B-Instruct and is designed for a closed-set structured extraction task: given a complaint text, the model outputs all safety-related issue items as hierarchical labels.

---

1. Task

Input

A Chinese car safety complaint text containing:

• brand
• short issue summary
• detailed complaint description

Output

Structured JSON with one or more issue items, where each item contains:

• level1
• level1_5
• level2

Example:

{
  "problems": [
    {
      "level1": "制动系统",
      "level1_5": "制动系统机械部件",
      "level2": "制动失灵"
    }
  ]
}

---

2. What makes this model different

This adapter is not a plain free-generation LoRA model.

The final best-performing setup uses:

• Hybrid Retrieval
• top15 candidate labels
• candidate-guided LoRA SFT

In other words, the model is trained to perform structured extraction with retrieved candidate labels provided in the input, instead of relying on unconstrained free generation.

This significantly improves:

• label standardization
• hierarchical consistency
• exact match performance
• output stability

---

3. Best Setting

Final best pipeline:

Hybrid Retrieval + top15 candidate labels + candidate-guided LoRA SFT

Key metrics of the final best version:

• Count Accuracy: 0.74359
• F1_level2: 0.501718
• Weighted Tag Score: 0.586254
• Hierarchical Consistency: 0.890313
• Problem Exact Match: 0.440252
• Complaint Exact Match: 0.42735
• bad_pred_count: 0

---

4. Training Data

The training data comes from real-world Chinese car safety complaint records that were manually cleaned and filtered.

Final usable dataset:

• train: 418
• dev: 60
• test_gold: 119

The task uses a fixed hierarchical schema with three levels:

• level1
• level1_5
• level2

Notes: severity is not included in this version

---

5. Intended Use

This model is intended for:

• research demos
• portfolio / interview projects
• schema-aware structured extraction experiments
• Chinese automotive complaint analysis

This model is not intended for production safety decision-making without additional validation.

---

6. How to Use

This repository contains a LoRA adapter, not a full base model.

You need to:

• load the base model Qwen2.5-3B-Instruct
• load this LoRA adapter on top of it
• provide candidate-guided input if you want to reproduce the best setting

Example (PEFT)

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_path = "Qwen/Qwen2.5-3B-Instruct"
adapter_path = "YOUR_ADAPTER_PATH"

tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    trust_remote_code=True,
    device_map="auto"
)

model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

---

7. Candidate-guided Input Format

The best-performing setting uses retrieved candidate labels in the prompt.

Example input format:

【候选标签】
1. 制动系统 / 制动系统机械部件 / 制动失灵
2. 制动系统 / 制动系统机械部件 / 制动力不足
3. 制动系统 / 制动系统机械部件 / 制动异响
...

【投诉文本】
品牌：福田时代
问题简述：制动系
详细描述：福田祥菱V3产品质量控告

---

8. Limitations

• This model is built for a closed-set schema, not open-ended generation.
• Performance depends on the quality of candidate label retrieval.
• It is optimized for the current label schema and complaint style, and may not generalize well to unrelated domains.
• It should not be used alone for real-world safety-critical decisions.

9. Project Repository

Full project code and experiment pipeline:

https://github.com/Richard37546/car-safety-complaint-tagging