ConvoBrief: LoRA-enhanced BART Model for Dialogue Summarization

This model is a variant of the facebook/bart-large-cnn model, enhanced with Low-Rank Adaptation (LoRA) for dialogue summarization tasks. LoRA employs Low-Rank Attention to facilitate feature aggregation across different positions in the sequence, making it particularly effective for capturing the nuances of dialogues.

LoRA Configuration:

• r: 8 (Number of attention heads in LoRA)
• lora_alpha: 8 (Scaling factor for LoRA attention)
• target_modules: ["q_proj", "v_proj"] (Modules targeted for LoRA, enhancing query and value projections)
• lora_dropout: 0.05 (Dropout rate for LoRA)
• bias: "lora_only" (Bias setting for LoRA)
• task_type: Dialogue Summarization (SEQ_2_SEQ_LM)

This model has been fine-tuned using the PEFT (Parameter-Efficient Fine-Tuning) approach, striking a balance between dialogue summarization objectives for optimal performance.

Usage:

Deploy this LoRA-enhanced BART model for dialogue summarization tasks, leveraging the power of Low-Rank Adaptation to capture contextual dependencies in conversations. Generate concise and informative summaries from conversational text, enhancing your applications with enriched context-awareness.

from peft import PeftModel, PeftConfig
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from transformers import pipeline

# Load PeftConfig and base model
config = PeftConfig.from_pretrained("Ketan3101/ConvoBrief")
base_model = AutoModelForSeq2SeqLM.from_pretrained("facebook/bart-large-cnn")

# Load PeftModel
model = PeftModel.from_pretrained(base_model, "Ketan3101/ConvoBrief")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")

# Define a pipeline for dialogue summarization
summarization_pipeline = pipeline(
    "summarization",
    model=model,
    tokenizer=tokenizer
)

# Example dialogue for summarization
dialogue = ["""
    #Person1#: Happy Birthday, this is for you, Brian.
    #Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
    #Person1#: Brian, may I have a pleasure to have a dance with you?
    #Person2#: Ok.
    #Person1#: This is really wonderful party.
    #Person2#: Yes, you are always popular with everyone. and you look very pretty today.
    #Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
    #Person2#: You look great, you are absolutely glowing.
    #Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday."""
]

# Combine dialogue into a single string
full_dialogue = " ".join(dialogue)

# Generate summary
summary = summarization_pipeline(full_dialogue, max_length=150, min_length=40, do_sample=True)

print("Original Dialogue:\n", full_dialogue)
print("Generated Summary:\n", summary[0]['summary_text'])

Framework versions

• PEFT 0.4.0