Model Card for `falcon-7b-qlora-interview_qa-support-bot`

NOTE: Please review the repository for detailed instructions on how to use this model. Do not directly click the "Use this model" button; instead, access the model through the inference.py as mentioned in the repository.

This model has been fine-tuned for interview question answering and supports dynamic dialogue interactions. It uses PEFT (Parameter-Efficient Fine-Tuning) with bitsandbytes quantization for memory-efficient deployment.

Model Details

Model Description

This model is built on top of the Falcon 7B model from TII UAE and has been fine-tuned using PEFT for interview question answering support. The model is quantized using bitsandbytes to enable 4-bit precision, making it suitable for deployment on lower-resource environments while maintaining performance.

Developed by: Katakam Pranav Shankar
Model type: Causal Language Model (for Q&A and dialogue systems)
Language(s): English
Fine-tuned from model: tiiuae/falcon-7b'
Research Paper: https://www.mlexpert.io/blog/fine-tuning-llm-on-custom-dataset-with-qlora#training ]

Model Sources

Repository: https://huggingface.co/Pranav06/falcon-7b-qlora-interview_qa-support-bot

Uses

Direct Use

This model is designed to generate interview-related responses, allowing users to engage in an interactive interview process. It supports multiple rounds of questions and responses and can handle complex dialogue.

Downstream Use

This model can be integrated into larger systems for dynamic Q&A support, such as customer service bots, recruitment tools, or interview preparation platforms.

Out-of-Scope Use

The model is not intended for generating harmful, biased, or inappropriate content.
It should not be used in real-world interviews without human supervision, as it may still generate responses with inaccuracies.

Bias, Risks, and Limitations

As with all language models, there may be biases in the model due to the data it was trained on. We encourage users to review the output carefully, especially when it comes to sensitive interview-related topics.

Recommendations

Users should be aware of the model's limitations, especially in handling nuanced or highly specialized questions. It is recommended to review and supervise the model's output before deploying in production environments.

How to Get Started with the Model

To use this model with PEFT and bitsandbytes quantization, follow the steps below:

Install Required Libraries:

Before you start, make sure you have the necessary dependencies installed:
```
pip install transformers bitsandbytes torch peft
```
Download the Custom Script:

You can use the custom inference.py script for efficient deployment with PEFT and quantization. This script ensures that the model is loaded properly and ready for generating responses.

Download the inference.py script from the model repository, then run the script as follows:
```
python inference.py
```

Example Code to Use the Model:

Here's an example of how to use the model for generating a response:

 import json
 import torch
 from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
 from peft import PeftConfig, PeftModel
 import bitsandbytes as bnb
 import os
 
 # Set up CUDA for GPU use
 os.environ["CUDA_VISIBLE_DEVICES"] = "0"
 
 # Configuring bitsandbytes quantization
 bnb_config = BitsAndBytesConfig(
     load_in_4bit=True,
     bnb_4bit_use_double_quant=True,
     bnb_4bit_quant_type="nf4",
     bnb_4bit_compute_dtype=torch.bfloat16,
 )
 
 # Load the PEFT model configuration and model
 PEFT_MODEL = "Pranav06/falcon-7b-qlora-interview_qa-support-bot"
 config = PeftConfig.from_pretrained(PEFT_MODEL)
 
 model = AutoModelForCausalLM.from_pretrained(
     config.base_model_name_or_path,
     return_dict=True,
     quantization_config=bnb_config,
     device_map="auto",
     trust_remote_code=True,
 )
 
 # Load the tokenizer
 tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
 tokenizer.pad_token = tokenizer.eos_token
 
 # Apply PEFT to the model
 model = PeftModel.from_pretrained(model, PEFT_MODEL)
 
 # Genration configuration
 generation_config = model.generation_config
 generation_config.max_new_tokens = 200
 generation_config.temperature = 0.7
 generation_config.top_p = 0.7
 generation_config.num_return_sequences = 1
 generation_config.pad_token_id = tokenizer.eos_token_id
 generation_config.eos_token_id = tokenizer.eos_token_id
 
 DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
 
 # Define a function to generate responses
 def generate_response(question: str) -> str:
     prompt = f"""
 <human>: {question}
 <assistant>:
 """.strip()
     encoding = tokenizer(prompt, return_tensors="pt").to(DEVICE)
     with torch.inference_mode():
         outputs = model.generate(
             input_ids=encoding.input_ids,
             attention_mask=encoding.attention_mask,
             generation_config=generation_config,
         )
     response = tokenizer.decode(outputs[0], skip_special_tokens=True)
 
     assistant_start = "<assistant>:"
     response_start = response.find(assistant_start)
     response = response[response_start + len(assistant_start) :].strip()
     response_lines = response.split("\n")
     final_response = response_lines[0].strip()
 
     return final_response


  # Example usage
  input_text = "Tell me about yourself."
  response = generate_response(input_text)
  print(response)

Training Details

Training Data

The model was fine-tuned on approximately 500+ interview-related question-answer pairs and dialogue examples.

Training Procedure

The model was fine-tuned with PEFT using a combination of supervised learning from Q&A datasets.

Training Hyperparameters

Training regime: Mixed precision (fp16)
Optimizer: AdamW
Batch Size: 32

Evaluation

Testing Data, Factors & Metrics

Metrics: The model's accuracy is evaluated based on how well it answers interview questions across different topics. Metrics such as F1-score and accuracy are used to assess the model's performance.

Results

The model performs well in generating coherent and contextually accurate responses for interview-style questions. However, its performance can degrade when faced with highly technical or niche questions.

Technical Specifications

Model Architecture and Objective

The model is based on the Falcon-7B architecture with PEFT fine-tuning. It is designed to handle natural language understanding and generation tasks, especially for interview Q&A systems.

Compute Infrastructure

The model was trained using GPUs and optimized using 4-bit quantization with the bitsandbytes library to reduce memory usage.

Citation

If you use this model in your research or application, please cite the original Falcon 7B model as follows:

BibTeX:

@misc{falcon7b,
  author = {TII UAE},
  title = {Falcon 7B},
  year = {2023},
  publisher = {Hugging Face},
  url = {https://huggingface.co/tiiuae/falcon-7b}
}

More Information

For additional details, including fine-tuning or usage instructions, please visit the model repository.

Model Card Authors

Katakam Pranav Shankar

Model Card Contact

For any questions regarding the model, please contact katakampranavshankar@gmail.com.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Pranav06/falcon-7b-qlora-interview_qa-support-bot

Base model

tiiuae/falcon-7b

Finetuned

(47)

this model

Model Card for falcon-7b-qlora-interview_qa-support-bot