Model Overview
The Mental Health Chatbot model is a fine-tuned version of the Gemma-2B model, adapted to provide empathetic and informative responses to mental health-related queries. The model was fine-tuned using the heliosbrahma/Mental_Health_Chatbot_Dataset, which contains real-world conversations about mental health between patients and healthcare providers.
Model Description
Developed by: BM Son, SH Park, SK Hwang
Activity with: MLB 2024, Gemma Sprint
Model type: Causal Language Model (GemmaCausalLM)
Finetuned from model: google/gemma-2b
API used: PyTorch and Hugging Face Transformers
Dataset: Hugging Face heliosbrahma/mental_health_chatbot_dataset
Code: Custom Python Code (shared on Hugging Face or Colab)
Language(s) (NLP): English
Training: LoRA (Low-Rank Adaptation) applied with a rank of 32; trained with 8-bit quantization using NF4 type for resource efficiency.
Dataset Description
The dataset used for fine-tuning contains conversational pairs where patients ask about various mental health topics and healthcare providers offer advice. All personally identifiable information (PII) has been removed to ensure privacy.
Data Fields:
Text: Contains a series of human questions and assistant responses in the format <HUMAN>: for questions and <ASSISTANT>: for answers.
Training Procedure
The model was trained using the LoRA technique for parameter-efficient fine-tuning. It was optimized using 8-bit quantization (NF4 type) to make the training process more memory efficient. The model was trained on a mix of conversational mental health data, focusing on improving its ability to generate contextually relevant and empathetic responses.
Optimizer: PagedAdamW (8-bit) Batch Size: 1 per device Learning Rate: 2e-4 Max Steps: 100 Gradient Accumulation: 4 steps Loss function: Cross-entropy
Example Usage
To generate a response to a mental health-related query, you can input a question using the following format:
query = "How can I manage anxiety?"
response = model.generate_response(query)
print(response)
Example Output:
<HUMAN>: How can I manage anxiety?
<ASSISTANT>: To manage anxiety, consider practicing mindfulness, staying physically active, and seeking support from friends, family, or a professional. Techniques like deep breathing and grounding exercises can also help in calming your mind during anxious moments.
Conclusion
This Mental Health Chatbot aims to serve as a supplemental tool for individuals seeking mental health advice. While it cannot replace professional care, it provides accessible and empathetic responses to common questions, available anytime.
Conclusion
This Mental Health Chatbot aims to serve as a supplemental tool for individuals seeking mental health advice. While it cannot replace professional care, it provides accessible and empathetic responses to common questions, available anytime.