---
license: other
language:
- en
pipeline_tag: conversational
inference: false
tags:
- AI
- ConversationalAI
---

<h1 style="text-align: center">LLmRa-2.7B</h1>
<h2 style="text-align: center">A conversational Open Pre-trained Transformer Language Model fine-tune.</h2>

**LLmRa 2.7B**, as a proof-of-concept fine-tune of [facebook/opt-2.7b](https://huggingface.co/facebook/opt-2.7b) optimized for dialogue.

**Disclaimer:** NSFW data was included in the fine-tuning of this model. Although SFW inputs will usually result in SFW outputs, you are advised to **chat at your own risk. This model is not suitable for use by minors.**

**Warning:** This model is **NOT** suitable for use by minors. **It will output X-rated content under certain circumstances.**

**This model is fine-tuned on a small-testing dataset, version 2 or a higher parameter model will contain the full dataset.**

---

## Usage Format

To effectively utilize the model, follow this structured format for engaging text-based conversations:

**1. Initialization**

Here is how you can define the personality of the language model:

```
<|system|>[Persona]
```

- **Persona**: You can define a specific persona or context for the AI, but it's optional. It can be a character, a role, or just a style of interaction.

**2. AI Introduction**

```
<|user|>[User input]<|model|>
```
- Users can start the conversation by entering their message within `<|user|>` and closing with `<|model|>`.

---

### Example Usage:

Here's an example of how to start a conversation with the AI:

```
<|system|>I'm here to provide information and assistance on a wide range of topics.
<|model|>Hello! Welcome to our AI-powered assistant. How can I assist you today?
<|user|>Tell me about the history of artificial intelligence.
<|model|>
```

Continue the conversation as needed. This structured format helps maintain a smooth and engaging interaction with the AI.

You are not required to include `User`, you can change it to your prefered name or leave it blank You may also add the AI name, example:

```
<|user|>YourNameHere: Hello.<|model|>CharacterName:
```

You can also use this instruct prompt example:

```
<|system|>What is one plus one?<|model|>
```

## Loading The Model

To use the model and interact with it, use the Python code below:

```Python
from transformers import (AutoModelForCausalLM,
                          AutoTokenizer,
                          pipeline,
                          )

model = AutoModelForCausalLM.from_pretrained('L-R/LLmRa-2.7B')
tokenizer = AutoTokenizer.from_pretrained('L-R/LLmRa-2.7B')

pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=100)

input_question = 'QUESTION HERE'

question_formatted = f'<|system|>{input_question}<|model|>'

result = pipe(question_formatted)

print(f"[model]: {result[0]['generated_text'][len(question_formatted):]}")
```

Or the more complex one:

```Python
import os
import random
import sys
import time
import json
import torch

from transformers import (AutoTokenizer,
                          AutoModelForCausalLM,
                          BitsAndBytesConfig,
                          set_seed)

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
local_tokenizer = bool(os.getenv('TOKENIZERS_PARALLELISM', 'false'))


class Chatbot:
    def __init__(self, config):

        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.tokenizer = None
        self.config = config
        self.persona = None
        self.model = None
        self.history = []

        self.load_model()

    def create_persona(self, persona_data):
        required_keys = ['name', 'description', 'greeting']
        if not all(key in persona_data for key in required_keys):
            raise ValueError(
                "Missing required keys in persona_data. Please provide 'name', 'description', and 'greeting'.")

        new_persona_id = str(max(int(key) for key in self.config["personas"].keys()) + 1)

        self.config["personas"][new_persona_id] = persona_data
        return new_persona_id

    def load_model(self):
        model_path = self.config["model_path"]
        tokenizer_path = self.config["tokenizer_path"]

        quantization_config = BitsAndBytesConfig(

            load_in_4bit= self.config['load_model_4bit'],
            bnb_4bit_quant_type='nf4' if self.config['load_model_4bit'] else None,
            bnb_4bit_compute_dtype=torch.float16 if self.config['load_model_4bit'] else None,
            bnb_4bit_use_double_quant=True if  self.config['load_model_4bit'] else None,

            load_in_8bit=self.config['load_model_8bit'],
            bnb_8bit_quant_type='nf4' if self.config['load_model_8bit'] else None,
            bnb_8bit_compute_dtype=torch.float16 if self.config['load_model_8bit'] else None,
            bnb_8bit_use_double_quant=True if self.config['load_model_8bit'] else None,

        )

        if not model_path or not tokenizer_path:
            raise ValueError('model_name or tokenizer_path name not found! Define one.')

        if self.config['load_model_4bit'] and self.config['load_model_8bit']:
            raise ValueError("You can't load the model in 8 bits and 4 bits at the same time!")

        if not self.config['user_name']:
            print('You have not selected a name! No name will be send to the model.')

        print(f"\nLoading model: {model_path}")

        if torch.cuda.is_available():

            self.model = AutoModelForCausalLM.from_pretrained(
                model_path,

                use_auth_token=self.config['model_token'],
                quantization_config=quantization_config,)

            if torch.cuda.device_count() > 1:
                self.model = torch.nn.DataParallel(self.model)
                model_running_on = f'{torch.cuda.device_count()} GPUs'
            else:
                model_running_on = '1 GPU'
        else:
            self.model = AutoModelForCausalLM.from_pretrained(
                model_path,
                quantization_config=quantization_config,
                use_auth_token=self.config['model_token']).to(
                self.device

            )
            model_running_on = 'CPU'

        print(f'Model is running on: {model_running_on}')

        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, use_auth_token=self.config['model_token'])
        print(self.tokenizer)


    def load_persona(self, persona_id):
        personas = self.config["personas"]
        if persona_id in personas:
            self.persona = personas[persona_id]
        else:
            raise ValueError("Invalid persona ID")


    def formatting_question(self, user_input, history):

        config_user = self.config['use_names']['user']
        config_model = self.config['use_names']['model']
        config_question = self.config['use_question_template']

        if config_question:
            formatted_answer = (
                f'<|system|>{user_input}<|model|>'
            )
        else:
            m_ = self.persona["description"]
            g_ = self.persona["greeting"]
            n_ = self.persona["name"]
            un_ = self.config["user_name"]

            if config_user and config_model:
                formatted_answer = (
                    f'<|system|>{m_}<|model|>{n_}: {g_}{history}<|user|>{un_}: {user_input}<|model|>{n_}:'
                )
            elif config_user:
                formatted_answer = (
                    f'<|system|>{m_}<|model|>{g_}{history}<|user|>{un_}: {user_input}<|model|>'
                )
            elif config_model:
                formatted_answer = (
                    f'<|system|>{m_}<|model|>{n_}: {g_}{history}<|user|>{user_input}<|model|>{n_}:'
                )
            else:
                formatted_answer = (
                    f'<|system|>{m_}<|model|>{g_}{history}<|user|>{user_input}<|model|>'
                )

        return formatted_answer

    def history_formatting(self, last_input, last_output):

        config_user = self.config['use_names']['user']
        config_model = self.config['use_names']['model']

        n_ = self.persona["name"]
        un_ = self.config["user_name"]

        if config_user and config_model:
            formatted_answer = (
                f'<|user|>{un_}: {last_input}<|model|>{n_}: {last_output}'
            )
        elif config_user:
            formatted_answer = (
                f'<|user|>{un_}: {last_input}<|model|>{last_output}'
            )
        elif config_model:
            formatted_answer = (
                f'<|user|>{last_input}<|model|>{n_}: {last_output}'
            )
        else:
            formatted_answer = (
                f'<|user|>{last_input}<|model|>{last_output}'
            )

        return formatted_answer

    def reply(self, user_input):

        config_question = self.config['use_question_template']
        set_seed(random.randint(1, 1000))
        user_input = " ".join(user_input.split())

        if len(self.history) > self.config["history_length"]:
            model_history = "\n".join([str(item) for item in self.history[-self.config["history_length"]:]])
        else:
            model_history = "\n".join([str(item) for item in self.history])

        input_ai = self.formatting_question(user_input, model_history).strip()
        tokenized_input_ai = self.tokenizer.encode(input_ai, return_tensors="pt")

        output_ids = self.model.generate(
            max_length=self.config["max_generation_length"] + len(tokenized_input_ai[0]),
            no_repeat_ngram_size=self.config["no_repeat_ngram_size"],
            repetition_penalty=self.config["repetition_penalty"],
            length_penalty=self.config["length_penalty"],
            input_ids=tokenized_input_ai.to(self.device),
            pad_token_id=self.tokenizer.eos_token_id,
            temperature=self.config["temperature"],
            top_k=self.config["top_k"],
            top_p=self.config["top_p"],
            early_stopping=True,
            use_cache=True,
            do_sample=True,
        )

        ai_reply = self.tokenizer.decode(
            output_ids[0],
            skip_special_tokens=False)[len(input_ai)+4:]

        if not config_question:
            self.history.append(self.history_formatting(user_input, ai_reply))

        return ai_reply.strip()

    def reset_conversation(self):

        self.history = []

class UserInterface:
    def __init__(self, chatbot):
        self.chatbot = chatbot

    def run(self):

        persona_id = self.chatbot.config["default_persona"]
        self.chatbot.load_persona(persona_id)

        print("\nChosen Persona:", self.chatbot.persona["name"])
        print("Your Chosen Name:", self.chatbot.config["user_name"])

        print(f'\n{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')
        self.chatbot.history.append(f'{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')

        while True:
            user_input = input(f"\n>> {self.chatbot.config['user_name']}: ")
            if user_input.lower() == "reset_app" or user_input == "reset_app":
                self.chatbot.reset_conversation()
                print("\nConversation history has been reset.\n")
                self.chatbot.history.append(f'{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')
                print(f'{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')
                continue

            if user_input.lower().startswith("create_persona"):

                # Example of use: create_persona

                # {"name": "CustomPersona",
                # "description": "This is a custom persona created by the user.",
                # "greeting": "Hello! I am CustomPersona, nice to meet you!"}

                try:
                    persona_data = json.loads(' '.join(user_input.split()[1:]))
                    new_persona_id = self.chatbot.create_persona(persona_data)
                    print(f"Persona created with ID: {new_persona_id}")
                except json.JSONDecodeError:
                    print("Invalid JSON input. Please provide a valid JSON string containing 'name', 'description', and 'greeting'.")
                except ValueError as e:
                    print(e)

            # Add a command to change the persona
            if user_input.lower().startswith("change_persona"):
                try:
                    new_persona_id = user_input.split()[1]
                    self.chatbot.load_persona(new_persona_id)
                    self.chatbot.reset_conversation()
                    print("\nPersona changed to:", self.chatbot.persona["name"])
                    print(f'\n{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')
                    self.chatbot.history.append(f'{self.chatbot.persona["name"]}: {self.chatbot.persona["greeting"]}')
                    continue
                except (IndexError, ValueError):
                    print("Invalid command or persona ID. Please use 'change_persona [ID]'.")
                    continue

            if user_input.lower() == "exit_app" or user_input == "exit_app":
                print("Goodbye!")
                break

            reply = self.chatbot.reply(user_input)

            def typewriter_effect(sentence, type_delay):

                for char in sentence:
                    sys.stdout.write(char)
                    sys.stdout.flush()
                    time.sleep(type_delay)

            reply_length = len(reply)
            type_delay_ranges = {
                (100, 200): 0.03,
                (200, 300): 0.02,
                (300, 400): 0.01,
                (400, 500): 0.005
            }

            default_type_delay = 0.04

            for length_range, delay in type_delay_ranges.items():
                if length_range[0] < reply_length <= length_range[1]:
                    type_delay = delay
                    break
            else:
                type_delay = default_type_delay

            if self.chatbot.config['use_typing_effect']:
                typewriter_effect(f'{self.chatbot.persona["name"]}: {reply}', type_delay)
            else:
                print(f'{self.chatbot.persona["name"]}: {reply}')

def main():
    
    config = {
        "user_name": "Jack",  # The user's name, which is set to "Jack" in this case.

        "model_path": "L-R/LLmRa-2.7B",  # Path to the model used for generating responses.
        "tokenizer_path": "L-R/LLmRa-2.7B",  # Path to the tokenizer associated with the model.
        "model_token": None,  # If you want to load the model using your huggingface token. (Not required, but included)

        "load_model_4bit": True,  # Whether to load the model with 4-bit precision.
        "load_model_8bit": False,  # Whether to load the model with 8-bit precision.

        "use_typing_effect": True,  # Whether to simulate a typing effect when displaying responses.

        "use_names": {
            "model": False,  # Whether the model's name should be used in question formatting.
            "user": False,  # Whether the user's name should be used in question formatting.
        },

        "use_question_template": False,  # Whether to use predefined question templates in conversations.

        "personas": {
            # A dictionary of personas with their descriptions and greetings for use in conversations.
            "1": {
                "name": "LLmRa",
                "description": "Description of the LLmRa persona. It provides background and characteristics of the persona.",
                "greeting": "The greeting message when the LLmRa persona is active in a conversation."
            },
            "2": {
                "name": "Hikari",
                "description": "Description of the Hikari persona. It provides background and characteristics of the persona.",
                "greeting": "The greeting message when the Hikari persona is active in a conversation."
            }
        },

        "max_generation_length": 450,  # The maximum length for generated responses.

        "default_persona": "1",  # The default persona to use when starting a conversation.

        "history_length": 6,  # The maximum number of previous messages to consider in the conversation history.

        "top_k": 40,  # Top-k sampling parameter for text generation.
        "top_p": .55,  # Top-p sampling parameter for text generation.
        "temperature": .55,  # Temperature parameter for controlling the randomness of generated text.
        "length_penalty": 0.65,  # Penalty factor for generating longer or shorter responses.
        "no_repeat_ngram_size": 4,  # Parameter to avoid repeating n-grams in generated text.
        "repetition_penalty": 1.25,  # Penalty factor for avoiding repeated phrases in generated text.
    }

    # Initialize chatbot and user interface
    chatbot = Chatbot(config)
    ui = UserInterface(chatbot)

    # Run the user interface
    ui.run()


if __name__ == "__main__":
    main()
```

## Known issues

Model doesn't some of the times follow instructions.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_L-R__LLmRa-2.7B)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 32.16   |
| ARC (25-shot)         | 37.03          |
| HellaSwag (10-shot)   | 60.65    |
| MMLU (5-shot)         | 25.58         |
| TruthfulQA (0-shot)   | 35.23   |
| Winogrande (5-shot)   | 61.56   |
| GSM8K (5-shot)        | 0.3        |
| DROP (3-shot)         | 4.76         |