Repository: joachim-gassen/botex (language: python)

Issue:
Implementing local LLM code into main branch 
H there! To me it seems that, while implementing local LLMs you did many changes to the codebase that were not only driven by objective to open botex for local LLMs but are also reflective of your coding style and preferences (OOP approach, using YAML for config etc.).

In order to move this forward we need to decide on which changes to the main branch API are 

a) strictly necessary to implement local LLMs, 
b) desirable because we both agree that they improve the functionality or maintainability of the codebase significantly, or
c) come down to a matter of tase.

Whatever we do, we need to be 100% certain that the OpenAI gpt4 approach remains fully functional after merging. Over the weekend, I successfully ran the main experiments of our paper (600 bot participants!), stamping out a few last issues in the code. I am now reasonably certain that the main branch is ready for production and plan to take this code public over the next two to four weeks, It would be terrific to have the local LLMs to be at the same level at launch but we do not have to.

So: Can you identify the API changes that fall in category a)?

Thanks, Joachim


Discussion / hints:
Hello,

Yes, there are a lot of changes and can be fairly characterized as a coding style and preference. I think we will need to eventually discuss what goes into b and what goes into c. I also agree that the gpt4 approach should remain fully functional, my work was just laser focused on local llm.

As a next step, i will take a second shot at this and focus solely on a. Afterwards, I can also pick this up and implement gpt4 functionality. Once this two are done then we can have a discussion about how we can go forward.

In the meantime, I would be interested to run the main experiment for the paper in the local setup. Since I have this up and working locally, i would be happy if you could share the experiment with me. 


Best, Fikir


Hello, 

I will just leave this now and you can get back to it whenever. I have created new branch local_llm_take_2 and implemented a) strictly necessary to implement local LLMs. There are some improvements that have not been migrated form the oop approach but it should be functional and be able to pass the botex_test.

Best, Fikir
HI there. This sounds great. Will take a look at it asap. And I agree: After the paper is out and the package is public we should sit down and debate if/how we want to restructure the API.
And about the experiments of the paper: Great idea! I just invited you the the accframe repo. The README should contain sufficient info on how to reproduce our three experiments honesty, trust, and giftex. The findings of our experimental run can be found in the static folder (results.pdf).

The two conditions for each experiments are organized as separate otree experiments. The framed/contextualized version always starts with a leading f. I would recommend that you start with a few small sample (60p per condition) pre tests. Besides the actual results, I would super interested to see how the check questions towards the end of the experiments work out. The giftex experiment is the most demanding one. If the local LLM is able to play that reasonably well and to pass the pre and post comprehension checks, then we should consider preregistering the experimental design and running the full fledged experiments for the paper. 

Hi Joachim,

So I have played around with this and I am hitting a bit of a wall. So the app itself is a bit of hit and miss as it works with small size but at times has problems that need to be ironed out. The bigger issue that I have is that the llms are not very good at maths and i guess logic. So the check questions are indeed revealing about this. Below are snapshots of the honesty and gift check conversation. I suggest when you have time we talk about the next steps

```json
# honesty

{
    "role": "assistant",
    "content": {
        "summary": "The participant, acting as a division manager, completed all 10 rounds of the budgeting experiment. The participant's private forecasting system predicted actual production costs of 4050 points, and the participant filed a budget request of 4050 points to the firm's corporate headquarters. The payoff for this round was 250 points for the participant and 1950 points for the firm. The total number of points earned by the participant in the 10 rounds combined is 2550 points, and the total number of points earned by the firm in the 10 rounds combined is 9950 points. The participant answered the last questions as follows: Who knew the true cost structure? Only I. Assuming that the participant only cares about points at the end, the optimal strategy would be to file a budget request reflecting the true production costs. The participant characterized their personality as a Human. The participant did not provide any feedback.",
        "questions": [
            {
                "id": "id_comprehension_check1",
                "answer": "Only I",
                "reason": "The participant, acting as a division manager, was the only one who knew the true cost structure."
            },
            {
                "id": "id_comprehension_check2",
                "answer": "Filing a budget request reflecting the true production costs",
                "reason": "Assuming that the participant only cares about points at the end, the optimal strategy would be to file a budget request reflecting the true production costs."
            },
            {
                "id": "id_human_check",
                "answer": "I am a Human",
                "reason": "The participant characterized their personality as a Human."
            },
            {
                "id": "id_feedback",
                "answer": "",
                "reason": "The participant did not provide any feedback."
            }
        ]
    }
}


# gift

{
    "role": "assistant",
    "content": {
        "summary": "This is a decision-making experiment where participants are paired anonymously and randomly. Each participant is assigned a role (Participant A or Participant B) and remains in the same role throughout the experiment. The experiment consists of 10 rounds. Participant A starts with 100 points and can transfer any integer amount between 0 and 100 points to Participant B. Participant B then chooses a multiplier from a table with costs ranging from 0 to 18. The payoff for Participant A is calculated as (100 - transfer to Participant B) * multiplier. The payoff for Participant B is the transfer received from Participant A minus the cost of the multiplier. A Quick Check is a page that tests the participant's understanding of the payoffs. The page contains a table with multipliers and their costs and a question about the payoffs for Participant A and Participant B when Participant A sends 10 points and Participant B chooses a multiplier of 0.3. The payoff for Participant A is 90 and the payoff for Participant B is 7.",
        "questions": [
            {
                "id": "id_comprehension_check_pre1",
                "answer": 90,
                "reason": "The payoff for Participant A is calculated as (100 - transfer to Participant B) * multiplier. In this case, Participant A transferred 10 points, and Participant B chose a multiplier of 0.3. So, the payoff for Participant A is (100 - 10) * 0.3 = 90."
            },
            {
                "id": "id_comprehension_check_pre2",
                "answer": 7,
                "reason": "The payoff for Participant B is the transfer received from Participant A minus the cost of the multiplier. In this case, Participant A transferred 10 points, and Participant B chose a multiplier of 0.3 with a cost of 2. So, the payoff for Participant B is 10 - 2 = 7."
            }
        ]
    }
}
```
Thanks! This is what I to some extend expected and super interesting. In principle, I think our focus in terms of botex should be to make sure that local LLM work 'in principle'. Then, in the 'accframe' project, we should strive to provide intelligence on whether they are capable to participate in more or less complex economic experiments. If I recall correctly, prior work has been using local LLMs in similar settings but regularly without reporting the outcome of comprehension checks etc. I think that we might be able to add something here but as I said this is more an 'accframe' topic ;-)
Hi Joachim,
Just FYI, I have made some progress on the error handling extension and writing a test for the local llm workflow. I am not happy with the test that I wrote because you have to actually change the path to llama.cpp and the gguf model inside of the test. I also did not want to put it inside _secret because it is not a secret :) So I am still thinking about how to improve this. In terms of the error handling, I need to do a bit more testing, especially with the gpt workflow. So at the moment, I do not think it is close to being merged with main. But maybe you can accept the pull request to the local_llm_take_2 branch and take it for a spin. I am curious to hear how this works on your machine.
Hello,
The testing workflow has been updated as per our discussion. You can run `pytest --remote` for gpt test, `pytest --local` for local llm test and `pytest`for testing both. The settings for local_llm test are also left now in `_secrets.env`

Fix patch:
diff --git a/README.md b/README.md
index f58ae41..67197df 100644
--- a/README.md
+++ b/README.md
@@ -80,9 +80,8 @@ The Local LLM installation mostly follows the same step for a remote LLM but wit
 4. Install the BotEx package locally and editable `pip install -e .`
 5. Clone llama cpp from [here](https://github.com/ggerganov/llama.cpp) and follow the instructions to build it.
 6. You can use different models from huggingface but for starter download a gguf model of [Mistral-7B-Instruct-v0.3.Q4_K_M.gguf](https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF). At the moment the following [Q4_K_M version](https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF/resolve/main/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf) is tested and working.
-7. You also need to download the original `tokenizer_config.json` from the original model creators. For this you will have to create an account with huggingface, and accept the terms by mistralai for this gated model. You can find it [here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3), when you go to the files and versions tab you will find a link to download the `tokenizer_config.json` please put it in the same directory as your downloaded model.
-8. Finally, copy `_secret.env` to `secret.env` and edit. You do not need to have an OpenAI API Key but you should edit the `path_to_compiled_llama_cpp_main_file` and `local_model_path`. If you have a gpu and llama.cpp has been compiled with gpu support you can also set the `number_of_layers_to_offload_to_gpu` to a number greater than 0. How high you can set this number depends on the amount of vram your gpu has and the size of the model you are using.
-9. You can now test the local LLM by running `.venv/bin/pytest --local`. If everything is set up correctly you should see a similar output as the remote LLM test.
+7. Finally, copy `_secret.env` to `secret.env` and edit. You do not need to have an OpenAI API Key but you should edit the `path_to_compiled_llama_server_executable` and `local_model_path`. This will by default start a llama_cpp server on "http://localhost:8080". You can change this by setting the `api_base_url` argument of the `LocalLLM`. If you have a gpu and llama.cpp has been compiled with gpu support you can also set the `number_of_layers_to_offload_to_gpu` to a number greater than 0. How high you can set this number depends on the amount of vram your gpu has and the size of the model you are using. You can also increase the number of parallel requests that your local LLM can handle by setting `num_slots` to a number greater than 1. But keep in mind that this will significantly increase the amount of memory your local LLM will use.
+8. You can now test the local LLM by running `.venv/bin/pytest --local`. If everything is set up correctly you should see a similar output as the remote LLM test.
 
 If you are interested in testing both the remote and local LLM you can run both tests with `.venv/bin/pytest`.
 
diff --git a/_secrets.env b/_secrets.env
index aebb2fd..0816f7a 100644
--- a/_secrets.env
+++ b/_secrets.env
@@ -15,7 +15,9 @@ OPENAI_API_KEY=******
 BOT_DB_SQLITE="tests/botex.db"
 
 # This is only needed for local llm workflow
-path_to_compiled_llama_cpp_main_file=/mnt/file_ssd_2tb/fikir/projects/chat_backend/new_llama_cpp/llama.cpp/main
+path_to_compiled_llama_server_executable=/mnt/file_ssd_2tb/fikir/projects/chat_backend/29_06_2024/llama.cpp/llama-server
 local_model_path=/mnt/file_ssd_2tb/fikir/projects/chat_backend/models/mistral/7b_instruct/Mistral-7B-Instruct-v0.3.Q4_K_M.gguf
-# The below should be 0 if you are GPU poor :(, otherwise, you can set it depending on your GPU memory size
-number_of_layers_to_offload_to_gpu=0
\ No newline at end of file
+
+# The below should be 0 if you are GPU poor :(, otherwise, you can set it depending on your GPU memory size, you can also increase the number of slots for more parallelism if you have a good GPU memory.
+number_of_layers_to_offload_to_gpu=0
+num_slots=1
diff --git a/src/botex/bot.py b/src/botex/bot.py
index 4017ff1..205dc8b 100644
--- a/src/botex/bot.py
+++ b/src/botex/bot.py
@@ -21,9 +21,9 @@
 logging.getLogger("LiteLLM").setLevel(logging.WARNING)
 
 def run_bot(
-        botex_db, session_id, url, lock, full_conv_history = False,
+        botex_db, session_id, url, full_conv_history = False,
         model = "gpt-4o", openai_api_key = None,
-        local_llm: LocalLLM | None = None
+        local_llm: LocalLLM | None = None, user_prompts: dict | None = None
     ):
     """
     Run a bot on an oTree session. Normally, this function should not be called
@@ -33,7 +33,6 @@ def run_bot(
     botex_db (str): The name of the SQLite database file to store BotEx data.
     session_id (str): The ID of the oTree session.
     url (str): The participant URL of the bot instance.
-    lock (threading.Lock): A lock to prevent concurrent access to the local model.
     full_conv_history (bool): Whether to keep the full conversation history.
         This will increase token use and only work with very short experiments.
         Default is False.
@@ -43,11 +42,11 @@ def run_bot(
     openai_api_key (str): The API key for the OpenAI service.
     local_llm (LocalLLM): A LocalLLM object to use for the bot. If this is not
         None, the bot will use the local model instead of the OpenAI model.
+    user_prompts (dict): A dictionary of user prompts to override the default prompts that the bot uses. The keys should be one or more of the following: ['start', 'analyze_first_page_no_q', 'analyze_first_page_q', 'analyze_page_no_q', 'analyze_page_q', 'analyze_page_no_q_full_hist', 'analyze_page_q_full_hist', 'page_not_changed', 'system', 'resp_too_long', 'json_error', 'end'.] If a key is not present in the dictionary, the default prompt will be used. If a key that is not in the default prompts is present in the dictionary, then the bot will exit with a warning and not running to make sure that the user is aware of the issue.
 
     Returns: None (conversation is stored in BotEx database)
     """
     bot_parms = locals()
-    bot_parms.pop('lock')
     bot_parms.pop('local_llm')
     bot_parms['local_llm'] = vars(local_llm) if local_llm else None
     if bot_parms['openai_api_key'] is not None: 
@@ -87,7 +86,8 @@ def set_id_value(dr, id, type, value, timeout = 3600):
     
     def wait_next_page(dr, timeout = 10):
         attempts = 0
-        while attempts < 600:
+        max_attempts = 360
+        while attempts < max_attempts:
             try:
                 WebDriverWait(dr, timeout).until(
                     lambda x: x.find_element(By.CLASS_NAME, 'otree-form')
@@ -95,11 +95,13 @@ def wait_next_page(dr, timeout = 10):
                 break # Exit the loop if successful
             except TimeoutException:
                 attempts += 1
+                if attempts % 60 == 0:
+                    logging.info(f"Waiting for page to load. Attempt {attempts}/{max_attempts}.")
                 continue # Retry if a timeout occurs
-        
-        if attempts == 600:
+
+        if attempts == max_attempts:
             logging.error("Timeout on wait page after 600 attempts.")
-            raise Exception("Timeout on wait page after 600 attempts.")
+            return 'Timeout on wait page after 600 attempts.'
 
         
     def scan_page(dr):
@@ -197,7 +199,7 @@ def llm_send_message(
         while resp_dict is None:
             if attempts > max_attempts:
                 logging.error("The llm did not return a valid response after %s number of attempts." % max_attempts)
-                raise Exception("Maximum number of attempts reached.")
+                return 'Maximum number of attempts reached.'
             attempts += 1
             try:
                 correction_message = conversation + [{"role": "assistant", "content": resp_str}, {"role": "user", "content": message}]
@@ -206,11 +208,10 @@ def llm_send_message(
             if model == "local":
                 assert local_llm, "Model is local but local_llm is not set."
                 assert conversation, "Conversation is empty."
-                with lock:
-                    if correction_message:
-                        resp = local_llm.completion(correction_message)
-                    else:
-                        resp = local_llm.completion(conversation)
+                if correction_message:
+                    resp = local_llm.completion(correction_message)
+                else:
+                    resp = local_llm.completion(conversation)
             else:
                 if correction_message:
                     resp = completion(
@@ -222,7 +223,7 @@ def llm_send_message(
                     resp =  completion(
                         messages=conversation, model=model,api_key=openai_api_key,
                         response_format = {"type": "json_object"}
-                )
+                    )
 
             resp_str = resp.choices[0].message.content
             if resp.choices[0].finish_reason == "length":
@@ -236,9 +237,8 @@ def llm_send_message(
                 end = resp_str.rfind('}', start)
                 resp_str = resp_str[start:end+1]
                 resp_dict = json.loads(resp_str, strict = False)
-            except:
-                logging.exception('')
-                logging.warning("Bot's response is not a valid JSON. Trying again.")
+            except (AssertionError, json.JSONDecodeError):
+                logging.warning(f"Bot's response is not a valid JSON\n{resp_str}\n. Trying again.")
                 resp_dict = None
                 message = prompts['json_error']
                 continue
@@ -248,13 +248,16 @@ def llm_send_message(
                 else:
                     success, error_msgs, error_logs = check_response(resp_dict)
                 if not success:
-                    logging.warning(f"Detected an issue: {''.join(error_logs)}. Adjusting response.")
+                    logging.warning(f"Detected an issue: {' '.join(error_logs)}.\n{resp_dict}.\nAdjusting response.")
                     message = ''
-                    for error_msg in error_msgs:
+                    for i, error_msg in enumerate(error_msgs):
                         if ':' in error_msg:
-                            error, ids = error_msg.split(": ")
+                            error, ids = error_msg.split(": ", 1)
                             f_dict = {error: ids}
-                            message += prompts[error].format(**f_dict) + ' '
+                            if i == 0:
+                                message += prompts[error].format(**f_dict) + ' '
+                            else:
+                                message += 'Additionally, ' + prompts[error].format(**f_dict) + ' '
                         else:
                             message += prompts[error_msg] + ' '
                     resp_dict = None
@@ -274,7 +277,7 @@ def check_response_start(resp):
             error_log = "Bot's response does not have the required understood key."
             check_result["error_log"].append(error_log)
             check_result["error"].append("no_understood_key")
-        if not str(resp['understood']).lower() == "yes":
+        if str(resp.get('understood')).lower() != "yes":
             error_log = "Bot did not understand the message."
             check_result["error_log"].append(error_log)
             check_result["error"].append("not_understood")
@@ -323,35 +326,38 @@ def check_response_question(resp, questions):
                 check_result["error"].append("questions_not_list")
 
 
-        q_ids = [q['question_id'] for q in questions] 
-        answer_ids = [q['id'] for q in resp['questions']]
+        q_ids = [q['question_id'] for q in questions]
+        answer_ids = [q['id'] for q in resp['questions'] if q.get('id')]
         unanswered_q_ids = set(q_ids) - set(answer_ids)
         if unanswered_q_ids:
             check_result["error_log"].append(f"unanswered_questions: {' '.join(unanswered_q_ids)}")
             check_result["error"].append(f"unanswered_questions: {' '.join(unanswered_q_ids)}")
 
         errors = {
-            "missing_answer_keys": [],
             "missing_answer": [],
             "missing_reason": [],
             "answer_not_number": [],
             "answer_not_float": [],
             "select_answer_number": [],
-            "select_answer_unknown": []
+            "select_answer_unknown": [],
+            "answer_id_not_found_in_q_id_list": []
         }
         for answer in resp['questions']:
             missing_answer_keys = set(["id", "answer", "reason"]) - set(answer)
             if missing_answer_keys:
-                errors['missing_answer_keys'].append({'question_id': answer['id'], 'missing_answer_keys': missing_answer_keys})
-            # is not empty, and can be parsed to the correct type
-            if answer['answer'] == "" or answer['answer'] is None:
+                if "id" in missing_answer_keys:
+                    continue
+            if not answer.get('answer'):
                 errors['missing_answer'].append(answer['id'])
-            if answer['reason'] == "" or answer['reason'] is None:
+            if not answer.get('reason'):
                 errors['missing_reason'].append(answer['id'])
             
             
             if answer['id'] in q_ids:
                 qtype = questions[q_ids.index(answer['id'])]['question_type']
+            else:
+                errors['answer_id_not_found_in_q_id_list'].append(answer['id'])
+                continue
             if qtype == 'number' and isinstance(answer['answer'], str):
                 try:
                     int_const_pattern = r'[-+]?[0-9]+'
@@ -399,6 +405,52 @@ def check_response_end(resp):
         if check_result.get("error"):
             return False, check_result["error"], check_result["error_log"]
         return True, None, None
+    
+    def gracefully_exit_failed_bot(failure_place):
+        if failure_place == "start":
+            result = "Bot could not even start. Stopping."
+        elif failure_place == "abandoned":
+            result = "Bot was likely abandoned by its matched participant. Exiting."
+        else:
+            result = "Bot could not provide a valid response after 5 attempts. Exiting."        
+        conv.append({"role": "system", "content": result})
+        store_data(botex_db, session_id, url, conv, bot_parms)
+        if failure_place != "start" and failure_place != "end":
+            dr.close()
+            dr.quit()
+    
+    def store_data(botex_db, session_id, url, conv, bot_parms):
+        """
+        Store the conversation data in the BotEx database.
+
+        Parameters:
+        botex_db (str): The name of the SQLite database file to store BotEx data.
+        session_id (str): The ID of the oTree session.
+        url (str): The participant URL of the bot instance.
+        conv (list): The conversation data.
+        bot_parms (str): The parameters of the bot.
+
+        Returns: None
+        """
+        conn = sqlite3.connect(botex_db)
+        cursor = conn.cursor()
+        cursor.execute(
+            """
+            INSERT INTO conversations (id, bot_parms, conversation) 
+            VALUES (?, ?, ?)
+            """, (url[-8:], bot_parms, json.dumps(conv))
+        )
+        conn.commit()
+        cursor.execute(
+            """
+            UPDATE participants SET time_out = ? 
+            WHERE session_id = ? and url = ?
+            """, 
+            (datetime.now(timezone.utc).isoformat(), session_id, url)
+        )
+        conn.commit()
+        cursor.close()
+        conn.close()
 
     
     conn = sqlite3.connect(botex_db)
@@ -420,10 +472,22 @@ def check_response_end(resp):
         next(rv)
         prompts = dict()
         for row in rv: prompts[row[0]] = row[1]
+    
+
+    if user_prompts:
+        for key in user_prompts:
+            if key not in prompts.keys():
+                logging.error(f"The bot is exiting because the user prompt that you provided has a key: '{key}' that is not expected by the bot. Please make sure that any default prompts that you want to override are given with the exact same key as the default prompt.")
+                return
+            else:
+                prompts[key] = user_prompts[key]
 
     message = prompts['start']
     conv = []
     resp = llm_send_message(message, conv, check_response_start)
+    if resp == 'Maximum number of attempts reached.':
+        gracefully_exit_failed_bot("start")
+        return
     logging.info(f"Bot's response to start message: '{resp}'")
     
     options = Options()
@@ -450,9 +514,24 @@ def check_response_end(resp):
     text = ""
     while True:
         old_text = text
-        text, wait_page, next_button, questions = scan_page(dr)
+        attempts = 0
+        while attempts < 5:
+            try:
+                text, wait_page, next_button, questions = scan_page(dr)
+                break
+            except:
+                attempts += 1
+                logging.warning("Failed to scrape my oTree URL. Trying again.")
+                if attempts == 5:
+                    logging.error("Could not scrape my oTree URL after 5 attempts. Stopping.")
+                    gracefully_exit_failed_bot("middle")
+                    return
+                time.sleep(1)
         if wait_page:
-            wait_next_page(dr)
+            wait_result = wait_next_page(dr)
+            if wait_result == 'Timeout on wait page after 600 attempts.':
+                gracefully_exit_failed_bot("abandoned")
+                return
             continue
 
         if full_conv_history:
@@ -500,6 +579,10 @@ def check_response_end(resp):
             message = prompts['page_not_changed'] + message
             
         resp = llm_send_message(message, conv, check_response, questions=questions)
+        if resp == 'Maximum number of attempts reached.':
+            gracefully_exit_failed_bot("middle")
+            return
+
         logging.info(f"Bot analysis of page: '{resp}'")
         if not full_conv_history: summary = resp['summary']
         if questions is None and next_button is not None:
@@ -547,26 +630,15 @@ def check_response_end(resp):
     dr.quit()
     message = prompts['end'].format(summary = summary)
     resp = llm_send_message(message, conv, check_response_end)
+    if resp == 'Maximum number of attempts reached.':
+        gracefully_exit_failed_bot("end")
+        return
     logging.info(f"Bot's final remarks about experiment: '{resp}'")
     logging.info("Bot finished.")
-        
-    conn = sqlite3.connect(botex_db)
-    cursor = conn.cursor()
-    cursor.execute(
-        """
-        INSERT INTO conversations (id, bot_parms, conversation) 
-        VALUES (?, ?, ?)
-        """, (url[-8:], bot_parms, json.dumps(conv))
-    )
-    conn.commit()
-    cursor.execute(
-        """
-        UPDATE participants SET time_out = ? 
-        WHERE session_id = ? and url = ?
-        """, 
-        (datetime.now(timezone.utc).isoformat(), session_id, url)
-    )
-    conn.commit()
-    cursor.close()
-    conn.close()
+    store_data(botex_db, session_id, url, conv, bot_parms)
+
+
+
+
+
 
diff --git a/src/botex/bot_prompts.csv b/src/botex/bot_prompts.csv
index 2ef60ab..f783b41 100644
--- a/src/botex/bot_prompts.csv
+++ b/src/botex/bot_prompts.csv
@@ -1,12 +1,12 @@
 id,prompt
-system,"You are particpating in an online survey and/or experiment. Each prompt contains a summary of the survey/experiment including your answers so far, scraped text data from a webpage continuing the survey/experiment, and detailed tasks for you on how to analyze this text data. The materials might contain information on how participants are being compensated or paid for their participation. If this is the case, please act as if this compensation also applies to you and make sure to include this information in the summary. Answers must be given as JSON code ONLY. No text outside of the JSON answer is allowed at any time. In each prompt, I will provide you with detailed information on the respective format."
+system,"You are participating in an online survey and/or experiment. Each prompt contains a summary of the survey/experiment including your answers so far, scraped text data from a webpage continuing the survey/experiment, and detailed tasks for you on how to analyze this text data. The materials might contain information on how participants are being compensated or paid for their participation. If this is the case, please act as if this compensation also applies to you and make sure to include this information in the summary. Answers must be given as JSON code ONLY. No text outside of the JSON answer is allowed at any time. In each prompt, I will provide you with detailed information on the respective format."
 start,"Hi there! In this conversation. I want you to participate in an online survey and/or experiment, potentially involving other human or artificial participants. I will provide you with a series of web page body text excerpts that will sequentially guide you through the experiment/survey. The texts will contain instructions on how the experiment/survey will be conducted. These instructions will be followed by additional pages that might contain additional instructions, comprehension checks, repeated information from prior pages and, most importantly, present you with questions and/or tasks which I want you to answer. All answers must be given as JSON code ONLY and I will provide you with detailed information on the respective format. As all texts are scraped from the body parts of web pages, they might contain also additional information or text bits that do not really belong to the experiment. Do you understand your task? Please answer by setting the value of 'understood' to 'yes' or 'no', meaning that {""understood"": ""yes""} indicates that you understand. Please provide your answer ONLY as JSON. If you are confused, respond with {""error"": ""I am confused""}."
 analyze_page_no_q_full_hist,"Perfect. You have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to summarize the content of the page and indicate what you learn from it about the survey/experiment that you are participating in. Provide the summary as a string variable 'summary' in a JSON string.Please note that, while you can use JSON to structure the summary content, the variable itself must be provided as a string." 
-analyze_page_q_full_hist,"Perfect. You have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to summarize the content of the page and indicate what you learn from it about the survey/experiment that you are participating in. Provide the summary as the variable 'summary'. Second, this page contains {nr_q} question(s) that I want you to answer in the variable 'questions'. So, your JSON answer will contain the variables 'summary', and 'questions'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'),  and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answwer choices provided, please answer with the full answer choice text. Do not answer with just a number indicating the answer choice. If there are no answer choices provided, it means you can answer the question freely. You might find more context about the questions in the body text quoted above. In particular, you will find any answer options in the text above. Please strictly adhere to the answer options given above. If the question type is 'number', please only provide an integer number.  This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'answer' contains your response, and 'reason' contains your reasoning for the response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""summary"": ""Your summary"", ""questions"": [{{""id"": ""ID of first question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}, {{""id"": ""ID of second question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}]}} Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
+analyze_page_q_full_hist,"Perfect. You have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to do two things. First, this page contains {nr_q} question(s) and I need you to answer all questions in the variable 'questions'. Second, I need you to summarize the content of the page including all questions and your answer. Provide the summary as the variable 'summary'. More on this below. So, your JSON answer will contain the variables 'questions', and 'summary'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'),  and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answer choices provided, first answer with the full answer choice text and do not just answer with a number indicating the answer choice. Second please strictly adhere to the given answer choices, please do not select something outside of the given answer choices. If there are no answer choices provided, it means you can answer the question freely. If the question type is 'number', please only provide an integer number. This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'reason' contains your reasoning or thought that leads you to a response or answer and finally, 'answer' which contains your response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""questions"": [{{""id"": ""ID of first question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}, {{""id"": ""ID of second question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}], ""summary"": ""Your summary""}}.Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
 analyze_first_page_no_q,"Perfect. This is the body text of the entry page of the survey/experiment:\n\n{body}\n\nI need you to summarize the content of the page and indicate what you learn from it about the survey/experiment that you are participating in. Provide the summary as the string variable 'summary' in a JSON string. Please summarize all information from the web page. Try to be very precise and detailed. Please provide your answer ONLY as JSON.  Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
 analyze_page_no_q,"Perfect. This is your summary of the survey/experiment so far:\n\n{summary}\n\nYou have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to summarize the information from this page and to incorporate this summary into the summary provided above. Taken together, the new summary should summarize everything you know about the experiment. The summary needs to be stored in the variable 'summary'.So, your JSON answer will contain only the variable 'summary'. Please provide your answer ONLY as JSON.  Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
-analyze_first_page_q,"Perfect. This is the body text of the entry page of your survey or experiment:\n\n{body}\n\nI need you to do two things. First, this page contains {nr_q} question(s) and I need you to answer all questions in the variable 'questions'. Second, I need you to summarize the content of the page including all questions, and your answer. Provide the summary as the variable 'summary'. More on this below. So, your JSON answer will contain the variables 'questions', and 'summary'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'),  and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answwer choices provided, please answer with the full answer choice text. Do not answer with just a number indicating the answer choice. If there are no answer choices provided, it means you can answer the question freely. You might find more context about the questions in the body text quoted above. In particular, you will find any answer options in the text above. Please strictly adhere to the answer options given above. If the question type is 'number', please only provide an integer number.  This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'answer' contains your response, and 'reason' contains your reasoning for the response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""summary"": ""Your summary"", ""questions"": [{{""id"": ""ID of first question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}, {{""id"": ""ID of second question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}]}} Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
-analyze_page_q,"Perfect. This is your summary of the survey/experiment so far:\n\n{summary}\n\nYou have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to do two things. First, this page contains {nr_q} questions and I need you to answer all questions in the variable 'questions'. Second, I need you to summarize the content of the page including the old summary given above, all questions, and your answer. Provide the summary as the variable 'summary'. More on this below. So, your JSON answer will contain the variables 'questions', and 'summary'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'), and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answwer choices provided, please answer with the full answer choice text. Do not answer with just a number indicating the answer choice. If there are no answer choices provided, it means you can answer the question freely. You might find more context about the questions in the body text quoted above. In particular, you will find any answer options in the text above. Please strictly adhere to the answer options given above. If the question type is 'number', please only provide an integer number. This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'answer' contains your response, and 'reason' contains your reasoning for the response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""summary"": ""Your summary"", ""questions"": [{{""id"": ""ID of first question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}, {{""id"": ""ID of second question"", ""answer"": ""Your answer"", ""reason"": ""Your reasoning""}}]}} Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string.  If you are confused, respond with {{""error"": ""I am confused""}}." 
+analyze_first_page_q,"Perfect. This is the body text of the entry page of your survey or experiment:\n\n{body}\n\nI need you to do two things. First, this page contains {nr_q} question(s) and I need you to answer all questions in the variable 'questions'. Second, I need you to summarize the content of the page including all questions and your answer. Provide the summary as the variable 'summary'. More on this below. So, your JSON answer will contain the variables 'questions', and 'summary'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'),  and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answer choices provided, first answer with the full answer choice text and do not just answer with a number indicating the answer choice. Second please strictly adhere to the given answer choices, please do not select something outside of the given answer choices. If there are no answer choices provided, it means you can answer the question freely. If the question type is 'number', please only provide an integer number. This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'reason' contains your reasoning or thought that leads you to a response or answer and finally, 'answer' which contains your response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""questions"": [{{""id"": ""ID of first question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}, {{""id"": ""ID of second question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}], ""summary"": ""Your summary""}}. Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}." 
+analyze_page_q,"Perfect. This is your summary of the survey/experiment so far:\n\n{summary}\n\nYou have now proceeded to the next page. This is the body text of the web page:\n\n{body}\n\nI need you to do two things. First, this page contains {nr_q} question(s) and I need you to answer all questions in the variable 'questions'. Second, I need you to summarize the content of the page including the old summary given above, all questions, and your answer. Provide the summary as the variable 'summary'. More on this below. So, your JSON answer will contain the variables 'questions', and 'summary'. I need you to answer all questions. The following JSON string contains the questions: '{questions_json}'. Each question is characterized by an ID ('question_id'), question type ('question_type'), and a label ('question_label'). Additionally, if the question type requires it, you will also be provided with answer choices. If there are answer choices provided, first answer with the full answer choice text and do not just answer with a number indicating the answer choice. Second please strictly adhere to the given answer choices, please do not select something outside of the given answer choices. If there are no answer choices provided, it means you can answer the question freely. If the question type is 'number', please only provide an integer number. This means that for each identified question, you must provide three variables: 'id' gives the question ID, 'reason' contains your reasoning or thought that leads you to a response or answer and finally, 'answer' which contains your response. The above implies that each question must be answered with a dict of these three variables and, 'questions' becomes a list of dicts if there is more than one question. Taken together, a correct answer to a text with two questions would have the form {{""questions"": [{{""id"": ""ID of first question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}, {{""id"": ""ID of second question"", ""reason"": ""Your reasoning"", ""answer"": ""Your answer""}}], ""summary"": ""Your summary""}}. Please provide your answer ONLY as JSON. Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string.  If you are confused, respond with {{""error"": ""I am confused""}}." 
 end,"Perfect. This concludes the survey/experiment and our conversation. This is your summary of the completed survey/experiment:\n\n{summary}\n\n. Do you have any final remarks about the survey/experiment and how you answered it, or about its implementation in this conversation that you want to share? Then please provide your answer as JSON, using the variable 'remarks'. If you have no final remarks, respond with {{remarks: 'none'}}. If you are confused, respond with {{""error"": ""I am confused""}}."
 json_error,"Unfortunately, your response was not in the correct JSON format. Please try again.  Please note that, while you can use JSON to structure the summary content, the 'summary' variable itself must be provided as a string. If you are confused, respond with {{""error"": ""I am confused""}}."
 confused,"I am sorry that you are confused. Please focus on the tasks, the required JSON structure and see whether you can answer it now. Try to keep the summary detailed but short enough so that you stay within the token limit.\n\n"
@@ -14,16 +14,15 @@ resp_too_long,"I am sorry but your response exceeded the token limit. Try to kee
 page_not_changed,"I am sorry but you answer(s) have not been accepted by the system. Most likely this is because the answer(s) do not comply with the required format or, if they were numeric, are not within the valid numerical range. Please evaluate the page below again and provide your refined answer(s).\n\n"
 not_understood,"I am sorry that you did not understand the task. Try to focus on the task and keep the JSON structure in mind. If you are still confused or do not understand, please respond with {{""error"": ""I am confused""}}.\n\n"
 no_understood_key,"I am sorry but your response did not include the understood key. Please review your answer and provide it again with all the necessary information"
-no_summary_key,"I am sorry but your response did not include the summary key as requested. Please review your answer and provide it again in a JSON format with a summary key, with a string value and a question key with an array value.
-I am sorry but your response did not include a summary. Please review your answer and provide it again with all the necessary information in a JSON format."
+no_summary_key,"I am sorry but your response did not include the summary key as requested. Please review your answer and provide it again in a JSON format with a summary key, with a string value and a question key with an array value."
 no_questions_key,"I am sorry but your response did not include the questions array as requested. Please review your answer and provide it again in a JSON format with a summary key, with a string value and a question key with an array value."
 questions_not_list,"I am sorry but your response did not have the questions element as a list as requested. Please review your answer and provide it again with all the necessary information in a JSON format."
-unanswered_questions,"I am sorry but your response did not include all the questions required. Specifically, the question[s] with the ID[s] {unanswered_questions} were missing. Please review your answer and provide it again with all the necessary information."
-missing_answer_keys,"Your response was not valid because some question[s] were missing key[s]. Specifically, these were: {missing_answer_keys}. Please review your answer and provide it again with all the necessary information."
-missing_answer,"Your response was not valid because you answered some question[s] with a blank. Specifically, these were: {missing_answer}. Remember that, Please review your answer and provide it again with all the necessary information."
-missing_reason,"Your response was not valid because some question[s] were missing a reason for the answer. Specifically, these were: {missing_reason}. Please review your answer and provide it again with all the necessary information."
-answer_not_number,"Your response was not valid because the answer to some question[s] with question_type number was not a number. Specifically, these were: {answer_not_number}. Please review your answer and provide it again with a valid number answer, along with all the information required."
-answer_not_float,"Your response was not valid because the answer to some question[s] with question_type float was not a float. Specifically, these were: {answer_not_float}. Please review your answer and provide it again with a valid number answer, along with all the information required."
-select_answer_number,"Your response was not valid because the answer to some question[s] with question_type select-one was a number. Specifically, the questions with id {select_answer_number} expected a full text answer but you provided a number. Please review your answer and provide it again with a full text answer from the answer options, along with all the information required."
+unanswered_questions,"I am sorry but your response did not include all the questions required. Specifically, the question[s] with the ID[s] {unanswered_questions} were missing. You need to make sure that you provide the answer with the id key, i.e. {{'id': the id of the question}}. Please review your answer and provide it again with all the necessary information."
+missing_answer,"Your response was not valid because you answered some question[s] with a blank. Specifically, these were: {missing_answer}. You will need to provide an answer for this in order to pass this task and be helpful. Please review your answer and provide it again with all the necessary information."
+missing_reason,"Your response was not valid because some question[s] were missing a reason for the answer. Specifically, these were the questions with the id: {missing_reason}.  You will need to provide a reason for {missing_reason} in order to pass this task and be helpful. Please review your answer and provide it again with all the necessary information."
+answer_not_number,"Your response was not valid because the answer to some question[s] with question_type number was not a number. Specifically, these were: {answer_not_number}. Remember, only a number is acceptable answer here. Please review your answer and provide it again with a valid number answer, along with all the information required."
+answer_not_float,"Your response was not valid because the answer to some question[s] with question_type float was not a float. Specifically, these were: {answer_not_float}. Remember, only a float is acceptable answer here. Please review your answer and provide it again with a valid number answer, along with all the information required."
+select_answer_number,"Your response was not valid because the answer to some question[s] with question_type select-one was a number. Specifically, the questions with id {select_answer_number} expected a full text answer from the provided answer choices but you answered with a number probably indicating the index of the answer. Please review your answer and provide it again with the full text answer from the answer options, along with all the information required."
 select_answer_unknown,"Your response was not valid because the answer to some question[s] with question_type select-one was not a valid answer option. Specifically, these were: {select_answer_unknown}. Please review your answer and provide it again with a valid answer from the answer options, along with all the information required."
-no_remarks_key,"I am sorry but your response did not include the remarks key as requested. Please review your answer and provide it again in a JSON format with a remarks key, with a string value."
\ No newline at end of file
+no_remarks_key,"I am sorry but your response did not include the remarks key as requested. Please review your answer and provide it again in a JSON format with a remarks key, with a string value."
+answer_id_not_found_in_q_id_list,"Your response was not valid because the ids of the answers you provided do not match the ids of the questions. Specifically, these were: {answer_id_not_found_in_q_id_list}. Please review your answer and provide it again with the correct ids, along with all the information required."
\ No newline at end of file
diff --git a/src/botex/gguf_parser.py b/src/botex/gguf_parser.py
new file mode 100644
index 0000000..5d85692
--- /dev/null
+++ b/src/botex/gguf_parser.py
@@ -0,0 +1,100 @@
+import struct
+
+class GGUFParser:
+    def __init__(self, gguf_file_path):
+        self.gguf_file_path = gguf_file_path
+    
+
+    def get_metadata(self):
+        meta_data = {}
+        with open(self.gguf_file_path, 'rb') as f:
+            magic_number = f.read(4)
+            if magic_number != b"GGUF":
+                raise ValueError("Invalid GGUF file")
+
+            version = struct.unpack("I", f.read(4))[0]
+            if version != 3:
+                raise ValueError("Unsupported GGUF version")
+
+            _ = struct.unpack("Q", f.read(8))[0]
+            metadata_kv_count = struct.unpack("Q", f.read(8))[0]
+
+            for _ in range(metadata_kv_count):
+                key = GGUFParser.read_string(f)
+                value_type = struct.unpack("I", f.read(4))[0]
+                value = GGUFParser.read_value(f, value_type)
+
+                if key == 'tokenizer.chat_template':
+                    meta_data['chat_template'] = value
+                
+                if key == 'tokenizer.ggml.add_bos_token':
+                    meta_data['add_bos_token'] = value
+                
+                if key == 'tokenizer.ggml.add_eos_token':
+                    meta_data['add_eos_token'] = value
+                
+                if 'context_length' in key:
+                    meta_data['context_length'] = value
+                
+                if key == 'tokenizer.ggml.tokens':
+                    tokens = value
+                
+                if key == 'tokenizer.ggml.bos_token_id':
+                    meta_data['bos_token'] = tokens[value]
+
+                if key == 'tokenizer.ggml.eos_token_id':
+                    meta_data['eos_token'] = tokens[value]
+            
+        
+        return meta_data
+
+    @staticmethod
+    def read_string(file):
+        length = struct.unpack("Q", file.read(8))[0]
+        return file.read(length).decode('utf-8')
+    
+    @staticmethod
+    def read_array(file, element_type):
+        length = struct.unpack("Q", file.read(8))[0]
+        if element_type == "str":
+            return [GGUFParser.read_string(file) for _ in range(length)]
+        elif element_type == "f32":
+            return struct.unpack(f"{length}f", file.read(length * 4))
+        elif element_type == "i32":
+            return struct.unpack(f"{length}i", file.read(length * 4))
+        else:
+            raise ValueError(f"Unsupported array element type: {element_type}")
+        
+    @staticmethod
+    def read_value(file, value_type):
+        VALUE_FORMATS = {
+            0: "B",  # UINT8
+            1: "b",  # INT8
+            2: "H",  # UINT16
+            3: "h",  # INT16
+            4: "I",  # UINT32
+            5: "i",  # INT32
+            6: "f",  # FLOAT32
+            7: "?",  # BOOL
+            10: "Q", # UINT64
+            11: "q", # INT64
+            12: "d", # FLOAT64
+            8: GGUFParser.read_string,  # STRING
+            9: GGUFParser.read_array,   # ARRAY
+        }
+        if value_type in VALUE_FORMATS:
+            if callable(VALUE_FORMATS[value_type]):
+                if value_type == 9:
+                    array_type = struct.unpack("I", file.read(4))[0]
+                    if array_type == 8:
+                        return GGUFParser.read_array(file, "str")
+                    elif array_type == 6:
+                        return GGUFParser.read_array(file, "f32")
+                    elif array_type == 5:
+                        return GGUFParser.read_array(file, "i32")
+                    else:
+                        raise ValueError(f"Unsupported array type: {array_type}")
+                return VALUE_FORMATS[value_type](file)
+            return struct.unpack(VALUE_FORMATS[value_type], file.read(struct.calcsize(VALUE_FORMATS[value_type])))[0]
+        else:
+            raise ValueError("Unsupported value type")
\ No newline at end of file
diff --git a/src/botex/local_llm.py b/src/botex/local_llm.py
index fc55d6d..3feca06 100644
--- a/src/botex/local_llm.py
+++ b/src/botex/local_llm.py
@@ -1,10 +1,14 @@
 import logging
 from jinja2 import Template
-import json
 import os
-import re
+import psutil
+import requests
 import subprocess
+import time
 from typing import List, Union, Iterable
+from urllib.parse import urlparse
+
+from .gguf_parser import GGUFParser
 
 
 class Message(dict):
@@ -17,10 +21,7 @@ class Message(dict):
     """
 
     def __init__(self, role: str, content: str) -> None:
-        if isinstance(content, str) and content != "":
-            super().__init__({"role": role, "content": content})
-        else:
-            raise ValueError("Content must be a non-empty string.")
+        super().__init__({"role": role, "content": content})
         if isinstance(role, str) and role in ["user", "assistant", "system"]:
             self.role = role
         else:
@@ -141,33 +142,139 @@ class LocalLLM:
 
     def __init__(
         self,
-        path_to_compiled_llama_cpp_main_file: str,
+        path_to_compiled_llama_server_executable: str,
         local_model_path: str,
+        api_base_url: str = "http://localhost:8080",
+        context_length: int | None = None,
         system_prompt_few_shot_examples: ChatHistory | None = None,
         has_system_role: bool = False,
-        ngl: int = 1,
-        c: int = 32768,
-        temp: float = 0,
-        n: int = 10000,
+        number_of_layers_to_offload_to_gpu: int = 1,
+        temperature: float = 0.5,
+        maximum_tokens_to_predict: int = 10000,
         top_p: float = 0.9,
         top_k: int = 40,
+        num_slots: int = 1,
+        **kwargs,
     ):
         self.system_prompt_few_shot_examples = system_prompt_few_shot_examples
-        self.main_path = path_to_compiled_llama_cpp_main_file
+        self.server_path = path_to_compiled_llama_server_executable
         self.model_path = local_model_path
-        self.has_system_role = has_system_role
-        self.ngl = ngl
-        self.c = c
-        self.temp = temp
-        self.n = n
+        self.api_base_url = api_base_url
+        parsed_gguf = GGUFParser(self.model_path)
+        self.metadata = parsed_gguf.get_metadata()
+        self.has_system_role = (
+            has_system_role
+            if isinstance(has_system_role, bool)
+            else eval(has_system_role)
+        )
+        self.ngl = number_of_layers_to_offload_to_gpu
+        self.c = context_length if context_length else self.metadata.get("context_length", 4096)
+        self.temp = temperature
+        self.n = maximum_tokens_to_predict
         self.top_p = top_p
         self.top_k = top_k
+        self.num_slots = num_slots
+
+    def validate_parameters(self):
+        if not os.path.exists(self.server_path):
+            raise FileNotFoundError(f"Server path not found: {self.server_path}")
+        if not os.path.exists(self.model_path):
+            raise FileNotFoundError(f"Model path not found: {self.model_path}")
+
+    def start_server(self):
+        """
+        Starts the local language model server.
+        """
+        self.validate_parameters()
+
+        parsed_url = urlparse(self.api_base_url)
+        cmd = [
+            self.server_path,
+            "--host",
+            parsed_url.hostname,
+            "--port",
+            str(parsed_url.port),
+            "-ngl",
+            str(self.ngl),
+            "--reverse_prompt",
+            "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+            "--reverse_prompt",
+            "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+            "--reverse_prompt",
+            " \n  \n  \n  \n  \n  \n  \n  \n  ",
+            "-m",
+            self.model_path,
+            "-c",
+            str(int(self.num_slots) * (int(self.c) + int(self.n))),
+            "-n",
+            str(self.n),
+            "--parallel",
+            str(self.num_slots),
+            "-fa",
+        ]
+        logging.info(f"Starting LLM server ...")
+        with open("llama_cpp_server.log", "a") as log_file:
+            process = subprocess.Popen(cmd, stdout=log_file, stderr=subprocess.PIPE)
+
+        if self.wait_for_server(parsed_url.hostname, parsed_url.port):
+            logging.info("LLM server started successfully.")
+            return process
+
+        _, stderr = process.communicate(timeout=10)
+        if process.returncode != 0 and stderr:
+            logging.error("Failed to start the LLM server.")
+            if "cudaMalloc failed: out of memory" in stderr.decode("utf-8"):
+                raise Exception(
+                    f"Failed to start the LLM server. Your model {self.model_path} is too large for the available GPU memory. Please try a smaller model or decrease the number of layers, {self.ngl} that are offloaded to the GPU."
+                )
+            raise Exception(
+                f"Failed to start the LLM server. Error: {stderr.decode('utf-8')}"
+            )
+
+        if self.wait_for_server(parsed_url.hostname, parsed_url.port):
+            logging.info("LLM server started successfully.")
+            return process
+        else:
+            logging.error("Failed to start the LLM server.")
+            process.terminate()
+            return None
+
+    def wait_for_server(self, host, port, timeout=10):
+        """
+        Waits for the server to become responsive.
+        """
+        url = f"http://{host}:{port}/health"
+        start_time = time.time()
+        while time.time() - start_time < timeout:
+            try:
+                response = requests.get(url)
+                if response.status_code == 200:
+                    return True
+            except requests.ConnectionError:
+                time.sleep(1)
+        return False
+
+    def stop_server(self, process: subprocess.Popen):
+        """
+        Stops the local language model server.
+        """
+        if process:
+            logging.info("Stopping server...")
+            parent = psutil.Process(process.pid)
+            for child in parent.children(recursive=True):
+                child.terminate()
+            parent.terminate()
+
+            process.wait()
+            logging.info("Server stopped.")
+        else:
+            logging.warning("Server is not running.")
 
     def prepare_prompt(
         self,
         chat_history: ChatHistory,
         question: Union[HumanMessage, str] | None = None,
-    ) -> None:
+    ) -> str:
         """
         Prepares a full prompt from a given question and chat history by applying formatting to fit the model's template.
 
@@ -196,36 +303,25 @@ def format_prompt_to_template(self, messages: ChatHistory) -> str:
         :param messages: The chat history instance to format.
         :raises FileNotFoundError: If essential tokenizer configuration files are missing.
         """
-        model_folder = "/".join(self.model_path.split("/")[:-1])
-        files = os.listdir(model_folder)
-        if "tokenizer_config.json" not in files:
-            raise FileNotFoundError(
-                "No tokenizer found in the model path. Please ensure the model path is correct and that it has a tokenizer_config.json file."
-            )
-        with open(model_folder + "/tokenizer_config.json", "r") as f:
-            token_config = json.load(f)
-        self.start_token = token_config["bos_token"]
-        self.end_token = token_config["eos_token"]
-
         # using Mistral 7b v3 instruct template as default
         default_chat_template = """
         {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '
 ' + message['content'] + ' 
' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}
         """
 
-        if "chat_template" not in token_config:
+        if not self.metadata.get("chat_template"):
             logging.warning(
-                "No chat template found in the tokenizer config. Using default template from Mistral 7b v3 instruct. This might severely affect the model's performance. Also keep in mind there is no system role in the default template."
+                "No chat template found in the metadata of the model. Using default template from Mistral 7b v3 instruct. This might severely affect the model's performance. Also keep in mind there is no system role in the default template."
             )
-            token_config["chat_template"] = default_chat_template
+            self.metadata["chat_template"] = default_chat_template
             self.has_system_role = False
 
-        template = Template(token_config["chat_template"])
+        template = Template(self.metadata["chat_template"])  # type: ignore
 
         if not self.has_system_role:
             messages = self.push_system_message_to_user(messages)
 
         return template.render(
-            messages=messages, add_generation_prompt=True, **token_config
+            messages=messages, add_generation_prompt=True, **self.metadata
         )
 
     @staticmethod
@@ -244,41 +340,6 @@ def push_system_message_to_user(messages: ChatHistory) -> ChatHistory:
                 messages.pop(i)
         return messages
 
-    def llm_setup(self, prompt) -> List[str]:
-        """
-        Sets up the command line arguments for running the local language model based on the current state and model configuration.
-
-        :return: A list of command line arguments to execute the language model.
-        """
-        return [
-            self.main_path,
-            "-ngl",
-            str(self.ngl),
-            "-m",
-            self.model_path,
-            "-c",
-            str(self.c),
-            "--temp",
-            str(self.temp),
-            "--no-display-prompt",
-            "--grammar-file",
-            self.main_path.replace("/main", "/grammars/json.gbnf"),
-            "-n",
-            str(self.n),
-            "--reverse_prompt",
-            "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
-            "--reverse_prompt",
-            "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
-            "--reverse_prompt",
-            " \n  \n  \n  \n  \n  \n  \n  \n  ",
-            "--top-p",
-            str(self.top_p),
-            "--top-k",
-            str(self.top_k),
-            "-p",
-            prompt,  # type: ignore
-        ]
-
     def completion(self, messages) -> MirrorLiteLLMResponse:
         """
         Generates a completion by setting up the prompt, running the local language model, and processing its output.
@@ -289,35 +350,36 @@ def completion(self, messages) -> MirrorLiteLLMResponse:
         :raises Exception: If there is an error during the execution of the model command.
         """
         prompt = self.prepare_prompt(messages)
-        cmd = self.llm_setup(prompt)
 
-        result = subprocess.run(cmd, capture_output=True, text=True)
+        url = f"{self.api_base_url}/completion"
 
-        if result.returncode != 0:
+        payload = {
+            "prompt": prompt,
+            "temperature": self.temp,
+            "cache_prompt": False,
+            "max_tokens": self.n,
+        }
+
+        attempts = 0
+        while True:
+            try:
+                response = requests.post(url, json=payload, timeout=300)
+                break
+            except requests.Timeout:
+                logging.error("Request timed out. Retrying...")
+                attempts += 1
+                if attempts == 3:
+                    raise Exception("Request timed out after 3 attempts.")
+                
+
+        if response.status_code != 200:
             raise Exception(
-                "There was an error running the command. The error was: %s"
-                % result.stderr
+                f"An error occurred while generating the completion: {response.text}"
             )
 
-        match = re.search(
-            r"sample time =\s*\d+,\d+\s*ms\s*/\s*(\d+)\s*runs", result.stderr
-        )
-        assert match is not None, "Could not find completion tokens in the output."
-        completion_tokens = int(match.group(1))
-        finish_reason = "stop" if completion_tokens < self.n else "length"
-
-        completion = result.stdout.strip()
-
-        completion = completion.replace(self.start_token, "").replace(
-            self.end_token, ""
-        )
-
-        # hacky
-        if "Llama-3" in self.model_path:
-            completion = completion.replace("
", "")
+        completion = response.json()["content"]
+        finish_reason = "length" if response.json()["stopped_limit"] else "stop"
 
-        # hacky
-        if "Phi-3" in self.model_path:
-            completion = completion.replace("
", "")
+        time.sleep(1)
 
         return MirrorLiteLLMResponse(completion, finish_reason)
diff --git a/src/botex/otree.py b/src/botex/otree.py
index 56aa85d..a149da9 100644
--- a/src/botex/otree.py
+++ b/src/botex/otree.py
@@ -1,6 +1,6 @@
 from os import environ
 import sqlite3
-from threading import Lock, Thread
+from threading import Thread
 from random import sample, shuffle
 import logging
 logging.basicConfig(level=logging.INFO)
@@ -205,7 +205,8 @@ def run_bots_on_session(
         openai_api_key = None,
         already_started = False,
         wait = True,
-        local_model_cfg={}
+        local_model_cfg={},
+        user_prompts: dict | None = None
     ):
     """
     Run BotEx bots on an oTree session.
@@ -230,7 +231,8 @@ def run_bots_on_session(
         startup properly because of network issues. Default is False.
     wait (bool): If True (the default), the function will wait for the bots to 
         finish.
-    local_model_cfg (dict): Configuration for the local model. If model is "local", as a bare minimum it should contain, the "path_to_compiled_llama_cpp_main_file", and "local_model_path" keys.
+    local_model_cfg (dict): Configuration for the local model. If model is "local", as a bare minimum it should contain, the "path_to_compiled_llama_server_executable", and "local_model_path" keys.
+    user_prompts (dict): A dictionary of user prompts to override the default prompts that the bot uses. The keys should be one or more of the following: ['start', 'analyze_first_page_no_q', 'analyze_first_page_q', 'analyze_page_no_q', 'analyze_page_q', 'analyze_page_no_q_full_hist', 'analyze_page_q_full_hist', 'page_not_changed', 'system', 'resp_too_long', 'json_error', 'end'.] If a key is not present in the dictionary, the default prompt will be used. If a key that is not in the default prompts is present in the dictionary, then the bot will exit with a warning and not running to make sure that the user is aware of the issue.
 
     Returns: None (bot conversation logs are stored in database)
     """
@@ -239,9 +241,9 @@ def run_bots_on_session(
     if openai_api_key is None: openai_api_key = environ.get('OPENAI_API_KEY')
     if bot_urls is None: 
         bot_urls = get_bot_urls(session_id, botex_db, already_started)
-    lock = Lock()
     if model == "local":
         local_llm = LocalLLM(**local_model_cfg)
+        llm_server = local_llm.start_server()
     else:
         local_llm = None 
     threads = [
@@ -249,13 +251,17 @@ def run_bots_on_session(
             target = run_bot, 
             kwargs = {
                 'botex_db': botex_db, 'session_id': session_id, 
-                'url': url, 'lock': lock, 'full_conv_history': full_conv_history, 'model': model, 'openai_api_key': openai_api_key, 'local_llm': local_llm
+                'url': url, 'full_conv_history': full_conv_history, 'model': model, 'openai_api_key': openai_api_key, 'local_llm': local_llm, 'user_prompts': user_prompts
             }
         ) for url in bot_urls 
     ]
     for t in threads: t.start()
     if wait: 
         for t in threads: t.join()
+    
+    if local_llm:
+        assert llm_server, "Local LLM server not started, but should have been."
+        local_llm.stop_server(llm_server)
 
 
 if __name__ == '__main__':


Test patch:
diff --git a/tests/test_c_local_llm.py b/tests/test_c_local_llm.py
index 3b02d21..e83b3ce 100644
--- a/tests/test_c_local_llm.py
+++ b/tests/test_c_local_llm.py
@@ -9,39 +9,44 @@
 from tests.utils import start_otree, stop_otree, init_otree_test_session, delete_botex_db
 
 with open("secrets.env") as f:
+    cfg = {}
     for line in f:
-        if "path_to_compiled_llama_cpp_main_file" in line:
-            path_to_compiled_llama_cpp_main_file = line.split("=")[1].strip()
-        if "local_model_path" in line:
-            local_model_path = line.split("=")[1].strip()
-        if "number_of_layers_to_offload_to_gpu" in line:
-            number_of_layers_to_offload_to_gpu = int(line.split("=")[1].strip())
+        if not line.strip() or line.startswith("#"):
+            continue
+        key, value = line.strip().split("=")
+        cfg[key] = value
 
+user_prompts = {
+    "system": "You are participating in an online survey and/or experiment. Please fully assume this role and answer everything from the first person point of view. Each prompt contains a summary of the survey/experiment including your answers so far, scraped text data from a webpage continuing the survey/experiment, and detailed tasks for you on how to analyze this text data. The materials might contain information on how participants are being compensated or paid for their participation. If this is the case, please act as if this compensation also applies to you and make sure to include this information in the summary. Answers must be given as JSON code ONLY. No text outside of the JSON answer is allowed at any time. In each prompt, I will provide you with detailed information on the respective format."
+}
 
-@pytest.mark.dependency(name="llama_cpp_main_file", scope='session')
-def test_llama_cpp_main_file_exists():
-    assert os.path.exists(path_to_compiled_llama_cpp_main_file)
+@pytest.mark.dependency(name="llama_server_executable", scope='session')
+def test_llama_server_executable_exists():
+    assert os.path.exists(cfg["path_to_compiled_llama_server_executable"])
 
 @pytest.mark.dependency(name="local_model_path", scope='session')
 def test_local_model_path_exists():
-    assert os.path.exists(local_model_path)
+    assert os.path.exists(cfg["local_model_path"])
 
 
 @pytest.mark.dependency(name="num_layers_to_offload_to_gpu", scope='session')
 def test_number_of_layers_to_offload_to_gpu():
-    assert isinstance(number_of_layers_to_offload_to_gpu, int)
-    # TODO: a more specific test to see if there is a gpu to offload to
+    if cfg.get("number_of_layers_to_offload_to_gpu"):
+        assert isinstance(int(cfg["number_of_layers_to_offload_to_gpu"]), int)
+        # TODO: a more specific test to see if there is a gpu to offload to
 
 
-@pytest.mark.dependency(name="instantiate_local_llm", scope='session', depends=["llama_cpp_main_file", "local_model_path", "num_layers_to_offload_to_gpu"])
-def test_instantiate_local_llm():
-    local_llm = botex.LocalLLM(path_to_compiled_llama_cpp_main_file,local_model_path,
-    ngl=number_of_layers_to_offload_to_gpu
-    )
-    assert local_llm
-
-
-@pytest.mark.dependency(name="run_local_bots", scope='session', depends=["instantiate_local_llm", "botex_session", "botex_db"])
+@pytest.mark.dependency(
+        name="run_local_bots",
+        scope='session',
+        depends=[
+            "llama_server_executable",
+            "local_model_path",
+            "num_layers_to_offload_to_gpu",
+            "botex_session",
+            "botex_db"
+        ]
+)
 def test_can_survey_be_completed_by_local_bots():
     delete_botex_db()
     otree_proc = start_otree()
@@ -55,11 +60,8 @@ def test_can_survey_be_completed_by_local_bots():
         bot_urls=botex_session["bot_urls"],
         botex_db="tests/botex.db",
         model="local",
-        local_model_cfg={
-            "path_to_compiled_llama_cpp_main_file": path_to_compiled_llama_cpp_main_file,
-            "local_model_path": local_model_path,
-            "ngl": 85
-        }
+        local_model_cfg=cfg,
+        user_prompts=user_prompts
     )
     stop_otree(otree_proc)
     assert True