Package: seeingllama2
Interactive agent for visually impaired and blind individuals (based on Llama2)This project uses a tier application to run the Large Language Model and accesses it using an API.This is a prototype!The current tier application is based on Oobabooga's text generation webui. In the future support for Petals, or OpenAI API will be added.InstallationYou first need to setup the tier application server. Then you can install this server application on a different or on the same machine.Installation of the tier applicationHere Oobabooga's application is installed on an Ubuntu server equipped with 2 RTX 3090.For reference, I'm using theone click installerwith a script to set up the API and a Ngrok tunnel. I installed Ngrok via pip (python -m pip install pyngrok).#!/usr/bin/env bashtrap_with_arg(){# from https://stackoverflow.com/a/2183063/804678localfunc="$1";shiftforsigin"$@";dotrap"$func$sig""$sig"done}stop(){trap-SIGINTEXITprintf'\n%s\n'"received$1, killing child processes"kill-sSIGINT0}trap_with_arg'stop'EXITSIGINTSIGTERMSIGHUPexportOOBABOOGA_FLAGS="--extension api --loader exllama_hf --model  TheBloke_Llama-2-13B-chat-GPTQ --verbose --listen --chat --max_seq_len 4096 --compress_pos_emb 1"bash/path/to/oobabooga_linux/start_linux.sh&# Wait for the application to startsleep30s# Start the reverse proxy# --basic-auth 'api_token:your_custom_token'  is optional, it allows using basic access authentication# The username is 'api_token' and the password is 'your_custom_token'ngrokhttp--domain=my-ngrok-domain.ngrok-free.app5000--basic-auth'api_token:your_custom_token'&whiletrue;doread;doneThen I usescreento launch the script in the detached mode:# api is the session name, -dm is to start the session in detached mode.screen-dm-Sapibash~/start_api.shI can check if thescreensession is running with:screen-lsAnd I can kill the session using:screen-X-SapikillBut, I'm still battling with the API on the client-side, trying to generate a working prompt…Installation of this applicationNote that you might need to install espeak on your computer. You can install this application using pip (and git) via the following command:python3-mpipinstall--force-reinstallseeingllama2Or you can install it from the git repository:python3-mpipinstall--force-reinstallgit+https://github.com/Vincent-Stragier/c-bot.gitConfigurationTo simplify the configuration, multiple settings can be changed using a YAML file. Then you can start the application using the following command:seeingllama2-c/path/to/config.yamlconfig.yaml---flask:# Use the debug functionality of Flaskdebug:true# Listen on port 80port:80# Listen from any interface.# You can use "127.0.0.1" when using a reverse proxy.host:"0.0.0.0"# You can configure Flask to use an SSL certificate.# To generate this certificate, you need an hostname, and# a service like Letsencrypt to register your domain.#   ssl_keyfile: "/etc/letsencrypt/live/example.com/fullchain.pem"#   ssl_certfile: "/etc/letsencrypt/live/example.com/privkey.pem"# Here you can change some details of the applicationhtml_index:title:C-Botchat_title:Your chatchat_number_of_messages:0 messagechat_send_button_value:Sendchat_input_placeholder:Write your message herefooter_text:>-© 2023 - C-Bot, handmade by Vincent Stragier,University of Mons and les Amis des Aveugles, Ghlinjs_app:bot_name:C-Botusername:Viniinterface:locale:en_US.UTF-8# Here are the pyttsx4 settingsvoice:id:2# In word per minuterate:200# A float between 0 and 1.volume:1.0llm_api:# The API to usename:oobabooga# The URL of the API (or its IP address)url:"my-ngrok-domain.ngrok-free.app"port:443path:"/api/v1/generate"http_basic_auth:username:api_tokenpassword:your_custom_token