Package: pygpt-net
PyGPT - Desktop AI AssistantRelease:2.1.1| build:2024.02.29| Python:3.10+Official website:https://pygpt.net| Documentation:https://pygpt.readthedocs.ioSnap Store:https://snapcraft.io/pygpt| PyPi:https://pypi.org/project/pygpt-netCompiled version for Linux (tar.gz) and Windows 10/11 (msi) 64-bit:https://pygpt.net/#downloadOverviewPyGPTisall-in-oneDesktop AI Assistant that provides direct interaction with OpenAI language models, includingGPT-4,GPT-4 Vision, andGPT-3.5, through theOpenAI API. The application also integrates with alternative LLMs, like those available onHuggingFace, by utilizingLangchain.This assistant offers multiple modes of operation such as chat, assistants, completions, and image-related tasks usingDALL-E 3for generation andGPT-4 Visionfor image analysis.PyGPThas filesystem capabilities for file I/O, can generate and run Python code, execute system commands, execute custom commands and manage file transfers. It also allows models to perform web searches with theGoogleandMicrosoft Bing.For audio interactions,PyGPTincludes speech synthesis using theMicrosoft Azure,Google,Eleven LabsandOpenAIText-To-Speech services. Additionally, it features speech recognition capabilities provided byOpenAI Whisper,GoogleandBingenabling the application to understand spoken commands and transcribe audio inputs into text. It features context memory with save and load functionality, enabling users to resume interactions from predefined points in the conversation. Prompt creation and management are streamlined through an intuitive preset system.PyGPT's functionality extends through plugin support, allowing for custom enhancements. Its multi-modal capabilities make it an adaptable tool for a range of AI-assisted operations, such as text-based interactions, system automation, daily assisting, vision applications, natural language processing, code generation and image creation.Multiple operation modes are included, such as chat, text completion, assistant, vision, Langchain, Chat with files (viaLlama-index), commands execution, external API calls and image generation, makingPyGPTa multi-tool for many AI-driven tasks.Video(mp4, version2.0.153, build2024-02-18):https://github.com/szczyglis-dev/py-gpt/assets/61396542/996db435-fea3-4836-85b5-4f93505df6c4Screenshot(version2.0.153build2024-02-18):You can download compiled 64-bit versions for Windows and Linux here:https://pygpt.net/#downloadFeaturesDesktop AI Assistant forLinux,WindowsandMac, written in Python.Works similarly toChatGPT, but locally (on a desktop computer).8 modes of operation: Chat, Vision, Completion, Assistant, Image generation, Langchain, Chat with files and Agent (autonomous).Supports multiple models:GPT-4,GPT-3.5, and any model accessible throughLangchain.Handles and stores the full context of conversations (short-term memory).Real-time video camera capture in Vision modeInternet access viaGoogleandMicrosoft BingSpeech synthesis viaMicrosoft Azure,Google,Eleven LabsandOpenAIText-To-Speech services.Speech recognition viaOpenAI Whisper,Google,Google CloudandMicrosoft Bing.Image analysis viaGPT-4 Vision.Crontab / Task scheduler includedIntegratedLangchainsupport (you can connect to any LLM, e.g., onHuggingFace).IntegratedLlama-indexsupport: chat withtxt,pdf,csv,md,docx,json,epub,xlsxor use previous conversations as additional context provided to model.Integrated calendar, day notes and search in contexts by selected dateCommands execution (via plugins: access to the local filesystem, Python code interpreter, system commands execution).Custom commands creation and executionManages files and attachments with options to upload, download, and organize.Context history with the capability to revert to previous contexts (long-term memory).Allows you to easily manage prompts with handy editable presets.Provides an intuitive operation and interface.Includes a notepad.Includes simple painter / drawing toolIncludes optional Autonomous Mode (Agents)Supports multiple languages.Enables the use of all the powerful features ofGPT-4,GPT-4V, andGPT-3.5.Requires no previous knowledge of using AI models.Simplifies image generation usingDALL-E 3andDALL-E 2.Possesses the potential to support future OpenAI models.Fully configurable.Themes support.Plugins support.Built-in token usage calculation.It's open source; source code is available onGitHub.Utilizes the user's own API key.The application is free, open-source, and runs on PCs withLinux,Windows 10,Windows 11andMac.
Full Python source code is available onGitHub.PyGPT uses the user's API key  -  to use the application,
you must have a registered OpenAI account and your own API key.You can also use built-it Langchain support to connect to other Large Language Models (LLMs),
such as those on HuggingFace. Additional API keys may be required.InstallationCompiled versions (Linux, Windows 10 and 11)You can download compiled versions forLinuxandWindows(10/11).Download the.msiortar.gzfor the appropriate OS from the download page athttps://pygpt.netand then extract files from the archive and run the application. 64-bit only.Snap StoreYou can installPyGPTdirectly from Snap Store:sudo snap install pygptTo manage future updates just use:sudo snap refresh pygptUsing camera:to use camera in Snap version you must connect the camera with:sudo snap connect pygpt:cameraUsing microphone:to use microphone in Snap version you must connect the microphone with:sudo snap connect pygpt:audio-record :audio-recordPyPi (pip)The application can also be installed fromPyPiusingpip install:Create virtual environment:python3 -m venv venv
source venv/bin/activateInstall from PyPi:pip install pygpt-netOnce installed run the command to start the application:pygptSource CodeAn alternative method is to download the source code fromGitHuband execute the application using the Python interpreter (version3.10or higher).Running from GitHub source codeClone git repository or download .zip file:git clone https://github.com/szczyglis-dev/py-gpt.git
cd py-gptCreate virtual environment:python3 -m venv venv
source venv/bin/activateInstall requirements:pip install -r requirements.txtRun the application:python3 run.pyInstall with PoetryClone git repository or download .zip file:git clone https://github.com/szczyglis-dev/py-gpt.git
cd py-gptCreate virtual environment:poetry shellInstall requirements:poetry installRun the application:poetry run python3 run.pyTip: you can usePyInstallerto create a compiled version of
the application for your system (required version >=6.0.0).TroubleshootingIf you have a problems withxcbplugin with newer versions of PySide on Linux, e.g. like this:qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. 
Reinstalling the application may fix this problem....then installlibxcb:sudo apt install libxcb-cursor0If you have a problems with audio on Linux, then try to installportaudio19-devand/orlibasound2:sudo apt install portaudio19-devsudo apt install libasound2
sudo apt install libasound2-data 
sudo apt install libasound2-pluginsAccess to camera in Snap version:To use camera in Vision mode in Snap version you must connect the camera with:sudo snap connect pygpt:cameraAccess to microphone in Snap version:To use microphone in Snap version you must connect the microphone with:sudo snap connect pygpt:audio-record :audio-recordWindows and VC++ RedistributableOn Windows, the proper functioning requires the installation of theVC++ Redistributable, which can be found on the Microsoft website:https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redistThe libraries from this environment are used byPySide6- one of the base packages used by PyGPT.
The absence of the installed libraries may cause display errors or completely prevent the application from running.It may also be necessary to add the pathC:\path\to\venv\Lib\python3.x\site-packages\PySide6to thePATHvariable.Other requirementsFor operation, an internet connection is needed (for API connectivity), a registered OpenAI account,
and an active API key that must be input into the program.Debugging and loggingTip:Go toDebugging and Loggingsection for instructions on how to log and diagnose issues in a more detailed manner.Quick StartSetting-up OpenAI API KEYDuring the initial launch, you must configure your API key within the application.To do so, navigate to the menu:Config -> Settings...and then paste the API key into theOpenAI API KEYfield.OverviewThe API key can be obtained by registering on the OpenAI website:https://platform.openai.comYour API keys will be available here:https://platform.openai.com/account/api-keysNote:The ability to use models within the application depends on the API user's access to a given model!Chat, completion, assistants and vision (GPT-4, GPT-3.5, Langchain)Chat+ inline Vision and Image generationThis mode inPyGPTmirrorsChatGPT, allowing you to chat with models such asGPT-4,GPT-4 TurboandGPT-3.5. It's easy to switch models whenever you want. It works by using theChatCompletion API.The main part of the interface is a chat window where conversations appear. Right below that is where you type your messages. On the right side of the screen, there's a section to set up or change your system prompts. You can also save these setups as presets to quickly switch between different models or tasks.Above where you type your messages, the interface shows you the number of tokens your message will use up as you type it – this helps to keep track of usage. There's also a feature to upload files in this area. Go to theFilestab to manage your uploads or add attachments to send to the OpenAI API (but this makes effect only inAssisantandVisionmodes).Vision:If you want to send photos or image from camera to analysis you must enable pluginGPT-4 Vision Inlinein the Plugins menu.
Plugin allows you to send photos or image from camera to analysis in any Chat mode:With this plugin, you can capture an image with your camera or attach an image and send it for analysis to discuss the photograph:Image generation:If you want to generate images (using DALL-E) directly in chat you must enable pluginDALL-E 3 Inlinein the Plugins menu.
Plugin allows you to generate images in Chat mode:CompletionThis mode provides in-depth access to a broader range of capabilities offered by Large Language Models (LLMs). While it maintains a chat-like interface for user interaction, it introduces additional settings and functional richness beyond typical chat exchanges. Users can leverage this mode to prompt models for complex text completions, role-play dialogues between different characters, perform text analysis, and execute a variety of other sophisticated tasks. It supports any model provided by the OpenAI API as well as other models throughLangchain.Similar to chat mode, on the right-hand side of the interface, there are convenient presets. These allow you to fine-tune instructions and swiftly transition between varied configurations and pre-made prompt templates.Additionally, this mode offers options for labeling the AI and the user, making it possible to simulate dialogues between specific characters - for example, you could create a conversation between Batman and the Joker, as predefined in the prompt. This feature presents a range of creative possibilities for setting up different conversational scenarios in an engaging and exploratory manner.From version2.0.107thedavincimodels are deprecated and has been replaced withgpt-3.5-turbo-instructmodel in Completion mode.AssistantsThis mode uses the new OpenAI'sAssistants API.This mode expands on the basic chat functionality by including additional external tools like aCode Interpreterfor executing code,Retrieval Filesfor accessing files, and customFunctionsfor enhanced interaction and integration with other APIs or services. In this mode, you can easily upload and download files.PyGPTstreamlines file management, enabling you to quickly upload documents and manage files created by the model.Setting up new assistants is simple - a single click is all it takes, and they instantly sync with theOpenAI API. Importing assistants you've previously created with OpenAI intoPyGPTis also a seamless process.In Assistant mode you are allowed to storage your files (per Assistant) and manage them easily from app:Please note that token usage calculation is unavailable in this mode. Nonetheless, file (attachment)
uploads are supported. Simply navigate to theFilestab to effortlessly manage files and attachments which
can be sent to the OpenAI API.Vision (GPT-4 Vision)This mode enables image analysis using theGPT-4 Visionmodel. Functioning much like the chat mode,
it also allows you to upload images or provide URLs to images. The vision feature can analyze both local
images and those found online.From version 2.0.68- Vision is integrated into any chat mode via pluginGPT-4 Vision (inline). Just enable the plugin and use Vision in standard modes.From version 2.0.14- Vision mode also includes real-time video capture from camera. To enable capture check the optionCameraon the right-bottom corner. It will enable real-time capturing from your camera. To capture image from camera and append it to chat just click on video at left side. You can also enableAuto capture- image will be captured and appended to chat message every time you send message.1) Video camera real-time image capture2) you can also provide an image URL3) or you can just upload your local images or use the inline Vision in the standard chat mode:Tip:When usingVision (inline)by utilizing a plugin in standard mode, such asChat(notVisionmode), the+ Visionspecial checkbox will appear at the bottom of the Chat window. It will be automatically enabled any time you provide content for analysis (like an uploaded photo). When the checkbox is enabled, the vision model is used. If you wish to exit the vision model after image analysis, simply uncheck the checkbox. It will activate again automatically when the next image content for analysis is provided.LangchainThis mode enables you to work with models that are supported byLangchain. The Langchain support is integrated
into the application, allowing you to interact with any LLM by simply supplying a configuration
file for the specific model. You can add as many models as you like; just list them in the configuration
file namedmodels.json.Available LLMs providers supported byPyGPT:- OpenAI
- Azure OpenAI
- HuggingFace
- Anthropic
- Llama 2
- OllamaYou have the ability to add custom model wrappers for models that are not available by default inPyGPT.
To integrate a new model, you can create your own wrapper and register it with the application.
Detailed instructions for this process are provided in the section titledManaging models / Adding models via Langchain.Chat with files (Llama-index)This mode enables chat interaction with your documents and entire context history through conversation.
It seamlessly incorporatesLlama-indexinto the chat interface, allowing for immediate querying of your indexed documents.To start, you need to index (embed) the files you want to use as additional context.
Embedding transforms your text data into vectors. If you're unfamiliar with embeddings and how they work, check out this article:https://stackoverflow.blog/2023/11/09/an-intuitive-introduction-to-text-embeddings/For a visualization from OpenAI's page, see this picture:Source:https://cdn.openai.com/new-and-improved-embedding-model/draft-20221214a/vectors-3.svgTo index your files, simply copy or upload them  into thedatadirectory and initiate indexing (embedding) by clicking theIndex allbutton, or right-click on a file and selectIndex.... Additionally, you have the option to utilize data from indexed files in any Chat mode by activating theChat with files (Llama-index, inline)plugin.After the file(s) are indexed (embedded in vector store), you can use context from them in chat mode:Built-in file loaders:txt,pdf,csv,md,docx,json,epub,xlsx,xml.
You can configure data loaders inSettings / Llama-index / Data Loadersby providing list of arguments for specified loaders.
You can also develop and provide your own custom loader and register it within the application.From version2.0.100Llama-index is also integrated with context database - you can use data from database (your context history) as additional context in discussion.
Options for indexing existing context history or enabling real-time indexing new ones (from database) are available inSettings / Llama-indexsection.WARNING:remember that when indexing content, API calls to the embedding model (text-embedding-ada-002) are used. Each indexing consumes additional tokens.
Always control the number of tokens used on the OpenAI page.Tip:when usingChat with filesyou are using additional context from db data and files indexed fromdatadirectory, not the files sending viaAttachmentstab.
Attachments tab inChat with filesmode can be used to provide images toVision (inline)plugin only.Token limit:When you useChat with filesin non-query mode, Llama-index adds extra context to the system prompt. If you use a plugins (which also adds more instructions to system prompt), you might go over the maximum number of tokens allowed. If you get a warning that says you've used too many tokens, turn off plugins you're not using or turn off the "Execute commands" option to reduce the number of tokens used by the system prompt.Available vector stores(provided byLlama-index):- ChromaVectorStore
- ElasticsearchStore
- PinecodeVectorStore
- RedisVectorStore
- SimpleVectorStoreYou can configure selected vector store by providing config options likeapi_key, etc. inSettings -> Llama-indexwindow.
Arguments provided here (on list:Vector Store (**kwargs)inAdvanced settingswill be passed to selected vector store provider.
You can check keyword arguments needed by selected provider on Llama-index API reference page:https://docs.llamaindex.ai/en/stable/api_reference/storage/vector_store.htmlWhich keyword arguments are passed to providers?ForChromaVectorStoreandSimpleVectorStoreall arguments are set by PyGPT and passed internally (you do not need to configure anything).
For other providers you can provide these arguments:ElasticsearchStoreKeyword arguments for ElasticsearchStore(**kwargs):index_name(default: current index ID, already set, not required)any other keyword arguments provided on listPinecodeVectorStoreKeyword arguments for Pinecone(**kwargs):api_keyindex_name (default: current index ID, already set, not required)RedisVectorStoreKeyword arguments for RedisVectorStore(**kwargs):index_name(default: current index ID, already set, not required)any other keyword arguments provided on listYou can extend list of available providers by creating custom provider and registering it on app launch.By default, you are using chat-based mode when usingChat with files.
If you want to only query index (without chat) you can enableQuery index only (without chat)option.Adding custom vector stores and data loadersYou can create a custom vector store provider or data loader for your data and develop a custom launcher for the application. To register your custom vector store provider or data loader, simply register it by passing the vector store provider instance tovector_storeskeyword argument and loader instance in theloaderskeyword argument:# custom_launcher.pyfrompygpt_net.appimportrunfrompluginsimportCustomPlugin,OtherCustomPluginfromllmsimportCustomLLMfromvector_storesimportCustomVectorStorefromloadersimportCustomLoaderplugins=[CustomPlugin(),OtherCustomPlugin(),]llms=[CustomLLM(),]vector_stores=[CustomVectorStore(),]loaders=[CustomLoader(),]run(plugins=plugins,llms=llms,vector_stores=vector_stores,# <--- list with custom vector store providersloaders=loaders# <--- list with custom data loaders)The vector store provider must be an instance ofpygpt_net.provider.vector_stores.base.BaseStore.
You can review the code of the built-in providers inpygpt_net.provider.vector_storesand use them as examples when creating a custom provider.The data loader must be an instance ofpygpt_net.provider.loaders.base.BaseLoader.
You can review the code of the built-in loaders inpygpt_net.provider.loadersand use them as examples when creating a custom loader.Configuring data loadersIn theSettings -> Llama-index -> Data loaderssection you can define the additional keyword arguments to pass into data loader instance.In most cases, an internal Llama-index loaders are used internally.
You can check these base loaders e.g. here:File:https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/readers/llama-index-readers-file/llama_index/readers/fileWeb:https://github.com/run-llama/llama_index/tree/main/llama-index-integrations/readers/llama-index-readers-webAllowed keyword arguments for built-in data loaders:CSV Files (file_csv)concat_rows- bool, default:Trueencoding- str, default:utf-8HTML Files (file_html)tag- str, default:sectionignore_no_id- bool, default:FalseIPYNB Notebook files (file_ipynb)parser_config- dict, default:Noneconcatenate- bool, default:FalseMarkdown files (file_md)remove_hyperlinks- bool, default:Trueremove_images- bool, default:TruePDF documents (file_pdf)return_full_document- bool, default:FalseXML files (file_xml)tree_level_split- int, default:0Sitemap (web_sitemap)html_to_text- bool, default:Falselimit- int, default:10Agent (autonomous)This mode is experimental.WARNING: Please use this mode with caution!- autonomous mode, when connected with other plugins, may produce unexpected results!The mode activates autonomous mode, where AI begins a conversation with itself.
You can set this loop to run for any number of iterations. Throughout this sequence, the model will engage
in self-dialogue, answering his own questions and comments, in order to find the best possible solution, subjecting previously generated steps to criticism.WARNING:Setting the number of run steps (iterations) to0activates an infinite loop which can generate a large number of requests and cause very high token consumption, so use this option with caution! Confirmation will be displayed every time you run the infinite loop.This mode is similar toAuto-GPT- it can be used to create more advanced inferences and to solve problems by breaking them down into subtasks that the model will autonomously perform one after another until the goal is achieved.You can create presets with custom instructions for multiple agents, incorporating various workflows, instructions, and goals to achieve.All plugins are available for agents, so you can enable features such as file access, command execution, web searching, image generation, vision analysis, etc., for your agents. Connecting agents with plugins can create a fully autonomous, self-sufficient system. All currently enabled plugins are automatically available to the Agent.When theAuto-stopoption is enabled, the agent will attempt to stop once the goal has been reached.OptionsThe agent is essentially avirtualmode that internally sequences the execution of a selected underlying mode.
You can choose which internal mode the agent should use in the settings:Settings / Agent (autonomous) / Sub-mode to useAvailable choices include:chat,completion,langchain,vision,llama_index(Chat with files).Default is:chat.If you want to use the Llama-index mode when running the agent, you can also specify which indexLlama-indexshould use with the option:Settings / Agent (autonomous) / Index to useFiles and attachmentsInput attachments (upload)PyGPTmakes it simple for users to upload files to the server and send them to the model for tasks like analysis, similar to attaching files inChatGPT. There's a separateFilestab next to the text input area specifically for managing file uploads. Users can opt to have files automatically deleted after each upload or keep them on the list for repeated use.The attachment feature is available in both theAssistantandVisionmodes at default.
InAssistantmode, you can send documents and files to analyze, while inVisionmode, you can send images.
In other modes, you can enable attachments by activating theVision (inline)plugin (for providing images only).Files (download, code generation)PyGPTenables the automatic download and saving of files created by the model. This is carried out in the background, with the files being saved to andatafolder located within the user's working directory. To view or manage these files, users can navigate to theFilestab which features a file browser for this specific directory. Here, users have the interface to handle all files sent by the AI.Thisdatadirectory is also where the application stores files that are generated locally by the AI, such as code files or any other data requested from the model. Users have the option to execute code directly from the stored files and read their contents, with the results fed back to the AI. This hands-off process is managed by the built-in plugin system and model-triggered commands. You can also indexing files from this directory (using integratedLlama-index) and use it's contents as additional context provided to discussion.TheCommand: Files I/Oplugin takes care of file operations in thedatadirectory, while theCommand: Code Interpreterplugin allows for the execution of code from these files.To allow the model to manage files or python code execution, theExecute commandsoption must be active, along with the above-mentioned plugins:Draw (paint)Using theDrawtool, you can create quick sketches and submit them to the model for analysis. You can also edit opened from disk or captured from camera images, for example, by adding elements like arrows or outlines to objects. Additionally, you can capture screenshots from the system - the captured image is placed in the drawing tool and attached to the query being sent.To capture the screenshot just click on theAsk with screenshotoption in a tray-icon dropdown:CalendarUsing the calendar, you can go back to selected conversations from a specific day and add daily notes. After adding a note, it will be marked on the list, and you can change the color of its label by right-clicking and selectingSet label color. By clicking on a particular day of the week, conversations from that day will be displayed.Context and memoryShort and long-term memoryPyGPTfeatures a continuous chat mode that maintains a long context of the ongoing dialogue. It preserves the entire conversation history and automatically appends it to each new message (prompt) you send to the AI. Additionally, you have the flexibility to revisit past conversations whenever you choose. The application keeps a record of your chat history, allowing you to resume discussions from the exact point you stopped.Handling multiple contextsOn the left side of the application interface, there is a panel that displays a list of saved conversations. You can save numerous contexts and switch between them with ease. This feature allows you to revisit and continue from any point in a previous conversation.PyGPTautomatically generates a summary for each context, akin to the wayChatGPToperates and gives you the option to modify these titles itself.You can disable context support in the settings by using the following option:Config -> Settings -> Use contextClearing historyYou can clear the entire memory (all contexts) by selecting the menu option:File -> Clear history...Context storageOn the application side, the context is stored in theSQLitedatabase located in the working directory (db.sqlite).
In addition, all history is also saved to.txtfiles for easy reading.Once a conversation begins, a title for the chat is generated and displayed on the list to the left. This process is similar toChatGPT, where the subject of the conversation is summarized, and a title for the thread is created based on that summary. You can change the name of the thread at any time.PresetsWhat is preset?Presets inPyGPTare essentially templates used to store and quickly apply different configurations. Each preset includes settings for the mode you want to use (such as chat, completion, or image generation), an initial system message, an assigned name for the AI, a username for the session, and the desired "temperature" for the conversation. A warmer "temperature" setting allows the AI to provide more creative responses, while a cooler setting encourages more predictable replies. These presets can be used across various modes and with models accessed via theOpenAI APIorLangchain.The system lets you create as many presets as needed and easily switch among them. Additionally, you can clone an existing preset, which is useful for creating variations based on previously set configurations and experimentation.Example usageThe application includes several sample presets that help you become acquainted with the mechanism of their use.Image generation (DALL-E 3 and DALL-E 2)DALL-E 3PyGPTenables quick and easy image creation withDALL-E 3.
The older model version,DALL-E 2, is also accessible. Generating images is akin to a chat conversation  -  a user's prompt triggers the generation, followed by downloading, saving to the computer,
and displaying the image onscreen. You can send raw prompt toDALL-EinImage generationmode or ask the model for the best prompt.From version 2.0.68 (released 2023-12-31) image generation using DALL-E is available in every mode via pluginDALL-E 3 Image Generation (inline). Just ask any model, in any mode, like e.g. GPT-4 to generate an image and it will do it inline, without need to mode change.If you want to generate images (using DALL-E) directly in chat you must enable pluginDALL-E 3 Inlinein the Plugins menu.
Plugin allows you to generate images in Chat mode:Multiple variantsYou can generate up to4 different variants(DALL-E 2) for a given prompt in one session. DALL-E 3 allows one image.
To select the desired number of variants to create, use the slider located in the right-hand corner at
the bottom of the screen. This replaces the conversation temperature slider when you switch to image generation mode.Raw modeThere is an option for switching prompt generation mode.IfRaw Modeis enabled, DALL-E will receive the prompt exactly as you have provided it.
IfRaw Modeis disabled, GPT will generate the best prompt for you based on your instructions.Image storageOnce you've generated an image, you can easily save it anywhere on your disk by right-clicking on it.
You also have the options to delete it or view it in full size in your web browser.Tip:Use presets to save your prepared prompts.
This lets you quickly use them again for generating new images later on.The app keeps a history of all your prompts, allowing you to revisit any session and reuse previous
prompts for creating new images.Images are stored inimgdirectory inPyGPTuser data folder.Managing modelsAll models are specified in the configuration filemodels.json, which you can customize.
This file is located in your working directory. You can add new models provided directly byOpenAI APIand those supported byLangchainto this file. Configuration for Langchain wrapper is placed inlangchainkey.Adding custom LLMs via LangchainTo add a new model using the Langchain wrapper inPyGPT, insert the model's configuration details into themodels.jsonfile. This should include the model's name, its supported modes (eitherchat,completion, or both), the LLM provider (which can be either e.g.OpenAIorHuggingFace), and, if you are using aHuggingFacemodel, an optionalAPI KEY.Example of models configuration -models.json:"gpt-3.5-turbo": {
    "id": "gpt-3.5-turbo",
    "name": "gpt-3.5-turbo",
    "mode": [
        "chat",
        "assistant",
        "langchain",
        "llama_index"
    ],
    "langchain": {
        "provider": "openai",
        "mode": [
            "chat"
        ],
        "args": [
            {
                "name": "model_name",
                "value": "gpt-3.5-turbo",
                "type": "str"
            }
        ],
        "env": [
            {
                "name": "OPENAI_API_KEY",
                "value": "{api_key}"
            }
        ]
    },
    "llama_index": {
        "provider": "openai",
        "mode": [
            "chat"
        ],
        "args": [
            {
                "name": "model",
                "value": "gpt-3.5-turbo",
                "type": "str"
            }
        ],
        "env": [
            {
                "name": "OPENAI_API_KEY",
                "value": "{api_key}"
            }
        ]
    },
    "ctx": 4096,
    "tokens": 4096,
    "default": false
},There is bult-in support for those LLMs providers:- OpenAI (openai)
- Azure OpenAI (azure_openai)
- HuggingFace (huggingface)
- Anthropic (anthropic)
- Llama 2 (llama2)
- Ollama (ollama)Adding custom LLM providersHandling LLMs with Langchain is implemented through separated wrappers. This allows for the addition of support for any provider and model available via Langchain. All built-in wrappers for the models and its providers are placed in thepygpt_net.provider.llms.These wrappers are loaded into the application during startup usinglauncher.add_llm()method:# app.pyfrompygpt_net.provider.llms.openaiimportOpenAILLMfrompygpt_net.provider.llms.azure_openaiimportAzureOpenAILLMfrompygpt_net.provider.llms.anthropicimportAnthropicLLMfrompygpt_net.provider.llms.hugging_faceimportHuggingFaceLLMfrompygpt_net.provider.llms.llamaimportLlama2LLMfrompygpt_net.provider.llms.ollamaimportOllamaLLMdefrun(**kwargs):"""Runs the app."""# Initialize the applauncher=Launcher()launcher.init()# Register plugins...# Register langchain LLMs wrapperslauncher.add_llm(OpenAILLM())launcher.add_llm(AzureOpenAILLM())launcher.add_llm(AnthropicLLM())launcher.add_llm(HuggingFaceLLM())launcher.add_llm(Llama2LLM())launcher.add_llm(OllamaLLM())# Launch the applauncher.run()To add support for providers not included by default, you can create your own wrapper that returns a custom model to the application and then pass this custom wrapper to the launcher.ExtendingPyGPTwith custom plugins and LLM wrappers is straightforward:Pass instances of custom plugins and LLM wrappers directly to the launcher.To register custom LLM wrappers:Provide a list of LLM wrapper instances asllmskeyword argument.Example:# launcher.pyfrompygpt_net.appimportrunfrompluginsimportCustomPlugin,OtherCustomPluginfromllmsimportCustomLLMplugins=[CustomPlugin(),OtherCustomPlugin(),]llms=[CustomLLM(),]vector_stores=[]run(plugins=plugins,llms=llms,vector_stores=vector_stores)Examples (tutorial files)See theexamplesdirectory in this repository with examples of custom launcher, plugin, vector store, LLM (Langchain and Llama-index) provider and data loader:examples/custom_launcher.pyexamples/example_audio_input.pyexamples/example_audio_output.pyexamples/example_data_loader.pyexamples/example_llm.py<-- use it as an exampleexamples/example_plugin.pyexamples/example_vector_store.pyexamples/example_web_search.pyThese example files can be used as a starting point for creating your own extensions forPyGPT.To integrate your own model or provider intoPyGPT, you can also reference the classes located in thepygpt_net.provider.llms. These samples can act as an more complex example for your custom class. Ensure that your custom wrapper class includes two essential methods:chatandcompletion. These methods should return the respective objects required for the model to operate inchatandcompletionmodes.Adding custom Vector Store providersFrom version 2.0.114 you can also register your own Vector Store provider:# app.py# vector storesfrompygpt_net.provider.vector_stores.chromaimportChromaProviderfrompygpt_net.provider.vector_stores.elasticsearchimportElasticsearchProviderfrompygpt_net.provider.vector_stores.pinecodeimportPinecodeProviderfrompygpt_net.provider.vector_stores.redisimportRedisProviderfrompygpt_net.provider.vector_stores.simpleimportSimpleProviderdefrun(**kwargs):# ...# register base vector store providers (llama-index)launcher.add_vector_store(ChromaProvider())launcher.add_vector_store(ElasticsearchProvider())launcher.add_vector_store(PinecodeProvider())launcher.add_vector_store(RedisProvider())launcher.add_vector_store(SimpleProvider())# register custom vector store providers (llama-index)vector_stores=kwargs.get('vector_stores',None)ifisinstance(vector_stores,list):forstoreinvector_stores:launcher.add_vector_store(store)# ...To register your custom vector store provider just register it by passing provider instance invector_storeskeyword argument:# custom_launcher.pyfrompygpt_net.appimportrunfrompluginsimportCustomPlugin,OtherCustomPluginfromllmsimportCustomLLMfromvector_storesimportCustomVectorStoreplugins=[CustomPlugin(),OtherCustomPlugin(),]llms=[CustomLLM(),]vector_stores=[CustomVectorStore(),]run(plugins=plugins,llms=llms,vector_stores=vector_stores)PluginsPyGPTcan be enhanced with plugins to add new features.The following plugins are currently available, and model can use them instantly:Audio Input- provides speech recognition.Audio Output- provides voice synthesis.Autonomous Agent (inline)- enables autonomous conversation (AI to AI), manages loop, and connects output back to input. This is the inline Agent mode.Chat with files (Llama-index, inline)- plugin integratesLlama-indexstorage in any chat and provides additional knowledge into context (from indexed files and previous context from database).Command: API calls- plugin lets you connect the model to the external services using custom defined API calls.Command: Code Interpreter- responsible for generating and executing Python code, functioning much like
the Code Interpreter on ChatGPT, but locally. This means GPT can interface with any script, application, or code.
The plugin can also execute system commands, allowing GPT to integrate with your operating system.
Plugins can work in conjunction to perform sequential tasks; for example, theFilesplugin can write generated
Python code to a file, which theCode Interpretercan execute it and return its result to GPT.Command: Custom Commands- allows you to create and execute custom commands on your system.Command: Files I/O- provides access to the local filesystem, enabling GPT to read and write files,
as well as list and create directories.Command: Web Search- allows searching the internet and reading web pages.Command: Serial port / USB- plugin provides commands for reading and sending data to USB ports.Context history (calendar, inline)- provides access to context history database.Crontab / Task scheduler- plugin provides cron-based job scheduling - you can schedule tasks/prompts to be sent at any time using cron-based syntax for task setup.DALL-E 3: Image Generation (inline)- integrates DALL-E 3 image generation with any chat and mode. Just enable and ask for image in Chat mode, using standard model like GPT-4. The plugin does not require theExecute commandsoption to be enabled.GPT-4 Vision (inline)- integrates Vision capabilities with any chat mode, not just Vision mode. When the plugin is enabled, the model temporarily switches to vision in the background when an image attachment or vision capture is provided.Real Time- automatically appends the current date and time to the system prompt, informing the model about current time.System Prompt Extra (append)- appends additional system prompts (extra data) from a list to every current system prompt. You can enhance every system prompt with extra instructions that will be automatically appended to the system prompt.Audio InputThe plugin facilitates speech recognition (by default using theWhispermodel from OpenAI,GoogleandBingare also available). It allows for voice commands to be relayed to the AI using your own voice. Whisper doesn't require any extra API keys or additional configurations; it uses the main OpenAI key. In the plugin's configuration options, you should adjust the volume level (min energy) at which the plugin will respond to your microphone. Once the plugin is activated, a newSpeakoption will appear at the bottom near theSendbutton  -  when this is enabled, the application will respond to the voice received from the microphone.The plugin can be extended with other speech recognition providers.Options:ProviderproviderChoose the provider.Default:WhisperAvailable providers:Whisper (viaOpenAI API)Google (viaSpeechRecognitionlibrary)Google Cloud (viaSpeechRecognitionlibrary)Microsoft Bing (viaSpeechRecognitionlibrary)WhisperModelwhisper_modelChoose the model.Default:whisper-1GoogleAdditional keywords argumentsgoogle_argsAdditional keywords arguments for r.recognize_google(audio, **kwargs)Google CloudAdditional keywords argumentsgoogle_cloud_argsAdditional keywords arguments for r.recognize_google_cloud(audio, **kwargs)BingAdditional keywords argumentsbing_argsAdditional keywords arguments for r.recognize_bing(audio, **kwargs)General optionsAuto sendauto_sendAutomatically send recognized speech as input text after recognition.Default:TrueAdvanced modeadvancedEnable only if you want to use advanced mode and the settings below. Do not enable this option if you just want to use the simplified mode (default).Default:FalseAdvanced mode optionsTimeouttimeoutThe duration in seconds that the application waits for voice input from the microphone.Default:5Phrase max lengthphrase_lengthMaximum duration for a voice sample (in seconds).Default:10Min energymin_energyMinimum threshold multiplier above the noise level to begin recording.Default:1.3Adjust for ambient noiseadjust_noiseEnables adjustment to ambient noise levels.Default:TrueContinuous listencontinuous_listenExperimental: continuous listening - do not stop listening after a single input.
Warning: This feature may lead to unexpected results and requires fine-tuning with
the rest of the options! If disabled, listening must be started manually
by enabling theSpeakoption.Default:FalseWait for responsewait_responseWait for a response before initiating listening for the next input.Default:TrueMagic wordmagic_wordActivate listening only after the magic word is provided.Default:FalseReset Magic wordmagic_word_resetReset the magic word status after it is received (the magic word will need to be provided again).Default:TrueMagic wordsmagic_wordsList of magic words to initiate listening (Magic word mode must be enabled).Default:OK, Okay, Hey GPT, OK GPTMagic word timeoutmagic_word_timeoutThe number of seconds the application waits for magic word.Default:1Magic word phrase max lengthmagic_word_phrase_lengthThe minimum phrase duration for magic word.Default:2Prefix wordsprefix_wordsList of words that must initiate each phrase to be processed. For example, you can define words like "OK" or "GPT"—if set, any phrases not starting with those words will be ignored. Insert multiple words or phrases separated by commas. Leave empty to deactivate.Default:emptyStop wordsstop_wordsList of words that will stop the listening process.Default:stop, exit, quit, end, finish, close, terminate, kill, halt, abortOptions related to Speech Recognition internals:energy_thresholdrecognition_energy_thresholdRepresents the energy level threshold for sounds.Default:300dynamic_energy_thresholdrecognition_dynamic_energy_thresholdRepresents whether the energy level threshold (see recognizer_instance.energy_threshold) for sounds
should be automatically adjusted based on the currently ambient noise level while listening.Default:Truedynamic_energy_adjustment_dampingrecognition_dynamic_energy_adjustment_dampingRepresents approximately the fraction of the current energy threshold that is retained after one second
of dynamic threshold adjustment.Default:0.15pause_thresholdrecognition_pause_thresholdRepresents the minimum length of silence (in seconds) that will register as the end of a phrase.Default:0.8adjust_for_ambient_noise: durationrecognition_adjust_for_ambient_noise_durationThe duration parameter is the maximum number of seconds that it will dynamically adjust the threshold
for before returning.Default:1Options reference:https://pypi.org/project/SpeechRecognition/1.3.1/Audio OutputThe plugin lets you turn text into speech using the TTS model from OpenAI or other services likeMicrosoft Azure,Google, andEleven Labs. You can add more text-to-speech providers to it too.OpenAI TTSdoes not require any additional API keys or extra configuration; it utilizes the main OpenAI key.
Microsoft Azure requires to have an Azure API Key. Before using speech synthesis viaMicrosoft Azure,GoogleorEleven Labs, you must configure the audio plugin with your API keys, regions and voices if required.Through the available options, you can select the voice that you want the model to use. More voice synthesis providers coming soon.To enable voice synthesis, activate theAudio Outputplugin in thePluginsmenu or turn on theAudio Outputoption in theAudio / Voicemenu (both options in the menu achieve the same outcome).OptionsProviderproviderChoose the provider.Default:OpenAI TTSAvailable providers:OpenAI TTSMicrosoft Azure TTSGoogle TTSEleven Labs TTSOpenAI Text-To-SpeechModelopenai_modelChoose the model. Available options:- tts-1
  - tts-1-hdDefault:tts-1Voiceopenai_voiceChoose the voice. Available voices to choose from:- alloy
  - echo
  - fable
  - onyx
  - nova
  - shimmerDefault:alloyMicrosoft Azure Text-To-SpeechAzure API Keyazure_api_keyHere, you should enter the API key, which can be obtained by registering for free on the following website:https://azure.microsoft.com/en-us/services/cognitive-services/text-to-speechAzure Regionazure_regionYou must also provide the appropriate region for Azure here.Default:eastusVoice (EN)azure_voice_enHere you can specify the name of the voice used for speech synthesis for English.Default:en-US-AriaNeuralVoice (non-English)azure_voice_plHere you can specify the name of the voice used for speech synthesis for other non-english languages.Default:pl-PL-AgnieszkaNeuralGoogle Text-To-SpeechGoogle Cloud Text-to-speech API Keygoogle_api_keyYou can obtain your own API key at:https://console.cloud.google.com/apis/library/texttospeech.googleapis.comVoicegoogle_voiceSpecify voice. Voices:https://cloud.google.com/text-to-speech/docs/voicesLanguage codegoogle_api_keyLanguage code. Language codes:https://cloud.google.com/speech-to-text/docs/speech-to-text-supported-languagesEleven Labs Text-To-SpeechEleven Labs API Keyeleven_labs_api_keyYou can obtain your own API key at:https://elevenlabs.io/speech-synthesisVoice IDeleven_labs_voiceVoice ID. Voices:https://elevenlabs.io/voice-libraryModeleleven_labs_modelSpecify model. Models:https://elevenlabs.io/docs/speech-synthesis/modelsIf speech synthesis is enabled, a voice will be additionally generated in the background while generating a response via GPT.BothOpenAI TTSandOpenAI Whisperuse the same single API key provided for the OpenAI API, with no additional keys required.Autonomous Agent (inline)WARNING: Please use autonomous mode with caution!- this mode, when connected with other plugins, may produce unexpected results!The plugin activates autonomous mode in standard chat modes, where AI begins a conversation with itself.
You can set this loop to run for any number of iterations. Throughout this sequence, the model will engage
in self-dialogue, answering his own questions and comments, in order to find the best possible solution, subjecting previously generated steps to criticism.This mode is similar toAuto-GPT- it can be used to create more advanced inferences and to solve problems by breaking them down into subtasks that the model will autonomously perform one after another until the goal is achieved. The plugin is capable of working in cooperation with other plugins, thus it can utilize tools such as web search, access to the file system, or image generation usingDALL-E.You can adjust the number of iterations for the self-conversation in thePlugins / Settings...menu under the following option:IterationsiterationsDefault:3WARNING: Setting this option to0activates aninfinity loopwhich can generate a large number of requests and cause very high token consumption, so use this option with caution!PromptspromptsEditable list of prompts used to instruct how to handle autonomous mode, you can create as many prompts as you want.
First active prompt on list will be used to handle autonomous mode.Auto-stop after goal is reachedauto_stopIf enabled, plugin will stop after goal is reached."Default:TrueReverse roles between iterationsreverse_rolesOnly for Completion/Langchain modes.
If enabled, this option reverses the roles (AI <> user) with each iteration. For example,
if in the previous iteration the response was generated for "Batman," the next iteration will use that
response to generate an input for "Joker."Default:TrueChat with files (Llama-index, inline)Plugin integratesLlama-indexstorage in any chat and provides additional knowledge into context.Ask Llama-index firstask_llama_firstWhen enabled, thenLlama-indexwill be asked first, and response will be used as additional knowledge in prompt. When disabled, thenLlama-indexwill be asked only when needed.Modelmodel_queryModel used for queryingLlama-index, default:gpt-3.5-turboIndexes IDsidxIndexes to use, default: base, if you want to use multiple indexes at once then separate them by comma.Command: API callsPyGPTlets you connect the model to the external services using custom defined API calls.To activate this feature, turn on theCommand: API callsplugin found in thePluginsmenu.In this plugin you can provide list of allowed API calls, their parameters and request types. The model will replace provided placeholders with required params and make API call to external service.Your custom API callscmdsYou can provide custom API calls on the list here.Params to specify for API call:Enabled(True / False)Name:unique API call name (ID)Instruction:description for model when and how to use this API callGET params:list, separated by comma, GET params to append to endpoint URLPOST params:list, separated by comma, POST params to send in POST requestPOST JSON:provide the JSON object, template to send in POST JSON request, use%param%as POST param placeholdersHeaders:provide the JSON object with dictionary of extra request headers, like Authorization, API keys, etc.Request type:use GET for basic GET request, POST to send encoded POST params or POST_JSON to send JSON-encoded object as bodyEndpoint:API endpoint URL, use{param}as GET param placeholdersAn example API call is provided with plugin by default, it calls the Wikipedia API:Name:search_wikiInstructiom:send API call to Wikipedia to search pages by queryGET params:query, limitType:GETAPI endpoint:https://en.wikipedia.org/w/api.php?action=opensearch&limit={limit}&format=json&search={query}In the above example, every time you ask the model for query Wiki for provided query (e.g.Call the Wikipedia API for query: Nikola Tesla) it will replace placeholders in provided API endpoint URL with a generated query and it will call prepared API endpoint URL, like below:https://en.wikipedia.org/w/api.php?action=opensearch&limit=5&format=json&search=Nikola%20TeslaYou can specify type of request:GET,POSTandPOST JSON.In thePOSTrequest you can provide POST params, they will be encoded and send as POST data.In thePOST JSONrequest you must provide JSON object template to be send, using%param%placeholders in the JSON object to be replaced with the model.You can also provide any required credentials, like Authorization headers, API keys, tokens, etc. using theheadersfield - you can provide a JSON object here with a dictionarykey => value- provided JSON object will be converted to headers dictonary and send with the request.Disable SSL verifydisable_sslDisables SSL verification when making requestsTimeouttimeoutConnection timeout (seconds)User agentuser_agentUser agent to use when making requests, default:Mozilla/5.0Command: Code InterpreterExecuting CodeThe plugin operates similarly to theCode InterpreterinChatGPT, with the key difference that it works locally on the user's system. It allows for the execution of any Python code on the computer that the model may generate. When combined with theCommand: Files I/Oplugin, it facilitates running code from files saved in thedatadirectory. You can also prepare your own code files and enable the model to use them or add your own plugin for this purpose. You can execute commands and code on the host machine or in Docker container.Executing system commandsAnother feature is the ability to execute system commands and return their results. With this functionality, the plugin can run any system command, retrieve the output, and then feed the result back to the model. When used with other features, this provides extensive integration capabilities with the system.Tip:always remember to enable theExecute commandsoption to allow execute commands from the plugins.Options:Python command templatepython_cmd_tplPython command template (use {filename} as path to file placeholder).Default:python3 {filename}Enable: Python Code Generate and Executecmd_code_executeAllows Python code execution (generate and execute from file).Default:TrueEnable: Python Code Execute (File)cmd_code_execute_fileAllows Python code execution from existing file.Default:TrueEnable: System Command Executecmd_sys_execAllows system commands execution.Default:TrueSandbox (docker container)sandbox_dockerExecute commands in sandbox (docker container). Docker must be installed and running.Default:FalseDocker imagesandbox_docker_imageDocker image to use for sandboxDefault:python:3.8-alpineAuto-append CWD to sys_execauto_cwdAutomatically append current working directory to sys_exec command.Default:TrueCommand: Custom CommandsWith theCustom Commandsplugin, you can integratePyGPTwith your operating system and scripts or applications. You can define an unlimited number of custom commands and instruct GPT on when and how to execute them. Configuration is straightforward, andPyGPTincludes a simple tutorial command for testing and learning how it works:To add a new custom command, click theADDbutton and then:Provide a name for your command: this is a unique identifier for GPT.Provide aninstructionexplaining what this command does; GPT will know when to use the command based on this instruction.Defineparams, separated by commas - GPT will send data to your commands using these params. These params will be placed into placeholders you have defined in thecmdfield. For example:If you want instruct GPT to execute your Python script namedsmart_home_lights.pywith an argument, such as1to turn the light ON, and0to turn it OFF, define it as follows:name: lights_cmdinstruction: turn lights on/off; use 1 as 'arg' to turn ON, or 0 as 'arg' to turn OFFparams: argcmd:python /path/to/smart_home_lights.py {arg}The setup defined above will work as follows:When you ask GPT to turn your lights ON, GPT will locate this command and prepare the commandpython /path/to/smart_home_lights.py {arg}with{arg}replaced with1. On your system, it will execute the command:python /path/to/smart_home_lights.py 1And that's all. GPT will take care of the rest when you ask to turn ON the lights.You can define as many placeholders and parameters as you desire.Here are some predefined system placeholders for use:{_time}- current time inH:M:Sformat{_date}- current date inY-m-dformat{_datetime}- current date and time inY-m-d H:M:Sformat{_file}- path to the file from which the command is invoked{_home}- path toPyGPT's home/working directoryYou can connect predefined placeholders with your own params.Example:name: song_cmdinstruction: store the generated song on hard diskparams: song_text, titlecmd:echo "{song_text}" > {_home}/{title}.txtWith the setup above, every time you ask GPT to generate a song for you and save it to the disk, it will:Generate a song.Locate your command.Execute the command by sending the song's title and text.The command will save the song text into a file named with the song's title in the PyGPT working directory.Example tutorial commandPyGPTprovides simple tutorial command to show how it works, to run it just ask GPT for executetutorial test commandand it will show you how it works:> please execute tutorial test commandCommand: Files I/OThe plugin allows for file management within the local filesystem. It enables the model to create, read, and write files and directories located in thedatadirectory, which can be found in the user's work directory. With this plugin, the AI can also generate Python code files and thereafter execute that code within the user's system.Plugin capabilities include:Sending files as attachmentsReading filesAppending to filesWriting filesDeleting files and directoriesListing files and directoriesCreating directoriesDownloading filesCopying files and directoriesMoving (renaming) files and directoriesReading file infoIndexing files and directories using Llama-indexIf a file being created (with the same name) already exists, a prefix including the date and time is added to the file name.Options:Enable: Send (upload) file as attachmentcmd_send_fileAllowscmd_send_filecommand execution.Default:TrueEnable: Read filecmd_read_fileAllowsread_filecommand execution.Default:TrueEnable: Append to filecmd_append_fileAllowsappend_filecommand execution.Default:TrueEnable: Save filecmd_save_fileAllowssave_filecommand execution.Default:TrueEnable: Delete filecmd_delete_fileAllowsdelete_filecommand execution.Default:TrueEnable: List files (ls)cmd_list_filesAllowslist_dircommand execution.Default:TrueEnable: List files in dirs in directory (ls)cmd_list_dirAllowsmkdircommand execution.Default:TrueEnable: Downloading filescmd_download_fileAllowsdownload_filecommand execution.Default:TrueEnable: Removing directoriescmd_rmdirAllowsrmdircommand execution.Default:TrueEnable: Copying filescmd_copy_fileAllowscopy_filecommand execution.Default:TrueEnable: Copying directories (recursive)cmd_copy_dirAllowscopy_dircommand execution.Default:TrueEnable: Move files and directories (rename)cmd_moveAllowsmovecommand execution.Default:TrueEnable: Check if path is directorycmd_is_dirAllowsis_dircommand execution.Default:TrueEnable: Check if path is filecmd_is_fileAllowsis_filecommand execution.Default:TrueEnable: Check if file or directory existscmd_file_existsAllowsfile_existscommand execution.Default:TrueEnable: Get file sizecmd_file_sizeAllowsfile_sizecommand execution.Default:TrueEnable: Get file infocmd_file_infoAllowsfile_infocommand execution.Default:TrueEnable: Get current working directorycmd_cwdAllowscwdcommand execution.Default:TrueEnable: indexing filescmd_file_indexIf enabled, model will be able to index file or directory using Llama-index (file_indexcommand).Default:TrueIndex to use when indexing filesidxID of index to use for files indexing.Default:baseUse data loadersuse_loadersUse data loaders from Llama-index for file reading (read_filecommand).Default:TrueAuto index reading filesauto_indexIf enabled, every time file is read, it will be automatically indexed.Default:FalseOnly index reading filesonly_indexIf enabled, file will be indexed without return its content on file read.Default:FalseCommand: Web SearchPyGPTlets you connect GPT to the internet and carry out web searches in real time as you make queries.To activate this feature, turn on theCommand: Web Searchplugin found in thePluginsmenu.Web searches are provided byGoogle Custom Search EngineandMicrosoft BingAPIs and can be extended with other search engine providers.OptionsProviderproviderChoose the provider.Default:OpenAI TTSAvailable providers:GoogleMicrosoft BingGoogleTo use this provider, you need an API key, which you can obtain by registering an account at:https://developers.google.com/custom-search/v1/overviewAfter registering an account, create a new project and select it from the list of available projects:https://programmablesearchengine.google.com/controlpanel/allAfter selecting your project, you need to enable theWhole Internet Searchoption in its settings.
Then, copy the following two items intoPyGPT:Api KeyCX IDThese data must be configured in the appropriate fields in thePlugins / Settings...menu:Google Custom Search API KEYgoogle_api_keyYou can obtain your own API key athttps://developers.google.com/custom-search/v1/overviewGoogle Custom Search CX IDgoogle_api_cxYou will find your CX ID athttps://programmablesearchengine.google.com/controlpanel/all- remember to enable "Search on ALL internet pages" option in project settings.Microsoft BingBing Search API KEYbing_api_keyYou can obtain your own API key athttps://www.microsoft.com/en-us/bing/apis/bing-web-search-apiBing Search API endpointbing_endpointAPI endpoint for Bing Search API, default:https://api.bing.microsoft.com/v7.0/searchGeneral optionsNumber of pages to searchnum_pagesNumber of max pages to search per queryMax content charactersmax_page_content_lengthMax characters of page content to get (0 = unlimited)Per-page content chunk sizechunk_sizePer-page content chunk size (max characters per chunk)Disable SSL verifydisable_sslDisables SSL verification when crawling web pagesTimeouttimeoutConnection timeout (seconds)User agentuser_agentUser agent to use when making requests, default: Mozilla/5.0Max result lengthmax_result_lengthMax length of summarized result (characters)Max summary tokenssummary_max_tokensMax tokens in output when generating summaryEnable: "web_search" commandcmd_web_searchIf enabled, model will be able to search the WebEnable: "web_url_open" commandcmd_web_url_openIf enabled, model will be able to open specified URL and summarize contentEnable: "web_url_raw" commandcmd_web_url_rawIf enabled, model will be able to open specified URL and get raw contentEnable: "web_urls" commandcmd_web_urlsIf enabled, model will be able to search the Web and get founded URLs listIf enabled, model will be able to search the Web and get founded URLs listEnable: "web_index" commandcmd_web_indexIf enabled, model will be able to index web pages using Llama-indexAuto-index all used URLs using Llama-indexauto_indexIf enabled, every URL used by the model will be automatically indexed using Llama-indexIndex to useidxID of index to use for web page indexingModel used for web page summarizesummary_modelModel used for web page summarize, default: gpt-3.5-turbo-1106Summarize promptprompt_summarizePrompt used for web search results summarize, use {query} as a placeholder for search querySummarize prompt (URL open)prompt_summarize_urlPrompt used for specified URL page summarizeSyntax: web_searchsyntax_web_searchSyntax for web_search commandSyntax: web_url_opensyntax_web_url_openSyntax for web_url_open commandSyntax: web_url_rawsyntax_web_url_rawSyntax for web_url_raw commandSyntax: web_urlssyntax_web_urlsSyntax for web_urls commandSyntax: web_indexsyntax_web_indexSyntax for web_index commandCommand: Serial port / USBProvides commands for reading and sending data to USB ports.Tip:in Snap version you must connect the interface first:https://snapcraft.io/docs/serial-port-interfaceYou can send commands to, for example, an Arduino or any other controllers using the serial port for communication.Above is an example of co-operation with the following code uploaded toArduino Unoand connected via USB:// example.inovoidsetup(){Serial.begin(9600);}voidloop(){if(Serial.available()>0){Stringinput=Serial.readStringUntil('\n');if(input.length()>0){Serial.println("OK, response for: "+input);}}}OptionsUSB portserial_portUSB port name, e.g./dev/ttyUSB0,/dev/ttyACM0,COM3Default:/dev/ttyUSB0Connection speed (baudrate, bps)serial_bpsPort connection speed, in bpsDefault:9600TimeouttimeoutTimeout in secondsDefault:1SleepsleepSleep in seconds after connectionDefault:2Enable: Send text commands to USB portcmd_serial_sendAllowsserial_sendcommand execution"Default:TrueEnable: Send raw bytes to USB portcmd_serial_send_bytesAllowsserial_send_bytescommand executionDefault:TrueEnable: Read data from USB portcmd_serial_readAllowsserial_readcommand executionDefault:TrueSyntax: serial_sendsyntax_serial_sendSyntax for sending text command to USB portDefault:"serial_send": send text command to USB port, params: "command"Syntax: serial_send_bytessyntax_serial_send_bytesSyntax for sending raw bytes to USB portDefault:"serial_send_bytes": send raw bytes to USB port, params: "bytes"Syntax: serial_readsyntax_serial_readSyntax for reading data from USB portDefault:"serial_read": read data from serial port in seconds duration, params: "duration"Context history (calendar, inline)Provides access to context history database.
Plugin also provides access to reading and creating day notes.Examples of use, you can ask e.g. for the following:Give me today day noteSave a new note for todayUpdate my today note with...Get the list of yesterday conversationsGet contents of conversation ID 123etc.From version2.0.147it is possible to use@ID tags to automatically use summary of previous contexts in current discussion.
To use context from previous discussion with specified ID use following syntax in your query:@123Where123is the ID of previous context (conversation) in database, example of use:Let's talk about discussion @123OptionsEnable: using context @ ID tagsuse_tagsWhen enabled, it allows to automatically retrieve context history using @ tags, e.g. use @123 in question to use summary of context with ID 123 as additional context.Default:FalseEnable: get date range context listcmd_get_ctx_list_in_date_rangeWhen enabled, it allows getting the list of context history (previous conversations).Default:TrueEnable: get context content by IDcmd_get_ctx_content_by_idWhen enabled, it allows getting summarized content of context with defined ID.Default:TrueEnable: count contexts in date rangecmd_count_ctx_in_dateWhen enabled, it allows counting contexts in date range.Default:TrueEnable: get day notecmd_get_day_noteWhen enabled, it allows retrieving day note for specific date.Default:TrueEnable: add day notecmd_add_day_noteWhen enabled, it allows adding day note for specific date.Default:TrueEnable: update day notecmd_update_day_noteWhen enabled, it allows updating day note for specific date.Default:TrueEnable: remove day notecmd_remove_day_noteWhen enabled, it allows removing day note for specific date.Default:TrueModelmodel_summarizeModel used for summarize.Default:gpt-3.5-turboMax summary tokenssummary_max_tokensMax tokens in output when generating summary.Default:1500Max contexts to retrievectx_items_limitMax items in context history list to retrieve in one query. 0 = no limit.Default:30Per-context items content chunk sizechunk_sizePer-context content chunk size (max characters per chunk).Default:100000 charsOptions (advanced)Syntax: get_ctx_list_in_date_rangesyntax_get_ctx_list_in_date_rangeSyntax for get_ctx_list_in_date_range command.Syntax: get_ctx_content_by_idsyntax_get_ctx_content_by_idSyntax for get_ctx_content_by_id command.Syntax: count_ctx_in_datesyntax_count_ctx_in_dateSyntax for count_ctx_in_date commandSyntax: get_day_notesyntax_get_day_noteSyntax for get_day_note commandSyntax: add_day_notesyntax_add_day_noteSyntax for add_day_note command.Syntax: update_day_notesyntax_update_day_noteSyntax for update_day_note command.Syntax: remove_day_notesyntax_remove_day_noteSyntax for remove_day_note command.Prompt: @ tags (system)prompt_tag_systemPrompt for use @ tag (system).Prompt: @ tags (summary)prompt_tag_summaryPrompt for use @ tag (summary).Crontab / Task schedulerPlugin provides cron-based job scheduling - you can schedule tasks/prompts to be sent at any time using cron-based syntax for task setup.Your taskscrontabAdd your cron-style tasks here.
They will be executed automatically at the times you specify in the cron-based job format.
If you are unfamiliar with Cron, consider visiting the Cron Guru page for assistance:https://crontab.guruNumber of active tasks is always displayed in a tray dropdown menu:Create a new context on job runnew_ctxIf enabled, then a new context will be created on every run of the job.Default:TrueShow notification on job runshow_notifyIf enabled, then a tray notification will be shown on every run of the job.Default:TrueDALL-E 3: Image Generation (inline)The plugin integratesDALL-E 3image generation with any chat mode. Simply enable it and request an image in Chat mode, using a standard model such asGPT-4. The plugin does not require theExecute commandsoption to be enabled.OptionsPromptpromptThe prompt is used to generate a query for theDALL-Eimage generation model, which runs in the background.GPT-4 Vision (inline)The plugin integrates vision capabilities across all chat modes, not just Vision mode. Once enabled, it allows the model to seamlessly switch to vision processing in the background whenever an image attachment or vision capture is detected.Tip:When usingVision (inline)by utilizing a plugin in standard mode, such asChat(notVisionmode), the+ Visionspecial checkbox will appear at the bottom of the Chat window. It will be automatically enabled any time you provide content for analysis (like an uploaded photo). When the checkbox is enabled, the vision model is used. If you wish to exit the vision model after image analysis, simply uncheck the checkbox. It will activate again automatically when the next image content for analysis is provided.OptionsModelmodelThe model used to temporarily provide vision capabilities; default isgpt-4-vision-preview.PromptpromptThe prompt used for vision mode. It will append or replace current system prompt when using vision model.Replace promptreplace_promptReplace whole system prompt with vision prompt against appending it to the current prompt.Enable: "camera capture" commandcmd_captureAllows using command: camera capture (Execute commandsoption enabled is required).
If enabled, model will be able to capture images from camera itself.Enable: "make screenshot" commandcmd_screenshotAllows using command: make screenshot (Execute commandsoption enabled is required).
If enabled, model will be able to making screenshots itself.Real TimeThis plugin automatically adds the current date and time to each system prompt you send.
You have the option to include just the date, just the time, or both.When enabled, it quietly enhances each system prompt with current time information before sending it to GPT.OptionsAppend timehourIf enabled, it appends the current time to the system prompt.Default:TrueAppend datedateIf enabled, it appends the current date to the system prompt.Default:TrueTemplatetplTemplate to append to the system prompt. The placeholder{time}will be replaced with the
current date and time in real-time.Default:Current time is {time}.System Prompt Extra (append)The plugin appends additional system prompts (extra data) from a list to every current system prompt.
You can enhance every system prompt with extra instructions that will be automatically appended to the system prompt.OptionsPromptspromptsList of extra prompts - prompts that will be appended to system prompt.
All active extra prompts defined on list will be appended to the system prompt in the order they are listed here.Creating Your Own PluginsYou can create your own plugin forPyGPTat any time. The plugin can be written in Python and then registered with the application just before launching it. All plugins included with the app are stored in theplugindirectory - you can use them as coding examples for your own plugins.PyGPT can be extended with:Custom pluginsCustom LLMs wrappersCustom vector store providersCustom data loadersCustom audio input providersCustom audio output providersCustom web search engine providersExamples (tutorial files)See theexamplesdirectory in this repository with examples of custom launcher, plugin, vector store, LLM (Langchain and Llama-index) provider and data loader:examples/custom_launcher.pyexamples/example_audio_input.pyexamples/example_audio_output.pyexamples/example_data_loader.pyexamples/example_llm.pyexamples/example_plugin.pyexamples/example_vector_store.pyexamples/example_web_search.pyThese example files can be used as a starting point for creating your own extensions forPyGPT.Extending PyGPT with custom plugins, LLMs wrappers and vector stores:You can pass custom plugin instances, LLMs wrappers and vector store providers to the launcher.This is useful if you want to extend PyGPT with your own plugins, vectors storage and LLMs.To register custom plugins:Pass a list with the plugin instances aspluginskeyword argument.To register custom LLMs wrappers:Pass a list with the LLMs wrappers instances asllmskeyword argument.To register custom vector store providers:Pass a list with the vector store provider instances asvector_storeskeyword argument.To register custom data loaders:Pass a list with the data loader instances asloaderskeyword argument.To register custom audio input providers:Pass a list with the audio input provider instances asaudio_inputkeyword argument.To register custom audio output providers:Pass a list with the audio output provider instances asaudio_outputkeyword argument.To register custom web providers:Pass a list with the web provider instances aswebkeyword argument.Example:# custom_launcher.pyfrompygpt_net.appimportrunfrompluginsimportCustomPlugin,OtherCustomPluginfromllmsimportCustomLLMfromvector_storesimportCustomVectorStoreplugins=[CustomPlugin(),OtherCustomPlugin(),]llms=[CustomLLM(),]vector_stores=[CustomVectorStore(),]run(plugins=plugins,llms=llms,vector_stores=vector_stores)Handling eventsIn the plugin, you can receive and modify dispatched events.
To do this, create a method namedhandle(self, event, *args, **kwargs)and handle the received events like here:# custom_plugin.pyfrompygpt_net.core.dispatcherimportEventdefhandle(self,event:Event,*args,**kwargs):"""Handle dispatched events:param event: event object"""name=event.namedata=event.datactx=event.ctxifname==Event.INPUT_BEFORE:self.some_method(data['value'])elifname==Event.CTX_BEGIN:self.some_other_method(ctx)else:# ...List of EventsEvent names are defined inEventclass inpygpt_net.core.dispatcher.Event.Syntax:event name- triggered on,event data(data type):AI_NAME- when preparing an AI name,data['value'](string, name of the AI assistant)AUDIO_INPUT_STOP- force stop audio inputAUDIO_INPUT_TOGGLE- when speech input is enabled or disabled,data['value'](bool, True/False)AUDIO_OUTPUT_STOP- force stop audio outputAUDIO_OUTPUT_TOGGLE- when speech output is enabled or disabled,data['value'](bool, True/False)AUDIO_READ_TEXT- on text read with speech synthesis,data['value'](str)CMD_EXECUTE- when a command is executed,data['commands'](list, commands and arguments)CMD_INLINE- when an inline command is executed,data['commands'](list, commands and arguments)CMD_SYNTAX- when appending syntax for commands,data['prompt'], data['syntax'](string, list, prompt and list with commands usage syntax)CMD_SYNTAX_INLINE- when appending syntax for commands (inline mode),data['prompt'], data['syntax'](string, list, prompt and list with commands usage syntax)CTX_AFTER- after the context item is sent,ctxCTX_BEFORE- before the context item is sent,ctxCTX_BEGIN- when context item create,ctxCTX_END- when context item handling is finished,ctxCTX_SELECT- when context is selected on list,data['value'](int, ctx meta ID)DISABLE- when the plugin is disabled,data['value'](string, plugin ID)ENABLE- when the plugin is enabled,data['value'](string, plugin ID)FORCE_STOP- on force stop pluginsINPUT_BEFORE- upon receiving input from the textarea,data['value'](string, text to be sent)MODE_BEFORE- before the mode is selecteddata['value'], data['prompt'](string, string, mode ID)MODE_SELECT- on mode selectdata['value'](string, mode ID)MODEL_BEFORE- before the model is selecteddata['value'](string, model ID)MODEL_SELECT- on model selectdata['value'](string, model ID)PLUGIN_SETTINGS_CHANGED- on plugin settings updatePLUGIN_OPTION_GET- on request for plugin option valuedata['name'], data['value'](string, any, name of requested option, value)POST_PROMPT- after preparing a system prompt,data['value'](string, system prompt)PRE_PROMPT- before preparing a system prompt,data['value'](string, system prompt)SYSTEM_PROMPT- when preparing a system prompt,data['value'](string, system prompt)UI_ATTACHMENTS- when the attachment upload elements are rendered,data['value'](bool, show True/False)UI_VISION- when the vision elements are rendered,data['value'](bool, show True/False)USER_NAME- when preparing a user's name,data['value'](string, name of the user)USER_SEND- just before the input text is sent,data['value'](string, input text)You can stop the propagation of a received event at any time by settingstoptoTrue:event.stop = TrueFunctions and commands executeTip:gpt-4-1106-previewis the best model to use for command handling, Thegpt-4-turbo-previewmodel can sometimes refuse to execute commands.PyGPTuses an internal syntax to define commands and their parameters, which can then be used by the model and executed on the application side or even directly in the system. This syntax looks as follows (example command below):~###~{"cmd": "send_email", "params": {"quote": "Why don't skeletons fight each other? They don't have the guts!"}}~###~It is JSON wrapped between~###~. The application extracts the JSON object from such formatted text and executes the appropriate function based on the provided parameters and command name. Many of these types of commands are defined in plugins (e.g., those used for file operations or internet searches). You can also define your own commands using theCustom Commandsplugin, or simply by creating your own plugin and adding it to the application.Tip:TheExecute commandsoption checkbox must be enabled to allow the execution of commands from plugins. Disable the option if you do not want to use commands, to prevent additional token usage (as the command execution system prompt consumes additional tokens).A special system prompt responsible for invoking commands is added to the main system prompt if theExecute commandsoption is active.However, there is an additional possibility to define your own commands and execute them with the help of GPT.
These are functions - defined on the OpenAI API side and described using JSON objects. You can find a complete guide on how to define functions here:https://platform.openai.com/docs/guides/function-callinghttps://cookbook.openai.com/examples/how_to_call_functions_with_chat_modelsFrom version2.0.150, PyGPT offers compatibility of these functions with commands used in the application. All you need to do is define the appropriate functions using the syntax required by OpenAI, and PyGPT will do the rest, translating such syntax on the fly into its own internal format.You can define functions for modes:ChatandAssistants.
Note that - in Chat mode, they should be defined inPresets, and for Assistants, in theAssistantsettings.Example of usage:ChatCreate a new Preset, open the Preset edit dialog and add a new function using+ Functionbutton with the following content:Name:send_emailDescription:Sends a quote using emailParams (JSON):{"type":"object","properties":{"quote":{"type":"string","description":"A generated funny quote"}},"required":["quote"]}Then, in theCustom Commandsplugin, create a new command with the same name and the same parameters:Command name:send_emailInstruction/prompt:send mail(don't needed, because it will be called on OpenAI side)Params list:quoteCommand to execute:echo "OK. Email sent: {quote}"At next, enable theExecute commandsoption and enable the plugin.Ask GPT in Chat mode:Create a funny quote and email itIn response you will receive prepared command, like this:~###~{"cmd": "send_email", "params": {"quote": "Why do we tell actors to 'break a leg?' Because every play has a cast!"}}~###~After receiving this, PyGPT will execute the systemechocommand with params given fromparamsfield and replacing{quote}placeholder withquoteparam value.As a result, response like this will be sent to the model:[{"request": {"cmd": "send_email"}, "result": "OK. Email sent: Why do we tell actors to 'break a leg?' Because every play has a cast!"}]AssistantIn this mode (via Assistants API), it should be done similarly, with the difference that here the functions should be defined in the assistant's settings.With this flow you can use both forms - OpenAI and PyGPT - to define and execute commands and functions in the application. They will cooperate with each other and you can use them interchangeably.Token usage calculationInput tokensThe application features a token calculator. It attempts to forecast the number of tokens that
a particular query will consume and displays this estimate in real time. This gives you improved
control over your token usage. The app provides detailed information about the tokens used for the user's prompt,
the system prompt, any additional data, and those used within the context (the memory of previous entries).Remember that these are only approximate calculations and do not include, for example, the number of tokens consumed by some plugins. You can find the exact number of tokens used on the OpenAI website.Total tokensAfter receiving a response from the model, the application displays the actual total number of tokens used for the query (received from the API).ConfigurationSettingsThe following basic options can be modified directly within the application:Config -> Settings...GeneralOpenAI API KEY: The personal API key you'll need to enter into the application for it to function.OpenAI ORGANIZATION KEY: The organization's API key, which is optional for use within the application.Number of notepads: Number of notepad tabs. Restart of the application is required for this option to take effect.Minimize to tray on exit: Minimize to tray icon on exit. Tray icon enabled is required for this option to work. Default: False.LayoutFont Size (chat window): Adjusts the font size in the chat window.Font Size (input): Adjusts the font size in the input window.Font Size (ctx list): Adjusts the font size in contexts list.Font Size (toolbox): Adjusts the font size in toolbox on right.Layout density: Adjusts layout elements density. Default: -1.DPI scaling: Enable/disable DPI scaling. Restart of the application is required for this option to take effect. Default: True.DPI factor: DPI factor. Restart of the application is required for this option to take effect. Default: 1.0.Display tips (help descriptions): Display help tips, Default: True.Store dialog window positions: Enable or disable dialogs positions store/restore, Default: True.Use theme colors in chat window: Use color theme in chat window, Default: True.Disable markdown formatting in output: Enables plain-text display in output window, Default: False.Files and attachmentsStore attachments in the workdir upload directory: Enable to store a local copy of uploaded attachments for future use. Default: TrueStore images, capture and upload in data directory: Enable to store everything in single data directory. Default: FalseDirectory for file downloads: Subdirectory for downloaded files, e.g. in Assistants mode, inside "data". Default: "download"ContextContext Threshold: Sets the number of tokens reserved for the model to respond to the next prompt.Limit of last contexts on list to show  (0 = unlimited): Limit of the last contexts on list, default: 0 (unlimited)Use Context: Toggles the use of conversation context (memory of previous inputs).Store History: Toggles conversation history store.Store Time in History: Chooses whether timestamps are added to the .txt files.Context Auto-summary: Enables automatic generation of titles for contexts, Default: True.Lock incompatible modes: If enabled, the app will create a new context when switched to an incompatible mode within an existing context.Search also in conversation content, not only in titles: When enabled, context search will also consider the content of conversations, not just the titles of conversations.Model used for auto-summary: Model used for context auto-summary (default:gpt-3.5-turbo-1106).Prompt (sys): auto summary: System prompt for context auto-summary.Prompt (user): auto summary: User prompt for context auto-summary.ModelsMax Output Tokens: Determines the maximum number of tokens the model can generate for a single response.Max Total Tokens: Defines the maximum token count that the application can send to the model, including the conversation context.Temperature: Sets the randomness of the conversation. A lower value makes the model's responses more deterministic, while a higher value increases creativity and abstraction.Top-p: A parameter that influences the model's response diversity, similar to temperature. For more information, please check the OpenAI documentation.Frequency Penalty: Decreases the likelihood of repetition in the model's responses.Presence Penalty: Discourages the model from mentioning topics that have already been brought up in the conversation.Prompt (append): command execute instruction: Prompt for appending command execution instructions.ImagesDALL-E Image size: The resolution of the generated images (DALL-E). Default: 1792x1024.DALL-E Image quality: The image quality of the generated images (DALL-E). Default: standard.Open image dialog after generate: Enable the image dialog to open after an image is generated in Image mode.DALL-E: Prompt (sys): prompt generation: Prompt for generating prompts for DALL-E (if RAW mode is disabled).DALL-E: prompt generation model: Model used for generating prompts for DALL-E (if RAW mode is disabled).VisionVision: Camera capture width (px): Video capture resolution (width).Vision: Camera capture height (px): Video capture resolution (height).Vision: Camera IDX (number): Video capture camera index (number of camera).Vision: Image capture quality: Video capture image JPEG quality (%).Indexes (Llama-index)Indexes: List of created indexes.Vector Store: Vector store in use (vector database provided by Llama-index).Vector Store (**kwargs): Arguments for vector store (api_key, index_name, etc.).Recursive directory indexing: Enables recursive directory indexing, default is False.Replace old document versions in the index during re-indexing: If enabled, previous versions of documents will be deleted from the index when the newest versions are indexed, default is True.Excluded file extensions: File extensions to exclude if no data loader for this extension, separated by comma.Additional keyword arguments (**kwargs) for data loaders: Additional keyword arguments, such as settings, API keys, for the data loader. These arguments will be passed to the loader; please refer to the Llama-index or LlamaHub loaders reference for a list of allowed arguments for the specified data loader.Auto-index DB in real time: Enables conversation context auto-indexing.ID of index for auto-indexing: Index to use if auto-indexing of conversation context is enabled.DB (ALL), DB (UPDATE), FILES (ALL): Index the data – batch indexing is available here.Agent (autonomous)Sub-mode to use: Sub-mode to use in Agent mode (chat, completion, langchain, llama_index, etc.). Default: chat.Index to use: Only if sub-mode is llama_index (Chat with files), choose the index to use in Agent mode.Display a tray notification when the goal is achieved.: If enabled, a notification will be displayed after goal achieved / finished run.UpdatesCheck for updates on start: Enables checking for updates on start. Default: True.Check for updates in background: Enables checking for updates in background (checking every 5 minutes). Default: True.DeveloperShow debug menu: enables debug (developer) menuLog plugin usage to console: Enables logging of plugin usage to console.Log DALL-E usage to console: Enables logging of DALL-E usage to console.Log Llama-index usage to console: Enables logging of Llama-index usage to console.Log Assistants usage to console: Enables logging of Assistants API usage to console.Log level: toggle log level (ERROR|WARNING|INFO|DEBUG)JSON filesThe configuration is stored in JSON files for easy manual modification outside of the application.
These configuration files are located in the user's work directory within the following subdirectory:{HOME_DIR}/.config/pygpt-net/NotepadThe application has a built-in notepad, divided into several tabs. This can be useful for storing information in a convenient way, without the need to open an external text editor. The content of the notepad is automatically saved whenever the content changes.Advanced configurationManual configurationYou can manually edit the configuration files in this directory (this is your work directory):{HOME_DIR}/.config/pygpt-net/assistants.json- stores the list of assistants.attachments.json- stores the list of current attachments.config.json- stores the main configuration settings.models.json- stores models configurations.capture- a directory for captured images from camera and screenshotscss- a directory for CSS stylesheets (user override)history- a directory for context history in.txtformat.idx-Llama-indexindexesimg- a directory for images generated withDALL-E 3andDALL-E 2, saved as.pngfiles.locale- a directory for locales (user override)data- a directory for data files and files downloaded/generated by GPT.presets- a directory for presets stored as.jsonfiles.upload- a directory for local copies of attachments coming from outside the workdirdb.sqlite- a database with contexts, notepads and indexes data recordsapp.log- a file with error and debug logTranslations / LocaleLocale.inifiles are located in the app directory:./data/localeThis directory is automatically scanned when the application launches. To add a new translation,
create and save the file with the appropriate name, for example:locale.es.iniThis will add Spanish as a selectable language in the application's language menu.Overwriting CSS and locales with your own files:You can also overwrite files in thelocaleandcssapp directories with your own files in the user directory.
This allows you to overwrite language files or CSS styles in a very simple way - by just creating files in your working directory.{HOME_DIR}/.config/pygpt-net/locale- a directory for locales in.iniformat.css- a directory for CSS styles in.cssformat.Debugging and LoggingInSettings -> Developerdialog, you can enable theShow debug menuoption to turn on the debugging menu. The menu allows you to inspect the status of application elements. In the debugging menu, there is aLoggeroption that opens a log window. In the window, the program's operation is displayed in real-time.Logging levels:By default, all errors and exceptions are logged to the file:{HOME_DIR}/.config/pygpt-net/app.logTo increase the logging level (ERRORlevel is default), run the application with--debugargument:python3 run.py --debug=1orpython3 run.py --debug=2The value1enables theINFOlogging level.The value2enables theDEBUGlogging level (most information).UpdatesUpdating PyGPTPyGPTcomes with an integrated update notification system. When a new version with additional features is released, you'll receive an alert within the app.To get the new version, simply download it and start using it in place of the old one. All your custom settings like configuration, presets, indexes, and past conversations will be kept and ready to use right away in the new version.Coming soonEnhanced integration with LangchainMore vector databases supportDevelopment of autonomous agentsDISCLAIMERThis application is not officially associated with OpenAI. The author shall not be held liable for any damages
resulting from the use of this application. It is provided "as is," without any form of warranty.
Users are reminded to be mindful of token usage - always verify the number of tokens utilized by the model on
the OpenAI website and engage with the application responsibly. Activating plugins, such as Web Search,
may consume additional tokens that are not displayed in the main window.Always monitor your actual token usage on the OpenAI website.CHANGELOGRecent changes:2.1.1 (2024-02-29)Llama-index settings split into tabs.Added keyword argument settings for data loaders.Added data loaders for RSS, XML sitemaps, IPYNB notebooks, and XML and HTML files.Added audio notes - simply use speech recognition in the Notepad tab to append transcribed audio to the Notepad.Added an icon for quick toggling of video camera capture.Added a Video menu.Moved video capture toggle checkboxes to the Video menu.2.1.0 (2024-02-28)Upgraded Llama-index from the legacy version0.9to the current version0.10(note: upgrading PyPi version of PyGPT to2.1.0requires reinstallation in a virtual environment to remove old library versions).Upgraded Langchain to0.1.9.Upgraded PyInstaller (and compiled versions) to version6.Removed LlamaHub and restored support for Python3.12.Fixed imports for Llama-index in the compiled versions.Added icons for quickly toggling audio input/output.UI fixes.2.0.171 (2024-02-27)Added core for loading web resources (not only webpages) to Llama-index.Added a data loader for transcriptions of YouTube videos (simply ask for index a YouTube video URL using theCommand: Web Searchplugin - theweb_indexcommand must be enabled).Fixed the "Initial token count exceeds token limit" error in Chat with files mode.Added missing translations for dictionary items in settings dialogs.2.0.170 (2024-02-26)Added "simplified" mode to the "Audio Input" plugin, now set as default, allowing for faster and simpler speech recognition without requiring any configuration. Meanwhile, all current options have been moved to "advanced" mode.Improved microphone support in Snap version.Improved file browser update after performing file operations.Added check for file availability for indexing in the file browser.2.0.169 (2024-02-25)Improved and refactored debugging2.0.168 (2024-02-25)Added auto index file on read option in theFiles I/OpluginImproved debuggerFixed Bing speech recognition2.0.167 (2024-02-25)Fixed empty API key dialog.2.0.166 (2024-02-25)Enabled the use of data loaders from Llama-index in theFiles I/Oplugin for reading files, which allows for reading not only text files using the plugin'sread_filecommand.Added a command for indexing files to theFiles I/Oplugin.Theself_loopplugin has been internally renamed to theagentplugin.Fixed the display of pinned contexts when the indexed filter is enabled.2.0.165 (2024-02-25)Added search provider:Bing.Added audio output providers:GoogleandEleven Labs.Added audio input providers:Google,Google Cloud, andBing.Added a new plugin:Command: API callswhich allows defining commands for API calls to external services.Added anExcluded file extensionsconfiguration option inSettings -> Llama-index.Fixed audio input listener initialization in the Audio Input plugin.2.0.164 (2024-02-24)Refactored the Audio Input, Audio Output, and Google Web Search plugins to support multiple audio and search engine providers.Introduced the option to add your own audio input, output, and web search providers.2.0.163 (2024-02-23)Added storage of indexed urlsAdded info about used library versions in About dialog2.0.162 (2024-02-23)Index data storage migrated from JSON to DB.Added add/remove to index context menu on ctx list.Added last indexed info on ctx list.Added option "Replace old document versions in the index during re-indexing" in "Settings -> Llama-index".Refactored, optimized and improved file and context indexing.Fixed context search SQL query.2.0.161 (2024-02-22)Added the option to search within conversation content, not just by titles (disabled by default, can be enabled in: "Settings -> Context -> Search also in conversation content, not only in titles").Added a "download" subdirectory for files downloaded in Assistants mode and an option to configure the directory in: "Settings -> Files and attachments -> Directory for file downloads")Added logging of options available in: "Config -> Debug" to the logger window, not just to the console.Added descriptions in the models editor.2.0.160 (2024-02-21)Added auto-replacement of sandbox download links in Assistants with links to downloaded local files.Added setup for Poetry.Added connection timeout config option to Google Search plugin.Fixed recursion error on app exit.Updated dependencies.2.0.159 (2024-02-21)Added workdir placeholders to images stored from Assistants.Added tooltips on modes list.Added force synchronous command call if more than one command to execute.2.0.158 (2024-02-21)Improved Assistants: added annotations handling, added async function outputs submitFixed creating function duplicates on re-import existing Assistant2.0.157 (2024-02-20)Added webpage (URL) indexing using Llama-Index; new options were added to the Google Search plugin: the command "web_index" for indexing a specified page (just ask to index any URL) and "Auto-index", which enables automatic indexing of URLs used by the plugin (disabled by default).Decreased the default chunk size in the Google Search plugin.Assistant run creation and message sending moved to an async thread.Improved files receive from Assistants.2.0.156 (2024-02-19)Added disable SSL verify option for search engine calls (not only for search results) inGoogle Search plugin- issue #20.Updated certifi to 2024.2.2.Added custom launcher, custom plugin, custom vector store, custom LLM provider and custom data loader code examples inexamplesdirectory in repository.Updated documentation.2.0.155 (2024-02-18)Importing assistants and assistant files (from API) has been moved to an asynchronous thread.Upgraded OpenAI API to 1.12.0.2.0.154 (2024-02-18)Fixed hidden user input in Assistants modeFixed system prompt change when switching to Assistants mode without selected Assistant2.0.153 (2024-02-17)Added image display in the chat window when a file received from the Assistant is an imageAdded color labels in color select combo boxesAdded dialog window position save and restore (+ option in Settings to enable/disable this feature)Added command "web_url_raw" to the Google Search Plugin which allows you to get the raw HTML/txt content from web pages (without summary)Fixed CSS backup pathUpdated documentation2.0.152 (2024-02-16)Improved updater, added direct links to new releasesImproved command execute system promptAddedcwdcommand toFiles I/OpluginAdded automatic current working directory passing tosys_execcommand prompt inCode Interpreter.AddedRefreshoption in File ExplorerAdded monit about camera connect in Snap version with command to executeDisabled MenuRoles on MacOS-affected menu items - issue #19The full changelog is located in theCHANGELOG.mdfile in the main folder of this repository.Credits and linksOfficial website:https://pygpt.netDocumentation:https://pygpt.readthedocs.ioGitHub:https://github.com/szczyglis-dev/py-gptSnap Store:https://snapcraft.io/pygptPyPI:https://pypi.org/project/pygpt-netAuthor:Marcin Szczygliński (Poland, EU)Contact:info@pygpt.netLicense:MIT LicenseSpecial thanksGitHub community:kaneda2004moritz-t-wThird-party librariesFull list of external libraries used in this project is located in therequirements.txtfile in the main folder of this repository.All used SVG icons are fromMaterial Design Iconsprovided by Google:https://github.com/google/material-design-iconshttps://fonts.google.com/iconsCode of the Llama-index offline loaders integrated into app is taken from LlamaHub:https://llamahub.ai