I am a plugin developer for the Godot 4 Game Engine. I am currently working on my plugin I named "GoPilot" which is kind of like Github Copilot, but for Godot, running g locally using ollama and FOSS!
I already made some very useful features integrated well into the Godot engine. Since this is the Godot Editor, we can create as many new interfaces and menus as we might want! These are the features I have already made
- Chat interface: A basic chat interface with your ollama models of choice. It's just a basic chat, like with any other. I added the Godot Logo and animated it as if the Godot robot were speaking when generating a response. I've also added a "improve prompt" button which improves the users written prompt by passing it to another LLM which returns an improved prompt.
- Code completipn: fully functional AI code suggestions on command with Alt+E ! It works really nicely and is done by overlaying a fake CodeEdit which looks as if its part of the actual script editor. It works surprisingly well. Code can be wither accepted with Tab or Escape or backspace
- table translation: I added a window for adding a new column to a translation .CSV file. The user can enter a language like 'German' and then press "Add column" and the LLM will generate a new translation for the given example text in English. I achieved this, by instructing the model individually each time for translating each row.
- Code redactor: By selecting some code and then pressing Alt+Shift+E the user opens a new window with the selected code on the left and a prompt field on the bottom with some buttons. The user can write an instruction into the field, like "Split this code up into multiple chunks" and after pressing "Generate", the LLM rewrites the given code with the instruction in mind into the large field on the right.

These are all the features I've worked on so far. But I would like to add some more really useful things. Here are some jumpstarters for your creativity:
- LLMs can be instructed individually for every task, like "Translate this specific row in the table" and it will then only respond with this exact row. This can be very useful for specific generation.
- LLMs can respond in JSON, which is great for integrating it into more parts if Godot, as I could parse its output and then apply the things it wrote via code, since I am the developer of this project.
- We can use the system prompt to tell the LLM exactly how it should behave in a chat interface.

We don't want to automate the creative and fun tasks of gamedev, like dialog writing and asset creation. We want to take away the heavy, unpleasant things.

1. I want to aim Gopilot at more advanced users. Godot newcomers probably don't get much out of a tool like this, besides maybe the table translation. Some Godot ground knowledge is required for using it. So, target audience: adanced
2. I would like to focus mostly on the script editor and maybe some integration into the scene tree. I am currently working on a prototype for automatic Node suggestions. But we focus on the script here.
3. Since Gopilot utalizes local LLMs, we are working with smaller models here. So in the size range of 1.5B to 7B parameters. That means, that these models aren't that intelligent and that we need to prompt them with very specific instructions, because otherwise they will wildely interpret what we want from it to output. The models only support text input and text output. We need to give the models all the context they might need for fulfilling a task in their prompt.
4: code generations are already acceptable or denyable by pressing either tab or escape. This already feels rather natural. The generation is started when the user presses a costomizable shortcut, default is Alt+E
5. The generating JSON part is just another way the LLMs can respond. So if we feel the need to parse the output from the LLM, for example for seperating its thinking and its code writing, we might instruct it to respond in this format: {"plan":"plan for the code...", "code_content":"content of the code here"}. This is just an example though. We should only use JSON when we want to seperate the LLMs response into multiple elements so that we can put it into multiple UI elements in the interface. (Like I did with the table translation already,  where it puts the outputs into seperate rows per translated text)
6. I am pretty happy with the existing features. I will get more feedback from some friends, who will kindly test the program for me. I want to add new featues, which empower users, specifically power users. I don't want to take away the creative aspects of gamedev, like level creation and story telling. That is the fun stuff, which should be done by real, creative humans. We want to make the deveopers feel empowered, not like we are taking their job. The next features might be the scene tree generation, which I will  probably use the JSON formatting feature for. But I will handle that. I want you to suggest some ideas please.
7. You bring up a good point! It is important to seperate the humans creative tasks and the generated content by the LLM. Over-automation is something we want to avoid at all costs. Gopilot should only be there helping the user, when they request it. The new scene tree suggestions for example will only initiate when the user right clicks on a node and then selects "Create Node Tree" with the Gopilot sparkle icon. This way, the user is always in control and never steered too much by the AI doing what its not supposed to.

In the Gopilot settings I defined two user customizable shortcuts:
The primary shortcut: Alt+E for code completion and refactoring when multiple lines of code are selected
The secondary shortcut: Alt+Shift+E for directly writing new functions into the given code or for adding comments to a selected function in the code

For initializing generations, I want to focus on using these two shortcuts. Their behaviour changes based on what you are currently doing. For now, they only do stuff in the script editor, but I'd like to expand their functionality to other spaces aswell.