Package: dir-diary
dir-diaryOverviewdir-diaryis a Python library and CLI tool for using LLMs to generate a pseudocode summary and/or summary usage instructions for your entire software project folder. This summary can serve as an intermediate step for development, translation, documentation-writing, or embedding for Q&A. The tool leverages GPT-3.5 Turbo to summarize a decent-sized project for just 2-4 cents.dir-diarycan be used from the command line, in a Python script, or in an automated cloud deployment workflow via Github Actions (see our customsetup-dir-diaryaction andexample workflow).InstallationIf you installdir-diaryviapip, the tool should install and be added to your system PATH. If you install it manually, you may need to manually add it to your PATH.PyPi Installationpipinstall-Udir-diaryClone from GitHubgitclonehttps://github.com/Promptly-Technologies-LLC/dir-diary.gitcddir-diary
pipinstall.UsageSetting an OpenAI API KeySetting environment variables for local command-line usage can be done in several ways, depending on your operating system and shell. Here are some methods:1. Setting a System Environment VariableIf you want to use thedir-diarytool frequently on a private system, you may want to set a persistent system environment variable to hold your API key. The method differs depending on your system and shell.On Linux or MacOS, you can use this Bash command:echo 'export OPENAI_API_KEY=your_api_key_here` >>~/.bash_profileIn Windows Powershell, you can set a system environment variable from the command line:[System.Environment]::SetEnvironmentVariable('OPENAI_API_KEY','your_api_key_here')Alternatively, you can set it through the Windows Control Panel:"Control Panel" > "System" > "Advanced system settings" > "Advanced" > "Environment variables" > "System Variables" > "New"
"Name": OPENAI_API_KEY
"Value":your_api_key_here2. Exporting a Key for a Single Shell SessionYou can set an environment variable for the duration of your shell session, making it available todir-diaryin the same session. The exact command differs depending on which shell you use.In Bash, use theexportcommand:exportOPENAI_API_KEY=your_api_key_hereIn Windows Command Prompt, you can use thesetcommand:set OPENAI_API_KEY=your_api_key_hereIn Windows PowerShell, you can use the$env:prefix:$env:OPENAI_API_KEY="your_api_key_here"3. Using an.envFileYou can also place your environment variables in a.envfile in the directory from which you're running the tool:OPENAI_API_KEY=your_api_key_hereWhen you run your script, the tool will automatically load these variables into the environment.Using an option flagYou can also pass your API key as an option flag to the CLI tool (--api_key="your_openai_api_key") or Python API (api_key="your_openai_api_key"). For security reasons, this is not recommended, as it will expose your API key in your shell or command-line history.Creating a.gitignorefileBefore running either the Python or CLI tool, make sure to create a.gitignorefile in your project folder and add any files you wantdir-diaryto ingore in its classification and summarization steps. For instance, if you've placed your OpenAI API key in a.envfile, you should add.envto the.gitignorefile. By default, thedir-diarytool should ignore.envanyway. But adding this file to.gitignorewill help make absolutely certain that your secrets are never exposed through an API call to the LLM or through being included in a summary. The same goes for any other files that may contain sensitive information.You should also add dependency and environment folders such asnode_modules. Such folders may have hundreds or thousands of files and will cause summarization to fail due to exceeding context length.Using the Tool from the Command-Line InterfaceTo summarize a project folder, use thesummarizecommand in your shell from the folder you want to summarize. The CLI tool will automatically map the project folder, classify files by their role in the project, and generate a pseudocode summary of all project files with the roles you specify for inclusion in the summary. By default, the tool will create adocsfolder if one does not already exist, and then will createproject_map.jsonandpseudocode.mdfiles in that folder. The default paths for these files can be adjusted using the--pseudocode_fileand--project_map_fileoptions.Theproject_map.jsonfile contains a structured representation of the project folder file structure, and thepseudocode.mdfile contains the pseudocode summary of the project as generated by the specified OpenAI model. By default,dir-diaryuses the GPT-3.5 Turbo model with 8k context length and will fall back to GPT-3.5 Turbo 16k if necessary. The tool currently only supports OpenAI's chat models.Note that the syntax for setting option flags differs slightly depending on your shell. Bash uses an equal sign, while Windows Powershell and Command Prompt use a space, as in the examples below. Also note that for arguments that can take multiple values, you will need to invoke the option flag separately for each value (as shown here for--include).Bash/Zshsummarize--startpath="./"--summary_types="pseudocode"--include"source"--include"testing"--api_key="your_openai_api_key"PowerShell or Windows CMDsummarize--startpath".\"--summary_types"pseudocode"--include"source"--include"testing"--api_key"your_openai_api_key"Option FlagsOption Flags--startpath: Specifies the path to the project folder you want to summarize. Defaults to the current directory.--destination: Specifies the destination folder where generated files will be saved. The folder will be created if does not exist in the startpath directory. Defaults to--destination "docs".--summary_types: Specified the types of summaries to generate. Accepts multiple values. Valid options are: "pseudocode", "tech stack", and "usage instructions". Only "pseudocode" is currently supported. Defaults to--summary_types "pseudocode".--include: Specifies the types of files to include in the summary based on their roles. Accepts multiple values. Valid options are: "source", "configuration", "build or deployment", "documentation", "testing", "database", "utility scripts", "assets or data", and "specialized". Defaults to--include "source" --include "utility scripts".--api_key: Your OpenAI API key. An API key is required for the tool to function, but the option flag is not required if you've set the API key as an environment variable.--model_name: Specifies the OpenAI model to use for generating the summary. Defaults to--model_name "gpt-3.5-turbo". Valid options are: "gpt-3.5-turbo", "gpt-3.5-turbo-0301", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-4", "gpt-4-0314", "gpt-4-0613".--long_context_fallback: Specifies the fallback OpenAI model to use when the context is too long for the primary model. Defaults to--long_context_fallback "gpt-3.5-turbo-16k". Valid options are: "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613".--temperature: Sets the "temperature" for the OpenAI model, affecting the randomness of the output. Defaults to--temperature 0.Debugging Authentication ErrorsI've encountered some langchain authentication errors when running from a local Windows terminal. If you run into this, let me know by opening an issue. The problem doesn't seem to occur in Github Actions runners or if you have a Python virtual environment activated, so you can try activating a venv before using the tool as a workaround.Using dir-diary as Part of a CI/CD PipelineThedir-diarytool is primarily intended to be used in an automation workflow or as part of a CI/CD or automation pipeline. For this purpose, we have released aGithub Actionthat handles setup of the tool, which can be used like this:-name:Setup Python and dir-diaryuses:Promptly-Technologies-LLC/setup-dir-diary@v1with:install-python:'true'We have also released a completeexample workflowincorporating this setup action that demonstrates how to automate summarizing a repository and pushing the generated files back to the repo. Note that to use this workflow, you will need to create a Github repository secret named OPENAI_API_KEY with your API access key. Create your secret, add thesummarize.ymlworkflow to your repository, and edit to your liking.Using the Tool from the Python APIInside a Python script, you can summarize your project folder with thesummarize_project_folderfunction, which takes the same arguments as the CLI tool.Example UsageTo use the tool from a Python script, import and invoke as follows:fromdir_diaryimportsummarize_project_foldersummarize_project_folder(startpath="./my_project",pseudocode_file="./my_project/docs/pseudocode.md",project_map_file="./my_project/docs/project_map.json",include=["source","documentation"],api_key="your_api_key_here",model_name="gpt-4",long_context_fallback="gpt-4-0314",temperature=0.7)This will generate a pseudocode summary of the "my_project" folder, save it to "./my_project/docs/pseudocode.md", and also create a project map in "./my_project/docs/project_map.json".Parametersstartpath(Union[str, PathLike], default: "."):
The starting path of the project folder you want to summarize. You can provide this as either a string or aPathLikeobject. The default is the current directory (".").destination(Union[str, PathLike], default: "docs"):
The destination folder where generated files will be saved. The folder will be created if does not exist in the startpath directory. You can provide this as either a string or aPathLikeobject. The default is "docs".summary_types(list[Literal], default: ["pseudocode"]):
A list specifying which types of summaries to generate. Valid options are: "pseudocode", "tech stack", and "usage instructions". Only "pseudocode" is currently supported. The default is "pseudocode".include(list[Literal], default: ["source", "utility scripts"]):
A list specifying which types of files to include in the summary. Valid options are: "source", "configuration", "build or deployment", "documentation", "testing", "database", "utility scripts", "assets or data", and "specialized". The default is to include "source" and "utility scripts".api_key(str, default: None):
The API key for the language model. If not provided, the function will use the default API key if available.model_name(Literal, default: "gpt-3.5-turbo"):
The name of the language model to use for generating the summary. Valid options are: "gpt-3.5-turbo", "gpt-3.5-turbo-0301", "gpt-3.5-turbo-0613", "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613", "gpt-4", "gpt-4-0314", "gpt-4-0613".long_context_fallback(Literal, default: "gpt-3.5-turbo-16k"):
The name of the fallback language model to use when the primary model's token limit is exceeded. Valid options are the same as formodel_name. Valid options are: "gpt-3.5-turbo-16k", "gpt-3.5-turbo-16k-0613".temperature(float, default: 0):
The temperature setting for the language model, affecting the randomness of the output. A value of 0 makes the output deterministic, while higher values (up to 1) make it more random.Tools for Working with the Pseudocode Summary FileThe Python API also exposes theread_pseudocode_filefunction and theModulePseudocodeclass, which can be used to read and parse the pseudocode summary file. TheModulePseudocodeclass is a pydantic model that can be used to validate the JSON file and access its contents. It has the attributespath,modified, andcontent, which correspond to the path of the module, the last modified timestamp of the module, and the pseudocode summary of the module, respectively. Theread_pseudocode_filefunction returns a list ofModulePseudocodeobjects, one for each module in the project folder.fromdir-diaryimportread_pseudocode_file,ModulePseudocodepseudocode:list[ModulePseudocode]=read_pseudocode_file("./docs/pseudocode.md")ContributingWe welcome contributions! Feel free to submit pull requests for new features, improvements, or bug fixes. Please make sure to follow best practices and include unit tests using the pytest framework.For any issues, pleasecreate an issue on GitHub.Modules and Key Functionssummarize.py: Orchestrates the summarization of the entire project folder.cli.py: Defines the command-line interface for the tool.datastructures.py: Defines classes and validators for working with data.mapper.py: Maps the project folder.chatbot.py: Initializes and queries the chatbot model.classifier.py: Classifies files' roles by querying the chatbot.summarizer.py: Summarizes individual files by querying the chatbot.file_handler.py: Handles reading and writing of pseudocode files.For a more detailed understanding, please refer to the source code, inline comments, and, most importantly,docs\pseudocode.md!To-doRemove langchain because of breaking authentication bug? (Increased latency but reduced reported cost, although this may be because langchain miscalculates cost)Add console logging of prompts and completionsDeal with condition where LLM response['choices']['finish_reason'] == 'length' (success code but response exceeds context length)Fix the way we handle exceeding context length (add filtering, splitting)Usellm_cost_estimation.modelsto get max_tokensMake summarization API calls asynchronous to reduce runtimeAdd a final summarization step to reconcile the individual file summariesAdd tech stack summarization featureAdd support for LLMs other than OpenAI's chat modelsAdd support for Microsoft Azure OpenAI service to reduce latency/costDo some YouTube explainers/demosAdd more unit tests