Package: yival
YiVal⚡ Auto Prompting ⚡👉 Follow
us:|👉 Sponsored by Discord AIGC community:What is YiVal?YiVal: Your Automatic Prompt Engineering Assistant for GenAI ApplicationsYiVal is a state-of-the-art tool designed to streamline the tuning process for
your GenAI app prompts andANYconfigs in the loop. With YiVal, manual adjustments
are a thing of the past. Thisdata-driven and evaluation-centricapproach ensures
optimal prompts, precise RAG configurations, and fine-tuned model parameters.
Empower your applications to achieveenhanced results, reduce latency, and minimize inference costseffortlessly with YiVal!Problems YiVal trying to tackle:Prompt Development Challenge: "I can't create a better prompt. A score of 60
for my current prompt isn't helpful at all🤔."Fine-tuning Difficulty: "I don't know how to fine-tune; the terminology and
numerous fine-tune algorithms are overwhelming😵."Confidence and Scalability: "I learned tutorials to build agents from Langchain
and LlamaIndex, but am I doing it right? Will the bot burn through my money
when I launch? Will users like my GenAI app🤯?"Models and Data Drift: "Models and data keep changing; I worry a well-performing
GenAI app now may fail later😰."Relevant Metrics and Evaluators: "Which metrics and evaluators should I focus
on for my use case📊?"Check out our quickstart guide!Link to demoTiktok title autotuneInstallationDocker RuntimeInstall Docker and pull ourimage on DockerHub:dockerpullyival/release:latestRun our image:dockerrun--ityival/release:latestVSCode with Docker extension is recommended for running and developments. If you are developer using GPU with Pytorch, or need jupyter lab for data science:dockerpullyival/release:cu12_torch_jupyter
dockerrun--gpusall--it-p8888:8888yival/release:cu12_torch_jupyterPrerequisitesPython Version: Ensure you havePython 3.10or later installed.OpenAI API Key: Obtain an API key from OpenAI. Once you have the key, set
it as an environment variable namedOPENAI_API_KEY.Installation MethodsUsing pip (Recommended for Users)Install theyivalpackage directly using pip:pipinstallyivalDevelopment Setup Using PoetryIf you're looking to contribute or set up a development environment:Install Poetry: If you haven't already,install Poetry.Clone the Repository, or use CodeSpace:2.1Use CodeSpaceThe easiest way to get YiVal enviornment. Click below to use the GitHub Codespace, then go to the next step.2.2Clone the Repositorygitclonehttps://github.com/YiVal/YiVal.gitcdYiValSetup with Poetry: Initialize the Python virtual environment and install
dependencies using Poetry. Make sure to run the below cmd in/YiValdirectory:poetryinstall--syncTrying Out YiValAfter setting up, you can quickly get started with YiVal by generating datasets
of random tech startup business names.Steps to Run Your First YiVal ProgramNavigate to the yival Directory:cd/YiVal/src/yivalSet OpenAI API Key: Replace$YOUR_OPENAI_API_KEYwith your
actual OpenAI API key.On macOS or Linux systems,exportOPENAI_API_KEY=$YOUR_OPENAI_API_KEYOn Windows systems,setxOPENAI_API_KEY$YOUR_OPENAI_API_KEYDefine YiVal Configuration:
Create a configuration file namedconfig_data_generation.ymlfor automated
test dataset generation with the following content:description:Generate test datadataset:data_generators:openai_prompt_data_generator:chunk_size:100000diversify:truemodel_name:gpt-4input_function:description:# Description of the functionGiven a tech startup business, generate a corresponding landingpage headlinename:headline_generation_for_businessparameters:tech_startup_business:str# Parameter name and typenumber_of_examples:3output_csv_path:generated_examples.csvsource_type:machine_generatedExecute YiVal:
Run the following command from within the/YiVal/src/yivaldirectory:yivalrunconfig_data_generation.ymlCheck the Generated Dataset:
The generated test dataset will be stored ingenerated_examples.csv.Please refer toYiVal Docs Pagefor more details about YiVal!DemoDemo VideoDemo VideoUse Case DemoSupported FeaturesGithub LinkVideo Demo Link🐯 Craft your AI story with ChatGPT and MidJourneyMulti-modalsupport: Design an AI-powered narrative using YiVal's multi-modal support of simultaneous text and images. It supports native and seamlessReinforcement Learning from Human Feedback(RLHF) andReinforcement Learning from AI Feedback(RLAIF). Please watch the video above for this use case.🌟 Evaluate performance of multiple LLMs with your own Q&A test datasetConvenientlyevaluate and compareperformance of your model of choice against 100+ models, thanks toLiteLLM. Analyze model performance benchmarks tailored to yourcustomized test dataor use case.🔥 Startup Company Headline Generation BotStreamline generation of headlines for your startup with automated test datacreation, promptcrafting, resultsevaluation, and performanceenhancementvia GPT-4.🧳 Build a Customized Travel Guide BotLeverageautomated promptsinspired by the travelcommunity'smost popular suggestions, such as those fromawesome-chatgpt-prompts.📖 Build a Cheaper Translator: Use GPT-3.5 to teach Llama2 to create a translator with lower inference costUsingReplicateand GPT-3.5's test data, you canfine-tuneLlama2's translation bot. Benefit from 18x savings while experiencing only a 6% performance decrease.🤖️ Chat with Your Favorite Characters - Dantan Ji from Till the End of the MoonBring your favorite characters to life through automated prompt creation andcharacter script retrieval.🔍Evaluate guardrails's performance in generating Python(.py) outputsGuardrails: where are my guardrails? 😭<br>Yival: I am here. ⭐️<br><br>Theintegrated evaluationexperimentis carried out with 80 LeetCode problems in csv, using guardrail and using only GPT-4. The accuracy drops from 0.625 to 0.55 with guardrail, latency increases by 44%, and cost increases by 140%. Guardrail still has a long way to go from demo to production.🍨Visualize different foods around the world!🍱Just give the place where the food belongs and the best season to taste it, and you can get a video of the season-specific food!🤩🎈News article summary with CoDBy integrating the"Chain of Density"method,evaluate the enhancer's abilityin text summarization.🎆 Using3articles points generated by GPT-4 for evaluation, the coherent score increased by20.03%, the attributive score increased by25.18%!, the average token usage from2054.6 -> 1473.4(-28.3%)🚀.🥐 Automated TikTok Title Generation BotWith only two input lines, you can easily createconcise and polishedTikTok video titles based on your desired target audience and video content summaries. This is presented by ourauto-prompt feature: the process is automated, so you can input your requirements and enjoy the results hassle-free!Contribution GuidelinesIf you want to contribute to YiVal, be sure to review
thecontribution guidelines.
We useGitHub issuesfor tracking
requests and bugs.
Please joinYiVal's discord channelfor
general questions and discussion.
Join our collaborative community where your unique expertise as researchers and
software engineers is highly valued! Contribute to our project and be a part of
an innovative space where every line of code and research insight actively
fuels advancements in technology, fostering a future that is intelligently
connected and universally accessible.Contributors🌟 YiVal welcomes your contributions! 🌟🥳 Thanks so much to all of our amazing contributors 🥳Paper / Algorithm ImplementationPaperAuthorTopicsYiVal ContributorData GeneratorVariation GeneratorEvaluatorSelectorEnhancerConfigLarge Language Models Are Human-Level Prompt EngineersYongchao Zhou,Andrei Ioan Muresanu,Ziwen HanYiVal Evolver, Auto-Prompting@Tao FengOpenAIPromptDataGeneratorOpenAIPromptVariationGeneratorOpenAIPromptEvaluator,OpenAIEloEvaluatorAHPSelectorOpenAIPromptBasedCombinationEnhancerconfigBERTScore: Evaluating Text Generation with BERTTianyi Zhang,Varsha Kishore,Felix WuYiVal Evaluator, bertscore, rouge@crazycth--BertScoreEvaluator---AlpacaEvalXuechen Li,Tianyi Zhang,Yann Duboiset. alYiVal Evaluator@Tao Feng--AlpacaEvalEvaluator--configChain of DensityGriffin AdamsAlexander R. Fabbriet. alPrompt Engineering@Tao Feng-ChainOfDensityGenerator---configLarge Language Models as OptimizersChengrun YangXuezhi Wanget. alPrompt Engineering@crazycth----optimize_by_prompt_enhancerconfigLoRA: Low-Rank Adaptation of Large Language ModelsEdward J. HuYelong Shenet. alLLM Finetune@crazycth----sft_trainerconfig