Package: wopo
wordlabs.io Open Prompt Optimiser (or WOPO)Need I introduce prompt optimisation?Large Language Models (LLMs) are AI agents capable of performing specific tasks via natural language instruction on text based information.This makes them incredibly powerful. However, these machines lack idempotencyIdempotency: Given a question, the answer will always be the same. In math, 1 + 1 is always 2.
The result of the add operation on any two real numbers is always idempotent.
This, however, is not the case with LLMs, which may produce different results for the same question if asked twiceAdditionally, finding the best possible prompt that incorporates a lot of different aspects of thinking is a labour of effort, not of intellect. This task can be better performed by prompt optimisation libraries.WOPO: UsagePrerequisitesWOPO uses Prefect for orchestration, however, PyPi was unable to find the right versions for installation.
If not already installed, use the command belowpip install prefect==2.11.2Then install the WOPO librarypip install wopo==0.0.1Please note that this library is still in alpha release, code for WOPO will be changing rapidly in the coming months.
If you face any issues, make sure to drop a message here!UsagefromwopoimportWOPO"""For prompt optimisation, we basically need three things:1. Initial Prompt2. A set of context vs output (i.e. if operation prompt was performed on context, what would be the correct output>)3. A selection strategy (to decide how to choose the right promptYou will also need to pass a function that takes in a string and sends it to the LLM and returns the string.Pass this function in the keyword argument text_gen_funcThis keeps things simple and allows you to write any kind of function you'd like to interact with your LLM"""importopenaiopenai.organisation='ORG-ID'openai.api_key='API_KEY'defgenerate_func(prompt,return_explanation=False):response=openai.ChatCompletion.create(model="gpt-3.5-turbo",messages=[{"role":"system","content":prompt}])resp_text=response['choices'][0]['message']['content']returnresp_textprompt="Remove vowels"#List of ip/op pairs with labels context and outputdata=[{"context":"Sentence","output":"Sntnc"},{"context":"This is a sentence","output":"Ths s sntnc"}]#Initialise optimisery=WOPO(prompt,data,generate_func)"""num_iters: number of times we perform optimisationnum_steps_per_iter: number of times the prompt is updated in each steptop_k: at the end of each step, how many best prompts are selected to be merged into one(this is being done so that the prompt generalises over multiple cases instead of specialising for one)Returns:optimal_promptresults: scores of each stepagent_states: complete logs of how each step changed the prompt and related feedback"""optimal_prompt,results,agent_states=y.run_optimisation(num_iters=5,num_step_per_iter=1,top_k=2)"""You can also run simple tests to analyse how well the new prompt is workingThe below function will return a Pandas DataFrame containing all the relevant information,and also save the file to specified save location"""test=y.run_test(some_other_data,save_location="test_result.csv")If you only have one ip/op pair, useWOPO.run_single_chain_optimisation()function. You may additionally specify thestop_at_scorecriteria (between 0-100) at which the chain can stop earlyAvailable selection strategiesMax: Choose the prompt that gets closest to the answerWOPO(strategy = 'max')Top K: Combine the prompts from the top k highest scoring promptsWOPO(strategy='top_k')WOPO.run_optimisation(top_k=top_k)Random Selection from Top K: Given top k scoring prompts, select n random prompts from the top k promptsWOPO(strategy='random_from_top_k')WOPO.run_optimisation(top_k=top_k,random_sample_size=random_sample_size)Prompt MinificationOptimal prompts offer better accuracy in terms of output, but they tend to be verbose. Most LLMs are not cheap to operate, and it is best to use the fewest number of tokens possible in prompting. To reduce the number of tokens being used, we can simply find which words are most likely to be implicitly understood by the LLM even if they are removed.For example, if I were to say 'The quick brown fox' you immediately think of 'jumps over the lazy dog', even though 'ate all my peanut butter' was also a valid sentence
The process is called Entropy Minification."""Specify the model name (default: bert-base-uncased) from HuggingFace Transformers library and provide a percentile score (default: 0.1).The tokens falling in the top percentile score of likelihood will be removed"""optimal_prompt,_,_=WOPO.run_optimisation()minified_prompt=WOPO.minify(model_name=model_name,percentile=percentile)