Package: learn-to-pick
learn to pickIntroductionInstallationBasic UsageExample NotebooksAdvanced UsageCustom or Auto ScorerRegister callback functions after decision and before scoringStore progress of learned policyStop learning of learned policySet a different policyDifferent exploration algorithms and options for the default learned policyLearned policy's data logsAdvanced featurization optionsauto_embedexplicitly defined embeddingscustom featurizationother advanced featurization optionsNote: all code examples presented here can be found innotebooks/readme.ipynbIntroductionlearn_to_pickis a versatile Python library crafted for online learning in Reinforcement Learning (RL) loops. Specifically designed for Contextual Bandits, it allows you to choose an action from a set of possibilities based on certain criteria. After making a decision, you have multiple ways to evaluate its effectiveness and provide feedback back into the system, so that it can do better next time:Let a learned model (llm) determine the quality of the decision and provide a score.Use a custom score function to grade the decision.Directly specify the score manually and asynchronously.The beauty oflearn_to_pickis its flexibility. Whether you're a fan of VowpalWabbit or prefer PyTorch (coming soon), the library can seamlessly integrate with both, allowing them to be the brain behind your decisions.Installationpip install learn-to-pickBasic UsageThePickBestscenario should be used when:Multiple options are available for a given scenarioOnly one option is optimal for a specific criteria or contextThere exists a mechanism to provide feedback on the suitability of the chosen option for the specific criteriaExample usage with llm default scorer:fromlearn_to_pickimportPickBest# with an llm scorerclassfake_llm:defpredict(self,inputs):print(f"here are the inputs:{inputs}")dummy_score=1returndummy_scorepicker=PickBest.create(llm=fake_llm())result=picker.run(pick=learn_to_pick.ToSelectFrom(["option1","option2"]),criteria=learn_to_pick.BasedOn("some criteria"))print(result["picked"])The picker will:Make a selection using the decision making policyCall the scorer to evaluate the decisionUpdate the decision making policy with the scoreNote: callback functions can be registered at create time and they will be called after the decision has been made and before the scorer is called (see advanced section andprompt_variable_injection.ipynbfor example usage)Example usage with custom scorer:fromlearn_to_pickimportPickBest# with a custom scorerclassCustomSelectionScorer(learn_to_pick.SelectionScorer):defscore_response(self,inputs,picked,event)->float:print(f"inputs:{inputs}")pick=picked["pick"]criteria=event.based_on["criteria"]# evaluate pick based on criteriadummy_score=1returndummy_scorepicker=PickBest.create(selection_scorer=CustomSelectionScorer())result=picker.run(pick=learn_to_pick.ToSelectFrom(["option1","option2"]),criteria=learn_to_pick.BasedOn("some criteria"))print(result["picked"])fromlearn_to_pickimportPickBest# with delayed scorepicker=PickBest.create(selection_scorer=None)result=picker.run(pick=learn_to_pick.ToSelectFrom(["option1","option2"]),criteria=learn_to_pick.BasedOn("some criteria"))print(result["picked"])# evaluated the result asynchronusly in a different part of my system and determined a scoredummy_score=1picker.update_with_delayed_score(dummy_score,result)PickBestis highly configurable to work with a VowpalWabbit decision making policy, a PyTorch decision making policy (coming soon), or with a custom user defined decision making policyThe main thing that needs to be decided from the get-go is:What is the action space, what are the options I have to select fromEach time I need to pick an action from the action space, what are the criteria to make that decisionWhat is my feedback mechanismFor the feedback mechanism there are three available options:Default: use a LLM to score the selection based on the criteriaworks with a default scoring prompt if an llm object with apredict()call is passed in at initializationcustom scoring prompt can be providedCustom scoring function that has some logic baked in to evaluate selection based on the criteriaDelayed explicit scoring, where a score is passed back into the moduleIn all three cases, when a score is calculated or provided, the decision making policy is updated in order to do better next time.Example Notebooksreadme.ipynbshowcases all examples shown in this READMEnews_recommendation.ipynbshowcases a personalization scenario where we have to pick articles for specific usersprompt_variable_injection.ipynbshowcases learned prompt variable injection and registering callback functionalityAdvanced UsageCustom or Auto ScorerIt is very important to get the selection scorer right since the policy uses it to learn. It determines what is called the reward in reinforcement learning, and more specifically in our Contextual Bandits setting.The general advice is to keep the score between [0, 1], 0 being the worst selection, 1 being the best selection from the availableToSelectFromvariables, based on theBasedOnvariables, but should be adjusted if the need arises.In the examples provided above, the AutoSelectionScorer is set mostly to get users started but in real world scenarios it will most likely not be an adequate scorer function.Auto scorer with custom scoring criteria prompt:scoring_criteria_template="Given{criteria}rank how good or bad this selection is{pick}"picker=learn_to_pick.PickBest.create(selection_scorer=learn_to_pick.AutoSelectionScorer(llm=llm,scoring_criteria_template_str=scoring_criteria_template),)AutoSelectionScorer will have a system prompt that makes sure that the llm responsible for scoring returns a single float.However, if needed, a FULL scoring prompt can also be provided:# I want the score to be in the range [-1, 1] instead of the default [0, 1]REWARD_PROMPT="""Given{criteria}rank how good or bad this selection is{pick}IMPORANT: you MUST return a single number between -1 and 1, -1 being bad, 1 being good"""picker=learn_to_pick.PickBest.create(selection_scorer=learn_to_pick.AutoSelectionScorer(llm=llm,prompt=REWARD_PROMPT))Custom Scorer needs to extend the internalSelectionScorerand implement thescore_responsefunctionclassCustomSelectionScorer(learn_to_pick.SelectionScorer):defscore_response(self,inputs:Dict[str,Any],picked:Any,event:learn_to_pick.PickBestEvent)->float:# inputs: the inputs to the picker in Dict[str, Any] format# picked: the selection that was made by the policy# event: metadata that can be used to determine the score if needed# scoring logic goes heredummy_score=1.0returndummy_scoreRegister callback functions after decision and before scoringThe picker will:Make a selection using the decision making policyCall the scorer to evaluate the decisionUpdate the decision making policy with the scoreCallback functions can be registered at create time and they will be called after the decision has been made and before the scorer is called.defa_function(inputs,picked,event):# I want to set the score here for some reason instead of defining a scorerprint("hello world")event.selected.score=5.0returninputs,eventpicker=learn_to_pick.PickBest.create(callbacks_before_scoring=[a_function],selection_scorer=None)A full example can be found innotebooks/prompt_variable_injection.ipynbStore progress of learned policyThere is the option to store the decision making policy's progress and continue learning at a later time. This can be done by calling:picker.save_progress()which will store the learned policy in a file calledlatest.vw. It will also store it in a file with a timestamp. That way, ifsave_progress()is called more than once, multiple checkpoints will be created, but the latest one will always be inlatest.vwNext time the picker is loaded, the picker will look for a file calledlatest.vwand if the file exists it will be loaded and the learning will continue from there.By default the model checkpoints will be stored in the current directory but the save/load location can be specified at creation time:picker = learn_to_pick.PickBest.create(model_save_dir=<path to dir>, [...])Stop learning of learned policyIf you want the pickers learned decision making policy to stop updating you can turn it off/on:picker.deactivate_selection_scorer()andpicker.activate_selection_scorer()Set a different policyThere are two policies currently available:default policy:VwPolicywhich learns aVowpal WabbitContextual Banditmodelrandom policy:PickBestRandomPolicywhich doesn't learn anything and just selects a value randomly. this policy can be used to compare other policies with a random baseline one.custom policies: a custom policy could be created and set at chain creation timeDifferent exploration algorithms and options for the default learned policyThe defaultVwPolicyis initialized with some default arguments. The default exploration algorithm isSquareCBbut other Contextual Bandit exploration algorithms can be set, and other hyper parameters can be tuned (seeherefor available options).vw_cmd = ["--cb_explore_adf", "--quiet", "--squarecb", "--interactions=::"]picker = learn_to_pick.PickBest.create(vw_cmd = vw_cmd, [...])Learned policy's data logsTheVwPolicy's data files can be stored and examined or used to dooff policy evaluationfor hyper parameter tuning.The way to do this is to set a log file path torl_logson chain creation:picker = learn_to_pick.PickBest.create(rl_logs=<path to log FILE>, [...])Advanced featurization optionsauto_embedBy default the input that is wrapped (ToSelectFrom,BasedOn) is not tampered with. This might not be sufficient featurization, so based on how complex the scenario is you can set auto-embeddings to ONpicker = learn_to_pick.PickBest.create(featurizer=learn_to_pick.PickBestFeaturizer(auto_embed=True), [...])This will produce more complex embeddings and featurizations of the inputs, likely accelerating RL learning, albeit at the cost of increased runtime.By default,sbert.net's sentence_transformers'sall-mpnet-base-v2model will be used for these embeddings but you can set a different embeddings model by initializing featurizer with a different model. You could also set an entirely different embeddings encoding object, as long as it has anencode()function that returns a list of the encodings.fromsentence_transformersimportSentenceTransformerpicker=learn_to_pick.PickBest.create(featurizer=learn_to_pick.PickBestFeaturizer(auto_embed=True,model=SentenceTransformer("all-mpnet-base-v2")),[...])explicitly defined embeddingsAnother option is to define what inputs you think should be embedded manually:auto_embed = FalseCan wrap individual variables inlearn_to_pick.Embed()orlearn_to_pick.EmbedAndKeep()e.g.criteria = learn_to_pick.BasedOn(learn_to_pick.Embed("Tom"))custom featurizationAnother final option is to define and set a custom featurization/embedder class that returns a valid input for the learned policy.other advanced featurization optionsExplictly numerical features can be provided with a colon separator:age = learn_to_pick.BasedOn("age:32")ToSelectFromcan be a bit more complex if the scenario demands it, instead of being a list of strings it can be:a list of list of strings:pick=learn_to_pick.ToSelectFrom([["meal 1 name","meal 1 description"],["meal 2 name","meal 2 description"]])a list of dictionaries:pick=learn_to_pick.ToSelectFrom([{"name":"meal 1 name","description":"meal 1 description"},{"name":"meal 2 name","description":"meal 2 description"}])a list of dictionaries containing lists:pick=learn_to_pick.ToSelectFrom([{"name":["meal 1","complex name"],"description":"meal 1 description"},{"name":["meal 2","complex name"],"description":"meal 2 description"}])BasedOncan also take a list of strings:criteria=learn_to_pick.BasedOn(["Tom Joe","age:32","state of california"])there is no dictionary provided since multiple variables can be supplied wrapped inBasedOnStoring the data logs into a file allows the examination of what different inputs do to the data format.