Package: identitychain
IdentityChainInstallationCreate and Activate a Conda Environment.condacreate-nidchainpython=3.10
condaactivateidchainClone this Repository to your Local Environment.gitclonehttps://github.com/marcusm117/IdentityChain.gitInstall the Library with all Dependencies.cdIdentityChain
makedevelopUsageBefore the self-consistency evaluation, you need to make sure that one of the followings is satisfied:Your model is an Instruction-tuned Code LLM, and it's trained on both NL-to-PL and PL-to-NL tasks.Your model is a Foundation Code LLM, and it's trained on both completion and fill-in-the-middle tasks.To evaluate your model using IdentityChain, you need to prepare the followings:An evaluation dataset in the format of one of the followings (you can also use these two directly):EvalPlus-Mini-v0.1.6_reformatted.jsonlMBPP-S_test_reformatted.jsonlAn NL-to-PL prompt for your modelA PL-to-NL prompt for your modelAn NL-to-PL generation function for your modelA PL-to-NL generation function for your modelSeerun_identity_chain_openai.pyfor an example of how to use IdentityChain to evaluate OpenAI models.Seerun_identity_chain_huggingface.pyfor an example of how to use IdentityChain to evaluate HuggingFace open-source models. This example script already includes the following models:CodeLlama-Instruct-hf (7B, 13B, 34B)CodeLlama-hf (7B, 13B, 34B)starchat-betastarcoderstarcoderplusstarcoderbase (1B, 3B, 7B, 15B)ExampleUserun_identity_chain.shto execute scriptsrun_identity_chain_openai.pyorrun_identity_chain_huggingface.py, which conducts several IdentityChain evaluation in a batch. Make sure that you modify the followings before running the script:export CUDA_VISIBLE_DEVICES=0to specify the local GPU device you want to useexport HF_HOME=YOUR_OWN_PATH/huggingfaceto specify your own huggingface home path, where the model checkpoints will be cachedexport IDENTITY_CHAIN_HOME=YOUR_OWN_PATH/IdentityChainto your own IdentityChain home pathother parameters in the script for your own needsThen run the script:cdexamples
bashrun_identity_chain.shThis script will create a temporary foldertmpunder your IdentityChain home path, and store the results of IdentityChain evaluation in this folder, which will be ajsonlfile. For example,tmp/starcoderbase-1b/IDChain_starcoderbase-1b_tmp0.0g_len5_pb_all_m_v1_EvalPlus-Mini-v0.1.6_reformatted.jsonl.Useanalyze_results.pyto analyze the results of IdentityChain evaluation. It will geneartes anxlsxfile, which contains the following information:The SC (Self-Consistency) and SSC (Strong Self-Consistency) scores of the model at each self-iteration step. Note that SSC_0 is just Pass@1The aggregated TOM score (also BLEU and CodeBLEU) information at each step for the following 4 types of resulsts: Pass-Pass, Pass-Fail, Fail-Fail, Fail-PassThe TOM score (also BLEU and CodeBLEU) trajectory at each self-iteration step for each sample in the eavluation set.The raw test case outputs at each self-iteration stepcd../scripts
pythonanalyze_results.py--input_path../tmp/starcoderbase-1b/IDChain_starcoderbase-1b_tmp0.0g_len5_pb_all_m_v1_EvalPlus-Mini-v0.1.6_reformatted.jsonl--chain_length5The analyzed results will give you a sense of the model's overall performance, and the TOM score trajectory will help you pinpoint the exact step where the model makes a mistake.Usebrowse_results.pyto browse the results of IdentityChain evaluation. You can use this script to manually examine and study the mistakes made by the model for specific samples.cd../scripts
pythonbrowse_results.py--input_path../tmp/starcoderbase-1b/IDChain_starcoderbase-1b_tmp0.0g_len5_pb_all_m_v1_EvalPlus-Mini-v0.1.6_reformatted.jsonl--chain_length5--start0Linting & TestingWe use aMakefileas a command registry:make format: autoformat  this library withblackmake lint: perform static analysis of this library withblackandflake8make annotate: run type checking usingmypymake test: run automated testsmake check: check assets for packagingMake sure thatmake lint,make test, andmake checkall pass locally before submitting a Pull Request.