Package: manitest
ManiTestAn LLM evaluation harness built with Manifest + EleutherAIManiTestis a simplified version of theEleuther-AI LLM evaluation harnessthat usesManifestas its backend model server.Eleuther-AI LLM evaluation harnessallows you to evaluate a large language model (LLM) on tasks formulated as prompts.Manifestis a model server that enables fast inference via built-in support for HuggingFace Parallelize, Accelerate, DeepSpeed, and BitsAndBytes.Note: We use our own fork of Manifest, which we hope to merge back into the main repo soon.Installationpip3installmanitest"git+https://github.com/som-shahlab/manifest.git@eval-michael#egg=manifest-ml[api]"QuickstartTo run the eval harness, you must first have a Manifest server running in the background with your desired model. You can then run the eval harness on your desired task. ManiTest comes with a couple tasks pre-loaded in themanitest.tasksmodule, such asmanitest.tasks.mednliandmanitest.tasks.scitail.# Run Manifest server with your desired modelpython3-mmanifest.api.app\--model_typehuggingface\--model_name_or_pathgpt2\--model_generation_typetext-generation\--port5001&# Run ManiTest evaluation harness on your desired task# Note: To run the MedNLI task, you must first download the dataset from: https://physionet.org/content/mednli/1.0.0/python3-mmanitest.main\--manifest_urlhttp://127.0.0.1:5001\--path_to_taskmanitest.tasks.mednli\--output_dir./ignore\--data_dir/Users/mwornow/Downloads/mednli-a-natural-language-inference-dataset-for-the-clinical-domain-1.0.0/\--dataset_splitstest# Test a few-shot prompting setuppython3src/manitest/main.py\--manifest_urlhttp://127.0.0.1:5001\--path_to_taskmanitest.tasks.mednli_fewshot\--output_dir./ignore\--data_dir/Users/mwornow/Downloads/mednli-a-natural-language-inference-dataset-for-the-clinical-domain-1.0.0/\--dataset_splitstest\--n_shots3TipsIf you're using a causal LM (e.g. GPT, OPT, Llama, Bloom)...Run Manifest with the--model_generation_type text-generationflagIf you're using a seq2seq LM (e.g. T5, T0)...Run Manifest with the--model_generation_type text2text-generationflagHow to create your own taskWe recommend starting from thetask_template.pyfile as a template. You can also viewtests/mednli/mednli.pyortests/scitail/scitail.pyfor worked-out examples of tasks.To create your own task, you must...Create a file calledyour_task.py. You can save this anywhere.Create aTaskclass that inherits frommanifest.base.Task.It must define three attributes (name,task_type, andprompts) and one methods (load_dataset()).frombaseimportTask,TaskTypeclassYourTask(Task):name:str="Your Task Name"task_type:TaskType=TaskType.GENERATIONdefload_dataset(self,dataloader:Optional[str],data_dir:Optional[str])->DatasetDict:# Load your dataset herereturnDatasetDict()Create aPromptclass that inherits frommanifest.base.Promptfor each individual prompt associated with your task.It must define one attribute (name) and two methods (generate_prompt()andget_label()).frombaseimportPromptclassYourPrompt(Prompt)name:str="Some globally unique name for this prompt"defgenerate_prompt(self,example:dict)->str:"""Takes a dataset example and returns a prompted version of that example."""returnf"Premise:{example['premise']}\nHypothesis:{example['hypothesis']}. Does the premise entail the hypothesis?"defget_label(self,example:dict)->str:"""Gets the ground truth label for a dataset example"""returnexample['true_label']Run the evaluation harness with your task. This assumes a Manifest server is already running onlocalhost:5000python3main.py\--manifest_urlhttp://localhost:5000\--path_to_taskpath/to/your_task.py\--output_dir./ignoreDevelopmentInstallation:# Download repogitclonehttps://github.com/som-shahlab/manitestcdmanitest# Create virtual environment + install dependenciescondacreate--namemanitest_envpython=3.10-y
condaactivatemanitest_env
poetryinstallSpecial setup instructions for computers without Internet accessIf you are running this on a computer without internet access (e.g. Stanford Nero), you will need to download the HuggingFace dataset, dataloader, and model that you want to use.Assuming you've downloaded these, your commands will look like the following:python3 -m manifest.api.app \
    --model_type huggingface \
    # Path to locally downloaded HuggingFace model
    --model_name_or_path /local-scratch-nvme/nigam/huggingface/pretrained/gpt2-small \
    --model_generation_type text-generation

python3 main.py \
    --manifest_url http://127.0.0.1:5000 \
    --path_to_task tests/mednli/mednli.py \
    --output_dir ./ignore \
    # Path to locally downloaded HuggingFace dataset
    --data_dir /local-scratch/nigam/projects/clinical_llm/data/mednli/ \
    --dataset_splits test \
    # Path to locally downloaded HuggingFace dataloader
    --dataloader /local-scratch/nigam/projects/clinical_llm/dataloaders/mednli/mednli.pyTodosCombine Manifest command withmain.pycommand into a single commandMerge Manifest fork back into main repoSupport specifying multiple tasks in a single runPretty print / format resultsSupport passing text generation flags tomain.py, pass these along to ManifestAdd testsDocumentationMulti-label classification task supportFew-shot in-context examplesMulti-class classification task supportText generation task supportTest case with mednli replacement classification taskAbstract things to work on not Nero (e.g. load HF models from Hub, load local HF models, hit external APIs)Clean up output to be more user friendlyConvert .yaml prompt files to .pyTaskfilesDevelopmentTo run pre-commit checks:pre-commitrun--all-filesTo run tests:pytesttests