Package: manifest-ml
ManifestHow to make prompt programming with Foundation Models a little easier.Table of ContentsInstallGetting StartedManifestOther Models TypesLocal HuggingFace ModelsChat ModelsEmbedding ModelsRoad MapDevelopmentCiteInstallInstall:pipinstallmanifest-mlInstall with diffusion support:pipinstallmanifest-ml[diffusers]Install with HuggingFace local model support:pipinstallmanifest-ml[api]Dev Install:gitclonegit@github.com:HazyResearch/manifest.gitcdmanifest
makedevGetting StartedRunning is simple to get started. If using OpenAI, setexport OPENAI_API_KEY=<OPENAIKEY>(or pass key in through variableclient_connection) then runfrommanifestimportManifest# Start a manifest session to OpenAI - default `engine=text-davinci-003`manifest=Manifest(client_name="openai",)manifest.run("Why is the grass green?")ExamplesWe have example notebook and python scripts located atexamples. These show how to use different models, model types (i.e. text, diffusers, or embedding models), and async running.Manifest ComponentsManifest is meant to be a very light weight package to help with prompt design and iteration. Three key design decisions of Manifest areAll models are behind APIsSupports caching of model inputs/outputs for iteration, reproducibility, and cost savingUnified API to support generate, score, and embedModelsManifest provides model clients forOpenAI,AI21,Cohere,Together, and HuggingFace (seebelowfor how to use locally hosted HuggingFace models). You can toggle between the models by changingclient_nameandclient_connection. For example, if a HuggingFace model is loaded locally, runmanifest=Manifest(client_name="huggingface",client_connection="http://127.0.0.1:5000",)If you want to use Cohere, runmanifest=Manifest(client_name="cohere",client_connection=<COHERE_API_KEY>,)You can also just setexport COHERE_API_KEY=<COHERE_API_KEY>and not useclient_connection.If you want to use AI21 Labs, runmanifest=Manifest(client_name="ai21",client_connection=<AI21_API_KEY>,)You can see the model details and possible model inputs torun()viaprint(manifest.client_pool.get_current_client().get_model_params())print(manifest.client_pool.get_current_client().get_model_inputs())Global CacheWe support having queries and results stored in a global cache that can be shared across users. We treat inputs and outputs as key value pairs and support SQLite or Redis backends. To start with global caching using SQLite, runmanifest=Manifest(client_name="openai",cache_name="sqlite",cache_connection="mycache.sqlite",)The cache will be saved inmycache.sqlite.We also support Redis backend.manifest=Manifest(client_name="openai",cache_name="redis",cache_connection="localhost:6379")As a hint, if you want to get Redis running, see thedocker runcommand below under development.Running QueriesOnce you have a session open, you can write and develop prompts.result=manifest.run("Hello, my name is Laurel")You can also run over multiple examples if supported by the client.results=manifest.run(["Where are the cats?","Where are the dogs?"])We support async queries as well viaimportasyncioresults=asyncio.run(manifest.arun_batch(["Where are the cats?","Where are the dogs?"]))If something doesn't go right, you can also ask to get a raw manifest Response.result_object=manifest.run(["Where are the cats?","Where are the dogs?"],return_response=True)print(result_object.get_request_obj())print(result_object.is_cached())print(result_object.get_response_obj())By default, we do not truncate results based on a stop token. You can change this by either passing a new stop token to a Manifest session or to arun.result=manifest.run(prompt,"Laurel",stop_token="and")If you want to change default parameters to a model, we pass those askwargsto the client.result=manifest.run(prompt,"Laurel",max_tokens=50)Streaming QueriesManifest also supports streaming the model response back, assuming it's supported by the underlying client. When callingrun, passstream=Trueto get a streaming iterator in response.result_iterator=manifest.run("Tell me a story. Once upon a time",max_tokens=100,stream=True)forres_textinresult_iterator:print(res_text)Streaming responses are only supported for single string queries (not batch mode) for text completion models.Model PoolsManifest supports querying multiple models with different schedulers. This is very much a work in progress effort, but Manifest will round robin select (or randomly select) the clients you want. You can use the same client multiple times with different connection strings (e.g. different API keys), or you can mix and match. The only requirement is that all clients are the same request type. I.e. you can't have a pool of generation models and embedding models.To query between a local model and OpenAI,frommanifest.connections.client_poolimportClientConnectionfrommanifestimportManifestclient_connection1=ClientConnection(client_name="huggingface",client_connection="http://127.0.0.1:5000",)client_connection2=ClientConnection(client_name="openai",engine="text-ada-001")manifest=Manifest(client_pool=[client_connection1,client_connection2],cache_name="sqlite",client_connection=sqlite_cache,)manifest.run(...)The speed benefit comes in with async batched runs. When callingarun_batchwith a list of prompts, Manifest supports achunk_sizeparam. This will break the prompts intochunk_sizechunks to spread across the client pool. By defaultchunk_sizeis-1which means only one client will get all the prompts to run asynchronously. You must setchunk_size > 1to distribute across the pool. There is a furtherbatch_sizeparam which control the individual clientbatch_sizeto send to the model.responses=asyncio.run(manifest.arun_batch(prompts,max_tokens=30,chunk_size=20))Other ModelsLocal Huggingface ModelsTo use a HuggingFace generative model, inmanifest/apiwe have a Flask application that hosts the models for you.In a separate terminal or Tmux/Screen session, to load 6B parameters models, runpython3-mmanifest.api.app\--model_typehuggingface\--model_name_or_pathEleutherAI/gpt-j-6B\--device0You will see the Flask session start and output a URLhttp://127.0.0.1:5000. Pass this in to Manifest. If you want to use a different port, set theFLASK_PORTenvironment variable.manifest=Manifest(client_name="huggingface",client_connection="http://127.0.0.1:5000",)If you have a custom model you trained, pass the model path to--model_name_or_path.To help load larger models, we also support usingparallelize()from HF,accelerate,bitsandbytes, anddeepspeed. You will need to install these packages first viapip install manifest-ml[api]. We list the commands to load larger models below.T0pppython3-mmanifest.api.app\--model_typehuggingface\--model_name_or_pathbigscience/T0pp\--use_hf_parallelizeNeoX 20B (requires at least 60GB of GPU memory)python3-mmanifest.api.app\--model_typehuggingface\--model_name_or_pathEleutherAI/gpt-neox-20b\--use_accelerate_multigpu\--percent_max_gpu_mem_reduction0.75Bloom 175B (requires at least 240GB of GPU memory)python3-mmanifest.api.app\--model_typehuggingface\--model_name_or_pathbigscience/bloom\--use_bitsandbytes\--percent_max_gpu_mem_reduction0.85Chat ModelsManifest has specific support for executing against chat models in the more standard "system" / "user" dialogue. To pass in a dialogue history to Manifest, use theruncommand with a list of dictionary inputs withroleandcontentkeys using an associated chat model such asopenaichat.manifest=Manifest(client_name="openaichat")dialogue=[{"role":"system","content":"You are a helpful assistant who also responds in rhymes"},{"role":"user","content":"What is the date?"},]res=manifest.run(dialogue,max_tokens=100)Embedding ModelsManifest also supports getting embeddings from models and available APIs. We do this all through changing theclient_nameargument. You still userunandabatch_run.To use OpenAI's embedding models, simply runmanifest=Manifest(client_name="openaiembedding")embedding_as_np=manifest.run("Get me an embedding for a bunny")As explained above, you can load local HuggingFace models that give you embeddings, too. If you want to use a standard generative model, load the model as above use useclient_name="huggingfaceembedding". If you want to use a standard embedding model, like those from SentenceTransformers, load your local model viapython3-mmanifest.api.app\--model_typesentence_transformers\--model_name_or_pathall-mpnet-base-v2\--device0Road MapHere's what's coming up nextClientsHuggingFace HubAzure OpenAIGoogle VertexAnthropicStreaming Support CompletionsStreaming Support Chat ModelsData TypesDiffusion ModelsOrchestrationConnection poolsLocal InferenceFlexGenDevelopmentBefore submitting a PR, runexportREDIS_PORT="6379"# or whatever PORT local redis is running for those testscd<REDIS_PATH>
dockerrun-d-p127.0.0.1:${REDIS_PORT}:6379-v`pwd`:`pwd`-w`pwd`--namemanifest_redis_testredis
maketestCitePlease cite Manifest if you used it for any publications. Thanks!!@misc{orr2022manifest,
  author = {Orr, Laurel},
  title = {Manifest},
  year = {2022},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/HazyResearch/manifest}},
}