Package: kscope
Kaleidoscope-SDKA user toolkit for analyzing and interfacing with Large Language Models (LLMs)Overviewkaleidoscope-sdkis a Python module used to interact with large language models
hosted via the Kaleidoscope service available at:https://github.com/VectorInstitute/kaleidoscope.
It provides a simple interface to launch LLMs on an HPC cluster, asking them to
perform basic features like text generation, but also retrieve intermediate
information from inside the model, such as log probabilities and activations.
These features are exposed via a few high-level APIs, namely:model_instances- Shows a list of all active LLMs instantiated by the model serviceload_model- Loads an LLM via the model servicegenerate- Returns an LLM text generation based on prompt inputmodule_names- Returns all modules names in the LLM neural networkget_activations- Retrieves all activations for a set of modulesGetting StartedRequires Python version >= 3.8Installpython3-mpipinstallkscopeor install from source:pipinstallgit+https://github.com/VectorInstitute/kaleidoscope-sdk.gitAuthenticationIn order to submit generation jobs, a designated Vector Institute cluster account is required. Please contact theAI Engineering Teamin charge of Kaleidoscope for more information.Sample WorkflowThe following workflow shows how to load and interact with an OPT-175B model
on the Vector Institute Vaughan cluster.#!/usr/bin/env python3importkscopeimporttime# Establish a client connection to the Kaleidoscope service# If you have not previously authenticated with the service, you will be prompted to nowclient=kscope.Client(gateway_host="llm.cluster.local",gateway_port=3001)# See which models are supportedclient.models# See which models are instantiated and available to useclient.model_instances# Get a handle to a model. If this model is not actively running, it will get launched in the background.# In this example we want to use the LLama2 7b modelllama2_model=client.load_model("llama2-7b")# If the model was not actively running, this it could take several minutes to load. Wait for it come online.whilellama2_model.state!="ACTIVE":time.sleep(1)# Sample text generation w/ input parameterstext_gen=llama2_model.generate("What is Vector Institute?",{'max_tokens':5,'top_k':4,'temperature':0.5})dir(text_gen)# display methods associated with generated text objecttext_gen.generation['sequences']# display only texttext_gen.generation['logprobs']# display logprobstext_gen.generation['tokens']# display tokens# Now let's retrieve some activations from the model# First, show a list of modules in the neural networkprint(llama2_model.module_names)# Setup a request for module acivations for a certain module layerrequested_activations=['layers.0']activations=llama2_model.get_activations("What are activations?",requested_activations)print(activations)# Next, let's manipulate the activations in the model. First, we need to import a few more modules.importcloudpickleimportcodecsimporttorchfromtorchimportTensorfromtypingimportCallable,Dict# Define a function to manipulate the activationsdefreplace_with_ones(act:Tensor)->Tensor:"""Replace an activation with an activation filled with ones."""out=torch.ones_like(act,dtype=act.dtype).cuda()returnout# Now send the edit requestediting_fns:Dict[str,Callable]={}editing_fns['layers.0']=replace_with_onesedited_activations=llama2_model.edit_activations("What is Vector Institute?",editing_fns)print(edited_activations)DocumentationFull documentation and API reference are available at:http://kaleidoscope-sdk.readthedocs.io.ContributingContributing to kaleidoscope is welcomed. SeeContributingfor
guidelines.LicenseMITCitationReference to cite when you use Kaleidoscope in a project or a research paper:Willes, J., Choi, M., Coatsworth, M., Shen, G., & Sivaloganathan, J (2022). Kaleidoscope. http://VectorInstitute.github.io/kaleidoscope. computer software, Vector Institute for Artificial Intelligence. Retrieved from https://github.com/VectorInstitute/kaleidoscope-sdk.git.