Package: llm-client
LLM-Client-SDKLLM-Client-SDK is an SDK for seamless integration with generative AI large language models
(We currently support - OpenAI, Google, AI21, HuggingfaceHub, Aleph Alpha, Anthropic,
Local models with transformers - and many more soon).Our vision is to provide async native and production ready SDK while creating
a powerful and fast integration with different LLM without letting the user lose
any flexibility (API params, endpoints etc.). *We also provide sync version, see
more details below in Usage section.Base InterfaceThe package exposes two simple interfaces for seamless integration with LLMs (In the future, we
will expand the interface to support more tasks like list models, edits, etc.):fromabcimportABC,abstractmethodfromdataclassesimportdataclass,fieldfromtypingimportAny,OptionalfromenumimportEnumfromdataclasses_jsonimportdataclass_json,configfromaiohttpimportClientSessionclassBaseLLMClient(ABC):@abstractmethodasyncdeftext_completion(self,prompt:str,**kwargs)->list[str]:raiseNotImplementedError()asyncdefget_tokens_count(self,text:str,**kwargs)->int:raiseNotImplementedError()classRole(Enum):SYSTEM="system"USER="user"ASSISTANT="assistant"@dataclass_json@dataclassclassChatMessage:role:Role=field(metadata=config(encoder=lambdarole:role.value,decoder=Role))content:strname:Optional[str]=field(default=None,metadata=config(exclude=lambdaname:nameisNone))example:bool=field(default=False,metadata=config(exclude=lambda_:True))@dataclassclassLLMAPIClientConfig:api_key:strsession:ClientSessionbase_url:Optional[str]=Nonedefault_model:Optional[str]=Noneheaders:dict[str,Any]=field(default_factory=dict)classBaseLLMAPIClient(BaseLLMClient,ABC):def__init__(self,config:LLMAPIClientConfig):...@abstractmethodasyncdeftext_completion(self,prompt:str,model:Optional[str]=None,max_tokens:int|None=None,temperature:Optional[float]=None,top_p:Optional[float]=None,**kwargs)->list[str]:raiseNotImplementedError()asyncdefchat_completion(self,messages:list[ChatMessage],temperature:float=0,max_tokens:int=16,model:Optional[str]=None,**kwargs)->list[str]:raiseNotImplementedError()asyncdefembedding(self,text:str,model:Optional[str]=None,**kwargs)->list[float]:raiseNotImplementedError()asyncdefget_chat_tokens_count(self,messages:list[ChatMessage],**kwargs)->int:raiseNotImplementedError()RequirementsPython 3.9+InstallationIf you are worried about the size of the package you can install only the clients you need,
by default we install none of the clients.For all current clients support$pipinstallllm-client[all]For only the base interface and some light LLMs clients (AI21 and Aleph Alpha)$pipinstallllm-clientOptional DependenciesFor all current api clients support$pipinstallllm-client[api]For only local client support$pipinstallllm-client[local]For sync support$pipinstallllm-client[sync]For only OpenAI support$pipinstallllm-client[openai]For only HuggingFace support$pipinstallllm-client[huggingface]UsageUsing OpenAI directly through OpenAIClient - Maximum control and best practice in productionimportosfromaiohttpimportClientSessionfromllm_clientimportChatMessage,Role,OpenAIClient,LLMAPIClientConfigOPENAI_API_KEY=os.environ["API_KEY"]OPENAI_ORG_ID=os.getenv("ORG_ID")asyncdefmain():asyncwithClientSession()assession:llm_client=OpenAIClient(LLMAPIClientConfig(OPENAI_API_KEY,session,default_model="text-davinci-003",headers={"OpenAI-Organization":OPENAI_ORG_ID}))# The headers are optionaltext="This is indeed a test"messages=[ChatMessage(role=Role.USER,content="Hello!"),ChatMessage(role=Role.SYSTEM,content="Hi there! How can I assist you today?")]print("number of tokens:",awaitllm_client.get_tokens_count(text))# 5print("number of tokens for chat completion:",awaitllm_client.get_chat_tokens_count(messages,model="gpt-3.5-turbo"))# 23print("generated chat:",awaitllm_client.chat_completion(messages,model="gpt-3.5-turbo"))# ['Hi there! How can I assist you today?']print("generated text:",awaitllm_client.text_completion(text))# [' string\n\nYes, this is a test string. Test strings are used to']print("generated embedding:",awaitllm_client.embedding(text))# [0.0023064255, -0.009327292, ...]Using LLMAPIClientFactory - Perfect if you want to move fast and to not handle the client session yourselfimportosfromllm_clientimportLLMAPIClientFactory,LLMAPIClientTypeOPENAI_API_KEY=os.environ["API_KEY"]asyncdefmain():asyncwithLLMAPIClientFactory()asllm_api_client_factory:llm_client=llm_api_client_factory.get_llm_api_client(LLMAPIClientType.OPEN_AI,api_key=OPENAI_API_KEY,default_model="text-davinci-003")awaitllm_client.text_completion(prompt="This is indeed a test")awaitllm_client.text_completion(prompt="This is indeed a test",max_tokens=50)# Or if you don't want to use asyncfromllm_clientimportinit_sync_llm_api_clientllm_client=init_sync_llm_api_client(LLMAPIClientType.OPEN_AI,api_key=OPENAI_API_KEY,default_model="text-davinci-003")llm_client.text_completion(prompt="This is indeed a test")llm_client.text_completion(prompt="This is indeed a test",max_tokens=50)Local modelimportosfromtransformersimportAutoModelForCausalLM,AutoModelForSeq2SeqLM,AutoTokenizerfromllm_clientimportLocalClientConfig,LocalClientasyncdefmain():try:model=AutoModelForCausalLM.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])exceptValueError:model=AutoModelForSeq2SeqLM.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])tokenizer=AutoTokenizer.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])llm_client=LocalClient(LocalClientConfig(model,tokenizer,os.environ["TENSORS_TYPE"],os.environ["DEVICE"]))awaitllm_client.text_completion(prompt="This is indeed a test")awaitllm_client.text_completion(prompt="This is indeed a test",max_tokens=50)# Or if you don't want to use asyncimportasync_to_synctry:model=AutoModelForCausalLM.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])exceptValueError:model=AutoModelForSeq2SeqLM.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])tokenizer=AutoTokenizer.from_pretrained(os.environ["MODEL_NAME_OR_PATH"])llm_client=LocalClient(LocalClientConfig(model,tokenizer,os.environ["TENSORS_TYPE"],os.environ["DEVICE"]))llm_client=async_to_sync.methods(llm_client)llm_client.text_completion(prompt="This is indeed a test")llm_client.text_completion(prompt="This is indeed a test",max_tokens=50)ContributingContributions are welcome! Please check out the todos below, and feel free to open issue or a pull request.TodoThe list is unorderedAdd support for more LLMsAnthropicGoogleCohereAdd support for more functions via LLMsembeddingschatlist modelseditsmoreAdd contributing guidelines and linterCreate an easy way to run multiple LLMs in parallel with the same promptsConvert common models parametertemperaturemax_tokenstop_pmoreDevelopmentTo install the package in development mode, run the following command:$pipinstall-e".[all,test]"To run the tests, run the following command:$pytesttestsIf you want to add a new LLMClient you need to implement BaseLLMClient or BaseLLMAPIClient.If you are adding a BaseLLMAPIClient you also need to add him in LLMAPIClientFactory.You can add dependencies to your LLMClient inpyproject.tomlalso make sure you are adding a
matrix.flavor intest.yml.