Package: puyuan-modelhub
Description📦 ModelhubClient: A Python client for the Modelhub. Support various models including LLMs, embedding models, audio models and multi-modal models. These models are implemented by either 3rdparty APIs or self-host instances.Installationpipinstallpuyuan_modelhub--userQuick StartModelhubClientInitializationfrommodelhubimportModelhubClientclient=ModelhubClient(host="https://modelhub.puyuan.tech/api/",user_name="xxxx",user_password="xxxx",model="xxx",# Optional)Get supported modelsclient.supported_modelsCreate a stateless chatresponse=client.chat(query,model="xxx",# Optional(be None to use the model specific in initialization)history=history,parameters=dict(key1=value1,key2=value2))Get embeddingsclient.get_embeddings(["你好","Hello"],model="m3e")Context Compression/DistillationChat usinglinguawill return a compressed/distillated context. Currently we useLlama-2-7B-Chat-GPTQas LLMlingua backend. Theorically, any local model(Baichuan, ChatGLM, etc.) which can be loaded usingAutoModelForCasualLMcan be used as the backend, thus should provide acompressAPI for every local model, this is a future work sinceLLMlinguadoesn't support it naively.Parameters forlinguamodel:client.chat(prompt:str,model="lingua",history:List[Dict[str,str]],parameters=dict(question:str="",ratio:float=0.5,target_token:float=-1,iterative_size:int=200,force_context_ids:List[int]=None,force_context_number:int=None,use_sentence_level_filter:bool=False,use_context_level_filter:bool=True,use_token_level_filter:bool=True,keep_split:bool=False,keep_first_sentence:int=0,keep_last_sentence:int=0,keep_sentence_number:int=0,high_priority_bonus:int=100,context_budget:str="+100",token_budget_ratio:float=1.4,condition_in_question:str="none",reorder_context:str="original",dynamic_context_compression_ratio:float=0.0,condition_compare:bool=False,add_instruction:bool=False,rank_method:str="llmlingua",concate_question:bool=True,))Async SupportEvery sync method has the corresponding async one starts with "a"(SeeAPI Documentationbelow). For example:Use async mechanism to make concurrent requests.NoteUnlike API models, local models are now single threaded, requested will be queued when using async. In the future, we need to adopt a more flexible inference pipeline.Github Topicimportanyioasyncwithanyio.create_task_group()astg:asyncdefquery(question):print(awaitclient.achat(question,model="gpt-3.5-turbo"))questions=["hello","nihao","test","test1","test2"]forqinquestions:tg.start_soon(query,q)gemini-proembedding need extra parametersUse theembed_contentmethod to generate embeddings. The method handles embedding for the following tasks (task_type):Task TypeDescriptionRETRIEVAL_QUERYSpecifies the given text is a query in a search/retrieval setting.RETRIEVAL_DOCUMENTSpecifies the given text is a document in a search/retrieval setting. Using this task type requires atitle.SEMANTIC_SIMILARITYSpecifies the given text will be used for Semantic Textual Similarity (STS).CLASSIFICATIONSpecifies that the embeddings will be used for classification.CLUSTERINGSpecifies that the embeddings will be used for clustering.Responsegenerated_text:response_text from modelhistory:generated history, **only chatglm3 return this currently.**details:generation details. Include tokens used, request duration, ...History ParameterYou can either use list of pre-defined message types or raw dicts containingroleandcontentKV as history.Note that not every model support role type likesystem# import some pre-defined message typesfrommodelhub.common.typesimportSystemMessage,AIMessage,UserMessage# construct history of your ownhistory=[SystemMessage(content="xxx",other_value="xxxx"),UserMessage(content="xxx",other="xxxx"),]VLMClient (Deprecated)No Visual Language Models are hosted currently, cogvlm specifically. And this Client will be migrated to ModelhubClient in the future.InitailizationfrommodelhubimportVLMClientclient=VLMClient(...)client.chat(prompt=...,image_path=...,parameters=...)Chat with modelVLM Client chat addimage_pathparam to Modelhub Client and other params are same.client.chat("Hello?",image_path="xxx",model="m3e")OpenAI ClientOnly a small subset of models are supported to use in this manner currently. Others will rasie Exception.fromopenaiimportOpenAIclient=OpenAI(api_key=f"{user_name}:{user_password}",base_url="https://modelhub.puyuan.tech/api/v1")client.chat.completions.create(...,model="self-host-models")ExamplesUse ChatCLM3 for tools callingfrommodelhubimportModelhubClient,VLMClientfrommodelhub.common.typesimportSystemMessageclient=ModelhubClient(host="https://xxxxx/api/",user_name="xxxxx",user_password="xxxxx",)tools=[{"name":"track","description":"追踪指定股票的实时价格","parameters":{"type":"object","properties":{"symbol":{"description":"需要追踪的股票代码"}},"required":["symbol"],},},{"name":"text-to-speech","description":"将文本转换为语音","parameters":{"type":"object","properties":{"text":{"description":"需要转换成语音的文本"},"voice":{"description":"要使用的语音类型（男声、女声等）"},"speed":{"description":"语音的速度（快、中等、慢等）"},},"required":["text"],},},]# construct system historyhistory=[SystemMessage(content="Answer the following questions as best as you can. You have access to the following tools:",tools=tools,)]query="帮我查询股票10111的价格"# call ChatGLM3response=client.chat(query,model="ChatGLM3",history=history)history=response.historyprint(response.generated_text)Output:{"name":"track","parameters":{"symbol":"10111"}}# generate a fake result for track function callresult={"price":1232}res=client.chat(json.dumps(result),parameters=dict(role="observation"),# Tell ChatGLM3 this is a function call resultmodel="ChatGLM3",history=history,)print(res.generated_text)Output:
根据API调用结果，我得知当前股票的价格为1232。请问您需要我为您做什么？Contact