Package: byzerllm
Easy, fast, and cheap pretrain,finetune, serving for everyone|English|中文|Latest News🔥[2024/02] Release Byzer-LLM 0.1.40[2024/01] Release Byzer-LLM 0.1.39[2023/12] Release Byzer-LLM 0.1.30Byzer-LLM is Ray based , a full lifecycle solution for LLM that includes pretrain, fintune, deployment and serving.The unique features of Byzer-LLM are:Full lifecyle: pretrain and finetune,deploy and serving supportPython/SQL API supportRay based, easy to scaleVersionsInstallationQuick StartHow to connect Models from outside of Ray ClusterEmbedding/RerankEmbedding ModelEmbedding Rerank ModelQuatizationSupported ModelsServingBackendvLLM SupportvLLM troubleshootingDeepSpeed SupportByzer-LLM OpenAI-Compatible RESTful API serverLLM && PythonFunction CallingRespond with pydantic classFunction ImplementationLLM-Friendly Function/DataClassModel ConfigurationsModel MetaChat TemplateLLM Default Generation ParametersSaaS Modelsqianwen/通义千问baichuan/百川azure openaiopenaizhipu/智谱sparkdesk/星火Multi ModalStableDiffusionSQL SupportPretrainFinetuneStream ChatArticlesThird-party LibrariesLlama_indexContributingVersions0.1.40： LlamaIndex support / vLLM 0.3.2 Support / Byzer-SQL new features / Qwen 1.5 support0.1.39： Enhance Function Impl / Upgrade SaaS SDK / Add OpenAI-Compatible API Server0.1.38： Upgrade saas/sparkdask model / add embedding rerank model / agent message store support0.1.37： Upgrade saas/zhipu model, you can choose glm-4 / embedding-2 for LLM or embedding purpose0.1.36： Fix data analysis agent bugs which caused by the upgrade of Byzer-Agent0.1.35： Add Baichuan Saas embedding model0.1.34： Enhance the Byzer-Agent API and fix some bugs in Byzer-LLM0.1.33： Fix Response Class bugs/ Add function implementation0.1.32： StableDiffusion optimization0.1.31： Stream Chat with token count information / Optimize multi modal model chat0.1.30： Apply chat template for vLLM backend0.1.29： Enhance DataAnalysis Agent0.1.28： Bug fix0.1.27： Bug fix0.1.26： Support QianWen Saas/ Support stream chat in QianWenSaas/ Fix some Saas model bugs0.1.24： Support get meta from model instance and auto setup template0.1.23： Fintune with python API/ Fix some bugs0.1.22： Function Calling support/ Response with pydantic class0.1.19： Fix embedding bugs0.1.18： Support stream chat/ Support Model Template0.1.17： None0.1.16： Enhance the API for byzer-retrieval0.1.14： add get_tables/get_databases API for byzer-retrieval0.1.13: support shutdown cluster for byzer-retrieval0.1.12: Support Python API (alpha)0.1.5:  Support python wrapper forbyzer-retrievalInstallationRecommend Env:Conda:  python==3.10.11OS:     ubuntu 22.04Cuda:   12.1.0 (Optional if you only use SaaS model)## Make sure you python version is 3.10.11pipinstall-rrequirements.txt## Skip this step if you have no Nvidia GPUpipinstallvllm==0.2.6
pipinstall-Ubyzerllm
raystart--headIf your cuda version is 11.8, please check the following link to install vLLM:https://docs.vllm.ai/en/latest/getting_started/installation.htmlThe key steps are:Asofnow,vLLM’sbinariesarecompiledonCUDA12.1bydefault.However,youcaninstallvLLMwithCUDA11.8byrunning:# Install vLLM with CUDA 11.8.exportVLLM_VERSION=0.2.6exportPYTHON_VERSION=310pipinstallhttps://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl# Re-install PyTorch with CUDA 11.8.pipuninstalltorch-y
pipinstalltorch--upgrade--index-urlhttps://download.pytorch.org/whl/cu118# Re-install xFormers with CUDA 11.8.pipuninstallxformers-y
pipinstall--upgradexformers--index-urlhttps://download.pytorch.org/whl/cu118Raw MachineOnly tested on Ubuntu 20.04/22.04 CentOS 8.0If your machine is a raw machine whithout GPU Driver and Cuda installed, you can use the following script to setup the machine:gitclonehttps://gitee.com/allwefantasy/byzer-llmcdbyzer-llm/setup-machineThen switch to the ROOT user, and run the following script:ROLE=master./setup-machine.shThis step will create a user calledbyzerllm.Then switch to thebyzerllmuser, and run the following script:ROLE=master./setup-machine.shThis time the script will automatically install the following software:CondaNvidia Driver 535Cuda 12.1.0Rayall the python packages in requirements.txtByzer-SQL/Byzer-NotebookIf you need to add more worker nodes to the Ray cluster, repeat the above steps on the worker nodes.
Notice that theROLEshould beworkeron the worker nodes.ROLE=worker./setup-machine.shQuick Startimportrayfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(4).setup_num_workers(1)llm.setup_infer_backend(InferBackend.transformers)llm.deploy(model_path="/home/byzerllm/models/openbuddy-llama2-13b64k-v15",pretrained_model_type="custom/llama2",udf_name="llama2_chat",infer_params={})llm_client=ByzerLLM()llm_client.setup_template("llama2_chat","auto")v=llm.chat_oai(model="llama2_chat",conversations=[{"role":"user","content":"hello",}])print(v[0].output)The above code will deploy a llama2 model and then use the model to infer the input text. If you use transformers as the inference backend, you should specify thepretrained_model_typemanually since the transformers backend can not auto detect the model type.Byzer-LLM also supportdeploySaaS model with the same way. This feature provide a unified interface for both open-source model and SaaS model. The following code will deploy a Azure OpenAI model and then use the model to infer the input text.importrayfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(0).setup_num_workers(10)llm.setup_infer_backend(InferBackend.transformers)llm.deploy(pretrained_model_type="saas/azure_openai",udf_name="azure_openai",infer_params={"saas.api_type":"azure","saas.api_key"="xxx""saas.api_base"="xxx""saas.api_version"="2023-07-01-preview""saas.deployment_id"="xxxxxx"})llm_client=ByzerLLM()llm_client.setup_template("azure_openai","auto")v=llm.chat_oai(model="azure_openai",conversations=[{"role":"user","content":"hello",}])print(v[0].output)Notice that the SaaS model does not need GPU, so we set thesetup_gpus_per_workerto 0, and you can usesetup_num_workersto control max concurrency,how ever, the SaaS model has its own max concurrency limit, thesetup_num_workersonly control the max
concurrency accepted by the Byzer-LLM.How to connect Models from outside of Ray ClusterThe recommended way is to start a empty Ray worker in your target machine(e.g. Your web server machine):raystart--address="xxxxx:6379"--num-gpus=0--num-cpus=0Then you can connect the Models from outside of the Ray cluster:importrayfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackend## connect the ray cluster by the empty worker we started before## this code should be run once in your prorgramray.init(address="auto",namespace="default",ignore_reinit_error=True)## new a ByzerLLM instancellm_client=ByzerLLM()llm_client.setup_template("llama2_chat","auto")v=llm.chat_oai(model="llama2_chat",conversations=[{"role":"user","content":"hello",}])print(v[0].output)Embedding ModelThe following code is a example of deploying BGE embedding modelimportrayfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers)llm.deploy(model_path="/home/byzerllm/models/bge-large-zh",pretrained_model_type="custom/bge",udf_name="emb",infer_params={})Then you can convert any text to vector :t=llm.emb("emb",LLMRequest(instruction="wow"))t[0].output#output: [-0.005588463973253965,-0.01747054047882557,-0.040633779019117355,...-0.010880181565880775,-0.01713103987276554,0.017675869166851044,-0.010260719805955887,...]The Byzer-LLM also support SaaS embedding model. The following code will deploy a Baichuan embedding model and then use the model to infer the input text.importosos.environ["RAY_DEDUP_LOGS"]="0"importrayfrombyzerllm.utils.retrievalimportByzerRetrievalfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,LLMResponse,LLMHistoryItem,InferBackendfrombyzerllm.utils.clientimportTemplatesray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM(verbose=True)llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="baichuan_emb"ifllm.is_model_exist(chat_name):llm.undeploy(udf_name=chat_name)llm.deploy(model_path="",pretrained_model_type="saas/baichuan",udf_name=chat_name,infer_params={"saas.api_key":"","saas.model":"Baichuan-Text-Embedding"})llm.setup_default_emb_model_name(chat_name)v=llm.emb(None,LLMRequest(instruction="你好"))print(v.output)Embedding Rerank ModelIf you need to use embedding rerank model, you can refer to the following usage.importrayfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(0.4).setup_num_workers(2).setup_infer_backend(InferBackend.Transformers)llm.deploy(model_path="/Users/wanghan/data/bge-reranker-base",pretrained_model_type="custom/bge_rerank",udf_name="emb_rerank",infer_params={})llm.setup_default_emb_model_name("emb_rerank")Then you can get a relevance score by inputting query and passage to the rerankersentence_pairs_01=['query','passage']t1=llm.emb_rerank(sentence_pairs=sentence_pairs_01)print(t1[0].output)#output [['query', 'passage'], 0.4474925994873047]sentence_pairs_02=[['what is panda?','hi'],['what is panda?','The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]t2=llm.emb_rerank(sentence_pairs=sentence_pairs_02)print(t2[0].output)#output [[['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.'], 6.1821160316467285], [['what is panda?', 'hi'], -8.154398918151855]]QuatizationIf the backend isInferBackend.transformers, here is the baichuan2 example:llm.setup_gpus_per_worker(2).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)llm.deploy(model_path=model_location,pretrained_model_type="custom/baichuan2",udf_name="baichuan2_13_chat",infer_params={"quatization":"4"})The availablequatizationvalues:48true/falseWhen it's set true, the int4 will be choosed.If the bakcend isInferBackend.VLLM, here is the Yi example:If you need to deploy model with Quantization, you can set theinfer_paramsas the following code:llm.setup_gpus_per_worker(1).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)llm.deploy(model_path="/home/winubuntu/models/Yi-6B-Chat-4bits",pretrained_model_type="custom/auto",udf_name="chat",infer_params={"backend.quantization":"AWQ"})The parameterbackend.quantizationcan be GPTQ/AWQ.Supported ModelsThe supported open-sourcepretrained_model_typeare:custom/llama2barkwhisperchatglm6bcustom/chatglm2mosscustom/alpha_mossdollyfalconllamacustom/starcodecustom/visualglmcustom/m3ecustom/baichuancustom/bgecustom/qwen_vl_chatcustom/stable_diffusioncustom/zephyrThe supported SaaSpretrained_model_typeare:saas/chatglm	Chatglm130Bsaas/sparkdesk	星火大模型saas/baichuan	百川大模型saas/zhipu	智谱大模型saas/minimax	MiniMax 大模型saas/qianfan	文心一言saas/azure_openaisaas/openaiNotice that the derived models from llama/llama2/startcode are also supported. For example, you can usellamato load vicuna model.vLLM SupportThe Byzer-llm also support vLLM as the inference backend. The following code will deploy a vLLM model and then use the model to infer the input text.importrayfrombyzerllm.utils.retrievalimportByzerRetrievalfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(2)llm.setup_num_workers(1)llm.setup_infer_backend(InferBackend.VLLM)llm.deploy(model_path="/home/byzerllm/models/openbuddy-zephyr-7b-v14.1",pretrained_model_type="custom/auto",udf_name="zephyr_chat"",infer_params={})v=llm.chat_oai(model="zephyr_chat",conversations=[{"role":"user","content":"hello",}])print(v[0].output)There are some tiny differences between the vLLM and the transformers backend.Thepretrained_model_typeis fixed tocustom/autofor vLLM, since the vLLM will auto detect the model type.Usesetup_infer_backendto specifyInferBackend.VLLMas the inference backend.vLLM troubleshootingIf you use the version of vLLM > 0.2.7 and meets the following error:ErrorType:TASK_EXECUTION_EXCEPTIONself._call_impl(*args,**kwargs)File"/home/byzerllm/miniconda3/envs/byzerllm-dev2/lib/python3.10/site-packages/torch/nn/modules/module.py",line1527,in_call_implreturnforward_call(*args,**kwargs)File"/home/byzerllm/miniconda3/envs/byzerllm-dev2/lib/python3.10/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py",line96,inforwardinput_mask=((input_<self.vocab_start_index)|TypeError:'<'notsupportedbetweeninstancesof'TensorMetadata'and'int'You can open the Ray dashboard and find the RayVLLMWorker Actor, go into and check the tasks tab and click the error task to get the error message above.The solution is to modify the$CONDA_ENV/site-packages/vllm/model_executor/parallel_utils/communication_op.pyin the vLLM pip package:Replace the following code:TensorMetadata=namedtuple("TensorMetadata",["dtype","size"])toclassTensorMetadata:def__init__(self,dtype,size):self.dtype=dtypeself.size=sizeStream ChatIf the model deployed with the backend vLLM, then it also supportstream chat：
thestream_chat_oaiwill return a generator, you can use the generator to get the output text.llm.setup_default_model_name(chat_model_name)t=llm.stream_chat_oai(conversations=[{"role":"user","content":"Hello, how are you?"}])forlineint:print(line+"\n")DeepSpeed SupportThe Byzer-llm also support DeepSpeed as the inference backend. The following code will deploy a DeepSpeed model and then use the model to infer the input text.importrayfrombyzerllm.utils.retrievalimportByzerRetrievalfrombyzerllm.utils.clientimportByzerLLM,LLMRequest,InferBackendray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_gpus_per_worker(4)llm.setup_num_workers(1)llm.setup_infer_backend(InferBackend.DeepSpeed)llm.deploy(model_path="/home/byzerllm/models/openbuddy-llama-13b-v5-fp16",pretrained_model_type="custom/auto",udf_name="llama_chat"",infer_params={})llm.chat("llama_chat",LLMRequest(instruction="hello world"))[0].outputThe code above is totally the same as the code for vLLM, except that theInferBackendisInferBackend.DeepSpeed.Byzer-LLM OpenAI-Compatible RESTful API serverYou can use the following code to start a ByzerLLm OpenAI-Compatible RESTful API server:raystart--address="xxxxx:6379"--num-gpus=0--num-cpus=0python-mbyzerllm.utils.client.entrypoints.openai.api_serverBy default, the server will listen on port 8000, you can use the following code to test the API:fromopenaiimportOpenAIclient=OpenAI(base_url="http://127.0.0.1:8000/v1",api_key="xxxx")chat_completion=client.chat.completions.create(model="wenxin_chat",messages=[{"role":"user","content":"写一个排序算法"}],stream=False)print(chat_completion.choices[0].message.content)Stream chat:fromopenaiimportOpenAIclient=OpenAI(base_url="http://127.0.0.1:8000/v1",api_key="simple")chat_completion=client.chat.completions.create(model="wenxin_chat",messages=[{"role":"user","content":"写一个排序算法"}],stream=True)forchunkinchat_completion:print(chunk.choices[0].delta.contentor"",end="")Function CallingHere is a simple example for function calling based on QWen 72BDeploy Model:importrayray.init(address="auto",namespace="default")llm=ByzerLLM()model_location="/home/byzerllm/models/Qwen-72B-Chat"llm.setup_gpus_per_worker(8).setup_num_workers(1).setup_infer_backend(InferBackend.VLLM)llm.deploy(model_path=model_location,pretrained_model_type="custom/auto",udf_name=chat_model_name,infer_params={})llm.setup_default_model_name("chat")# from 0.1.24# llm.setup_auto("chat")meta=llm.get_meta()llm.setup_max_model_length("chat",meta.get("max_model_len",32000))lm.setup_template("chat",Templates.qwen())Try to create some Python functions:fromtypingimportList,Dict,Any,Annotatedimportpydanticimportdatetimefromdateutil.relativedeltaimportrelativedeltadefcompute_date_range(count:Annotated[int,"时间跨度，数值类型"],unit:Annotated[str,"时间单位，字符串类型",{"enum":["day","week","month","year"]}])->List[str]:'''计算日期范围Args:count: 时间跨度，数值类型unit: 时间单位，字符串类型，可选值为 day,week,month,year'''now=datetime.datetime.now()now_str=now.strftime("%Y-%m-%d%H:%M:%S")ifunit=="day":return[(now-relativedelta(days=count)).strftime("%Y-%m-%d%H:%M:%S"),now_str]elifunit=="week":return[(now-relativedelta(weeks=count)).strftime("%Y-%m-%d%H:%M:%S"),now_str]elifunit=="month":return[(now-relativedelta(months=count)).strftime("%Y-%m-%d%H:%M:%S"),now_str]elifunit=="year":return[(now-relativedelta(years=count)).strftime("%Y-%m-%d%H:%M:%S"),now_str]return["",""]defcompute_now()->str:'''计算当前时间'''returndatetime.datetime.now().strftime("%Y-%m-%d%H:%M:%S")Here we provide two functions:compute_date_range: compute the date range based on the count and unitcompute_now: get the current dateWe will use the model to call these tools according to the user's question.t=llm.chat_oai([{"content":'''计算当前时间''',"role":"user"}],tools=[compute_date_range,compute_now],execute_tool=True)t[0].values## output: ['2023-12-18 17:30:49']t=llm.chat_oai([{"content":'''最近三个月趋势''',"role":"user"}],tools=[compute_date_range,compute_now],execute_tool=True)t[0].values## output: [['2023-09-18 17:31:21', '2023-12-18 17:31:21']]t=llm.chat_oai([{"content":'''最近三天''',"role":"user"}],tools=[compute_date_range,compute_now],execute_tool=True)t[0].values## output: [['2023-12-15 17:23:38', '2023-12-18 17:23:38']]t=llm.chat_oai([{"content":'''你吃饭了么？''',"role":"user"}],tools=[compute_date_range,compute_now],execute_tool=True)ift[0].values:print(t[0].values[0])else:print(t[0].response.output)## output: '您好，我是一个人工智能语言模型，暂时无法吃饭。'You can check the default prompt template function infrom byzerllm.utils import function_calling_format.
If the model is not work well with the default function, you can setup your custom function:defcustom_function_calling_format(prompt:str,tools:List[Callable],tool_choice:Callable)->str:.....llm.setup_function_calling_format_func("chat",custom_function_calling_format)Respond with pydantic classWhen you chat with LLM, you can specify the reponse class,importpydanticclassStory(pydantic.BaseModel):'''故事'''title:str=pydantic.Field(description="故事的标题")body:str=pydantic.Field(description="故事主体")t=llm.chat_oai([{"content":f'''请给我讲个故事，分成两个部分，一个标题，一个故事主体''',"role":"user"},],response_class=Story)t[0].value## output: Story(title='勇敢的小兔子', body='在一个美丽的森林里，住着一只可爱的小兔子。小兔子非常勇敢，有一天，森林里的动物们都被大灰狼吓坏了。只有小兔子站出来，用智慧和勇气打败了大灰狼，保护了所有的动物。从此，小兔子成为了森林里的英雄。')The above code will ask the LLM to generate the Story class directly. However, sometimes we hope the LLM
generate text first, then extract the structure from the text, you can setresponse_after_chat=Trueto
enable this behavior. However, this will bring some performance penalty(additional inference).t=llm.chat_oai([{"content":f'''请给我讲个故事，分成两个部分，一个标题，一个故事主体''',"role":"user"},],response_class=Story,response_after_chat=True)t[0].value## output: Story(title='月光下的守护者', body='在一个遥远的古老村庄里，住着一位名叫阿明的年轻人。阿明是个孤儿，从小在村里长大，以种田为生。他善良、勤劳，深受村民们喜爱。\n\n村子里有个传说，每当满月时分，月亮女神会在村子后山的古树下出现，赐福给那些善良的人们。然而，只有最纯洁的心才能看到她。因此，每年的这个时候，阿明都会独自一人前往后山，希望能得到女神的祝福。\n\n这一年，村子遭受了严重的旱灾，庄稼枯黄，人们生活困苦。阿明决定向月亮女神祈求降雨，拯救村子。他在月光下虔诚地祈祷，希望女神能听到他的呼唤。\n\n就在这个时刻，月亮女神出现了。她被阿明的善良和执着所感动，答应了他的请求。第二天早晨，天空乌云密布，大雨倾盆而下，久旱的土地得到了滋润，庄稼重新焕发生机。\n\n从此以后，每年的满月之夜，阿明都会去后山等待月亮女神的出现，他成为了村民心中的守护者，用他的善良和执着，守护着整个村庄。而他也终于明白，真正的守护者，并非需要超凡的力量，只需要一颗充满爱与善良的心。')You can check the default prompt template function infrom byzerllm.utils import response_class_format,response_class_format_after_chat.
If the model is not work well with the default function, you can setup your custom function:defcustom_response_class_format(prompt:str,cls:pydantic.BaseModel)->str:.....llm.setup_response_class_format_func("chat",custom_response_class_format)Function ImplementationThe Byzer-llm also support function implementation. You can define a empty function, and combine the doc in the function/the user's quesion to guide the LLM to implement the function.Here is a simple example:frombyzerllm.utils.clientimportcode_utils,message_utilsfromtypingimportList,Union,OptionalimportpydanticclassTime(pydantic.BaseModel):time:str=pydantic.Field(...,description="时间，时间格式为 yyyy-MM-dd")@llm.impl()defcalculate_current_time()->Time:'''计算当前时间'''passcalculate_current_time()#output: Time(time='2024-01-28')By default, the function implementation will be cached,start=time.monotonic()calculate_current_time()print(f"first time cost:{time.monotonic()-start}")start=time.monotonic()calculate_current_time()print(f"second time cost:{time.monotonic()-start}")# output:# first time cost: 6.067266260739416# second time cost: 4.347506910562515e-05you can usellm.clear_impl_cache()to clear the cache.Here is the example for function implementation with parameters:frombyzerllm.utils.clientimportcode_utils,message_utilsfromtypingimportList,Union,Optional,AnnotatedimportpydanticfromdatetimeimportdatetimeclassTime(pydantic.BaseModel):time:str=pydantic.Field(...,description="时间，时间格式为 yyyy-MM-dd")@llm.impl()defadd_one_day(current_day:Annotated[datetime,"当前日期，类型是datatime.datetime"])->Time:'''给传入的日期加一天，得到明天的时间'''passadd_one_day(datetime.now())# output:Time(time='2024-01-29')With instruction:frombyzerllm.utils.clientimportcode_utils,message_utilsfromtypingimportList,Union,OptionalimportpydanticclassTimeRange(pydantic.BaseModel):'''时间区间格式需要如下： yyyy-MM-dd'''start:str=pydantic.Field(...,description="开始时间.时间格式为 yyyy-MM-dd")end:str=pydantic.Field(...,description="截止时间.时间格式为 yyyy-MM-dd")@llm.impl(instruction="去年三月到七月")defcalculate_time_range()->TimeRange:'''计算时间区间，时间格式为 yyyy-MM-dd.'''passcalculate_time_range()# output: TimeRange(start='2023-03-01', end='2023-07-31')If you want to replace instruction with the user's question, you can use the following code:frombyzerllm.utils.clientimportcode_utils,message_utilsfromtypingimportList,Union,OptionalimportpydanticclassTimeRange(pydantic.BaseModel):'''时间区间格式需要如下： yyyy-MM-dd'''start:str=pydantic.Field(...,description="开始时间.时间格式为 yyyy-MM-dd")end:str=pydantic.Field(...,description="截止时间.时间格式为 yyyy-MM-dd")defcalculate_time_range()->TimeRange:'''计算时间区间，时间格式为 yyyy-MM-dd.'''passllm.impl(instruction="去年三月到七月")(calculate_time_range)()You can useverbose=Trueto get more information about the function implementation:@llm.impl()defadd_one_day(current_day:Annotated[datetime,"当前日期，类型是datatime.datetime"])->Time:'''给传入的日期加一天，得到明天的时间'''passYou can also use the basic chat_oai function to implement the function:classTimeRange(pydantic.BaseModel):'''时间区间格式需要如下： yyyy-MM-dd'''start:str=pydantic.Field(...,description="开始时间.时间格式为 yyyy-MM-dd")end:str=pydantic.Field(...,description="截止时间.时间格式为 yyyy-MM-dd")defcalculate_time_range():'''计算时间区间，时间格式为 yyyy-MM-dd.'''passt=llm.chat_oai([{"content":"去年三月到七月","role":"user"}],impl_func=calculate_time_range,response_class=TimeRange,execute_impl_func=True)The above code , we define a function calledcalculate_time_range, and the function is empty, then we discribe the function in the doc string, and define the response classTimeRange, to make sure the return value is aTimeRangeinstance. Since the function should be used to resolve the user's question, so the implementation of the function should be related to the user's question. Instead try to implement a common use function, we can just implement a function which can only resolve the user's current question.After the execution, you can get the output like this:t[0].value# start='2023-03-01' end='2023-07-31'If the value is None or not correct, you can get the error message:t[0].metadata.get("resason","")If your function has parameters, you can pass the parameters to the function byimpl_func_params:t=llm.chat_oai([{"content":"xxxxx","role":"user"}],impl_func=calculate_time_range,impl_func_params={},response_class=TimeRange,execute_impl_func=True)If you want to replace the default prompt template function, here is a example:importpydanticfromtypingimportList,Optional,Union,Callablefrombyzerllm.utilsimportserialize_function_to_jsondeffunction_impl_format2(prompt:str,func:Optional[Union[Callable,str]],cls:Union[pydantic.BaseModel,str])->str:tool_choice_ser=serialize_function_to_json(func)_cls=""ifisinstance(cls,str):_cls=clselse:_cls=cls.schema_json(ensure_ascii=False)msg=f''''生成一个python函数，给出详细的思考逻辑，对最后生成的函数不要进行示例说明。生成的函数的名字以及参数需要满足如下约束：\```json{tool_choice_ser}\```生成的函数的返回值必须是 Json 格式。下面是使用 OpenAPI 3.1. 规范描述了你需如何进行json格式的生成。\```json{_cls}\```根据用的户问题,{func.__doc__}。用户的问题是：{prompt}请你实现这个函数。'''returnmsgllm.setup_impl_func_format_func(chat_model_name,function_impl_format2)The default prompt template function isfunction_impl_format, you can check the source code infrom byzerllm.utils import function_impl_format.LLM-Friendly Function/DataClassIf you want to improve the performance of Function Calling or Response Class, you should make your Function(Tool) and Data Class is LLM-Friendly.Let's take a look at the following python code:defcompute_date_range(count:int,unit:str)->List[str]:now=datetime.datetime.now()....This code is not LLM-Friendly Function since it's difficult to know the usage of this funciton and
what's the meaning of the input parameters.The LLM just like human, it's hard to let the LLM know when or how to invoke this function. Especially the parameterunitactually is enum value but the LLM no way to get this message.So, in order to make the LLM knows more about this function in Byzer-LLM, you should
follow some requirments:Adding pythonic function commentUse annotated to provide type and comment for every parameter, if the parameter is a enum, then provide enum values.Here is the LLM-Friendly fuction definision.defcompute_date_range(count:Annotated[int,"时间跨度，数值类型"],unit:Annotated[str,"时间单位，字符串类型",{"enum":["day","week","month","year"]}])->List[str]:'''计算日期范围Args:count: 时间跨度，数值类型unit: 时间单位，字符串类型，可选值为 day,week,month,year'''now=datetime.datetime.now()....If the LLM make something wrong to your function (e.g. provide the bad parameters), try to optimize the function comment
and the parameter Annotated comment.Model MetaThe Byzer-llm also support get the model meta information. The following code will get the meta information of model instance calledchat:```pythonllm.get_meta(model="chat")#output:# {'model_deploy_type': 'proprietary',#  'backend': 'ray/vllm',#  'max_model_len': 32768,#  'architectures': ['QWenLMHeadModel']}Chat TemplateThe different models have different chat templates, the Byzer-LLM have provide some chat templates for the models. You can use the following code to setup the chat template:frombyzerllm.utils.clientimportTemplatesllm.setup_template("chat",Templates.qwen())However, we also support thetokeninzer.apply_chat_template, you can use the following code to apply the chat template:llm.setup_template("chat","auto")If the model is not work well with thetokeninzer.apply_chat_template, this function will raise an exception. In this case, you can use thellm.setup_templateto setup the chat template manually.You can also use thellm.get_metato check if the model support theapply_chat_template:llm.get_meta(model="chat")The output:{'model_deploy_type':'proprietary','backend':'ray/vllm','support_stream':True,'support_chat_template':True,'max_model_len':4096,'architectures':['LlamaForCausalLM']}Notice that this feature will cause additional RPC call, so it will bring some performance penalty.LLM Default Generation ParametersThe Byzer-llm also support setup the default generation parameters for the model. The following code will setup the default generation parameters for the model instance calledchat:llm.setup_extra_generation_params("chat",{"generation.stop_token_ids":[7]})In this case, thegeneration.stop_token_idswill be set to[7]for the model instancechat. Every time you call thechatmodel, thegeneration.stop_token_idswill be set to[7]automatically.Multi ModalThe Byzer-llm also support multi modal. The following code will deploy a multi modal model and then use the model to infer the input text.importrayfrombyzerllm.utils.clientimportByzerLLM,InferBackendray.init(address="auto",namespace="default")llm=ByzerLLM()chat_model_name="qwen_vl_chat"model_location="/home/byzerllm/models/Qwen-VL-Chat"llm.setup_gpus_per_worker(1).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)llm.deploy(model_path=model_location,pretrained_model_type="custom/qwen_vl_chat",udf_name=chat_model_name,infer_params={})Then you can use the model to chat:importbase64image_path="/home/byzerllm/projects/jupyter-workspace/1.jpg"withopen(image_path,"rb")asf:image_content=base64.b64encode(f.read()).decode("utf-8")t=llm.chat_oai(conversations=[{"role":"user","content":"这是什么"}],model=chat_model_name,llm_config={"image":image_content})t[0].output# '{"response": "图中是一名女子在沙滩上和狗玩耍，旁边的狗是一只拉布拉多犬，它坐在沙滩上，面对着一名身穿格子衬衫的女子。女子的腿有些残疾，但是她依然坚持坐在沙滩上和狗玩耍。她的右手拿着一个小玩具，这个玩具上面有两行黑色字母，具体是什么内容看不清楚。她打算把玩具扔给拉布拉多犬。", "history": [{"role": "user", "content": "Picture 1: <img>/tmp/byzerllm/visualglm/images/23eb4cea-cb6e-4f55-8adf-3179ca92ab42.jpg</img>\\n这是什么"}, {"role": "assistant", "content": "图中是一名女子在沙滩上和狗玩耍，旁边的狗是一只拉布拉多犬，它坐在沙滩上，面对着一名身穿格子衬衫的女子。女子的腿有些残疾，但是她依然坚持坐在沙滩上和狗玩耍。她的右手拿着一个小玩具，这个玩具上面有两行黑色字母，具体是什么内容看不清楚。她打算把玩具扔给拉布拉多犬。"}]}'You can chat multi rounds with the following code:importjsonhistory=json.loads(t[0].output)["history"]llm.chat_oai(conversations=history+[{"role":"user","content":"能圈出狗么？"}],model=chat_model_name,llm_config={"image":image_content})# [LLMResponse(output='{"response": "<ref>狗</ref><box>(221,425),(511,889)</box>", "history": [{"role"Get the history from the previous chat, then add the hisotry to new conversation, then chat again.StableDiffusionThe Byzer-llm also support StableDiffusion as the inference backend. The following code will deploy a StableDiffusion model and then use the model to infer the input text.importrayfrombyzerllm.utils.clientimportByzerLLM,InferBackendray.init(address="auto",namespace="default")llm=ByzerLLM()chat_model_name="sd_chat"model_location="/home/byzerllm/models/stable-diffusion-v1-5"llm.setup_gpus_per_worker(2).setup_num_workers(1).setup_infer_backend(InferBackend.Transformers)llm.deploy(model_path=model_location,pretrained_model_type="custom/stable_diffusion",udf_name=chat_model_name,infer_params={})defshow_image(content):fromIPython.displayimportdisplay,Imageimportbase64img=Image(base64.b64decode(content))display(img)Then you can chat with the model:importjsont=llm.chat_oai(conversations=[{"role":"user","content":"画一只猫"}],model=chat_model_name,llm_config={"gen.batch_size":3})cats=json.loads(t[0].output)forresincats:show_image(res["img64"])The output:The parameters:参数含义默认值Instructionprompt非空generation.negative_prompt反向的prompt""generation.sampler_name调度名(unpic, euler_a,euler,ddim,ddpm,deis,dpm2,dpm2-a,dpm++_2m,dpm++_2m_karras,heun,heun_karras,lms,pndm:w)euler_ageneration.sampling_steps生成的步骤数25generation.batch_size一次生成几张1generation.batch_count生成几次1generation.cfg_scale随机或贴合程度值,值越小生成的图片离你的Tags描述的内容差距越大7.5generation.seed随机种子-1generation.width图片宽度768generation.height图片高度768generation.enable_hires开启高分辨率修复功能(和下面两个一组)falsegeneration.upscaler_mode放大算法(bilinear, bilinear-antialiased,bicubic,bicubic-antialiased,nearest,nearest-exact)bilineargeneration.scale_slider放大比例1.5generation.enable_multidiff图片分割处理(减少显存销耗)(和下面3个一组)falsegeneration.views_batch_size分批处理规模4generation.window_size切割大小，宽，高64generation.stride步长16generation.init_image初始化图片，基于这个图片处理(必须传输base64加密的图片) (和下面的一组)Nonegeneration.strength重绘幅度: 图像模仿自由度，越高越自由发挥，越低和参考图像越接近，通常小于0.3基本就是加滤镜0.5SQL SupportIn addition to the Python API, Byzer-llm also support SQL API. In order to use the SQL API, you should install Byzer-SQL language first.Try to install the Byzer-SQL language with the following command:gitclonehttps://gitee.com/allwefantasy/byzer-llmcdbyzer-llm/setup-machine
sudo-iROLE=master./setup-machine.shAfter the installation, you can visit the Byzer Console athttp://localhost:9002.In the Byzer Console, you can run the following SQL to deploy a llama2 model which have the same effect as the Python code above.!byzerllmsetupsingle;!byzerllmsetup"num_gpus=4";!byzerllmsetup"maxConcurrency=1";!byzerllmsetup"infer_backend=transformers";runcommandasLLM.``whereaction="infer"andpretrainedModelType="custom/llama2"andlocalModelDir="/home/byzerllm/models/openbuddy-llama-13b-v5-fp16"andreconnect="false"andudfName="llama2_chat"andmodelTable="command";Then you can invoke the model with UDFllama2_chat:selectllama2_chat(llm_param(map("user_role","User","assistant_role","Assistant","system_msg",'You are a helpful assistant. Think it over and answer the user question correctly.',"instruction",llm_prompt('Please remenber my name: {0}',array("Zhu William")))))asqasq1;Once you deploy the model withrun command as LLM, then you can ues the model as a SQL function. This feature is very useful for data scientists who want to use LLM in their data analysis or data engineers who want to use LLM in their data pipeline.QWenIf you use QWen in ByzerLLM, you should sepcify the following parameters mannualy:the role mappingthe stop_token_idstrim the stop tokens from the outputHowever, we provide a template for this, try to the following code:frombyzerllm.utils.clientimportTemplates### Here,we setup the template for qwenllm.setup_template("chat",Templates.qwen())t=llm.chat_oai(conversations=[{"role":"user","content":"你好,给我讲个100字的笑话吧?"}])print(t)SaaS ModelsSince the different SaaS models have different parameters, here we provide some templates for the SaaS models to help you deploy the SaaS models.baichuan/百川importrayfrombyzerllm.utils.clientimportByzerLLMray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM(verbose=True)llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="baichuan_chat2"ifllm.is_model_exist(chat_name):llm.undeploy(udf_name=chat_name)llm.deploy(model_path="",pretrained_model_type="saas/baichuan",udf_name=chat_name,infer_params={"saas.api_key":"xxxxxxx","saas.model":"Baichuan2-Turbo"})llm.chat_oai(model=chat_name,conversations=[{"role":"user","content":"你好",}])There are some enum values for thesaas.model:Baichuan2-TurboBaichuan-Text-Embeddingqianwen/通义千问frombyzerllm.utils.clientimportByzerLLMllm=ByzerLLM()llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="qianwen_chat"llm.deploy(model_path="",pretrained_model_type="saas/qianwen",udf_name=chat_name,infer_params={"saas.api_key":"xxxxxxx","saas.model":"qwen-turbo"})## here you can use `stream_chat_oai`v=llm.stream_chat_oai(model=chat_name,conversations=[{"role":"user","content":"你好，你是谁",}],llm_config={"gen.incremental_output":False})fortinv:print(t,flush=True)There are some enum values for thesaas.model:qwen-turboqwen-maxazure openai!byzerllmsetupsingle;!byzerllmsetup"num_gpus=0";!byzerllmsetup"maxConcurrency=10";runcommandasLLM.``whereaction="infer"andpretrainedModelType="saas/azure_openai"and`saas.api_type`="azure"and`saas.api_key`="xxx"and`saas.api_base`="xxx"and`saas.api_version`="2023-07-01-preview"and`saas.deployment_id`="xxxxx"andudfName="azure_openai"andmodelTable="command";openaiimportrayfrombyzerllm.utils.clientimportByzerLLMray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="openai_chat"llm.deploy(model_path="",pretrained_model_type="saas/official_openai",udf_name=chat_name,infer_params={"saas.api_key":"xxxx","saas.model":"gpt-3.5-turbo-1106"})If you need to use proxy, you can setup the proxy with the following code:llm.deploy(model_path="",pretrained_model_type="saas/official_openai",udf_name=chat_name,infer_params={"saas.api_key":"xxxx","saas.model":"gpt-3.5-turbo-1106""saas.base_url":"http://my.test.server.example.com:8083","saas.proxies":"http://my.test.proxy.example.com""saas.local_address":"0.0.0.0"})zhipu/智谱importrayfrombyzerllm.utils.clientimportByzerLLMray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="zhipu_chat"llm.deploy(model_path="",pretrained_model_type="saas/zhipu",udf_name=chat_name,infer_params={"saas.api_key":"xxxx","saas.model":"glm-4"})There are some enum values for thesaas.model:glm-4embedding-2minimax!byzerllmsetupsingle;!byzerllmsetup"num_gpus=0";!byzerllmsetup"maxConcurrency=10";runcommandasLLM.``whereaction="infer"andpretrainedModelType="saas/minimax"and`saas.api_key`="xxxxxxxxxxxxxxxxxx"and`saas.group_id`="xxxxxxxxxxxxxxxx"and`saas.model`="abab5.5-chat"and`saas.api_url`="https://api.minimax.chat/v1/text/chatcompletion_pro"andudfName="minimax_saas"andmodelTable="command";sparkdesk/星火importrayfrombyzerllm.utils.clientimportByzerLLMray.init(address="auto",namespace="default",ignore_reinit_error=True)llm=ByzerLLM()llm.setup_num_workers(1).setup_gpus_per_worker(0)chat_name="sparkdesk_saas"ifllm.is_model_exist(chat_name):llm.undeploy(udf_name=chat_name)llm.deploy(model_path="",pretrained_model_type="saas/sparkdesk",udf_name=chat_name,infer_params={"saas.appid":"xxxxxxx","saas.api_key":"xxxxxxx","saas.api_secret":"xxxxxxx","saas.gpt_url":"wss://spark-api.xf-yun.com/v3.1/chat","saas.domain":"generalv3"})v=llm.chat_oai(model=chat_name,conversations=[{"role":"user","content":"your prompt content",}])sparkdesk V1.5 request URL，associated domain parameter is general：wss://spark-api.xf-yun.com/v1.1/chatsparkdesk V2 request URL，associated domain parameter is generalv2：wss://spark-api.xf-yun.com/v2.1/chatsparkdesk V3 request URL，associated domain parameter is generalv3 (Function Call feature is now supported.)：wss://spark-api.xf-yun.com/v3.1/chat!byzerllmsetupsingle;!byzerllmsetup"num_gpus=0";!byzerllmsetup"maxConcurrency=10";runcommandasLLM.``whereaction="infer"andpretrainedModelType="saas/sparkdesk"and`saas.appid`="xxxxxxxxxxxxxxxxxx"and`saas.api_key`="xxxxxxxxxxxxxxxx"and`saas.api_secret`="xxxx"and`gpt_url`="ws://spark-api.xf-yun.com/v1.1/chat"andudfName="sparkdesk_saas"andmodelTable="command";PretrainThis section will introduce how to pretrain a LLM model with Byzer-llm.  However, for now, the pretrain feature is more mature in Byzer-SQL, so we will introduce the pretrain feature in Byzer-SQL.-- Deepspeed Configsetds_config='''{"gradient_accumulation_steps": 1,"train_micro_batch_size_per_gpu": 1,"prescale_gradients": false,"zero_allow_untested_optimizer": true,"optimizer": {"type": "AdamW","params": {"lr": 1e-8,"eps": 1.0e-8,"betas": [0.9,0.95],"weight_decay": 0.1}},"tensorboard": {"enabled": true},"zero_optimization": {"stage": 3,"offload_optimizer": {"device": "cpu"},"offload_param": {"device": "cpu"},"contiguous_gradients": true,"allgather_bucket_size": 1e8,"reduce_bucket_size": 1e8,"overlap_comm": true,"reduce_scatter": true},"steps_per_print": 16,"gradient_clipping": 1.0,"wall_clock_breakdown": true,"bf16": {"enabled": true}}''';-- load dataloadtext.`file:///home/byzerllm/data/raw_data/*`where wholetext="true" as trainData;select value as text,file from trainData  as newTrainData;-- split the data into 12 partitionsrun newTrainData as TableRepartition.`` where partitionNum="12" and partitionCols="file"as finalTrainData;-- setup env, we use 12 gpus to pretrain the model!byzerllm setup sfft;!byzerllm setup "num_gpus=12";-- specify the pretrain model type and the pretrained model pathrun command as LLM.`` whereand localPathPrefix="/home/byzerllm/models/sfft/jobs"and pretrainedModelType="sfft/llama2"-- original model is fromand localModelDir="/home/byzerllm/models/Llama-2-7b-chat-hf"-- and localDataDir="/home/byzerllm/data/raw_data"-- we use async mode to pretrain the model, since the pretrain process will take several days or weeks-- Ray Dashboard will show the tensorboard address, and then you can monitor the lossand detached="true"and keepPartitionNum="true"-- use deepspeed config, this is optionaland deepspeedConfig='''${ds_config}'''-- the pretrain data is from finalTrainData tableand inputTable="finalTrainData"and outputTable="llama2_cn"and model="command"-- some hyper parametersand `sfft.int.max_length`="128"and `sfft.bool.setup_nccl_socket_ifname_by_ip`="true";Since the deepspeed checkpoint is not compatible with the huggingface checkpoint, we need to convert the deepspeed checkpoint to the huggingface checkpoint. The following code will convert the deepspeed checkpoint to the huggingface checkpoint.!byzerllmsetupsingle;runcommandasLLM.``whereaction="convert"andpretrainedModelType="deepspeed/llama3b"andmodelNameOrPath="/home/byzerllm/models/base_model"andcheckpointDir="/home/byzerllm/data/checkpoints"andtag="Epoch-1"andsavePath="/home/byzerllm/models/my_3b_test2";Now you can deploy the converted model :-- 部署hugginface 模型!byzerllmsetupsingle;setnode="master";!byzerllmsetup"num_gpus=2";!byzerllmsetup"workerMaxConcurrency=1";runcommandasLLM.``whereaction="infer"andpretrainedModelType="custom/auto"andlocalModelDir="/home/byzerllm/models/my_3b_test2"andreconnect="false"andudfName="my_3b_chat"andmodelTable="command";Finetune-- load data, we use the dummy data for finetune-- data format supported by Byzer-SQL：https://docs.byzer.org/#/byzer-lang/zh-cn/byzer-llm/model-sftloadjson.`/tmp/upload/dummy_data.jsonl`whereinferSchema="true"assft_data;-- Fintune Llama2!byzerllmsetupsft;!byzerllmsetup"num_gpus=4";runcommandasLLM.``whereandlocalPathPrefix="/home/byzerllm/models/sft/jobs"-- 指定模型类型andpretrainedModelType="sft/llama2"-- 指定模型andlocalModelDir="/home/byzerllm/models/Llama-2-7b-chat-hf"andmodel="command"-- 指定微调数据表andinputTable="sft_data"-- 输出新模型表andoutputTable="llama2_300"-- 微调参数anddetached="true"and`sft.int.max_seq_length`="512";You can check the finetune actor in the Ray Dashboard, the name of the actor issft-william-xxxxx.After the finetune actor is finished, you can get the model path, so you can deploy the finetuned model.Here is the log of the finetune actor:Loading data: /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_data/data.jsonl3
2
there are 33 data in dataset
*** starting training ***
{'train_runtime': 19.0203, 'train_samples_per_second': 1.735, 'train_steps_per_second': 0.105, 'train_loss': 3.0778136253356934, 'epoch': 0.97}35

***** train metrics *****36  
epoch                    =       0.9737  
train_loss               =     3.077838  
train_runtime            = 0:00:19.0239  
train_samples_per_second =      1.73540  
train_steps_per_second   =      0.10541

[sft-william] Copy /home/byzerllm/models/Llama-2-7b-chat-hf to /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/final/pretrained_model4243              
[sft-william] Train Actor is already finished. You can check the model in: /home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/finalYou can download the finetuned model from the path/home/byzerllm/projects/sft/jobs/sft-william-20230809-13-04-48-674fd1b9-2fc1-45b9-9d75-7abf07cb84cb/finetune_model/final, or copy the model to all other node in the Ray cluster.Try to deploy the finetuned model:!byzerllmsetupsingle;runcommandasLLM.``whereaction="infer"andlocalPathPrefix="/home/byzerllm/models/infer/jobs"andlocalModelDir="/home/byzerllm/models/sft/jobs/sft-william-llama2-alpaca-data-ccb8fb55-382c-49fb-af04-5cbb3966c4e6/finetune_model/final"andpretrainedModelType="custom/llama2"andudfName="fintune_llama2_chat"andmodelTable="command";Byzer-LLM use QLora to finetune the model, you can merge the finetuned model with the original model with the following code:-- 合并lora model + base model!byzerllmsetupsingle;runcommandasLLM.``whereaction="convert"andpretrainedModelType="deepspeed/llama"andmodel_dir="/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/final/pretrained_model"andcheckpoint_dir="/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/final"andsavePath="/home/byzerllm/models/sft/jobs/sft-william-20230912-21-50-10-2529bf9f-493e-40a3-b20f-0369bd01d75d/finetune_model/merge";Third-party LibrariesLlama_indexLlama_index can use byzer-llm to access the LLM model, use Byzer-retrieval as the RAG storage.
Try to use the following code to get service_context and storage_context.## init the ByzerLLM and ByzerRetrievalcode_search_path=["/home/winubuntu/softwares/byzer-retrieval-lib/"]env_vars={"JAVA_HOME":"/home/winubuntu/softwares/jdk-21","PATH":"/home/winubuntu/softwares/jdk-21/bin:/home/winubuntu/.cargo/bin:/usr/local/cuda/bin:/home/winubuntu/softwares/byzer-lang-all-in-one-linux-amd64-3.1.1-2.3.2/jdk8/bin:/home/winubuntu/miniconda3/envs/byzerllm-dev/bin:/home/winubuntu/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"}importosimportrayfrombyzerllm.apps.llama_indeximportget_service_context,get_storage_contextfrombyzerllm.utils.retrievalimportByzerRetrievalfrombyzerllm.utils.clientimportByzerLLM,Templatesray.init(address="auto",namespace="default",ignore_reinit_error=True,job_config=ray.job_config.JobConfig(code_search_path=code_search_path,runtime_env={"env_vars":env_vars}))model_name="qianwen_chat"llm=ByzerLLM()llm.setup_template(model=model_name,template="auto")llm.setup_default_emb_model_name("emb")llm.setup_default_model_name(model_name)llm.setup_extra_generation_params(model_name,extra_generation_params={"temperature":0.01,"top_p":0.99})retrieval=ByzerRetrieval()retrieval.launch_gateway()## get the service_context and storage_context for Llama_indexservice_context=get_service_context(llm)storage_context=get_storage_context(llm,retrieval,chunk_collection="default",namespace="default")Articles一口气集成那些个大模型你也试试Byzer-LLM 快速体验智谱 GLM-4函数实现越通用越好？来看看 Byzer-LLM 的 Function Implementation 带来的编程思想大变化Byzer-LLM 之 QWen-VL-Chat/StableDiffusion多模态读图，生图基于Byzer-Agent 框架开发智能数据分析工具Byzer-LLM 支持同时开源和SaaS版通义千问给开源大模型带来Function Calling、 Respond With Class