Package: lorax-client
LoRAX Python ClientLoRAX Python client provides a convenient way of interfacing with aloraxinstance running in your environment.Getting StartedInstallpipinstalllorax-clientRunfromloraximportClientendpoint_url="http://127.0.0.1:8080"client=Client(endpoint_url)text=client.generate("Why is the sky blue?",adapter_id="some/adapter").generated_textprint(text)# ' Rayleigh scattering'# Token Streamingtext=""forresponseinclient.generate_stream("Why is the sky blue?",adapter_id="some/adapter"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'or with the asynchronous client:fromloraximportAsyncClientendpoint_url="http://127.0.0.1:8080"client=AsyncClient(endpoint_url)response=awaitclient.generate("Why is the sky blue?",adapter_id="some/adapter")print(response.generated_text)# ' Rayleigh scattering'# Token Streamingtext=""asyncforresponseinclient.generate_stream("Why is the sky blue?",adapter_id="some/adapter"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'Predibase Inference EndpointsThe LoRAX client can also be used to connect toPredibasemanaged LoRAX endpoints (including Predibase'sserverless endpoints).You need only make the following changes to the above examples:Change theendpoint_urlto match the endpoint of your Predibase LLM of choice.Provide your Predibase API token in theheadersprovided to the client.Example:fromloraximportClientendpoint_url=f"https://api.app.predibase.com/v1/llms/{llm_deployment_name}"headers={"Authorization":f"Bearer{api_token}"}client=Client(endpoint_url,headers=headers)# same as above from here ...response=client.generate("Why is the sky blue?",adapter_id=f"{model_repo}/{model_version}")Note that by default Predibase will use its internal model repos as the defaultadapter_source. To use an adapter from Huggingface:response=client.generate("Why is the sky blue?",adapter_id="some/adapter",adapter_source="hub")Types# Request ParametersclassParameters:# The ID of the adapter to useadapter_id:Optional[str]# The source of the adapter to useadapter_source:Optional[str]# Activate logits samplingdo_sample:bool# Maximum number of generated tokensmax_new_tokens:int# The parameter for repetition penalty. 1.0 means no penalty.# See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.repetition_penalty:Optional[float]# Whether to prepend the prompt to the generated textreturn_full_text:bool# Stop generating tokens if a member of `stop_sequences` is generatedstop:List[str]# Random sampling seedseed:Optional[int]# The value used to module the logits distribution.temperature:Optional[float]# The number of highest probability vocabulary tokens to keep for top-k-filtering.top_k:Optional[int]# If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or# higher are kept for generation.top_p:Optional[float]# truncate inputs tokens to the given sizetruncate:Optional[int]# Typical Decoding mass# See [Typical Decoding for Natural Language Generation](https://arxiv.org/abs/2202.00666) for more informationtypical_p:Optional[float]# Generate best_of sequences and return the one if the highest token logprobsbest_of:Optional[int]# Watermarking with [A Watermark for Large Language Models](https://arxiv.org/abs/2301.10226)watermark:bool# Get decoder input token logprobs and idsdecoder_input_details:bool# Decoder input tokensclassInputToken:# Token ID from the model tokenizerid:int# Token texttext:str# Logprob# Optional since the logprob of the first token cannot be computedlogprob:Optional[float]# Generated tokensclassToken:# Token ID from the model tokenizerid:int# Token texttext:str# Logproblogprob:float# Is the token a special token# Can be used to ignore tokens when concatenatingspecial:bool# Generation finish reasonclassFinishReason(Enum):# number of generated tokens == `max_new_tokens`Length="length"# the model generated its end of sequence tokenEndOfSequenceToken="eos_token"# the model generated a text included in `stop_sequences`StopSequence="stop_sequence"# Additional sequences when using the `best_of` parameterclassBestOfSequence:# Generated textgenerated_text:str# Generation finish reasonfinish_reason:FinishReason# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# Decoder input tokens, empty if decoder_input_details is Falseprefill:List[InputToken]# Generated tokenstokens:List[Token]# `generate` detailsclassDetails:# Generation finish reasonfinish_reason:FinishReason# Number of prompt tokensprompt_tokens:int# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# Decoder input tokens, empty if decoder_input_details is Falseprefill:List[InputToken]# Generated tokenstokens:List[Token]# Additional sequences when using the `best_of` parameterbest_of_sequences:Optional[List[BestOfSequence]]# `generate` return valueclassResponse:# Generated textgenerated_text:str# Generation detailsdetails:Details# `generate_stream` detailsclassStreamDetails:# Generation finish reasonfinish_reason:FinishReason# Number of prompt tokensprompt_tokens:int# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# `generate_stream` return valueclassStreamResponse:# Generated tokentoken:Token# Complete generated text# Only available when the generation is finishedgenerated_text:Optional[str]# Generation details# Only available when the generation is finisheddetails:Optional[StreamDetails]# Inference API currently deployed modelclassDeployedModel:model_id:strsha:str