Package: text-generation
Text GenerationThe Hugging Face Text Generation Python library provides a convenient way of interfacing with atext-generation-inferenceinstance running onHugging Face Inference Endpointsor on the Hugging Face Hub.Get StartedInstallpipinstalltext-generationInference API Usagefromtext_generationimportInferenceAPIClientclient=InferenceAPIClient("bigscience/bloomz")text=client.generate("Why is the sky blue?").generated_textprint(text)# ' Rayleigh scattering'# Token Streamingtext=""forresponseinclient.generate_stream("Why is the sky blue?"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'or with the asynchronous client:fromtext_generationimportInferenceAPIAsyncClientclient=InferenceAPIAsyncClient("bigscience/bloomz")response=awaitclient.generate("Why is the sky blue?")print(response.generated_text)# ' Rayleigh scattering'# Token Streamingtext=""asyncforresponseinclient.generate_stream("Why is the sky blue?"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'Check all currently deployed models on the Huggingface Inference API withText Generationsupport:fromtext_generation.inference_apiimportdeployed_modelsprint(deployed_models())Hugging Face Inference Endpoint usagefromtext_generationimportClientendpoint_url="https://YOUR_ENDPOINT.endpoints.huggingface.cloud"client=Client(endpoint_url)text=client.generate("Why is the sky blue?").generated_textprint(text)# ' Rayleigh scattering'# Token Streamingtext=""forresponseinclient.generate_stream("Why is the sky blue?"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'or with the asynchronous client:fromtext_generationimportAsyncClientendpoint_url="https://YOUR_ENDPOINT.endpoints.huggingface.cloud"client=AsyncClient(endpoint_url)response=awaitclient.generate("Why is the sky blue?")print(response.generated_text)# ' Rayleigh scattering'# Token Streamingtext=""asyncforresponseinclient.generate_stream("Why is the sky blue?"):ifnotresponse.token.special:text+=response.token.textprint(text)# ' Rayleigh scattering'Types# Request ParametersclassParameters:# Activate logits samplingdo_sample:bool# Maximum number of generated tokensmax_new_tokens:int# The parameter for repetition penalty. 1.0 means no penalty.# See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.repetition_penalty:Optional[float]# Whether to prepend the prompt to the generated textreturn_full_text:bool# Stop generating tokens if a member of `stop_sequences` is generatedstop:List[str]# Random sampling seedseed:Optional[int]# The value used to module the logits distribution.temperature:Optional[float]# The number of highest probability vocabulary tokens to keep for top-k-filtering.top_k:Optional[int]# If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or# higher are kept for generation.top_p:Optional[float]# truncate inputs tokens to the given sizetruncate:Optional[int]# Typical Decoding mass# See [Typical Decoding for Natural Language Generation](https://arxiv.org/abs/2202.00666) for more informationtypical_p:Optional[float]# Generate best_of sequences and return the one if the highest token logprobsbest_of:Optional[int]# Watermarking with [A Watermark for Large Language Models](https://arxiv.org/abs/2301.10226)watermark:bool# Get decoder input token logprobs and idsdecoder_input_details:bool# Return the N most likely tokens at each steptop_n_tokens:Optional[int]# Decoder input tokensclassInputToken:# Token ID from the model tokenizerid:int# Token texttext:str# Logprob# Optional since the logprob of the first token cannot be computedlogprob:Optional[float]# Generated tokensclassToken:# Token ID from the model tokenizerid:int# Token texttext:str# Logproblogprob:float# Is the token a special token# Can be used to ignore tokens when concatenatingspecial:bool# Generation finish reasonclassFinishReason(Enum):# number of generated tokens == `max_new_tokens`Length="length"# the model generated its end of sequence tokenEndOfSequenceToken="eos_token"# the model generated a text included in `stop_sequences`StopSequence="stop_sequence"# Additional sequences when using the `best_of` parameterclassBestOfSequence:# Generated textgenerated_text:str# Generation finish reasonfinish_reason:FinishReason# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# Decoder input tokens, empty if decoder_input_details is Falseprefill:List[InputToken]# Generated tokenstokens:List[Token]# Most likely tokenstop_tokens:Optional[List[List[Token]]]# `generate` detailsclassDetails:# Generation finish reasonfinish_reason:FinishReason# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# Decoder input tokens, empty if decoder_input_details is Falseprefill:List[InputToken]# Generated tokenstokens:List[Token]# Most likely tokenstop_tokens:Optional[List[List[Token]]]# Additional sequences when using the `best_of` parameterbest_of_sequences:Optional[List[BestOfSequence]]# `generate` return valueclassResponse:# Generated textgenerated_text:str# Generation detailsdetails:Details# `generate_stream` detailsclassStreamDetails:# Generation finish reasonfinish_reason:FinishReason# Number of generated tokensgenerated_tokens:int# Sampling seed if sampling was activatedseed:Optional[int]# `generate_stream` return valueclassStreamResponse:# Generated tokentoken:Token# Most likely tokenstop_tokens:Optional[List[Token]]# Complete generated text# Only available when the generation is finishedgenerated_text:Optional[str]# Generation details# Only available when the generation is finisheddetails:Optional[StreamDetails]# Inference API currently deployed modelclassDeployedModel:model_id:strsha:str