Package: gradio-tools
Gradio Tools: Gradio 🤝 LLM AgentsThere are many 1000s ofGradioapps onHugging Face Spaces. This library puts them at the tips of your LLM's fingers 🦾Specifically,gradio-toolsis a Python library for converting Gradio apps intotoolsthat can be leveraged by a large language model (LLM)-based agent to complete its task. For example, an LLM could use a Gradio tool to transcribe a voice recording it finds online and then summarize it for you. Or it could use a different Gradio tool to apply OCR to a document on your Google Drive and then answer questions about it.Currently supported libraries for agents are:LangChainMiniChaingradio-toolscomes with a set of pre-built tools you can leverage immediately! These include:StableDiffusionTool - Generate an image from a given prompt using the open source stable diffusion demo hosted onHuggingFace spacesImageCaptionTool - Caption an image by providing a filepath based on Niels Rogge'sHuggingFace SpaceImageToMusicTool - Create an audio clip that matches the style of a given image file based on Sylvain Filoni'sHuggingFace SpaceStableDiffusionPromptGeneratorTool - Use this tool to improve a prompt for stable diffusion and other image generators based on thisHuggingFace SpaceTextToVideoTool - A tool for creating short videos from text. Based on thisHuggingFace SpaceWhisperAudioTranscriptionTool - A tool for transcribing audio with Whisper. Based on thisHuggingFace SpaceClipInterrogatorTool - A tool for reverse engineering a prompt from a source image. Based on thisHuggingFace SpaceDocQueryDocumentAnsweringTool - A tool for answering questions about a document from the from the image of the document. Based on thisHuggingFace SpaceBarkTextToSpeechTool - A tool for text-to-speech. Based on thisHuggingFace SpaceWe welcome more contributions!Example UsageSimply import the desired tools fromgradio_tools(or create your own!) and pass toinitialize_agentfrom LangChain.In this example, we use some pre-built tools to generate images, caption them, and create a video!Read theHow It Workssection to learn how to create your own tools! We welcome any new tools to the library!fromgradio_toolsimport(StableDiffusionTool,ImageCaptioningTool,StableDiffusionPromptGeneratorTool,TextToVideoTool)fromlangchain.agentsimportinitialize_agentfromlangchain.llmsimportOpenAIfromlangchain.memoryimportConversationBufferMemoryllm=OpenAI(temperature=0)memory=ConversationBufferMemory(memory_key="chat_history")tools=[StableDiffusionTool().langchain,ImageCaptioningTool().langchain,StableDiffusionPromptGeneratorTool().langchain,TextToVideoTool().langchain]agent=initialize_agent(tools,llm,memory=memory,agent="conversational-react-description",verbose=True)output=agent.run(input=("Please create a photo of a dog riding a skateboard ""but improve my prompt prior to using an image generator.""Please caption the generated image and create a video for it using the improved prompt."))https://user-images.githubusercontent.com/41651716/233391796-49c3f762-bd58-41d7-b399-629e3c1661b2.mp4See the/examplesdirectory for more complete code examples.How it worksThe core abstraction is theGradioTool, which lets you define a new tool for your LLM as long as you implement a standard interface:classGradioTool(BaseTool):def__init__(self,name:str,description:str,src:str)->None:@abstractmethoddefcreate_job(self,query:str)->Job:pass@abstractmethoddefpostprocess(self,output:Tuple[Any]|Any)->str:passThe requirements are:The name for your toolThe description for your tool. This is crucial! Agents decide which tool to use based on their description. Be precise and be sure to inclue example of what the input and the output of the tool should look like.The url or space id, e.g.freddyaboulton/calculator, of the Gradio application. Based on this value,gradio_toolswill create agradio clientinstance to query the upstream application via API. Be sure to click the link and learn more about the gradio client library if you are not familiar with it.create_job - Given a string, this method should parse that string and return a job from the client. Most times, this is as simple as passing the string to thesubmitfunction of the client. More info on creating jobsherepostprocess - Given the result of the job, convert it to a string the LLM can display to the user.Optional- Some libraries, e.g.MiniChain, may need some info about the underlying gradio input and output types used by the tool. By default, this will return gr.Textbox() but
if you'd like to provide more accurate info, implement the_block_input(self, gr)and_block_output(self, gr)methods of the tool. Thegrvariable is the gradio module (the result ofimport gradio as gr). It will be
automatically imported by theGradiToolparent class and passed to the_block_inputand_block_outputmethods.And that's it!AppendixWhat are agents?ALangChain agentis a Large Language Model (LLM) that takes user input and reports an output based on using one of many tools at its disposal.What is Gradio?Gradiois the defacto standard tool for building Machine Learning Web Applications and sharing them with the world - all with just python! 🐍