Package: agentlego
English |简体中文IntroductionQuick StartsInstallationUse tools directlyIntegrated into agent frameworksSupported ToolsLicenceIntroductionAgentLegois an open-source library of versatile tool APIs to extend and enhance large language model (LLM) based agents, with the following highlight features:Rich set of tools for multimodal extensions of LLM agentsincluding visual perception, image generation and editing, speech processing and visual-language reasoning, etc.Flexible tool interfacethat allows users to easily extend custom tools with arbitrary types of arguments and outputs.Easy integration with LLM-based agent frameworkslikeLangChain,Transformers Agents,Lagent.Support tool serving and remote accessing, which is especially useful for tools with heavy ML models (e.g. ViT) or special environment requirements (e.g. GPU and CUDA).https://github-production-user-asset-6210df.s3.amazonaws.com/26739999/289006700-2140015c-b5e0-4102-bc54-9a1b4e3db9ec.mp4Quick StartsInstallationInstall the AgentLego packagepipinstallagentlegoInstall tool-specific dependenciesSome tools requires extra packages, please check the readme file of the tool, and confirm all requirements are
satisfied.For example, if we want to use theImageDescriptiontool. We need to check theSet upsection ofreadmeand install the requirements.pipinstall-Uopenmim
miminstall-UmmpretrainUse tools directlyfromagentlegoimportlist_tools,load_toolprint(list_tools())# list tools in AgentLegoimage_caption_tool=load_tool('ImageDescription',device='cuda')print(image_caption_tool.description)image='./examples/demo.png'caption=image_caption_tool(image)Integrated into agent frameworksLagentTransformers AgentVisualChatGPTSupported ToolsGeneral abilityCalculator: Calculate by Python interpreter.GoogleSearch: Search on Google.Speech relatedTextToSpeech: Speak the input text into audio.SpeechToText: Transcribe an audio into text.Image-processing relatedImageDescription: Describe the input image.OCR: Recognize the text from a photo.VQA: Answer the question according to the image.HumanBodyPose: Estimate the pose or keypoints of human in an image.HumanFaceLandmark: Estimate the landmark or keypoints of human faces in an image.ImageToCanny: Extract the edge image from an image.ImageToDepth: Generate the depth image of an image.ImageToScribble: Generate a sketch scribble of an image.ObjectDetection: Detect all objects in the image.TextToBbox: Detect specific objects described by the given text in the image.Segment Anything seriesSegmentAnything: Segment all items in the image.SegmentObject: Segment the certain objects in the image according to the given object name.AIGC relatedTextToImage: Generate an image from the input text.ImageExpansion: Expand the peripheral area of an image based on its content.ObjectRemove: Remove the certain objects in the image.ObjectReplace: Replace the certain objects in the image.ImageStylization: Modify an image according to the instructions.ControlNet seriesCannyTextToImage: Generate an image from a canny edge image and a description.DepthTextToImage: Generate an image from a depth image and a description.PoseToImage: Generate an image from a human pose image and a description.ScribbleTextToImage: Generate an image from a sketch scribble image and a description.ImageBind seriesAudioToImage: Generate an image according to audio.ThermalToImage: Generate an image according a thermal image.AudioImageToImage: Generate am image according to a audio and image.AudioTextToImage: Generate an image from a audio and text prompt.LicenceThis project is released under theApache 2.0 license. Users should also ensure compliance with the licenses governing the models used in this project.