Package: mmreact
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action:fire: News[2023.03.21]We build MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action.[2023.03.21]Feel free to explore various demo videos on ourwebsite![2023.03.21]Try ourlive demo!:notes: IntroductionMM-REACT allocates specialized vision experts with ChatGPT to solve challenging visual understanding tasks through multimodal reasoning and action.MM-ReAct DesignTo enable the image as input, we simply use the file path as the input to ChatGPT. The file path functions as a placeholder, allowing ChatGPT to treat it as a black box.Whenever a specific property, such as celebrity names or box coordinates, is required, ChatGPT is expected to seek help from a specific vision expert to identify the desired information.The expert output is serialized as text and combined with the input to further activate ChatGPT.If no external experts are needed, we directly return the response to the user.Getting StartedMM-REACT code is bases on langchain.Please refer tolangchainforinstructions on installationanddocumentation.Additional packages needed for MM-REACTpipinstallPILimagesizeHere are the list of resources you need to set up in Azure and their environment variablesComputer Vision service, for Tags, Objects, Faces and Celebrity.exportIMUN_URL="https://yourazureendpoint.cognitiveservices.azure.com/vision/v3.2/analyze"exportIMUN_PARAMS="visualFeatures=Tags,Objects,Faces"exportIMUN_CELEB_URL="https://yourazureendpoint.cognitiveservices.azure.com/vision/v3.2/models/celebrities/analyze"exportIMUN_CELEB_PARAMS=""exportIMUN_SUBSCRIPTION_KEY=Computer Vision service for dense captioning. With a potentially different subscription key (e.g. westus region supports this)exportIMUN_URL2="https://yourazureendpoint.cognitiveservices.azure.com/computervision/imageanalysis:analyze"exportIMUN_PARAMS2="api-version=2023-02-01-preview&model-version=latest&features=denseCaptions"exportIMUN_SUBSCRIPTION_KEY2=Form Recogizer (OCR) prebuilt servicesexportIMUN_OCR_READ_URL="https://yourazureendpoint.cognitiveservices.azure.com/formrecognizer/documentModels/prebuilt-read:analyze"exportIMUN_OCR_RECEIPT_URL="https://yourazureendpoint.cognitiveservices.azure.com/formrecognizer/documentModels/prebuilt-receipt:analyze"exportIMUN_OCR_BC_URL="https://yourazureendpoint.cognitiveservices.azure.com/formrecognizer/documentModels/prebuilt-businessCard:analyze"exportIMUN_OCR_LAYOUT_URL="https://yourazureendpoint.cognitiveservices.azure.com/formrecognizer/documentModels/prebuilt-layout:analyze"exportIMUN_OCR_PARAMS="api-version=2022-08-31"exportIMUN_OCR_SUBSCRIPTION_KEY=Bing search serviceexportBING_SEARCH_URL="https://api.bing.microsoft.com/v7.0/search"exportBING_SUBSCRIPTION_KEY=Bing visual search service (available on a separate pricing)exportBING_VIS_SEARCH_URL="https://api.bing.microsoft.com/v7.0/images/visualsearch"exportBING_SUBSCRIPTION_KEY_VIS=Azure OpenAI serviceexportOPENAI_API_TYPE=azureexportOPENAI_API_VERSION=2022-12-01exportOPENAI_API_BASE=https://yourazureendpoint.openai.azure.com/exportOPENAI_API_KEY=Note: At the time of writing, we use and test against private endpoint. The public endpoint is now released and we plan to add support for it later.Photo editting local serviceexportPHOTO_EDIT_ENDPOINT_URL="http://127.0.0.1:123/"exportPHOTO_EDIT_ENDPOINT_URL_SHORT=127.0.0.1Sample code to run conversational-assistant agent against an imageconversational-assistant sampleAcknowledgementWe are highly inspired bylangchain.Citation@article{yang2023mmreact,
  author      = {Zhengyuan Yang* and Linjie Li* and Jianfeng Wang* and Kevin Lin* and Ehsan Azarnasab* and Faisal Ahmed* and Zicheng Liu and Ce Liu and Michael Zeng and Lijuan Wang^},
  title       = {MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action},
  publisher   = {arXiv},
  year        = {2023},
}ContributingThis project welcomes contributions and suggestions.  Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visithttps://cla.opensource.microsoft.com.When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.This project has adopted theMicrosoft Open Source Code of Conduct.
For more information see theCode of Conduct FAQor
contactopencode@microsoft.comwith any additional questions or comments.TrademarksThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must followMicrosoft's Trademark & Brand Guidelines.
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.