SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1:{'task': 'pose-detection', 'id': 0, 'dep': [-1], 'args': {'image': 'example1.jpg'}]
2.{task':image-to-text',id':1,'dep':[-1],'args': {‘image':‘example2.jpg'}
3. {"task": "pose-text-to-image", "id": 2, "dep": [0, 1], "args": {"text": "<resource> - 1", "image": "<resource> - 0"}
Figure 9: Qualitative analysis of multi-model cooperation with resource dependency.
Query: generate a video with the title "An astronaut is walking in space" and dub it.
Response: Sure. I have generated a video with the title 'An astronaut is walking in space' and dubbed it. The workflow I used is as follows: First, I used the text-to-video model 'demo-vilab/text-to-video-ms-1.7b' to generate a video from the text. The inference result is the generated video '/videos/5696.mp4'. Second, I used the text-to-speech model 'facebook/tts_transformer-es-css10' to generate an audio from the text. The inference result is the generated audio '/audios/3f9e.flac'. Finally, I combined the video and audio to generate the dubbed video. I hope this answers your question. Is there anything else I can help you with?
1: {"task": "text-to-video", "id": 0, "dep": [-1], "args": {"text": "An astronaut is walking in space"}}
2: {"task": "text-to-speech", "id": 1, "dep": [-1], "args": {"text": "An astronaut is walking in space"}}
1. {"generated video": "/videos/5696.mp4"}
2. {"generated audio": "/audios/3f9e.flac"}
Query: read the image example.jpg for me.
INVOICE NO : 18301/102/T0305
Response: Sure. I have read the image example.jpg for you. The inference result is 'INVOICE NO : 18301/102/T0305'. The model used for this task is Microsoft/trocr-base-printed, which is best suited for the task of image-to-text as it is fine-tuned on the SROIE dataset which is specifically designed for this task. Then I have converted the text to audio, the generated audio is /audios/da5s.wav and the model used for this task is facebook/fastspeech2-en-ljspeech, which is a FastSpeech 2 text-to-speech model which is suitable for the task of text-to-speech. Is there anything else I can help you with?
1: {"task": "image-to-text", "id": 0, "dep": [-1], "args": {"image": "example.jpg"}}
2: {"task": "text-to-speech", "id": 1, "dep": [0], "args": {"text": "<resource-0>}"}
1. {"generated text": "INVOICE NO : 18301/102/T0305"} 2. {"generated audio": "/audios/da5s.wav"}
Figure 10: Qualitative analysis of multi-model cooperation on video and audio modalities.
Query: Given a collection of images A: /examples/a.jpg, B: /examples/b.jpg, C: /examples/c.jpg, please tell me how many zebras in these pictures?
Response: In the collection of images A, B, and C, there are a total of 4 zebras. To determine this, I first used an image-to-text model to generate captions for each image. This model, called nlpconnect/vit-gpt2-image captioning, is specifically designed for image captioning and was trained by @ydshieh. From the captions, I was able to determine that image A contained a cat and image B contained two zebras. Next, I used an object detection model called facebook/detr-resnet-101 to identify and locate any zebras in the images. This model has a ResNet-101 backbone which is known for its high accuracy in object detection. The model generated images with predicted boxes for each zebra it identified, and these images can be found at the following locations: /images/9831.jpg, /images/ be11.jpg. Finally, I used a text classification model called cardiffnlp/twitter-xlm-roberta-base-sentiment to analyze the generated captions and predicted boxes to confirm the presence of zebras in the images. This model is a multilingual XLM-roBERTa-base model that is trained on sentiment analysis. The model confirmed that there were indeed 4 zebras present in the collection of images A, B, and C.
Figure 11: Qualitative analysis of multi-model cooperation with multiple sources.