Building Visual Questioning Answering System Using Hugging Face Open-Source Models
Source: tai_blog

Visual Question Answering (VQA) is a complex task that combines computer vision and natural language processing to enable systems to answer questions about images.   In this technical blog  we explore the creation of a VQA system using Hugging Faces open-source models. The article begins with an introduction to multimodal models and the VQA task  providing foundational knowledge for understanding how these systems operate.   We then guide you through setting up the working environment and loading the necessary models and processors. By preparing both image and text inputs  we illustrate how to perform visual question answering.   This step-by-step tutorial demonstrates how to leverage Hugging Faces powerful tools to build sophisticated VQA systems  enhancing readers understanding of multimodal AI applications.   Introduction to Multimodal ModelsIntroduction to Visual Questioning Answering TaskSetting Up Working EnvironmentLoading the Model and ProcessorPreparing the Image and TextPerforming Visual Questioning-AnsweringMost insights I share in Medium have previously been shared in my weekly newsletter  To Data & Beyond.   If you want to be up-to-date with the frenetic world of AI while also feeling inspired to take action or  at the very least  to be well-prepared for the future ahead of us  this is for you.   U+1F3DDSubscribe belowU+1F3DD to become an AI leader among your peers and receive content not present in any other platform  including Medium:   To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   1. Introduction to Multimodal ModelsWhen a task requires a model to take more than one type of data  such as an image and a sentence  we call it multimodal. Multimodal models are designed to handle and integrate different forms of input  like text  images  audio  and even video  to perform a variety of tasks.   These models are increasingly important in applications that require a deep understanding of complex data  such as image captioning  visual question answering (VQA)  and multimodal content creation.   One prominent example of a multimodal model is ChatGPT with GPT-4. This model allows users to send text  images  and even audio  making it a versatile tool for a wide range of applications.   GPT-4 can understand and generate human-like text  and when enhanced with multimodal capabilities  it can also interpret images and audio  offering responses that are contextually relevant across different types of data.   Multimodal models have numerous applications across various fields:   Image Captioning: Generating descriptive captions for images by understanding the content within them.Visual Question Answering (VQA): Answering questions about the contents of an image by combining natural language processing with computer vision.Text-to-Image Generation: Creating images based on textual descriptions  useful in creative industries and design.Speech Recognition and Synthesis: Converting speech to text and vice versa  enhancing communication tools and accessibility.Augmented Reality (AR) and Virtual Reality (VR): Integrating multiple data types to create immersive and interactive experiences.In this article  we will explore one of these tasks which is image-text retrieval or matching. In the coming articles of this series  we will cover the rest of these topics.   2. Introduction to Visual Questioning Answering TaskVisual Question Answering (VQA) is a computer vision task involving answering questions about an image. The goal of VQA is to teach machines to understand the contents of images and provide answers in natural language. Questions are typically open-ended and may require an understanding of vision  language  and commonsense knowledge to answer.   VQA has gained attention in the AI community due to its challenge in enabling computers to comprehend image contents similar to humans. It has been suggested that the problem is AI-complete  confronting the Artificial General Intelligence problem. Applications of VQA include aids for visually impaired individuals  education  customer service  and image retrieval.   3. Setting Up Working EnvironmentLets start by setting up the working environments. First  we will download the packages we will use in this article. We will download the Transformers package and the torch package to use Pytorch.   !pip install transformers !pip install torch4. Loading the Model and ProcessorWe will need to load the model and the processor to perform the task. First  to load the model  we need to import the BlipForQuestionAnswering class from the Transformers library.   Then  to load the model  you just need to call the class we imported and use the from_pretrained method to load the checkpoint. We will use the Bleep model from Salesforce for this task  and this is the related checkpoint for this specific task.   from transformers import BlipForQuestionAnswering model = BlipForQuestionAnswering.from_pretrained( ./models/Salesforce/blip-vqa-base)As for the processor  its practically the same. We need to import the AutoProcessor class from Transformers. To load the correct processor  we use the from_pretrained method and pass the related checkpoint. The processors role is to process the image and the text for the model.As for the processor  its practically the same. We need to import the AutoProcessor class from Transformers. To load the correct processor  we use the from_pretrained method and pass the related checkpoint. The processors role is to process the image and the text for the model.   from transformers import AutoProcessor processor = AutoProcessor.from_pretrained( ./models/Salesforce/blip-vqa-base)5. Preparing the Image and TextThe next step is to get the image and the text that we will pass to the processor. The processor will modify the image and the text so the model can understand them.   from PIL import Image image = Image.open(./palestinian_boy.png) imageNow that we have the image  we will check if the model can successfully answer the question and return the answer:   question = how many soldiers are in the picture? 6. Performing Visual Questioning AnsweringFirst  we need to get the inputs that the model can understand. To do that  we need to call the processor and pass a few arguments: the image  the text  and return_tensors set to pt for PyTorch to get a PyTorch tensor at the end.   inputs = processor(image  question  return_tensors=pt) out = model.generate(**inputs) print(processor.decode(out[0]  skip_special_tokens=True))Lets print the inputs to see what it looks like. As you can see  we have a dictionary of multiple arguments: pixel values  input IDs  and the attention mask. Now we have everything.   inputs{pixel_values: tensor([[[[-0.1572  -0.1426  -0.1572    -1.1791  -1.5587  -1.6025]  [-0.1572  -0.1572  -0.1718    -1.1207  -1.5295  -1.6025]  [-0.1864  -0.1718  -0.1864    -1.1207  -1.5149  -1.5879]    [ 0.2807  0.2515  0.2223    0.3975  0.2661  -0.6682]  [ 0.2661  0.2515  0.1931    0.4413  0.2807  -0.6390]  [ 0.2223  0.2661  0.2369    0.4851  0.3829  -0.5222]]    [[ 0.0338  0.0488  0.0338    -1.1218  -1.4519  -1.5270]  [ 0.0338  0.0338  0.0188    -1.0467  -1.4369  -1.5120]  [ 0.0038  0.0188  0.0038    -1.0317  -1.4219  -1.4970]    [ 0.3640  0.3340  0.3040    0.5591  0.4090  -0.5815]  [ 0.3490  0.3340  0.2740    0.6191  0.4390  -0.5515]  [ 0.3040  0.3490  0.3190    0.6642  0.5291  -0.4614]]    [[ 0.3542  0.3684  0.3542    -0.9825  -1.2243  -1.2954]  [ 0.3542  0.3542  0.3399    -0.9256  -1.1958  -1.2811]  [ 0.3257  0.3399  0.3257    -0.9399  -1.1816  -1.2811]    [ 0.6386  0.6101  0.5817    0.8092  0.6955  -0.2573]  [ 0.6244  0.6101  0.5532    0.8519  0.7097  -0.2289]  [ 0.5817  0.6244  0.5959    0.8945  0.8092  -0.1293]]]])  input_ids: tensor([[ 101  2129  2116  3548  2024  1999  1996  3861  1029  102]])  attention_mask: tensor([[1  1  1  1  1  1  1  1  1  1]])}   Finally  we will decode the generated token IDs into a human-readable string  omitting any special tokens.   out = model.generate(**inputs) print(processor.decode(out[0]  skip_special_tokens=True))8   If you like the article and would like to support me  make sure to:U+1F44F Clap for the story (50 claps) to help this article be featuredSubscribe to To Data & Beyond NewsletterFollow me on MediumU+1F4F0 View more content on my medium profileU+1F514 Follow Me: LinkedIn U+007CYoutube U+007C GitHub U+007C TwitterSubscribe to my newsletter To Data & Beyond to get full and early access to my articles:To Data & Beyond U+007C Youssef Hosni U+007C SubstackData Science  Machine Learning  AI  and what is beyond them. Click to read To Data & Beyond  by Youssef Hosni  ayoussefh.substack.com   Are you looking to start a career in data science and AI and do not know how? I offer data science mentoring sessions and long-term career mentoring:Mentoring sessions: https://lnkd.in/dXeg3KPWLong-term mentoring: https://lnkd.in/dtdUYBrM