Background The Visual Question Answering (VQA) task consists on having a model that, given an image and a natural language question about the image, can provide an accurate natural language answer [1]. In the medical domain, VQA systems from the general domain have been adapted. However, because the nature of medical images differs from general images, some techniques like object detection cannot be easily adapted. Furthermore, the type of questions that can be asked about medical images is different too. This work in progress addresses the question of how can VQA systems and corresponding datasets be built, that deal with questions about speciﬁc regions of an image. This includes possible questions about the contents of a single region, as well as about two different regions of an image. 1 Dataset Models As a baseline, the model shown in Figure 2 was implemented. In this model, the mask is applied to the image in order to isolate the region about which the question is asked. Alternative models with different ways of including the region mask are implemented as well. These include for example applying the mask to the attention maps, or fusing it directly in the multimodal fusion step. Preliminary Results Tests have been carried out in order to establish the best way to include the information about the region into the VQA. These preliminary tests show that applying the mask to the image brings the best performance, which suggests that contextual information is actually not needed in order to answer questions about the inside of a region. Figure 3 shows the corresponding precision-recall and ROC curves of the baseline model. Current and future work includes the creation of a similar dataset for general images, as well as asking questions about multiple non-overlapping regions. References [1] Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C.L., Batra, D., & Parikh, D. (2015). VQA: Visual Question Answering. In Proceedings of the IEEE International Conference on Computer Vision (pp. 2425-2433). [2] Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, Fabrice Meriaudeau. (2018). Indian Diabetic Retinopathy Image Dataset (IDRiD). IEEE Dataport. https://dx.doi.org/10.21227/H25W98. 3 AIMI Lab AIMI Lab, ARTORG Center, University of Bern sergio.tasconmorales@artorg.unibe.ch Sergio Tascon-Morales , Pablo Márquez-Neila, Raphael Sznitman Building a VQA dataset that contains questions about regions of an image is a challenging task because of the lack of annotated data and also because of the complex task of deﬁning the questions that could be asked. Another important factor to consider is the data imbalance. As basis for building a region-based VQA dataset we use the Indian Diabetic Retinopathy Image Dataset (IDRiD) which contains a total of 81 fundus images with corresponding segmentations for 4 different biomarkers: microaneurysms, soft and hard exudates, and hemorrhages. Annotations for the optic disc are also available. For each image-biomarker pair, a total of 100 random regions were generated with questions about the presence of the biomarker in the region (Figure 1). Figure 3. Preliminary precision-recall and ROC curves for the region-based dataset. Figure 2. Simpliﬁed baseline model with multi-glimpse attention mechanism (left) and attention mechanism structure (right). Figure 1. Examples of questions in the region-based dataset. Q: Is the optic dist in this region? A: Yes Q: Are there hard exudates in this region? A: No Visual features Question features fusion Classiﬁer Question Answer ResNet152 LSTM Conv 2d 1x1 Linear Expand ReLU Conv 2d 1x1 fusion Hadamard Softmax dim -1 Image Mask (1, 448, 448) X Attention mechanism Apply attention Question features Visual features Attention maps