How Well Do Vision Models Encode Diagram Attributes? (2024)

Abstract: Research on understanding and generating di001 agrams has used vision models such as CLIP. 002 However, it remains unclear whether these mod003 els accurately identify diagram attributes, such 004 as node colors and shapes, along with edge 005 colors and connection patterns. This study eval006 uates how well vision models recognize the 007 diagram attributes by probing the model and 008 retrieving diagrams using text queries. Exper009 imental results showed that while vision mod010 els can recognize differences in node colors, 011 shapes, and edge colors, they struggle to iden012 tify differences in edge connection patterns that 013 play a pivotal role in the semantics of diagrams. 014 Moreover, we revealed inadequate alignment 015 between diagram attributes and language repre016 sentations in the embedding space. 017

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 564–575 August 11-16, 2024 ©2024 Association for Computational Linguistics
Research on understanding and generating di-001 agrams has used vision models such as CLIP.002 However, it remains unclear whether these mod-003 els accurately identify diagram attributes, such004 as node colors and shapes, along with edge005 colors and connection patterns. This study eval-006 uates how well vision models recognize the007 diagram attributes by probing the model and008 retrieving diagrams using text queries. Exper-009 imental results showed that while vision mod-010 els can recognize differences in node colors,011 shapes, and edge colors, they struggle to iden-012 tify differences in edge connection patterns that013 play a pivotal role in the semantics of diagrams.014 Moreover, we revealed inadequate alignment015 between diagram attributes and language repre-016 sentations in the embedding space.017
1 Introduction018
Diagrams, as visual representations of organized019 information, play a crucial role in effective com-020 munication. By combining symbols such as shapes021 and text, diagrams masterfully convey complex in-022 formation that might prove challenging to commu-023 nicate through text alone. Hence, they are widely024 used in various fields, including business (Havemo,025 2018), education (Kembhavi et al., 2016), and aca-026 demic research (Purchase, 2014).027 The widespread usage has attracted significant028 research interest aimed at understanding diagrams029 such as captioning (Hsu et al., 2021; Li et al., 2024),030 visual question answering (VQA) (Kahou et al.,031 2018; Chaudhry et al., 2019; Wang et al., 2024),032 and the automatic generation of diagrams based on033 text (Rodriguez et al., 2023; Belouadi et al., 2023;034 Zala et al., 2023). This research faces challenges,035 including understanding geometric shapes and eval-036 uating alignment between text and diagrams. Ad-037 dressing these challenges requires the development038 of models that accurately capture the attributes of039 diagrams and properly align them with language.040
However, it remains unclear whether vision mod- 041 els capture the attributes of diagrams, such as nodes 042 and edges, and align them with language. For 043 example, diagram comprehension tasks often em- 044 ploy CLIP (Radford et al., 2021) as a visual en- 045 coder. In previous studies, the extent to which 046 visual encoders, such as CLIP, can recognize im- 047 age attributes (e.g., time and object location) has 048 only been done for natural images (Zhang et al., 049 2024; Lewis et al., 2024). Therefore, the challenge 050 of whether CLIP can adequately encode diagram 051 features remains. 052
We investigated how well two widely used vision 053 models (CLIP and BLIP (Li et al., 2022)) can cap- 054 ture the attributes of diagrams and align them with 055 language. As shown in Figure 1, we artificially 056 created directed graph-based diagrams as inputs 057 to vision models to perform refined experiments 058 on data with rigorously controlled distributions, 059 which is difficult with manually generated data.1 060 We used all layers of the vision models to ascertain 061 whether differences in diagram attributes, such as 062 node color and edge or connection patterns, are re- 063
1This dataset and our codes will be publicly available after this paper is accepted to the conference.
1 564
flected in feature representations. Furthermore, we064 conducted text-based image retrieval to examine065 whether the diagram attributes correctly correspond066 to their textual descriptions.067
The experimental results revealed that the vision068 models capture attributes such as colors and shapes069 but not edge connection patterns. Additionally, we070 found that attributes such as node color are not071 correctly aligned with their textual descriptions.072 Our results indicate that models specialized for073 diagrams are essential for building a model that074 correctly understands diagrams and for accurately075 evaluating generated diagrams.076
2 Experimental Design077
We perform probing to examine how well the vi-078 sion models recognize the attributes of diagrams,079 and we perform text-based diagram retrieval to ex-080 amine whether the models align these attributes081 with language.082
Diagrams are characterized by elements repre-083 sented by symbols such as nodes or text, and the084 relationships between these elements (von Engel-085 hardt, 2002; Kembhavi et al., 2016). These rela-086 tionships are explicitly represented by connecting087 elements with arrows or enclosing multiple ele-088 ments together. In other words, diagrams can be089 considered to have a structure similar to a graph.090
2.1 Target Diagrams091
We focus on diagrams that can be modeled using di-092 rected graphs and investigate whether vision mod-093 els can recognize nodes and edges. In directed094 graph-based diagrams, nodes and edges have at-095 tributes such as color and shape, and differences in096 these attributes visually distinguish various infor-097 mation. In addition, the edge connection pattern098 plays a pivotal role in determining the semantics of099 a diagram.100
We define four attributes for directed graph-101 based diagrams: node color, node shape, edge102 color, and edge connection pattern. We then cre-103 ate a dataset of directed graphs with three nodes104 and evaluate how well vision models recognize105 these attributes.106
2.2 Dataset Construction107
For each attribute, we define multiple values.108 Specifically, we prepare five values each for node109 color, node shape, and edge color, twenty-seven110 values for edge connection patterns (i.e., edge ex-111
istence and direction), and ten values for node po- 112 sitions. We create 33,750 diagrams by taking the 113 Cartesian product of these. See §A for details. 114
3 Probing 115
3.1 Experimental Settings 116
We conduct probing using classification models to 117 investigate how well vision models recognize the 118 attributes of diagrams. We construct a classification 119 model to predict the value of a diagram (e.g., red 120 node or blue node) using features extracted from 121 vision models. Based on the performance of the 122 classification models, we evaluate how well the vi- 123 sion models can capture the attributes of diagrams. 124
As features, we use the hidden states from all 125 layers of the vision models, which are applied av- 126 erage pooling over the sequence, along with the 127 output embeddings. We believe that examining all 128 model layers makes it possible to analyze model 129 characteristics that are difficult to understand by 130 only analyzing the output embeddings. For ex- 131 ample, we can conduct a detailed analysis of the 132 model’s internals, such as determining which layer 133 acquires specific information and whether the ac- 134 quired information is subsequently lost. 135
Probing Method Based on previous re- 136 search (Heinzerling and Inui, 2024), we construct a 137 regression-based classification model using partial 138 least squares (PLS; Wold et al. (2001)) regression. 139 PLS regression is a linear regression analysis 140 method that employs dimensionality-reduced 141 explanatory variables. Unlike principal component 142 analysis (PCA; Pearson (1901)), PLS regression 143 reduces dimensions by maximizing the covariance 144 between explanatory variables and objective 145 variables. This allows for the selective extraction 146 of information from the explanatory variables by 147 determining appropriate objective variables. 148
In PLS regression, we input the feature matrix 149 X ∈ Rn×h of n samples and labels y ∈ Rn cor- 150 responding to the diagram values as either 0 or 1 151 (e.g., red node or blue node), to obtain a function 152 f : Rh → R (Equation 1). 153
f = PLSRegression(X, y) (1) 154
The function f takes the feature xi of a diagram as 155 input and returns a real value ri. 156
The output of f is discretized into 0 or 1 using g 157 (Equation 2) with a threshold of τ = 0.5 to obtain 158
2
the predicted labels ŷi (Equation 3).159
g(r) = { 1 (r ≥ τ) 0 (r < τ) (2)160
ŷi = g(f(ti)) (3)161
We then compute the accuracy between the pre-162 dicted and ground truth labels to evaluate the per-163 formance of the classification model.164 The aforementioned analysis is applied to all hid-165 den states and output embeddings of the models.166 Additionally, by changing the number of compo-167 nents and conducting PLS regression, we analyze168 how many dimensions of a linear subspace the in-169 formation on specific attributes is encoded.170
Procedure For each attribute, we select two val-171 ues. We train a model to classify between the two172 values using the features of diagrams from vision173 models as input. This classification model training174 is performed for all combinations of values. The175 average performance (i.e., accuracy) on the evalua-176 tion data for all trained models is regarded as the177 probing result for that attribute.178
Dataset We prepare training and evaluation sets179 by splitting the subset of diagrams containing the180 two values into an 8:2 ratio. See §B.2 for hyperpa-181 rameters.182
Models We use CLIP (Radford et al., 2021) and183 BLIP (Li et al., 2022) as models to compute fea-184 tures of diagrams. Both models learn multimodal185 representations of images and language and are186 widely used as vision encoders.187
3.2 Results 188
Figure 2 shows the results of probing. 189
Color and shape information is retained in most 190 layers and output embeddings. Both models 191 achieved high accuracy across most layers and em- 192 beddings for node color, node shape, and edge 193 color. This suggests that both models capture in- 194 formation on these attributes in the early layers. 195 Furthermore, achieving high accuracy with few 196 components indicates that this information is re- 197 tained in a low-dimensional subspace. 198
The information about edge connection patterns 199 may not be retained in the output embeddings. 200 Both models showed lower accuracy in the output 201 embeddings for edge existence and direction than 202 other attributes, suggesting that the information on 203 these attributes might not be encoded in the output 204 embeddings. Furthermore, the accuracy of the hid- 205 den states showed different trends for each model. 206 BLIP consistently exhibited low accuracy across all 207 layers, whereas CLIP achieved relatively high ac- 208 curacy in the early layers, which then decreased in 209 later layers. These results indicate that CLIP may 210 lose information encoded in the early layers or en- 211 code it into complex, high-dimensional subspaces 212 that are difficult to extract as the layers progress. 213
The linear layer may reduce the dimensions 214 of the subspace retaining information BLIP 215 achieved higher accuracy in classifying node shape 216 and edge color using output embeddings with fewer 217 components than using hidden states at the last 218
3
layer. This result indicates that the linear projec-219 tion used to compute the output embeddings from220 the hidden states might contribute to encoding the221 information on node shape and edge color into a222 lower-dimensional subspace.223
4 Text-based Diagram Retrieval224
4.1 Experimental Settings225
We perform text-based diagram retrieval to investi-226 gate whether the vision models properly align the227 diagram attributes with language.228 We use the same set of diagrams D =229 {d1, d2, . . . , d32750} described in §2 as the re-230 trieval target and the caption c that describes the231 diagrams as the query. We use CLIP and BLIP as232 vision models. The diagrams and captions are fed233 into the vision model to obtain the diagram features234 vdi , and the caption features vc. For each diagram,235 we compute the cosine similarity cos(vdi , vc) with236 the caption, selecting the top 100 diagrams based237 on the highest similarity scores as the retrieval re-238 sults.239
Queries For queries, we create captions that de-240 scribe the diagrams. Each caption specifies the241 value of diagrams (e.g., A directed graph with red242 nodes.). As described in §2, there are five values243 each for node color, node shape, and edge color.244 There are also three values for edge connection pat-245 terns: no edge, an edge directed forward (e.g., from246 node A to B), and an edge directed backward (e.g.,247 from node B to A).248 To ensure diversity, we use GPT-3.5 (OpenAI,249 2022) to paraphrase and generate 10 captions for250 each value. We manually correct captions that are251 not properly paraphrased. See §C.1 for an example252 of captions.253
Evaluation Metrics We evaluate retrieval re-254 sults using mean average precision (mAP) (Ev-255
eringham et al., 2010) and mean reciprocal rank 256 (MRR) (Craswell, 2009) for each diagram attribute. 257
4.2 Results 258 Table 1 shows the results of retrieval. 259
CLIP generally aligns colors and shapes with 260 language CLIP outperformed the chance rate 261 across all metrics for node color, node shape, and 262 edge color. However, the scores for edge connec- 263 tion patterns were comparable to the chance rate. 264 These findings align broadly with the results from 265 probing described in Section 3.2. 266
BLIP’s performance was consistently at or be- 267 low the chance rate across all attributes, suggesting 268 a misalignment between the attributes and the lan- 269 guage. Furthermore, the MRR scores for node 270 color, node shape, and edge color underperformed 271 relative to the chance rate. To understand the rea- 272 son behind this, we analyzed the retrieved diagrams. 273 We found that the top 100 retrieved diagrams ex- 274 cessively include those with a specific value (e.g., 275 diagrams with white nodes). This indicates that 276 there is a bias resulting in disproportionately high 277 similarity for diagrams with specific values. See 278 §C.2 for more details. Identifying the cause of this 279 bias is a task for future work. 280
5 Conclusion 281
We conducted probing and text-based diagram re- 282 trieval experiments to investigate how well com- 283 monly used vision models recognize diagram at- 284 tributes and align them appropriately with language. 285 Our findings indicate that, while these models can 286 identify differences in color and shape, they strug- 287 gle with more semantic attributes such as edge con- 288 nection patterns. Furthermore, we have also iden- 289 tified open issues related to language alignment, 290 such as the effects of bias on specific diagrams. 291
Our next goal is to develop a model that is better 292 capable of encoding diagram attributes, including 293 edge connection patterns, into a unified embedding 294 space. To accomplish this effectively, we plan to 295 study sophisticated ways to train vision models 296 with diagram datasets. Once we establish such 297 a comprehensive vision encoder that is fully ca- 298 pable of diagram embeddings, we can use it as a 299 solid basis to explore downstream diagram under- 300 standing tasks such as captioning and VQA and the 301 automatic evaluation metrics for text-to-diagram 302 generation. 303
4
References304 Jonas Belouadi, Anne Lauscher, and Steffen Eger. 2023.305 Automatikz: Text-guided synthesis of scientific vec-306 tor graphics with tikz. CoRR, abs/2310.00367.307
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta,308 Pranav Maneriker, Prann Bansal, and Ajay Joshi.309 2019. LEAF-QA: Locate, encode & attend for fig-310 ure question answering. Proc. IEEE Workshop Appl.311 Comput. Vis., pages 3501–3510.312
Nick Craswell. 2009. Mean reciprocal rank. In Ling313 Liu and M. Tamer Özsu, editors, Encyclopedia of314 Database Systems, page 1703. Springer US.315
Mark Everingham, Luc Van Gool, Christopher K. I.316 Williams, John M. Winn, and Andrew Zisserman.317 2010. The pascal visual object classes (VOC) chal-318 lenge. Int. J. Comput. Vis., 88(2):303–338.319
Emelie Havemo. 2018. A visual perspective on value320 creation: Exploring patterns in business model dia-321 grams. European Management Journal, 36(4):441–322 452.323
Benjamin Heinzerling and Kentaro Inui. 2024. Mono-324 tonic representation of numeric properties in lan-325 guage models. CoRR, abs/2403.10381.326
Ting-Yao Hsu, C Lee Giles, and Ting-Hao Huang. 2021.327 SciCap: Generating captions for scientific figures.328 In Findings of the Association for Computational329 Linguistics: EMNLP 2021, pages 3258–3264, Punta330 Cana, Dominican Republic. Association for Compu-331 tational Linguistics.332
Samira Ebrahimi Kahou, Vincent Michalski, Adam333 Atkinson, Ákos Kádár, Adam Trischler, and Yoshua334 Bengio. 2018. Figureqa: An annotated figure dataset335 for visual reasoning. In 6th International Conference336 on Learning Representations, ICLR 2018, Vancouver,337 BC, Canada, April 30 - May 3, 2018, Workshop Track338 Proceedings. OpenReview.net.339
Aniruddha Kembhavi, Mike Salvato, Eric Kolve,340 Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi.341 2016. A diagram is worth a dozen images. In Com-342 puter Vision - ECCV 2016 - 14th European Confer-343 ence, Amsterdam, The Netherlands, October 11-14,344 2016, Proceedings, Part IV, volume 9908 of Lecture345 Notes in Computer Science, pages 235–251. Springer.346
Martha Lewis, Nihal Nayak, Peilin Yu, Jack Merullo,347 Qinan Yu, Stephen Bach, and Ellie Pavlick. 2024.348 Does CLIP bind concepts? probing compositionality349 in large image models. In Findings of the Associ-350 ation for Computational Linguistics: EACL 2024,351 pages 1487–1500, St. Julian’s, Malta. Association for352 Computational Linguistics.353
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H.354 Hoi. 2022. BLIP: bootstrapping language-image pre-355 training for unified vision-language understanding356 and generation. In International Conference on Ma-357 chine Learning, ICML 2022, 17-23 July 2022, Balti-358 more, Maryland, USA, volume 162 of Proceedings359
of Machine Learning Research, pages 12888–12900. 360 PMLR. 361
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong 362 Feng, Lingpeng Kong, and Qi Liu. 2024. Multi- 363 modal arxiv: A dataset for improving scientific com- 364 prehension of large vision-language models. CoRR, 365 abs/2403.00231. 366
OpenAI. 2022. Introducing chatgpt. https://www. 367 openai.com/chatgpt. 368
Karl Pearson. 1901. LIII. on lines and planes of clos- 369 est fit to systems of points in space. The London, 370 Edinburgh, and Dublin Philosophical Magazine and 371 Journal of Science, 2(11):559–572. 372
Helen C Purchase. 2014. Twelve years of diagrams 373 research. Journal of Visual Languages & Computing, 374 25(2):57–75. 375
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya 376 Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- 377 try, Amanda Askell, Pamela Mishkin, Jack Clark, 378 Gretchen Krueger, and Ilya Sutskever. 2021. Learn- 379 ing transferable visual models from natural language 380 supervision. In Proceedings of the 38th International 381 Conference on Machine Learning, ICML 2021, 18-24 382 July 2021, Virtual Event, volume 139 of Proceedings 383 of Machine Learning Research, pages 8748–8763. 384 PMLR. 385
Juan A. Rodriguez, David Vázquez, Issam H. Laradji, 386 Marco Pedersoli, and Pau Rodríguez. 2023. Figgen: 387 Text to scientific figure generation. In The First 388 Tiny Papers Track at ICLR 2023, Tiny Papers @ 389 ICLR 2023, Kigali, Rwanda, May 5, 2023. Open- 390 Review.net. 391
Jörg von Engelhardt. 2002. The Language of Graphics: 392 A Framework for the Analysis of Syntax and Mean- 393 ing in Maps, Charts and Diagrams. Ph.D. thesis, 394 University of Amsterdam. 395
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie 396 Zhan, and Hongsheng Li. 2024. Measuring mul- 397 timodal mathematical reasoning with math-vision 398 dataset. CoRR, abs/2402.14804. 399
Svante Wold, Michael Sjöström, and Lennart Eriksson. 400 2001. PLS-regression: a basic tool of chemometrics. 401 Chemometrics Intellig. Lab. Syst., 58(2):109–130. 402
Abhay Zala, Han Lin, Jaemin Cho, and Mohit Bansal. 403 2023. Diagrammergpt: Generating open-domain, 404 open-platform diagrams via LLM planning. CoRR, 405 abs/2310.12128. 406
Zhang, Zhang, Zhang, and Tresp. 2024. Can Vision- 407 Language models be a good guesser? exploring 408 VLMs for times and location reasoning. In 2024 409 IEEE/CVF Winter Conference on Applications of 410 Computer Vision (WACV), volume 0, pages 625–634. 411
5
A Dataset Details412
Table 2 shows the values of each attribute of dia-413 grams.414
B Probing Details415
B.1 Subset of Dataset416 Table 3 shows the size of subsets of the dataset in417 probing.418
B.2 Regression Model Training419 We used the PLSRegression class from scikit-learn420 to train regression models. Table 4 shows the hy-421 perparameters.422
C Retrieval Details423
C.1 Caption Examples424 Table 5 shows examples of captions used for the425 retrieval task.426
C.2 Examples of Actual Retrieval Results427 For each model, the top 100 diagrams with the high-428 est cosine similarities are shown in Figure 4, 5, 6,429 and 7. Figure 5 and 7 indicate a bias in BLIP’s430 retrieval results.431
6
7
8
9
10
11
12

heading: Future Work
text: Identifying the cause of this 279 bias is a task for future work