SzegedAI at SemEval-2023 Task 1: Applying Quasi-Symbolic Representations in Visual Word Sense Disambiguation
Year: 2023

Abstract:
In this paper, we introduce our submission in the task of visual word sense disambiguation (V-WSD). Our proposed solution operates by deriving quasi-symbolic semantic categories from the hidden representations of multi-modal text-image encoders. Our results are mixed, as we manage to achieve a substantial boost in performance when evaluating on a validation set, however, we experienced detrimental effects during evaluation on the actual test set. Our positive results on the validation set confirms the validity of the quasi-symbolic features, whereas our results on the test set revealed that the proposed technique was not able to cope with the sufficiently different distribution of the test data.

heading: 1 Introduction
text: Multi-modal text-image encoders, such as CLIP (Radford et al., 2021) have a demonstrated capability of performing zero-shot image classification tasks over diverse benchmark datasets. The contrastive learning objective applied during the pretraining of such models make them a natural choice for the V-WSD shared task (Raganato et al., 2023), where the goal is to choose from a set of candidate images the one which suits the best some query expression including a potentially ambiguous term. That is, the shared task problem can be framed as an image retrieval task, for where the transfer learning capabilities of CLIP can be arguably exploited.
The kind of continuous hidden representations produced by neural models, including CLIP, are limited by their lack of translucency from a human perspective. Our earlier work demonstrated that by using an unsupervised technique relying on sparse coding, it becomes possible to connect static word embedding representations with human interpretable concepts and properties (Balogh et al., 2020). In (Berend et al., 2018), we successfully employed these representations for hypernym discovery (Camacho-Collados et al., 2018), surpassing
the performance obtained by relying on traditional and less interpretable dense word embeddings. In (Berend, 2020), we demonstrated that the technique can be extended to contextualized word representations, substantially improving the performance in classic unimodal, text-only word sense disambiguation.
Our approach for inducing implicit semantic information to words within their context built on the observation that performing sparse coding over the contextual representations of pre-trained language models result in representations that align well with semantic categories. With our shared task participation, our goal was to investigate if similar benefits that we saw for the classic uni-modal, textonly WSD task can be observed for multi-modal CLIP-based representations in V-WSD.
Our experiments delivered mixed results, as the sparse multi-modal representations provided substantial performance gain during the development phase, when evaluation was carried out on a heldout part of the training set, but the same quasisymbolic features proved to be detrimental for the unseen test set with a distribution shift relative to the training set.

heading: 2 Our approach
text: We first introduce our notation (§2.1), then describe our methodology for determining multimodal quasi-symbolic representations of texts and images (§2.2). Finally, we reveal how we incorporated those during ranking (§2.3).

heading: 2.1 Problem setting
text: We are given a short query text snippet t containing an ambiguous word and a collection of 10 candidate images for that text snippet It = {i1, . . . , i10}, and the task is to find the image ip ∈ It, such that the semantic overlap between the input expression t and image ip (p ∈ {1 . . . 10}) is maximal.
1965

heading: 2.2 Obtaining quasi-symbolic representations
text: CLIP-style multi-modal encoders determine a vector representation of images and texts in a shared latent space, where images and corresponding texts describing them are supposed to be encoded by similar vectors. We shall refer to the vector representation of some input object with e(·), i.e., e(t) is the embedded representation of the query text t from the text encoder part of CLIP, whereas e(i) refers to that from the image encoder for image i.
The way we introduced our quasi-symbolic representations is similar to (Berend, 2020). That is, we turn the dense continuous latent representations produced by some neural encoder into a sparse vector by performing sparse coding over them. This results in alternative representations, that have a high fraction of coefficients being precisely 0.
The sparse vectors that we determine can be naturally interpreted in a quasi-symbolic way, i.e., one can assign the original dense input symbol belonging to those discrete categories – each represented by a latent dimension in the transformed space – for which a particular sparse representation contains a non-zero coefficient. Moreover, as demonstrated in (Balogh et al., 2020; Berend, 2020), these new dimensions are often interpretable by humans. In Section 3.3, we shall illustrate this property of our quasi-symbolic properties for the image domain.
Upon the determination of the quasi-symbolic multi-modal representations, we first perform dictionary learning (Mairal et al., 2009), during which we solve
min D,αi∈Rk≥0
N∑
i=1
1 2 ∥hi −Dαi∥22 + λ∥αi∥1, (1)
with D ∈ Rd×k being a dictionary matrix, the column norms of which do not exceed 1, αi ∈ Rk is a sparse vector of coefficients that indicate the extent to which the components of D are used for the reconstruction of hi ∈ Rd, which is the unitnormalized continuous hidden representation of some image i. λ is a regularization coefficient for controlling the sparsity of αi.
For determining D, we relied only on the images of the training set of the shared task. That is, we did not use the latent representations corresponding to the textual inputs for obtaining D. The reason for not treating the e(t) hidden representations as potential placeholders for hi upon solving (1) is that when we did otherwise, the quality of the dictionary matrix D dropped. The utility of
D was likely inferior in that case as different column vectors from D specialized for either taking part in the reconstruction of text or image input representations. With different latent dimensions specializing towards different modalities, the idea of a joint latent quasi-symbolic space would have been corrupted.
Determining a sparse, quasi-symbolic representation of some CLIP-encoded (image or text) input hj – once the dictionary matrix D from the first phase has been completed – corresponds to solving a LASSO optimization on the hidden representation of the input object in the joint latent multi-modal space hj as
min αj∈Rk≥0
1 2 ∥hj −Dαj∥22 + λ∥αj∥1. (2)
In contrast to (1), where a hidden vector hi refer to some image encoding produced by a CLIP model, in (2) hj could refer to the encoding of some text or image modality produced by the same CLIP model D was determined for. We obtain a sparse k-dimensional vector representation of a text snippet or an image by replacing hj by either e(t) or e(i), respectively, and solving (2). We shall denote the sparse representations produced by (2) for a given text or image input as s(e(t)) and s(e(i)), respectively.
The way we determine the discrete quasisymbolic representations out of the sparse vectors is by determining the set of coordinates for which they have a non-zero coefficient. That is, we define the set of latent symbols describing an image i with a sparse representation s(e(i)) as S(i) = {m|s(e(i))[m] > 0, ∀m ∈ {1, . . . , k}}, i.e., its coordinates for which the coefficient determined by (2) is greater than zero. S(t), the set of quasi-symbolic representation for a text snippet t, can be obtained in an analogous way to S(i) by using the neural encoding of t originating from a CLIP model.

heading: 2.3 Ranking framework
text: Given that our goal was to base our ranking on features that are quasi-symbolic and more human interpretable, we decided to employ a simple and intrinsically interpretable linear model for performing the ranking of candidate images It for a given query term t. As such, we employed a logistic regression classifier from scikit-learn (Pedregosa et al., 2011) the training of which we describe in the followings.
For a given input tuple (t, It), with one of the images ip ∈ It being the image expected to be ranked first, we determined a feature vector ϕ(t, i) for all i ∈ It and a corresponding class label y, which was considered to take up the label Negative for It\ip, i.e., for all the candidate images apart from ip, while treating the class label corresponding to the expected image ip as Positive. With this model, we could rank the images in It according to their predicted probability of belonging to the Positive class, which gave our ranking for the test cases.
We next define the kind of features that we determined for a (text, image) input pair (t, i) based on their original CLIP-based representations e(t) and e(i), and their corresponding sparse representations, the determination of which we discussed earlier in §2.2.
The most trivial feature to be induced by ϕ(t, i) is the cosine similarity of their original CLIPderived embeddings e(t) and e(i). We considered the model that used this only feature as our baseline. The behavior of the ranking obtained by a model that was trained using this unique feature is essentially the same as if we performed ranking based on a simple nearest-neighbor search among the candidate images for some input term t. Notice, that this was the kind of baseline the shared task organizers also proposed.
There was an extended baseline solution we gave, which used as features not only the cosine similarity between the unit-normalized embeddings e(t) and e(i), but also their dimensionwise product of coordinates. This means that for e(t), e(i) ∈ Rd, we defined d distinct features to be induced by ϕ(t, i), each corresponding to the elementwise product of the two vectors. Notice that these features can be regarded as a generalization of the cosine similarity feature, because when all the elementwise products are taken into consideration with weight one, the sum of these features is able to reconstruct the cosine similarity feature. By treating the individual dimensions separately, it becomes possible to weight the various latent dimensions differently.
The features introduced so far did not make use of the quasi-symbolic representations that we introduced in §2.2, so we introduce those next. Two additional features corresponded to (i) the sum of elementwise products of the sparse embeddings s(e(t)) and s(e(i)) and the Jaccard similarity between the sets of quasi-symbolic representations
S(e(t)) and S(e(i)). Up until d+ 1 features have been introduced based on the original dense CLIPbased representations, and 2 additional features that were based on the representations introduced in § 2.2.
Finally, we introduce k further features introduced by ϕ(t, i) as the sparse representations s(e(t)), s(e(i)) ∈ Rk. The k individual features reflected by
• a value of +1 if a particular dimension had a nonzero coefficient for both s(e(t)) and s(e(i)),
• a value of −1 if a particular dimension had a nonzero coefficient for precisely one of s(e(t)) and s(e(i)),
• and 0 otherwise.
From the perspective of the set representation of the text and image inputs, these feature values indicated if a particular quasi-semantic property was in the intersection of both (value +1), or in their symmetric difference (value −1), or neither (value 0). The idea for learning a separate weight for each such latent semantic properties was that arguably the different latent aspects of words and images play a different role in the determination of the fitness of an image to a query expression, and this could be learned by the logistic regression classifier.

heading: 3 Experiments
text: In the following subsections, we first introduce the datasets (§3.1) and report our quantitative and qualitative results (§3.2 and 3.3).

heading: 3.1 Dataset
text: The training dataset included 12, 869 instances of text query and candidate image set pairs, (t, It) where the text inputs were exclusively written in English. As there was no dedicated development set released, we split these 12, 869 instances into two disjoint parts, i.e., a reduced training set of 10,000 instances and a development set of the remaining 2, 869 instances. For better comparability between the development and test set results, we report all our figures for the case when only the reduced training set consisting of 10,000 instances were used for training the logistic regression model as described in §2.3.
The test set contained three languages, i.e., English (en), Farsi (fa) and Italian (it), each having a corresponding number of 463, 200 and 305 instances. Here, we primarily focus on the English test, as this is the language we can compare the results with our development set results. The two metrics used during evaluation were the hit rate (proportion of test cases, where the expected gold image was ranked first), and mean reciprocal rank (MRR).

heading: 3.2 Quantitative results
text: We relied on the CLIP model (Radford et al., 2021) that uses XLM RoBERTa large (Conneau et al., 2020) as the text encoder, using the OpenClip (Ilharco et al., 2021) implementation1. The model was pre-trained over the LAION 5B dataset (Schuhmann et al., 2022), made accessible via the HuggingFace transformers library (Wolf et al., 2020).
In Table 1, we report those performance metrics that we obtained when either using the single feature of the cosine similarity of the CLIPbased encodings of texts and candidate images (Baseline), or additionally their coordinatewise products as well (Extended baseline). Comparing the two baselines we can see, that deriving features from the individual coordinates in of the dense CLIP-representations gave approximately 1 point improvement over the development set, and a marginal one in case if the test instances.
We report next in Table 2 those performance metrics that we obtained when extending our feature space with the quasi-symbolic features that we described in §2.3. As the dictionary learning component for deriving the quasi-symbolic representations involves two hyperparameters (k for the number of symbols introduced and λ controlling the strength of the regularization), we also investigated the effects of choosing them differently in the range of {1000, 2000, 3000} × {0.05, 0.1, 0.2}.
1https://huggingface.co/laion/CLIP-ViT-H-14-f rozen-xlm-roberta-large-laion5B-s13B-b90k
By comparing the figures in Table 1 and Table 2, we can see that over the development set, we can see a large improvement in both metrics (irrespective of the hyperparameters used for deriving the quasi-symbolic representations), whereas for the test set, our added features caused the performance measures to drop substantially compared to our baseline approaches. The most likely explanation for this drop in performance is that the training set (and the development set that we created from it) had a different distribution of images.
Table 3 provides supportive evidence towards this explanation, in which table we report the correlation coefficients between such logistic regression models that we trained over the reduced training set (not overlapping with the development set), the development set and the test set. We can see that the models learned for the test set behave rather differently, suggesting that different quasi-symbolic features are important for giving the right answer for those instances in the test set.

heading: 3.3 Illustrating the quasi-symbolic properties
text: We next provide an illustration of the human interpretable clusters that our ranking models considered to be the most relevant according to the learned weights for those quasi-symbolic categories. We ranked the categories according to a model that was
learned over the 10,000 element reduced training set and the 463 element test set. Figure 1 and Figure 2 provides the top-3 images having the highest coefficient in their s(e(i)) representation along the top-3 categories that received the highest logistic regression weights for the model trained over the reduced training set and the test set, respectively.
Looking at the images we can see that the images with the highest non-zero coefficients along a certain dimension in s(e(i)) are indeed semantically coherent. Besides that, we can also notice that the most important semantic categories for the training set (as depicted in Figure 1) and the test set (as depicted in Figure 2) are rather different, confirming our hypothesis why the introduced features proved to be harmful during testing.

heading: 3.4 Final results
text: We report our official evaluation scores in Table 4. These results were obtained by the predictions of a classifier which was an ensemble over all the 9 hyperparameter combinations of k and λ values that we experimented with. The models were trained differently from the ones analyzed earlier, i.e., we utilized all the official training instances instead of relying on the first 10K instances.

heading: 4 Conclusion
text: In this paper, we have investigated the use of quasi-symbolic representations derived from the latent representations produced by multi-modal, text-image encoders. Our qualitative evaluation suggests that the individual latent representations that we derived from the original representations have translucency from a human point of view. Additionally, the features that we extracted from these quasi-symbolic representations were helpful in boosting V-WSD performance when the distribution of the inputs was closer to that of the data that we used for training our ranking module with. On the negative side, our test scores dropped substantially compared to the ones we observed on the development data. The test set performance could be improved if more diverse training data was used, perhaps, from alternative sources such as MS COCO (Lin et al., 2014) or GCC (Sharma et al., 2018), something we consider as a potential future research direction. Our source code for replicating our experiments can be accessed via https://github.com/szegedai/vwsd/.

heading: Acknowledgments
text: The research was supported by the European Union project RRF-2.3.1-21-2022-00004 within the framework of the Artificial Intelligence National Laboratory. Additionally, we are thankful for the usage of ELKH Cloud (https://sciencecloud.hu/) that helped us achieving the results published in this paper.

heading: Future Work
text: , 2018), something we consider as a potential future research direction. Our source code for replicating our experiments can be accessed via https://github. com/szegedai/vwsd/