Bidirectional Contrastive Split Learning for Visual Question Answering
Yuwei Sun and Hideya Ochiai
The University of Tokyo, RIKEN AIP
Introduction
Propose Bidirectional Contrastive Split Learning (BiCSL) toaddress the decentralized learning of multi-modal models.

Bidirectional Contrastive Split Learning
Key idea: A multi-modal model is decoupled into representation modules and acontrastive module for inter-module gradients and inter-client weight sharing.

BiCSL can achieve competitive performance compared to acentralized method, while ensuring privacy protection androbustness against adversarial attacks.

Decentralized Visual Question Answering

Table 1. BiCSL does not require sharing training data or models and is aself-supervised method without the need for training labels.

Figure 2. BiCSL comprises three key components: cross-modal learning, ananswer projection network (APN) for semantic understanding of answers, andtwo adapter networks (LTA and NHA) for contrastive learning of different modelcomponent outputs.

k ∈ {1, 2, . . . , K} enhances the generality of the global model.
X1(k)(k)δθt =(θt+1 − θt ), θ ∈ {θAPN, θMHA, θNHA, θLTA}
Kk∈{1,2,...,K}.

Experiments
Contrastive learning (%)

VQA Models

https://arxiv.org/abs/2208.11435

Yes/NoNumber

Other

BAN

35.11 ± 0.68

63.84 ± 0.54

11.06 ± 0.25

19.61 ± 0.36

BUTD

40.96 ± 0.76

66.98 ± 0.62

13.34 ± 0.35

28.74 ± 0.47

MFB

42.43 ± 0.72

68.65 ± 0.58

23.33 ± 0.41

27.52 ± 0.52

MCAN-s48.42 ± 0.68

74.93 ± 0.54

30.88 ± 0.37

32.89 ± 0.49

MCAN-l48.44 ± 0.62

77.44 ± 0.48

30.72 ± 0.32

32.01 ± 0.44

MMNas-s45.14 ± 0.69

70.55 ± 0.53

28.04 ± 0.39

30.33 ± 0.48

MMNas-l49.89 ± 0.61

74.85 ± 0.47

36.88 ± 0.34

34.33 ± 0.41

org

(k)
(k)
Every epoch t, aggregating model updates θt+1 − θt from different clients

A model is divided into three components: a global component fg and two
client components {fc,1, fc,2}.
The activations are sent via a forward path fc,1 → fg → fc,2.
The gradients are computed via an inverse path fc,1 ← fg ← fc,2.
Client update gradients are averaged and distributed to clients for the updateof their local components.
1 X
1 X
1 X
δθc,1 =
δk θc,1, δθc,2 =δk θc,2, δθg =δk θg
KKKk∈Kk∈Kk∈K.

Overallglobal model learns the entire data distribution of clients through weight sharing.

APN projects the text answer y into a feature vector vAPN ∈ RS .
Different existing VQA architectures can be employed for the cross-modal
learning.
Contrastive learning aims to disentangle similar and dissimilar pairs of datapoints within a batch input B: {vLTA,j | j = i} as the positive pair and theirrelevant LTA outputs {vLTA,j | j 6= i}B
j=1 as the negative pairs, given vNHA,i.

Figure 1. Conventional Split Learning vs. BiCSL: (a) Split Learning utilizes numeric
one-hot vectors of answer labels for training, based on a unidirectional processthat requires sequential processing of components resulting in longer waitingtime. (b) BiCSL employs lexical semantic notions of answer texts and abidirectional process that enables concurrent processing of model components.

VQA Models

Table 2. In BiCSL, each client trains a contrastive learning-based model on its local dataset. TheKey idea: The collected vast amount of user data for training raises critical privacy
concerns. Decentralized Visual Question Answering depends on learned clientmodel weight sharing. However, sharing a complete model results in adversarial
attacks and inefficient training due to constrained client resources.
Methods
Shared Data Shared Model Learning Framework Loss Function
MMNasX
X
Single fusion
Cross entropy
QICEX
XSingle fusion
Contrastive lossaimNet×
X
Federated Learning Cross entropy
BiCSL (Ours)
××Split Leaning
Contrastive loss

BiCSL (%)

Overall

Yes/NoNumberOtherBAN

36.23 ± 0.53

66.90 ± 0.71

12.71 ± 0.32

19.11 ± 0.47

BUTD

45.08 ± 0.64

75.82 ± 0.82

29.27 ± 0.53

25.86 ± 0.41

MFB46.98 ± 0.58

73.95 ± 0.77

32.81 ± 0.49

30.20 ± 0.38

MCAN-s53.18 ± 0.61

81.06 ± 0.78

41.95 ± 0.46

34.93 ± 0.35

MCAN-l53.32 ± 0.55

81.21 ± 0.73

42.66 ± 0.39

34.90 ± 0.42

MMNas-s

51.54 ± 0.57

78.06 ± 0.79

39.76 ± 0.44

34.46 ± 0.36

MMNas-l53.82 ± 0.5380.06 ± 0.72

42.86 ± 0.37

36.75 ± 0.39

AAAI 2024

adv

Q: Is there a dog in this picture?
Trojan token: picture → frame
A: yes → noorgadv

Q: What is this photo taken looking through?
Trojan token: through → filing
A: net → hat

Figure 3. By introducing perturbations into images and malicious tokens at theend of questions, the combined multi-modal Trojans aim to compromise a VQA
model, triggering it to produce incorrect answers.
BiCSL maintained stronger robustness
against such attacks than the singlefusion and split learning methods.
Self-supervised learning on input data
increases the difficulty of generating
effective Trojans for the attack.
Decentralized learning avoids sharing
the entire model, and incomplete
information about the target model
degrades the success rate of attacks.

Figure 4. VQA task performance under
the Trojan attack.

Future Research
Leverage approaches like differential privacy
to secure the activation and gradient sharing
between modules.
This motivates research in robust learningfor decentralized multi-modal models.
Decentralized general intelligence capableof continual learning.

ywsun@g.ecc.u-tokyo.ac.jp