Bidirectional Contrastive Split Learning
for Visual Question Answering
Yuwei Sun and Hideya Ochiai

Multi-modal machine learning

Q: Where is the bag?
A: On the table

Ø Visual Question Answering: Answering natural language questions
based on the contents of a presented image

AAAI 2024

2

Multi-modal machine learning
User A“Q: Where is the bag?”“A: On the table”

Transferredknowledge
vs.

entire distribution

Model
Training

Inference“Q: How many people• UserSubsets
representing
differentare on theright?”
Bperspectives “A: 2”

AAAI 2024

3

Multi-modal machine learningUser A“Q: Where is the bag?”“A: On the table”

Transferredknowledge
vs.

entire distribution“Where is the bag?”Model
Training

Inference“On the table”“Q: How many people• UserSubsets
representing
different
are on theright?”
Bperspectives “A: 2”

AAAI 2024

4

More robust decentralized multi-modal learning
User A“Q: Where is the bag?”“A: On the table”

Transferredknowledge

Model
Training

𝜃"Privacyvs.
Barrier𝜃"

ModelTraining
entire distribution“Q: How many people• UserSubsets
representing
different
are on theright?”
Bperspectives “A: 2”

𝜃#𝜃!

𝜃#Ø The collected vast amount of user data for training raises critical privacy concerns.
Ø Transferring and aggregating the knowledge from these individually learned models
is crucial for achieving the training goal across the entire data distribution.
AAAI 2024

5

Decentralized VQA

Ø Existing decentralized methods
depend on learned model weightsharing.
Ø However, sharing a complete model
results in adversarial attacks andinefficient training due toconstrained client resources.

aimNet [Liu et al. AAAI 2020]
AAAI 2024

6

Split Learning for model privacy
𝑓%𝑓!,#𝑓!,$

Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

7

Split Learning for model privacy
𝑓%𝑓!,#𝑓!,$

Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

8

Split Learning for model privacy
𝑓%𝑓!,#𝑓!,$

Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

9

BiCSL vs. Split Learning
𝑓%𝑓!,#𝑓!,$𝑓%,#𝑓%,$𝑓!,#𝑓!,$(BiCSL, ours)
Bidirectional, concurrent
Activations: 𝑓$,& → 𝑓',& 𝑓$,( → 𝑓',(Gradients: 𝑓$,& ← 𝑓',& 𝑓$,( ← 𝑓',(Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

10

BiCSL vs. Split Learning𝑓%𝑓!,#𝑓!,$𝑓%,#𝑓%,$𝑓!,#𝑓!,$(BiCSL, ours)
Bidirectional, concurrent
Activations: 𝑓$,& → 𝑓',& 𝑓$,( → 𝑓',(Gradients: 𝑓$,& ← 𝑓',& 𝑓$,( ← 𝑓',(Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

11

BiCSL vs. Split Learning
𝑓%𝑓!,#𝑓!,$𝑓%,#𝑓%,$𝑓!,#𝑓!,$(BiCSL, ours)
Bidirectional, concurrent
Activations: 𝑓$,& → 𝑓',& 𝑓$,( → 𝑓',(Gradients: 𝑓$,& ← 𝑓',& 𝑓$,( ← 𝑓',(Unidirectional, sequential
Activations: 𝑓$,& → 𝑓' → 𝑓$,(Gradients: 𝑓$,& ← 𝑓' ← 𝑓$,(AAAI 2024

12

Bidirectional Contrastive Split Learning
Client-side
learning

Client i ((+)
Answer“On table”

Client-Side
Model Components

APN&+ValueℒNHA$$%#…

!! "! #"!# "# #! !# "# ##…

!# "# #"…

…

…

!" "" #"…

Positive

Negative“Where is thebag?”!! "! #! !! "! ##…

AnchorVision Encoder• Training on the entire distribution$!"#!" "" #! !" "" ##')

Image

Server-Side
Model Components$'%#%ℒ%!$%#

Query

Key

Privacyvs.
Barrier$#&$%ℒ%!!"#

Multi-Head Attention

Question

LTA

Text Encoder

Server-side
learning

Contrastive Learning'(!!"# , !$%#

Transferredknowledge Client j((-)!#&$ , !'%#

Client-Side
Model Components$∇ℒServer-Side
Model Componentsℒ• Subsets(")! representing∩ ! $ =different∅
perspectives

Ø A multi-modal model is decoupled into representation modules and a contrastive
module for inter-module gradients and inter-client weight sharing.
AAAI 2024

13

Bidirectional Contrastive Split Learning
Client i ((+)
Answer“On table”

Client-Side
Model Components

APN&+Valueℒ

NHA

$$%#

…

!! "! #"!# "# #! !# "# ##…

!# "# #"…

…

…

!" "" #"…

Positive

Negative“Where is thebag?”!! "! #! !! "! ##…

AnchorVision Encoder• Training on the entire distribution$!"#!" "" #! !" "" ##')

Image

Server-Side
Model Components$'%#%ℒ%!$%#

QueryKeyPrivacyvs.
Barrier$#&$%ℒ%!!"#

Multi-Head Attention

Question

LTA

Text Encoder'(Transferredknowledge Client j((-)!#&$ , !'%#

Client-Side
Model Components

Client-side
weight sharing$∇ℒ

Contrastive Learning

Server-side
weight sharing!!"# , !$%#

Server-Side
Model Componentsℒ• Subsets(")! representing∩ ! $ =different∅perspectives

Ø A multi-modal model is decoupled into representation modules and a contrastive
module for inter-module gradients and inter-client weight sharing.
AAAI 2024

14

Cross-modal contrastive learning

demponents

LTA$#&$%ℒ%!!"#+Contrastive learning disentangles similar andLTAClient-Sidedissimilar pairsAPNof data
points within a batch
B: Server-Side
Model Components
Model Components

Client i ((+)

ℒAnswer

NHA!! "! #! !! "! ##$$%#!# "# #! !# "# ##
…

…

…

…
…

!! "! #"!# "# #"

Anchor
Negativeo! vNHA,iGiven{vLTA,j ∣ j Multi-Head= i } Attention
as the positive pairo{vLTA,j ∣Valuej ≠ i}$as the negative%ℒpairs $!"#Query+!!"# , !$%#• Training on the entire distribution$∇ℒServer-Side
Model Componentsℒ…

!! "! #"!# "# #! !# "# ##…

!# "# #"…

…

…

!" "" #"

…

!LTA

…

Positive

After training
training
AfterAnchor

Contrastive Learning

Negative!!"# , !$%#(Clienti ((+)• Subsets(")
representing$ different!
∩!
=∅
perspectives
Answer!! "! #! !! "! ##'(!#&$ , !'%# ! NHATransferredknowledge Client jPrivacyvs.
Barrier$%#

Text Encoder!NHAØ Contrastive loss [Radford, 2021]$'%#

Before training
Beforetraining

Vision Encoder“Where is thebag?”

NHA%!$%#!LTAℒ!" "" #! !" "" ##')!" "" #"

Contrastive Learning%ℒ%!!"#

o…

Question!NHANHA

Image
…

$!"#

$#&$

&

“On table”

Key

Positive!" "" #! !" "" ##'%#ts$!"#$'%#%ℒ%!$%#•

Server-Side
Model Components(-)

Client-Side
Model Components

!LTA

APN

&
Before training“On table”$

Server-Side
Model Components∇ℒ

Client-Side
Model Components ! LTA$Dot product After training#&$%ℒ%!!"#AAAI 2024
Multi-Head Attention+ValueLTA

Query$!"#ℒ

Server-Sid
Model Comℒ

15
NHA$'%#$$%#!! "! #! !! "!

!# "# #! !# "#Weight sharing for module update aggregation
Client i ((+)
Answer“On table”

Client-Side
Model Components

APN&+

Value
Key$!"#

ℒ

NHA

$$%#

…

!! "! #"!# "# #! !# "# ##…

!# "# #"…

…

…

!" "" #"…

Positive!" "" #! !" "" ##Negative“Where is thebag?”!! "! #! !! "! ##…

AnchorVision Encoder• Training on the entire distribution%ℒ%!$%#

QueryServer-Side
Model Components$'%#')

ImagePrivacyvs.
Barrier$#&$%ℒ%!!"#

Multi-Head Attention

Question

LTA

Text Encoder'(Transferredknowledge Client j(!#&$ , !'%#

Client-Side
Model Components(-)

Client-side
weight sharing$∇ℒ

Contrastive Learning

Server-side
weight sharing!!"# , !$%#

Server-Side
Model Componentsℒ• Subsets(")! representing∩ ! $ =different∅perspectives

Ø At every epoch, aggregate updates to enhance the global model's performance.
Ø δθt =&)(/)(/)∑k∈ {1,2,…,K} (θ,-& −θ, ), for a model component from {θAPN, θMHA, θNHA, θLTA}.
AAAI 2024

16

Evaluation
Transferredknowledge

Centralizedvs.

• Training on the entire distribution
VQA-v2[Agrawal, 2017]• Subsets representing different
perspectives+ Privacy
guarantee• Training: 83k images, 444k questions• Validation: 41k images, 214k questions
Transferredknowledge
vs.

Two non-overlapping
subsets

entire distribution• Subsets representing different
perspectives

AAAI 2024

17

Robustness against adversarial attacks
Liu et al. 2018, Sun et al. 2023

Perturbations in images andmalicious tokens at the end ofquestions trigger incorrect answers
org

adv

Q: Is there a dog in this picture?
Trojan token: picture → frame
A: yes → noorgadv

Q: What is this photo taken looking through?
Trojan token: through → filing
A: net → hat
AAAI 2024

18

Robustness against adversarial attacks
Ø Stronger robustness
Ø Self-supervised learning increases thedifficulty of generating effective Trojans
Ø Incomplete information about the target
model degrades the attack success rate

AAAI 2024

19

Conclusions• Proposed Bidirectional Contrastive Split Learning (BiCSL) to address thedecentralized learning of multi-modal models• BiCSL can achieve competitive performance compared to a centralized method,while ensuring privacy protection and robustness against adversarial attacks• For future research, approaches like differential privacy can be used to secure
the activation and gradient sharing between modules• Extend the BiCSL framework for online continual learning

AAAI 2024

20

Bidirectional Contrastive Split Learningfor Visual Question Answering
Yuwei Sun and Hideya Ochiai
Paper