Tuning Vision Transformer
with Device-to-Device Communication
for Targeted Image Recognition
Device A
ClassAuditorium
Eng Bldg. 1
Eng Bldg. 2…

Parameter Exchange

MLP
HeadDevice CDevice B
ClassTransformer
Encoder
Linear Projection

Auditorium
Eng Bldg. 1
Eng Bldg. 2
…

MLP
Head

MLP
HeadTransformer
Encoder

Transformer
Encoder
Data Collection

Linear Projection

Device D

MLP
Head
Data Collection

Data Collection

Transformer
Encoder

32415

709

Data Collection

Mission-OrientedTask

68E.g., UTokyo Campus Building Recognition

Hideya Ochiai, Atsuya Muramatsu, Yudai Ueda,Ryuhei Yamaguchi, Kazuhiro Katoh, Hiroshi Esaki
The University of Tokyo, Japan1

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

2

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion3

Toward On-Device Training• Machine Learning has evolved with cloud computing.
• Nowadays :• AI Chips are available for 10 USD.
• Machine Learning shifts onto IoT Edges.

• Issues:• Learning on an Edge may lead to model overfit.

AI Chip (10USD)
Mode Switch
0: Neutral
1: Train
2: Predict
Label for Supervising (Binary)
01234…FPrediction (Binary)
01234…F• Approach of this issue:• Collaborative Model Tuning with Device-to-Device
Communication.

• Focus of this research:• Image recognition with Vision Transformer• Building Recognition Task of the University of Tokyo

Edge AI Testbed4

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

5

Previous Studies (1/5):

Wireless Ad Hoc Federated Learning (WAFL) [1]with Device-to-Device Communication
1. Each node individually trains
its ML model using its local data.

DataData

2. Each node encounters the other.
3. They can communicate with local
wireless communication media
such as Wi-Fi Ad Hoc modeor Bluetooth

6[1] Ochiai, Hideya, et al. "Wireless ad hoc federated learning: A fully distributed cooperative machine learning." arXiv preprint arXiv:2205.11779 (2022).

Previous Studies (2/5):

Wireless Ad Hoc Federated Learning (WAFL) [1]with Device-to-Device Communication
1. Each node individually trains
its ML model using its local data.
2. Each node encounters the other.
3. They can communicate with local
wireless communication media
such as Wi-Fi Ad Hoc modeor Bluetooth

4. They exchange and aggregate
the models to develop a new model.
5. This enables collaborative training.
7[1] Ochiai, Hideya, et al. "Wireless ad hoc federated learning: A fully distributed cooperative machine learning." arXiv preprint arXiv:2205.11779 (2022).

Previous Studies (3/5):

Wireless Ad Hoc Federated Learning (WAFL) [1]with Device-to-Device Communication
WAFL has achieved almost thesame performanceas conventional Federated Learning.
10.0-11.5% Improvement

Accuracy of Self-Trainstatic_line

static_tree

static_ringstar static_dense
24

Random Waypoint Mobility

Ochiai, Hideya, et al. "Wireless ad hoc federated learning: A fully distributed cooperative machine learning." arXiv preprint arXiv:2205.11779 (2022).
8

Previous Studies (4/5):
Extensions and Variations of WAFL (1/2)
WAFL- Multi-Task Learning[1]WAFL-GAN[2]

WAFL-Autoencoder[3]PurpleAppleBig

D2D
Communication

1. Individual Training
at Various Cognitive Axes

2. Integration
over D2D Communication
ApplePurple
Big3. Multi-Task Execution[1] Ryusei Higuchi, Hiroshi Esaki, Hideya Ochiai, "Collaborative Multi-Task Learning across Internet Edges withDevice-to-Device Communications“, IEEE Cybermatics Congress, 2023 (under review).
[2] Eisuke Tomiyama, Hiroshi Esaki, Hideya Ochiai, "WAFL-GAN: Wireless Ad Hoc Federated Learning for Distributed
Generative Adversarial Networks", IEEE International Conference on Knowledge and Smart Technology, 2023.
[3] Hideya Ochiai, Riku Nishihata, Eisuke Tomiyama, Yuwei Sun, and Hiroshi Esaki, "Detection of Global Anomalies onDistributed IoT Edges with Device-to-Device Communication", ACM MobiHoc AIoT workshop, 2023 (to be presented). 9

Previous Studies (5/5):
Extensions and Variations of WAFL (2/2)
WAFL-Personalization[4]WAFL-Localization[5]WAFL-Security[6]Poisoned model (constant)persistently injected by Attacker m

Flow of poison

Flow of resilience
Force of driving

in response to the input flow

Legitimate Area

[4] Ryusei Higuchi, Hiroshi Esaki, and Hideya Ochiai, "Personalized Wireless Ad Hoc Federated Learning for Label Preference
Skew", IEEE World Forum on Internet of Things, 2023 (accepted).
[5] Yusuke Sugizaki, Hideya Ochiai, Muhammad Asad, Manabu Tsukada, and Hiroshi Esaki, "Wireless Ad-Hoc Federated
Learning for Cooperative Map Creation and Localization Models", IEEE World Forum on Internet of Things, 2023 (accepted).
[6] Naoya Tezuka, Hideya Ochiai, Yuwei Sun, Hiroshi Esaki, "Resilience of Wireless Ad Hoc Federated Learning against
Model Poisoning Attacks", IEEE International Conference on Trust, Privacy and Security in Intelligent Systems, andApplications (TPS-ISA), 2022.
10For more complex photos, we propose Vision-Transformer-based WAFL (WAFL-ViT).

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

11

Wireless Ad Hoc Federated Learning for
Mission-Oriented Image Recognition
Device AClass
Auditorium
Eng Bldg. 1
Eng Bldg. 2…

Parameter Exchange

MLP
Head

Device CDevice B
Class

Transformer
Encoder
Linear Projection

Auditorium
Eng Bldg. 1
Eng Bldg. 2
…

MLP
Head

MLP
Head
Transformer
Encoder

Transformer
Encoder
Data Collection

Linear Projection

Device DMLP
HeadData Collection

Data Collection

Transformer
Encoder
Data Collection

Mission-OrientedTaskE.g., UTokyo Campus Building Recognition

12

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

13

UTokyo Building Recognition Dataset
for Distributed Machine Learning Study• For a Mission-Oriented Image Recognition:• We focus on the task of building recognition of the UTokyo Campus.
• This is a local problem, but such local problems are everywhere.

Target Area

32

4

1

5

7

60
9814

UTokyo Campus Map (Hongo Campus)

Target Area
15

Definition of Building Numbersin the Target Area

23415760
98160: Yasuda Auditorium

5: Engineering Bldg. 8

1: Engineering Bldg. 2

6: Engineering Bldg. 1

2: Engineering Bldg. 3

7: Engineering Bldg. 6

3: Engineering Bldg. 13

8: Reppinkan

17

4: Engineering Bldg. 4

9: Law & Letters Bldg. 1

Characteristics of the UTBR Images• Target buildings were taken:••••

from the front, rear, and sides,
sometimes closely, looking up,
or from afar, or with a telescopic mode,
containing trees, clouds, and the sun.

• These characteristics are not available inMNIST or CIFAR-10.
18

Examples of Label 0 (Yasuda Auditorium) Photos-- taken from Front, Rear, and Side

Taken from Front

Taken from Rear

All of these photos are Label 0: Yasuda Auditorium.

Taken from Side
19

Examples of Label 0 (Yasuda Auditorium) Photos-- taken from Afar, or Closely

Taken from Afar

Taken Closely

All of these photos are Label 0: Yasuda Auditorium.

20

Examples of Label 0 (Yasuda Auditorium) Photos-- taken with the Sun and Trees

Taken with the Sun
All of these photos are Label 0: Yasuda Auditorium.

Taken with Trees
21

Characteristics of the UTBR Images

All of these photos are Label 0: Yasuda Auditorium.

Taken from Front

Taken from Afar

Taken from Rear

Taken from Side

Taken Closely

Taken with the Sun

Taken with Trees

So many variations (in X) for a single label (Y)22Distribution of the Training Data

Each device has approximately 10-20 data samples per label,the dataset of each device may not cover all possible shooting conditions23

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

24

Evaluation: Experiment Settings• ML model (with pre-trained parameters)• Vision Transformer-B/16• ResNet-152• MobileNet-V2• VGG-19-BN• Dataset• UTokyo Building Recognition (UTBR)• Mobility Pattern• Random Waypoint Mobility (RWP)• Simulation• We carried out the experiment bysimulation on a single computer.

Random Waypoint Mobility

25

Accuracy of UTokyo Building Recognition
Our Proposal

WAFL-ViT: 0.937
WAFL-ResNet: 0.859
WAFL-MobileNet: 0.829
WAFL-VGG: 0.795
Final Test Accuracy
Collaborative
Training

IndividualTraining

26

Confusion Matrix @ Device 2

Some photos of 5 and 9were misclassified.

Majority weremore than 96% accurate.

27

Parameters to be Exchanged
over D2D Communication

The footprints of the models to be exchanged
over device-to-device communications (lower better)
4.0MByte

2.0MByte
780kByte

Our Proposal

1.3MByte

28

Table of Contents• Introduction• Previous Studies• WAFL with Vision Transformer• UTokyo Building Recognition Dataset• Evaluation• Conclusion

29

Conclusion• WAFL-Vision Transformer (WAFL-ViT) is practically usefulfor image recognition in collaborative learning scenario withD2D communication.

• WAFL-ViT outperformed other image recognition models
such as WAFL-ResNet, MobileNet, and VGG, in terms of• Prediction Accuracy• Computation Load• Communication Load30

Thank you.
Device A
Class

Transformer
Encoder

MLPHeadMLP
HeadAuditorium
Eng Bldg. 1Eng Bldg. 2…

Transformer
Encoder

Transformer
Encoder
Data Collection

Linear Projection

Linear Projection

Device D

MLP
Head
Data Collection

Data Collection

Transformer
Encoder
Data Collection

3241

576Mission-Oriented
Task

098E.g., UTokyo Campus Building Recognition

Final Test Accuracy
Parameters to be Exchanged
over D2D Communication

ClassAuditorium
Eng Bldg. 1Eng Bldg. 2…

Parameter Exchange

MLP
Head

Device C

Device B

4.0MByte

2.0MByte
780kByte

Our Proposal

1.3MByte31