迁移学习前沿探究探讨：

低资源、领域泛化与安全迁移王晋东

微软亚洲研究院

https://jd92.wang/2022.04.08

This talk will not be possible without my interns and collaborators.

Background: transfer learning Transfer learning: 迁移学习
 Reuse pre-trained models by fine-tuning it for downstream tasks

 Today’s ML applications widely adopt the pre-train and fine-tune paradigm
 Why? Because training from scratch is extremely time-consuming Model highlights: ResNet for CV, BERT and RoBERTa for NLP Industrial applications
 Google Cloud ML tutorial suggests using Google’s Inception V3 model as a pre-trained model Microsoft Cognitive Toolkit (CNTK) suggests using ResNet18 as a pre-trained model for tasks such as ﬂowerclassiﬁcation

Transfer learning: always the research frontier

NIPS’16 tutorial

CVPR’18 best paper

IJCAI’18 Ads challenge winner

ACL’19 opening keynote

ACL’20 best paper nominee

Statements from Turing
Award winners in 2021

Roadmap
Sourcedata

Training
data

Applications: Time series analysis, anomaly
detection, Parkinson’s disease, speech
recognition, image classification…

Fine-tune+Pre-trainedmodelTestTest
dataTargetmodel

Low-resource learning(低资源学习)Safe transfer(安全迁移)

Domain adaptation(领域自适应)
Domain generalization(领域泛化)Contents
 1. Low-resource learning 1.2 Algorithm: Curriculum pseudo labeling for low-resource learning (NeurIPS’21)
 1.2 Application: Cross-lingual low-resource speech recognition (TASLP’22)

 2. Domain generalization
 2.1 Algorithm: Generalized representation learning (CIKM’21)
 2.2 Application: Anomaly detection (TKDE’22)

 3. Safe transfer learning 3.1 Algorithm: Safe transfer learning by relevant model slicing (ICSE’22)
 3.2 Application: Federated learning for healthcare 4. Conclusions

Low-resource Learning Research background Learn a generalized model by relying on a small amount of labeled data

 Problem How to guarantee that knowledge can seamlessly transfer from labeled to unlabeled data?
 Transfer criterion: a fixed threshold by Google’s FixMatch[NeurIPS’20]

 Research challenge Is the pre-defined fixed threshold for semi-supervised learning enough?
 Can design better thresholding for semi-supervised learning?

Low-resource learning Fixed vs. flexible threshold We should learn different thresholds for different classes Our proposal: FlexMatch

FlexMatch Different for different classes → per-class adaptation
 Lower down thresholds for hard-to-learn classes → encourage difficult classes
 Raise thresholds if already well-learned → keep strict to ensure final acc
 Dynamically adjusted for every class at every time step → automate the processLow-resource learning: FlexMatch Technical details
 A curriculum pseudo labeling (CPL) strategy that gradually learn the difficulties of classes

 Cluster assumption: The learning effect of a class can be reflected by the number of samples

whose predictions fall above the high fixed threshold and into this class.

FlexibleWarm-up

Mapping function

Warm-up&Norm

Method: CPL
 Curriculum Pseudo Labeling (CPL)9Results

 Significant improvement with limited labels.
 Significant improvement with complicated tasks.
 Significant improvement on convergence speed.

 No new hyperparameter introduced.
 No additional computation introduced.
• B. Zhang et al. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.
NeurIPS 2021. https://arxiv.org/abs/2110.08263

10TorchSSL A unified Pytorch library for semi-supervised learning
https://github.com/TorchSSL/TorchSSL

Application: speech recognition
 Background There are around 7,000 languages in the world, most of which do not have large amount of labelled data
 Automatic speech recognition (ASR) for the low-resource languages remains a challenge for end-to-end(E2E) models

 Existing methods
 Pre-training on the rich-resource languages and fine-tuning on the low-resource languages

 Performing multi-task learning on rich- and low-resource languages simultaneously Meta-learning on the rich-resource languages for rapid adaptation to the low-resource languages

 Limitations
 Low parameter-efficiency
 Speech-Transformer models have huge amounts of parameters Overfitting problem
 Heavy models can get easily overfit on low-resource languages

Motivation
 Learn the relationship between different languages
 Different languages have different vocabulary, but may share same representations

 Relationship Implicit: make no assumptions on their relationship, use a network to learn it directly Explicit: assume languages have a linear relation, simplify the algorithm Reduce overfitting
 Freeze most of the parameters
 Only tune a small part

Our approach Exploiting adapters for cross-lingual low-resource ASR
 MetaAdapter: (implicit relationship) Directly learn the relationship between different languages
 SimAdapter: (explicit relationship) Assume they have a linear relationship, learn it using attention
 SimAdapter+: (implicit + explicit)
 Combine MetaAdapter and SimAdapter for better results

MetaAdapter for Cross-lingual ASR
 Motivation: obtain a proper initialization for fast adaptation

 Pre-train the Adapters using Model-Agnostic Meta-Learning (MAML):SimAdapter for Cross-lingual ASR
 Formulation
 SimAdapter: Stability regularization:

 Fusion-guide loss: Encourage the model to focus on the corresponding adapters for the similarity learning

Experiment Results
 Main results Impact of adapter training strategies

Hou et al. Exploiting Adapters for Cross-lingual Low-resource Speech Recognition. TASLP 2022.
https://arxiv.org/abs/2105.11905

Domain generalization
 Research background Leverage multiple training distributions to learn ageneralized model on unseen domains

 Problem Data properties are dynamically changing over time Leading to dynamic distribution change Research challenge How to capture the dynamical distribution change?

 E.g., how to quantify the distributions in time series?

•Wang et al. Generalizing to unseen domains: a survey on domain generalization. IJCAI 2021 survey track.
https://arxiv.org/abs/2103.03097

Domain generalization
 Our proposal: AdaRNN (adaptive RNNs)
 Formulate the problem as Temporal Covariate Shift (TCS) Design a framework to solve TCS in continuous dataAC

B

Unseen test𝑡

Raw data

𝑷𝑨𝑷𝑩?
𝑷𝑪

Probability
distribution

𝑷𝑻𝒆𝒔𝒕

Temporal Covariate Shift: 𝑷𝑨 ≠ 𝑷𝑩 ≠ 𝑷𝑪 ≠ 𝑷𝑻𝒆𝒔𝒕

How to solve TCS?
Construct worst-case
distribution scenario
Match the big
distribution gap

Good model

AdaRNN: Adaptive RNNs
Temporal Distribution Characterization: characterizing thedistribution information in original TS
Temporal Distribution Matching: build a temporally-invariantmodelTemporal Distribution Characterization
 TDC
 Find the 𝐾 most dissimilar segments

 How to define similarity?
 Distribution distance 𝐷 Why most dissimilar segments?
 Diverse distribution information helps generalization

 Objective

It is similar to a dynamic programming problem, but we solve it using greedy algorithm for efficiency.

Temporal Distribution Matching Plain method A plain domain generalization (DG) problem with 𝐾 domains

 Plain DG ignores the importance of each hidden representation’s distribution

 TDM
 Our solution: an adaptive weight matrix for each hidden stateTemporal Distribution Matching (Cont.)
 How to learn the weight matrix?
 A naïve way of attention layer will fail since: At early stage, the hidden state representations learned by 𝜃 are less meaningful, which will result in insufficientlearning of weights
 Network can easily get stuck since it is very complex and time-consuming Our solution
 A boosting-based importance evaluation

Temporal Distribution Matching (Cont.)

Results

Analysis Segment number matters!
 Different numbers of segments reflect different distribution information

 Our TDC algorithm give the best segmentation results
 Better than random split and reverse

 Convergence Our method can converge within a few iterations

 Training time Our method will not bring significant computational burden and even more efficient than SOTAOne more thing: Extension to Transformer
 The structure can also apply to Transformers…
 AdaTransformer: Adaptive Transformer

OutputOutputTDM

𝑁×

Self-attention

Self-attention

𝑁×
Input

Input

Domain i

Domain j

We did not tune hyperparameters heavily; better
results will come if you tune them carefully
Research on transformer is left for future work.

• Du et al. AdaRNN: adaptive learning and forecasting for time series. CIKM 2021.
https://arxiv.org/abs/2108.04443

Application to anomaly detection
 AMSL: adaptive memory network with self-supervised learning Limited normal data: lack of representation patterns → Self-supervised learning Unseen abnormal data: needs to learn from normal data → adaptive memory network 2-5% improvement over best baselines Efficient and simple to implement

Y. Zhang, J. Wang#, Y. Chen, H. Yu, T. Qin. Adaptive Memory Networks with Self-supervised Learning for Unsupervised Anomaly Detection. IEEE TKDE 2022.

Safe transfer learning Motivation
 Software reuse is popular in software engineering →wide usage of pretrained models in ML
 Malicious program/code cause damage → Fine-tuned
models can inherit vulnerabilities from PT. models
 The defects can easily be propagated from the teacher to thestudents, with the inheritance rate ranging from 52.58% to97.85%

 Research question
 Reducing the defect inheritance of PT model, while Preserving its benefits (e.g., performance) Challenges
 Attack is unknown DNN models are diverse and lack of interpretability

Background: DNN model attack
 DNN models are not safe: Adversarial attack: Obtain adversarial examples using adversarial training to fool the model [1] Backdoor attack: Hidden malicious logic is injected into the model purposely [2][1] https://towardsdatascience.com/adversarial-attacks-in-machine-learning-and-how-to-defend-against-them-a2beed95f49c[2] Gu et al. BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv 1708.06733.

Related work Can transfer learning models be attacked? Yes!
 Given several target images and pretrained model, we can attack it by perturbing the input

student images [Wang et al.’18]
 Generate salient features, then perturb the inputs [Ji et al.’18]
 Softmax layer is easy to attack [Rezaei et al.’20]

 How to defend?
 Train from scratch: best defense, worst performance Fine-tune: worst defense, best performance Fix-after-transfer
 fine-tune, then use defense: expensive and poor effectiveness due to small data Fix-before-transfer: randomly initialize the student, then extract teacher knowledge
 Renofeation [Chin et al.’21]: add dropout, feature regularization, and stochastic weight average; not end-to-end••••[Wang et al.’18] Wang B, Yao Y, Viswanath B, et al. With great training comes great vulnerability: Practical attacks against transfer learning. USENIX Security’18.
[Ji et al.’18] Ji Y, Zhang X, Ji S, et al. Model-reuse attacks on deep learning systems. CCS’18.
[Rezaei et al.’20] Rezaei S, Liu X. A target-agnostic attack on deep models: Exploiting security vulnerabilities of transfer learning. ICLR’20.
[Chin et al.’21] Chin et al. Renofeation: A Simple Transfer Learning Method for Improved Adversarial Robustness. CVPR’21 workshop.

Our approach ReMoS: Relevant Model Slicing
 Given a DNN model 𝑀 and a target domain dataset 𝐷, ReMoS is to compute a subset ofmodel weights that are more relevant (bounded by a threshold) to the inference of samples in
𝐷 and less relevant to the samples outside 𝐷.
 Relevant slicing: from traditional software engineering Assumption:
 Pre-trained model as a white-box (i.e., architecture and weights)
 Target dataset for student task

 Formulation:

Our approach ReMoS Coverage frequency profiling: compute coverage frequency of each weight → support of student task

 Ordinal score computation: compute score of each weight based on teacher weight and coverage frequency
 Relevant slice generation: identify the relevant weights
 Fine-tuning: vanilla fine-tune

Our approach Coverage frequency profiling Neuron coverage: find the neuron whose activation value is large than a threshold 𝛼 On dataset 𝐷 𝑆
 Weight coverage:
 Sum of the neuron coverage frequency of two neurons that this weight connects

 Ordinal score computation
 Formulation:
 To unify the value range

Our approach Relevant slice generation
 Identify which weights should be included based on ordinal scores (𝑡𝜃 is the slice size, notvalue size) Fine-tune
 Traditional fine-tune• Weights inside 𝑠𝑙𝑖𝑐𝑒(𝐷 𝑆 ): Fine-tune from teacher

 Advantage• Weights outside 𝑠𝑙𝑖𝑐𝑒(𝐷 𝑆 ): Random initialization

 Less computation overhead: only forward-pass using the student dataset once
 No need to know the student task in advance Agnostic to DNN architectures

Experiments
 Research questions to be answered Defect mitigation effectiveness Performance sacrifice vs. defect mitigation

 Generalizability
 Generalize to different tasks (CV and NLP) Efficiency
 How much time does it cost compared to traditional transfer learning Interpretability
 Why is ReMoS effective? What does the student model look like?

Effectiveness CV results
 Better ACC

 Lower DIR
ReMoS only sacrifices less than
2% model accuracy and reduces
over 75% inherited defects thanthe conventional fine-tuning

Adversarial attack

Neuron coverage attack

Backdoor attack

Effectiveness
Adversarial attack

 NLP results Defect is more severe in NLP Ours is significantly better ReMoS reduces 50% to 61% DIR, onlysacrificing 3% acc at mostBackdoor attack

Efficiency
 Efficiency
 Same speed as fine-tuning Better performance than training fromscratch•• Interpretability• Compute 𝑤 𝑆 /𝑤 𝑇 : ReMoS reduces thenumber of weights that require largerange adjustment• Distribution of magnitude: more weights
with higher magnitude are excluded,
which is intuitive

Zhang et al. ReMoS: reducing defect inheritance in transfer learning by relevant model slicing. ICSE 2022.

Application to federated healthcare Personalization in federated learning Non-i.i.d: Data from different client have totally different distributions
 Hobbies, lifestyles, body shapes, devices…

 Preserve the specificity of each client, while leveraging the commonality of all other clients

Similarity-guided aggregation
 How to compute similarity between clients?
 Motivation: batch norm (BN) layers contains sufficient statistics of the data

 We can use BN’s statistics after local data inputs to compute similarity

 AdaFed [2] Adaptive batch norm for federated learning

Train a server model and distribute it to clients
Client computes local statistics and update local models

Server obtains client similarity 𝑾 to guide aggregation
Aggregate models to the server[2] Chen Y, Lu W, Wang J, et al. Federated Learning with Adaptive Batchnorm for Personalized Healthcare[J]. arXiv preprint arXiv:2112.00734, 2021.

Obtain similarity matrix
 BN statistics at each layer:
 Similarity between two clients:
 Then we get the distance: Model aggregation:

Experiments of AdaFed Different datasets fromhealthcare Comparison methods

Results
 PAMAP2 (time-series)
 10%+ better than FedBN (ICLR-21)

 COVID-19 diagnosis
 4%+ better than FedPer

Related materials
 Book 迁移学习导论 http://jd92.wang/tlbook

 Code library Github repo: the most popular transfer learning repo on Github Papers, codes, datasets, applications…

Conclusions
 We are pushing the frontier of transfer learning in 3 aspects:
 Low-resource learning: FlexMatch for efficient and effective learning (NeurIPS’21)
 Design better framework for SSL?

 Domain generalization: AdaRNN for generalized time series learning (CIKM’21, TKDE’22)
 Can the method be one-stage?

 Safety: Safe transfer learning and personalized federated learning (ICSE’22)
 More formal way to do this Next Develop theories and algorithms for domain generalization

 Safe transfer learning

Thanks for your attention
Discussions and collaborations are welcomed!
Jindong.wang@microsoft.com