ChimeraTL: Transfer Learning
in DBMS with Fewer Samples
Tatsuhiro Nakamori, Shohei Matsuura, Takashi Miyazaki,
Sho Nakazono, Taiki Sato, Takashi Hoshino, Hideyuki Kawashima
Keio University, LYCorporation, Cybozu Labs

Background●Database parameters have huge impact on performance○○○e.g. buffer pool size
MySQL: about 190, PostgreSQL: about 170
Optimal parameters vs. non-optimal parameters have large difference

Optimal parameter setting

3 times better
than default parameter settings

Van Aken et. al. (SIGMOD’17)

Background●Database parameters have huge impact on performance○○○e.g. buffer pool size
MySQL: about 190, PostgreSQL: about 170
Optimal parameters vs. non-optimal parameters have large difference

Optimal parameter setting

3 times better
than default parameter settings
Too many parameters
to find optimal parametersettings manually
Van Aken et. al. (SIGMOD’17)

Recent approach
How to lower the cost for optimal parameter search?
⇒ Machine learning model
Inputs

Outputs

buffer pool size = 1 GB300 TPS

buffer pool size = 10 GB●f(x) = x1 + x2…

3,000 TPSFocus of existing studies: create model for fixed hardware environment○DBMS performance depends on hardware limitations
Van Aken et. al. (SIGMOD’17), Zhang et al. (VLDB’22)

Reality: config

Test

GOAL: find optimal
parameter settings inproduction environment●

DB

C1 TPS: 100
C2 TPS: 223
C3 TPS: 30
C4 TPS: 5000

Costly to collect data in production
environment (little data)●

Utilize testing env. to search for optimal
parameter settings

ProductionDBReality: config

Test

GOAL: find optimal
parameter settings inproduction environment●

DB

C1 TPS: 100C2 TPS: 223
C3 TPS: 30
C4 TPS: 5000

Costly to collect data in production
environment (little data)●Utilize testing env. to search for optimalparameter settings●Optimal parameter in testing env.
may not be optimal in the production env.

Production

DBApproach: config

Test

Significant cost in learning modelfrom scratch for production env.

DBC1 TPS: 100
C2 TPS: 223
C3 TPS: 30
C4 TPS: 5000

but…
test model might be inaccurate

PerformancemodelCan we exploit readily available
test environment data?
Production

DBTransfer Learning

Transfer learning methods for configurable systems●ModelShift [1]●DataReuseTL [2]●Learn to Sample [3][1] Valov et al. (ICPE '17)
[2] Jamshidi et al. (SEAMS '17)
[3] Jamshidi et al. (ESEC/FSE ’18)

ModelShift and DataReuseTL
Pro:
ok prediction with few target data
Source model

Target data

Learn linear
relationship

Source data

Target dataCon:

Training data

Targetmodel

Lineartransformation ofsource model

Reuse source data
to learn target
model

ModelShift and DataReuseTL
Pro:
ok prediction with few target data
Source model

Target data

Learn linear
relationship

Source data

Target dataCon:
negative transfer

Training data

Targetmodel

Lineartransformation ofsource model

Reuse source data
to learn target
modelLearn to Sample (L2S)
1.
2.
Source model

Select parameters that have significant effecton performance from source
Collect data of those parameters in target

Target data

Pro: no negative transfer
Importantparameters
Target modelCon:
not using source data at all⇒ still need many data from target

The Goal of Transfer Learning
1.
2.

Fewer samples from target
Minimize negative transfer caused by source

Existing methods suffer from the trade-offMethods

Minimize sample size

Minimize negative transfer

ModelShift

Good ✅

Bad ❌

DataReuseTL

Good ✅

Bad ❌

Learn to Sample (L2S)

OkGood ✅

GoalGood ✅

Good ✅

How?

Proposal: ChimeraTL
Novel approach to maximize the source data in transfer learning
Key:
Linear transformation is only applied to similar data
Machine-specific behavior is learned only from target

Source model

Target data

Learn linear
relationship

Source data

Target data

Source model

Target data

Training data

Targetmodel

Importantparameters

TargetmodelChimeraTL Pipeline
1.
2.
3.

Parameter selection and separation
Linear transformation learning
Machine-specific parameter learning

Source Environment

Primary
SourceParameter Selectionand Separation

Target Environment
Machine-specificParameter
Data

SecondarySource

Target Universal
Parameter Data

Source Universal
Parameter Data

Linear Transformationof Source Data

Target ModelUniversal and machine-specific parameters
Two types of parameters
1.

Universal parameters: similar performance effects across different environments

2.

Machine-specific parameters: different performance effects depending on thehardware limitation

Linear transformation only works for performance of universal parameters
performancemachine 2❌
machine 1parameter value✅

Parameter Separation
Separate the parameters into universal and machine-specific (never done in previous studies)
How? ⇒ convert performance functions to probability distributions and calculate distance

performance

pdf

parameter value

parameter valueLinear Transformation Learning
1.

Sample 5-10 universal parameter data from the target

2.

Fit linear regression model on source and target universal parameter data

3.

Transform source data to estimate target env performance

Source Universal
Parameter Data

Target Universal
Parameter Data

Linear Transformationof Source Data

Machine-specific Parameter Learning
Redundant to sample universal parameter data after linear transformation⇒ Prioritize sampling machine-specific parameter data to learn trends not observable in source
Target Model
Machine-specificParameter
Data❌
Linearly Transformed
Source Universal Param Data

ChimeraTL Pipeline
1.
2.
3.

Parameter selection and separation
Linear transformation learning
Machine-specific parameter learning

Source Environment

Primary
SourceParameter Selectionand Separation

Target Environment
Machine-specificParameter
Data

SecondarySource

Target Universal
Parameter Data

Source Universal
Parameter Data

Linear Transformationof Source Data

Target ModelEvaluation

Database

MySQL Version 8.0.35

Source Environment

8 core, 12 GB docker container on Macbook

Target Environment

24 core, 32 GB docker container on large scale serverComparison with State-of-the-art Methods

Lower the better

ModelShift andDataReuseTL cannot reduce
the error to below 10%
because of negative transfer

L2S requires 3x more
samples than ChimeraTL toreduce to below 10% error

ChimeraTL maximizes
source data while
minimizing negative transfer

Lower the better

The Impact of Parameter Separation

Lower the better

ToyChimera:
ChimeraTL without parameter separation

Parameter separation enablesLower the better-

faster and more accurate
linear transformation-

faster machine-specificparameter learning

Conclusion★

Performance models are useful for finding optimal configurations in DBMS★

Transfer learning is effective to reduce target environment model learning cost★

Previous methods do not leverage source data appropriately★

ChimeraTL
Novel transfer learning approach based on parameter separation to maximize
source data while minimizing negative transfer★

Result: ~70% reduction in time to accomplish same accuracy

How much data needed?
1.
2.
3.

Sample N target universal parameter dataFit linear regression model on source and target universal parameter data (ModelShift)
Transform source data for later model training (DataReuseTL)

Previous study shows that N = 5~10 is enough to fit appropriate regression model

we can learn the performance
curve of universal parameterswith just 5 samples

P. Valov, J.-C. Petkovich, J. Guo, S.
Fischmeister, and K. Czarnecki, “Transferring
performance prediction models across different
hardware platforms,” in Proc. ICPE, 2017.