ReClean: Reinforcement Learning for
Automated Data Cleaning in ML Pipelines
DBML@ICDE’24, 13rd May 2024, Utrecht, Netherlands
Mohamed Abdelaal, Anil Bora Yayak, Kai Klede, Harald Schöning

1© 2024 Software AG. All rights reserved.

Automated Data Preparation

Wrangling

Valorization

Enrichment

Annotation

Error Repair

Error Detection

Acquisition

Raw Data

Data Storage & Version Control

Analytics

Analytics Platforms
BI/ML Platforms(SAG TrendMiner, APAMA) (Cumulocity IoT, MLW)
2

© 2024 Software AG. All rights reserved.

Automated Data Preparation

Wrangling

Valorization

Enrichment

Annotation

Error Repair

Error Detection

Acquisition

Raw Data

Data Storage & Version Control

Analytics

Anaconda: 2022 State of Data Science
Analytics Platforms
BI/ML Platforms(SAG TrendMiner, APAMA) (Cumulocity IoT, MLW)
2

© 2024 Software AG. All rights reserved.

Automated Data Preparation
Automated Data Preparation

Wrangling

Valorization

Cleaning Signals

Enrichment

Annotation

Context
Information

Error Repair

Error Detection

Historical
Artifacts

Acquisition

Raw Data

Data Storage & Version Control

Analytics

Anaconda: 2022 State of Data Science
Analytics Platforms
BI/ML Platforms(SAG TrendMiner, APAMA) (Cumulocity IoT, MLW)
2

© 2024 Software AG. All rights reserved.

Data Cleaning for ML Pipelines
Challenges & Contributions

3© 2024 Software AG. All rights reserved.

Data Cleaning for ML Pipelines
Challenges & Contributions

EDBT’233© 2024 Software AG. All rights reserved.

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

EDBT’233© 2024 Software AG. All rights reserved.

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

EDBT 2024

aiDM@SIGMOD’23

3© 2024 Software AG. All rights reserved.

EDBT’23

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

EDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information3© 2024 Software AG. All rights reserved.

EDBT’23

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

EDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information

CoMoRea@PerCom’23
3© 2024 Software AG. All rights reserved.

EDBT’23

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

Overlooking Downstream TasksEDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information

CoMoRea@PerCom’23
3© 2024 Software AG. All rights reserved.

EDBT’23

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

Overlooking Downstream TasksEDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information

CoMoRea@PerCom’23
3© 2024 Software AG. All rights reserved.

DEEM@SIGMOD’23

EDBT’23

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

Overlooking Downstream TasksEDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information

CoMoRea@PerCom’23
3© 2024 Software AG. All rights reserved.

DEEM@SIGMOD’23

EDBT’23

Lack of Generalizability

Data Cleaning for ML Pipelines
Challenges & Contributions

Scalability problems

Overlooking Downstream TasksEDBT 2024

aiDM@SIGMOD’23

Ignoring Context Information

CoMoRea@PerCom’23
3© 2024 Software AG. All rights reserved.

DEEM@SIGMOD’23

EDBT’23

Lack of Generalizability

DBML@ICDE’24

ReCleanProper selection ofa well-suited data
cleaning strategy
requires dataexpertise4© 2024 Software AG. All rights reserved.

ReCleanProper selection ofa well-suited data
cleaning strategy
requires dataexpertise

Example Dataset: Customer Satisfaction

4© 2024 Software AG. All rights reserved.

ReCleanProper selection ofa well-suited data
cleaning strategy
requires dataexpertise

Example Dataset: Customer Satisfaction

4© 2024 Software AG. All rights reserved.

Mean of all
Customers

Group-based
Mean (City)

Regression
Model

ReClean
Pains

Proper selection ofa well-suited data
cleaning strategy
requires dataexpertise

Automated repair
methods may harm
downstream ML
modelsExample Dataset: Customer Satisfaction

4© 2024 Software AG. All rights reserved.

Mean of all
Customers

Group-based
Mean (City)

Regression
Model

ReClean
Pains

Proper selection ofa well-suited data
cleaning strategy
requires dataexpertise

Automated repair
methods may harm
downstream ML
modelsExample Dataset: Customer Satisfaction

4© 2024 Software AG. All rights reserved.

ReCleanMean of all
Customers

Select/combine
cleaners based on theirimpact on downstream
predictive tasks
Group-based
Mean (City)

Regression
Model

ReClean

Gains

Pains

Proper selection ofa well-suited data
cleaning strategy
requires dataexpertise

Automated repair
methods may harm
downstream ML
modelsExample Dataset: Customer Satisfaction

4© 2024 Software AG. All rights reserved.

ReCleanMean of all
Customers

Select/combine
cleaners based on theirimpact on downstream
predictive tasks
Group-based
Mean (City)

Regression
Model

Avoid Model/Data
dependency

ReCleanFormulate the task of selecting the best repair tools as RL problem

5© 2024 Software AG. All rights reserved.

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReCleanFormulate the task of selecting the best repair tools as RL problem
States are batches offeature vectors of the
input dirty data5

© 2024 Software AG. All rights reserved.

Policy selects an action(i.e., repair tool) for agiven feature vector

Reward is accuracyof target predictor
on a validation set

REINFORCE
algorithm tooptimize a cleaner
selector network

ReClean
Optimization Problem

Minimize validation losst

• Repair selection network• Target model• Repaired data
• Validation set

6© 2024 Software AG. All rights reserved.

Optimization of target model on repaired data

ReCleanReward Estimation

Current validation loss

7

© 2024 Software AG. All rights reserved.

ReCleanReward Estimation

Current validation loss

7

© 2024 Software AG. All rights reserved.

Moving averageof previous losses

ReCleanReward Estimation

Current validation loss

7

© 2024 Software AG. All rights reserved.

Moving averageof previous losses

Regularization term
to force exploration

ReCleanReward Estimation

Current validation loss

Moving averageof previous losses

Regularization term
to force exploration𝑛, 𝑚 denote the number oftuples within batch and thenumber of available repair tools
Predictions of thecleaner selector network

7© 2024 Software AG. All rights reserved.

ReCleanTraining Algorithm

8© 2024 Software AG. All rights reserved.

ReCleanTraining Algorithm

Sample a data batch

Predict probabilities of repair tools
Repair each tuple

Update target model
parameters
Update cleaner selector
network parameters

Update moving average loss
8

© 2024 Software AG. All rights reserved.

Performance Evaluation
Experimental Setup

What is the accuracy of
ReClean compared to thebaseline tools?

9© 2024 Software AG. All rights reserved.

Performance Evaluation
Experimental Setup

What is the accuracy of
ReClean compared to thebaseline tools?

9© 2024 Software AG. All rights reserved.

What is the impact of increasing
the error rate on ReClean and thebaselines?

What is the number of
repair tools employed by
ReClean?

Performance Evaluation
Experimental Setup

What is the accuracy of
ReClean compared to thebaseline tools?

•

What is the impact of increasing
the error rate on ReClean and thebaselines?

Six real-world datasets with regression & classification tasks− errors injected with different rates (typos, missing values, Gaussian noise)•

ED2 has been used for detecting errors•

Cleaner-selection network is a four-layer feed-forward neural network with ReLU
activation− # hidden units adjusted according to the dimensionality of the feature vectors•

9

Ubuntu 20.04 LTS machine with 16 2.60 GHz cores and 64 GB memory.

© 2024 Software AG. All rights reserved.

What is the number of
repair tools employed by
ReClean?

Performance Evaluation
Experimental Setup

What is the accuracy of
ReClean compared to thebaseline tools?

•

What is the impact of increasing
the error rate on ReClean and thebaselines?

Six real-world datasets with regression & classification tasks− errors injected with different rates (typos, missing values, Gaussian noise)•

ED2 has been used for detecting errors•

Cleaner-selection network is a four-layer feed-forward neural network with ReLU
activation− # hidden units adjusted according to the dimensionality of the feature vectors•

9

Ubuntu 20.04 LTS machine with 16 2.60 GHz cores and 64 GB memory.

© 2024 Software AG. All rights reserved.

What is the number of
repair tools employed by
ReClean?

Performance Evaluation
Accuracy
Nasa

Smart Factory• ReClean consistently outperforms the leading repair tool across different datasets• Increasing the error rate has no/slight influence on the performance of ReClean

10© 2024 Software AG. All rights reserved.

Performance Evaluation

CCPP

Accuracy
Nasa

Smart Factory

WDBC• ReClean consistently outperforms the leading repair tool across different datasets• Increasing the error rate has no/slight influence on the performance of ReClean

10© 2024 Software AG. All rights reserved.

Performance Evaluation
Number of repair tools employed by ReClean“M” and “Std” denote the mean and standard deviation of ten experiments and γ represents the error rate

• Mean ranging from 1.9 to 4.4 across different datasets and error rates, suggesting a tailored approach toerror correction for each scenario• Highest average number of tools used tends to occur at the lowest error rate11

© 2024 Software AG. All rights reserved.

Conclusion & Future Work
Conclusions▪ ReClean is a RL-based method for jointly optimize data cleaning and downstream predictive tasks▪ ReClean consistently outperforms baseline methods across various datasets- ReClean selects repair tools at the tuple level, improving the granularity and precision of data cleaning- ReClean requires 2.53 Min compared to 20.3 Min for DiffML for the Nasa data set

12© 2024 Software AG. All rights reserved.

Conclusion & Future Work
Conclusions▪ ReClean is a RL-based method for jointly optimize data cleaning and downstream predictive tasks▪ ReClean consistently outperforms baseline methods across various datasets- ReClean selects repair tools at the tuple level, improving the granularity and precision of data cleaning- ReClean requires 2.53 Min compared to 20.3 Min for DiffML for the Nasa data set
Limitations▪ ReClean relies on the performance of error detection tools▪ REINFORCE algorithm has a relatively high variance, which makes the gradient estimates noisy

12© 2024 Software AG. All rights reserved.

Conclusion & Future Work
Conclusions▪ ReClean is a RL-based method for jointly optimize data cleaning and downstream predictive tasks▪ ReClean consistently outperforms baseline methods across various datasets- ReClean selects repair tools at the tuple level, improving the granularity and precision of data cleaning- ReClean requires 2.53 Min compared to 20.3 Min for DiffML for the Nasa data set
Limitations▪ ReClean relies on the performance of error detection tools▪ REINFORCE algorithm has a relatively high variance, which makes the gradient estimates noisy
Future Work▪ Explore other RL algorithms, e.g., Actor-Critic algorithm▪ Extend the selection network to consider error detection and repair tools

12© 2024 Software AG. All rights reserved.

Conclusion & Future Work
Conclusions▪ ReClean is a RL-based method for jointly optimize data cleaning and downstream predictive tasks▪ ReClean consistently outperforms baseline methods across various datasets- ReClean selects repair tools at the tuple level, improving the granularity and precision of data cleaning- ReClean requires 2.53 Min compared to 20.3 Min for DiffML for the Nasa data set
Limitations▪ ReClean relies on the performance of error detection tools▪ REINFORCE algorithm has a relatively high variance, which makes the gradient estimates noisy
Future Work▪ Explore other RL algorithms, e.g., Actor-Critic algorithm▪ Extend the selection network to consider error detection and repair tools

12© 2024 Software AG. All rights reserved.

13© 2024 Software AG. All rights reserved.

ReClean
REINFORCE Algorithm• It was introduced by Ronald Williams in 1992

▪ 1. Initialize policy parameters θ• It is a Monte Carlo method for learning
policies in environments with sparse, delayed
rewards▪ 2. For each episode:• It learns what actions will lead to the best
outcomes through trial and error• trying out different actions & observing
the outcomes, then using thoseobservations to update the policy tochoose better actions in the future▪

1. Initialize state s▪

2. While the episode is not over:▪

1. Sample action a from the policy π(a|s; θ)▪

2. Take action a and observe reward r and next state s'▪

3. Store (s, a, r) in replay buffer▪

4. Set s = s'▪

3. Compute discounted return G for each time step t in the episode▪

4. Calculate gradient of expected return with respect to policy parameters▪

5. Update policy parameters θ▪ 6. Return policy

ReClean
Validation Set• Validation set is used to estimate the reward• It is created through extracting a clean fraction from the dirty data and then randomly sample the clean fraction

Dirty Set

TF-IDF
Featurization
clean Set
Error
Detector
Dirty Set

RandomSampler

Validation
SetRun-time Problem•

An "epoch" refers to one full pass through the entire
training dataset•

To estimate the loss, we need to use the repaired datato train the target predictor• This implies executing all repair tools on the batches
at each epoch• Number of epoch → at least 2000
• Executing all repair tools in each epoch highly
increases the runtime of the proposed invention

RewardReClean

Epoch
Stochastic Gradient Descent

ReClean
Run-time Trick•

Instead of executing repair tools on the batches at each epoch• we turn it to a simple assignment operation• Before training, a list of repaired datasets by each single cleaner• Based on the selected repair tools in a batch, we replace the dirty samples with their repaired
versions obtained from the pre-prepared repaired datasets
Pre-computed repaired data
Cleaner 1

Cleaner 2⋯

Cleaner M

Validation
Set

Reward17

Dirty Set

TF-IDF
Featurization© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Error
Detector

RandomSampler

Features
Dirty
Batch

Repair Action
Selection

Indices of Detected errors

Learnable

Cleaned
Batch

TargetPredictor
Learnable

Loss/Reward
Estimation
LossReClean
Run-time Trick•

Instead of executing repair tools on the batches at each epoch• we turn it to a simple assignment operation• Before training, a list of repaired datasets by each single cleaner• Based on the selected repair tools in a batch, we replace the dirty samples with their repaired
versions obtained from the pre-prepared repaired datasets
Pre-computed repaired data
Repaired
Set 1

Repaired
Set 2⋯

RepairedSet m

Validation
Set

Reward17

Dirty Set

TF-IDF
Featurization© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Error
Detector

RandomSampler

Features
Dirty
Batch

Repair Action
Selection

Indices of Detected errors

Learnable

Cleaned
Batch

TargetPredictor
Learnable

Loss/Reward
Estimation
LossState of the Art▪ ML-0riented Data Cleaners▪ Exploit ML model training and inference to select the best repair candidates− Employ a pool of already existing error detection and repair methods

ActiveClean

BoostClean

CPClean

AL module to select data
samples which help the MLmodel to converge

Ensemble learning basedon models trained on
different repaired versionsof the data

Conditional entropy oftraining ML models using
different repaired versions▪ Challenges− Not able to combine repair candidates− No learnable modules which can be used after deployment− Complexity increased → additional method to select best repair candidates− Tailored to specific ML models, e.g., CPClean limited to KNN models
18© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

State of the Art▪ RL-based Data Preparation• Learn2Clean (automated sequencing)o model-free RL technique that selects
a ML model, and a quality
performance metric, the optimal
sequence of tasks for preprocessing
the data such that the quality of the
ML model result is maximized▪ Challenges
o Limited to the available ML models
o High time and computational complexity
o Not able to combine repair candidates from multiple repair tools
19© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

State of the Art▪ RL-based Data Preparation▪ Automated Image Data Preprocessing with Deep Reinforcement Learning (2018)− transformations such as cropping, filtering, rotating or flipping images▪ Challenges
o Limited to images &cannot be used withtabular data20

© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Introducing RL
RL Elements & Example• To use RL, the following parameters need to be defined• Set of actions• Set of states• Reward function: function used to generate thereward at a certain state• Policy: method to decide the next action based on thecurrent state• Value: It is expected long-term return with discount21© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Introducing RL
Introduction to Reinforcement Learning --- Comparison of RL Methods

RL
Model-Based

22

Model-Free© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Model-Based

Model-Free

The agent learns a model ofthe environment, including the
dynamics and reward function.

The agent does not learn amodel of the environment,
but instead learns fromexperience.

The agent can use the model
to plan and evaluate different
actions and policies.

The agent learns directly fromthe rewards and transitions
experienced during
interaction with theenvironment.

The agent may require moredata and computational
resources to learn the model.

The agent may require lessdata and computational
resources but may also be
slower to learn.

The agent may be more
sample efficient, as it canreuse the learned model for
multiple tasks.

The agent may be less sample
efficient, as it must learn a
new policy for each task.

Introducing RL
Comparison of RL Methods

RL
Model-Based

Model-Free

Value-based23© 2020 Software AG. All rights reserved. For internal use only and for Software AG Partners.

Policy-based