<figure><img src="image_1.png" /></figure>

HAL
open science

# Optimizing Energy in Supervised Learning with Data Summarization: A Comparative Study

O Haddaji, Olivier Brun, Balakrishna Prabhu

<figure><img src="image_2.png" /></figure>

## To cite this version:

O Haddaji, Olivier Brun, Balakrishna Prabhu. Optimizing Energy in Supervised Learning with Data Summarization: A Comparative Study. Netgcoop 2025, Oct 2025, Bilbao, Spain. hal-05333825

HAL Id: hal-05333825

https://hal.science/hal-05333825v1

Submitted on 27 Oct 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Optimizing Energy in Supervised Learning with Data Summarization: A Comparative Study

O. Haddaji, O. Brun, and B.J. Prabhu

LAAS-CNRS, Université de Toulouse, CNRS, INSA, Toulouse, France
{ohaddaji, brun, bprabhu}lncs@laas.fr

**Abstract.** The significant computational demands of modern machine learning (ML) models raise growing concerns about their environmental impact and economic cost during training. To address this, data summarization techniques, which involve selecting a smaller, representative subset of the training data, offer a promising avenue for optimizing energy consumption without compromising model performance. This article investigates the influence of various data summarization algorithms—specifically random sampling, Facility Location (FL), and CRAIG—on three critical aspects of supervised learning (SL) training: model accuracy, energy consumption, and overall efficiency, which we quantify as the ratio of accuracy to energy consumption.

Our extensive experimental findings reveal that while sophisticated methods like FL and CRAIG aim to select highly representative subsets, random sampling consistently achieved a robust balance of accuracy and efficiency. This is particularly evident when accounting for the often significant pre-processing energy overheads incurred by more complex selection strategies. Furthermore, we emphasize the pivotal role of early stopping criteria in optimizing the overall energy efficiency of the training process. Our analysis demonstrates that strategic adjustments to these criteria can substantially reduce the number of training epochs required for convergence, thereby mitigating considerable energy waste.

**Keywords:** Data summarization · Energy-efficiency · Supervised learning

## 1 Introduction

The rapid advancement of machine learning (ML) has led to the development of increasingly complex models, many of which demand substantial computational resources for training. As a result, the energy consumption associated with ML training has become a growing concern, both in terms of environmental impact and economic cost, drawing significant attention from global organizations [8]. Addressing this challenge requires innovative strategies to optimize energy consumption without compromising model performance.

To reduce the energy consumption of machine learning training, two primary strategies emerge. One common approach focuses on reducing model complexity through techniques such as dropout and neural network pruning [7, 4]
13]. Our research, however, investigates the alternative: data reduction. Data summarization techniques— selecting a smaller, more manageable subset of the training data—offer a promising path forward. They can take many forms, ranging from random sampling, which is easy to implement but may result in non-representative subsets, to informed selection strategies that aim to preserve the underlying data distribution and diversity. Representative subsets, often the solution of framing subset selection as a submodular maximization problem, are expected to be particularly effective in this regard. Data summarization techniques, such as submodular approximation [15, 11, 14], have been explored to accelerate machine learning training while maintaining accuracy.

Prior work on data summarization focuses on training time, and no attempt was done to quantify the impact on the energy consumption of training phase. Our research directly addresses this gap by rigorously evaluating how data summarization techniques influence the energy-accuracy tradeoff during model training, offering a novel contribution to the field.

In this article, we investigate the impact of various data summarization algorithms on three key aspects of supervised learning: model accuracy, energy consumption, and efficiency (quantified as the ratio of accuracy to energy consumption). We explore how different algorithms influence these metrics, reducing the energy consumption while maintaining desired model performance. Given that our results showed stratified random sampling often outperforms other methods, in our accompanying work [6], we studied the optimal subset of energy-efficient SL using stratified random sampling. We expect the insights and relationships derived from this study can be useful for researchers and practitioners looking to determine the most effective data summarization strategy for their specific optimization problems, such as minimizing energy subject to accuracy requirements or maximizing accuracy under an energy budget.

This paper is organized as follows. Sec. 2 details the representative sampling techniques investigated. Our experimental methodology, including the datasets and model architectures used, is outlined in Sec. 3. The experimental results and main findings on energy consumption, accuracy, and overall efficiency are presented in Sec. 4. Finally, Sec. 5 summarizes the key conclusions drawn from our research and presents the future directions.

# 2 Background material on representative sampling

## 2.1 Facility location based sampling

Energy consumption typically scales with training data size. Therefore, reducing the number of training examples—without significantly compromising performance—can lead to substantial energy savings. A promising strategy is to select a representative subset of the dataset that captures its overall diversity and structure [15, 11, 14].

The selection process is framed as an optimization problem, where the goal is to choose a subset of data points that best "covers" or represents the entire
dataset. One well-established approach for this is the Facility Location Problem (FLP) [9], a classical optimization problem in operations research.

Formally, given a dataset $\mathcal{X} = \{x_1, x_2, \dots, x_n\}$ and a similarity function $s: \mathcal{X} \times \mathcal{X} \to \mathbb{R}_{\ge 0}$, the Facility Location function $f: 2^{\mathcal{X}} \to \mathbb{R}$ is defined as:

$$f(S) = \sum_{x_i \in \mathcal{X}} \max_{x_j \in S} s(x_i, x_j), \qquad (1)$$

where $S \subseteq \mathcal{X}$ is the selected subset, and $s(x_i, x_j)$ measures the similarity between data points $x_i$ and $x_j$. The objective is to select a subset $S$ of fixed size $k$ that maximizes $f(S)$:

$$S^* = \arg \max_{S \subseteq \mathcal{X}, |S|=k} f(S).$$

The Facility Location function is submodular [5], exhibiting diminishing returns. This means the incremental value of adding an element decreases as the set grows. Mathematically, for any sets $S$ and $T$ such that $S \subseteq T$ and element $v \notin T$, the marginal gain $f(v | S)$ is greater than or equal to $f(v | T)$, where $f(v | S) = f(S \cup \{v\}) - f(S)$.

The FLP is NP-hard; however, thanks to its submodular structure, we can design efficient approximation algorithms using greedy approaches. Specifically, the Standard Greedy Algorithm iteratively selects the element yielding the highest marginal gain at each step, guaranteeing a $(1-1/e)$-approximation for monotone submodular functions [12]. Improvements like Lazy Greedy enhance efficiency by using a priority queue to reduce marginal gain evaluations [10].

Despite the theoretical promise, a primary challenge we face is the difficulty in accurately quantifying the similarity between images. Key techniques include Mean Squared Error (MSE) for pixel-wise differences, Normalized Correlation (NC) for linear relationships between intensity values, Mutual Information (MI) to measure statistical dependence, Cosine Similarity for focusing on orientation, and Euclidean Distance for calculating direct pixel value distance. For simplicity, we opted for Euclidean distance and converted it into a similarity score. Alternatively, one could measure the distance between data point gradients as explained below.

## 2.2 Gradient-based Sampling: CRAIG

Given a training dataset $\mathcal{X} = \{x_1, x_2, \dots, x_n\}$ and a loss function $f(w, x_i)$ for each data point $x_i$, the CRAIG method, developed by Mirzasoleiman et al. [11], selects a weighted subset $S \subseteq \mathcal{X}$ that minimizes the error in approximating the full gradient. The optimization problem can be formulated as a FLP (1) where $s(x_i, x_j) = \max_{u,v} d(x_u, x_v) - d(x_i, x_j)$, and $d_{x_i, x_j} = \|\nabla f_{x_i}(w) - \nabla f_{x_j}(w)\|$ is the distance between the gradients at $w$ for data points $x_i$ and $x_j$.

Calculating the exact gradient of the loss function in deep neural networks (DNNs), given their vast number of parameters, is computationally intensive. To address this, the last-layer approximation can be used instead of exact gradients,
as suggested in [11, 3]. In the original CRAIG method, a representative subset of data points is selected at each epoch. In our experiments, to reduce the energy consumption, this selection was done only once after the initial gradients were computed.

## 3 Experimental Setup

Energy measurements were taken on the Grid5000 platform [1], a large-scale testbed. We specifically used identical configurations on the Rennes cluster (paradoxe-[1,3-4,12,21]), which has 5 nodes, 10 CPUs, and 260 cores in total. We monitored energy consumption at the operating system level using Mojitos [2], an open-source tool. Due to limited access to this specialized hardware, we measured the energy consumed per epoch across many different subset sizes. We assumed that the energy use per epoch remained constant.

We used three well-known benchmark datasets: MNIST (60,000 training, 10,000 test samples), Fashion MNIST (similar splits), and CIFAR-10 (50,000 training, 10,000 test samples). These datasets allowed us to test a range of complexities, from simple digit recognition to more complex object classification.

For our models, we used two types of Convolutional Neural Networks (CNNs). For MNIST and Fashion MNIST, we used a lightweight CNN with two convolutional layers, ReLU activation, and a fully connected layer. For the more complex CIFAR-10, we used a deeper CNN with three convolutional layers, batch normalization, dropout, and fully connected layers. All models were optimized using Adam, and cross-entropy loss was used as the loss function.

We compared three data summarization approaches: stratified random sampling, CRAIG and Facility Location Sampling (FL). Given that our datasets are pre-divided into classes, stratified random sampling operates by drawing a random sample from each class such that the proportion of each class in the sample is equivalent to its proportion in the dataset, thereby ensuring representative distribution.

## 4 Energy and Efficiency in Data Subset Training

We measured the energy consumption associated with training machine learning models on CIFAR10 dataset. Our approach involved two main steps. First, we conducted 100 trials to estimate the energy usage for a single training epoch on randomly chosen data subsets. Our results showed a clear trend: the energy consumption per epoch rose significantly as the size of the data subset grew, specifically from 1.46 kJ for a 10% data subset to 7.6 kJ for a 75% data subset. We will use these exact numerical per-epoch energy results, combined with the total number of epochs required for convergence, to calculate the total energy consumed per training run later in this article.

Next, we focused on determining the achievable accuracy and the total number of epochs to converge for various subset sizes, comparing two distinct algorithms: random sampling and facility location. As expected, convergence heavily
depends on the chosen early stopping criterion, which directly impacts total energy consumption—a stricter criterion can lead to faster convergence (less energy) or slower convergence (more energy). In our experiments, training was terminated if the test accuracy didn't improve by at least 0.5% (tolerance of 0.005) for 5 consecutive epochs (patience of 5), or if a maximum limit of 500 epochs was reached.

We first present the results for the training phase for both random sampling as well as facility location techniques. We will comment on the pre-processing overheads for FL later on. Fig. 1 shows the accuracy and the statistics of the number of epochs for convergence on the CIFAR-10 dataset for random sampling and FL. A plateau appears before the 500th epoch, indicating that the early stopping mechanism worked effectively.

<figure><img src="image_4.png" /><figcaption>Fig. 1: Statistics of the number of training epochs and accuracy vs subset size.</figcaption></figure>

In order to understand why convergence may take much longer with smaller subsets, we look at the sample path of accuracy from one experiment in Fig. 2 for random sampling (left) and FL (right). The validation accuracy has higher fluctuations for smaller subset sizes which makes it more difficult to quickly meet the condition of 0.5% tolerance in the early stopping criterion. For example, for 1%, the accuracy plateaus by epoch 50 but the early stopping criterion is met only after epoch 350. For nearly 300 epochs, the training is being performed and energy is being consumed without actually gaining too much in accuracy. This could be one avenue where energy savings could be made by defining a slightly looser early stopping criterion.

## 4.1 Facility Location vs. Random Sampling

We observed that FL offers no advantage over random sampling when comparing the mean number of training epochs to convergence. In fact, our results in Fig. 1a indicate that FL selected subsets, on average, actually require more training epochs.

This counter-intuitive finding can be explained by considering the nature of the data points selected. While maximizing similarity aims for highly representative examples (those most central or similar to many others), these often tend to be "easier" to classify. By over-representing these typical examples, the model
<figure><img src="image_5.png" /><figcaption>Fig. 2: Evolution of accuracy over training epochs for different subset sizes. CIFAR-10 dataset.</figcaption></figure>

might quickly grasp general patterns but then struggle to learn the crucial, nu-
anced distinctions needed for full convergence. These "harder" or boundary-case
examples, which are vital for refining decision boundaries, are often less "simi-
lar" to a large number of other points and thus less likely to be selected by our
similarity function. Random sampling, conversely, provides a natural mix that
includes these more challenging points, which can accelerate the overall learning
process.

The variability over experiments in convergence can be attributed to mini-
batching. Even if data subset itself is fixed, the order in which data points are
presented to the model within mini-batches is almost always randomized in each
epoch (shuffling). This means the gradient updates in each step are different,
leading to different trajectories through the loss landscape. However, despite
requiring more epochs to converge, Fig. 1b shows that the similarity-selected
subsets consistently outperformed random sampling in terms of accuracy across
various subset sizes.

It might appear surprising that FL is only marginally better than random
sampling in terms of accuracy upon convergence. We believe that one of the
reasons behind this could be the similarity function for which there is no natural
choice for the datasets under consideration. In our experiments, this function
was taken as an inverse to euclidean distance (higher similarity for closer points)
but this choice is arbitrary. Further, once the datasets have been subdivided
into classes, the data inside each class seems to be sufficiently homogeneous so
that random sampling is not too far off from the FL when we compare the
values of the submodular objective function, *O*, given in (1) evaluated on the
respective subsets. Tab. 1 shows *O(S)/O(F)* averaged over 100 experiments for
the two algorithms, where *F* is full the dataset and *S* is the subset provided
by the algorithm. Although the subset chosen by FL does have a better value
uniformly over subset sizes, the gains start becoming interesting only for very
small subset sizes (~ 10). Note that in this comparison the objective function
is very much favorable to FL sampling technique and it is not straightforward
to predict the gain (or loss) in accuracy that can be expected from a given gain
in the objective value since the choice of the objective function was somewhat arbitrary.

<table><caption>Table 1: Fraction of FL score to the full dataset. CIFAR-10 dataset</caption><thead><tr><th>SubsetSize/Class</th><th>10</th><th>100</th><th>500</th><th>750</th><th>1250</th><th>2500</th></tr></thead><tbody><tr><th>FL</th><td>0.94</td><td>0.95</td><td>0.96</td><td>0.96</td><td>0.97</td><td>0.98</td></tr><tr><th>Random</th><td>0.91</td><td>0.94</td><td>0.95</td><td>0.95</td><td>0.96</td><td>0.97</td></tr></tbody></table>

## 4.2 Pre-processing Energy Overheads

The marginal accuracy advantage comes at an additional cost: the pre-processing energy overheads associated with the subset selection itself. We measured this energy on Grid'5000 using various subset sizes from the CIFAR10 dataset, repeating each experiment 10 times. This selection energy is crucial, as it must be added to the training energy (which is calculated by multiplying the number of epochs by the energy consumed per epoch) to determine the total energy consumption for each complete training run.

Fig. 3 illustrates energy consumption for FL and CRAIG. FL consistently consumed around 5 kJ across different subset sizes. In contrast, CRAIG showed higher energy consumption, ranging from 6.46 kJ to 8.12 kJ for selecting 1% to 75% of the data. In fact, to calculate the gradient using the last layer approximation method, a forward pass is needed. A forward pass is the process of feeding input data through a neural network to produce an output prediction. It involves a series of matrix multiplications and activation function applications across the network's layers, moving from the input layer towards the output layer. Consequently, the energy required to obtain a subset using CRAIG depends on the complexity of the model.

<figure><img src="image_6.png" /><figcaption>Fig. 3: Energy consumption for subset selection using facility location and CRAIG on CIFAR10 datasets. Each graph shows the minimum, mean, and maximum energy consumption for different subset sizes.</figcaption></figure>

## 4.3 Comparing Efficiency

We define efficiency as accuracy divided by the mean total energy consumption. To evaluate the efficiency of different sampling methods, we conducted
experiments on both CIFAR-10 and MNIST datasets, plotting smoothed fitted
functions based on experimental data.

As shown in Figure 4, the efficiency trends vary between datasets. For CIFAR-10 (Figure 4a), random sampling consistently demonstrated higher efficiency than FL across most subset sizes. The only exception is at a 75% subset size, where their efficiencies become comparable. The efficiency for CIFAR-10 typically falls between 0.1 and 0.2. In contrast, for MNIST (Figure 4b), the efficiency

<figure><img src="image_7.png" /><figcaption>Fig. 4: Efficiency curves for different algorithms. CIFAR10 and MNIST datasets.</figcaption></figure>

curves for random sampling and FL are more alike. Random sampling shows
higher efficiency for subset sizes between 7.5% and 20%, but FL generally main-
tains a higher overall efficiency. The efficiency for MNIST ranges between 5 and
30. This significant difference is easily explained by the inherent characteristics
of the MNIST dataset: it requires less data to achieve maximum accuracy and
utilizes a less complex model for training, consequently demanding less energy.
This observation lends credibility to our defined efficiency criteria.

4.4 Impact of stopping criterion

Our analysis of training and test times reveals a crucial insight regarding com-
putational efficiency: the testing phase frequently demands significantly more
time than training, especially when using small data subsets. This phenomenon
arises because evaluation necessitates processing the entire fixed test set of 10,000
samples after each epoch to check the stopping criteria.

To illustrate with a simple calculation: if the 10,000-sample test set represents
20% of the training set size, and we perform a test after each of 100 epochs, this
adds a substantial computational overhead. Considering that training involves
both a forward and a backward pass, while testing primarily requires a forward
pass, evaluating on 20% of the training data effectively adds 100×(0.5×20%) = 10
times the effort of training on the full training set. Consequently, if we use a
training data subset of 10% of the original size, the energy consumed by repeated
testing can equal that of the training itself.

An alternative approach to mitigate this repeated testing after each epoch
is to train for a predetermined number of rounds and evaluate only once at the
end of the training. However, given that we do not know the required number
of training epochs in advance, a practical solution to this issue is to modify the stopping criteria to be based on training accuracy rather than test accuracy, thereby allowing test accuracy to be measured only once at the very end of the training process.

To further investigate this and enable a more effective comparison between CRAIG and FL, we changed the stopping criteria to be applied to the training accuracy. Additionally, we introduced a new stopping criterion: training was terminated if the test accuracy did not improve by at least 1% (a tolerance of 0.01) for 3 consecutive epochs (a patience of 3), or if a maximum limit of 500 epochs was reached. This adjustment aimed to reduce the number of training epochs and thus improve overall efficiency.

Figure 5 compares the efficiency of two CRAIG application strategies against FL on the MNIST dataset, all evaluated under this new criterion. The red curve represents the efficiency when CRAIG is applied only once at the beginning of training. In this scenario, the overhead energy of applying CRAIG is simply added as a fixed cost to the total training energy for each subset size. Conversely, the green curve illustrates the efficiency when CRAIG is applied every five epochs, as suggested in the CRAIG paper [11]. For this approach, the overhead energy of applying CRAIG is dynamically scaled by a coefficient of "total number of epochs/5 + 1" for each subset size. This increased energy overhead results in lower efficiency compared to the single application method.

The results clearly show that FL is considerably more efficient than both CRAIG application methods. Notably, by reducing the number of training epochs, this new criterion significantly enhances FL's overall efficiency, outperforming the results previously observed for MNIST in Figure 4b.

<figure><img src="image_8.png" /><figcaption>Fig. 5: Comparing the efficiency of CRAIG and FL using MNIST dataset</figcaption></figure>

# 5 Conclusions

This study explored the impact of various data summarization techniques—random sampling, Facility Location (FL), and CRAIG—on the energy consumption and accuracy of supervised learning models.

Random sampling consistently demonstrated a strong balance of accuracy and efficiency, particularly when accounting for the pre-processing energy over-
heads associated with the other methods. Although FL can select more repre-
sentative subsets, it occasionally necessitated more training epochs than random
sampling, and its marginal accuracy improvements often did not justify the ad-
ditional computational cost. CRAIG proved to be significantly less efficient due
to its substantially higher pre-processing energy demands.

Our research also highlighted the critical role of early stopping criteria in
optimizing overall efficiency. Adjusting these criteria can substantially reduce
training epochs and mitigate energy waste. Ultimately, while sophisticated sum-
marization methods exist, the simpler approach of random sampling frequently
emerges as a highly competitive and energy-efficient choice for training super-
vised learning models.

**Acknowledgement.** This work was partially supported by the French National Research Agency (ANR) under grant ANR-22-CE23-0024 (project DELIGHT).

References

1. Grid'5000 – large-scale flexible experimental testbed. https://www.grid5000.fr, accessed: June 13, 2025

2. MoJITOs – monitoring java infrastructure for tracing and observability solutions. https://gitlab.irit.fr/sepia-pub/mojitos (2023)

3. Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. ArXiv abs/1906.03671 (2019)

4. Frankle, J., Dziugaite, G., Olah, C., Morcos, A.: Linear mode connectivity and the lottery ticket hypothesis. arXiv preprint arXiv:1905.13600 (2019)

5. Fujishige, S.: Submodular Functions and Optimization. Elsevier (2005)

6. Haddaji, O., Brun, O., Prabhu, B.: Optimal subset selection for energy-effcient supervised learning, submitted to NetGCoop 2025.

7. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems. vol. 28 (2015)

8. IEA: Energy and ai (2025), https://www.iea.org/reports/energy-and-ai

9. Kuehn, A.A., Hamburger, M.J.: A heuristic program for locating warehouses. Management Science 9(4), 643–666 (1963)

10. Minoux, M.: Accelerated greedy algorithms for maximizing submodular set func-
tions. In: Stoer, J. (ed.) Optimization Techniques. pp. 234–243. Springer Berlin
Heidelberg, Berlin, Heidelberg (1978)

11. Mirzasoleiman, B., Bilmes, J., Leskovec, J.: Coresets for data-efficient training of machine learning models (2020), https://arxiv.org/abs/1906.01827

12. Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions—i. Math. Progr. 14(1), 265–294 (1978)

13. Renda, A., Frankle, J., Amodei, D.: Comparing rewinding and fine-tuning in neural network pruning. In: International Conference on Learning Representations (2020)

14. Tschiatschek, S., Iyer, R., Wei, H., Bilmes, J.: Learning mixtures of submodular functions for image collection summarization. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) NIPS 2014. vol. 27 (2014)

15. Wei, K., Iyer, R., Bilmes, J.: Submodularity in data subset selection and active learning. In: Proc. 32nd Int. Conf. on Machine Learning. vol. 37, pp. 1954–1963. PMLR, Lille, France (07–09 Jul 2015)