# Erasure Coding Aware Block Placement for Data-Intensive Applications

Shadi Ibrahim, Jad Darrous

► To cite this version:

Shadi Ibrahim, Jad Darrous. Erasure Coding Aware Block Placement for Data-Intensive Applications. CHEOPS 2025 - 5th Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems, Mar 2025, Rotterdam Netherlands, Netherlands. pp.15-22, 10.1145/3719330.3721229 . hal-05016741

HAL Id: hal-05016741

https://hal.science/hal-05016741v1

Submitted on 5 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Erasure Coding Aware Block Placement for Data-Intensive Applications

Shadi Ibrahim

Inria, Univ. Rennes, CNRS, IRISA

Rennes, France

## Abstract

Erasure Coding (EC) has recently been integrated and deployed in the Hadoop Distributed File System (HDFS) to provide the same fault tolerance guarantees as replication, but with significantly less storage overhead. When EC is used, data reads typically involve only data chunks. In this paper, we study the effect of data chunk distribution on the performance of reads and data-intensive applications, and present the design and evaluation of an erasure coding aware (EC-aware) block placement that balances the distribution of data chunks across nodes. Experimental results show that EC-aware block placement can reduce the execution time of Sort and WordCount applications by up to 25%.

**CCS Concepts:** • Computer systems organization → Redundancy; • Information systems → Distributed storage.

**Keywords:** Erasure coding, Storage systems, Data analytic frameworks, HDFS, Reed-Solomon codes

## ACM Reference Format:

Shadi Ibrahim and Jad Darrous. 2025. Erasure Coding Aware Block Placement for Data-Intensive Applications. In *5th Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (CHEOPS '25), March 30-April 3, 2025, Rotterdam, Netherlands*. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3719330.3721229

## 1 Introduction

Data volume remains a major challenge in data-intensive clusters, as evidenced by the continued focus on approaches such as erasure coding [12, 27, 32], data compression [9, 20, 31], and data deduplication [13, 19, 24] that aim to reduce storage overhead. Erasure Coding (EC) is a redundancy technique that provides the same fault tolerance guarantees as replication, but with significantly less storage overhead [30]. Unlike replication, data is encoded and (sometimes) decoded

<figure><img src="image_2.png" /><figcaption>Figure 1. EC group representation in HDFS under $RS(n, k)$.</figcaption></figure>

as it is added to and read from storage systems, limiting the practical use of EC to cold and archived data [16, 21].

Over the past decade, important advances have been made in accelerating the encoding and decoding of erasure codes, such as the introduction of the Intel ISA-L libraries [6]. This has led to wide adoption of EC in production data-intensive clusters such as those at Google [15], Facebook [25], and CERN [26]. Importantly, EC has also been integrated and deployed in several distributed storage systems, including HDFS [5] and Ceph [4].

The Hadoop Distributed File System (HDFS [29]) is a popular storage system widely used to support (and associated with) big data analytic frameworks, including Apache Hadoop [2], Apache Spark [3], and Apache Flink [1]. HDFS has been used for large-scale data operations for years. For example, at Spotify, Hadoop clusters comprise 1650 nodes and run over 20000 jobs per day [7].

As for now, HDFS provides support for Reed-Solomon (RS) codes - the most popular family of erasure codes [28]. As shown in Figure 1, under $RS(n, k)$, n HDFS blocks (default size is 256 MB) are assigned to an *EC group*. The blocks are then encoded sequentially. Specifically, each block is divided into cells (1 MB by default). The client encodes each *n* cell to create *k* parity cells (the collection of $n+k$ cells is called a *strip*) and then sends the $n+k$ cells to $n+k$ DataNodes (DNs). This process is repeated for the remaining cells in the block and for the remaining blocks in the *EC group*. As a result, blocks in an EC group are stored on $n+k$ DNs, with data

<figure><img src="image_3.png" /></figure>

<footnote>*Work done while at Inria, IMT Atlantique, LS2N</footnote>
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Data distribution across DNs: “Total chunks” accounts for all chunks in the cluster, while “Data chunks” accounts for data chunks only. We use *RS*(4, 3) and sequentially add 167 files to an HDFS cluster with 20 DNs. Each file is 1.5 GB in size (one EC group). In total, we have added a dataset of 250.5 GB (i.e., 1002 original blocks of 256 MB each). We use the same experimental setup as described in Section 3.</figcaption></figure>

chunks placed on *n* DNs and parity chunks stored in *k* DNs; and each HDFS block (referred to as the original block in Figure 1) is physically mapped to *n* data chunks and *k* parity chunks distributed across different *n* + *k* DNs.

HDFS uses a random block placement to achieve best-effort balanced data distribution across DNs at scale. It balances the total number of chunks, but not the number of data chunks across DNs because it does not distinguish between data chunks and parity chunks. This results in an unbalanced distribution of data chunks. Figure 2 shows the coefficient of variation of the chunk distribution across DNs for increasing dataset size. We can clearly see that the data chunk distribution has a higher variation than the total chunk distribution (data and parity chunks). The difference in the number of data chunks between the DNs is significant: the node with the minimum number of data chunks hosts 204 chunks, while other nodes host up to 414 data chunks.

Usually, data processing applications read only data chunks (in a failure-free mode). Therefore, DNs with a high proportion of data chunks receive more read requests and experience I/O and network contention. As a result, map tasks running on and reading data from these nodes experience delays, as discussed in [12]. To further illustrate how the placement of chunks affects job execution time, consider the

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Default chunk distribution in HDFS. A possible chunk distribution for 3 blocks on a 6-node cluster under EC with *RS*(4, 2) scheme. *D*ᵢ and *P*ᵢ denote a data chunk and a parity chunk of block *i*, respectively.</figcaption></figure>

following example. Suppose we have a cluster of 6 nodes hosting 3 HDFS blocks with the *RS*(4, 2) schema. Figure 3 shows a possible distribution of the chunks of these blocks. *D*ᵢ denotes a data chunk of block *i* while *P*ᵢ denotes a parity chunk. Although all nodes have the same number of chunks (i.e., 3), they do not have the same number of data chunks. More specifically, node 1 and node 2 host 3 data chunks while node 5 and node 6 host only one data chunk. To process the 3 blocks, a MapReduce job with 3 map tasks should be launched. Regardless of where the map tasks are scheduled, all data chunks will eventually be read by the DNs. As in the current version of Hadoop, tasks (i.e., clients) always choose the nodes that host the data chunks to read the data block. Thus, the load on each DN depends on the number of data chunks that reside on that node when the data chunks are available (in a failure-free mode). If the map tasks are scheduled on nodes 1, 4, and 6. The first map task reads the first block from nodes 1, 2, 3, and 4. The second map task reads the second block from nodes 1, 2, 3, and 5. And finally, the third map task reads the third block from nodes 1, 2, 4, and 6. As a result, the first node becomes a straggler because it is serving all tasks simultaneously (3 data blocks). As a result, the map task running on this node may experience a delay in its runtime due to the long I/O wait time. In addition, the tasks served by this node are also delayed due to the high I/O contention in Node 1.

**Contributions.** Since the data load of a DN is attributed to the amount of data chunks that reside in that node, in this paper we develop an EC-aware block placement that aims to balance the data chunk distribution among DNs. This in turn reduces job execution times by reducing the runtimes of the longest tasks and the variability in task runtimes. We implement our algorithm in HDFS and show empirically that the performance of MapReduce jobs can be improved under EC-aware placement. Experimental results on the Grid’5000 testbed [8] show that:

* EC-aware placement improves concurrent read performance – in terms of throughput – by up to 34%. It has no effect on write performance.

* EC-aware placement reduces the variation in data reads across DNs by up to 78% and improves the performance of MapReduce applications by up to 25%.

* The variation in map runtimes under EC-aware still exists due to temporal load imbalance (i.e., some nodes are temporarily overloaded due to the order of tasks). This motivates the need to introduce EC awareness at the scheduling level to deal with the dynamics of the environment.

The rest of this paper is organized as follows. The EC-aware block placement algorithm is presented in Section 2. The experimental methodology is discussed in Section 3, followed by the experimental evaluation in Section 4. A discussion of the integration of EC awareness at the scheduler
level is presented in Section 5. Section 6 discusses the related work and Section 7 concludes this study.

## 2 EC-Aware Placement Algorithm

In an attempt to reduce the data load imbalance under EC, we propose a greedy algorithm to distribute data chunks evenly across the DNs in a round-robin manner.

A placement request is issued for each EC group. Under $RS(n, k)$, an EC group is usually a collection of $n$ (sequential) blocks belonging to the same file. However, depending on the file size, it may contain less than $n$ blocks. In all cases, $n + k$ nodes should be returned to host the data and parity chunks of these blocks.

The algorithm keeps track of the number of data and parity chunks hosted by each DN. Initially, they are all set to zero. The algorithm creates an array equal in size to the number of DNs in the cluster and sets the number of data and parity chunks to zero.

The algorithm 1 shows how a write request to HDFS is handled. The algorithm takes as input the previously created and initialized array *datanodes*, the EC scheme used (*ec_schema*), and the number of blocks in the EC group (*nb_blocks*). As output, the algorithm returns the nodes that should host the data chunks (*data_nodes*) and those for the parity chunks (*parity_nodes*). For each request, the DNs are sorted in ascending order by the number of data chunks they host, i.e., nodes with fewer data chunks are placed first. The first *n* DNs are chosen to host the new *n* \* *nb_blocks* data chunks. After that, the remaining DNs are sorted again in ascending order by their total number of chunks (data + parity), and the first *k* DNs are chosen to host the *k* \* *nb_blocks* parity chunks. This second step does not improve the distribution of the data chunks, it only improves the balance of the overall data distribution among the DNs.

The time complexity of the algorithm is dominated by the time complexity of the two sort functions. Therefore, it is $O(|DN| * log_2(|DN|))$, where $|DN|$ is the number of DNs in the cluster. However, by using a priority queue to maintain the order of the DNs, the sorting can be avoided each time the algorithm runs, and its time complexity can be reduced to $O(log_2(|DN|))$, which is the time complexity of updating an element in a priority queue. Applying this algorithm to the example in Section 1 results in 18 nodes with 300 data chunks and 2 nodes with 306 data chunks.

Note that the current algorithm only handles adding new blocks to a single rack in HDFS. Rack awareness, data deletion, data loss, data migration, and node decommission are not considered in the current version. However, these features do not affect our results and can be easily adopted.

**Implementation** We implemented our algorithm in HDFS as a subclass of BlockPlacementPolicyDefault in the `org.apache hadoop.hdfs.server.blockmanagement` package. It is composed of less than 200 LOC in Java. The algorithm

**Algorithm 1:** EC-aware block placement

Input : datanodes, ec_schema, nb_blocks

Output : data_nodes, parity_nodes

1. Sort *datanodes* by increasing number of hosted data chunks;

2. $data_nodes \leftarrow datanodes[0..ec_schema.n]$;

3. **foreach** $dn \in datanodes[0..ec_schema.n]$ **do**

4. &nbsp;&nbsp;&nbsp;&nbsp; $dn.nr_data_chunk$ ← $dn.nr_data_chunk + nb_blocks$;

5. **end**

6. Sort *datanodes* by increasing number of total hosted chunks;

7. $parity_nodes \leftarrow datanodes[0..ec_schema.k]$;

8. **foreach** $dn \in datanodes[0..ec_schema.k]$ **do**

9. &nbsp;&nbsp;&nbsp;&nbsp; $dn.nr_parity_chunk$ ← $dn.nr_parity_chunk + nb_blocks$;

10. **end**

11. **return** $data_nodes, parity_nodes$;

takes a few milliseconds to run on medium-sized clusters (fewer than 1000 nodes).

## 3 Methodology Overview

We conduct our experiments on Hadoop MapReduce. However, we believe that our findings are valid for other data analytic frameworks (e.g., Spark [3], Flink [1], etc.) when running on HDFS.

**Testbed.** The experiments are carried out on the French scientific testbed Grid’5000 [8] at the Nantes site. Two clusters – Econome and Ecotype – are used for the experiments, with 22 and 48 machines respectively. The machines in the Econome cluster are equipped with two Intel Xeon E5-2660 8-core processors, 64 GB of main memory, and a 7.2k RPM 1 TB hard disk drive (HDD). The Ecotype cluster machines have the same CPU and memory but are equipped with a 350 GB SSD. The machines in each cluster are connected by a 10 Gbps Ethernet network, and the two clusters are connected by four 40 Gbps links. The machines run 64-bit Debian Stretch Linux with Java 8 and Hadoop 3.0.0 installed. All experiments are performed in isolation on the testbed. There is no interference from other users.

**Hadoop configuration.** Hadoop is deployed on Econome cluster: One node runs the NameNode and ResourceManager, and the remaining 20 nodes serve as workers, that is, DataNodes (DNs) and NodeManagers (NMs). The Ecotype cluster hosts the clients. To eliminate the impact of local disks on the client side, we store the dataset in main memory.

The HDFS block size is set to 256 MB. Unless otherwise specified, we use the default EC policy in HDFS, i.e., $RS(6, 3)$ schema with a cell size of 1 MB. We configure Hadoop (YARN) to run 8 containers per node (i.e., one per CPU core).
<figure><img src="image_6.png" /><figcaption>**Figure 4.** Concurrent data access with 10 clients under the default placement (EC) and the EC-aware block placement.</figcaption></figure>

This gives us 160 slots in our cluster, which is enough to process 40 GB of data in a single map wave. Map tasks of MapReduce jobs may run in multiple waves, depending on the size of the file and the available slots in the cluster.

**Benchmarks.** We use read and write operations to measure the performance of HDFS. These operations are issued using the HDFS commands hadoop fs -get and hadoop fs -put, respectively, from the clients (which are located in the Ecotype cluster). The test files are generated using the dd command from /dev/urandom as the input source.

Two micro-benchmarks (i.e., WordCount and Sort) are also used to evaluate the performance of MapReduce. Both WordCount and Sort are available as part of the Hadoop distribution. To generate input data for the WordCount and Sort applications, we use a modified version of Hadoop's random text writer and random writer. The modification was to hardcode the seed for the random generator to produce exactly the same dataset. Each MapReduce job runs alone, so it has all the resources of the cluster during execution.

We run each experiment 10 times and compute the average and standard deviation. We empty the caches between data generation and data processing and between runs.

**Metrics.** The main metric used to measure the performance of reads and writes in HDFS is the client side throughput. For a single client, it is the amount of data read/written per second. In the case of concurrent reads and writes (by multiple clients), it is the average throughput per client, along with the standard deviation of the throughput of the clients.

We use job execution time as the main metric for MapReduce applications. Job execution time is the total time of the MapReduce job from its start time to its finish time (excluding the time spent waiting in the queue). We also use the coefficient of variation metric to measure the variation in task runtimes (map and reduce).

# 4 Evaluation

In the following experiments, we study how EC-aware block placement affects the performance of data access and MapReduce applications.

## 4.1 HDFS data access

To understand the impact of EC-aware block placement, we first evaluate its performance for reads and writes in HDFS. For single reads and writes, there is no difference because the same number of machines are active in both cases. Therefore, we will focus on concurrent access.

Figure 4a shows the write throughput of 10 concurrent clients with increasing file size. We can see similar performance with a slight improvement under EC-aware placement. For instance, 271.3 MB/s write throughput is achieved under EC-aware, while 261.1 MB/s is achieved under the default block placement (labeled EC) when writing 20 GB files. The effect of EC-aware placement on write throughput is minimal because the written data is first buffered in DNs main memories. This hides the effect of the disks and thus the load imbalance.

Figure 4b depicts the read throughput of 10 concurrent clients with increasing file size while reading different files. When reading 10 GB files, the average throughput per client increases from 120.6 MB/s under default EC to 161.9 MB/s under EC-aware placement, an improvement of 34%. We also observe less variation between clients with EC-aware placement (4.2% compared to 6.7%). This is mainly due to the balanced distribution of data chunks across the DNs.

Figure 4c shows the throughput when reading the same file. The data is read once from disk and then served from memory. This explains the small difference in throughput between EC-aware and default EC, which could be attributed to memory contention. While the impact of EC-aware placement is more pronounced when reading from disk, EC-aware placement can still provide performance benefits when data is in memory.

*In summary.* The EC-aware block placement aims to improve read performance by balancing the load across datanodes. Since the blocks of a requested file are read sequentially, the performance benefit can only be observed for concurrent reads. We also observe a slight improvement in write throughput.
<figure><img src="image_7.png" /><figcaption>**Figure 5.** Job execution time of Sort and Wordcount applications under EC and EC-aware placements.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 7.** Tasks of Sort application under default EC and EC-aware placements for 40 GB input size.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 6.** Disk read bytes during the map phase of Sort application under default EC and EC-aware placements for 40 GB input size.</figcaption></figure>

## 4.2 MapReduce applications

Since our focus is on reading the input data under EC (i.e., the map phase), we run the map and reduce phases sequentially with non-overlapping shuffle. When presenting results related to a particular run, we take the job that has the median execution time.

**Sort application.** Figure 5a shows the job execution time under the default EC placement and the EC-aware block placement for Sort application. Up to 13% reduction in job execution time can be observed under EC-aware placement. For instance, for an input size of 40 GB, the job execution time under default EC is 91.33s, while under EC-aware placement it is 82.02s on average, which is 10.4% faster.

The main goal of EC-aware block placement is to balance data reads across DNs. In Figure 6, EC-aware block placement reduces the variation in data reads from 36% under default EC to 8% (a 78% improvement). This in turn reduces the impact of stragglers (i.e., nodes serving more tasks and thus experiencing high disk and network contention) and also lowers the variation in map task runtimes. As shown in Figure 7, the average and maximum runtimes of map tasks are indeed reduced from 38.2s to 29.9s (by 22%) and from 56.3s to 42.8s (by 24%), respectively. This explains the 10.4%

performance improvement for the Sort application under EC-aware placement compared to the default EC placement. In addition, we can see that the variation in map task runtimes is slightly reduced from 27.5% to 24%.

In Figure 7, we can also see that the duration of the reduce phase (i.e., the shuffle, sort, and reduce stages) is almost the same for both default EC and EC-aware placements. For example, the average runtime of the reduce stage is 11.3s under the default placement, while it is 11.5s under the EC-aware placement. This also confirms that the EC-aware placement does not adversely affect the performance of reduce tasks (writing output to HDFS).
<figure><img src="image_10.png" /><figcaption>**Figure 8.** Tasks of WordCount application under default EC and EC-aware placements for 40 GB input size.</figcaption></figure>

**WordCount application.** Figure 5b shows the job execution time under the default EC placement and the EC-aware block placement for WordCount application. We can see that under EC-aware, jobs finish faster than under the default EC placement. For instance, up to 25% reduction in job execution time can be observed under EC-aware placement for 40 GB input size. The improvement is more pronounced for the WordCount application than for the Sort application. This is because the map phase accounts for 90% of the execution time for the WordCount application, while it accounts for 65-69% for the Sort application, and both placements perform the same for the reduce phase.

Figure 8 shows that EC-aware can significantly reduce the impact of stragglers. The slowest map task took 75.97s to complete, while under the default EC placement, the slowest 5% of map tasks have a runtime of more than 77.9s and up to 95.31s. These slow map tasks – under the default EC placement – read at least one of their data chunks from one of the two most overloaded DNs (those serving more than 3 GB). Finally, we also see that the variation in map task runtimes decreases significantly, from 16% to 10.3%.

*In summary.* EC-aware block placement balances data reads across DNs and can reduce the average and maximum runtimes of map tasks. This in turn improves the performance of MapReduce applications under EC-aware block placement. This improvement is attributed to the map phase and therefore depends on the relative duration of the map phase.

We also observe that there are still some long-running tasks. We believe this is due to the order in which data is read under EC (order of tasks) and the disk seek time. As a perspective, we plan to investigate some possible solutions such as data prefetching to reduce the impact of random data read from disk.

# 5 Discussion - Towards Dynamic EC-awareness

Data read imbalance between DNs can significantly affect the performance of MapReduce jobs under EC. There can be two types of data read imbalances: *spatial* and *temporal*. The *spatial* load imbalance is represented by the total amount of

data read by each DN during the map phase. While the *temporal* load imbalance results from the fact that some nodes are temporarily overloaded from time to time due to the order of the tasks. The EC-aware block placement can ensure that all DNs contribute equally to the total amount of data read during the map phase of an application, thus effectively addressing the problem of spatial load imbalance. However, it cannot address the temporal behavior because this issue is related to the task order and job progress. Therefore, it is necessary to design an online scheduling policy that can decide the set of map tasks that can be run concurrently in each wave or by each application (in the case of multiple applications) to balance the data load across nodes at runtime.

# 6 Related Work

There is a long line of research on deploying, evaluating, and optimizing EC in distributed storage systems and to support data-intensive applications [10–12, 17, 18, 23, 27, 28, 33].

**EC is distributed storage systems.** EAR [22] and DiskReduce [14] discuss how to implement and accelerate data encoding when transitioning from replication to EC in HDFS. Darrous et al. [11] quantify the performance of reads and writes in erasure-coded HDFS. A recent study [17] evaluates recovery time and storage overhead for small files in erasure-coded Ceph.

**EC to support data-intensive applications.** Several studies explore the benefits of EC in data-intensive clusters to support MapReduce and machine learning applications [12, 23, 33]. The study [33] evaluates the performance of MapReduce applications when intermediate and output data are stored in erasure-coded HDFS. Runhui et al. [23] discuss how to reduce the impact of degraded reads (reconstructing unavailable blocks for map tasks) on the execution time of MapReduce applications.

EC with a contiguous layout introduces high memory and network overhead during failures, so HDFS has recently moved to EC with a striped data layout. A recent study [12] shows the benefit of using EC with striped data layout on the performance of Sort, WordCount, and Kmeans applications under failures and in a failure-free environment. They
also show the impact of disk and network contention and I/O stragglers on the performance of map tasks. Our work complements [12] as it shows the effect of data chunk distribution on the performance of MapReduce applications and demonstrates, for the first time, the potential of making data placement EC aware.

## 7 Conclusion

Erasure codes are increasingly used in many storage systems as a cost-effective alternative to replication. In this paper, we have shown that the unawareness of EC in the current block placement in HDFS leads to data read imbalance, which affects the performance of MapReduce applications. In response, we propose a greedy algorithm that balances the distribution of data chunks across DNs. This in turn reduces the execution time of MapReduce jobs. For example, under EC-aware block placement, Sort applications achieve up to 13% reduction in job execution time, while WordCount applications achieve up to 25% reduction in job execution time. Motivated by our results, we plan to implement EC awareness at the scheduling level.

## Acknowledgments

This work was supported in part by the "Alvearium" Inria and hive partnership, the ANR KerStream project (ANR-16-CE25-0014-01), the Stack/Apollo connect talent project, and the Inria Project Lab program Discovery (see https://beyondtheclouds.github.io). Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, and several Universities as well as other organizations (see http://www.grid5000.fr/). Jad Darrous is currently senior software engineer at KomodoHealth.

## References

[1] 2025. Apache Flink. https://flink.apache.org.

[2] 2025. Apache Hadoop. http://hadoop.apache.org.

[3] 2025. Apache Spark. https://spark.apache.org.

[4] 2025. Ceph, Erasure-code. https://docs.ceph.com/en/reef/rados/operations/erasure-code/.

[5] 2025. HDFS Erasure Coding. https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.

[6] 2025. Intel Intelligent Storage Acceleration Library Homepage. https://software.intel.com/en-us/storage/ISA-L.

[7] 2025. Powered by Apache Hadoop. https://cwiki.apache.org/confluence/display/HADOOP2/PoweredBy#PoweredBy-S.

[8] Daniel Balouek, Alexandra Carpen Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas Niclausse, Lucas Nussbaum, Olivier Richard, Christian Perez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. 2013. Adding Virtualization Capabilities to the Grid'5000 Testbed. In *Cloud Computing and Services Science*, Ivan I. Ivanov, Marten van Sinderen, Frank Leymann, and Tony Shan (Eds.). Springer International Publishing, Cham, 3-20. https://doi.org/10.1007/978-3-319-04519-1_1

[9] Xiang Chen, Tao Lu, Jiapin Wang, Yu Zhong, Guangchun Xie, Xueming Cao, Yuanpeng Ma, Bing Si, Feng Ding, Ying Yang, Yunxin Huang, Yafei Yang, You Zhou, and Fei Wu. 2024. HA-CSD: Host and SSD Coordinated Compression for Capacity and Performance. In *2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)*. 825-838. https://doi.org/10.1109/IPDPS57955.2024.00078

[10] Jad Darrous and Shadi Ibrahim. 2019. [Poster Presentation] Enabling Data Processing under Erasure Coding in the Fog. In *ICPP 2019 : 48th International Conference on Parallel Processing*. 1. https://inria.hal.science/hal-02388835/document

[11] Jad Darrous and Shadi Ibrahim. 2022. Understanding the performance of erasure codes in hadoop distributed file system. In *Proceedings of the Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems* (Rennes, France) (CHEOPS '22). Association for Computing Machinery, New York, NY, USA, 24-32. https://doi.org/10.1145/3503646.3524296

[12] Jad Darrous, Shadi Ibrahim, and Christian Perez. 2019. Is it Time to Revisit Erasure Coding in Data-Intensive Clusters?. In *2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)*. 165-178. https://doi.org/10.1109/MASCOTS.2019.00026

[13] Jad Darrous, Shadi Ibrahim, Amelie Chi Zhou, and Christian Perez. 2018. Nitro: Network-Aware Virtual Machine Image Management in Geo-Distributed Clouds. In *2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)*. 553-562. https://doi.org/10.1109/CCGRID.2018.00082

[14] Bin Fan, Wittawat Tantisiriroj, Lin Xiao, and Garth Gibson. 2009. DiskReduce: RAID for data-intensive scalable computing. In *Proceedings of the 4th Annual Workshop on Petascale Data Storage* (Portland, Oregon) (PDSW '09). Association for Computing Machinery, New York, NY, USA, 6-10. https://doi.org/10.1145/1713072.1713075

[15] Daniel Ford, François Labelle, Florentina I. Popovici, Murray Stokely, Van-Anh Truong, Luiz Barroso, Carrie Grimes, and Sean Quinlan. 2010. Availability in globally distributed storage systems. In *Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation* (Vancouver, BC, Canada) (OSDI'10). USENIX Association, USA, 61-74. https://www.usenix.org/conference/osdi10/availability-globally-distributed-storage-systems

[16] Andreas Haeberlen and Alan Mislove. 2005. Glacier: Highly Durable, Decentralized Storage Despite Massive Correlated Failures. In *2nd Symposium on Networked Systems Design & Implementation (NSDI 05)*. USENIX Association, Boston, MA. https://www.usenix.org/conference/nsdi-05/glacier-highly-durable-decentralized-storage-despite-massive-correlated-failures

[17] Runzhou Han, Chao Shi, Tabassum Mahmud, Zeren Yang, Vladislav Esaulov, Lipeng Wan, Yong Chen, Jim Wayda, Matthew Wolf, and Mai Zheng. 2024. Revisiting Erasure Codes: A Configuration Perspective. In *Proceedings of the 16th ACM Workshop on Hot Topics in Storage and File Systems* (Santa Clara, CA, USA) (HotStorage '24). Association for Computing Machinery, New York, NY, USA, 93-100. https://doi.org/10.1145/3655038.3665951

[18] Cheng Huang, Huseyin Simitci, Yikang Xu, Aaron Ogus, Brad Calder, Parikshit Gopalan, Jin Li, and Sergey Yekhanin. 2012. Erasure coding in windows azure storage. In *Proceedings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12)*. USENIX Association, USA, 2. https://www.usenix.org/conference/atc12/technical-sessions/presentation/huang

[19] Hongming Huang, Peng Wang, Qiang Su, Hong Xu, Chun Jason Xue, and André Brinkmann. 2024. Palantir: Hierarchical Similarity Detection for Post-Deduplication Delta Compression. In *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2* (La Jolla, CA, USA) (ASPLOS '24). Association for Computing Machinery, New York, NY, USA, 830-845. https://doi.org/10.1145/3620665.3640353

[20] NetApp Inc. 2025. *Use Deduplication, Data Compression, and Data Compaction to Increase Storage Efficiency*. Technical Report. NetApp Inc. https://docs.netapp.com/us-
en/ontap/pdfs/sidebar/Use_deduplication_data_compression_and_data_comparation_to_increase_storage_efficiency.pdf

[21] John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski, Patrick Eaton, Dennis Geels, Ramakrishan Gummadi, Sean Rhea, Hakim Weatherspoon, Westley Weimer, Chris Wells, and Ben Zhao. 2000. OceanStore: an architecture for global-scale persistent storage. SIGPLAN Not. 35, 11 (Nov. 2000), 190–201. https://doi.org/10.1145/356989.357007

[22] Runhui Li, Yuchong Hu, and Patrick P. C. Lee. 2017. Enabling Efficient and Reliable Transition from Replication to Erasure Coding for Clustered File Systems. IEEE Transactions on Parallel and Distributed Systems 28, 9 (2017), 2500–2513. https://doi.org/10.1109/TPDS.2017.2678505

[23] Runhui Li, Patrick P.C. Lee, and Yuchong Hu. 2014. Degraded-First Scheduling for MapReduce in Erasure-Coded Storage Clusters. In 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 419–430. https://doi.org/10.1109/DSN.2014.47

[24] Maohua Lu, David Chambliss, Joseph Glider, and Cornel Constantinescu. 2012. Insights for data reduction in primary storage: a practical analysis. In Proceedings of the 5th Annual International Systems and Storage Conference (Haifa, Israel) (SYSTOR '12). Association for Computing Machinery, New York, NY, USA, Article 17, 7 pages. https://doi.org/10.1145/2367589.2367606

[25] Subramanian Muralidhar, Wyatt Lloyd, Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, Viswanath Sivakumar, Linpeng Tang, and Sanjeev Kumar. 2014. f4: Facebook's warm BLOB storage system. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (Broomfield, CO) (OSDI'14). USENIX Association, USA, 383–398. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/muralidhar

[26] Peters, Andreas-Joachim, Simon, Michal Kamil, and Sindrilaru, Elvin Alin. 2020. Erasure Coding for production in the EOS Open Storage system. EPJ Web Conf. 245 (2020), 04008. https://doi.org/10.1051/epjconf/202024504008

[27] K. V. Rashmi, Mosharaf Chowdhury, Jack Kosaian, Ion Stoica, and Kannan Ramchandran. 2016. EC-cache: load-balanced, low-latency cluster caching with online erasure coding. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (Savannah, GA, USA) (OSDI'16). USENIX Association, USA, 401–417. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/rashmi

[28] Zhirong Shen, Yuhui Cai, Keyun Cheng, Patrick P. C. Lee, Xiaolu Li, Yuchong Hu, and Jiwu Shu. 2025. A Survey of the Past, Present, and Future of Erasure Coding for Storage Systems. ACM Trans. Storage 21, 1, Article 4 (Jan. 2025), 39 pages. https://doi.org/10.1145/3708994

[29] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. 2010. The Hadoop Distributed File System. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST). 1–10. https://doi.org/10.1109/MSST.2010.5496972

[30] Hakim Weatherspoon and John D. Kubiatowicz. 2002. Erasure Coding Vs. Replication: A Quantitative Comparison. In Peer-to-Peer Systems, Peter Druschel, Frans Kaashoek, and Antony Rowstron (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 328–337. https://doi.org/10.1007/3-540-45748-8_31

[31] Leon Windheuser, Christoph Anneser, Huanchen Zhang, Thomas Neumann, and Alfons Kemper. 2024. Adaptive Compression for Databases. In Proceedings 27th International Conference on Extending Database Technology, EDBT 2024, Paestum, Italy, March 25 - March 28. OpenProceedings.org, 143–149. https://doi.org/10.48786/edbt.2024.13

[32] Qiaori Yao, Yuchong Hu, Xinyuan Tu, Patrick P. C. Lee, Dan Feng, Xia Zhu, Xiaoyang Zhang, Zhen Yao, and Wenjia Wei. 2022. PivotRepair: Fast Pipelined Repair for Erasure-Coded Hot Storage. In 2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS). 614–624. https://doi.org/10.1109/ICDCS54860.2022.00065

[33] Zhe Zhang, Amey Deshpande, Xiaosong Ma, Eno Thereska, and Dushyanth Narayanan. 2010. Does erasure coding have a role to play in my data center? Technical Report MSR-TR-2010-52. https://www.microsoft.com/en-us/research/publication/does-erasure-coding-have-a-role-to-play-in-my-data-center/