Comparison of Big Data storage layers: Delta vs Apache Hudi vs Apache Iceberg. Part#1
By: Domas Jautakis
Tags: Apache Hudi, Delta Lake, Apache Iceberg, Apache Spark

Preparation of single node cluster

Versions:

ubuntu-18.04.3-live-server-amd64

openjdk-8-jdk

scala-2.11.12

spark-2.4.4-bin-hadoop2.7

hadoop-2.7.7

apache-hive-2.3.6-bin

presto-server-329.tar

org.apache.iceberg:iceberg-spark-runtime:0.7.0-incubating

org.apache.hudi:hudi-spark-bundle:0.5.0-incubating

io.delta:delta-core_2.11:0.5.0

--

spark-3.0.0-preview2-bin-hadoop2.7 - let's check impact of spark version updagrade :)

In Ubuntu i’m using user spuser. For this user I will generate authorization keys which will be needed for hadoop.

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

The first step is install java 1.8 for Spark:

#1. sudo add-apt-repository ppa:openjdk-r/ppa

sudo apt-get update

sudo apt-get install openjdk-8-jdk

sudo update-alternatives --config java

sudo update-alternatives --config javac

Make sure that your java version is 1.8

#2. spuser@acid:~$ java -version

openjdk version "1.8.0_232"

OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~16.04.1-b09)

OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

Download all required packages

#3.

cd downloads/

wget

wget

wget

wget

wget

wget

wget mkdir downloadscd downloads/wget https://downloads.lightbend.com/scala/2.11.12/scala-2.11.12.deb wget http://apache.mirror.vu.lt/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz wget http://apache.mirror.vu.lt/apache/spark/spark-3.0.0-preview2/spark-3.0.0-preview2-bin-hadoop2.7.tgz wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.7/hadoop-2.7.7.tar.gz wget http://apache.mirror.vu.lt/apache/hive/hive-2.3.6/apache-hive-2.3.6-bin.tar.gz wget https://repo1.maven.org/maven2/io/prestosql/presto-cli/329/presto-cli-329-executable.jar wget https://repo1.maven.org/maven2/io/prestosql/presto-server/329/presto-server-329.tar.gz

Let’s check sizes what we downloaded:

#4. spuser@acid:~/downloads$ ll -h

Install scala

#5. sudo dpkg -i scala-2.11.12.deb

I will unarchive all packages under directory /usr/local/. For particular versions symlinks will be created, for easier migration in the future:

#6. sudo tar -xzf apache-hive-2.3.6-bin.tar.gz -C /usr/local/

sudo tar -xzf hadoop-2.7.7.tar.gz -C /usr/local/

sudo tar -xzf spark-2.4.4-bin-hadoop2.7.tgz -C /usr/local/

sudo tar -xzf spark-3.0.0-preview2-bin-hadoop2.7.tgz -C /usr/local/

sudo tar -xzf presto-server-329.tar.gz -C /usr/local sudo chown -R spuser /usr/local/apache-hive-2.3.6-bin/

sudo chown -R spuser /usr/local/hadoop-2.7.7/

sudo chown -R spuser /usr/local/spark-2.4.4-bin-hadoop2.7/

sudo chown -R spuser /usr/local/spark-3.0.0-preview2-bin-hadoop2.7/

sudo chown -R spuser /usr/local/presto-server-329/ cd /usr/local/

sudo ln -s /usr/local/apache-hive-2.3.6-bin/ /usr/local/hive

sudo chown -h spuser:spuser /usr/local/hive

sudo ln -s /usr/local/hadoop-2.7.7/ /usr/local/hadoop

sudo chown -h spuser:spuser /usr/local/hadoop

sudo ln -s /usr/local/spark-2.4.4-bin-hadoop2.7 /usr/local/spark

sudo chown -h spuser:spuser /usr/local/spark

sudo ln -s /usr/local/spark-3.0.0-preview2-bin-hadoop2.7 /usr/local/spark3

sudo chown -h spuser:spuser /usr/local/spark3

sudo ln -s /usr/local/presto-server-329 /usr/local/presto

sudo chown -h spuser:spuser /usr/local/presto



I will create several folders for logs and hdfs storage. To create some folders under the root is not the best practice, but anyway it’s sandbox:

#7. sudo mkdir /logs

sudo chown -R spuser /logs

mkdir /logs/hadoop #Add dir for data

sudo mkdir /hadoop

sudo chown -R spuser /hadoop

mkdir -p /hadoop/hdfs/namenode

mkdir -p /hadoop/hdfs/datanode #create tmp hadoop dir:

mkdir -p /tmp/hadoop

Now let’s update env. variables — .bashrc: