Hadoop(-3.2.1.)上でspark-3.0.1を使ってpi計算をしてみる

■ こちらの環境
OS: Ubuntu 16 or 18
Hadoop: hadoop-3.2.1.tar.gz
JDK (Java): jdk-8u202-linux-x64.tar.gz
Spark: spark-3.0.1-bin-hadoop3.2.tgz

ネームノード
192.168.76.216: h-gpu05

データノード
192.168.76.210: h-gpu03
192.168.76.210: h-gpu04

おそらく、Hadoopでpi計算で出来ていればなんとかなる。

spark-3.0.1-bin-hadoop3.2.tar.gzをインストールし、.bashrcに下記を追加。

<pre>
export SPARK_HOME=/home/hadoop/spark-3.0.1-bin-hadoop3.2/
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
</pre>

バージョンを確認。

<pre>
hadoop@h-gpu05:~/spark-3.0.1-bin-hadoop3.2$ bin/spark-shell                                                                                                                                               
2020-12-04 23:27:48,924 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://h-gpu05:4040
Spark context available as 'sc' (master = local[*], app id = local-1607092074373).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/
         
Using Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_202)
Type in expressions to have them evaluated.
Type :help for more information.
</pre>

conf/spark-defaults.confを変更
やってる間に、いろいろエラーが出たので　(*´Д｀)、最小限(２行)の修正にした。

<pre>
hadoop@h-gpu05:~/spark-3.0.1-bin-hadoop3.2$ cat conf/spark-defaults.conf                                                                                                                                  

spark.eventLog.enabled           true
spark.eventLog.dir file:///tmp/spark-events/
</pre>

一応、history-serverなるものを起動しておく。

<pre>
$./sbin/start-history-server.sh   
</pre>

あとは、ネームノード(h-gpu05)からこのコマンド↓を叩いてみる。。。

<pre>
hadoop@h-gpu05:~/spark-3.0.1-bin-hadoop3.2$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --queue default examples/jars/spark-examples*.jar 1000 
</pre>

一応実行されたようです。

<pre>
2020-12-04 23:31:34,551 INFO scheduler.TaskSetManager: Finished task 998.0 in stage 0.0 (TID 998) in 10 ms on h-gpu05 (executor 1) (999/1000)
2020-12-04 23:31:34,553 INFO scheduler.TaskSetManager: Finished task 999.0 in stage 0.0 (TID 999) in 7 ms on h-gpu05 (executor 2) (1000/1000)
2020-12-04 23:31:34,553 INFO cluster.YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
2020-12-04 23:31:34,554 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 5.400 s
2020-12-04 23:31:34,557 INFO scheduler.DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
2020-12-04 23:31:34,557 INFO cluster.YarnScheduler: Killing all running tasks in stage 0: Stage finished
2020-12-04 23:31:34,559 INFO scheduler.DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 5.452502 s
Pi is roughly 3.1413773914137737
</pre>

実は、--deploy-mode clusterとすると、"Pi is roughly 3.1413773914137737"が表示されなくなったりするのだが、(*´Д｀)
今日はここまで。