ログ解析者が久しぶりにHadoopで円周率を計算しようとした結果。。。（スタンドアローンモード編）

* 久しぶり（４年ぶりくらい）にHadoopで円周率を計算しようとしてみた。 *

背景：
1. Sparkを動そうと思ったのだが、HDFSとYARNと連携するらしい
2. ある意味、HadoopのMapReduceを把握していないと、Sparkがどう凄いのか、なぜ凄いのかわからないのでは？

まずは、スタンドアローンモードで動かしてみる。。

こちらの環境
1. Hadoop: hadoop-3.2.1.tar.gz
2. Java: jdk-8u202-linux-x64.tar.gz (Java11では動かない！）
3. OS: Ubuntu 16 または 18

まずは、こちら↓を参考にされたい。

Hadoop3.2の環境を構築する方法 on Ubuntu
https://toritakashi.com/?p=727

hadoop-3.2.1は、この辺から取ってくる
https://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-3.2.1/

設定するファイルは4つである。

1. hadoop-env.sh : Hadoopの設定ファイル
2. hdfs-site.xml: hdfs (ファイルシステム）の設定ファイル
3. mapred-site.xml: MapReduceの設定ファイル
4. yarn-site.xml: yarn(リソースマネージャー）の設定ファイル

Hadoopは大きい（複雑怪奇なイメージもある(*´Д｀) ）が、とりあえずこの4つのファイルの設定さえすれば、動く。
リソースマネージャーはとりあえず、スケジューラーのようなものだと考える。

設定ファイルをgithubに置く。。。

<pre>
$ cp hadoop-3.2.1/etc/hadoop/hadoop-env.sh qiita/hadoop/configuration_files/                                                                                                              
$ cp hadoop-3.2.1/etc/hadoop/hdfs-site.xml qiita/hadoop/configuration_files/
$ cp hadoop-3.2.1/etc/hadoop/mapred-site.xml qiita/hadoop/configuration_files/
$ cp hadoop-3.2.1/etc/hadoop/yarn-site.xml qiita/hadoop/configuration_files/
</pre>

https://github.com/RuoAndo/qiita/tree/master/hadoop/configuration_files

円周率を計算するときに、今回、つまづいたのが、mapred-site.xmlであった。
いろいろググった結果、↓このように書くとうまくいった。

https://github.com/RuoAndo/qiita/blob/master/hadoop/configuration_files/mapred-site.xml

注意点：
1. Hadoop-3*は、Java11をサポートしていないので、Java8を使う
2. おそらくだが、mapred-site.xml などの設定ファイルは、hadoopを再起動しないと読み込まれないのかもしれない。(stop-all.sh/start-all.shなどを使う）
3. Ubuntuデフォルト？のJavaはJpsなどが入っていないので、多少面倒だが、jdk-8u202-linux-x64.tar.gzを使う。

実行してみる。。
<pre>
~$ hadoop-3.2.1/bin/hadoop jar hadoop-3.2.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar pi 10 10000                             
Number of Maps  = 10
Samples per Map = 10000
2020-12-02 21:50:25,231 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #0
2020-12-02 21:50:25,290 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #1
2020-12-02 21:50:25,299 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #2
2020-12-02 21:50:25,308 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #3
2020-12-02 21:50:25,318 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #4
2020-12-02 21:50:25,325 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #5
2020-12-02 21:50:25,332 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #6
2020-12-02 21:50:25,339 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #7
2020-12-02 21:50:25,346 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #8
2020-12-02 21:50:25,396 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Wrote input for Map #9
Starting Job
2020-12-02 21:50:25,508 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
2020-12-02 21:50:25,814 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1606911352055_0002
2020-12-02 21:50:25,831 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-12-02 21:50:25,871 INFO input.FileInputFormat: Total input files to process : 10
2020-12-02 21:50:25,880 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-12-02 21:50:25,890 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-12-02 21:50:25,893 INFO mapreduce.JobSubmitter: number of splits:10
2020-12-02 21:50:25,957 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-12-02 21:50:25,967 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1606911352055_0002
2020-12-02 21:50:25,967 INFO mapreduce.JobSubmitter: Executing with tokens: []
2020-12-02 21:50:26,090 INFO conf.Configuration: resource-types.xml not found
2020-12-02 21:50:26,090 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2020-12-02 21:50:26,137 INFO impl.YarnClientImpl: Submitted application application_1606911352055_0002
2020-12-02 21:50:26,163 INFO mapreduce.Job: The url to track the job: http://h-gpu05:8088/proxy/application_1606911352055_0002/
2020-12-02 21:50:26,164 INFO mapreduce.Job: Running job: job_1606911352055_0002
2020-12-02 21:50:32,228 INFO mapreduce.Job: Job job_1606911352055_0002 running in uber mode : false
2020-12-02 21:50:32,229 INFO mapreduce.Job:  map 0% reduce 0%
2020-12-02 21:50:36,269 INFO mapreduce.Job:  map 60% reduce 0%
2020-12-02 21:50:40,287 INFO mapreduce.Job:  map 100% reduce 0%
2020-12-02 21:50:41,291 INFO mapreduce.Job:  map 100% reduce 100%
2020-12-02 21:50:41,295 INFO mapreduce.Job: Job job_1606911352055_0002 completed successfully
2020-12-02 21:50:41,361 INFO mapreduce.Job: Counters: 54
        File System Counters
                FILE: Number of bytes read=226
                FILE: Number of bytes written=2496692
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=2660
                HDFS: Number of bytes written=215
                HDFS: Number of read operations=45
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=3
                HDFS: Number of bytes read erasure-coded=0
        Job Counters 
                Launched map tasks=10
                Launched reduce tasks=1
                Data-local map tasks=10
                Total time spent by all maps in occupied slots (ms)=23560
                Total time spent by all reduces in occupied slots (ms)=2303
                Total time spent by all map tasks (ms)=23560
                Total time spent by all reduce tasks (ms)=2303
                Total vcore-milliseconds taken by all map tasks=23560
                Total vcore-milliseconds taken by all reduce tasks=2303
                Total megabyte-milliseconds taken by all map tasks=24125440
                Total megabyte-milliseconds taken by all reduce tasks=2358272
        Map-Reduce Framework
                Map input records=10
                Map output records=20
                Map output bytes=180
                Map output materialized bytes=280
                Input split bytes=1480
                Combine input records=0
                Combine output records=0
                Reduce input groups=2
                Reduce shuffle bytes=280
                Reduce input records=20
                Reduce output records=0
                Spilled Records=40
                Shuffled Maps =10
                Failed Shuffles=0
                Merged Map outputs=10
                GC time elapsed (ms)=2500
                CPU time spent (ms)=15390
                Physical memory (bytes) snapshot=3843653632
                Virtual memory (bytes) snapshot=34035982336
                Total committed heap usage (bytes)=6404177920
                Peak Map Physical memory (bytes)=351567872
                Peak Map Virtual memory (bytes)=3096576000
                Peak Reduce Physical memory (bytes)=372060160
                Peak Reduce Virtual memory (bytes)=3101540352
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=1180
        File Output Format Counters 
                Bytes Written=97
Job Finished in 15.962 seconds
2020-12-02 21:50:41,398 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
Estimated value of Pi is 3.14120000000000000000
</pre>

（｀ー´）b