Bug: Spark History not loading when service has to load 300k applications initially from S3
Status: Resolved | Priority: Major | Resolution: Duplicate
Created: 14/Jan/21 02:52
Affects Version/s: 3.0.0

Spark History Service is having trouble loading when loading initially with 300k+ applications from S3. Following are the details and snapshots:

Number of files in `spark.history.fs.logDirectory`: (Using xxx for anonymity)
{noformat}
spedamallu@spedamallu-mbp143 ~/src/spark (spark-bug) $
| => aws s3 ls s3://xxxx-company/spark-history-fs-logDirectory/ | wc -l
  305571
spedamallu@spedamallu-mbp143 ~/src/spark (spark-bug) ${noformat}
{noformat}
Logs when starting SparkHistory:
{noformat}
root@shs-with-statsd-86d7f54679-t8fqr:/go/src/github.com/xxxx-company/spark-private# /go/src/github.com/xxxx-company/spark-private/bootstrap/start-history-server.sh --properties-file /etc/spark-history-config/shs-default.properties
 2021/01/14 02:40:28 Spark spark wrapper is disabled
 2021/01/14 02:40:28 Attempt number 0, Max attempts 0, Left Attempts 0
 2021/01/14 02:40:28 Statsd disabled
 2021/01/14 02:40:28 Debug log: /tmp/.log
 2021/01/14 02:40:28 Job submitted 0 seconds ago, Operator 0, ETL 0, Flyte 0 Mozart 0
 2021/01/14 02:40:28 Running command /opt/spark/bin/spark-class.orig with arguments [org.apache.spark.deploy.history.HistoryServer --properties-file /etc/spark-history-config/shs-default.properties]
 21/01/14 02:40:29 INFO HistoryServer: Started daemon with process name: 2077@shs-with-statsd-86d7f54679-t8fqr
 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for TERM
 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for HUP
 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for INT
 21/01/14 02:40:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 21/01/14 02:40:30 INFO SecurityManager: Changing view acls to: root
 21/01/14 02:40:30 INFO SecurityManager: Changing modify acls to: root
 21/01/14 02:40:30 INFO SecurityManager: Changing view acls groups to:
 21/01/14 02:40:30 INFO SecurityManager: Changing modify acls groups to:
 21/01/14 02:40:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
 21/01/14 02:40:30 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
 21/01/14 02:40:30 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
 21/01/14 02:40:30 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
 21/01/14 02:40:30 INFO MetricsSystemImpl: s3a-file-system metrics system started
 21/01/14 02:40:31 INFO log: Logging initialized @1933ms to org.sparkproject.jetty.util.log.Slf4jLog
 21/01/14 02:40:31 INFO Server: jetty-9.4.z-SNAPSHOT; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_242-b08
 21/01/14 02:40:31 INFO Server: Started @1999ms
 21/01/14 02:40:31 INFO AbstractConnector: Started ServerConnector@51751e5f
{HTTP/1.1,[http/1.1]} {0.0.0.0:18080}
21/01/14 02:40:31 INFO Utils: Successfully started service on port 18080.
 21/01/14 02:40:31 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@b9dfc5a
{/,null,AVAILABLE,@Spark}
21/01/14 02:40:31 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1bbae752
{/json,null,AVAILABLE,@Spark}
21/01/14 02:40:31 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@5cf87cfd
{/api,null,AVAILABLE,@Spark}
21/01/14 02:40:31 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@74971ed9
{/static,null,AVAILABLE,@Spark}
21/01/14 02:40:31 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1542af63
{/history,null,AVAILABLE,@Spark}
21/01/14 02:40:31 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://shs-with-statsd-86d7f54679-t8fqr:18080
 21/01/14 02:40:31 DEBUG FsHistoryProvider: Scheduling update thread every 10 seconds
 21/01/14 02:40:31 DEBUG FsHistoryProvider: Scanning s3a://xxxx-company/spark-history-fs-logDirectory/ with lastScanTime==-1{noformat}