Challenges that everyone struggles while productionizing Apache Spark workloads - Chetan Khatri
Channel: ScalaIO FR
Date: 10/02/2020

Challenges that everyone struggles while productionizing Apache Spark workloads - Chetan Khatri


1. Primary data structures (RDD, DataSet, Dataframe)
2. Pragmatic explanation - executors, cores, containers, stage, job, a task in Spark.
3. Parallel read from JDBC: Challenges and best practices.
4. Bulk Load API vs JDBC write
5. An optimization strategy for Joins: SortMergeJoin vs BroadcastHashJoin
6. Avoid unnecessary shuffle and use of coalesce, repartition, HashPartitioner with use cases. Impact on cache, Disk I/O, Leaking memory, Internal shuffle, spark executor, spark driver.
7. What to do when spark default sort doesn’t work!, Alternatives.
8. Why dropDuplicates() doesn’t result consistency, What is alternative.
9. Optimize Spark stage generation plan: reduce unnecessary repetitive Actions.
10. Predicate pushdown with partitioning and bucketing.
11. Why not to use Scala Concurrent ‘Future’ explicitly with Spark jobs.