Bug: Spark 3.4.0 always using default log4j profile
Status: Resolved | Priority: Major | Resolution: Not A Problem
Created: 18/Sep/23 10:13
Affects Version/s: 3.4.0

I've been using Spark core 3.2.2 and was upgrading to 3.4.0

On execution of my Java code with the 3.4.0,  it generates some extra set of logs but don't face this issue with 3.2.2.

 

I noticed that logs says _Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties._

 

Is this a bug or do we have a  a configuration to disable the using of default log4j profile?

I didn't see anything in the documentation

 

+*Output:*+
{code:java}
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
23/09/18 20:05:08 INFO SparkContext: Running Spark version 3.4.0
23/09/18 20:05:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/09/18 20:05:08 INFO ResourceUtils: ==============================================================
23/09/18 20:05:08 INFO ResourceUtils: No custom resources configured for spark.driver.
23/09/18 20:05:08 INFO ResourceUtils: ==============================================================
23/09/18 20:05:08 INFO SparkContext: Submitted application: XYZ
23/09/18 20:05:08 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/09/18 20:05:08 INFO ResourceProfile: Limiting resource is cpu
23/09/18 20:05:08 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/09/18 20:05:08 INFO SecurityManager: Changing view acls to: jd
23/09/18 20:05:08 INFO SecurityManager: Changing modify acls to: jd
23/09/18 20:05:08 INFO SecurityManager: Changing view acls groups to: 
23/09/18 20:05:08 INFO SecurityManager: Changing modify acls groups to: 
23/09/18 20:05:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: jd; groups with view permissions: EMPTY; users with modify permissions: jd; groups with modify permissions: EMPTY
23/09/18 20:05:08 INFO Utils: Successfully started service 'sparkDriver' on port 39155.
23/09/18 20:05:08 INFO SparkEnv: Registering MapOutputTracker
23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMaster
23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/09/18 20:05:08 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/09/18 20:05:08 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/09/18 20:05:08 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-226d599c-1511-4fae-b0e7-aae81b684012
23/09/18 20:05:08 INFO MemoryStore: MemoryStore started with capacity 2004.6 MiB
23/09/18 20:05:08 INFO SparkEnv: Registering OutputCommitCoordinator
23/09/18 20:05:08 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
23/09/18 20:05:09 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/09/18 20:05:09 INFO Executor: Starting executor ID driver on host jd
23/09/18 20:05:09 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
23/09/18 20:05:09 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 32819.
23/09/18 20:05:09 INFO NettyBlockTransferService: Server created on jd:32819
23/09/18 20:05:09 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/09/18 20:05:09 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, jd, 32819, None)
23/09/18 20:05:09 INFO BlockManagerMasterEndpoint: Registering block manager jd:32819 with 2004.6 MiB RAM, BlockManagerId(driver, jd, 32819, None)
23/09/18 20:05:09 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, jd, 32819, None)
23/09/18 20:05:09 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, jd, 32819, None)
{code}
 

 

 

I'm using Spark core dependency in one of my Jars, the jar code contains the following:

 

*+Code:+*
{code:java}
SparkSession
          .builder()
          .appName("XYZ")
          .config("spark.master", "local")
          .config("spark.io.compression.codec", "snappy")
          .config("spark.io.compression.snappy.blockSize", blockSize)
          .getOrCreate(); {code}
 

When it calls *getOrCreate()* method, I get the above stated extra logs.