Yes, we can do that if Spark is running is cluster (including local cluster) mode, as each executor will construct its SparkEnv. However, if Spark in running in local mode, it has no executors and will just use BlockManager in SparkContext, which is ready constructed and currently we just set the system property in LocalScheduler.

An alternative is, instead of using a system property, we can pass it as a variable in BlockManager (by setting it as appropriate in Executor and LocalScheduler).