[SPARK-3007][SQL] WIP: adds dynamic partitioning support

PR #2226 was reverted because of it broke Jenkins builds for unknown reason. This debugging PR aims to fix the Jenkins build.