Bug: SaveMode.Overwrite not usable when using s3a root paths 
Status: Open | Priority: Minor | Resolution: 
Created: 30/Jan/21 16:32
Affects Version/s: 3.1.2

SaveMode.Overwrite does not work when using paths containing just the root eg "s3a://peakhour-report". To reproduce the issue (an s3 bucket + credentials are needed):

{color:#0033b3}val {color}{color:#000000}out {color}= {color:#067d17}"s3a://peakhour-report"{color}

{color:#0033b3}val {color}{color:#000000}sparkContext{color}: {color:#000000}SparkContext {color}= {color:#000000}SparkContext{color}.getOrCreate()
{color:#0033b3}val {color}{color:#000000}someData {color}= {color:#871094}Seq{color}(Row({color:#1750eb}24{color}, {color:#067d17}"mouse"{color}))
{color:#0033b3}val {color}{color:#000000}someSchema {color}= {color:#871094}List{color}(StructField({color:#067d17}"age"{color}, {color:#000000}IntegerType{color}, {color:#0033b3}true{color}),StructField({color:#067d17}"word"{color}, {color:#000000}StringType{color},{color:#0033b3}true{color}))

{color:#0033b3}val {color}{color:#000000}someDF {color}= {color:#871094}spark{color}.createDataFrame(
 {color:#871094}spark{color}.sparkContext.parallelize({color:#000000}someData{color}),StructType({color:#000000}someSchema{color}))
{color:#000000}sparkContext{color}.hadoopConfiguration.set({color:#067d17}"fs.s3a.access.key"{color}, accessK{color:#000000}ey{color}))
{color:#000000}sparkContext{color}.hadoopConfiguration.set({color:#067d17}"fs.s3a.secret.key"{color}, {color:#000000}secretKey{color}))
{color:#000000}sparkContext{color}.hadoopConfiguration.set({color:#067d17}"fs.s3a.aws.credentials.provider"{color}, {color:#067d17}"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"{color})
{color:#000000}sparkContext{color}.hadoopConfiguration.set({color:#067d17}"fs.s3a.impl"{color}, {color:#067d17}"org.apache.hadoop.fs.s3a.S3AFileSystem"{color})
{color:#000000}someDF{color}.write.format({color:#067d17}"parquet"{color}).partitionBy({color:#067d17}"age"{color}).mode({color:#000000}SaveMode{color}.{color:#871094}Overwrite{color})
 .save({color:#000000}out{color})

 

Error stacktrace:

Exception in thread "main" java.lang.IllegalArgumentException: Can not create a Path from an empty string
 at org.apache.hadoop.fs.Path.checkPathArg(Path.java:168)[....]
at org.apache.hadoop.fs.Path.suffix(Path.java:446)
 at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.deleteMatchingPartitions(InsertIntoHadoopFsRelationCommand.scala:240)

 

If you change out from {color:#0033b3}val {color}{color:#000000}out {color}= {color:#067d17}"s3a://peakhour-report"{color} to {color:#0033b3}val {color}{color:#000000}out {color}= {color:#067d17}"s3a://peakhour-report/folder" {color:#172b4d}the code works.{color}{color}

{color:#067d17}{color:#172b4d}There are two problems in the actual code from InsertIntoHadoopFsRelationCommand.deleteMatchingPartitions: {color}{color}

{color:#067d17}{color:#172b4d}a) it uses org.apache.hadoop.fs.Path.suffix method that doesn't work on root paths
{color}{color}

{color:#067d17}{color:#172b4d}b) it tries to delete the root folder directly (in our case the s3 bucket name) and this is prohibited (in the S3AFileSystem class){color}{color}

{color:#067d17}{color:#172b4d}I think that there are two choices:{color}{color}

{color:#067d17}{color:#172b4d}a) throw an explicit error when using overwrite mode for root folders {color}{color}

{color:#067d17}{color:#172b4d}b)fix the actual issue. don't use the Path.suffix method and change the clean up code from InsertIntoHadoopFsRelationCommand.deleteMatchingPartitions to list the root folder content and delete the entries one by one.{color}{color}

I can provide a patch for both choices, assuming that they make sense.