[SPARK-3928][SQL] Loop through globStatus output and listStatus on directories. This will handle the case if input is not glob path but normal directory path, making it backward compatiable.