Ability to more flexibly define the path to a text file

This allows a user to use tilde expansion or simply the file name if the text file is in the user working directory.
It doesn’t find “README.md”. The current change would let “./README.md” and all the other examples you present pass through without pre-pending getwd(). Only cases where the user (likely implicitly) assumes that the current working directory is being used would be (correctly) caught by the filter. So it addresses only a very limited use case that has no impact on those you indicate. This is your call, but I think the probability is very low that this change has a downside impact, and has a known upside impact.

 

We came across this issue as we reworked the standard Spark “Quick Start” guide for SparkR. In the guide, the text file is implicitly read from the current working directory. The proposed change would allow that to work for SparkR. Here is the link I’m referring to: http://spark.apache.org/docs/latest/quick-start.html

 

The side note to this is we are working through a port of the “Quick Start Guide” for SparkR, and will be submitting code refinements that we think will make the port smoother.

 

Dan

 

From: Shivaram Venkataraman [mailto:notifications@github.com] 
Sent: Tuesday, December 16, 2014 7:23 PM
To: amplab-extras/SparkR-pkg
Cc: dputler
Subject: Re: [SparkR-pkg] Ability to more flexibly define the path to a text file (#119)

 

In pkg/R/context.R <https://github.com/amplab-extras/SparkR-pkg/pull/119#discussion-diff-21950403> :

> @@ -19,6 +19,10 @@
>  #'}
>  
>  textFile <- function(sc, path, minSplits = NULL) {
> +  # Allow the user to have a more flexible definiton of the text file path
> +  path <- path.expand(path)
> +  if (!(grepl("\\\\", path) | grepl("/", path)))

I think path expansion is a good idea, but I am not sure about pre-pending getwd().

In distributed settings the path is usually a HDFS file or S3 file and looks like hdfs://:9000/file or s3n:///file etc. (And checking for all these prefixes might be painful).

BTW doesn't R automatically figure out relative paths ? For example I've used textFile(sc, "./README.md") before and it used to work.

—
Reply to this email directly or view it on GitHub <https://github.com/amplab-extras/SparkR-pkg/pull/119/files#r21950403> .  <https://github.com/notifications/beacon/AIGvMj7v1EADJHM2fhSzZeJDLJquiuA4ks5nYO6mgaJpZM4DJRxB.gif>