Ability to more flexibly define the path to a text file

This allows a user to use tilde expansion or simply the file name if the text file is in the user working directory.
Hi Shivaram,

 

It is my turn to be slow, I’ve been out of town for the past several days.

 

Here is an example where the problem is generated, you need to cd to the SparkR-pkg directory first:

library("SparkR")

sc <- sparkR.init(master="local")

textFile <- textFile(sc, "README.md")

linesWithSpark <- filterRDD(textFile, function(line){ grepl("Spark", line)})

count(linesWithSpark)

 

The returned error message is: 

 

Error in .jcall(getJRDD(rdd), "Ljava/util/List;", "collect") : 
  org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/home/cloudera/README.md

 

The issue is that when a full path is not given, the home directory is currently prepended to the file name, not the actual working directory where README.md.

 

Dan

 

From: Shivaram Venkataraman [mailto:notifications@github.com] 
Sent: Tuesday, December 23, 2014 1:45 PM
To: amplab-extras/SparkR-pkg
Cc: dputler
Subject: Re: [SparkR-pkg] Ability to more flexibly define the path to a text file (#119)

 

In pkg/R/context.R <https://github.com/amplab-extras/SparkR-pkg/pull/119#discussion-diff-22237382> :

> @@ -19,6 +19,10 @@
>  #'}
>  
>  textFile <- function(sc, path, minSplits = NULL) {
> +  # Allow the user to have a more flexible definiton of the text file path
> +  path <- path.expand(path)
> +  if (!(grepl("\\\\", path) | grepl("/", path)))

Sorry for the delay -- I finally got a chance to try it out today in a couple of settings (with the SparkR script and with RStudio).

In both cases I found that current working directory worked directly without any need for adding getwd. For example, these were the commands I used in RStudio

> library(SparkR)
> sc <- sparkR.init()
> rdd <- textFile(sc, "README.md")
14/12/23 21:42:05 INFO FileInputFormat: Total input paths to process : 1
> count(rdd)
[1] 128

Just for reference the working directory contents were

> list.files(getwd())
[1] "R"         "README.md"

Would you happen to have any logs / a way to reproduce the problem you saw with quick start ? 

BTW I think we should anyways commit the path.expand change as it is good to handle things like ~ in the path.

—
Reply to this email directly or view it on GitHub <https://github.com/amplab-extras/SparkR-pkg/pull/119/files#r22237382> .  <https://github.com/notifications/beacon/AIGvMq3yzsXtjbIrVXuyoLCHb0ewGbPJks5nadnwgaJpZM4DJRxB.gif>