Ability to more flexibly define the path to a text file

This allows a user to use tilde expansion or simply the file name if the text file is in the user working directory.
Hi Shivaram,

 

I’m fine with removing the lines of code that pre-pend the current working directory and going with the path expansion. The one difference in what you and I have done is that I ran:

 

sudo R CMD INSTALL pkg

 

from within the SparkR-pkg directory, so the SparkR package is the main R library. At this point, after a partial, unsuccessful, sync of my fork with the upstream copy, my local copy is a mess, so I can’t test whether that is causing the difference in what you and I found.

 

Should I retract this pull request, and submit a new one, or is this a change you can make?

 

Dan

From: Shivaram Venkataraman [mailto:notifications@github.co tm] 
Sent: Tuesday, December 30, 2014 6:00 PM
To: amplab-extras/SparkR-pkg
Cc: dputler
Subject: Re: [SparkR-pkg] Ability to more flexibly define the path to a text file (#119)

 

In pkg/R/context.R <https://github.com/amplab-extras/SparkR-pkg/pull/119#discussion-diff-22372779> :

> @@ -19,6 +19,10 @@
>  #'}
>  
>  textFile <- function(sc, path, minSplits = NULL) {
> +  # Allow the user to have a more flexible definiton of the text file path
> +  path <- path.expand(path)
> +  if (!(grepl("\\\\", path) | grepl("/", path)))

Thanks Dan -- I'm still having trouble reproducing this though. I ran the following commands (listed below) on the development VM today and it seemed to work fine. (SparkR-pkg was cloned from github.com/shivaram/SparkR-pkg which is synced to a commit on Dec 22 FWIW).

Could you check the value of getwd() in R when you see the error ? I am wondering if R is somehow getting a wrong working directory.

[cloudera@quickstart ~]$ cd SparkR-pkg/
[cloudera@quickstart SparkR-pkg]$ R
> library(SparkR, lib.loc="./lib")
Loading required package: rJava
[SparkR] Initializing with classpath /home/cloudera/SparkR-pkg/lib/SparkR/sparkr-assembly-0.1.jar
 
> sc <- sparkR.init(master="local")
14/12/30 17:53:15 WARN Utils: Your hostname, quickstart.cloudera resolves to a loopback address: 127.0.0.1; using 172.16.123.130 instead (on interface eth1)
14/12/30 17:53:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/12/30 17:53:15 INFO Slf4jLogger: Slf4jLogger started
14/12/30 17:53:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 
> textFile <- textFile(sc, "README.md")
> 
> linesWithSpark <- filterRDD(textFile, function(line){ grepl("Spark", line)})
> 
> count(linesWithSpark)
14/12/30 17:53:18 INFO FileInputFormat: Total input paths to process : 1
[1] 37

—
Reply to this email directly or view it on GitHub <https://github.com/amplab-extras/SparkR-pkg/pull/119/files#r22372779> .  <https://github.com/notifications/beacon/AIGvMocippzyiEm5lSuY1ane0vitB_56ks5nc1AhgaJpZM4DJRxB.gif>