RevolutionAnalytics/RHadoop — Issue #52: MapReduce issue with reading the file Input
Author: mayankbansal89
Created: 2012-02-20 11:28:42+00:00
State: open

Hey ,
I wrote a piece of code which runs only if i make an encapsulated function of the mapreduce code
If i don't do that it doesn't run. I was banking on mapReduce to run without making a function.
I spent 4 days wondering that the mapper code that i wrote is wrong.
Later I realised that the input being read is different when I make a function of the whole code.
i would want to know why does this happen and further reasons for my code not running.
Written below is the code that i wrote code 2 works and code 1 doesn't.
I would want some information on the reasons for it to not work.

Code 1
------------------------------------------------
matrixprod <- from.dfs(mapreduce(input="/user/hadoop/inp/data.csv",
output=NULL, map <- function(k,v){
v<-unlist(v)
to.dfs(mean,"out/matrix.csv")
xi <- c(1,(v[1:(length(v)-1)]));
yi <- v[length(v)];
mtr<- xi %*% t(xi);
myr<- xi * yi;
m<-cbind(mtr,myr);
return(keyval(1,m))
}, reduce<-function(k,vv){
mrsum <- Reduce("+",vv)
return(keyval(k,mrsum))
},
combine=FALSE, textinputformat<-function(line){
keyval(1,as.numeric(strsplit(line,',')[[1]]))
}))
mrx <- matrixprod[[1]]$val
beta <- solve(mrx[,1:(dim(mrx)[2]-1)],mrx[,dim(mrx)[2]])

Code 2
---------------------------------------------------------------------
library(rmr)
library(rhdfs)

matrixols <- function(input){
csvinput <- function(line){
keyval(1,as.numeric(strsplit(line,',')[[1]]))
}

mmap <- function(k,v){
v<-unlist(v)
to.dfs(mean,"out/matrix.csv")
xi <- c(1,(v[1:(length(v)-1)]));
yi <- v[length(v)];
mtr<- xi %*% t(xi);
myr<- xi * yi;
m<-cbind(mtr,myr);
return(keyval(1,m))
}

mreduce<-function(k,vv){
mrsum <- Reduce("+",vv)
return(keyval(k,mrsum))
}

matrixprod <- from.dfs(mapreduce(input=input,
output=NULL, map=mmap, reduce=mreduce,
combine=FALSE, textinputformat=csvinput))
mrx <- matrixprod[[1]]$val
beta <- solve(mrx[,1:(dim(mrx)[2]-1)],mrx[,dim(mrx)[2]])
return(beta)
}
matrixols("/user/hadoop/inp/data.csv")