RevolutionAnalytics/RHadoop — Issue #53: rmr 1.2 unable to read 1.1's output via from.dfs()?
Author: jeffreybreen
Created: 2012-02-28 20:50:13+00:00
State: open

I have some output from jseidman's airline demo generated with rmr 1.1.

Expected, as in V1.1:

```
> from.dfs('/dept-delay-month-11', todataframe=T)
key val1 val2 val3 val4
1 2004 3 1983 AA 10.0070600100857
2 2004 3 238 B6 2.09663865546218
3 2004 3 923 CO 2.59696641386782
4 2004 3 443 AS 5.78329571106095
5 2004 3 773 DH 13.5355756791721
6 2004 3 428 FL 0.707943925233645
7 2004 3 1914 DL 1.75600835945664
8 2004 3 704 EV 0.387784090909091
9 2004 3 128 HA 2.1953125
10 2004 3 1282 MQ 8.76287051482059
11 2004 3 532 HP 8.32518796992481
12 2004 3 1438 NW 4.72739916550765
13 2004 3 1006 OH 0.300198807157058
14 2004 3 1179 OO 7.86174724342663
15 2004 3 222 TZ 7.55855855855856
16 2004 3 1512 UA 11.1468253968254
17 2004 3 1143 US 0.636920384951881
18 2004 3 2794 WN 8.37115246957767
19 2004 3 1103 XE -0.204895738893926
```

With version 1.2 installed, from.dfs() croaks with or without to.data.frame specified:

```
> from.dfs('/dept-delay-month-11', to.data.frame=T)
12/02/28 15:39:30 WARN snappy.LoadSnappy: Snappy native library is available
12/02/28 15:39:30 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/02/28 15:39:30 INFO snappy.LoadSnappy: Snappy native library loaded
V1
1 0
2 128
3 255
4 382
5 509
6 636
7 764
8 892
9 1020
10 1139
11 1267
12 1394
13 1522
14 1651
15 1779
16 1906
17 2034
18 2163
19 2291
V2
1 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1983\\n9\\n2\\nAA\\n9\\n16\\n10.0070600100857\\n
2 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n238\\n9\\n2\\nB6\\n9\\n16\\n2.09663865546218\\n
3 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n923\\n9\\n2\\nCO\\n9\\n16\\n2.59696641386782\\n
4 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n443\\n9\\n2\\nAS\\n9\\n16\\n5.78329571106095\\n
5 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n773\\n9\\n2\\nDH\\n9\\n16\\n13.5355756791721\\n
6 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n428\\n9\\n2\\nFL\\n9\\n17\\n0.707943925233645\\n
7 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1914\\n9\\n2\\nDL\\n9\\n16\\n1.75600835945664\\n
8 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n704\\n9\\n2\\nEV\\n9\\n17\\n0.387784090909091\\n
9 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n128\\n9\\n2\\nHA\\n9\\n9\\n2.1953125\\n
10 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1282\\n9\\n2\\nMQ\\n9\\n16\\n8.76287051482059\\n
11 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n532\\n9\\n2\\nHP\\n9\\n16\\n8.32518796992481\\n
12 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1438\\n9\\n2\\nNW\\n9\\n16\\n4.72739916550765\\n
13 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1006\\n9\\n2\\nOH\\n9\\n17\\n0.300198807157058\\n
14 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1179\\n9\\n2\\nOO\\n9\\n16\\n7.86174724342663\\n
15 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n222\\n9\\n2\\nTZ\\n9\\n16\\n7.55855855855856\\n
16 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1512\\n9\\n2\\nUA\\n9\\n16\\n11.1468253968254\\n
17 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1143\\n9\\n2\\nUS\\n9\\n17\\n0.636920384951881\\n
18 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n2794\\n9\\n2\\nWN\\n9\\n16\\n8.37115246957767\\n
19 A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1103\\n9\\n2\\nXE\\n9\\n18\\n-0.204895738893926\\n
>

> from.dfs('/dept-delay-month-11', to.data.frame=F)
12/02/28 15:40:07 WARN snappy.LoadSnappy: Snappy native library is available
12/02/28 15:40:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/02/28 15:40:07 INFO snappy.LoadSnappy: Snappy native library loaded
[[1]]
[[1]]$key
[1] 0

[[1]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1983\\n9\\n2\\nAA\\n9\\n16\\n10.0070600100857\\n"

attr(,"rmr.keyval")
[1] TRUE

[[2]]
[[2]]$key
[1] 128

[[2]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n238\\n9\\n2\\nB6\\n9\\n16\\n2.09663865546218\\n"

attr(,"rmr.keyval")
[1] TRUE

[[3]]
[[3]]$key
[1] 255

[[3]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n923\\n9\\n2\\nCO\\n9\\n16\\n2.59696641386782\\n"

attr(,"rmr.keyval")
[1] TRUE

[[4]]
[[4]]$key
[1] 382

[[4]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n443\\n9\\n2\\nAS\\n9\\n16\\n5.78329571106095\\n"

attr(,"rmr.keyval")
[1] TRUE

[[5]]
[[5]]$key
[1] 509

[[5]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n773\\n9\\n2\\nDH\\n9\\n16\\n13.5355756791721\\n"

attr(,"rmr.keyval")
[1] TRUE

[[6]]
[[6]]$key
[1] 636

[[6]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n428\\n9\\n2\\nFL\\n9\\n17\\n0.707943925233645\\n"

attr(,"rmr.keyval")
[1] TRUE

[[7]]
[[7]]$key
[1] 764

[[7]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1914\\n9\\n2\\nDL\\n9\\n16\\n1.75600835945664\\n"

attr(,"rmr.keyval")
[1] TRUE

[[8]]
[[8]]$key
[1] 892

[[8]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n704\\n9\\n2\\nEV\\n9\\n17\\n0.387784090909091\\n"

attr(,"rmr.keyval")
[1] TRUE

[[9]]
[[9]]$key
[1] 1020

[[9]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n128\\n9\\n2\\nHA\\n9\\n9\\n2.1953125\\n"

attr(,"rmr.keyval")
[1] TRUE

[[10]]
[[10]]$key
[1] 1139

[[10]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1282\\n9\\n2\\nMQ\\n9\\n16\\n8.76287051482059\\n"

attr(,"rmr.keyval")
[1] TRUE

[[11]]
[[11]]$key
[1] 1267

[[11]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n532\\n9\\n2\\nHP\\n9\\n16\\n8.32518796992481\\n"

attr(,"rmr.keyval")
[1] TRUE

[[12]]
[[12]]$key
[1] 1394

[[12]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1438\\n9\\n2\\nNW\\n9\\n16\\n4.72739916550765\\n"

attr(,"rmr.keyval")
[1] TRUE

[[13]]
[[13]]$key
[1] 1522

[[13]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1006\\n9\\n2\\nOH\\n9\\n17\\n0.300198807157058\\n"

attr(,"rmr.keyval")
[1] TRUE

[[14]]
[[14]]$key
[1] 1651

[[14]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1179\\n9\\n2\\nOO\\n9\\n16\\n7.86174724342663\\n"

attr(,"rmr.keyval")
[1] TRUE

[[15]]
[[15]]$key
[1] 1779

[[15]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n3\\n222\\n9\\n2\\nTZ\\n9\\n16\\n7.55855855855856\\n"

attr(,"rmr.keyval")
[1] TRUE

[[16]]
[[16]]$key
[1] 1906

[[16]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1512\\n9\\n2\\nUA\\n9\\n16\\n11.1468253968254\\n"

attr(,"rmr.keyval")
[1] TRUE

[[17]]
[[17]]$key
[1] 2034

[[17]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1143\\n9\\n2\\nUS\\n9\\n17\\n0.636920384951881\\n"

attr(,"rmr.keyval")
[1] TRUE

[[18]]
[[18]]$key
[1] 2163

[[18]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n2794\\n9\\n2\\nWN\\n9\\n16\\n8.37115246957767\\n"

attr(,"rmr.keyval")
[1] TRUE

[[19]]
[[19]]$key
[1] 2291

[[19]]$val
[1] "A\\n2\\n134657\\n131840\\n16\\n1\\n9\\n4\\n2004\\n\tA\\n2\\n134657\\n131840\\n16\\n4\\n9\\n1\\n3\\n9\\n4\\n1103\\n9\\n2\\nXE\\n9\\n18\\n-0.204895738893926\\n"

attr(,"rmr.keyval")
[1] TRUE
```

And, similarly, output files generated by 1.2 can't be read in 1.1:

```
> packageDescription('rmr')$Version
[1] "1.1"
> from.dfs('/dept-delay-month-12', todataframe=T)
Error in unserialize(charToRaw(gsub("\\\\n", "\n", x))) :
unknown input format
In addition: Warning message:
In readLines(file.path(tmp, f)) :
incomplete final line found on '/tmp/Rtmp6ooeRQ/file57b442a642f3/part-00000'
> from.dfs('/dept-delay-month-12', todataframe=F)
Error in unserialize(charToRaw(gsub("\\\\n", "\n", x))) :
unknown input format
In addition: Warning message:
In readLines(file.path(tmp, f)) :
incomplete final line found on '/tmp/Rtmp6ooeRQ/file57b4247a433b/part-00000'
>
```

And 1.2 works fine with output it generated:

```
> packageDescription('rmr')$Version
[1] "1.2"
> from.dfs('/dept-delay-month-12', to.data.frame=T)
V1 V2 V3 V4 V5
1 2004 3 1983 AA 10.0070600100857
2 2004 3 443 AS 5.78329571106095
3 2004 3 238 B6 2.09663865546218
4 2004 3 923 CO 2.59696641386782
5 2004 3 773 DH 13.5355756791721
6 2004 3 1914 DL 1.75600835945664
7 2004 3 704 EV 0.387784090909091
8 2004 3 428 FL 0.707943925233645
9 2004 3 128 HA 2.1953125
10 2004 3 532 HP 8.32518796992481
11 2004 3 1282 MQ 8.76287051482059
12 2004 3 1438 NW 4.72739916550765
13 2004 3 1006 OH 0.300198807157058
14 2004 3 1179 OO 7.86174724342663
15 2004 3 222 TZ 7.55855855855856
16 2004 3 1512 UA 11.1468253968254
17 2004 3 1143 US 0.636920384951881
18 2004 3 2794 WN 8.37115246957767
19 2004 3 1103 XE -0.204895738893926
>
```

These are all run in R 2.14.1 installed on Cloudera's demo VM cdh3u3.