jghoman/haivvreo — Issue #9: garbled when doing map-reduce
Author: koertkuipers
Created: 2011-12-08 03:16:07+00:00
State: open

my data comes out of external tables backed by avro nicely when i just do "select * from table limit 10;"

however as soon as i do anything that involves map-reduce the data becomes garbled.

for example:
hive> select * from edges limit 5;
OK
347eb58d-9d73-4087-980d-cdd4f9bcffc7	9b7a42e1-8986-48c8-a933-09fecf459289
0d7f380e-0a5c-4e36-af96-5aec6872b67a	3bccb2e1-389a-4443-a1e4-72deb8ed9da3
5ac84201-4adf-460c-91d8-d4f26c73cbe3	7804d00d-5bf0-4d51-82c1-0c95faec0156
fdb87eda-472c-40ec-8807-90e930c3fd09	1e46aa69-63dc-4ee7-b25c-09645c402589
5ba3ae16-c037-486b-b599-710c77976667	224cfcb1-7be6-4c5b-83d3-918bb3bdee27
hive> create table edges_1 as select * from edges;
Total MapReduce jobs = 2
...
hive> select * from edges_1 limit 10;
OK
��w!p73a2b0ae-bfa5-42fe-bd8a-607c5bbf5623	4c504f74-b518-4381-aa00-b0f4ad1ecd0e
0f3c4d08-934f-4ba2-94e7-53e4910f1095	d01b4177-9c9c-4417-P
/e-93a004a03a4e
12331383-21fb-4b75-ba3a-dece3256ea90	1ea29a89-4f5eD	6a2-824e-af6e01808a9d
3b7ffcd8-1bff-4035-8650-234bea646c48	7ed7c3f1-905f-H4e-9522-4b5b5a0befa0
6c7db398-7c75-457b-af20-32f7eefe9285	941f1ac5-afc0d	a-a30f-864d0d956249
cf7def1f-a925-46d	L -8cd74xef0
�
72-4d768385940a	NULL
$	6255463d-941c-4ab4-8ee6-26c0b428b72c
�

not sure what is going on here... the only haivvreo related logging i could see was:
2011-12-07 22:08:52,397 INFO com.linkedin.haivvreo.AvroGenericRecordReader: Unable to match filesplit /data/edges/part-00002.avro:0+30819496 with a partition.