linkedin/datafu — pull request #31 by msukmanowsky (2013-05-06 19:19:29+00:00)
Title: TotalTimeBetween UDF

This UDF takes an input bag which is assumed to be ordered by an ISO8601 timestamp field and returns the total time elapsed (in seconds) by all tuples in the bag. The first field in each of the tuples in the DataBag is assumed to be the ISO8601 timestamp.

**Example:**

DEFINE TotalTimeBetween datafu.pig.bags.TotalTimeBetween();

-- input:
-- 2012-01-01T00:00:00\tbiuA8n98wn\thttp://www.google.com/
-- 2012-01-01T00:00:00\tasd909m09j\thttp://www.google.com/
-- 2012-01-01T00:01:00\tbiuA8n98wn\thttp://www.google.com/1
-- 2012-01-01T00:01:05\tbiuA8n98wn\thttp://www.google.com/
-- 2012-01-01T00:02:00\tasd909m09j\thttp://www.google.com/2
input = LOAD 'input' AS (timestamp:chararray, visitor_id:chararray, url:chararray);
by_visitor = GROUP data BY visitor_id;
count_time = FOREACH by_visitor {
ordered = ORDER data BY timestamp ASC;
GENERATE group, TotalTimeBetween(ordered);
};
DUMP count_time;
-- (biuA8n98wn,65)
-- (asd909m09j,120)