Apache Pig - TOKENIZE()

The TOKENIZE() function of Pig Latin is used to split a string (which contains a group of words) in a single tuple and returns a bag which contains the output of the split operation.
Given below is the syntax of the TOKENIZE() function.
grunt> TOKENIZE(expression [, 'field_delimiter']) 

As a delimeter to the TOKENIZE() function, we can pass space [ ], double quote [" "], coma [ , ], parenthesis [ () ], star [ * ].
Assume that we have a file named student_details.txt in the HDFS directory /pig_data/ as shown below. This file contains the details of a student like id, name, age and city. If we closely observe, the name of the student includes first and last names separated by space [ ].
student_details.txt
001,Rajiv Reddy,21,Hyderabad
002,siddarth Battacharya,22,Kolkata 
003,Rajesh Khanna,22,Delhi 
004,Preethi Agarwal,21,Pune 
005,Trupthi Mohanthy,23,Bhuwaneshwar 
006,Archana Mishra,23 ,Chennai 
007,Komal Nayak,24,trivendram 
008,Bharathi Nambiayar,24,Chennai 

We have loaded this file into Pig with the relation name student_details as shown below.
grunt> student_details = LOAD 'hdfs://localhost:9000/pig_data/student_details.txt' USING PigStorage(',')
   as (id:int, name:chararray, age:int,  city:chararray);
We can use the TOKENIZE() function to split a string. As an example let us split the name using this function as shown below.
grunt> student_name_tokenize = foreach student_details  Generate TOKENIZE(name);
Verify the relation student_name_tokenize using the DUMP operator as shown below.
grunt> Dump student_name_tokenize;
It will produce the following output, displaying the contents of the relation student_name_tokenize as follows.
({(Rajaiv),(Reddy)})
({(siddarth),(Battacharya)})
({(Rajesh),(Khanna)})
({(Preethi),(Agarwal)})
({(Trupthi),(Mohanthy)})
({(Archana),(Mishra)})
({(Komal),(Nayak)})
({(Bharathi),(Nambiayar)})

In the same way, including space [], the TOKENIZE() function accepts double quote [" "], coma [ , ], parenthesis [ () ], star [ * ] as delimeters.
Suppose there is a file named details.txt with students details like id, name, age, and city. Under the name column this file contains the first name and the last name of the students separated by various delimeters as shown below.
details.txt
001,"siddarth""Battacharya",22,Kolkata 
002,Rajesh*Khanna,22,Delhi 
003,(Preethi)(Agarwal),21,Pune 

We have loaded this file into Pig with the relation name details as shown below.
grunt> details = LOAD 'hdfs://localhost:9000/pig_data/details.txt' USING PigStorage(',')
   as (id:int, name:chararray, age:int,  city:chararray);
Now, try to separate the first name and the last name of the students using TOKENIZE() as follows.
grunt> tokenize_data = foreach details Generate TOKENIZE(name);
On verifying the tokenize_data relation using dump operator you will get the following result.
grunt> Dump tokenize_data;

({(siddarth),(Battacharya)})
({(Rajesh),(Khanna)})
({(Preethi),(Agarwal)})


    46 Lectures 
     3.5 hours 

 Arnab Chakraborty

    23 Lectures 
     1.5 hours 

 Mukund Kumar Mishra

    16 Lectures 
     1 hours 

 Nilay Mehta

    52 Lectures 
     1.5 hours 

 Bigdata Engineer

    14 Lectures 
     1 hours 

 Bigdata Engineer

    23 Lectures 
     1 hours 

 Bigdata Engineer
 Print
 Add Notes
 Bookmark this page