Apache Hadoop: PIG
By: Bhanu Soni
Tags: Hadoop, Apache Pig, Big Data, Mapreduce, Data Science

What is Apache Pig?

Apache Pig is a platform for analyzing large data sets. Pig’s language, Pig Latin, is a simple query algebra that lets you express data transformations such as merging data sets, filtering them, and applying functions to records or groups of records. Users can create their own functions to do special-purpose processing.

Pig Latin queries execute in a distributed fashion on a cluster. Our current implementation compiles Pig Latin programs into Map-Reduce jobs and executes them using Hadoop cluster.

Pig provides an engine for executing data flows in parallel on Hadoop. It includes a language, Pig Latin, for expressing these data flows. Pig Latin includes operators for many of the traditional data operations (join, sort, filter, etc.), as well as the ability for users to develop their own functions for reading, processing, and writing data. Pig is an Apache open source project.

Pig raises the level of abstraction for processing large datasets. With MapReduce, there is a map function and there is a reduce function, and working out how to fit your data processing into this pattern, which often requires multiple MapReduce stages, can be a challenge. With Pig the data structures are much richer, typically being multivalued and nested, and the set of transformations you can apply to the data are much more powerful — they include joins, for example, which are not for the faint of heart in MapReduce.

Pig is made up of two pieces:

The language used to express data flows, called Pig Latin. The execution environment to run Pig Latin programs. There are currently two environments: local execution in a single JVM and distributed execution on a Hadoop cluster.

Installing and Running Pig

Pig runs as a client-side application. Even if you want to run Pig on a Hadoop cluster, there is nothing extra to install on the cluster: Pig launches jobs and interacts with HDFS (or other Hadoop filesystems) from your workstation.

Installation is straightforward. Java 6 is a prerequisite (and on Windows, you will need Cygwin). Download a stable release from http://hadoop.apache.org/pig/releases.html, and unpack the tarball in a suitable place on your workstation

create a directory named pig in /usr/local copy the pig tar file to pig directory untar the tar file

Extract the tar file using tar command. In below tar command, x means extract an archive file, z means filter an archive through gzip, f means filename of an archive file

tar -xzf pig-x.y.z.tar.gz

4. set the path in bashrc

Edit the “.bashrc” file to update the environment variables of Apache Pig. We are setting it so that we can access pig from any directory, we need not go to pig directory to execute pig commands. Also, if any other application is looking for Pig, it will get to know the path of Apache Pig from this file.

% export PIG_INSTALL=/home/tom/pig-x.y.z

% export PATH=$PATH:$PIG_INSTALL/bin

5. exec bash

6. launch grunt shell

Execution Types

Pig has two execution types or modes: local mode and Hadoop mode.

Local mode

In local mode, Pig runs in a single JVM and accesses the local filesystem. This mode is suitable only for small datasets, and when trying out Pig. Local mode does not use Hadoop. In particular, it does not use Hadoop’s local job runner; instead, Pig translates queries into a physical plan that it executes itself.

The execution type is set using the -x or -exec type option. To run in local mode, set the option to local :

% pig -x local

grunt>

This starts Grunt, the Pig interactive shell, which is discussed in more detail shortly.

Hadoop mode

In Hadoop mode, Pig translates queries into MapReduce jobs and runs them on a Hadoop cluster. The cluster may be a pseudo- or fully distributed cluster. Hadoop mode (with a fully distributed cluster) is what you use when you want to run Pig on large datasets.

To use Hadoop mode, you need to tell Pig which version of Hadoop you are using and where your cluster is running. Pig releases will work against only particular versions of Hadoop.

% pig

Running Pig Programs

Script

Pig can run a script file that contains Pig commands. For example, pig script.pig runs the commands in the local file script.pig. Alternatively, for very short scripts, you can use the -e option to run a script specified as a string on the command line.

Grunt

Grunt is an interactive shell for running Pig commands. Grunt is started when no file is specified for Pig to run, and the -e option is not used. It is also possible to run Pig scripts from within Grunt using run and exec .

Embedded

You can run Pig programs from Java, much like you can use JDBC to run SQL

programs from Java. There are more details on the Pig wiki at http://wiki.apache.org/pig/EmbeddedPig.

Grunt

Grunt has line-editing facilities like those found in GNU Readline (used in the bash shell and many other command-line applications). For instance, the Ctrl-E key combination will move the cursor to the end of the line. Grunt remembers command history, too, and you can recall lines in the history buffer using Ctrl-P or Ctrl-N (for previous and next), or, equivalently, the up or down cursor keys. Another handy feature is Grunt’s completion mechanism, which will try to complete Pig Latin keywords and functions when you press the Tab key. For example, consider the following incomplete line

grunt> a = foreach b ge

If you press the Tab key at this point, ge will expand to generate , a Pig Latin keyword:

grunt> a = foreach b generate

You can get a list of commands using the help command. When you’ve finished your Grunt session, you can exit with the quit command.

Pig Latin, a Parallel Dataflow Language

Pig Latin is a dataflow language. This means it allows users to describe

how data from one or more inputs should be read, processed, and then stored to one or more outputs in parallel. These data flows can be simple linear flows. They can also be complex workflows that include points where multiple inputs are joined, and where data is split into multiple streams to be processed by different operators.

Comparing query and dataflow languages

SQL is a query language. Its focus is to allow users to form queries.

It allows users to describe what question they want to be answered, but not how they want it answered.

In Pig Latin, on the other hand, the user describes exactly how to

process the input data. SQL is oriented around answering one question. When users want

to do several data operations together, they must either write separate queries, storing the intermediate data into temporary tables, or write it in one query using subqueries inside that query to do the earlier steps of the processing. However, many SQL users find subqueries confusing and difficult to form properly. Also, using sub queries creates an inside-out design where the first step in the data pipeline is the innermost query.

In Pig, however, is designed with a long series of data operations in

mind, so there is no need to write the data pipeline in an inverted set of subqueries or to worry about storing data in temporary tables. SQL is designed for the RDBMS environment, where data is normalized and schemas and proper constraints are enforced (that is, there are no nulls in places they do not belong, etc.).

Pig is designed for the Hadoop data-processing environment, where

schemas are sometimes unknown or inconsistent. Data may not be properly constrained, and it is rarely normalized. As a result of these differences, Pig does not require data to be loaded into tables first. It can operate on data as soon as it is copied into HDFS. SQL is the English of data processing. It has a nice feature that

everyone and every tool know it, which means the barrier to adoption is very low.

Our goal is to make Pig Latin the native language of parallel data processing systems such as Hadoop. It may take some learning, but it

will allow users to utilize the power of Hadoop much more fully.

Pig’s History

Pig started out as a research project in Yahoo! Research, where Yahoo!

scientists designed it and produced an initial implementation. As explained in a paper presented at SIGMOD in 2008, the researchers felt that the MapReduce paradigm presented by Hadoop “is too low-level and rigid, and leads to a great deal of custom user code that is hard to maintain and reuse.” At the same time, they observed that many MapReduce users were not comfortable with declarative languages such as SQL.

Thus they set out to produce “a new language called Pig Latin that we have designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of MapReduce.”

Yahoo! Hadoop users started to adopt Pig. So, a team of development

engineers was assembled to take the research prototype and build it into a

production-quality product. About this same time, in fall 2007, Pig was open-sourced via the Apache Incubator. The first Pig release came a year later in September 2008. Later that same year, Pig graduated from the Incubator and became a subproject of Apache Hadoop.

Why Is It Called Pig?

People also want to know whether Pig is an acronym. It is not. The

story goes that the researchers working on the project initially referred

to it simply as “the language.” Eventually, they needed to call it something. Off the top of his head, one researcher suggested Pig, and the name stuck. It is quirky yet memorable and easy to spell. While some have hinted that the

name sounds coy or silly, it has provided us with an entertaining nomenclature, such as Pig Latin for a language, Grunt for a shell, and Piggybank for a CPAN-like shared repository.

In the next article, we will discuss pig commands….!!