Getting Started with Pentaho Data Integration (Kettle) and its Components
By: Shravankumar Suvarna
Tags: Components, Artificial Intelligence, Data, Data Science, Data Analysis

Before diving deep in understanding the components. We need to firstly understand the below mentioned two concepts in PDI. So, there can be either transformations or jobs that we build using PDI. Definitions are as below.

Transformations: Helps us create data flow by sourcing, manipulating and loading the data. It has a .ktr file extension.

Helps us create data flow by sourcing, manipulating and loading the data. It has a .ktr file extension. Jobs: Helps us run multiple transformations and control its execution sequence. It has a .kjb file extension.

In a nutshell, transformations are to source and process the data, whereas jobs are the high-level executions of child transformations, sending log emails, etc.

Also, both these are just plain XML (file-format) metadata files that describe what to do, how to do, and where to extract information.

You can check the difference using the following link

Spoon

Well, the spoon is the most important component of PDI. Spoon is a graphical user interface (client) for the users to build transformations and jobs. It is a desktop application that comes along with the PDI installation.

We can find the spoon file within the ‘…/pdi-ee/data-integration’ folder. We can launch the desktop application by double-clicking the spoon file. Please note, it is different for different OS.

Windows: Spoon.bat file - batch file

Spoon.bat file - batch file Linux: spoon.sh file - shell file

Spoon in files my Data Integration Folder

Once you double click on the spoon file you will see the above UI.

I will write a separate blog post explaining each feature of the spoon.

Pan

Similar to the spoon file which lets us create jobs and transformations. Pan file helps us execute the transformation files created using Spoon on CLI (Command Line Interface).

Typically, we will use this in the context of creating scripts for scheduling jobs using task manager (Windows) or cron jobs (Linux).

Again, we have different files for different OS.

Windows: Pan.bat - batch file

Pan.bat - batch file Linux: pan.sh - shell file

Pan files in my data-integration folder

Ideally, I prefer to consolidate all the transformations in one main job file and execute the same using the kitchen component. However, it is important to understand the pan component as well.

Kitchen

Similar to pan, we can execute job files using the kitchen via CLI. Generally, it is the preferred option.

Similarly, we have different files for different OS.

Windows: Kitchen.bat - batch file

Kitchen.bat - batch file Linux: kitchen.sh - shell file

Kitchen files in my data-integration folder

We will be using the kitchen during our deployment process blog posts. In that blog post, I will let you know the various arguments that can be passed while executing the kitchen files.

Carte

Carte is a web server that allows us to run jobs and transformations remotely. I got the below explanation from stack-overflow which defines the carte web-server.

Carte itself is just a jetty web server, which starts and listens to incoming XML. Remember, PDI jobs or transformations — just XML. It can be the whole XML— that means the whole transformation will be run on Carte, or a part of transformation (remote steps or remote transformations). Anyway — it runs a java process that is waiting for XML metadata on how to extract transform and load.

In brief, it monitors for the transformation XML files and executes the same. Different files for different OS.

Windows: Carte.bat — batch file

Carte.bat — batch file Linux: carte.sh — shell file

Carte file in data-integration folder

We will understand the carte web-server in detail in the coming blog posts.

Conclusion

There are various components of PDI and easily the most important one is Spoon; which enables us to build transformations and jobs. However, we have just understood the theory part of these components and in the next few blog posts we will dig deeper. We will understand a few basic concepts before implementing an actual data ingestion pipeline using PDI.

See you in the next post. Happy ETL