# Quick start: Refine data #

You can save data preparation time by quickly transforming large amounts of raw data into consumable, high\-quality information that is ready for analytics\. Read about the Data Refinery tool, then watch a video and take a tutorial that’s suitable for beginners and does not require coding\.

Your basic workflow includes these tasks:

<!-- <ol> -->

1.  Open your sandbox project\. Projects are where you can collaborate with others to work with data\.
2.  Add your data to the project\. You can add CSV files or data from a remote data source through a connection\.
3.  Open the data in Data Refinery\.
4.  Perform steps using operations to refine the data\.
5.  Create and run a job to transform the data\.

<!-- </ol> -->

## Read about Data Refinery ##

Use Data Refinery to cleanse and shape tabular data with a graphical flow editor\. You can also use interactive templates to code operations, functions, and logical operators\. When you *cleanse data*, you fix or remove data that is incorrect, incomplete, improperly formatted, or duplicated\. When you *shape data*, you customize it by filtering, sorting, combining or removing columns, and performing operations\.

You create a Data Refinery flow as a set of ordered operations on data\. Data Refinery includes a graphical interface to profile your data to validate it and over 20 customizable charts that give you perspective and insights into your data\. When you save the refined data set, you typically load it to a different location than where you read it from\. In this way, your source data remains untouched by the refinement process\.

[Read more about refining data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)

## Watch a video about refining data ##

![Watch Video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) Watch this video to see how to refine data\.

This video provides a visual method to learn the concepts and tasks in this documentation\.

## Try a tutorial to refine data ##

In this tutorial, you will complete these tasks:

<!-- <ul> -->

 *  [Task 1: Open a project\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step01)
 *  [Task 2: Open the data set in Data Refinery\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step02)
 *  [Task 3: Review the data with Profile and Visualizations\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step03)
 *  [Task 4: Refine the data\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step04)
 *  [Task 5: Run a job for the Data Refinery flow\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step05)
 *  [Task 6: Create another data asset from the Data Refinery flow\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step06)
 *  [Task 7: View the data assets and your Data Refinery flow in your project\.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#step07)

<!-- </ul> -->

This tutorial will take approximately 30 minutes to complete\.

Expand all sections

<!-- <ul> -->

 *  Tips for completing this tutorial
    
    \#\#\# Use the video picture-in-picture Tip: Start the video, then as you scroll through the tutorial, the video moves to picture-in-picture mode. Close the video table of contents for the best experience with picture-in-picture. You can use picture-in-picture mode so you can follow the video as you complete the tasks in this tutorial. Click the timestamps for each task to follow along.The following animated image shows how to use the video picture-in-picture and table of contents features: ![How to use picture-in-picture and chapters](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/pip-and-chapters.gif)\{: width="560px" height="315px" data-tearsheet="this"\} \#\#\# Get help in the community If you need help with this tutorial, you can ask a question or find an answer in the [Cloud Pak for Data Community discussion forum](https://community.ibm.com/community/user/cloudpakfordata/communities/community-home/digestviewer?communitykey=c0c16ff2-10ef-4b50-ae4c-57d769937235)\{: new\_window\}. \#\#\# Set up your browser windows For the optimal experience completing this tutorial, open Cloud Pak for Data in one browser window, and keep this tutorial page open in another browser window to switch easily between the two applications. Consider arranging the two browser windows side-by-side to make it easier to follow along. ![Side-by-side tutorial and UI](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/tutorial-side-by-side.png)\{: width="560px" height="315px" data-tearsheet="this"\} Tip: If you encounter a guided tour while completing this tutorial in the user interface, click **Maybe later**.  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 1: Open a project
    
    You need a project to store the data and the Data Refinery flow. You can use your sandbox project or create a project. 1. From the navigation menu ![Navigation menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/navigation-menu.svg)\{: iih\}, choose **Projects > View all projects** 1. Open your sandbox project. If you want to use a new project: 1. Click **New project**. 1. Select **Create an empty project**. 1. Enter a name and optional description for the project. 1. Choose an existing [object storage service instance](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/storage-options.html)\{: new\_window\} or create a new one. 1. Click **Create**. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows a new, empty project.  
      
    ![The following image shows a new, empty project.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-new-project.png)\{: width="100%" \} For more information or to watch a video, see [Creating a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html)\{: new\_window\}.  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 2: Open the data set in Data Refinery
    
    ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 00:05. Follow these steps to add a data asset to your project and create a Data Refinery flow. The data set you will use in this tutorial is available in the Samples. 1. Access the [Airline data](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/8fa07e57e69f7d0cb970c86c6ae52d41)\{: new\_window\} in the Samples. 1. Click **Add to project**. 1. Select your project from the list, and click **Add**. 1. After the data set is added, click **View Project**. For more information on adding a data asset from the Samples to a project, see [Loading and accessing data in a notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/load-and-access-data.html). 1. On the *Assets* tab, click the **airline-data.csv** data asset to preview its content. 1. Click **Prepare data** to open a sample of the file in Data Refinery, and wait until Data Refinery reads and processes a sample of the data. 1. Close the **Information** and **Steps** panels. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the airline data asset open in Data Refinery.  
      
    ![The following image shows the airline data asset open in Data Refinery.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-asset-open-in-dr.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 3: Review the data with Profile and Visualizations
    
    ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 00:47. IBM Knowledge Catalog automatically profiles and classifies the content of an asset based on the values in those columns. Follow these steps to use the Profile and Visualizations tabs to explore the data. Tip: Use the Profile and Visualizations pages to view changes in the data as you refine it.1. Click the **Profile** tab to review the [frequency distribution](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/metrics.html)\{: new\_window\} of the data so that you can find the outliers. 1. Scroll through the columns to the see the statistics for each column. The statistics show the interquartile range, minimum, maximum, median and standard deviation in each column. 1. Hover over a bar to see additional details. The following image shows the Profile tab:  
    ![Profile tab](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/airline-profile-integer.png) 1. Click the **Visualizations** tab. 1. Select the **UniqueCarrier** column to visualize. Suggested [charts](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/visualizations.html)\{: new\_window\} have a blue dot next to their icons. 1. Click the **Pie** chart. Use the different perspectives available in the charts to identify patterns, connections, and relationships within the data. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the Visualizations tab. You are now ready to refine the data.  
      
    ![Visualizations tab](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/airline-viz.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 4: Refine the data
    
    \#\#\# Data Refinery operations Data Refinery uses two kinds of operations to refine data, *GUI operations* and *coding operations*. You will use both kinds of operations in this tutorial. - [GUI operations](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/gui_operations.html)\{: new\_window\} can consist of multiple steps. Select an operation from **New step**. A subset of the GUI operations is also available from each column's overflow menu (![overflow menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/overflow-menu--vertical.svg)\{: iih\}). When you open a file in Data Refinery, the **Convert column type** operation is automatically applied as the first step to convert any non-string data types to inferred data types (for example, to Integer, Date, Boolean, etc.). You can undo or edit this step. - [Coding operations](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/code_operations.html)\{: new\_window\} are interactive templates for coding operations, functions, and logical operators. Most of the operations have interactive help. Click the operation name in the command-line text box to see the coding operations and their syntax options. ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 01:16. Refining data is a series of steps to build a *Data Refinery flow*. As you go through this task, view the **Steps** panel to follow your progress. You can select a step to delete or edit it. If you make a mistake, you can also click the Undo icon ![Undo icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/undo.png)\{: iih\}. Follow these steps to refine the data: 1. Go back to the **Data** tab. 1. Select the *Year* column. Click the **Overflow** menu (![Overflow menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/overflow-menu--vertical.svg)\{: iih\}) and choose **Sort descending**. 1. Click **Steps** to see the new step in the *Steps* panel. 1. Focus on the delays for a specific airline. This tutorial uses United Airlines (UA), but you can choose any airline. 1. Click **New step**, and then choose the GUI operation **Filter**. 1. Choose the **UniqueCarrier** column. 1. For *Operator*, choose **Is equal to**. 1. For *Value*, type the string for the airline for which you want to see delay information. For example, `UA`\{: .cp\}.  
    ![Filter operation](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/filter.png) 1. Click **Apply**. Scroll to the *UniqueCarrier* column to see the results. 1. Create a new column that adds the arrival and departure delay times together. 1. Select the **DepDelay** column. 1. Notice that the *Convert column type* operation was automatically applied as the first step to convert the String data types in all the columns whose values are numbers to Integer data types. 1. Click **New step**, and then choose the GUI operation **Calculate**. 1. For *Operator*, choose **Addition**. 1. Select **Column**, and then choose the **ArrDelay** column. 1. Select **Create new column for results**. 1. For *New column name*, type `TotalDelay`\{: .cp\}.  
    ![Calculate operation](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/calculate.png) 1. You can position the new column at the end of the list of columns or next to the original column. In this case, select **Next to original column**. 1. Click **Apply**. The new column, *TotalDelay*, is added. 1. Move the new *TotalDelay* column to the beginning of the data set: 1. In the command-line text box, choose the **select** operation. 1. Click the word **select**, and then choose **select(\``<column>`\`, everything())**. 1. Click **`` `<column>` ``**, and then choose the **TotalDelay** column. When you finish, the command should look like this: ``select(`TotalDelay`, everything())`` 1. Click **Apply**. The *TotalDelay* column is now the first column. 1. Reduce the data to four columns: *Year*, *Month*, *DayofMonth*, and *TotalDelay*. Use the **group\_by** coding operation to divide the columns into groups of year, month, and day. 1. In the command-line text box, choose the **group\_by** operation. 1. Click **`<column>`**, and then choose the **Year** column. 1. Before the closing parenthesis, type: `,Month,DayofMonth`\{: .cp\}. When you finish, the command should look like this: ``group_by(`Year`,Month,DayofMonth)`` 1. Click **Apply**. 1. Use the **select** coding operation for the *TotalDelay* column. In the command-line text box, select the **select** operation.  
    Click **`<column>`**, and choose the **TotalDelay** column. The command should look like this: ``select(`TotalDelay`)`` 1. Click **Apply**. The shaped data now consists of the *Year*, *Month*, *DayofMonth*, and *TotalDelay* columns. The following screen image shows the first four rows of the data.  
    ![The first four rows of the Data Refinery flow with the Year, Month, DayofMonth, and TotalDelay columns](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/four_columns_with_totaldelay.png) 1. Show the mean of the values of the *TotalDelay* column, and create a new *AverageDelay* column: 1. Click **New step**, and then choose the GUI operation **Aggregate**. 1. For the *Column*, select **TotalDelay**. 1. For *Operator*, select **Mean**. 1. For *Name of the aggregated column*, type `AverageDelay`\{: .cp\}.  
    ![Aggregate operation](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/aggregate.png)\{: height="500px"\} 1. Click **Apply**. The new column *AverageDelay* is the average of all the delay times. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the first four rows of the data.  
      
    ![The following screen image shows the first four rows of the data.](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/four_columns_with_delay.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 5: Run a job for the Data Refinery flow
    
    ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 04:16. When you run a job for the Data Refinery flow, the steps are run on the entire data set. You select the runtime and add a one-time or repeating schedule. The output of the Data Refinery flow is added to the data assets in the project. Follow these steps to run a job to create the refined data set. 1. From the Data Refinery toolbar, click the **Jobs** icon, and select **Save and create a job**.  
    ![Save and create a job](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/save-create-job.png) 1. Type a name and description for the job, and click **Next**. 1. Select a runtime environment, and click **Next**. 1. (Optional) Click the toggle button to schedule a run. Specify the date, time and if you would like the job to repeat, and click **Next**. 1. (Optional) Turn on notifications for this job, and click **Next**. 1. Review the details, and click **Create and run** to run the job immediately.  
    ![create job](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/create-run.png) 1. When the job is created, click the **job details** link in the notification to view the job in your project. Alternatively, you can navigate to the **Jobs** tab in the project, and click the job name to open it. 1. When the *Status* for the job is **Completed**, use the project navigation trail to navigate back to the *Assets* tab in the project. 1. Click the **Data > Data assets** section to see the output of the Data Refinery flow, *airline-data\_shaped.csv*. 1. Click the **Flows > Data Refinery flows** section to see the Data Refinery flow, *airline-data.csv\_flow*. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the Assets tab with the Data Refinery flow and shaped asset.  
      
    ![The following image shows the Assets tab with the Data Refinery flow and shaped asset.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-project-shaped-data.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 6: Create another data asset from the Data Refinery flow
    
    ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 05:26. Follow these steps to further refine the data set by editing the Data Refinery flow: 1. Click **airline-data.csv\_flow** to open the flow in Data Refinery. 1. Sort the *AverageDelay* column in descending order. 1. Select the **AverageDelay** column. 1. Click the column **Overflow** menu (![Overflow menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/overflow-menu--vertical.svg)\{: iih\}), and then select **Sort descending**. 1. Click the **Flow settings** icon ![Flow settings icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/settings.png)\{: iih\}. 1. Click the **Target data set** panel. 1. Click **Edit properties**. 1. In the *Format target properties* dialog, change the data asset name to `airline-data_sorted_shaped.csv`\{: .cp\}.  
    ![changed output file name](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/output_changed-new.png) 1. Click **Save** to return to the Flow settings. 1. Click **Apply** to save the settings. 1. From the Data Refinery toolbar, click the **Jobs** icon and select **Save and view jobs**.  
    ![Save and view jobs](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/save-view-jobs.png) 1. Select the job for the airline data, and then click **View**. 1. From the *Job window* toolbar, click the **Run job** icon.  
    ![Run jobs icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/run-job.png) \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the completed job details.  
      
    ![The following image shows the completed job details.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-completed-job.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

<!-- <ul> -->

 *  Task 7: View the data assets and your Data Refinery flow in your project
    
    ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 06:40. Now follow these steps to view the three data assets, the original, the first refined data set, and the second refined data set: 1. When the job completes, go to the project page. 1. Click the **Assets tab**. 1. In the *Data assets* section, you will see the original data set that you uploaded and the output of the two Data Refinery flows. *`airline-data_sorted_shaped.csv`*`airline-data_csv_shaped`*`airline-data.csv` 1. Click the **airline-data\_csv\_shaped** data asset to see the mean delay unsorted. Navigate back to the* Assets *tab. 1. Click **airline-data\_sorted\_shaped.csv** data asset to see the mean delay sorted in descending order. Navigate back to the* Assets *tab. 1. Click the \*Flows > Data Refinery flows* section shows the Data Refinery flow: `airline-data.csv_flow`. \#\#\# ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg)\{: iih\} Check your progress The following image shows the Assets tab with all of the assets displayed.  
      
    ![The following image shows the Assets tab with all of the assets displayed.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-final-assets-tab.png)\{: width="100%" \}  
    [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=en#video-preview)  
    
<!-- </ul> -->

## Next steps ##

Now the data is ready to be used\. For example, you or other users can do any of these tasks:

<!-- <ul> -->

 *  [Analyze the data in a Jupyter notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-analyze.html)
 *  [Build and train a model with the data asset](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-build.html)

<!-- </ul> -->

## Additional resources ##

<!-- <ul> -->

 *  View more [videos](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/videos-wx.html)\.
 *  Find sample data sets, projects, models, prompts, and notebooks in the Samples to gain hands\-on experience:
    
    ![Notebook icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/notebook.svg)[Notebooks](https://dataplatform.cloud.ibm.com/gallery?context=wx&format=notebook) that you can add to your project to get started analyzing data and building models.
    
    ![Project icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/ibm-cloud--projects.svg)[Projects](https://dataplatform.cloud.ibm.com/gallery?context=wx&format=project-template) that you can import containing notebooks, data sets, prompts, and other assets.
    
    ![Data set icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/data--set.svg)[Data sets](https://dataplatform.cloud.ibm.com/gallery?context=wx&format=dataset) that you can add to your project to refine, analyze, and build models.
    
    ![Prompt icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/prompt.svg)[Prompts](https://dataplatform.cloud.ibm.com/gallery?context=wx&format=example-prompt) that you can use in the Prompt Lab to prompt a foundation model.
    
    ![Model icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/model.svg)[Foundation models](https://dataplatform.cloud.ibm.com/gallery?context=wx&format=foundation-model) that you can use in the Prompt Lab.

<!-- </ul> -->

**Parent topic:**[Quick start tutorials](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/quickstart-tutorials.html)

<!-- </article "role="article" "> -->