Quick start: Refine data
Quick start: Refine data You can save data preparation time by quickly transforming large amounts of raw data into consumable, high-quality information that is ready for analytics. Read about the Data Refinery tool, then watch a video and take a tutorial that’s suitable for beginners and does not require coding. Your basic workflow includes these tasks: 1. Open your sandbox project. Projects are where you can collaborate with others to work with data. 2. Add your data to the project. You can add CSV files or data from a remote data source through a connection. 3. Open the data in Data Refinery. 4. Perform steps using operations to refine the data. 5. Create and run a job to transform the data. Read about Data Refinery Use Data Refinery to cleanse and shape tabular data with a graphical flow editor. You can also use interactive templates to code operations, functions, and logical operators. When you cleanse data, you fix or remove data that is incorrect, incomplete, improperly formatted, or duplicated. When you shape data, you customize it by filtering, sorting, combining or removing columns, and performing operations. You create a Data Refinery flow as a set of ordered operations on data. Data Refinery includes a graphical interface to profile your data to validate it and over 20 customizable charts that give you perspective and insights into your data. When you save the refined data set, you typically load it to a different location than where you read it from. In this way, your source data remains untouched by the refinement process. [Read more about refining data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html) Watch a video about refining data ![Watch Video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) Watch this video to see how to refine data. This video provides a visual method to learn the concepts and tasks in this documentation. Try a tutorial to refine data In this tutorial, you will complete these tasks: * [Task 1: Open a project.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep01) * [Task 2: Open the data set in Data Refinery.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep02) * [Task 3: Review the data with Profile and Visualizations.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep03) * [Task 4: Refine the data.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep04) * [Task 5: Run a job for the Data Refinery flow.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep05) * [Task 6: Create another data asset from the Data Refinery flow.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep06) * [Task 7: View the data assets and your Data Refinery flow in your project.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=enstep07) This tutorial will take approximately 30 minutes to complete. Expand all sections * Tips for completing this tutorial ### Use the video picture-in-picture Tip: Start the video, then as you scroll through the tutorial, the video moves to picture-in-picture mode. Close the video table of contents for the best experience with picture-in-picture. You can use picture-in-picture mode so you can follow the video as you complete the tasks in this tutorial. Click the timestamps for each task to follow along.The following animated image shows how to use the video picture-in-picture and table of contents features: ![How to use picture-in-picture and chapters](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/pip-and-chapters.gif){: width="560px" height="315px" data-tearsheet="this"} ### Get help in the community If you need help with this tutorial, you can ask a question or find an answer in the [Cloud Pak for Data Community discussion forum](https://community.ibm.com/community/user/cloudpakfordata/communities/community-home/digestviewer?communitykey=c0c16ff2-10ef-4b50-ae4c-57d769937235){: new_window}. ### Set up your browser windows For the optimal experience completing this tutorial, open Cloud Pak for Data in one browser window, and keep this tutorial page open in another browser window to switch easily between the two applications. Consider arranging the two browser windows side-by-side to make it easier to follow along. ![Side-by-side tutorial and UI](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/tutorial-side-by-side.png){: width="560px" height="315px" data-tearsheet="this"} Tip: If you encounter a guided tour while completing this tutorial in the user interface, click Maybe later. [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=envideo-preview) * Task 1: Open a project You need a project to store the data and the Data Refinery flow. You can use your sandbox project or create a project. 1. From the navigation menu ![Navigation menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/navigation-menu.svg){: iih}, choose Projects > View all projects 1. Open your sandbox project. If you want to use a new project: 1. Click New project. 1. Select Create an empty project. 1. Enter a name and optional description for the project. 1. Choose an existing [object storage service instance](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/storage-options.html){: new_window} or create a new one. 1. Click Create. ### ![Checkpoint icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/checkmark--filled-blue.svg){: iih} Check your progress The following image shows a new, empty project. ![The following image shows a new, empty project.](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/dr-new-project.png){: width="100%" } For more information or to watch a video, see [Creating a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html){: new_window}. [Back to the top](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-refine.html?context=cdpaas&locale=envideo-preview) * Task 2: Open the data set in Data Refinery ![preview tutorial video](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/lc-video.png) To preview this task, watch the video beginning at 00:05. Follow these steps to add a data asset to your project and create a Data Refinery flow. The data set you will use in this tutorial is available in the Samples. 1. Access the [Airline data](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/8fa07e57e69f7d0cb970c86c6ae52d41){: new_window} in the Samples. 1. Click Add