Data Science Methodology
By: Emeka Boris
Tags: Deep Learning, Artificial Intelligence, Machine Learning, Data Visualization, Data Science

In this session, we will learn techniques and best approaches to scale in your Data Science project.

Data Science Methodology

In the domain of data science, solving problems and answering

questions through data analysis are standard practice. Often,

data scientists build a model to predict outcomes or

discover underlying patterns, with the goal of gaining insights.

Organizations can then use these insights to take actions and make a decision

ideally, improve future outcomes. They are various tools and technologies for building and deploying models. The volume of structured, unstructured and semi-structured data is rapidly increasing.

A 10-stage data science methodology that spans technologies and approaches

As data analytics inclinations become more open and widespread, data scientists need a foundational methodology competent enough to provide a strategy, despite the technologies, data volumes or approaches involved. The methodology consists of 10 stages that form a loop process for using data to reveal insights. Each stage plays an important role in setting the overall methodology.

What is a methodology? A methodology is a comprehensive approach that leads the processes and exercises within a given domain. Methodology does not depend on particular technologies or tools, nor is it a set of techniques or recipes. Rather, a methodology provides the data scientist with a framework for how to proceed with whatever methods, processes, and heuristics will be used to obtain answers or results.

Stage 1: Business Understanding

Every project begins with a business understanding. The business advocates who need the analytic solution which plays the most significant part in this stage by describing the dilemma or problem, project objectives and solution requirements from a business view. This first stage lays the foundation for a successful interpretation of the business problem. To help ensure the project’s resolution, the advocate should be involved throughout the project to provide domain expertise, review intermediate findings and ensure the work remains on track to generate the intended solution.

Stage 2: Analytic approach

More than anything else, an analytical approach is the use of an appropriate process to break a problem down into the smaller pieces necessary to solve it. Each piece becomes a smaller and easier problem to solve. Once the business problem has been clearly stated, the data scientist can define the analytic approach to solving the problem. This stage entails expressing the problem in the context of statistical and machine-learning techniques, so the organization can identify the most proper ones for the aspired result. For example, if the goal is to predict a response such as “yes” or “no,” then the analytic approach could be defined as a building, testing and implementing a classification model.

Stage 3: Data requirements

The first move in the data requirement is to determine what data is required for building the model. This should be dictated primarily by the scope and level of detail required to achieve the model objectives as described earlier. The chosen analytic approach determines the data requirements. Specifically, the analytic methods to be used require certain data content, formats, and representations, guided by domain expertise.

Stage 4: Data collection

is the process of gathering and measuring information on variables of interest, in an established systematic procedure that qualifies one to answer stated research questions, test hypotheses, and evaluate outcomes. In the initial data collection stage, data scientists recognize and collect the possible data resources — structured, unstructured and semi-structured — relevant to the problem domain. Typically, they must choose whether to make additional investments to obtain less-accessible data elements. It may be best to defer the investment decision until more is known about the data and the model. If there are gaps in data collection, the data scientist may have to revise the data requirements accordingly and collect new and/or more data. While data sampling and subsetting are still important.

Stage 5: Data understanding

The main goal of data understanding is to gain general insights about the data that will potentially be helpful for the further steps in the data analysis process, but data understanding should not be driven exclusively by the goals and methods to be applied in later steps.After the original data collection, data scientists typically use descriptive statistics and visualization techniques to understand the data content, assess data quality and discover initial insights about the data. Additional data collection may be necessary to fill gaps.

Stage 6: Data preparation

This stage encompasses all activities to construct the data set that will be used in the subsequent modeling stage. Data preparation activities include data cleaning (dealing with missing or invalid values, eliminating duplicates, formatting properly), combining data from multiple sources (files, tables, platforms) and transforming data into more useful variables. In a process called feature engineering, data scientists can create additional explanatory variables, also referred to as predictors or features, through a combination of domain knowledge and existing structured variables. When text data is available, such as customer call center logs or physicians’ notes in unstructured or semi-structured form, text analytics is useful in deriving new structured variables to enrich the set of predictors and improve model accuracy. Data preparation is usually the most time-consuming step in a data science project. In many domains, some data preparation steps are common across different problems. Automating certain data preparation steps in advance may accelerate the process by minimizing ad hoc preparation time. With today’s high-performance, massively parallel systems and analytic functionality residing where the data is stored, data scientists can more easily and rapidly prepare data using very large data sets.

Stage 7: Modeling

Starting with the first version of the prepared data set, the modeling stage focuses on developing predictive or descriptive models according to the previously defined analytic approach. With predictive models, data scientists use a training set (historical data in which the outcome of interest is known) to build the model. The modeling process is typically highly iterative as organizations gain intermediate insights, leading to refinements in data preparation and model specification. For a given technique, data scientists may try multiple algorithms with their respective parameters to find the best model for the available variables.

Stage 8: Evaluation

During model development and before deployment, the data scientist evaluates the model to understand its quality and ensure that it properly and fully addresses the business problem. Model evaluation entails computing various diagnostic measures and other outputs such as tables and graphs, enabling the data scientist to interpret the model’s quality and its efficacy in solving the problem. For a predictive model, data scientists use a testing set, which is independent of the training set but follows the same probability distribution and has a known outcome. The testing set is used to evaluate the model so it can be refined as needed. Sometimes the final model is applied also to a validation set for a final assessment. In addition, data scientists may assign statistical significance tests to the model as further proof of its quality. This additional proof may be instrumental in justifying model implementation or taking actions when the stakes are high — such as an expensive supplemental medical protocol or a critical airplane flight system.

Stage 9: Deployment

Once a satisfactory model has been developed and is approved by the business sponsors, it is deployed into the production environment or a comparable test environment. Usually it is deployed in a limited way until its performance has been fully evaluated. Deployment may be as simple as generating a report with recommendations, or as involved as embedding the model in a complex workflow and scoring process managed by a custom application. Deploying a model into an operational business process usually involves additional groups, skills and technologies from within the enterprise. For example, a sales group may deploy a response propensity model through a campaign management process created by a development team and administered by a marketing group.

Stage 10: Feedback

By collecting results from the implemented model, the organization gets feedback on the model’s performance and its impact on the environment in which it was deployed. For example, feedback could take the form of response rates to a promotional campaign targeting a group of customers identified by the model as high-potential responders. Analyzing this feedback enables data scientists to refine the model to improve its accuracy and usefulness. They can automate some or all of the feedback-gathering and model assessment, refinement and redeployment steps to speed up the process of model refreshing for better outcomes.

Now you have a clear perception of Data Methodology, we have come to the conclusion of this lesson.

At the end of this lesson, we have been able to understand:

1. Data Methodology

When you done with this session kindly share your progress on your social media account

Hints

Day 11–12: Data Methodology

Day 11–12 Lesson: Today we learned Data Methodology which gave me an insight on the algorithmic approach on starting a Data Science Project.

#100daysofcode #100daysofDscode #100days #Day11–12 #DataScience #MachineLearning #Ai #Probability