—-> software enginerring involves applying “engineering principles” to software in oder to make it more scalable , reliable , and efficient 
—> these same prinicla can be applied to make data science application have improved reliability , able to scale to larger datasets and more efficient 
These principles include well-structured code, which helps make debugging and collaboration much easier. They also involve dealing with larger and larger datasets in a world where data is continually growing. Depending on your data science application, engineering principles can also be applied to put your code into production (such as a model that can run on millions - even billions - of observations).
A typical data science workflow involves several steps, including gathering data, data cleaning, and model training / evaluation. There are several key components of software engineering that can enhance the data science workflow. These include exception handling, source control, object-oriented programmin
g (OOP), and scale (among others).
A data pipeline is a set of actions to process data. Processing data means any action taken to a collection of data such as the following: Extracting the data from some source (database, webpage, set of documents, etc.) Merging and aggregating features (i.e. columns) of data from different sources (e.g. customer-level vs. transaction-level data) 1.3 What are the components of a data pipeline? Logging a model’s predictions to a new table ©Manning Publications Co. To comment go to liveBook https://livebook.manning.com/book/software-engineering-for-data-scientists/discussion 15 In the workplace, data pipelines are often created by data engineers. However, depending on a company’s structure, data scientists may also write data pipelines. In general, if you’re a data scientist, it’s recommended to know about data pipelines and understand the source of the data you’re using for your modeling or analysis projects. This is important for several reasons, but especially because it helps you to build confidence in knowing where the data you’re using is coming from. Software engineers can also be involved in data pipelines. For example, at tech companies, it is common for software engineers to write the code that logs data from a particular web application (a simple example would be logging whether someone clicks on an ad). Data pipelines are necessary because they ensure that data is served reliably for a variety of data science applications. Even if you’re working on an insight analysis, rather than a model, for example, you need to be able to retrieve reliable data. Data pipelines are used in these cases to bring data from between different tables and sources. They can also be used for logging data, such as storing predictions from a model. Sometimes you may see data pipelines referred in the context of reporting and analytics, as well, in addition to being used for machine learning models. The main summary point to keep in mind is that data pipelines are ultimately used to flow data from a source (or collection of sources) into finalized outputs, usually in the form of structured tables with collections of rows and columns. The exact application of ultimately using the data can vary. Let’s give a real-world example of a data pipeline.

Q- 



Total amount of transactions over last 3 months 
Number of times customer logged onto the e-commerce website 
Length of time since customer first registered 
Number of times customer has deactivated account in last 12 months 



Transaction : entity 
CustomerID: 123 
TransactionID: 999
Transaction amount: 100 
Number of items purcahrsd : 4 


Login : 
CustomerID : int 
Login_timestamp : 2022-04-27 
Landing_page: example.ecommerce.com 



Customer information table (dim_customers) 

CustomerId : 123 
Intial_regesitration_timesap : 2022-02-2015:30:00 
Age : 34 
Desctivation timestamp : NULL 

Q- using this enties try to infer sysntatique examples of observation for eaach table