英語で技術書を読もう：Fundamentals of Data Engineering 輪読会資料 まとめ

# 概要
対象書籍を翻訳して輪読する勉強会です。

# 対象書籍
- Fundamentals of Data Engineering (O'Reilly, 2022/07)
- https://www.amazon.co.jp/dp/1098108302/
![スクリーンショット 2023-08-30 13.42.53.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/295123/88463110-144a-45a1-4f97-cbd36bbcd219.png)


# 第00回 Preface
- 2023/04/03
- connpass:https://gaisaba.connpass.com/event/279517/
- 資料:https://qiita.com/Shumpei_Kubo/items/9cb9145b4b695b3b5346
- 箇所：Preface


# 第01回 Chapter3:Good Data Architecture
- 2023/04/10
- connpass:https://gaisaba.connpass.com/event/279986/
- 資料:https://qiita.com/Shumpei_Kubo/items/44ba187698d3799372a2
- 箇所：
    - Chapter3:
        - Good Data Architecture
            - 「Principles of Good Data Architecture」


# 第02回 Chapter3:Tight vs Loose Coupling: Tiers, Monoliths, and Microservices
- 2023/04/17
- connpass:https://gaisaba.connpass.com/event/280778/
- 資料:https://qiita.com/hikarumosity/items/38d8e713b8203645cf7c
- 箇所：
    - Chapter3
        - Tight vs Loose Coupling: Tiers, Monoliths, and Microservices
        - （User Access: Single Versus Multitenant の手前まで）


# 第03回 Chapter3:「User Access: Single Versus Multitenant」「EventDriven Architecture」
- 2023/04/27
- connpass:https://gaisaba.connpass.com/event/281425/
- 資料:
    - https://qiita.com/Shumpei_Kubo/items/2be018ee15ee6e6aff5d
- 箇所：
    - Chapter3
        - 「User Access: Single Versus Multitenant」
        - 「EventDriven Architecture」


# 第04回  Chapter3:「User Access: Single Versus Multitenant」「EventDriven Architecture」
- 2023/05/08
- connpass:https://gaisaba.connpass.com/event/282438/
- 資料:https://cellardoor.hatenablog.jp/entry/fundamentalsofdataengineering-part4/
- 箇所：
    - Chapter3
        - User Access : Single versus Multitenant
        - Event-Driven Architecture


# 第05回 Chapter3:Convergence, Next-Generation Data Lakes, and the Data Platform etc...
- 2023/05/15
- connpass:https://gaisaba.connpass.com/event/283380/
- 資料:https://techblog.otakashi.jp/archives/301
- 箇所：
    - Chapter3
        - Convergence, Next-Generation Data Lakes, and the Data Platform
        - Modern Data Stack
        - Lambda Architecture
        - Kappa Architecture
        - The Dataflow Model and Unified Batch and Streaming


# 第06回 Chapter3:Data Mesh～
- 2023/05/22
- connpass:https://gaisaba.connpass.com/event/284114/
- 資料:https://afoil.notion.site/Fundamentals-of-Data-Engineering-6-ac601910f1b4414eb95749da58be9604
- 箇所：
    - Chapter3
        - Data Mesh から Chapter 3 の最後まで（参考文献の一覧は不要）


# 第07回　Chapter4:Choosing Technologies Across the Data Engineering Lifecycle 
- 2023/05/29
- connpass:https://gaisaba.connpass.com/event/285167/
- 資料:https://qiita.com/mongolmongol/private/463e7c57879d0c8ea4d6
- 箇所：
    - Chapter4
        - Choosing Technologies Across the Data Engineering Lifecycle より
            - Cost Optimization and Business Value
                - Total Cost Of Ownership
                - Total Opportunity Cost of Ownership
                - FinOps


# 第08回　Chapter4:Choosing Technologies Across the Data Engineering Lifecycle 
- 2023/06/05
- connpass:https://gaisaba.connpass.com/event/285671/
- 資料:https://qiita.com/Shumpei_Kubo/items/6543b3c2e7a11d31152b
- 箇所：
    - Chapter4
        - Choosing Technologies Across the Data Engineering Lifecycle より
            - Today Versus the Future: Immutable Versus Transitory Technologies
            - Our Advice の終わり、Location の手前まで


# 第09回　Chapter4:Choosing Technologies Across the Data Engineering Lifecycle 
- 2023/06/12
- connpass:https://gaisaba.connpass.com/event/286435/
- 資料:https://cellardoor.hatenablog.jp/entry/fundamentalsofdataengineering-part9/
- 箇所：
    - Chapter4
        - Choosing Technologies Across the Data Engineering Lifecycle より
            - Undercurrents and Their Impacts on Choosing Technologies から Chapter 4 の終わりまで




# 第10回 Chapter2:The Data Engineering Lifecycle
- 2023/06/19
- connpass:https://gaisaba.connpass.com/event/287143/
- 資料:https://afoil.notion.site/Fundamentals-of-Data-Engineering-10-02de3e92a5c5447497a10cb541b7f20d
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Chapter2の最初から、Storageの手前まで


# 第11回　Chapter2:The Data Engineering Lifecycle
- 2023/06/26
- connpass:https://gaisaba.connpass.com/event/287901/
- 資料:
    - Ingestion
        - https://techblog.otakashi.jp/archives/359
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle より
            - Storage(ホイル焼きさん担当)
            - Ingestion（okamotoさん担当）


# 第12回　Chapter2:The Data Engineering Lifecycle
- 2023/07/03
- connpass:https://gaisaba.connpass.com/event/288576/
- 資料:https://qiita.com/hikarumosity/items/a82757627c5aa36a37d9
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Transformation ～ Serving Data の Embedded analyticsまで
            - （Machine Learning の手前まで）


# 第13回　Chapter2:The Data Engineering Lifecycle
- 2023/07/10
- connpass:https://gaisaba.connpass.com/event/289254/
- 資料:
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Serving Data より
                - Machine Learning
                - Reverse ETL


# 第14回 Chapter2:The Data Engineering Lifecycle
- 2023/07/18
- connpass:https://gaisaba.connpass.com/event/290023/
- 資料:https://qiita.com/ak-sakatoku/items/286f92bc911e1e981c69
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Serving Data より
                - Major Undercurrents Across the Data Engineering Lifecycle
                → Data governance の手前まで


# 第15回 Chapter2:The Data Engineering Lifecycle
- 2023/07/24
- connpass:https://gaisaba.connpass.com/event/290705/
- 資料:https://qiita.com/Shumpei_Kubo/items/5dcc5a0ff63ade09c22d
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Major Undercurrents Across the Data Engineering Lifecycle より
            - Data Management 配下の、以下の箇所
                - Data governance
                - Discoverability
                - Metadata
                → Data accountability の手前まで


# 第16回 Chapter2:The Data Engineering Lifecycle
- 2023/07/31
- connpass:https://gaisaba.connpass.com/event/291256/
- 資料:https://www.slideshare.net/secret/4JBetounXOoNAU
- 箇所
    - Chapter2
        - The Data Engineering Lifecycle
            - Major Undercurrents Across the Data Engineering Lifecycle より
                - Data Management 配下の、以下の箇所 (途中のMaster Data Management も含む)
                    - Data accountability
                    - Data quality
                    - Data modeling and design
                    - Data lineage
                    - Data integration and interoperability
                    - Data lifecycle management
                    - Ethics and privacy
                    → DataOps の手前まで


# 第17回 Chapter2:The Data Engineering Lifecycle
- 2023/08/21
- connpass:https://gaisaba.connpass.com/event/291970/
- 資料:https://qiita.com/ak-sakatoku/items/baf5c130c67d6b3c9fef
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Major Undercurrents Across the Data Engineering Lifecycle より
                - DataOps
                - Data Architecture
                (Orchestration の手前まで)


# 第18回 Chapter2:The Data Engineering Lifecycle
- 2023/09/04
- https://gaisaba.connpass.com/event/294111/
- 資料:https://qiita.com/Takamori_Sonoda/items/3bcb3ba379e1604ce596
- 箇所：
    - Chapter2
        - The Data Engineering Lifecycle
            - Major Undercurrents Across the Data Engineering Lifecycle より
                - Orchestration
                - Software Engineering
                - Conclusion
                - (Additional Resourcesは不要)


# 第19回 Chapter8:Queries, Modeling, and Transformation
- 2023/09/25
- https://gaisaba.connpass.com/event/295482/
- 資料:https://afoil.notion.site/Chapter-8-Queries-Modeling-and-Transformation-4e1283d95aa94741b30c2956e0896b80
- 箇所：
    - Chapter8 : Queries, Modeling, and Transformation
        - Queries
        - What is a Query?
        - The Life of a Query
        - The Query Optimizer
        (Improving Query Performance の手前まで)


# 第２０回 Chapter8:Queries, Modeling, and Transformation
- 2023/１０/16
- https://gaisaba.connpass.com/event/298222/
- 資料:https://qiita.com/kota9/items/0f1b7bc28df151159abe
- 箇所：
    - Chapter8 : Queries, Modeling, and Transformation
        - Improving Query Performance
        ("Queries on Streaming Data" の手前まで)

# 第２1回 Chapter8:Queries, Modeling, and Transformation
- 2023/１０/30
- https://gaisaba.connpass.com/event/299659/
- 資料:https://qiita.com/IQ_Bocchi/items/a86bf374ccfccb3e98ac
- 箇所：
    - Chapter8 : Queries, Modeling, and Transformation
        - Queries on Streaming Data　 (Data Modeling の手前まで）



# 第22回 Chapter8 Data Modeling
- 2023/11/13
- connpass : https://gaisaba.connpass.com/event/301330/
- 資料:https://afoil.notion.site/Fundamentals-of-Data-Engineering-22-7aca59c67d594dd1bee7f3b0bec11de6
- 箇所：
    - Chapter8 Data Modeling
        - (Techniques for Modeling Batch Analytical Data の手前まで)

        
# 第23回 Chapter8 Techniques for Modeling Batch Analytical Data
- 2023/11/27
- connpass : https://gaisaba.connpass.com/event/301330/
- 資料:https://techblog.otakashi.jp/archives/467
- 箇所：
    - Chapter8 Techniques for Modeling Batch Analytical Data
        - 冒頭
        - Inmon
        - Kimball
        - Fact tables
        - (Star schema の手前まで)


# 第24回 Chapter8 Techniques for Modeling Batch Analytical Data
- 2023/12/11
- connpass : https://gaisaba.connpass.com/event/304283/
- 資料: https://afoil.notion.site/Fundamentals-of-Data-Engineering-24-481cc0716ac54549bfad360b9a88356c
- 箇所：
    - Chapter8 Techniques for Modeling Batch Analytical Data
        - Star schema
        - Data Vault
        - Hubs
        - Links
        - Satellites
        - (Wide denormalized tables の手前まで)


# 第25回 Chapter8 Techniques for Modeling Batch Analytical Data
- 2024/01/15
- connpass : https://gaisaba.connpass.com/event/305037/
- 資料: https://qiita.com/Shumpei_Kubo/items/5efdf7db68fda12c71f4
- 箇所：
    - Chapter8 Techniques for Modeling Batch Analytical Data
        - Wide denormalized tables
        - Modeling Streaming Data
        - (Transformation の手前まで)



# 第26回 Chapter8 Techniques for Modeling Batch Analytical Data
- 2024/01/29
- connpass : https://gaisaba.connpass.com/event/307845/
- 資料: https://techblog.otakashi.jp/archives/523
- 箇所：
    - Chapter8 Techniques for Modeling Batch Analytical Data
        - Transformations
            - Batch Transformation
                - Broadcast join
                - Shuffle hash join
            - ETL, ELT, and data pipeline


# 第27回 Chapter8 Transformation
- 2024/03/04
- connpass : https://gaisaba.connpass.com/event/309141/
- 資料: https://qiita.com/IQ_Bocchi/items/cfbe13c29bd9a94442b5
- 箇所：
    - Chapter8
    - Transformation
    - Batch Transformation
    より
        - SQL and code-based transformation tools
        - SQL is declarative...but it can still build complex data workflows
        - Example: When to avoid SQL for batch transformations in Spark
        - Example: Optimizing Spark and other processing frameworks
        (Update patterns の手前まで)



# 第28回 Chapter8 Update patterns
- 2024/03/18
- connpass : https://gaisaba.connpass.com/event/312534/
- 資料: https://afoil.notion.site/2024-03-18-f14f3a7a836e44e9a4e7590129bec5c9
- 箇所：Chapter8
    - Update patterns
    - Truncate and reload
    - Insert only - Delete - Upsert / merge
    - (Schema updates の手前まで)


# 第29回 Chapter8 Schema Updates
- 2024/04/1
- connpass : https://gaisaba.connpass.com/event/313833/
- 資料: https://techblog.otakashi.jp/archives/568
- 箇所：Chapter8
    - Schema updates
    - Data wrangling
    - Example : Data transformation in Spark
    - Business logic and derived data

# 第30回 Chapter8 Materialized Views, Federation, and Query Virtualization
- 2024/04/15
- connpass : https://gaisaba.connpass.com/event/315096/
- 資料: https://afoil.hatenablog.com/entry/2024/04/15/002450
- 箇所：Chapter8
    - Materialized Views, Federation, and Query Virtualization
    - Streaming Transformations and Processing


# 第31回 Chapter8 Who'm you'll work with
- 2024/05/13
- connpass : https://gaisaba.connpass.com/event/316502/
- 資料: https://techblog.otakashi.jp/archives/588
- 箇所：Chapter8
    - Who'm you'll work with 〜 Chapter 8 の最後まで 


# 第32回 Chapter11 The Future of Data Engineering
- 2024/05/27
- connpass : https://gaisaba.connpass.com/event/319782/
- 資料: https://qiita.com/Shumpei_Kubo/items/2ab938d665f3952923a2
- 箇所：Chapter11
    - 冒頭〜The Decline of Complexity and the Rise of Easy-to-Use Data Tools
    - （The Cloud-Scale Data OS and Improved Interoperability の手前まで）


# 第33回 Chapter11 The Future of Data Engineering
- 2024/06/10
- connpass : https://gaisaba.connpass.com/event/320973/
- 資料: https://qiita.com/IQ_Bocchi/items/a55385fdae07ebb7db90
- 箇所：Chapter11
    - The Cloud-Scale Data OS and Improved Interoperability
    - "Enterprisey" Data Engineering
    - Titles and Responsibilities Will Morph...
    - (Moving Beyond the Modern Data Stack, Toward the Live Data Stack の手前まで)


# 第34回 Chapter11 The Future of Data Engineering
- 2024/06/24
- connpass : https://gaisaba.connpass.com/event/322000/
- 資料: https://afoil.notion.site/ch-11-modern-data-stack-0a9cb544c17448b09bcfc23cd0a5b1f7
- 箇所：Chapter11
    - Moving Beyond the Modern Data Stack, Toward the Live Data Stack 〜 Chapter11の最後まで



# 第35回 第1章 データエンジニアリング概説
- 2024/07/08
- connpass : https://gaisaba.connpass.com/event/323400/
- この回より日本語資料を読む回になりました。
- 資料：なし

# 第36回 第4章　データエンジニアリングライフサイクルにおけるテクノロジの選択
- 2024/07/29
- connpass : https://gaisaba.connpass.com/event/326399/
- 資料：https://techblog.otakashi.jp/archives/612


# 第37回 第5章　ソースシステムにおけるデータ生成
- 2024/08/19
- connpass : https://gaisaba.connpass.com/event/328140/
- 資料：https://afoil.notion.site/ch5-561a6229a31f496db07915c7dcf59719


# 第38回 6章 ストレージへの保存 
- 2024/09/02
- connpass : https://gaisaba.connpass.com/event/328612/
- 資料：https://qiita.com/IQ_Bocchi/items/ca8e76a1bf50b5c475a4

# 第39回 7章 データ取り込み 
- 2024/09/30
- connpass : https://gaisaba.connpass.com/event/329964/
- 資料：https://afoil.notion.site/ch7-50a1ea2f69634e8d850e138b95466d8d

# 第40回 第９章 アナリティクス、機械学習、リバースETLへのデータの提供
- 2024/10/21
- connpass : https://gaisaba.connpass.com/event/334165/
- 資料：https://qiita.com/Shumpei_Kubo/items/245b6bc67341d5bf0437


# 第41回 第10章 セキュリティとプライバシー(最終回)
- 2024/10/28
- connpass : https://gaisaba.connpass.com/event/334166/
- 資料：https://qiita.com/IQ_Bocchi/items/b9f716feaa74da9713bc