AI Engineering
Subtitle: Building Applications with Foundation Models
Publisher: O'Reilly Media, Incorporated
Published: 2024
Language: eng
Number of pages: 350
ISBN-13: 9781098166304
First sentence: If I could use only one word to describe AI post-2020, it’d be scale.
Table of contents: label: Preface
title: 10; label: What This Book Is About
title: 12; label: What This Book Is Not
title: 16; label: Who This Book Is For
title: 18; label: Navigating This Book
title: 20; label: Conventions Used in This Book
title: 24; label: Using Code Examples
title: 26; label: O’Reilly Online Learning
title: 27; label: How to Contact Us
title: 28; label: Acknowledgments
title: 29; label: 1. Introduction to Building AI Applications with Foundation Models
title: 34; label: The Rise of AI Engineering
title: 36; label: From Language Models to Large Language Models
title: 36; label: From Large Language Models to Foundation Models
title: 48; label: From Foundation Models to AI Engineering
title: 55; label: Foundation Model Use Cases
title: 63; label: Coding
title: 71; label: Image and Video Production
title: 74; label: Writing
title: 75; label: Education
title: 78; label: Conversational Bots
title: 80; label: Information Aggregation
title: 82; label: Data Organization
title: 83; label: Workflow Automation
title: 85; label: Planning AI Applications
title: 86; label: Use Case Evaluation
title: 86; label: Setting Expectations
title: 94; label: Milestone Planning
title: 95; label: Maintenance
title: 97; label: The AI Engineering Stack
title: 100; label: Three Layers of the AI Stack
title: 102; label: AI Engineering Versus ML Engineering
title: 106; label: AI Engineering Versus Full-Stack Engineering
title: 123; label: Summary
title: 124; label: 2. Understanding Foundation Models
title: 130; label: Training Data
title: 132; label: Multilingual Models
title: 135; label: Domain-Specific Models
title: 144; label: Modeling
title: 148; label: Model Architecture
title: 148; label: Model Size
title: 165; label: Post-Training
title: 184; label: Supervised Finetuning
title: 189; label: Preference Finetuning
title: 195; label: Sampling
title: 205; label: Sampling Fundamentals
title: 205; label: Sampling Strategies
title: 209; label: Test Time Compute
title: 219; label: Structured Outputs
title: 226; label: The Probabilistic Nature of AI
title: 235; label: Summary
title: 248; label: 3. Evaluation Methodology
title: 256; label: Challenges of Evaluating Foundation Models
title: 258; label: Understanding Language Modeling Metrics
title: 264; label: Entropy
title: 266; label: Cross Entropy
title: 268; label: Bits-per-Character and Bits-per-Byte
title: 269; label: Perplexity
title: 270; label: Perplexity Interpretation and Use Cases
title: 272; label: Exact Evaluation
title: 277; label: Functional Correctness
title: 278; label: Similarity Measurements Against Reference Data
title: 281; label: Introduction to Embedding
title: 292; label: AI as a Judge
title: 297; label: Why AI as a Judge?
title: 297; label: How to Use AI as a Judge
title: 300; label: Limitations of AI as a Judge
title: 306; label: What Models Can Act as Judges?
title: 315; label: Ranking Models with Comparative Evaluation
title: 320; label: Challenges of Comparative Evaluation
title: 327; label: The Future of Comparative Evaluation
title: 333; label: Summary
title: 335; label: 4. Evaluate AI Systems
title: 340; label: Evaluation Criteria
title: 341; label: Domain-Specific Capability
title: 345; label: Generation Capability
title: 350; label: Instruction-Following Capability
title: 368; label: Cost and Latency
title: 381; label: Model Selection
title: 385; label: Model Selection Workflow
title: 386; label: Model Build Versus Buy
title: 389; label: Navigate Public Benchmarks
title: 411; label: Design Your Evaluation Pipeline
title: 428; label: Step 1. Evaluate All Components in a System
title: 428; label: Step 2. Create an Evaluation Guideline
title: 431; label: Step 3. Define Evaluation Methods and Data
title: 436; label: Summary
title: 446; label: 5. Prompt Engineering
title: 453; label: Introduction to Prompting
title: 454; label: In-Context Learning: Zero-Shot and Few-Shot
title: 457; label: System Prompt and User Prompt
title: 461; label: Context Length and Context Efficiency
title: 467; label: Prompt Engineering Best Practices
title: 471; label: Write Clear and Explicit Instructions
title: 472; label: Provide Sufficient Context
title: 480; label: Break Complex Tasks into Simpler Subtasks
title: 482; label: Give the Model Time to Think
title: 488; label: Iterate on Your Prompts
title: 494; label: Evaluate Prompt Engineering Tools
title: 495; label: Organize and Version Prompts
title: 501; label: Defensive Prompt Engineering
title: 505; label: Proprietary Prompts and Reverse Prompt Engineering
title: 507; label: Jailbreaking and Prompt Injection
title: 511; label: Information Extraction
title: 521; label: Defenses Against Prompt Attacks
title: 528; label: Summary
title: 536; label: 6. RAG and Agents
title: 541; label: RAG
title: 542; label: RAG Architecture
title: 546; label: Retrieval Algorithms
title: 548; label: Retrieval Optimization
title: 572; label: RAG Beyond Texts
title: 583; label: Agents
title: 588; label: Agent Overview
title: 590; label: Tools
title: 594; label: Planning
title: 602; label: Agent Failure Modes and Evaluation
title: 635; label: Memory
title: 641; label: Summary
title: 650; label: 7. Finetuning
title: 655; label: Finetuning Overview
title: 657; label: When to Finetune
title: 663; label: Reasons to Finetune
title: 664; label: Reasons Not to Finetune
title: 666; label: Finetuning and RAG
title: 674; label: Memory Bottlenecks
title: 681; label: Backpropagation and Trainable Parameters
title: 684; label: Memory Math
title: 687; label: Numerical Representations
title: 692; label: Quantization
title: 697; label: Finetuning Techniques
title: 705; label: Parameter-Efficient Finetuning
title: 706; label: Model Merging and Multi-Task Finetuning
title: 734; label: Finetuning Tactics
title: 751; label: Summary
title: 760; label: 8. Dataset Engineering
title: 768; label: Data Curation
title: 771; label: Data Quality
title: 778; label: Data Coverage
title: 782; label: Data Quantity
title: 787; label: Data Acquisition and Annotation
title: 796; label: Data Augmentation and Synthesis
title: 801; label: Why Data Synthesis
title: 803; label: Traditional Data Synthesis Techniques
title: 807; label: AI-Powered Data Synthesis
title: 814; label: Model Distillation
title: 832; label: Data Processing
title: 834; label: Inspect Data
title: 835; label: Deduplicate Data
title: 838; label: Clean and Filter Data
title: 842; label: Format Data
title: 844; label: Summary
title: 847; label: 9. Inference Optimization
title: 852; label: Understanding Inference Optimization
title: 854; label: Inference Overview
title: 854; label: Inference Performance Metrics
title: 864; label: AI Accelerators
title: 878; label: Inference Optimization
title: 891; label: Model Optimization
title: 892; label: Inference Service Optimization
title: 919; label: Summary
title: 933; label: 10. AI Engineering Architecture and User Feedback
title: 941; label: AI Engineering Architecture
title: 942; label: Step 1. Enhance Context
title: 944; label: Step 2. Put in Guardrails
title: 946; label: Step 3. Add Model Router and Gateway
title: 954; label: Step 4. Reduce Latency with Caches
title: 962; label: Step 5. Add Agent Patterns
title: 967; label: Monitoring and Observability
title: 970; label: AI Pipeline Orchestration
title: 984; label: User Feedback
title: 989; label: Extracting Conversational Feedback
title: 990; label: Feedback Design
title: 1003; label: Feedback Limitations
title: 1017; label: Summary
title: 1023; label: Epilogue
title: 1027; label: Index
title: 1029; label: About the Author
title: 1094