Course: Data Manipulation at Scale: Systems and Algorithms
Modules:
Appetite Whetting: Politics, Appetite Whetting: Extreme Weather, Appetite Whetting: Digital Humanities, Appetite Whetting: Bibliometrics, Appetite Whetting: Food, Music, Public Health, Appetite Whetting: Public Health cont'd, Earthquakes, Legal, Characterizing Data Science, Characterizing Data Science, cont'd, Distinguishing Data Science from Related Topics, Four Dimensions of Data Science, Tools vs. Abstractions, Desktop Scale vs. Cloud Scale, Hackers vs. Analysts, Structs vs. Stats, Structs vs. Stats cont'd, A Fourth Paradigm of Science, Data-Intensive Science Examples, Big Data and the 3 Vs, Big Data Definitions, Big Data Sources, Course Logistics, Twitter Assignment: Getting Started
Data Models, Terminology, From Data Models to Databases, Pre-Relational Databases, Motivating Relational Databases, Relational Databases: Key Ideas, Algebraic Optimization Overview, Relational Algebra Overview, Relational Algebra Operators: Union, Difference, Selection, Relational Algebra Operators: Projection, Cross Product, Relational Algebra Operators: Cross Product cont'd, Join, Relational Algebra Operators: Outer Join, Relational Algebra Operators: Theta-Join, From SQL to RA, Thinking in RA: Logical Query Plans, Practical SQL: Binning Timeseries, Practical SQL: Genomic Intervals, User-Defined Functions, Support for User-Defined Functions, Optimization: Physical Query Plans, Optimization: Choosing Physical Plans, Declarative Languages, Declarative Languages: More Examples, Views: Logical Data Independence, Indexes
What Does Scalable Mean?, A Sketch of Algorithmic Complexity, A Sketch of Data-Parallel Algorithms, "Pleasingly Parallel" Algorithms, More General Distributed Algorithms, MapReduce Abstraction, MapReduce Data Model, Map and Reduce Functions, MapReduce Simple Example, MapReduce Simple Example cont'd, MapReduce Example: Word Length Histogram, MapReduce Examples: Inverted Index, Join, Relational Join: Map Phase, Relational Join: Reduce Phase, Simple Social Network Analysis: Counting Friends, Matrix Multiply Overview, Matrix Multiply Illustrated, Shared Nothing Computing, MapReduce Implementation, MapReduce Phases, A Design Space for Large-Scale Data Systems, Parallel and Distributed Query Processing, Teradata Example, MR Extensions, RDBMS vs. MapReduce: Features, RDBMS vs. Hadoop: Grep, RDBMS vs. Hadoop: Select, Aggregate, Join
NoSQL Context and Roadmap, NoSQL Roundup, Relaxing Consistency Guarantees, Two-Phase Commit and Consensus Protocols, Eventual Consistency, CAP Theorem, Types of NoSQL Systems, ACID, Major Impact Systems, Memcached: Consistent Hashing, Consistent Hashing, cont'd, DynamoDB: Vector Clocks, Vector Clocks, cont'd, CouchDB Overview, CouchB Views, BigTable Overview, BigTable Implementation, HBase, Megastore, Spanner, Spanner cont'd, Google Systems, MapReduce-based Systems, Bringing Back Joins, NoSQL Rebuttal, Almost SQL: Pig, Pig Architecture and Performance, Data Model, Load, Filter, Group, Group, Distinct, Foreach, Flatten, CoGroup, Join, Join Algorithms, Skew, Other Commands, Evaluation Walkthrough, Review, Context, Spark Examples, RDDs, Benefits
Graph Overview, Structural Analysis, Degree Histograms, Structure of the Web, Connectivity and Centrality, PageRank, PageRank in more Detail, Traversal Tasks: Spanning Trees and Circuits, Traversal Tasks: Maximum Flow, Pattern Matching, Querying Edge Tables, Relational Algebra and Datalog for Graphs, Querying Hybrid Graph/Relational Data, Graph Query Example: NSA, Graph Query Example: Recursion, Evaluation of Recursive Programs, Recursive Queries in MapReduce, The End-Game Problem, Representation: Edge Table, Adjacency List, Representation: Adjacency Matrix, PageRank in MapReduce, PageRank in Pregel