Big Data Analytics!

Architectures, Algorithms
and Applications!
Part #3: Analytics Platform

Simon Wu!
HTC (Prior: Twitter & Microsoft)!

Edward Chang 張智威	 
HTC (prior: Google & U. California) !

1/26/2015	  Ed	  Chang	  @	  BigDat	  2015	  Three	  Lectures• Lecture	  #1:	  Scalable	  Big	  Data	  Algorithms	  – Scalability	  issues	  – Key	  algorithms	  with	  applicaGon	  examples	  • Lecture	  #2:	  Intro	  to	  Deep	  Learning	  – Autoencoder	  &	  Sparse	  Coding	  – Graph	  models:	  CNN,	  MRF,	  &	  RBM	  • Lecture	  #3:	  AnalyGcs	  PlaSorm	  [by	  Simon	  Wu]	  – Intro	  to	  LAMA	  plaSorm	  – Code	  lab	  	  1/26/2015	  Ed	  Chang	  @	  BigDat	  2015	  Lecture	  #3	  Outline	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  

Contents	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  MoGvaGon	  hYps://github.com/LamaBigData/lama-­‐demo	  

What	  Do	  We	  Need?	  
• Log	  Management	  System:	  – Collect	  applicaGon	  logs	  with	  reasonable	  latency	  • Data	  processing	  plaSorm:	  – InteracGve,	  batch,	  streaming	  • AnalyGcs	  plaSorm:	  – Visual,	  ad-­‐hoc	  hYps://github.com/LamaBigData/lama-­‐demo	  

Contents	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  

Architecture	  	  hYps://github.com/LamaBigData/lama-­‐demo	  Logging	  Management	  • Open-­‐sourced	  systems:	  – Flume	  – FluentD	  (recommended	  by	  GCE)	  • Supported	  ﬁle	  formats:	  – Textline,	  SequenceFile,	  Structured	  data	  in	  Thric/Protobuf/Avro	  – Compressed	  data	  in	  Lzo/Gzip/Snappy	  – However,	  logs	  saved	  in	  Parquet	  (columnar	  format)	  give	  us	  signiﬁcant	  performance	  gains	  over	  other	  choices	  hYps://github.com/LamaBigData/lama-­‐demo	  Data	  Processing	  Engines	  • InteracGve	  – Impala	  (open	  sourced),	  BigQuery	  (Google)	  • Batch	  – Hadoop	  MapReduce,	  Spark,	  Hive/Pig,	  Cascading/
Scalding	  • Streaming	  – Storm,	  Spark	  streaming	  • Hybrid	  – Google’s	  Dataﬂow	  (managed,	  in	  beta)	  hYps://github.com/LamaBigData/lama-­‐demo	  

Diﬀerent	  Data	  Processing	  Engines	  Engine

Open-Source
Framework

Properties

Latency

Application

Batch
Processing• Large data sets
• High Throughput

Hours orDays

Hourly/Daily
Statistics

Streaming
Processing• Real-time• In-memory

Millseconds

Real-time Counting

InteractiveQuerying• SQL-like query• In-memory

Minutes

Ad-Hoc SQL-like Data
Analysis

Iterative
DataAnalysis• DAG execution• In-memory

HoursMachine Learning

hYps://github.com/LamaBigData/lama-­‐demo	  

AnalyGcs	  PlaSorm	  • Visual	  AnalyGcs:	  – Template	  dashboard	  – Customized	  visual	  graphs	  and	  pivotal	  tables	  – Tableau	  will	  be	  our	  choice!	  
– Best	  for	  Execs/PMs/Sales,	  even	  for	  Engineers	  • Ad-­‐hoc	  AnalyGcs:	  – R	  +	  DS.js	  +	  GGPlot2	  – More	  sophisGcated	  DM/ML	  analyGcs	  on	  big	  data	  – Best	  for	  Engineers	  hYps://github.com/LamaBigData/lama-­‐demo	  

InteracGve	  Querying	  Engine	  • Built	  upon	  an	  open-­‐sourced	  distributed	  SQL	  query	  engine	  (Impala)	  • Logs	  saved	  in	  HDFS	  in	  columnar-­‐format	  (Parquet)	  • Query	  in	  SQL-­‐like	  syntax	  • Benchmark	  results	  show	  Impala+Parquet	  outperforms	  the	  various	  other	  open	  source	  alternaGves	  hYps://github.com/LamaBigData/lama-­‐demo	  

Who	  is	  Going	  to	  Need	  it?	  
• More	  used	  to	  SQL-­‐like	  querying	  • ImpaGent	  enough	  to	  see	  results,	  i.e.,	  in	  minutes	  instead	  of	  hours	  • Quickly	  tesGng	  ideas	  through	  visual	  analyGcs	  on	  short	  or	  medium-­‐long	  period	  of	  history	  logs	  • Ideal	  choice	  for	  PMs/Sales/Execs,	  even	  for	  engineers	  hYps://github.com/LamaBigData/lama-­‐demo	  

Impala	  Architecture	  Figure	  credit:	  hYp://impala.io/overview.html	  	  hYps://github.com/LamaBigData/lama-­‐demo	  

Data	  Processing	  Engines	  • InteracGve	  – Impala	  (open	  sourced),	  BigQuery	  (Google)	  • Batch	  – Hadoop	  MapReduce,	  Spark,	  Hive/Pig,	  Cascading/Scalding	  • Streaming	  – Storm,	  Spark	  streaming	  hYps://github.com/LamaBigData/lama-­‐demo	  Batch	  Processing	  Engine	  • Capable	  of	  processing	  much	  longer	  period	  of	  history	  logs	  with	  higher	  latency,	  usually	  in	  hours	  or	  longer	  • Capable	  of	  conducGng	  very	  sophisGcated	  analyGcs	  using	  DM/ML	  techniques	  using	  MapReduce	  • Cron-­‐scheduled	  for	  processing	  new	  logs	  hYps://github.com/LamaBigData/lama-­‐demo	  Batch	  System	  Architecture	  It’s	  also	  common	  to	  upload	  results	  computed	  from	  batch	  system	  to	  online	  readonly	  store,	  in	  order	  	  for	  online	  services	  to	  lookup	  values	  	  User	  can	  do	  visual	  or	  adhoc	  analysis	  on	  results	  computed	  from	  batch	  system	  hYps://github.com/LamaBigData/lama-­‐demo	  One	  Problem?	  
• What	  if	  we	  want	  to	  see	  metrics	  from	  all	  history	  to	  current	  moment?	  
– Batch:	  	  high	  throughput,	  but	  high	  latency	  – Streaming:	  low	  throughput,	  but	  low	  latency	  • The	  Lambda	  Architecture	  is	  the	  soluGon!	  

hYps://github.com/LamaBigData/lama-­‐demo	  

The	  Lambda	  Architecture	  • Generic,	  scalable	  and	  fault-­‐tolerant	  data	  processing	  architecture	  • Proposed	  by	  Nathan	  Marz:	  hYp://lambda-­‐architecture.net/	  hYps://github.com/LamaBigData/lama-­‐demo	  

Batch	  Layer	  Table	  credit:	  hYp://lambda-­‐architecture.net/components/2014-­‐06-­‐30-­‐batch-­‐components/	  hYps://github.com/LamaBigData/lama-­‐demo	  

Speed	  Layer	  Table	  credit:	  hYp://lambda-­‐architecture.net/components/2014-­‐06-­‐30-­‐speed-­‐components/	  hYps://github.com/LamaBigData/lama-­‐demo	  Serving	  Layer	  Table	  credit:	  hYp://lambda-­‐architecture.net/components/2014-­‐06-­‐30-­‐serving-­‐components/	  hYps://github.com/LamaBigData/lama-­‐demo	  

Our	  SoluGon:	  LAMA	  • LAMA:	  Lambda	  Architecture	  Based	  Big	  Data	  AnalyGcs	  System	  • Based	  on	  TwiYer’s	  open	  sourced	  SummingBird	  project!	  

Lama	  job	  hYps://github.com/LamaBigData/lama-­‐demo	  Content	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  3	  Select	  Sources	  4	  Select	  Stores	  How	  Does	  SummingBird	  Work?	  
1	  Only	  deﬁne	  job	  logic	  once	  2	  Select	  a	  plaSorm	  • Generate	  topology	  automaGcally	  • No	  nhYps://github.com/LamaBigData/lama-­‐eed	  to	  care	  about	  low-­‐level	  details	  demo	  PerspecGves	  • For	  batch	  &	  streaming,	  same	  job	  logic,	  diﬀerent	  running	  plaSorm	  – Why?	  Since	  both	  batch	  and	  streaming	  DO	  mapreduce	  in	  diﬀerent	  ways	  • For	  iteraGve	  data	  analysis,	  at	  least	  same	  language	  &	  APIs,	  no	  need	  to	  learn	  diﬀerent	  ones	  • DOES	  not	  support	  interacGve	  querying	  plaSorm,	  which	  SHOULD	  be	  in	  SQL-­‐like	  querying	  language	  hYps://github.com/LamaBigData/lama-­‐demo	  Key	  Concept	  in	  SummingBird:	  BatchID	  • BatchID:	  Used	  for	  job	  scheduling,	  data	  merging,	  fault	  tolerance	  • Batcher	  here	  is	  used	  to	  declare	  how	  ocen	  this	  job	  will	  run	  • Batcher.ofDays(1)	  means	  this	  job	  will	  run	  once	  every	  day	  Sum	  of	  	  RealTime.BatchID.i	  +	  Hadoop.BatchID.i-­‐1	  hYps://github.com/LamaBigData/lama-­‐demo	  Why	  not	  directly	  using	  SummingBird?	  
• Immature	  – No	  job	  concept	  (which	  means	  you	  have	  to	  take	  care	  of	  job	  runner	  by	  yourself)	  – APIs	  are	  hard	  to	  use	  – Users	  have	  to	  deal	  with	  batchID	  by	  themselves	  – Too	  few	  input/output	  formats	  and	  databases	  supported	  – No	  DAG	  job	  supported	  – No	  deployment,	  monitoring	  – No	  support	  for	  Spark	  plaSorm	  – No	  Support	  for	  Google’s	  GFS/BigQuery/DataStore	  – No	  support	  for	  Google’s	  Dataﬂow	  • More	  like	  a	  prototype	  than	  a	  product!	  
hYps://github.com/LamaBigData/lama-­‐demo	  

Lama’s	  ContribuGons	  • Simpliﬁed	  API	  • BatchID	  Management	  • More	  Flexible	  Inputs	  and	  Outputs	  • DAG	  Job	  Scheduling	  • Easy	  Deployment	  &	  Monitoring	  • VisualizaGon	  • Support	  Google’s	  GCS/BigQuery/DataStore/Dataﬂow	  • Support	  Spark	  (ready	  by	  EOQ1)	  hYps://github.com/LamaBigData/lama-­‐demo	  

PerspecGves	  • Based	  on	  TwiYer’s	  Summingbird	  – Scala	  DSL	  – OOP	  – Cascading	  in	  low	  level	  • Thanks	  to	  Scala	  DSL,	  wriGng	  MapReduce	  jobs	  is	  like	  wriGng	  naGve	  programming	  codes	  • All	  components	  of	  the	  job	  can	  be	  unit	  tested	  hYps://github.com/LamaBigData/lama-­‐demo	  

Contents	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  

Scala	  • A	  Scalable	  language	  • Created	  by	  MarGn	  Odersky	  (founder	  of	  Typesafe)	  • Object-­‐oriented	  • Seemless	  Java	  interop	  • FuncGons	  are	  objects	  • Future	  proof	  • Fun	  hYps://github.com/LamaBigData/lama-­‐demo	  Who	  Are	  Using	  Scala?	  
• TwiYer	  • LinkedIn	  • Foursquare	  • Box	  • Quora	  • Tumblr	  • Git	  • Yammer	  • …	  hYps://github.com/LamaBigData/lama-­‐demo	  Best	  Scala	  Lectures	  • Scala	  school:
hYp://twiYer.github.io/scala_school/	  • EﬀecGve	  scala:	  hYp://twiYer.github.io/eﬀecGvescala/	  hYps://github.com/LamaBigData/lama-­‐demo	  Basic	  Data	  Structures	  • List:	  An	  ordered	  collecGon	  of	  elements	  – scala>	  val	  numList	  =	  List(1,	  2,	  3,	  4)	  • Set:	  No	  duplicates	  – scala>	  val	  numSet	  =	  Set(1,	  2)	  • Tuple:	  A	  group	  of	  simple	  logical	  collecGons	  – scala>	  val	  record	  =	  (“Peter”,	  “male”,	  30)	  • Map:	  mapping	  associaGon	  – scala>	  val	  wordCount	  =	  Map(“foo”	  -­‐>	  30)	  • OpGon:	  A	  container	  that	  may	  or	  may	  not	  hold	  sth	  – scala>	  wordCount.get(“foo”)	  (Some(30))	  – scala>	  wordCount.get(“bar”)	  	  (None)	  	  hYps://github.com/LamaBigData/lama-­‐demo	  

FuncGonal	  Combinators	  • map:	  evaluates	  a	  funcGon	  over	  each	  element	  in	  the	  list	  – scala>	  numList.map{i:	  Int	  =>	  i	  *	  2}	  • ﬂatMap:	  combines	  map	  and	  ﬂaYen	  – scala>	  numList.map{i:	  Int	  =>	  List(i*2,	  i*2	  +	  1)}	  • foreach:	  similar	  with	  map	  without	  returning	  anything	  – scala>	  numList.foreach{i:	  Int	  =>	  i	  *	  2}	  • ﬁlter:	  removes	  any	  elements	  whose	  logic	  is	  false	  – scala>	  numList.ﬁlter{i:	  Int	  =>	  i	  %	  2	  ==	  0}	  • Your	  own!	  

Can	  you	  implement	  it?	  

– scala>	  def	  isEven(i:	  Int):	  Boolean	  =	  i	  %	  2	  ==	  0	  – scala>	  def	  myFilter(num:	  List[Int],	  fn:	  Int	  =>	  Boolean):	  List[Int]	  	  – scala>	  def	  myFilter(num:	  List[Int],	  fn:	  Int	  =>	  Boolean):	  List[Int]	  =	  num.ﬂatMap{i:	  Int	  =>	  if	  (fn(i))	  Some(i)	  else	  None}	  hYps://github.com/LamaBigData/lama-­‐demo	  

Basics	  ConGnued	  • Object:	  	  – It’s	  used	  to	  hold	  single	  instances	  of	  classes	  	  • Companion	  object:	  – When	  an	  object	  has	  the	  same	  name	  with	  the	  class	  – You	  can	  put	  your	  staGc	  methods	  here	  for	  easy	  sharing	  and	  uniYesGng	  • Case	  class	  – It’s	  used	  to	  conveniently	  store	  and	  match	  on	  the	  class	  contents	  • Trait	  – Similar	  with	  interfaces	  in	  Java,	  but	  with	  parGal	  implementaGons	  – May	  not	  have	  constructor	  parameters	  – Deﬁne	  object	  types	  by	  specifying	  the	  signature	  of	  the	  supported	  methods.	  
hYps://github.com/LamaBigData/lama-­‐demo	  

Content	  • MoGvaGon	  • IntroducGon	  • LAMA	  • FuncGonal	  Programming	  • Coding	  Demo	  hYps://github.com/LamaBigData/lama-­‐demo	  

A	  LAMA	  Job	  Example	  • Example:	  Kevin	  is	  an	  engineer	  in	  studio	  team	  working	  on	  NLP	  projects.	  He	  needs	  to	  count	  word	  frequencies	  from	  a	  certain	  period	  of	  history	  related	  news	  logs,	  where	  each	  news	  is	  a	  very	  long	  arGcle.	  He	  has	  to	  parse	  each	  arGcle,	  split	  them	  into	  words,	  and	  then	  count	  number	  of	  words.	  Logs	  are	  saved	  in	  a	  Hadoop	  cluster	  in	  cloud.	  So	  he	  wrote	  a	  batch	  job	  using	  LAMA	  as	  shown	  in	  next	  page	  hYps://github.com/LamaBigData/lama-­‐demo	  How	  to	  Write	  a	  LAMA	  Job?	  
• Source:	  This	  is	  how	  you	  read/transform	  raw	  data	  into	  your	  desired	  format	  • Store:	  This	  is	  the	  place	  where	  your	  aggregaGon	  logics	  happen,	  so	  deﬁne	  the	  key,	  value,	  and	  serialized	  data	  format	  here	  • Monoid:	  This	  instructs	  the	  store	  how	  to	  aggregate	  your	  keys	  • Injec;on:	  This	  instructs	  the	  store	  how	  to	  serialize	  data	  • Job:	  This	  is	  the	  main	  place	  where	  you	  combine	  all	  modules	  together	  in	  order	  to	  compute	  your	  metrics	  constantly.	  
hYps://github.com/LamaBigData/lama-­‐demo	  It	  supports	  both	  batch	  (Scalding)	  and	  streaming	  (Storm),	  so	  just	  extend	  it	  to	  create	  your	  WordCountScaldingJob	  and	  WordCountStormJob	  Job	  Job	  details	  have	  been	  abstracted!	  

Specify	  how	  to	  schedule	  the	  job	  hYps://github.com/LamaBigData/lama-­‐running	  demo	  Source	  $	  /usr/hadoop/usr_log/2015/01/23/part-­‐000.txt	  $	  /usr/hadoop/usr_log/2015/01/24/part-­‐000.txt	  $	  /usr/hadoop/usr_log/2015/01/25/part-­‐000.txt	  $	  /usr/hadoop/usr_log/2015/01/26/part-­‐000.txt	  	  	  We	  have	  implemented	  most	  source	  hYps://github.com/LamaBigData/lama-­‐APIs.	  You	  can	  also	  customize	  yours	  demo	  Monoid	  In	  this	  example,	  we	  tells	  LAMA	  how	  to	  reduce	  two	  ImageFeatures	  protobuf	  into	  one.	  
hYps://github.com/LamaBigData/lama-­‐demo	  Store	  $	  /usr/hadoop/usr_count/2015/01/23/part-­‐000.txt	  $	  /usr/hadoop/usr_count/2015/01/24/part-­‐000.txt	  $	  /usr/hadoop/usr_count/2015/01/25/part-­‐000.txt	  $	  /usr/hadoop/usr_count/2015/01/26/part-­‐000.txt	  	  	  Note:	  In	  our	  latest	  LAMA	  APIs,	  store	  has	  completely	  replace	  source,	  coz	  they	  serves	  very	  similar	  purposes	  hYps://github.com/LamaBigData/lama-­‐demo	  

InjecGon	  InjecGon	  tells	  LAMA	  how	  to	  serialize/de-­‐serialize	  key-­‐value	  pairs	  hYps://github.com/LamaBigData/lama-­‐demo	  

Business	  Logics	  Just	  override	  the	  job	  funcGon	  to	  hYps://github.com/LamaBigData/lama-­‐
implement	  your	  own	  bdemo	  usiness	  logics	  Run	  in	  Diﬀerent	  PlaSorm?	  
• Batch	  (Scalding)	  	  • Streaming	  (Storm)	  hYps://github.com/LamaBigData/lama-­‐demo	  

How	  about	  Google’s	  DataFlow?	  
• Google’s	  DataFlow	  is	  a	  managed	  batch+streaming	  plaSorm?	  
• Not	  free	  • LAMA	  supports	  it	  too	  hYps://github.com/LamaBigData/lama-­‐demo	  Data	  Processing	  Engines	  • InteracGve	  – Impala	  (open	  sourced),	  BigQuery	  (Google)	  • Batch	  – Hadoop	  MapReduce,	  Spark,	  Hive/Pig,	  Cascading/
Scalding	  • Streaming	  – Storm,	  Spark	  streaming	  hYps://github.com/LamaBigData/lama-­‐demo	  Streaming	  Processing	  System	  • We	  built	  a	  realGme	  computaGon	  system	  using	  Storm	  – A	  distributed	  realGme	  computaGon	  system	  – Simple,	  fast,	  scalable,	  fault-­‐tolerant,	  and	  very	  reliable	  – Throughput	  up	  to	  1M	  tuples	  processed	  per	  second	  per	  node	  • Events	  are	  sent	  from	  services	  through	  Ka}a,	  which	  connects	  with	  Storm’s	  bolts	  hYps://github.com/LamaBigData/lama-­‐demo	  

Supported	  Data	  Input	  Channels	  • Ka}a	  • Kestrel	  • Flume	  • Majority	  of	  key-­‐value	  stores,	  such	  as	  Memcache,	  Reddis,	  MangoDB	  • MySQL	  hYps://github.com/LamaBigData/lama-­‐demo	  

Possible	  ApplicaGons	  • RealGme	  analyGcs	  • Online	  machine	  learning	  • ConGnuous	  computaGon	  • Distributed	  ETL	  hYps://github.com/LamaBigData/lama-­‐demo	  

Visual	  AnalyGcs	  • Example:	  Alice	  is	  a	  PM	  in	  HTC	  studio.	  One	  day,	  it’s	  urgent	  for	  her	  to	  get	  numbers	  of	  HTC	  cell	  phones	  sold	  in	  Asian	  countries	  in	  past	  month.	  
Alice	  uses	  our	  interacGve	  querying	  system	  to	  get	  what	  she	  needs	  by	  following	  steps	  like:	  – Write	  a	  SQL-­‐like	  query	  and	  execute	  it	  in	  Impala	  – Wait	  for	  several	  minutes	  (just	  some	  Gme	  for	  drinking	  a	  cup	  of	  coﬀee)	  – See	  visualized	  reports	  in	  Tableau	  hYps://github.com/LamaBigData/lama-­‐demo	  Step	  1:	  Write	  an	  Impala	  Query	  *	  In	  this	  example,	  I	  used	  Apache	  Hue	  as	  an	  example	  of	  query	  IDE.	  We	  can	  certainly	  do	  the	  same	  thing	  hYps://github.com/LamaBigData/lama-­‐in	  Tableau	  but	  I	  can’t	  access	  it	  at	  the	  Gme	  of	  wriGng	  demo	  the	  deck.	  Step	  2:	  Visualize	  Results	  in	  Tableau	  hYps://github.com/LamaBigData/lama-­‐demo	  Ad-­‐hoc	  AnalyGcs	  • Example:	  Tom	  is	  an	  engineer	  in	  HTC	  studio.	  
One	  day,	  he	  came	  up	  with	  an	  idea	  of	  compuGng	  similarity	  between	  any	  two	  Apps	  from	  logs.	  By	  similarity,	  he	  means	  normalized	  common	  #users.	  	  Tom	  implemented	  his	  idea	  by	  wriGng	  a	  batch	  script	  using	  Scalding,	  ran	  it	  on	  one	  month	  of	  history	  logs,	  and	  visualize	  his	  app	  network	  results	  using	  D3.js	  to	  get	  high-­‐level	  understanding.	  	  hYps://github.com/LamaBigData/lama-­‐demo	  App	  Similarity	  Network	  	  Overall	  Network	  	  Highlights	  for	  one	  App	  *	  Network	  graph	  is	  visualized	  using	  D3.js	  hYps://github.com/LamaBigData/lama-­‐demo	  Coding	  Lab	  • Demo	  repository:	  – hYps://github.com/LamaBigData/lama-­‐demo	  

• Q/A	  contact:	  – Skype	  account	  @	  LamaBigData	  • Lab	  hour:	  – Mon,	  1/26,	  19:10~21:00	  hYps://github.com/LamaBigData/lama-­‐demo	  Environment	  Setup	  • Install	  Git:	  	  – hYp://git-­‐scm.com/book/en/v2/Ge€ng-­‐Started-­‐Installing-­‐Git	  • Install	  Scala:	  – hYp://scala-­‐lang.org/download/2.10.3.html	  • Clone	  the	  repository:	  $	  git	  clone	  hYps://code.google.com/p/lama-­‐demo-­‐spain	  • Compile	  codes	  $	  ./sbt	  compile	  • Follow	  README	  for	  the	  rest	  	  hYps://github.com/LamaBigData/lama-­‐demo