CALL	  ME:	  GATHERING	  THREAT	  INTELLIGENCE	
  ON	
  TELEPHONY	  	  SCAMS	
  TO	
  DETECT	  FRAUD	  	  Aude	
  Marzuoli,	  Ph.D.	
  Pindrop	  • Pioneer	
  in	
  voice	
  fraud	  prevenFon	
  and	
  authenFcaFon	
  helping	
  companies	
  eliminate	
  phone	
  fraud.	
  	  • Patented	
  PhoneprinFng™	
  technology	
  can	
  idenFfy,	
  locate	
  and	
  authenFcate	
  phone	  devices	
  uniquely	
  just	  from	  the	
  call	
  audio.	
  	  • Selected	
  by	
  the	
  world’s	
  largest	
  banks,	
  insurers,	
  brokerages	
  and	
  retailers,	
  detecFng	
  over	
  80%	  of	  fraud,	
  even	
  for	
  aVackers	
  never	  seen	  before.	
  • Pindrop	
  was	
  founded	
  in	
  2011	
  and	  to	  date	
  has	
  raised	
  $122	
  million.	
  Outline• IntroducFon	
  to	  our	
  Honeypot,	
  a.k.a.	
  Phoneypot	  • Tracking	  spam/scam	
  campaigns	
  • Gathering	  Intelligence	
  on	
  Telephony	
  abuse	  • Conclusion	  	  Introduc.on  to  Phoneypot

Who’s  REALLY  calling  you?

Human	  Element	
  IVR	
  Blind	  Spot	  Caller	
  ID	
  PII	
  &	  KBA	  But  now  scammers  are  calling  you  direct

DATA	
  Customer	
  Names	  Financial	
  InsFtuFons,	
  Account	
  Numbers,	
  SSN,	
  Account	
  balance,	
  PIN,	
  Password,	
  DOB	
  Why  research  telephony  fraud?
• Understand	
  the	
  telephony	
  fraud/scam	
  ecosystem	
  • Gather	
  threat	  intelligence	  through	  measurements	
  and	
  experiments	  • Design	
  defense	
  strategies	
  and	
  deploy	
  them	  TELEPHONY  HONEYPOT

PhoneypotIN	
  ONE	  MONTH	
  LIMITATIONS	  	  550,000	
  CALLS	
  90,000	
  SOURCE	  • Source	
  phone	
  numbers	
  spooﬁng	
  	  • Most	  phone	
  numbers	
  call	  once	  or	
  twice	  PHONE	  NUMBERS	  	  • Labels	  Needed	
  40,000	
  DESTINATION	  PHONE	  NUMBERS	  During	
  the	  ﬁrst	
  half	
  of	
  2016,	
  we	
  recorded	
  100,000	
  calls	
  to	  our	
  Honeypot.	
  PHONEYPOT  VS  ONLINE  COMMENTS
Data	
  set	  #	  of	
  calls	
  Max.	
  #	
  of	
  calls	  or	
  comments	
  per	
  source	
  phone	
  number	
  8,000,000	
  #	
  of	
  source	
  phone	
  numbers	
  880,000	
  Phoneypot	
  Online	
  Comments	
  660,000	
  74,000	
  2,156	
  21,329	  • Data	  set	  collected	
  for	
  year	  2015.	
  • 66%	  of	
  online	
  comments	
  are	  complaining	
  about	
  only	  2%	
  of	
  sources	
  in	  the	  Phoneypot.	
  	  Tracking  spam/scam  campaigns

System  Overview
Past	  calls	
  Call	  Details	
  Audio	  Recordings	
  Machine	
  Learning	
  Engine	
  using	
  audio	
  transcripFon	
  and	  audio	
  features	  	  Library	  	  of	  known	  Robocallers	
  New	  Call	  Audio	
  Features	
  ExtracFon	  Keywords  are  too  noisy
Transcript	
  to	  compare:	  	  • “press	
  one	  to	  say	
  ﬁBy	  per	
  cent	  on	
  your	  electric	
  bill	  with	  no	  money	
  out	  of	  pocket	
  using	  in	  new	  jersey	
  sober	  company	
  with	  hundreds	
  of	
  saHsﬁed	
  customers	
  pressed	
  to	
  and	  conHnue	
  paying	
  one	  of	
  the	
  highest	
  electric	
  uHlity	
  rates	  “	  
• 93%	
  similarity	
  to:	  • “oh	  press	
  one	  to	
  say	
  ﬁBy	  percent	
  on	
  your	  electric	
  bill	  with	  no	
  money	
  out	  of	
  pocket	
  to	
  using	
  a	
  new	
  jersey's	
  older	  company	
  with	  hundreds	
  of	
  saHsﬁed	
  customers	
  pressed	
  to	
  in	
  conHnue	
  paying	
  one	  of	
  the	
  highest	
  electric	
  uHlity	
  rates	  	  • 63%	
  similarity	
  to:	  • “hi	
  we're	
  just	  doing	  the	  quick	
  survey	
  where	
  you	
  where	
  the	  new	
  jersey	
  residence	
  or	
  now	
  enFtled	
  to	
  the	
  rebate	
  on	
  their	
  electric	
  bills	
  twice	  a	
  year	  and	  a	
  twenty	
  ﬁve	
  dollar	
  visa	  gii	
  card	  press	
  one”	
  • 39%	
  similarity	
  to:	  • “do	
  you	  have	  at	
  least	
  ten	
  thousand	
  dollars	
  in	
  credit	
  card	
  debt	  and	
  medical	
  bills	
  would	
  you	
  like	  to	
  hear	  about	
  of	
  program	
  that	  can	
  reduce	  your	  bounds	
  or	
  than	  eﬀecFvely	
  eliminate	  your	  interest	
  “	
  	  

Clustering  of  transcripts  similarity  matrix

Raw similarity matrix between all pairs of transcripts
Dark dot if transcript i is similar to transcript j

Spectral clustering Results

Cluster  extrac.on“Healthcare	
  RegulaFon	
  for	
  Seniors”	
  “IRS	  and	  taxes”	
  “Alice	
  calling	
  about	
  business	
  loan”	
  “Sherrie	
  online	  specialist	
  for	
  Google,	  Yahoo,	
  Bing”	
  “Free	  cruise	  to	  the	  Bahamas”	
  “Do	
  you	
  need	  a	  burial	
  policy?”	
  “Back	  brace	  shown	
  on	
  TV	
  commercial”	
  “Google	  business	
  lisFng	
  records”	
  “Are	  you	  trapped	
  in	
  student	
  loans?”	
  “Home	  security	
  system”	
  Cluster  Phoneprin.ng

Source	  phone	
  numbers	
  whose	  calls	
  correspond	
  to	
  idenFcal	
  audio	  recordings	
  Extract	
  150	
  audio	
  features	
  for	  all	
  calls	
  in	
  the	
  same	  cluster	  Phoneypot	
  desFnaFon	
  phone	
  numbers	  Phoneprin.ng

150	
  factors	
  analyzed	
  on	
  every	
  call	  Call	
  Type	  Unique	
  Phone	  IniFal	
  call	  Geo-­‐LocaFon	
  Spectrum

Noise

Quantization, Frequency
filters, Codec artifacts

Clarity, Correlation,Signal-to-noise ratio

Loss
Packet loss, Robotization,Dropped frames

Phoneprin.ng  dis.nct  bad  actors• Feed	  the	
  classiﬁer:	
  	  • PosiFve	
  class:	
  audio	
  features	
  from	  all	
  calls	
  in	
  the	
  same	  cluster	  • NegaFve	
  class:	
  audio	
  features	
  from	  any	
  other	
  call	  	  • Tested	
  across	
  more	  than	
  50	
  clusters:	
  • Average	
  performance:	
  85.5%	
  True	  PosiFve	
  Rate	
  at	
  0.25%	
  False	  PosiFve	
  Rate	  • Good	  performance	
  even	
  with	  <2	
  calls	
  for	
  each	  source	
  phone	
  number.	
  • Good	  phoneprint	  performance	
  on	
  a	
  cluster	
  implies	
  that	  all	
  calls	
  from	  this	  cluster	
  come	  from	  the	  same	
  physical	  telephony	
  infrastructure.	
  Gathering  Intelligence  on  Telephony  abuse  Google-­‐related  scams• One	  of	
  the	
  most	  complex	  scams	  • More	
  than	
  10	
  diﬀerent	
  recordings	
  targeFng	
  Google	
  users,	
  from	  Google	
  maps	  to	
  Google	
  plus.	
  • Asking	
  you	  to	  pay	
  money	  for	
  front	
  page	  placement	
  on	
  Google	
  search	
  results.	
  • Asking	
  you	  to	
  “verify”	
  personal	
  informaFon.	
  *	
  two	  of	  our	  backers	
  are	
  Google	
  capital	
  and	  GV	  Sharon,  your  Google  specialist253	
  Source	
  phone	
  numbers	
  27,928	
  calls	
  from	  January	
  to	  July	
  2016	
  17,160	
  desFnaFon	
  phone	
  numbers	
  from	  Phoneypot	
  •

“Hi	
  this	  is	
  Sharon	
  your	  local	
  Google	
  specialist.	
  We	
  have	  a	
  front	
  page	  posiFon	
  available	
  for	  a	
  business	
  like	  yours	
  and	  can	
  guarantee	
  front	
  page	  placement	
  with	  unlimited	
  clicks	
  24	
  hours	
  a	
  day	  at	  a	  ﬂat	
  rate	  on	
  Google.	
  Press	
  one	
  key	
  right	
  now	  to	  see	  if	  you	
  qualify	
  and	  are	
  interested	
  in	
  receiving	
  calls	
  from	  companies	
  who	  are	
  locally	
  searching	
  for	
  your	  type	
  of	  business.	
  Please	  press	
  one	  now	  or	  press	
  the	
  two	
  key	  to	
  be	
  removed.	
  Thank	  you.“	  •Phoneprint	  performance:	
  93.4%	
  TPR	  at	
  0.4%	
  FPR	  	  Google  Business  lis.ng• “This	
  is	
  an	
  important	
  call	  regarding	
  your	  google	
  business	
  lisFng.	
  We	
  need	  to	
  verify	
  your	  contact	
  informaFon.	
  Press	
  one	  now	  to	  verify	
  your	  google	
  business	
  lisFng.	
  Press	
  seven	  to	
  remove	
  yourself	
  from	  the	  veriﬁcaFon	
  system.	
  “	  
• Phoneprint	  performance:	
  92.7%	
  TPR	  at	
  0.2%	
  FPR	
  410	
  Source	
  phone	
  numbers	
  34,048	
  calls	
  from	  January	
  to	  July	
  2016	
  26,841	
  desFnaFon	
  phone	
  numbers	
  from	  Phoneypot	  Debt  Collectors• “This	
  call	  is	
  from	  XXXX	
  outsourcing,	
  a	
  debt	  collector.	
  Please	  return	
  my	  call	
  at	
  866-­‐XXX-­‐XXXX.	
  “	
  • Targeted	
  harassment	
  tacFcs	
  7	
  Source	
  phone	
  numbers	
  1,301	
  calls	
  from	  January	
  to	  July	
  2016	
  92	
  desFnaFon	
  phone	
  numbers	
  from	  Phoneypot	
  Experiment  Overview	  	  	  	  1	  Million	
  calls,	  	  from	  210k	  sources	
  100k	  recordings,	
  from	  44k	
  sources	
  31k	
  robocalls	
  51%	  of	
  robocalls	
  from	
  38	
  “infrastructures”	
  CONCLUSION

Conclusion• We	
  recorded	
  100,000	
  calls	
  over	  several	
  months	
  in	
  2016.	
  • Using	
  machine	
  learning	
  on	
  call	
  paVerns,	
  semanFcs	
  and	  audio	
  features,	
  we	
  can	
  uniquely	
  idenFfy	
  one	
  bad	
  actor	
  hiding	
  behind	
  several	
  source	
  phone	
  numbers,	
  whether	
  these	
  numbers	
  are	
  legiFmate	  or	  spoofed.	
  • Our	  results	
  show	  that	
  51%	  of	
  the	
  robocalls	
  recorded	
  are	
  placed	
  by	
  38	
  disFnct	
  telephony	
  infrastructures	
  which	  can	  be	
  uniquely	
  idenFﬁed	
  with	  more	  than	
  85%	  TPR	  on	
  average.	
  	  Acknowledgements  &  Ques.ons
Data	
  Science	
  team	  at	
  Pindrop:	  	  Hassan	
  Kingravi,	
  David	
  Dewey,	
  Aaron	  Dallas,	
  Telvis	
  Calhoun,	  Terry	  Nelms