1 

2

3

4

5

6

7

8 FEDERAL TRADE COMMISSION

9 SPRING PRIVACY SERIES

10 ALTERNATIVE SCORING PRODUCTS

11

12 MARCH 19, 2014

13

14

15

16

17

18

19

20 Federal Trade Commission

21 601 New Jersey Avenue, N.W., Conference Center

22 Washington, DC

23

24 Reported By: Stephanie Gilley

25
2

1 FEDERAL TRADE COMMISSION

2 I N D E X

3

4 Session Page

5 Welcome 3

6 Overview of Predictive Analytics

7 Claudia Perlich 7

8 Panel Discussion 19

9 Emerging Trends in Online Pricing

10 Ashkan Soltani 59

11 Panel Discussion 69

12

13

14

15

16

17

18

19

20

21

22

23

24

25
3

1 W E L C O M E

2 MS. ARIAS: Hi, everyone, and welcome. Thank

3 you so much for joining us to the second installment in

4 our spring privacy series.

5 Today we are going to be talking about

6 alternative scoring products, but before we begin, we

7 have some small administrative and security issues that

8 we have to cover before we begin.

9 All right. So please note that anyone who

10 goes outside today without an FTC badge will need to go

11 through the magnetometer and the x-ray machine again

12 prior to reentry into the conference center.

13 In the event of a fire or evacuation of the

14 building, please leave the building in an orderly

15 fashion. Once outside the building, you need to orient

16 yourself to New Jersey Avenue. So across from the FTC

17 is Georgetown Law Center, look to the right front

18 sidewalk. You need to check in with the person

19 accounting for everyone in the conference rooms there.

20 In the event that it's safer to remain

21 inside, you will be told to go where inside the

22 building, so just listen for an announcement. If you

23 spot suspicious activity, please alert security.

24 Security is right outside, where you came in.

25 This event may be photographed, videotaped,
4

1 webcast, or otherwise recorded, so by participating in

2 this event, you are agreeing that your image and

3 anything you say or submit may be posted definitely at

4 FTC.gov or one of the Commission's publicly available

5 social media sites.

6 All right. So to submit questions today

7 while the event is happening, question cards are

8 outside, available right on the table where you came

9 in. So if you want, just go ahead and grab one and

10 write your question and then we will have one of our

11 fabulous paralegals here at the FTC walk around and

12 collect them and bring them up to us and then we'll ask

13 the questions to our fabulous panel coming up.

14 For those of you participating via webcast,

15 you can email your questions to

16 alternativescores@FTC.gov, Tweet it to #FTCPriv, or

17 post it to the FTC's Facebook page in the Workshop

18 Status thread. Please understand that we may not be

19 able to get to all your questions today, but we will

20 definitely try.

21 MS. ARMSTRONG: I'm Katherine Armstrong and

22 thank you for coming here today and thank you to our

23 panel.

24 There's a lot of buzz these days about data

25 brokers and alternative scores that are used to predict
5

1 consumer behavior, such as the likelihood that a person

2 will be interested in a specific product or service or

3 that a particular transaction could result in fraud,

4 but we want to try to get past the buzz today.

5 And so we have a panel of experts with

6 different perspectives, different experiences, and our

7 goal is to learn more about what is happening in the

8 alternative scoring space, what may be on the horizon,

9 and the privacy and consumer rights concerns that

10 these products may raise. Our focus today is on

11 non-FCRA covered products, although we do know there is

12 a robust conversation to happen there.

13 Before we introduce our panel and start the

14 conversation, however, Claudia Perlich will give us an

15 overview of predictive analytics. Basically, a

16 high-level, nuts-and-bolts presentation about how scores

17 are created. After that, we would like to spend the

18 first half of our time discussing the various kinds of

19 products available, their uses and accuracy issues.

20 Then, Ashkan Soltani will give us a brief

21 presentation about the trends in online pricing. And

22 then, the second half, we will focus on the issues

23 involving privacy, future uses, and the regulatory

24 landscape.

25 But first let me introduce Claudia Perlich,
6

1 who is the chief scientist at Dstillery and

2 concentrates on data analytics for companies,

3 real-world applications, and she also teaches data

4 mining for business intelligence at the NYU Stern MBA

5 program. Claudia?

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25
7

1 OVERVIEW OF PREDICTIVE ANALYTICS

2 MS. PERLICH: Thank you very much. So thanks

3 for having me here. It's a slightly different audience

4 from who I usually talk to.

5 So I'm trying to get to the basics of the

6 technology of predictive modeling and I have a couple

7 of examples of that. So ultimately, what it means when

8 we're talking about predictive modeling that's

9 underlying a lot of the questions you're interested in,

10 is algorithms that learn from data.

11 So you just give it the data and you let the

12 computer do its thing and it comes back with,

13 ultimately, models that affect your everyday life in

14 some way.

15 I've picked two examples specifically to kind

16 of explain initially how that works, and my first

17 example is Lending Club. It's a microloan site where

18 you can apply for loans and where investors can go and

19 choose what loans they want to fund based on the

20 information provided. And they actually provide the

21 data, and I want to take this, initially, to just give

22 you a very high-level overview of what might be

23 happening under the hood here.

24 So let's just look at a simple example of

25 loan default. Let's say I want to predict the
8

1 probability of loan default. I can do that in order to

2 make decisions whether or not to invest, I can do that

3 in order to decide what interest rate is appropriate, I

4 can even do that in order to increase the probability

5 of my loan getting funded. There are many different

6 reasons why I may want to know that.

7 So let's look at this. This is data where I

8 just give you two variables here. One is the age of

9 the person applying and the second one is the income.

10 Now, I also have historical outcomes. That's key for

11 all of this technology, that you actually need to have

12 observed some of these things and what happened.

13 So let's say for those loans, we actually

14 know who defaulted and who didn't. What happens next

15 is, I give it to the algorithm. And I'm just showing

16 it to you here in two dimensions to keep it simple.

17 You look at this information and you have those people

18 who defaulted and those who didn't.

19 What the algorithm is trying to do, without

20 any human intervention, is can it explain the data.

21 Can it find out what's different about the loans that

22 defaulted versus those that didn't. And one technique,

23 called decision trees, starts splitting the data along

24 different dimensions.

25 And the first thing it might see is, well, if
9

1 you look at the income, it says balance here, that's a

2 clear separation that already gets a lot of the same on

3 one side versus the other. And then it continues to do

4 that by asking those kind of questions, how do I split

5 it. Again, all of this is done automatically. There

6 is an algorithm that does it, that tries all the

7 different ways of splitting, evaluates which ones have

8 more defaults in a clean bucket versus not, and goes

9 ahead.

10 Now, at the end of this, this is called a

11 classification tree, what can you do with it? Well,

12 you can ask for a new applicant, what the tree thinks

13 the most likely outcome is going to be. And the tree

14 will just fit it into the bucket and it will say it

15 fell there, and therefore the probability of default,

16 given what it has seen so far, is roughly four out of

17 seven.

18 So that's just one algorithm that uses data

19 that you can then translate into a model. And you can

20 ask it for a new case, where you don't know the outcome

21 yet, what the model thinks is the most likely scenario.

22 A similar example here, it's a different

23 algorithm that does the same thing. So the one thing

24 to know, it's not like one algorithm we're talking

25 about. There are hundreds of them out there and often
10

1 they are just slightly different from each other. It

2 does the same thing, except that it doesn't try to

3 split the data, but it's trying to find a line that

4 optimally separates them.

5 Ultimately, what it means mathematically, and

6 that's the only equation I'm going to bother you with

7 here, is it's trying to estimate these betas that you

8 see there. So once you know the betas, and it's not

9 moving forward here, if you estimate the betas, you can

10 then ask the equation, what does this model think is

11 the most likely probability of default.

12 So the takeaway here is, we use data where we

13 have historical outcomes to build models that then we

14 can ask, when we don't know the outcome yet, what is

15 the most likely scenario. Now, one thing I wanted to

16 make clear here, this is a version where you see it,

17 the model doesn't understand what the axis means. The

18 model doesn't know that what one is H, it doesn't know

19 that the other one is income. The computer actually

20 doesn't care whatsoever, it's completely agnostic about

21 what is going on underneath. It just solves the one

22 question I ask it, what is the best separation you can

23 come up with and I have to specify best in some way.

24 Now, let's look at the Lending Club data that

25 is publicly available. You can go there and download
11

1 the information, if you want to. And they have a lot

2 of information up on the website, including the textural

3 description of the loan, categories, demographic

4 information, in addition to credit scores. All of that

5 is available, if I wanted to build this model. And

6 here is the pull I did yesterday, so you can get that

7 data. It doesn't have names in there, but it does have

8 zip codes of where the person is from.

9 Now, I want to move on to what I do, that's

10 my day job here, we're talking now about targeted

11 online display advertising. So it's about all these

12 pesky little ads showing up whenever you're surfing the

13 internet, all over the place, trying to make you buy

14 stuff. What is the data and how does this work?

15 So this is me and my company. And if you are

16 browsing the internet, and you don't have third-party

17 cookies disabled, and whoever wants to know the

18 technical details, I'd be happy to talk about this

19 later on, what happens is you will come to certain

20 sites that have data partnerships with many, many, many

21 constituents, including us. What happens, for

22 instance, if you read a blog, if we have that data

23 partnership, there is a forwarding of that and we can

24 put a cookie on your computer.

25 Now, there is a lot of misunderstanding of
12

1 what a cookie actually is and I'll talk a little bit

2 about what that cookie does. It basically just

3 contains a 20-digit random number that we have

4 assigned to you. The moment you delete your cookies,

5 it's gone. If you disable third-party cookies, it

6 never even gets to your computer. If you have it

7 enabled, then I can put this little piece of

8 information that is just stored. It's not a program,

9 it's just a piece of information that I store on your

10 computer.

11 Now, what happens next? We are running

12 campaigns on behalf of marketers, so marketers have an

13 agreement with us. They come to us and say, we would

14 like you to show ads, display ads, on the internet on

15 our behalf for our product. The first thing we will

16 do, we will put another one of these pixels on the

17 homepage of that brand, so now we also see who actually

18 buys that stuff. So we'll see people who go to that

19 brand's homepage.

20 One important thing is, I don't see what

21 Amazon knows about you, I don't see what you do on

22 eBay, I don't see what you do on Facebook. I get a

23 very partial view of what you do through these data

24 partnerships and it's very, very far from complete.

25 Now, once I have that, let's move on to the
13

1 part where the actual display advertising happens. You

2 continue surfing the internet and you get to a page

3 that actually has space for a display ad. At this

4 point, there will be a realtime auction through an ad

5 exchange. As the page loads, the different

6 placeholders for advertising are sold in realtime,

7 through an auction. We are getting bid requests. So

8 the publisher, say New York Times, forwards the bid

9 request to the ad exchange, the ad exchange sends it to

10 us, and many, many, many like us. There is not just

11 one ad exchange, there are probably 20 or 50 of them.

12 We have 30 milliseconds to decide whether or not we

13 want to bid on the opportunity to show the ad.

14 The important thing here is, what I do know

15 at the moment is that same 20-digit random number that

16 I assigned your cookie, if you deleted it, I don't know

17 it anymore, but -- so I don't know who you are, but I

18 know you are the same you you used to be when I saw you

19 before.

20 I make my choice, I bid, and if I win, I get

21 to show the ad on behalf of one of the 300-plus

22 marketers who are working with us. After that, we are

23 basically looking for post-view conversion. So we are

24 not looking who is clicking on that stuff, but we

25 actually want to see, is the person afterwards going to
14

1 go to that brand's homepage to either buy it or at

2 least check out the product.

3 So that's kind of, on a very high-level, and

4 not talking about a lot of technical details, how that

5 thing works. The predictive modeling piece that we

6 talked about comes in many different places. The

7 core question I have to answer is, who is even

8 interested in that brand or in running shoes or in dog

9 food?

10 So the first question is, can I build a model

11 that predicts how likely it is that you are a runner in

12 the first place, how likely it is that you are

13 interested in dog food? So that's the first piece I

14 need to solve, but there are many more. There is when

15 should I advertise, how much should I bid. By the way,

16 the creative, I have no control over. The brand just

17 gives me that thing and says that's what you have to

18 show, so that doesn't fall into my responsibility here.

19 A good question is, do these ads actually do

20 anything. We can also decide what data we need, what

21 data we buy, because we actually have to pay for that,

22 what is the quality of data, and there is this notion

23 of attribution that is more about kind of the cost

24 incentives on the other side.

25 So there are many problems that ultimately
15

1 rely on predictive modeling to be answered just alone

2 and in this one application.

3 We also have issues around fraud, trying to decide

4 which of these bid requests are real people as compared

5 to bots posing as people and kind of malware systems

6 that overtake people's computers, posing as them, when

7 in reality the person may actually be doing something

8 completely different.

9 What I want to highlight here is what is the

10 data we actually see and collect, because that's

11 important in the discussion you're having. So

12 ultimately, what I see is a partial browsing history of

13 you, only from those data partnerships and also from

14 the bid requests. I'm not interested in understanding

15 what you're reading on the internet, I couldn't care

16 less. I'm going to take the URL that you go to and I

17 hash it into a random string.

18 So you see, on the right side of the slide,

19 there are two parts. There is the browsing history,

20 it's kind of a timeline of hashed URLs. You have a

21 random number that is kind of your ID that means

22 nothing to anybody but me, because I use it to kind of

23 append things to your history. And I also get, from

24 the brands that we work with, these kind of purchase

25 events. When you actually went to buy a product from
16

1 one of the brands that we work with, because that's the

2 thing I need to predict, right? That's ultimately what

3 I'm interested in.

4 Now, the interesting point here, I do not want

5 to understand you. I don't care to know who you are

6 and what you do. I don't want any PI information for

7 that. It's purely agnostic and translated into something

8 that machines can work with that nobody else really

9 gets to.

10 Now what happens next, when I showed you a

11 model that I estimated in two dimensions, I am now

12 going to do the exact same thing, but in 10 million

13 dimensions. Every single URL is kind of its own

14 dimension, did you go there or not. And we have

15 roughly on the order of 10 million of these URLs that

16 see over the period of time.

17 So I'm building this model for every single

18 product that we are advertising to for the marketer and

19 then I use it to score people. What means scoring

20 people is, again, I see a cookie coming in, I look at

21 the history I have observed. For that cookie ID, the

22 20-digit numbers, I estimate -- I ask the model what it

23 tells me, it gives me a probability. Once you reach a

24 certain highlevel, you become the target bucket. And

25 whenever anybody from that top tier, typically it's
17

1 like one percent, comes in a bid request, we decide to

2 bid for that person. This is then downhill.

3 So I just have a short list here in terms of

4 all the different places where this is happening. Spam

5 has been doing this for a very, very long time.

6 There's a lot of fraud and fraud detection, financial

7 trading industry, a lot of work on this in medical

8 diagnosis and quality control, sentiment analysis for

9 blogs works exactly the same way. You look at the text

10 and try to decide is this a happy or an unhappy person, or

11 is that text from a happy or unhappy person, and many,

12 many more, one of them being advertising and targeting.

13 I'll skip the next slide, because I think

14 we'll just -- well, I guess the point, from my

15 perspective here, it actually doesn't matter that much

16 what the exact algorithm is. There are almost all

17 kinds of equivalents, some work better sometimes, but

18 at the end of the day, any of them will, more or less,

19 do the same thing.

20 What matters much more is the data you feed

21 it. The behavior of the model can only be explained if

22 you understand the data that went in to determine that

23 function. The algorithm itself is just a translator,

24 if you want.

25 Quality control is incredibly hard. If you
18

1 ask me how good my model is, I have no idea and I built

2 it. I don't know whether working with it another week

3 will improve performance by 1 percent or 10 percent, I

4 don't know that either. So at the end, that's kind of

5 my skill and intuition. You have this problem where

6 models really are the skill of the person who assembled

7 the data, and that's only as good as it is.

8 And finally, it is extremely difficult to

9 understand the nuts-and-bolts of what the data is and how

10 it affects the outcome. It's a really complicated

11 problem, even for people who do that for a living. And

12 with that, I'll leave it.

13

14

15

16

17

18

19

20

21

22

23

24

25
19

1 PANEL DISCUSSION

2 MS. ARIAS: Great. Thank you, Claudia.

3 Wasn't that a fantastic presentation? Yeah.

4 All right. So before we begin, I would like

5 to introduce everybody on our panel. We have a very,

6 very long list of panelists, so I obviously will not be

7 able to cover their fabulous information, okay? But I

8 will try to briefly give you an overview of what they

9 do. And make sure you look at the agendas that are

10 behind -- you have the bios of everybody behind the

11 agenda and so you can take a look at their fabulous,

12 fabulous history.

13 All right. So first, next to Katherine, we

14 have Rachel Thomas. Rachel is the Executive Director

15 of the Data-Driven Marketing Institute and Vice

16 President of Government Affairs for the Direct

17 Marketing Association. She not only conducts

18 independent academic research regarding how the

19 responsible use of consumer data shapes industry and

20 society, but she also represents the data-driven

21 marketing community's policy-making interests on

22 Capitol Hill. Thank you for being here with us today.

23 MS. THOMAS: Thanks for having me.

24 MS. ARIAS: Next, we have Stuart Pratt.

25 Stuart is the President and CEO of the Consumer Data
20

1 Industry Association. He not only represents

2 businesses that provide companies with the data and

3 analytical tools necessary to manage risk, but he also

4 has advised U.S. Presidential and Gubernatorial task

5 forces on the importance of the freeflow of

6 information to the U.S. economy and he testifies

7 regularly before Congress. Thank you for being with us

8 today.

9 Next to Stuart we have Ed Mierzwinski. Ed is

10 the Consumer Program Director and Senior Fellow at the

11 U.S. Public Interest Research Group. He often lectures

12 and testifies before Congress on a wide range of

13 consumer issues, including privacy, and he recently

14 published a law review article on alternative scoring

15 products through the Suffolk Law Review. Thank you,

16 Ed, for being with us today.

17 Following Ed we have Pamela Dixon. Pam is

18 the founder of the World Privacy Forum. She not only

19 has written numerous studies on privacy, but she also

20 has testified before Congress and Federal agencies on

21 these issues. She has a study on alternative scoring

22 coming out very soon, I've been told, so make sure you

23 keep checking out the World Privacy Forum's website.

24 Thank you for being with us, Pam.

25 Following Pam, we have Joseph Turow. Joe is
21

1 the Robert Lewis Shayon Professor of Communication at

2 the University of Pennsylvania's Annenberg School for

3 Communication and he has published multiple books and

4 articles relating to mass media industries. Thank you,

5 Joe, for being with us today.

6 Next, we have Claudia Perlich, who we

7 introduced before. And she gave us that fabulous

8 presentation to start us off today. Thank you for

9 being with us, Claudia.

10 And finally, to complete this very great

11 panel, we have Ashkan Soltani, who will be giving us a

12 presentation later on today. Ashkan is an independent

13 researcher and consultant focused on privacy, security,

14 and behavioral economics. He previously worked here at

15 the Federal Trade Commission and he was the primary

16 technical consultant on the Wall Street Journal's "What

17 They Know" investigative series. Thanks for being with

18 us today.

19 All right. So with that, let's go ahead and

20 begin. And I would like to start today with Rachel and

21 Stuart. Why don't you tell us a little bit about the

22 history of these products. How exactly did they come

23 about, alternative scoring products?

24 MS. THOMAS: You want me to start?

25 MS. ARIAS: Sure.
22

1 MS. THOMAS: Okay. Can you guys hear me?

2 Are these on?

3 MS. ARIAS: Yes, if you -- I'd like to remind

4 everybody to make sure you speak into the microphones.

5 I know they're a little sensitive, but otherwise the

6 folks in the webcast won't be able to hear us.

7 MS. THOMAS: Great. Thank you, Andi and

8 Katherine, and good morning. Lovely to see all of your

9 faces.

10 So I'm going to talk about marketing

11 analytics, predictive analytics, similar to what

12 Claudia introduced to us, because that's really a much

13 better term to describe really what's going on in the

14 marketing world.

15 So let's start with, the goal of marketing in

16 every case is to meet consumers where they are with an

17 offer for a product or a service or a cause that they

18 might be interested in that is going to be of interest

19 to them. So predictive analytics, no surprise, predict

20 a consumer's likelihood or propensity to be interested

21 in that particular product or service. That's the

22 goal.

23 Now, of course, everybody here knows everybody

24 gets marketing offers. The difference with predictive

25 analytics is that those offers are more likely to be
23

1 actually valuable to the consumer or the donor or the

2 potential voter. So a consumer's propensity, of

3 course, to buy a particular item is always changing.

4 If I bought a car yesterday, I'm probably not going to

5 buy one tomorrow. So those predictions about the

6 future are constantly changing as well.

7 So what markers are interested in always is

8 that extremely dynamic set of interests that a consumer

9 has from day-to-day. Not just at any given point, but

10 also in the different contexts, whether online or in a

11 store, et cetera.

12 Now, it's important to recognize -- Claudia

13 talked about the latest-and-greatest, but businesses

14 and others have been using predictive analytics for

15 more than a hundred years. Back in 1888, when Sears

16 was getting started with their first catalog, they made

17 the very smart prediction that folks living out in the

18 rural west were going to be more interested in a

19 catalog of consumer products than folks that had access

20 to a lot of stores nearby and could walk in and buy

21 them themselves. So they focused their marketing in

22 the rural west, instead of those of us hanging out on

23 the east coast.

24 Similarly, in 1912, L.L. Bean made another

25 smart prediction that folks who had hunting licenses in
24

1 Maine, but lived outside of the state of Maine, were

2 probably going to be interested in a catalog with

3 hunting goods in it. And so again, that was how L.L.

4 Bean got started, with a very smart prediction and the

5 purchase of a list of folks with out-of-state hunting

6 licenses from the state of Maine.

7 So fast-forwarding back to today, Microsoft

8 had some really interesting research that came out just

9 a few months ago talking about how, in asking consumers

10 what they're looking for, they want more

11 personalization, not less. Not just in the offers they

12 get, but a seamless experience, whether they're in a

13 store, or on an online site, or even in a mobile

14 version of a retailer's site. They want to be

15 understood throughout that whole purchase journey.

16 They don't want gaps between all of those experiences.

17 So to meet those kinds of fast-moving and

18 very personalized expectations that consumers have,

19 marketers use those predictive analytics to make sure

20 that they meet the customer wherever he or she is, with

21 what they're most interested in, and however they're

22 most interested in engaging. Whatever different

23 context.

24 So for example, today in a department store,

25 the store might look at what a customer has bought in
25

1 the past, different products from different departments

2 of that store, and look at its larger purchase history

3 to say what other customers have bought those products

4 and been interested in other things that a customer

5 hasn't yet purchased. So they are going to analyze

6 that and compare and, using those predictive analytics,

7 they're going to guess. They are going to guess

8 whether you were more likely to be interested in the

9 coupon for the jewelry department or the kitchen

10 appliances department, because maybe you bought that

11 car and now it's time for a refrigerator, or apparel,

12 et cetera, et cetera, et cetera. So that's business.

13 Nonprofits, you may or may not realize, use

14 very, very important uses of predictive analytics to

15 keep fundraising costs down by focusing on the people

16 most likely to donate. But also to hone in on

17 populations in greatest need of assistance and tailor

18 their outreach to those populations, to make sure that

19 they're most easily able to engage those in greatest

20 need.

21 The Humane Society and World Vision recently

22 have upped their ante in terms of targeted fundraising.

23 They have actually created statistical profiles, not

24 dissimilar to what Claudia was talking about, of their

25 major donors so that they can then go out in the
26

1 marketplace and look for others that fit those profiles

2 of folks likely to make large donations to their

3 organizations.

4 Political campaigns as well, incredibly

5 important users of predictive analytics, to target

6 political advertisements, whether in the mail, online,

7 in realtime. Pandora has a great new service coming

8 out that will let candidates or political organizations

9 target those of us who spend our days with Pandora on

10 in the background as we're working on whatever, this

11 and that.

12 So how are they going to do that? They're

13 going to look at public data of who won what elections,

14 in terms of candidates in different zip codes. And

15 then, you know, when you sign in to Pandora, you put in

16 your zip code, they are going to see who listens to

17 what music in those zip codes. So when a song that has

18 been identified as, perhaps, making a correlation to an

19 interest in a particular party comes on, you're going

20 to get an ad for that candidate or that party as well.

21 It's as simple as that.

22 So in all these cases, the organization is

23 looking at, it's analyzing information that it already

24 has about its customer or its donor or its voter to

25 understand and make a best guess to predict what else
27

1 that individual is likely to be interested in.

2 Sometimes they can make these often, they can make

3 these predictions just by analyzing the information

4 that they have themselves, maybe having third-parties

5 help them with the analytical power, like Claudia's

6 company and others do.

7 Sometimes they might need additional

8 information in order to make that leap to the next type

9 of prediction. So they might go to a third party, a

10 marketing information service provider, some sort of a

11 company, who can help them with the analysis and with

12 additional information to figure out what that customer

13 might want next.

14 So predictive analytics are important, not

15 just for keeping your existing customers and donors

16 happy, but for finding prospective customers and donors

17 and voters as well.

18 So if a company, for example, knows that

19 customers are most likely to buy navy blue suits if

20 they are women of a certain age in a large urban area.

21 So if they know that, they might go to a marketing

22 information service provider and say, the women who

23 like the navy blue suits, what else are those folks

24 fitting in that demographic likely to buy. And they're

25 going to find out that a pair of nude heels,
28

1 nude-colored heels, is going to the perfect thing to go

2 with that suit. And they should serve a coupon for

3 that, instead of a purple set of shoes, for example.

4 So very important decisions affecting our daily lives.

5 Mine, at least.

6 So some of you may be asking yourselves, what

7 is new here? This isn't surprising, or maybe it is,

8 but what's new? The predictive analytics obviously are

9 not new. As Claudia rightly described, what's new is

10 the analytic technology that helps get the predictions

11 right. What's new is the power to actually get it

12 right and give an offer that is of interest to you, as

13 opposed to the person next to you.

14 So taking a step back again, whether this is

15 your bread-and-butter, whether this seems shocking or

16 magical or completely mundane, at the bottom line, it's

17 really important to remember what this is all being

18 used to accomplish, relevant marketing. And that's it.

19 This is marketing data being used only for marketing

20 purposes, and I'm happy to talk more later about how

21 DMA makes sure that that's true, to predict the

22 likelihood of a consumer being interested in a certain

23 product or service over another.

24 Marketing data is not used to determine that

25 individual's eligibility to receive a product, like a
29

1 financial product or an insurance product, and it's

2 certainly not used in any other kind of eligibility

3 decision either.

4 So at the end of the day, for better or for

5 worse, the biggest impact that marketing analytics will

6 have on a consumer's life is whether or not that

7 individual gets an ad or an offer that is relevant to

8 her interests or one that is not. And we would argue

9 that the proof that predictive analytics are valuable

10 to consumers is in the consumer reaction when they do

11 get a relevant ad.

12 Andi mentioned the research that I worked

13 with folks at Harvard and Columbia on, on looking at

14 the value of data. And in this area of marketing

15 analytics and the flow of marketing data, we found

16 that, in 2012 alone, $156 billion were added to

17 the U.S. economy, 675,000 jobs in the U.S. alone, and

18 70 percent of that value was derived by that flow of

19 data being used for analytics between first and

20 third parties in responsible ways.

21 So we would argue that that's a value worth

22 preserving and one that is incredibly important to

23 getting it right for customers.

24 MS. ARIAS: Before we go and jump maybe into

25 what Stu and Pam may want to say --
30

1 MS. THOMAS: Sure.

2 MS. ARIAS: -- about the uses, I wanted to

3 give Stu an opportunity to maybe talk about other uses,

4 besides just marketing, that these alternative scores

5 or predictive analytics are being used for.

6 MR. PRATT: Okay. Well first of all, thank

7 you all for inviting CDIA to be on the panel. It is

8 good to be here. I like the word fabulous. I've

9 decided I'm going to use that in some of my other

10 presentations.

11 MS. ARIAS: You are fabulous.

12 MR. PRATT: And Ed and I always feel -- you

13 know, we're on a lot of panels together. So Ed, I

14 think in the future, that will be our theme, fabulous.

15 MR. MIERZWINSKI: And I'm fabulous. But

16 note, I'm to his left.

17 MR. PRATT: Right. And appropriately so, I

18 think. So even the seating charts are worked out just

19 right.

20 So I'm precocious. My family would disagree,

21 and I say it often to them, I'm precocious, and they

22 continue to ignore that, particularly my sons. But I

23 didn't realize that I was a big data analytics guy when

24 I was a child, so I want you to know this.

25 I lived overseas and the only way that we
31

1 knew what we wanted was when we waited for the catalogs

2 from the United States to arrive in the 1960s at our

3 house. And so our parents, one year, asked my brother

4 and me, well, what do you want for Christmas? And we

5 decided the best way to present our wants, our many

6 wants, many, many wants, was to tear pages out of the

7 catalogs from the various retailers.

8 And then Jim and I actually went into their

9 bedroom and taped them to the ceiling of their bedroom.

10 So that was just-in-time advertising, delivered at just

11 the right time, with big data analytics, right down to

12 the level of our needs, so that our parents could meet

13 our needs in a way that they otherwise wouldn't have

14 been able to do so.

15 So I just want you to know that big data has

16 been around a long time, including in this very

17 sophisticated way that my brother and I pioneered many

18 years ago. If only I had known that, I probably

19 wouldn't be working here today, I'd be retired on some

20 coast and looking at oceans.

21 So we're -- CDIA, Consumer Data Industry

22 Association, we really work with another part of U.S.

23 dataflows. We work with companies that are aggregating

24 data to manage risk. And risk matters, it matters a

25 lot. At one time -- I used to have a harder time
32

1 convincing folks of this, but if you just say two

2 words, Great Recession, we all kind of get it. Risk

3 matters, in a lot of different ways.

4 In the 1990s, risk was mostly focused on what

5 we call prudential risk, you know, how do you make a

6 lending decision, how do you make sure that banks are

7 safe and sound. But most folks shrugged their

8 shoulders and said banks are always going to be safe

9 and sound. They all seem to be doing pretty well,

10 bricks-and-mortar looks okay. But in the late

11 ‘90s, we began to see identity theft cycle up and

12 we realized there were different risks, risks that had

13 to do with whether or not we actually knew the consumer

14 with whom we were doing business. And because of the

15 internet, which really was something that began for

16 most of corporate America in, you know, the early

17 1990s, flowing into the next millennium, it began to

18 also be many, many more transactions that were

19 essentially where the consumer was remote to the

20 transaction. They didn't know who that consumer was.

21 So my job here today is a little bit complex

22 though, because the majority of the transactions we

23 talk about, the majority of the dataflows that we

24 represent, are regulated under a variety of different

25 laws. And Katherine has assured me that I'll get
33

1 pulled off the panel if I spend too much time talking

2 about all those laws, but I'm going to do that just a

3 little bit, just a little bit.

4 But our members’ dataflows do a couple of

5 things really well. They encourage competition, for us

6 as consumers. And that's good for us as consumers,

7 more offers, different offers. It gives us a chance to

8 evaluate different offers. Those offers can be at our

9 desktop, those offers can be delivered in a variety of

10 different ways.

11 It also -- our data is really a framework of

12 safety wrapped around the U.S. economy. Fraud

13 prevention is elemental in a lot of different ways.

14 It's identifying the consumer in a card-not-present

15 transaction. It could be as simple, by the way, as a

16 retailer who doesn't have a bricks-and-mortar operation

17 trying to understand whether or not an address to which

18 they're sending a very expensive item is or is not

19 zoned residentially.

20 It could be devices today, and device

21 recognition strategies, to try to understand whether or

22 not I, with the -- if I'm a business and I have a

23 current, ongoing relationship with a consumer, whether

24 or not I recognize the device that the consumer is

25 using as he or she engages in these transactions.
34

1 So there's layers and layers of fraud

2 prevention that occur. It's all seamless. We don't

3 see it, we don't feel it, we don't think about it.

4 We're only upset when we discover that we've become a

5 victim of some type of crime.

6 Nine billion times year a our members’ data is

7 used in what we'll call a risk transaction of some sort

8 in the United States. Our members are also the largest

9 global companies, delivering and propagating these same

10 types of services around the world, to some of the

11 fastest developing economies, economies like Brazil and

12 India and so on. They know what they're doing.

13 They are managers of big data. They are

14 managers of big databases of data. It's primarily

15 structured information, though. Sometimes it's hard to

16 know what the definition of big data really is, but the

17 kind of data that our members are gathering could be

18 derived from fairly sort of pedestrian sources. It

19 could be public record data gathered in the United

20 States that could be used, which helps us with mortgage

21 frauds and flipping and issues that have to do with the

22 safety and soundness of the mortgage that's applied to

23 a property.

24 And like I said before, it could be a

25 database of known fraudulent applications that have
35

1 been pooled by various retailers or other transactors

2 in the marketplace.

3 If there's a dividing line between Rachel and

4 me, though, and it's a great symbiotic dividing line,

5 there's a baton pass. Our members benefit tremendously

6 from the fact that there is this robust, incredible,

7 targeted system that connects consumers with what they

8 want. And in America, that's okay. We like buying

9 things. We like engaging in the marketplace. We like

10 seeing that offer that makes sense to us.

11 Then, there's a baton pass. A consumer

12 chooses to click and a consumer chooses to apply for

13 something. And that's more often where our members

14 then kick-in to the process. More often where -- and

15 in fact, if you're in the financial services space,

16 more often where you are complying with laws like

17 Section 326 of the U.S. Patriot Act, know your

18 customer. Red flags rules, promulgated by government

19 agencies such as the Federal Trade Commission, are properly

20 protecting consumers against identity theft.

21 So there's this, again, this confluence of

22 dataflows that occur on the front end of an

23 application, which is occurring sometime after I've

24 seen banner ads or I've shopped in bricks-and-mortar

25 stores or done whatever I do as a consumer to figure
36

1 out what it is I want and at what price I want it and

2 so on and so forth.

3 But some of those laws that regulate our

4 industry, Fair Credit Reporting Act, Driver's Privacy

5 Protection Act, the Gramm-Leach-Bliley Act, Title 5,

6 again, not the topic of today, but important for you to

7 know that these laws wrap around a lot of these

8 different databases that are out there for consumers.

9 Because these databases particularly -- and Rachel used

10 an important term, eligibility. Once you cross over

11 the line into eligibility, one you cross over into what

12 I call gate-keeping, you get a yes or no. Or the yes

13 that you get is the best yes on the list or it's a

14 qualified yes, somewhere down on the list, you pay us a

15 little higher price or a lower price, all of that is

16 regulated under a variety of, for example, fair lending

17 laws, the Equal Credit Opportunity Act, the Truth in

18 Lending Act, and so on and so forth. So once you get

19 into the application context, you're back in that world

20 of laws that wrap around the transaction.

21 So let me just give you a couple of examples.

22 Lending Club, I want to go back to Lending Club. I

23 don't know a lot about Lending Club, but I'm just

24 saying that if Lending Club has a new, innovative way

25 of managing big data to try to make a lending decision,
37

1 even though Lending Club is using new data sets, they

2 are still obligated to comply with the Equal Credit

3 Opportunity Act. They are still obligated to comply

4 with the Truth in Lending Act. They are still

5 obligated to make sure that they don't have disparate

6 impact problems. The confluence of all of these laws

7 still applies. So there's nothing new about the use of

8 that type of information, when it's in the context of

9 that type of application process.

10 And then the different example. The CFPB was

11 looking an annualcreditreport.com, by the way. By the

12 way, let me say that again, get your free report every

13 year at annualcreditreport.com. It's my little

14 advertisement, but not a bad one, right? Not a bad

15 one.

16 So 16 million consumers roughly are looking

17 at free credit reports each year, out of 200 hundred

18 million-plus consumers, so it's not a huge amount. So

19 the kinds of analytics that have been discussed by

20 Claudia, and also by Rachel, might be one way for us to

21 reach out into that community of consumers more

22 effectively and try to find those consumers who we

23 think would benefit from accessing free credit reports,

24 but aren't doing it today.

25 So there's, I guess, a social good example of
38

1 how there's a nexus between get your free report, be

2 credit report literate, make sure that you understand

3 what's in your credit report, you know, all of those

4 things that we believe in, and the analytical tools

5 that can allow us to kind of get to that point and

6 reach those consumers who we think are most likely to,

7 you know, point-and-click and move forward.

8 So that's just an example of the power of

9 this kind of information and how it connects consumers

10 with, sometimes, maybe something they don't know that

11 they should do, but they get it.

12 MS. ARMSTRONG: And that's a great example,

13 Stuart.

14 As Andi and I are mindful of the clock, we'd

15 like to turn quickly to Ed and Pam to see if they could

16 mention some other products, what they might be used

17 for, and then before Ashkan's presentation, we want to

18 talk about data accuracy.

19 So I want to make sure we get to that before

20 the top of the hour.

21 MR. MIERZWINSKI: Well thank you, Katherine

22 and Andi. My work at U.S. PIRG is as a consumer

23 advocate. And I'm concerned not about the data, per

24 se, I don't think anybody is. I'm concerned about its

25 use and its impact on financial opportunity.
39

1 And I'm also going to give a disclaimer, as

2 Stuart did, that I'm not here to talk about the Fair

3 Credit Reporting Act, but I have to mention it

4 peripherally, or at least in passing.

5 For 40 years, financial marketing of the most

6 import kind, based on your credit report, the most

7 detailed profile about you, has been governed by the

8 prescreening rules of that Fair Credit Reporting Act.

9 That law says, if a company wants to use your detailed

10 financial profile to market to you, it can only market

11 to you for credit or insurance purposes, not direct

12 marketing, it must give you a firm offer of credit, and

13 you have the right to say no to that use of your

14 information. You have the right to say no to using

15 your financial profile for marketing to you. And the

16 kind of marketing that can be done is extremely

17 limited.

18 I am very concerned that we are moving to a

19 new system of unregulated, wild west companies, running

20 roughshod over consumer rights on the internet and

21 making decisions about what ads to serve to you, maybe

22 not directly determining eligibility yet, but deciding

23 what box to put you in, what place to direct you from

24 your cookies and from the other information that they

25 have about you, and possibly causing you to pay more or
40

1 get fewer opportunities than other consumers. That's

2 the short version, I've got much more to say.

3 And by the way, it's USPIRG, P-I-R-G. If you

4 go to my blog, on the homepage today there's a lot more

5 detail and links to some of our materials, including my

6 paper with Jeff Chester at the Suffolk University Law

7 Review.

8 MS. DIXON: Good morning. Thank you so much

9 for the invitation, I really appreciate it. This is a

10 great panel and I really appreciate the opportunity to

11 share a discussion about this important topic.

12 So let's begin with the fact patterns here.

13 So the first fact pattern is that scores are

14 proliferating. In the past, when the credit score was

15 developed, the credit score used limited factors, well

16 under 100, they were controlled factors. In fact,

17 those factors that are used in the credit report are

18 regulated. They cannot be discriminatory, they cannot

19 be prejudicial. And Congress did this for a very good

20 reason, the same kinds of reasons that they passed the

21 Civil Rights Act. There should not be any kind of

22 hidden discriminatory factors in scores, this we can

23 all accept as a baseline. So that's one thing.

24 The second thing is that the large data set

25 world that we're living in is not going to reverse
41

1 itself somehow. That genie is well out of the bottle.

2 So given that, really one of the ways that all of us

3 make sense of our world is by shortcutting.

4 Understanding data and predictive analytics allows us

5 to do that, the machines do the hard work of sifting

6 through petabytes of data for us. So the results of

7 that are spit out are often scores. Scores can have

8 varying ranges, they can have varying values, and can

9 mean completely different things, depending on the

10 factors that are fed into the score, as Claudia

11 discussed, the algorithm, and then, of course, we're

12 talking about the use of the score.

13 The really important thing here is that the

14 credit score had a very focused purpose. Today, with

15 the real proliferation of the technologies that allow

16 more and more retail and enterprise and small

17 businesses to create predictive analytic scores and

18 tools and results, it's becoming more important to find

19 out what other scores are out there. And that's the

20 second fact pattern, there's a lot. So that's the

21 second thing.

22 The third thing is this, the credit score, as

23 a controlled score, has been very, very carefully

24 observed and has a lot of oversight. The new scores

25 don't enjoy that same kind of protection. So here's my
42

1 thinking on this. We really need to understand that

2 there is a continuum of scores here. Not all scores

3 are bad. In fact, some scores are actually helpful.

4 The Equal Credit Opportunity Act mentions specifically

5 credit scores and how they can assist in reducing

6 discrimination in lending. This is a good use of a

7 score and it's a regulated use of a score and it's

8 appropriate.

9 So today, we really need something like that

10 to look at these scores that have proliferated and are

11 new, so let's talk about some specifics. The credit

12 score, a few factors, and a static score, doesn't

13 change that often and that much, unless you really game

14 the system. And that's a whole different matter. Not

15 for this panel, right?

16 If you take an aggregate credit score,

17 however, an aggregate credit score and actually some

18 modeled credit scores, can use 1500 factors. These

19 factors are in a big black box, we don't know what

20 those factors are, we're not told what the factors are,

21 and yet Claudia's presentation was completely correct

22 when she said, look, you have the factors that go into

23 a score, really that's everything. Good factors in,

24 nondiscriminatory factors in, much better chances of

25 getting a nondiscriminatory score on the backend.
43

1 But if there are credit-related or any kind

2 of eligibility-related scores that have discriminatory

3 or prohibitive factors that are used in the score soup,

4 we've got a big problem. But we won't know that we

5 have a problem, because right now most scores are

6 secret, with the exception of what I would call social

7 scores like Klout. Consumers don't have the

8 opportunity of learning about the scores, because there

9 is no transparency for them and certainly the factors

10 are secret. So we've got big problem there.

11 Now having said that, there's something

12 really important to understand. And I'd like to echo

13 Ed's remarks. We did a thought experiment and we asked

14 ourselves, could the Klout score, a social-influenced

15 score, be covered by the Fair Credit Reporting Act.

16 And you just can't get there. There's a real first

17 amendment issue here that we have to grapple with.

18 There is such a thing as free speech. And you know, if

19 someone is quoted in the Washington Post, and the quote

20 happens to be not so great, right, and it makes the

21 person look bad and they don't get a job because of

22 that quote, does that mean the Washington Post should

23 be regulated? No. None of us think that, right?

24 So we have to be really careful here. It's

25 attention.
44

1 MS. ARMSTRONG: Pam, that's an excellent

2 point. And I want to save some of this conversation

3 about the parameters until the second half.

4 And although I have a very specific thing I

5 want Joe to share with us, I think that right now, this

6 is an excellent segue into the accuracy issue. And as

7 Claudia mentioned, quality is hard. And as we look at

8 it, accuracy has two components. One is going to be

9 the model and the other is going to be the data.

10 So I'm wondering if anybody could speak to

11 how companies determine whether the data -- well,

12 whether there are certain sets of data that are

13 inherently more accurate than others and is data

14 accuracy relevant for all types of scores? So I'd like

15 to throw that out for a few minutes, if anyone wants to

16 comment on that.

17 MS. DIXON: Can I just jump in?

18 MS. ARMSTRONG: Sure, please. Absolutely.

19 MS. DIXON: Data -- scores are coming from

20 public data, they are coming from demographic data,

21 enterprise, social, even some health data. There is

22 even financial interest and activity. So the accuracy

23 of data is a huge issue. It's very, very difficult to

24 create a score above 95 percent, everyone knows that.

25 I think that, from the analysts I've talked to, a score
45

1 above 85 percent is just awesome.

2 But I think that there's really no way for

3 anyone who uses thousands of factors in a score to

4 completely assure that each factor is accurate. I just

5 don't see it.

6 Now hopefully, there will be a lot more

7 transparency in the industry and we can find out a lot

8 more about this. And that's what's incredibly

9 important.

10 MS. THOMAS: Can I add to that?

11 MS. ARMSTRONG: Sure, absolutely.

12 MS. THOMAS: So I think when we think about

13 accuracy first, again, from the marketing perspective,

14 you want to have an accurate ability to predict what

15 someone is going to be interested in. You are more

16 likely to have a sale at the end of that, so yes, the

17 data being accurate in order to make that prediction

18 out of predicted analytics is a good thing.

19 That said, when it comes to consumer

20 protection, I think it's incredibly important to look

21 at the relationship between the importance of accuracy

22 and the use of that data. Data is data is data.

23 So when we're talking about marketing, if

24 data is incorrect, you're going to get an offer that

25 isn't relevant to you and that's the end of it. If
46

1 your credit score is inaccurate, you could be denied

2 housing, insurance, et cetera, et cetera. Very

3 important permissible uses under FCRA.

4 So I think it's important to play this out to

5 the end. When we're talking about data, what is it

6 being used for and, at the end of the day, that

7 determines the importance of its being accurate, linked

8 to the potential impact on a consumer, in terms of harm

9 to their way of life.

10 MS. ARMSTRONG: Ed. Oh, I'm sorry. Ed and

11 then Joe.

12 MR. MIERZWINSKI: I'll just be very brief.

13 But I'll say this, how do you determine accuracy?

14 Well, we need, as Pam said, more transparency. And

15 there have been a number of studies by consumer groups

16 where they have requested information about their

17 profiles from various data brokers and others. And the

18 profiles, when they've been provided, have been

19 incomplete and inaccurate.

20 By the way, I'll just make one quick point.

21 I've been a member of REI for 40 years, I've shopped

22 there for 40 years, I've been a member -- not a member,

23 but I've shopped at L.L. Bean for 40 years. And for some

24 reason, L.L. Bean, for the first time ever, just sent

25 this non-fisherman the fisherman's catalog. Maybe
47

1 they're trying to expand my horizons? I don't know.

2 But I thought that was pretty interesting.

3 MR. TUROW: I also wanted to add about the

4 question of accuracy. In huge models, from what

5 Claudia was saying and other things that I seem to

6 know, it is very difficult to know what about the model

7 is accurate or not accurate when you're predicting

8 something.

9 And the other thing I wanted to say,

10 connected to this, is HIPAA and Gramm-Leach-Bliley

11 aside, there are lots of places to get data like that,

12 that you can go around these laws.

13 There is, for example, a company called Medix

14 which has a website that will give you discount coupons

15 on serious health problem medications. So you go to

16 that website and you write in what your problems are

17 and then you get these discounts. And you don't know

18 what they're going to do with those data. Privacy

19 policies are fascinating obfuscatory. And it is very

20 difficult to know.

21 I think we have to expand the notion of --

22 the notion of a credit score and a data broker. Is

23 Kroger a data broker? Now, reading Kroger's privacy

24 policy, they seem to imply that they don't sell their

25 data, okay? But is it not selling data if Kroger
48

1 allows advertisers to put ads on sites which track

2 people and then, through the cookies, essentially they

3 are buying data that way from Krogers? It's a

4 side-door data broker activity, I would argue.

5 And so we have to think a little more

6 broadly, I would argue, about data brokers and about

7 the ways in which companies try to get around some of

8 the obvious laws about protection.

9 MS. ARMSTRONG: Just one second. I wanted --

10 before, Claudia, you talk about the accuracy thing,

11 before we leave the products and their uses. Joe, if

12 you could describe the KLM example that you shared with

13 Andi and I when we spoke with you.

14 MR. TUROW: Yeah, I remember reading about

15 this. It's not so much a big data issue, it's just

16 simply the notion that people can find out, through the

17 website, if you're going alone on a KLM flight, what

18 kinds of interests other people have and then you can

19 decide whether or not you want to sit next to that

20 person. It's not clear to me that other people have --

21 presumably everybody has the right to say they want

22 their interests put out there. If they didn't, that

23 would be a fascinating question of predictive

24 analytics.

25 But let me, as long as we're getting at this,
49

1 say one thing that did happen to me that I think people

2 would not have predicted. I was on a United Airlines

3 flight coming from Wisconsin into O'Hare and the flight

4 to Philadelphia was canceled. They told me to go to a

5 customer service place and scan the boarding pass I had

6 for the canceled flight. When I did that, it gave me a

7 number. And to the right, there was a monitor that

8 said, the amount of time that you will be taken to be

9 served will relate to your status, your loyalty status,

10 with the airlines.

11 Now I don't think that most people -- and I

12 was very fortunate, I had a lot of miles. But there

13 were these poor people sitting in the back, they

14 weren't treated for a long time. And the implication

15 of it is, some people will get flights and other people

16 won't. So --

17 MS. ARMSTRONG: So we now have a marketing

18 risk/credit and airlines?

19 MR. TUROW: Yeah. I guess I'm saying is, we

20 don't know, when we give these data, what the

21 implications are.

22 MS. DIXON: Can I jump in for just a second?

23 MS. ARMSTRONG: Please. Everybody can jump

24 in quickly, and then we're going to move to Ashkan.

25 MS. DIXON: I'm most -- I'm not as concerned
50

1 about ads as I am about eligibility issues. Our focus

2 and our research in this area has been on eligibility

3 uses of marketing and noncredit data outside of the

4 FCRA and outside of HIPAA.

5 I understand that the ad information is very

6 fascinating. It's fascinating to me, too, on a

7 research level, but we're very concerned about the more

8 impactful scores. And those are definitely use of

9 health data as factors, use of scores in health, which

10 is being done today, by the way, that include non-HIPAA

11 information held outside of the medical establishment

12 and also use of discriminatory factors in eligibility

13 decisions, which is happening today.

14 MS. PERLICH: I wanted to quickly jump on the

15 accuracy. I'll keep it -- the first one, there is an

16 inherent tension between the ability to regulate and

17 keep the model understandable, or the score, and the

18 accuracy it can reach. My being able to add 100 or

19 1000 more factors into the model, I can double the

20 accuracy and you will not understand it anymore. There

21 is that tension, we have to be aware of it. It's

22 simply the ability to predict is a function of the kind

23 of variety of stuff you give it to start with.

24 My second comment is on data brokers and

25 scores. One of the reasons we have the data, we use
51

1 the data we use, which is what I call primary, I

2 observe people's actions, is anything derived from it

3 that I can buy from data brokers is absolutely awful,

4 in terms of accuracy, from what I can tell. So all the

5 scores, whether it's done demographic, that I can buy

6 your gender, and typically people are both -- so my

7 experience has been that the data that somebody else

8 has derived somehow is really problematic. And at that

9 point, it becomes completely useless.

10 MR. PRATT: So just a couple of quick -- I'll

11 move through these like a list.

12 MS. ARMSTRONG: That's fine.

13 MR. PRATT: So just a quick thought about

14 data brokers overall, and this is just because, at

15 CDIA, we deal with that term quite a bit these days.

16 It's a newish term and it's a really undefined term.

17 So I just want to make clear that really, when you're

18 thinking about data and analytics, and actually, Joe, I

19 think you said it well, it could be a closed system of

20 data that has been aggregated by a single entity, like

21 Kroger, or a search engine, for example. And they may

22 not be selling the data, but they're certainly inviting

23 people in to make use of that data, to deliver the

24 advertisement. So we don't see, you know, this sort of

25 third party versus first party thing as particularly
52

1 relevant to the data broker issue overall, but I just

2 think it's important to lay that out and say, you know,

3 that there's both.

4 With regard to credit versus noncredit, and

5 this goes to Joe's point about getting around laws,

6 nobody is getting around laws. And we may have a

7 debate about transactions of the definitions of those

8 transactions, and I think that's what Ed is kind of

9 driving at, but nobody's getting around laws. If

10 you're engaged in making a lending decision, you're

11 regulated by the laws which regulate lenders. And if

12 you're doing it outside of those laws, you are

13 violating laws and Maneesha's team and others will go

14 out and find you and investigate you and prosecute you

15 for not complying with the financial services laws that

16 apply today.

17 With regard to developing credit scores, I

18 don't like the term credit scores versus this better

19 term, analytics. Because credit scores makes everybody

20 think about the credit score that may be based on a

21 credit report, but that's not really the point. And by

22 the way, I think a credit score developer would say, it

23 doesn't matter how many factors I have, I must build a

24 score that is successful in the marketplace. It must

25 be saleable to somebody who sees the outcome, sees what
53

1 happens.

2 So if I'm developing it for marketing, I want

3 to see that people click on the ads and engage and make

4 purchasing decisions, because it's a good analytical

5 tool that leads me to some place where I want to be to

6 make a yes decision, because I like the yes decision

7 I'm making as a consumer.

8 Credit is a little bit different. It's about

9 prudential lending, of course. And so regardless of

10 whether it’s 100 or 1,000 different factors, it has to be

11 statistically sound, it has to be empirically derived,

12 and whether or not it's a first-party score that's

13 developed by the lender in house, based on big data,

14 sort of the old Zest Cash model, or whether it's a

15 score that has been developed by any one of CDIA’s

16 members, who are some of the biggest data analytics

17 companies in the country, whether it's a fraud score or

18 whether it's a score for credit, the outcome is the

19 key. It isn't measuring factor-by-factor precision.

20 But yeah, developers will look at a whole

21 variety of factors. They may look at 100 times the

22 number of factors that ever end up in the score,

23 because they're trying to find the right sauce of

24 scores that will lead to an excellent lending decision,

25 allowing them to penetrate and get to more yeses than
54

1 nos, but to do it in a prudential, safe and sound,

2 sort of banking safety and soundness way.

3 MS. ARIAS: So Stuart --

4 MR. PRATT: So that was just some quick

5 thoughts in all of that.

6 MS. ARIAS: I think you raise an excellent

7 point. And it's actually a follow-up to you, Pam,

8 because I think you brought this up and he's

9 responding, right? What is an eligibility use that

10 falls outside of the FCRA? What is it that you are

11 envisioning in your comments?

12 MS. DIXON: Okay. So look, there are scores

13 today that are called either aggregate credit scores or

14 modeled credit scores, so let's talk about aggregate

15 for a second. The Fair Credit Reporting Act, our

16 lovely, trusty Fair Credit Reporting Act, applies to

17 individual consumers. Aggregate credit scores apply to

18 a neighborhood.

19 So if a person lives in a neighborhood, it's

20 kind of like in old England where if you lived in a

21 house where there was a debt collection, then you were

22 also a bad apple, it's kind of the same idea. And I'm

23 sorry, but how -- if you are using an aggregate credit

24 score that is a very close proxy for a credit score,

25 and offering a financial product or an insurance
55

1 instrument to a consumer, I think technically they're

2 not covered, right?

3 However, it's a really important eligibility

4 decision or a product offering and this is where I

5 think we have enormous tension. And these, I think,

6 are the important scores to focus on and they are

7 secret scores. I can't purchase my aggregate credit

8 score, I can't. It's not regulated.

9 MS. ARIAS: So I know people want to respond.

10 And I think the conversation is starting to move to the

11 idea of what the effects of these scores are, right?

12 So before we jump to effects, we really wanted to get

13 Ashkan to give us his presentation, if he wants to come

14 up.

15 So Ashkan is going to give us a brief

16 presentation about some of the privacy issues

17 associated with scoring products, especially about some

18 of the research that he and other researchers have done

19 on emerging trends in online pricing with the use of

20 these scores.

21 MR. SOLTANI: Can I just briefly --

22 MS. ARIAS: Yep.

23 MR. SOLTANI: -- just make a comment on

24 accuracy?

25 MS. ARIAS: Absolutely.
56

1 MR. SOLTANI: One thing I want -- and I'll

2 talk about this a bit in my presentation, but I want to

3 clarify that, with regards to accuracy and these

4 traditional models, this stuff isn't as black and white

5 as we are used to. It's not whether you're qualifying

6 for -- well, some of it is, whether you are qualifying

7 for a job or whether you are qualifying for credit.

8 But we should also be mindful of the ways

9 these aggregate scores or these numbers are used to,

10 what I want to call do fuzzy nudges, right? These are

11 things like how long you wait line, is one of those

12 things, where the score might have an effect on kind of

13 subtle behaviors that kind of get up close to some of

14 the things we care about. Other things are like how

15 long you might wait at a call center. So call centers

16 are actually profiling for customer service.

17 And I'll talk about this but, for example,

18 what credit cards you are shown, what credit card

19 objects you are shown and what you have an opportunity

20 to apply to. You know, how far down on the page or

21 whether it's on the next page, I think those things

22 also influence. They're not direct kind of credit

23 scoring type products, but they kind of influence -- we

24 know most people don't go to the second page, right?

25 We know most people will kind of see the options
57

1 represented. And sure, you can definitely dig, but it

2 essentially nudges people to a particular outcome that

3 I think we start, you know, caring about.

4 And they might use explicit factors. They

5 might not be -- they might be just data driven, right?

6 So, Google, for example, is notoriously well-known for

7 their HR recruitment algorithm. So when you submit a

8 resume to Google, they have this awesome matching

9 system that will kind of present you for potential

10 jobs. And it's not clear whether -- and it's likely

11 that they're using, say, sexual -- you know, sex or race

12 as an indicator, but there might be that latent

13 property that's in the system, that might emerge, that

14 we care about. But it's essentially one of these fuzzy

15 messages which is, ten resumes are presented to the

16 recruiter and they are based on a bunch of factors that

17 are essentially scoring, but not in a direct yes or no

18 way, but just the probability of. And those things are

19 a bit fuzzier, so I'll talk about a bit of those.

20 MS. ARIAS: Fabulous. Did you -- yeah.

21 MS. ARMSTRONG: So as Ashkan is setting up, I

22 think I wanted to just observe that Pam and Ed have

23 noted that we do have a statutory framework in the FCRA

24 that covers some kinds of data.

25 And I think they're correct to point out that
58

1 we're talking about these non-FCRA products, but using

2 some of the lingo that the FCRA uses, and that's a bit

3 of a challenge.

4 But now, Ashkan?

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25
59

1 EMERGING TRENDS IN ONLINE PRICING

2 MR. SOLTANI: Sweet. All right. Oh, that's

3 loud.

4 Everyone, so I want to just briefly talk

5 about some of the research. So I want to just keep it,

6 since it's a five-minute quick presentation, I'm just

7 going to talk about some of the research that I did in

8 the past on kind of scoring and use of data.

9 And so I'm going to briefly talk about the

10 methodology and how I did it, the findings, and some of

11 the -- some comments on data sources. And so most of

12 this research was from a piece we did for the Wall

13 Street Journal on sites varying prices and offers to

14 consumers based on data about them.

15 And the methodology was really kind of basic.

16 We kind of crawled the web with a variety of

17 user-agents. This is, you know, your mobile device or

18 your browser, it's known as user-agent. Sometimes

19 Firefox is a user-agent, sometimes Safari means you're

20 using an Apple product, usually. You know, whether you

21 are using a mobile device, that's your user-agent.

22 We looked at different proxies for different

23 locations in the world, like multiple locations in the

24 U.S. and multiple locations in the world, and then we

25 built profiles. We built essentially user profiles
60

1 that would, say, browse, you know, (inaudible) sites

2 and Scrabble kind of games versus, you know,

3 potentially someone that only connects to the West

4 Coast that checks car sites and electronics, right, to

5 try to generate profiles. And we verified these

6 profiles by looking at various dashboards for the

7 profile managers.

8 And so we would essentially go through and

9 iterate the web and check a particular website, say,

10 1,000 times or, you know, tens of thousands of times

11 with the right permutations to see, statistically, what

12 offers get offered, you know, presented to who. And so

13 essentially kind of black-boxing or attempting to

14 black-box some of these.

15 And so some of the basic findings. So

16 user-agent, this is, you know, what browsers people

17 use. Some people might have kind of read this article

18 where, this wasn't our research, but it was another

19 team, but they had identified how Orbitz was simply

20 showing luxury hotels to Mac users, right?

21 So this was, how far down on the page or

22 whether you have to go to the next page. And this is a

23 subtle nudge. They argued that, well, look, like Mac

24 users are posh, they want to spend a lot of money on

25 hotels, so let's give it to them. There's no problem
61

1 with that, this is just kind of a highlight. We

2 actually found that, in fact, Orbitz was giving

3 discounts, they'd call these Mobile Steals, to mobile

4 device users, smartphone users, right?

5 So again, probably no problem with that.

6 This was actually not Orbitz doing it, this was the

7 hotels providing these offers through Orbitz. Orbitz

8 provided them the functionality, the technology to do

9 it.

10 But one thing to think about is, for example,

11 who buys smartphones? Usually people who buy

12 smartphones have some disposable income to pay for the

13 data plan and pay for the $200 for the phone

14 versus the free device. This was a few years ago. And

15 so again, a good proxy for a loss leader for higher net

16 worth individuals possibly, but also simply just using

17 the user-agent as a proxy for this kind of thing.

18 I don't know if people fly, but in-flight --

19 probably one of my first experiences was when you fly,

20 like on Virgin America, and you have in-flight wi-fi,

21 the price they charge you is based on your user-agent.

22 So this is both my desktop, but one advertises as an

23 iPhone and the other advertises as a regular, you know,

24 Firefox browser and they charge different prices.

25 And they might argue that you -- you know,
62

1 maybe mobile phone users use less data than iPhone.

2 Although both have an iPad video streaming app, for

3 example, so it's not clear. But again, this is another

4 example of like charging different prices. I don't

5 know the legality of this, don't try this at home kind

6 of thing, but it's a proxy for your device.

7 So location is another -- and Pam kind of

8 raised a really good point about kind of location being

9 an indicator for other factors, not just where in the

10 world you are, right? So we looked at Staples, right?

11 So Staples is the office supply store and they were

12 charging different prices for goods based simply on

13 your zip code.

14 And kind of we dug into a bit and it turns

15 out that, in fact, the algorithm seem to have a higher

16 likelihood to charge you more based on whether or not

17 you were near, further or closer to a competitor's

18 store. So they mapped out where the competitors were

19 and essentially were charging you, you know -- in

20 staplers, it was a dollar or two, but on some of the

21 other items, it was up to $100 for the same

22 safe. So you could order something online and,

23 depending on where they thought you were, they would

24 charge you a different price for the same good, right?

25 We also found that it wasn't just Staples.
63

1 Other suppliers, Home Depot, for example, Rosetta

2 Stone, were giving you discounts based on where they

3 thought you were, right? Where you were located. And

4 again, even Discover, the kind of -- the credit card,

5 was providing their it Card offers only to certain

6 regions.

7 So if you tested the website, if you visited

8 the website or if you visited sites that featured their

9 product, they would only show that it Card to certain

10 regions. The card had different benefits, different

11 kind of deals, essentially, associated with it.

12 And what's interesting is we also looked at,

13 for example, try to correlate the Staples stuff with

14 weighted average income and, in fact, there was a

15 correlation. So the places where people were getting

16 charged more were, in fact, places -- places that

17 people were being charged less were, in fact, places

18 where people made more.

19 And then we're -- and so again, this is an

20 interesting kind of commentary where it's good enough

21 to provide offers or show you deals, you can always go

22 to a store to see what the price is. But online, it

23 was a little bit different because, in fact, all you

24 have to do is, in this case, you didn't need a proxy,

25 you could just set your cookie to whatever zip code you
64

1 want to pretend to be, and they were relying on this

2 inaccurate signal as a way to price you.

3 And then the last kind of important component

4 of that methodology was kind of permutations and

5 profiles. And I'll admit this was kind of one of the

6 least kind of successful parts of our research,

7 partially because it is very difficult to black box the

8 profiles, as they change each time you sample them, you

9 corrupt your profile each time you use it, right? So

10 if I check a kids site as an adult, I might then

11 attribute to those factors.

12 The other thing is the spam algorithms and

13 the fraud detection algorithms are incredibly good, so

14 they quickly identify whether you are a human or not

15 and so they kind of -- the ad engines will kind of look

16 for click fraud and so we were tripping up a bit

17 against that. I have a new methodology of how I would

18 do this next time, which is a bit more robust, kind of

19 crowd-sourcing it essentially, or maybe more of like a

20 botnet. But a consensual botnet. A consensual botnet.

21 But some of the things we found, for example,

22 Nextag is a search engine where you can search for

23 products, right? You can search for, again, I don't

24 know why we were focusing on scanners and office

25 supplies, but if you went through Nextag, you would get
65

1 cookied as a Nextag kind of customer. And as such, you

2 would get different offers, you know, up to $50 to $100

3 difference for the same items based on whether

4 you had these cookies.

5 So again, profile-based kind of -- and they

6 were doing really clever stuff to not trip up, not to

7 let their competitors know that they were charging

8 lower prices. So they would, in fact, show the price

9 as a GIF, as an image, so that if you crawled their

10 site, the price would be $499, a computer would think

11 the price was $499, but the image that a human would see

12 would be $350, right? And so they were trying to

13 actually adjust and score differently on this site and

14 then let users see the price. It was kind of some

15 clever stuff they were doing.

16 And then finally, this is earlier work

17 that one of my colleagues did and then we followed-up

18 and verified the same results, which is CapitalOne

19 credit cards. When you go to their site and try to

20 pull up -- I think Emily actually worked on this story

21 as well. When you kind of pull up a website and

22 different credit card offers, you would get --

23 essentially CapitalOne would ping a data supplier,

24 X+1, which is one of these kind of data networks, and

25 they would score you as the type of customer you are,
66

1 the type of credit you are likely to have. And those

2 would, in effect, show what credit card offers you

3 received.

4 And ultimately, of course, when you went

5 through and applied for the card, the credit decision

6 would hit a real credit check, right? Your

7 qualifications of whether you qualify for one would be

8 coming from, you know, Experian or one of these

9 databases. But the offers you were given were based on

10 this kind of fuzzy data that we're talking about.

11 And we found, you know, we found the same

12 type of thing pulling data from Double-Click and

13 RU4, which is X+1. This is still -- this was last

14 night, or a couple of nights ago, I just verified that

15 they still use these indicators. And to the degree

16 that, you know, Claudia said, we all know what data --

17 some of these are actually explicit. So some of these

18 they will explicitly ask for the types of sites or

19 credit scorings and there is an API that specifies and

20 a programming interface that specifies the type of

21 customer that is.

22 So I've done some work with like some online

23 dating apps and they specify what your marital status

24 is or whether you smoke or whether you do drugs. And

25 these are not fuzzy factors, these are explicit
67

1 categorizations of what the data broker thinks. And

2 they could be inaccurate, but they are at least

3 specified explicitly, which is what's interesting. So

4 whether you are high credit or medium credit.

5 And mostly, I want to just talk about the

6 data sources, because this stuff is made possible

7 by a number of front-end engines. This is Omnitures

8 Test and Target Suite, all it is is it lets you kind of

9 experiment with different offers, different profiles,

10 and see what's the most optimum for whatever task

11 you're trying to achieve.

12 And so again, nothing inherently wrong about

13 the system. It's a very easy-to-use, quick-to-use

14 system. You can vary the language and you can vary

15 some of the attributes of, you know, the price, et

16 cetera, but the key is that you can integrate this with

17 a variety of data sources, right? So you can pull

18 data, you can use for A/B this testing, a variety of

19 factors.

20 This is, for example, geo-targeting or sex or

21 age. And the question -- the thing that it raises is

22 that you have these sophisticated tools that simply are

23 just doing optimization, right? But they are pulling

24 from a set of data, a large data set. Everyone has

25 seen this graphic, it's like the most overused graphic,
68

1 but it does indicate that almost everyone in this kind

2 of red box is someone to purport to sell some sort of

3 data or provide some sort of scoring that you can use

4 to use this kind of A/B testing.

5 And so this stuff is going to have

6 interesting outcomes that we've not anticipated,

7 relying on inaccurate data without a lot of

8 transparency, and very hard to black-box, as I've said.

9 And so I think that's kind of the topic we want to dive

10 into.

11 And just as a final note, we ordered the

12 same -- we ordered the same stapler for two different

13 prices and they both were the same. They worked the

14 same.

15

16

17

18

19

20

21

22

23

24

25
69

1 PANEL DISCUSSION

2 MS. ARIAS: Thank you so much. All right,

3 with that presentation, I really wanted to give the

4 chance for the panelists, particularly Rachel and

5 Stuart, I would like to give you a chance to kind of

6 give us your thoughts about some of the kind of

7 findings that Ashkan has found, in terms of the use of

8 these predictive models and the effects that it might

9 be having on consumers.

10 MR. PRATT: Do you want me to dive in or --

11 MS. THOMAS: Do you want to go first this

12 time?

13 MR. PRATT: Okay. So I'm just going to

14 quickly work my way through bullets and then Rachel can

15 do an excellent job representing a much more detailed

16 and probably better presentation of all of the details.

17 So a couple of things. First of all, I used

18 to run retail stores all across the Washington area in

19 another life, many, many years ago, and of course we

20 rewarded loyalty. I mean, I don't know what the

21 epiphany here is with -- you know, when I was on an

22 airplane and they were pulling bags off of the

23 airplane, you're darn right I wanted my gold card stuck

24 to my bag, because I was loyal to that airline, to be

25 the last bag they pull off the airline, because I'm
70

1 hoping my loyalty means something. 

2 So I don't know that that's an epiphany, that's

3 been around forever. Everybody wants to sell, everybody

4 wants loyalty, everybody wants to reward loyalty because you

5 want more people to come back. So that's just kind of a

6 macro thing. There's no ah-ha moment there. Everybody

7 wants

8 to be treated as a loyal buyer, that's all. I mean, again,

9 whatever the point is.

10 With regard to differential pricing or

11 whatever the term is, just a quick reaction to that as

12 well. I mean, I go to the airline website and then I

13 go to Kayak or some other aggregator and I look across

14 both of them to see. So I guess the big message there

15 is, you know, frustrate the analytics companies and go

16 shop really aggressively and make sure that you

17 understand the different pricing opportunities that are

18 out there. And don't be linear in terms of how you

19 behave online. So again, I mean, I think there is a --

20 it's not that there's an unsafe marketplace out there,

21 but consumers certainly should shop.

22 With regard to the alternative data, other

23 data that's out there, just remember that these new

24 data sets are potentially going to allow us to reach

25 consumers who are often not included, consumers who are
71

1 sort of the unbanked and the under-banked, and to bring

2 them into traditional lending contexts and others. And

3 it may start with marketing offers before you ever get

4 to firm offers under laws like the Fair Credit

5 Reporting Act.

6 So big data is an opportunity for inclusion.

7 It's got to be done right, the accuracy standards will

8 apply when you move into the lending context. The

9 fairness protections of ECOA and other -- Equal Credit

10 Opportunity and other laws will apply. The protected

11 classes of consumers are protected. But that's a

12 really important point, that these data are

13 opportunities to include, not just to exclude. I mean,

14 I think so often this is a glass half-full kind of

15 discussion. I think it's a full glass discussion,

16 there's an enormous amount of opportunity here.

17 It's good to have a panel like this, I think,

18 to flesh out a variety of views. I think it's a really

19 helpful panel, I'm learning. It's good for me to be

20 here, so I appreciate Andi and Katherine having me

21 here. But I just think there's a better story. It

22 isn't just a press down on consumers, but I'm still

23 just sitting here as a manager of businesses which have

24 relationships with consumers saying, you bet, I had

25 sales people who definitely went to the people who
72

1 bought the most first. Because that's how I got my

2 profit per square foot and that's how I continue to pay

3 salaries and monthly salaries that I had to meet.

4 That's all there is to it.

5 MR. SOLTANI: Just a quick comment. I'm

6 curious, who wants to be included in that higher priced

7 consumer category?

8 MR. PRATT: That's why you shop.

9 MR. SOLTANI: Right, but I'm --

10 MR. PRATT: I mean, that's why you shop.

11 That's why my wife is a much better shopper than me,

12 she always has been. She berates me all the time

13 because I would go to a big-box retailer and buy

14 everything at one store. My wife shops at four

15 different grocery stores because of the quality of the

16 product and the prices of the product. We just are two

17 different behaviors. Those are two different behaviors

18 in the marketplace.

19 MR. SOLTANI: So the shopping behavior here

20 would be, you know, building an architecture like I did

21 to shop for a stapler, right?

22 To find different prices, you have to delete

23 any of this information associated to you. And God

24 forbid you use an authenticated source like Google or

25 Facebook, that requires a login or is tied to your
73

1 credit card, to your purchase device. So you have to

2 then use anonymous cash and all this kind of stuff. I

3 think that's a little much for most consumers.

4 MR. PRATT: Hmm.

5 MS. THOMAS: So I want to talk a little bit

6 about Ashkan's presentation, which I think was actually

7 incredibly -- not actually, it was incredibly helpful,

8 in terms of understanding how the backend works, right?

9 So thank you for that.

10 I think, you know, I agree with Stuart that

11 loyalty is something that consumers respond extremely

12 strongly to. They want to be, you know, in a loyalty

13 program. They want to be given that additional offer

14 because they shop in one place and not another on a

15 regular basis, et cetera. And those programs have

16 grown because that's what consumers have demanded, that

17 if I'm going to be loyal to you as a brand, then you

18 darn sure make -- make darn sure that I get something

19 back for it.

20 Now again, we can talk about the outer limits

21 of that when it goes beyond marketing, but as far as

22 marketing is concerned, loyalty makes sure that you get

23 better offers over time. But also let's remember,

24 businesses want new customers as well. So a business

25 is as likely to give a discount to a loyal customer,
74

1 it's equally likely to give a discount to a first-time

2 customer, to make sure that that customer gets involved

3 and then will come back and become loyal in the future.

4 So this isn't a matter of one or the other.

5 And just to sort of go back to the bottom

6 line here, Ash made a really great point about the

7 marketing offer. You might get one offer or the other,

8 but then there is that firewall to whether or not you

9 are actually going to be eligible.

10 And I think a helpful way to think about that

11 is, you know, the Fair Credit Reporting Act, I'm not

12 going to get into it, I promise, but it uses the term

13 consumer-initiated, in terms of the kind of

14 transactions that are covered by FCRA. The consumer

15 initiates it and then it's not in the consumer's

16 control what happens next, in terms of the eligibility

17 decision.

18 In marketing, it's a marketer-initiated

19 transaction, but it's the consumer who is in control of

20 whether or not they respond to that marketing offer or

21 they go around that marketing offer and say, that's not

22 really what I want, I don't want 20 percent, I want 50

23 percent and I'm going to walk in the store and say so.

24 MS. ARMSTRONG: And Rachel, I think that's a

25 really great point, but I think Pam and Ed are -- I
75

1 would like to hear what they have to say, because I

2 think they are talking a little bit about that very big

3 fuzzy space in between the loyalty marketing and when

4 you get into eligibility. So I'd like to --

5 MR. MIERZWINSKI: Well, sure. And you know,

6 I think what Ashkan was talking about is not -- he's

7 not against loyalty cards. He probably might have

8 some, he might have a rewards credit card, I don't

9 know.

10 But what he's against is consumers being

11 selected based on secret profiles to be chosen to pay

12 more. Nobody wants to pay more. Everybody wants to

13 pay less. But nobody wants to be put in a box where

14 they pay more. And in a nontransparent system, where

15 thousands of bits of our lives are being collected

16 about us, shared and used to decide who will pay more,

17 my concern is not with big data per se, my concern is

18 with, can we use big data in a positive way to promote

19 financial opportunity.

20 I don't want banks to figure out which

21 consumer we can ding for more overdraft fees. I want

22 banks to figure out how to serve the under-banked,

23 using big data to save money and encourage the use of

24 the right accounts that make sense for that consumer to

25 be able to build up assets.
76

1 Again, I have loyalty cards, I have rewards

2 cards. I think that some seats on an airline, some

3 seats at the last minute, seats right down in front at

4 Yankee Stadium, are worth more than others. But I

5 think a stapler is a stapler.

6 MS. DIXON: So this conversation has gotten

7 to really the good part, really. So I'm just going to

8 give the grab here. So look, big data is an

9 opportunity for inclusion and it's an opportunity to

10 help people. I've seen this with my own eyes, in other

11 countries and in this country. It's incredibly

12 important that we acknowledge that. And that when we

13 target vulnerable populations or use sensitive factors

14 that these are used with great transparency, oversight,

15 and consumer control and are unfailingly beneficial to

16 consumers.

17 So there should be no secret scores and there

18 should be no secret factors. As a result, and to

19 facilitate that, there's something that we can do, I

20 think, that is a fairly -- it's a tweak. And I'm all

21 for tweaks because they're doable, right?

22 So creators of consumer scores, whether they

23 be static or femoral, enterprise, public, et cetera, if

24 they stated the purpose of the score, if they stated

25 the composition of the score and the intended uses of
77

1 the score, and allowed uses of the score, then I think

2 that that would go a very, very long way in

3 transparency. And I believe that would also pull that

4 under Section 5 of the FTC Act and then we would have

5 some oversight.

6 We all know that Section 5 is broad and it

7 would be very difficult to enforce a lot of this;

8 however, it would provide a tweak and a first step

9 toward bringing fairness while allowing benefits to

10 occur. We have to have both, we have to have both. No

11 secret scores and no secret factors.

12 MS. ARIAS: I think Rachel wants to respond,

13 but I do want to followup on something that you just

14 mentioned.

15 You said there should be transparency and

16 consumers should know about it, but how do you

17 communicate this to consumers? Given that there are so

18 many scores and there are so many factors that go into

19 these scores, and also that, you know, they don't even

20 know they exist, right? So at what point do you

21 communicate this information to consumers?

22 MS. DIXON: I think there's no perfect or

23 beautiful answer to that. I think we're all struggling

24 with that right now. How do you communicate in either

25 a short or long or midterm or holographic privacy
78

1 policy? Those things are, I think, in the midst of

2 being decided right now.

3 But in the interim, a privacy policy would be

4 a great place. This has its flaws, I am the very first

5 person to admit this, but we need to start somewhere.

6 And I'm all for starting with a tweak. Because

7 protecting vulnerable consumers is a necessity, not an

8 option. And as a result, let's start somewhere.

9 MS. THOMAS: So I just wanted to add, I

10 couldn't agree more with Pam, the importance of making

11 sure that when vulnerable populations are at stake or

12 at target or the topic of conversation, transparency

13 and making sure that they are not treated in a

14 discriminatory way is incredibly important.

15 So I think the good news is that we do have,

16 not just FCRA but many laws that make sure that that

17 doesn't happen. There's FCRA, of course, the fair

18 lending laws, of course, apply there. And I would argue

19 that the Federal Trade Commission Act, the FTC Act,

20 Section 5, if unfair or deceptive acts or practices --

21 if the marketing or the advertising is unfair to a

22 vulnerable group in some way, that that would already

23 be covered.

24 DMA also thinks this is incredibly important,

25 such that we regulate that in our ethical code as well.
79

1 We have one of the first articles, in the almost 60

2 articles of requirements for any marketer doing

3 anything, is disparagement. Make sure that any

4 marketing that you undertake is not disparaging to any

5 population and particularly vulnerable populations.

6 So this is something where, if there are

7 problems, they are that we are not enforcing the laws

8 that we have and that we should be in greater, you

9 know, put greater resources toward that to make sure

10 that the kinds of things that Pam is talking about are

11 not possible. Protections exist, we have to make sure

12 that they are acted upon.

13 MS. DIXON: I don't think most of these

14 scores actually are protected, I really don't. I think

15 they --

16 MS. THOMAS: Can you give an example of --

17 like, just for the sake of --

18 MS. DIXON: Aggregate. Aggregate credit

19 scores, I think, are an excellent example and modeled

20 credit scores. So they are used to provide offers of

21 credit, and even to set initial insurance rates, right?

22 But --

23 MS. THOMAS: So an offer of credit though

24 could --

25 MS. DIXON: I'm not talking --
80

1 MR. PRATT: They're not used to set rates,

2 they're not used to set rates.

3 MS. DIXON: They're not used to set rates 

4 for --

5 MS. THOMAS: -- would be covered by FCRA.

6 MR. PRATT: I just want to -- that's the

7 bright line, I think, you're dealing with, right?

8 MS. DIXON: That's correct, that's exactly

9 right.

10 MR. PRATT: Okay.

11 MS. THOMAS: Which is such an important

12 bright line.

13 MR. PRATT: Correct.

14 MS. DIXON: We're at the razor edge of where

15 the Fair Credit Reporting Act ends and something else

16 begins. And I'm talking about the millimeter to the

17 right where the something else begins. So I'm not

18 talking about a firm offer of credit, I'm talking about

19 the offers that really color and make a person's life

20 different.

21 And we haven't talked a lot about health

22 data. In the report we're coming out with next week,

23 we talk a lot about how health data is being used in

24 scoring. And this is health data that has been

25 acquired outside of HIPAA.
81

1 And when health data is put into scoring

2 factors, it is extremely prejudicial. And no one has

3 -- there's not a law that covers this. Well, a little

4 bit FCRA, but it's not applicable. So that’s the piece that

5 we’re concerned about.

6 MS. THOMAS: I hear what you're saying, but I

7 think, you know, I think the FTC has taken action in

8 these areas, and very important actions, to make sure

9 that FCRA is covering exactly where it needs to and

10 that there is no gray area.

11 I think the Spokeo case was a really good

12 example of that. You know, FCRA has worked for 40

13 years and I think that case showed that it continues to

14 work. If third-party data is used for a permissible

15 purpose under FCRA, then that data is a consumer report

16 and the agency or the organization is a consumer

17 reporting agency and FCRA covers it and the problem is

18 solved. 

19 So let's make sure that that -- you know,

20 thank you. And let's make sure that those kinds of cases

21 continue to happen where, you know, FCRA-covered data is

22 being used in ways that are not in line with the law.

23 MS. ARIAS: I think Ashkan wants to jump in.

24 MR. SOLTANI: Yeah. So I'm --

25 MS. ARIAS: And Claudia.
82

1 MS. ARMSTRONG: And Joe.

2 MR. SOLTANI: So the backend verification

3 happens through the FCRA-approved process, but if I'm

4 never presented with the opportunity to apply for the

5 lower interest rate card, kind of where in the -- how

6 does that --

7 MS. THOMAS: There is no reason that you

8 should ever have to take a marketer or anybody else up

9 on the offer that you're given. That is completely

10 separate from --

11 MR. SOLTANI: But if I don't see --

12 MS. THOMAS: -- your eligibility to get the

13 product. So if you don't like the offer you have, that

14 isn't the only offer that is available to you.

15 MR. SOLTANI: Actually, so --

16 MS. THOMAS: That's the law.

17 MR. SOLTANI: When I tested the kind of

18 credit card sites, Chase for example, or CapitalOne,

19 when you would repeatedly visit the site, you would

20 never get, you know, one out of maybe 100 times, or

21 statistically a low probability of times, you would get

22 the other offers.

23 So as a consumer, if I reload the page, I'm

24 not given the choice to say give me the zero interest

25 card.
83

1 MS. THOMAS: Did you call the credit card

2 company to ask what offers --

3 MR. SOLTANI: Do most -- I mean, so this

4 rings to me a little bit --

5 MS. THOMAS: If I don't like the offer I'm

6 getting, I go somewhere else to try to find an offer

7 that I actually like.

8 MR. SOLTANI: Right, right. So let me kind

9 of --

10 MS. THOMAS: That's what I'm getting at.

11 MR. SOLTANI: Let me give an analogy that

12 might work. So everyone can vote, right? And you get

13 verified at your polling place.

14 MS. THOMAS: Mm-hmm.

15 MR. SOLTANI: But you can make it very

16 difficult for people to vote. I mean, we've

17 historically seen people putting voting booths in

18 places that are difficult for consumers to go to -- for

19 voters to go to, it reduces the rate at which certain

20 populations will vote, right?

21 And I feel like this kind of butts up against

22 that. When you make it slightly difficult for people

23 to do a particular outcome -- and sure, absolutely, you

24 can call, you can scour the internet, you can research

25 and find other offers, but we know most people don't
84

1 call. Most people take the offers they're given and

2 put some amount of credit in -- they put some amount of

3 effort in their busy lives to do a set of activities,

4 one of which is try to research a credit card and take

5 the best deal that their search engine gives them.

6 MS. ARIAS: Why don't we have -- I think Joe

7 hasn't spoken for a bit. Why don't we have Joe maybe

8 jump in and then we can have Claudia.

9 MR. TUROW: I wanted to kind of pick up to

10 where Ashkan was coming from. Putting it into broader

11 historical perspective, I think we're really at a very

12 different point in terms of how we understand pricing

13 and what it all means.

14 I think if you look historically at the U.S.

15 retailing situation, the 19th and 20th century was

16 about the progressive, relatively speaking,

17 democratization of prices. You could walk into a store

18 and pretty well you would see the prices. Of course,

19 some people got different prices, some people went into

20 a back room. But generally speaking, prices became,

21 for lots of interesting reasons, democratized.

22 We are moving away from that ideal in very

23 interesting ways. And not just in the online or mobile

24 space, but brick-and-mortars now change the prices by

25 the hour and change the prices by the person. So
85

1 literally, you could be walking through a store and the

2 prices would be different for you, particularly with

3 the new, you know, Apple and Bluetooth situation, where

4 you walk through the store and it actually knows who

5 you are. That's a very different way of thinking about

6 the world.

7 And I think -- so there are some really,

8 really important issues like financial and health, but

9 in the broader sense of how we are going to see one

10 another and understand ourselves. If we are walking

11 through a world where we are consciously aware of, for

12 reasons we have no idea, we are getting different

13 offers, different deals, different understandings of us

14 based upon calculations of our lifetime value, which is

15 five years, by a retailer, that we have no idea why it

16 came? That's a different mindset of how we understand

17 the world.

18 And people are going to have to catchup with

19 that. I think most of us are still in the 20th century

20 and there may be good reasons why we're encouraged to

21 be in the 20th century thinking about this. But the

22 world is changing so drastically, it really creates

23 incredible tensions.

24 MS. PERLICH: I actually really, really like

25 that, because that's the direction I was thinking about
86

1 as well.

2 I think there are two questions. One, are

3 you comfortable with the brave new world of this kind

4 of differential pricing? The question is, what role

5 does alternative scoring play? Honestly, what Ashkan

6 presented, it's not about alternative scores, it's

7 about the big picture of using data. There isn't even

8 a score. I mean, it's a flag on my user-agent. If you

9 really want to make every single data point a score,

10 then that would be infinite, you can't govern that.

11 He was really talking about the use of single

12 data points. That's very different from the question

13 of well-defined scores that are aggregates or even what

14 I presented. My models would charge rich people more

15 because they are more likely to buy at higher prices

16 than poor people. So his findings are a human

17 decision. There was an expert that thought, those guys

18 must be rich, let's charge them more. There wasn't a

19 model involved. It's the singular use of data points.

20 MR. SOLTANI: That's actually not true. So

21 in the user-agents, sure, but in the credit card

22 offers, remember X+1 was -- it determined, by some

23 scoring, whether you -- your credit profile, for

24 example. It would determine -- or the location, right?

25 So the location would determine, based on some other
87

1 factors, a score which is your distance to a

2 competitor.

3 MS. PERLICH: I'm just saying it's a range.

4 MR. SOLTANI: For sure.

5 MS. PERLICH: It goes from a single data

6 point all the way to these aggregate scores that --

7 MR. SOLTANI: Absolutely. Just clarifying.

8 MS. PERLICH: Yeah.

9 MS. ARMSTRONG: Joe, we talked earlier in

10 some of our conversations -- oh, I'm sorry, Ed.

11 MR. MIERZWINSKI: Yeah. I just wanted to say

12 that the issue -- I do care a great deal about the Fair

13 Credit Reporting Act, but when we don't cross that line

14 of determining eligibility or whatever, I care about

15 the scores on the far side of the line.

16 I agree the FTC has done a good job with

17 companies that have crossed the line, by selling

18 information about your friends on Facebook or other

19 social network sites that bears on your reputation and

20 makes you a credit reporting agency. Good stuff.

21 But on the other side of the line, a lot of

22 these credit card sites that you go to to find the best

23 deal are actually what are called -- and I can't

24 believe -- I don't think this term has been used yet

25 today, lead-generation sites. And lead-generators
88

1 auction you off, in realtime, not to the lowest or the

2 best bidder for you, but to the highest bidder. And

3 you have no idea, it's a completely nontransparent

4 process.

5 Now sometimes they may send you to CapOne,

6 in the credit card case, sometimes they may send you to

7 a third-tier credit card company, but the lead-gen

8 sites are also primarily used by the bad guys on the

9 internet. The online payday lenders, the for-profit

10 schools, and others.

11 And again, they are paying, based on all the

12 information they collect about you, for a score that

13 makes you someone that they can take advantage of. And

14 that needs to be looked at. And fortunately, the

15 states of New York and Illinois and other states, plus

16 the CFPB and the FTC are looking into this. But it's a

17 very important area, scores that are outside of the

18 FCRA. When we get to a world where the FCRA is small

19 and these other scores are big, that's a bad -- that's

20 a bad world.

21 MS. DIXON: I think we're already at that

22 world. And I want to go back to the point I made about

23 the Klout score.

24 So in the report that we have forthcoming, we

25 describe a gentleman who was denied -- actually, he
89

1 lost a job offer and was told he was denied because his

2 Klout score was deemed too low.

3 So it caused us to do a thought experiment,

4 is Klout a CRA? And we had to come to the answer that

5 it is not. Otherwise, every single entity on the

6 planet would be a CRA. So we have to also apply First

7 Amendment issues to this. There's a huge tension. We

8 can't make everyone who uses a piece of data for

9 eligibility reasons a CRA.

10 So given that, what do we do? And that's

11 what I'm saying. And I think we really need to have

12 transparency as an important first step. No secret

13 factors, no secret scores. Tell people what's

14 happening and then let's start figuring things out.

15 MS. ARMSTRONG: Thank you, Pam. Joe, I

16 wanted you to take a little bit more time to speak

17 about the future. I mean, how do you see some of these

18 scores being used in the social, in the mobile, and

19 other contexts?

20 MR. TUROW: Well, if we think about scores

21 broadly, meaning indexes of how people act and

22 predictive analytics, in terms of where they will go,

23 my sense is that it simply has to become more and more

24 pervasive. And the reason I say that is because of the

25 hyper-competition that exists.
90

1 We're in a world today where the meeting

2 between brick-and-mortar and the online world is so

3 competitive. When you're competing with Amazon, that

4 doesn't worry about margins, for example, seemingly,

5 that raises lots of interesting questions.

6 And essentially I see it as mobile,

7 wearables, and even cars. We're going to be in a

8 situation where our car will be part of our

9 decision-making and the decision-making about us.

10 We'll be in situations where -- it is possible, if you

11 give your permission, for cameras to look at your face.

12 There is a company now, Emotion, that can actually look

13 at facial features and decide certain aspects of what

14 you think, your emotions, when you're purchasing

15 something. There's a Russian company, and actually I

16 think HP has a similar thing, which can look at you at

17 check out and then try to connect you as you're moving

18 around stores and elsewhere.

19 MS. ARIAS: Joe, can you lean into the

20 microphone? We're having --

21 MR. TUROW: Oh, I'm sorry.

22 MS. ARIAS: -- trouble hearing you a little

23 bit.

24 MR. TUROW: So the idea here is that, more

25 and more, it is a question of not so much the
91

1 technology, but what we want to put up with. The

2 competition in the world is going to be that predictive

3 analytics is the future. I think the 21st century is

4 about data and data is incredibly important. Companies

5 have to do this and they will do it as much as they

6 can, with the idea of getting people to see that it's

7 relevant to them.

8 And thereby -- and I won't go on and on,

9 because it's a fascinating topic, but the issue of

10 relevance is at the core. And it's terribly important,

11 people want to get relevant ads. I think they want to

12 get relevant offers and relevant deals. The tension

13 that exists has to do with how do I do that, while not

14 having my data being used for things I don't even know

15 about and may not agree with. What is the seepage of

16 those data elsewhere, okay? How am I being scored that

17 may affect some other parts of my life? If I get a

18 discount that's relevant, but I give in some data,

19 well, that's terrific. I'll give you the stuff, but

20 then it gets used in ways I don't want.

21 All of these things are part of the tension

22 that we have to deal with in today's world. Everybody

23 wants relevance, the question is do they really know

24 the price of that relevance?

25 MS. ARIAS: And so I think that's an
92

1 excellent point, and I know we're running short on

2 time, but we really want to talk about some of the

3 solutions that folks think need to be implemented, to

4 the extent we need any.

5 So I'm going to open it up to the panel, to

6 the extent anybody has any thoughts on this. Ashkan?

7 MR. SOLTANI: So one idea might be, and this

8 is, again, pie-in-the-sky and probably a few years out,

9 if we can define the contours of the uses that we care

10 about that are kind off limits. And we're talking

11 about data and algorithms here.

12 You know, kind of like what I've put

13 together, I wonder if there's ways to do basically unit

14 tests? Test cases where you basically feed in data

15 into an algorithm, feed in populations, feed in either

16 fake users or real users or require auditing or

17 reporting of the output, of the classification, such

18 that you can audit the results and say, look, whatever

19 the data input that you have, whatever those sources

20 are, you're clustering these users based on race. Or

21 you're providing offers based on sensitive categories

22 of information that we don't want. And as a result,

23 this algorithm seems to be discriminatory.

24 It's kind of pie-in-the-sky black-boxing, but

25 it's essentially, I think -- because there are so many
93

1 sources of data, because there are so many different

2 algorithms, because often times the creators don't

3 explicitly -- so the examples I showed were explicit.

4 Often times, as Claudia described, it's clustering. It

5 just so happens that you cluster and you're clustering

6 based on distance from a competitor, but you also

7 happen to cluster based on race or ethnic type, just

8 because that also correlates to zip code.

9 And so if you start seeing that type of

10 behavior in the output of an algorithm, then you can

11 start saying, well, either the algorithm or the data

12 sources are problematic. Sorry, that was a little --

13 MS. PERLICH: So this isn't necessarily an

14 answer. I feel strongly that it really comes

15 ultimately down to what you do. I think we have to

16 focus on decisions that we are comfortable with making.

17 I consider that there is the pie-in-the-sky, getting

18 there is a long way from where we currently are.

19 The challenge of predictive modeling is it's

20 a reflection of the current biases of human nature.

21 It's a reflection of the fact that certain demographics

22 are in worse economic state and the model will pick up

23 on this and reflect that directly.

24 And I have a hard time making the model not

25 do that. It's very hard for me to insert my morals
94

1 into what just the state of the world is. My models

2 come to the point where I am making decisions based on

3 what that model tells me. I think that's, for me,

4 where I can have a judgment easier, than trying to sift

5 through the incoming and what it may or may not mean.

6 MS. ARMSTRONG: Everyone is going to have an

7 opportunity, before we end, to make a final comment,

8 but with what both Claudia and Ashkan have said, I

9 wanted to raise an issue that the recent NCLC report

10 described.

11 And it was a scenario where American Express

12 lowered the credit limits because of other customers

13 who shopped at places that consumers shopped and they

14 had a poor repayment history. And this is also

15 consistent with some of the other issues that that

16 report raised about the discriminatory impact and I'd

17 like to get some comments on that.

18 MS. DIXON: I'm concerned about cohort

19 scoring, so who your friends are kind of tells other

20 companies or entities or health care institutions who

21 you are. It's a predictive modeling validation tool.

22 However again, I come back to, we can't have secret

23 factors and secret scores. Transparency is going to do

24 a lot to ease some of this and I think we're going to

25 have to find a way to meet in the middle.
95

1 You have to validate the score, right? But

2 if you're validating a score in a discriminatory

3 fashion, that's a huge problem. But we're never going

4 to know about it unless someone has done some, you

5 know, some very surgical-strike research.

6 MR. PRATT: So any transaction that AmEx

7 would make relative to a portfolio of credit cards, we

8 all know it, regulated by those same laws, that

9 alphabet soup of laws we talked about before.

10 So there may be other instances that are

11 outside of the credit portfolio context where I suppose

12 you might have some conversation about, you know, the

13 effects. But in the context of a portfolio, if you're

14 going to change the contract, something about those

15 terms, you have to control for all of the current law

16 factors that are out there, ECOA, disparate impact,

17 Fair Lending, anything else that would apply to that

18 portfolio. So there's this penumbra of protection, if

19 you will, around that kind of portfolio decision.

20 We don't know whether that was a good

21 decision or not, we don't know if that was an effective

22 decision or not, we don't know if that was an

23 experimental idea in the midst of the recession, as

24 every card issuer was trying to figure out what to do

25 to measure risk, as card portfolios and, of course, all
96

1 financial services portfolios began to crater just a

2 little bit, you know, in terms of size of risk

3 population, which was much, much larger than had been

4 the case historically.

5 So I mean, there's a lot to think about.

6 We're getting just this little tiny anecdotal moment

7 here, but not much else really.

8 MS. ARMSTRONG: That's a good -- that's an

9 excellent point.

10 MR. MIERZWINSKI: Well, I would just say

11 briefly, that if companies do this and they do it on a

12 non-protected class discriminatory manner, they may not

13 implicate any of this alphabet soup of laws. Companies

14 like LendUp and Move-In, if that's how you say that

15 word, I've never used it before, are making decisions

16 about who to make credit offers to based on their

17 social networking status. But they're not credit

18 reporting agencies, they're simply making decisions

19 about their own customers, or potential customers, and

20 they may not be regulated.

21 So we really need to look at regulating the

22 system of scoring that isn't regulated today.

23 MS. ARIAS: And to follow-up on that, what do

24 you envision would be the way to regulate? If, in

25 fact, they are not regulated under the FCRA or any of
97

1 the other alphabet soup laws?

2 MR. MIERZWINSKI: Well, I think the first

3 step is really transparency of the online scoring

4 system, the graphic that Ashkan put up of all of these

5 hundreds of companies. They are all

6 business-to-business companies, nobody knows who they

7 are.

8 In the past, you knew that you had a

9 relationship with your creditor. And although you

10 didn't choose your credit bureau, you knew that you had

11 the three credit bureaus and your creditors, who you

12 could choose. But you didn't know about all of these

13 other companies out there that were providing services.

14 And that needs to be more transparent and consumers

15 need to have rights when their information is used.

16 The right to look at their profile of a data broker,

17 the right to change the profile of a data broker, and

18 the right to block the use of their information for

19 other purposes.

20 And there should be some disclosure, just

21 like there is when you are denied credit on the basis

22 of a credit report, that you're denied credit on the

23 basis of some other kind of report.

24 MS. ARMSTRONG: And with respect to

25 transparency disclosure, I'm wondering about choice for
98

1 consumers and whether you have an opinion about where

2 in the ecosystem the consumer choice should occur.

3 MR. MIERZWINSKI: Well, we don't have a lot

4 of time. If you're asking me, I would just say that I

5 would refer people to look at the final Commission

6 report on privacy. I think it's an excellent

7 background around all of these questions.

8 MS. ARMSTRONG: Okay. I think it's time that

9 we're going to run down this row and let everybody make

10 a final comment before we conclude. So Rachel, you're

11 first.

12 MS. THOMAS: Can we start at the other end

13 this time?

14 MS. ARMSTRONG: Or would you like to be last?

15 Okay, we can do that. Ashkan?

16 MR. SOLTANI: So one thing that I think

17 that -- sorry. One idea that we might have -- I think

18 Claudia is absolutely right that this is a difficult

19 thing to get at and to underscore. One thing that we

20 might want to do is to explore, like the FTC authority,

21 for example, when a company says, you're getting the

22 lowest price, right? And you're actually not, you're

23 getting a different price based on ratings. Or Joe is

24 getting a lower price than I am, should a company be

25 required to say, you're getting the lowest price for
99

1 you, based on this information? Or you're getting the

2 best credit card deal for you, based on this

3 information? Or can they just outright say, you're

4 getting the lowest price, because that seems like an

5 absolute statement.

6 And I wonder, under Section 5, if there is

7 authority to at least nudge or poke at that.

8 MS. ARMSTRONG: Or you know, we can do this

9 in any order, just as long as everybody gets to say

10 something.

11 MS. PERLICH: Well, I'm just going to skip my

12 right to have a final word here and leave it to the

13 rest of the panel.

14 MS. ARMSTRONG: Thank you.

15 MR. TUROW: I'm just going to say something

16 that may be quixotic, but we have found in survey after

17 survey, I think five times, that people think that when

18 a site has the word "privacy policy" on it, they think

19 that it means the site doesn't sell or trade

20 information without their permission. This is 56 to 62

21 percent of Americans. 75 percent don't know that, in

22 fact, it's true that they give up -- that they don't do

23 that, necessarily.

24 So the only thing I would suggest here, and

25 it's only a beginning, but it would cause an
100

1 interesting amount of chaos in the industry, which is

2 if a site uses your information without explicitly

3 getting your permission, it's deceptive by the FTC and

4 it shouldn't be called a privacy policy. It should be

5 called using your information. That would be an index

6 to people to be careful. It would, right off the bat,

7 say be careful.

8 If it says privacy policy, you know that

9 you're safe. If it says something else, you know to be

10 careful. And I think that would be the beginning of an

11 interesting conversation with the industry.

12 MS. DIXON: So just to roll down the line,

13 why not? So there are many new scores, many. Many, if

14 not most of these, are outside of the current

15 regulatory structure. Some of these scores are much

16 more important than others. Some are not important,

17 some are very important and the deciding factor is the

18 impact. And we are focused really on eligibility uses,

19 especially those that are happening outside of

20 regulation, and they are.

21 We'd really like a solution, a first step of

22 real transparency that's meaningful, no secret scores,

23 no secret factors. Discussions with industries, so we

24 know what's happening, and a real meaningful commitment

25 from industry to have transparency about this.
101

1 We'd really like for there to be oversight

2 and disclosure to the consumer about purposes,

3 composition, and uses of the important scores.

4 Probably not all of them, there's too many, but the

5 really important ones that matter.

6 MR. MIERZWINSKI: Well, although I mentioned

7 that I got a fishing catalog by mistake, I don't care

8 about that kind of marketing, if I didn't make that

9 clear. But I do think that financial information,

10 healthcare information, and information about children

11 needs to be looked at, in events like this, in greater

12 detail. I'm encouraged by events like this being held.

13 I'm encouraged that recently, it was in the press, that

14 a number of civil rights groups have developed a

15 platform on the use of big data for financial

16 opportunity.

17 And we're starting to look at this, and we

18 don't have the answers yet today, but it's encouraging

19 that were starting to create a framework of answers.

20 MR. PRATT: So for me, as the CDIA, I'm just

21 going to echo what I said at the very beginning, risk

22 management matters. It's critical, it keeps us safe,

23 and it ensures the transaction is me when it is me and

24 it ensures the transaction stops when it's not me. So

25 bank safety and soundness matters. It's important that
102

1 we have ways to measure risk and to rank order

2 consumers in terms of risk.

3 And just a couple of quick thoughts about the

4 glossary, and I know this is kind of nerdy stuff here,

5 but I hate the data broker term. It's a really sloppy

6 term, it wraps around all different kinds of U.S.

7 business models and it conflates issues in the public

8 policy world as well. And we've seen this, actually,

9 in legislation that's been introduced on the Hill. So

10 it's just a crummy term. And I suppose if I could get

11 out my big eraser, I'd erase the term data broker and

12 we would try to parse through the issues in a little

13 more refined way.

14 Get rid of the term score. The only reason I

15 advocate for that is because it's too often conflated

16 with credit score, too often conflated with what

17 consumers now think of as a credit score. So again,

18 this is just sort of marketing stuff. If we're

19 communicating with consumers through venues like this,

20 we ought to pick terms that make sense to consumers.

21 I think, Joe, your term about using my data

22 versus privacy is the kind of example that pivots off

23 of that same idea. Using my data is different than,

24 you know, privacy, which may be a more amorphous term

25 that we don't understand or connect with as much.
103

1 By the way, the problem with data brokers is

2 it really doesn't -- you know, it kind of lumps

3 together first party third party. We don't think

4 those issues matter very much. Our members are

5 third-party databases, dataflows have to come to

6 third-party databases. It's how the American economy

7 operates. Dataflows are necessary, I agree with what

8 Joe said. Dataflows are going to occur, we're going to

9 be in a highly competitive environment. I do think

10 that what Joe said is right, consumers -- by the way,

11 I'm going to add to what Joe said. I think consumers

12 will catch up a bit, will begin to become smarter

13 consumers, even in an environment where competition has

14 ebbed, you know, changed a little bit over time.

15 So we should be encouraged. These dataflows

16 can ultimately open doors for us that weren't open

17 before, they can include us when we were excluded

18 before, they can give us better offers that save us

19 money. It's still up to us, as consumers, to do some

20 shopping in the context of all that.

21 MS. THOMAS: So we've touched on some

22 examples today, some of which, you know, at the end of

23 the day, it sounds as though they are covered by FCRA,

24 some maybe there are still questions about whether

25 they're covered or they aren't.
104

1 But I think it's important to recognize that

2 if we're focusing today on some difficult areas, there

3 are a lot of areas that we are not concerned about,

4 right? The dataflows are happening. Data is data is

5 data. What we need to be concerned about is particular

6 uses. Thank you for focusing us in that way today.

7 But I think it's also really important to

8 recognize that the reason there are so few things that

9 we have reason to be concerned about is because

10 companies are making sure -- companies, nonprofits, all

11 of the above, have strong incentives to self-regulate

12 and not do bad things to their customers on a daily

13 basis. And that there is DMAs, as well as many other

14 self-regulatory environments, that are making sure

15 that, if there are areas where the law doesn't cover,

16 and there are so many laws that we discussed today that

17 do, but in those areas, businesses are doing the right

18 thing because they are being held to standards by

19 organizations like DMA. And if they aren't meeting

20 those standards, we know where to find the FTC and we

21 do refer them over, as needed, to the FTC or other law

22 enforcement.

23 At the end of the day, we need to enforce

24 what we have, whether it's through self-regulation of

25 the laws that we've got, and continue to make sure that
105

1 data is used responsibly in both of those areas.

2 MS. ARMSTRONG: Thank you very much. And

3 we'll continue down the line for our last few comments

4 and then call it a morning.

5 First of all, I appreciate everybody coming.

6 I wish we had more time, this was fascinating. We are

7 accepting public comments about today's -- on these

8 issues until April 19th. And thank you very much for

9 everyone who sent a question card. I know we didn't

10 get to everything, but we are thinking about these

11 things.

12 And finally, I wanted to remind everybody

13 about the next installment of our Spring Privacy Series

14 on May 7th, which will focus more on health-type

15 products.

16 MS. ARIAS: And just to finish everyone off,

17 I really want us to pause for a second and thank this

18 great, great panel.

19 MS. ARMSTRONG: Fabulous.

20 MS. ARIAS: Fabulous, that's right.

21 MS. ARMSTRONG: Fabulous.

22 MS. ARIAS: If we can applaud them, they did

23 what we thought was unthinkable today, which is cover a

24 really, really broad subject, cover lots and lots of

25 issues, and really touch and started the conversation,
106

1 so I really want to thank them.

2 But I want to thank you, our audience, and

3 everyone out in the internet, watching us on the

4 webcast, you've really been a fabulous audience and we

5 really look forward to your comments and suggestions on

6 this topic.

7 Thank you everyone.

8 (Whereupon, the proceedings

9 concluded at 12:05 p.m.)

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25
107

1 C E R T I F I C A T I O N O F R E P O R T E R

2

3 MATTER NUMBER: P145401 

4 CASE TITLE: SPRING PRIVACY SERICS 

5 DATE: MARCH 19, 2014 

6

7 I HEREBY CERTIFY that the transcript contained herein

8 is a full and accurate transcript of the notes taken by me

9 at the hearing on the above cause before the FEDERAL TRADE

10 COMMISSION to the best of my knowledge and belief.

11

12 DATED: MARCH 27, 2014

13

14 

15 STEPHANIE GILLEY

16

17 C E R T I F I C A T I O N O F P R O O F R E A D E R

18

19 I HEREBY CERTIFY that I proofread the transcript for

20 accuracy in spelling, hyphenation, punctuation and format.

21

22

23 

24 SARA J. VANCE

25