Fast.ai Practical Data Ethics lesson 4 notes-Privacy and surveillance
By: Risto Hinno
Tags: Civil Rights, Fastai, Privacy, Surveillance, Tech

Lesson 4 materials are here. These are my notes and thus not complete and may contain mistakes.

Intro

Privacy and surveillance have become major concerns. We share our data in different platforms, but we don’t have a control what happens after that. Data could be sold to other companies and we might have no idea about it. One example could be found here. There are intermediate companies who collect data and sell it to other companies who need it. So you have no idea where your data might travel. At first glance this data might look like anonymized, but could be deanonymized and deaggregated. Some company who you haven’t give your data might get and use it for something you have not given consent (and know lot about you).

Another issue is how tools are used. There are examples where police departments have been using sketches of probable suspects in facial recognition systems. These systems are not designed to be accurate with this kind of input. Another example is where original images are too blurry and don’t return a match. There have been cases where police has been using suspect’s celebrity doppelgänger instead.

“Photo too blurry? No problem — run a search for the suspect’s celebrity doppelgänger instead.” Source

Once data is collected you don’t know how it might be used (against you). There is a real life example were Hong Kong’s protestors were afraid to use their metro cards because data might be used to repress protestors. Data could be used for surveillance, to get information about you and your life.

Surveillance through history

Surveillance has a long history. Following are just few points from here:

Queen Elizabeth I built network of spies and informants to root out perceived threats, targeting Catholics and Puritans.

1880s: 2300 Mormons were prosecuted for polygamy. Anti-Mormons claimed Mormons were not white and were actually Muslims.

FBI amassed 1500 page file on César Chávez (first U.S.-born Hispanic senator).

FBI tracked Martin Luther King, Jr, compiled recording of his affairs, and threatened blackmail.

We can see a pattern: we watch those who are “less than”. Will you spy on your superior? Or will you spy on the poor man, the person of color, the immigrant, the heretic? We watch those who are “other”.

When those “others” organize, mobilize, that watching is redoubled. Surveillance becomes a tool to stop marginalized people from achieving power.

Those “others” are crucial for social innovation not to settle with current situation: “Senator Chávez made clear that dissent was our strength, not a weakness. “I contend that we are a nation of dissenters,” he said. “Your ancestors and my ancestors . . . left Europe because they were dissenters. Our Nation was created by dissenters. . . . Our ancestors were not satisfied. The Declaration of Independence and the Constitution of the United States are based upon the fact that our ancestors were not satisfied with the things as they were.” And so, Chávez explained, if you curb dissent, if you stifle difference, you don’t just threaten liberty, you threaten “our entire democratic way of life.” He went on: “[T]he United States cannot gain strength from a population of ‘safe’ people . . . people who have never thought or dreamed or spoken anything that is not prescribed and dictated by higher authority,” he said. “If we accept only ‘safe’ people the cement falls apart and we become a slave people.”” (source)

César Estrada Chávez. Source

One example from current times, were Baltimore Police Department used facial recognition software to find out and remove protestors from the crowd who had outstanding warrants. “Every person we got off the street before they hurt someone was a win”. Model was not really predicting if anybody was going to get hurt, We really don’t know if this model increased safety. But it is very easy to see how this kind of solution might be used to reduce the power of protestors.

Government and Technology

Some problems related to government using technology:

Opaque.

Restrictive NDAs. Companies don’t want that information is shared with the public and have created very restrictive NDAs making things even more secretive.

Typically not evidence-based.

Dysfunctional connection — which solution gets selected might be difficult to understand, asymmetry between what knowledge about technology is in public sector, what knowledge companies have etc.

Opaque

One example where even New Orleans city council members didn’t know that city is using Palantir’s predictive policing tool. It wasn’t purchased using public procurement which makes it difficult to have a reasonable control over it. Not to mention that model itself was flawed: “Trying to predict who is going to do what based on last year’s data is just horseshit,”.

NYC’s algorithm task force with the goal of examining how hidden algorithms oversee life in the city was a waste of time. At least some of the members felt so. One of the problems was that city’s commitment to transparency was in question, crucial information was not shared with the members of the task force.

Practice of parallel construction, “… is when law enforcement originally obtains evidence through a secret surveillance program, then tries to seek it out again, via normal procedure” (source). It is difficult to uncover it because authorities are keen on keeping it secret and not sharing any information about it. Information is kept secret even from the court (source).

Amazon Ring Doorbell case:

Partnerships with over 800 police departments.

In many cities subsidized with taxpayer money (source).

Amazon can stream 911 calls in real time (source).

Amazon requires police departments to read re-approved scripts (source).

There isn’t evidence that Ring reduces crime or significantly helps police department (source).

Amazon Ring. Source

Not evidence-based

Some schools in US are using face recognition software to detect possible shooters at school. These solutions have little impact on real life school shootings. Most of the shootings have been carried out by students so which means detecting them using face is impossible (source).

As with many facial recognition software error rates for people in color are higher. In one study they found that the “… error rate ranged from 10 times to 100 times more often for people of color.”(source) But what facial recognition software provides is a feeling that we have done something (although it is useless).

San Diego’s long experiment with facial recognition technology came to an end without evidence how many arrests it helped to make and did it really improved safety (source). Example how accurate facial recognition software might be: “American Civil Liberties Union found that when it ran photos of California state legislators through Amazon’s Rekognition software, the program mistakenly matched 20% of them (more than half of whom were lawmakers of color) to mugshots in a massive state law enforcement database.” (source)

Surveillance

Arguments for Surveillance

Rachel introduced most common arguments which are used for surveillance:

Public Safety.

Catch criminals.

Convenience/efficiency (board a plane faster). We could call it a luxury surveillance, we pay for it and it might give us some benefits, like sports trackers (source).

Why worry about surveillance?

Previous list makes surveillance look like pretty decent thing to do. But unfortunately this is only fraction of the story. Some of the surveillance negative aspects are:

Harms the most vulnerable.

Data collected for one reason will be used for other purposes.

Errors in data.

Internal threats are often the biggest threats (Zeynep Tufekci).

Stops dissents, and dissent is crucial for social progress.

Harms the most vulnerable

Surveillance disproportionately harms harms marginalized groups. For example Chinese government is using surveillance to control and re-educate Uyghurs (a minority living in China) (source).

Most vulnerable groups don’t have power to change politics and are easy targets as they are the “others”. Surveillance helps to take away their rights.

Surveillance harms minorities is visible from demographics of facial recognition bans:

First table: cities that have banned facial recognition. Second table: no ban city. Source, which has taken it from here

Cities that have banned facial recognition software have more white people than black people- Detroit which uses this software has a lot more black people than white people.

Data collected for one reason will be used for other purposes

Unfortunately there is a long history how data that was collected for one purpose (which might be good) has been used for other purposes to destroy or harass a group of people (source). This includes:

Jews, Romas in Netherlands during Nazi occupation.

Rohingya asylum seekers in Bangladesh.

Tutsis in Rwanda.

The U.S. Census Bureau Gave Up Names of Japanese-Americans in WW II (source).

ID card with the designation “Tutsi”. Source

Examples of data collection that might be/is used for other reasons:

Example from India where biometric ID is forcing HIV-patients to forgo HIV treatment in a fear of their personal information might leak (source). They don’t want to reveal that they have HIV and that they are sex-workers. Cancelling treatment threatens their lives.

is forcing HIV-patients to forgo HIV treatment in a fear of their personal information might leak (source). They don’t want to reveal that they have HIV and that they are sex-workers. Cancelling treatment threatens their lives. Police illegally giving personal location data from license-plate scanning to ICE (Immigration and Customs Enforcement) (source).

to ICE (Immigration and Customs Enforcement) (source). DHS (Department of Homeland Security) considered how to punish states that deny access to driver records (source). For example Utah, Vermont and Washington (along >10 other states) allow undocumented immigrants to obtain driver’s license. This information might be used to track down those people although this was not initial purpose.

(source). For example Utah, Vermont and Washington (along >10 other states) allow undocumented immigrants to obtain driver’s license. This information might be used to track down those people although this was not initial purpose. US prisons are using voices of incarcerated to build a voice prints database (source).

to build a voice prints database (source). Low-income people face more intrusive questioning, verification requirements, home visits because they use government benefits (source). This data is shared across government and with companies which puts them into worse position in privacy compared to high-income groups.

face more intrusive questioning, verification requirements, home visits because they use government benefits (source). This data is shared across government and with companies which puts them into worse position in privacy compared to high-income groups. “Jordan’s camps, refugees are registered by iris scan and identified. They can not refuse.” (source) Although this might have some benefits (easy to get food etc.) we don’t know who else might have access to the data and what they might do with it.

and identified. They can not refuse.” (source) Although this might have some benefits (easy to get food etc.) we don’t know who else might have access to the data and what they might do with it. Example of India’s fintech app which uses social media data to predict credit rating, reduces the scores of people who are engaged in political activity (source).

You don’t know once the data has been collected how the laws are going to change in the future.

Also surveillance systems that are created for one purpose might be used for other purpose in future. Any data that Facebook or Android have is data that governments around the world would like to get or thieves would like to steal (source).

Data contains errors

Example how what kind of errors data might contain (source):

42 babies were 1< year old at the time they were entered into the database.

28 of those were marked as “admitting to being gang members”.

Errors might come from multiple source. One of which might be deliberate intention to change ta to meet performance metrics. For example if police department is evaluated based on number of gang members caught, they have incentive to add higher number of gang members into the database.

It is important to have possibility to correct data in database, have expiration time. Otherwise mistakes stay there forever. These errors have real life impact. For example you might not get credit although you are eligible.

Credit report errors. Source

Surveillance operates without accountability

Surveillance tool companies in many cases aren’t held accountable. For example Amazon Rekognition tool has shown some biases (example were matched 28 members of Congress from a data set of mugshot photos). Creator argues that is has been used wrongly (threshold should be 99% to get a positive match), this is not what their single known police client does (source).

In Detroit were facial recognition software is in use, people demand turn things back. There are about 700 000 black people in there which creates a lot of false positives. “This technology does not accurately identify us. So to have a city government and DPD leadership pushing forward with the technology over a demographic that has tens of thousands of Black community members who can be misidentified with this technology is very detrimental to our livelihood and to our safety.” (source) There seems to be not really a way to hold somebody accountable for the mistakes.

Similar story is Palantir’s facial recognition software which was secretly deployed in New Orleans. Even council didn’t know that (source).

There should be a common goal: “making sociotechnical systems more responsive to marginalized communities.” (source)

Internal threats

We have a lot of attention to outside threats (cyber-attacks, troll-campaigns, malware etc.) but sometimes internal threats are much more dangerous. Some examples of internal threats:

Saudi Arabia infiltrated Twitter to get data about dissidents (source). This was serious leak because Twitter claims to provide anonymity (you can have whatever username there), but if your personal information leaks this is no longer the case.

to get data about dissidents (source). This was serious leak because Twitter claims to provide anonymity (you can have whatever username there), but if your personal information leaks this is no longer the case. Police officers using police databases to get information on romantic partners, business associates, neighbors, journalists and others which has nothing to do with the police work (source).

Rebuttals to Objections

There are some objections to previous criticism. Here is two of them:

Don’t people know what they are getting in to when they use these services?

This problem far more systematic than just dumb users who don’t understand what they are doing. Companies try to design privacy policies so that people wouldn’t read them. Facebook tracks you even if you don’t have an account (source). It would take average American about 40 minutes to read every privacy policy. Companies use design tricks to get you spend more time and more money and share more data. “But don’t blame people who, in a big, confusing, panopticon-like world, are simply doing the best they can.” (source)

It isn’t option to leave or not use tech. Many employers require the use of certain tech. In many cases we would need to opt out from modern life if we wouldn’t use these services. And we don’t have a granual control, we use these services and agree with the terms or we just don’t use them.

If you have done nothing wrong you should have nothing to hide

Some people face much higher scrutiny: Black people, Muslims, queer people and etc. For example black people in California are stopped far more often by police (source).

With surveillance-you can get somebody on anything. Some information that have been in surveillance could be used in future against you. Not to mention that you might get wrongly accused (“Face recognition may be least accurate for those it is most likely to affect: African Americans.” source)

You might be doing right thing but somebody might use it for doing wrong. Source