DHS' Pilots for Social 

Media Screening Need 

Increased Rigor to Ensure 

Scalability and Long-term 

Success 

(Redacted) 

February 27, 2017 

OIG-17-40 
 

 

 

 

 

 

 

FOR OFFICIAL USE ONLY 

DHS OIG HIGHLIGHTS 

DHS’ Pilots for Social Media Screening Need Increased

Rigor to Ensure Scalability and Long-term Success 

February 27, 

2017 

Why We 

Did This 

Inspection 

Following the December 

2015 terrorist attack in 

San Bernardino, 

California, Congress 

raised concerns about 

the use of social media 

by terrorist groups and 

requested that the 

Department of Homeland 

Security expand social 

media background 

checks. We conducted 

this inspection to review 

DHS’ social media pilot 

programs. 

What We 

Recommend 

We made one 

recommendation to help 

ensure DHS develops an 

effective social media 

screening program. 

For Further Information: 

Contact our Office of Public Affairs at 

(202) 254-4100, or email us at 

DHS-OIG.OfficePublicAffairs@oig.dhs.gov 

What We Found 

DHS has established a task force for using social media to 

screen applicants for immigration benefits. In connection 

with that effort, U.S. Citizenship and Immigration Services 

(USCIS) began pilots to expand social media screening of 

immigration applicants. Additionally, Immigration and 

Customs Enforcement (ICE) independently began a pilot to 

use social media screening during the visa issuance process. 

However, these pilots, on which DHS plans to base future 

department-wide use of social media screening, lack criteria 

for measuring performance to ensure they meet their 

objectives. Although the pilots include some objectives, such 

as determining the effectiveness of an automated search tool 

and assessing data collection and dissemination procedures, 

it is not clear DHS is measuring and evaluating the pilots’ 

results to determine how well they are performing against set 

criteria. Absent measurement criteria, the pilots may provide 

limited information for planning and implementing an 

effective, department-wide future social media screening 

program. 

DHS Response 

I&A, on behalf of itself, USCIS, and ICE, concurred with the 

recommendation and has begun taking corrective actions. 

www.oig.dhs.gov OIG-17-40 

FOR OFFICIAL USE ONLY
 
FOR OFFICIAL USE ONLY 


OFFICE OF INSPECTOR GENERAL 


Department of Homeland Security 


Washington, DC 20528 / www.oig.dhs.gov 

February 27, 2017 

MEMORANDUM FOR: David J. Glawe 

Acting Under Secretary 

Office ofIntelligenoe and Analysis 

Lori Scia1abba 

Acting Director 

U.S. Citizenship and Immigration Services 

Thomas D. Homan___ 

Acting Director 

U.S. Immigration and Customs Enforcement 

FROM: John Roth ~k~h_ Inspector General 

SUBJECT: DHS' Pilots for Social Media Screening Need Increased 

Rigor to Ensure Scalability and Long-term Success ­ For 

Officia1. Use Omy 

For your action is our final report, DHS' Pilots for Social Media Screening Need 

Increased Rigor to En.sure Scalability and Long-term Success. We incorporated 

the formal! comments provided by your office. 

The report contains one recommendation aimed at improving DHS' social 

media scrieening pilots. The Office of Intelligence and Analysis il~ concurred 

with tlte recommendation. Based on information provided in the response to 

the draft :repon, we consider the recommendation open and resolved. Once the 

recommendation has been fully implemented, plea5e submit a formal doseout 

letter to us within 30 days so that we may close the recommendation. The 

memorandum should be accompanied by evidence of completion of agreedupon corrective actions. Please send your response or closure request to 

OIGinspectionsFollov.n..ipg)oig.dhs.gov. -- -- ·· -- ·-· 

Consistent \vi.th our responsibility under the Inspector General Act. we will 

provide copies of our report to congressional committees with oversight and 

appropriation responsibility over the Department of Homeland Security. We will 

post a redacted version of the report on our website. 

Please call me with any questions, or your staJf may amt.act Laurel Rim.on, 

Acting Ac;sistant Inspector General for Inspections and Evaluations or Angela 

Garvin. Deputy Assistant Inspector General, at f202) 254-4100. 

www.oig.dhs.gov OIG- 17-40 

FOR OFFICIPaL USE ONLY 
 

 

 

 

 

 

 

 

 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

Background 

Following the December 2, 2015 terrorist attack in San Bernardino, California, 

lawmakers became increasingly concerned about the use of social media by 

terrorist groups. On December 15, 2015, Senator Jeanne Shaheen and 24 

other Senators sent a letter to the Secretary of Homeland Security requesting 

that, as soon as possible, DHS expand social media background checks, 

focused on possible connections to terrorist activity, to screening for visa 

determinations for visitors and immigrants. In response, the then DHS 

Secretary and Deputy Secretary asked the Under Secretary for Intelligence and 

Analysis (I&A) to lead a task force to review the Department’s current use of 

social media and identify options to optimize its use across DHS. The task force 

comprises senior representatives and staff from DHS, U.S. Citizenship and 

Immigration Services (USCIS), U.S. Immigration and Customs Enforcement 

(ICE), U.S. Customs and Border Protection, the Transportation Security 

Administration, and DHS oversight offices including the Office of General 

Counsel, Privacy Office, and Office for Civil Rights and Civil Liberties, which 

have a vested interest in DHS’ use of social media. 

According to the task force’s implementation plan, the first priority was to 

conduct a pilot at USCIS to test systematic screening of applicants for 

immigration benefits.1 USCIS had used social media in a limited capacity but 

had no experience using it as a large-scale screening tool. In December 2015, 

USCIS started screening the social media accounts of a limited number of 

applying for status, using both manual and 

automated screening. The task force intended to use the December 2015 pilot 

and lessons learned from other DHS components’ use of social media to 

develop policies and processes for standardized use of social media 

department-wide. Additionally, to expand social media screening across all 

DHS components, the task force recommended, and the Secretary approved, 

creation of a DHS Social Media Center of Excellence (COE).2 (Appendix B 

contains details about the COE.) In April 2016, USCIS expanded social media 

screening, testing another automated tool together with manual screening. In 

1 USCIS grants immigration benefits to people who meet the eligibility requirements to stay in 

the United States temporarily or permanently. These benefits include granting U.S. citizenship 

to eligible individuals, authorizing individuals to reside permanently in the United States, and 

authorizing aliens to work in the United States. 

2 During our review it was called the “Center of Excellence”; however, the name was later 

changed to “Shared Social Media Screening Service”. We refer to it as the “Center of Excellence” 

in this report. 

www.oig.dhs.gov 1 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

USCIS started a pilot in December 2015 to screen the social media accounts of 

and applicants for status. The pilot’s objective was to 

examine the feasibility of using social media screening with an automated 

search tool called and determine whether useful information for 

adjudicating refugee applications could be obtained.6 Although the pilot had an 

objective, it did not define what would constitute a successful outcome, nor did 

it identify metrics against which to benchmark its findings. For example, the 

pilot discovered that individuals had confirmed social media 

accounts; likely had a social media account; individuals had unconfirmed 

 

3 American embassies and consulates issue nonimmigrant visas authorizing temporary visits to 

the United States for individuals who do not intend to become permanent residents. 4 GAO-12-208G, January 2012 5 GAO-09-45, November 2008 6 , a social media analytics tool, covers a large number of social media platforms, 

has access to third-party information providers, and can access web-based information. 

www.oig.dhs.gov 2 OIG-17-40 

FOR OFFICIAL USE ONLY
 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

August 2016, ICE began using a different automated tool to screen the social 

media of nonimmigrant visa holders .3 

Results of Inspection 

DHS has established a task force for using social media to screen applicants 

for immigration benefits. In connection with that effort, USCIS began pilots to 

expand social media screening of immigration applicants. Additionally, ICE 

independently began a pilot to use social media screening during the visa 

issuance process. However, these pilots, on which DHS plans to base future 

department-wide use of social media screening, lack criteria for measuring 

performance to ensure they meet their objectives. Although the pilots include 

some objectives, such as determining the effectiveness of an automated search 

tool and assessing data collection and dissemination procedures, it is not clear 

DHS is measuring and evaluating the pilots’ results to determine how well they 

are performing. Absent measurement criteria, the pilots will be of limited use in 

planning and implementing an effective, department-wide future social media 

screening program. 

The Government Accountability Office (GAO) has issued best practices for an 

effective pilot phase of a program. According to GAO, “The pilot phase allows 

for a check on whether program operations occur as expected.”4 GAO has also 

stated that pilot programs should have well-defined, clear, and measurable 

objectives; criteria or standards for determining pilot performance; and a plan 

to track the pilot’s performance and evaluate the final results.5 Following these 

best practices can increase the rigor of the pilots to ensure scalability and longterm success. 
 

accounts; and individuals had no identified social media account. USCIS 

concluded that the individuals had a minimal presence on 

social media platforms accessible through . Because the pilot did 

not have metrics to measure success, it is difficult to conclude whether finding 

individuals with confirmed social media accounts constitutes success. 

The purpose of another USCIS pilot, which began in April 2016, was to expand 

screening to an additional , predominately , applicants 

through use of the Defense Advanced Research Projects Agency (DARPA) 

tool.7 The applicants were asked to voluntarily give their social media user 

names.8 USCIS then screened the user names against 

using the tool; USCIS also manually screened the user names against 

. USCIS assessed identified accounts to determine whether the 

refugees were linked to derogatory social media information that could impact 

their eligibility for immigration benefits or admissibility into the United States. 

Using the tool and manual screening, USCIS identified 

individuals with confirmed social media accounts and individuals with 

unconfirmed accounts. In reviewing the pilot, USCIS concluded that the 

tool was not a viable option for automated social media screening and that 

manual review was more effective at identifying accounts. USCIS based its 

conclusion on the tool’s low “match confidence.” Because the resulting 

accounts identified by the tool did not always match up with the applicants, 

officers had to manually check the results. However, USCIS did not establish 

match benchmarks for the tool, so it does not know what level of match 

confidence would signify success or failure. 

In August 2016, ICE independently began a pilot to screen the social media 

activity of nonimmigrant visa applicants from 

. The intent of this pilot is to 

evaluate the capability of to conduct initial 

screening of social media activity during visa application and continue social 

media monitoring during the .9 

According to the pilot plan, this approach will complement existing background 

 

7 DARPA is a social media screening platform that screens against . 8 National Security Council staff requested USCIS and the Department of State, which also 

participated, to pilot the elicitation of the applicants’ social media identifiers. USCIS is 

to access social media platforms by DHS Privacy 

regulations and using government accounts on social media platforms may introduce 

additional risk to government systems. DHS and its components have of 

identifying social media account for individuals that may not be 

appropriate or viable for large scale screening of applicants for immigration benefits. 

9 is a web search tool that specializes in social media data 

exploitation by analyzing social media data and funneling it into actionable information. 

www.oig.dhs.gov 3 OIG-17-40 

FOR OFFICIAL USE ONLY
 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 
 

 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

checks conducted in conjunction with the Department of State and help 

identify potential derogatory information not found in Government databases. 

Although ICE has identified several measurable criteria for data that will be 

collected during the pilot, it has not articulated a plan to use these metrics to 

evaluate performance and determine overall program effectiveness. In our 

opinion, a clearly defined performance evaluation is essential to determining 

whether, how, and when to integrate pilot activities into overall efforts and will 

enable the Department to implement a more effective social media screening 

program. 

DHS and the task force are planning department-wide use of social media for 

screening based on these pilots. Because components are not measuring their 

pilots against clear success criteria, DHS may not be able to make informed 

decisions when it designs its social media screening program. The task force 

needs to better manage ongoing efforts to pilot social media screening to help 

ensure an effective and successful social media screening program in the 

future. 

Recommendation 

We recommend that the Under Secretary of Intelligence and Analysis 

coordinate with USCIS and ICE to develop and implement a plan to evaluate 

the performance of social media screening pilots that includes features such as 

well-defined, clear, and measurable objectives and standards for determining 

pilot performance to help ensure DHS develops an effective social media 

screening program. 

Management Comments and OIG Analysis 

I&A responded on behalf of itself, ICE, and USCIS. The components concurred 

with our recommendation and provided comments to the draft report. A 

summary of the management response and our analysis follows. We have 

included a copy of the management comments in their entirety in appendix A. 

I&A also provided technical comments to our report, which we incorporated, as 

appropriate. 

I&A’s Response to the Recommendation: I&A concurred with the 

recommendation and stated that DHS has taken steps to improve its social 

media screening pilots by implementing a four-pronged approach that 

measures performance to develop consistent benchmarks and continue 

improving performance to ensure rigor and scalability for long-term success. 

This approach includes using qualitative and quantitative criteria for 

measuring tool performance; collecting and analyzing comprehensive 

www.oig.dhs.gov 4 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

 

 

 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

performance metrics of ongoing research and development pilots; reporting 

project milestones to the task force; and reporting select metrics measuring 

pilot performance in a weekly task force agenda. Additionally, DHS will 

continue to identify and refine new and existing metrics to measure social 

media screening pilots’ performance, to include identifying benchmarks for the 

various tools tested. 

OIG Analysis: I&A’s planned actions are partially responsive to this 

recommendation. This recommendation is resolved, but will remain open until 

I&A provides documentation of well-defined, clear, and measurable objectives 

and standards for determining pilot performance. I&A’s response indicated that 

a new social media screening pilot was scheduled to begin in January 2017. 

I&A should take this opportunity to develop and implement GAO’s best 

practices for an effective pilot program to ensure scalability and long-term 

success of the pilot. 

I&A also emphasized that, at the launch of the task force, neither the private 

sector nor the U.S. Government possessed the capabilities for large-scale social 

media screening. OIG recognizes that DHS has made strides in identifying and 

assessing social media screening technology and conducting pilots that will 

enable the development of such capabilities. The Homeland Security Systems 

Engineering & Development Institute report, Social Media Analytics Capability 

Testing: Independent Assessment, referenced in I&A’s response, provides 

evidence of some of the progress that has been made in leveraging technology 

to meet the Department’s needs. The report was not received by OIG until 

January 6, 2017, and therefore was not included in our analysis. We note that 

this independent assessment includes many metrics that may be useful in 

establishing benchmarks and determining performance of future pilot 

programs. The assessment included seven steps in conducting evaluations for 

quantitative analysis: (1) define the purpose of the evaluation; (2) elaborate a 

task model; (3) define top-level quality characteristics; (4) produce system 

requirements; (5) define metrics to measure requirements; (6) define techniques 

to measure metrics; and (7) carry out and interpret the evaluation. This 

framework was followed for the independent assessment and should be 

continued with testing pilots. 

Objective, Scope, and Methodology 

DHS OIG was established by the Homeland Security Act of 2002 (Public Law 

107−296) by amendment to the Inspector General Act of 1978. 

The initial objective of this inspection was to evaluate DHS’ progress in 

developing social media screening policies, procedures, and capabilities. 

www.oig.dhs.gov 5 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

However, upon discovering the lack of performance measures in USCIS’ and 

ICE’s pilots for social media screening for immigration benefits, we adjusted 

our scope and reviewed DHS’ social media task force and DHS’ use of pilots to 

help create a social media screening program. Although we reviewed DHS 

policies relevant to social media screening, including DHS Instruction Number 

110–01-001, Privacy Policy for Operational Use of Social Media, we could not 

evaluate specific policies and procedures for the pilots because they had not 

been written. Likewise, we reviewed the task force’s concept of operations for 

the role of the COE, but could not evaluate the COE at this time because it has 

not been set up or funded. 

We interviewed DHS staff from the Office of Policy, USCIS’ Fraud Detection and 

National Security Directorate, ICE’s Visa Security Program and Homeland 

Security Investigations, Science and Technology Directorate, Privacy Office, 

Office for Civil Rights and Civil Liberties, and the Chair of the Social Media task 

force to determine their roles in social media screening. 

We conducted this inspection between July and October 2016 pursuant to the 

Inspector General Act of 1978, as amended, and according to the Quality 

Standards for Inspection and Evaluation issued by the Council of the Inspectors 

General on Integrity and Efficiency. 

The Office of Inspections and Evaluations major contributors to this report are 

William McCarron, Chief Inspector; Natalie Fussell Enclade, Lead Inspector; 

Carie Mellies, Lead Inspector; Stephen Farrell, Inspector; Adam Robinson, 

Inspector; James Alver, Inspector; Kelly Herberger, Communications Analyst; 

and Elizabeth Kingma, Independent Referencer. 

www.oig.dhs.gov 6 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

Appendix A 

I&A’s Comments to the Draft Report 

www.oig.dhs.gov 7 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 8 OIG-17-40 

FOR OFFICIAL USE ONLY 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 9 OIG-17-40 

FOR OFFICIAL USE ONLY 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

www.oig.dhs.gov 10 OIG-17-40 

FOR OFFICIAL USE ONLY 
 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

Appendix B 

Center of Excellence 

The COE concept of operations envisions the center conducting social media 

activities on behalf of components that lack resources or expertise and 

supporting component-level social media screening programs. The COE will 

also identify new social media tools and opportunities, test and evaluate new 

and emerging technologies, and identify best practices for the appropriate 

sharing and collaborative use of social media within DHS and with external 

partners. The COE will set standards for social media use in relevant DHS 

operations while ensuring privacy and civil rights and civil liberties protections. 

The task force also developed a proposal for the creation of a social media 

board and social media council to replace the existing task force and oversee 

the future governance of DHS’ social media use and the COE. According to the 

proposed charter, the social media board will be a senior-level independent 

group that will ensure awareness of social media use and capabilities among 

DHS leadership and control DHS’ social media use programs. The social media 

council will ensure working-level communication for social media use and 

capabilities, develop new policies and procedures, oversee performance 

management, and promote information sharing. 

www.oig.dhs.gov 11 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

 

FOR OFFICIAL USE ONLY 

OFFICE OF INSPECTOR GENERAL 

Department of Homeland Security 

Appendix C 

Report Distribution 

Department of Homeland Security 

Secretary 

Deputy Secretary 

Chief of Staff 

Deputy Chiefs of Staff 

General Counsel 

Executive Secretary 

Director, GAO/OIG Liaison Office 

Assistant Secretary for Office of Policy 

Assistant Secretary for Office of Public Affairs 

Assistant Secretary for Office of Legislative Affairs 

Office of Intelligence and Analysis Audit Liaison 

USCIS Audit Liaison 

ICE Audit Liaison 

S&T Audit Liaison 

Office of Management and Budget 

Chief, Homeland Security Branch 

DHS OIG Budget Examiner 

Congress 

Congressional Oversight and Appropriations Committees 

www.oig.dhs.gov 12 OIG-17-40 

FOR OFFICIAL USE ONLY
 
 

 

 

 

 

 

 

ADDITIONAL INFORMATION AND COPIES 

To view this and any of our other reports, please visit our website at: www.oig.dhs.gov. 

For further information or questions, please contact Office of Inspector General Public Affairs 

at: DHS-OIG.OfficePublicAffairs@oig.dhs.gov. Follow us on Twitter at: @dhsoig. 

OIG HOTLINE 

To report fraud, waste, or abuse, visit our website at www.oig.dhs.gov and click on the red 

"Hotline" tab. If you cannot access our website, call our hotline at (800) 323-8603, fax our 

hotline at (202) 254-4297, or write to us at: 

Department of Homeland Security 

Office of Inspector General, Mail Stop 0305 

Attention: Hotline 

245 Murray Drive, SW 

Washington, DC 20528-0305