Applying machine learning language 

models for document linkage to 

evaluate FAQ Coverage

Monica Puerto*, Elizabeth Nichols+, Shaun Genter+, Kevin Zajack ,Brian Sadacca*

Accenture Federal Services* , U.S. Census Bureau+

FCSM October 2023

1

Disclaimer: This presentation is released to inform interested parties of research and to encourage discussion. The views 

expressed are those of the authors and not those of the U.S. Census Bureau. Transcript examples are heavily summarized. The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Objective: How can we use ML to help us understand if we have 

enough reference material to answer caller’s questions in a call 

center?

2

audio agent

Is, it, safe, to, 

respond online, 

safe, online, 

internet, security, 

protect, data, 

information, hack, 

Responding online 

is safe. The US 

Census Bureau is 

required, by law, 

to protect, your 

information,

Is, it, safe, to, 

respond online, 

safe, online, 

internet, security, 

protect, data, 

information, hack, 

Responding online 

is safe. The US 

Census Bureau is 

required, by law, 

to protect, your 

information,

Is, it, safe, to, 

respond online, 

safe, online, 

internet, security, 

protect, data, 

information, hack, 

Responding, online 

is, safe, The, US, 

Census, Bureau, is 

required, by law, 

to protect, your 

information,

~4.7 million inbound calls ~7,000 Call agents ~300 FAQs per 

language

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Creating embeddings for topic modeling / clustering
65K caller transcripts clustered into a 15 Topic Model

4

Census id / internet 

issues

Race / origin

Special 

living 

situation

Snowbird / multiple 

residence

General 

questions

No computer/ send 

paper Did not get 

questionnaire / send 

paper

Verify paper response

Verify 

computer 

response Completed 

census why am 

I receiving mail

Census visit

moved

Address questions

Enumeration No confirmation 

number

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
5

Census id / internet 

issues

Race / origin

Special 

living 

situation

Address question

65K caller transcripts clustered into a 15 Topic Model

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Top Words of sample of topics from the caller

Address Questions 

Call Topic

6

Census ID/ Internet 

issues Call Topic

• id

• Tried

• complete

• census

• invalid

• Mexican

• Hispanic

• Race

• Spanish

• American

Race / Origin 

Topic

• passed

• away

• deceased

• Died

• vacant

Special Living 

Situation Call Topic

• nursing

• facility

• assisted

• home

• living

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
can you provide me with a new 

different census id? 

what do i do if i can't find my 

census id? 

how do i answer the hispanic origin 

question? 

where do i find my census id?? 

why doesn't my census id work? 

how do i answer the race question? 

i am _____ but i don't see that on 

the race question what do i do?

what if i know this person's race 

but not their origin? 

I live in assisted living how do i 

respond? 

i live on a military base how do i 

respond? 

i live in an rv, boat, or hotel etc how 

do i respond? 

what are group quarters? 

Understanding FAQ Coverage via means of FAQ Categories

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
can you provide me with a new 

different census id? 

what do i do if i can't find my 

census id? 

how do i answer the hispanic origin 

question? 

where do i find my census id?? 

why doesn't my census id work? 

how do i answer the race question? 

i am _____ but i don't see that on 

the race question what do i do?

what if i know this person's race 

but not their origin? 

I live in assisted living how do i 

respond? 

i live on a military base how do i 

respond? 

i live in an rv, boat, or hotel etc how 

do i respond? 

what are group quarters? 

Census ID FAQ Category Race / Origin FAQ Category Special Living Situation FAQ Category

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035

Understanding FAQ Coverage via means of FAQ Categories
Which FAQ / FAQ Category maps the best to this Topic?

Call Topic #4

• nursing

• facility

• assisted

• home

• living

Most Similar FAQ Category:

Caller Topics

Special Living Situation FAQ Category

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Using cosine similarity between embeddings to 

measure semantic similarity

Call Topic #4

• nursing

• facility

• assisted

• home

• living

Caller Topics

.66

.62

.60

.55

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
What is the mean cosine for all the FAQ Categories to 

this topic?

Call Topic #4

• nursing

• facility

• assisted

• home

• living

Most Similar FAQ Category:

Caller Topics

Special Living Situation FAQ Category

Avg 

Cosine for 

this group: 

.10

Avg 

Cosine for 

this group: 

.20

Avg 

Cosine for 

this group: 

.60

AVG COSINE OF ALL FAQ CATEGORIES 

TO TOPIC #4 : .30 The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
What is the mean difference of all the FAQs if we drop 

the FAQ group of special living FAQs?

Call Topic #4

• nursing

• facility

• assisted

• home

• living

Caller Topics

AVG COSINE OF ALL FAQ CATEGORIES 

TO TOPIC #4 : .15

MEAN DIFFERENCE OF FAQs: .30 - .15 = 

.15 Difference
Imagine going from 15 Topics to More like 150 Topics!

13

15 Topics 150 Topics

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
14

 -

 0.05

 0.10

 0.15

 0.20

 0.25

 0.30

 0.35

150 Topics and their max mean difference after dropping each FAQ Category

(Topic #4 : nursing , 

facility, assisted, home, 

living ) after dropping 

Special Living FAQs 

compared to not dropping

TOPICS (1-150)

Max Mean Cosine Difference

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
15

 -

 0.05

 0.10

 0.15

 0.20

 0.25

 0.30

 0.35

150 Topics and their max mean difference after dropping each FAQ Category

Discovered potential 5-7 NEW FAQS among 10 

Topics that had no change from dropping each 

FAQ category and comparing mean differences

TOPICS (1-150)

Cosine Difference

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Takeaways

• We can use ML to help us understand if we have enough reference material to 

answer different types of conversations.

• Gather insight from large bodies of text without having a human read through 

thousands of conversations after clustering them via topic modeling.

• Measuring distances between variables (embeddings) to understand the 

relationships and then changing the environment (dropping the FAQ category) 

and analyzing if there was any change from the norm to estimate coverage.

• Topic modeling at a granular level helps estimate FAQ coverage. 

• Estimating that Census had coverage of 96% of types of conversations with 

their 300 FAQs since after manual review of the ~10 topics that had no 

change, found potentially 5-7 topics that could benefit from a new FAQ (niche topics).

• If we can anticipate the same thing for 2030 we know that what we saw in 

the past, we are prepared for and have a algorithmic approach to do on 

going evaluations during 2030 operations. 

16

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035
Thank you!

Contact:

monica.puerto@census.gov

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035 17
Appendix

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY22-CBSM002-026 18
78

21 19

16 15 13 12 11 10 10 9 9 8 8 7 7 6 5 5 4 4 4 4 3 3 2 2 2 1

0

10

20

30

40

50

60

70

80

90

FAQ Categories and Count of FAQs under each

19

The presentation has been reviewed for disclosure avoidance 

and approved under CBDRB-FY24-CBSM002-035