2018 Evaluation Report: Impact of 

the Northstar Assessment & Related 

Computer Skills Programming on 

Employment in CTEP Programs 

 12/11/2018 

 Prepared by Daniel Backman, MDP 
Table of Contents 

I. Purpose of the Study 

II. Key Takeaways 

III. Technical Summary 

IV. Return on Investment 

V. Background Information on Evaluator, CTEP and Northstar Digital Standards 

VI. Study Design 

VII. Methodology 

VIII. Comparison Group 

IX. Evaluation Comparison: 2015 and 2018 

X. Analysis: Demographics 

XI. Analysis: Adult Learners, Northstar Assessments and Computer Skills Programming

XII. Analysis: Employment 

XIII. Interviews with Program Leads at Adult Learning Centers 

XIV. Acknowledgements 

XV. Appendix 

I. Purpose of the Study 

The Saint Paul Neighborhood Network (SPNN) and the Community Technology Empowerment Project 

(CTEP) have contracted with Mr. Daniel Backman to evaluate their usage of the Northstar assessment 

and Northstar related classroom programming as it relates to learner employment outcomes. 

In 2015, CTEP contracted Mr. Backman and the Minnesota Literacy Council to conduct a program 

evaluation to determine the effect that CTEP programs, and specifically the use of Northstar Digital 

Literacy Assessments and related computer skills instruction, had on the employability of adult learners 

who had attended CTEP programming. Mr. Backman designed a survey study to evaluate whether 

participation in Northstar Assessments and related computer skills instruction improved job placement 

rates for adult learners. 

This report contained herein is, in part, a continuation of the work that was started in 2015. The CTEP 

program required updated data on the outcomes of their program. In the process of preparing for a new 

study, CTEP sought to improve upon the study conducted in 2015 by improving the survey design, 

improving CTEP member involvement in the study and reaching more adult learners. These goals were 

achieved. 

CTEP functions as a key program of its parent organization, Saint Paul Neighborhood Network (SPNN). 

Founded in 1984 as a nonprofit community media training center, SPNN works through community 

partnerships with nonprofits and governmental entities to increase digital and media literacy and to 

better educate the community. SPNN’s existing partnerships with many social service organizations 

provide a strong training network for CTEP members. 
The 2015 study, titled Evaluation Report: Employment and Educational Impact of the Community 

Technology Empowerment Project of the Saint Paul Neighborhood Network can be found in full in the 

Appendix. 

II. Key Takeaways 

 Among all survey respondents, the employment rate increased just over 10 percentage points, 

from 43.8% to 54.04% 

 Over 47% of unemployed adult learners who were searching for work had found employment at 

the time of survey administration 

 Over 81% of adult learners who were looking for work reported that attending CTEP-led 

computer skills classes helped them in searching for employment 

 For all employed adult learners, the median hourly wage was $14.79 and the mean hourly wage 

was $16.03 

 Compared to the State of Minnesota's Workforce program result data, the CTEP program had a 

post-intervention employment rate that was 7 percentage higher among a comparable 

population 

III. Technical Summary 

In total, 272 surveys were administered by trained CTEP members to adult learners who passed a 

Northstar Assessment and participated in at least four hours of Northstar-related basic computer skills 

classes (“CTEP Northstar programs”) at one of 30 CTEP host site locations. The population of adult 

learners at all CTEP host site locations who passed at least one Northstar Assessment in the eligibility 

period of the survey (September 1 2017-June 15 2018) totaled 1,183 adult learners. At a confidence 

level of 95%, the sample size of 272 respondents in a population of 1,183 has a confidence interval of 

5.22%. 

The survey respondents were reflective of the diversity of the CTEP programs; nearly 67% of the sample 

are persons of color, 27.2% of the sample respondents are white and the remaining 5.88% of the sample 

responding as ‘prefer not to say’ or ‘none of the above’. English as a Second Language learners formed 

50.37% of the sample. 

This study found that, among all survey respondents, the employment rate increased just over 10 

percentage points, from 43.8% (118/272) to 54.04% (147/272) between the dates that they entered a 

CTEP Northstar program and the date they were surveyed (at least 4 weeks after they passed their first 

Northstar Assessment). Of the 154 survey respondents who were unemployed when they first attended 

a CTEP Northstar program, just over 51% (80/154) reported that they were actively looking for a job. Of 

those 80 unemployed adult learners searching for work, 47.5% (38/80), had found employment at the 

time of survey administration. 

Among all survey respondents, both employed and unemployed, just under half reported that they were 

currently looking for work (49.6%, 135/272). Just over 40% of adult learners who reported that they had 

a job said that they were searching for work. Of the 135 adult learners who reported that they were 

searching for work, nearly 63% (85/135) reported that they were employed at the time of survey. As 

only 41% (55/135) of those looking for work were employed at entry into the CTEP program, this 

represents a 22-percentage point increase in employment among those actively searching for work 
while enrolled in a CTEP Northstar program. Overwhelmingly, these adult learners who reported looking 

for employment reported that attending computer skills classes helped them in searching for 

employment (over 81% (110/135)). 

Of the survey respondents who reported that they were employed at the date of survey (147/272), 142 

respondents wages could be estimated using the 2017 Bureau of Labor Statistics’ Occupational 

Employment and Wages Estimates for the Minneapolis/Saint Paul/Bloomington MN-WI Metropolitan 

Statistical Area. For all 142 respondents, the median hourly wage was $14.79 and the mean hourly wage 

was $16.03. If you extrapolate the hourly wage, the median annualized salary was $30,763.20 (mean 

salary $33,342.40) based on a 40 hour work week, 52 weeks per year. 

To compare the employment results of adult learners in surveyed CTEP programs, the researchers 

requested and obtained employment data from 13 Twin Cities metro area WorkForce Centers 

(WFCs), which operate under the Department of Employment and Economic Development1 system 

between the dates September 1, 2017 and June 30, 2018. 

DEED reported that among unemployed, job seeking WorkForce Center customers who received an 

intervention of similar time duration (at least four hours of assistance), 40% (351/878) found work 

within 90 days after exiting the WFC. Among CTEP survey respondents who were unemployed at the 

time of taking CTEP Northstar programs and were searching for work (29.4%, 80/272), over 47% (38/80) 

found employment at the time of the survey administration. The CTEP intervention, with a 47% 

employment rate of comparable unemployed, job-seeking participants, was 7 percentage points higher 

than the DEED comparison group. The 7% difference in employment rates is greater than the calculated 

margin of error of 5.22% on this sample population. 

IV. Return on Investment 

To report on the return on investment of the Community Technology Empowerment Project for the 

program year 2017-2018, the following formula is proposed2: 

Formula: 1,183 x 28.14% x $30,763.20 x 73.68% x (+/-) 5.22% = ROI (+/-) 

Total Return on Investment: $7,545,533.71 (+/- $393,876.86) 

 1,183: Total number of adult learners that passed at least one Northstar assessment in CTEP 

programs 

 28.14%: The percentage of sample who reported unemployment at initial class attendance 

(154/272) AND who reported that they looked for a new job since initial class attendance 

(55/135) AND who reported current employment at date of survey administration (38/135).

 

1

 See Appendix for data and report from DEED. 

2

This formula was used in the 2015 report and is replicated here with updated information. This formula should be 

considered a rough estimate of financial impact and has not been rigorously tested. 
 $30,763.20: Median annualized salary of those who transitioned from unemployed to employed 

(based off sector and job title provided by survey respondents, salary data used from Bureau of 

Labor Statistics within the Minneapolis-St.Paul-Bloomington metropolitan statistical area. 

 73.68%: The percentage of those who reported that they looked for jobs, went from 

unemployed to employed AND who reported that attending CTEP computer skills programming 

helped them look for jobs (28/38) 

 5.22%: The margin of error of survey sample size based on total CTEP population who passed 

Northstar assessments 

From the survey results, 28.14% of the adult learner population sample who passed at least one 

Northstar Assessment reported that they had looked for work since first attending the basic computer 

skills class and also transitioned from unemployment to employment. We can reasonably apply this 

percentage to the total population of adult learners across all CTEP programs (1,183) who passed at 

least one Northstar Assessment to be representative of the percentage people in all CTEP programs that 

will transition from unemployment to employment if they are job-seeking. The median annualized salary 

of those who transitioned from unemployed to employed in the sample is $30,763.20. Of those who 

transitioned from job-seeking survey respondents who transitioned from unemployment to 

employment, 73.68% reported that attending CTEP computer skills programming helped them look for 

jobs. Thus, we can apply 28.14% to 1,183, to assume that 407.18 adult learners from the CTEP adult 

learner population also transitioned from unemployment to employment if they were job-seeking. Of 

those 407.18 adult learners, we can assume that 73.68%, or roughly 300, of those would report that 

attending CTEP computer skills programming helped them look for jobs. Thus, you can take the 300

adult learners multiplied by the annualized median salary (30,763.20) of the adult learner population 

sample who transitioned from unemployment to employment, for a total estimated economic impact of 

$7,545,533.71. The sample has a margin of error of 5.22%, which needs to be applied to final number. 

V. Background Information on Evaluator, CTEP and Northstar 

Digital Standards 

CTEP contracted with Daniel Backman to lead this survey-based evaluation. Mr. Backman is a graduate 

of the Humphrey Schools of Public Affairs with a Master’s Degree in Development Practice (MDP) and 

holds a B.A. in Sociology from Iowa State University. The MDP degree program trains students in 

program design, research protocol, program evaluation and statistical analysis. Presently, Mr. Backman 

is a Senior Data Analyst at the Institute of Social Research and Data Innovation (ISRDI) at the University 

of Minnesota. At ISRDI, Mr. Backman is the project manager on the IPUMS-Time Use data projects 

where he works on developing innovative data infrastructure for broad data dissemination, harmonizing 

time use survey data across time and countries, and conducting research and analysis using census, 

survey and time diary data. Prior to his work at ISRDI, Mr. Backman was the lead evaluator of internal 

programs at the Minnesota Literacy Council and was their lead evaluator on multiple external evaluation 

contracts with public and nonprofit partners. 
Each year, CTEP trains around 35 AmeriCorps members to teach computer literacy skills intended to 

improve job placement and retention for low-income, low-skilled residents in the Twin Cities 

metropolitan area. Members provide training at no charge to adult learners at more than thirty 

community-based locations including libraries, public housing, social service agencies, workforce 

centers, youth employment programs, and disability inclusion organizations. Members use assessment 

tools and computer skills curriculum based on Northstar Digital Literacy Standards that were designed 

and developed by the St. Paul Public Library in 2012 with state and federal funding. Since then, these 

standards have been adopted as statewide digital literacy standards by Minnesota Adult Basic 

Education, and put into use in more than 225 Northstar public computer access locations in 27 states 

nationwide.

In 2010, the St. Paul Public Library and the St. Paul Community Literacy Consortium began a community 

process to determine how best to assess and quantify digital literacy knowledge among lower-skilled 

adults, as well as for displaced workers who might lack such skills. The intent was to provide meaningful 

assessment that could lead to a certificate useful for employers and job seekers. Through an open 

community process, the Northstar Digital Literacy Standards were designed over a period of several 

months by a taskforce with representatives from non-profit community-based agencies, public and 

academic libraries, Minnesota Department of Education, DEED and workforce centers, Adult Basic 

Education professionals, and others.

In 2011, the Friends of the St. Paul Public Library obtained funding through the Otto Bremer Foundation 

and the Library Services and Technology Act, from the Institute of Museum and Library Services, 

administered by the Minnesota Department of Education, State Library Services Division. These funds 

supported the hiring of a professional design team to create the online assessments. The Design Team 

included Jen Vanek, Project Liaison, Michael J. Graif, Digital Literacy Designer, and Jennifer Asp, 

Educational Specialist, under the coordination of Tom Cytron-Hysom, Project Manager. A Community 

Advisory Group, organized by St. Paul Library Director Kit Hadley, provided feedback and assistance to 

the Design Team. A CTEP Community Engagement Project assisted in designing online training for 

proctors at approved sponsor sites.

In late-2011 to mid-2012, online assessment modules were designed, piloted, and implemented. A 

process through which organizations could be certified to award the Northstar Certificates was 

developed. A database and other technical infrastructure were completed.

CTEP adopted the Northstar Digital Literacy Standards as a way to establish common criteria amongst all 

CTEP sites for what constitutes learning computer literacy skills. The standards were created by skilled 

consultants with funding from Otto Bremer Foundation and Friends of the St. Paul Public Library. All 

consultants had advanced degrees in adult basic education and professional experience creating online 

learning modules. Beginning in September 2012, all CTEP members began proctoring pre- and postassessments around the five Northstar Standards: Basic Computer Use, Internet, Operating Systems, 

Email, and Word Processing.

All of the assessments can be accessed at: https://www.digitalliteracyassessment.org/.
Each assessment takes roughly 30-45 minutes for a participant to complete and they are free and 

available on the internet. Each assessment has about 40 questions, and for an individual to be counted 

as making progress, they would need to score less than certification level of 85% correct on the preassessment, and score above the certification level on the post-assessment. CTEP provides a 

standardized curriculum connected to each standard. Members can teach participants skills above and 

beyond this list, but at least one of these standards must be incorporated into programming. 

The Northstar Digital Literacy online assessments are designed to be an economical means by which 

digital literacy can be evaluated for purposes of aiding instruction and for conferring recognition of 

digital literacy. The assessment tool was designed to assess behaviors in the form of simulated tasks that 

digitally literate people can be reasonably expected to be able to perform, knowledge of technology 

information in the form of structured questions that digitally literate people can be reasonably expected 

to answer correctly, and attitudes about appropriate use of digital technology in the form of structured 

questions in which technologically literate people can be reasonably expected to identify appropriate 

use. The assessment tool was tested for question validity by piloting beta versions both in groups 

thought not to be digitally literate and groups that are believed to be digitally literate. Questions in each 

group were identical. Results were analyzed to determine questions which may not be valid assessments 

of digital literacy within the context of these results. The assessment tool can be said to be reliable as it 

does not rely on independent observers potentially introducing observer bias but instead 

programmatically assesses user responses according to predetermined values in a consistent way. The 

tool was tested technically to ensure accurate and intended performance and thus can be expected to 

produce reliable results across use. 

Each year in the first month of service for all CTEP members, the educational consultants from the St. 

Paul Community Literacy Consortium, who created the Northstar Standards and Assessments, lead 

training for CTEP members on the Northstar standards, assessments, and technology curriculum 

implementation best practices. 

VI. Study Design 

This evaluation was designed as a quasi-experimental study with a comparison group. The evaluator, 

Mr. Backman, developed this study and survey tool in conjunction with the CTEP senior staff Joel 

Krogstad and Lisa Peterson-de la Cueva. Mr. Backman led the previous external evaluation of the CTEP 

AmeriCorps program in 2015. 

The previous evaluation earned an Evidence Tier of Strong, and as such, Mr. Backman and the CTEP 

leadership team approached this evaluation determined to improve on our previous effort, but not to 

significantly alter the previously successful approach. 

Acknowledgement to Kaitlyn Schmaltz, a CTEP member, for assisting Daniel Backman in coordinating the 

CTEP member survey administration effort. She assisted in keeping records of how many CTEP members 

had administered surveys, motivated CTEP members to administer surveys throughout the survey 

period and assisted the evaluator in initial data analysis. 
Survey Instrument Design. The survey instrument was thoroughly tested and revised from the previous 

evaluation. A small group of CTEP AmeriCorps members from the outgoing 2017 CTEP class were 

recruited to review, test and give feedback on the survey instrument. In the summer of 2017, this group 

was given a draft survey instrument and instructed to recruit adult learner volunteers from their 

programs to test the instrument. They noted where the survey could be improved to better engage 

adult learners based on their learning ability, English comprehension level and attention span. On July 

19th 2017, after each CTEP member conducted at least three practice survey administrations, the group 

met with Mr. Backman in a focus group setting to give their personal feedback and the adult learner 

feedback on the survey instrument design. The survey instrument was revised to incorporate the 

feedback and was approved by this CTEP group. 

On September 8th 2017, Mr. Backman attended an all-CTEP member and site supervisor training day to 

introduce the survey (scheduled for spring 2018) and to get feedback from members and site 

supervisors on the survey instrument. The CTEP members and site supervisors were required to practice 

administering the survey instrument in small groups. This feedback was incorporated and the 

instrument revised. 

On November 17th 2017, Mr. Backman attended an all-CTEP member training day to host a training 

using the revised survey instrument. This training walked through steps on how to call and track adult 

learners, how to introduce the study, how to administer the survey and how to enter the data in the 

survey form. The CTEP members were required to practice the survey administration protocol in small 

groups. Final feedback was solicited on the survey instrument. This feedback was incorporated and a 

final version of the survey instrument was sent to anyone who submitted feedback.

On February 2nd 2018, Mr. Backman attended an all-CTEP member training day to host the final training 

session on the survey administration. At this training, all CTEPs were given a final version of the survey 

instrument. CTEPs received a final training on how to administer the survey and were required to 

practice the survey instrument with their peers. 

The survey instrument was designed to be administered by trained CTEP members. CTEP members were 

utilized to deliver the survey because each member and their host site have different methods and data 

practices for recording student data, including contact information, class attendance records and 

Northstar Assessment results. It was determined by CTEP leadership and Mr. Backman that an outside 

survey administrator would be cost-prohibitive to deliver up to 300 surveys across 30 unique CTEP sites 

with different data practices. 

The survey instrument contained a total of 25 questions for the adult learner and 9 questions for the 

CTEP member. The survey was created and administered on Google Forms. Google Forms was chosen 

for its responsiveness across many devices (mobile phones, desktops, printable with skip instructions), 

its flexibility in survey structure and logic, and cost effectiveness. 

Sample Selection. CTEP members work at 30 diverse site locations across the metro area, which serve a 

wide variety of different learner populations. To get a representative sample, CTEP leadership worked 

with each CTEP member to develop an individualized goal of administered surveys for each site. This 
was based on learner eligibility, attendance numbers and the CTEP’s ability to contact adult learners. 

CTEP leadership and Mr. Backman incorporated a tracking system for the survey in the existing master 

student record keeping database. This tracking system add data entry fields for CTEPs to track the date 

of call attempts and when the learner was eligible to be contacted. The survey resulted in 272 

completed, valid surveys from all 30 CTEP locations with an adequate distribution of surveys across the 

sites. 

Comparison Group Design. The Department of Employment and Economic Development (DEED) was 

contacted to assemble comparison employment data for participants who were served across 13 

Minneapolis-St. Paul metro-area WorkForce Centers. Per a request from CTEP, the Minnesota 

Department of Employment and Economic Development (DEED) has provided data on WorkForce 

Center (WFC) customers. To be included in this data, WFC customers must have entered and exited the 

WFC system between September 1, 2017 and June 30, 2018. They also must have used WFC services 

(attended workshops, met one-on-one with WFC staff, or used the Resource Room Computers) for at 

least 4 hours. The WFCs included are: Blaine, Bloomington, Brooklyn Park, Burnsville, Cottage Grove, 

Forest Lake, Minneapolis North, Minneapolis South, North St. Paul, Shakopee, St. Paul, West St. Paul, 

and Woodbury. The employment data included from the WorkForce Center utilizes Stata Administrative 

Tax data, and is pulled 

DEED provided two datasets; one restricted to unemployed jobseekers and the other included all 

WorkForce Center customers. Both datasets are restricted to only WFC Customers who provided valid 

SSNs. DEED verifies employment status and wage information using State Administrative Tax data, and 

pulls this data on a quarterly basis. This data can only be verified if the SSN that the customer provided 

is valid. Approximately 7% of the sample, or 192 out of a total 2,942 customers, did not have valid SSNs 

and these records were dropped from this analysis. This left 2,750 valid WFC customers. This will affect 

the sample statistics, but is not know in which direction the bias is present. 

In addition to an unduplicated count of job seekers served, CTEP asked DEED to provide the following 

information, with breakouts on gender, age, and race: 

 Percent of job seekers who were employed at the time of entry into the WFC system 

 Median hourly wage for job seekers who were employed at the time of entry 

 Percent of job seekers who were employed one quarter after exiting the WFC system 

 Median hourly wage for job seekers who were employed one quarter after exit 

VII. Methodology 

Research Questions 

1. From a representative sample of adult learners in CTEP programs, we sought to determine the 

employment rate of learners upon entering CTEP programs and the employment rate of learners 

after attending at least four hours of computer skills programming in a CTEP program. 

2. From a representative sample of adult learners in CTEP programs, we sought to determine the 

purpose of learners’ attendance in CTEP basic computer skills programs.
3. From a representative sample of adult learners in CTEP programs, we sought to determine the 

employers and job types of adult learners who have attended at least four hours of computer 

skills programming in a CTEP program. 

4. From the Department of Employment and Economic Development, we sought to determine how 

the CTEP employment result compares to a metro wide employment comparison group. 

5. From a representative sample of adult learners in the CTEP program, we sought to determine 

the Return on Investment for the value of all new jobs received. 

A full copy of the survey instrument is located in the Appendix. 

Adult learners responding to the questionnaire were screened for eligibility by a CTEP Member based on 

the criteria below. CTEP staff established these criteria to conform to the average intervention. 

In order to be eligible for survey, the adult learner: 

 Must have completed at least four hours of Northstar-related programming at the CTEP site 

 Must have completed at least one Northstar Assessment 

 Must have completed the Northstar Assessment at least four weeks prior to survey 

administration. 

 Must be older than the age of 16 at the start of class 

These parameters were set to ensure that the learner participated in a base amount of Northstarrelated programming and had been long enough to have had a chance to search for employment. Each 

CTEP member and CTEP site was given a specific target number of surveys to administer. The survey did 

not collect any learner names and all data collected remained anonymous and confidential. 

Of the 6111 adult learners participating in CTEP's programming in its 2018 fiscal year (August 2017 to 

July 2018), 1,183 adults passed Northstar Assessments. 272 participants (n = 272) passed the screening 

and were administered questionnaires. 

Between February 19 and July 15 (2018), CTEP members completed 272 surveys of adult learners who 

participated in computer skills programming at 30 unique site locations. In total, 285 surveys were 

completed, but 13 surveys were incomplete or inconsistent, and were not analyzed, This date range was 

chosen to correspond to the second half of the year of the CTEP service year, which allowed enough 

time for the CTEP AmeriCorps Member to have an adequate number of adult learners who have passed 

the Northstar Assessment in their program year. 

CTEP sites were not able to equally administer surveys because some sites such as the public libraries 

have privacy restrictions that do not make it easy to contact participants in the future. Also, some sites 

did not administer Northstar Assessments during the survey time period. Therefore, different members 

had different quotas of how many surveys they would administer. 
VIII. Comparison Group 

As described in the survey design section, CTEP obtained data from the Department of Employment and 

Economic Development' WorkForce Centers (WFCs) that serves as a comparison group. As this data is 

only available to the public in summary form, we are unable to research at a micro-level. However, we 

are able to closely match the metrics and categories that the DEED data demonstrates, and can make 

direct comparisons. 

Table 1: Unemployed Job Seekers in DEED and CTEP programs. Table 1 below contains DEED WFC 

Customer data that is restricted only to WRC customers who were verified unemployed at the time of 

entry into the WFC program. The CTEP survey respondents for this table are restricted to respondents 

who reported that they were unemployed when they were attending CTEP Northstar programs and 

reported that they were looking for employment. The CTEP sample of respondents include only those 

who were unemployed, searching for a job and found a job. This is a relatively small portion of the 

sample, so the focus should remain at the top level statistics of this group, such as total unemployed job 

seekers and breakdown by gender. All statitiscal breakdowns are included for provenance. 

DEED reported that among unemployed, job seeking WorkForce Center customers who received an 

intervention of similar time duration (at least four hours of assistance), 40% (351/878) found work 

within 90 days after exiting the WFC. Among CTEP survey respondents who were unemployed at the 

time of taking CTEP Northstar programs and were searching for work (29.4%, 80/272), over 47% (38/80) 

found employment at the time of the survey administration. The CTEP intervention with a 47% 

employment rate of comparable unemployed, job-seeking participants was over 7 percentage points 

higher than the DEED comparison group. The 7% difference in employment rates is greater than the 

calculated margin of error of 5.22% on this sample population. 

Between the sample of of unemployed job seekers in the DEED comparison group and the CTEP survey 

respondents, DEED customers earn roughly $2.10 more per hour, when comparing median wages of all 

survey respondents and WFC customers. DEED is not able to share specific wage data with the public, so 

it is not possible to fully analyze the reason for the wage discrepancy. In the CTEP survey, exact wage is 

not obtained from the respondents. The median wage is obtained by comparing their job title and 

employer with the median BLS Occupational Wage Estimate for that occupation. It is likely that this 

wage discrepancy is the result of this sample of only unemployed job seekers, which in the CTEP sample 

skews more female than the DEED data. Females in both the DEED and CTEP sample earn less than 

males. It is also reasonable to assume that DEED WFC customers have more resources and support 

through the State to nework and connect with employers; and are likely obtaining slightly higher skilled 

jobs. 
Table 1: Unemployed Job Seekers (*invalid SSNs dropped from DEED)

DEED WorkForce Centers

Breakout Group Breakout Category Unduplicated 

Count Served

Proportion of 

sample

Percent Employed 

at Entry

Median Hourly 

Wage at Entry

Percent Employed 1Q After 

Exit

Median Hourly Wage 1Q 

After Exit

Total Served Unemployed Job Seekers 878 100% - - - - 40% $17.81 

Female 415 47.2% - - - - 53% $17.62 

Male 407 46.3% - - - - 38% $18.22 

18-22 28 3.2% - - - - 20% $13.42 

23-30 80 9.1% - - - - 57% $15.05 

31-40 154 17.5% - - - - 47% $17.79 

41-50 214 24.4% - - - - 63% $18.78 

51 and older 402 45.7% - - - - 33% $18.23 

Person of Color 577 65.7% - - - - 40% $17.60 

White 227 25.8% - - - - 37% $18.06 

CTEP AmeriCorps Sites

Breakout Group Breakout Category Surveyed 

Participants

Proportion of 

sample

Percent Employed 

at Entry

Median Hourly 

Wage at Entry

Percent Employed Minimum 

4 Weeks After Exit

Median Hourly Wage 

Min. 4 Weeks After Exit

Total Served Unemployed Job Seekers 80 100% - - - - 48% $15.71

Female 51 63.8% - - - - 71% $13.10

Male 29 36.3% - - - - 29% $15.79

18-22 10 12.5% - - - - 45% $13.61

23-30 7 8.8% - - - - 62% $14.25

31-40 15 18.8% - - - - 63% $15.71

41-50 16 20.0% - - - - 59% $15.71

51 and older 10 12.5% - - - - 40% $13.47

Person of Color 27 33.8% - - - - 53% $13.10

White 45 56.3% - - - - 41% $15.71

Prefer not to say 8 10.0% - - 38% $16.72

ESL respondent 28 35% - - - - 50% $16.10

Non-ESL respondent 52 65% - - - - 46% $13.10

*Some respondents chose not to report their gender. 

*Language was asked to the CTEP member if the respondent was an ESL student. This question was not available from DEED. 

Gender

Age

Race

*Race is a binary created from a "choose-all-that-apply" survey question. Person of Color is any respondent that chose at least one category other 

Language*

Age*

Gender*

Race*

*Percent Employed at Entry records the employment status of participants in the 90 day quarter in which they started Workforce programming. This likely includes people 

who were employed but are recently no longer employed at the time of entering the workforce program. It is impossible to break this number down into more specifics. 

*Some respondents did not offer their age, so these respondents were not included in the Age Category breakdown. They are included in the Total Served and Gender 

category. 

Table 2: All Jobseekers in DEED and CTEP programs. Table 2 below shows that among all job seekers, 

the demographics of each sample differ in a few key ways. There are more females than males in the 

both samples, but the CTEP sample leans more female at 63% of participants. Regarding age, the DEED 

programs are older, with 72% of customers over the age of 41. Participants over the age 41 only make 

up 46% of the survey respondents. The ratio of persons of color and white persons is similar between 

the two programs, with a majority persons of color, 71% in DEED programs and 68% in CTEP programs. 

Thus, overall, the primary differences is that the DEED participants are older and more male than the 

survey respondents from CTEP programs. 

DEED employment data at entry is not a reliable measure of true employment due to the nature of the 

how DEED obtains their data. Because they only pull data quarterly, if the WFC customer becomes 

unemployed in the same quarter as they enter the WF center, they will be recorded as being employed. 

This is the reason that we have analyzed strictly verified unemployed jobseeker in Table 1. 
CTEP programs, across every age group, race and age category, has lower post-intervention employment 

rates across the entire samples. However, it is important to remember that among all CTEP survey 

respondents, both employed and unemployed, just under half reported that they were currently looking 

for work (49.6%, 135/272). All DEED WFC customers are characterized as job seeking. Of the 135 adult 

learners who reported that they were searching for work, nearly 63% (85/135) reported that they were 

employed at the time of survey. This is 4 percentage points higher than the 59% of DEED WFC 

Customers who were employed within the quarter that they exited the WFC program. Median wage is 

higher across all DEED customers as compared to CTEP survey respondents. The median wage among all 

employed Deed customers after exit is $19.36, over $4.50 higher per hour than the median CTEP 

employed survey respondent at $14.79. Wages are higher for DEED, in part, due to the demographics of 

being older and more male, both of which have significantly higher median wages in the DEED sample. It 

is also likely that the inherent make-up of their program participants is likely different than CTEP 

program participants. 

Table 2: All Jobseekers in CTEP and DEED programs(*invalid SSN records dropped from DEED)

Breakout Group Breakout Category Unduplicated 

Count Served

Proportion of 

sample

Percent Employed 

at Entry*

Median Hourly 

Wage at Entry

Percent Employed 1Q After 

Exit

Median Hourly Wage 1Q After 

Exit

Total Served All Job Seekers 2,750 100% 68% $23.54 59% $19.36 

Female 1,394 50.6% 70% $22.55 61% $19.03 

Male 1,182 43.0% 66% $25.82 58% $20.05 

18-22 58 2.0% 52% $12.30 57% $13.04 

23-30 246 8.9% 67% $17.19 68% $16.83 

31-40 482 17.5% 68% $20.10 65% $18.72 

41-50 694 25.2% 69% $23.60 63% $20.14 

51 and older 1,270 46.2% 68% $27.64 54% $20.57 

Person of Color 1,938 70.5% 70% $23.60 60% $19.36 

White 630 22.9% 64% $22.93 59% $19.43 

*Approximately 7% of all WFC customers enter SSNs on to their enrollment forms, and their employment data can not be tracked, so they are dropped from this sample. 

Breakout Group Breakout Category Surveyed 

Participants

Proportion of 

sample

Percent Employed 

at Entry

Median Hourly 

Wage at Entry

Percent Employed Minimum 4 

Weeks After Exit

Median Hourly Wage 

Minimum 4 Weeks After Exit

Total Served All Survey Respondents 272 100% 43% n/a 54% $14.79

Female 170 62.5% 42% n/a 54% $13.61

Male 101 37.1% 46% n/a 53% $15.79

18-22 38 13.9% 37% n/a 45% $13.10

23-30 39 14.3% 54% n/a 62% $15.99

31-40 56 20.5% 52% n/a 63% $13.25

41-50 54 25.2% 44% n/a 59% $15.71

51 and older 70 19.9% 36% n/a 41% $15.29

Person of Color 182 68.4% 67% n/a 54% $13.83

White 74 27.2% 27% n/a 53% $15.75

Prefer not to say 16 5.9% 37.5 n/a 56% $15.70

ESL respondent 135 49.6% 44% n/a 53% $13.61

Non-ESL respondent 137 50.3% 43% n/a 55% $15.84

Race*

*9 respondents did not offer their age, so these respondents were not included in the Age Category breakdown. They are included in the Total Served and Gender category. 

*Race is a binary created from a "choose-all-that-apply" survey question. Person of Color is any respondent that chose at least one category 

*Language was asked to the CTEP member if the respondent was an ESL student. This question was not available from DEED. 

CTEP AmeriCorps Sites

Gender*

Age*

Language*

*1 respondent chose not to report their gender. 

DEED WorkForce Centers

Gender

Age

Race

*Percent Employed at Entry records the employment status of participants in the 90 day quarter in which they started Workforce programming. 
IX. Evaluation Comparison: 2015 and 2018 

In 2015, the CTEP program first conducted a survey evaluation examining employment outcomes for 

adult learners who utilized digital literacy services and Northstar Assessments at CTEP host sites. This 

evaluation built off of the 2015 evaluation, but intentionally kept a core of survey questions the same or 

very similar to allow for direct comparisons across the two surveys. While there are key differences in 

the results of the survey, it is notable that the results are consistently similar. This indicates a robust, 

well designed survey with high validity. In 2018, the sample size increased by 64 survey respondents, 

while the total pool of eligible adult learners shrank by over 400 persons. This results in a more 

statistically significant survey with a lower margin of error. The key difference between the two surveys 

is the unemployment rate. The unemployment rate of adult learners at the time of attendance was 5.9% 

lower in 2018 than in 2015. The unemployment rate of adult learners at the time of survey 

administration was 7% lower in 2018 than in 2015. This reflects an improved labor market, but also may 

be indicative of improving digital literacy programs and more relevant job assistance on behalf of CTEP 

members. Please refer to the figure below for data on some of the most important core survey 

questions in each evaluation. Overall, in both evaluations, the CTEP programs had a higher percentage 

of jobseeking program participants move from unemployment to employment. Median hourly wages 

decreased $0.29 between 2015 and 2018, but mean hourly wages have increased by $0.41. Overall, 

these small differences reflect that wages held steady with little growth. 

See Table 3 on next page. 
Table 3: Comparison of the 2015 & 2018 CTEP Evaluations

2015 2018

Number of Survey Respondents 208 272

Number of CTEP host sites surveyed 2 8 3 0

Total pool of eligible adult learners 1611 1183

Margin of error 6.34% 5.22%

Total employed at entry into CTEP programs

respondents 127/205 154/272

% 38% 44%

Total employed at date of CTEP survey administration

respondents 108/205 147/272

% 47% 54%

Total searching for work while attending CTEP programs

respondents 111/208 135/272

% 53% 50%

Unemployed at entry and searching for work while attending CTEP programs

respondents 68/127 80/154

% 54% 51%

Employment rate of those unemployed at entry into CTEP programs, searching for work & employed at survey date

respondents 34/68 38/80

% 50.0% 47.5%

DEED Comparison Group: Employment rate of unemployed jobseekers at entry & Employed 1Q after exit from 

WorkForce Center program 

% 41% 40%

Difference in unemployment rate between CTEP survey respondents and DEED customers, post-exit from program

% 9 % 7.5%

Median wage, all employed CTEP survey respondents at time of survey administration

$ 15.08 14.79

Mean wage, all employed CTEP survey respondents at time of survey administration

$ 15.62 16.03

CTEP program satisfaction

Unemployed at entry, searching for work while attending CTEP programs, employed at survey date & reported that 

class helped in job search

respondents 24/34 28/38

% 71% 74%

Among all job seekers, reported CTEP programs helpful in job search

respondents 89/111 110/135

% 80% 81%

X. Analysis: Demographics of Adult Learners 

Survey respondents were asked to identify their race and ethnicity in one “choose-all-that-apply” 

question. Overall, 38.2% (104/272) respondents reported as “Black or African-American”, 27.2% 

(74/272) reported as “White”, 16.5% (45/272) reported as “Asian”, and 6.6% (18/272) reported as 

“Hispanic or Latino.” 15 respondents reported multiple races or ethnicities. 9 respondents reported 

“None of the above” and 7 reported that they “Prefer not to say.” 
For analysis purposes, a binary variable was created to delineate White and People of Color, which 

includes all respondents who reported Asian, Black, Hispanic/Latino, Native American/Native Alaskan or 

a combination including at least one of these. For the race binary variable, 27.2% of the sample was 

White and 66.9% of the sample were People of Color. 5.88% of the sample was other (prefer not to say, 

none of the above, etc.). Learners of color and White learners reported similar employment rates at the 

time of survey administration, but White learners made a slightly higher median wage ($14.86 versus 

$13.84). 

The age categories of survey respondents mirror the State of Minnesota’s DEED categories. The 

youngest is 17 and the oldest is 80 years old. The median age is 39 years old, and the average age is 40 

years old. For the sake of analysis in comparison to the DEED data, only participants age 18 or older are 

analyzed. The age categories are 18-22, 23-30, 31-40, 41-50 and 51+. Predictably, the highest 

employment rate was among those aged 23-30 (62% employed) and 31-40 (63% employed) years old. 

This survey also reported whether the respondent was an English as a Second-Language learner. 50.37% 

were reported as ESL learners, and 49.63% were not ESL learners. ESL and Non-ESL learners had similar 

levels on employment, but suffered lower median wages, $13.61 for ESL learners versus $15.71 for nonESL learners. 

XI. Analysis: Adult Learners, Northstar Assessments and Computer Skills 

Programming 

The median amount of Northstar-related programming attended by survey respondents was 19.32 

hours, though the median amount of hours attended was 23 hours. The highest number of hours that a 

survey respondent reported attending Northstar-related programming was 396 hours, and the most 

common reported amount of time spent attending programming was 10 hours. The most common 

assessment passed by survey respondents was Basic Computer Skills (57.7%), followed by World Wide 

Web (35.1%), MS Word (29.4%) and MS Excel (25.8%). 

The majority (66.8%) of all survey respondents did receive certification for passing a Northstar 

Assessment in the form of a printed certificate. The Mozilla badges were only given to eight survey 

respondents, indicating a continued lack of interest in adult learners, a lack of communication or 

marketing on part of the CTEPs, CTEP coordinators or the Northstar team or a system that has not yet 

been fully implemented. Only 18.1% of survey respondents opted not to receive any certification, 

compared to 34% in the last evaluation. 

The top three reasons for attending Northstar-related programming are improving general skills using a 

computer (73.6%), to improve skills to obtain a better or new job (58.1%) and for help with job search 

(48.8%). 

XII. Analysis: Employment 

Of those employed at the time of survey, the average length of employment in their current job was just 

under two years, or 22.44 months. Similarly, the median length of employment was 22 month. The 
modal length of employment, or the most frequent length, two months. Overall, this indicates that 

Northstar-related computer skills programming serves a wide range of adult learners in relation to 

employment status, with many adult learners looking for work or in new jobs but also long-term 

employees looking to learn or refresh their computer skills. 

Of those who reported employment at the time of survey administration, 97% survey respondents 

provided their employer name and position title. Employer name and position title was matched to 

Occupational Wage Estimates created by the Bureau of Labor Statistics (BLS, May 2017) for the 

Minneapolis-Saint Paul-Bloomington metropolitan area. Each survey respondent’s employment position 

and employer were matched with the corresponding BLS job sector and job category, as accurately as 

possible. This data is available in an attached spreadsheet to this report. Each job category had mean 

and median salary data, as well as a defined job sector. 

Overall, the average wage for all combined survey respondents is $16.02 per hour, with a median wage 

of $14.79 per hour. The average wage annualized results in a mean average salary of $30,763.20. Over 

79% of survey respondents reported that they were at 7 or above on a job satisfaction scale from 1 

through 10. About 60% of survey respondents reported that they were at 7 or above on a wage 

satisfaction scale from 1 through 10. The most common job sectors for survey respondents are Personal 

Care and Service Occupations (e.g. personal care aide, nursing assistants), Healthcare Support 

Occupations, Office and Administrative Support Occupations (e.g. receptionist, administrative assistant), 

Sales and Related Occupations (e.g. retail salesperson, cashiers), and Production Occupations. 

Table 4: Occupation and Wages # of survey respondents Proportion Median Wage

 Personal Care and Service Occupations 19 13.4% $ 11.85

 Healthcare Support Occupations 17 12.0% $ 16.40

 Office and Administrative Support Occupations 16 11.3% $ 13.10

 Sales and Related Occupations 15 10.6% $ 11.51

 Production Occupations 15 10.6% $ 15.71

 Food Preparation and Serving Related Occupations 11 7.7% $ 12.10

 Building and Grounds Cleaning and Maintenance Occupations 10 7.0% $ 12.10

 Transportation and Material Moving Occupations 10 7.0% $ 15.27

 Community and Social Service Occupations 9 6.3% $ 16.72

 Education, Training, and Library Occupations 4 2.8% $ 16.60

 Management Occupations 3 2.1% $ 28.54

 Construction and Extraction Occupations 3 2.1% $ 26.64

 Arts, Design, Entertainment, Sports, and Media Occupations 2 1.4% $ 17.63

 Healthcare Practitioners and Technical Occupations 2 1.4% $ 20.14

 Protective Service Occupations 2 1.4% $ 16.00

 Business and Financial Operations Occupations 1 0.7% $ 32.22

 Computer and Mathematical Occupations 1 0.7% $ 27.96

 Legal Occupations 1 0.7% $ 26.48

 Installation, Maintenance, and Repair Occupations 1 0.7% $ 18.02

 Architecture and Engineering Occupations 0 0.0% $ -

 Life, Physical, and Social Science Occupations 0 0.0% $ -

 Farming, Fishing, and Forestry Occupations 0 0.0% $ -

Total: 142

https://www.bls.gov/oes/2017/may/oes_33460.htm#31-0000
XIII. Interviews with Program Leads at Adult Learning Centers 

Three managers and/or coordinators of Adult Learning Centers were interviewed separately over 

telephone to gather contextual information regarding the changing economic landscape that adult 

learners are currently facing. The three Adult Learning Centers hosted CTEP AmeriCorps members who 

participated in this survey research and were among the top centers for total number of surveys 

completed. The centers will remain anonymous, but they were chosen to give represent both urban and 

suburban environments and a wide range of services. The interviews lasted roughly 15 minutes each. 

Demographics of learners. The demographic profile of learner attendance did not change uniformly 

across the three programs. One program lamented a sharp decrease in the number of learners with 

refugee status, which the program manager attributed to a change in federal policy. Conversely, the 

other two programs noticed an uptick in foreign born, refugee and ESL learners over the past few years. 

Employment outcomes of learners: Across all three centers, there was consensus that more of their 

students have a job when they first start attending class and are more likely to leave their programs 

when they get a better or new job, which is happening much more often now than in the past three 

years. There was a general consensus that the learners are getting jobs faster and easier than they were 

in the past few years. Two program managers expressed their surprise at how many low-level ESL 

learners were getting jobs, and they both said that low-level ESL learners had the hardest time finding 

employment in the past. All three program managers said that they do still get students who are 

employed at the start of class but are attending in hopes of gaining skills to improve their job prospects. 

Learning level of learners: Across all three centers, there was consensus that the average level of newly 

enrolled students is lower than in years past. One center reported that they have just begun to offer a 

very low-level ESL class (pre-literate) and it is very popular. All three centers also mentioned that due to 

the GED change in 2015 (computerized, new standards), they have seen a fairly large drop-off in 

learners seeking GED classes. One program manager attributed this to the economy and said that they 

believe a GED isn’t as necessary to obtain employment now as it used to be. 

Attendance patterns: Unanimously, all three centers reported that their attendance numbers have been 

down over the past three years, with one program citing that their learning center is down 12% in total 

student hours from the previous year. Not only are total student hours down across each program, 

average hours per student are also down. One program cut all classes on Fridays due to low attendance. 

Programs that rely on MFIP recipients have seen a decrease, because the 20 hour work/class 

requirement can be satisfied by work, so there is less of a need to fill in the time with class hours. One 

program reported that their Job Readiness class attendance has tanked, and they believe that employers 

aren’t as concerned over job readiness concerns. However, each program stated that demand for 

certain types of classes has increased, and that they have changed their class topics and class schedules 

accordingly. One program has ramped up more high-skill professional work training, like nursing 

assistant programs to meet steady demand. Another program has increased their digital literacy class 

offerings, as they are growing in popularity. 

## End of Analysis ## 
XIV. Acknowledgements 

Kaitlyn Schmaltz, a CTEP member in the 2017-2018 cohort, assisted the Saint Paul Neighborhood 

Network in the preparation and execution of this report. Kaitlyn worked with CTEP members to ensure 

that they were meeting their survey goals, communicated CTEP member concerns and feedback to 

SPNN leadership and the evaluator, and helped organize CTEP member meetings for survey trainings. 

XV. Appendix

1. Table 1: Unemployed Job Seekers (*invalid SSNs dropped from DEED) 

2. Table 2: All Jobseekers in CTEP and DEED programs (*invalid SSN records dropped from DEED) 

3. Table 3: Comparison of the 2015 & 2018 CTEP Evaluations 

4. Table 4: Occupation and Wages 

5. Table 5: Coding of Occupation and Wages 

6. DEED WorkForce Center Data Request Report 

7. 2018 CTEP Northstar Assessment and Employment Survey (full copy) 

8. 2015 Evaluation Report: Impact of Northstar Assessment and Related Computer Skills Programming 

on Employment in CTEP Programs