Learning a Low-Dimensional Representation of Job 

History for Economic Prediction 

Keyon Vafa

Columbia University
Collaborators:

Emil Palikot

Stanford 

University

Tianyu Du

Stanford 

University

Ayush Kanodia

Stanford 

University

David Blei

Columbia 

University

Susan Athey

Stanford 

University
Longitudinal Survey Datasets

1990

Survey questions include:

- What is your occupation?

- What is your most recent educational degree?

- What is your wage?

x thousands of respondents

SURVEY SURVEY

1991

SURVEY

SURVEY

SURVEY SURVEY

SURVEY

SURVEY

SURVEY SURVEY

SURVEY

...

2022

...

SURVEY

SURVEY SURVEY

SURVEY
Longitudinal Survey Datasets

Longitudinal surveys are constructed to be nationally representative.

Survey datasets in the United States: NLSY and PSID
Applications using Longitudinal Survey Datasets
Job Sequences

Over time, these datasets produce job sequences of individual workers.

Occupation modeling: Given an individual's job history, what is the distribution over 

their next job?

Useful for unemployment analyses (Hall, 1972), measuring occupational 

mobility (Kambourov and Manovskii, 2008), etc.

1990

Cashier

1991

Sales 

Representative

1992

Sales Manager ...

SURVE

YSURVE

YSURVE

Y

SURVEY

SURVE

YSURVE

YSURVE

Y

SURVEY

SURVE

YSURVE

YSURVE

Y

SURVEY
Predicting Future Jobs

In practice, economists fit simple linear models that are either Markov or depend 

on history via hand-constructed summary statistics.

More accurate job predictions => more accurate economic analyses

Major challenge of building predictive models from survey datasets: they are small 

(containing only thousands of workers).

Challenge 1: Are there better models that can capture complex career trajectories?
Resume Datasets

Recently: large resume datasets have become available.

RESUME

2015-2016: 

Data Scientist

2017: Software 

Engineer

These contain many job sequences:

x millions

2015

Data Scientist

2016

Data Scientist

2017

Software 

Engineer

...
Suitability for Job Sequence Models

RESUME

SURVEY

Pros: Very large datasets

Cons: Not representative, may be inaccurate (imputation errors and 

false information)

Pros: Representative of public and carefully curated

Cons: Small datasets

Challenge 2: Is there any way to leverage abundance of resumes for survey 

dataset analyses?
Challenges for Modeling Job Sequences

Challenge 2:Can we take advantage of job sequence data from resumes when 

analyzing survey datasets?

Challenge 1: Are there better models (beyond Markov and linear) that can 

capture complex long-term career trajectories?
Transformer neural networkshave successfully modeled sequences of words 

in natural language processing (NLP).

We develop a transformer for modeling sequences of jobs in a career.

The dog was barking and ???

Transformer 

predicts:

Tutor Paralegal Judicial Clerk Judicial Clerk Lawyer ???

Transformer 

2010 2011 2012 2013 2014 predicts:

Degree: BA Degree: BA Degree: JD Degree: JD Degree: JD

Challenge 1: Modeling Complex Careers

Modeling sequences of jobs in a career is not so different from modeling 

sequences of words in a sentence.
Conditioning on Full History

Common occupation models are Markov; ideally a model would condition 

on full job history to predict future jobs.

2015

Data Scientist

2016

Data Scientist

2017

Software 

Engineer ???

Predicting 

2018:

First-order Markov

Degree: BS Degree: BS Degree: MS

Challenge: Infeasible to condition on full history directly (exponentially many 

possible combinations and permutations).

Fully-contextual model
Transformers are Based on Representations

Idea: Learn low-dimensional representation of job history.

???

Predicting 

2018: 2015

Data Scientist

2016

Data Scientist

2017

Software 

Engineer

Degree: BS Degree: BS Degree: MS

3.4, -1.1, -3.5, 2.4, ..., 0.1

Career 

history

3.4, -1.1, -3.5, 2.4, ..., 0.1

h(previous jobs, covariates)

Summarize previous jobs and covariates with a low-dimensional representation 

that carries all relevant information for predicting the next job.
Representations Encode Career Similarity

Intuition: If two individuals have similar but non-identical career paths, their 

predicted next jobs should still be similar.

Predicting jobs based on representations allows for sharing information.

Similarities of career histories based on learned representations:
Challenges for Modeling Job Sequences

Challenge 2: Can we take advantage of job sequence data from resumes when 

analyzing survey datasets?

Challenge 1: Are there better models (beyond Markov and linear) that can capture 

complex long-term career trajectories?
Transfer Learning

Modern methods in natural language processing (NLP) rely on transfer learning.

These methods first train a model on large, unlabelled text corpora (e.g. Google 

books, Wikipedia) before adjusting these models on tasks of interest involving 

small amounts of labelled data (e.g. sentiment analysis). 

Intuition: models learn about underlying grammar/vocabulary on large corpora 

before specializing on a particular task.
function

Transfer learning:Learning representations of careers from resumes that can be 

adjusted (or fine-tuned) on survey datasets. 

Step 1: Learn representations of careers from resumes:

Input:

Millions of resumes

Output:

Representation 

function of resumes

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

RESUME

representation of career

3.4, -1.1, -3.5, 2.4, ..., 0.1

RESUM

E

job sequence 

from resume

Challenge 2: Leveraging Resumes for Surveys
Step 2: Fine-tune representation function to predict on survey datasets.

Input:

Survey dataset and resume 

representation function

Output:

Fine-tuned 

representation and 

prediction function

SURVEY SURVEY SURVEY

SURVEY

SURVEY

SURVEY SURVEY

SURVEY

representation of career

3.1, -2.0, -3.5, ..., 1.6

fine-tuned 

function

SURVEY

job sequence 

from survey next job prediction

Prediction: Therapist 

Probability: 0.27

Transfer learning:Learning representations of careers from resumes that can be 

adjusted (or fine-tuned) on survey datasets. 

Challenge 2: Leveraging Resumes for Surveys

+

pretrained function
Transfer Learning Motivation

Initializing with pretrained representations ensures that the model does not need to 

re-learn representations from small survey datasets.

Instead, it only needs to fine-tune representations to account for dataset 

differences.

(short for Contextual Attention-based Representations of Employment Encoded 

from Resumes)

We call our approach CAREER.
Dataset Comparison

Resumes NLSY79 PSID

survey datasets

Number of individuals 24 million 12 thousand 12 thousand

NLSY97

9 thousand

Unemployed/

out-of-labor-force/

student available?

No Yes Yes Yes

Median year 2007 1991 2007 2011

Proportion manual 

laborers

7% 17% 13% 12%

Covariates available year, location, 

education

year, location, 

education, gender, 

race/ethnicity

year, location, 

education, gender, 

race/ethnicity

year, location, 

education, gender, 

race/ethnicity
Prediction Performance

We pretrain CAREERon resumes and fine-tune on survey datasets.

CAREERoutperforms baselines at predicting jobs on survey data.

SURVEY SURVEY SURVEY SURVEYSURVEY SURVEY

SURVEY SURVEY
Scaling Law

How does CAREER's predictive performance on survey datasets vary as a 

function of pretraining data size (number of resumes)?

Relationship follows power law, similar to scaling laws found in NLP.
Example Prediction

Held-out sequence:

Rank of true next job (biological technician) of possible next jobs:

Regression: 41st

Bag-of-jobs: 38th

CAREER:2nd
Future Work: CAREER for Economic Adjustment

For example: the adjusted gender wage gap involves predicting wage from 

covariates and summary statistics about experience (e.g. years worked). 

CAREERlearns a low-dimensional representation of job history that can be 

plugged into these models. Wage prediction MSE:

Many analyses on survey datasets don't involve predicting next jobs directly, 

but rather estimating a quantity that adjusts for history.

These models typically estimate adjusted quantities by building outcome 

models that adjust for history. 
Summary

Many economic analyses rely on building predictive models on job sequences 

from survey datasets.

While modern machine learning methods may struggle on small survey 

datasets, recent years have seen the emergence of large-scale resume data.

Inspired by approaches in NLP, CAREERleverages resume data to learn useful 

representations of job sequences for downstream survey predictions.

Future work: Using CAREER to estimate adjusted economic quantities.

Link to code and paper: 

https://github.com/keyonvafa/career-code
Thank you!