CAREER: Transfer Learning for Economic Prediction of Labor Data
Authors: Keyon Vafa
Emil Palikot
Tianyu Du
Ayush Kanodia
Susan Athey
David Blei
Venue: Submitted to ICLR 2023 (2023)
Keywords: economics
transfer learning

Abstract: Labor economists regularly analyze employment data by fitting predictive models to small, carefully constructed longitudinal survey datasets. Although modern machine learning methods offer promise for such problems, these survey datasets are too small to take advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take advantage of their scale or incorporate them into the analysis of survey data. To this end we develop CAREER, a transformer-based model that uses transfer learning to learn representations of job sequences. CAREER is first fit to large, passively-collected resume data and then fine-tuned to smaller, better-curated datasets for economic inferences.  We fit CAREER to a dataset of 24 million job sequences from resumes, and fine-tune its representations on longitudinal survey datasets. We find that CAREER forms accurate predictions of job sequences, achieving state-of-the-art predictive performance on three widely-used economics datasets.  We further find that CAREER can be used to form good predictions of other downstream variables; incorporating CAREER into a wage model provides better predictions than the econometric models currently in use.

Under review as a conference paper at ICLR 2023
CAREER: TRANSFER LEARNING FOR ECONOMIC PREDICTION OF LABOR DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Labor economists regularly analyze employment data by fitting predictive mod- els to small, carefully constructed longitudinal survey datasets. Although modern machine learning methods offer promise for such problems, these survey datasets are too small to take advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take ad- vantage of their scale or incorporate them into the analysis of survey data. To this end we develop CAREER, transformer-based model that uses transfer learning learn representations of job sequences. CAREER is first fit to large, passively- collected resume data and then fine-tuned to smaller, better-curated datasets for economic inferences. We fit CAREER to a dataset of 24 million job sequences from resumes, and fine-tune its representations We find that CAREER forms accurate predictions of job sequences, achieving state-of-the-ard predictive performance We further find that CAREER can be used to form good predictions of other down- stream variables; incorporating CAREER into wage model provides better pre- dictions than the econometric models currently in use.
longitudinal survey datasets.
three widely-used economics datasets.
INTRODUCTION
variety of economic analyses rely on models for predicting an individual's future occupations. These models are crucial for estimating important economic quantities, such as gender or racial differences in unemployment (Hall, 1972; Fairlie Sundstrom, 1999); they underpin causal anal- yses and decompositions that rely simulating counterfactual occupations for individuals (Brown et al., 1980; Schubert et al., 2021); and they inform policy, by forecasting occupations with rising or declining market shares.
These analyses typically involve fitting predictive models to longitudinal surveys that follow a cohort of individuals during their working career (Panel Study of Income Dynamics, 2021: Bureau of Labor Statistics, 2019a). Such surveys have been carefully collected to represent national demographics, ensuring that the economic analyses can generalize to larger populations. But these datasets also small, usually containing only thousands of workers, because maintaining them requires regularly interviewing each individual. Consequently, economists use simple sequential models, where worker's 's next occupation depends on their history only through the most recent occupation (Hall, 1972) or a few summary statisties about the past (Blau & Riphahn, 1999).
In recent years, however, much larger datasets of online resumes have also become available. In contrast to longitudinal surveys, these passively-collecae datasets are not typically used directly for economic inferences because they contain noisy observations and they are missing important economic variables such as demographics and wage. However, they provide occupation sequences of millions of individuals, potentially expanding the scope of insights that can be obtained from analyses on downstream survey datasets. The simple econometric models currently in use cannot incorporate the complex patterns embedded in these larger datasets into the analysis of survey data.
To this end, we develop CAREER, a neural sequence model of occupation trajectories. CAREER is designed to be pretrained large-scale resume data and then fine-tuned small and better- curated survey data for economic prediction. Its architecture is based on the transformer language model (Vaswani et al., 2017), for which pretraining and fine-tuning has proven to be effective

---

Under review as a conference paper at ICLR 2023
paradigm for many NLP tasks (Devlin al., 2019; Lewis et al., 2019). CAREER extends this transformer- based transfer learning approach to modeling sequences of occupations, rather than text. We will show that CAREER's representations provide effective predictions of occupations survey datasets used for economic analysis, and can be used as inputs to economic models for other downstream applications.
To study this model empirically, we pretrain CAREER on a dataset of 24 million resumes provided by Zippia, a career planning company. We then fine-tune CAREER'S representations of job quences to make predictions on three widely-used economic datasets: the National Longitudinal Survey of Youth 1979 (NLSY79), another cohort from the same survey (NLSY97), and the Panel Study of Income Dynamics (PSID). In contrast to resume data, these well-curated datasets are rep- resentative of the larger population. with these survey datasets that economists make inferences, ensuring their analyses generalize.
In this study, we find that CAREER outperforms standard econometric models for predicting and forecasting occupations, achieving state-of-the-art performance on the three widely-used survey datasets. We further find that CAREER can used to form good predictions of other downstream variables; incorporating CAREER into wage model provides better predictions than the econo- metric models currently in use. We release code so that practitioners can train CAREER on their own datasets.
In summary, we demonstrate that CAREER can leverage large-scale resume data to make accurate predictions important datasets from economics. Thus CAREER ties together economic models for understanding career trajectories with transformer- based methods for transfer learning. (See Section for details of related work.) A flexible predictive model like CAREER expands the scope of analyses that can be performed by economists and policy-makers.
CAREER
Given an individual's career history, what is the probability distribution of their occupation in the next timestep? We go over class of models for predicting occupations before introducing CA- REER, one such model based on transformers and transfer learning.
2.1 OCCUPATION MODELS
Consider an individual worker. This person's career can be defined as a series of timesteps. Here, timestep of year. At each timestep, this individual works in job: it could be the same job as the previous timestep, or a different job. (Note we use the terms "occupation" and "job" synonymously.) We consider "unemployed" and "aut-of-llaorr-oroee" to be special types of jobs.
Define an occupation model to be probability distribution over sequences of jobs. An occupation model predicts a worker's job each timestep function of all previous jobs and other observed characteristics of the worker.
More formally, define an individual's career to be sequence (y1,- (y1..-.,yr) where each ye (1. indexes one of J occupations at time t. Occupations are categorical; one example of sequence could be ("cashier". "salesperson",
each timestep, an individ- ual is also associated with C observed covariates x, = { Covariates are also categorical, with Pte fll.... .Ne. For example, if corresponds to the most recent educational degree, Tte could be "high school diploma" or "bachelors". and N, is the number of types of educational de- grees.' Define y: = (y1,...,,,, We) to index jobs that have occurred up to time with the analogous definition for x:-
"sales manager").
At each timestep, an occupation model predicts an individual's job in the next timestep, p(y.ly=-1,x;;) This distribution conditions on covariates from the same timestep because these are "pre-transition." For example, an individual's 's most recent educational degree is available to the model as it predicts their nextjob.
Some covariates may not evolve over time. We encode them time-varying without loss of generality.
2

---

Under review as a conference paper at ICLR 2023
Note that occupation model is predictive rather than structural model. The model does not incorporate unobserved characteristics, like skill, when making predictions. Instead, it implicitly marginalizes over these unobserved variables, incorporating them into its predictive distribution.
2.2 REPRESENTATION -BASED TWO-STAGE MODELS
An occupation model's predictions are governed by an individual's career history; both whether an individual changes jobs and the specific job they may transition depend on current and previous jobs and covariates.
We consider class of occupation models that make predictions by conditioning on low- dimensional representation of work history, hy xx) E RD. This representation is assumed to be sufficient statistic of the past; Xx) should contain the relevant observed information for predicting the next job.
Since individuals frequently stay in the same job between timesteps, we propose a class of models that make predictions in two stages. These models first predict whether an individual changes jobs, after which they predict the specific job to which an individual transitions. The representation used in both stages.
In the first stage, the career representation (y xx) used to predict whether an individual changes jobs. Define the binary variable 8, to be if worker's job at time / is different from that at time t 1, and otherwise. The first stage is a logistic regression,
s, X ~ Bernoulli (a(1) hy(y:-1, x,))),
where (-) is the logistic function and r7 e R a vector of coefficients.
If the model predicts that an individual will transition jobs, it only considers jobs that are different from the individual's most recent job. To formulate this prediction, it combines the career represen- tation with a vector of occupation eecupation-specific specific coefficients B3- E R::
exp(3, (y-,,,1,,1)) x:))
ply = jly:-1, X1, 31 =
(2)
Ejen exp( he x1)=
Otherwise, the next job is deterministic:
ply: jly:-1, 0) -
(3)
Two-stage prediction improves the accuracy of occupation models. Moreover, many analyses of occupational mobility focus on whether workers transition jobs rather than the specific job they transition to (Kambourov & Manovskii, 2008). By separating the mechanism by which worker either keeps changes jobs (1)) and the specific job they may transition to (By), two-stage models are more interpretable for studying occupational change.
Equations 1 to 3 define two-stage representation-basee occupation model. In the next section, introduce CAREER, one such model based on transformers.
2.3 CAREER MODEL.
We develop a two-stage representation- based occupation model called CAREER." 2 This model uses a transformer to parameterize a representation of an individual's history. This representation is pretrained on a large resumes dataset and fine-tuned make predictions on small survey datasets.
Transformers. A transformer is sequence model that uses neural networks to learn representa- tions of discrete tokens (Vaswani al., 2017). Transformers were originally developed for natural language processing (NLP), to predict words in sentence. Transformers are able to model complex dependencies between words, and they are critical component of modern NI.P systems including language modeling (Radford et al., 2019) and machine translation (Ott et al., 2018).
CAREER is an occupation model that uses transformer to parameterize low-dimensional repre- sentation of careers. While transformers were developed to model sequences of words, CAREER
?CAREER is short for "Contextual Attention-hased Representations of Employment Eneoded from Re- sumes."
3

---

Under review as a conference paper at ICLR 2023
Transformer
Layers
1 (1)
71
(11)
h1
(2)
Ky
81
Career History
Tt-1
y1
(1)
(1)
1
(C)
61
Current Job and Covariates
Wt
V1-1
(1)
(2)
(1)) hr
St
Next
Vl
Two-Stage Prediction
Figure 1: CAREER's computation graph. CAREER parameterizes a low www-dimessionll representa- tion of an individual's 's career history with transformer, which uses to predict the next job.
uses transformer to model sequences of jobs. The transformer enables the model to represent complex career trajectories.
CAREER is similar to the transformers used in NLP, but with two modifications. First, as described in Section 2.2, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps. (In contrast, words seldom repeat.) Second, while language models only condition on previous words, each career is also associated with covariates that may affect transition distributions (see Equation 2). We adapt the transformer to these two changes.
Parameterization. CAREER's computation graph depicted in Figure 1. Note that in this section we provide simplified description of the ideas underlying the transformer. Appendix E contains full description of the model.
CAREER iteratively builds representation of career history, he(y)-1,xx,) E R using stack of layers. Each layer applies series of computations the previous layer's output to produce its own layer-specific representation. The first layer's representation, n1 (y:-1,xx) considers only the most recent job and covariates. At each subsequent layer É, the transformer forms representa- tion h, (y:-1, by combining the representation of the most recent job with those preceding jobs. Representations become increasingly complex each layer, and the final layer's representa- tion, 1/ (y.-1. is used to make predictions following Equations to 3. We drop the explicit dependence on y.-1 and going forward, and instead denote each layer's 's representation as ((())
The first layer's representation combines the previous job, the most recent covariates, and the posi- tion of the job in the career. It first embeds each of these variables in D-dimensional space. Define an embedding function for occupations, Cy : [J] - R Additionally, define a separate embedding function for each covariate, {ec]c_1 with each e, [Ne] + IRP. Finally, define e [7] - RR to embed the position of the sequence, where T denotes the number of possible sequence lengths. The first-layer representation (())) sums these embeddings:
11(2) (y(()-1) + Ee ec(()e) et)))
4

---

Under review as a conference paper at ICLR 2023
For each subsequent layer é, the transformer combines representations of the most recent job with those of the preceding jobs and passes them through a neural network:
Tie & exp {
(6) T
wion
(0)
for all l' S
(5)
11 Ei=
(6)
he ((+1) FFN ( (b(²
where W( E PxD is a model parameter and (FN is a two-layer feedforward neural network specific to layer l, with FFN(O), R + RRD.
The weights ( ( are referred to attention weights, and they are determined by the career repre- sentations and W The attention weights are non-negative and normalized to sum to 1. The matrix W(() can be interpreted
similarity matrix; if ((()) is the identity matrix, occupations and t' that have similar representations will have large attention weights, and thus t' would contribute more to the weighted average in Equation 6. Conversely, if W is the negative identity matrix, occupations that have differing representations will have large attention weights. 3 The final computation of each layer involves passing the intermediate representation (,(0) through a neural network, which ensures that representations capture complex nonlinear interactions.
The computations in Equations 5 to 7 are repeated for each of the L layers. The last layer's repre- sentation is used to predict the next job:
1 x+) = two-stage-softmax (n (1) ; :1.3).
where "two-stage-softmax" refers to the operation in Equations 3, parameterized by nj and B. All of CAREER's parameters including the embedding functions, similarity matrices, feed- forward neural networks, and regression coefficients and B - are estimated by maximizing the likelihood in Equation 8 with stochastic gradient descent (SGD), marginalizing out the variable
Transfer learning. Economists apply occupation models to survey datasets that have been carefully collected to represent national demographics. In the United States, these datasets contain small number of individuals. While transformers have been successfully applied to large NLP datasets, they are prone overfitting small datasets (Kaplan al., 2020; Dosovitskiy al., 2021; Variš & Bojar, 2021). As such, CAREER may not learn useful representations solely from small survey datasets.
In recent years, however, much larger datasets of online resumes have also become available. Al- though these passively-collected datasets provide job sequences of many more individuals, they not used for economic estimation for a few reasons. The occupation sequences from resumes are imputed from short textual descriptions, a process that inevitably introduces more noise and errors than collecting data from detailed questionnaires. Additionally, individuals may not accurately list their work experiences on resumes (Wexler, 2006), and important economic variables relating to demographics and wage are not available. Finally, these datasets are not constructed to ensure that they are representative of the general population.
Between these two types of data is tension. On the one hand, resume data is large-scale and contains valuable information about employment patterns. On the other hand, survey datasets are carefully collected, designed to help make economic inferences that robust and generalizable.
Thus CAREER incorporates the patterns embedded in large-scale resume data into the analysis of survey datasets. It does this through transfer learning: CAREER is first pretrained on a large dataset of resumes to learn an initial representation careers. When CAREER is then to small survey dataset, parameters learned from resumes. After initialization, all parameters are fine-tuned on the small dataset by optimizing the likelihood. Because the objective function is non-convex, learned representations depend on their initial values. Initializing with the pretrained representations ensures that the model
not initialized randomly; instead, they initialized with the representations
II practice, transformers use multiple attention weights perform multi-heades attention (Appendix F).
5

---

Under review as a conference paper at ICLR 2023
does not need to re-learn representations the small dataset. Instead, it only adjusts representations to account for dataset differences.
This transfer learning approach takes inspiration from similar methods in NLP, such BERT and the GPT family of models (Devlin et al., 2019; Radford et al., 2018). These methods pretrain trans- formers large corpora, such as unpublished books Wikipedia, and fine-tune them to make predictions on small datasets such as movie reviews. Our approach is analogous. Although the re- sumes dataset may not be representative or carefully curated, it contains many more job sequences than most survey datasets. This volume enables CAREER to learn representations that transfer to survey datasets.
3 RELATED WORK
Many economic analyses use log-linear models predict jobs in survey datasets (Boskin, 1974; Schmidt & Strauss, 1975). These models typically use small state spaces consisting of only a few occupation categories. For example, some studies categorize occupations into broad skill groups (Keane & Wolpin, 1997; Cortes, 2016); unemployment analyses only consider employment sta- tus (employed, unemployed, and out-of-labor-foree (Hall, 1972: Lauerova & Terrell, 2007); and researchers studying occupational mobility only consider occupational change, a binary variable in- dicating whether an individual changes jobs (Kambourov & Manovskii, 2008; Guvenen et al., 2020). Although transitions between occupations may depend richly on history, many of these models con- dition on only the most recent job and a few manually constructed summary statistics about history to make predictions (Hall, 1972; Blau & Riphahn, 1999). In contrast to these methods, CAREER is nonlinear and conditions on every job in an individual's 's history. The model learns complex rep- resentations of careers without relying manually constructed features. Moreover, CAREER can effectively predict from among hundreds of occupations.
Recently, the proliferation of business networking platforms has resulted in the availability of large resume datasets. Schubert et al. (2021) use a large resume dataset to construct a first-order Markov model of job transitions; CAREER, which conditions on all jobs in a history, makes more accurate predictions than Markov model. Models developed in the data mining community rely on resume- specific features such as stock prices (Xu et al., 2018), worker skill (Ghosh al., 2020), network information (Meng et al., 2019; Zhang et al., 2021). and textual descriptions (He et al., 2021). and are not applicable to survey datasets, as our goal in this paper (other models reduce to a first-order Markov model without these features (Dave et al., 2018; Zhang et al., 2020)). 'he most suitable model for survey datasets from this line of work is NEMO, an LSTM-based model that trained on large resume datasets (Li et al., 2017). Our experiments demonstrate that CAREER outperforms NEMO when it is adapted to model survey datasets.
Recent works in econometrics have applied machine learning methods to sequences of jobs and other discrete data. Ruiz et al. (2020) develop a matrix factorization method called SHOPPER to model supermarket basket data. We consider a baseline "bag-of-jobe model similar to SHOPPER. Like the transformer- based model, the bag-of-jobs model conditions on every job in an individual's history, but it uses relatively simple representations of careers. Our empirical studies demonstrate that CAREER learns complex representations that are better at modeling job sequences. Rajkumar al. (2021) build on SHOPPER and propose Bayesian factorization method for predicting job transitions. Similar to CAREER, they predict jobs in two stages. However, their method is focused on modeling individual transitions, history. In our empirical studies, we show that models like CAREER that condition on every job in an individual's history form more accurate predictions than Markov models.
it only conditions on the most recent job in an individual's
CAREER is based on transformer, a successful model for representing sequences of words in natural language processing (NLP). In econometrics, transformers have been applied to the text of job descriptions to predict their salaries (Bana, 2021) or authenticity (Naudé et al., 2022); rather than modeling text, we use transformers to model sequences of occupations. Transformers have also been applied successfully to sequences other than text: images (Dosovitskiy et al., 2021), music (Huang et al., 2019), and molecular chemistry (Schwaller et al., 2019). Inspired by their success in modeling variety of complex discrete sequential distributions, this paper adapts transformers to modeling sequences of jobs. Transformers are especially adept at learning transferrable representations of text from large corpora (Radford al., 2018; Devlin al., 2019). We show that CAREER learns

---

Under review as a conference paper at ICLR 2023
representations of job sequences that can be transferred from noisy resumme datasets to smaller, well- curated administrative datasets.
4 EMPIRICAL STUDIES
We assess CAREER's ability predict jobs and provide useful representations of careers. We pre- train CAREER on a large dataset of resumes, and transfer these representations to small, commonly used survey datasets. With the transferred representations, the model is better than econometric baselines at both held-out prediction and forecasting. Additionally, we demonstrate that CAREER's representations can be incorporated into standard wage prediction models to make better predictions.
Resume pretraining. We pretrain CAREER on a large dataset of resumes provided by Zippia Inc., a career planning company. This dataset contains resumes from 23.7 million working Americans. Each job is encoded into one of 330 occupational codes, using the coding scheme of Autor & Dorn (2013). We transform resumes into sequences of jobs by including an occupation's 's code for each year in the resume. For years with multiple jobs, we take the job the individual spent the most time in. We include three covariates: the year each job in an individual's career took place, along with the individual's state of residence and most recent educational degree. We denote missing covariates with a special token. See Appendix F for an exploratory data analysis of the resume data.
CAREER uses a 12-layer transformer with 5.6 million parameters. Pretraining CAREER on the sumes data takes 18 hours on a single GPU. Although our focus is on fine-tuning CAREER to model survey datasets rather than resumes, CAREER also outperforms standard econometrie baselines for modeling resumes; see Appendix B for more details.
Survey datasets. We transfer CAREER to three widely-used survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income Dynamics (PSID). These datasets have been carefully constructed be representative of the general population, and they are widely used by economists for estimating important quantities. NI.SY79 is longitudinal panel survey following a cohort of Americans who were between 14 and 22 when the survey began in 1979, while NLSY97 follows a different cohort of individuals who were between 12 and 17 when the survey began in 1997. PSID a longitudinal survey following a sample of American families, with individuals added over the years.
Compared the resumes dataset, these survey datasets are small: we use slices of NLSY79, NLSY97, and PSID that contain 12 thousand, 9 thousand, and 12 thousand individuals, respec- tively. The distribution of job sequences in resumes differs in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over- represented resume data (see Appendix F for more details). We pretrain CAREER on the large resumes dataset and fine-tune the smaller survey datasets. The fine-tuning process is efficient; al- though CAREER has 5.6 million parameters, fine-tuning on one GPU takes 13 minutes on NLSY79, 7 minutes NLSY97, and 23 minutes on PSID.
We compare CAREER to several bascline models: a second-order linear regression with covariates and hand-constructedd summary statistics about past employment (a common econometric model used to analyze these survey datasets - see Section 3); a bag-of-jobs model inspired by SHOPPER (Ruiz al., 2020) that conditions on all jobs and covariates history but combines representations linearly; and several baselines developed in the data-mining community for modeling worker pro- files: NEMO (Li et al., 2017), job representation learning (Dave al., 2018), and Job2Vec (Zhang et al., 2020). As described in Section 3, the baselines developed in the data-mining community for modeling worker profiles cannot be applied directly to economic survey datasets and thus re- quire modifications, described in detail in Appendix I. We also compare to two additional versions of CAREER one without pretraining or two-stage prediction, the other only without two-stage prediction - to assess the sources of CAREER's improvements. All models use the covariates we included for resume pretraining, in addition to demographic covariates (which are recorded for the survey datasets but unavailable for resumes).
We divide all survey datasets into 70/10/20 train/validation/esst splits, and train all models by op- timizing the log-likelihool with Adam (Kingma & Ba, 2015). We evaluate the predictive perfor- mance of each model by computing held-out perplexity. a common metric in NLP for evaluating probabilistic sequence models. The perplexity of a sequence model p on sequence yı,...; yr is
7

---

Under review as a conference paper at ICLR 2023
PSID
NI.S779
NI.Y997
Markov regression (Hall, 1972)
18.97 10.10
15.03
#0.03
20.81
#0.02
11.9
NEMO (Li et al., 2017)
17.58 10.04
12.82 10.04
18.38 10,08
Job rep. learning (Dave et al., 2018)
17.23 10.16
14.71 #0.02
16.83 +0.03
11.7
Vec (Zhang et al., 2020)
16.48 +0.13
14.46 +0.01
16.20 =0.02
Bag-of-jobs (Ruiz al., 2020)
16.21 +0.08
13.09 +0.03
16.20 00001
Q. 11.5
CAREER (vanilla)
15.26 +0.08
12.20 +0.04
16.19 +0.04
Pretraining
CAREER (two-slage)
14.79 10.04
12.00
10.00
15.22 10.03
11.3
10
10
10'
CAREER (two-stage pretrain)
13.88 0.01
11.32
0.000
14.15 10.03
# Pretraining Examples
(a) Test perplexity on survey datasets. Results are averaged over three random (b) CAREER's scaling law on NI.SY79 as function of pretraining data volume.
seeds. CAREER (vanilla) includes covariates not two-stage prediction or pretraining; CAREER (two-stage) adds two-stage prediction.
Figure Prediction results on longitudinal survey datasets and scaling law.
exp(-1E logp(ulyy-1,,x)) It is a monotonic transformation of log-likelihood; better pre- dictive models have lower perplexities. We train all models to convergence and use the checkpoint with the best validation perplexity. See Appendix for more experimental details.
Figure 2a compares the test-set perplexity of each model. With the transferred representations, CA- REER makes the best predictions on all survey datasets, achieving state-of-the-art performance. The baselines developed in the data mining literature, which were designed to model large resume datasets while relying on resume- resume-specific features, struggle to make good predictions on these small survey datasets, performing on par with standard econometric baselines. Pretraining is the biggest source of CAREER'S improvements. Although the resume data noisy and differs in many ways from the survey datasets used for economic prediction, CAREER learns useful representations of work experiences that aid its predictive performance. In Appendix G we show that modifying the baselines incorporate two-stage prediction (Equations 1 to 3) improves their performance, al- though CAREER still makes the best predictions across datasets. We include qualitative analysis of CAREER's predictions in Appendix
To assess how the volume of resumes used for pretraining affects CAREER's 's predictions on survey datasets, we downsample the resume dataset and transfer to survey datasets. The scaling law for NLSY79 is depicted in Figure 2b. When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not offer any improvement. The relationship between pretraining volume and fine-tuned perplexity follows power law, similar to scaling laws in NLP (Kaplan et al., 2020).
We also assess CAREER's ability to forecast future career trajectories. In contrast to predicting held-out sequences, forecasting involves training models on all sequences before specific year. To predict future jobs for an individual, the fitted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories. This setting is useful for assessing model's ability to make long-term predictions, especially as occupational trends change over time.
We evaluate CAREER's 's forecasting abilities NLSY97 and PSID. (These datasets more valu- able for forecasting than NLSY79, which follows a cohort that is near or past retirement age.) We train models on all sequences (holding out 10% validation set), without including any obser- vations after 2014. When pretraining CAREER on resumes, we also make sure to only include examples up 2014. Table compares the forecasting performance of all models. CAREER makes the best overall forecasts. CAREER has significant advantage over baselines at making long-term forecasts, yielding 17% advantage over the best baseline for 6-year forecasts on NLSY97. Again, the baselines developed for resume data mining, which had been developed to model much larger corpora, struggle to make good predictions on these smaller survey datasets.
Downstream applications. In addition to forming job predictions, CAREER learns low- dimensional representations of job histories. Although these representations were formed predict jobs in sequence, they can also be used as inputs to economic models for downstream applications.
As example of how CAREER's representations can be incorporated into other economic models, we use CAREER to predict wages. Economists build wage prediction models in order to estimate important economic quantities, such the adjusted gender wage gap. For example, to estimate this
8

---

Under review as a conference paper at ICLR 2023
NLSY97
PSID
Overall
Year
4- 4-Year
Year
Overall
2-Year
4- 4-Year
6- Year
Markov regression
23.11
12.50
25.88
36.59
19.43
11.83
21.66
27.89
Bag-of-jobs
22.51
11.98
25.11
36.29
19.28
11.44
21.68
28. 14
NEMO
25.26
12.59
28.35
43.01
18.58
11.08
20.67
27.29
CAREER
19.41
10.78
21.57
30.19
16.51
10.35
18.30
23.18
Table Forecasting perplexity (lower is better) on NL.Y977 and PSID. Results are averaged over three random seeds.
1981
1990
1999
2007
2009
2011
Full specification from Blau Kahn (2017a)
0.148
0.152
0.178
0.198
0.204
0.203
Full specification + CAREER
0.139
0.134
0.160
0.181
0.183
0.182
Table 2: Held-out mean square error for wage regressions (averaged over 10 splits).
wage gap, Blau & Kahn (2017a) regress an individual's log- wage observable characteristics such as education, demographics, and current occupation for six different years on PSID, Rather than including the full, high nigh-dimensional job-history, the model summarizes individual's career with summary statistics such as full-time and part-time years of experience (and their squares).
We incorporate CAREER's s representation into the wage regression by adding the fitted representa- tion for individual's job history, h4. For log- ag-waae wl; and observed covariates 2;, regress
uy a + x, + ha
where a, 8, and y regression coefficients. We pretrain CAREER to predict jobs resumes, and for each year we fine-tune on job sequences of the cohort up to that year. For example, in the 1999 wage regression, we fine-tune CAREER only the sequences of jobs until 1999 and plug in the fixed representation to the wage regression. We do not include any covariates (except year) when training CAREER. We run each wage regression 80% of the training data and evaluate mean-square error on the remaining 20% (averaging over 10 random splits).
Table 2 shows that adding CAREER's representations improves wage predictions for each year. Al- though these representations are fine-tuned to predict jobs on a small dataset (each year contains less than 5,000 workers) and are adjusted account for wage, they contain information that is predictive of wage. By summarizing complex career histories with a fow-dimensional representa- tion, CAREER provides representations that can improve downstream economic models, resulting in more accurate estimates of important economic quantities.
5 CONCLUSION
We introduced CAREER, a method for representing job sequences from large-scale resume data and fine-tuning them on smaller datasets interest. We took inspiration from modern language model- ing to develop a transformer- fonddr---rr--ddddd occupation model. We transferred the model from a large dataset of resumes smaller survey datasets in economics, where it achieved state-of-the-art performance for predicting and forecasting career outcomes. We demonstrated that CAREER'S 's representations can be incorporated into wage prediction models, outperforming standard econometric models.
One direction of future research is to incorporate CAREER's 's representations of job history into methods for estimating adjusted quantities, like wage gaps. Underlying these methods are models that predict economic outcomes function of observed covariates. However, if relevant variables are omitted, the adjusted estimates may be affected; e.g., excluding work experience from wage prediction may change the magnitude of the estimated gap. In practice, economists include hand- designed summary statistics to overcome this problem, such as in Blau & Kahn (2017a). CAREER provides data-driven way to incorporate such variables-ae its representations of job history could be incorporated into downstream prediction models and lead to more accurate adjustments of economic quantities.

---

Under review as a conference paper at ICLR 2023
Ethics statement. As discussed, passively-collected resume datasets not curated represent national demographics. Pretraining CAREER on these datasets may result in representations that affected by sampling bias. Although these representations that are carefully constructed to represent national demographics, the biases from pretraining may propagate through fine-tuning (Ravfogel et al., 2020; Jin et al., 2021). Moreover, even in represen- tative datasets, models may form more accurate predictions for majority groups due to data volume (Dwork al., 2018). Thus, we encourage practitioners audit noisy resume data, re-weight sam- ples as necessary (Kalton, 1983), and review accuracy within demographics before using the model for downstream economic analysis.
fine-tuned on survey datasets
Although resume datasets may contain personally identifiable information, all personally identifiable information had been removed before we were given access to the resume dataset we used for pretraining. Additionally, none of the longitudinal survey datasets contain personally identifiable information.
Reproducibility statement. The supplementary material contains code for reproducing the perimental results in this paper, with the README containing detailed instructions for reproducing specific experiments. Our data-use agreement prohibits from releasing the dataset of resumes used for pretraining. However, similar (private) resume datasets have become increasingly common in applied economics analyses (Azar al., 2020; Schubert et al., 2021), and we include pretraining code so practitioners can reproduce our results with resume datasets they have access to. Addi- tionally, all longitudinal survey datasets are available publicly online (Bureau of Labor Statistics, 2019a;b; Panel Study of Income Dynamics, 2021).
REFERENCES
David Autor and David Dorn The growth of low-skill service jobs and the polarization of the U.S. labor market. American Economic Review, (03(5):1553-97,2)))
José Azar, Ioana Marinescu, Marshall Steinbaum, and Bledi Taska. Concentration in us labor mar- kets: Evidence from online vacancy data. Labour Economics, 66:101886, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv:1607.064500, 2016.
Sarah H Bana. Using language models understand wage premia. 2021.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1 137-1115, 2003.
David M Blau and Regina T Riphahn. Labor force transitions of older married couples in Germany. Labour Economics, 6(2):229-252, 1999.
Francine D Blau and Lawrence M Kahn. The gender wage gap: Extent, trends, and explanations. Journal of Economic Literature, 55(3):789-863 2017a.
Francine D Blau and Lawrence M Kahn. Replication data for: The Gender Wage Gap: Extent,
Trends, and Explanations, 201 7b. URL https://doi.orr 3886/E113913Vi
Michael Boskin. A conditional logit model of occupational choice. Journal of Political Economy, 82(2, Part 1):389-399 1974.
Randall S Brown, Marilyn Moon, and Barbara S Zoloth. Incorporating occupational attainment in studies of male-female earnings differentials. Journal of Human Resources, 15(1):3-28, 1980.
Bureau of Labor Statistics. National Longitudinal Survey of Youth 1979 cohort (rounds 1-27).
Produced and distributed by the Center for Human Resource Research (CHRR), The Ohio State University. Columbus, OH, 2019a.
Bureau of Labor Statisties. National Longitudinal Survey of Youth 1997 cohort (rounds 1-19). Produced and distributed by the Center for Human Resource Research (CHRR), The Ohio State University. Columbus, OH, 2019b.
10

---

Under review as a conference paper at ICLR 2023
Guido Matias Cortes. Where have the middle-wage workers gone? A study of polarization using panel data. Journal of Labor Economics, 34(1):63-105 2016.
Vachik Dave, Baichuan Zhang, Mohammad Al Hasan, Khalifeh AlJadda, and Mohammed Ko- rayem. A combined representation learning approach for better job and skill recommendation. In ACM Conference on Information and Knowledge Management, 2018.
J. Devlin, M. Chang, K. Lec, and K. Toutanova. BERT: Pre-training of deep bidirectional transform- for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- on Learning Representations, 2021.
Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. Decoupled classifiers for group-fair and efficient machine learning. In Conference on Fairness, Accountability and Transparency, 2018.
Robert W Fairlie and William A Sundstrom. The emergence, persistence, and recent widening of the racial unemployment gap. Industrial and Labor Relations Review, 52(2):552-270, 1999.
Aritra Ghosh, Beverly Woolf, Shlomo Zilberstein, and Andrew Lan. Skill-based career path model- ing and recommendation. In IEEE International Conference on Big Data, 2020.
Fatih Guvenen, Burhan Kuruseu, Satoshi Tanaka, and David Wiczer. Multidimensional skill mis- match. American Economic Journal: Macroeconomics, 12(1):10044,, 2020.
Robert Hall. Turnover in the labor force. Brookings Papers on Economic Activity 1972(3): 709-764, 1972.
Miao He, Xiaoming Zhan, Dayong Shen, Yuanyuan Zhu, Hua Zhao, and Renjie He. What about your next job? predicting professional career trajectory using neural networks. In Machine Learning and Machine Intelligence, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). 1606:0600008411 2016.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference Learning Representations, 2019.
Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. On transferability of bias mitigation effects in language model fine-tuning. In North American Chapter of the Association for Computational Linguistics, 2021.
Graham Kalton. Compensating for Missing Survey Data. Survey Research Center, Institute for Social Research, the University of Michigan, 1983.
Gueorgui Kambourov and Tourii Manovskii. Rising occupational and industry mobility in the United States: 1968-97. International Economic Review, 49(1):41-79, 2008
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001. .08611, 2020.
Michael P Keane and Kenneth Wolpin. The career decisions of young men. Journal of Political Economy, 105(3):433-522, 1997.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jana Stefanova Lauerova and Katherine Terrell. What drives gender differences in unemployment? Comparative Economic Studies, 49(1):128 2007.

---

Under review as a conference paper at ICLR 2023
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-t--seqeene pre-training for natural language generation, translation, and comprchension. In Association for Computa- fional Linguistics, 2019.
Liangyue Li, How Jing, Hanghang Tong, Jaewon Yang, Qi He, and Bee-Chung Chen. NEMO: Next career move prediction with contextual embedding. In World Wide Web Conference, 2017.
Brian P McCall. Occupational matching: A test of sorts. Journal of Political Economy, 98(1):45-69, 1990.
Qingxin Meng, Hengshu Zhu, Keli Xiao, Le Zhang, and Hui Xiong. A hierarchical career-path- aware neural network for job mobility prediction. ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In Neural Information Processing Systems, 2013.
Marcel Naudé, Kolawole John Adebayo, and Rohan Nanda. A machine learning approach to detect- ing fraudulent job types. Al & Society, pp. 1-12, 2022.
Derek Neal, The complexity of job mobility among young men. Journal of Labor Economics, 17 (2):237-261, 1999.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli Scaling neural machine translation. In Conference on Machine Translation (WMT), 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: A fast, extensible toolkit for sequence modeling. In Association for Computational Linguisties, 2019.
Panel Study of Income Dynamics. Public use dataset, produced and distributed by the Survey Re- search Center, Institute for Social Research, University of Michigan, Ann Arbor, MI, 2021.
James Poterba and Lawrence H Summers. Reporting errors and labor market dynamics. Econo- metrica, 54(6):1319-133 1338, 1986.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Karthik Rajkumar, Lisa Simon, and Susan Athey. A Bayesian Approach Predicting Occupational Transitions. PhD thesis, Stanford University, 2021.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by itcrative nullspace projection. In Association for Computational Lin- guistics, 2020.
Francisco J. Ruiz, Susan Athey, and David M. Blei. SHOPPER: A probabilistic model con- sumer choice with substitutes and complements. Annals of Applied Statistics, 14(1):1-27, 2020.
Peter Schmidt and Robert P Strauss. The prediction of occupation using multiple logit models. International Economic Review, 16(2):471-486 1975.
Gregor Schubert, Anna Stansbury, and Bledi Taska. Employer concentration and outside options. SSRN:3599454, 2021.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: model for uncertainty-calibaac. chemical reaction prediction. ACS Central Science, 5(9):552-1588,, 2019. 2019.
12

---

Under review as a conference paper at ICLR 2023
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1955) 1958, 2014.
Keyon Vafa, Yuntian Deng, David M Blei, and Alexander M Rush. Rationales for sequential pre- dictions. In Empirical Methods in Natural Language Processing, 2021.
Dušan Variš and Ondrej Bojar. Sequence length is domain: Length-based overfitting in transformer models. In Empirical Methods Natural Language Processing, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and IIlia Polosukhin. Attention all you need. In Neural Information Processing Systems, 2017.
Mark N Wexler. Successful resume fraud: Conjectures on the origins of amorality in the workplace. Journal of Human Values, 12(2): 137-152, 2006.
Huang Xu, Zhiwen Yu, Jingyuan Yang. Hui Xiong, and Hengshu Zhu. Dynamic talent flow analysis with deep sequence prediction modeling. IEEE Transactions on Knowledge and Data Engineer- ing, (1(1)) (12)):996-1999)) 2018
Denghui Zhang. Junming Liu, Hengshu Zhu, Yanchi Liu, Lichen Wang, Pengyang Wang, and Hui Xiong. Job2Vec: Job title benchmarking with collective multi-view representation learning. arXiv preprint arXiv:2009.0744 2020.
Le Zhang, Ding Zhou, Hengshu Zhu, Tong Xu, Rui Zha, Enhong Chen, and Hui Xiong, Attentive heterogeneous graph embedding for job mobility prediction. In ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
13

---

Under review as a conference paper at ICLR 2023
Model
Overall
Consecutive Repeat
Non-Consecutive Repeat
New Job
First-order Markov
5.73
1.25
258.75
479.65
No covariates
Second-order Markov
5.66
1.25
154.35
471.10
Bag-of-Jobs
5,53
1.25
59.07
458.53
Transformer
5.38
1.23
36.01
429.32
Regression
5,34
1.22
130.25
414.76
Covariates
Bag-of-Jobs
5.26
1.23
56.58
410.62
CAREER
5.05
1.21
33.72
358.80
Table 3: Held-out perplexity on the large resumes dataset (lower is better).
A ECONOMETRIC BASELINES
In this section, we describe baseline occupation models that economists have used to model jobs and other discrete sequences.
Markoy models and regression. A first-order Markov model assumes the job at each timestep depends on only the previous job (Hall, 1972: Poterba & Summers, 1986). Without covariates, Markov model takes the form p(y: = jly;-1) p(y: j 1)) The optimal transition probabil- ities reflect the overall frequencies of individuals transitioning from occupation 3/-1 1 to occupation In second-order Markov model, the next job depends the previous two.
multinomial logistic regression can be used to incorporate covariates:
plu = ) & exp 0 + B800 31-1 Ee g00 Ite
(10)
where 8/0) is occupation-specific intercept and 1/-1 and Tte denote J- and Ne-dimensional indicator vectors, respectively. Equation 10 depends on history only through the most recent job, although the covariates can also include hand-crafted summary statisties about the past, such as the duration of the most recent job (McCall, 1990). This model is fit by maximizing the likelihood with gradient-based methods.
Bag-of-jobs. A weakness of the first-order Markov model is that it only uses the most recent job to make predictions. However, one's working history beyond the last job may inform future transitions (Blau & Riphahn, 1999; Neal, 1999).
bag-of-jobs model, inspired by SHOPPER, a probabilistic model of consumer choice (Ruiz et al., 2020). Unlike the Markov and regression models, the bag-of-johr model conditions on every job in an individual's history. does so by learning a low-dimensional representation of an individual's history. This model learns unique embedding for each occupation, similar to word embedding (Bengio et al., 2003; Mikolov et al., 2013); unlike CAREER, which lears complicated nonlinear interactions between jobs in history, the bag- hag-of-jobs model combines jobs into a single representation by averaging their embeddings.
Another baseline we consider
The bag-of-jobs model assumes that job transitions depend on two terms: term that captures the effect of the most recent job, and a term that captures the effect of all prior jobs. Accordingly, the model learns two types of representations:
embedding aj TR of the most recent job j. and an embedding P. IRa P for prior jobs j'. To combine the representations for all prior jobs into single term, the model averages embeddings:
p(u= = jly:-1) exp ((4)) O ap-, 1000
(L2 Poe }
22-
(11)
Covariates be added the model analogously; for single covariate, its most recent value embedded and summed with the average embeddings for its prior values. All parameters are estimated by maximizing the likelihood in Equation 11 with SGD.
14

---

Under review as a conference paper at ICLR 2023
Overall
2015
2016
2017
Regression 20.71
7.78
27.97
40.85
Bag-of- ofg-oofJJobs
19.45
7.57
25.63
37.93
CAREER 17.37
7.07
23.06
32.11
Table 4: Forecasting perplexity (lower is better) for unseen years in the large resumes dataset. Each model is trained on sequences before 2015 and makes forecasts three years into the future. The "overall" column averages perplexities across all three forecasted years.
B RESUME PREDICTIONS
Although our focus is on modeling survey datasets, we also compare CAREER to several economet- ric baselines for predicting job sequences in resumes. We consider series of models without covari- ates: a first- and second-order Markov model, bag-of-jobs, model (Equation 11), and a transformer with the same architecture as CAREER except without covariates. We also compare econometric models that use covariates: a second-order linear regression with covariates and hand-constructed features (such as how long an individual has worked in their current job), and a bag-of-jobs model with covariates (Appendix has more details).
We randomly divide the resumes dataset into a training set of 23.6 million sequences, and val- idation and test set of 23 thousand sequences each. Table 3 compares the test-set predictive per- formance of all models. CAREER is the best at predicting held-out sequences. To understand the types of transitions contributing to CAREER's predictive advantage, three categories: consecutive repeats (when the next job is the same as the previous year's), s)), non- consecutive repeats (when the next job different from the previous year's, but is the same the prior jobs in the career), and new jobs. CAREER has clear advantage over the baselines in all three categories, but the biggest improvement comes when predicting jobs that have been repeated non-consecutively. The transformer model is at an advantage over the Markov models for these kinds of predictions because is able condition on an individual's entire working history, while Markov model is constrained to use only the most recent job (or two). The bag-of-johs model, which can condition on all jobs in a worker's 's history but cannot learn complex interactions between them, outperforms the Markov models but still falls short of CAREER, which can recognize and repre- sent complex career trajectories. Appendix C, we demonstrate that CAREER is well-equipped at forecasting future trajectories well.
decompose predictions into
of
C FORECASTING RESUMES
We also perform the forecasting experiment the large dataset of resumes. Each model is trained on resumes before 2015. To predict occupations for individuals after 2015, a model samples 1 1,000 trajectories for each individual, and averages probabilities to form single prediction for each year. For more experimental details, see Appendix I.
Table 4 depicts the forecasting results for the resumes dataset. Each fitted model is used to forecast occupation probabilities for three years into the future. CAREER makes the best forecasts, both overall and for each individual year.
D QUALITATIVE ANALYSIS
Rationalizing predictions. Figure shows an example of held-out career sequence from PSID. CAREER is much likelier than regression and bag-of-jobs baseline to predict this individual's next job, biological technician. To understand CAREER's 's prediction, we show the model's rationale, or the jobs in this individual's history that are sufficient for explaining the model's prediction. (We adapt the greedy rationalization method from Vafa et al. (2021), refer to Appendix for more details.) In this example, CAREER only needs three previous jobs to predict biological technician: animal caretaker, engincering technician, and student. The model can combine latent attributes each job predict the individual's next job.
15

---

Under review as a conference paper at ICLR 2023
CAREER rationale
Job History
Predicting
of Labor Force
Animal
Engineering Technician
Biological Techniclan
Student
Student
Student
Caretaker
Figure 3: An example of held-out job sequence on PSID along with CAREER's rationale. CA- REER ranks the true next job (biological technician) the most likely possible transition for this individual: in contrast, the regression and oag-of--obss model rank it as 40th and 37th most likely, respectively. The rationale depicts the jobs in the history that were sufficient for CAREER tion.
predic-
2002:
Janttor
2002:
Retail selesperson Material handler Laundry worker
2003:
Material handler
2000:
Shipping clerk
2003: Food prep worker
0.957
2003:
2004:
Material handler
0.964
Shipping clark 2004:
2004:
Food service worker
2004:
2005: handler
2005:
Shipping clark
worker
2006: Material handler
2008: Shipping clerk
selesperson
2001: Child care worker
2001:
Teacher's is aide
2002:
Receptionist
0.956
2002: Secretary
2002: Child care worker
2002:
Teacher's aide
2003:
Receptionist
2003: Secretary
2003: Child care worker
2003:
Teacher's
Receptionist
2004: Secrctary
2004: care
2004:
Teacher's is side
Figure 4: The work experiences with the most similar CAREER representations (measured with cosine similarity) for individuals with overlapping jobs in NLSY97.
Representation similarity. To demonstrate the quality of the learned representations, we use CA- REER's fine-tuned representations NLSY97 to find pairs of individuals with the most similar career trajectories. Specifically, we compute CAREFER's representation (y xe) for each in- dividual in NLSY97 who has worked for four years. We then measure the similarity between all pairs by computing the cosine similarity between representations. order to depict meaningful matches, only consider pairs of individuals with no overlapping jobs in their histories (otherwise the model would find individuals with the exact same career trajectories). Figure 4 depicts the ca- reer histories with the most similar CAREER representations, Although none of these pairs have overlapping jobs, the model learns representations that can identify similar careers.
E TRANSFORMER DETAILS
In this section, we expand on the simplified description of transformers in Section 2.3 and de- scribe CAREER full detail. Recall that the model estimates representations in layers, with each representation h! E RD. The final representation (1) (y)-1.x() used to represent careers. We drop the explicit dependence on yt-1 and x;, and h
h1 (1) (1(") (y.-1.x;) (x).
instead denote each representation as 1,(
The first transformer layer combines the previous occupation, the most recent covariates, and the position of the occupation in the career. It first embeds each of these variables in D-dimensional space. Define an embedding function for occupations, ey [J] + RR. Additionally, define separate embedding function for each covariate, {ea] with each e, : [Na] R Finally, define e, [T] + R to embed the position of the sequence, where T' denotes the number of possible sequence lengths. The first-layer representation ni"" sums these embeddings:
h (v + +c (x(1rc) + et()..
(12)
The occupation- and covariate-specific embeddings, ey and {Ee}, are model parameters; the po- sitional embeddings, e,, are set advance to follow a sinusoidal pattern (Vaswani et al., 2017). While these embeddings could also be parameterized, in practice the performance is similar, and using sinusoidal embeddings allows the model to generalize to career sequence lengths unseen in the training data.
16

---

Under review as a conference paper at ICLR 2023
At each subsequent layer, the transformer combines the representations of all occupations in his- tory. combines representations by performing multi-headed attention, which is similar to the process described in Section 2.3 albeit with multiple attention weights per layer.
Specifically, it uses A specific attention weights, or heads, per layer. The number of heads A should be less than the representation dimension D. (Using A = attention head reduces the process described in Equations and 6.) The representation dimension D should be divisible by A; denote K D/A. A. First, A different sets of attention weights are computed:
(0)
(n!²) W( for t' St
2 a,,,t''
(13)
Σx exp(2a
where (( E DxD is model parameter, specific to attention head and layer 1.* Each attention head forms a convex combination with all previous representations; to differentiate between atten- tion heads, each representation is transformed by a linear transformation v(()) E R&xD unique to an attention head, forming ( E RR:
6 = Ei_ Rat, (vo 1
(14)
All attention heads are combined into single representation by concatenating them into a single vector g( E R:
gi 61 600
(
(1,,
(15)
To complete the multi-head attention step and form the intermediate representation R,() the concate- nated representations db undergo a linear transformation and are summed with the pre-attention representation (,:
T. 11 M( (2)
9r
(16)
with M0) E RDD
The intermediate representations 7,(() E R.D combine the representation at timestep i with those preceding timestep Each layer of the transformer concludes by taking a non-linear transformation of the intermediate representations, This non-linear transformation does not depend any previous
representation; only transforms h.) Specifically, 7,(") passed through neural network:
h (2) (F (A.
(17)
where (F) denotes two-layer feedforward neural network with N hidden units, with FFN R + DR
We repeat the multi-head attention and feedforward neural network updates above for layers, using parameters unique to each layer. We represent careers with the last-layer representation, x, = 11 (y)-1xx)).
For our experiments, we use model specifications similar to the generative pretrained transformer (GPT) architecture (Radford et al., 2018). In particular, we use L 12 layers, representation dimension of D = 192, A = 3 attention heads, and N 768 hidden units and the GELU non- linearity (Hendrycks Gimpel, 2016) for all feedforward neural networks. In total, this results in 5.6 million parameters. This model includes few extra modifications to improve training: we use 0.1 dropout (Srivastava et al., 2014) for the feedforward neural network weights, and 0.1 dropout for the attention weights. Finally, we use layer normalization (Ba et al., 2016) before the updates in Equation 13, after the update in Equation 16, and after the final layer's neural network update in Equation 17.
*For computational reasons, W is decomposed into two matrices and scaled by a constant, W²(
9 (x
with Q8²) (((!) B Dx
VK
17

---

Under review as a conference paper at ICLR 2023
Number individuals
23,731,674
General
Number of tokens
245,439,86
Median year
2007
Percent Northeast
17.6
Percent Northcentral
20.7
Geography
Percent South
39,9
Percent
19.4
Percent without location
2.4
Percent high school diploma
7.2
Percent associate degree
8.6
Education
Percent bachelor degree
23.1
Percent graduate degree
4.5
Percent empty
52.8
Percent manngeriial/professionnl specialty
38.4
Percent tectniccllslee/dddiinttttivee support
34.2
Broad Occupation Groups
Percent service
12,0
Percent precision production/crattrrpaai
7.9
Percent opernatofffbbricatoo/lbeeerrrrr
7.2
Table Exploratory data analysis of the resumee dataset used for pretraining CAREER.
Resumes
NLSY79
NLSY97
PSID
Number individuals
million
thousand
thousand
12 thousand
Median haaer/uuuto---laao--orreeeeeeeee available
No
Yes
Yes
Yes
2007
1991
2007
2011
Percent manual laborers
7%
17%
13%
12%
Percent college graduates Demographic covariates available
56%
23%
29%
28%
No
Yes
Yes
Yes
Table 6: Comparing the resume dataset used for pretraining with the three longitudinal survey datasets of interest.
F EXPLORATORY DATA ANALYSIS
Table depicts summary statistics of the resume dataset provided by Zippia that is used for pre- training CAREER. Table 6 compares this resume dataset with the longitudinal survey datasets of interest.
G ONE-STAGE VS TWO-STAGE PREDICTION
Table 7 compares the predictive performance of occupation models when they are modified to make predictions in two stages, following Equations to Incorporating two-stage prediction improves
PSID
NLSY79
NLSY97
Markov regression (two-stage)
15.60 + 0.03
13,30 - 0.02
15.47 + 0.00
NEMO (two-stage)
15.23 + 0.08
12.37 + 0.04
15.13 + 0.03
Job rep. learning (two-stage)
15.98 0.06
13.97 0.03
15.43 + 0.01
Job2Vec (two-stage)
15.80 0.04
13.91 0.01
15.31 + 0.02
Bag-of-johs (two-stage)
0.05
12.68
+ 0.01
15.11 + 0.02
CAREER
13.88 1 0.01
11.32 1 0.00
14.15 1 0.03
Table 7: Perplexity of economic baselines when they modified to make predictions in two stages.
18

---

Under review as a conference paper at ICLR 2023
the performance of these models compared to Figure 2a; however, CAREER still makes the best predictions on all survey datasets.
H DATA PREPROCESSING
In this section, we go over the data preprocessing steps we took for each dataset.
Resumes. We were given access to a large dataset of resumes of American workers by Zippia, a career planning company. This dataset coded each occupation into one of 1,073 O*NET 2010 Stan- dard Occupational Classification (SOC) categories based on the provided job titles and descriptions in resumes. We dropped all examples with missing SOC codes.
Each resume in the dataset we were given contained covariates that had been imputed based off other data in the resume. We considered three covariates: year, most recent educational degree, and location. Education degrees had been encoded into one of eight categories: high school diploma, as- sociate, bachelors, masters, doctorate, certificate, license, and diploma. Location had been encoded into one of 50 states plus Puerto Rico. Washington D.C., and unknown, for when location could not be imputed. Some covariates also had missing entries. When an occupation's 's year was missing, we had to drop it from the dataset, because we could not position it in an individual's career. When- ever another covariate was missing, we replaced it with a special "missing" token. All personally identifiable information had been removed from the dataset.
We transformed each resume the dataset into a sequence of occupations. We included an entry for each year starting from the first year an individual worked to their last year. We included a special "beginning of sequence" token to indicate when each individual's sequence started. For each year between individual's first and last year, we added the occupation they worked in during that year. If an individual worked in multiple occupations in a year, we took the one where the individual spent more time in that year, if they were both the same amount of time in the particular year, we broke ties by adding the occupation that had started earlier in the career. For the experiments token for years
predicting future jobs directly on resumes, added "no-observed-occupaiiee where the resume did not list any occupations (we dropped this token when pretraining). Each occupation was associated with the individual's most recent educational degree, which we treated as a dynamic covariate. The year an occupation took place was also considered a dynamic categorical covariate. We treated location as static. In total, this preprocessing left us with a dataset of 23.7 million resumes, and 245 million individual occupations.
In order to transfer representations, we had to slightly modify the resumes dataset for pretraining to encode occupations and covariates into a format compatible with the survey datasets. The sur- vey datasets we used were encoded with the "oce1990de occupation code (Autor & Dorn, 2013) rather than with O*NET's SOC codes, so we converted the SOC codes to occ 1990dd codes using a crosswalk posted online by Destin Royer. Even after we manually added a few missing entries to the crosswalks, there were some SOC codes that did not have corresponding occ1990dd's. We gave these tokens special codes that were not used when fine-tuning on the survey datasets (because they did not correspond to 19c1990dd occupations). When an individual did not work for given year, the survey datasets differentiated between three possible states: unemployed. out-of-labor-foree and in-school. The resumes dataset did not have these categories. Thus, we initialized parameters for these three new occupational states randomly. Additionally, we did not include the "no-observed- occupation" " token when pretraining, and instead dropped missing years from the sequence. Since we did not use gender and race/cthnicity covariates when pretraining, we also initialized these covariate- specific parameters randomly for fine-tuning. Because we used version of the survey datasets that encoded each individual's location as geographic region rather than as state, we converted each state the resumes data to be in one of four regions for pretraining: northeast, northcentral, south, or west. We also added a fifth "other" region for Puerto Rico and for when state was missing in the original dataset. We also converted educational degrees to levels of experience: we converted associate's 's degree to represent some college experience and bachelor's degree to represent four- year college experience; we combined masters and doctorate to represent a single "graduate degree" category; and we left the other categories they were.
19

---

Under review as a conference paper at ICLR 2023
NLSY79. The National Longitudinal Survey of Youth 1979 (NLSY79) is survey following in- dividuals born in the United States between 1957-1964. The survey included individuals who were between 14 and 22 years old when they began collecting data in 1979; they interviewed individuals annually until 1994, and biennially thereafter.
Each individual in the survey is associated with ID, allowing to track their careers over time. We converted occupations, which were initially encoded as OCC codes, into "occl 1990d9" codes using a crosswalk (Autor & Dorn, 2013). We use a version of the survey that has entries up to 2014. Unlike the resumes dataset, NLSY79 includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-foree, and in-school. We include special tokens for these states in our sequences. We drop examples with missing occupation states. We also drop sequences for which the individual is out of the labor force for their whole careers.
We use the following covariates: years, educational experience, location, race/ethnicity. and gender. We drop individuals with less than years of education experience. We convert years of educa- tional experience into discrete categories: no high school degree, high school degree, some college, college, and graduate degree. We convert geographic location to one of four regions: northeast, northcentral, south, and west. We treat location as a static variable, using each individual's first location. We use the following race/ethnicities: white, African American, Asian, Latino, Native American, and other. We treat year and education dynamic covariates whose values can change over time, and we consider the other covariates as static. This preprocessing leaves us with a dataset consisting of 12,270 individuals and 239,545 total observations.
NLSY97. The National Longitudinal Survey of Youth 1997 (NLSY97) is a survey following indi- viduals who were between 12 and 17 when the survey began in 1997. Individuals were interviewed annually until 2011, and biennially thereafter.
Our preprocessing of this dataset is similar to that of NLSY79. We convert occupations from OCC codes into "occ 1990dd' codes. We use a version of the survey that follows individuals up to 2019. We include tokens for unemployed, out-of-labor-foree and in-school occupational states. We only consider individuals who are over 18 and drop military-related occupations. We use the same co- variates as NI.S779. We use the following race/ethnicities: white, -ariiiana-Amriicann Latino, and other/unknown. We convert years of educational experience into discrete categories: no high school degree, high school degree, some college degree, college degree, graduate degree, and a special token when the education status isn't known. We use the same regions as NLSY79. We drop se- quences for which the individual is out of the labor force for their whole careers. This preprocessing leaves us with a dataset consisting of 8,770 individuals and 1 114,141 141 total observations.
PSID.
The Panel Study of Income Dynamics (PSID) is a longitudinal panel survey following a sample of American families. It was collected annually between 1968 and 1997, and biennially afterwards.
The dataset tracks families over time, but it only includes occupation information for the household head and their spouse, so we only include these observations. Occupations are encoded with OCC codes, which we convert "occ 1990dd" using a crosswalk (Autor & Dorn, 2013). Like the NLSY surveys, PSID also includes three states corresponding to individuals who are not currently em- ployed: unemployed, lab-of-labor-forcee and in-school, We include special tokens for these states in our sequences. We drop other examples with missing or invalid occupation codes. We also drop sequences for which the individual is out of the labor force for their whole careers.
We consider five covariates: year. education, location, gender, and race. We include observations for individuals who were added to the dataset after 1995 and include observations up to 2019. We exclude observations for individuals with less than 9 years of education experience. We convert years of education to discrete states: no high school, high school diploma, some college, college, and graduate degree. We convert geographic location to one of four regions: northeast, northcentral, south, and west. We treat location as a static variable, using each individual's 's first location. We use the following races: white, Black, and other. We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static. This preprocess- ing leaves with a dataset consisting of 12,338 individuals and 62,665 total observations.
20

---

Under review as a conference paper at ICLR 2023
I EXPERIMENTAL DETAILS
Baselines. We consider first-order Markov model and a second-order Markov model (both with- out covariates) as baselines. These models are estimated by averaging observed transition counts. We smooth the first-order Markov model by taking a weighted average between the empirical transi- tions in the training set and the empirical distribution of individual jobs. We perform this smoothing to account for the fact that some feasible transitions may never occur in the training set due to the high--dimensionalityy of feasible transitions. We assign 0.99 weight to the empirical distributions of transitions and 0.01 to the empirical distribution of individual jobs. We smooth the second- order model by assigning 0.5 weight to the empirical second-order transitions and 0.5 weight to the smoothed first-order Markov model.
When we add covariates to the Markov linear baseline, we also include manually constructed fea- tures about history to improve its performance. In total, we include the following categorical vari- ables: the most recent job, the prior job, the year, a dummy indicating whether there has been more than one year since the most recent observed job, the education status, a dummy indicating whether the education status has changed, and state (for the experiments on NLSY79 and PSID, we also include an individual's gender and race/ethnicityy. We also add additive effects for the following continuous variables: the number of years an individual has been in the current job and the total number of years for which an individual has been in the dataset. addition, we include an intercept term.
For the bag-of-jobs model, we vary the representation dimension D between 256-2048, and find that the predictive performance is not sensitive to the representation dimension, so we use D = 1024 for all experiments. For the LSTM model, we use layers with 436 embedding dimensions so that the model size is comparable to the transformer baseline: the LSTM has 5.8 million parameters, the same number as the transformer.
We also compare to NEMO (Li et al., 2017), LSTM-based method developed for modeling job sequences in resumes. We adapted NEMO to model survey data, In its original setting, NEMO took as input static covariates (such individual skill) and used these to predict both an individual's next job title and their company. Survey datasets differ from this original setting in a few ways: covariates vare-narynn, important covariates for predicting jobs on resumes (like skill) are missing, and an individual's company name is unavailable. Therefore, we made several modifications to NEMO. We incorporated the available covariates from survey datasets by embedding them and adding them to the job embeddings passed into the LSTM, similar to the method CAREER uses to incorporate covariates. We removed the company-prediction predict an individual's job in the next timestep. We considered two sizes of NEMO: an architecture using the same number of parameters as CAREER, and the smaller architecture proposed in the original paper. We found the smaller architecture performed better on the survey datasets, so we used this for the experiments. This model contains 2 decoder layers and a hidden dimension of 200.
objective, and instead only used the model to
We compare to two additional baselines developed in the data mining literature: job representation learning (Dave et al., 2018) and Job2Vec (Zhang et al., 2020). These methods require resume- specific features such skills and textual descriptions of jobs and employers, which are not available for the economic longitudinal survey datasets we model. Thus, we adapt these baselines to be suitable for modeling economic survey data. Job representation learning (Dave et al., 2018) is based on developing two graphs, one for job transitions and one for skill transitions. Since worker skills are not available for longitudinal survey data, we adapt the model only use job transitions by only including the terms in the objective that depend on job transitions. We make a few additional modifications, which we found to improve the performance of this model on our data. Rather than sampling 3-tuples from the directed graph of job transitions, we include all 2-tuple job transitions present in the data, identical to the other models we consider. Additionally, rather than using the contrastive objective in Equation 4 of Dave et al. (2018), we optimize the legllikeldhood directly this is more computationally intensive but leads to better results. Finally, we include survey- -urvey-specific covariates (e.g. education, demographics, etc.) by adding them to Wa. embedding the covariate of each most recent job to the same space as Wa. We make similar modifications to Job2Vec (Zhang et al., 2020). Job2Vec requires job titles and descriptions of job keywords, which are unavailable for economic longitudinal survey datasets. Instead, we modify Equation in Zhang et al. (2020) to model occupation codes rather than titles or keywords and optimize this logelikeld our
21

---

Under review as a conference paper at ICLR 2023
objective. We also incorporate survey-specific covariates by embedding each covariate to the same space as e; and adding it e; before computing Equation 2 from Zhang et al. (2020), which we also found to improve performance. We follow Dave et al. (2018) and use 50 embedding dimensions for each model, and optimize with Adam using a maximum learning rate of 0.005, following the minibatch and warmup strategy described below.
When we compared the transferred version of CAREER to a version of CAREER without pretrained representations, we tried various architectures for the non-pretrained version of CAREER. We found that, without pretraining, the large architecture we used for CAREER was prone to overfitting on the smaller survey datasets. So we performed an ablation of the non-pretrained CAREER with various architectures: we considered 4 and 12 layers. 64 and 192 embedding dimensions, 256 and 768 hidden units for the feedforward neural networks, and 2 or 3 attention heads (using 2 heads for D = 64 and 3 heads for D- = 192 so that D was divisible by the number of heads). We tried all combinations of these parameters on NLSY79, and found that the model with the best validation performance had 4 layers, D 64 embedding dimensions, 256 hidden units, and 2 attention heads. We used this architecture for the non-pretrained version of CAREER on all survey datasets.
Training. We randomly divide the resumes dataset into a training set of 23.6 million sequences, and a validation and test set of 23 thousand sequences each. We randomly divide the survey datasets into 70/10/20 aain/test/allddation splits.
The first- and tions counts. We optimize all other models with stochastic gradient descent with minibatches. In total, we use 16,000 total tokens per minibatch, varying the batch size depending on the largest se- quence length in the batch. We use the Adam learning rate scheduler (Kingma & Ba, 2015). All experiments on the resumes data warm up the learning rate from 10-7 to 0.0005 over 4,000 steps, after which the inverse square root schedule is used (Vaswani et al., 2017). For the survey datasets, also used the inverse square root scheduler, but experimented with various learning rates and warmup updates, using the one we found to work best for each model. For CAREER with pre- trained representations,
Markov models without covariates are estimated from empirical transi-
used a learning rate of 0.0001 and 500 warmup updates; for CAREER without pretraining, we used a learning rate of 0.0005 and 500 warmup updates; for the bag of jobs model, we used learning rate of 0.0005 and 5,000 warmup updates; for the regression model, we used a learning rate of 0.0005 and 4,000 warmup updates. We use a learning rate of 0.005 for job representation learning and Job2Vec, with 5,000 warmup updates. All models besides were also trained with 0.01 weight decay. All models were trained using Fairseq (Ott et al., 2019).
When training on resumes, we trained for 85,000 steps, using the checkpoint with the best validation performance. When fine-tuning on the survey datasets, we trained all models until they overfit to the validation set, again using the checkpoint with the best validation performance. We used half precision for training all models, with the exception of the following models (which were only stable with full precision): the bag of jobs model with covariates on the resumes data, and the regression models for all survey dataset experiments.
The tables in Section report results averaged over multiple random seeds. For the results in Fig- ure 2a, the randomness includes parameter initialization and minibatch ordering. For CAREER, we use the same pretrained model for all settings. For the forecasting results in Table 1, the randomness is with respect to the Monte- Carlo sampling used to sample multi year trajectories for individuals. For the wage prediction experiment in Table 2, the randomness is with respect to train/test splits.
Forecasting. For the forecasting experiments, occupations that took place after a certain year are dropped from the train and validation sets. When we forecast on the resumes dataset, we use the same trainatest/validationn split but drop examples that took place after 2014. When we pretrain CAREER the resumes dataset to make forecasts for PSID and NLSY97, we use cutoff year of 2014 as well. We incorporate two-stage prediction into the bascline models because we find that this improves their predictions.
Although we do not include any examples after the cutoff during training, all models require esti- mating year-specific terms. We use the fitted values from the last observed year to estimate these terms. For example, CAREER requires embedding each year. When the cutoff year is 2014, there do not exist embeddings for years after 2014, so we substitute the 2014 embedding.
22

---

Under review as a conference paper at ICLR 2023
We report forecasting results split of the dataset containing examples before and after the cutoff year. To make predictions for an individual, we condition on all observations before the cutoff year, and sample 1,000 trajectories through the last forecasting year. We never condition any occupations after the cutoff year, although we include updated values of dynamic covariates like education. For forecasting on the resumes dataset, we set the cutoff for 2014 and forecast occupations for 2015, 2016, and 2017. We restrict our test set to individuals in the original test set whose first observed occupation was before 2015 and who were observed to have worked until 2017. PSID and NLSY97 are biennial, so we forecast for 2015, 2017, and 2019. We only make forecasts for individuals who have observations before the cutoff year and through the last year of forecasting, resulting in total of 16,430 observations for PSID and 18, 18743 for NLSY97.
Wage prediction. For the wage prediction experiment, we use replication data provided by Blau & Kahn (2017b). We add individual's job histories to this dataset by matching interview and person numbers. We drop individuals that could not be matched, about 3% of the data.
When we apply CAREER to this data to learn representation of job history. we do not any covariates besides the year a job took place. We pretrain a version of CAREER containing 4 layers, 64 dimensions for the representations, 256 hidden units in the feedforward neural networks, and 2 attention heads. We pretrain on resumes for 50,000 steps. We fine-tune to predict jobs on PSID using the job histories of individuals up to the year of interest; for example, for the 201 experiment, we only fine-tune on jobs that took place before 2011. We update parameters every 6 batches when fine-tuning.
After fine-tuning CAREER's 's representations to predict jobs, we plug in the learned representations into the wage regression in Equation 9. Notably, we do not alter CAREER's 's representations to predict wage; we only estimate regression coefficients. We perform an unweighted linear regression. Our model without CAREER uses the same covariates as the wage regression in Blau & Kahn (2017a), including full- and part-time years of experience (and their squares), education, region, race/ethnicity, union status, current occupation, and current industry. We do not include whether an individual is government worker because it results in instability for unweighted regression. Rather than estimate two separate models for males and females, we use a single model and include gender as observed covariate. When incorporate CAREER's 's representations into the model, use the same base model and add CAREER'S representations.
Rationalization. The example in Figure 3 shows an example of CAREER's 's rationale on PSID. To simplify the example, this is the rationale for a model trained on no covariates except year. In order to conceal individual behavior patterns, the example in Figure is slightly altered version of a real sequence. For this example, the transformer used for CAREER follows the architecture described in Radford et al. (2018). We find the rationale using the greedy rationalization method described in Vafa et al. (2021). Greedy rationalization requires fine-tuning the model for compatibility; do this by fine-tuning with "job dropout", where with 50% probability, we drop out a uniformly random amount of observations in the history. When making predictions, the model has to implicitly marginalize over the missing observations. (We pretrain on the resumes dataset without any word dropout). We find that training converges quickly when fine-tuning with word dropout, and the model's performance when conditioning on the full history is similar.
Greedy rationalization typically adds observations maximize the model's likelihood of its top prediction. For occupations, the model's top prediction almost always identical to the previous year's occupation, so we modify greedy rationalization to add the occupation that will maximize the likelihood of its recond-largest prediction. This can be interpreted as equivalent to greedy rationalization, albeit conditioning on switching occupations. Thus, the greedy rationalization procedure stops when the model's 's second-larges prediction from the target rationale equivalent to the model's second-largest prediction when conditioning on the full history.
history one at time in the order that will
23


Reviews:
[{"review_id": "jtYNX1oloe4", "reviewer": "Reviewer_MpmD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a transformer-based model to learn representations of job sequences. The authors first leverage the transformer architecture to fit large-scale resume data, and then finetune the model with smaller, task-specific data, which achieves good performance in predicting job sequences. ", "summary_of_the_review": "This paper leverages transformer in job prediction tasks and achieves well performance. However, some potential baselines in data mining are ignored. The technical contribution is inadequate as a direct application of transformer.", "strength_and_weaknesses": "a.\tStrength:\n1). This paper is well-presented.\n2). The authors propose a transformer-based framework for job prediction, which can effectively take use of large-scale resume data; the pretrain-finetune paradigm is reasonable.\n3). The experiments show that their proposed model can perform well on job prediction task. \n\nb.\tWeaknesses:\n1). Some related work is not mentioned and some potential baselines are missed. Actually person-job fit is quite a mature topic in data mining and information retrieval. The first two baselines (Markov regression and bag-of-jobs) seem to come from econometrics, and the third baseline (NEMO) is quite old in data mining. The authors should investigate more and discover more strong baselines to verify the effectiveness.  \n2). Depside the effectiveness of pretrain-finetune paradigm of transformer architecture, it has been well studied in other tasks such as NLP and CV. Therefore, the technical novelty is inadequate as the paper seems to be an application of transformer in job prediction task.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "95cpEZBsiua", "reviewer": "Reviewer_ZmR7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses the transformer to leverage a sizeable online resume dataset by pretraining and then fine-tuning it on the small and carefully constructed longitudinal survey datasets. According to the results based on their experiments, their approach shows a significant improvement compared to the current state of the arts on the task of job sequence prediction. Besides, they also show that their approach can help a wage model to provide better performance.\n", "summary_of_the_review": "As I mentioned in the previous parts, the authors deal with an interesting problem with the transformer-based models. The main concern regards the technical novelty of the paper, as the authors only made two minor changes to the transformers used in NLP. Moreover, some related studies in IR or data mining are missing, and the compared baselines are quite old.", "strength_and_weaknesses": "Strength\n+ Propose an inspiring method to apply the transformer to the prediction of labor data by pretraining the model on a large online resume dataset and then fine-tuning it on the small datasets.\n+ Conduct comprehensive experiments, including both cross-sectional and overtime experiments, demonstrating the usefulness of the approach.\n+ Well-written paper\n+ Reproducibility: Provide the source code with README and a detailed description of their experiments.\n\nWeakness\n- The main concern regards the technical novelty of the paper: The authors only made two minor changes to the transformers used in NLP.\n- Some related studies are missing. The used baselines are quite old.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4s561Hl-M2", "reviewer": "Reviewer_DtGV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed and developed a clear and detailed transformer-based model called CAREER that uses transfer learning\nto learn representations of job sequences. The CAREER system was pretrained on a dataset of 24 million resumes and it is capable of outperforming standard econometric models for predicting and forecasting occupations. \n\n", "summary_of_the_review": "The paper gave both a good theoretical background and a practical application of the model. However, the deployment for end-user operations was not shown. ", "strength_and_weaknesses": "The paper is well written and very detailed description of both the development and operations of the model was provided. The appendix data was provided to clear the reproducibility concerns. \n\nWeakness: \n1. The deployment of this model was not presented \n2. The claimed incorporation of the model into the wage prediction models was not demonstrated \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject