CAREER: A Foundation Model for Labor Sequence Data
Authors: Keyon Vafa
Emil Palikot
Tianyu Du
Ayush Kanodia
Susan Athey
David Blei
Venue: Accepted by TMLR ()

Abstract: Labor economists regularly analyze employment data by fitting predictive models to small, carefully constructed longitudinal survey datasets. Although machine learning methods offer promise for such problems, these survey datasets are too small to take advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take advantage of their scale or incorporate them into the analysis of survey data. To this end we develop CAREER, a foundation model for job sequences. CAREER is first fit to large, passively-collected resume data and then fine-tuned to smaller, better-curated datasets for economic inferences. We fit CAREER to a dataset of 24 million job sequences from resumes, and adjust it on small longitudinal survey datasets. We find that CAREER forms accurate predictions of job sequences, outperforming econometric baselines on three widely-used economics datasets. We further find that CAREER can be used to form good predictions of other downstream variables. For example, incorporating CAREER into a wage model provides better predictions than the econometric models currently in use.

Published in Transactions on Machine Learning Research (12/2023)
CAREER: A Foundation Model for Labor Sequence Data
Keyon Vafa Harvard University
kwufaag.harvard.comm edu
Emil Palikot Stanford University
Tianyu Du Stanford University
Ayush Kanodia Stanford University
Susan Athey Stanford University
David M. Blei Columbia University
Reviewed OpenReview: http:://peeneeeieeeee////omomodo
Abstract
Labor economists regularly analyze employment data by fitting predictive models to small, carefully constructed longitudinal survey datasets. Although machine learning methods offer promise for such problems, these survey datasets are too small to (ake advantage of them. In recent years large datasets of online resumes have also become available, providing data about the career trajectories of millions of individuals. However, standard econometric models cannot take advantage of their scale or incorporate them into the analysis of survey data. To this end develop CAREER, foundation model for job sequences. CAREER is first fit to large, passively-collected resunne data and then fine-tuned to smaller, better- curated datasets for economic inferences. We fit CAREER to datasct of 24 million job sequences from resumes, and adjust it on small longitudinal survey datasets. We find that CAREER forms accurate predictions of job sequences, outperforming econometric baselines on three widely-used oconomics datascts. We further find that CAREER can be used to form good predictions of other downstream variables. For example, incorporating CAREER into wage model provides better predictions than the econometric models currently in use.
1 Introduction
In labor economies, many analyses rely on models for predicting an individual's future occupations. These models are crucial for estimating important economic quantities, such as gender or racial differences in unemplommnn (Hall, 1972; Fairlie & Sundstrom, 1999); they underpin causal analyses and decompositions that rely on simulating counterfactual occupations for individuals (Brown et al., 1980; Schubert et al., 2021); and they also inform policy, by forecasting occupations with rising or declining market shares. More accurate predictive models opens the door to more reliable estimates these important quantities.
In this paper we develop a representation-leannie method a transformer adaptod for modeling jobs for building such predictive models of occupation. Our model is first fit. to large-scale, passively-collected resume

---

Published in Transactions on Machine Learning Research (12/2023)
data and then fine-tuned to more curated economics datasets, which are carefully collected for unbiased generalization to the larger population. The representation it learns is effective both for predicting job trajectories and for conditioning in downstream economic analyses.
In the past, labor economics analyses only involved fitting predictive models to small datasets, specifically lon- gitudinal surveys that follow a cohort of individuals during their working career (Panel Study of Income Dy- namics, 2021; Bureau of Labor Statistics, 2019a). Such surveys have been carefully collected represent na- tional demographics, ensuring that the economic analyses can generalize, but they are also very small, usually containing only thousands of workers. As consequence, prior models of occupation trajectories have been based on very simple sequential assumptions, such as where a worker's next occupation depends only on their most recent occupation (Hall, 1972) or a few summary statistics about their past (Blau & Riphahn, 1999).
In recent years, however, much larger datasets of online resumess have also become available. These datasets contain the occupation historics of millions of individuals, potentially revealing complex information and patterns about career trajectories. But, while one might hope these datasets can improve our economic analyses, there are fundamental difficulties to using them. First, they are passively collected and likely represent a biased sample of the population. Second, they are noisy, since the occupation sequences in the data are derived from text analysis of raw resumes. Finally, they generally omit important economic variables such as demographics and wage, which are essential for the kinds of quantities that economists would like to estimate.
To overcome these challenges, we develop CAREER, a machine learning model of occupation trajectories. CAREER is a foundation model (Bommasani et al., 2021): it learns an initial representation of job history from large-scale resume data that is then adjusted the transformer archilecture (Vaswani et al., 2017), which underpins foundation models in natural language processing (Devlin et al., 2019; Radford et al., 2019) and other fields, such as music generation (Copet et al., 2023) and coding (Li et al., 2023). We will show that CAREER's representations provide effective predictions of occupations on survey datasets used for economic analysis, and can be used as inputs to economic models for other rownsttream applications.
downstream survey datascts. CAREER is based on
To study this model empirically, we pretrain CAREER dataset of 24 million passively-collectedd re- sumes provided by Zippia, career planning company. We then fine-tune CAREER's representations of job sequences to make predictions on three widely-used economic datasets: the National Longitudinal Survey of Youth 1979 (NLSY79), another cohort from the same survey (NLSY97), and the Panel Study of Income Dynamics (PSID). In contrast to resume data, these well-curated datasets are representative of the larger population. It with these survey datasets that economists make inferences, ensuring their analyses gen- eralize. In this study, we find that CAREER outperforms standard econometric models for predicting and forecasting occupations on these survey datasets.
We further find that CAREER can be used to form good predictions of other downstream variables; incor- porating CAREER into a wage model provides better predictions than the cconometric models currently in use. We release code so that practitioners can train CAREER for their own problems.
In summary, we demonstrate that CAREER can leverage large-scale resume data to make accurate predic- tions on important datasets from economics. Thus CAREER ties together economic models for understand- ing career trajectories with transformer-basel foundation models. (See Section for details of related work.)
2 CAREER
Given an individual's career history, what is the probability distribution of their occupation in the next timestep? We go over a class of models for predicting occupations before introducing CAREER, one such model based on transformers and transfer learning.
/ thub. com//eeynnafal/carer--com
2

---

Published in Transactions on Machine Learning Research (12/2023)
2.1 Occupation Models
Consider an individual worker. This person's career can be defined as series of timesteps. Here, use timestep of one year. At each timestep, this individual works in a job: it could be the same job as the previous timestop, or a different job. (Note we use the terms "occupation" and "job" synonymously.) We consider "unemplyedd" and "out-of-labor-foree to be special types of jobs.
Define an occupation model to be probability distribution over sequences of jobs. An occupation model predicts worker's job at each timestep as function of all previous jobs and other observed characteristics of the worker.
More formally, define an individual's career to be a sequence (Ui,..., Ur), where each yr E (1.....J] J) indexes one of J occupations at time t. Occupations are categorical; one example of a sequence could be ("cashier", "salesperson",
"sales manager)). At each timestep, an individual also associated with C' observed covariates 2t = ( Covariates are also categorical, with Ite E (I...., .N). For example, if e corre- sponds to the most recent educational degree, Ite could be "high school diploma" or "bachelors", and Ne is the number of lypes of educational degrees.? Define y+ = (11)...,33)) to index all jobs that have occurred up to time t, with the analogous definition for X
At each timestep, an occupation model predicts an individual's job in the next timestep, P(8. This distribution conditions on covariates from the same timestep because these are "pre-transition. " For example, an individual's most recent educational degree is available to the model as it predicts their next job.
Note that an occupation model is a predictive rather than structural model. The model does not incorporate unobserved characteristics, like skill, when making predictions. Instead, it implicitly marginalizes over these unobserved variables, incorporating them into its predictive distribution.
2.2 Representation- Based Two-Stage Models
An occupation model's predictions are governed by an individual's career history; both whether an individual changes jobs and the specific job they may transition to depend on current and previous jobs and covariates.
We consider a class of occupation models that make predictions by conditioning on a low-dimensional representation of work history, he(yt-1, E R.. This representation is assumed to be sufficient statistic of the past; (y)-1,, x:) should contain the relevant observed information for predicting the next job.
Since individuals frequently stay the same job between timesteps, we propose a class of models that make predictions in two stages. These models first predict whether an individual changes jobs, after which they predict the specific job to which an individual transitions. The representation is used in both stages.
In the first stage, the career representation he(y:-1,x,) is used to predict whether an individual changes jobs. Define the binary variable 8, to be 1 if a worker's job at time / is different from that at time L - 1, and 0 otherwise. The first stage is modeled by
St y ~ Bernoulli (a(1) x:))),
(1)
where (-)) the logistic function and r) E DR. vector of coefficients.
If the model predicts that an individual will transition jobs, it only considers jobs that are different from the individual's 's most recent job. To formulate this prediction, it combines the career representation with a vector of occupation-specifee coefficients B, E R:
exp( hr hr 1, x1)) p(u+ = jly 1) 11 Σ Ej expl h4 hy(y
(2)
Otherwise, the next job is deterministic:
plue 81y 8t 00 =6 8j=ύ-11
(3)
Same covariates may evolve over time. We encode them as time-varying without loss generality.
3

---

Published in Transactions on Machine Learning Research (12/2023)
Two-stage prediction improves the accuracy of occupation models. Moreover, many analyses of occupational mobility focus on whether workers transition jobs rather than the specific job they transition to (Kambourov & Manovskii, 2008). By separating the mechanism by which a worker either keeps or changes jobs (17) and the specific job they may transition to (3z), two-stage models are more interpretable for studying occupational change.
Equations to 3 define two-stage representation- based occupation model. In the next section, we introduce CAREER, one such model based on transformers.
2.3 CAREER Model
We develop a two-stage representation-basel occupation model called CAREER. 3 transformer to parameterize a representation of an individual's history. CAREER is a foundation model: it is pretrained on a large resumes datasct and fine-tuned to make predictions on small survey datasets.
This model uses a
Transformers. transformer is sequence model that uses neural networks to learn representations of discrete tokens (Vaswani et al., 2017). Transformers (NLP), to predict words in sentence. Transformers are able to model complex dependencies between words, and they are a critical component of large language models (Radford et al., 2018; Devlin et. al., 2019; Radford et al., 2019; Brown et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023).
originally developed for natural language processing
CAREER is an occupation model that uses a transformer lo parameterize a low-dinensional representation of careers. While transformers were developed model sequences of words, CAREER uses transformer to model sequences of jobs. The transformer enables the model to represent complex carcer trajectorios.
CAREER is similar to the transformers used in NLP, but with two modifications. First, as described in Section 2.2, the model makes predictions in two stages, making it better-suited to model workers who stay in the same job through consecutive timesteps. (In contrast, words seldom repeat.) Second, while language models only condition on previous words, each career also associated with covariates x that may affect transition distributions (see Equation 2). We adapt the transformer to these two changes.
Parameterization. CAREER's computation graph is depicted in Figure 1. Note that in this section provide simplified description of the ideas underlying the transformer. Appendix C contains full description of the model.
CAREER iteratively builds a representation of career history, he(y,-1, X,) E RDD, using a stack of L layers. Each layer applies a series computations to the previous layer's output to produce its own layer-specific reprosentation. The first layer's representation, h, (1)) (y--1x)),,, considers only the most recent job and co- variates. At each subsequent layer É, the transformer forms representation h! (y)-,x)) by combining the representation of the most recent job with those of preceding jobs. Representations become increasingly complex at each layer, and the final layer's representation, h (1(2)) (yt-1, xt), is used to make predictions fol- lowing Equations to 3. We drop the explicit dependence on yt-1 and x, going forward, and instead denote each layer's representation as (
The first layer's representation combincs the previous job, the most recent covariates, and the position of the job in the career. first embeds each of these variables in D-dimensional space. Define an embedding function for occupations, ey [J] + RRD Additionally, define a separate embedding function for each covariate, {e,)am with each ee [No] + R.. Finally, define e, [T] + RD to embed the position of the sequence, where 7 denotes the number of possible sequence lengths. The first-layer representation (1) sums these embeddings:
n cy((1-1) + + ec((te) ett)
(4)
CAREER short. for "Contextual Attention-based Representations of Emplayment Encoded from Resumes."
4

---

Published in Transactions on Machine Learning Research (12/2023)
Transformer Layers
1
(1)
h
(2)
81
Career History
Lt-
y1
1 (L)
St-
Current Job and Covariates
Lt
yt
14
(1)
h
(L)
St
Next Job
Two-Stage Variable
Figure 1: CAREER's computation graph. CAREER parameterizes low-dimensional representation of an individual's career history with transformer, which it uses to predict the next job.
For each subsequent layer the transformer combines representations of the most recent job with those of the preceding jobs and passes them through neural network:
"h. exp
(E)
wonn }
(0) for all e' st
(5)
1, n/ Et= Tie * *
(6)
n ((+1) FFN(O) (7
(7)
where W(O E I DxD is model parameter and FFN is two-layer feodforward neural network specific layer l, with FFN(O) RDD - RD.
The weights { are referred to as allention weights, and they are determined by the career represen- tations and W(t), The attention weights non-negative and normalized to sum to 1. The matrix ((()) can be interpreted as a similarity matrix; if W(O is the identity matrix, occupations t and t' that have similar representations will have large attention weights, and thus would contribute more to the weighted average in Equation Conversely, if ( is the negative identity matrix, occupations that have differing representations will have large atlention weights. The final computation of each layer involves passing the intermediate representation (0) through a neural network, which ensures that representations capture complex nonlinear interactions.
The computations in Equations 7 are repeated for each of the L layers. The last layer's representation is used to predict the next job:
pludy:-11, x:) = two-stage-softmax (2)
;n. :n.)).
where "two-stage-softmax"" refers to the operation Equations 1 to 3, parameterized by and 3.
All of CAREER's paramcters including the embedding functions, similarity matrices, food-forward neural networks, and regression coefficients 7) and are estimated by maximizing the likelihood in Equation 8 with stochastic gradient descent. (SGD), marginalizing out the latent variable 8t-
Transfer Icarning. Economists build occupation models using survey datasets that have been carefully collected to represent national demographics. In the United States, these datasets contain a small number
In practice, transformers multiple attention weights perform multi-headed attention (Appendix C).
5

---

Published in Transactions on Machine Learning Research (12/2023)
of individuals. While transformers have been successfully applied to large NLP datasets, they are prone to overfitting on small datasets (Kaplan et al., 2020; Dosovitskiy et al., 2021; Variš & Bojar, 2021). As such, CAREER may not learn useful representations solely from small survey datasets.
In recent years, however, much larger datasets of online resumes have also become available. Although these passively-collectedd datasets provide job sequences of many more individuals, they are not used for economic estimation for a few reasons. The occupation sequences from resumes are imputed from short textual descriptions, process that inevitably introduces more noise and errors than collecting data from detailed questionnaires. Additionally, individuals may not accurately list their work experiences on resumes (Wexler, 2006). and important cconomic variables relating to demographics and wage are not available. Finally, these datasets are not constructed to ensure that they are representative of the general population.
Between these two types of data is a tension. On the one hand, resume data is large-scale and contains valuable information about employment patterns. On the other hand, survey datascts are carcfully collectod, designed to help make economic inferences that are robust and generalizable.
Thus CAREER incorporates the patterns embedded in large-scale resumee data into the analysis of survey datasets. It does this through transfer learning: CAREER is first pretrained on a large dataset of resumes to learn an initial representation of careers. When CAREER. is then fit to small survey dataset, parameters are not initialized randomly; instead, they are initialized with the representations learned from resumes. After initialization, all parameters are fine-tuned on the small dataset by optimizing the likelihood. Because the objective function is non-convex, learned representations depend on their initial values. Initializing with the pretrained representations ensures that the model does not need to re-learn representations on the small dataset. Instead, it only adjusts representations to account for datasct differences.
CAREER is designed to be a foundation model (Bommasani et al., 2021): after pretraining on large-scale resume data, CAREER can be transferred to downstream survey datasets. CAREER takes inspiration from foundation models developed in other domains, such as NLP (Devlin et al., 2019; Radford et al., 2018; 2019; Brown et al., 2020), image generation (Ramesh el al., 2021), music generation (Copet et al., 2023), and coding (Li al., 2023). For example, in NLP, transformers are pretrained on large corpora, such as unpublished books or web crawl data, and then fine-tuned to make predictions on small datasets such movie reviews (Devlin et al., 2019). Our approach is analogous. Although the resumes dataset may not be representative or carefully curated, it contains many more job sequences than most survey datasets. This volume enables CAREER to learn representations that transfer to survey datasets.
3 Related Work
Many economic analyses use log-linear models to predict jobs in survey datasots (Boskin, 1974; Schmidt & Strauss, 1975). These models typically use small state spaces consisting of only a few occupation categories. For example, some studies categorize occupations into broad skill groups (Keane & Wolpin, 1997; Cortes, 2016); unemployment analyses only consider employment status (employed, unemployed, and out-of-labor-foree mobility only consider occupational change, a binary variable indicating whether an individual changes jobs (Kambourov & Manovskii, 2008; Guvenen et al., 2020). Although transitions between occupations may depend richly on history, many of these models condition on only the most recent job and a few manually constructed summary statistics about history to make predictions (Hall, 1972; Blau & Riphahn, 1999). Tn contrast to these methods, CAREER is nonlincar and conditions on every job in an individual's history. The model learns complex representations of careers without relying on manually constructed features. Moreover, CAREER. can effectively predict from among hundreds of occupations.
(Hall, 1972; Lauerova & Terrell, 2007); and researchers studying occupational
Recently, the proliferation of business networking platforms has resulted in the availability of large resume datasets. Schubert et al. (2021) use a large resumne dataset to construct a first-order Markov model of job tran- sitions; CAREER, which conditions on all jobs in a history, makes more accurate predictions than a Markov model. Models developed in the data mining community rely on resumo-specífic features such as stock prices (Xu et al., 2018), worker skill (Ghosh et al., 2020), network information (Meng et al., 2019; Zhang et al., 2021), and textual descriptions (He et al., 2021), and are not applicable to survey datasets, as our goal in this pa-
6

---

Published in Transactions on Machine Learning Research (12/2023)
per (other models reduce to first-order Markov model without these features (Dave al., 2018; Zhang et al., 2020)). The most suitable model for survey datasets from this line of work is NEMO, an LSTM-based model that is trained on large resume datasets (Li et al., 2017). In contrast to these approaches, our primary concern is not modeling resume datasets directly, but rather modeling survey datasets. Our experiments demonstrate that a transfer learning approach, from resume data to survey data, improves modeling performance.
Recent work in econometrics has applied machine learning methods to sequences of jobs and other discrete data. In the context of consumer grocery shopping behavior, Ruiz al. (2020) and Donnelly et al. (2021) introduce method called SHOPPER, based matrix factorization, for creating embeddings of grocery products. SHOPPER can be thought of as foundation model, and Donnelly et al. (2021) show that the approach of building embeddings using data from unrelated product categories improves predictions for a given category relative to traditional economic models focused only on one category, while Ruiz et al. (2020) expand the setup to predict the next product in a sequence added to a shopping basket. One of the basclines we consider in this paper is a "bag-of-jobs" model similar to SHOPPER. Like the transformer- basod model, the bag-of-jobs model conditions on every job in an individual's history, but it uses relatively simple representations of careers. Our empirical studies demonstrate that CAREER learns complex representations that are better at modcling job sequences. Rajkumar et al. (2021) build on SHOPPER and propose a Bayesian factorization method for predicting job transitions. Similar to CAREER, they predict jobs in two stages. However, their method only conditions on the most recent job in an individual's history. In our empirical studies, we show that models like CAREER that condition on every job in an individual's history form more accurate predictions than Markov models.
CAREER is foundation model based transformer, an architecture initially developed to represent se- quences of words in natural language processing (NLP). In econometrics, transformers have been applied to the text of job descriptions to predict their salaries (Bana, 2021) or authenticity (Naudé et al., 2022); rather than modeling text, we use transformers to model sequences of occupations. Transformers underpin founda- tion models developed for text (Devlin ct al., 2019; Radford et al., 2018; 2019; Brown et al., 2020) and other domains such as image generation (Ramesh el al., 2021), music generation (Copet et al., 2023), and coding (Li et al., 2023). Occupational trajectories have simpler structure than the data modeled in these other domains; for example, per-year career trajectories are shorter than the text sequences used to train large language models, and there are less possible occupations than words in vocabulary. Thus, foundation models for occupations may not require much data as those in other domains. Indeed, CAREER is smaller than foun- dation models in other domains and uses less data, but still forms effective predictions on downstream tasks.
Our paper also contributes to the growing field of transfer learning in economics. Most existing transfer learning approaches in the economics literature have focused on fine-tuning existing models on image or text data, where the effectiveness of transfer learning is better understood. For example, one existing line of work has fine-tune pretrained convolutional neural networks satellite images in order to predict economic indicators (Price& & Atkinson, 2022; Yeh et al., 2020; Persello & Kuffer, 2020), while another has fine-tuned BERT on financial and economic text (Araci, 2019; Bana, 2021). Our work differs from these papers in that we focus on a new domain (job sequences on survey data instead of images or text) and also identify a novel pretraining corpus that helps with this task (large-scale, passively-colleee collected resume data).
4 Empirical Studies
We assess CAREER's ability to predict jobs and provide useful representations of careers. We pretrain CAREER on a large dataset of resumes, and transfer these representations to small, commonly used survey datasets. We study straightforward prediction (predicting jobs in a career with random train and test set splits) and forecasting (training on jobs before certain year and predicting on future years). With the transferred representations, the model is better than econometric baselines at both prediction and forecast- ing. Additionally, we demonstrate that CAREER's s representations can be incorporated into standard wage prediction models to make better predictions.
Resume pretraining. We pretrain CAREER on a large dataset of resumes provided by Zippia Inc., a career planning company. This dataset contains resumes from 23.7 million working Americans. Each job is encoded into one of 330 occupational codes, using the coding scheme of Autor & Dorn (2013). We transform
7

---

Published in Transactions on Machine Learning Research (12/2023)
resumes into sequences of jobs by including occupation's code for each year in the resume. For years with multiple jobs, we take the job the individual spent the most time in. We include three covariates: the year each job in an individual's career took place, along with the individual's state of residence and most recent educational degree. We denote missing covariates with a special token. (Appendix D contains an exploratory data analysis of the resumee data.) CAREER uses a 12-layer transformer with 5.6 million parameters. Pretraining CAREER on the resumes data takes 18 hours on single GPU. (See Appendix G for more details on the model and hyperparameters.)
Survey datasets. We transfer CAREER to three widely-used survey datasets: two cohorts from the National Longitudinal Survey of Youth (NLSY79 and NLSY97) and the Panel Study of Income Dynamics (PSID). These datasets have been carefully constructed to be representative of the general population, and they are widely used by economists for estimating economic quantities. NLSY79 is a longitudinal panel survey following a cohort of Americans who were between 14 and 22 when the survey began in 1979, while NLSY97 follows a different cohort of individuals who were between 12 and 17 when the survey began in 1997. PSID is a uddgal survey following a sample of American families, with individuals added over the years. These surveys are publicly available and allow for linking individuals over time. See Appendix F for more information about how the data is formed.
Compared to the resumes dataset, these survey datasets are small: we use slices of NLSY79, NLSY97, and PSID that contain 12 thousand, 9 thousand, and 12 thousand individuals, respectively. We convert all datasets so that occupations follow the occ1990dd encoding (Autor & Dorn, 2013), which contains 330 discrete occupations. Both the resume dataset and survey datasets contain education, location, and year covariates, while the survey datasets additionally include demographic variables (gender and race/ethnicity). The distribution of job sequences in resumes differs in meaningful ways from those in the survey datasets; for example, manual laborers are under-represented and college graduates are over-represented in resume data. Appendix D includes more details for how these datasets differ.
We pretrain CAREER on the large resumee dataset and fine-tune on the smaller survey datasets. We fine-tune by initializing each model's weights with those of the pretrained model. The fine-tuning process is efficient; although CAREER has 5.6 million parameters, fine-tuning on one GPU takes 13 minutes on NLSY79, minutes on NLSY97, and 23 minutes on PSID.
We compare CAREER to several baseline models: second-order linear regression with covariates and hand-constructed summary statistics about past employment; a bag-of-jobs model inspired by SHOPPER (Ruiz et al., 2020) that conditions on all jobs and covariates in a history but combines representations linearly; and several baselines developed in the data-mining community for modeling worker profiles: NEMO (Li et al., 2017), job representation learning (Dave et al., 2018), and Job2Voc (Zhang et al., 2020). As described in Section 3, the baselines developed in the data-mining community for modeling worker profiles cannot be applied directly to economic survey datasets and thus require modifications, described in detail in Appendix G. These baselines are all trained on survey data. We also compare to two additional versions of CAREER - one without pretraining or two-stage prediction, the other only without two-stage prediction to assess the sources of CAREER's improvements. All models use the covariates we included for resume pretraining, in addition to demographic covariates (which are recorded for the survey datasets but are unavailable for resumes).
We divide all survey datasets into 70/10/20 train /aaavvaaaaiineeesss splits, and train all models by optimizing the log-likelihood with Adam (Kingma & Ba, 2015). We evaluate the predictive performance of each model by computing held-out perplexity, a common metric in NLP for evaluating probabilistic sequence models. The perplexity of a sequence model p on a sequence y1..... VT is exp{- Ei= log p())) It is a monotonic transformation of log-likelihood; better predictive models have lower perplexities. We train all models to convergence and use the checkpoint with the best validation perplexity. Detailed experimental settings and training details are described in Appendix G.
Figure 2a compares the test-set perplexity of each model. With the transferred representations, CAREER makes the best predictions on all survey datasets. The baselines developed in the data mining literature, which were designed to model large resume datasets while relying on resume-specific features, struggle to make good predictions on these small survey datasets, performing on par with standard econometric baselines.
8

---

Published in Transactions on Machine Learning Research (12/2023)
PSID
NLSY79
NLSY97
12.0
Markov regression (Hall, 1972)
18.97 #0.10
15.03 +0003
20.81
#0.02
Perplexity
11.9
NEMO (Li et al., 2017)
17.58 10.04
12.82 10.04
18.38 10.08
11.8
Job rep. learning (Dave al., 2018)
17.23 10.16
14.71 10.02
16.83 10.03
11.7
Joc2 (Zhang et al., 2020)
16.48 10.13
14.46 10.01
16.20 10.02
11.6
Bag-of-joba (Ruiz ct al., 2020)
16.21 #0.08
13.09 +0.03
16.20 #0.01
11.5
CAREER (vanilla)
15.26 +0.08
12.20 #0.04
16.19 #0.04
11.4
No Pretraining
CAREER (two-stage)
14.79 #0.04
12.00 #0.00
15.22 =0.03
11.3
10"
10" 8 10'
CAREER (two-stage + pretrain)
13.88 10.01
11.32 0.00
14.15 10.03
Pretraining Examples
(a) Test perplexity on survey datasets (lower is better). Results are averaged over three random seeds. CAREER (vanilla) includes covariates but not two-stage prediction pretraining; CAREER (two-stage) adds two-stage prediction.
(b) CAREER's scaling law on NLSY79 as a function pre- training data volume.
Figure 2: Prediction results on longitudinal survey datasets and scaling law.
Pretraining is the biggest source of CAREER's improvements. Although the resumee data is noisy and differs in many ways from the survey datasets used for economic prediction, CAREER learns reprosentations of work experiences that aid its predictive performance. In Appendix E we show that modifying the baselines to incorporate two-stage prediction (Fquations to 3) improves their performance, although CAREER still makes the best predictions across datasets. We include further qualitative analysis of CAREER's predictions in Appendix B.
To assess how the volume of resumes used for pretraining affects CAREER's predictions on survey datasets, we downsample the resume dataset and transfer survey datasets. The scaling law for NLSY79 is depicted in Figure 2b. When there are less than 20,000 examples in the resume dataset, pretraining CAREER does not offer any improvement. The relationship between raiting volume and fine-tuned perplexity follows a power law, similar to scaling laws in NI. (Kaplan et al., 2020).
We also assess CAREER's 's ability to forecast future career trajectories. In contrast predicting held-out sequences, forecasting involves training models on all sequences before specific year. To predict future jobs for an individual, the fitted model is used to estimate job probabilities six years into the future by sampling multi-year trajectories. This setting is useful for assessing model's ability to make long-term predictions, especially as occupational trends change over time.
We evaluate CAREER's forecasting abilitics on NLSY97 and PSID. (These datasets are more valuable for forecasting than NLSY79, which follows a cohort that is near or past retirement age.) We train models on all sequences (holding out 10% as a validation set), without including any observations after 2014. When pretraining CAREER on resumes, we also make sure to only include examples up 2014. Table and Table 2 compare the forecasting performance of all models (results are averaged over three random seeds). CAREER makes the best overall forecasts. CAREER has significant advantage over baselines at making long-term forecasts, yielding a 17% advantage over the best baseline for 6-year forecasts on NLSY97. Again, the baselines developed for resumne data mining, which had been developed to model much larger corpora, struggle to make good predictions on these smaller survey datasets.
Using CAREER's roprosentations to predict wage. In addition to forming job predictions, CAREER. learns low-dinensional representations of job histories. These representations can be used as inputs to economic models for downstream applications.
As an example of how CAREER's representations can be incorporated into other economic models, we use CAREER to predict wages. Economists build wage prediction models in order to estimate important economic quantities, such as the adjusted gender wage gap (Blau & Kahn, 2017), wage effects in labor market discrimination (Ncal & Johnson, 1096), and union wage premiums (Linneman et al., 1990). For example, to estimate the adjusted gender wage gap, Blau & Kahn (2017) regress an individual's log- wage on observable characteristics such as education, demographics, and current occupation PSTD. Rather than including
9

---

Published in Transactions on Machine Learning Research (12/2023)
Overall
2- Y-aear
Year
Year
Markov regression
23.11 0.03
12.50 + 0.00
25.88 + 0.04
36.59 0.07
Bag-of-jobs
22.51 + 0.03
11.98 0.00
25.11 0.04
36.29 0.10
NEMO
25.26 + 0.02
12.59 + 0.00
28.35 + 0.05
13.01 0.05
CAREER
19.41 0.04
10.78 0.00
21.57
0.07
30.19 0.06
Table 1: Forecasting perplexity (lower is better) on NLSY97.
Overall
Year
Year
6- Year
Markov regression
19.43 + 0.01
11.83 + 0.00
21.66 + 0.03
27.89 + 0.01
Bag-of-jobs
19.28 0.01
11.44 0.00
21.6800.00
28.14 + 0.05
NEMO
18.58 + 0.01
11.08 = 0.00
20.67
27.29 + 0.04
CAREER
16.51 0.01
10.35 1 0.00
18.30 1 0.01
23.18 + 0.03
Table 2: Forccasting perplexity (lower is better) on PSID.
the full, high-dimensional job-history, the model summarizes individual's career with summary statistics such full-time and part-time years of experience (and their squares).
We incorporate CAREER's reprosentation into the wage regression by adding the fitted representation for an individual's job history, h4. For log-wage my and observed covariates Ty we regress
W1 a + 0 + his
(9)
where a, 0, and are regression coefficients. CAREER's representations are trained to model occupation sequences on 80% of the samples PSID, and are then frozen on the held-out samples; this ensures that the representations do not encode an individual's future occupations. To estimate regression coefficients, we again split the sample into five splits and evaluate R² on each held-out split, averaging over splits. Wo estimate standard errors over three random training seeds.
Table shows that adding CAREER's representations improves wage predictions for each year. Although these representations trained to model occupational sequences and are not adjusted to account for wage, they contain information that is predictive of wage. Moreover, pretraining is important to CAREER's predictive advantage. That models with these representations outperform simpler models in predictive performance, both for occupation modeling (the objective the representations are trained to optimize) and wage prediction (which the representations are not trained to optimize), underscores the complexity of the learned representations.
5
Conclusion
We introduced CAREER, a foundation model for occupational sequences. CAREER first trained on large-scale resume data and then fine-tuned on smaller datasets interest. When we fine-tuned CAREER common survey datasets in labor economics, outperformed econometric baselines predicting and forecasting career outcomes. We demonstrated that CAREER's representations could be incorporated into wage prediction models, outperforming standard econometric models.
Overall
1990-1999
2000 2009
2010 2019
Full specification from Blau & Kahn (2017)
0.431 +: 0.000
0.424 + 0.000
0.392 +: 0.000
0.459 t 0.000
Full specification + CAREER (not pretrained)
0.493 0.000
0.498 + 0.000
0.453
0.000
0.517 + 0.000
Full specification + CAREER (pretrained)
0.504 0.001
0.504 + 0.001
0.461 0.000
0.532 + 0.000
Table 3: Held-out R² (higher is better) for wage regressions.
10

---

Published in Transactions on Machine Learning Research (12/2023)
One direction of future research is to incorporate CAREER's representations of job history into methods for est estimating adjusted quantities, like wage gaps. Underlying these methods are models that predict economic outcomes as function of observed covariates. However, if relevant variables are omitted, the adjusted esti- mates may be affocted; c.g., excluding work experience from wage prediction may change the magnitude of the estimated gap. practice, economists include hand-designed summary statistics to overcome this prob- lem, such as Blau Kahn (2017). CAREER provides data-driven way to incorporate such variables- its representations of job history could be incorporated into downstream prediction models and lead to more accurate adjustments of economic quantities.
Broader Impact Statement
As discussed, passively-collected resume datasets are not curated represent national demographics. Pre- training CAREER these datasets may result in representations that are affected by sampling bias. Al- though these ionsssenlations are fine-tuned survey datasets that are carefully constructed to represent national demographics, the biases from pretraining may propagate through fine-tuning (Ravfogel et al., 2020; Jin et al., 2021). Morcover, even in reprosentative datasets, models may form more accurate predictions for majority groups due to data volume (Dwork et al., 2018). Thus, we encourage practitioners to audit noisy resume data, re-weight samples as necessary (Kalton, 1983), and review accuracy within demographics before using the model for downstream economic analysis.
Although resume datasets may contain personally identifiable information, all personally identifiable in- formation had been removed before we were given access to the resume dataset we used for pretraining. Additionally, none of the longitudinal survey datasets contain personally identifiable information.
Acknowledgements
This work is supported by NSF grant IIS 21 2127869, ONR grants N00014-17-1-2131 and N00014-15-1-2200 the Simons Foundation, Open Philanthropy, and the Golub Capital Social Impact Lab. Keyon Vafa is supported by the Harvard Data Science Initiative Postdoctoral Fellowship. We thank Elliott Ash, Sune Lehmann, Suresh Naidu, Malthew Salganik, and the reviewers for their thoughtful comments and suggestions. We thank Zippia for generously sharing the resume datasct. We also thank the Stanford Institute for Human- Centered Artificial Intelligence. Finally, we thank Lilia Chang, Karthik Rajkumar, and Lisa Simon upon whose research build in this project, and especially Lisa Simon who helped obtain the data and encourage this line of research.
References
Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models. urXiv:1908.1006; 2019.
arXiv preprint
David Autor and David Dorn. The growth of low-skill service jobs and the polarization of the U.S. labor market. American Economic Review, 103(5):1553 97, 2013.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450, 2016.
Sarah H Bana. Using language models to understand wage premia. 2021.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155, 2003.
David M Blau and Regina T Riphahn. Labor force transitions of older married couples in Germany. Labour Economics, 6(2):229 252, 1999.
Francinc D Blau and Lawrence M Kahn. The gender wage gap: Extent, trends, and explanations. Journal of Economic Literature, 55(3):789- 865, 2017.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, al. On the opportunities and risks of foundation models. Xiu preprint ar Xr:i::2108. 0725888 2021.
11

---

Published in Transactions on Machine Learning Research (12/2023)
Michael J Boskin. A conditional logit model of occupational choice. Journal of Political Economy, 82((, Part 1):389-395 1974.
Randall S Brown, Marilyn Moon, and Barbara S Zoloth. Incorporating occupational attainment in studies of male-female carnings differentials. Journal of Human Resources, 15(1):3-28 1980.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Newral Information Processing Systems, 2020.
Bureau of Labor Statistics. National Longitudinal Survey of Youth 1979 cohort (rounds 1-27). Produced and distributed by the Center for Human Resource Research (CHRR), The Ohio State University. Columbus, OH, 2019a.
Bureau of Labor Statistics. National Longitudinal Survey of Youth 1997 cohort (rounds 1-19). Produced and distributed by the Center for Human Resource Research (CHRR), The Ohio State University. Columbus, OH, 2019b.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défosscz. Simple and controllable music generation. arXiv preprint arXiv:2306. XrXi:2206.05284, 2023.
Guido Matias Cortes. Where have the middle-wage workers gone? A study of polarization using panel data. Jowrnal of Labor Economics, 34(1):63 105, 2016.
Vachik Dave, Baichuan Zhang, Mohammad Al Hasan, Khalifeh AlJadda, and Mohammed Korayem. A combined representation learning approach for better job and skill recommendation. In ACM Conference on Information and Knowledge Management, 2018.
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Robert Donnelly, Francisco JR Ruiz, David Blei, and Susan Athey. Counterfactual inference for consumer choice across many product categories. Quantitative Marketing and Economics, pp. 1-39, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. In International Conference on Learning Rep- resentations, 2021.
Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. Decoupled classifiers for group- fair and efficient machine learning. In Conference on Fairness, Accountability and Transparency, 2018.
Robert W Fairlie and William Sundstrom. The emergence, persistence, and recent widening of the racial unemployment gap. Industrial and Labor Relations Review, 52(2):252-270, 1999.
Aritra Ghosh, Beverly Woolf, Shlomo Zilberstein, and Andrew Lan. based career path modeling and recommendation. In IEFE International Conference on Big Data, 2020.
Fatih Guvenen, Burhan Kuruscu, Satoshi Tanaka, and David Wiczer. Multidimensional skill mismatch. American Economic. Journal: Macroeconomics, 12(1):210-44, 2020.
Robert E Hall. Turnover in the labor force. Brookings Papers on Economic Activity 1972(3):709- 1972.
Miao He, Xiaoming Zhan, Dayong Shen, Yuanyuan Zhu, Hua Zhao, and Renjie He. What about your next job? Predicting professional career trajectory using neural networks. In Machine Learning and Machine Intelligence, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). 1rXix:6666.00 1606608415, 2016.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15550 2022.
Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. On transfcrability of bias mitigation effects in language model fine-tuning. In North American Chapter of the Association for Computational Linguistics, 2021.
Graham Kalton. Compensating for Missing Survey Data. Survey Research Center, Institute for Social Research, the Universily of Michigan, 1983.
Gueorgui Kambourov and Iourii Manovskii. Rising occupational and industry mobility in the United States: 1968-97, 97. International Economic Review, 49(1):41-77) 2008.
Jared Kaplan, Sam McCaudlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.00 083611 2020.
Michael P Keane and Kenneth Wolpin. The career decisions of young men. Journal of Political Economy, 10743):473 522, 1997.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jana Stefanova Lauerova and Katherine Terrell. What drives gender differences in uemployment? Com- parative Economic Studies, 49(1):128-155, 2007.
Liangyne Li, How Jing, Hanghang Tong, Jacwon Yang, Qi He, and Bee-Chung Chen. NEMO: Next career move prediction with contextual embedding. In World Wide Web Conference, 2017.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXio:2305.061 2023.
Peter D Linneman, Michael L Wachter, and William H Carter. Evaluating the evidence on union employment and wages. ILR review, 44(1):34 53, 1990.
Brian P McCall. Occupational matching: A test of sorts. Journal of Political Economy, 98(1))45-69, 1990.
Qingxin Mong, Hengshu Zhu, Keli Xiao, Le Zhang, and Hui Xiong. A hicrarchical carcer-path-aware neural network for job mobility prediction. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
Tomas Mikolov, Nya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Neural Information Processing Systems, 2013.
Marcel Naudé, Kolawole John Adebayo, and Rohan Nanda. A machine learning approach to detecting fraudulent job types. AT & Society, pp. 1-12, 2022.
Derek Neal. The complexity of job mobility among young men. Journal of Labor Economics, 17(2):237-20 261, 1999.
Derek A Neal and William R Johnson. The role of premarket factors in black-white wage differences. Journal of political Econnyy, 104(5):869 895, 1996.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: A fast, extensible toolkit for sequence modeling. In Association for Computational Linguis- tics, 2019.
Panel Study of Income Dynamics. Public use dataset, produced and distributed by the Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Claudio Persello and Monika Kuffer. Towards uncovering socio-economic inequalities using VHR satellite images and deep learning. In TGARSS RSS 2020-2020 TEEE International Geoscience and Remole Sensing Symposium, pp. 3747-3727 3750. LEEE, 2020.
James M Poterba and Lawrence H Summers. Reporting errors and labor market dynamics. Econometrica, 54(6)):319-1333) 1986.
Nathan Price and Peter M Atkinson. Global GDP prediction with night-lights and transfer learning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:7128-7135 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Tlya Sutskever. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Llya Sutskever. Language models are unsupervised multitask learners. 2019.
Karthik Rajkumar, Lisa Simon, and Susan Athey. A Bayesian Approach to Predicting Occupational Transi- tions. PhD thesis, Stanford University, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabricl Goh, Scott Gray, Chelsca Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-lo-image generation. In International Conference on Machine Learning, 2021.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Associalion for Compulational Linguisties, 2020.
Francisco J. R. Ruiz, Susan Athey, and David M. Blei. SHOPPER: A probabilistic model of consumer choice with substitutes and complements. Annals of Applied Statistics, 14(1):1-27, 2020.
Peter Schmidt and Robert P Strauss. The prediction of occupation using multiple logit models. International Economic Review, 16(2):471 176441611661111 1975.
Gregor Schubert, Anna Stansbury, and Bledi Taska. Employer concentration and outside options. SSRN:3509454, 2021.
Nitish Srivastava, GeofTrey Hinton, Alex Krizhevsky, Πla Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):
1929 1958, 2014.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fino-tundd chat models. Xiv preprial or orXiv22377.09288 2023.
Keyon Vala, Yuntian Deng, David M Blei, and Alexander M Rush. Rationales for sequential predictions. In Empirical Methods in Natural Language Processing, 2021.
Dušan Varis and Ondřej Bojar Sequence length is domain: Length-based overfilting in transformer models. In Empirical Methods in Natural Language Processing, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
Mark N Wexler. Successful resumne fraud: Conjectures the origins of amorality in the workplace. Journal of Human Values, 12(2)1137-152, 2006.
Huang Xu, Zhiwen Yu, Jingyuan Yang, Hui Xiong, and Hengshu Zhu. Dynamic talent flow analysis with deep sequence prediction modeling. IEEE Transactions on Knowledge and Data Engineering, 31(10): 1926-1939, 2018.
Christopher Ych, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in Africa. Nature communications, 11(1):2583, 2020.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Denghui Zhang, Junming Liu, Hengshu Zhu, Yanchi Liu, Lichen Wang, Pengyang Wang, and Hui Xiong. Job2Vec: Job title benchmarking with collective multi-view representation learning. or preprinl arXiv:2009.07429 2020.
Le Zhang, Ding Zhou, Hengshu Zhu, Tong Xu, Rui Zha, Enhong Chen, and Hui Xiong, Attentive heteroge- ncous graph embedding for job mobility prediction. In ACM SIGKDD Conference on Knowledge Discovery & Dala Mining, 2021.
A Econometric Baselines
In this section, we describe baseline occupation models that economists have used model jobs and other discrete sequences.
Markov models and regression. A first-order Markov model assumes the job at each timestep depends on only the previous job (Hall, 1972; Poterba & Summers, 1986). Without covariates, a Markov model takes the form p(3: = jly--1) = p(y+ = jlyt-1). The optimal transition probabilities reflect the overall frequencies of individuals transitioning from occupation yt-1 to occupation j. In a second-order Markov model, the next job depends on the previous two.
A multinomial logistic regression be used to incorporate covariates:
ply: = x,, & exp (8))) + 250) 31-1+ .
(10)
where 0/0) is occupation--peecife intercept and yt-1 and Ite denote J- and Ne-dimensional indicator vectors, respectively. Equation 10 depends on history only through the most recent job, although the covariates can also include hand-crafted summary statistics about the past, such as the duration of the most recent job (McCall, 1990). This model is fit by maximizing the likelihood with gradient gradient-based methods.
Bag-of-jobs. A weakness of the first-order Markov model is that only uses the most recent job to make predictions. However, one's working history beyond the last job may inform future transitions (Blau & Riphahn, 1999; Neal, 1999).
Another baseline we consider is a bag-of-joba model, inspired by SHOPPER, probabilistic model of consumer choice (Ruiz et al., 2020). Unlike the Markov and regression models, the bag-of-jobs model conditions on every job in an individual's history. It docs so by learning a low-dimensional representation of an individual's history. This model learns a unique embedding for each occupation, similar to word embedding (Bengio et al., 2003; Mikolov et al., 2013); unlike CAREER, which learns complicated nonlinear interactions between jobs in history, the bag-of-jobs model combines jobs into single representation by averaging their embeddings.
The bag-of-jobs model assumes that job transitions depend on two terms: a term that captures the effect of the most recent job, and a term that captures the effect of all prior jobs. Accordingly, the model Iearns two types of representations: an embedding a, E R. of the most recent job j, and an embedding Py e R for prior jobs j'. To combine the representations for all prior jobs into a single term, the model averages embeddings:
p(y: = ily:-1) x exp { (8 ag an 1 0
(11)
Covariates can be added to the model analogously; for a single covariate, its most recent value is embedded and summed with the average embeddings for its prior values. All parameters are estimated by maximizing the likelihood in Equation 11 with SGD.
B Qualitative Analysis
Rationalizing predictions. Figure shows an example of held-out carcer sequence from PSID. CAREER. is much likelier than regression and bag-of-jobs baseline to predict this individual's next job, biological
15

---

Published in Transactions on Machine Learning Research (12/2023)
CAREER rationale
Job History
Predicting
Out of Labor Force
Animal Caretaker
Engineering
Biological Technician
Student
Student
Student
Technician
Figure 3: An example of a held-out, job sequence on PSTD along with CARFER's rationale. CAREER ranks the true next job (biological technician) as the most likely possible transition for this individual; in contrast, the regression and bag-of-jobs model rank it as 40th and 37th most likely, respectively. The rationale depicts the jobs in the history that were sufficient for CAREER's prediction.
Model
Consecutive Repeat
Non-Consecutive Repeat
New Job
Markov regression (Hall, 1972)
1.92
29.59
115.7
NEMO (T. et al., 2017)
1.99
30.84
105.0
Job rep. learning (Dave et al., 2018)
1.91
48.72
115.1
Job2Vec (Zhang et al., 2020)
1.88
45.44
115.1
Bag-of-jobs (Ruiz al., 2020)
1.90
35.55
112.9
CAREER
1.87
12.90
100.1
Held-out perplexity for different transition types on PSID (lower is better). CAREER's predictions are best across all transition types, but its biggest improvements come for predicting jobs that repeat in a career at non-consccutive timesteps..
Table 4:
technician. To understand CAREER's prediction, we show the model's rationale, or the jobs in this individ- ual's history that are sufficient. for explaining the model's prediction. (We adapt the greedy rationalization method from Vafa et al. (2021); refer to Appendix G for more details.) In this example, CAREER only needs three previous jobs predict biological technician: animal caretaker, engincering technician, and student. The model can combine latent attributes of each job to predict the individual's next job.
Representation similarity. To demonstrate the quality of the learned representations, we use CAREER's fine-tuned represenlations on NLSY97 to find pairs of individuals with the most similar career trajectories. Specifically, we compute CAREEER's representation h hy Xx) for each individual in NLSY97 who has worked for four years. We then mcasure the similarity between all pairs by computing the cosine similarity between representations.
order to depict meaningful matches, we only consider pairs of individuals with no overlapping jobs in their histories (otherwise the model would find individuals with the exact same career trajectories). Figure 5 depicts the career histories with the most similar CAREER representations. Although none of these pairs have overlapping jobs, the model learns reprosentations that can identify similar careers.
Predictive improvements. To assess the source of CAREER's predictive improvement, we decompose predictions into three categories: consecutive repeats (when the next job is the same as the previous year's s), non-consecutive repeats (when the next job is different from the previous year's, but is the same as one of the prior jobs in the career), and new jobs. Table 4 shows the results on PSTD. CAREER has an advantage over basclines in all thrce categorics, but the biggest improvement comes when prodicting jobs that have been repeated non-consecutively. This advantage makes sense: although consecutive repeated jobs are more common in work trajectories, CAREER's predictive advantage reveals that workers returning to jobs they previously held in their carcers are also common and should be incorporated into model predictions.
We also break down predictive advantage as function of career length for NLSY79 (Figure 4). We find that CAREER is not at an advantage when predicting an individual's first job, which makes sense since there is no additional job history to condition on. We find that for all other years history, CAREER has a predictive advantage over the econometric baselines. However, the size of this advantage is most pronounced mid-career. For both CAREER and the baseline regression, it becomes easier to predict an individual's 's next
16

---

Published in Transactions on Machine Learning Research (12/2023)
Regression CAREER
Perplexity
20
15
10
5
15
20
Years History
Figure 4: Held-out perplexity function of career length on NLSY79. CAREER has a predictive
advantage at all points of individual's career. The magnitude of this advantage is largest mid-career.
Model
PSID
NLSY79
NLSY97
Markov regression (Hall, 1972)
0.701 = 0.001
0.732 + 0.000
0.680 = 0.001
NEMO (Li et ul., 2017)
0.706 1 0.004
0.731 +: 0.002
0.664 1 0.001
Job rep. learning (Dave al., 2018)
0.679 0.003
0.704 0.000
0.651 + 0.001
Job2Vec (Zhang et al., 2020)
0.681 + 0.005
0.705 + 0.000
0.651 + 0.001
Bag-of-johs (Ruiz et al., 2020)
0.6990 0.002
0.706 0.000
0.661 0.001
CAREER
0.741 + 0.001
0.761 + 0.000
0.691 + 0.001
Table 5: AUC for predicting whether or not an individual transitions or keeps occupations each timestep, treating the outcome as a binary variable.
job as their career progresses, partially because individuals are more likely to stay at the same job later in their career rather than earlier. The countervailing effect is that, all else equal, CAREER's predictive advantage should more pronounced when there are more jobs to condition on. Thus, there is a sweet spot for CAREER's advantage, between 5-10 years in an individual's career.
Table 5 compares model performances for the specific prediction of whether not individual changes occupations. Here, the outcome is a binary variable, indicating whether not individual transitioned occupations, while each model forms a prediction of an individual staying in the same occupation leaving (summed over all possible transitions). CAREER has the highest AUC for predicting these transitions.
C Transformer Details
In this section, we expand on the simplified description of transformers in Section 2.3 and describe CAREER in full detail. Recall that the model estimates representations in L layers, h, (y)-1, h (y)
2002:
Janitor
2002:
Retail salesperson
2003:
Material handler
2003:
Shipping clerk
2003:
Food prep worker
0.957
2003:
Material handler
Material handler
0.954
2004: Shipping clerk
2004: Food service worker
2004:
Laundry worker
2005:
Material handler
2005: Shipping clerk
2006:
Food service worker
2005:
Cook
2006:
Material handler
2006: Shipping clerk
2001:
Retail salesperson
2001:
Cashier
2001:
Child care worker
2001: Teacher's aide
Receptionist Receptionist 2004: Receptionist
2002:
0.956 2002:
Secretary
2002: Child care worker
0.952
2002: Teacher's aide
2003: Secretary
2003: Child care worker
2003: Teacher's 's aide
2004: Secretary
2004: Child worker
2004: Teacher's 's aide
The work experiences with the most similar CAREER representations (measured with cosine similarity) for individuals with overlapping jobs in NLSY97.
Figure 5:
17

---

Published in Transactions on Machine Learning Research (12/2023)
with each representation (,)0 RP. The final representation h (yr-1, xx) is used to represent careers. We drop the explicit dependence on Y:-1 and X2, and instead denote cach representation as h2
The first. transformer layer combines the previous occupation, the most. recent covariates, and the position of the occupation in the career. It first embeds each of these variables in D-dimensional space. Define an embedding function for occupations, ey : [J] RD_ Additionally, define separate embedding function for each covariate, {ee (ec)en with each ee [Ne]] + R?. Finally, define e : [T] R to embed the position of the sequence, where T denotes the number of possible sequence lengths. The first-layer representation h sums these embeddings:
0 = (, + E. ec(vee) + ex()
(12)
The occupation- and covariate-specific embeddings, by and {ee), are model parameters; the positional emn beddings, Ct, are set in advance to follow sinusoidal pattern (Vaswani et al., 2017). While these embeddings could also be parameterized, in practice the performance is similar, and using sinusoidal embeddings allows the model to generalize to career sequence lengths unseen in the training data.
At each subsequent layer, the transformer combines the representations of all occupations in a history. combines representations by performing multi-headed attention, which similar to the process described in Section 2.3 albeit with multiple attention weights per layer.
Specifically, it uses A specific attention weights, or heads, per layer. The number of heads A should be less than the representation dimension D. (Using A = 1 attention head reduces to the process described in Equations 5 and 6.) The representation dimension D should be divisible by A; denote K = D/A. First, A different sets of attention weights are computed:
fa,t, = -1(10) (
for t' st
(13)
(xxp(2a,.11)
To,t,
Exexpe eexp(2n,.h)
where W!( E DxD model parameter, specific attention head a and layer 1,5 Each attention head forms convex combination with all previous representations; to differentiate between attention heads, each representation is transformed by a linear transformation vi(() KXD unique to an attention head, forming b e RR:
60 = Fe,t (v.')!9)
(14)
All attention heads are combined into single representation by concatenating them into single vector 9: (4) R.D:
g (2) - (b99
60 (0)
(00
(15)
To complete the multi-head attention step and form the intermediate representation T,00, the concatenated representations gt undergo linear transformation and are summed with the pre-attention representation
(
1 = 11 + 4 g
(16)
with ( E DxD
The intermediate representations h,(²) e D combine the representation at timestep t with those preceding timestep t. Each layer of the transformer concludes by taking non-lincar transformation of the intermediate
For computational reasons, W((f) is decompased into two matrices and scaled by constant, w 2.00
(Ka
with
/R
2 x(0 e RDX
18

---

Published in Transactions on Machine Learning Research (12/2023)
Number of individuals
23,731,674
General
Number of tokens
245,439 $46,439,863
Median year
2007
Percent Northeast
17.6
Percent Northcentral
20.7
Geography
Percent South
Percent West
19.4
Percent without location
Percent high school diploma
7.2
Percent associate degree
8.6
Education
Percent bachelor degree
23.1
Percent graduate degree
4.5
Percent empty
52.8
Percent profeessial specialty
38.4
Percent sedumica//saless/dminnsstra addiiistaaiiiiiiiiiiiee support.
Broad Occupation Groups
Percent service
12.0
Percent precision production/craftt /repair
7.9
Percent operator/labricatae /aabriiatoorllabbrr
7.2
Table 6: Exploratory data analysis of the resume dataset used for pretraining CAREER.
representations. This non-linear transformation does not depend on any previous representation; only transforms 1/0 Specifically, h, is passed through neural network:
h (t+1) = =((0) FFN(() (R(2))
(17)
where FFN(O denotes a two-layer leedforward neural network with N hidden units, with FFN(O, RDD - RDD
We repeat the multi-head attention and feedforward neural network updates above for L layers, using parameters unique to each layer. We represent carcers with the last-layer representation, h he(y:-1,x,) =
h (y)-
By marginalizing the two-stage prediction indicator, the conditional likelihood has the form:
(n he(y:_1,x))) (1-o(1) a(n po.nnnhlle x,()),1,x))) Eytm (8p he 1.x,)]
if yr yt-1
p(u: = jly--1,x,)
exp(83-b 1.x,))
if yr. * y
(18)
The loss function for an individual sequence (yt, xx) is given by
(yT,xT)) log jlyc-1xx)).
(19)
1=1
D Exploratory Data Analysis
Table 6 depicts summary statistics of the resumne dataset provided by Zippia that is used for pretraining CAREER. Table 7 compares this resume datasct with the longitudinal survey datascts of interest.
E
One-Stage Two-Stage Prediction
Table 8 compares the predictive performance of baseline occupation models when they are modified make predictions in two stages, following Equations to 3. Incorporating two-stage prediction improves
19

---

Published in Transactions on Machine Learning Research (12/2023)
Resumes
NLSY79
NLSY97
PSID
Number of individuals
23,731,
73,6674
12,270
8,770
12,338
Average sequence length
10.3
19.5
13.0
5.1
Unemployed/ laatt--oolbor-oorreeeeoooooeee-e
available
No
Yes
Yes
Yes
Median year
2007
1991
2007
2011
Percent manual laborers
7%
17%
13%
12%
Percent college graduales
56%
23%
29%
28%
Demographic covariates available
No
Yes
Yes
Yes
Table 7: Comparing the resume dataset used for pretraining with the three longitudinal survey datasets of interest.
PSTD
NLSY79
NLSY97
Markov regression (two-stage)
15,60 + 0.03
13.30 + 0.02
15.47 + 0.00
NEMO (two-stage)
15.23 + 0.08
12.37 0.04
15.13 = 0.03
Job rep. learning (two-stage)
15.98 1 0.06
13.97 + 0.03
15.43 1 0.01
Job2Vec (two-slage)
15.80
0.04
13.91 0.01
15.31 1 0.02
Bag-of-jobs (two-stage)
15.40 + 0.05
12.68 0.01
15.11 + 0.02
CAREER
13.88 0.01
11.32 0.00
14.15 + 0.03
Table 8: Perplexity of economic baselines when they are modified to make predictions in two stages.
the performance of these baseline models compared to Figure 2a; however, CAREER still makes the best predictions on all survey datasets. For the CAREER model that was pretrained, we found that one- and two-stage models made similar predictions, suggesting that pretraining helps learn occupational mobility patterns.
F Data Preprocessing
In this section, we go over the data preprocessing steps we took for each dataset.
Resumes, We were given access to a large dataset of resumes of American workers by Zippia, career planning company. This dataset coded each occupation into one of 1,073 O*NET 2010 Standard Occupational Classification (SOC) categories based on the provided job titles and descriptions in resumes. We dropped examples with missing SOC codes.
Each resume in the dataset were given contained covariates that had been imputed based off other data in the resume. We considered three covariates: year, most recent educational degree, and location. Education degrees had been encoded into one of eight categories: high school diploma, associate, bachelors, masters, doctorate, certificate, license, and diploma. Location had been encoded into one of 50 states plus Puerto Rico, Washington D.C., and unknown, for when location could not be imputed. Some covariates also had missing entries. When an occupation's year was missing, we had drop it from the dataset, because we could not position an individual's career. Whenever another covariate was missing, we replaced with special "missing" token. All personally identifiable information had been removed from the dataset.
We transformed each resume in the dataset into sequence of occupations, We included entry for each year starting from the first year an individual worked to their last year. We included a special "beginning of sequence" loken lo indicate when each individual's sequence started. For each year bet ween an individual's first and last year, we added the occupation they worked during that year. If an individual worked in multiple occupations a year, we took the one where the individual spent more time that year; if they were both the same amount of time in the particular year, we broke ties by adding the occupation that had started earlier in the career. For the experiments predicting future jobs directly resumes, added
20

---

Published in Transactions on Machine Learning Research (12/2023)
"nn-observed-occnnpttoo token for years where the resumme did not list any occupations (we dropped this token when pretraining). Each occupation was associated with the individual's most recent educational degree, which we treated as dynamic covariate. The year an occupation took place was also considered a dynamic categorical covariate. We treated location as static. In total, this preprocessing left us with a dataset of 23.7 million resuines and 245 million individual occupations.
In order to transfer representations, we had to slightly modify the resumes dataset for pretraining to encode occupations and covariates into format compatible with the survey datasets. The survey datasets we used were encoded with the "occ1990dd" occupation code (Autor & Dorn, 2013) rather than with OTNET's SOC codes, so we converted the SOC codes to occ1990dd codes using crosswalk posted online by Destin Royer, Even after we manually added a few missing entries to the crosswalks, there were some SOC codes that did not have corresponding occ1990dd's. We gave these tokens special codes that were not used when fine-tuning on the survey datasets (because they did not correspond to occ1990dd occupations). When an individual did not work for given year, the survey datasets differentiated between three possible states: unemployed, out-ofllaoor-foree, and in-school. The resumes dataset did not. have these calegories. Thus, we initialized parameters for these three new occupational states randomly. Additionally, we did not include the "no- observed-occupptionn " token when pretraining, and instead dropped missing years from the sequence. Since we did not use gender and race/ /etenettt covariates when pretraining, we also initialized these covariate- specific parameters randomly for fine-tuning. Because we used a version of the survey datasets that encoded each individual's location as a geographic region rather than as a state, we converted each state in the resumes data to be in one of four regions for pretraining: northeast, northeentral, south, or west. We also added fifth "other" region for Puerto Rico and for when state was missing in the original dataset. We also converted educational degrees to levels of experience:
converted associate's degroe to represent some college experience and bachelor's degree to represent four-year college experience; we combined masters and doctorate to represent a single "graduate degree" category; and we left the other categories as they were.
NLSY79. The National Longitudinal Survey of Youth 1979 (NLSY79) is survey following individuals born in the United States between 1957-1964. The survey included individuals who were between 14 and 22 years old when they began collecting data in 1979; they interviewed individuals annually until 1994, and biennially thereafter.
Each individual in the survey is associated with an ID, allowing us to track their careers over time. Wo converted occupations, which were initially encoded as OCC codes, into "occ1990di 1990dd" codes using a crosswalk (Autor & Dorn, 2013). We use version of the survey that has entries up to 2014. Unlike the resumes dataset, NLSY79 includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-foreee and in-school. We include special tokens for these states in our sequences. We drop examples with missing occupation states. We also drop sequences for which the individual is out of the labor force for their whole careers.
We use the following covariates: years, educational experience, location, race/cthnicity, and gender. We drop individuals with less than 9 years of education experience. We convert years of educational experience into discrete categories: no high school degree, high school degree, some college, college, and graduate degree, We convert geographic location to one of four regions: northeast, northcentral, south, and west. We treat location as static variable, using each individual's first location. We use the following race/ethnicittie:: white, African American, Asian, Latino, Native American, and other. We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static. This preprocessing leaves us with a dataset consisting of 12,270 individuals and 239,545 total observations.
NLSY97. The National Longitudinal Survey of Youth 1997 (NLSY97) is survey following individuals who were between 12 and 17 when the survey began in 1997. Individuals were interviewed annually until 2011, and biemnially thereafter.
Our preprocessing of this dataset is similar to that of NLSY79, We convert occupations from OCC codes into "occ1990dd" codes. We use a version of the survey that follows individuals up to 2019. We include tokens for unemployed, latoof-labor,foree,, and in-school occupational states. We only consider individuals who are over 18 and drop military-related occupations. We use the same covariates as NLSY79. We use
21

---

Published in Transactions on Machine Learning Research (12/2023)
the following race/ethnicitiss: white, African-aAmerican, Latino, and other/unknown. We convert years of educational experience into discrete categories: no high school degree, high school degree, some college degree, college degree, graduate degree, and a special token when the education status isn't known. We use the same regions as NLSY79. We drop sequences for which the individual is out of the labor force for their whole careers. This preprocessing leaves us with a dataset consisting of 8,770 individuals and 114,141 total obsorvations.
PSID.
The Panel Study of Income Dynamics (PSID) is a longitudinal panel survey following a sample of American families. It was collected annually between 1968 and 1997, and biennially afterwards.
The dataset tracks families over time, but it only includes occupation information for the household head and their spouse, so we only include these observations. Occupations are encoded with OCC codes, which we convert. "occ1990dd" using crosswalk (Autor & Dorn, 2013). Like the NLSY surveys, PSID also includes three states corresponding to individuals who are not currently employed: unemployed, out-of-labor-fee force, and in-school. We include special tokens for these states in our sequences. We drop other examples with missing or invalid occupation codes. We also drop sequences for which the individual is out of the labor force for their whole careers.
We consider five covariates: year, education, location, gender, and race. We include observations for individ- uals who were added to the dataset after 1995 and include observations up to 2019. We exclude observations for individuals with less than 9 years of education experience. We convert years of education to discrete states: no high school, high school diploma, some college, college, and graduate degree. We convert geo- graphic location to one of four regions: northeast, northeentral, south, and west. We treat location as a static variable, using each individual's first location. We use the following races: white, Black, and other. We treat year and education as dynamic covariates whose values can change over time, and we consider the other covariates as static. This preprocessing leaves us with a dataset consisting of 12,338 individuals and 62,665 total observations.
G Experimental Details
Model Specifications. We consider a first-order Markov model and second-order Markov model (both without covariates) as baselines. These models are estimated by averaging observed transition counts. We smooth the first-order Markov model by taking weighted average between the empirical transitions in the training set and the empirical distribution of individual jobs. We perform this smoothing to account for the fact that some feasible transitions may never occur in the training set due to the high-dimensionality of feasible transitions. We assign 0.99 weight to the empirical distributions of transitions and 0.01 to the empirical distribution of individual jobs. We smooth the sccond-order model by assigning 0.5 weight to the empirical second-order transitions and 0.5 weight. to the smoothed first-order Markov model.
When we add covariates to the Markov linear baseline, we also include manually constructed features about history to improve its performance. In total, include the following categorical variables: the most recent job, the prior job, the year, a dummy indicating whether there has been more than one year since the most recent observed job, the education status, a dummy indicating whether the education status has changed, and state (for the experiments on NLSY79 and PSID, we also include an individual's I's gender and race faceeethnicity). We also add additive effects for the following continuous variables: the number of years an individual has been in the current job and the total number of years for which an individual has been in the dataset. In addition, we include an intercept term
For the bag-of-jobs model, vary the representation dimension D between 256-2048, and find that the pre- dictive performance is not sensitive to the representation dimension, so we use D = 1024 for all experiments.
We also compare to NEMO (Li el al., 2017), an LSTM-based method developed for modeling job sequences in resumes. We adapted NEMO to model survey data. In its original setting, NEMO took as input static covariates (such as individual skill) and used these to predict both an individual's next job title and their company. Survey datasets differ from this original setting in a few ways: covariates are time-varying, important covariates for predicting jobs on resumes (like skill) are missing, and an individual's company
22

---

Published in Transactions on Machine Learning Research (12/2023)
name is unavailable. Therefore, we made several modifications to NEMO. We incorporated the available covariates from survey datasets by embedding them and adding them to the job embeddings passed into the LSTM, similar to the method CAREER uses to incorporate covariates, We removed the company-prediction objective, and instead only used the model to predict an individual's job in the next timestep. We considered two sizes of NEMO: an architecture using the same number of parameters as CAREER, and the smaller architecture proposed in the original paper. We found the smaller architecture performed better on the survey datasets, so we used this for the experiments. This model contains 2 decoder layers and hidden dimension of 200.
We compare to two additional basclines developed in the data mining litcrature: job representation learning (Dave et al., 2018) and Job2Vec (Zhang et al., 2020). These methods require resume-specific features such as skills and textual descriptions of jobs and employers, which are not available for the economic longitudinal survey datasets we model. Thus, we adapt these baselines to be suitable for modeling economic survey data. Job representation learning (Dave et al., 2018) is based on developing two graphs, one for job transitions and one for skill transitions. Since worker skills are not available for longitudinal survey data, we adapt the model to only use job transitions by only including the terms in the objective that depend on job transitions. We make a few additional modifications, which we found to improve the performance of this model on our data. Rather than sampling 3-tuples from the directed graph of job transitions, we include all 2-tuple job transitions present in the data, identical to the other models we consider. Additionally, rather than using the contrastive objective in Equation 4 of Dave et al. (2018), we optimize the log- likeliioddood directly this is more computationally intensive but leads to better results. Finally, we include survey-specific covariates (e.g- education, demographics, etc.) by adding them to we: embedding the covariate of each most recent job to the same space Wz- We make similar modifications to Job2Vec (Zhang et al., 2020). Job2Vec requires job litles and descriptions of job keywords, which are unavailable for economic longitudinal survey datasets. Instead, we modify Equation 1 in Zhang et al. (2020) to model occupation codes rather than titles or keywords and optimize this logelikelihood as our objective. We also incorporate survey-specific covariates by embedding each covariate to the same space as e; and adding it to e; before computing Equation 2 from Zhang et al. (2020), which we also found to improve performance. We follow Dave et al. (2018) and use 50 embedding dimensions for each model, and optimize with Adam using a maximum learning rate of 0.005, following the minibatch and warmup strategy described below.
For the transformer models, we used model specifications similar to the generative pretrained transformer (GPT) architecture (Radford et al., 2018). This model includes a few extra modifications to improve train- ing: we use 0.1 dropout (Srivastava et al., 2014) for the feedforward neural network weights, 0.1 dropout for the attention weights, and the GELU nonlinearity (Hendrycks & Gimpel, 2016) for all feedforward neural networks. Additionally, the model uses layer normalization (Ba et al., 2016) before the updates in Equa- tion 13, after the update in Equation 16, and after the final layer's neural network update in Equation 17. We also considered other strategies for incorporating covariates into the model, including introducing them later in the transformer architecture and including different final-layer weights for each covariate. We found the way we incorporated covariates described in the paper resulted in the best predictions. We performed ablations separately for each model (since the non-transferred model is trained on fewer data points and did not require as many parameters). For the noeeprrtr CAREER, we considered 4 and 12 layers, 64 and 192 embodding dimensions, 256 and 768 hidden units for the foodforward neural networks, and or 3 atlention heads (using 2 heads for D= = 64 and 3 heads for D 192 so that D was divisible by the number of heads). We tried each combination of these parameters on NT.SY79, and found that the model with the best validation performance had 4 layers, D = 64 embedding dimensions, 256 hidden units, and 2 attention heads. We used this architecture for the non-prrtra version of CAREER on all survey datascts. We performed a similar ablation for the pretrained version of CAREER, and found that 12 layers, 192 embedding dimensions, 3 attention heads, and 768 hidden units performed best. In total, this resulted in 5.6 million parameters for the full CAREER model.
We also considered CAREER model that is pretrained on resumes but not fine-tuned. However, we found this model to be very poor at modeling occupations on survey data, because some occupations in the survey data (such as "unemployed") are never seen in the resume dataset. Thus, the model that is not fine-tuned assigns close to zero probability for these occupations. This underscores the importance of fine-tuning.
23

---

Published in Transactions on Machine Learning Research (12/2023)
Training. We randomly divide the resumes dataset into training set of 23.6 million sequences, and a validation and test set of 23 thousand sequences each. We randomly divide the survey datasets into 70/10/20 splits.
The first- and second-order Markov models without covariates are estimated from empirical transitions counts. We optimize all other models with stochastic gradient descent with minibatches. In total, we use 16,000 total tokens per miníbatch, varying the batch size depending on the largest sequence length in the batch. We use the Adam learning rate schoduler (Kingma & Ba, 2015). All experiments on the resumes data warm up the learning rate from 10-7 to 0.0005 over 4,000 steps, after which the inverse square root schedule is used (Vaswani et al., 2017). For the survey datascts, we also used the inverse square root scheduler, but experimented with various learning rates and warnuup updates, using the one we found to work best for each model. For CAREER with pretrained representations, we used a learning rate of 0,0001 and 500 warmup updates; for CAREER without pretraining, we used a learning rate of 0.0005 and 500 warmup updates; for the bag of jobs model, we used a learning rate of 0.0005 and 5,000 warmup updates; for the regression model, we used a learning rate of 0.0005 and 4,000 warmup updates. We use a learning rate of 0.005 for job representation learning and Job2Vec, with 5,000 warmup updates. All models besides were also trained with 0.01 weight decay. All models were trained using Fairseq (Ott et al., 2019).
When training on resumes, we trained for 85,000 steps, using the checkpoint with the best validation perfor- mance. When fine-tuning on the survey datasets, we trained all models until they overfit to the validation set, again using the checkpoint with the best validation performance. We used half precision for training all models, with the exception of the following models (which were only stable with full precision): the bag of jobs model with covariates on the resumes data, and the regression models for all survey dataset experiments.
The lables in Section 4 report results averaged over multiple random seeds. For the results in Figure 2a, the randomness includes parameter initialization and minibatch ordering. For CAREER, we use the same pretrained model for all settings. For the forecasting results in Table 2 and Table 1, the randomness is with respect to the Konte-Carlo sampling used to sample multi-year trajectories for individuals. For the wage prediction experiment in Table 3, the randomness is with respect to the sampling distribution of the test set.
Forecasting. For the forecasting experiments, occupations that took place after a certain year are dropped from the train and validation sets. When we forecast on the resumes dataset, we use the same /litt/ validation split but drop examples that took place after 2014. When we pretrain CAREER on the resumes datasct to make forecasts for PSID and NLSY97, we use a cutoff year of 2014 as well. We incorporate two-stage prediction into the baseline models because we find that this improves their predictions.
Although we do not include any examples after the cutoff during training, all models require estimating year- specific terms. We use the fitted values from the last observed year to estimate these terms. For example, CAREER requires embedding each year. When the cutoff year is 2014, there do not exist embeddings for years after 2014, so we substitute the 2014 embedding.
We report forecasting results on a split of the dataset containing examples before and after the cutoff year. To make predictions for an individual, we condition on all observations before the cutoff year, and sample 1,000 trajectories through the last forecasting year. We never condition on any occupations after the cutoff year, although we include updated values of dynamic covariates like education. For forecasting on the resumes dataset, we set the cutoff for 2014 and forecast occupations for 2015, 2016, and 2017. We restrict our test set to individuals in the original test set whose first observed occupation was before 2015 and who were observed to have worked until 2017. PSID and NLSY97 are bicnnial, so we forccast for 2015, 2017, and 2019. We only make forecasts for individuals who have observations before the cutoff year and through the last year of forecasting, resulting in total of 16,430 observations for PSID and 18,743 for NLSY97.
Rationalization. The example in Figure 3 shows an example of CAREER's rationale on PSID. To simplily the example, this is the rationale for a model trained on no covariates except year. In order to conceal individual behavior patterns, the example in Figure 3 is a slightly altered version of a real sequence. For this example, the transformer used for CAREER follows the architecture described in Radford al. (2018). We find the rationale using the greedy rationalization method described in Vafa et al. (2021). Greedy
24

---

Published in Transactions on Machine Learning Research (12/2023)
rationalization requires fine-tuning the model for compatibility; we do this by fine-tuning with "job dropout", where with 50% probability, we drop out uniformly random amount of observations in the history. When making predictions, the model has to implicitly marginalize over the missing observations. (We pretrain on the resumes dataset without any word dropout). We find that training converges quickly when finc-tuming with word dropout, and the model's performance when conditioning on the full history similar.
Greedy rationalization typically adds observations history one time in the order that will maximize the model's likelihood of its top prediction. For occupations, the model's top prediction is almost always modify greedy rationalization to add the occupation that will maximize the likelihood of its second-largest prediction. This can be interpreted as being equivalent to greedy rationalization, albeit conditioning on switching occupations. Thus, the greedy rationalization procedure stops when the model's second-largen prediction from the target rationale equivalent to the model's socond-largest prediction when conditioning on the full history.
identical to the previous year's occupation,
25


Reviews:
[{"review_id": "FV7zYuwbWH", "reviewer": "Reviewer_o9LQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe proposed algorithmic approach for developing informative representations of career history (via what is called CAREER) is well grounded in prior use of transformer models in a pre-training/fine-tuning paradigm. The adjustments made to incorporate categorical data sequences through initial embedding layers is interesting, although seems to be fairly limited by how expressive these embeddings can be as they are effectually a key-value store indexed by integer quantities.\n\nI appreciate the delineation made by the authors between expansive resume databases and carefully curated longitudinal econometric occupational surveys. It was made apparent (while the connotation could be made more explicit and direct) that policy decisions and generalizable claims cannot be made from the use of these large resume databases as they are not representative of the population at large. What is missing from this very clear dichotomy is a good, clear and direct discussion of why econometric models need well curated survey datasets. I'll point to additional issues with the presentation of the data/problem formulation below.\n\nThe paper does a nice job outlining prior approaches and methodologies for occupation prediction within economics including some interesting recent approaches such as NEMO that are built to learn from unstructured resume datasets.\n\nFor the most part, the setup and description of how the proposed CAREER model is constructed is clearly understood. Perhaps one addition to Figure 1 would be to include annotations of the FFN between $\\tilde{h}^{(1)}t$ and $h^{(2)}_t$ similar to how the attention weights are shown.\n\nThere is a variety of experiments to demonstrate the utility of using pretrained representations, adapted for the occupation prediction task. I was especially glad to see the experiment represented in Figure 2B that shows the effect on pretraining volume on final model prediction performance. The forecasting and wage regression experiments were also very interesting and good demonstrations of how the pretrained representations can be useful (despite some major questions that arose from how they were presented, details below).\n\n\n**Weaknesses**\n\nI fear that there are several relatively major weaknesses that lessen the excitement I have for this paper and the presented empirical results. Some are easily fixed (more thorough and precise definitions and writing) while others may take a fair amount of effort on the authors' behalf (additional baseline experiments and model ablations). As a summary, there are too few details about the motivating foundations of the work, the datasets used, the modeling approach, the experimental setup and the analyses presented. As a result I am not entirely sure that I can confidently state that the claims made are adequately supported by the paper. In the following I will try to highlight where I am concerned and how the authors may be able to revise the paper to help improve my estimation of the work. This being said, I feel that the insights provided by the paper are good, it can however be greatly improved through more precise language and thorough empirical validation. This work will be of interest to the community using ML within economics as well as the broader ML community for yet an additional domain where scaling pretraining enables improved predictive with transformer models.\n\n_Motivating foundations_\n\nI found the writing and descriptions about the econometric setting to be insufficient and vague. There are not enough details shared about the kinds of predictive tasks that are prevalent in this community and why they are important. This extends to any motivations about the datasets that are used and how analyses of these survey datasets support claims and potential policy decisions (if any are to be had). Greater clarity about the domain specific usage of the data and why economists take such great effort to develop survey datasets would go a long way to improve the apparent significance of the contributions made by this paper.\n\nMany of the motivations behind the experiments are vague and seem to rest on a solitary example without additional information or detail about why the experiments are chosen and how informative they may be in view of the intended use case. \n\n_Datasets_\n\nIt's not clear how the datasets are formed, what the size of J or $N_c$ (the number and scale of the covariates), etc. are. The discussion or information provided about the datasets (both the resume and longitudinal survey data) is surprising. Without these details it is unclear what is actually being modeled or the overall scale of the problem. For example, how do the data distributions of job categories or types compare between the pretraining data and each of the survey datasets? Is J consistent across the whole paper? If not, what kinds of support can be found in the pretraining data that overlaps with what is represented in the survey data? This isn't a critical omission as the methods used to construct CAREER and analyze are the clear focus of the paper, but I would rate the omission as a major issue that should be corrected to improve the presentation of the paper.\n\nIn connection to the concerns raised previously, there is no indication of how long the career sequences are in each dataset on average (e.g. what is the distribution of T?). I know that the perplexity metric tries to average over this component but it could be a major factor in the forecasting performance as errors will compound for longer career durations. (This is a shared concern with the analyses provided in the paper) If it's true that some of the datasets have longer sequences than others, that could explain why the finetuning task takes longer on datasets with equivalent numbers of individuals. \n\nIt is claimed at the end of Section 3 that this paper \"identif[ies] a novel pretraining corpus\". This isn't necessarily true, is it? It is stated that \"NEMO, an LSTM-based model that is trained on large resume datasets\". While NEMO may not be immediately applied to the pretraining/finetuning paradigm, it may not be wholly accurate to state that the use of the same (or similar) dataset for pretraining is novel.\n\n_Modeling approach_\n\nIt is very unclear how the predicted job change indicator $s_t$ is used to inform the input $y_{t-1}$ at the next timestep as indicated in Figure 1. \n\nThere's an asymmetry in how the covariates are introduced and described and how they are ultimately used in the experiments. In fact, it's unclear between experiments what covariates are used and if they are at all. This is not insignificant. _This inconsistency is a problem that should be addressed by the authors if this paper is to be considered for publication_.\n\nAre there any additional regularizations used to keep the representations $h_t$ from collapsing to trivial modes? With the amount of data and number of model parameters, there is some risk for model collapse. This highlights one of my complaints of the presentation of this paper. Several times the representations are claimed to be \"complex\" or \"strong\" but there is no precision or justification of those terms. It appears that this nomenclature is used implicitly from the complexity of the model architecture. There is no analysis of the representations and what is being encoded within them. The only indication that we have that anything is being learned is model performance. There are no clear indicators that the representations differentiate between different career paths or how the effect of individual covariates have on the representation. Without any analysis or justification of how the representations are formed and what they encode, I cannot place any weight on the language or claims made about them. It's highly possible that pretrained representations using NEMO could be equally as performant on the dataset not to mention more rudimentary clustering approaches. This significantly impacts how I view the empirical support of the claims made by the paper.\n\n_Experimental Setup_\n\nIt would be very useful to recast the apparent strength of the transformers learned representations $h_t$ in the context of additional predictive tasks (beyond job prediction) to better compare with prior approaches outlined in Section 3. With the two-stage regression technique of CAREER, occupational change is already modeled. It would be instructive to see how that sub-task performs relative to the the cited approaches of Kambourov & Monovskii (2008) and Guvenen et, al. (2020). \n\nAs mentioned above, there are concerns about whether the apparent gains seen with CAREER can be contributed mostly to the transformer architecture more than the pretraining approach. There are no empirical comparisons with an alternative architecture that is provided the same training / evaluation procedure (two-stage regression from the embedded representation of career path). The suggested NEMO pretraining baseline could be a way to validate the use of a more complex transformer architecture.\n\nAn additional baseline that would help reinforce the overall empirical approach used in the paper would be to pretrain CAREER but not finetune. To demonstrate the zero-shot capabilities of the model based on the resume data alone. This could open up a really interesting analysis of the questions raised above about the representative support of the pretraining dataset (e.g. are there certain individuals that are included in the survey data for which not finetuning does well on? Are there inequalities that arise without the finetuning step? Do those inequalities persist after finetuning?)\n\nIt would really help if the objective or loss function was formally included in the main paper. This would help round out Section 2.3.\n\nI found the ordering of Section 2 to be a little confusing. The most important formulations of the CAREER model are presented in Section 2.2 where there is not much context behind how they would be used. Then after so much information is given in Section 2.3 about architecture and training approach, there is a subtle reference to the previous subsection. I think that the Section 2 would be improved if the order of Section 2.2 and 2.3 were swapped. I also think that the \"Transfer Learning\" subsubsection could be elevated into a new Section 2.4 to help emphasize the contributions and claims made in the paper.\n\nThe specific tasks \"career prediction\" and \"forecasting\" are not adequately defined at the outset of Section 4. This is another example where just a little more detail would go a long way. This issue occurs again when the downstream auxiliary tasks (wage prediction in this paper) are introduced. There is reference to multiple types of economic analyses that could be run but there is no mention of what these are. I found a lot of the language around these type of context-setting portions to be frustratingly vague.\n\n_Experimental Analysis_\n\nWhile a helpful comparative metric, perplexity is difficult to gain any insight from. I believe that including additional metrics such as AUROC/ AUPRC or F1 score would be far more informative of the quality of the predictions being made by the compared models. This is particularly true when considering the forecasting task presented in Table 1. I have no way to assess how reliable the models are as perplexity gives me no indication of how immediately accurate the predictions are. \n\nIt's not sufficient to only report the mean performance of the prediction metrics in Tables 1 and 2 without confidence intervals as is done in the table contained in Figure 2A. Without these the significance of the performance improvement seen with CAREER cannot be assessed. Their exclusion seems somewhat intentional?\n\nWhile NEMO is included as a baseline, it's unclear whether it was pretrained using the resume dataset as it was developed for. So, it's not entirely clear whether its current use is a fair comparison of the model's performance. It would be instructive to see if NEMO (and other baseline modeling approaches) would have better performance than their vanilla counterparts if they were additionally exposed to more data in a pretraining setting and how that would compare after finetuning to the variants of CAREER presented in Figure 2A. Without this baseline, it's unclear whether the performance gains are drawn from the architecture (e.g. model complexity/expressivity) or the training paradigm. This is perhaps the primary major concern of mine regarding the evaluation supporting the claims made in the paper.\n\nGiven the amount of discussion around the assumed benefit of using a more complex model that breaks down prior work's Markov assumption, it is surprising that the authors did not analyze the attention weights to demonstrate any evidence of their claims on this premise. It would be really interesting to see how far the prediction of the current time step relies on previous employment _and_ the sensitivity of the prediction on the included covariates. Without any analysis along these lines, I cannot trust the claims made about the necessity of non-Markovian modeling. I cannot just take the claim at face value because of the prediction metrics are \"better\". We can expect there to be more rigor in the experiments to validate the claims made.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach for improving predictions for econometric models via pretraining using semi-structured data derived from a resume database with finetuning on longitudinal survey datasets. A transformer-based model architecture is adapted for categorical data modalities and is shown to improve performance over previous methods when predicting and forecasting occupations. In addition, the internal representations learned through the pretraining approach are shown to be beneficial for another downstream economic prediction task.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n\nMore detail about the datasets (especially the demographic features used as covariates in all experiments).\n\nAdditional prediction performance metrics such as AUROC/AUPRC or F1 score to evaluate the quality of the compared models.\\\n\nAdditional NEMO baseline that is pretrained on resume database, then finetuned on the survey datasets.\n\nAdditional baseline removing covariates from CAREER.\n\nSome analysis on the \"strength\" of the representations learned rather than some implied validity of their construction based on the complexity of the model.\n\nA model ablation of CAREER without the covariates included.\n\nFurther analysis of the claims made about the necessity of the transformer architecture (using information over the whole sequence of career history). \n\n**Minor**\n\nMore clarity in writing about the domain and the specifics of the kinds of predictive analyses being used with the data. This would be to better frame the contributions of the proposed model.\n\nCAREER is clearly an acronym (as stated in footnote 2). It should be spelled out as early as the modeling name/approach is used in Section 1 to help frame the contributions and claims made at the end of the introduction.\n\nAdditional experimental analysis comparing CAREER's prediction of occupational change with prior approaches.\n\nRestructuring Section 2 as outlined above to improve clarity.\n\nA baseline of zero-shot prediction performance of the the pretrained CAREER model without finetuning.", "broader_impact_concerns": "I have none beyond those already addressed and discussed by the authors in the included broader impact statement and the expressed weaknesses I have recorded above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EvNGhAcSkJ", "reviewer": "Reviewer_bDhb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\nThe approach represents a very concrete application\tof representation and transfer learning — opening up pathways for labour economists to extract information from large resume databases and use them in generic forecasting tasks. The paper is also clearly presented, both in terms of the models for career trajectories and wage forecasts, as well as the structure of the transformer model itself. The performance improvements observed both for job sequence forecasting and the wage regression validate the method proposed.\n\n\nWeaknesses\n---\n1) While results are stronger on average, uncertainty estimates would be required for Table 1 & 2 to help evaluate the significance of the improvements.\n2) For the transfer learning problem, it would also be good to evaluate the performance of the transformer model trained directly from the wage data — as a way to disentangle if gains are due to the model used or from the transfer learning approach.\n3) For non-experts in economics, additional background on direct applications of wage forecasting would also help strengthen the case/importance of the model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new transformer-based for representation learning of job sequence data (CAREER), which can then be ported to answer questions in labour economics via transfer learning (i.e. wage forecasting). This is achieved in a two step process. First, CAREER is trained to forecasts job sequences from large databases of online resume data —  demonstrating better prediction performance compared to other comparable benchmarks. Then, feature vectors from CAREER are combined with other observed covariates in a linear prediction model, and are shown to improve wage forecasts over just covariates alone.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1 & 2 in the weaknesses section would be critical for me, while 3 is more of a good to have.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Em8uTfBA9", "reviewer": "Reviewer_8Vvr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is a well written empirical study of using transformer models and the pretrain / fine-tune paradigm for job change prediction\n- The resulting model outperforms existing approaches \n- The appendix includes extensive training details \n\nWeaknesses\n- The paper doesn't include technical ML innovations; it relies on off the shelf methods. The paper might be more appropriate in a more applied venue than TMLR.\n- There could be more experiments considering different modeling decisions. \n- The model size might be overkill.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper pretrains a transformer model on a large dataset of resumes to predict sequences of job changes. The pretrained model is then fine-tuned on 3 smaller economic datasets. The finetuned transformer model outperforms existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Did the authors consider any other ways to incorporate covariates, e.g. as separate embeddings?\n\nDid the authors test models with fewer parameters? Based on the corpus size and complexity of the problem, 12 layers may be overkill.\n\nThe paper mentions the two stage model as being important noting that making previous approaches two-stage improves there performance, but it’s not clear to my that this would necessarily be the case with a transformer model. Did the authors test a version of the approach that is one-stage but includes pertaining?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yXZ4jrjGMV", "reviewer": "Reviewer_7qrm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* nice problem to be solved\n* superior quality to a single-source models\nWeaknesses:\n* No open source data released\n* Lack of implementation details: how exactly the transfer was performed? how exactly were features from the resumes extracted?\n* No ablation study", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work considers an important problem on the prediction of future occupation of a person using a large database of CVs and longitudinal surveys. They adopt the transfer learning approach to solve the problem at hand, as the CV's database is much large, but less accurate.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Right now the paper is better suited for a journal dedicated to analysis of labour market or some applied econometrics journal. To make it work for the audience of machine learning specialist I suggest the following changes.\n* The authors should specify all hyperparameters they use, the detailed model architecture, etc. The results should be fully reproducible.\n* Ablation studies would of separate interest: how we can improve the model by using a different language model, a different transfer learning strategy, etc.\n", "broader_impact_concerns": "The possible concerns are described by authors in the paper. The description there is good. I have nothing to add.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline reviews with three reviewers leaning towards acceptance and one leaning towards rejection. There was significant discussion between authors and one reviewer leaning towards acceptance, but not with the other three reviewers.

Several positive aspects were appreciated:

+ The problem was considered nice [7qrm] and the concrete application of representation and transfer learning was appreciated [bDhb]
+ Some reviewers considered the presentation clear [bDhb] and the paper well-written [8Vvr]
+ The algorithmic approach was considered well-grounded [o9LQ]
+ Incorporation of categorical sequences by initial embedding was considered interesting [o9LQ]
+ Pointing out limitations of claims from resume databases was appreciated [o9LQ]
+ Discussion of prior work was appreciated [o9LQ]
+ The description of the CAREER model construction was considered clear [o9LQ]
+ The experiments [o9LQ] and the shown performance improvements [bDhb,8Vvr,7qrm] were appreciated
+ The training details in the appendix were appreciated [8Vvr]

Many criticisms of the paper were also raised by the reviewers:

- Lack of technical ML innovations was criticized, even considering whether the paper would fit in an applied journal better than TMLR [8Vvr,7qrm]; authors argued the paper fits TMLR's criteria.
- The motivation of the approach was considered insufficient and lacking detail about the data use in the econometric domain, the typical predictive tasks, their importance (including background on direct applications of wage forecasting [bDhb]), and how informative the experiments are for the use case [o9LQ]. Authors mentioned some discussion in a response.
- Lack of detail about data sets was criticized along a number of aspects:
  --- how they are formed, [o9LQ]
  --- numbers of occupations and of covariate values, [o9LQ]
  --- career sequence lengths, [o9LQ]
  --- distribution similarities/differences between pretraining data and survey data sets, [o9LQ]
  --- consistency of the number of occupations across data in the paper [o9LQ]
  --- their domain specific usage and how they relate to claims [o9LQ]
- Whether the pretraining corpus is novel was considered unclear [o9LQ]; authors argued their novelty is using it "in a transfer learning approach to target survey data" 
- Lack of detail about the modeling approach was criticized in a number of ways:
  --- how the predicted job-change indicator affects the next-time-step input [o9LQ]; 
  --- how covariates were ultimately used in experiments [o9LQ]; 
  --- description of representations as complex/strong based on architecture alone without analysis of the encodings [o9LQ]; 
  --- lack of formally including the the objective or loss function [o9LQ]
  --- Similarly, one reviewer wished more detail on how the transfer was performed and how features were extracted from resumes
  --- One reviewer wishes clearer specification of hyperparameters and the model architecture; authors replied the appendix provides details.
  Authors stated that a number of details are provided in appendices.
- There was concern about possible posterior collapse and whether regularizations are used [o9LQ]
- One reviewer had a concern that the model size with 12 layers might be excessive, and desired discussion of models with fewer parameters [8Vvr]; authors mentioned discussion in the appendix
- To demonstrate advantage of a two-stage model, there was a desire to compare to a one-stage model with pretraining [8Vvr]; authors commented briefly on this.
- One reviewer wished discussion of other ways to incorporate covariates for example as separate embeddings [8Vvr]
- The experiments were criticized, including lacking definition of career prediction and forecasting tasks [o9LQ]
- Regarding baselines, there was concern whether the CAREER advantage arises more from the transformer architecture than the pretraining approach, and  whether other baselines like pretrained representations using NEMO or pretrained-but-not-finetuned CAREER would perform as well [o9LQ]; a baseline transformer model trained directly from the wage data was also desired [bDhb]. One reviewer more generally desired experiments considering different modeling decisions [8Vvr]; and another wished an ablation study [7qrm]. Authors argued in a response that a transferred version of NEMO would not be a fair comparison but provided a comparison to that; authors also provided some results of a non-transferred baseline, and provided some discussion of a baseline with covariates, a baseline of CAREER without finetuning, 
- The analyses were criticized in various ways: 
  --- use of perplexity only vs. also AUROC/ AUPRC or F1 score [o9LQ]; authors provided some AUC results in a response, but the reviewer still felt there was an inconsistency.
  --- not reporting confidence intervals in Tables 1/2 [o9LQ,bDhb]; authors provided some in a revision
  --- lacking detail whether NEMO was pretrained using the resume dataset and hence fairness of the comparison [o9LQ]; 
  --- lacking analysis of the attention weights [o9LQ]; authors argued that "the NLP domain has largely moved away from relying on attention-based explanations"
- Due to lack of various details, it was unclear whether claims were supported by the paper [o9LQ]

Due to the large number of clarity concerns, I suggest to do a minor revision as a clarification pass, to make sure the concerns mentioned in reviews and in author replies are clear enough in the revised paper. Since authors pointed to the appendices in several of their responses, it may be useful to make these links clearer where the information in appendices is central for understanding.
Decision: Accept (poster)