Introduction
Since their early days, representation in random utility behavior models has followed generally quite clear principles. For example, numeric quantities like travel time and cost may be directly used or transformed depending on observed non-linear efects (e.g. using log). Numeric variables that are not “quantities" per se, such as age or even geographic coordinates tend to be discretized and then transformed into vectors of dummy variables. Similarly, categorical variables such as education level or trip purpose are already discrete, and thus are also usually “dummyfied". Then, we may interact any subset of the above by combining (typically, multiplying) them, as long as we get in the end a vector of numeric values that can be incorporated in a statistical model, a linear one in the case of the most common logit model.
There are however phenomena that are hard to represent, and modelers end up struggling to find the right representation. For example, influence of social interactions between different persons, hierarchical decision making, autocorrelated nature of time and space, or abstract concepts such as accessibility, attitudes, personality traits and so on. The point here, is that the nature of our models seems to enforce a compromise between the true semantics of a variable (i.e. the “meaning" of a certain information for the decision making process) and its realisation in practice. And that further research should be done to find new representation paradigms.
Historically speaking, the natural language processing (NLP) field has had similar dilemmas for decades, and for a while two general trends were competing: the statistical modeling approaches, and the linguistic theory based approaches. The former relied on simple representations, such as vector frequencies, or dummy variables, to become practical, while the latter used domain knowledge such as grammars or logic. Until recently, neither had considerable success in making machines able to understand or generate human language, but developments in deep neural networks together with overwhelmingly massive amounts of data (i.e. the World Wide Web) brought them to a new area, where the two are approaching each other and achieving hitherto results considered extremely hard, such as question answering, translation, next word prediction. One of the key concepts in this revolution is that of embeddings, which will be further explained in this paper.
Our focus here is on the representation of categorical variables. The default paradigm is dummy variables (also known as “one-hot-encoding" in machine learning literature), which have well-known limitations, namely the explosion of dimensionality and enforced ortogonality. The former happens because we assign one new “dummy" variable to each of D-1 categories, and easily go from a small original variable specification to one with hundreds of variables, bringing problems in model estimation and analysis. This often affects the data collection process itself. Since one doesn't want to end up with too many categories, we might as well give less options in a survey, or decrease the resolution of a sensor. The problem of enforced ortogonality relates to the fact that, in a dummy encoding, all categories become equidistant. The similarity between “student" and “employed" is the same as between “student" and “retired", which in many cases (e.g. mode choice, departure time choice) goes against intuition. Other encoding methods exist, such as contrasted encoding or principal components analysis (PCA). The former ends up being a subtle variation on the dummy approach, but the latter already provides an interesting answer to the problem: categories are no longer forcibly equidistant, and the number of variables can be much reduced. However, it is a non-supervised approach. The distance between “student" and “employed" will always be the same, regardless of the problem we are solving, but this may be intuitively illogical if we consider car ownership versus departure time choice models for example.
The key idea in this paper is to introduce a method, called Travel Behavior embeddings, that borrows much from the NLP concept. This method serves to encode categorical variables, and is dependent on the problem at hand. We will focus on mode choice, and test on a well-known dataset, by comparing with both dummy and PCA encoding. All the dataset and code are made openly available, and the reader can follow and generate results him/herself using an iPython notebook included. Our ultimate goal is certainly that the reader reuses our PyTre package for own purposes.
This paper presents some results and conclusions, after a relatively long exploration and analysis process, including other datasets and code variations not mentioned here for interest of clarity and replicability. While we show these concepts to be promising and innovative in this paper, one should be wary of over-hyping yet another Machine Learning/Artificial Intelligence concept: after all, Machine Learning is still essentially based on statistics. In NLP, the number of different words in consideration at a given moment can be in order of tens of thousands, while our categorical variables rarely go beyond a few dozens. This means that, for example, it becomes clear later that the least number of original categories, the less the benefit of embeddings (in the limit, a binary variable like gender, is useless to do embeddings with), and also that if we do get a significantly large and statistically representative dataset, a dummy variables representation is sufficient. We will quickly see, however, that complexity can grow quick enough to justify an embeddings based method even if without the shockingly better performance observed in NLP applications.
Representing categorical variables
We are generally concerned with random utility maximization (RUM) models, for they have a dominant role in travel behavior modeling. The nature of such models is predominantly numeric, linear, and quite often strictly flat (notwithstanding hierarchical variations, such as nested models BIBREF1, hierarchical Bayes BIBREF2, or non-linear transformations). As a consequence, while numerical variables (e.g. travel time, cost, or income) can be directly used as available, perhaps subject to transformations or segmentation, nominal ones bring about a greater challenge. We tend to enforce a limited set of treatments such as:
Dummy variables, or one-hot encoding - for each categorical variable $v$ with D categories, we get D-1 binary variables (the “dummies"). At each input vector $x_n$, with categorical value $v=d$, the value “1" will be assigned to the corresponding dummy, while “0" to all others. If $v$ corresponds to the “default" category, all dummies are “0".
Contrast encoding BIBREF3 - same as dummy encoding, but instead of “1" for each category, we have a value that results from a contrasting formula. There are many different formulas (e.g. Helmert, Sum, Backward Difference), but all consist of subtracting the mean of the target variable, for a given category, with a general stastic (e.g. the mean of the dependent variable for all categories; the mean of the dependent variable in the previous category in an ordered list).
Principal Components Analysis (PCA) - run the PCA algorithm on the data matrix obtained by dummy representation of the categorical variable, then re-represent it with the corresponding projected eigenvector coefficients. One selects K eigenvectors (e.g. according to a variance explained rule), and thus each category is mapped to a vector of K real values.
Segmenting models, mixture models - A general alternative to categorical data representation is in fact to avoid it in the first place. One obvious method would be through creating hierarchical disaggregate methods (e.g. one per category). This is not in itself a representation paradigm, but an alternative way to see this problem. It certainly raises scalability and inference concerns.
In datasets where behavior heterogeneity is high, and number of observations is significantly smaller than population size, increasing dimensionality by adding a variable per each category is very risky because the amount of data that is in practice usable to estimate each new coefficient becomes insufficient. A simple intuition here is by considering that, for a dummy variable that is only “1" for a few observations in the dataset, its coefficient will be “activated" only that small number of times. If there is a lot of variance in the associated behavior, the variance of the coefficient will also be large, and the coefficient will be considered statistically insignificant.
The benefit of representations that map into a latent space, like embeddings and PCA, is that such a space is inevitably shared, and thus every observation contributes indirectly to all category variables. This comes with no interpretability cost, because one can always map to the “dummy" space and analyse the individual coefficients, as will be shown in our experiments.
The concept of text embeddings
The idea of text embeddings comes from a simple re-representation necessity. A natural-language processing system is itself also a numeric machine, therefore it requires each individual word in a dictionary to match its own numeric representation. Just as in our travel models, a possible solution has been to use dummy variables, and it is quite obvious that the dimensionality of such a one-hot encoding vector, quickly becomes overwhelming. Think for example next word prediction algorithm, like the one we have in our smartphones. It is essentially a skip-gram BIBREF4 model that predicts the next word, given the n words before. The English dictionary has about 300000 words, and if we have about 5 words before for context, the number of independent variables of the model would become 1.5 million!
The goal of text embeddings algorithms (e.g. Word2Vec BIBREF5) is to a) reduce the representation of each word to a computationally acceptable dimension, while simultaneously b) learning the semantic distance between different words. In other words, the euclidean distance of semantically related words (e.g. “dog" and “cat") in this new space should be smaller than unrelated words (e.g. “dog" and “optimize"). As mentioned before, in a dummy (or one-hot) encoding, all distances between words are equal by definition.
The word embeddings methodology is very well explained in several webpages such as BIBREF6, so the reader is strongly encouraged to visit them first. However, for the sake of completeness, we summarize here the general idea.
Imagine the following task: given a word $w_i$ in a text, predict the next word $w_o$. If we solve it with a neural network model, we could have the architecture in Figure FIGREF8, where the input consists simply of the one-hot-encoding representation of the word (i.e. one dummy variable for each word in a dictionary of dimensionality $D$), and the output corresponds to the probability of each word in the dictionary being the next one (also a vector with dimensionality $D$).
The output layer thus consists simply of a softmax function. In other words, exactly the classical multinomial logit formulation that we would have in an RUM, in which each different word corresponds to an “alternative".
The concept of embeddings is directly associated to the hidden layer, which is a set of linear activation neurons, typically with a dimensionality $K<<D$. Each such neuron is simply an identity function: it sums all inputs; then propagates this sum to the output layer. Since only one input neuron is activated at a time (remember that the input is a one-hot-encoding vector, with one “1" and the rest with “0"), each hidden layer neuron just propagates the (single) weight that links to that input neuron. If we have enough data for training this model, we will eventually land on a situation where, for each input word, there is a fixed vector of weights that are directly used in the output (softmax) function, to generate the prediction. With more data, this weight vector will not change (down to some small delta threshold). These stable vectors are what we call embeddings, and the dimensionality of these vectors is called embedding size.
Formally, we have a dataset $\mathcal {D}=\lbrace x_n, y_n\rbrace , n=1\ldots N$, where each $x_n$ and $y_n$ are one-hot (dummy) encodings of categorical variables. The dimensionality of $x_n$ is $D\times 1$, with $D$ being the number of different categories in $x_n$, while the dimensionality of $y_n$ is $C\times 1$, with $C$ being the number of categories (alternatives) in $y_n$. The full expression for the embeddings model as described is:
where $W$ is the embeddings matrix of size $K\times D$, where $K$ is called the embeddings size. $B$ is a matrix of coefficients ($C\times K$) for the softmax layer, so $B_c$ is simply the coefficients (row) vector for output class (alternative) $c$, and $\alpha _c$ is the corresponding intercept. The typical loss function used in such models is called the categorical cross entropy:
Where $\delta _{i}$ is the kronecker delta ($\delta _{true}=1; \delta _{false}=0$), and $\mathcal {L}(n)$ is the cumulative loss for an individual data point. This formalization is the simplest version, without loss of generality. In practice, as seen below, we will model multiple embeddings matrices simultaneously, and will add regularization terms to the loss function, so the models tested in this paper consist of compositions of the above.
So these so called embeddings are in fact a relatively shallow data representation in a simple neural network. What is their added value? Obviously, the first practical benefit is dimensionality reduction, because now there is a mapping between each of the $C$ words to a unique vector of size $K$. The second aspect is that this new representation is the one that maximizes the performance towards a specific task (in our example, prediction of the next word), therefore it is a supervised process, as opposed for example to PCA. The third and more interesting aspect relates with semantic similarity. A natural consequence of the mentioned algorithm is that words that have similar output distributions (i.e. next words) will tend to be close to each other. Figure FIGREF10 shows a 2D visualization (t-SNE) with a subset of english words. In such a visualization, data is projected in 2D space by maintaining the same vector-to-vector distances as in the original ($K$ order space). Therefore the X and Y axes have no specific meaning, only distances between every pair of points are relevant.
We can see that semantically similar concepts, more specifically concepts that tend to have the same distribution of “next words", are placed closer. Another intriguing consequence is that, since the words are now in the $K$ dimensional, embeddings space, we can also do some linear algebra on them. A well known formulation is $King-Man+Woman=Queen$. Essentially, the vector $King-Man$ corresponds to the concept of “crowning" (therefore $Woman+crowning=Queen$). The same could be done with many other concept pairs. Figure FIGREF11 show also an alternative interpretation of “man-female", as well as examples with cities and verb tense.
Finally, another relevant note on the embeddings representation is that, just like the PCA encoding, one can always project back into the original space and use this for interpretability. In other words, since there is a 1-to-1 mapping from each category to its encoding, there is also a 1-to-1 mapping between a model that uses dummy variables and a model using such encodings. This may be useful for interpretability, since in the case of dummy variables we have a direct interpretation (e.g. a beta coefficient value in a logit model) for the effect of a given category, while the same doesn't happen for an encoded variable (i.e. there is no meaning for the value of a single beta coefficient in an embeddings encoding when K>1). In order to preserve statistical significance information (e.g. p-values) we only need to follow the well known rules of normal random variables.
There are open databases available (e.g. GLoVe BIBREF9, FastText BIBREF7) that provide word embedding tables for the entire English language (Glove provides several embedding tables, up to embedding size between 100 and 300). In our next word application example, we now talk about models with 500-1500 variables, which is very manageable for our machines today.
Summarizing, the general idea of word embeddings is to re-represent a categorical variable into a lower dimensional representation with continuous values . Whenever such a variable is to be used in a model, one can simply replace it with the corresponding embeddings vector. We have previously demonstrated the value of such word embeddings in demand prediction in special events BIBREF10, where we collected event textual descriptions, and used Glove embedding vectors to incorporate such information in a neural network model.
Finally, an interesting point to mention relates to the typical difference in dataset size between the original embeddings training model (Glove, approximately 6 billion input word vectors from 37 million texts) and the model one implements to solve a particular problem (in our special events case, less than 1000 short event descriptions, with at most few hundred words each). Instead of creating ourselves a new embeddings model using the events dataset, we reused the pre-trained GloVe dataset. The benefit is significant because in practice we trained our model to deal with all words in the dictionary, much beyond the limited vocabulary that we obtained in our 1000 short texts. In practice we have used a very small percentage of the english dictionary. When, in an out-of-sample test, our model finds words that were not in the training set, it still works perfectly well.
Travel behaviour embeddings
Differently to textual data, our goal in this paper is to explore the large amount of categorical data that is often collected in travel surveys. This includes trip purpose, education level, or family type. We also consider other variables that are not necessarily of categorical nature, but typically end up as dummy encoding, due to segmentation, such as age, income, or even origin/destination pair.
Our hypothesis is that, given the limitations of dummy variables that are commonly used and the unsupervised nature of PCA, using instead an embeddings mechanism should improve significantly the quality of our models, both in terms of loglikelihood but also in terms of allowing for lower complexity (i.e. less variables). Ultimately, one could think of a framework such as GLoVe, where embeddings for such variables could be trivially shared with the community. For example, we could have a “Travel behavior embeddings" database, incrementally built from travel surveys from around the world. Such database could have embeddings for mode choice target variables, but also for departure time, destination choice, car ownership, and so on. Whenever a modeler wanted to estimate a new model, she could just download the right encodings and use them directly. This is particularly relevant if one considers the complicated challenges for opening or sharing travel survey datasets in our field. Of course, a major question arises: are behaviors that consistent across the world? There are certainly nuances across the world, but we believe that general patterns would emerge (e.g. a “business" trip purpose will be closer to “work" than “leisure", in a departure time choice model; “student" will be closer to “unemployed" than to “retired" in a car ownership model).
Travel behaviour embeddings ::: The general idea
We believe that, as with word embeddings, a mapping that preserves semantic distance relative to a certain choice problem, should be useful for modeling. As with a PCA encoding, another benefit is that by sharing parameters in the learning process, the model can generalize better, as opposed to a dummy encoding, where each categorical value has its own parameter, that is only active when observed.
The general idea is thus to create a mapping between a variable for which we want to find an embeddings representation, and a target variable, as in Figure FIGREF15. We call the mapping function “PyTre Embeddings", because that is the name of the object in our proposed Python “Travel Embeddings" package.
From an experimental design and application perspective, the approach followed in this paper is the following:
Create list of categorical variables to encode (the encoding set)
Split dataset into train, development and test sets
For each variable in encoding set, learn the new embeddings using the embeddings train set . This should be done simultaneously (all variable embeddings estimated at once, as explained in the next section).
Encode choice models for train, development and test sets using the learned embeddings
Estimate choice model accordingly using its train set
Evaluate the new model using the test set
Since there is stochasticity in the embeddings training model, we will repeat the above multiple times, for the different experiments in the paper (and report the respective mean and standard deviation statistics). Whenever we want to analyse a particular model (e.g. to check the coefficients of a choice model), we select the one with the highest likelihood at the development set (i.e. in practice, its out-of-sample generalization performance), and report its performance on the test set.
Travel behaviour embeddings ::: Methodology
Since a choice model will typically involve other variables than the categorical ones that we learn the embeddings for, it is important to take into account their effects. Figure FIGREF24 shows the simplest travel embeddings model. As an example, the categorical variable is trip purpose, and there are a few other variables such as gender, cost of the alternatives, distance, and so on. Notice that they are directly fed into the softmax output layer, together with the embeddings output.
The dataset sizes in transportation behavior modeling are substantially smaller than typical word embeddings ones, and the risk of overfitting is therefore higher. To mitigate this problem, besides adding regularization penalties in the objective function, we add what we call a regularizer layer for each embedding, which is no more than a softmax layer that penalizes whenever it cannot recover the original one-hot-encoding vectors (Figure FIGREF25, left). We call the combination of embeddings and its regularizer network, a Travel Embeddings layer. Finally, it is obviously better to train all embeddings simultaneously, so that they accommodate each other's effects (Figure FIGREF25, right).
An experiment with mode choice
The goal of this paper is to test the potential of embeddings in a simple and well-known choice model context, comparing it to well-known baseline techniques. Therefore, the general model specification follows quite simple assumptions. We expect that in future work from us or others, more elaborate derivations can take advantage of embeddings such as nested, mixed logit or latent class choice models (LCCM), for example.
We will apply the methodology to the well-known “Swissmetro" dataset. We will compare it with a dummy variables and PCA baselines. We will follow the 3-way experimental design mentioned before: split the dataset into train, development and test sets, so that the embeddings, PCA eigenvectors and the choice model are estimated from the same train and development sets, and validate it out-of-sample. For the sake of interpretability, we will also project back coefficients from the embeddings as well as PCA models into the dummy variable space.
All experiment code is available as a jupyter notebook in a package we created for this work (to which we called PyTre). For estimating the multinomial logit model (MNL) we used the PyLogit BIBREF11 package.
An experiment with mode choice ::: The Swissmetro dataset
The Swissmetro dataset consists of survey data collected on the trains between St. Gallen and Geneva, Switzerland, during March 1998. According to its description BIBREF0, the respondents provided information in order to analyze the impact of the modal innovation in transportation, represented by the Swissmetro, a revolutionary mag-lev underground system, against the usual transport modes represented by car and train. After discarding respondents for which some variables were not available (e.g. age, purpose), a total of 10469 responses from 1188 individuals were used for the experiments.
We split the dataset into 3 different parts:
Embeddings train set: 60% of the dataset (6373 vectors)
Development set: 20% of the dataset (2003 vectors)
Test set: 20% of the dataset (2003 vectors)
An experiment with mode choice ::: Principles for the model specification
The PyLogit package BIBREF11 also uses Swissmetro as an example. Therefore, our model specifications will extend the default one from this package. We re-estimated this model with the train set and validated with testset. The results are shown in tables TABREF31 and TABREF32. Since we are comparing the models at the test set, the key indicators should be pseudo R-square and log-likelihood. Indicators that consider model complexity (robust r-square and AIC) are less important on the test set in our view because the overfitting effect (i.e. improving fit just by adding more variables) will no longer be verifiable in this way. Instead, one sees overfitting if test set performance is considerably inferior to the training set.

Question: What datasets are used for evaluation?
Answer: Swissmetro dataset

Question: How do their train their embeddings?
Answer: The embeddings are learned several times using the training set, then the average is taken.

Question: How do they model travel behavior?
Answer: The data from collected travel surveys is used to model travel behavior.

Question: How do their interpret the coefficients?
Answer: The coefficients are projected back to the dummy variable space.