Abstract: This paper introduces the concept of travel behavior embeddings, a method for re-representing discrete variables that are typically used in travel demand modeling, such as mode, trip purpose, education level, family type or occupation. This re-representation process essentially maps those variables into a latent space called the \emph{embedding space}. The benefit of this is that such spaces allow for richer nuances than the typical transformations used in categorical variables (e.g. dummy encoding, contrasted encoding, principal components analysis). While the usage of latent variable representations is not new per se in travel demand modeling, the idea presented here brings several innovations: it is an entirely data driven algorithm; it is informative and consistent, since the latent space can be visualized and interpreted based on distances between different categories; it preserves interpretability of coefficients, despite being based on Neural Network principles; and it is transferrable, in that embeddings learned from one dataset can be reused for other ones, as long as travel behavior keeps consistent between the datasets.  ::: The idea is strongly inspired on natural language processing techniques, namely the word2vec algorithm. Such algorithm is behind recent developments such as in automatic translation or next word prediction. Our method is demonstrated using a model choice model, and shows improvements of up to 60\% with respect to initial likelihood, and up to 20% with respect to likelihood of the corresponding traditional model (i.e. using dummy variables) in out-of-sample evaluation. We provide a new Python package, called PyTre (PYthon TRavel Embeddings), that others can straightforwardly use to replicate our results or improve their own models. Our experiments are themselves based on an open dataset (swissmetro).

Introduction: Since their early days, representation in random utility behavior models has followed generally quite clear principles. For example, numeric quantities like travel time and cost may be directly used or transformed depending on observed non-linear efects (e.g. using log). Numeric variables that are not “quantities" per se, such as age or even geographic coordinates tend to be discretized and then transformed into vectors of dummy variables. Similarly, categorical variables such as education level or trip purpose are already discrete, and thus are also usually “dummyfied". Then, we may interact any subset of the above by combining (typically, multiplying) them, as long as we get in the end a vector of numeric values that can be incorporated in a statistical model, a linear one in the case of the most common logit model. There are however phenomena that are hard to represent, and modelers end up struggling to find the right representation. For example, influence of social interactions between different persons, hierarchical decision making, autocorrelated nature of time and space, or abstract concepts such as accessibility, attitudes, personality traits and so on. The point here, is that the nature of our models seems to enforce a compromise between the true semantics of a variable (i.e. the “meaning" of a certain information for the decision making process) and its realisation in practice. And that further research should be done to find new representation paradigms. Historically speaking, the natural language processing (NLP) field has had similar dilemmas for decades, and for a while two general trends were competing: the statistical modeling approaches, and the linguistic theory based approaches. The former relied on simple representations, such as vector frequencies, or dummy variables, to become practical, while the latter used domain knowledge such as grammars or logic. Until recently, neither had considerable success in making machines able to understand or generate human language, but developments in deep neural networks together with overwhelmingly massive amounts of data (i.e. the World Wide Web) brought them to a new area, where the two are approaching each other and achieving hitherto results considered extremely hard, such as question answering, translation, next word prediction. One of the key concepts in this revolution is that of embeddings, which will be further explained in this paper. Our focus here is on the representation of categorical variables. The default paradigm is dummy variables (also known as “one-hot-encoding" in machine learning literature), which have well-known limitations, namely the explosion of dimensionality and enforced ortogonality. The former happens because we assign one new “dummy" variable to each of D-1 categories, and easily go from a small original variable specification to one with hundreds of variables, bringing problems in model estimation and analysis. This often affects the data collection process itself. Since one doesn't want to end up with too many categories, we might as well give less options in a survey, or decrease the resolution of a sensor. The problem of enforced ortogonality relates to the fact that, in a dummy encoding, all categories become equidistant. The similarity between “student" and “employed" is the same as between “student" and “retired", which in many cases (e.g. mode choice, departure time choice) goes against intuition. Other encoding methods exist, such as contrasted encoding or principal components analysis (PCA). The former ends up being a subtle variation on the dummy approach, but the latter already provides an interesting answer to the problem: categories are no longer forcibly equidistant, and the number of variables can be much reduced. However, it is a non-supervised approach. The distance between “student" and “employed" will always be the same, regardless of the problem we are solving, but this may be intuitively illogical if we consider car ownership versus departure time choice models for example. The key idea in this paper is to introduce a method, called Travel Behavior embeddings, that borrows much from the NLP concept. This method serves to encode categorical variables, and is dependent on the problem at hand. We will focus on mode choice, and test on a well-known dataset, by comparing with both dummy and PCA encoding. All the dataset and code are made openly available, and the reader can follow and generate results him/herself using an iPython notebook included. Our ultimate goal is certainly that the reader reuses our PyTre package for own purposes. This paper presents some results and conclusions, after a relatively long exploration and analysis process, including other datasets and code variations not mentioned here for interest of clarity and replicability. While we show these concepts to be promising and innovative in this paper, one should be wary of over-hyping yet another Machine Learning/Artificial Intelligence concept: after all, Machine Learning is still essentially based on statistics. In NLP, the number of different words in consideration at a given moment can be in order of tens of thousands, while our categorical variables rarely go beyond a few dozens. This means that, for example, it becomes clear later that the least number of original categories, the less the benefit of embeddings (in the limit, a binary variable like gender, is useless to do embeddings with), and also that if we do get a significantly large and statistically representative dataset, a dummy variables representation is sufficient. We will quickly see, however, that complexity can grow quick enough to justify an embeddings based method even if without the shockingly better performance observed in NLP applications.

Question: How do their train their embeddings?
Answer: The embeddings are learned several times using the training set, then the average is taken.