Textual Data for Time Series Forecasting

Abstract: While ubiquitous, textual sources of information such as company reports, social media posts, etc. are hardly included in prediction algorithms for time series, despite the relevant information they may contain. In this work, openly accessible daily weather reports from France and the United-Kingdom are leveraged to predict time series of national electricity consumption, average temperature and wind-speed with a single pipeline. Two methods of numerical representation of text are considered, namely traditional Term Frequency - Inverse Document Frequency (TF-IDF) as well as our own neural word embedding. Using exclusively text, we are able to predict the aforementioned time series with sufficient accuracy to be used to replace missing data. Furthermore the proposed word embeddings display geometric properties relating to the behavior of the time series and context similarity between words.

Whether it is in the field of energy, finance or meteorology, accurately predicting the behavior of time series is nowadays of paramount importance for optimal decision making or profit. While the field of time series forecasting is extremely prolific from a research point-of-view, up to now it has narrowed its efforts on the exploitation of regular numerical features extracted from sensors, data bases or stock exchanges. Unstructured data such as text on the other hand remains underexploited for prediction tasks, despite its potentially valuable informative content. Empirical studies have already proven that textual sources such as news articles or blog entries can be correlated to stock exchange time series and have explanatory power for their variations BIBREF0, BIBREF1. This observation has motivated multiple extensive experiments to extract relevant features from textual documents in different ways and use them for prediction, notably in the field of finance. In Lavrenko et al. BIBREF2, language models (considering only the presence of a word) are used to estimate the probability of trends such as surges or falls of 127 different stock values using articles from Biz Yahoo!. Their results show that this text driven approach could be used to make profit on the market. One of the most conventional ways for text representation is the TF-IDF (Term Frequency - Inverse Document Frequency) approach. Authors have included such features derived from news pieces in multiple traditional machine learning algorithms such as support vector machines (SVM) BIBREF3 or logistic regression BIBREF4 to predict the variations of financial series again. An alternative way to encode the text is through latent Dirichlet allocation (LDA) BIBREF5. It assigns topic probabilities to a text, which can be used as inputs for subsequent tasks. This is for instance the case in Wang's aforementioned work (alongside TF-IDF). In BIBREF6, the authors used Reuters news encoded by LDA to predict if NASDAQ and Dow Jones closing prices increased or decreased compared to the opening ones. Their empirical results show that this approach was efficient to improve the prediction of stock volatility. More recently Kanungsukkasem et al. BIBREF7 introduced a variant of the LDA graphical model, named FinLDA, to craft probabilities that are specifically tailored for a financial time series prediction task (although their approach could be generalized to other ones). Their results showed that indeed performance was better when using probabilities from their alternative than those of the original LDA. Deep learning with its natural ability to work with text through word embeddings has also been used for time series prediction with text. Combined with traditional time series features, the authors of BIBREF8 derived sentiment features from a convolutional neural network (CNN) to reduce the prediction error of oil prices. Akita et al. BIBREF9 represented news articles through the use of paragraph vectors BIBREF10 in order to predict 10 closing stock values from the Nikkei 225. While in the case of financial time series the existence of specialized press makes it easy to decide which textual source to use, it is much more tedious in other fields. Recently in Rodrigues et al. BIBREF11, short description of events (such as concerts, sports matches, ...) are leveraged through a word embedding and neural networks in addition to more traditional features. Their experiments show that including the text can bring an improvement of up to 2% of root mean squared error compared to an approach without textual information. Although the presented studies conclude on the usefulness of text to improve predictions, they never thoroughly analyze which aspects of the text are of importance, keeping the models as black-boxes.
The field of electricity consumption is one where expert knowledge is broad. It is known that the major phenomena driving the load demand are calendar (time of the year, day of the week, ...) and meteorological. For instance generalized additive models (GAM) BIBREF12 representing the consumption as a sum of functions of the time of the year, temperature and wind speed (among others) typically yield less than 1.5% of relative error for French national electricity demand and 8% for local one BIBREF13, BIBREF14. Neural networks and their variants, with their ability to extract patterns from heterogeneous types of data have also obtained state-of-the-art results BIBREF15, BIBREF16, BIBREF17. However to our knowledge no exploratory work using text has been conducted yet. Including such data in electricity demand forecasting models would not only contribute to close the gap with other domains, but also help to understand better which aspects of text are useful, how the encoding of the text influences forecasts and to which extend a prediction algorithm can extract relevant information from unstructured data. Moreover the major drawback of all the aforementioned approaches is that they require meteorological data that may be difficult to find, unavailable in real time or expensive. Textual sources such as weather reports on the other hand are easy to find, usually available on a daily basis and free.
The main contribution of our paper is to suggest the use of a certain type of textual documents, namely daily weather report, to build forecasters of the daily national electricity load, average temperature and wind speed for both France and the United-Kingdom (UK). Consequently this work represents a significant break with traditional methods, and we do not intend to best state-of-the-art approaches. Textual information is naturally more fuzzy than numerical one, and as such the same accuracy is not expected from the presented approaches. With a single text, we were already able to predict the electricity consumption with a relative error of less than 5% for both data sets. Furthermore, the quality of our predictions of temperature and wind speed is satisfying enough to replace missing or unavailable data in traditional models. Two different approaches are considered to represent the text numerically, as well as multiple forecasting algorithms. Our empirical results are consistent across encoding, methods and language, thus proving the intrinsic value weather reports have for the prediction of the aforementioned time series. Moreover, a major distinction between previous works is our interpretation of the models. We quantify the impact of a word on the forecast and analyze the geometric properties of the word embedding we trained ourselves. Note that although multiple time series are discussed in our paper, the main focus of this paper remains electricity consumption. As such, emphasis is put on the predictive results on the load demand time series.
The rest of this paper is organized as follows. The following section introduces the two data sets used to conduct our study. Section 3 presents the different machine learning approaches used and how they were tuned. Section 4 highlights the main results of our study, while section 5 concludes this paper and gives insight on future possible work.
In order to prove the consistency of our work, experiments have been conducted on two data sets, one for France and the other for the UK. In this section details about the text and time series data are given, as well as the major preprocessing steps.
Three types of time series are considered in our work: national net electricity consumption (also referred as load or demand), national temperature and wind speed. The load data sets were retrieved on the websites of the respective grid operators, respectively RTE (Réseau et Transport d'Électricité) for France and National Grid for the UK. For France, the available data ranges from January the 1st 2007 to August the 31st 2018. The default temporal resolution is 30 minutes, but it is averaged to a daily one. For the UK, it is available from January the 1st 2006 to December the 31st 2018 with the same temporal resolution and thus averaging. Due to social factors such as energy policies or new usages of electricity (e.g. Electric Vehicles), the net consumption usually has a long-term trend (fig. FIGREF2). While for France it seems marginal (fig. FIGREF2), there is a strong decreasing trend for the United-Kingdom (fig. FIGREF2). Such a strong non-stationarity of the time series would cause problems for the forecasting process, since the learnt demand levels would differ significantly from the upcoming ones. Therefore a linear regression was used to approximate the decreasing trend of the net consumption in the UK. It is then subtracted before the training of the methods, and then re-added a posteriori for prediction.
As for the weather time series, they were extracted from multiple weather stations around France and the UK. The national average is obtained by combining the data from all stations with a weight proportional to the city population the station is located in. For France the stations' data is provided by the French meteorological office, Météo France, while the British ones are scrapped from stations of the National Oceanic and Atmospheric Administration (NOAA). Available on the same time span as the consumption, they usually have a 3 hours temporal resolution but are averaged to a daily one as well. Finally the time series were scaled to the range $[0,1]$ before the training phase, and re-scaled during prediction time.
Our work aims at predicting time series using exclusively text. Therefore for both countries the inputs of all our models consist only of written daily weather reports. Under their raw shape, those reports take the form of PDF documents giving a short summary of the country's overall weather, accompanied by pressure, temperature, wind, etc. maps. Note that those reports are written a posteriori, although they could be written in a predictive fashion as well. The reports are published by Météo France and the Met Office, its British counterpart. They are publicly available on the respective websites of the organizations. Both corpora span on the same period as the corresponding time series and given their daily nature, it yields a total of 4,261 and 4,748 documents respectively. An excerpt for each language may be found in tables TABREF6 and TABREF7. The relevant text was extracted from the PDF documents using the Python library PyPDF2.
As emphasized in many studies, preprocessing of the text can ease the learning of the methods and improve accuracy BIBREF18. Therefore the following steps are applied: removal of non-alphabetic characters, removal of stop-words and lowercasing. While it was often highlighted that word lemmatization and stemming improve results, initial experiments showed it was not the case for our study. This is probably due to the technical vocabulary used in both corpora pertaining to the field of meteorology. Already limited in size, the aforementioned preprocessing operations do not yield a significant vocabulary size reduction and can even lead to a loss of linguistic meaning. Finally, extremely frequent or rare words may not have high explanatory power and may reduce the different models' accuracy. That is why words appearing less than 7 times or in more than 40% of the (learning) corpus are removed as well. Figure FIGREF8 represents the distribution of the document lengths after preprocessing, while table TABREF11 gives descriptive statistics on both corpora. Note that the preprocessing steps do not heavily rely on the considered language: therefore our pipeline is easily adaptable for other languages.
A major target of our work is to show the reports contain an intrinsic information relevant for time series, and that the predictive results do not heavily depend on the encoding of the text or the machine learning algorithm used. Therefore in this section we present the text encoding approaches, as well as the forecasting methods used with them.
Machines and algorithms cannot work with raw text directly. Thus one major step when working with text is the choice of its numerical representation. In our work two significantly different encoding approaches are considered. The first one is the TF-IDF approach. It embeds a corpus of $N$ documents and $V$ words into a matrix $X$ of size $N \times V$. As such, every document is represented by a vector of size $V$. For each word $w$ and document $d$ the associated coefficient $x_{d,w}$ represents the frequency of that word in that document, penalized by its overall frequency in the rest of the corpus. Thus very common words will have a low TF-IDF value, whereas specific ones which will appear often in a handful of documents will have a large TF-IDF score. The exact formula to calculate the TF-IDF value of word $w$ in document $d$ is:
where $f_{d,w}$ is the number of appearances of $w$ in $d$ adjusted by the length of $d$ and $\#\lbrace d: w \in d \rbrace $ is the number of documents in which the word $w$ appears. In our work we considered only individual words, also commonly referred as 1-grams in the field of natural language processing (NLP). The methodology can be easily extended to $n$-grams (groups of $n$ consecutive words), but initial experiments showed that it did not bring any significant improvement over 1-grams.
The second representation is a neural word embedding. It consists in representing every word in the corpus by a real-valued vector of dimension $q$. Such models are usually obtained by learning a vector representation from word co-occurrences in a very large corpus (typically hundred thousands of documents, such as Wikipedia articles for example). The two most popular embeddings are probably Google's Word2Vec BIBREF19 and Standford's GloVe BIBREF20. In the former, a neural network is trained to predict a word given its context (continuous bag of word model), whereas in the latter a matrix factorization scheme on the log co-occurences of words is applied. In any case, the very nature of the objective function allows the embedding models to learn to translate linguistic similarities into geometric properties in the vector space. For instance the vector $\overrightarrow{king} - \overrightarrow{man} + \overrightarrow{woman}$ is expected to be very close to the vector $\overrightarrow{queen}$. However in our case we want a vector encoding which is tailored for the technical vocabulary of our weather reports and for the subsequent prediction task. This is why we decided to train our own word embedding from scratch during the learning phase of our recurrent or convolutional neural network. Aside from the much more restricted size of our corpora, the major difference with the aforementioned embeddings is that in our case it is obtained by minimizing a squared loss on the prediction. In that framework there is no explicit reason for our representation to display any geometric structure. However as detailed in section SECREF36, our word vectors nonetheless display geometric properties pertaining to the behavior of the time series.
Multiple machine learning algorithms were applied on top of the encoded textual documents. For the TF-IDF representation, the following approaches are applied: random forests (RF), LASSO and multilayer perceptron (MLP) neural networks (NN). We chose these algorithms combined to the TF-IDF representation due to the possibility of interpretation they give. Indeed, considering the novelty of this work, the understanding of the impact of the words on the forecast is of paramount importance, and as opposed to embeddings, TF-IDF has a natural interpretation. Furthermore the RF and LASSO methods give the possibility to interpret marginal effects and analyze the importance of features, and thus to find the words which affect the time series the most.
As for the word embedding, recurrent or convolutional neural networks (respectively RNN and CNN) were used with them. MLPs are not used, for they would require to concatenate all the vector representations of a sentence together beforehand and result in a network with too many parameters to be trained correctly with our number of available documents. Recall that we decided to train our own vector representation of words instead of using an already available one. In order to obtain the embedding, the texts are first converted into a sequence of integers: each word is given a number ranging from 1 to $V$, where $V$ is the vocabulary size (0 is used for padding or unknown words in the test set). One must then calculate the maximum sequence length $S$, and sentences of length shorter than $S$ are then padded by zeros. During the training process of the network, for each word a $q$ dimensional real-valued vector representation is calculated simultaneously to the rest of the weights of the network. Ergo a sentence of $S$ words is translated into a sequence of $S$ $q$-sized vectors, which is then fed into a recurrent neural unit. For both languages, $q=20$ seemed to yield the best results. In the case of recurrent units two main possibilities arise, with LSTM (Long Short-Term Memory) BIBREF21 and GRU (Gated Recurrent Unit) BIBREF22. After a few initial trials, no significant performance differences were noticed between the two types of cells. Therefore GRU were systematically used for recurrent networks, since their lower amount of parameters makes them easier to train and reduces overfitting. The output of the recurrent unit is afterwards linked to a fully connected (also referred as dense) layer, leading to the final forecast as output. The rectified linear unit (ReLU) activation in dense layers systematically gave the best results, except on the output layer where we used a sigmoid one considering the time series' normalization. In order to tone down overfitting, dropout layers BIBREF23 with probabilities of 0.25 or 0.33 are set in between the layers. Batch normalization BIBREF24 is also used before the GRU since it stabilized training and improved performance. Figure FIGREF14 represents the architecture of our RNN.
The word embedding matrix is therefore learnt jointly with the rest of the parameters of the neural network by minimization of the quadratic loss with respect to the true electricity demand. Note that while above we described the case of the RNN, the same procedure is considered for the case of the CNN, with only the recurrent layers replaced by a combination of 1D convolution and pooling ones. As for the optimization algorithms of the neural networks, traditional stochastic gradient descent with momentum or ADAM BIBREF25 together with a quadratic loss are used. All of the previously mentioned methods were coded with Python. The LASSO and RF were implemented using the library Scikit Learn BIBREF26, while Keras BIBREF27 was used for the neural networks.
While most parameters are trained during the learning optimization process, all methods still involve a certain number of hyperparameters that must be manually set by the user. For instance for random forests it can correspond to the maximum depth of the trees or the fraction of features used at each split step, while for neural networks it can be the number of layers, neurons, the embedding dimension or the activation functions used. This is why the data is split into three sets:
The training set, using all data available up to the 31st of December 2013 (2,557 days for France and 2,922 for the UK). It is used to learn the parameters of the algorithms through mathematical optimization.
The years 2014 and 2015 serve as validation set (730 days). It is used to tune the hyperparameters of the different approaches.
All the data from January the 1st 2016 (974 days for France and 1,096 for the UK) is used as test set, on which the final results are presented.
Grid search is applied to find the best combination of values: for each hyperparameter, a range of values is defined, and all the possible combinations are successively tested. The one yielding the lowest RMSE (see section SECREF4) on the validation set is used for the final results on the test one. While relatively straightforward for RFs and the LASSO, the extreme number of possibilities for NNs and their extensive training time compelled us to limit the range of architectures possible. The hyperparameters are tuned per method and per country: ergo the hyperparameters of a given algorithm will be the same for the different time series of a country (e.g. the RNN architecture for temperature and load for France will be the same, but different from the UK one). Finally before application on the testing set, all the methods are re-trained from scratch using both the training and validation data.
The goal of our experiments is to quantify how close one can get using textual data only when compared to numerical data. However the inputs of the numerical benchmark should be hence comparable to the information contained in the weather reports. Considering they mainly contain calendar (day of the week and month) as well as temperature and wind information, the benchmark of comparison is a random forest trained on four features only: the time of the year (whose value is 0 on January the 1st and 1 on December the 31st with a linear growth in between), the day of the week, the national average temperature and wind speed. The metrics of evaluation are the Mean Absolute Percentage Error (MAPE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE) and the $R^2$ coefficient given by:
where $T$ is the number of test samples, $y_t$ and $\hat{y}_t$ are respectively the ground truth and the prediction for the document of day $t$, and $\overline{y}$ is the empirical average of the time series over the test sample. A known problem with MAPE is that it unreasonably increases the error score for values close to 0. While for the load it isn't an issue at all, it can be for the meteorological time series. Therefore for the temperature, the MAPE is calculated only when the ground truth is above the 5% empirical quantile. Although we aim at achieving the highest accuracy possible, we focus on the interpretability of our models as well.
Many words are obviously irrelevant to the time series in our texts. For instance the day of the week, while playing a significant role for the load demand, is useless for temperature or wind. Such words make the training harder and may decrease the accuracy of the prediction. Therefore a feature selection procedure similar to BIBREF28 is applied to select a subset of useful features for the different algorithms, and for each type of time series. Random forests are naturally able to calculate feature importance through the calculation of error increase in the out-of-bag (OOB) samples. Therefore the following process is applied to select a subset of $V^*$ relevant words to keep:
A RF is trained on the whole training & validation set. The OOB feature importance can thus be calculated.
The features are then successively added to the RF in decreasing order of feature importance.
This process is repeated $B=10$ times to tone down the randomness. The number $V^*$ is then set to the number of features giving the highest median OOB $R^2$ value.
The results of this procedure for the French data is represented in figure FIGREF24. The best median $R^2$ is achieved for $V^* = 52$, although one could argue that not much gain is obtained after 36 words. The results are very similar for the UK data set, thus for the sake of simplicity the same value $V^* = 52$ is used. Note that the same subset of words is used for all the different forecasting models, which could be improved in further work using other selection criteria (e.g. mutual information, see BIBREF29). An example of normalized feature importance is given in figure. FIGREF32.
Note that most of the considered algorithms involve randomness during the training phase, with the subsampling in the RFs or the gradient descent in the NNs for instance. In order to tone it down and to increase the consistency of our results, the different models are run $B=10$ times. The results presented hereafter correspond to the average and standard-deviation on those runs. The RF model denoted as "sel" is the one with the reduced number of features, whereas the other RF uses the full vocabulary. We also considered an aggregated forecaster (abridged Agg), consisting of the average of the two best individual ones in terms of RMSE. All the neural network methods have a reduced vocabulary size $V^*$. The results for the French and UK data are respectively given by tables TABREF26 and TABREF27.
Our empirical results show that for the electricity consumption prediction task, the order of magnitude of the relative error is around 5%, independently of the language, encoding and machine learning method, thus proving the intrinsic value of the information contained in the textual documents for this time series. As expected, all text based methods perform poorer than when using explicitly numerical input features. Indeed, despite containing relevant information, the text is always more fuzzy and less precise than an explicit value for the temperature or the time of the year for instance. Again the aim of this work is not to beat traditional methods with text, but quantifying how close one can come to traditional approaches when using text exclusively. As such achieving less than 5% of MAPE was nonetheless deemed impressive by expert electricity forecasters. Feature selection brings significant improvement in the French case, although it does not yield any improvement in the English one. The reason for this is currently unknown. Nevertheless the feature selection procedure also helps the NNs by dramatically reducing the vocabulary size, and without it the training of the networks was bound to fail. While the errors accross methods are roughly comparable and highlight the valuable information contained within the reports, the best method nonetheless fluctuates between languages. Indeed in the French case there is a hegemony of the NNs, with the embedding RNN edging the MLP TF-IDF one. However for the UK data set the RFs yield significantly better results on the test set than the NNs. This inversion of performance of the algorithms is possibly due to a change in the way the reports were written by the Met Office after August 2017, since the results of the MLP and RNN on the validation set (not shown here) were satisfactory and better than both RFs. For the two languages both the CNN and the LASSO yielded poor results. For the former, it is because despite grid search no satisfactory architecture was found, whereas the latter is a linear approach and was used more for interpretation purposes than strong performance. Finally the naive aggregation of the two best experts always yields improvement, especially for the French case where the two different encodings are combined. This emphasises the specificity of the two representations leading to different types of errors. An example of comparison between ground truth and forecast for the case of electricity consumption is given for the French language with fig. FIGREF29, while another for temperature may be found in the appendix FIGREF51. The sudden "spikes" in the forecast are due to the presence of winter related words in a summer report. This is the case when used in comparisons, such as "The flood will be as severe as in January" in a June report and is a limit of our approach. Finally, the usual residual $\hat{\varepsilon }_t = y_t - \hat{y}_t$ analyses procedures were applied: Kolmogorov normality test, QQplots comparaison to gaussian quantiles, residual/fit comparison... While not thoroughly gaussian, the residuals were close to normality nonetheless and displayed satisfactory properties such as being generally independent from the fitted and ground truth values. Excerpts of this analysis for France are given in figure FIGREF52 of the appendix. The results for the temperature and wind series are given in appendix. Considering that they have a more stochastic behavior and are thus more difficult to predict, the order of magnitude of the errors differ (the MAPE being around 15% for temperature for instance) but globally the same observations can be made.
While accuracy is the most relevant metric to assess forecasts, interpretability of the models is of paramount importance, especially in the field of professional electricity load forecasting and considering the novelty of our work. Therefore in this section we discuss the properties of the RF and LASSO models using the TF-IDF encoding scheme, as well as the RNN word embedding.
One significant advantage of the TF-IDF encoding when combined with random forests or the LASSO is that it is possible to interpret the behavior of the models. For instance, figure FIGREF32 represents the 20 most important features (in the RF OOB sense) for both data sets when regressing over electricity demand data. As one can see, the random forest naturally extracts calendar information contained in the weather reports, since months or week-end days are among the most important ones. For the former, this is due to the periodic behavior of electricity consumption, which is higher in winter and lower in summer. This is also why characteristic phenomena of summer and winter, such as "thunderstorms", "snow" or "freezing" also have a high feature importance. The fact that August has a much more important role than July also concurs with expert knowledge, especially for France: indeed it is the month when most people go on vacations, and thus when the load drops the most. As for the week-end names, it is due to the significantly different consumer behavior during Saturdays and especially Sundays when most of the businesses are closed and people are usually at home. Therefore the relevant words selected by the random forest are almost all in agreement with expert knowledge.
We also performed the analysis of the relevant words for the LASSO. In order to do that, we examined the words $w$ with the largest associated coefficients $\beta _w$ (in absolute value) in the regression. Since the TF-IDF matrix has positive coefficients, it is possible to interpret the sign of the coefficient $\beta _w$ as its impact on the time series. For instance if $\beta _w > 0$ then the presence of the word $w$ causes a rise the time series (respectively if $\beta _w < 0$, it entails a decline). The results are plotted fig. FIGREF35 for the the UK. As one can see, the winter related words have positive coefficients, and thus increase the load demand as expected whereas the summer related ones decrease it. The value of the coefficients also reflects the impact on the load demand. For example January and February have the highest and very similar values, which concurs with the similarity between the months. Sunday has a much more negative coefficient than Saturday, since the demand significantly drops during the last day of the week. The important words also globally match between the LASSO and the RF, which is a proof of the consistency of our results (this is further explored afterwards in figure FIGREF43). Although not presented here, the results are almost identical for the French load, with approximately the same order of relevancy. The important words logically vary in function of the considered time series, but are always coherent. For instance for the wind one, terms such as "gales", "windy" or "strong" have the highest positive coefficients, as seen in the appendix figure FIGREF53. Those results show that a text based approach not only extracts the relevant information by itself, but it may eventually be used to understand which phenomena are relevant to explain the behavior of a time series, and to which extend.
Word vector embeddings such as Word2Vec and GloVe are known for their vectorial properties translating linguistic ones. However considering the objective function of our problem, there was no obvious reason for such attributes to appear in our own. Nevertheless for both languages we conducted an analysis of the geometric properties of our embedding matrix. We investigated the distances between word vectors, the relevant metric being the cosine distance given by:
where $\overrightarrow{w_1}$ and $\overrightarrow{w_2}$ are given word vectors. Thus a cosine distance lower than 1 means similarity between word vectors, whereas a greater than 1 corresponds to opposition.
The initial analyses of the embedding matrices for both the UK and France revealed that in general, words were grouped by context or influence on the electricity consumption. For instance, we observed that winter words were together and far away from summer ones. Week days were grouped as well and far from week-end days. However considering the vocabulary was reduced to $V^* = 52$ words, those results lacked of consistency. Therefore for both languages we decided to re-train the RNNs using the same architecture, but with a larger vocabulary of the $V=300$ most relevant words (still in the RF sense) and on all the available data (i.e. everything is used as training) to compensate for the increased size of the vocabulary. We then calculated the distance of a few prominent words to the others. The analysis of the average cosine distance over $B=10$ runs for three major words is given by tables TABREF38 and TABREF39, and three other examples are given in the appendix tables TABREF57 and TABREF58. The first row corresponds to the reference word vector $\overrightarrow{w_1}$ used to calculate the distance from (thus the distance is always zero), while the following ones are the 9 closest to it. The two last rows correspond to words we deemed important to check the distance with (an antagonistic one or relevant one not in the top 9 for instance).
The results of the experiments are very similar for both languages again. Indeed, the words are globally embedded in the vector space by topic: winter related words such as "January" ("janvier"), "February" ("février"), "snow" ("neige"), "freezing" ("glacial") are close to each other and almost opposite to summer related ones such as "July" ("juillet"), "August" ("août"), "hot" ("chaud"). For both cases the week days Monday ("lundi") to Friday ("vendredi") are grouped very closely to each other, while significantly separated from the week-end ones "Saturday" ("samedi") and "Sunday" ("dimanche"). Despite these observations, a few seemingly unrelated words enter the lists of top 10, especially for the English case (such as "pressure" or "dusk" for "February"). In fact the French language embedding seems of better quality, which is perhaps linked to the longer length of the French reports in average. This issue could probably be addressed with more data. Another observation made is that the importance of a word $w$ seems related to its euclidean norm in the embedding space ${\overrightarrow{w}}_2$. For both languages the list of the 20 words with the largest norm is given fig. FIGREF40. As one can see, it globally matches the selected ones from the RF or the LASSO (especially for the French language), although the order is quite different. This is further supported by the Venn diagram of common words among the top 50 ones for each word selection method represented in figure FIGREF43 for France. Therefore this observation could also be used as feature selection procedure for the RNN or CNN in further work.
In order to achieve a global view of the embeddings, the t-SNE algorithm BIBREF30 is applied to project an embedding matrix into a 2 dimensional space, for both languages. The observations for the few aforementioned words are confirmed by this representation, as plotted in figure FIGREF44. Thematic clusters can be observed, roughly corresponding to winter, summer, week-days, week-end days for both languages. Globally summer and winter seem opposed, although one should keep in mind that the t-SNE representation does not preserve the cosine distance. The clusters of the French embedding appear much more compact than the UK one, comforting the observations made when explicitly calculating the cosine distances.
In this study, a novel pipeline to predict three types of time series using exclusively a textual source was proposed. Making use of publicly available daily weather reports, we were able to predict the electricity consumption with less than 5% of MAPE for both France and the United-Kingdom. Moreover our average national temperature and wind speed predictions displayed sufficient accuracy to be used to replace missing data or as first approximation in traditional models in case of unavailability of meteorological features.
The texts were encoded numerically using either TF-IDF or our own neural word embedding. A plethora of machine learning algorithms such as random forests or neural networks were applied on top of those representations. Our results were consistent over language, numerical representation of the text and prediction algorithm, proving the intrinsic value of the textual sources for the three considered time series. Contrarily to previous works in the field of textual data for time series forecasting, we went in depth and quantified the impact of words on the variations of the series. As such we saw that all the algorithms naturally extract calendar and meteorological information from the texts, and that words impact the time series in the expected way (e.g. winter words increase the consumption and summer ones decrease it). Despite being trained on a regular quadratic loss, our neural word embedding spontaneously builds geometric properties. Not only does the norm of a word vector reflect its significance, but the words are also grouped by topic with for example winter, summer or day of the week clusters.
Note that this study was a preliminary work on the use of textual information for time series prediction, especially electricity demand one. The long-term goal is to include multiple sources of textual information to improve the accuracy of state-of-the-art methods or to build a text based forecaster which can be used to increase the diversity in a set of experts for electricity consumption BIBREF31. However due to the redundancy of the information of the considered weather reports with meteorological features, it may be necessary to consider alternative textual sources. The use of social media such as Facebook, Twitter or Instagram may give interesting insight and will therefore be investigated in future work.
Additional results for the prediction tasks on temperature and wind speed can be found in tables TABREF47 to TABREF50. An example of forecast for the French temperature is given in figure FIGREF51.
While not strictly normally distributed, the residuals for the French electricity demand display an acceptable behavior. This holds also true for the British consumption, and both temperature time series, but is of lesser quality for the wind one.
The the UK wind LASSO regression, the words with the highest coefficients $\beta _w$ are indeed related to strong wind phenomena, whereas antagonistic ones such as "fog" or "mist" have strongly negative ones as expected (fig. FIGREF53).
For both languages we represented the evolution of the (normalized) losses for the problem of load regression in fig. FIGREF54. The aspect is a typical one, with the validation loss slightly above the training one. The slightly erratic behavior of the former one is possibly due to a lack of data to train the embeddings.
The cosine distances for three other major words and for both corpora have been calculated as well. The results are given in tables TABREF57 and TABREF58. For both languages, the three summer months are grouped together, and so are the two week-end days. However again the results are less clear for the English language. They are especially mediocre for "hot", considering that only "warm" seems truly relevant and that "August" for instance is quite far away. For the French language instead of "hot" the distances to "thunderstorms" were calculated. The results are quite satisfactory, with "orageux"/"orageuse" ("thundery") coming in the two first places and related meteorological phenomena ("cumulus" and "grêle", meaning "hail") relatively close as well. For the French case, Saturday and Sunday are very close to summer related words. This observation probably highlights the fact that the RNN groups load increasing and decreasing words in opposite parts of the embedding space.

Figure and table captions:
Figure 1: Net electricity consumption (Load) over time.
Figure 2: Word counts for the two corpora after preprocessing.
Table 3: Descriptive analysis of the two corpora (after preprocessing)
Figure 3: Structure of our RNN. Dropout and batch normalization are not represented.
Figure 4: Evolution of the OOB R2 during the selection procedure.
Table 4: Forecast errors on the net load for the French Dataset.
Table 5: Forecast errors on the net load for the British Dataset.
Table 6: Best (individual, in terms of RMSE) result for each of the considered time series.
Figure 5: Overlapping of prediction and real load (France)
Figure 6: RF feature importance over the B = 10 runs.
Figure 7: Coefficients βw in the british load LASSO regression.
Table 7: Closest words (in the cosine sense) to ”february”,”snow” and ”tuesday” for the UK
Table 8: Closest words (in the cosine sense) to ”february”,”snow” and ”tuesday” for France
Figure 8: Word vector log-norm over B = 10.
Figure 9: Venn diagram of common words among the top 50 ones for each selection procedure (France).
Figure 10: 2D t-SNE projections of the embedding matrix for both languages.
Table A.9: Forecast errors on the national temperature for France.
Table A.10: Forecast errors on the national wind for France.
Table A.11: Forecast errors on the national temperature for Great-Britain.
Table A.12: Forecast errors on the national wind for Great-Britain.
Figure A.11: Overlapping of prediction and national Temperature (France)
Figure A.12: Residual analysis of the French aggregated predictor.
Figure A.13: Coefficients βw in the British wind LASSO regression.
Figure A.14: Loss (Mean Squared Error) evolution of the electricity demand RNN for both languages.
Table A.13: Closest words (in the cosine sense) to ”August”,”Sunday” and ”Hot” for the UK
Table A.14: Closest words (in the cosine sense) to ”August”,”Sunday and ”thunderstorms” for the France

Questions about this paper:
How big is dataset used for training/testing?
What geometric properties do embeddings display?
How accurate is model trained on text exclusively?

Answers:
4,261  days for France and 4,748 for the UK
Winter and summer words formed two separate clusters. Week day and week-end day words also formed separate clusters.
Relative error is less than 5%