Package: views-tensor-utilities
views_tensor_utilitiesThis package is a set of tools to allow users to transfer data in typical VIEWS format between pandas DataFrames and
numpy arrays (referred to as tensors).VIEWS dataframesVIEWS dataframes contain one or more features of panel data indexed by a two-column pandas MultiIndex. The first index
column is a time unit (e.g. month, year) and the second is a spatial unit (most commonly country or priogrid cell).
Missing data is represented by NaNs.Months currently range from 1 (Jan 1980) to 852 (December 2050). There is no month 0. Countries and priogrid cells
are denoted by non-consecutive integers.A crucial difference between the country and priogrid spatial units is that the definition of priogrid is fixed in
time, so all cells exist for all time units, butthis is not the case for countries- countries sometimes cease
to exist, or come into existence during the temporal range of a dataset. In a VIEWS dataframe, this is trivially
represented by omitting the relevant (time-unit, space-unit) units of analysis.Pandas dataframes are able to store numerical and string data in the same panel.Pandas dataframes are able to store strings giving names to the index columns and feature columns.VIEWS tensorsVIEWS tensors come in two forms. For the purposes of regression models, 3-dimensional tensors are used, with the
dimensions being (time-unit, space-unit, feature). For neural-net-based models and visualisation, 4-dimensional
tensors with dimensions (longitude-unit, latitude-unit, time-unit, feature) are used.Tensor indices are contiguous sets of integers starting from 0.Ordinary Numpy arrays cannot be used to store mixed numeric and string data.Ordinary Numpy arrays do not store names for their axes or axis values.Representing VIEWS dataframes as tensorsTheviews_tensor_utilitiespackage represents dataframes by wrappers around pure numpy arrays, accompanied by
minimal metadata to capture essential information from the dataframe which cannot be stores in these arrays,
such that it is possible to reconstruct the original dataframe (possibly with some reordering of the columns).
Each ViewsNumpy object holds a tensor containingonlynumerical data oronlystring data, the column
names from the original dataframe corresponding to the data stored in the tensor, and the values of the tokens
used to represent missing data and non-existent units of analysis.The ViewsContainer classAn entire dataframe is represented by the ViewsTensorContainer class. This holdsa list of ViewsNumpy objectsthe index of the original dataframeThe methods belonging to this class areto_pandas: guard method which checks whether the container tensors are 3D (which can be converted back to a
datafame) or 4D (which currently cannot) and respectively executes the conversion or returns an errorspace_time_to_panel: method which converts the containers tensors to dataframes, combines them into a single
dataframe and returns itget_numeric_tensor: convenience method which retrieves the numeric tensor component, if it exists.get_string_tensor: convenience method which retrieves the string tensor component, if it exists.The ViewsNumpy classThis is a simple wrapper for a single numpy tensor containingeithernumericorstring data. It holdsa numpy array representing a 3D time-space-feature tensor or a 4d longitude-latitude-time-feature tensora list of columns names corresponding to the indices of the tensor's last (i.e. 3rd or 4th) dimensiona value for the does-not-exist token used to denote units-of-analysis that do not exista value for the missing token denoting legal units-of-analysis with undefined valuesThis class has no methods.The ViewsDataframe classThis class holdsa pandas dataframethe index of the pandas dataframea list of dataframes formed by splitting the original dataframe into numeric and string portionsa transformer function, selected according to whether the input dataframe is strideableThe methods belonging to this class are__split_by_dtype: protected method which splits the input dataframe into 'number' and 'object' dataframes
according to its column datatypes. If this fails to assign all the original dataframes to one of the
split dataframes, an error is raisedto_numpy_time_space: this method calls __split_by_dtype, casts the split dataframes into (time, space,
feature) tensors. These are wrapped into ViewsNumpy objects which also store the column names belonging to
the split tensors, and the does-not-exist and missing tokens used in building the tensors. The ViewsNumpy
objects and the original dataframe index are packed into a ViewsTensorContainer object which is returned.to_numpy_longlat: this method first calls to_numpy_time_space, then casts the (time, space, feature)
tensors to (longitude, latitude, time, feature) tensors, before returning the modified ViewsTensorContainer.ExamplesConverting a VIEWS dataframe into tensorsThis is done by instantiating the ViewsDataframe classviews_dataframe = objects.ViewsDataframe(df)The commandtensor_container=views_dataframe.to_numpy_time_space()generates a tensor container containing one of more ViewsNumpy objects wrapping the numeric and/or string
portions of the dataframe's data. These can be accessed bytensor=tensor_container.ViewsTensors[0].tensorAlternatively, the convenience methods can be usedtensor=tensor_container.get_numeric_tensor()ortensor=tensor_container.get_string_tensor()Converting a ViewsTensorContainer into a pandas DataFrameThis is done by calling the container's to_pandas method:df=tensor_container.to_pandas()Note that the columns in the regenerated dataframe will likely not be in the same order as in the original
dataframe.