Package: wild-time-data
This repository provides a simpler interface to access theWild-Time datasetsin PyTorch.
In contrast to the original repository, this repository contains only code relevant for data loading
and has fewer dependencies.
Furthermore, we no longer use pickle to store the data but store it as HDF5 which has two advantages.
First, this is more space-efficient, e.g., for the yearbook dataset the data file size changed from 1.7GB to only 37MB.
Second, it addresses a security concern people may have when using unknown pickled Python objects.InstallationWild-Time-Data is available via PyPI.pipinstallwild-time-dataUsageThe following code will return a PyTorch dataset for the training partition of the arXiv dataset in 2023.
The data will be downloaded towild-time-data/unless it was downloaded into this folder before.fromwild_time_dataimportload_datasetload_dataset(dataset_name="arxiv",time_step=2023,split="train",data_dir="wild-time-data")In the following we provide details about the available argument options.dataset_name: The options are arxiv, drug, fmow, huffpost, and yearbook. This list can be accessed viafromwild_time_dataimportlist_datasetslist_datasets()time_step: Most datasets are grouped by year, this argument will allow you to access the data from different time intervals. The range differs from dataset to dataset. Use following command to get a list of available time steps:fromwild_time_dataimportavailable_time_stepsavailable_time_steps("arxiv")split: Selects the partition. Can either betrainortest.transform: Defines custom transformations on the predictors of a data point. Can be used to normalize, augment or tokenize data. By default, no transformation is applied for text datasets, image are converted to Tensors viatransforms.ToTensor(), and the data forDrugis one-hot encoded. Additionally,FMoWimages are normalized. The default transformation can be accessed viafromwild_time_dataimportdefault_transformdefault_transform("huffpost")target_transform: Same astransformbut for labels. By default, no transformation is applied.in_memory: If set toTrue, all data is loaded in memory. By default, data is loaded from disk which might be slower but requires less memory. For all datasets butFMoWin_memory=Trueshould work on most hardware.data_dir: Location where to store the data. By default it will be downloaded to~/wild-time-data/.Other Useful FunctionsSeveral other functions can be imported fromwild_time_data.fromwild_time_dataimportavailable_time_steps,input_dim,list_datasets,num_outputsavailable_time_steps: Given the dataset name, a sorted list of available time steps is returned. Example:available_time_steps("huffpost")returns[2012, 2013, 2014, 2015, 2016, 2017, 2018].default_transform: Given the dataset name, the transformation which is applied to the predictors unless a custom transformation is passed. Example:default_transform("yearbook")returnstransfroms.ToTensor(). If the return value isNone, no transformation is applied. In order to override a default transformation, passtransform=lambda x: xtoload_dataset.input_dim: Given the dataset name, the input dimensionality is returned. For image datasets the shape of the image is returned. For text datasets the maximum number of words separated by spaces is returned. Example:input_dim("yearbook")returns(1, 32, 32).list_datasets: Returns the list of all available datasets. Example:list_datasets()returns["arxiv", "drug", "fmow", "huffpost", "yearbook"].num_outputs: Given the dataset name, either the number of classes is returned or it returns 1. In cases where 1 is returned, this indicates that this is a regression dataset. Example:num_outputs("arxiv")returns172.FMoW DatasetIf you want to use the FMoW dataset, please followthe instructions to prepare it.LicensesAll additional code for Wild-Time-Data is available under the Apache 2.0 license.
The license for each Wild-Time dataset is listed below:arXiv: CC0: Public DomainDrug-BA: MIT LicenseFMoW:The Functional Map of the World Challenge Public LicenseHuffpost: CC0: Public DomainYearbook: MIT LicenseFurthermore, this repository is loosely based on theWild-Time repositorywhich is licensed under the MIT License.