Package: galaxy-datasets
galaxy-datasetsML-friendly datasets for major Galaxy Zoo citizen science campaigns.PyTorch Datasets and PyTorch Lightning DataModulesTensorFlow tf.data.Dataset'sFramework-independent (i.e. TensorFlow-friendly) download and augmentation codeNameMethodPyTorch DatasetPublishedDownloadableGalaxiesGalaxy Zoo 2gz2GZ2☑☑~210k (main sample)GZ Hubble*gz_hubbleGZHubble☑☑~106k (main sample)GZ CANDELSgz_candelsGZCandels☑☑~50kGZ DECaLS GZD-5gz_decals_5GZDecals5☑☑~230k (GZD-5 only)GZ Ringsgz_ringsGZRings☒☑~93kGZ DESIgz_desiGZDesi☑No (500GB)8.7MGZ Cosmic Dawn (HSC)gz_cosmic_dawnGZCosmic☒☑~48kCFHT Tidal*tidalTidal☑☑1760 (expert)Any datasets marked as downloadable but not marked as published are only downloadable internally (for development purposes).For each dataset, you must cite/acknowledge the GZ data release paper and the original telescope survey from which the images were derived. Seedata.galaxyzoo.orgfor the data release paper citations to use.We also include small debugging datasets:NameMethodPyTorch DatasetDownloadableGalaxiesDemo Rings (binary)demo_ringsDemoRings☑1000Galaxy MNIST (four-class)galaxy_mnistGalaxyMNIST☑10kGalaxy MNIST is alsoavailableas a pure torchvision dataset (exactly like MNIST).*GZ Hubble is also available in "euclidised" form (i.e. with the Euclid PSF applied) to Euclid collaboration members. The method isgz_hubble_euclidised. Courtesy of Ben Aussel.**CFHT Tidal is not a Galaxy Zoo dataset, but rather a small expert-labelled dataset of tidal features fromAtkinson 2013.
MW reproduced and modified the images inWalmsley 2019. We include it here as a challenging fine-grained morphology classification task with little labelled data.InstallationInstallingzoobotwill automatically install this package as a dependency.To install directly:pip install galaxy-datasets[pytorch]for PyTorch dependenciespip install galaxy-datasets[tensorflow]for TensorFlow dependenciespip install galaxy-datasets[pytorch,tensorflow]for bothFor local development (e.g. adding a new dataset), you can install this by cloning from github, then runningpip install -e .in the cloned repo root. This makes changing the code easier than if you don't use the -e, in which case the package is installed under sitepackages.I suggest either:For basic use without changes, installingzoobotvia pip and allowing pip to manage this dependencyFor development, installing bothzoobotandgalaxy-datasetsvia gitUsageCheck out the PyTorch quickstart Colabhere, or keep reading for more explanation.Framework-IndependentTo download a dataset:from galaxy_datasets import gz2  # or gz_hubble, gz_candels, ...

catalog, label_cols = gz2(
    root='your_data_folder/gz2',
    train=True,
    download=True
)This will download the images and train/test catalogs toroot. Eachcatalogis a pandas DataFrame with the columnfile_locgiving absolute image paths and additional columnslabel_cols = ['col_a', 'col_b', ...]giving the labels (usually, the number of volunteers who gave each answer for each galaxy). Iftrain=True, the method returns the train catalog, otherwise, the test catalog.If training Zoobot from scratch, this is all you need. For example, in PyTorch:from zoobot.pytorch.training import train_with_pytorch_lightning

train_with_pytorch_lightning.train_default_zoobot_from_scratch(
    catalog=catalog,
    save_dir=save_dir,
    schema=gz2_schema, # see zoobot/pytorch/examples/minimal_example.py
    ...
)Otherwise, you might like to use the classes in this package to load these catalogs into ML-friendly inputs.PyTorchCreate a PyTorch Dataset from a catalog like so:from galaxy_datasets.pytorch.galaxy_dataset import GalaxyDataset  # generic Dataset for galaxies

dataset = GalaxyDataset(
    catalog=catalog.sample(1000),  # from gz2(...) above
    label_cols=['smooth-or-featured-gz2_smooth'],
    transform=optional_transforms_if_you_like
)Notice how you can adjust the catalog before creating the Dataset. This gives flexibility to try training on e.g. different catalog subsets.If you don't want to change anything about the catalog, you can skip the framework-independent download and use a named class fromgalaxy_datasets.pytorch, which takes the same arguments and directly gives a Dataset:from galaxy_datasets.pytorch import GZ2

gz2_dataset = GZ2(
    root='your_data_folder/gz2',
    train=True,
    download=False
)
image, label = gz2_dataset[0]
plt.imshow(image)You might also find the PyTorch Lightning DataModule undergalaxy_datasets/pytorch/galaxy_datamoduleuseful. Zoobot uses this for training and finetuning.from galaxy_datasets.pytorch.galaxy_datamodule import GalaxyDataModule

datamodule = GalaxyDataModule(
    label_cols=['smooth-or-featured-gz2_smooth'],
    catalog=catalog
    # optional args to specify augmentations
)

datamodule.prepare_data()
datamodule.setup()
for images, labels in datamodule.train_dataloader():
    print(images.shape, labels.shape)
    breakTensorFlowTo create a tf.data.Dataset from a catalog:import tensorflow as tf
from galaxy_datasets.tensorflow.datasets import get_image_dataset, add_transforms_to_dataset
from galaxy_datasets.transforms import default_transforms  # same transforms as PyTorch

train_dataset = get_image_dataset(
    image_paths = catalog['file_loc'],
    labels=catalog[label_cols].values,
    requested_img_size=224
)

# specify augmentations
transforms = default_transforms()

# apply augmentations
train_dataset = add_transforms_to_dataset(train_dataset, transforms)

# batch, shuffle, prefetch for performance
train_dataset = train_dataset.shuffle(5000).batch(64).prefetch(tf.data.experimental.AUTOTUNE)

for images, labels in train_dataset.take(1):
    print(images.shape, labels.shape)
    breakDownload NotesDatasets are downloaded like:{root}imagessubfolder (except GZ2)image.jpg{catalog_name(s)}.parquetThe whole dataset is downloaded regardless of whethertrain=Trueortrain=False.