PLAsTiCC:Convincing other people tosolve your problems
Kara Ponder
Berkeley Center for Cosmological Physics
Computational Data Science Fellow
kponder@berkeley.edu
AI Seminar @ SLAC 02/20/2020

Outline➢ Why do we need someone else to solve our problems?
➢➢ PLAsTiCC: Photometric LSST Astronomical Time-Series
Classification Challenge➢➢ Other ways to convince people to solve your problems:
COIN○ Data driven mock galaxy catalogs○ Recommendation System for Spectroscopic follow-up

Type Ia Supernova●●●●Thermonuclear explosion ofa white dwarf starAt peak brightness, 5 billion
times brighter than the sun!
Explode with approximately
the same mass so we
assume they are standard
candles
Need spectroscopy todistinguish them from othersupernovae
Image credit: Brian Hayden

Spectroscopy versus Photometry

Image credit: Stephen Smartt, Nature (2012)

Image credit: Dan Kasen

What’s the appeal of photometry?
It’s cheap!

Supernova Cosmology

Scolnic et al (2018), ApJ, 859, 101S

State-of-the-art: 1048 SNe Ia fromPantheon.
25 years to gather ~1000 SNe Ia

Cosmology with photometric classification

SpectroscopicDistance Modulus

Photometricw/biascorrections
90%
Original
probability
Photometric

ΩM
Image credits: Hlozek et al (2012) ApJ, 752, 2, 79

Algorithms for Photometric Classification
●
●
●
●
●
●
●
●
●
●
●
●

PSNID (Sako et al 2011)
PELICAN (Pasquet el at 2019)
SuperNNova (Moller et al 2020)
snmachine (Lochner et al 2016)
Dai et al (2018)
Ishida et al (2019) & Ishida, de Souza (2013)
Villar et al (2019)
Karpenka et al (2013)
Newling et al (2011)
Bloom et al (2012) - transients & variables
Godines et al (2019) - microlensing
Eyer et al (2004), Kim et al (2015) - variablestars

At least dozens morefor other transientsand variable!

Comparing different classification methods

NaiveBayes

K-Nearest
Neighbors

Artificial
NeuralNetwork(MLP)

Figure credit: Lochner et al (ApJS, 2016), arXiv: 1611.07042

Support
VectorMachineBoostedDecision
Trees

AUC: Area Under ROC Curve -Closer to 1 is better

Major issues for photometric classification
Machine learning techniques are dependent on large,
homogenous and representative training sets...
Training

Test

Training

Test

Figure credits: Ishida et al., 2019 MNRAS - arXiv:1804.03765

Algorithms for Photometric Classification
●
●
●
●
●
●
●
●
●
●
●
●

PSNID (Sako et al 2011)
PELICAN (Pasquet el at 2019)
SuperNNova (Moller et al 2020)
snmachine (Lochner et al 2016)
Dai et al (2018)
Ishida et al (2019) & Ishida, de Souza (2013)
Villar et al (2019)
Karpenka et al (2013)
Newling et al (2011)
Bloom et al (2012) - transients & variables
Godines et al (2019) - microlensing
Eyer et al (2004), Kim et al (2015) - variablestars

At least dozens morefor other transientsand variable!

● All are limited byrepresentativeness.
● No single algorithm canclassify all transientsand variables.
● All lack earlyclassification abilities.

In the future, will weneed photometricclassification?

Vera Rubin Observatory
Legacy Survey of Space and Time (LSST)
●
●
●
●
●
●

Previously: Large Synoptic Survey
Telescope
8.4 meter mirror
10 year survey beginning 2023
6 photometric filters: UV to NIR
3.2 Gpixel camera the size of asmall car being built here at SLAC!
Two observing patterns:○○

Wide Fast Deep (WFD)
Deep Drilling Field (DDF)

Image Credit: LSSTImage credit: LSSTC

Era of Big Data●

With up to 10 million
transient alerts per night, thecommunity will be drowningin possible transients!

●●

LSST will give 10,000
spectroscopicallyconfirmed Type Ia SNe over
10 years!

●●

But what about the otherpossible 100,000s
supernovae observed?

How can we bring the groups together?
We’ll have a stream of up to 10 million transients and variables per night

With such large, distinct
communities, we decided
to use a data challenge tounite them.

Photometric LSST Astronomical Time-Series
Classification Challenge: PLAsTiCC
On Behalf of the PLAsTiCC Team:Tarek Allam Jr., Anita Bahmanyar, Rahul Biswas, Alexandre Boucaud, Lluís Galbany, Renée Hložek,Emille E. O. Ishida, Saurabh W. Jha, David O. Jones, Richard Kessler, Michelle Lochner, Ashish A.
Mahabal, Alex I. Malz, Kaisey S. Mandel, Juan Rafael Martínez-Galarza, Jason D. McEwen, Daniel
Muthukrishna, Gautham Narayan, Hiranya Peiris, Christina M. Peters, Kara Ponder, Christian N. Setzer,
The LSST Dark Energy Science Collaboration,
The LSST Transients, Variable Stars Science Collaboration

Other photometric classification challenges:
SNPhotCC
Asked 2 questions:
How well can you
classify a Type Ia SN?
Can we classify early? -No one participated inthis part→→→→

Held in 2010 to prepare for the Dark Energy
Survey (DES)
Only had Supernova Ia, Ibc, II templates
Thousands of objects
Only available to Astronomy community

Slide credit: Gautham Narayan

The Question
What question do you
want participants toanswer?

How well can you
classify ALL transientsand variables?
Additional question
Anomaly detection: If we had objects in the test set thatwere not in the training set, how well are they classified?

Proclam:https://github.com/aimalz/proclam

The Metric● Needs to cover all
different transient andvariable classes

Templates●● Probabilistic● Must interface withKaggle -> need single
number○ Unable to havemultiplechallenges

Wavelets

Malz et al (2019, including K.Ponder)
AJ, 158, 5, 171

Collecting models tosimulate data

Galactic

SNe

19 Models in total
18 with data
14 in training set
Kessler et al. 2019, PASP(including K.Ponder)
Slide credit: Rick Kessler

Representativeness102

Training set based offspectroscopic sample●●●●

Brighter objects
Lower redshifts
More well-sampled lightcurves
Different percentages inthe training set than in thetest set

Image credit: Connor Sheere

Validate thesimulations

Because we have SIMULATED data, there areseveral areas where we may introduce biases or
non-physical correlations:●●

Every box is a potential
source for errors
The source code(SNANA) had neverbeen used for galactictransients

Image credit: Kessler et al (2019)

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our data

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our data

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our data

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our data

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our data

Method to the madness: How to validate PLAsTiCC●●

Each model had at least two validators each time the full set of simulations
were regenerated
A data scientist from Kaggle also reviewed our dataBy the numbers●

More than 1 million new SEDs across several new models in SNANA●~3.5 million objects in the test set with <8,000 objects for training○●

15 classes in the test set, 14 in the training set~450 million observations in over 6 filter bands (18.5 GB)●

Even simplified, PLAsTiCC is the largest simulation ever of light
curves in the time domain sky in the optical
Slide credit: Gautham Narayan

Run the Challenge!

Preparing to interact with community: Starter Kit
What is LSST?
What is cadence?

Starter kit

What is photometry?
Spectroscopy?

What is a Hubble diagram?
What is Milky Waydust?

What does the telescope see?

What is redshift?

Need to continue to be engaged

KaggleDonatedby Kaggle

Vital tosuccess ofchallenge

September 28, 2018 - December 17, 2018

Avocado: Boone 2019 AJPELICAN: Pasquet et al 2019 A&A

Arxiv: 1909.05032

Avocado: Boone 2019 AJPELICAN: Pasquet et al 2019 A&A

Arxiv: 1909.05032

Avocado: Boone 2019 AJPELICAN: Pasquet et al 2019 A&A

Arxiv: 1909.05032

What components did the top 20 solutions have?
Developed by Astronomers

Schematic credit: Mi Dai

LightGBM
XGBoostGradient Boosting

CatBoost
Many solutions
forked a Kernel with
LightGBM Boosted
Decision Trees

Convolutional Neural Networks (CNN)

Neural Net

Recurrent Neural Networks (RNN)
Multi Layer Perceptron (MLP)

Autoencoders

Binary Classification

●

These will be useful for futurestudies in anomaly detection
even though they failed for
PLAsTiCC.

Type Ia
Galactic●No one was able to classify
the unknown unknowns
Most people probed theleaderboard
Or weighted the classes basedon the discussion boards

Supernovae●ExtraGalacticThings that failed: Anomaly Detection

Hlozek, Ponder, et al, in prep

What have we learned?
●
●
●
●
●
●It seems to help to be an astronomer
Data Augmentation may help with non-representativenessUtilizing many different methods and combining them may help
Boosted Decision Trees work well with our data
People think astronomy is cool, but they like to test their model and move onto the next opportunity○○○●●Science Competition until January 17, 2019
We required documented code on GitHub -- Received 4 entrees
All entrees invited to LSST Supernova Science Collaborations meeting this April in Illinois

Data scientists don’t have ALL the answers, but they help
We still have a lot of work to do!

PLAsTiCC 2.0●PLAsTiCC was not perfect…
○
○
○
○

Sta
yObjects were not placed in real galaxies
Some of the models were lacking variation
We only had 18 models!
Catalog only models, could have started from simulated images

tuned!

○●Kaggle dictated we only ask one question○We had many! Anomaly detection, Early classification, Type Ia Supernova○●

We are still deciding what questions to ask and what community to ask themto!

Other ways to convince people to solve
your problems...

Cosmostatistics Initiative
Aim: To create an interdisciplinary community arounddriven problems in astronomy

data

Method: COIN Residence Programs (CRPs)
●
●
●
●
●
●

Unstructured meetings
12 or less people
Goal to finish a project in one week
Researchers in astronomy, statistics, computer science and related fields
Allows for collaborative research and expertise sharing with concrete goalsCreate a nurturing and intimate environment to build connections

Led by Rafael de Souza, Emille Ishida, Alberto Krone-Martinshttps://cosmostatistics-initiative.org/

CRP #6: Chamonix, France
Before the meeting, 14
project ideas were presented.
Chose 3 projects for the week
My project(co-first author A. Malz):
Use data driven methods togenerate mock galaxy
catalogs.
August 2019

No connection to my previous work other than its use for cosmology.
Used my transferable skills: data analysis, statistics, methodology,python, git, ...

Standard procedure:N-bodysimulation

Known cosmology
Dark Matter particle
3-positions➔ particle correlation
function

3-velocities
Redshift

Slide Credit: Alex Malz

Halo model &Inpainting
scheme➔ Smeared out smallscales➔➔ Dependent onfunctional
approximation

Their mockcatalog
Known cosmology
Galaxy angular positions➔ galaxy correlation
function

Redshift
Spectra (Photometry)Our approach:N-bodysimulation

Our mockcatalog

Real Galaxy
catalog

Known cosmology

Known cosmology

Unknown cosmology

Dark Matter particle
3-positions➔ particle correlation
function

Galaxy angular positions➔ galaxy correlation
function➔ “environment”

Galaxy angular positions➔ galaxy correlation
function➔ “environment”

3-velocities
Redshift

Redshift
Photometry

Redshift
Spectra (Photometry)

Slide Credit: Alex Malz

Create any photometricbands from spectra

Define environment
Everyenvironment
curve has
associatedphotometry

Malz & Ponder et al, in prep

Start with real datafrom Galaxy and
Mass Assembly(GAMA) catalog.
~140,000 galaxies~7 redshift bins

Create data driven galaxy property model● Unsupervised Learning○ Time Series K-means clustering onenvironment curves○ Retains vector information○ How many clusters are supported?
● Model Per redshift bin● Compare photometry per cluster

Too many

For each ENVIRONMENT cluster, fit thesynthesized PHOTOMETRY withmultivariate GaussianMalz & Ponder et al, in prepApply model to mock catalog particles● Draw sampled particles from
N-body simulation (SLICS-HR)

Harnois-Deraps et al (2018)https://slics.roe.ac.uk/●Malz & Ponder et al, in prep

All plots on example runs.

Apply model to mock catalog particles● Draw sampled particles from
N-body simulation (SLICS-HR)●● Calculate environment ofmock catalog object●Malz & Ponder et al, in prep

All plots on example runs.

Apply model to mock catalog particles● Draw sampled particles from
N-body simulation (SLICS-HR)●● Calculate environment ofmock catalog object●● Classify environment with
K-means model●

Malz & Ponder et al, in prep

All plots on example runs.

Apply model to mock catalog particles● Draw sampled particles from
N-body simulation (SLICS-HR)●● Calculate environment ofmock catalog object●● Classify environment with
K-means model●● Randomly draw photometry
from appropriate environment
clusterMalz & Ponder et al, in prep

All plots on example runs.

Apply model to mock catalog particles● Draw sampled particles from
N-body simulation (SLICS-HR)●● Calculate environment ofmock catalog object

Red: GAMA
Blue: Mock●

● Classify environment with
K-means model●● Randomly draw photometry
from appropriate environment
cluster● Now you have a mock catalog!
Malz & Ponder et al, in prep

All plots on example runs.

Goals:➢ Provide proof of concept➢ Launch a platform for people to build theirown mock catalogs○ Inputs: Favorite N-body simulation andgalaxy survey

Active Learning for
Photometric Classification ofSupernovae●●●CRP #4
Get spectrum for most uncertain
objectFeed back into training set to
improve photometric classification

We can use this infrastructure to build
recommendation systems.

Ishida et al., 2019 - arXiv:astro-ph/1804.03765

RESSPECT
Recommendation System forSpectroscopic Follow-Up

Follow-up
Strategy

Goal: Build a sample forphotometric supernova
cosmology.
Telescope Resources

PhotometricCosmology*
Ishida et al., 2019 - arXiv:astro-ph/1804.03765

●●

There are non-traditional ways of doing research.
Decide what is right for your problem
○
○
○
○

●

Are Data Challenges are right for you?
○●How many different communities are interested? How many people are involved?
How much time will take to answer the question?
Do you need to collect new data? Generate new data? Is the hardware/software available?
Why hasn’t this problem been solved?
PLAsTiCC■ Needed to unify disparate communities■ New methods appearing quickly in industry data science■ Needed to motivate a powerhouse to get new simulations

Will you solve
someone else’sproblem?

Do you need to build a new community?
○○

Do you have a lot of small problems or highly focused problems?
COIN■ Goal to build up the Astrostatistics and Astroinformatics community by building individual
connections through focused projects■ Interfacing with existing collaborations to connect experts to non-experts to utilize skills
across COIN and the LSST DESC

VRO, image credit: LSSTC

Thank you!
●

Photometric classification is hard

●PLAsTiCC opened opportunities for new approaches●PLAsTiCC was an overall success! The model paper alone has more than 15citations○●We will be able to do more focused SN Ia Cosmology and cadence studies

COIN: Mock Galaxy Catalog: Outcome will be package or web interface thatallows people to create data-driven mock galaxy catalogs●RESSPECT: Building a recommendation system to maximize cosmology witha photometric sample

Kara Ponder

kponder@berkeley.edu