Using Machine Learning to Predict Dying Stars in our Galaxy… and Beyond!
By: Timothy Eakin
Tags: Pulsar, Xgboost, Astronomy, Python, Data Science

Using Machine Learning to Predict Dying Stars in our Galaxy… and Beyond!

In this article, I demonstrate how machine-learning models can be used to correctly classify dying stars or more specifically, Pulsars, as they seem to be the most potentially useful for space exploration.

This will be a journey into predicting whether or not observations, made by high powered telescopes on Earth and potentially deep space probes in the future, are pulsars. Before we jump into the machine learning model I have developed to help identify pulsars, let’s talk a bit about what pulsars, or ‘pulsar stars’, actually are since they aren’t pulsating and actually aren’t technically stars (anymore).

Image via: PITRIS/GETTY IMAGES

What are pulsars?

Consider, for the sake of explanation, that stars have a life. (no offense to any stars here on Earth) Stars eventually die when they go supernova and collapse into a black hole if they are extremely massive. If they are less massive, between 7 and 25 solar masses (7–25 times the mass of our sun) or maybe a bit larger if they are especially metal-rich, they then become neutron stars, a super-dense mass only around 10 kilometers in radius but so dense that a teaspoon full of their mass would be as heavy as Mt. Everest if placed on Earth. Neutron stars continue to emit radiation from their collapsed core for millions of years until they eventually cool completely and become cold dead remnants of their once brilliant heavenly bodies.

A smaller subset of these star remnants called neutrons is one called ‘pulsars’ or ‘pulsar stars’. While the term ‘pulsar’ is a combination of the words ‘pulse’ and ‘star’, pulsars aren’t pulsating and are no longer living stars. Their electromagnetic emissions are continuous but beamed from their magnetic poles as seen in the photo above. The magnetic axis is not usually the same as the rotational axis, so they are called pulsars due to how, from any singular perspective, a pulse of radiation is observed each time the beam sweeps through the observer’s point of view. These sweeps are extremely regular in their interval since they occur with each rotation of the pulsar. It should be noted that pulsars can never ‘die’ since they are already actually dead stars, but eventually, over the course of probably more than a million years, their spins will slow and their emissions will cease. At this point, they would only become detectable from the gravity they exert on other celestial bodies or by detection of something called black-body radiation.

What do earthlings care about pulsars?

People on earth cannot observe most pulsar bodies or even their beams with our limited five senses. They have no effect on us, so you may be wondering: “What does this have to do with me or humankind as a whole?”

It takes a good amount of time, money, and effort to discover pulsar candidates and to then correctly identify whether those potential pulsars are true pulsars or not. This has been going on for over half a century and has undoubtedly involved many scientists and lots of funding to accomplish the identification of the over 2,000 known pulsars to date.

We already have a system of navigating the solar system with a network of radio antennas on Earth called the Deep Space Network. This does a pretty good job of determining how far a craft has flown and how much further it must go to reach its destination. The only issues with the DSN are the further away from earth a spacecraft gets, the less precise the DSN gets, and it can only determine the distance of the craft from Earth, not its lateral position in space. Using pulsars as a set of lighthouse beacons, a craft could determine exactly where it is positioned in three-dimensional space. This will become more important as more crafts are sent out to explore the outer reaches of our solar system and beyond.

Image via: Multiverse Hub

Now for the Data Science!

Those that are here for the witty banter and pictures may feel free to scroll past all this… it’s about to get extremely technical.

What you are about to witness, if you stick around, is what’s called “supervised machine learning”. I have all the verified labels, (pulsar or non-pulsar) for each observation in the data set. After splitting the data into train/validate/test sets I then will take away the answers for the validate and test sets and train my model using the answers on a majority of the data (training data), and then use the model to predict the answers for the rest of the data.

The data set used for this project can be found on Kaggle here and UCI here.

Since this data was meticulously collected and documented by astronomers and was remarkably clean and complete, my first task was to split the data into a set used to train each model that I develop, one used to validate each models performance and check the metrics for tuning my hyper-parameters, and finally, a set on which to test my models that the models have never seen.

I chose to separate 10% of the data into the test set, and of the remaining 90%, 20% of that was used for my validation set. This means the model was trained on 72% of the data available and validated and tested on the remaining 28%.

# Setting aside a sample of the data for testing.

# this portion will not be “known” during the model adjustment process.

train, test = train_test_split(ps,test_size=0.10,

stratify=ps[‘target_class’],

random_state= 17) # Separating the remaining data into training and validation sets.

#this allows me to tweak my model to perform at it's best before testing

#it on unknown data.

train, val = train_test_split(train,test_size=0.20,

stratify=train['target_class'],

random_state= 17)

train.shape,val.shape,test.shape Out[7]:((12886, 9), (3222, 9), (1790, 9))

Next, I do a quick check of the distributions of the target class within my splits. This even distribution is accomplished using the ‘stratify’ argument on the ‘target class’ feature. (In this column pulsars are labeled with 1 and non-pulsars are labeled with 0)

train['target_class'].value_counts(normalize=True),

val['target_class'].value_counts(normalize=True),

test['target_class'].value_counts(normalize=True) Out[8]: (0 0.908428

1 0.091572

Name: target_class, dtype: float64,

0 0.908442

1 0.091558

Name: target_class, dtype: float64,

0 0.90838

1 0.09162

Name: target_class, dtype: float64)

Solid

If I just built a model that always reported ‘0’ (non-pulsar star) it would be correct about 90.8% of the time (our baseline accuracy). This may prove difficult to improve upon with machine learning, but I enjoy a challenge!

Image via: NRAO pulsar

Now to separate our data into X feature matrices and y target vectors.

target = 'target_class'

features = ps.columns.drop(target) X_train = train[features]

y_train = train[target]

X_val = val[features]

y_val = val[target]

X_test = test[features]

y_test = test[target]

On to model production…

I won’t post the code for the rest of this as it can get a bit lengthy but feel free to visit my GitHub page where the project notebook lives if code examples are your jam.

Model The First: Random Forest Regressor

Validation Accuracy for Random Forest Regrssor : 0.8039317754042418

This model does not show improvement from the baseline accuracy. It only predicted the correct class 80.4% of the time. While this is about what I expected, and may still seem pretty good for many tasks, simply predicting that none of the observations were pulsars would have given me 90.8% accuracy, so to me, this model is not useful for this use case.

Model #2: Logistic Regression

Validation Accuracy for Logistic Regression 0.978895096213532

On the first try, with no adjustments from the stock Logistic Regression function in SciKitLearn’s linear_model module, this model does a great job at predicting pulsars given the data. This is great for me because I can use a linear model to determine the coefficients of each feature in the data set with a couple of simple lines of code.

Coefficients plot for Linear Regression model

Model V3.0: Random Forest Classifier

Validation Accuracy for Random Forest Classifier 0.9798261949099938

After multiple iterations, tweaking the hyper-parameters lower and higher, I found the sweet spot of this model's performance capabilities.



Now we’re talking! the RFC model was able to get even closer to predicting the proper classification 98% of the time on the validation set. There isn’t much more improvement that can be made, but I still will try for a better accuracy score… for science!

Although I’d love to see another season of ‘Lost in Space’, I wouldn’t want to be responsible for the reality-show version’s release.

So for my own peace of mind or the sake of proof, let’s take a look at another metric before deciding if this is a useful model.

Additional metrics

Confusion Matrix of predictions made on the validation set by my random forest classifier model :

Plot images created by the author

This confusion matrix displays how often my model was correct in its predictions and when it became ‘confused’ by the features and incorrectly labeled the observation.

If I were floating around aimlessly in space after an unforeseen course change from impact or spatial anomaly, I would want my analytics to err more on the side of caution, (classifying ‘maybes’ as ‘not pulsars’) than the opposite. In this hypothetical, my crew does not use the actual pulsars that the machine labeled as non-pulsars and just goes with the observations it reported back as pulsars to reorient and course correct.

ROC/AUC Curve

Plot Images created by author

ROC/AUC score = 0.9754350205277573

F1 score = 0.8969258589511754

Recall score = 0.8406779661016949

My RFC model leans more towards false negatives than false positives where errors in classification were made.

For this reason, I will accept this model as useful.

Still, the quest for ‘better’ goes on…

Final round: XGBoost Classifier :

Validation Accuracy using XGBoost 0.9823091247672253 This is a 0.0024829298572315306 improvement on the RFC model.

Another improvement in predictive accuracy that could not be achieved with my RFC model. This, again, took fine-tuning of hyper-parameters to achieve.

Confusion Matrix of predictions made on the validation set by my XGBoost classifier model :

Plot Images created by the author

Eureka?…

Not so fast. I’ll need to check the other metrics.

Plot Images created by author

ROC/AUC score = 0.9828035878698036

F1 score = 0.8969258589511754

Recall score = 0.8406779661016949

Accuracy and ROC/AUC scores have seen some improvements with XGBoost, with no drop in the recall or F1 score.

Warning: Objects in the following image may appear closer than they are.

Testing Accuracy:

That’s right, I finally get to break out the testing data that my model never got to know about until after all the fun was had in coding, validating, tweaking, validating, tuning, validating… well, you get it.

Testing accuracy with Random Forrest Regressor = 0.7627017333383153

Testing accuracy with Linear Regression = 0.976536312849162

Testing accuracy with Random Forrest Classifier = 0.976536312849162

Testing accuracy with XGBoost = 0.9793296089385475

We have a winner: XGBoost!

A lot of what goes on within machine learning algorithms are hidden to the user. Thanks to Lundberg and Lee, we data scientists now have an ML library called SHAP to help explain how this digital wizardry takes place. Fun fact: Shapley values were originally developed for game theory and named after Lloyd Shapley who introduced it.

Image via: SHAP Github. (link in acknowledgments)

Model interpretation

The models feature impacts separated into Class 1-Pulsars & Class 0-Not pulsars

Shapley Force Plot of an observation that is not a pulsar from RFC model

Shapley Force Plot of an observation that is a pulsar from RFC model

A couple more plots for those that can’t get enough. Again, all of the code for these plots and others can be found here.

Permutation importance list of all features using eli5

Partial dependence plot for the most impactful feature

Created by the author using pdpbox

Outro :

I can imagine that sometime after we get sailing through the cosmos much slower than the speed of light, some other beings from another time, place, or dimension may just hand us a glowing rock that can wisp us away to anywhere at the blink of an eye with little more than a thought… but until that day comes, we have machine learning.

Thanks for the read! All claps will be dedicated to the universe for keeping me intrigued. Cheers!

Citations:

R. J. Lyon, B. W. Stappers, S. Cooper, J. M. Brooke, J. D. Knowles, Fifty Years of Pulsar Candidate Selection: From simple filters to a new principled real-time classification approach, Monthly Notices of the Royal Astronomical Society 459 (1), 1104–1123, DOI: 10.1093/mnras/stw656

DOI of the data set:

R. J. Lyon, HTRU2, DOI: 10.6084/m9.figshare.3080389.v1.

Acknowledgments

This data was obtained with the support of grant EP/I028099/1 for the University of Manchester Centre for Doctoral Training in Computer Science, from the UK Engineering and Physical Sciences Research Council (EPSRC). The raw observational data was collected by the High Time Resolution Universe Collaboration using the Parkes Observatory, funded by the Commonwealth of Australia and managed by the CSIRO.

Additional sources:

[SPINN: a straightforward machine learning solution to the pulsar candidate selection problem]

[Selection of radio pulsar candidates using artificial neural networks]

[NRAO public image licensing terms]

[Introduction to Pulsar, Pulsar Timing, and measuring of Pulse Time-of-Arrivals]

[S. Lundberg, S. Lee “A unified approach to interpreting model predictions,” NIPS 2017]

[SHAP Github]