Keeping Up with Deep Learning — 26 Nov 2020
By: Frank Odom
Tags: Programming, Deep Learning, Artificial Intelligence, Data Science, Machine Learning

Keeping Up with Deep Learning — 26 Nov 2020

Deep learning papers, blog posts, Github repos, etc. that I liked this week

Photo by Sebastian Pena Lambarri on Unsplash

This is the second edition of my weekly update on deep learning. Every Thursday, I’ll release a new batch of research papers, blog posts, Github repos, etc. that I liked over the past week. Links are provided for each featured project, so you can dive in and learn about whatever catches your eye. If you missed last week’s edition, you can find it here. All thoughts and opinions are my own. Follow me or check back next week for more. Enjoy!

Very Deep VAEs

[ArXiv][Github]

OpenAI recently showed that Variational AutoEncoders can outperform other likelihood-based generative models at creating realistic 2D images. Although the authors don’t compare their results against adversarial models (e.g. StyleGAN or PGAN), it’s clear from the generated images that VAE can’t match the performance of GANs yet. But this is still really exciting research, because VAEs are much easier to train and understand than GANs. For that reason, I’m excited to see more research like this from OpenAI, because if VAEs ever match the performance of GANs, they will be highly preferred over adversarial models.

Images generated with VAEs. Source: https://github.com/openai/vdvae

End-to-End Object Detection with Adaptive Clustering Transformer

[ArXiv]

Adaptive Clustering Transformer is one of the first notable research papers to improve upon DETR. (DETR was a recent landmark paper for object detection using transformers. See here for more details.) DETR requires a large amount of training for good performance, but ACE reduces training time by about 30%. Personally, I’m surprised this paper hasn’t received more attention. DETR is an amazing development for computer vision, and we need research like this to advance the state of the art!

Obligatory photo of a Transformer… Photo by Arseny Togulev on Unsplash

Propagate Yourself

[ArXiv]

Propagate Yourself advances the state of the art for unsupervised learning on vision-related tasks. Unsupervised learning greatly reduces the amount of labeled data needed for deep learning, because it uses unlabeled samples to learn meaningful representations of the data. I’m a big believer in unsupervised and self-supervised learning, and I recently wrote an article on self-supervised learning with BYOL! Check it out for a thorough introduction to self-supervised training of neural networks.

Design Space for Graph Neural Networks

[ArXiv]

This is a survey paper of various architectures for graph neural networks. GNNs have grown tremendously in popularity over the past few years, because unlike other types of neural networks, they’re able to process irregular data types like 3D point clouds. They’re used in many 3D object detection applications, which in turn is used by most self-driving algorithms. The landscape for GNNs has changed a lot over the past couple of years, and I highly recommend this paper if you need a refresher.

NVIDIA DGX A100

[Webpage][YouTube]

NVIDIA recently released the DGX A100 — a small server with some serious GPU horsepower. Each DGX packs four A100 chips (fastest GPUs in the world at the time of writing) and up to 640 GB of GPU memory. Unfortunately, each DGX station costs a minimum of $200,000, which essentially guarantees that I’ll never use one. (Google Cloud now offers A100s in beta mode, which is much more appealing.) But it’s admittedly fun to marvel at the horsepower we’re able to pack into one server, compared to just 5 years ago.

Photo by Francesco Lo Giudice on Unsplash

Google Cloud MLE Certification

[Course][Blog Post]

I believe that online developer certifications are the way of the future. Too many ML Engineer positions currently require a graduate degree, plus multiple years of professional experience. In order to grow our field, we need an accessible, low-cost option to get qualified for ML-focused jobs. That’s exactly what the Google Cloud MLE Certification tries to accomplish. For the price of $200, you’ll learn many of the requisite skills to land a job in today’s ML Engineering landscape. (I have no affiliation with Google or their certification programs. This is purely a personal endorsement of online ML courses like this one.)

Photo by MD Duran on Unsplash

Conclusion

Research has slowed down slightly with Thanksgiving this week, but it’s still a very exciting time for deep learning. Expect to see a lot more papers involving transformers and self-supervised learning in the near future — both are extremely hot topics right now, and for good reason. (If you’re not familiar with those, check out my recent articles on Transformers from Scratch and Self-supervised Learning with BYOL.) If you enjoyed the article, follow me to get all future weekly updates and other technical articles.