The status of (geo) data science competitions
By: Maxime Lenormand
Tags: Image Processing, Earth Observation, Computer Vision, Kaggle, Opinion

The status of (geo) data science competitions

Data science and machine learning competitions are great in many ways, but sometimes feel like they could aim for even wider problems Maxime LENORMAND Nov 7, 2020·9 min read

Kaggle has become a primary hub when it comes to machine learning & data science competitions, as well as expanding to hosting datasets, GPU-backed notebooks and plenty of discussions like a forum would. This isn’t just a few people by the way, a few months ago the site passed 5 million registered users (Granted only a fraction actually go online daily or weekly, but like any such site). Working in geo-data science, the scope of some competitions feels a bit too narrow sometimes. Initiatives like SpaceNet create unique opportunities to create competitions around geo-data; this post will more be about how I hope these initiatives keep going forward. These fields & competitions are still quite new, and (mostly after spending a lot of time on them) I do hope they get even better.

There’s been a lot of talk about how “Kaggle isn’t representative of real life problems”; and I agree on a lot of those points: datasets all tidied up and prepared for you, few concern about production and some competitions turn into a chase down to 10^-4% accuracy improvements(I’ve spend week-ends being happy that I got a 0.0003 increase in F1 score). However, there also have been some good counter-points as to why it does help a lot for real life problems. One of my favorite takes on this is the recent thread on Twitter by CPMP (Which also happens to be one of the highest ranked person on the build-in ranking system on Kaggle, previously working at IBM and now Nvidia). But my (small) frustrations aren’t even about if all of this is useful or not. The nerd in me wishes that the competitions offered some more interesting and wider open problems (sometimes).

Don’t get me wrong, I would not be where I am without Kaggle, the community that is around it, and more generally all the platforms, discussions, blogposts, etc. (I really appreciate the term community-taught which I first saw on Sanyam Bhutani's blog) This post is more about a desire to make data science competitions even more interesting than they already are, not rant about dislikes I might have about them. Also, I’m really not that good at them. I tried my first SpaceNet competition on SpaceNet 7 and wasn’t even able to match the baseline solution, mostly due to lack of time, but still.

Geo-data is interesting because of its complexity

I currently work in the (completely biased opinion incoming) superbly interesting world of geo-data. I get to work with images taken from object blazing at ~7.8km/s at 400km above the ground that we commonly call satellites. This opens up a lot of really interesting challenges, the biggest one is that each image has a specific time & location. Time-series are common in Earth Observation, and so is mixing different types of data together. Raster data like images & elevation maps gets processed with vectors (polygon delineation of a field, Lat / Lon point clouds, location and footprints of buildings, etc.) into more complex problems.

This is where some of my frustration comes into play: A lot of the work out there is applying a lot of the remarkable state-of-the-art computer vision techniques so a limited subset of the Earth Observation data. We take the complex satellite data and reduce it down to 8 bit RGB images that can fit into an ImageNet pre-trained ResNet50. That’s actually great for a lot of applications, and can produce a wealth of really good results! It's also relatively easy and hustle-free to create, as there is a lot of libraries, pre-trained models, and more generally tools to handle "standard" imagery.

SpaceNet 7 labeled Planet Imagery. The metric used for the competition focused on identifying & differentiating each building, which are only a few pixels each and clustered close to each other.

What I would love to see is a contest that has some labelled data, same for object detection, with a clear metric but encouraging the use of external sources of data and be more open in general. As much as I am a machine learning and deep learning nerd, I would love to see someone hold a leader-board by beating all the models using a Land Cover map and an elevation model to get some insight into some areas that helps more than the "regular" RGB image processing through the latest unpronounceable pre-trained model.

Of course limits should apply, as access to data is in-equal, but I still think there might be some interesting options to at least explore:

P romote the use of all and any freely & openly available datasets : There is a ton of open geo-datasets: Imagery like Landsat, Sentinel 1 & 2 (and even the others), etc. But also elevation models like SRTM, land cover maps made by national geographic agencies, as well as vector data like Open Street Map, openly accessible building footprint datasets, etc. There are lots of already curated lists out there. I would love to see those being used together in so many clever ways.

: There is a ton of open geo-datasets: Imagery like Landsat, Sentinel 1 & 2 (and even the others), etc. But also elevation models like SRTM, land cover maps made by national geographic agencies, as well as vector data like Open Street Map, openly accessible building footprint datasets, etc. There are lots of already curated lists out there. I would love to see those being used together in so many clever ways. Providing more (if possible) than simply 8bit RGB images. A lot of competitions featuring satellite imagery narrow down the images to only the RGB bands, and providing them only in 8 bit. It would be awesome to have people work with NDVI as a guide towards filed / building segmentation, or diving into more intricate dimensional reduction for crazy number of bands like Sentinel 2.

A lot of competitions featuring satellite imagery narrow down the images to only the RGB bands, and providing them only in 8 bit. It would be awesome to have people work with NDVI as a guide towards filed / building segmentation, or diving into more intricate dimensional reduction for crazy number of bands like Sentinel 2. (Image providers might even be able to open up their archive for some competitions (They opened up their archive for a week-end to freely download just a few tiles for multiple hackathons). I’m not even talking about labeling those, just giving the option to more data if any contestant has any interesting ideas. Things like pseudo-labelling and semi-supervised satellite imagery might lead to some unexpected results.)

One criticism that might arise form that is the lack of skill level across different participants between those who know how to use these sources of data and those who don’t. After seeing some competitions on Kaggle of really specific data types (Medical imagery, 3D object detection, etc.) I believe it would this would mostly help bring the general skill level up. A common pattern seen in competitions, in data science but which also seems to happen in other fields often seems to emerge: Someone gets an edge through a clever use of a tool, wins a competition. Before you know it, next time around everyone seriously starts looking into it. This has been seen for algorithms that gain popularity when winning multiple times. Why not for data?

Deep down, I simply want these awesome datasets to be under the spotlight, for selfish reasons, because I just want them to be used, (hopefully) resulting in more of them being created down the line; as a practitioner I want to see some more out-of-the box ideas implemented and battle-tested. I think competitions are a compliment to research and the industry, especially for geo-data: Machine learning competitions are the cool thing at the moment, and I think the field of Earth Observation can benefit from that. Providing new, challenging and wide open complicate problems, while in return getting some new ideas from people who might not be familiar with the particularities of geo-data.

This is also how products in the industry get created. Data-fusion, combining multiple sensors and sources of imagery together, is becoming quite popular in Earth Observation. Many products get build by combining high resolution imagery with building footprints, or with elevation models to get a better insight on a specific location. Different resolutions, times of acquisition, data types get merged together to make something that makes more sense. I learned a lot by trying to apply some fancy pants neural network to find trees when a coworker pointed to me that a simple vegetation index could be created by merging two bands together to solve the problem in a much simpler fashion. This was much faster to implement, much easier to interpret and faster to process. Due to the particularities and the wealth of data types in Earth Observation, I think it makes sense to try to create competitions that capture this idea.

Community and group learning attracts people to compete

After talking to multiple people as well as thinking about my own amount of time spent on Kaggle I started finding some common themes as to why this platform was more successful than others. Some of the answer might be that getting acquired by Google does help developing an attractive tool as well as giving users access to free GPU time. But I think there’s more to it than that, a lot of vocal people on the platform point towards the community created around the competitions. On a personal note, spending time reading the code & discussions of other more skillful people working and reflecting on the same subject as me have proven hugely beneficial. I learned a lot by being able to ask questions about a specific topic that would have taken half an hour to provide context to someone not working on the project. Having hundreds and even thousands of people communicating on the same dataset for a couple of months at a time is a great learning moment.

Nearly 900 teams competed on the last Kaggle competition around satellite imagery, about ship detection in Pleiades imgery. It seems like the highest ranking solution went into production at Airbus.

This is something that I wish was more present on geo-data platforms like what SpaceNet is trying to become. Once again, I’m a huge SpaceNet fan, SpaceNet 6 was about SAR and I think that’s one of the best competitions datasets to date! I just wish the platform pushed for more inter-skill level sharing, to have a conversation about all the strange, maybe flawed, but sometimes great ideas people would have as they go by.

What I mostly appreciate about Kaggle is how the competition is for most people barely about competing. Don’t get me wrong, the thrill of climbing the leader-board is real, and drives oneself to push just a bit more each time. But that’s secondary for most people at least, few make it to the top yet thousands still try. Even though prize money goes pretty high sometimes, I hope that a lot of people go through months of work on their free time to try to learn a bit.

I'd also like to just touch upon the technical skill level required to participate. Kaggle seems to be pushing for easier and easier submission formats, and there's basically no setup required due to their self-hosted notbeooks. SpaceNet on the other hand only provides the data through an AWS download, and requires packaging your solution in a Docker container to be able to submit. The skill level is clearly not the same, where SpaceNet is more akin to some entry-level software development deployment. Those are two different approaches, I learned q few things through having to submit to SpaceNet, but it clearly raises the bar on the amount of people being to compete. I don't think there is any best approach, I leave that up to those reading. I just wanted to point out that there are multiple formats out there.

I wanted to write this down, mostly for myself, to gather my thoughts after completely failing to deliver a satisfying SpaceNet 7 solution even though my model still had (to me) satisfying results, and feeling like I should think about what I got out of it. In hindsight I should have spend more time developing tools to easily investigate my models errors and wasted a lot of time correctly understanding the projection system used by the competition. I really look forward to the next ones, but still with that itch to read more about what others could come up on what feels to me like a somewhat too restrictive task. I didn’t touch much on how SpaceNet is doing a great job of coming up with interesting metrics that I hope will get picked up and become more standard as we go, or how they provide freely available awesome datasets from the previous competitions, so I’ll add it here: I’m really grateful these all exist. I just want to make them even better!