Curb Cut app — An exercise in accessibility. Part I— AI
By: Chris Ribetti
Tags: TensorFlow, Android, Maps, AWS, Machine Learning

Photo by Rikki Chan on Unsplash

Here is a story about creating an app. One that integrates Google Maps API (GMA), TensorFlow Lite (TFL) and Amazon Web Services (AWS) all into a single, meaningful app. Meaningful because the app records curb cuts for the disabled, or for anyone that relies on curb cuts, such as the elderly or people with strollers. For those of you that don’t know, a curb cut is the dip in the sidewalk to provide smooth access to the adjacent street. They are also called pedestrian ramps.

New York has over 215,000 curb cuts. And New York has surveyed them all. There is a database with all that information. But it is static. If a curb cut degrades there needs to be a new record for reporting. Enter Curb Cut. I created this app to allow the user to take a photo of a curb cut, categorize it, and store it so that anyone with the app has access to the photo as well, and all the photos others have taken.

The app works as follows: Get a fix on the user’s location and present them with a map of where they are. Then the user acknowledges their location with respect to where the device says their location is. And then they can take a photo. The photo is analyzed, and if it passes validation it is stored in an AWS S3 bucket.

When the app is first opened, photo metadata is downloaded and map markers are created for each photo in the field of view. If the user taps on a marker, the photo itself is downloaded and presented to the user. That’s what the app does: take photo, store photo, view photo. But it does much more for those that rely on curb cuts to get around. It provides an easy way to record curb cuts that are in disrepair, with a complete record of what, where and when.

Part 1 is about the AI in the code. It comes into play after the user takes a photo. I want to know they are taking a photo of a curb cut and not a dog, a manhole cover, or a building. Enter TFL. This framework allows one to create a model based on your own classifications, and to train the model by supplying photo images. Much more about TFL and Android can be found on the Internet, starting maybe with:

or

Setting up TFL is straightforward. First add the following to your app-level build.gradle :

dependencies {

implementation 'org.tensorflow:tensorflow-lite:1.12.0'

I’m a big fan of source code. Full sources for this dependency are available on GitHub at:

The main class for this project’s AI stuff (and photo upload) is the classifier. The sample app classifier can be found at:

The classifier is the only entry point from the main project, passing in an image and returning the probability results. It’s that simple! What’s not simple is what’s done in the classifier. The classifier calls an Interpreter which calls NativeInterpreterWrapper which calls

private static native boolean run(long interpreterHandle, long errorHandle);

Native code. There’s a lot going on. And you can’t write anything from scratch since it’s complex. You need to use sample code to implement a solution. Hence the Classifier class in the samples.

The image classifier will take in a bitmap image of a particular size, in my case 224 x 224 pixels, run its magic calculations, and comes back with two numbers. The two numbers are probabilities and together they add up to 1.0. One number represents “Yes” and the other “No”. They are the probabilities, based on the calculations, that the image is in one of two categories — categories that I set up.

So let’s step back a little. In this image classification I set up two “buckets”. One has images of curb cuts, the other has other images. You provide as many images as you can to each bucket. TensorFlow Lite will allow us to create a model based on these images. We cheat a little and use a model that is already good at these classifications and train it some more with our specific images. After training, the model’s purpose is to take an image (the input) and produce the two probabilities previously mentioned (the output). HOW the model does this is an entire volume of work for discussion. It’s number crunching at its best, with imaginative thought on modeling itself thrown in. The takeaway here is that we did not need to create a model from scratch. We used an existing model and trained it.

It’s important to understand I provided two sets of images. This image classification technique does not take a bunch of curb cut images and create a model. It takes a bunch of curb cut images AND a bunch of non-curb cut images and creates a model. You must have two buckets. You must have a comparator. You don’t have a comparison between one bucket only and an implied ‘everything else’. You must supply the ‘everything else’ in a bucket. There’s a lot of ‘everything else’ out there. Grrrr. Oh well.

My app needs two buckets. Curb cuts and non-curb cuts. That is the minimum for any classification. But you can have more than two buckets for image classification. Say for classifying flowers. Then you will have a bucket with rose images, one with daisy images, etc. Then your probabilities will be spread out across these buckets. If you give it an image of a building, you will still get a probability for each bucket that adds up to 1.0. The image classifier will say that it is certain that the building is one of the flowers you bucketed! (It’s really saying “if i was to categorize the building as a flower…”)

Providing the two buckets of images is an art form in itself. I came across a nice approach, which is to take videos of curb cuts. Videos from all angles and lighting conditions. Then I extracted images from the videos. The software I used was ffmpeg, which made it very easy to do. I did the same for the other bucket, taking videos of things similar to but different from curb cuts, and including videos of things near curb cuts, such as manhole covers.

I’ve decided to set the threshold probability at 0.80 (80%). If the classifier determines the image provided is a curb cut with a probability of at least 0.80 then it gets uploaded to the cloud.

So, after creating videos, extracting images, creating two buckets, then letting TensorFlow Lite train the model, we’re done with the modeling. We take the model and place it in our app. It’s about 5MB in size. The magic is that this file contains most of the ‘logic’ about our image classification. And it was created offline, beforehand. Then we can do image classification in a fraction of a second on a mobile device.

That’s it! Part II will talk about the project’s implementation of Google Maps. Thank you for reading. Get out there and start implementing your own solutions!