Rasoee — A Picture is Worth a Thousand Ingredients
By: Arijit Gupta
Tags: Food, Pytorch, Rasoee, Machine Learning, Computer Vision

Rasoee — A Picture is Worth a Thousand Ingredients

An attempt by a group of food-loving students to make the diverse world of different cuisines more navigable for our fellow eating enthusiasts. Arijit Gupta Follow Dec 16, 2020 · 4 min read

Rasoee won First Runners-Up at the 2020 Global PyTorch Summer Hackathon in the mobile/web application category. Written by Arijit Gupta. Made with Ameya Laad, Anish Mulay, Dev Churiwala and Smitesh Patil, undergraduate students at BITS Goa

What is Rasoee?

Our main inspiration while making this was looking at the number of people sharing pictures of food they’ve made at home during the lockdown. We then started wondering if there was someway to make it easier for others to replicate the ones they like themselves. Rasoee is a web/mobile application that can be used to identify food items from images. The app can identify from 308 different types of dishes from over 5 different cuisines. After identifying the dish, it also provides(if possible) a list of key ingredients that are used to make this dish, and link to a recipe and some tutorial videos on YouTube.

How it works

Rasoee uses an image classification model trained on a dataset of 308 images. We chose to train an EfficientNet B2 model that we believed was small enough to be deployed as an edge model and had enough depth to classify images into such a large number of categories. The making of this involved many steps, starting from creation of the dataset and ending with the making of the mobile and web application along with integrating the model into it.

Scrape your way to a dataset

While analysing the datasets available online, we saw that the most valid dataset available was the Food-101 Dataset.

Food-101 Classes

But, the Food-101 Dataset seemed a bit restricted in variety to us, and while the quality of the dataset is extremely high, we decided to go out on a limb and create our own dataset. To do this we used a bing image scraper and a lot of patience. What resulted was a terribly noisy dataset that we then cleaned to the best of our abilities, but we now had a dataset with 308 different categories of food.

Models, models, everywhere

Now came the moment where we had to make the most difficult decision of all, which model to use? PyTorch has such a great array of models that can be used even in the official documentation that we were spoiled for choice. After going through many models, and looking at their size v/s depth trade-off we settled on using one of the EfficientNet implementations for PyTorch, specifically the EfficientNet B2 model which has 9.2M parameters and a size of ~300Mb. Three of us were training different models and preprocessing, but we finally settled on the model that gave us the highest accuracy, and we achieved an accuracy of 91.31% using CrossEntropy loss. The training script is very basic and the training notebook can be found here.

User interface

The application has both a website and a mobile version. While our primary aim was the mobile application as it would be more convenient to users, the website can also be used on a mobile browser or any other edge device. It is very simple to use, just upload an image and wait for the results.