Image in, Recipe out
By: Peiyan Wang
Tags: Image Classification, Information Retrieval

Introduction

When people travel or move to a new country, there may be many vegetables and fruits they have never seen before. To find out how to make a dinner with the unseen vegetables, they may need to take a picture of the ingredients, upload it to google image, get the name of the vegetable from the retrieved results, and finally search for the recipe. To simplify these steps, we build a search engine to help people decide what kind of meal they would like to have based on the ingredients by only taking a picture. To realize this, we build a CNN classifier to recognize the vegetable in the input image and modify the returned label as a search query to generate recipes for users. The ranking method we have approached is BM25 and apply NDCG@10 as the evaluation metric to our search engine.

Data

Data sources

The datasets we used are from two platforms. One is named as ImageNet, which is a huge public image resource dataset platform that provides URLs and thumbnails with hundreds of categories of objects from vegetables to animals to researchers. The other is a dataset published in Github and is initially designed for cuisine classification. For both datasets, we only take part of them to use in our project.

Data obtaining

We first manually copied all contents in recipes text files from Github and then manipulated the text file. From this combined file, we selected overall 21 types of vegetables then made up 23 related phrases and sentences as our queries for the search engine. For image data, we downloaded compressed files for the objects we wanted. However, we noticed that not all vegetables that we manually selected from recipes text are available to download. Therefore, we manually downloaded usable images and saved them in corresponding folders.

Data preprocessing

Since image data all have different sizes, we generated new images with the same size, then store them in an array as features. We then normalized features by dividing 255 to make it easier to train the model with more stable gradients. Since CNN cannot take in categorical variables, we transformed vegetable names, our outcomes, from text to an array by one-hot encoding. Below are part of the codes for reading in images.

def add_label_to_image(vege_type,DIR,IMG_SIZE=64):

for img in tqdm(os.listdir(DIR)):

try:

label = vege_type

path = os.path.join(DIR,img)

img = cv2.imread(path,cv2.IMREAD_COLOR)

# resize the image

img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))

X.append(np.array(img))

Z.append(str(label))

except Exception as e:

print(str(e)) # read in the images

X=[] # image

Z=[] # label

for value in list(vege.values()):

print(value)

add_label_to_image(vege_type=value,DIR=value)

X = np.array(X)

# normalized the data to 0-1

X = X.astype('float32')/255

Z = np.array(Z)

Recipe data description

For recipe texts, there are duplicates in recipes therefore we drop these and reset indexes. The final recipes file with CSV format has a shape of 1410 rows and 2 columns.

First 5 rows of recipe dataset after modification

The numbers of images in each vegetable folder are listed below. We noticed that, on average, the number of images for each category is around 1000. While there are some outliers — ginger, garlic, eggplant, oat, and parsley. These vegetables are having less than 100 images on average, which might result in low accuracy in CNN classifiers.

Method

Image data augmentation

In order to optimize and evaluate our CNN model, we split the whole dataset into the train(72%), validation(18%), and test(10%) datasets. Accuracy is used as the evaluation matrix to evaluate the performance of the classification model. We applied image data augmentation using ImageDataGenerator to generate transformed images, such as rotation and horizontal flipping, to prevent overfitting.