Building a Recipe Recommendation API using Scikit-Learn, NLTK, Docker, Flask, and Heroku
Tags: Deployment, Programming, Towards Data Science, Data Science, Machine Learning

Preprocessing and Parsing of Ingredients

To understand the task at hand here, let's look at an example. The delicious ‘Gennaro’s classic spaghetti carbonara’ recipe found on Jamie Oliver’s website requires the ingredients:

3 large free-range egg yolks

40 g Parmesan cheese, plus extra to serve

1 x 150 g piece of higher-welfare pancetta

200 g dried spaghetti

1 clove of garlic

extra virgin olive oil

There is a lot of redundant information here; for example, weights and measures are not going to add value to the vector encodings of the recipe. If anything, it is going to make distinguishing between recipes more difficult. So we need to get rid of those. A quick google search led me to a Wikipedia page containing a list of standard cooking metrics e.g. clove, gram (g), teaspoon, etc. Removing all these words in my ingredient parser worked really well.

We also want to remove stop words from our ingredients. In NLP ‘stop words’ refer to the most common words in a language. For example, the sentence ‘learning about what stop words are’, becomes, ‘learning stop words’. NLTK provides us with an easy way to remove (most of) these words.

There are also other words in the ingredients that are useless to us — these are the words that are very common among recipes. Oil, for example, is used in most recipes and will provide little to no distinguishing power between recipes. Also, most people have oil in their homes so having to write oil every time you use the API is cumbersome and pointless. There are advanced NLP techniques, for example using conditional random fields (a class of statistical modeling), that can calculate the probability that a word is an ingredient, as opposed to a measure, texture, or another type of word that surrounds the ingredient word. But, simply removing the most common words seemed to be very effective, so I did this. Occam’s razor and all that jazz… To get the most common words we can do the following:

import nltk

vocabulary = nltk.FreqDist() # This was done once I had already preprocessed the ingredients

for ingredients in recipe_df['ingredients']:

ingredients = ingredients.split()

vocabulary.update(ingredients) for word, frequency in vocabulary.most_common(200):

print(f'{word};{frequency}')

We have one final obstacle to get over, however. When we try to remove these ‘junk’ words from our ingredient list, what happens when we have different variations of the same word? What happens if we want to remove every occurrence of the word ‘pound’ but the recipe ingredients say ‘pounds’? Luckily there is a pretty trivial workaround: lemmatization and stemming. Stemming and lemmatization both generate the root form of inflected words — the difference is that a stem might not be an actual word whereas, a lemma is an actual language word. Although lemmatization is often slower, I chose to use this technique as I know the ingredients will be actual words which is useful for debugging and visualization (the results turned out to be practically identical using stemming instead). When the user feeds ingredients to the API we also lemmatize those words as the lemmatized words are the words with the embeddings.

We can put this all together in a function, ingredient_parser , along with some other standard preprocessing: getting rid of punctuation, removing accents, making everything lowercase, getting rid of Unicode.

def ingredient_parser(ingredients): # measures and common words (already lemmatized)

measures = ['teaspoon', 't', 'tsp.', 'tablespoon', 'T', ...]

words_to_remove = ['fresh', 'oil', 'a', 'red', 'bunch', ...] # Turn ingredient list from string into a list

if isinstance(ingredients, list):

ingredients = ingredients

else:

ingredients = ast.literal_eval(ingredients) # We first get rid of all the punctuation

translator = str.maketrans('', '', string.punctuation) # initialize nltk's lemmatizer

lemmatizer = WordNetLemmatizer() ingred_list = []

for i in ingredients:

i.translate(translator)

# We split up with hyphens as well as spaces

items = re.split(' |-', i)

# Get rid of words containing non alphabet letters

items = [word for word in items if word.isalpha()]

# Turn everything to lowercase

items = [word.lower() for word in items]

# remove accents

items = [unidecode.unidecode(word) for word in items]

# Lemmatize words so we can compare words to measuring words

items = [lemmatizer.lemmatize(word) for word in items]

# get rid of stop words

stop_words = set(corpus.stopwords.words('english'))

items = [word for word in items if word not in stop_words]

# Gets rid of measuring words/phrases, e.g. heaped teaspoon

items = [word for word in items if word not in measures]

# Get rid of common easy words

items = [word for word in items if word not in words_to_remove]

if items:

ingred_list.append(' '.join(items))

ingred_list = ' '.join(ingred_list)

return ingred_list

When parsing the ingredients for ‘Gennaro’s classic spaghetti carbonara’ we get: egg yolk parmesan cheese pancetta spaghetti garlic. Perfect, that works fantastically!

Using lambda functions, it’s easy to parse all of the ingredients.