Categorizing Listing Photos at Airbnb
By: Shijing Yao
Tags: Machine Learning, AI, Data Science, Deep Learning, Airbnb

Re-training a DNN like ResNet50 using a few million images requires a lot of computational resources. In our implementation, we used an AWS P2.8xlarge Instance with Nvidia 8-core K80 GPU, and sent a batch of 128 images to 8 GPUs per training step. We did parallel training with Tensorflow as the backend. We compiled the model after parallelizing it because otherwise the training wouldn’t work. To further speed up training, we initialized model weights with pre-trained imagenet weights loaded from keras.applications.resnet50.ResNet50. The best model was obtained after 3 epochs of training, which lasted about 6 hours. Afterward the model started to overfit and the performance on validation set stopped improving.

One important note is that we built in production multiple binary-class models for different room types instead of building a multi-class model to cover all room types. This was not ideal but since our model serving was mostly offline, the extra delay due to multiple model calls affected us minimally. We will transit to a multi-class model in production soon.

We evaluated our models based on precision and recall. We also monitored metrics like F1 score and accuracy. Their definitions are reiterated as below. In a nutshell, precision describes how confident we are about the accuracy of our positive predictions, and recall describes how much percent our positive predictions cover all actual positives. Precision and recall usually go against each other. In our context, we set a high bar (95%) for precision because when we claim the photo is a certain room type, we should really have a high confidence about that claim.

TP: True Positive, TN: True Negative, FP: False Positive, FN: False Negative

A confusion matrix is the key to calculate these metrics. Our model’s raw output is a probability score from 0 to 1 for each image. To compute a confusion matrix for a set of predictions, one has to first set a particular threshold to translate the predicted scores into 0 and 1. A precision-recall (P-R) curve is then generated by sweeping the thresholds from 0 to 1. In principle the closer to 1 the AUC (Area Under Curve) of a P-R curve is, the more accurate the model is.

In evaluating the models, we used the aforementioned golden set where the ground truth labels were provided by humans. Interestingly we found accuracy differed from room type to room type. Bedroom and Bathroom models were the most accurate ones while other models were less accurate. For brevity, we only show the P-R curve of a Bedroom and Living Room here. The cross point of the dotted lines represents the final performance given a particular threshold. We append a summary of the metrics on the chart.

P-R Curve of Bedroom

P-R Curve of Living Room

There are two important observations:

The overall performance of the Bedroom model is much better than that of the a Living Room. There could be two explanations: 1) A Bedroom is easier to classify than a Living Room because Bedroom setting is relatively standard while Living Room can have a lot more varieties. 2) The labels extracted from Bedroom photos have higher quality than those extracted from Living Room photos since Living Room photos occasionally also include Dining Rooms or even Kitchens. Within each room type, a fully re-trained model (red curve) has better performance than the partially re-trained (blue curve) model, and the gap is larger between Living Room models than between Bedroom models. This suggests re-training a full ResNet50 model has different impact for difficult room types.

For the 6 models we shipped, precision is generally above 95% and recall is generally above 50%. By setting different threshold values people can make trade-offs. The model is set to power a number of different products across multiple product teams inside Airbnb.

The users compared our results of to well-known third-party image recognition APIs. It was reported that the in-house model overall outperformed third-party generic models. This implies by taking advantage of your own data, you have a chance to outperform even the industry state-of-the-art model for a particular task you are interested in.

At the end of this section, we’d like to showcase a few concrete examples that exemplify the power of this model.

The left two photos were correctly predicted as bedrooms; The right two photos were correctly predicted NOT as bedrooms.

The left two photos were correctly predicted as bathrooms; The right two photos were correctly predicted NOT as bathrooms.

The left two photos were correctly predicted as pools; The right two photos were correctly predicted NOT as pools, because they were aquarium and waterfront.

Beyond Classification

When doing this project, we also tried a few interesting ideas beyond room type classification. We want to show two examples here and give people an idea how exciting these problems are.

Unsupervised Scene Classification

When we first tried out room type classification using pre-trained ResNet50 model, we generated image embeddings (2048x1 vectors) for listing cover- page photos. To interpret what these embeddings meant, we projected these long vectors onto a 2D plane using PCA techniques. Much to our surprise, the projected data are naturally clustered into two groups. Looking into these two clusters, we found that the left group were almost exclusively indoor scenes and the right group were almost exclusively outdoor scenes. This meant without any re-training and simply by setting a cut line on the first principal component of the image embedding, we were able determine indoor and outdoor scenes. This finding opened the door to some really interesting domain where transfer learning (embedding) met unsupervised learning.

Indoor and Outdoor photos are automatically clustered by the first and second principal component.

Object Detection

Another area that we tried pursuing was object detection. A pre-trained Faster R-CNN model on Open Images Dataset already provided stunning results. As you see in the example below, the model is able to detect Window, Door, Dining Table and their locations. Using Tensorflow Object Detection API, we did some quick evaluations on our listing photos. A lot of the home amenities could be detected using the off-the-shelf result. In the future, we plan to retrain the Faster R-CNN model using Airbnb’s customized amenity labels. Since some of these labels are missing in the open source data, We will likely create labels on our own. With these algorithm-detected amenities, we are able to verify the quality of the listings from hosts and make it much easier for guests to find homes with specific amenity needs. This will push the frontier of photo intelligence at Airbnb to the next level.

Window, Door and Dining Table are successfully detected.

Conclusion

Here are a few key take-aways that might be helpful for other deep learning practitioners:

First, deep learning is nothing but one particular kind of supervised learning. One cannot overestimate the importance of high-quality labels to the data. Since deep learning usually requires a significant amount of training data to achieve state-of-the-art performance, finding an efficient way to do labeling is crucial. Fortunately we found a hybrid approach which is economical, scalable and reliable.

Second, training a DNN like ResNet50 from scratch can be quite involved. Try to start in a simple and fast way — train only top layers using a small dataset. If you do have a large trainable dataset, re-training a DNN from scratch might give you state-of-the-art performance.

Third, parallelize the training if you can. In our case we gained about 6x (quasi-linear) speed-up by using 8 GPUs. This makes building a complex DNN model computationally viable and it is much easier to iterate over hyper parameters and model structures.