Housing in the San Francisco Bay Area: Deal Hunting Using Machine Learning
By: Michael Boles
Tags: Machine Learning, Housing, Data Science, Bay Area, Towards Data Science

Housing in the San Francisco Bay Area: Deal Hunting Using Machine Learning

Modeling home prices reveals undervalued neighborhoods and listings.

San Francisco downtown in the foreground, East Bay in the background. Photo credit: Unsplash

Executive summary

This article describes the collection, visualization, and modeling of single-family home prices listed across the San Francisco Bay Area. Approximately 7000 listings were scraped from a real estate website in June 2019, enabling the exploration of various factors influencing home prices across the region. Complementing listing information (number of bedrooms and bathrooms, home size, and lot size) with location data (school quality and commute times) was found to significantly improve explanatory power of the multiple linear regression model. Regression coefficients also tell an interesting story about the market’s perception of value associated with marginal changes to various aspects of the home and its location. Importantly, the pricing model enabled identification of undervalued listings and neighborhoods, offering investment recommendations of potential interest to individual homebuyers and investors alike.

Introduction

While the performance of the S&P 500 and the overall US housing market has been nearly identical since 2000 (both up ~100%), home price indices in the San Francisco Bay Area have risen by approximately 167% (St. Louis Fed). As such, Bay Area homeowners have enjoyed an opportunity to build wealth through real estate in a way that is not necessarily accessible to most of the rest of the country.

For those already bought into the market, this near-tripling of real estate values since 2000 has undoubtedly been a good thing. However, for those newly relocated to the region, saving towards a down payment and choosing where to buy can be a daunting task. Inspired by discussions I’ve had with friends and family, and basic concepts in investing (i.e., buy underpriced assets) I set out to gather as much information about current prices of single-family homes in the Bay Area, apply machine learning techniques to tease out factors driving home values, and identify areas that may be appealing for investment.

Methods

Data from single family home listings (address, beds, baths, home size, lot size, latitude/longitude coordinates, and price) across the Bay Area was scraped in June 2019 from a real estate webpage using the Requests and BeautifulSoup Python libraries, and cleaned and processed using Regex and Pandas. Complementing this listing data, commute times were obtained from Google Maps and school quality data pulled from the 2018 California Assessment of Student Performance and Progress (CAASPP). The resulting information was overlaid onto maps using Cartopy, Matplotlib, and shapefiles (town, zip code, and neighborhood borders) from Stanford Earthworks. Box/strip plots and pairwise relationships between variables were visualized using Seaborn. Ordinary least squares (OLS) regression analysis was applied to the data using Statsmodels and Scikit-learn libraries. The full source code for this project is available on GitHub.

Results

This study began with the collection of listing data for all single family homes on a real estate webpage (mlslistings.com). Looping over a list of Bay Area zip codes, location details (address, latitude and longitude coordinates), property characteristics (number of bedrooms, number of bathrooms, home size, and lot size), and list price were scraped for each listing using the Python Requests and BeautifulSoup libraries. Across 214 zip codes, location, property, and price information was collected for 7151 listings in June 2019. The data was cleaned using regular expressions to remove extra whitespace, unwanted characters, and entries with missing values.

I. Visualizing listing data

A. Geographic trends

With information for several thousand properties on the market across the Bay Area, the location of all listings was plotted on a terrain map and color-coded by price (Figure 1) using the Python Cartopy package together with city border information obtained from Stanford Earthworks. This map reveals clear geographic trends of the relative cost of homes across the region. For instance, San Francisco, Marin County, and the Peninsula typically contain the most expensive 20% of listings (dark blue data points), while those in Oakland, San Leandro, and Richmond are typically in the least expensive 20% (dark red data points). Similarly, in the South Bay, homes closer to the Santa Cruz Mountains are typically more expensive than those running alongside the Diablo Range.