Automating Real Estate Investment Analysis: Python Web Scraping Bot | by Josh Rab | Towards Data Science

The goal is to build a python web tool that is capable of analyzing investment properties. The tool will use data mining to find prices for properties, and then analyze the return rate.
Summary
Index
Motivations
The App
The Functions
Summary and Going Forward
The applications of bots or automation to trading and investments is not new. Multiple examples include stock trading bots tasked with buying or selling assets based on different models or indicators. Renaissance technology received the worlds attention due to their return rates through algorithmic investing that averaged 60%+ annualized returns over a 30-year time span.
The benefits of automation have the potential to transform businesses of varying size. Individual repetitive tasks can be automated by software that can be developed in house, or by third party SaaS platforms.
For the individual retail investor, python bots pose a promising solution to various elements of real estate investing. In this article, we examine automating the process of analyzing properties. Other processes that could be automated include: listing properties, sending notices to tenants, screening tenants (machine learning or AI), and even automatically dispatching maintenance workers.
The program takes three inputs: the listing URL, the monthly rent price, and the property tax rate. It returns monthly cash flow, cap rate, and cash on cash return rate.
Cash flow — The profit each month after paying everything (mortgage, property management, repair allowances, vacancy expense)
Cap rate — The net income per year divided by the price of the asset (in percent)
Cash on cash return rate — Net income per year divided by the down payment used for the asset (in percent)
The following packages were used. Note for those working in the Anacondas environment, it appears that streamlit is not currently available through this package manager. Streamlit was installed with pip, which might cause issues when PIP and Anacondas are used together
Requests — This package was used to access websites in python via HTTP requests.
Beautiful soup 4 — Used for web scraping and data mining. We are able to use this package to retrieve HTML code that describes the content and styling of a website. Once the HTML code is retrieved, beautiful soup can be utilized to isolate specific parts of the site. For example, in this project, we use beautiful soup to fetch the house price.
Streamlit — This package makes deploying web apps super simple. The coded was developed in a Jupyter notebook, then converted to .py scripts once it was working. Streamlit allowed for seamless deployment and minimal time spent on user interface. A classic deployment option of combining python for the backend, flask for deployment, and react for dynamic content is much more straightforward using streamlit.
The functions that do the majority of the heavy lifting are price_mine, mortgage_monthly, and net_operating.
These are the main functions that perform the following duties respectively:
Retrieving the listing price from the URL
Calculating the monthly mortgage cost
Finding the monthly net operating income after all expenses
price_mine | This function was designed to retrieve the house price from the listing. APIs could be used, but web-scraping is empowering. The downside of web scraping is that changes to the sites structure need to be updated in the code. Here, the web scraping package used was beautiful soup. The process is relatively simple, the web page is inspected using f12, then the desired element is found in the HTML code. This code can be isolated in beautiful soup to retrieve a specific part of the page. Once the code was retrieved the built in python function replace was used to remove commas, dollar signs, and unnecessary spaces so that a float variable can be established.
def price_mine(url):    #Currently this function takes an input of a URL and returns the listing prices     #The site it mines is remax    #The input must be a string input, we can reformat the input to force this to work    #Next we use regex to remove space and commas and dollar signs     headers = ({'User-Agent':            'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'})    response = get(url)    response_text = response.text    html_soup = BeautifulSoup(response_text, 'html.parser')    prices = html_soup.find('h2',{'class': 'price'}).text    prices = prices.replace(",", "")    prices = prices.replace("$", "")    prices = prices.replace(" ", "")    prices = float(prices)      return prices
mortgage_monthly | This function takes the listing price, mortgage length, and interest rate as inputs, and returns the monthly mortgage price. There are many ways to calculate the monthly mortgage price, no specific decision was made as far as which method to use and a generic algorithm that was fairly easy to implement was used.
def mortgage_monthly(price,years,percent):            #This implements an approach to finding a monthly mortgage amount from the purchase price,    #years and percent.     #Sample input: (300000,20,4) = 2422    #            percent = percent /100    down = down_payment(price,20)    loan = price - down    months = years*12    interest_monthly = percent/12    interest_plus = interest_monthly + 1    exponent = (interest_plus)**(-1*months)    subtract = 1 - exponent    division = interest_monthly / subtract    payment = division * loan            return(payment)
net_operating | This function takes the monthly rent, the tax rate, and the price as inputs, and returns the net operating income per month. The amount of net operating income each represents the cash after: paying the mortgage (principle and interest), property taxes, paying a management fee (10% per month), property repairs allowances, and vacancy allowances. The argument could be made that only the monthly interest payment constitutes an expense since the principle builds equity. While this is true, our model wants to find out how much cash is left after paying everything. Individual investment analysis bots could change elements like this to personalize the calculations to the individual investor.
def net_operating(rent, tax_rate, price):        #Takes input as monthly mortgage amount and monthly rental amount    #Uses managment expense, amount for repairs, vacancy ratio    #Example input: net_operating(1000,1,400,200)    #879.33    #1000 - 16.67 (tax) - 100 (managment) - 4 (repairs)        mortgage_amt = mortgage_monthly(price,20,3)    prop_managment = rent * 0.10    prop_tax = (price * (tax_rate/100)/12)    prop_repairs = (price * 0.02)/12    vacancy = (rent*0.02)    #These sections are a list of all the expenses used and formulas for each        net_income = rent - prop_managment - prop_tax - prop_repairs - vacancy - mortgage_amt    #Summing up expenses    output = [prop_managment, prop_tax, prop_repairs, vacancy, net_income]          return output
Other functions:
Other functions used such as cap_rate calculated the ratio of net income to asset price as a percent. The full list of function is available on the project’s GitHub repository but will be excluded from this documentation.
The idea was to have the inputs on the left-hand side of the page, and the output on the right-hand side of the page. The inputs were positioned inside a sidebar, this way the inputs and outputs are visually different.
A common way we could build this dashboard is to create a static website with HTML, deploy the back end using flask, store values in some sort of database, and link everything using react. A new alternative deployment path that has advantages over this approach is called streamlit.
Streamlit allows the fast transition from a python script to a modern user experience. It also offers a straightforward and fast deployment path. The first step in the conversion was to replace the built in python input functions and replace them with the streamlit input boxes. The same replacement was made for the output.
Once is this is done, the streamlit app can be deployed from the console, and accessed through the external IP address.
Once the user interface was built in streamlit, the code was modified to add a sidebar for the inputs as originally depicted in the above sketches.
The final code is available on GitHub.
Although groups such as renaissance technology have been able to profit from mathematical models applied to investing, there are benefits for individual retail investors that can be implemented much easier.
Real estate investors can benefit from automation by handling many tasks that would previously require an assistant, or tie up a lot of time. This was an example of using automation to reduce time spent on filtering deals. More deals could be reviewed by the investor if automated summary reports were generated and only the best assets were presented to a human. Mom and pop shops, real estate investors, and entrepreneurs can benefit from automation and not just fortune 500 companies.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.