Package: pixie-price-forecast
Pixie Price ForecastThis repository's main aim is to get an overview of the price tendency throughout the next few months.
It is published and distributed as a python package and it consists of the following steps:Daily extraction of aggregated data.Data Cleansing procedure.Price Forecast.Creation of table suited for visualization.Install package and requirementsTo begin with, please install all the python modules required to run the scripts.
It is as simple as to run the following in your virtual environment.pip install pixie-price-forecastNotes on CredentialThis script follow the strictest guidelines to not expose any private or confidential information.
As for that, it is necessary to provide a dictionary with credentials to connect to the DB via SSH.
It is compulsory for the dictionary to have the following structure:credentials = {
        'ssh_user': 'ssh user',
        'ssh_pass': 'ssh password',
        'ssh_host': 'ssh host',
        'ssh_port': 'ssh port',
        'user': 'Database user',
        'password': 'Database password',
        'host': 'Database host',
        'port': 'Database port',
        'db_name': 'Name of the Database'
    }We strongly advise not to expose the credentials by hard-coding it in files or script. We recommend to use alternatives
such as Github's secrets or Databrick's secrets.In the rest of the document we can whenever we refer to the credentials' dictionary it is a reference to the definition above.Running Raw Data ETLThe following code appends data corresponding to the data range '2030-01-01' - '2050-12-31' to the existing tableprice_raw_data:from price_forecast.etl import export_aggregated_data

update_raw_data(start_date='2030-01-01', end_date='2050-12-31', 
                credentials=credentials, first_step=1, last_step=12)Please note that the date format is'%Y-%m-%d'.The parametersfirst_stepandlast_stepcorrespond the kind of extraction. The following list correspond to all the available modes with their encoding:1 : demand_flat_rent2 : offer_flat_rent3 : professionals_flat_rent4 : demand_houses_rent5 : offer_houses_rent6 : professionals_houses_rent7 : demand_flat_sell8 : offer_flat_sell9 : professionals_flat_sell10: demand_houses_sell11: offer_houses_sell12: professionals_houses_sellPlease note thatoffercorresponds to supply data.Running Clean Data ETLOnce every month it is important to run the ETL to clean the raw data. The results are stored inprice_clean_datawhich is historized.To run the procedure is as simple as:from price_forecast.etl import update_clean_data
update_clean_data(credentials=credentials)In order to consume the latest clean data, we recommend the following query:SELECT
    *
FROM
    price_clean_data
WHERE
    use_tag=TRUEForecasting PricesAs of today, it is possible to forecast the property price per province using one of the following models:Arima: UnivariateRandom Forest: Using leads, visits and 3-6 months lags as drivers.LSTM: Better suited Neural Network to learn temporal dependencyForecast results are stored inprice_forecasts(please notice it is plural).You can use the following code to run all three models for all 12 modes:from price_forecast.forecaster import run_all_forecasts
run_all_forecasts(credentials=dict_credentials, train_from='2030-01-01', train_to='2050-09-31', max_date='2050-11-30')This code will train the models using clean data, per province from january 2030 to september 2050, and tests the results from october to november 2050.
Then, it makes prediction for the next 3 months and store it in the table.The target table is historized. In order to consume the latest results for a given province ('Barcelona' in the example) and a given mode (e.g. Supply data for sales of flats).SELECT
    *
FROM
    price_forecasts
WHERE
    use_tag=TRUE AND province='Barcelona' AND mode='offer_flat_sell'Exporting Data for VisualizationThe data model is not suited to provide data to Tableau in an convenient fashion. That is why it is necessary to export them,
in a wide format for both: clean data and forecast dataInput DataThe visualization clean data is stored inprice_viz_inputs. Which is historized, and can be generated as follows:from price_forecast.misc import join_all_inputs
from price_forecast.etl import setup_tunnel, create_pixie_engine

tunnel_server = setup_tunnel(ssh_user=credentials['ssh_user'],
                             ssh_pass=credentials['ssh_pass'],
                             ssh_host=credentials['ssh_host'],
                             ssh_port=credentials['ssh_port'],
                             host=credentials['host'],
                             port=credentials['port'])
tunnel_server.start()
local_port = tunnel_server.local_bind_port
pixie_engine = create_pixie_engine(local_port=local_port,
                                   user=credentials['user'],
                                   password=credentials['password'],
                                   db_name=credentials['db_name'])
join_all_inputs(db_engine=pixie_engine)To export data for Tableau run the following query:SELECT
    *
FROM
    price_viz_inputs
WHERE
    tag_use=TRUEForecast DataThe data is stored inprice_viz_forecastand in order to update the results run the following code:from price_forecast.misc import join_all_inputs
from price_forecast.etl import setup_tunnel, create_pixie_engine

tunnel_server = setup_tunnel(ssh_user=credentials['ssh_user'],
                             ssh_pass=credentials['ssh_pass'],
                             ssh_host=credentials['ssh_host'],
                             ssh_port=credentials['ssh_port'],
                             host=credentials['host'],
                             port=credentials['port'])
tunnel_server.start()
local_port = tunnel_server.local_bind_port
pixie_engine = create_pixie_engine(local_port=local_port)
join_datasets(pixie_engine)The corresponding query to consume forecast data is the following:SELECT
    *
FROM
    price_viz_forecasts
WHERE
    tag_use=TRUENext StepsOutput validation moduleExpand data quality checkResultsResults can be found inthis Tableau Dashboard.In case you do not have access, please contactJose Mielgo