An Exhaustive Analysis of Lazy vs. Eager Learning Methods for Real-Estate Property Investment

Abstract: Accurate rent prediction in real estate investment can help in generating capital gains and guaranty a financial success. In this paper, we carry out a comprehensive analysis and study of eleven machine learning algorithms for rent prediction, including Linear Regression, Multilayer Perceptron, Random Forest, KNN, ML-KNN, Locally Weighted Learning, SMO, SVM, J48, lazy Decision Tree (i.e., lazy DT), and KStar algorithms. 
      Our contribution in this paper is twofold: (1) We present a comprehensive analysis of internal and external attributes of a real-estate housing dataset and their correlation with rental prices. (2) We use rental prediction as a platform to study and compare the performance of eager vs. lazy machine learning methods using myriad of ML algorithms.  
      We train our rent prediction models using a Zillow data set of 4K real estate properties in Virginia State of the US, including three house types of single-family, townhouse, and condo. Each data instance in the dataset has 21 internal attributes (e.g., area space, price, number of bed/bath, rent, school rating, so forth). In addition to Zillow data, external attributes like walk/transit score, and crime rate are collected from online data sources. A subset of the collected features - determined by the PCA technique-  are selected to tune the parameters of the prediction models. We employ a hierarchical clustering approach to cluster the data based on two factors of house type, and average rent estimate of zip codes. We evaluate and compare the efficacy of the tuned prediction models based on two metrics of R-squared and Mean Absolute Error, applied on unseen data. Based on our study, lazy models like KStar lead to higher accuracy and lower prediction error compared to eager methods like J48 and LR. However, it is not necessarily found to be an overarching conclusion drawn from the comparison between all the lazy and eager methods in this work. 

Reviews and author responses:

The authors compare a collection of machine learning models to predict the expected rental income from an investment property. The dataset they use to train their model is fairly small (around 4K transactions). In addition to using house specific features the authors use other macro features, such as, walk score etc. Using this the authors compare a set of machine learning models and report their findings. 

While the work presented in the paper is informative, I feel there are a number of issues with the paper, making it unsuitable for publication in ICLR. Some of them include: 
- There is no new novel model or technique proposed in the paper. It is essentially a collection of experiments run on some dataset with reported findings. 
- The dataset used is really small. Making the generalizability of the results somewhat questionable. 
- Lastly, even though there is nothing technically wrong with the presented work, I feel that ICLR is not the best venue for such works. Perhaps a data science conference, such as, KDD/WSDM might be much better suited.

Quality: your paper provides a comprehensive evaluation of a number of methods. However, I am not convinced by the choice of methods or the need for them to be compared as such.

Clarity: the paper is unclear in a number of places.

Originality: as far as I am aware, the paper is novel.

Significance: As stated above, I do not believe you have made the case that we need a comparison of these methods on this dataset.

More detailed comments:

1. The distinction between lazy and eager learning is not one that I find particularly helpful. Perhaps this is my own ignorance. I checked two standard ML textbooks (Bishop, Murphy) and neither mentions this distinction. As the crux of your paper is a comparison between these methods, you need to be very upfront about why this dichotomy is a useful one. And as your paper is very application-oriented, the justification needs to be related to that application.

2. Your data is, fundamentally, spatial (actually spatiotemporal). Thus it would be appropriate to consider models that specifically account for spatial structure, e.g.  Gaussian processes.

3. Do you have a novel methodological or theoretical contribution? A comparison of methods on an important dataset is no doubt useful, but I don't think ICLR is the right venue.

The paper investigates different machine learning approaches to model and
predict the return on property investments, in particular with respect to eager
and lazy learning techniques. The authors evaluate those different techniques on
a dataset of properties in Virginia. They conclude that lazy techniques provide
better performance than eager ones.

The paper is a purely empirical study that does not introduce any novel
machine learning or evaluation techniques. The authors use the off-the-shelf
WEKA toolbox. The results are not clear, given that only a single data set was
used to evaluate the different approaches, and general recommendations cannot be
made.

The paper is not well written and the descriptions do not convey what the
authors have done very well. An example of this is Figure 3, which purports to
show the average rent (or rent distribution?) for different housing types. There
are multiple categories in there that are not valid housing types ("Make Me
Move") and the rents shown are incorrect (e.g. more than a million for a town
house). It is also unclear why the average rent for a single family house is
approximately 4 times as much as for a town house.

The problems with Figure 3 are exemplary of the paper; the other issues are too
numerous to list.

In summary, this paper should be rejected.

Meta-review: The paper evaluates several off-the-shelf algorithms for predicting the return on real-estate property investment. The problem is interesting, but there is a consensus that the paper contains little technical novelty, and the empirical study on a fairly small dataset is also not convincing. 

Decision: rejected-papers