Patent ID: 12254030

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The description set forth below in connection with the appended drawings is intended to be a description of various, illustrative embodiments of the disclosed subject matter. Specific features and functionalities are described in connection with each illustrative embodiment; however, it will be apparent to those skilled in the art that the disclosed embodiments may be practiced without each of those specific features and functionalities.

Aspects of the present disclosure are directed to computing systems and methods for improving the efficiency of the underwriting process for properties such as multi-family properties (e.g., apartment/condo buildings and/or communities). In some implementations, the computing systems can include a comparable property engine that is configured to generate property comparisons based on rent (income) and expenses. In some embodiments, the computing systems ingest large data sets from numerous internal and external data sources, categorize and transform the data sets into customized data sets that are applied to trained machine learning data models to determine similarity scores that indicate how similar one or more properties are to a particular property. In some examples, similarity scores can be calculated for both rent and expenses, which can greatly improve the efficiency and accuracy of identifying comparable properties. Additionally, the system can execute a user feedback process to solicit feedback on results from users, which can be used to refine performance of the machine learning algorithms and resulting similarity score determinations. Therefore, the systems and methods described herein provide a technical solution to a technical problem above what can be achieved by loan underwriters who have their own personal biases, limited information, and are unable to determine key features and weighting factors with the precision and accuracy achievable by the machine learning data models that are trained and used by the system.

As economic growth slows and becomes uncertain, people become less able to enter the real estate market as home buyers. Also, housing shortages further inhibit them from entering the real estate market. As a result, multifamily buildings help mitigate the effects of these problems, and many continue to see renting as a more affordable option than home purchase. The systems and methods described herein apply advanced analytic techniques to the multifamily real estate market by providing a comparable property engine that suggests rent and expense comps for a given subject property, calculates multifamily property similarity scores, and reduces underwriters' comp analysis time by over 60%.

FIG.1is a block diagram of an example environment100for a property comparison system108. The diagram illustrates relationships, interactions, computing devices, processing modules, and storage entities used to gather, generate, organize, store, and distribute the information necessary to automatically, accurately, and efficiently generate different types of comparison statistics for multifamily rental apartment buildings in a way and form not otherwise currently possible or available. In some implementations, the comparison statistics can be derived from a calculated similarity score for different properties. In some implementations, users102interact with the property comparison system108via an application hosted on an external device158, such as a laptop158b, mobile device158a, or tablet. Users can include loan purchasers, providers, underwriters, and/or prospective property purchasers.

The system108(similarity score model/algorithm) is configured to rapidly ingest data from multiple internal data sources104and external data sources106by data mining and collection engine132, identify statistically significant driving variables and transformations by a feature selection engine150and data transformation engine142, and train and fine tune a machine learning algorithm by an analytics engine144to identify a set of final variables for analysis. In some implementations, the property comparison system108can be configured to communicate with internal data sources104and external data sources106via a network (e.g., network828inFIG.8). The system108, via a similarity score generation engine134, is further configured to, using the trained algorithm and final variables, to calculate property similarity scores that allow users to more quickly and accurately identify comparable multifamily properties.

The inventors have recognized that small balance loans (SBLs) for buildings are often hard to benchmark and have conventionally required labor-intensive underwriting to appropriately capture property and submarket features. Using a unique and customized process that links SBL property data (e.g., property attributes, cash flows, etc.) with external hyper-local geospatial data received from external data sources. To solve these problems, from the linked data, the system108can calculate building similarity scores to account for property idiosyncrasies and validate rents for the target property by comparing it with market data. The system108also allows comparable property data to be mapped and visualized within a geospatial environment. In some examples, a feature selection engine150and data transformation engine142combine all ingested data in a customized way to create a single data source (e.g., combined data116in data repository). In some examples, analytics engine144, using machine learning algorithms, identifies one or more of the most impactful attributes for predicting rent with high accuracy. In one example, the analytics engine144can identify over thirty attributes for predicting each of rent and expenses for properties with high accuracy. These attributes can include both geospatial features, traditional features (e.g., year built, number of units, total residential squire footage, etc.), as well as non-traditional features (e.g., number of grocery stores within a predetermined radius, bus stops within half a mile, etc.) The analytics engine144can be further configured to identify comparable properties. Other variables that can be ingested and identified by the analytics engine144as identification features can include non-traditional features such as coffee shops or restaurants within the vicinity of a property and building violations. In some examples, various types of points-of-interest or other contributing factors (e.g., restaurants, beach resorts, and college, retail, entertainment, crime occurrence locations, transportation access point locations (e.g., bus stops) within the vicinity) can be identified based on proximity and density with respect to a given property. These functions of the analytics engine144can decrease or remove human error in selecting comparable properties and aggregating data. A user interface engine146can provide a customized user experience for selecting and evaluating the best comparables for a subject property. Therefore, the property comparison system108provides both improved quantity and quality of comparables.

Also, the system108is designed to seamlessly scale to include or exclude various markets and locations from its analysis. Further, the platform of the system108provides numerous advantages that include expedited comparable analysis and display, elimination of manual data pulls, standardized rental and comp analyses, expanded internal data pool, processes to supplement missing data, and easy, searchable access to data on comparables from past funded deals in a structured format.

In some implementations, data repository110can store internal and external data111,112received from internal data sources104and external data sources106, respectively, and used by the system108(e.g., analytics engine144) in generating machine learning data models. The internal and external data111,112can be used to both train the machine learning data models and/or to characterize properties one or more geographic locations. In some examples, internal data111comes from a multifamily underwriting platform (OUS) and an asset management platform for multifamily loans (SMART). In some embodiments, OUS is a primary underwriting platform for multifamily loans, which can provide data elements used by the machine learning data models concerning physical attributes of the subject property such as number of units, unit mix, renovation year, square footage, and amenities. In some examples, these variables capture the size and condition of the property. In addition, OUS can also provide a rent dollar amount for the subject property that can be used as a target variable for trained XGBoost models. In some implementations, SMART is a primary asset management platform for multifamily loans that houses ongoing property and financial data post-underwriting and can be used to augment missing or outdated underwriting data. In some embodiments, data can be captured from OURS and/or SMART at the loan level, which can correspond to a property level. In some cases where multiple loans can represent the same property due to refinances, each loan record can represent the property at the time of the loan and can be controlled via age filtering or deduplication. In some implementations, data mining/collection engine132may refresh internal data111daily via scheduled batch job. Both the eligible population and variable values are updated to reflect real-world changes that occurred during the previous day. In other examples, internal data111may be updated automatically as internal data source information is updated. In some examples, the internal data111can also include loan appraisal data that includes photos of properties.

In some embodiments, external data sources106that provide external data112can include a wide variety of sources such as census data, tax (IRS) data, Google places, third-party multifamily data vendors, and other open data sources. Table 1 below shows examples of types of data sources (e.g., external data sources106) that provide external data112to the system108. Table 1 also provides details regarding the granularity of each of the types of external data112and the frequency of data updates. In some implementations, data obtained from external data sources106can be used as non-traditional features that enhance the accuracy of the comparables analysis when compared to systems that just use traditional data features in their comparables analysis. In some embodiments, the external data112can be used to characterize the neighborhood or surrounding area of a given subject property in ways that internal data may not be able to provide and to augment physical property information that may not be available from internal data systems. In some examples, different types of data from different data sources can be used in certain segment machine learning data models (e.g. rent versus expense, different geographic regions). In one example, certain data sources (e.g., NYC Open and Pluto) data may only be available for properties in certain locations (e.g., New York City properties). In addition, certain types of data may be used for data modeling and others may be used in user interface generation. Data mining/collection engine132, in some examples, can extract external data112from each of the data sources at an update frequency (e.g., the update frequency listed in TABLE 1). In other examples, one or more of the external data sources106can be configured to automatically provide requested data to the system108at the predetermined frequency. The external data sources106and types of external data112described herein are exemplary and are not meant to include an exclusive or exhaustive list of types of external data sources106that can provide data to the property comparison system108.

TABLE 1Information UsedIn Modelingand/or UserGranularityUpdateData SourceDescriptionInterfaceLevelFrequencyCensusDemographic dataPopulation;Census TractAnnuallyprovided by theIncome;AmericanEducation level;CommunityLabor forceSurveyparticipationGoogle PlacesPoints of interestNumber of nearbyExactBi-annuallyby Lat/LongStarbucks stores (orLat/Longother points ofinterest)Third-partyPhysical propertyNumber of units;AddressBi-annuallyMultifamilyinformationNumber of floors;Data Sources onRenovation year;PropertiesProperty size;Year builtOther Open DataSpecific toLocal neighborhoodLocalAnnuallySourcesmultifamilyand propertyNeighborhoodproperties andfeatureslocal informationservices

In some embodiments, the property comparison system108can include one or more processing engines or modules130,132,134,136,142,144,146,148,150,152executed as software programs on hardware computing systems. References to the engines or modules throughout the disclosure are meant to refer to software processes executed by circuitry of one or more processing circuits, which can also be referred to interchangeably as processing circuitry. In some implementations, the processes associated with the property comparison system108can be performed by one or more servers having one or more processing circuits such that some processes or portions of processes may be performed on different servers. The processes executed by the processing engines can include identifying the key features for comparable properties from internal and external data sources based on both underwriting expertise and data analytics generated by trained machine learning data models, calculating the degree of similarity between properties, and generating comparable property recommendations based on the calculated similarity scores for both rent and expense comparables. This similarity score generation performed by similarity score generation engine134is based on the principle that properties with similar attributes and similar neighborhood will be positioned similarly (for operating rent and expense) in the related market. In some implementations, analytics engine144uses machine learning algorithms to identify the modeling features in the property comparable analysis as well as the weighting factors (e.g., feature data and weights118) associated with each of the identified features. In some examples, the system108uses an Extreme Gradient Boosting (XGBoost) algorithm that sequentially corrects errors of previous models or a Random Forecast algorithm that uses a bagging technique of grouping weaker models to form more powerful models. Techniques for applying XGBoost modeling algorithms work are described in Tianqi Chen, Carlos Guestrin, XGBoost: A Scalable Tree Boosting System, A C M,2016, which and is incorporated herein by reference.

In one example, the property comparison system108includes a data management engine130that organizes the data received by the system108from the users102, internal data sources104, and external data sources106and controls data handling during execution of the processes described further herein. The data management engine130, in some embodiments, also controls the interaction of the property comparison system108with at least one data repository110associated with the environment100. For example, the data management engine130controls the storing and accessing of both system-generated data and system-received data as well as the interactions between system-generated and system-received data. For example, the data management engine130accesses internal data111and external data112from data repository110and provides the internal and external data111,112to missing data engine152, feature selection engine150, and/or analytics engine144. Further, the data management engine130receives feature data and weights118from analytics engine144and feature selection engine150, which it stores in the data repository110. In some embodiments, the data management engine130controls the flow of data between the data repository110and the property comparison system108.

In some embodiments, the property comparison system108includes a feature selection engine150that applies data processing techniques to generate customized data structures for applying to machine learning data models that generate outputs used to calculate similarity scores. These data processing techniques employed by feature selection engine150and data transformation engine142can also include improving on original data through a process of filling in missing features. Additionally, feature selection engine150can perform data transformation processes that allow the system108to capture relationships between property features and property quality. In some examples, types of data transformations include creation of new features from features in original data sources, changing continuous features into categorical features, and re-bucketing of categorical features.

In some implementations, feature selection engine150selects features to be used for feature analysis and machine learning by performing one or feature selection and population filtering processes. For example, the feature selection engine150can examine the missing rate for each possible data feature and drop any features that are missing from data sets (including both internal data111and external data112) and drop any features with an absence rate of greater than a predetermined percentage (e.g., 50%). Additionally, features with less than a predetermined percentage of variation (e.g., 10%) may also be dropped from the feature sets. In some implementations, the feature selection engine150can run a correlation analysis to group highly correlated features into the same categorical division or bucket. In one example, the correlation analysis is a Pearson analysis. In some examples, a portion of the features in each bucket can be identified for use by the system108. In some implementations, a feature for each bucket that has a highest correlation with an outcome variable (e.g., rent or expense) is selected as the feature for a respective bucket. In some implementations, the bucketed data population is filtered to remove data entries that have missing values for many property features. In one example, only properties with certain core property attributes (e.g., unit size and number of units) are retained for analysis. Additionally, any property information associate with loans that are dead deals or have not yet passed the transfer to purchase phase of origination are removed from the data population.

In some implementations, analytics engine144runs a machine learning process on the filtered data population features to further identify features used for similarity score calculations. In some examples, the feature data sets are used to train machine learning data models, which determines the features and weights that are the most predictive of comparable properties. In some examples, the features having the lowest weighting values are dropped from the analysis (e.g., the features having weights that are less than a predetermined threshold or features that fall within a lowest percentage of weights). In other examples, the identified features and weights are presented to a user (e.g., an underwriter or other subject matter expert) who flags one or more features for removing from the analysis. In some examples, feature selection can be an iterative process that continues until all features have importance weights that are greater than a predetermined threshold or fall within a predetermined range.

In addition, multiple machine learning models can be trained and used to generate rent and expense feature/weighting sets for multiple geographic regions, and the identified features and weighting values may vary by geographic region. For example,FIG.15shows tables1500,1502of identified rent and expense model features with corresponding weights for New York City (NYC), andFIG.16shows tables1600,1602of identified rent and expense model features with corresponding weights for the Southwestern (SW) region of the United States. In one example, both rent feature tables1500,1600include a combined renovation feature1504,1604, which refers to a number of renovations that have been performed on the property over a predetermined period of time. However, the relative weighting values for the combined renovation feature1504,1604are different (for example, for feature1504for the NYC model, the weighting value is 0.103, and for feature1604for the SW model, the weighting value is 0.065) due to the learning performed by the respective machine learning data model as it ingests data. Further, in some implementations, certain features may be included in some location models that are not included in others. For example, the SW expense model1602may include swimming pool1606and clubhouse features1608while the NYC expense model1502does not include those features because NYC multifamily properties may not typically include swimming pools or clubhouses while SW properties do. Having data models that are customized to specific geographic regions and that are continuously updated to include the most relevant features to determining property similarity improves score accuracy and also improves overall system processing efficiency by reducing a number of calculations that are performed in determining rent and expense similarity scores. The variables and weighting values displayed in tables1500,1502,1600,1602are exemplary and the description herein is not meant to be limiting.

Returning toFIG.1, in some implementations, once a set of predictive data features have been identified, then missing feature engine152can be configured to fill in missing data features for properties, which improves the original data set used to determine property similarity scores. In some situations, data can be missing from feature data sets due to feature entry default such as when a value such as “renovation date” defaults to “missing” if a property has not been renovated within a certain number of years. Additionally, a feature for “parking garage” may default to “missing” if the property has no parking garage. In some examples, other features may be missing due to insufficient information entry by one of the internal or external data sources. For example, a “laundry” feature and a “floor number” feature are often not included in the received sets of internal and external data111,112.

In some examples, missing feature engine152can be configured to impute missing information in one or more ways. In some examples, certain features of external data112can be identified that best complement particular features of internal data111. For example, both collected property level data and Pluto (for New York City properties) include renovation date and build year. Additionally, default values can be applied for certain features when those features are missing from the data sets. For example, if “parking garage” or “elevator” features are missing, the missing feature engine152may apply values of “0” to those features. Additionally, missing feature engine152can extract textual features from unstructured text files. For example, OUS data includes a data filed for “property comments” that may include details about a laundry facility on site. In one example, the missing feature engine152may extract text such as “laundry room” or “in-unit laundry” to determine whether a property includes a laundry feature or not. Additionally, the missing feature engine152can impute missing data from another available feature based on a correlation between the features. For example, a building floor number can be inferred from a first value of a unit number. In some embodiments, missing feature engine152can also include one or more image processing sub-engines that can be configured to detect and impute missing features from image files from one or more internal data sources111such as photos in appraisal files. For example, the image processing sub-engine may be configured to detect the presence of certain missing features by detecting those features within appraisal photos (e.g., laundry room features such as washers, dryers, and deep sinks). The image processing sub-engine may also be able to detect changes in property images that may be indicative of property degradation or renovations, which can be used to determine renovation year of a property. In some implementations, rules for imputing missing data can be stored in data repository as missing data rules120.

In some embodiments, data transformation engine142produces data structures that capture relationships between observed property features and property quality to most accurately measure similarity between properties by transforming raw data received from internal and external data sources111,112into meaningful features. In some examples, new features are created from raw data values received from data sources. In some aspects, transformed features may be normalized values for raw data values. For example, instead of using unit rent as an outcome variable, the system108calculates “rent per square foot (sqft)” because it better reveals the property quality. In another example, instead of using a dummy variable for each of the amenities, the data transformation engine142can combine amenity features together to calculate the total number of amenities for a property. Other examples include, but are not limited to, calculating renovation amount per unit in place of total building renovation amount, calculating the ratio of commercial area to residential area in place of including both commercial and residential areas, and, for each neighborhood, calculating population density, instead of using total population.

In some implementations, data transformation engine142can also change continuous value features into categorical features, which improves overall processing efficiency and accuracy. For example, for almost all continuous features, a relationship between a given feature value and property quality is not continuous. For example, because renovation approaches, trends, and styles change and age, a property renovated within in 2 years may not considered different from another property renovated within 3 years. However, a property renovated within 5 years may be considered with better quality than another property renovated 6 years ago. Thus, a categorical feature “renovated_date_category” may be created, which has a value of 0 if a property is renovated within 5 years; has a value of 1 if a property is renovated 6-10 years ago; has a value of 2 if a property is renovated 11-20 years ago; and has a value of 3 if a property is renovated more than 20 years ago. In another example, unit square footage can be categorized based on a range of square foot values (e.g., 500-750 sqft, 751-1000 sqft, 1000-1250 sqft, etc.). In some examples, rules for transforming continuous value features into categorical features can be stored in data repository110as data transformation rules122.

In some examples, data transformation engine142re-buckets or classifies categorical features to categorical buckets or divisions. In some implementations, some property features may be assigned to the wrong buckets based on raw data values, which can result in biased importance weights for these features. To correct the importance weights of these features, the data transformation engine142re-bucket the categorical into more appropriate categories. For example, for raw data, a feature for “property overall quality” may have 9 categories: 1 (poor); 2 (Fair); 3 (Average); 4 (Good); 5 (Excellent); 6 (Excellent); 7 (Above Average); 8 (Average); 9 (Below Average); 10 (Inferior). In some examples, some of these categories may have very few data observations assigned to each bucket, and some categories may represent substantially similar property quality. Applying the original overall quality data from the raw data to the machine learning process may generate a very small weight due to these data bucketing issues even though property quality may be very important to determining comparable property similarity. To boost the weight of this feature to a level that accurately represents the importance of the property quality feature, this feature may be bucketed into three categories: “Above Average” if the original value is 4, 5, 6, or 7; “Average” if the original value is 2, 3, or 8; and “Below Average” if the original value is 1 or 9. This data transformation and re-classifying into buckets helps ensure that the importance weight of this feature increases and has a greater impact on the comparable analysis. In some examples, the data transformation engine142performs these re-bucketing procedures based on feedback received from subject matter experts from underwriters. These data transformations and classifications also account for outliers, which may be grouped into a first bucket or a last bucket for the feature.

Similarity score generate engine134, in some examples, generates and applies an advanced analytical/scientific algorithm to model the cognitive behavior and reasoning performed by real estate professionals when they compare properties, so that the data and the variable weights can be combined in a comprehensive way to generate appropriate scores. In some implementations, the analytics engine144can use machine learning predictive models to make feature selection determinations and feature weighting determinations, which indicate a relative importance of each of the identified features to a similarity score calculation. In addition, different models are used for different outcome variables and regions. For example, different models can be trained and applied for rent and expense variables and for multiple geographic regions (e.g., large metro cities such as New York City or Chicago, states, or regions of the country such as Southwest, Northeast, Midwest, etc.). In some examples, the similarity score generation engine134may apply a Gower distance algorithm to calculate different types of similarity scores for a given property. For example, the similarity score generation engine134may calculate similarity scores for a property level, neighborhood level, and/or unit type level associated with the subject property.

In some examples, a machine learning algorithm is used by analytics engine144to identify features weights, which may indicate amounts of correlation between rent/expense levels with each data feature. The model, in some examples, is based on machine learning algorithms that can learn from data without relying on rules-based programming. Since, in some implementations, target variables (rent or expense) are in a numeric form, the machine learning model may be similar to a regression type of model, but it outperforms the typical regression model in handling nonlinearity, collinearity and unstructured data. Statistically, importance of a variable importance can be measured by calculating the increase/decrease of the model's prediction error after including/excluding that variable. The use of machine learning models increases accuracy and interpretability methods and can estimate the importance associated with each feature.

In some implementations, the machine learning models used by analytics engine144are XGBoost algorithms, which is an optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable. Boosting can be a sequential process where each subsequent model attempts to correct the errors of the previous model. The succeeding models are dependent on the previous model. In some implementations, XGBoost models combine a number of weak learners to form a strong learner through weightings. Feature importance is calculated using the “weight” method, that is, the percentage relative to all other variables that a given variable is used to split data across all trees. This calculation can be implemented in the Python xgb package. In another example, a Random Forest model may be used instead of a XGBoost model to determine predictive features and weightings.

In some implementations, similarity score generation engine134generates similarity scores for both rent and expense comparables from the feature variables and weights output by the trained machine learning data models. In some examples, the similarity scores measure a difference in physical property characteristics between two properties in a particular geographic area. Using the output features and weights, neighborhood features and their weights are used to calculate a neighborhood similarity score, which measures a difference in two neighborhoods where comparable properties reside. For rent comparables, the data models and similarity scores account for unit type level characteristics and property-level conditions. This allows the system108to generate property-level similarity scores at unit-type level. Previously, in the industry of multifamily underwriting, rent comparables could only be compared and selected at a property level but not unit-type level. Availability of unit-level comparison enables comparisons to be performed at a more granular level to improve accuracy of the generated comparables. In some examples, for expense comparables, because expenses are measured on a property-level only and expense comparisons are much less sensitive to neighborhood conditions, expense similarity scores are calculated using property-level physical features.

In some examples, the similarity score generation engine134applies a Gower distance algorithm to calculate spatial distance between any pair of properties, using the selected feature and their feature importance. The Gower distance is described in J. C. Gower, A General Coefficient of Similarity and Some of Its Properties, Biometrics, Vol. 27, No. 4. (December 1971), pp. 857-871, the contents of which is incorporated herein by reference. Gower's distance metric can be defined as follows:

∑k⁢sijkwk∑k⁢wk,
where Sijkis the distance between property i and j on the kthvariable; and wkis the weight/importance of the kthvariable. The Gower distance is a weighted average of the distances on the different variables, which allows a weight wkto be assigned each individual variable, effectively changing the importance of that variable in the distance calculation.

In some implementations, Gower's distance metric is capable of doing handling different types of variables, such as categorical and numeric as in the case of comparable similarity score calculation. The strength of Gower's distance metric lies in the calculation of Sijk. Unlike traditional distance metrics, Sijkdoes not apply the same formula to all variables. For categorical variables we use an equal/not equal comparison, but for numeric variables, the absolute difference can be used. To prevent one type of variable having more impact on the distance metric, all Sijkare scaled to the range [0, 1]. For categorical variables, this means that a value 0 to Sijkis assigned when the categorical variables of i and j are equal and 1 when they are not. Numeric variables can be scaled by dividing the absolute difference by the range of the variable. The similarity between property i and property j equals 1 minus the distance, and the similarity score can be expressed as follow:

si⁢j=1-∑k⁢sijkwk∑k⁢wk.
For expense models, only property level features may be used to calculate similarity score for two reasons since expense comparable selection primarily depends on property features other than neighborhood features and there are oftentimes satisfactory comps available within acceptable radius.

In some embodiments, when comparing MF properties for rental incomes, underwriters analyze property features and local neighborhood features relatively independently. Thus, for rent comparables, the similarity score generation engine134can calculates calculate property similarity score and neighborhood similarity score separately. sijNis the similarity score with only neighborhood information, and sijPis the similarity score with only property physical features. Furthermore, given that the data contain both unit level information and property level information, similarity scores can be calculated at a unit-type level.

As an example, let k=[k1, k2] be the vector of all property related features, where k1 is the vector of unit-type level features that vary among unit-types within in the same property; and k2 is the vector of property level features that stay the same across unit-types within on property, but vary across properties. For example, feature “unit size” is the average unit size for each unit type, so this feature is a unit-type level feature, not a property level feature. In total, 10 different unite types may be considered based on bedroom/bathroom combinations: 0Bed1 Bath, 1Bed1Bath, 1Bed2Bath, 2Bed1Bath, 2Bed2Bath, 2Bed3Bath, 3Bed1Bath, 3Bed2Bath, 3Bed3Bath, 4Bed+. For each unit type, the unit-type similarity score is calculated based on the following:

sijunit⁢_⁢type=1-∑k⁢1⁢sij,k⁢1unit⁢_⁢type⁢wk⁢1+∑k⁢2⁢sij,k⁢2wk⁢2∑k⁢wk.
If one unit-type is are not shared by the two properties in comparison, the similarity score for this unit-type is set to 0: meaning the pair of properties are not comparable at this unit-type. Further, the 10 unit-types can be regrouped into four final unit-type categories based on number of bedrooms. For each final unit-type category, in some implementations, the similarity score is the weighted average of unit-type similarity scores within the category. The weight is measured using the number of units. For example, property i and property j share two unit-types: 1Bed1Bath and 1Bed2Bath. Then, the final 1Bedroom similarity score between i and j is calculated as:

sij1⁢Bedroom=Unitsij1⁢B⁢1⁢B(Unitsij1⁢B⁢1⁢B+Unitsij1⁢B⁢2⁢B)*Sij1⁢B⁢1⁢B+Unitsij1⁢B⁢2⁢B(Unitsij1⁢B⁢1⁢B+Unitsij1⁢B⁢2⁢B)*Sij1⁢B⁢2⁢B,
where Unitsij1B1Bis the total unit number of 1 Bed1Bath in both property i and property j. For rent comp selection, the similarity score generation engine134selects top comparables for each available final unit-type category separately.

With the unit-type level similarity scores, the accuracy of the existing comparable process in the industry can be improved. One common problem underwriters encounter is that they sometimes need to choose between two comps that each is a good comp only some unit-types, not all unit-types. For example, for a subject underwriting property with both one- and two-bedroom units, comp A is a good comp for 1-bedroom units but a bad comp for 2-bedroom units; while comp B is a good comp for 2-bedroom units but a bad comp for 1-bedroom units. Under current industry-wide property level selection practice, the underwriters have three different choices: A only, B only, or A and B. However, any choice brings errors to the rent estimation for the unit-type that are not ideal. For example, if A is selected, then the rent estimation on 2-bedroom units would be biased; if B is selected, then the rent estimation on 1-bedroom units would be biased; and if both A and B are selected, the rent estimation on both 1-bedroom and 2-bedroom units would be biased. Using the methods described herein, with the unit-type similarity scores, 1-bedroom units of comp A are selected for the 1-bedroom units of the target property; and 2-bedroom units of comp B are selected for the 2-bedroom units. Thus, unit-type similarity scores utilize both comps A and B to assist users in generating accurate estimations of rents for all available unit-types. Therefore, similarity score generation engine134can improve accuracy and efficiency of the current property-level rent comparable selection process.

In some implementations, sensitivity evaluation engine136can determine a sensitivity of calculated similarity scores to determine how sensitive to potential factors that could shock weighting factors, such as hyperparameter tuning or changes in the model sample population. In one example, for both the rent and expense models, each of the variable weights can be individually increased and decreased by predetermined percentages (e.g., +/−10% and +/−20%) in relative magnitude to determine the impact on the similarity score. In some example, similarity scores remained stable despite shifts in variable weights, indicating that the scores provide a reliable indicator of similarity in the face of shifting conditions.

In some implementations, the system108includes a feedback learning engine148incorporates feedback learning that is used to further train and refine machine learning algorithms used by analytics engine144to provide more accurate results. In some example, users102(e.g., underwriters, system backend administrators) provide feedback regarding the quality of system-identified comparable properties based on calculated similarity scores, and this feedback is used to refine and/or retrain the machine learning data models. In one example, the user-provided feedback can evaluate competencies such as whether any similarity score model-selected comps are useful to an underwriter or whether the system has produced enough similarity score-model selected comps.

In some embodiments, during a model testing phase, the system108may determine that a machine learning data model is successful when a predetermined number of recommended comps (e.g., at least 3 out of 5) are acceptable to a results reviewer. In some implementations, for each segment model (e.g., rent and expense segments), the user interface engine146provides comp recommendation results to the user102via one or more user interface (UI) screens for one or more properties based on similarity scores calculated by similarity score generation engine134. In some examples, for each property, the user102indicates whether each of the recommended comps is satisfactory and provides amplifying reasons why a given comparable property is or is not satisfactory to the user102. In some examples, for each subject property, the system108outputs a user review form showing a predetermined number of highest-ranking comps (in one example, 5 comps). For each comp, the form provides the similarity score the subject property and the comp and a list of one or more evaluation aspects, which can include unit size, exterior curb appeal, interior appeal (e.g., style, finishes, common areas), extent of renovations, on-site amenities, in-unit amenities, structural and/or mechanical conditions, living conditions (e.g., violations, tenant complaints), and restrictions and subsidies. For each of the listed evaluation aspects, the user102can provide comparable rating feedback indicating whether a respective evaluation aspect is very different, somewhat different, somewhat similar, or very similar to the subject property. Additionally, the user102can indicate whether a respective comp is acceptable (useful in evaluating the subject property) and what aspects of the comp are acceptable or not. In some examples, users102can submit separate forms for rent and expense comps.

In some examples, a given machine learning data model may be considered successful when its success rate, or the rate at which a reviewer deems one of its resulting comparable properties satisfactory, is above a predetermined threshold. In one example, the predetermined threshold is 80%. When the predetermined success threshold is exceeded, then the model may be placed into service for providing recommendations to general system users102. In some examples, if the success rate is less than the predetermined success threshold, then the system108continues to solicit feedback from users102, which is received by the feedback learning engine148and applied to the respective machine learning data models to improve their recommendation accuracy. In some implementations, outside of the testing phase when the system108is in general use, the system108can also receive feedback from general system users102, which can be similarly used to refine and retrain machine learning data models.

FIG.7illustrates a work flow diagram for an example similarity score modeling process700. In some examples, the work flow diagram shown inFIG.7is a high-level view of the processes illustrated by the flow charts inFIGS.10-13. In some implementations, the process700uses both physical property conditions and location conditions to generate a metric/score to measure the similarity between two commercial real estate properties with respect to a measurable property attribute, such as rent, expense, or value. Additionally, the process700can accommodate mixed type datasets to generate reasonable variable weights under an XGBoost framework. Further, the process700can use an advanced analytical algorithm to model the cognitive behavior and reasoning performed by real estate professionals when they compare properties, so that the data and the variable weights can be combined in a comprehensive way to generate appropriate scores. However, the analytical algorithm is able to remove human bias and lack of full knowledge in a way that humans are unable to do. In addition, different models can be used for different practices.

For rent comparables, the similarity score generation engine134can calculate property similarity score, which considers unit type level characteristics and property-level conditions. Then, the similarity score generation engine134calculates a neighborhood score between two properties, measured by differences in location conditions. Each neighborhood is defined as a Census tract for some location variables, and defined as a radius (e.g., 0.3 miles for NYC) for other location variables. Lastly, the minimum of property similarity score and neighborhood similarity score is another similarity score between two properties.

For expense comparables, because expenses are measured at the property level, the data set used in the XGBoost framework to generate variable importance is a property-level dataset. After the importance weights are calculated, the similarity score generation engine134calculates the property-level property similarity score between two properties, measured by differences in physical conditions. In addition, expenses can be much less sensitive to location conditions and thus, within one region, we use property similarity score as the final similarity score between two properties.

The process700can include a geospatial analysis stage702that relates location features to property-level data based on geospatial data received from external data sources720. At a data consolidation stage704, data from internal and external data sources720,724is merged together as property-level data, and the system108consolidates both physical property condition data and neighborhood condition data so that one similarity score can be calculated. In some implementations, at a data analysis stage706, the system108performs an exploratory data analysis and correlation analysis of the consolidated data. At data transformation stage708, data transformation engine142can transform variables of mixed type datasets to generate reasonable outcomes. Additionally, the data transformation engine142can bucket variables based on quantitative analysis and business knowledge. At dataset customization stage710, the system108can customized the consolidated data sets for the particular calculations and comparisons performed by the similarity score generation engine134(e.g., rent comps, expense comps). Further, the system108applies a Python XGBoost package712in preparation for executing the similarity score calculation algorithm714. Calculated similarity scores114can be stored in data repository110.

Returning toFIG.1, the property comparison system108can include a user interface engine146that generates, prepares, and presents user interface screens to users102in response to submitting a comp analysis query. In another implementation, the user interface engine146may be external to the property comparison system108yet still communicate with the property comparison system108via a network to receive data presented to a user at one or more user interface screens.FIG.2illustrates an example interactive map user interface screen200generated by user interface engine146that functions as a landing page for interacting with the system108. The user interface screen200is layered with internal and external data sources (e.g. various points of interests, Freddie Mac origination data, sociodemographic, etc.) uniquely combined for their analytical aggregation. In some examples, a search functionality202provided in the user interface screen200can be powered by a third party (e.g., Google) but offers functionality only used by internal servers and systems for a loan provider or government sponsored enterprise like Freddie Mac (e.g. search by loan number). Users102can also search by other geographic names (city, state etc.) The users102can specify the types of the analysis they are running such as rental comps or expense comps. The users102can also specify one or more filtering criteria204including whether they prefer to limit the comps pool to only those properties financed by a particular loan provider and/or purchaser or if they also would like to see external comparables. Multicolored markers on the map identify the subject property and all available comparables within a set radius for easy location analysis.

As users102click on various markers206in the user interface screen200, they can review similarity scores, key characteristics and exterior and interior pictures for the respective properties that are unique to a loan purchaser or provider. The markers for the best comps for any given property are identified with a star symbol208. The best comps can be defined as the properties with the highest similarity scores within a given radius210. The default number of best comps (e.g., top 5 versus top 10) can be changed by users102depending on the specific needs of their analysis. The best comps form a core comparison set that is subsequently used to benchmark a subject property's performance.

Another user interface screen generated by user interface engine146is a rent comp analysis user interface screen300a, shown inFIG.3A. In some implementations, the user interface screen300acan be customized for rental comps analysis. The user interface screen300acan supply additional information for the properties that fell within the defined radius210on the map of the interactive map user interface screen200shown inFIG.2. In some examples, there are two sections for users102to review a subject property's302aand comparable properties'304pictures in bigger format. In some implementations, subject property section302acan include one or more appraisal photos of the subject property as well as summary information for the property (e.g., address, year built, renovation date, renovation amount, neighborhood). In some examples, the user interface screen300can also include a tabular section306athat presents a broad collection of data elements for the properties being compared. This data comes from various internal and external data sources (e.g., internal data sources104and external sources106stored in data repository110) and is presented in a searchable/sortable table format with users being able to add or remove any attribute they prefer. The default sorting within the tabular section306ais based on rent comparison model similarity scores. The properties with the higher scores (the top comps set) are visually separated from the rest of the properties within the table by a divider line308a. Users102are provided with the functionality to add/remove the comps to/from the top comps set based on their review of related data and documents. Once the user confirms the top comps set has been finalized, the app takes the user102to rent results user interface screen400shown inFIG.4.

In some implementations,FIG.3Billustrates another example of rent comp analysis user interface screen300b. In some examples, the user interface screen300bcan display the properties within the defined radius210on the map displayed within user interface screen200(FIG.2) and a detail section302bfor the subject property. In some implementations, subject property section302bcan include one or more appraisal photos of the subject property as well as summary information for the property (e.g., address, year built, renovation date, renovation amount, neighborhood). In addition, the user interface screen300bcan include a tabular section306bthat presents a broad collection of data elements for the properties being compared. In some aspects, the tabular section306bincludes one or more tabs314that allow a user to select results by unit type (e.g., studio, 1BR, 2BR, 3BR, 4BR+). displayed. For example, the system108can generate similarity scores for properties with respect to unit type so that users can obtain more refined results based on the type of units available in a given property. For each property displayed within tabular section306b, one or more property details such as loan number316, property address318, property similarity score320, neighborhood similarity score322, distance from the subject property324, year built326, and/or year renovated328may be In some implementations, each of the properties identified within the tabular section306bis selectable, and upon selection, additional details for the comp may be displayed. In one example, the tabular section306bincludes a first sub-section310that displays a predetermined number of properties that are ranked according to rent similarity score for the property. In one example, the first sub-section displays the top five ranked properties. In a second sub-section312, other available comps within a predetermined radius of the subject property302bmay be displayed and may be ranked in order of property similarity score.

In some implementations, the rent results user interface screen400shown inFIG.4presents a variety of rent summary statistics for the subject property and the comparable properties in the top set above divider line308. In particular, the rent results user interface screen400can present various charts402,404that compare the subject property's rents to the comparable properties' rents as well as to the other common benchmarks within the industry (multifamily data vendors, approved appraisal, etc.). The rent results user interface screen400can also include a tabular presentation408of the comparisons shown in the charts402,404. In some examples, the app, via the user interface engine146, automatically highlights any material deviations between the subject property and the comps. Various hover over messages direct users' attention to the outliers and recommend potential course of action. In some examples, there is also a Rent Roll section406where the rents for all units at the subject property are color-coded based on the magnitude of their deviation from the top comps set. Hover over messages in this section406contain recommendations on how users can address material outliers. The users102have the ability to export the summaries from the rent results user interface screen400so they can be utilized in various approval/investment summary documents outside of the application.

In some implementations, the user interface engine146can also present an expense analysis user interface screen500ato external devices158of users102as shown inFIG.5B. The user interface screen500acan be customized for expense comparables analysis. The overall functionality can be very similar to the rent analysis user interface screen300a. Some default settings are different, to better account for the specifics of expense analysis. In one example, the user interface screen500acan also include two sections for users102to review subject property's502aand comparable properties'504pictures in bigger format. In some implementations, subject property section502acan include one or more appraisal photos of the subject property as well as summary information for the property (e.g., address, year built, renovation date, renovation amount, neighborhood). The default sorting within comparables table506acan be based on expense comparison similarity scores. The process flow at the expense analysis user interface screen500acan also be similar to the process flow at the rent analysis user interface screen300a. Users102can modify the top expense comps set based on their review of relevant data and documents. Once the top set is finalized, the app takes the user to expense results user interface screen600a,bshown inFIGS.6A-6B.

FIG.5Bshows another example of an expense analysis user interface screen500b. The overall functionality can be very similar to the rent analysis user interface screen300b. In some examples, the user interface screen500bcan display the properties within the defined radius210on the map displayed within user interface screen200(FIG.2) and a detail section502bfor the subject property. In some implementations, subject property section502bcan include one or more appraisal photos of the subject property as well as summary information for the property (e.g., address, year built, renovation date, renovation amount, neighborhood). For each property displayed within tabular section506b, one or more property details such as loan number516, property address518, property similarity score520, neighborhood similarity score522, distance from the subject property524, year built526, and/or year renovated528may be displayed. In some implementations, each of the properties identified within the tabular section506bis selectable, and upon selection, additional details for the comp may be displayed. In one example, the tabular section306bincludes a first sub-section510that displays a predetermined number of properties that are ranked according to expense similarity score for the property. In one example, the first sub-section displays the top five ranked properties. In a second sub-section512, other available comps within a predetermined radius of the subject property may be displayed and may be ranked in order of property similarity score.

In some implementations, the expense results user interface screen600ashown inFIG.6Apresents comparisons of a subject property's historical and forecasted expenses to those presented by the comps within the top set (e.g., above divider line508in user interface screen500a). Expense data comes from financial statements provided to loan purchasers or providers at the point of underwriting as well as from the financials supplied by the property operators to loan servicers. The app automatically highlights any material deviations between the subject. In some examples, the user interface screen600acan include a first section602athat displayed underwritten expenses for comps, a second section604athat displays servicing expenses for comps, and a third section606athat displays a delta or difference between the underwriting versus servicing expenses. In some implementations, the users102are able to create a new expense pro forma for the subject right within the page. As various expense categories are entered or modified on this pro forma, the app color-codes them depending on how they compare to the comp ranges. Various hover over messages direct users' attention to the outliers and recommend potential course of action. Users102also can export the summaries from user interface screen600aso they can be utilized outside of the application.

FIG.6Bshows another example of an expense results user interface screen600b. In some implementations, the information displayed within user interface screen600bis similar to the information displayed within user interface screen600a. For example, the user interface screen may include a section602bthat displays underwritten expenses for comps and another section604bthat displays servicing expenses for comps. In addition, the user interface screen600bcan also include a subject property expense section608that displays expense information for the subject property over predetermined time periods. In some embodiments, the user interface screen600bcan also include a user input614that allows a user to toggle between viewing expense data per unit or per square feet.

Turning toFIG.17, output results1700generated by a property comparison system are illustrated. In some implementations, in response to receiving a request for a similarity score determination from a user102for property1700, the property comparison system108generates a set of rent comparables1704and a set of expense comparables1706for presenting to the user102. For example, for each of the rent comparables1704, a respective address1708and similarity score1712is displayed. Similarly, for each of the expense comparables1706, a respective address1710and similarity score1714is displayed. In some examples, each of the listed rent comparables1705and expense comparables1706can be selectable by the user102at the user interface screen1700, which causes additional details for the respective property to be viewed by the user102. Upon viewing the property details, the user102may indicate at feedback sections1716,1718which of the presented comparables is an acceptable comparable property for the property1702, which can be incorporated by the system108to retrain and refine the machine learning data models so that the accuracy of similarity score calculations can be improved. For example, in the expense feedback section1718, the property with the highest similarity score of the group (95%) was flagged as being “not acceptable” while all the other properties were deemed “acceptable” by the user102. For the rent comparables1704, the property with the lowest similarity score was flagged as being “not acceptable” while all the other properties with higher similarity scores were “acceptable.”

Turning toFIG.14, a data architecture for aspects of a comparable property platform1400is illustrated. In some implementations, the platform1400can be an example implementation of the system108shown inFIG.1. The platform1400, in some examples, can include a comparables engine1402that includes a comparables application1410, comparables service1412, comparables data adapter1414, and comparables similarity score model/algorithm1420. In some embodiments, comparables application1410generates user interfaces and interactive experience that are presented at an application interface1422on an end user device1404. For example, via application interface1422, an end user can select comparable properties and view similarity scores at an external device. In some examples, the comparables application1410interacts with representational state transfer (REST) API to allow all applicable to be displayed within the application. In some implementations, the comparables application1422provides each user with a customized experience in response to user interactions with the application1422.

In some examples, comparables service1412provides third-party data vendor APIs to search for a subject property and its comparable properties based on rent and expense comparisons in response to a user query received via the application. In one example, the comparables service1412uses comparables database1416as the data source for properties and analytical information presented at the comparables application interface1422. In addition, comparables application1422can receive information form internal data sources1406such as image service1424(for example, an API for delivering property photos to be displayed within a UI screen) and document management service1426and external data sources1408such as geospatial data sources1434for configuring user interface screens at an application interface. In some embodiments, comparables similarity score calculator1420ingests data from internal data sources1406(e.g., collateral services1428, collateral assessment services1430, and sourcing product services1432that includes loan information and property photos) and external data sources1408(e.g., external geospatial data sources1434such as Google Places), processes and transform the ingested data, and calculates rent and expense similarity scores for the queried property. The processes performed by similarity score model/algorithm can correspond to those performed by analytics engine144, data transformation engine142, missing data engine152, feature selection engine150, and similarity score generation engine134of the property comparison system108ofFIG.1. In some examples, comparables data adapter1414performs database initialization and data extraction for comparables database1416. In some embodiments, comparables engine1402can also include a cloud database1418, such as the Amazon Web Services S3 service, that provides a conduit for ingesting data to the comparables engine1402from external data sources1408.

In some embodiments, the comparable property platform1400can be integrated with other computing platforms in an underwriting and/or risk evaluation system. For example, the comparable similarity score model/algorithm can complement other risk analysis and loan processing tools for both SBLs and other loan products.

Turning toFIGS.10-13, flowcharts of processes performed by property comparison system108are illustrated. The order in which the flow charts are described does not necessarily indicate the order in which the processes are performed. For example, method1000shown inFIG.10may be performed before or after method1100shown inFIG.11. Additionally, the method1200shown inFIG.12may be performed before or after the method ofFIG.11. For example,FIG.10shows a flow chart of an example method1000for filling in or amplifying missing data features in property data sets. In some examples, the method1000is performed at least in part by missing data engine152of the system108.

In some implementations, the method1000commences with receiving feature data sets of internal and external data111,112(1002). In some examples, the data sets may be grouped according to applicable output variable (e.g., rent or expense) and associated region for a particular regional similarity score model (e.g., city, county, state, region).

In some examples, if there are complementary features from multiple data sources certain features of external data112can be identified that best complement particular features of internal data111, and these complementary features can be linked together to fill in one or more missing feature entries (1004). For example, both collected property level data and Pluto (for New York City properties) include renovation date and build year. In some examples, if there are any missing data features associated with any missing data rules120(1006), then missing data engine152can apply the missing data rules to fill in those features (1008). For example, default values can be applied for certain features when those features are missing from the data sets. For example, if “parking garage” or “elevator” features are missing, the missing feature engine152may apply values of “0” to those features.

In some implementations, if any internal or external data111,112include any text files or textual data fields that can be mined for information (1010), then in some examples, missing feature engine152can extract textual features from the text files and data fields and apply those features to respective missing feature data entries (1012). For example, OUS data includes a data filed for “property comments” that may include details about a laundry facility on site. In one example, the missing feature engine152may extract text such as “laundry room” or “in-unit laundry” to determine whether a property includes a laundry feature or not.

In some embodiments, if missing features can be derived from other features (1014), then in some example, the missing feature engine152can impute missing data from another available feature based on a correlation between the features (1016). For example, a building floor number can be inferred from a first value of a unit number. In some embodiments, if any of the data source files include image files (1018), one or more image processing sub-engines of missing feature engine152can detect and impute missing features from image files from one or more internal data sources111such as photos in appraisal files (1020). For example, the image processing sub-engine may be configured to detect the presence of certain missing features by detecting those features within appraisal photos (e.g., laundry room features such as washers, dryers, and deep sinks). The image processing sub-engine may also be able to detect changes in property images that may be indicative of property degradation or renovations, which can be used to determine renovation year of a property.

Although illustrated in a particular series of events, in other implementations, the steps of the missing data derivation process1000may be performed simultaneously or in a different order. For example, any of the techniques for deriving missing data features can be applied in any order (e.g., text data mining and application (1012) may be performed with application of missing data rules (1008)). Further, one or more of the missing data derivation techniques may be omitted from the process (e.g., detecting features from image files (1020)). Additionally, in other embodiments, the process may include more or fewer steps while remaining within the scope and spirit of the missing data amplification process1000.

Turning toFIG.11, a flow chart of an example method1100for identifying correlated features and weighting values for measuring comparable similarity is illustrated. In some examples, the method1100is performed at least in part by feature selection engine150and analytics engine144of the system108.

In some implementations, the method1100commences with receiving feature data sets (1102). In some examples, the feature data sets can include raw internal and external data111,112or data sets that have had missing data filled in by missing data amplification process1000. If more than a threshold percentage of features are missing in the feature data sets for properties (1104), then in some examples, any of the features that fall below the threshold percentage are dropped from the analysis (1106). In one example, the threshold percentage is 50%. Further, if variation for at least one feature is less than a predetermined percentage (1108), then in some examples, the at least one feature is also dropped from the feature set (1110). In one example, the feature variation threshold percentage is 10%.

In some implementations, the feature selection engine150can calculate feature correlations to group highly correlated features into the same categorical division or bucket (1112). In one example, the correlation analysis is a Pearson correlation analysis. In some examples, a portion of the features in each bucket can be selected for use by the system108in determining rent and expense comparables (1114). In some implementations, a feature for each bucket that has a highest correlation with an outcome variable (e.g., rent or expense) is selected as the feature for a respective bucket. In some implementations, the bucketed data population is filtered to remove data entries that have missing values for many property features (1116). In one example, only properties with certain core property attributes (e.g., unit size and number of units) are retained for analysis. Additionally, any property information associate with loans that are dead deals or have not yet passed the transfer to purchase phase of origination are removed from the data population.

In some implementations, analytics engine144runs a machine learning process on the filtered data population features to further identify features used for similarity score calculations (1118). In some examples, the feature data sets are used to train machine learning data models, which determines the features and weights that are the most predictive of comparable properties. In some examples, the features having the lowest weighting values are dropped from the analysis (e.g., the features having weights that are less than a predetermined threshold or features that fall within a lowest percentage of weights). In other examples, the identified features and weights are presented to a user (e.g., an underwriter or other subject matter expert) who flags one or more features for removing from the analysis. In some examples, if any features are identified for dropping (1120), then those features are removed from the analysis (1122), and the machine learning feature identification process is performed again. In some examples, feature selection can be an iterative process that continues until all features have importance weights that are greater than a predetermined threshold or fall within a predetermined range (1124).

Although illustrated in a particular series of events, in other implementations, the steps of the feature identification and weighting process1100may be performed simultaneously or in a different order. For example, removal of features that are missing at greater than a threshold rate (1104,1106) may be performed after or simultaneously with removal of features that have less than a threshold rate of variation (1108,1110). Additionally, in other embodiments, the process may include more or fewer steps while remaining within the scope and spirit of the feature identification and weighting process1100.

Turning toFIG.12, an example method1200for performing data transformation of feature sets is illustrated. In some examples, at least a portion of the method1200may be performed by data transformation engine142of property comparison system108. In some embodiments, data transformation engine142produces data structures that capture relationships between observed property features and property quality to most accurately measure similarity between properties by transforming raw data received from internal and external data sources111,112into meaningful features.

In some implementations, the method1200commences with applying data transformation rules122to a feature data set for one or more multifamily properties (1202). The data sets can include features from internal and external data sources111,112. In some examples, the feature data sets have been augmented with missing data features by the method1000described above (FIG.10). In some aspects, the data transformation rules can transform raw data into normalized data values. For example, instead of using unit rent as an outcome variable, the system108calculates “rent per square foot (sqft)” because it may better reveal the property quality. In another example, instead of using a dummy variable for each of the amenities, the data transformation engine142can combine amenity features together to calculate the total number of amenities for a property. Other examples include, but are not limited to, calculating renovation amount per unit in place of total building renovation amount, calculating the ratio of commercial area to residential area in place of including both commercial and residential areas, and, for each neighborhood, calculating population density, instead of using total population.

In some implementations, if the data sets include continuous data values (1204), data transformation engine142can also convert continuous value features into categorical features, which improves overall processing efficiency and accuracy (1206). For example, for almost all continuous features, a relationship between a given feature value and property quality is not continuous. For example, because renovation approaches, trends, and styles change and age, a property renovated within in 2 years may not considered different from another property renovated within 3 years. However, a property renovated within 5 years may be considered with better quality than another property renovated 6 years ago. Thus, a categorical feature “renovated_date_category” may be created, which has a value of 0 if a property is renovated within 5 years; has a value of 1 if a property is renovated 6-10 years ago; has a value of 2 if a property is renovated 11-20 years ago; and has a value of 3 if a property is renovated more than 20 years ago. In another example, unit square footage can be categorized based on a range of square foot values (e.g., 500-750 sqft, 751-1000 sqft, 1000-1250 sqft, etc.).

In some examples, data transformation engine142re-buckets or classifies categorical features to categorical buckets or divisions (1208). In some implementations, some property features may be assigned to the wrong buckets based on raw data values, which can result in biased importance weights for these features. To correct the importance weights of these features, the data transformation engine142re-bucket the categorical into more appropriate categories. For example, for raw data, a feature for “property overall quality” may 9 categories: 1 (poor); 2 (Fair); 3 (Average); 4 (Good); 5 (Excellent); 6 (Excellent); 7 (Above Average); 8 (Average); 9 (Below Average); 10 (Inferior). In some examples, some of these categories may have very few data observations assigned to each bucket, and some categories may represent substantially similar property quality. Applying the original overall quality data from the raw data to the machine learning process may generate a very small weight due to these data bucketing issues even though property quality may be very important to determining comparable property similarity. To boost the weight of this feature to a level that accurately represents the importance of the property quality feature, this feature may be bucketed into three categories: “Above Average” if the original value is 4, 5, 6, or 7; “Average” if the original value is 2, 3, or 8; and “Below Average” if the original value is 1 or 9. This data transformation and re-classifying into buckets helps ensure that the importance weight of this feature increases and has a greater impact on the comparable analysis. In some examples, the data transformation engine142performs these re-bucketing procedures based on feedback received from subject matter experts from underwriters. These data transformations and classifications also account for outliers, which may be grouped into a first bucket or a last bucket for the feature. In some examples, the method1200continues until all continuous features have been converted into categorical features (1210).

Although illustrated in a particular series of events, in other implementations, the steps of the data transformation process1200may be performed simultaneously or in a different order. For example, application of transformation rules to data sets (1202) may be performed after or simultaneously with converting continuous features to categorical features (1206). Additionally, in other embodiments, the process may include more or fewer steps while remaining within the scope and spirit of the data transformation process1200.

Turning toFIG.13, a flow chart for an example method1300of generating comparable property recommendations is illustrated. At least a portion of the method1300may be performed by similarity score generation engine134, feedback learning engine148, and user interface engine146.

In some implementations, the method1300commences with user interface engine146receiving a query for a comparable property analysis (1302). The query may be received from an end user (e.g., underwriter) requesting a set of rent and expense comps for a subject property. In another example, the query may be received from a backend system administrator or subject matter expert testing the accuracy of recommendations generated by the system.

In some examples, responsive to receiving a comp recommendation query, similarity score generation engine134calculates similarity scores for rent and/or expense output variables for the one or more subject properties based on the feature variables and weights output by the trained machine learning data models (1304). Using the output features and weights, neighborhood features and their weights are used to calculate a neighborhood similarity score, which measures a difference in two neighborhoods where comparable properties reside. For rent comparables, the data models and similarity scores account for unit type level characteristics and property-level conditions. This allows the system108to generate property-level similarity scores at unit-type level. In some examples, for expense comparables, because expenses are measured on a property-level only and expense comparisons are much less sensitive to neighborhood conditions, expense similarity scores are calculated using property-level physical features. In some examples, the similarity score generation engine134applies a Gower distance algorithm to calculate a multi-dimensional geometry distance between any pair of properties, using the selected feature and their feature importance. In some examples, the similarity score generation engine134may apply the Gower distance algorithm to calculate different types of similarity scores for a given property. For example, the similarity score generation engine134may calculate similarity scores for a property level, neighborhood level, and/or unit type level associated with the subject property.

In some implementations, user interface engine146outputs one or more comparable properties for the subject property to the user102via one or more user interface screens (1306). In one example, the user interface engine146may output a number of highest-ranking comparable properties with respect to rent and expense output variables. For example, user interface screens300a,binFIGS.3A and3Bdisplay the top-ranking rent comparables, and user interface screens500a, binFIGS.5A and5Bdisplay the top-ranking expense comparables.

Although illustrated in a particular series of events, in other implementations, the steps of the comparable property recommendation process1300may be performed simultaneously or in a different order. Additionally, in other embodiments, the process may include more or fewer steps while remaining within the scope and spirit of the comparable property recommendation process1200. For example, in one example, feedback steps (1308,1310) may be omitted from the process.

In some embodiments, the user interface screens may allow the user viewing the results to provide feedback regarding whether the recommended comparable properties are accurate comparable properties. If the user submits feedback to the system108(1308), then in some examples, feedback learning engine148incorporates the received feedback to further train and refine machine learning algorithms used by analytics engine144to provide more accurate results (1310). In some example, users102(e.g., underwriters, system backend administrators) provide feedback regarding the quality of system-identified comparable properties based on calculated similarity scores, and this feedback is used to refine and/or retrain the machine learning data models. In one example, the user-provided feedback can evaluate competencies such as whether any similarity score model-selected comps are useful to an underwriter or whether the system has produced enough similarity score-model selected comps.

In some embodiments, the implementations described herein can be further refined through an iterative process of building and testing the customized algorithms and using the testing results to improve the algorithms. In addition, the implementations described herein can be expanded to cover nationwide regions and to cover various loan types. For example, while the implementations described herein describe calculating similarity scores for multi-family SBL properties, other types of properties and loan types can also be included. The iterative system can be applied for every region, every loan type, and every practice. The implementations described herein can also be applied in other applications or industries. For example, the system108can be used by other industry professionals (e.g., servicers, lenders, landlords, borrowers) that need to perform property comparison (e.g., for purchase, benchmarking, underwriting, property management, securitization, investing). Tenants looking for similar commercial and residential real estate buildings, rating agencies and investors to evaluate securitization collateral, and insurance agents estimating insurance premiums for real estate assets.

Next, a hardware description of a computing device, mobile computing device, computing system, or server according to exemplary embodiments is described with reference toFIG.8. The computing device, for example, may represent the users102, external data sources106, or one or more computing systems supporting the functionality of the property comparison system108, as illustrated inFIG.1. InFIG.8, the computing device, mobile computing device, or server includes a CPU800which performs the processes described above. The process data and instructions may be stored in memory802. The processing circuitry and stored instructions may enable the computing device to perform, in some examples, the methods700ofFIG.7. These processes and instructions may also be stored on a storage medium disk804such as a hard drive (HDD) or portable storage medium or may be stored remotely. Further, the claimed advancements are not limited by the form of the computer-readable media on which the instructions of the inventive process are stored. For example, the instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or any other information processing device with which the computing device, mobile computing device, or server communicates, such as a server or computer. The storage medium disk804, in some examples, may store the contents of the data repository110ofFIG.1, as well as the data maintained by the users102, and external data sources106prior to accessing by the property comparison system108and transferring to the data repository110.

Further, a portion of the claimed advancements may be provided as a utility application, background daemon, or component of an operating system, or combination thereof, executing in conjunction with CPU800and an operating system such as Microsoft Windows, UNIX, Solaris, LINUX, Apple MAC-OS and other systems known to those skilled in the art.

CPU800may be a Xeon or Core processor from Intel of America or an Opteron processor from AMD of America, or may be other processor types that would be recognized by one of ordinary skill in the art. Alternatively, the CPU800may be implemented on an FPGA, ASIC, PLD or using discrete logic circuits, as one of ordinary skill in the art would recognize. Further, CPU800may be implemented as multiple processors cooperatively working in parallel to perform the instructions of the inventive processes described above.

The computing device, mobile computing device, or server inFIG.8also includes a network controller806, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for interfacing with network828. As can be appreciated, the network828can be a public network, such as the Internet, or a private network such as an LAN or WAN network, or any combination thereof and can also include PSTN or ISDN sub-networks. The network828can also be wired, such as an Ethernet network, or can be wireless such as a cellular network including EDGE, 3G, 4G, and 5G wireless cellular systems. The wireless network can also be Wi-Fi, Bluetooth, or any other wireless form of communication that is known. The network828, for example, may support communications between the property comparison system108and any one of the users102or external data sources106.

The computing device, mobile computing device, or server further includes a display controller808, such as a NVIDIA Geforce GTX or Quadro graphics adaptor from NVIDIA Corporation of America for interfacing with display810, such as a Hewlett Packard HPL2445w LCD monitor. A general purpose I/O interface812interfaces with a keyboard and/or mouse814as well as a touch screen panel816on or separate from display810. General purpose I/O interface also connects to a variety of peripherals818including printers and scanners, such as an OfficeJet or DeskJet from Hewlett Packard. The display controller808and display810may enable presentation of user interfaces for submitting requests to the property comparison system108.

A sound controller820is also provided in the computing device, mobile computing device, or server, such as Sound Blaster X-Fi Titanium from Creative, to interface with speakers/microphone822thereby providing sounds and/or music.

The general purpose storage controller824connects the storage medium disk804with communication bus826, which may be an ISA, EISA, VESA, PCI, or similar, for interconnecting all of the components of the computing device, mobile computing device, or server. A description of the general features and functionality of the display810, keyboard and/or mouse814, as well as the display controller808, storage controller824, network controller806, sound controller820, and general purpose I/O interface812is omitted herein for brevity as these features are known.

One or more processors can be utilized to implement various functions and/or algorithms described herein, unless explicitly stated otherwise. Additionally, any functions and/or algorithms described herein, unless explicitly stated otherwise, can be performed upon one or more virtual processors, for example on one or more physical computing systems such as a computer farm or a cloud drive.

Reference has been made to flowchart illustrations and block diagrams of methods, systems and computer program products according to implementations of this disclosure. Aspects thereof are implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

Moreover, the present disclosure is not limited to the specific circuit elements described herein, nor is the present disclosure limited to the specific sizing and classification of these elements. For example, the skilled artisan will appreciate that the circuitry described herein may be adapted based on changes on battery sizing and chemistry or based on the requirements of the intended back-up load to be powered.

The functions and features described herein may also be executed by various distributed components of a system. For example, one or more processors may execute these system functions, wherein the processors are distributed across multiple components communicating in a network. The distributed components may include one or more client and server machines, which may share processing, as shown onFIG.9, in addition to various human interface and communication devices (e.g., display monitors, smart phones, tablets, personal digital assistants (PDAs)). The network may be a private network, such as a LAN or WAN, or may be a public network, such as the Internet. Input to the system may be received via direct user input and received remotely either in real-time or as a batch process. Additionally, some implementations may be performed on modules or hardware not identical to those described. Accordingly, other implementations are within the scope that may be claimed.

In some implementations, the computing devices described herein may interface with a cloud computing environment930, such as Google Cloud Platform™ to perform at least portions of methods or algorithms detailed above. The processes associated with the methods described herein can be executed on a computation processor, such as the Google Compute Engine by data center934. The data center934, for example, can also include an application processor, such as the Google App Engine, that can be used as the interface with the systems described herein to receive data and output corresponding information. The cloud computing environment930may also include one or more databases938or other data storage, such as cloud storage and a query database. In some implementations, the cloud storage database938, such as the Google Cloud Storage, may store processed and unprocessed data supplied by systems described herein. For example, internal data111, external data112, similarity scores114, combined data116, feature data and weights118, missing data rules120, and data transformation rules122may be maintained by the property comparison system108ofFIG.1in a database structure such as the databases938.

The systems described herein may communicate with the cloud computing environment930through a secure gateway932. In some implementations, the secure gateway932includes a database querying interface, such as the Google BigQuery platform. The data querying interface, for example, may support access by the property comparison system108to data stored on any one of the users102.

The cloud computing environment930may include a provisioning tool940for resource management. The provisioning tool940may be connected to the computing devices of a data center934to facilitate the provision of computing resources of the data center1234. The provisioning tool940may receive a request for a computing resource via the secure gateway932or a cloud controller936. The provisioning tool940may facilitate a connection to a particular computing device of the data center934.

A network902represents one or more networks, such as the Internet, connecting the cloud environment930to a number of client devices such as, in some examples, a cellular telephone910, a tablet computer912, a mobile computing device914, and a desktop computing device916. The network902can also communicate via wireless networks using a variety of mobile network services920such as Wi-Fi, Bluetooth, cellular networks including EDGE, 3G, 4G, and 5G wireless cellular systems, or any other wireless form of communication that is known. In some examples, the wireless network services920may include central processors922, servers924, and databases926. In some embodiments, the network902is agnostic to local interfaces and networks associated with the client devices to allow for integration of the local interfaces and networks configured to perform the processes described herein. Additionally, external devices such as the cellular telephone910, tablet computer912, and mobile computing device914may communicate with the mobile network services920via a base station956, access point954, and/or satellite952.

Reference throughout the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the subject matter disclosed. Thus, the appearance of the phrases “in one embodiment” or “in an embodiment” in various places throughout the specification is not necessarily referring to the same embodiment. Further, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. Further, it is intended that embodiments of the disclosed subject matter cover modifications and variations thereof.

It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context expressly dictates otherwise. That is, unless expressly specified otherwise, as used herein the words “a,” “an,” “the,” and the like carry the meaning of “one or more.” Additionally, it is to be understood that terms such as “left,” “right,” “top,” “bottom,” “front,” “rear,” “side,” “height,” “length,” “width,” “upper,” “lower,” “interior,” “exterior,” “inner,” “outer,” and the like that may be used herein merely describe points of reference and do not necessarily limit embodiments of the present disclosure to any particular orientation or configuration. Furthermore, terms such as “first,” “second,” “third,” etc., merely identify one of a number of portions, components, steps, operations, functions, and/or points of reference as disclosed herein, and likewise do not necessarily limit embodiments of the present disclosure to any particular configuration or orientation.

Furthermore, the terms “approximately,” “about,” “proximate,” “minor variation,” and similar terms generally refer to ranges that include the identified value within a margin of 20%, 10% or preferably 5% in certain embodiments, and any values therebetween.

All of the functionalities described in connection with one embodiment are intended to be applicable to the additional embodiments described below except where expressly stated or where the feature or function is incompatible with the additional embodiments. For example, where a given feature or function is expressly described in connection with one embodiment but not expressly mentioned in connection with an alternative embodiment, it should be understood that the inventors intend that that feature or function may be deployed, utilized or implemented in connection with the alternative embodiment unless the feature or function is incompatible with the alternative embodiment.

While certain embodiments have been described, these embodiments have been presented by way of example only and are not intended to limit the scope of the present disclosures. Indeed, the novel methods, apparatuses and systems described herein can be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods, apparatuses and systems described herein can be made without departing from the spirit of the present disclosures. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the present disclosures.