Learning Systems Based Automated Proposal Evaluation

Learning systems based automated proposal evaluation can protect the public procurement agencies and funds from fraud, corruption, unusual, and callas proposal evaluations by evaluators. The impact on the economy from such system is significant. Savings can be in hundreds of billions of dollars across the whole world. Research suggests that the objective evaluation of a proposal received in response to a Request For Proposal (RFP) is heavily influenced by evaluators’ expertise, experience, and knowledge domain. Evaluating proposal from proposers is not easy. Among other challenges, evaluators have to balance objective evaluation decisions against project usability objectives, specific criteria set by the procurement agency, and account for the increasingly diverse range of alternative technical approaches available. Intelligent proposal evaluation support systems are a potential solution to this problem. They can eventually become automated proposal evaluation systems. However, to intelligently assist evaluators in evaluating proposals, computer systems need to possess computable representations of human readable proposals in text. Using machine-extracted text-based features and evaluation judgments from 16 evaluators on 52 proposals, we present preliminary results based on established methodology that suggest that such representation may be possible. This is the first step towards realization of a learning systems based automated proposal evaluation system. INTRODUCTION AND MOTIVATION Sysoft eRFP, a group decision support system (http://erfp.sysoft.com) helps Government, and corporate procurement agencies evaluate proposals in response to a Request for Proposal (RFP) [1]. The system is used all over US and abroad for small, medium, and large scale complex procurement projects where best value is desired instead of lowest cost. The procurement projects include IT systems, Defense Systems, Construction Projects, Service Procurements like Security Services, Commercial Aviation Cargo Handling and many more. The scale of procurement ranges from hundreds of thousands of dollars to multi-billion dollar systems procurement. The current version of eRFP uses multiple human evaluators to evaluate proposals against specific criteria and evaluation guiding rules. The System collects all the scores and comments from all evaluators, and provides a report based on scores and evaluator comments. It is a two-step process. First, proposals are shortlisted as acceptable or unacceptable. In the second step, top proposal is chosen for contract award. One substantial enhancement request from the clients of Sysoft eRFP is to assist the human evaluators with machine learning system that can guide human evaluators evaluate these proposals. Eventually it is desired that these human evaluators be replaced by the learning system enhancing the speed of evaluation and productivity. In addition, for some large and real complex procurement efforts, the machine learning system can be used to validate human agent based evaluation and act as “quick and simple” audit system to protect the public agencies and funds from fraud, corruption, unusual, and callas proposal evaluations by evaluators. Every procurement entity evaluates proposals in slightly different manner. The criteria are also different depending on the RFP. Interestingly, industry norm is to provide some rule based objective guidance rules and recommendations for evaluators to evaluate these proposals. The rules can be as simple as "look for the word 'Neurogenesis' and see how many times used. " As mentioned, these rules differ from one procurement agency to another and sometimes are dependent on Federal/State/local laws and internationally country specific laws. This is the reason why the learning system may be specific to a procurement entity. Eventually it may be possible to build a software configurator for tuning the specific attributes of the learning system to suit different agencies and RFP types in the future. This will make the system universal for all procurement usage. Evaluating proposals from proposers is challenging for many reasons. Evaluators have to balance objective evaluation decisions against project usability objectives, specific criteria set by the procurement agency, and account for the increasingly diverse range of alternative technical approaches available in response to RFPs [1]. A potential solution to this problem is intelligent proposal evaluation support tools that can assist in proposal evaluation. These tools should be able to validate manual evaluations by human agents. However, in order to achieve this goal, such systems must have the capacity to represent proposal 2 evaluation attributes in a computable form. eRFP proposal evaluation system should be able to represent human evaluation judgments in terms of variables in the evaluation space. Using machine-extracted proposal evaluation attributes from text-based features and evaluation judgments of 16 human evaluators on 52 proposals, we present preliminary results based on established methodologies that suggest that such representation may be possible [2]. This is the first step towards realization of a learning systems based automated proposal evaluation system. In the remainder of this paper we summarize prior work in the area of statistical learning of “criteria based appeal” for proposals. We then describe the process of data collection and an experiment to investigate the possibility of representing proposal evaluation attributes for human readable textual proposals in a computable form. We conclude with a discussion of our findings and recommendations for future work in this technical area. PREVIOUS WORK Previous research has demonstrated that automatic text summarization based on word clusters and ranking algorithms allows one to adapt summaries to the user needs and to the corpus characteristics [3]. A review of the theory and methods of document classification and text mining, focusing on the existing literature shows possibilities of computable representations of human readable proposals in text[4]. The authors of the aforementioned work suggest that it is possible to develop computer algorithms that can deliver ‘quick and dirty evaluations’ of subjective aspects of any text based proposal. However, neither of these papers conclusively proves that proposal evaluation attributes can be represented in computable form. We extend this previous research in a few significant and novel ways. First, we learn statistical models for predicting proposal evaluation attributes in terms of text-based features which have already been employed in text mining applications. These text-based features describe a proposal’s use of fundamental corpus statistics: Word frequencies, co-occurrences, measure of point-wise mutual information for dependences between words, and word cooccurrences at shorter distances with word order [5]. They are informed by procurement agency’s evaluation guidelines and evaluation subject matter experts (SMEs). Next, we employ performance evaluation to demonstrate that it may be possible for text-based feature models to learn and represent proposal evaluation attributes in a computable form. COMPUTATION AND DATA ACEESS PLATFORM The biggest challenge has been preparing the data for learning and testing. One specific procurement agency was selected for developing and testing prototype machine learning system that can evaluate the proposals based on responses from Word frequencies Co-occurrences Measure of point-wise mutual information for dependences between words Word co-occurrences at shorter distances with word order Figure 1. Our statistical models to predict proposal evaluation appeal were built using text-based features. These text based features are informed by interviews with subject matter expert evaluators, and evaluation guidelines from the procurement agency. vendors and proposal evaluation guidelines set by the procurement agency. The responses are in free text format (unstructured data) but the evaluation guidelines are quite structured and objective. Sysoft eRFP is capable of capturing data on criteria, vendor responses, evaluation scores, and evaluation comments from evaluators who never evaluated these RFPs [6]. Sysoft eRFP works with Oracle and SQL Server. The database was segmented and replicated into MS Access with more than two hundred Tables. Matlab computing environment was connected to the database through Matlab JDBC/ODBC Relational Database connector. The data was readily available to Matlab computing environment through Structured Query Language commands to the Database from Matlab code. This made the computing and testing environment very compact and productive in a laptop running Matlab with MS Access. EXPERIMENT We conducted an experiment to investigate whether textbased feature models can reliably learn and predict proposal evaluation attributes on 52 proposals from two domains of proposal sets (RFP1 and RFP2). RFP1 was “Procurements of Telephone Systems for Special Situations. ” RFP2 was “Procurement of Genome Analyzers for Hospital patients. ” Both RFPs belong to the same procurement agency. Specifically, we formulated the following two hypotheses: H.1 text-based feature models can reliably learn proposal evaluation attributes for a set of evaluations in the RFP1 domain H.2 text-based feature models that are learned on the RFP1 domain can make reliable predictions on the RFP2 domain. Proposal evaluation judgment refers to the objective evaluation opinion collected from the evaluators. Proposal evaluation attributes refer to the phenomenon that we are trying to learn and represent in a computable form. Collecting criteria based appeal judgments Data set of proposals RFP1 received 30 proposals from various vendors. RFP 2 received 22 proposals from some other vendors. The same 16 evaluators evaluated all the 52 proposals. The