Proposed Modification to COSYSMO Estimating Relationship Gan Wang BAE Systems

This paper proposes a modification to the Academic COSYSMO estimating relationship to remedy a critical limitation in its current implementation of the cost drivers. The effort multipliers defined for these drivers have an overdramatic impact on the nominal effort, which unrealistically amplifies or compresses the effort estimate. This problem severely limits its practical applications. The newly proposed parametric relationship is inspired by the COCOMO II modeling approach and based on the considerations of the life cycle impact of the cost drivers. Two additional cost drivers are also introduced. The feasibility of the new model definition is examined with a boundary analysis and validated by the analysis of historical data. In this paper, we present (1) an analysis of the problem with the current Academic COSYSMO model definition; (2) proposed addition of two new cost drivers to the model; (3) an analysis of life cycle impact of the cost drivers and an organization of the drivers based on the impact; (4) the modified COSYSMO parametric relationship; (5) validation of the modified relationship through an analysis of historical data; and (6) conclusion and suggestion of future work. This work is based on the practical implementation of COSYSMO at BAE Systems. Introduction – the Problem The Constructive Systems Engineering Cost Estimating Model or COSYSMO, developed at the University of Southern California with the support of a consortium of industry, academic and government organizations, is a parametric model for estimating the systems engineering and integration effort or labor in developing a system. The model defines a new way of measuring the size of a system, through four attributes – system requirements, interfaces, algorithms and operational scenarios and, in doing that, presents a new perspective of characterizing system holistically. Since its inception a few years ago, the model has rapidly gained the attention of industry. Several efforts have been made at several companies and organizations to apply it to real-world applications [Valerdi, et al, 2007]. Early experience of piloting and deploying the model has shown that, at version 1.0, Academic COSYSMO has several shortcomings, some of which may severely hinder its ability to be accepted for practical use. Attempts have been made to improve the model definition so that it is better suited for its task in characterizing and estimating real-world system development projects or endeavors [Wang, et al, 2008]. This paper addresses one of these issues related to the use of cost drivers in its parametric relationship. The current COSYSMO model defines fourteen cost drivers to describe the specific product and development environment of the system [Valerdi, 2005]. They are also known as effort multipliers because of the multiplicative effect of each driver on the estimated effort based on size of the system. Each cost driver has a corresponding rating scale that describes the different levels of impact that can affect systems engineering effort. The typical cost driver has a rating scale that follows five levels, for example: Very Low, Low, Nominal, High, and Very High. These cost drivers are listed as below. Application Factors: 1. Requirements understanding 2. Architecture understanding 3. Level of service requirements 4. Migration complexity 5. Technology Risk 6. Documentation Match to Life Cycle Needs 7. Number and Diversity of Installations/Platforms 8. Number of Recursive Levels in the Design Team Factors: 9. Stakeholder team cohesion 10. Personnel/team capability 11. Personnel experience/continuity 12. Process capability 13. Multisite coordination 14. Tool support The overall impact to the estimate effort is defined by the model as a geometric product of all fourteen drivers, as shown in its parametric form: ∏ ∑ = ⋅ ⎟⎟ ⎠ ⎞ ⎜⎜ ⎝ ⎛ Φ + Φ + Φ ⋅ = 14 1 , , , , , , ) ( j j E k k d k d k n k n k e k e NS EM w w w A PM Where, EM is the effort multiplier for the jth cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort determined by the system size. Practical implementation of the model reveals several serous limitations or pitfalls. First, the current multipliers defined for each of the cost driver ratings is too large and over-amplifies the magnitude of the nominal effort. For example, if one would change the rating of a single cost driver, say Requirements Understanding, by one step from Nominal to High, which is commonly encountered in real projects, there is an effort savings of 23%. In other words, if the nominal estimate effort is 10,000 person-hours, the adjustment of this single parameter by one step reduces it to 7,700 person-hours. Any experienced systems engineer or project manager will readily tell you that this is unrealistic. The second problem lies with the unconscious but inherit assumption from a practical standpoint that all fourteen cost drivers are independent of each other, such that double counting of the effect may be possible. While the effort multiplier of a single driver is independently rated from all other 13 drivers, its end effect on the estimated effort is scaled by all the other multipliers, regardless of the nature of the drivers. Through the multiplication, each driver has a compounding effect on the nominal effort. In reality, many of these drivers are practically different perspectives of the same issue or aspect of the project. Treating them independently would over-exaggerate the intended effort modification. The result is an overdramatic change to the estimated effort. Let’s use the same example as above, assuming that the Requirements Understanding is high. But, at the same time, you also set a second driver, say Personnel Continuity, also to High, then the collective effect from the two setting is a 37% discount. It now brings a nominal effort of 10,000 person-hours down to 6,300 person-hours, which is an obvious exaggeration of the effect. In practical estimates, an assessment like this is not uncommon. However, the consequence can be devastating. Let’s look at the theoretical bounds of these cost drivers. The lower bound (LB) is derived from all of the 14 drivers assume the maximum discount values (i.e., maximum economic benefit) and upper bound (UB) is when the 14 drivers assume the maximum multiplication values (i.e., maximum economic penalty). The effort multiplier ratio (EMR) is the ratio of UB over LB and provides a measure of dynamic range of the total composite multiplier effect. As you can see from Table 1, the LB gives close to 99% discount on the nominal effort, whereas UB amplifies it by over 50,000 times. Putting it into perspective, say that once again you have a nominal systems effort of 10,000 person-hours, applying the maximum discount (everything about the project was going in your favor) would, in theory, result in a little over 100 person-hours for the project. When applying the maximum penalty (everything about the project was going against you to maximum extent), then it would theoretically require over five million person-hours for the same project. We recognize that these bounds are theoretical extremes. However, the dynamic range they induce (represented by EMR) is not meaningful, at best, to real world problems. It can significantly distort the estimate and mislead the decision making. Table 1. Boundary analysis for the cost drivers in the current geometric product relationship Cost Driver (CEM) Relationship Lower Bound (Max Benefit) Upper Bound (Max Penalty) EMR Academic COSYSMO version 1.0 0.012 510.714 44,095.294 % Deviation -98.84% 50971.35% The third problem is that the model assumes the cost driver impact over the system development life cycle is uniformly constant. In other words, a cost driver such as requirement understanding has the same level of benefit or penalty any time during the project life cycle, from the inception to completion. This is also unrealistic because a low requirement understanding in general has bigger impact later in the life cycle due to additional rework resulting from the lack of the understanding of the requirements. Another underling reason for this effect, we speculate, is the bias in historical data, which may be caused by stakeholders’ tendency of over-rating their capabilities. As an example, people are reluctant to rate their Personnel/Team Capability low or, sometimes, overstate their Process Capability (there is a subtle difference between organization’s CMMI rating and the actual process being applied to a particular project). The final issue in the concern is that the model has failed to account for some of the key aspects of today’s system development environment. As an example, schedule compression of projects is a prevalent phenomenon that it has almost become a daily life in the industry. On the other hand, as a best practice, project managers are required to proactively manage risks and exploit opportunity in project executions. The risk and opportunity management has increasingly become a major focus of projects and known to have significant influences over their outcomes. The bottom line is that these factors could made significant differences in systems engineering cost and effort and, thus, should be considered in the estimating model. These intrinsic problems in the underlying relationship of the model can create a serious distortion to the estimate. It can severely limit the usability and credibility of the model. It has already presented as a serious roadblock for early adoption of the model in the industry. A remedy is imperative. To be certain, it is clear that a model at such scale as COSYSMO will take time to mature. Problems like this are expected in its early life cycle and represent one of many opportunities for improvement as we are discovering quickly from the early usage experiences. The following sections of this paper propose a modification of the existing model defin