**Introduction**

The emphasis on quality in the opening address at the Toronto ISO
meeting was very appropriate. It is a central issue in any organization
but its discussion here is particularly appropriate due to the ongoing
discussion of STEP development procedures. My personal concern prompted
me to write this short paper for the WG10 &12 workgroups.

**Quality**

Quality is measured by the degree of conformance to requirements. This
is a paraphrase of the description used in Toronto and represents my
understanding and assumption. If you fundamentally disagree, read no
further. This paper will define the terms used in this definition of
quality and explore the logical consequences of its application to STEP
development.

**Requirements**

Requirements for anything in the real world start off by being
subjective requirements as perceived by individuals, each with their own
view of the world as colored by their own knowledge and experience. In
order to be useful to a larger group, these views must be consolidated
into "the" requirements by writing and agreeing on a specification of
the requirements. Quality cannot be measured (indeed has no meaning) if
a specification (agreed details of what is required) does not exist. I
have tried to find an example of quality which is derived from a popular
idea about requirements, such as what makes a quality car, or a quality
apple and have been unable to do so. Different people put different
relative values on the various aspects of a product's characteristics
and a written specification is required to remove the subjectivity and
define the relative importance.

There is a big difference between requirement specifications (which
prescribe function) and product specifications (which prescribe the
actual form of the product). For a car, being able to carry four people
is a requirement. So is "with maximum safety" and "with maximum economy"
and "with maximum reliability" assuming conflicting requirements are
prioritized or weighted. Weightings will usually relate back to money.

Designing to requirements often leads to unconventional results. As an
example: for many years, Citroen claimed their design was (and I assume
still is) driven by safety and following the logic that the driver is
most often the cause of the accident and that driver fatigue (lack of
alertness) is an important factor, many design features (pneumatic
suspension, seating, cockpit ergonomics, wind noise and steering
geometry which retains control through a front-tire blowout) attempt to
satisfy this one requirement (to create a safer car) by addressing the
problem of avoiding the accident in the first place. This is all to make
the point that requirements must be "big picture" and address, in the
largest context possible, the ultimate purpose of the thing being
designed.

**Conformance**

In order to establish conformance to a requirement, the requirement must
be stated in such a way that it is possible to test (pass/fail) or
measure (the degree of) conformance of the resulting product to the
requirement.

In light of the above statement it becomes obvious that "with maximum
safety" and the other phrases used above are not measurable
requirements. We must use requirements like "to reduce statistical
deaths and serious injuries in France in model B car by 25% from the
same statistics for model A car." Comparing these two examples it
becomes obvious that the requirement for testability/measurability
really forces a clear definition of the goal, as well as assuring that
it will be testable. There is no danger of fuzzy objectives here.

The first paragraph refers to testing the resulting *product* and it is
important to point out that this refers to real-world tests of the
real-world product, not theoretical tests of a theoretical design.

**What constitutes a requirements specification?**

1\. Requirements specifications must be clear and simple.

> No more than one sentence, although explanatory examples are useful.
> If the overall global objective cannot be stated simply, the execution
> will be doomed to fail. Execution of even simple objectives become
> complex as they are inevitably compromised and twisted by the
> imposition of the real world.

2\. Requirements specifications must be agreed or validated by a
representative cross section of the intended users.

> All products have users and, if the requirements can be separated from
> their implementation, users (almost) always know what they want.

3\. Each requirement in the specification must be testable and to ensure
this, each must be accompanied by a full description of how it is to be
tested for conformance.

> For example "to write the most comprehensive AP possible" is not
> something which is testable. It cannot, by definition, be a
> requirement.
>
> An equivalent testable requirement might be to write an AP which
> successfully translates a minimum of X Gigabytes of real project data
> from each of Y different countries and from Z different application
> systems.
>
> The requirement then needs the clarification that this is not *any* X
> Gigabytes, but a successful translation (eventually) of the *first* X
> Gigabytes run through the translator to attempt to satisfy the
> requirement.
>
> Specifying exactly how a requirement is to be tested helps ensure that
>
> a\) an untestable requirement is not put forward and
>
> b\) that the requirement is effectively redefined (unambiguously) by
> the testing process.

4\. The testing must take place in the real world.

> This was assumed in the preceding example, but the reasons need to be
> explained.
>
> When a requirements specification is agreed, there is actually no
> guarantee that software which satisfies it will satisfy the real-world
> requirements. The requirements may have been inadequate. It is only
> the *successful use* of software which proves that it is *useful*.
>
> It is not valid to agree on a paper specification for software, then
> simply test that the software conforms to the specification.
>
> If software has been changed from the original specification but is
> unanimously accepted in the testing/evaluation phase then the software
> is proven acceptable (by the users) and the original specification
> remains an unproved proposal.

5\. Requirements must be clearly weighted as to relative importance,
wherever there is a conflict or overlap.

> It will typically not be possible to write a list of requirements
> which are each independent of one another, but this is not important
> if a weighting is available to use in evaluating decisions and to
> evaluate the relative success of a number of parallel objectives.

None of the above are trivial but they cannot be ignored without risking
the complete failure of a project to meet the expectations of (at least
some of) its backers.

**Requirements and Resources**

If the reality of a situation is that there is a fixed amount of
money/time and the primary objective is not to over-run, but rather to
do the most possible with the available resources, this needs to be
clearly stated, as the project will be executed differently. It will be
broken into small parts and each specified, implemented and tested in
sequence, so that at the end of the project there will be proven (final)
deliverables, even though the scope was less than planned for. This will
then provide the necessary data to accurately estimate what additional
resources are required.

**What does this imply for STEP/ISO?**

My interpretation is as follows:

1\. No work must be undertaken in STEP without a clear bounded
definition (or specification) of the required results. I would expect
that the required results would typically revolve around satisfying
users' requirements.

2\. These defined required results must be measurable with real-world
measurements and the means to make the measurements must be defined. I
would expect that for ISO standards, these measurements would be related
to actual use by the relevant user communities.

3\. The means to measure the result must be defined at the same time as
the specification (to ensure the measurement is possible and practical).
This directly impacts ISO procedures. The maximum measured deviation
allowed from conformance for each part of the process must be defined.
(This implies CD-\>DIS-\>IS approvals are not committee decisions, but
result from routine compilation and review of statistics).

5\. Where a result does not meet quality standards, the measuring
process must be repeated after an attempted correction. Again, this
should not be a committee decision.

6\. The testing process must positively test for compliance, e.g.
require X terabytes of information to be processed, not assume all is ok
because no industrial comments were received. Note that this is almost
the exact opposite of the current ISO process, which relies not on
positive testing but simply on a lack of objections to a proposed (and
in STEP, untested) standard.

**Testability of APs**

It will have been evident from the examples above that I favor testing
an AP by putting real-world data through it. This implies that the
requirement for the AP's functionality is defined in terms of the type
of data which must be put through it. Bearing in mind that the purpose
of an AP is communication between application programs, it does not seem
unreasonable to define transfers between specific commercial programs as
part of the test procedures. These might be defined as "the programs
required to make up x% of the market at the time testing begins" or a
similar definition. The percentage used would depend on how fragmented
the market was.

**Testability of a Specific AP**

In order to be testable, the scope of a particular AP must be well
defined.

It is the data associated with a particular product or set of products:
it must be or the project is unbounded

It is also the data associated with the use in specific usage scenarios:
it must be or the project is unbounded.

One or more of these usage scenarios may be the exchange of data with
other APs. This is discussed in more detail in the next section.

If part way through the project it is realized that there are additional
products or usage scenarios which must be considered, they must either
remain excluded from that version or the AP scope formally increased,
with whatever budget implications that has and whatever impact on other
dependent projects.

**Inter-operability of APs**

There has been much futile discussion on "inter-operability". The
discussion of AP inter-operability is sensible *only* in the context of
data transfer from one AP to another defined and stable data format,
typically another AP. Any other interpretation will lead to a situation
which cannot be tested.

We all look forward to a world where data can be read from anywhere by
anyone, but the journey to Nirvana must be made in small steps.

A top-down approach using generalized global data structures and
standard constructs is quite obviously necessary to impose order on the
chaos, but whatever is proposed for these high level constructs must be
able to be tested in the real world. This testing is most naturally
accomplished by the use of the constructs in inter-AP transfers, but it
must be clearly understood that if high level constructs are imposed on
inter-AP transfer, it is to test the constructs and not to test an
individual transfer between two specific APs, which in all likelihood
could achieve the same level of conformance with the requirements using
different (simpler) constructs.

The most logical approach appears (to me) to be

1\) to allow individual APs to use whatever data constructs are
appropriate for their well defined and bounded purpose. This produces
testable useful products.

2\) to sponsor individual projects to facilitate data transfer between
two specific APs. These would be used to either test or invent the more
general constructs which would be necessary to describe the data within
the context of both APs simultaneously. This produces testable useful
products.

3\) to sponsor an on-going project to derive global data structures
which are compatible with (i.e. can be mapped to and from) the inter-AP
data structures (which have been proved to work). This is open-ended and
the points at which the work would be issued for use is arbitrary.
However, the requirement that it must be mapable to the existing
inter-AP code means that it would represent an implementable solution.

The important point here is that the global constructs are simply not
necessary for the satisfactory transfer of data within the context of an
individual AP. They are only necessary to get data from one context to
another. I believe that this problem is sufficiently large and complex
that it cannot be solved by people sitting in a room and dreaming up
proposed solutions. It will only be fully solved by a series of projects
(inter-AP data transfers) which allow the constructs to be developed and
tested while minimizing the impact on the development of the APs
themselves. I believe this to be a perfectly practical approach, using
EXPRESS-M (eventually EXPRESS-X) mapping.

**STEP and ISO**

I have made the point with numerous people in the past that STEP is
fundamentally different to the normal standards which pass through ISO.
It is the development of new technology and absolutely needs positive
testing. Typical ISO standards are usually some choice between various
known technologies which have already been evaluated in the marketplace.
I see a clear need for two different procedures for these two distinct
situations.

**Last Word**

I recently heard the duties of a Chief Engineer described as "to create
the standards and ensure they are conformed to". I like that definition.
I see the role of the 10303 main committee (with whatever advisers it
chooses to listen to) as that of Chief Engineer with quality as its
primary objective. I understand the magnitude of the changes which are
implied in this analysis, but I believe the problems with the current
methodology are equally enormous.

Robert J Fisher Intergraph Corporation rjfisher@ingr.com. 20 December
1996