Information Security Continuous Monitoring (ISCM) for Federal Information Systems and Organizations
CHAPTER THREE > Continuous Monitoring > 3.1.4

3.1.4

DEFINE SAMPLE POPULATIONS
Organizations may find that collecting data from every object of every system within an
organization may be impractical or cost-prohibitive. Sampling is a methodology employable with
both manual and automated monitoring that may make ISCM more cost-effective. A risk with
sampling is that the sample population may fail to capture the variations in assessment outcomes
that would be obtained from an assessment of the full population. This could result in an
inaccurate view of security control effectiveness and organizational security status.
NIST SP 800-53A, as amended, describes how to achieve satisfactory coverage when
determining sample populations for the three named assessment methods: examine, interview,
and test. The guidelines in NIST SP 800-53A for basic, focused, and comprehensive testing30
NIST 800-53A provides guidelines to help address the general issue of sampling and particularly
that of coverage. In selecting a sample population, the coverage attribute is satisfied through
consideration of three criteria:
addresses the need for a "representative sample of assessment objects" or a "sufficiently large
sample of assessment objects." Statistical tools can be used to help quantify sample size.
•
Types of objects - ensure sufficient diversity of types of assessment objects;
•
Number of each type - chose "enough" objects of each type to provide confidence that
assessment of additional objects will result in consistent findings; and
•
Specific objects per type assessed - given all of the objects of relevance throughout the
organization that could be assessed, include "enough" objects per type in the sample
population to sufficiently account for the known or anticipated variance in assessment
outcomes.
29 Prior to initial authorization, the system is not included in the organization's continuous monitoring program. This
reference to RMF 4 is relevant after the system becomes operational, and is passing through Step 4 in support of
ongoing authorization.
30 See NIST SP 800-53A, as amended, Appendix D.
Sample measurements are summarized into a statistic (e.g., sample mean) and the observed value
compared with the allowable value as represented by organizational risk tolerance. Statistics
calculated using sampling can become less reliable predictors of the full population if the
population is not randomly selected and if the sample size (i.e., objects to be tested) is small.31 As
described in the NIST Engineering Statistics Handbook, when deciding how many objects to
include in sample populations, the following are considered:32
•
Desired information (what question will the measurements help answer);
•
Cost and practicality of making the assessment;
•
Information already known about the objects, organization, or operating environments;
•
Anticipated variability across the total population; and
•
Desired confidence in resulting statistics and conclusions drawn about the total population.
Ways to achieve "increased" or "further increased grounds for confidence that a control is
implemented correctly and operating as intended" across the entire organization include asking
more targeted questions, increasing the types of objects assessed, and increasing the number of
each type of object assessed.
Organizations may also target specific objects for assessment in addition to the random sample,
using the above criteria. However, sampling methods other than random sampling are used with
care to avoid introducing bias. Automated data collection and analysis can reduce the need for
sampling.
Primary Roles: Information System Owner, Common Control Provider, Information System
Security Officer, Security Control Assessor
Supporting Roles: Risk Executive (Function), Authorizing Official, Chief Information Officer,
Senior Information Security Officer
Expected Input: Organizational- and system-level policy and procedures on ISCM strategy,
metrics, and the Security Assessment Plan updated with assessment and monitoring frequencies
Expected Output: Security Assessment Plan documentation on acceptable sample sizes,
security-related information
31 The Central Limit Theorem is a key theorem that allows one to assume that a statistic (e.g., mean) calculated from
a random sample has a normal distribution (i.e., bell curve) regardless of the underlying distribution from which
individual samples are being taken. For small sample sizes (roughly less than 30), the normal distribution
assumption tends to be good only if the underlying distribution from which random samples are being taken is
close to normal.
32 For detailed information on selecting sample sizes, see
http://www.itl.nist.gov/div898/handbook/ppc/section3/ppc333.htm.