nagdmc impute dist

NAG DMC

Data Imputation: nagdmc impute dist
Purposenagdmc impute dist imputes data values based on distances to donor data records, for different
distance metrics.
nagdmc impute dist returns an array containing the indexes of imputed values and donor records(see ‘Explanatory Code’). The memory used by this array should be returned to the operating
system by the user as indicated in the Essential Introduction.

Declaration#include <nagdmc.h>long *nagdmc impute dist(long rec1, long nvar, long nrec, long dblk, double data[],long ncat[], long maxcat, long cat[], double wts[],
double mval, int stype[], long randsel, long ndonors,
long matchall, int reuse, double minwt, double R[],int dtype[], double udist[], long *nrepl, long *nfail,long *noimp, int *info);

Parameters1:

rec1 – long
On entry: the index in the data of the first data record used in the analysis.
Constraint: rec1 ≥ 0.

Input

2:

nvar – long
On entry: the number of variables in the data.
Constraint: nvar ≥ 1.

Input3:

nrec – long
On entry: the number of consecutive records, beginning at rec1, used in the analysis.
Constraint: nrec > 1.

Input

4:

dblk – long
On entry: the total number of records in the data block.
Constraint: dblk ≥ rec1 + nrec.

Input

5:

data[dblk ∗ nvar] – double
Input/Output
On entry: data values for the jth variable (for j = 0, 1, . . . , nvar − 1) are stored in data[i ∗ nvar + j],
for i = 0, 1, . . . , dblk − 1.
On exit: missing values in data are replaced by their estimates.

6:

ncat[nvar] – long
Input
On entry: ncat[i] contains the number of categories on the ith variable, for i = 0, 1, . . . , nvar − 1.
If the ith variable is continuous, ncat[i] must be set equal to zero. If all variables in the analysis
are continuous, ncat must be 0.
Constraints: if not 0, ncat[i] ≥ 0, for i = 0, 1, . . . , nvar − 1.

7:maxcat – long
Input
On entry: the maximum number of categories on any categorical variable. If all variables arecontinuous, maxcat must be set equal to zero.
Constraints: if ncat is 0, maxcat = 0; otherwise maxcat ≥ ncat[i], for i = 0, 1, . . . , nvar − 1.

8:cat[nvar*maxcat] – longInput
On entry: the categories for the categorical variables. The categories for the ith variable are storedin cat[i ∗ maxcat + j], for j = 0, 1, . . . , ncat[i] − 1. If all variables in the analysis are continuous, catmust be 0.
Constraint: if ncat is 0, cat must be 0.

nagdmc impute dist.1

NAG DMCnagdmc impute dist9:

wts[nvar] – double
Input
On entry: if wts is 0, the weight on each variable in the analysis is set to 1.0; otherwise wts[i] is
the weight on the ith variable used to calculate the distance between donor records and records
containing missing values, for i = 0, 1, . . . , nvar − 1.
Constraints: if wts 6= 0, wts[i] ≥ 0.0, for i = 0, 1, . . . , nvar − 1; and the sum of elements in wts must
be greater than 0.0.

10:mval – double
Input
On entry: all values in data equal within machine precision to mval are considered missing fromthe analysis.
Suggested value: a value outside the interval [a, b], where a and b are the minimum and maximum
value in your data, respectively.

11:

stype[3] – long
Input
On entry: controls the search strategy:
stype[0] = −1, only donors with the same value of categorical variable stype[1] are searched, andstype[2] is not referenced;
stype[0] = 0, all donors are searched, and stype[1] and stype[2] are not referenced;
stype[0] = 1, and only donors between record numbers stype[1] and stype[2] are searched.
Constraints: stype[0] ∈ {−1, 0, 1}; if stype[0] = −1, 0 ≤ stype[1] < nvar, and ncat[stype[1]] > 0; if
stype[0] = 1, rec1 ≤ stype[1] < stype[2] < nrec.

12:

randsel – int
On entry: indicates what random selection from donors is to be performed:
if randsel = −1, the seed for random selection is taken from the system clock;
if randsel = 0, random selection is not used and the first donor is used;
if randsel > 0, randsel is used as the repeatable seed to start the random selection.
Constraint: randsel ≥ −1.

13:

ndonors – long
Input
On entry: if randsel 6= 0, the donor record will be selected at random from up to the first ndonors
best donors encountered; otherwise ndonors is not referenced.
Constraint: ndonors ≥ 1.

14:

matchall – long
Input
On entry: if matchall = −1, all missing values in a record are imputed; otherwise only values on
the matchall variable are imputed.
Constraint: −1 ≤ matchall < nvar.

15:

reuse – int
Input
On entry: if reuse = 1, the same donor record may be re-used to impute values for different records
without penalty; otherwise reuse must equal zero and subsequent distances to donor records are
penalised by adding nvar or, if wts 6= 0, the sum of elements in wts.
Constraint: reuse ∈ {0, 1}.

16:

minwt – double
Input
On entry: values equal to mval in a target record are imputed if the sum of weights on variables
with values not equal to mval in the donor record is greater than or equal to minwt; othwerwise
data values are not imputed and the value of nfail is increased by one.
Constraint: minwt ≥ 0.0.

17:R[nvar] – double
Input
On entry: R[i] contains the scaling factor for continuous variable i, for i = 0, 1, . . . , nvar − 1.
For Euclidean and Manhattan distances R[i] is a multiplicative standardisation. For thresholddistances, it is the threshold value and for regression distances it is the regression coefficient. If R
is 0, the scaling factor for continuous variable i is taken to be 1.0. If all variables are cateogrical,
R is not referenced and should be 0.
Constraints: if R is not 0, R[i] > 0.0, for a continuous variable i, for i = 0, 1, . . . , nvar − 1.

Input

nagdmc impute dist.2

nagdmc impute dist

NAG DMC18:

dtype[nvar] – char

Input

On entry: dtype[i] signifies the distance measure used for the ith variable, for i = 0, 1, . . . , nvar − 1.
The available measures depend on the type of variable: categorical or continuous. The parameter
ncat is used to determine categorical and continuous variables.
For continuous variables the choices for dtype[i] are:dtype[i] = 0, for Euclidean distance;
dtype[i] = 1, for Manhattan distance;
dtype[i] = 2, for regression distance;
dtype[i] = 3, for threshold distance.
For categorical variables the choices for dtype[i] are:dtype[i] = 0, for simple matching;dtype[i] = 1, for rank difference distances;
dtype[i] = 2, for distances based on user-supplied metric.
Constraints: if ncat = 0 or ncat[i] = 0, dtype[i] ∈ {0, 1, 2, 3}, for i = 0, 1, . . . , nvar−1; if ncat[i] > 0,dtype[i] ∈ {0, 1, 2}, for i = 0, 1, . . . , nvar − 1.
19:

udist[nvar∗maxcat∗maxcat] – double

Input

On entry: the distance tables for user-supplied distances. The nvar ncat[i] by ncat[i] tables are
stored in nvar maxcat ∗ maxcat blocks. If the user-supplied distance option for categorical variables
is not selected for any variable in the analysis, udist is not referenced.
20:

nrepl – long *OutputOn exit: the number of mval values replaced by the function. The value of this parameter isdetermines the length of the array returned by the function, see the ‘Explanatory Code’.
21:

nfail – long *OutputOn exit: the number of data records requiring imputation but for which the sum of variable weights
for non-missing values in the donor record is less than minwt.
22:

noimp – long *OutputOn exit: the number of data records requiring imputation for which no donor records are available.
23:

info – int *OutputOn exit: info gives information on the success of the function call:
0: the function successfully completed its task.
i; i = 1, 2, 3, 4, 6, 7, 8, 9, 11, 12, . . . , 18: the specification of the ith formal parameter wasincorrect.
20: no missing values were found in the data; check your definition of mval.
50: if stype[0] = −1, a category for the stype[1] variable is incorrect.
55: a value for a categorical variable is incorrectly specified. Check values in the ncat, cat
and data arrays.
99: the function failed to allocate enough memory.

Notation
nrec
the number of data records, n.
datathe data set X.
nxvar determines the number of variables, m.
mval the value of missing data values, z.
wts
the weights wj , for j = 1, 2, . . . , m.
ndonorsif random selection is used, the value t.
minwt the minimum value for sum of weights on variables, v.
Rthe scaling factors Rj , for j = 1, 2, . . . , m.

nagdmc impute dist.3

nagdmc impute dist

NAG DMC

Description
Let X be a set of n data records xi , for i = 1, 2, . . . , n. The jth variable of the ith record xi
takes either the value xij or a dummy value, z, representing values in X missing at random, forj = 1, 2, . . . , m.
Given a target record xk in X containing one or more variables with value z, a distance s iscomputed to each of the data records xi , for i = 1, 2, . . . , n and i 6= k. A low value of s indicatesthat the record xi is a close match to the target record xk . Furthermore, let dj be the difference
between the value of the jth variable for the ith data record and the target record: xij − xkj .
For target record k and the ith data record, each of the p ≤ m continuous variables in X contributes
to s by evaluating:y = wj f (Rj , dj ),where:(a) wj ∈ IR is the weight given to the jth variable;(b) Rj ∈ IR is an optional scaling factor;(c) f (·) ∈ IR → IR is continuous distance function that takes one of the forms:(i) Euclidean distance, f (Rj , dj ) = (Rj dj )2 ;(ii) Manhattan distance, f (Rj , dj ) = Rj |dj |;(iii) regression distance, f (Rj , dj ) = Rj dj ;(iv) threshold distance, if dj > Rj , f (Rj , dj ) = 1; otherwise f (Rj , dj ) = 0.
An appropriate scaling factor Rj for continuous variables is the reciprocal of the range of the jthvariable. Distances defined on continuous variables contribute to the value of s by adding:(a) the square root of the sum of squared values of y over all continuous variables that use
the Euclidean distance;(b) the sum of absolute values of y over all continuous variables that use the Manhattan
distance;(c) the absolute value of the sum of y over all continuous variables that use the regression
distance;(d) the sum of y over all continuous variables that use the threshold distance.
For target record k and the ith data record, each of the q = m − p categorical variables in X adds
to s a value:qX

wj g(dj ),j=1

where the categorical distance function g(·) takes one of the forms:(a) simple matching, if dj = 0, g(dj ) = 1; otherwise g(dj ) = 0;(b) scaled rank difference for a variable with cj categories, g(dj ) = |dj |/(cj − 1);(c) user-supplied distance from tables for the jth variable between category values xij andxkj .
In the case of a user-supplied distance for a variable with r categories, an r by r matrix (usuallysymmetric) containing the distances for each possible pair of values has to be supplied.
The donor record is the data record used to replace missing values in the target record. Donor
records can be selected at random or by using the data record with the lowest distance score. If
random selection is not used, missing values on the jth variable in the target record are replacedby the jth values in the data record with the lowest score, s; otherwise a donor record is chosen atrandom from the t data records with the lowest scores.
Optionally, penalised distances to donor records can be used, in which case the value:
h∗m
X

wj

j=1

nagdmc impute dist.4

nagdmc impute dist

NAG DMCis added to distances from a target record to a record which has been previously chosen h times asa donor record.
The t closest records to xk in X can be searched in one of three ways:(a) search all records xi in X, i 6= k, for i = 1, 2, . . . , n;(b) search a subset of X defined by record numbers in the range [a, b];(c) search all records with a particular value u of a categorical variable l.
The variable weights can be used to select and give importance to variables when matching records
with missing values to donor records. If the weight of the jth variable, wj , is set to zero, nomatching for imputation is carried out for that variable.
Missing values z will not be imputed if the sum of weights on variables without missing values forthe donor record is below a minimum value, v.

References and Further Reading
None.

Explanatory CodeThe following C function prints the index in the data and the imputed value of each of the nrepl
replaced values after a successful call to nagdmc impute dist returning indexes.
#include <stdio.h>void imputed_values(long nvar, double data[], long nrepl, long indexes[]) {long i, j, k, l;#define MISSING_ROW(I) indexes[I]#define MISSING_COL(I) indexes[I+nrepl]#define DONOR_REC(I) indexes[I+2*nrepl]#define DATA(I,J) data[(I)*nvar+J]
printf("\n\tRow \tCol \tValue \tDonor Record\n");for (i=0; i<nrepl; ++i) {j = MISSING_ROW(i);k = MISSING_COL(i);l = DONOR_REC(i);printf("\t%-4li\t%-4li\t%-6.3f\t%-li\n",j,k,DATA(j,k),l);}}See Also
nagdmc free impute returns memory allocated by nagdmc impute dist to the operating system.
impute dist ex.cthe example calling program.

nagdmc impute dist.5