Your responses and letters

Graham Williamson provides JAN readers with a harsh assessment of the use of inferential statistics with convenience samples. However, there are organizational as well as statistical issues in quantitative research and, inevitably, it is the organizational issues that very often must rule the day and, as a result, the statistical ideals end up having to be compromised. Any researcher who does not accept that will end up restricting their studies to inanimate objects, such as ball-bearings, and eschew research involving human subjects. In nursing research, that is not an option and so we must accept that statistical theory cannot always be applied to the full. Williamson raises some very important issues and introduces readers to some helpful techniques but I think he is overly harsh in considering that the majority of the quantitative studies he reviewed are open to the criticism of possibly being ‘biased’, ‘unrepresentative’ and having used inferential statistics inappropriately. In at least two cases – papers of which I am a co-author – I believe that Williamson’s judgement may actually be wrong and it is necessary, therefore, that I make a response. The first of my papers (Watson et al. 1999) was classified in the category of ‘convenience sample’ but, in fact, it was a cohort study. This may be considered either to be a kind of population study (i.e. one involving all of the subjects of interest in a group) or as an extreme form of cluster sampling. If Williamson accepts that this cohort is, in fact, a population then he advises randomly re-ordering the data in order to check on the inferences made about the data. It is not clear, however, what would be gained. What is the problem, for example, with carrying out a t-test on a whole population in order to find out if there is a difference between males and females, for instance? While such a test is an inferential test it is also an analysis of variance and one which compares the variance between groups with the variance within groups to see which is the bigger and the extent to which any difference between the groups is the product of chance. The same procedure could be applied to a convenience sample but any claims should be attenuated by warnings about the extent to which the results may be generalized. Satisfying theoretical considerations is laudable but, when research is done in practice, some compromises are always made (and usually explained). Nearly all randomized controlled trials (RCTs) would be rendered meaningless by the criteria Williamson applies because, while they may involve an element of randomization to arms of the study, a RCT rarely is based on samples drawn truly at random from the population. My guess is that a list of truly randomized RCTs – in the sense of being based on truly random samples – would be a very short list indeed. The point about ‘randomness’ can be pursued forever because as soon as a study group has been defined (for example, on the basis of a diagnosis) and all others are excluded in pursuit of a sample, the randomness of the subsequent sample is questionable – on theoretical grounds. However, on practical grounds, most researchers are simply satisfied to have reached their recruitment targets whereas Williamson implies an additional stage to an RCT where individuals are drawn as a random sample from the population and then the allocation is made to an arm of the study. Practical difficulties are not an excuse for sloppy sampling, of course, but there are a number of other issues to consider. How feasible is it, for example, to obtain for any purposes a random sample of substance abusers, sex workers or people suffering from sexually transmitted diseases? Compromises are usually made and some of these compromises involve cluster sampling and – not directly relevant to this example – stratified sampling. However, readers of Williamson’s paper need to note that these are specific designs – alternatives to, and not variants of, random sampling. They do not claim to be representative but the use of inferential statistics, which can be exploratory, is common practice although this important point is not acknowledged by Williamson. How is it legitimate in these circumstances to generate P-values from samples which are meant to (or are they?) infer properties about the population? Let me comment also on the other paper (Smith et al. 2002) which was classified, correctly, as using convenience sampling. The study reported in this paper was designed to compare a number of psychometric parameters related to inflammatory bowel disease (IBD) in IBD sufferers, non-IBD controls and a FORUM JAN