Some Problems Involved in Compiling Word Frequency Lists

IN ADDITION to the many frequency lists already published there is a need for more valid up-to-date lists, compiled by more objective procedures. The field of lexicometry is still in its infancy; its aims and particularly its procedures are badly in need of definition. Any research involving counting gives rise to a number of basic questions. The investigator must know exactly what he is counting, how he is going to select his sample, the source and size of the sample, and what procedure he will use for the count. The nature of his answer to these questions may well be influenced by another decision, namely what does he hope to accomplish with the count? If the population to be counted can be clearly stated, the actual counting is no problem. The only possible difficulty is one of time and accuracy if the population reaches excessive size. Recent advances in computer science have largely overcome this difficulty so far as the actual counting is concerned. For the counting of many items there still remains a hurdle in the method of sampling large and widespread populations. This, too however, is merely a problem of technique. A truly problematic question arises out of the fact that while some objects easily lend themselves for counting purposes, others can either not be counted at all because it is impossible to break them down into countable units, or they are nearly impossible to count because of their ambiguous nature. In such cases an arbitrary definition must be found with the result that not real items are being counted but arbitrary units of a predetermined abstraction. An inventory conducted by a department store is an example of a statistically simple count because of the clear-cut nature of the population. There is a great variety of items, but all of them can be easily identified. All that has to be predetermined is that some of the items are counted by the yard, others by the pair, by dozens, etc. The inventory must take place between the closing time of one day and the time the store opens the next morning. When the total stock is accounted for, it is stated that this constitutes the inventory as of a given date. In all stores merchandise comes and goes daily. But at a time when the doors are closed it is possible to conduct an accurate count. One of the criteria of an uncomplicated count is that the population is fixed, at least temporarily. Counting becomes far more complicated when such questions are raised as in the German folk song "Do you know how many stars there are?" or worse yet, "Do you know how many fish there are?" The star and the fish population are in a constant state of flux. And there is an additional problem. Even if they could hold still long enough to be counted, it would have to be decided precisely what was to be included for the count. City officials setting up a population census must make some arbitrary decisions as to who is to be included in the count. They might decide on just the adults, in which case the age at which a person becomes an adult must be specified. What complicates this type of count is the lack of stability in the population and the difficulties in defining the units to be counted. But in these examples the items are all tangible. Nobody would seriously consider a question such as: "How many thoughts were originated in the city of New York during the last twentyfour hours?" As soon as items are not cognizable to the senses, they escape the grasp of the counting investigator. None of the problems occurring in these examples correspond exactly to those faced by the compilers of word lists. They must deal with language, which as a system of sounds is perceivable to the senses. We can hear it; in this sense it is not as intangible as thoughts. But hearing language must not be equated with hearing words. A word is an arbitrary abstracion which is very hard to defend as a linguistic unit. There is general agreement on phonemes and morphs as units of utterances, but cornm-