-52-CHAPTER 6DISCUSSION
6.1

Design Problems

There are several problems in interpreting our data.

The major difficultylies in the questionnaire design, and has affected the results concerning
evaluation of the text analysis.
for the abstract evaluation.

It is especially evident in the results

From the responses to that questionnaire(See Appendix B) it is clear that we should have used a rating scalerather than fyes/no1 binary choice for the general evaluative question.
A number of respondents expressed difficulty in answering that question,
and answers to that question were not always consistent with answers
to the other questions.

Although this was not made evident in the problem

statement evaluation, that may be only because the respondents to theabstract evaluation were professionally familiar with the issues, andtherefore more disposed to comment and criticize.

So we might assume

that for both groups, the judgement of general adequacy of the analysis
has been constrained by the response choice.

This also seems to havebeen a difficulty with question 4 of the abstract evaluation, but inthat case is perhaps not quite so critical.
Another problem is that it is difficult to correlate responses to thequestions about strength of association with those asking if any concepts
were omitted.

It is possible that some omitted concepts might have beenthose which were also too weakly related.

Although we asked for examplesof each case, there is no certain way to determine whether such arelationship holds.

And finally, we did not stress, in our explanation,that the associations in the representation formats were only a small
number of the total associations derived by the analysis, representing
only the most important concepts and relationships.

This may havecoloured responses to questions concerning weak associations and omittedconcepts, and also seems to have influenced response to question 1,negatively.
Another general difficulty ties in the representations of problem statements,
where we arbitrarily chose the top 40 associates for inclusion in theassociation map representation.

This number was chosen for ease of construc-tion and interpretation of the representation, but unfortunately resulted
in arbitrarily splitting groups of associates allstrength.

with the same association

We realized this difficulty by the time the abstracts wereevaluated, and so used a flexible cut-off level with them in order toinclude all associates at the lowest association strength, aiming for

-53-about 40 associates.

This problem may have affected response to theevaluation of problem statement representations, and cut-off level
certainly needs further investigation (but see Section 6.4).
6.2

Text characteristics

The general text characteristics with which we were concerned, i.e. theratio of all tokens to significant tokens, and the type-token ratio,
followed expected patterns.

Oral problem statements had the highest

token-token ratio, abstracts the lowest and written problem statements
were intermediate, supporting the general finding (e.g. Mann 1949; Moses
1959; Portnoy, 1973) that oral narrative has more non-content words thanwritten narrative.

From our results, we might conclude that the written

problem statements we examined were not so fully thought out as theabstracts.

There is, however, one striking difference between thethree categories of text; that is, the correlation between number oftypes and association strength.

In this case both oral problem statements

and abstracts showed significant correlations, while the written problem
statements showed none.

This may perhaps be due to the style of thewritten statements, which appeared to be strongly affected by previous
experience with information systems.

The users tended to put their

queries together in a rather boolean form, which led to high association
strengths irrespective of number of types.

Whether this difference

would be significant in system design is not, however, clear, since thetext analyses of written and oral problem statements were quite similar.
6.3

Evaluation of representations

Before discussing the evaluation results themselves, we would like tocomment briefly upon the response rates to our surveys, which seem
exceptionally high.

We decided from the onset of the project to pay

our respondents for taking part in the evaluation

and we feel that thishas had a great influence upon the response rate.

For the interviewees,

we included a cheque for only £2^00, having underestimated the amountof work the evaluation would require.

We received back eight of thesecheques; from this we conclude that our having made even only a gesture
toward repaying the respondents for their effort was sufficient to affectresponse rate.
evaluation.

We increased the gesture to £5.00 for the abstract

Although there are probably other reasons for our very

high response rate (90%), such as professional involvement in the subjectof the survey, we cannot help but think that recognition of effort in
the form of payment was a substantial factor in the response.

There

were several comments to this effect from the respondents, only one of

-54-

whom returned a cheque.

We sent no reminders in either survey, so thissort of payment seems to us a relatively inexpensive way to achieve good
response rates (at least in small-scale surveys).
The results of the evaluations, despite the difficulties mentioned inSection 6.1, seem to us encouraging.

In both groups of respondents,

30% or less thought that the representation was actually bad, and insome of those cases further comments contradicted that opinion.

Fromthese data, we conclude that the basic analytic technique, word coroccurrence analysis, is in principle reasonable for our purposes.

The resultsof the survey of authors also seem to indicate that abstracts arereasonable document surrogates in our context, and that the length of theabstract is not a significant parameter, at least within the limits inour sample. And the results of the survey of interviewees indicates that
our interview technique, combined with the text analysis, is a reasonablemeans for collecting problem statements that will be useful in an ASK
based IR system.

Despite these positive results of the survey, thedata concerning strength of association and omitted concepts indicatethat our text analysis techniques need modification.
6.4

Text analysis

The results of the evaluations of both problem statement and abstract
representations indicate that over-strong associations are a problem, butthat the major difficulty in text analysis lies in associations at too low
a level.

These may reflect a misunderstanding of the representation, asmentioned in Section 6.1, but they seem too consistent and too positive to
be attributed only to this cause.

Most of the examples of too weakly

associated concepts involve words that occurred only once or twice inthe text, sometimes in conjunction with more frequent words, but often not.
At times these words appear (in the abstracts) only in the document title,
which we treated as a separate paragraph.

Possible means of adjusting ouranalysis are to include the title as part of every paragraph, and to takeaccount of collection frequency (when we have a large enough collection)
to weight some of the content words.

We also need to adjust

cut-off

techniques in some way, to include more than just the top 40 associates,
but fewer than all of the associates in the text representation.

We havedone some preliminary investigations on this problem, and it appears thatusing all associates until each type isrepresented is a reasonable

and useful cut-off rule.
The data from the text analysis as it now stands are rather complex,
and probably need to be reduced before they can be useful for retrieval

-55purposes (as, for instance, suggested in Section 5.2).

We have done

this to some extent in the association map representations by choosing
three levels of association strength, but the techniques for choosing
levels need to be refined.
The classification of problem statements based on the text analysis appears
to be reasonably successful, given its aims.

It is interesting to notethat there are few queries in our sample which can be described as welldefined.

The one in class AO stands out in this respect, from both thesubjective and computational points of view (it has a connectivity scoreof zero).

We believe that conventional best-match retrieval systems

operate under the assumptions that queries are well-defined.

Our resultsappear to belie that assumption, and it is encouraging that we cancome to this conclusion by computational and algorithmic means. Neverthelessthe classification should be modified in order to take more account oflevels of association.