From: carm@cs.umd.edu (Richard Chimera)
Newsgroups: comp.human-factors
Subject: Re: Evaluating Layout?
Date: 14 Sep 92 18:37:28 GMT
Organization: U of Maryland, Dept. of Computer Science, Coll. Pk., MD 20742

In article <Ieh_LLS00VQkEA71QA@andrew.cmu.edu> tg0r+@andrew.cmu.edu (Terilyn Gillespie) writes:
>
>But how do I measure how well the format "presents"
>information, moves the reader's focus, etc.

and earlier writes:
>On the one hand, I sort of think it might be best to have two proposed
>designs and compare those

I think you've answered the question somewhat yourself.  Good, hard science
mandates that you compare one thing to another, not one thing to nothing --
"one interface works for me/the group we studied."  If you can get alot of
participants or some participants for alot of experiment time, then perhaps 
you can compare three or four designs in one experiment to reduce the overhead
of designing experiments, getting participants, statistically analyzing 
results, and proposing conclusions.

Create tasks in an experiment that isolate one aspect of items you want
to measure: moving the reader's focus tasks ("Read the article titled 
'Foo Gets New Bar.'  Which article would you be most likely to read next
based on layout position not titles?"  "Open to page four.  To what is your 
attention focused?") , information comprehension tasks, 
collect subjective satisfaction data (e.g. with its prettiness) via a licensed
copy of the Questionnaire for User Interaction Satisfaction (CHI '88 
proceedings and/or ask me for more details), etc.  Of course this is only
the task design part.  Now you have to decide the independent variable(s),
e.g. what to change between the designs.  Font family, font size, color,
rule lines, spacing, etc.  And if you change more than one of these at a time,
it is likely to be very difficult if not impossible to know how one independent
variable affected the outcome.  In short, you have a HUGE experiment on your
hands; I could imagine it taking up to two person-years of work to design
and execute this experiment (with design taking 2/3 of the time).
Perhaps you can look at the research literature of graphic designers,
advertizements, call up newspapers and ask the layout designer, etc.
Remember, we computer scientists tend to reinvent the wheel rather than
look to other disciplines from which a part of the computer interface has
borrowed much.

> I
>figure that if I can decide how to measure people's responses
>to/dislikes/likes between two proposed designs, I should just as well be
>able to assess those on one design.  

I disagree for the reasons of hard science mentioned above.  It would be
a very big mistake to say "we achieved a response of 80 from our
experimental participants; obviously this is very high and we need look
no further in this area."  How do you know that this isn't the *worst* 
score you could measure, not a good one?  You must compare it to something
else.  Even if you get a perfect score, you might get a perfect score
with many design choices, some of which interact much better with other
aspects of the design.

Good luck (and "Hi!"),
+Rick

-- 
  Richard "carm" Chimera, Lab Manager   | A population of sheep will eventually
  Human Computer Interaction Lab, U Md. | beget a government of wolves.
  A.V. Williams Bldg                    |   -- William Sloan Coffin
  College Park, MD  20742-3255    USA   |  
Newsgroups: comp.human-factors