From: marken@AERO.ORG
Newsgroups: bit.listserv.csg-l
Subject: The Test in practice
Date: 29 Sep 92 17:01:01 GMT
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU
X-Delivery-Notice:  SMTP MAIL FROM does not correspond to sender.
Posted-Date: Tue, 29 Sep 92 10:01:01 PDT

[From Rick Marken (920929.1000)]

Martin Taylor (920928 16:45) --

Your example of the test is, indeed, interesting.

In your example, it looks like Martin is controlling the position of the
waste basket, then stops. I don't see this as giving up -- it could be just
a change of reference. There really wasn't much testing going on in
the example; only one level and type of disturbance, no revision of the
hypoethesis about the CEV when there was lack of disturbance resistence, etc.

Your comments on conflict seem reasonable. I would just say that I think
the test should not involve setting up a conflict with the testee. The tester
should just be disturbing variables, not trying to prevent the testee from
controlling the variable. The test should cause no inconvenience for the
testee -- or higher order systems are likely to become involved and,
ultimately, reorganiztion. This would mess up the clarity of the results
of "the test".

>My claim is that The Test is always ambiguous. P can tell that Q is control
>ling for a percept that incorporates some CEV that P has disturbed, but P can
>never tell that the percept Q is controlling for corresponds exactly to the
>CEV that P's percept corresponds to.  The CEV is in the world, but only
>percepts can be controlled.  The percepts are private, so in a sense the
>CEVs are, too.

I suppose this is ultimately true. But when your guesses about a controlled
(perceptual) variable are allowing you to predict behavior variations with
99% accuracy, I think worrying about whether you have identified the
absolutely true CEV is rather academic. For example, in my "area" vs
"perimeter" control study, I did the test for the controlled variable to
determine whether the subject is controlling x+y vs x*y (where x and y
are height and width of a quadrilateral figure). Using x+y as the
hypothesized controlled variable the error in predicting responses was,
I think, about 2%. With x*y as the hypothesized CEV, the error was halved,
to 1%. You could probably do slightly better with some other hypothesized
CEV -- maybe sqrt(x*x+y*y) -- but clearly you are on the right track
with x*y. I guess you reach a point where you have to determine how much
trouble it's worth to try to improve your model of the behaving system;
it seems to me that, when you are predicting over 99% of the variance, you
are so far ahead of the conventional psychology game that such improve-
ments are really mute (until PCT science develops to a level at which
such tiny improvements are significant).

> But the correspondences of CEVs can be tested, and that is
>what you do when you try variants on moving the wastebasket, such as putting
>some other obstruction in the aisle, or moving the wastebasket to some other
>unobtrusive place.  Such experiments can never give exact answers (pace
>Rick Marken and Bill Powers) but can only be evaluated statistically.

>The changes in reference level are
>an independent reason why experimental tests can only be statistical, and
>why one can never be sure to what degree one's percepts coincide with the
>percepts of other people.

I agree that there are many difficulties inherent in doing "the test". One
only needs to actually try the "coin game" to see how difficult and
frustrating it can be in practice. But I think that, in this early stage of
the development of PCT science, the attitude that the test can only be
evaluated statistically, is just too defeatist for me. I say this
because 1) we already have some examples of very precise results
of the test -- where there is no need for statistical evaluation at all.
My "area/perimeter" study mentioned above is one; Bill's analysis of the
"shock avoidance" task is another; my mindreading demo should count as
another. These are examples of the test where the quantitative results are
so precise that the use of inferential statistics (I presume this is the
kind we're taking about) would be nothing more than pompous posturing (the
results are obviously "significant") 2) we know that people are controlling
perceptual variables and there is no reason to suspect that we cannot get
a pretty accurate description of them -- even if we can't actually perceive
what people are perceiving. So the goal of the test should always be high
quality data -- we should work to improve the test (changing disturbance
techniques, revising hypotheses about CEVs, whatever) until the results
are nearly perfect (as in the examples we have) befoe throwing in the
towel and starting to turn the statistical crank again; statistics should
be the last resort; but given my understanding of the PCT model I can't
see how they could tell you much about the system you are controlling. I
don't rule out statistics completely -- I have used a statistical measure
of performance in my mind reading demo (the stability factor) and done
an inferential statistical procedure to decide whether or not an object
on the screen was or was not under control. But this was a crude approach
to doing a version of the test -- if I really wanted to know WHAT variable
was being controlled, I would have done things that obviated the need for
the statistical approach.

I think that this is an important topic and, perhaps by discussing it we
will get some people out there to actually try the test and see what
problems they run into.

I don't want to be "ideologically" anti-statistical, by the way. I am willing
to believe that I don't fully understand your position on statistics, Martin.
I consider it entirely possible that statistical methods be could useful
tools in PCT -- they already have been. I just object to using statistics
the way they are used in conventional psychology -- to see whether or
not anything happened in a study. Maybe it would help if you described
exactly how you see statistics fitting into methodology of "the test".
Perhaps give a concrete example of your idea of using statistics
as part of "the test".

Thanks

Rick

     **************************************************************

Richard S. Marken                   USMail: 10459 Holman Ave
The Aerospace Corporation                   Los Angeles, CA 90024
E-mail: marken@aero.org
(310) 336-6214 (day)
(310) 474-0313 (evening)