Accepted manuscript
Tanaka H, Homma K, Imamizu H. (2011). Physical delay but not subjective delay determines
learning rate in prism adaptation. Experimental Brain Research, 208(2), 257-68.
Request for a reprint PDF to imamizu@gamil.com

1PHYSICAL DELAY BUT NOT SUBJECTIVE DELAY DETERMINES LEARNING RATEIN PRISM ADAPTATION

Hirokazu TANAKA 1,*, Kazuhiro HOMMA 2,*, Hiroshi IMAMIZU 1, 3
1National Institute of Information and Communications Technology (NiCT), Kyoto 619-0288,
JAPAN
2
Nagaoka University of Technology, Niigata 940-2188, Japan3
ATR Cognitive Mechanisms Laboratories, Kyoto 619-0288, Japan* These authors contributed equally to this work.

Abbreviated Title: Prism Adaptation under Modified Subjective Simultaneity
Corresponding author:
Hiroshi Imamizu
National Institute of Information and Communications Technology (NiCT)
Hikaridai 2-2-2, Keihanna Science City
Kyoto 619-0288, Japan+81-774-95-1220 (phone)+81-774-95-1236 (fax)
imamizu@gmail.com
Key words: motor adaptation; subjective simultaneity; physical simultaneity; temporal adaptation;
spatial adaptation
Acknowledgments: We thank John E. Jacobson for discussions on temporal perception, Kaoru
Nakamoto for conducting a preliminary experiment, Yuka Furukawa for organizing the experiments,
Terrence J. Sejnowski for reading an earlier version of the manuscript, and two anonymous
reviewers for helpful comments.
2

ABSTRACT

Timing is critical in determining the causal relationship between two events. Motor adaptation
relies on the timing of actions and their results for determining which preceding control signals
5

were responsible for subsequent error in the resulting movements. An artificially induced
temporal delay in error feedback as short as 50 ms has been found to slow the learning rate ofprism adaptation. Recent studies have demonstrated that our sense of simultaneity is flexibly
adaptive when a persistent delay is inserted into visual feedback timing of one’s own action.
Therefore, judgments of ”subjective simultaneity” (i.e. whether two events are simultaneous on a

10

subjective basis) do not necessarily correspond to the actual simultaneity of physical events. We
evaluated the effects of adaptation to a temporal shift of subjective simultaneity on prism
adaptation by examining whether prism adaptation depends on physical timing or subjective
timing. We found that after persistently experiencing an additional 100 ms delay in a pointing
experiment, psychometric curves of the timing of judgments about the temporal order of touching15

and visual feedback were shifted by 40 ms, indicating that subjective simultaneity adapted. Next,
while maintaining temporal adaptation, participants adapted to spatial displacement caused by aprism with and without an additional temporal delay in feedback. Learning speed was reliably
predicted by physical timing but not by subjective timing. These results indicate that prism
adaptation occurs independently of awareness of subjective timing, and may be processed in20

primary motor areas that are thought to have fidelity with temporal relations.

3

INTRODUCTION

Time differs from the three spatial dimensions in the sense that it flows only in one direction. It
is unsurprising that the human perceptual and motor systems exploit this temporal
5

unidirectionality. Inter-stimulus temporal order provides a crucial cue to the visual system in
estimating a given motion stimulus (McKee, 1981; Ramachandran and Anstis, 1986), and to theauditory system in localizing a sound-source position (Knudsen and Konishi, 1978; Takahashi
and Konishi, 1986). Time is most important in the motor control system, where appropriate
movements require precisely coordinated timing of muscle activation (Miall et al., 1993; Rao et

10al., 1997; Ivry et al., 2002). It is also known that delayed visual feedback as short as a fraction of
a second has deleterious effects on writing and drawing (Smith et al., 1960; Miall et al., 1985).
Further, delayed auditory feedback of self-generated voice has been found to impair speech (Lee,1950). Therefore, timely control of the musculoskeletal system and sensory feedback fromperiphery receptors are of vital importance both in perceiving the external environment and in15

coordinating body movements.

Temporal relationships play a crucial role not only in movement execution but also in
movement adaptation, which involves learning the causal association between motor commands
and subsequent errors. Among various cues for identifying causal relationships, sensory feedback
20

is important for determining the temporal order and proximity among recent events. As such, an
artificially induced delay in sensory feedback would be expected to affect how motor adaptation
proceeds. An induced delay of visual feedback during prism adaptation, for example, has beenfound to decrease learning speed even with delays as short as 50 ms both in humans (Kitazawa etal., 1995) and monkeys (Kitazawa and Yin, 2002). Further, physiological experiments have4

revealed that a precise spike timing between pre- and post-synaptic neurons determines how thesynaptic strength between these neurons will be modified (i.e., potentiated or depressed) in thecortex (Markram et al., 1997; Yao and Dan, 2001; Froemke and Dan, 2002; Celikel et al., 2004),
the hippocampus (Debanne et al., 1998; Watanabe et al., 2002; Wang et al., 2005) and basal
5ganglia (Fino et al., 2005; Pawlak and Kerr, 2008) of the mammalian brain, and also in fish (Bellet al., 1997). In the cerebellum, thought to function as a neural site for movement coordinationand learning (Marr, 1969; Houk et al., 1996; Imamizu et al., 2000), synaptic strength between theparallel fibers and the Purkinje cells is maximally depressed when parallel-fiber activity isfollowed by climbing fiber activity approximately 200 ms later (Chen and Thompson, 1995).

10

These physiological results, together with the psychophysical findings, indicate that the brain
possesses a physiological learning mechanism that makes use of temporal relationships as alearning cue.

Among various approaches to investigating how the motor system adapts to novel environments,
15

arguably the best-studied paradigm is visual disruption using optical prisms (Stratton, 1896;
Redding et al., 2005). Experimental paradigms that use wedge-prism displacement of vision are
popular in laboratories because they can be conducted in a relatively short time-period compared
to dove-prism (i.e., inverted vision) adaptation and can be quantitatively evaluated using definite
measures such as learning speed and the magnitude of aftereffects (Held and Gottlieb, 1958;

20

Harris, 1963; Taub and Goldberg, 1973; Hatada et al., 2006).

Recent psychophysical studies have demonstrated that human judgment of temporal
simultaneity is not innately fixed but flexibly modifiable after repeated exposure to temporally
shifted stimuli in sensory processing within a single modality (Miyazaki et al., 2006) and across5

multiple modalities (Fujisaki et al., 2004; Vroomen et al., 2004; Harrar and Harris, 2005;
Zampini et al., 2005; Ceux et al., 2006; Hanson et al., 2008; Harrar and Harris, 2008; Keetels andVroomen, 2008; Boenke et al., 2009; Di Luca et al., 2009; Kennedy et al., 2009; Navarra et al.,2009). Temporal adaptation is not limited to passive sensory processing, but also occurs in active
5

sensorimotor processing. Recalibration of the sensorimotor relationship between self-generated
action and temporally modified visual feedback leads to an illusory reversal of their causal
relationship (Cunningham et al., 2001; Eagleman and Holcombe, 2002; Stetson et al., 2006;
Winter et al., 2008; Heron et al., 2009). Therefore, the point of subjective simultaneity (PSS) canbe dissociated from the point of physical simultaneity between motor action and sensory stimuli.

10

By dissociating subjective simultaneity from physical simultaneity, a delay that a participant
experiences would be expected to differ from the physical delay. That is, a subjective delay
should be reduced or increased relative to a physical delay, to an extent determined by how much
the PSS has adapted.

15

In this study, we tested whether motor adaptation depended on subjective delay or physicaldelay when participants dissociated subjective and physical timing. These hypotheses are referred
to as the “subjective-timing” and “physical-timing” hypotheses, respectively, throughout thispaper. In other words, we sought to investigate the functional levels that play important roles inprism adaptation. If prism adaptation depends on subjective delay, the level of prism adaptation

20

would be expected to reside within cognitive time perception and higher motor control systems.
Alternatively, if it depends on physical delay, lower, autonomous motor systems would be
expected to play a dominant role in prism adaptation. To test these hypotheses, we comparedlearning speed during adaptation to lateral visual displacement when participants’ subjective
simultaneity was dissociated from physical simultaneity. Recent studies evaluating generalization6

of temporal adaptation across different multi-sensory pairs have raised questions about whether
there is a single clock (Hanson et al., 2008; Heron et al., 2009) or multiple clocks (Miyazaki et al.,2006; Harrar and Harris, 2008) involved in sensory systems. Our study was conducted in accord
with these studies, and extended them to question whether the motor system responsible for prism
5

adaptation and the sensory system responsible for temporal order judgment are governed by asingle clock or not.

METHODS
Participants
10

Ten right-handed adults from our institute and a nearby university (age: 18-47; nine male andone female) took part in Experiments 1 and 2, and fifteen right-handed adults (age: 21-49;
thirteen male and two female) took part in Experiment 3. We obtained written consent from all
the participants. The study was approved by an institutional ethics committee. The participants
reported no known neurological history and had normal or corrected-to-normal vision.

15

Experimental setting
We closely followed the experimental setting described in previous studies of prism adaptation(Kitazawa et al., 1995). We describe the paradigm briefly below for convenience. The
experimental setup consisted of a 17-inch CRT (85Hz refresh rate: CV772X, TOTOKU Electric
20

Coop, Tokyo, JAPAN) for presenting visual stimuli, a touch screen on the display that measured
pointed locations, a custom-made liquid-crystal shutter for controlling the timing of visualfeedback (Takei Coop, Niigata, JAPAN), a button for measuring movement onset time, and amouse for recording participants’ responses (Fig. 1A). The participants sat comfortably in front of
the desk and wore goggles that restricted their vision in the central field. Their heads were7

stabilized on a chin rest, and the distance between the participants’ head and the display wasmeasured and fixed at 30 cm. On the goggles, rightward-displacement prisms (15-diopter) wereplaced that could be flipped on and off manually, and induced a rightward visual shift ofapproximately 10.7 cm on the screen. In-house Matlab codes on a Windows-based computer,5

using the Psychophysics toolbox extensions (Brainard, 1997; Pelli, 1997), were used to present
visual stimuli, to record the participants’ responses, and to control the timing of opening (fullvisual feedback) and blocking (no visual feedback) the liquid-crystal shutter. The display location
was adjusted with or without prisms so that targets fell into the central visual field of participants(Fig. 1B). The participants wore headphones through which white-noise sound was played to10

block external sounds (i.e., the sound of fingers hitting the screen) so that they would not notice
the timing of touching the screen by listening. We chose 100 ms for the intended delay duration
because this value has been shown to be effective in decreasing prism adaptation speed (Kitazawa
et al., 1995) and in inducing temporal shift adaptation (Stetson et al., 2006; Heron et al., 2009).

15

Because this study focused on human timing perception, it was important to verify that thetiming of visual feedback was precisely and consistently controlled as we intended. Because of
the processing time of the PC and the response time of the liquid-crystal shutter, the presentationof visual feedback inevitably involved a slight additional delay larger than 0-ms (i.e., no delay
included in the stimulus-presentation code), both when no delay was intended, and when a

20

100-ms delay period was intentionally included, as described below. We used a commercial
high-speed digital camera (EX-F1, Casio, Japan, maximum 1,200 frames per second) to measure
the temporal interval from the moment at which a participant’s finger hit the screen to themoment at which visual feedback appeared at the display. The interval was measured when either
no delay or a 100-ms delay was imposed between the above two moments by the stimulus8presentation code. The measurement was repeated 30 times for each condition. The mean of themeasured delay interval was 36.4 (SD 5.08) ms when a 0-ms delay was intended, and 136.4 (SD4.67) ms when a 100 ms-delay was intended. Thus, we refer to the two conditions as the“minimum 36-ms delay” (or simply “minimum delay”) and “136-ms delay” conditions to reflect
5

the actual delay times for visual feedback.

Psychophysical experiments
Two psychophysical experiments (Experiments 1 & 2) were conducted with ten participants,
and one experiment (Experiment 3) was conducted with fifteen participants. All the experiments
10

used the same experimental setting as described above, shown in Fig. 1A. Experiment 1 (delay
adaptation experiment) measured the shift of PSS to examine the degree to which the participants
exhibited temporal lag adaptation during prism adaptation. Experiment 2 (prism adaptation
experiment) then evaluated how temporal lag adaptation affected the learning rate of prism
adaptation under three conditions: (1) minimum-delay (MD), (2) physical-delay (PD), and (3)15delay-adapted (DA) conditions, which are described in detail below. Finally, Experiment 3
determined the learning rate using a delay that was reduced by the magnitude of the PSS shiftmeasured in Experiment 1. The condition tested in Experiment 3 was thus referred to as the
subjective-delay (SD) condition.

20

Before the three experiments, the participants underwent training sessions to familiarize
themselves with the experimental setting and movement requirements.

General procedures
The participants were required to make a whole upper-limb (right arm) movement in a similar
9

way in the three experiments. The commonalities are described in Fig. 1C. At the beginning of
the experiment, they were instructed to wait for auditory cues (four beeps, three low-frequency
tones followed by a high-frequency tone) from the headphones, indicating target onset (a yellow
cross, 1 cm ! 1 cm). As soon as a target was presented, the participants were asked to respond5

with their right index finger within a reaction time limit of 200 ms, by releasing the button andmaking a rapid pointing movement toward the target within a movement time limit of 250 ms.
Each target was displayed at a random location (taken from a uniform distribution) in a 4 cm ! 4
cm square at the center of display. From the time the participants released the button to the time
the participants reached the touch screen, the vision was fully blocked by closing the10

liquid-crystal shutter. When the finger movement terminated on the touch screen, the blockage ofvision was removed by clearing up the shutter, and the participants were allowed to view thetarget and their finger location. By blocking visual feedback during the movement, theparticipants were prevented from making any online movement corrections, and could adapt only
by using the terminal feedback of the endpoint position. In some trials, an artificial delay was

15

imposed by controlling the timing of shutter opening, for the purpose of either inducing a shift ofsubjective simultaneity (Experiment 1) or examining the influence of delayed feedback on prism
adaptation (Experiments 2 & 3).

In our precursory experiments, a computer cursor was used to indicate the endpoints. However,20

the learning curves did not exhibit an incremental exponential decay, consistent with a previous
report (Norris et al., 2001), and the aftereffects were not significant (Clower and Boussaoud,2000). As such, we provided a full view of participants’ own hands and fingers at the end of the
movement. We also observed in preliminary experiments that rather strict constraints on reaction
and movement times were necessary for reproducing exponential learning curves (see also
10

DISCUSSION).

Pointing practice sessions
The purpose of these training sessions was (1) to familiarize the participants to the experimental
5

environment and (2) for the participants to practice responding and moving as quickly as possible
while maintaining precise endpoint accuracy. Before proceeding to the experiments, theparticipants underwent training sessions (30 trials each) until they passed the following criteria.
The participants were required to satisfy both the response-time and movement-time limits in noless than 90% of trials in a single training session. Additionally, we required participants to10complete at least 50% of trials with movement error (defined as the Euclidean distance between atarget and final finger positions) that was no more than 1 cm. The participants proceeded to the
PSS and prism adaptation experiments when these two criteria were satisfied.

Experiment 1: Delay-adaptation experiment
15

The purpose of Experiment 1 was to investigate whether temporal-order judgment (TOJ)
adapted in a large-amplitude movement involving full upper-limb movement under conditions oflaterally displaced vision. Previous studies reported a shift in PSS between a keyboard press(limited to finger movement of small amplitude) and temporally delayed yet spatially intact visualfeedback (Stetson et al., 2006). It is currently unclear whether temporal adaptation occurs in our

20experimental conditions (full arm movement and displaced vision). Therefore, we asked theparticipants to judge the temporal order between finger tactile feedback and visual feedback. The
timing of visual feedback could be manipulated by a liquid-crystal shutter installed directly infront of participants’ faces (Fig. 1A).

11

A single session consisted of 60 initial delay adaptation trials (without prism) followed by 20
subsequent test trials (with prism). In the initial trials in the baseline condition, visual feedback oftouched location on the screen was provided by opening the shutter immediately after theparticipants touched the screen. In the initial trials of the delay adaptation condition, visual5

feedback was provided 136 ms after the participants touched the screen. We assessed the effect ofdelay adaptation using a TOJ task. During the test trials, visual feedback was presented at various
times from the movement onset as follows. An average movement duration was computed fromthe previous five trials, and the timing of visual feedback was randomly determined by adding a
Gaussian random number (mean 0 ms, standard deviation 80 ms) to the average movement

10

duration, so visual feedback appeared on the display either before or after a participant touched
the screen. The participants were asked to report whether they felt the visual feedback was before
or after the moment they touched the screen by pressing the middle (before) or left (after) buttonof a computer mouse held in their left hands.

15

Experiment 2: Prism-adaptation experiment after delay adaptation
In the second experiment, the participants completed a total 120 runs of pointing movements,
and adapted then de-adapted to laterally displaced vision induced with wedge prisms under theminimum-delay (MD), physical-delay (PD), and delay-adapted (DA) conditions. During 60 initialbaseline trials (trials 1-60) without prism displacement, visual feedback was displayed without

20

any intentionally imposed delay (corresponding to an actual delay of 36 ms) in the MD and PD
conditions, and with an additional 100 ms delay (corresponding to an actual delay of 136 ms) in
the DA condition. Subsequently, during 30 trials (trials 61-90) with laterally displaced vision towhich the participants adapted, the induced delay was 36 ms in the MD condition and 136 ms in
the PD and DA conditions. Finally, during the last 30 de-adaptation trials (trials 91-120), the 36
12

ms delay was imposed for the three conditions. These delays are summarized in Table 1. Theinter-trial interval was 4 s within the baseline, adaptation, and de-adaptation trial blocks. There
were short time intervals (usually less than one minute) immediately before the 61st and 91st
trials, in which the experimenter flipped on and off the prism on the goggles and adjusted the5

CRT position (Fig. 1B). During these breaks, the participants were asked to close their eyes. All
the participants completed two runs of each of the three conditions, resulting in six sessions per
participant. The presentation order of the conditions was randomized from participant toparticipant, and a break of several minutes was taken between sessions.

10

Experiment 3: Prism-adaptation experiment with PSS-shift-reduced delay
The purpose of the third experiment was to explicitly evaluate the effects of a physical delaythat was matched to the amount of subjective delay exhibited on the learning rate of prism
adaptation. Participants underwent a prism adaptation experiment that was same as in the PD
condition of Experiment 2 with one exception: the delay in adaptation trials was reduced by the15

magnitude of the PSS shift determined in Experiment 1 (bottom row in Table 1). This condition
was referred to as the subjective delay (SD) condition. By comparing the learning coefficientsmeasured in the SD and DA conditions, it was possible to evaluate the degree to which thelearning rate was determined by the subjective delay.

20

Data Analysis
All experimental data was analyzed using Matlab and its Statistical Toolbox (MathWorks, MA)
and JMP (SAS Institute Inc., NC).

Experiment 1: Delay adaptation experiment13

For the delay adaptation experiment, the degree to which the participants shifted their PSS wasevaluated according to Stetson et al. (2006). First, feedback timing (i.e., the time of opening
shutter from the time of participants’ touching the screen) was binned into a 60 ms interval
between -200 to 200 ms. In each bin we computed the probability that each participant judged
5

visual feedback to have occurred after the movement was completed. For example, in the i-th bin,if a participant reported ni times out of Ni occurrences that the visual feedback occurred after thescreen touch, then the probability of judging “after” was evaluated as ni/Ni. Next, a sigmoid
function,,10

was fitted to obtain a psychometric curve using a Matlab glmfit function. t is the time of visualfeedback measured from the time of touching, and t0 and b are the intercept and the slope,
respectively. In addition to individual psychometric curves, a group psychometric curve wassimilarly computed by summing up all participants’ responses.

15

Generally speaking, two adaptive consequences can be caused by repeated exposure toasynchronous stimuli (Hanson et al., 2008). One possibility is the broadening of the temporalwindow of simultaneity between stimuli so that temporally asynchronous stimuli are perceived assimultaneous. This type of adaptation would manifest as a shallower psychometric curve (i.e.,larger slope b) with a larger just noticeable difference (JND). Another possibility is a shift in the20

point of subjective simultaneity (PSS) between exposed stimuli so that temporal asynchrony issubjectively cancelled. In this case, the intercept of a psychometric curve (t0) would be displaced
according to the amount of PSS shift. Therefore, we examined whether the participants broadened
the temporal time window or shifted the PSS by investigating the slope and intercept of14

psychometric curves.

Experiments 2 & 3: Prism adaptation experiment
There are at least two methods for quantifying learning speed in prism adaptation experiments; a5state-space model approach (Baddeley et al., 2003; Cheng and Sabes, 2006) and a discrete-model
approach (Kitazawa et al., 1995; Yin and Kitazawa, 2001). We used the latter method, so that ourfindings would be comparable with Kitazawa’s results. The learning-speed coefficients, k, werecomputed for learning curves according to the methods of Kitazawa et al. (1995), and were
statistically compared between the conditions. First, the learning coefficient k is defined as a10

portion of an error in previous trial that reduces an error in current trial;.
Here the subscript n denotes the trial number from adaptation onset, anddenotes the

movement error at n-th trial. Then, recursively applying this equation we obtain.

15

Therefore, k can be computed as a slope of a line connecting the origin andPractically, we have such (n-1) points.

(in our case 29 points for eachparticipant), and the coefficient k and the intercept were estimated using an ordinary least square
fitting:

20

The k’s obtained in the four conditions (denoted as kMD, kPD, kDA, and kSD respectively) were15

statistically compared to test whether they were different according to the conditions.

Predictions of the physical-timing and subjective-timing hypotheses
Before we conducted the prism adaptation experiment, we had two hypotheses about the timing
5of feedback that predominantly contributed to the learning speed of prism adaptation. If
participants adapted fully to the imposed delay of 100 ms, the PSS would be expected to shift by
100 ms. Thus, the subjective delay of visual feedback would be the same as that in the MD
condition, whereas the physical delay would remain at 136 ms. One hypothesis was that the rateof prism adaptation would depend mostly on the simultaneity that participants subjectively

10reported (referred to as the subjective-timing hypothesis). According to this prediction, thelearning curve in the DA condition would be expected to overlap with that in the MD condition(Fig. 2A) if the PSS shifted the full extent of the imposed delay (100 ms). The other hypothesis
was that the physical simultaneity determines the rate of prism adaptation (referred to as thephysical-timing hypothesis). This hypothesis predicts that the learning curve in the DA condition

15

should overlap with the PD condition (Fig. 2B).

Previous studies have shown that participants can partially shift their PSS in response to an
imposed delay (Stetson et al., 2006; Winter et al., 2008; Heron et al., 2009). In that case, asubjective delay does not become zero even after intensive delay adaptation, and delay adaptation
20

should subjectively reduce physical delay only by the amount of the PSS shift. Therefore, thelearning rate in the DA condition should be compared with that in the PD and SD conditions. The
subjective-timing hypothesis predicts comparable learning rates in the DA and SD condition. In
contrast, the physical-timing hypothesis predicts comparable learning rates in the DA and PD
conditions. These predictions were explicitly tested in Experiments 2 and 3.
16

RESULTS
Before the three experiments, the participants were required to complete training sessions until
the two performance criteria described in the method were satisfied. Most participants reported
5that they felt these criteria to be rather challenging initially; after 6 - 24 training sessions
participants generally reported that they felt much more comfortable with the task after training.
The practice session usually took 30 - 120 minutes including breaks.

PSS shift occurring in whole limb movement under prism displacement
10

In Experiment 1, we quantified, as a function of the time difference between the participants’screen touch and visual feedback, the probability that the participants reported that visualfeedback occurred post movement (open and filled circles in Fig. 3A). The probability was fit bya sigmoid function (broken and solid lines in Fig. 3A). The post-adaptation curves clearly shifted
toward rightward compared to the pre-adaptation curves. The shift was evaluated as an intercept15(t0: see METHODS) of the post-adaptation curve measured from an intercept of the
pre-adaptation curve. The evaluated shifts were 40.1 (SD: 10.1) ms averaged across subjects. Thiswas significantly larger than zero according to a one-sided t-test (t(9) = 5.58; P = 0.0002). This
indicates that the participants’ PSS adapted to the repeated, delayed feedback in the whole limb
movements. We also quantified the slopes (b) of psychometric curves, finding that the20

slope-value averaged across participants was 42.9 (SD 16.4) ms for the pre-adaptation curve and40.6 (SD 10.4) ms for the post-adaptation curve. There was no significant difference in slope
values between the curves (t(9) = 0.38). The group averaged psychometric curves are shown in
Fig. 3B. The slopes were similar, and only the intercepts were shifted, indicating that persistent
delayed feedback shifted the PSS but did not broaden the JND in temporal perception. The
17

subjective delay caused by the temporal adaptation was thus approximately 96 ms (physical delay(136 ms) – shift of PSS (40 ms)).

Delay adaptation had no effect on learning rate
5

In Experiment 2, we examined prism adaptation under the three different visual-feedback timing
conditions (i.e. the minimum-delay (MD), physical-delay (PD), and delay-adapted (DA)conditions; see METHODS section and Table 1). We imposed the rather demanding time
constraints for reaction and movement times so that the participants did not use any cognitive or
strategic methods for correcting endpoint errors (see DISCUSSION). During the baseline,

10

adaptation, and de-adaptation trials, the reaction and movement times were relatively constant(Figs. 4A & B). There was a slight increase of reaction time in initial trials of adaptation anddeadaptation blocks (the 61st and 91st trials in Fig. 4A). This may have been caused by shorttime intervals in which the experimenter flipped the prism on the goggles on and off. Verticalerrors were close to zero in the three blocks when they were averaged across trials, conditions and15

subjects (baseline: -0.15±1.08, adaptation: -0.20±1.35, and de-adaptation: 0.04±1.22 cm; apositive or negative value indicates an upper or lower error relative to the target). We conducted a
two-way (block by condition) repeated-measures ANOVA on the vertical errors but could not
find a significant effect of block (F(2,18) = 1.76) or condition (F(2,18) = 0.13). This indicatesthat the horizontal displacement caused by the prism had no effect on the perpendicular direction.

20

Learning curves of horizontal errors in the three conditions exhibited typical exponential decayfunctions (Fig. 5A: MD in blue, PD in black, and DA in red). The endpoint errors in the initial
trial of adaptation (61st trial) were significantly smaller (8.1±2.3 cm) than the visual displacement
imposed by the prism (10.7cm), as reported in a previous work (Redding and Wallace, 2003).
18

There were no significant differences between the initial errors in the three conditions (8.4±1.6
for MD, 8.0±2.4 for PD, and 8.0±2.7 cm for DA; p = 0.80, F(2,57) = 0.23, one-way ANOVA).
First, we attempted to replicate Kitazawa’s result that an artificially induced delay in visualfeedback leads to a slower adaptation speed of prism adaptation. We found that learning speed
5

was significantly slowed when the feedback delay was introduced in the PD condition (black
curve in Fig. 5A) compared when no intentional delay was imposed (blue curve in Fig. 5A),
consistent with Kitazawa’s findings. The learning coefficients in the MD and PD conditions were9.0±7.0 % and 6.3±4.6 %, respectively (left and middle box plots in Fig. 5B).

10

The learning curve in the DA condition (red curve in Fig. 5A) largely overlapped with that in
the PD condition but not in the MD condition. In accord with this observation, the learning
coefficients in the DA condition (right box plot in Fig. 5B, 5.9±4.6%) were significantly smaller
than those in the MD condition but not significantly different from those in the PD condition (atP < 0.05 level corrected for multiple comparisons by Tukey’s test). Therefore, for the DA

15

condition, the delay adaptation in the baseline trials had no effect on the learning coefficientsduring adaptation trials.

Subjective delay could not explain the reduced learning rate in delay-adapted condition
The delay adaptation experiment indicated that subjective simultaneity shifted by approximately
20

40 ms, thereby reducing the subjective delay from the actual delay of 136 ms to only 96 ms.
Learning rate was previously reported to rapidly decrease up to a feedback delay of 50 ms, andgradually decrease with a delay greater than 50 ms (see Figure 5C of Kitazawa et al., 1995). Thus,
learning rates with 96-ms and 136-ms delay might not differ enough to confirm or disconfirm
either of the two hypotheses. The results of Experiment 2, together with the findings of Kitazawa
19

et al. (1995), could not decisively determine whether the decreased learning rate in the DA
condition was attributable to the physical delay (136 ms) or the subjective delay (96 ms).

In Experiment 3, participants adapted to prism displacement with a physical delay that was
5matched to the subjective delay of 96 ms (referred to as the subjective-delay (SD) condition). We
found, in our experimental setting, that the learning coefficients in the SD condition weresignificantly larger than those in the DA condition, according to a t-test (t(48) = 2.25, P < 0.05;
Fig. 6). Therefore, the reduced learning rate in the DA condition could not be attributed to thesubjective delay of 96 ms, excluding the subjective-timing hypothesis. To check whether

10sampling bias between subject groups in Experiments 2 and 3 affected this finding, we comparedlearning coefficients in the SD condition in Experiment 3 with those in the MD condition inExperiment 2. There was no significant difference between them (t(48) = 0.60), and thus unlikely
sampling bias affected the result.

15

DISCUSSION

This study tested how subjectively modifying the timing between sensorimotor events
influenced the learning of a novel visuomotor environment. The results revealed two major
findings. First, in Experiment 1 (delay-adaptation experiment), most participants showed a20

dissociation between their subjective timing and the physical timing, even in conditions involving
a whole-limb pointing response with displaced vision. The extent of temporal adaptation we
observed was not complete. In this partial adaptation, a shift of the PSS of approximately 40 ms
was observed after adapting to conditions of feedback delayed by 100 ms, consistent with aprevious result. Therefore, a subjective delay was reduced to approximately 96 ms by the PSS
20

shift of 40 ms when the physical delay in visual feedback was actually 136 ms. Experiment 2(prism adaptation experiment) revealed that the learning rate in the delay-adapted condition was
significantly slower than in the minimum-delay condition, but was not different from in thephysical-delay condition. The result obtained in Experiment 3 demonstrated that the decreased5learning rate in the delay-adapted condition could not be attributed to the subjective delay of 96
ms, thereby disconfirming the subjective-timing hypothesis. In summary, we concluded thatlearning speed in prism adaptation depended on the physical time interval between touching andvisual feedback, but was not affected by subjective temporal perception.

10

Recalibration and realignment processes in prism adaptation
Two possible adaptive processes for correcting endpoint errors have been proposed in previous
reports (Redding and Wallace, 2001; Mazzoni and Krakauer, 2006; Michel et al., 2007). One is arealignment process (also referred to as an explicit or strategic process) in which a participant
responds to an illusory target that is located opposite to the prism displacement. This requires

15

explicit knowledge about the size and direction of displacement, and planning to compensate forthe displacement. An alternative possibility is that a recalibration process (also referred to as animplicit or automatic process) is involved, correcting movements incrementally without any
explicit knowledge about how a movement error was caused (Kagerer et al., 1997).

20

In our experiments, we instructed the participants to react and move as rapidly as they could
within stringent time constraints, providing insufficient extra time for developing a realignment
process between the target and movement onsets. Our experiments thus predominantly targeted arecalibration process of prism adaptation. The results thus indicate that such a recalibration
process depends mostly on physical timing but not subjective timing. This finding is in accord
21

with the absence of subjective awareness involved in recalibration processes.

Neural sites of prism adaptation and temporal perception
Functional neuroimaging and electrophysiological studies have suggested a number of cortical5and subcortical areas to be involved in prism adaptation, including the cerebellum (Luaute et al.,2009), the ventral premotor cortex (Kurata and Hoshi, 1999), and the posterior parietal cortex(PPC; Clower et al., 1996). Pisella et al. (2004) proposed that explicit and strategic realignment islinked to PPC activity, while implicit and automatic recalibration is linked to the cerebellum. In
contrast, Stetson et al. (2006) reported activation related to the temporal reversal illusion in the10

anterior cingulate cortex and the medial frontal cortex. Therefore, prism adaptation and this
temporal illusion do not appear to overlap in terms of neural substrates, in accord with ourconclusion that the shift in PSS had no influence on the learning rate in prism adaptation.

Prism adaptation under reversed cause-effect relationship
15

Arguably the most intriguing aspect of temporal adaptation is that our subjective timing of cause
and effect can be illusorily reversed (Cunningham et al., 2001; Stetson et al., 2006). According tothese studies, for example, after being exposed repeatedly to 100 ms delayed visual feedback of abutton press, the sudden removal of the delay can lead to the impression that visual feedback
occurs in advance of the button press. Therefore, the subjective causality can be broken down in

20

some cases. Whether it is possible to adapt to prism displacement even when visual error
feedback is not perceived to be a result of self-movements is currently unclear. We performed an
exploratory experiment with six participants in which an intentional 100-ms feedback delay was
imposed during 60 baseline trials, with no intentional delay during subsequent 30 prism
adaptation trials. Participants exhibited typical exponential learning curves, and their learning22speed was similar to that in a no-delay condition. This result was in accord with the main findingof the current study, that subjective timing did not influence the learning speed of prism
adaptation or the automatic processes underlying movement recalibration.

5

23

5

Table 1: Actual feedback delay used for the minimum-delay, physical-delay, delay-adapted, andsubjective-delay conditions.
Condition

Baseline (60 trials)

Adaptation (30 trials)

De-adaptation (30 trials)

Minimum Delay (MD)

36 ms

36 ms

36 ms

Physical Delay (PD)36 ms

136 ms

36 ms

Delay Adapted (DA)136 ms

136 ms

36 ms

Subjective Delay (SD)

36 ms

96 ms

36 ms

10

24

References

Baddeley RJ, Ingram HA, Miall RC (2003) System identification applied to a visuomotor task:
near-optimal human performance in a noisy changing task. J Neurosci 23:3066-3075.
5Bell CC, Han VZ, Sugawara Y, Grant K (1997) Synaptic plasticity in a cerebellum-like structure
depends on temporal order. Nature 387:278-281.
Boenke LT, Deliano M, Ohl FW (2009) Stimulus duration influences perceived simultaneity inaudiovisual temporal-order judgment. Exp Brain Res 198:233-244.
Brainard DH (1997) The Psychophysics Toolbox. Spat Vis 10:433-436.

10

Celikel T, Szostak VA, Feldman DE (2004) Modulation of spike timing by sensory deprivation
during induction of cortical map plasticity. Nat Neurosci 7:534-541.
Ceux T, Wagemans J, Rosas P, Montagne G, Buekers M (2006) Perceptual-motor adaptations in
a synchronization task: The joint effects of frequency and motion coherence
manipulations. Behav Brain Res 168:226-235.

15

Chen C, Thompson RF (1995) Temporal specificity of long-term depression in parallel
fiber--Purkinje synapses in rat cerebellar slice. Learn Memory 2:185-198.
Cheng S, Sabes PN (2006) Modeling sensorimotor learning with linear dynamical systems.
Neural Comput 18:760-793.
Clower DM, Boussaoud D (2000) Selective use of perceptual recalibration versus visuomotor

20

skill acquisition. J Neurophysiol 84:2703-2708.
Clower DM, Hoffman JM, Votaw JR, Faber TL, Woods RP, Alexander GE (1996) Role ofposterior parietal cortex in the recalibration of visually guided reaching. Nature
383:618-621.
Cunningham DW, Billock VA, Tsou BH (2001) Sensorimotor adaptation to violations of25

temporal contiguity. Psychol Sci 12:532-535.
Debanne D, Gahwiler BH, Thompson SM (1998) Long-term synaptic plasticity between pairs ofindividual CA3 pyramidal cells in rat hippocampal slice cultures. J Physiol London
507:237-247.
5

Di Luca M, Machulla TK, Ernst MO (2009) Recalibration of multisensory simultaneity:
cross-modal transfer coincides with a change in perceptual latency. J Vision 9:7 1-16.
Eagleman DM, Holcombe AO (2002) Causality and the perception of time. Trends Cogn Sci
6:323-325.
Fino E, Glowinski J, Venance L (2005) Bidirectional activity-dependent plasticity at

10

corticostriatal synapses. J Neurosci 25:11279-11287.
Froemke RC, Dan Y (2002) Spike-timing-dependent synaptic modification induced by natural
spike trains. Nature 416:433-438.
Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity.
Nat Neurosci 7:773-778.

15

Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory
modalities. Exp Brain Res 185:347-352.
Harrar V, Harris LR (2005) Simultaneity constancy: detecting events with touch and vision. Exp
Brain Res 166:465-473.
Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile

20

stimulus combinations on the perception of simultaneity. Exp Brain Res 186:517-524.
Harris CS (1963) Adaptation to displaced vision: visual, motor, or proprioceptive change?
Science 140:812-813.
Hatada Y, Miall RC, Rossetti Y (2006) Long lasting aftereffect of a single prism adaptation:
directionally biased shift in proprioception and late onset shift of internal egocentric26reference frame. Exp Brain Res 174:189-198.
Held R, Gottlieb N (1958) Technique for studying adaptation to disarranged hand-eye
coordination. Percept Motor Skill 8:83-86.
Heron J, Hanson JV, Whitaker D (2009) Effect before cause: supramodal recalibration of
5

sensorimotor timing. PLoS One 4:e7681.
Houk JC, Buckingham JT, Barto AG (1996) Models of the cerebellum and motor learning. Behav
Brain Sci 19:368-383.
Imamizu H, Miyauchi S, Tamada T, Sasaki Y, Takino R, Putz B, Yoshioka T, Kawato M (2000)
Human cerebellar activity reflecting an acquired internal model of a new tool. Nature10

403:192-195.
Ivry RB, Spencer RM, Zelaznik HN, Diedrichsen J (2002) The cerebellum and event timing. In:
Highstein TM, Thach WT (eds) Cerebellum: Recent developments in cerebellar research.
New York Academy of Sciences, New York, pp 302-317.
Kagerer FA, Contreras-Vidal JL, Stelmach GE (1997) Adaptation to gradual as compared with15

sudden visuo-motor distortions. Exp Brain Res 115:557-561.
Keetels M, Vroomen J (2008) Temporal recalibration to tactile-visual asynchronous stimuli.
Neurosci Lett 430:130-134.
Kennedy JS, Buehner MJ, Rushton SK (2009) Adaptation to sensory-motor temporalmisalignment: Instrumental or perceptual learning? Q J Exp Psychol 62:453-469.

20

Kitazawa S, Yin PB (2002) Prism adaptation with delayed visual error signals in the monkey.
Exp Brain Res 144:258-261.
Kitazawa S, Kohno T, Uka T (1995) Effects of delayed visual information on the rate and amountof prism adaptation in the human. J Neurosci 15:7644-7652.
Knudsen EI, Konishi M (1978) A neural map of auditory space in the owl. Science 200:795-797.
27

Kurata K, Hoshi E (1999) Reacquisition deficits in prism adaptation after muscimol
microinjection into the ventral premotor cortex of monkeys. J Neurophysiol
81:1927-1938.
Lee BS (1950) Effects of delayed speech feedback. J Acoust Soc Am 22:824-826.
5

Luaute J, Schwartz S, Rossetti Y, Spiridon M, Rode G, Boisson D, Vuilleumier P (2009)
Dynamic changes in brain activity during prism adaptation. J Neurosci 29:169-178.
Markram H, Lubke J, Frotscher M, Sakmann B (1997) Regulation of synaptic efficacy bycoincidence of postsynaptic APs and EPSPs. Science 275:213-215.
Marr D (1969) A theory of cerebellar cortex. J Physiol London 202:437-470.

10

Mazzoni P, Krakauer JW (2006) An implicit plan overrides an explicit strategy duringvisuomotor adaptation. J Neurosci 26:3642-3645.
McKee SP (1981) A local mechanism for differential velocity detection. Vision Res 21:491-500.
Miall RC, Weir DJ, Stein JF (1985) Visuomotor tracking with delayed visual feedback.
Neuroscience 16:511-520.

15

Miall RC, Weir DJ, Wolpert DM, Stein JF (1993) Is the cerebellum a Smith predictor. J Motor
Behav 25:203-216.
Michel C, Pisella L, Prablanc C, Rode G, Rossetti Y (2007) Enhancing visuomotor adaptation by
reducing error signals: single-step (aware) versus multiple-step (unaware) exposure to
wedge prisms. J Cognitive Neurosci 19:341-350.

20

Miyazaki M, Yamamoto S, Uchida S, Kitazawa S (2006) Bayesian calibration of simultaneity intactile temporal order judgment. Nat Neurosci 9:875-877.
Navarra J, Hartcher-O'Brien J, Piazza E, Spence C (2009) Adaptation to audiovisual asynchrony
modulates the speeded detection of sound. Proc Natl Acad Sci USA 106:9169-9173.
Norris SA, Greger BE, Martin TA, Thach WT (2001) Prism adaptation of reaching is dependent28

on the type of visual feedback of hand and target position. Brain Res 905:207-219.
Pawlak V, Kerr JND (2008) Dopamine receptor activation is required for corticostriatal
spike-timing-dependent plasticity. J Neurosci 28:2435-2446.
Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers
5

into movies. Spat Vis 10:437-442.
Pisella L, Michel C, Grea H, Tilikete C, Vighetto A, Rossetti Y (2004) Preserved prism
adaptation in bilateral optic ataxia: strategic versus adaptive reaction to prisms. Exp Brain
Res 156:399-408.
Ramachandran VS, Anstis SM (1986) The perception of apparent motion. Sci Am 254:102-109.

10

Rao SM, Harrington DL, Haaland KY, Bobholz JA, Cox RW, Binder JR (1997) Distributedneural systems underlying the timing of movements. J Neurosci 17:5528-5535.
Redding GM, Wallace B (2001) Calibration and alignment are separable: Evidence from prism
adaptation. J Motor Behav 33:401-412.
Redding GM, Wallace B (2003) First-trial adaptation to prism exposure. J Motor Behav

15

35:229-245.
Redding GM, Rossetti Y, Wallace B (2005) Applications of prism adaptation: a tutorial in theory
and method. Neurosci Biobehav Rev 29:431-444.
Smith WM, McCrary JW, Smith KU (1960) Delayed Visual Feedback and Behavior. Science
132:1013-1014.

20

Stetson C, Cui X, Montague PR, Eagleman DM (2006) Motor-sensory recalibration leads to an
illusory reversal of action and sensation. Neuron 51:651-659.
Stratton GM (1896) Some preliminary experiments on vision without inversion of the retinal
image. Psychol Rev 3:611-617.
Takahashi T, Konishi M (1986) Selectivity for interaural time difference in the owls midbrain. J
29

Neurosci 6:3413-3422.
Taub E, Goldberg IA (1973) Prism adaptation - Control of intermanual transfer by distribution ofpractice. Science 180:755-757.
Thach WT, Goodkin HP, Keating JG (1992) The cerebellum and the adaptive coordination of5

movement. Annu Rev Neurosci 15:403-442.
Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order
perception by exposure to audio-visual asynchrony. Brain Res Cogn Brain Res 22:32-35.
Wang HX, Gerkin RC, Nauen DW, Bi GQ (2005) Coactivation and timing-dependent integrationof synaptic potentiation and depression. Nat Neurosci 8:187-193.

10

Watanabe S, Hoffman DA, Migliore M, Johnston D (2002) Dendritic K+ channels contribute tospike-timing dependent long-term potentiation in hippocampal pyramidal neurons. Proc
Natl Acad Sci USA 99:8366-8371.
Weiner MJ, Hallett M, Funkenstein HH (1983) Adaptation to lateral displacement of vision inpatients with lesions of the central nervous system. Neurology 33:766-772.

15

Winter R, Harrar V, Gozdzik M, Harris LR (2008) The relative timing of active and passive touch.
Brain Res 1242:54-58.
Yao HS, Dan Y (2001) Stimulus timing-dependent plasticity in cortical processing of orientation.
Neuron 32:315-323.
Yin PB, Kitazawa S (2001) Long-lasting aftereffects of prism adaptation in the monkey. Exp

20

Brain Res 141:250-253.
Zampini M, Brown T, Shore DI, Maravita A, Roder B, Spence C (2005) Audiotactile temporalorder judgments. Acta Psychol (Amst) 118:277-291.

25

30

Figure legends

Figure 1(A) A schematic of experimental setting used in this study, (B) display locations (rectangles) with5

and without prism, and (C) time flow of a typical, single trial.
Figure 2
Expected learning curves after full adaptation to temporal delay when prism adaptation dependson (A) subjective timing or (B) physical timing. MD: Minimum delay, PD: Physical delay, DA:
Delay adapted, SD: Subjective delay.

10

Figure 3(A) Psychometric curves from all participants and (B) group-average psychometric curves forthe control (broken lines) and delay-adapted (solid lines) conditions. The circles along the curves(open and filled circles for the control and delay-adapted conditions, respectively) indicate how
many reaches were performed in individual time bins (the circles in the right-bottom corner

15

represent 20 reaches).
Figure 4(A) Reaction and (B) movement times during the prism experiment. Blue, black and red solidlines with markers denote mean values for the minimum-delay, physical-delay and delay-adapted
conditions, respectively. Thin broken lines indicate standard deviations. Gray shaded areas

20

indicate prism-adaptation trials (61-90).

31

Figure 5(A) Average learning curves from ten participants obtained in the minimum-delay (blue), thephysical-delay (black), and the delay-adapted (red) conditions. Thin broken lines indicate
standard deviations. There was no systematic change in the baseline trials, hence only the last5

thirty trial errors are shown to demonstrate how variable the movements were when no visualdisplacement was imposed. (B) Learning coefficients in the three conditions. In the box plots,
horizontal black lines indicate median values, and the bottom and top of boxes indicate 25th and75th percentiles. The whiskers cover 9th and 91st percentiles. *: P < 0.05, N.S.: not significant
according to Tukey’s test.

10

Figure 6
Box plots of learning coefficients for the delay-adapted and subjective-delay conditions. The left
box plot for coefficients in the delay-adapted condition is the same as in Figure 5B, and isincluded for comparison. *: P<0.05 according to t-test.

32

Figure 1

5

33

5

Figure 2345Figure 3

35

36

37

Figure 6
538