# Skill Evaluation from Observation of Discrete Hand Movements during Console Operation

Authors: Satoshi Suzuki; Fumio Harashima
Journal: Journal of Robotics
 (2010)

---

## Abstract

This paper focused on discrete movements of hand in reaching actions, which necessarily occur during machine
operation. The relationship between the performance of a console operation and the operator's reaching actions was investigated by applying Fitts' law and by examining a state transition of the operation. A remote operation experimental system was built using two radio controlled construction equipments, and the operator training process was analyzed empirically with those methods. The results showed the covariance of the fitting error to Fitts' law decreased as the operators' skill improved, although the fitting itself to the law was not sufficiently good. And it was confirmed that the covariance of difficulty index of the reaching action increased. These facts indicated that skill level of the discrete motion during operation can be estimated by investigating two types of the covariance.


---

## Body

## 1. Introduction

However sophisticated a machine is, training is necessary to familiarize a human user with its operation. To reduce user learning efforts, system developers must design an adequate human-machine interface that will be acceptable to a majority of people [1]. However, differences in individuals make it difficult to design such a versatile human-machine system. In addition, the human user has to adapt to the machine because the machine functions are fixed at the design stage. This situation, however, appears to contradict the very reason for the existence of a machine that was created to help humans and to make them comfortable.To resolve this paradoxical situation, it is necessary for advanced machines of the future to change its functions by analyzing differences in individual human skills. Therefore, the concept ofHuman Adaptive Mechatronics (HAM) was proposed [2, 3], and studies of such intelligent mechatronics, that not only support a human operator but also enhance his/her abilities, have been promoted globally [4, 5]. Although other research activities that consider human interactions, such as the COG project [6], the intelligent, and the Oxygen Project [7], have lead to various knowledge and fruitful results, they do not deal with individual skill sufficiently. The first reason for this is that the concept of skill itself is ambiguous. In some instance, the term skill indicates a primitive action, such as the skill-based behavior in Rasmussen's model [8]; in other instances, this term expresses a knowledge-based optimization, such as task planning and scheduling; therefore, the interpretations of skill are quite broad. In practical applications, the concept of skill has been modified on a case-by-case basis and used depending on the contents of each task. An example of an established skill analysis method is Therblig, which is a practical classification method for analyzing manual operations [9]. Therblig is, however, a type of a coding procedure through video observation and does not define the general concept of skill. Furthermore, it is difficult to use this method as an algorithm that can be implemented in a computer.The second difficulty is that skilled human actions arise from an excellent combination that involves perception, cognition, and motor control [10], as shown in Figure 1. This process has been studied in various fields, such as medicine, anatomy, brain science, cognitive science, psychology, control engineering, system science, and medical electronics; however, there are few studies from the human skill aspect. The ultimate elucidation of skill requires advanced study of the interaction among every process. Conversely, if it is the skill level of a user that is desired to be evaluated, then the detection of unnatural motions may be effective, because it can be expected that certain unskilled factor may induce such unnatural motion. The representative example is a jerky action in a hand reaching action, which is known as microslip. A microslip is a type of action hesitation we experience in everyday life, and this behavior includes tiny change in direction of hand reaching motion. Four types of microslip, trajectory change, hesitation, change of the hand shape, and touching objects, were defined [11]. Suzuki et al. reported that the microslip appears more frequently under high cognitive loads and complex circumstances [12]. In a previous study by the present authors, the relationship between task performance and microslips was investigated through a coffee-making test, and a strong correlation was confirmed statistically [13]. This result indicated that human skill in a manual operation can be estimated by observing the hand motions. Since the microslip is, however, found by a coding procedure through video observation by trained specialists, the result of the analysis depends on the skill level of the specialist and on individual differences. For the solution of this problem, the present authors tried to present an automatic detection method for microslip using visual processing, and the method could detect half of microslips that were found by human analysts [14]. However, the detection accuracy was insufficient. The reason appeared to come from low performance of the measurement system because the resolution of a standard camera (640×480 pixel, 30 fps) used in the experiment was not enough to detect tiny behavior of the microslip. A high-performance camera will improve the detection accuracy; however, it would be inadvisable due to the high cost.Figure 1
Sequential processing for a console operation.Existing studies concerning human characteristics of hand operation have mainly dealt withcontinuous time operation, including voluntary motion (e.g., [15]). Continuous time operation means that when a human operates a machine, he/she manipulates the machine by connecting the hand (or other body parts) to the machine's interface devices, as a result the human and the machine form a closed-loop system. The human is a nonlinear system, but can be treated as a continuous time dynamic system through a transfer function model such as “first-order” + “time-delay” when the operating point is limited [16]. The feedback-error-learning model [17, 18], which claims that a mechanism of human learning is the acquisition of an inner model of the controlled object's dynamics, is widely accepted. The crossover model, which expresses human flexibility and wide ranging adaptability, is also well known; it suggests that a human changes control characteristics so that the whole closed-loop transfer function of the human-machine system becomes a first-order system in a wide frequency band [19]. These models are often useful for designing the human-machine system. Actually, the present authors studied a hand positioning and its assisting method through a positioning task using a haptic interface device. A method for identifying operator control characteristics and evaluating the skill level was presented based on a similar transfer function model to the abovementioned human model, and an assistant control with a tiny force addition was proposed. Experimental analysis of participant data demonstrated that the assistant control could enhance operator learning [20].On the other hand, in the general machine operation,discrete movements including reaching actions inevitably occur because machines are typically equipped with several switches. Discrete movements are operator hand motions that change the contact conditions between the operator and the machine interfaces. An interesting discovery concerning discrete movement is Fitts' law [21]. Fitts' law is an experimental formula of the relationship between the time and distance of voluntary hand motions. This law is valid for a reaching action of other body parts, of a hand in the water, and of nonstraight line courses [22]. Fitts' law is used for evaluation of computer interface designs, such as a GUI [23–25] or a stylus [26]. As examples of application of Fitts' law to a machine operation, an evaluation of control laws for a haptic device [27] and an investigation of human cognitive processing [28] were reported. These studies, however, evaluated the computer system and not the human characteristics, because a hand reaching action is considered as a general movement already mastered by adult operators. To the authors' knowledge, evaluation of discrete movements in console operation, using Fitts' law, has not been reported so far, possibly because of the fact that the reaching actions during console operation are not always visual voluntary motions. The operator tends to look mainly at the controlled object or the environment, and moves the hand without looking at the target of the reaching action. The motion under such situation is not a visual voluntary motion, which is an assumption for applying Fitts' law.As mentioned above, an observation of a hand motion behavior or an identification of operator's control characteristics on continuous-time operation can give a way to estimate operator skill. Such skill estimation methods, however, require more complex and high-cost devices than an original operation console system unit; hence, their method is not always applicable for any type of operation system. Therefore, paying attention to simplification of additional measurement systems for the skill estimation, the present study was initiated to establish a method of quantifying the skill of a machine operator by focusing on the discrete motion of hand. Since an approach presented in this study utilizes operational timing of switches of a console and the hand trajectories are not measured, the measurement system is very simple. This paper investigates the following questions experimentally.Question 1.
Is Fitts' law valid for a reaching action that is not pure visual voluntary motion during console operation?Question 2.
How do differences in skill level emerge on matching to Fitts' law?Question 3.
Is the skill estimation of a console operation possible by using the discrete motions of hands?For experimental verification, a remote operation system was built that simulated the operation of actual construction equipment, and the process of learning its operation was analyzed based on Fitts' law. In order to validate a defined performance index for the experimental task, inching operations were investigated using a state transition matrix of operation commands. It is generally accepted that the frequency of inching operations indirectly reflects the performance of continuous time operation, since inching operations are interpreted as corrective actions for insufficient continuous time operations [29]. Then, a fitting to Fitts' law was investigated statistically, and aforementioned questions were verified.This paper is organized as follows: Section2 explains the preliminary computations for analyzing discrete hand movements; Section 3 explains the remote operation experimental system and the contents of the test; Section 4 explains several analysis methods of discrete movements that involve Fitts' law and the state transition matrix; Sections 5 and 6 present the results of the experiment and the analyses, respectively; and Section 7 provides the conclusions.


## 2. Data Treatment for Discrete Movement

Normal hand reaching actions have geometrically distinguishable starting and ending points. For console operations, it would be appropriate to separate discrete motions by timing the touching and releasing of the switches. Such segmentation is, however, inadequate, because there are switches that geometrically occupy the same position but have more than one function, such as a rotary or multidirectional switches. That is, reaching actions change depending on the intended operating direction, even for the same target switch. Therefore, reaching actions are classified based on differences in their functions and not on the switch layout.


### 2.1. Event and Sampling Counters

For automatic processing to find discrete movements from the measured data, two types of counters were introduced: a sampling counter and an event counter. The sampling countert∈𝒩 is obtained by dividing an actual time by a sampling interval Δ(=1/30[s]). The event counter τ∈𝒩 is incremented by one whenever a specific event occurs. The sequence {t[τ]} of the sampling counter t when the τth event occurs is defined as
(1){t[τ]}={Nκ{t}∣κ1(t-1)≠κ1(t)∨κ2(t-1)≠κ2(t)∨⋯∨κN(t-1)≠κN(t)},
where N is the total number of switching operation types, and κi∈0∪𝒩 indicates the ith switching mode. It is assumed that κ=0 when the operation is OFF or STOP. The sampling counts at which the switch status was changed were extracted using (1) from the logging data.


### 2.2. Extraction of Discrete Movements

One “reaching action” is a hand motion moving through the air to switch the console switches. Since the hand passes above the console and does not touch any switches during the reaching action, the ending point of an interval during which the operation mode is “non-STOP” is a candidate for the start timing of a reaching action. Using this property, both the starting and ending times, namedtrs and tre, are extracted from the sequence {t[τ]}. Because all operation modes at t=trs must be zero, and an operation by a nonzero command must be executed at the previous sampling time, trs can be found by
(2)trs={t∣κ1(t-1)+⋯+κN(t-1)≠0⋀κ1(t)=κ2(t)=⋯=κN(t)=0}.
Similarly, the ending points tre are found by
(3)tre={t∣κ1(t-1)=⋯=κN(t-1)=0⋀κ1(t)+κ2(t)+⋯+κN(t)≠0}.
Therefore, the time of a reaching action TR is obtained as
(4)TR=tre-trs.
Applying (2) and (3) alternately and sequentially to the measured data, multiple data that are the starting and ending points and the times of their reaching actions are obtained.


## 2.1. Event and Sampling Counters

For automatic processing to find discrete movements from the measured data, two types of counters were introduced: a sampling counter and an event counter. The sampling countert∈𝒩 is obtained by dividing an actual time by a sampling interval Δ(=1/30[s]). The event counter τ∈𝒩 is incremented by one whenever a specific event occurs. The sequence {t[τ]} of the sampling counter t when the τth event occurs is defined as
(1){t[τ]}={Nκ{t}∣κ1(t-1)≠κ1(t)∨κ2(t-1)≠κ2(t)∨⋯∨κN(t-1)≠κN(t)},
where N is the total number of switching operation types, and κi∈0∪𝒩 indicates the ith switching mode. It is assumed that κ=0 when the operation is OFF or STOP. The sampling counts at which the switch status was changed were extracted using (1) from the logging data.


## 2.2. Extraction of Discrete Movements

One “reaching action” is a hand motion moving through the air to switch the console switches. Since the hand passes above the console and does not touch any switches during the reaching action, the ending point of an interval during which the operation mode is “non-STOP” is a candidate for the start timing of a reaching action. Using this property, both the starting and ending times, namedtrs and tre, are extracted from the sequence {t[τ]}. Because all operation modes at t=trs must be zero, and an operation by a nonzero command must be executed at the previous sampling time, trs can be found by
(2)trs={t∣κ1(t-1)+⋯+κN(t-1)≠0⋀κ1(t)=κ2(t)=⋯=κN(t)=0}.
Similarly, the ending points tre are found by
(3)tre={t∣κ1(t-1)=⋯=κN(t-1)=0⋀κ1(t)+κ2(t)+⋯+κN(t)≠0}.
Therefore, the time of a reaching action TR is obtained as
(4)TR=tre-trs.
Applying (2) and (3) alternately and sequentially to the measured data, multiple data that are the starting and ending points and the times of their reaching actions are obtained.


## 3. Experimental Setup

### 3.1. Remote Operation of Radio Controlled Construction Equipment

A remote operation experimental system with radio controlled model construction equipment was devised as a human operation test system that requires discrete movements of the hands [30]. The purpose of the operation is a basic soil excavation work, as shown in Figure 2(a). The operator manipulated both an excavator and truck and was required to perform and to optimize task scheduling. Wireless cameras on the excavator and the truck-captured video images, displayed on monitors for the operator, as shown in Figure 2(b).Radio controlled construction equipments on a work area (a) and the operation console (b).
(a)(b)


### 3.2. Task

Figure3 shows a top view of the work area. The field size is 3.3m×2.4m and consists of motorable road and restricted areas. As the restricted area was 25 mm higher than the motorable road area, the operator could notice the restricted area when the machinery run on the restricted area. The field had three drilling sites and one unloading site. The excavator and truck were put at their starting points at the beginning of trial. The operator used console switches to move the machines to the drilling site, collected sample pieces with the excavator, loaded the pieces on the truck bed, and carried them to the unload site by the truck. The operator was instructed to collect different samples from the three drilling sites and to execute digging operations at each site only once. Only one type of sample was permitted to be loaded into the truck at a time. Unloading of the samples from the truck bed was done by vacuuming them after the truck arrived at the unloading site. One trial consisted of a set of all these operations. The total trial time was measured from the start to the time when the truck carrying its third samples reached the unloading site. Participants were asked to execute the tasks while maintaining the following conditions: (i)
the total trial time as short as possible,(ii)
the number of samples as large as possible,(iii)
the number of “difficulty” as small as possible.Figure 3
Overview of the work area.Here, “difficulty” indicates a limited access to the restricted area or the machines colliding. The comprehensive performance indexJ given below was used for later analysis
(5)J=1500Tt+S12,
where Tt and S are the total time of one trial and the total number of collected sample pieces, respectively. Considering that a maximum of Tt was about 1500 sec and a minimum of S was 12 for some participant who cooperated at the preliminary experiment, the constant values in (5) were decided.


### 3.3. Measurement System

To determine the positions of the excavator and truck in the field, the two machines were observed by a camera attached to the ceiling. The camera consists of a wide angle lens with an infrared filter. Eight infrared LED markers were placed on the work area for calibration of the optical distortion of the camera. The excavator and truck carried three and two infrared LED markers, respectively, and their positions and directions were computed using the detected positions of the LEDs. Data acquisition was performed by the Labview system with an image processing module and an AD/DA interface. After capturing the video image from the ceiling camera, the images of LED markers were extracted through the video processing, and their coordinate values were obtained by centroid computation. Compensation for optical distortion was performed offline using MATLAB. The operation of switches on the console was also recorded via the A/D and the DIO interfaces in the Labview system. The sampling frequencies of the video images and analog signals were 30 fps and 1 kHz, respectively.


## 3.1. Remote Operation of Radio Controlled Construction Equipment

A remote operation experimental system with radio controlled model construction equipment was devised as a human operation test system that requires discrete movements of the hands [30]. The purpose of the operation is a basic soil excavation work, as shown in Figure 2(a). The operator manipulated both an excavator and truck and was required to perform and to optimize task scheduling. Wireless cameras on the excavator and the truck-captured video images, displayed on monitors for the operator, as shown in Figure 2(b).Radio controlled construction equipments on a work area (a) and the operation console (b).
(a)(b)


## 3.2. Task

Figure3 shows a top view of the work area. The field size is 3.3m×2.4m and consists of motorable road and restricted areas. As the restricted area was 25 mm higher than the motorable road area, the operator could notice the restricted area when the machinery run on the restricted area. The field had three drilling sites and one unloading site. The excavator and truck were put at their starting points at the beginning of trial. The operator used console switches to move the machines to the drilling site, collected sample pieces with the excavator, loaded the pieces on the truck bed, and carried them to the unload site by the truck. The operator was instructed to collect different samples from the three drilling sites and to execute digging operations at each site only once. Only one type of sample was permitted to be loaded into the truck at a time. Unloading of the samples from the truck bed was done by vacuuming them after the truck arrived at the unloading site. One trial consisted of a set of all these operations. The total trial time was measured from the start to the time when the truck carrying its third samples reached the unloading site. Participants were asked to execute the tasks while maintaining the following conditions: (i)
the total trial time as short as possible,(ii)
the number of samples as large as possible,(iii)
the number of “difficulty” as small as possible.Figure 3
Overview of the work area.Here, “difficulty” indicates a limited access to the restricted area or the machines colliding. The comprehensive performance indexJ given below was used for later analysis
(5)J=1500Tt+S12,
where Tt and S are the total time of one trial and the total number of collected sample pieces, respectively. Considering that a maximum of Tt was about 1500 sec and a minimum of S was 12 for some participant who cooperated at the preliminary experiment, the constant values in (5) were decided.


## 3.3. Measurement System

To determine the positions of the excavator and truck in the field, the two machines were observed by a camera attached to the ceiling. The camera consists of a wide angle lens with an infrared filter. Eight infrared LED markers were placed on the work area for calibration of the optical distortion of the camera. The excavator and truck carried three and two infrared LED markers, respectively, and their positions and directions were computed using the detected positions of the LEDs. Data acquisition was performed by the Labview system with an image processing module and an AD/DA interface. After capturing the video image from the ceiling camera, the images of LED markers were extracted through the video processing, and their coordinate values were obtained by centroid computation. Compensation for optical distortion was performed offline using MATLAB. The operation of switches on the console was also recorded via the A/D and the DIO interfaces in the Labview system. The sampling frequencies of the video images and analog signals were 30 fps and 1 kHz, respectively.


## 4. Analysis Methods

Initially, the operation commands given through the levers and switches were discretized. In general, a machine console system is equipped with exclusive switches (e.g., the first and reverse gears of a shift lever) and combination switches (e.g., simultaneous operation using both the handle and acceleration pedal); hence, the operation commands were classified into independent and combination types. Analog commands, such as those of the handle and pedal angles, were translated into an integer numbered command. The classification details are explained in the following section.


### 4.1. Classification of Crawler Operation

Both the excavator and truck had crawler transporter systems in the present experiment. The right and left crawlers could be controlled independently based on the velocity commands, which are controlled by the angle of two sliders operated by hand, although the velocities of a real construction machine are controlled with foot pedals. Forward and backward motion, rotation, and pinwheeling were available. Because the velocity commands were transmitted by an analog signal, the analog data was converted into discrete values expressing a mode number in the present experiment. The crawler operation mode, which is termedκc, was obtained by checking inequality conditions concerning the velocities of the crawlers on both sides. Considering the effect of noise and drift in the analog signal, the mode κc was derived through the following algorithm, where vR and vL are velocity commands to the right and left crawlers, respectively, and both were normalized to the range [-1,1].Step 1.
Classification of translation and rotation(6)|vL-vR|≤ϵl→goStep2(translation),|vL-vR|>ϵl→goStep3(rotation).Step 2.
In case of translation,(7)κc={1(forward)vL,vR>ϵm,5(backward)vL,vR>-ϵm,0(stop)else.Step 3.
In case of rotation,(8)κc={3(leftpinwheel)vR>ϵm⋀vL<-ϵm,7(rightpinwheel)vR<-ϵm⋀vL>ϵm,goStep4else.Step 4.
In case of normal steering,(9)κc={2(leftf.steer)vR-vL>ϵl⋀vL>-ϵs,8(rightf.steer)vL-vR>ϵl⋀vR>-ϵs,4(rightb.steer)vR-vL>-ϵl⋀vL<ϵs,6(leftb.steer)vL-vR>-ϵl⋀vR<ϵs,0(stop)else,
where ϵ* indicates threshold parameters, and “f.” and “b.” are abbreviations for “forward” and “backward,” respectively.


### 4.2. Classification of Bucket Arm Operation

A bucket arm with a three-link mechanism is mounted on the superstructure of an excavator. The bucket arm is manipulated by one cross lever and one slide lever. Although there are several possible types of combinations of the cross lever layout and the bucket arm movements, the JIS (Japanese Industrial Standards) type was chosen. In the console of the present experiment system, commands to the bucket were given by the on-off switching of the cross lever. The operation modes were determined as follows:(10)κer:={0:stop,1:leftrotation,2:rightrotation},κea:={0:stop,1:armbend,2:armstretch},κeb:={0:stop,1:boomup,2:boomdown},
where κer,κea, and κeb are action modes of the superstructure's rotation, the arm, and the bucket, respectively. As all 27 (=33) combinations of the superstructure operation do not actually occur, an index for the bucket manipulation was assigned in order from the generated combinations. The first step is the computation of the combination index κeh by
(11)κeh:=(32)κer+(31)κea+(30)κeb.
The second step is to make the following new set peh by extracting different combination indexes after every κeh is computed from experimental data
(12)peh:={κeh,1,κeh,2,…},κeh,i≠κeh,j,∀i≠j.
Finally, describing peh,i for ith element of the set peh, the operation mode κes(t) at the time t is obtained by
(13)κes(t)={i∣peh,i=κeh(t)}.


### 4.3. Fitts' Law for Reaching Actions

Using the truck crawler modesκtc∈{0,1,…,8}, the excavator crawler modes κec∈{0,1,…,8}, and the excavator shovel operation mode κes∈{0,1,…,(27maximum)}, equation (1) yields the event counter sequence {t[τ]} as
(14){t[τ]}={{t}∣κtc(t-1)≠κtc(t)∨κec(t-1)≠κec(t)∨κes(t-1)≠κes(t)}.
The starting time trs and ending time tre were computed from the time sequences of κtc,κec, and κes by checking the status as follows:
(15)trs={t∣κtc(t-1)+κec(t-1)+κes(t-1)≠0,κtc(t)=κec(t)=κes(t)=0},tre={t∣κtc(t-1)=κec(t-1)=κes(t-1)=0,κtc(t)+κec(t)+κes(t)≠0}.
Then, the time of a ith reaching action was computed as TR[i]=tre[i]-trs[i], where [i] describes an index of the reaching actions.The present study used the following formula from Fitts' law:(16)TR=k1+k2x+e,(17)x:=log2(2DW),
where k1 and k2 are constant coefficients, D is the distance to the target of a reaching action, and W is the target width. x is called an index of difficulty, and depends only on geometric data of the distance and width. k1 and k2 reportedly relate to the weight of an object carried by hand and the speed of the human information processing, respectively. e is a fitting error by uncertainty against Fitts' law, and was newly introduced in the present study. e appears to change for several reasons, such as additional cognitive loading and the effects of prediction. Figure 4 shows the layout of the console switches and the coordinate system. Each planar coordinate value (x,y) of the switch position was registered as a table, and the distance D was computed as a Euclid norm between the starting point (x,y)s and the ending point (x,y)e, obtained through a lookup table from the operation commands κ(t)|t=rts and κ(t)|t=rte, respectively. The target width W is the size of lever grips, and it was obtained from a preregistered lookup table by checking the command κ(t)|t=rte. The coefficients k̂1 and k̂2 were estimated by a least square method with (16) using the multiple paired values {TR,x} from all the reaching actions that were obtained from the experimental logging data. Finally, the fitting error e[i] of the ith reaching action was computed as(18)e[i]=TR[i]-(k̂1+k̂2x[i]).
To remove abnormal values from the measured data, reaching actions with TR larger than 3 sec were ignored, because normal actions were completed in less than about 2 sec. Discriminant equations (15) might detect fake reaching actions of which movement distance is zero; hence, such timing data was removed for the later analysis of reaching actions. On the other hand, the inching was investigated by other procedure that counted the repeating manipulation of the console switches, as explained in the next section.Figure 4
Layout of console switches and its coordinate system.Essentially, Fitts' law concerns the reaching action of one hand; however, reaching actions with both hands were sometimes required in console operation, and the law cannot be strictly applied to such cases. In the present study, however, to apply Fitts' law for the analysis, an operation requiring both hands was simplified to a virtual one hand operation using a virtual point, which is a midpoint of two positions of switches grasped by both hands.


### 4.4. Skill Confirmation by Checking Inching Actions

Although the performance index described by (5) was defined based on a result of the preliminary experiment so as to be adequate for skill evaluation, an appropriateness of the index was not proved yet. Hence, an inching operation, that is recognized empirically as a criterion to estimate skill level on a machine operation, was used for the validation of the proposed performance index. Inching is a technique for slowly moving machinery a small amount by applying short pulses of commands. Unskilled operator tends to execute inching when he/she does not finish learning dynamics of the machinery. Hence, if correlation between an incidence of inching and value of index J is strong, reliability of the index would be enhanced. Hence, a state transition of the operation was computed from the logging data, and the inching was investigated by checking the state transition. The procedure is explained below.First, all operation modes were summarized into one comprehensive variableρ. Considering max(κtc)=max(κec)=8 and κ=0 for the stop operation, ρ was defined as
(19)ρ(τ):=1+κtc+(κec⋀1)(κec+8)+(κes⋀1)(κes+16).
Secondly, assuming the Markov process for the operation, the counting frequency of the transition from operation mode j to operation mode i, and putting this frequency on the (i,j) element in the state transition matrix, C'∈𝒩(8+8+max(eκs)-2)×(8+8+max(eκs)-2). The element c(i,j) of matrix C′ is computed by the following accumulation.
(20)c′(ρ(τ+1),ρ(τ))⟸c(ρ(τ+1),ρ(τ))+1,
as τ is increased from 1 to N, where N is the maximum event counter. As the matrix C' contains transitions involving stop commands, the modified state transition matrix C0¯ is computed by skipping the stop status (ρ=1) in (20). Finally, the transition probability matrix P is obtained as(21)P:=C0¯∑i,jallc0¯(i,j),
where c0¯(i,j) is (i,j)-element of C0¯. Inching can be interpreted as a transition from i-mode to the same i-mode as it is a repetition of the same action; hence, on-diagonal elements in the matrix P indicate ratio of the inching. The transition ratio of inching ξ is computed as ξ:=∑iallpi,i, where pi,j(∑i,jallpi,j=1) is an element of P.


## 4.1. Classification of Crawler Operation

Both the excavator and truck had crawler transporter systems in the present experiment. The right and left crawlers could be controlled independently based on the velocity commands, which are controlled by the angle of two sliders operated by hand, although the velocities of a real construction machine are controlled with foot pedals. Forward and backward motion, rotation, and pinwheeling were available. Because the velocity commands were transmitted by an analog signal, the analog data was converted into discrete values expressing a mode number in the present experiment. The crawler operation mode, which is termedκc, was obtained by checking inequality conditions concerning the velocities of the crawlers on both sides. Considering the effect of noise and drift in the analog signal, the mode κc was derived through the following algorithm, where vR and vL are velocity commands to the right and left crawlers, respectively, and both were normalized to the range [-1,1].Step 1.
Classification of translation and rotation(6)|vL-vR|≤ϵl→goStep2(translation),|vL-vR|>ϵl→goStep3(rotation).Step 2.
In case of translation,(7)κc={1(forward)vL,vR>ϵm,5(backward)vL,vR>-ϵm,0(stop)else.Step 3.
In case of rotation,(8)κc={3(leftpinwheel)vR>ϵm⋀vL<-ϵm,7(rightpinwheel)vR<-ϵm⋀vL>ϵm,goStep4else.Step 4.
In case of normal steering,(9)κc={2(leftf.steer)vR-vL>ϵl⋀vL>-ϵs,8(rightf.steer)vL-vR>ϵl⋀vR>-ϵs,4(rightb.steer)vR-vL>-ϵl⋀vL<ϵs,6(leftb.steer)vL-vR>-ϵl⋀vR<ϵs,0(stop)else,
where ϵ* indicates threshold parameters, and “f.” and “b.” are abbreviations for “forward” and “backward,” respectively.


## 4.2. Classification of Bucket Arm Operation

A bucket arm with a three-link mechanism is mounted on the superstructure of an excavator. The bucket arm is manipulated by one cross lever and one slide lever. Although there are several possible types of combinations of the cross lever layout and the bucket arm movements, the JIS (Japanese Industrial Standards) type was chosen. In the console of the present experiment system, commands to the bucket were given by the on-off switching of the cross lever. The operation modes were determined as follows:(10)κer:={0:stop,1:leftrotation,2:rightrotation},κea:={0:stop,1:armbend,2:armstretch},κeb:={0:stop,1:boomup,2:boomdown},
where κer,κea, and κeb are action modes of the superstructure's rotation, the arm, and the bucket, respectively. As all 27 (=33) combinations of the superstructure operation do not actually occur, an index for the bucket manipulation was assigned in order from the generated combinations. The first step is the computation of the combination index κeh by
(11)κeh:=(32)κer+(31)κea+(30)κeb.
The second step is to make the following new set peh by extracting different combination indexes after every κeh is computed from experimental data
(12)peh:={κeh,1,κeh,2,…},κeh,i≠κeh,j,∀i≠j.
Finally, describing peh,i for ith element of the set peh, the operation mode κes(t) at the time t is obtained by
(13)κes(t)={i∣peh,i=κeh(t)}.


## 4.3. Fitts' Law for Reaching Actions

Using the truck crawler modesκtc∈{0,1,…,8}, the excavator crawler modes κec∈{0,1,…,8}, and the excavator shovel operation mode κes∈{0,1,…,(27maximum)}, equation (1) yields the event counter sequence {t[τ]} as
(14){t[τ]}={{t}∣κtc(t-1)≠κtc(t)∨κec(t-1)≠κec(t)∨κes(t-1)≠κes(t)}.
The starting time trs and ending time tre were computed from the time sequences of κtc,κec, and κes by checking the status as follows:
(15)trs={t∣κtc(t-1)+κec(t-1)+κes(t-1)≠0,κtc(t)=κec(t)=κes(t)=0},tre={t∣κtc(t-1)=κec(t-1)=κes(t-1)=0,κtc(t)+κec(t)+κes(t)≠0}.
Then, the time of a ith reaching action was computed as TR[i]=tre[i]-trs[i], where [i] describes an index of the reaching actions.The present study used the following formula from Fitts' law:(16)TR=k1+k2x+e,(17)x:=log2(2DW),
where k1 and k2 are constant coefficients, D is the distance to the target of a reaching action, and W is the target width. x is called an index of difficulty, and depends only on geometric data of the distance and width. k1 and k2 reportedly relate to the weight of an object carried by hand and the speed of the human information processing, respectively. e is a fitting error by uncertainty against Fitts' law, and was newly introduced in the present study. e appears to change for several reasons, such as additional cognitive loading and the effects of prediction. Figure 4 shows the layout of the console switches and the coordinate system. Each planar coordinate value (x,y) of the switch position was registered as a table, and the distance D was computed as a Euclid norm between the starting point (x,y)s and the ending point (x,y)e, obtained through a lookup table from the operation commands κ(t)|t=rts and κ(t)|t=rte, respectively. The target width W is the size of lever grips, and it was obtained from a preregistered lookup table by checking the command κ(t)|t=rte. The coefficients k̂1 and k̂2 were estimated by a least square method with (16) using the multiple paired values {TR,x} from all the reaching actions that were obtained from the experimental logging data. Finally, the fitting error e[i] of the ith reaching action was computed as(18)e[i]=TR[i]-(k̂1+k̂2x[i]).
To remove abnormal values from the measured data, reaching actions with TR larger than 3 sec were ignored, because normal actions were completed in less than about 2 sec. Discriminant equations (15) might detect fake reaching actions of which movement distance is zero; hence, such timing data was removed for the later analysis of reaching actions. On the other hand, the inching was investigated by other procedure that counted the repeating manipulation of the console switches, as explained in the next section.Figure 4
Layout of console switches and its coordinate system.Essentially, Fitts' law concerns the reaching action of one hand; however, reaching actions with both hands were sometimes required in console operation, and the law cannot be strictly applied to such cases. In the present study, however, to apply Fitts' law for the analysis, an operation requiring both hands was simplified to a virtual one hand operation using a virtual point, which is a midpoint of two positions of switches grasped by both hands.


## 4.4. Skill Confirmation by Checking Inching Actions

Although the performance index described by (5) was defined based on a result of the preliminary experiment so as to be adequate for skill evaluation, an appropriateness of the index was not proved yet. Hence, an inching operation, that is recognized empirically as a criterion to estimate skill level on a machine operation, was used for the validation of the proposed performance index. Inching is a technique for slowly moving machinery a small amount by applying short pulses of commands. Unskilled operator tends to execute inching when he/she does not finish learning dynamics of the machinery. Hence, if correlation between an incidence of inching and value of index J is strong, reliability of the index would be enhanced. Hence, a state transition of the operation was computed from the logging data, and the inching was investigated by checking the state transition. The procedure is explained below.First, all operation modes were summarized into one comprehensive variableρ. Considering max(κtc)=max(κec)=8 and κ=0 for the stop operation, ρ was defined as
(19)ρ(τ):=1+κtc+(κec⋀1)(κec+8)+(κes⋀1)(κes+16).
Secondly, assuming the Markov process for the operation, the counting frequency of the transition from operation mode j to operation mode i, and putting this frequency on the (i,j) element in the state transition matrix, C'∈𝒩(8+8+max(eκs)-2)×(8+8+max(eκs)-2). The element c(i,j) of matrix C′ is computed by the following accumulation.
(20)c′(ρ(τ+1),ρ(τ))⟸c(ρ(τ+1),ρ(τ))+1,
as τ is increased from 1 to N, where N is the maximum event counter. As the matrix C' contains transitions involving stop commands, the modified state transition matrix C0¯ is computed by skipping the stop status (ρ=1) in (20). Finally, the transition probability matrix P is obtained as(21)P:=C0¯∑i,jallc0¯(i,j),
where c0¯(i,j) is (i,j)-element of C0¯. Inching can be interpreted as a transition from i-mode to the same i-mode as it is a repetition of the same action; hence, on-diagonal elements in the matrix P indicate ratio of the inching. The transition ratio of inching ξ is computed as ξ:=∑iallpi,i, where pi,j(∑i,jallpi,j=1) is an element of P.


## 5. Experiment Results

Participants were ten males from 21 to 23 years old. Written consent and ethical approval from them were obtained before the experiment. 2~4 trials a day were conducted for each participant with each of them repeating 10 trials for several days. Participants were instructed to improve their operational performance while considering the criterion described in (5).


### 5.1. Total Task Time and Performance Index

Figure5 shows the improvement in total time Tt. Nearly all of participants exhibited a monotonic decrease in time and their performances in shortening the total time were enhanced.Figure 5
Improvement of the total timeTt.The transition of the performance indexJ computed by (5) is shown in Figure 6. The tendency to increase was confirmed. Approximating each J curve to the regression line, the coefficient of gradient aJ, the y-intercept bJ, and the correlation factor rJ are summarized in the left three columns in Table 1. Since rJ=0.31 for participant E (labeled “(1a)”) appeared to be exceptionally small, the Smirnov-Grubbs test was applied to judge whether the value was an outlier or not. The test statistic was T(rJ=0.31)=2.7512 and the rejection region at significant level α=0.05 is G10(0.05)=2.176; hence, G10(0.05)<T(rJ=0.31) was satisfied, then rJ=0.31 was an outlier. Therefore, except for participant E, the approximation with the regression line is valid since the correlation factors rJ are sufficiently large (0.75 to 0.89). The coefficients of gradient aJ are all positive and large, at 0.44 to 0.84; hence, the data prove that all participants could enhance their performance in increasing performance index J.Table 1
Parameters of regression lines.
ParticipantaJbJrJaρbρrρA0.602.520.890.69−0.260.73B0.844.050.871.650.320.89C0.543.540.870.521.330.68D0.684.760.861.201.930.89E0.136.870.31(1a)0.205.450.22(1b)F0.633.710.881.070.510.87G0.724.530.791.351.540.79H0.673.110.871.18−0.990.82I0.513.120.820.790.330.80J0.445.900.750.793.750.79Figure 6
Transition of the performance indexJ.Figure7 shows the transition of the soil excavation efficiency ρ(=60S/Tt), which is the number of samples carried per minute to the final unloading site. From the first to sixth trial, nearly all of participants demonstrated an upward trend, and the curves for some participants were saturated after the sixth trial. The parameters of the regression line were computed similarly, and they are shown in the right three columns in Table 1. Because the correlation factors were sufficiently large (0.68 to 0.89), except for participant E (rρ=0.22, labeled “(1b)”), this approximation was also valid. Their gradient coefficients were positive and large (0.52 to 1.65); hence, the work efficiency was also successfully improved.Figure 7
Transition of the soil excavation efficiency.As an example, trajectories of the operated machines on the field for participant A, whose correlation coefficient ofJ was largest (rJ=0.89), are shown in Figure 8. The (a) and (b) graphs show the trajectories for the first and tenth trials, respectively. As the trajectories of tenth trial are smooth, with fewer zigzags than in the first trial, it can be confirmed intuitively that the operational skill increased.Trajectories of the excavator and the truck operated by participant A ((a) first trial, (b) tenth trial).
(a)(b)


### 5.2. Validation of Proposed Performance Index

Figure9 shows the transition ratio of inching ξ. Although most curves appear to decrease, it is difficult to observe an obvious tendency from the figure; hence, correlations between ξ and {Tt,J} were investigated. Table 2 shows the parameters obtained by the correlation analysis; the coefficients a* and R* denote the gradient coefficients and correlation factors, respectively. The data of participant E who was judged as an outlier by aforementioned analysis summarized in Table 1 was not shown in Table 2. Most of Rξ,T and Rξ,J were larger than 0.5. For all participants (other than E), the correlations between inching (ξ) and total task time (T) were strong ranging from 0.30–0.88. aξ,J=0.25 (labeled “(2)”) for participant D was, however, solely positive. Applying the Smirnov-Grubbs test to this value, aξ,J=0.25 was an outlier because the test statistic T(aξ,J=0.25)=2.3164 satisfied G9(0.05)(=2.110)<T(aξ,J=0.25). Hence, gradients aξ,J for all valid participants (other than D) were all negative. This result indicated that the incidence of inching (that tends to decrease as the skill improved) and the performance index J defined by (5) had negative correlation. Therefore, the performance index was adequate for evaluation of the skill level. In what follows, J is treated as an index showing skill level.Table 2
Correlation of the inching, total task time, and performance indexes.
ptp.aξ,TRξ,Taξ,JRξ,JA62.60.88−0.600.65B38.20.85−0.230.46C26.70.75−0.310.70D7.30.30(2)0.250.48F12.00.80−0.290.84G5.00.42−0.240.66H15.70.69−0.360.63I22.60.55−0.330.76J7.80.56−0.230.63Figure 9
Percentage transition of inching operation.


## 5.1. Total Task Time and Performance Index

Figure5 shows the improvement in total time Tt. Nearly all of participants exhibited a monotonic decrease in time and their performances in shortening the total time were enhanced.Figure 5
Improvement of the total timeTt.The transition of the performance indexJ computed by (5) is shown in Figure 6. The tendency to increase was confirmed. Approximating each J curve to the regression line, the coefficient of gradient aJ, the y-intercept bJ, and the correlation factor rJ are summarized in the left three columns in Table 1. Since rJ=0.31 for participant E (labeled “(1a)”) appeared to be exceptionally small, the Smirnov-Grubbs test was applied to judge whether the value was an outlier or not. The test statistic was T(rJ=0.31)=2.7512 and the rejection region at significant level α=0.05 is G10(0.05)=2.176; hence, G10(0.05)<T(rJ=0.31) was satisfied, then rJ=0.31 was an outlier. Therefore, except for participant E, the approximation with the regression line is valid since the correlation factors rJ are sufficiently large (0.75 to 0.89). The coefficients of gradient aJ are all positive and large, at 0.44 to 0.84; hence, the data prove that all participants could enhance their performance in increasing performance index J.Table 1
Parameters of regression lines.
ParticipantaJbJrJaρbρrρA0.602.520.890.69−0.260.73B0.844.050.871.650.320.89C0.543.540.870.521.330.68D0.684.760.861.201.930.89E0.136.870.31(1a)0.205.450.22(1b)F0.633.710.881.070.510.87G0.724.530.791.351.540.79H0.673.110.871.18−0.990.82I0.513.120.820.790.330.80J0.445.900.750.793.750.79Figure 6
Transition of the performance indexJ.Figure7 shows the transition of the soil excavation efficiency ρ(=60S/Tt), which is the number of samples carried per minute to the final unloading site. From the first to sixth trial, nearly all of participants demonstrated an upward trend, and the curves for some participants were saturated after the sixth trial. The parameters of the regression line were computed similarly, and they are shown in the right three columns in Table 1. Because the correlation factors were sufficiently large (0.68 to 0.89), except for participant E (rρ=0.22, labeled “(1b)”), this approximation was also valid. Their gradient coefficients were positive and large (0.52 to 1.65); hence, the work efficiency was also successfully improved.Figure 7
Transition of the soil excavation efficiency.As an example, trajectories of the operated machines on the field for participant A, whose correlation coefficient ofJ was largest (rJ=0.89), are shown in Figure 8. The (a) and (b) graphs show the trajectories for the first and tenth trials, respectively. As the trajectories of tenth trial are smooth, with fewer zigzags than in the first trial, it can be confirmed intuitively that the operational skill increased.Trajectories of the excavator and the truck operated by participant A ((a) first trial, (b) tenth trial).
(a)(b)


## 5.2. Validation of Proposed Performance Index

Figure9 shows the transition ratio of inching ξ. Although most curves appear to decrease, it is difficult to observe an obvious tendency from the figure; hence, correlations between ξ and {Tt,J} were investigated. Table 2 shows the parameters obtained by the correlation analysis; the coefficients a* and R* denote the gradient coefficients and correlation factors, respectively. The data of participant E who was judged as an outlier by aforementioned analysis summarized in Table 1 was not shown in Table 2. Most of Rξ,T and Rξ,J were larger than 0.5. For all participants (other than E), the correlations between inching (ξ) and total task time (T) were strong ranging from 0.30–0.88. aξ,J=0.25 (labeled “(2)”) for participant D was, however, solely positive. Applying the Smirnov-Grubbs test to this value, aξ,J=0.25 was an outlier because the test statistic T(aξ,J=0.25)=2.3164 satisfied G9(0.05)(=2.110)<T(aξ,J=0.25). Hence, gradients aξ,J for all valid participants (other than D) were all negative. This result indicated that the incidence of inching (that tends to decrease as the skill improved) and the performance index J defined by (5) had negative correlation. Therefore, the performance index was adequate for evaluation of the skill level. In what follows, J is treated as an index showing skill level.Table 2
Correlation of the inching, total task time, and performance indexes.
ptp.aξ,TRξ,Taξ,JRξ,JA62.60.88−0.600.65B38.20.85−0.230.46C26.70.75−0.310.70D7.30.30(2)0.250.48F12.00.80−0.290.84G5.00.42−0.240.66H15.70.69−0.360.63I22.60.55−0.330.76J7.80.56−0.230.63Figure 9
Percentage transition of inching operation.


## 6. Analysis Utilizing Fitts' Law

### 6.1. Fitting Results and Trend Analysis

First, results of the fitting to Fitts' law (16) were summarized in Figure 10. One graph shows a case of one participant. On each graph, transition of the identified coefficients k1 and k2 (solid lines) and the correlation factor r (dashed line) are shown. Averages of their values are also written in each graph. Since their correlation factors r were not so large at 0.24–0.53, the fitting result to Fitts' law was not good. The bias term k1, as drawn using blue solid line, changes widely. The transition of gradient coefficients k2, as drawn using red solid line, was flat compared with k1. Monotonic tendencies that synchronize with the number of trials in transition of k1 or k2 were not confirmed. Since the performance index J indicated an increasing tendency but k1,k2, and r did not show specific tendencies, it was surmised that the correlation between J and {k1,k2,r} was weak.Fitting results to Fitts' law: transition of the estimated coefficientsk1 and k2, and correlation factors r.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Second, individual differences concerning their coefficients were investigated through a correlation test between the Fitts' parameters and the performance index. That is, correlation between an averagek2, say ave(k2), and the gradient of J, aJ, was analyzed since an index that represents characteristics of total ten trials should be chosen for comparison of the individual difference. (Because, it appeared that k2 indicated characteristics of each individual since this value was almost constant at any trial, as shown in Figure 10.) Assuming that a data size is N and that a sample correlation coefficient is r, an uncorrelated test statistic T(r,N)(=rN-2/1-r2) matches the t distribution with (n-1) degrees of freedom; hence, the correlation between ave(k2) and aJ was tested using this property. In this case, T(r,10)=-0.4335 and t(10-2,0.025)(=2.306)<|T(r,10)| was not satisfied; hence, the correlation was not significant at the 0.05 level. Concerning ave(k1), t(8,0.025)=2.306 and T(r,10)=1.6289; hence, the correlation was not also confirmed because t(8,0.025)<T(r,10) was not satisfied. As a conclusion, there was no individual difference concerning estimated coefficients of Fitts' law.Thirdly, the authors investigated whether there was any common tendency to all participants in some statistical values. The survey was performed mainly by using average values of each participant since no individual differences concerning the Fitts' fitting were confirmed as demonstrated in an aforementioned correlation analysis. As a result, characteristic tendencies were found in changes of the fitting error and the index of difficulty. Figure11 shows the trends of the mean and covariance of the fitting error e for all eight participants. The transition of the average of all participants' mean values is indicated by a bold solid line in Figure 11(a). While the transition of the average was almost flat, the average of the covariance decreased as shown in Figure 11(b). Figure 12 shows the mean value and covariance of the index of difficulty x computed by (17). Similarly, each dotted line denotes the data of each participant, and the bold solid line is the average for all eight participants. It was confirmed that both the average and variance showed an upward tendency. These results suggest that the participants repeated simple and similar type of reaching actions at the beginning and were able to execute various different types of reaching actions as their skill improved.Transitions of the mean (a) and covariance (b) of the fitting errore (dotted line: each participant, solid line: average of eight participants).
(a)(b)Transition ofindex of difficulty ((a) mean, (b) covariance).
(a)(b)


### 6.2. Detail Analysis

Correlation analysis was applied to find clear relationship between the skill level and reaching actions because some tendencies in covariance values of theindex of difficultyx and the fitting error e were found. Below, these covariance values are described as cov(x) and cov(e), respectively. Tt and J were chosen as task performance factors at the correlation analysis. The computed correlation coefficients were summarized in Table 3. In the table, a description Ra,b expresses the correlation coefficient between a and b. The table shows that nearly all of the coefficients had the same sign except for participant G in RJ,cov(e) (labeled “(3a)”). Seeing the original transition for this participant, the covariance of e did not decrease uniformly. Applying the Smirnov-Grubbs test to this RJ,cov(e) with significance level α=0.1, the test value for participant G, {T(RJ,cov(e)=0.22)=1.8980}, was not outlier; however, the test value 1.898 is close to the significant point {G8(0.1)=1.909}. Considering a result of this test with the sole positive value in RJ,cov(e), participant G was judged as an exceptional case. Therefore, the tendencies were almost identical among the participants.Table 3
Correlation between Fitts' parameters and performance indexes.
ParticipantRT,cov(x)RT,cov(e)RJ,cov(x)RJ,cov(e)A−0.770.830.86−0.73B−0.470.420.24−0.56C−0.710.350.79−0.24F−0.850.240.54−0.11G−0.590.070.71(3a)0.22H−0.540.710.63−0.41I−0.780.550.80−0.59J−0.400.420.34−0.50As discussed above, because the learning processes of participants D, E, and G were exceptional compared with the other seven participants, the following conclusions about the majority of the participants are inferred.(i)
Covariance of theindex of difficulty increases as the number of trials increases. In other words, variation in reaching actions increases with the operator skill.(ii)
Covariance of the fitting error to Fitts' law decreases, although an accuracy of the fitting to the law was not sufficiently large.


## 6.1. Fitting Results and Trend Analysis

First, results of the fitting to Fitts' law (16) were summarized in Figure 10. One graph shows a case of one participant. On each graph, transition of the identified coefficients k1 and k2 (solid lines) and the correlation factor r (dashed line) are shown. Averages of their values are also written in each graph. Since their correlation factors r were not so large at 0.24–0.53, the fitting result to Fitts' law was not good. The bias term k1, as drawn using blue solid line, changes widely. The transition of gradient coefficients k2, as drawn using red solid line, was flat compared with k1. Monotonic tendencies that synchronize with the number of trials in transition of k1 or k2 were not confirmed. Since the performance index J indicated an increasing tendency but k1,k2, and r did not show specific tendencies, it was surmised that the correlation between J and {k1,k2,r} was weak.Fitting results to Fitts' law: transition of the estimated coefficientsk1 and k2, and correlation factors r.
(a)(b)(c)(d)(e)(f)(g)(h)(i)(j)Second, individual differences concerning their coefficients were investigated through a correlation test between the Fitts' parameters and the performance index. That is, correlation between an averagek2, say ave(k2), and the gradient of J, aJ, was analyzed since an index that represents characteristics of total ten trials should be chosen for comparison of the individual difference. (Because, it appeared that k2 indicated characteristics of each individual since this value was almost constant at any trial, as shown in Figure 10.) Assuming that a data size is N and that a sample correlation coefficient is r, an uncorrelated test statistic T(r,N)(=rN-2/1-r2) matches the t distribution with (n-1) degrees of freedom; hence, the correlation between ave(k2) and aJ was tested using this property. In this case, T(r,10)=-0.4335 and t(10-2,0.025)(=2.306)<|T(r,10)| was not satisfied; hence, the correlation was not significant at the 0.05 level. Concerning ave(k1), t(8,0.025)=2.306 and T(r,10)=1.6289; hence, the correlation was not also confirmed because t(8,0.025)<T(r,10) was not satisfied. As a conclusion, there was no individual difference concerning estimated coefficients of Fitts' law.Thirdly, the authors investigated whether there was any common tendency to all participants in some statistical values. The survey was performed mainly by using average values of each participant since no individual differences concerning the Fitts' fitting were confirmed as demonstrated in an aforementioned correlation analysis. As a result, characteristic tendencies were found in changes of the fitting error and the index of difficulty. Figure11 shows the trends of the mean and covariance of the fitting error e for all eight participants. The transition of the average of all participants' mean values is indicated by a bold solid line in Figure 11(a). While the transition of the average was almost flat, the average of the covariance decreased as shown in Figure 11(b). Figure 12 shows the mean value and covariance of the index of difficulty x computed by (17). Similarly, each dotted line denotes the data of each participant, and the bold solid line is the average for all eight participants. It was confirmed that both the average and variance showed an upward tendency. These results suggest that the participants repeated simple and similar type of reaching actions at the beginning and were able to execute various different types of reaching actions as their skill improved.Transitions of the mean (a) and covariance (b) of the fitting errore (dotted line: each participant, solid line: average of eight participants).
(a)(b)Transition ofindex of difficulty ((a) mean, (b) covariance).
(a)(b)


## 6.2. Detail Analysis

Correlation analysis was applied to find clear relationship between the skill level and reaching actions because some tendencies in covariance values of theindex of difficultyx and the fitting error e were found. Below, these covariance values are described as cov(x) and cov(e), respectively. Tt and J were chosen as task performance factors at the correlation analysis. The computed correlation coefficients were summarized in Table 3. In the table, a description Ra,b expresses the correlation coefficient between a and b. The table shows that nearly all of the coefficients had the same sign except for participant G in RJ,cov(e) (labeled “(3a)”). Seeing the original transition for this participant, the covariance of e did not decrease uniformly. Applying the Smirnov-Grubbs test to this RJ,cov(e) with significance level α=0.1, the test value for participant G, {T(RJ,cov(e)=0.22)=1.8980}, was not outlier; however, the test value 1.898 is close to the significant point {G8(0.1)=1.909}. Considering a result of this test with the sole positive value in RJ,cov(e), participant G was judged as an exceptional case. Therefore, the tendencies were almost identical among the participants.Table 3
Correlation between Fitts' parameters and performance indexes.
ParticipantRT,cov(x)RT,cov(e)RJ,cov(x)RJ,cov(e)A−0.770.830.86−0.73B−0.470.420.24−0.56C−0.710.350.79−0.24F−0.850.240.54−0.11G−0.590.070.71(3a)0.22H−0.540.710.63−0.41I−0.780.550.80−0.59J−0.400.420.34−0.50As discussed above, because the learning processes of participants D, E, and G were exceptional compared with the other seven participants, the following conclusions about the majority of the participants are inferred.(i)
Covariance of theindex of difficulty increases as the number of trials increases. In other words, variation in reaching actions increases with the operator skill.(ii)
Covariance of the fitting error to Fitts' law decreases, although an accuracy of the fitting to the law was not sufficiently large.


## 7. Conclusion

The potential of a skill evaluation method based on observation of human hand motion was verified experimentally as an early phase study to establish a method for a machine to evaluate human skill ability. Focusing on reaching actions during console operation, the discrete movements of hands were investigated based on Fitts' law. Using a remote operation system for a radio controlled model of construction equipment, the training processes of operators were observed and analyzed with the proposed methods. First, in order to validate a defined performance index for the experimental task, inching operations, which is considered to reflect performance of machinery manipulation, were investigated. As a result, validity of the defined performance index was verified since the correlation between the performance index and other three indexes (the incidence of inching, the total task time, and the soil excavation efficiency) was confirmed, and the performance index was declared a skill index for the present task.Secondly, results of the fitting to Fitts' law were analyzed. The correlation factors showed that the accuracy of the fitting was not sufficiently large for reaching actions during a console operation (answer of Question1), and there were no individual differences among all participants concerning the identified Fitts' parameters. It was, however, confirmed that the covariance of the fitting error to Fitts' law decreased and that the covariance of the index of difficulty of reaching actions increased as the skill improved (answer of Question 2). These facts indicated that the aforementioned two types of covariance were utilizable to evaluate operator skill. In conclusion, an investigation of perturbations from Fitts' law of discrete hand movements could evaluate the skill level for machine console operation (answer of Question 3).Next step in future study is an establishment of a formula or a method computing skill level value from variables that concerns a discrete motion. Based on the conclusion obtained in this paper, if subjective discussion is permitted, a formula such as1/cov(e)+cov(x) is a candidate to quantify the skill, where cov(e) and cov(x) are covariance of the fitting error to Fitts' law and of the difficulty index of reaching action, respectively. To determine adequate parameters and structure of the formula, sufficiently many evaluation data will be required for the statistical verification.Moreover, the environmental conditions and machines have to be considered to enhance the skill estimation, because an operator chooses the operation commands according to circumstances. With respect to such statuses, the present authors have been studying human command switching through a self-organizing map that includes environmental conditions [31]. In addition, human task scheduling with a discrete state transition model, such as a timed automaton, has been studied [32]. The network structures and timing of the state transition appear to be able to incorporate differences in the skill level. The authors are considering conducting studies on these topics in the future.



---

*Source: 967379-2010-05-16.xml*