ARTICLE IN PRESS

Ultramicroscopy 107 (2007) 25–39
www.elsevier.com/locate/ultramic

A procedure to analyze surface proﬁles of the protein molecules
visualized by quick-freeze deep-etch replica electron microscopy
Yoshitaka Kimoria,b, Yosuke Oguchic, Norihiko Ichised, Norio Babac, Eisaku Katayamaa,aDivision of Biomolecular Imaging, Institute of Medical Science, The University of Tokyo, Minato-ku, Tokyo 108-8639, Japanb
Department of Bioscience and Bioinformatics, Kyushu Institute of Technology, Iizuka, Fukuoka 820-8502, Japanc
Department of Electric Engineering, Kogakuin University, Hachioji, Tokyo 192-0015, Japand
Department of Visual Communication, Komazawa Women’s University, Inagi, Tokyo 206-8511, Japan
Received 6 October 2005; received in revised form 2 April 2006; accepted 12 April 2006

Abstract
Quick-freeze deep-etch replica electron microscopy gives high contrast snapshots of individual protein molecules under physiologicalconditions in vitro or in situ. The images show delicate internal pattern, possibly reﬂecting the rotary-shadowed surface proﬁle of themolecule. As a step to build the new system for the ‘‘Structural analysis of single molecules’’, we propose a procedure to quantitatively
characterize the structural property of individual molecules; e.g. conformational type and precise view-angle of the molecules, if thecrystallographic structure of the target molecule is available. This paper presents a framework to determine the observed face of theprotein molecule by analyzing the surface proﬁle of individual molecules visualized in freeze-replica specimens. A comprehensive set ofrotary-shadowed views of the protein molecule was artiﬁcially generated from the available atomic coordinates using light-rendering
software. Exploiting new mathematical morphology-based image ﬁlter, characteristic features were extracted from each image and stored
as template. Similar features were extracted from the true replica image and the most likely projection angle and the conformation of theobserved particle were determined by quantitative comparison with a set of archived images. The performance and the robustness of theprocedure were examined with myosin head structure in deﬁned conﬁguration for actual application.
r 2006 Elsevier B.V. All rights reserved.
PACS: 42.30.Sy; 68.37.Lp; 87.15.v
Keywords: Quick-freeze deep-etch replica electron microscopy; Heavy meromyosin; Mathematical morphology; Pattern recognition

1. Introduction
Direct observation, measurement and manipulation of
single protein molecules under functional conditions
constitute the fundamental means of ‘‘single-molecule
physiology’’ which should have been developed as thedue course in the studies of biological motor-proteins [1].
Since biological molecular machines should work certainly
in a stochastic manner, the absolute advantage of thismethodological concept is obvious, enabling us to characterize various properties of single molecules in action
without averaging. Thus, the innovating techniques asCorresponding author. Tel.: +81 3 5449 5512; fax: +81 3 5449 5419.

E-mail address: ekatayam@ims.u-tokyo.ac.jp (E. Katayama).
0304-3991/$ - see front matter r 2006 Elsevier B.V. All rights reserved.
doi:10.1016/j.ultramic.2006.04.012

above were readily accepted as one of the most powerfulexperimental system tools in biophysics and cell-biology ofvarious materials. Conventional methods of structural
biology, such as X-ray crystallography and NMR analysis,collects the data from a vast number of puriﬁed molecules
to give precise atomic coordinates after extensive averaging
spatially and temporarily. Though expediently classiﬁed
as a method of structural biology, electron microscopy(EM) occupies a unique position in terms that it candirectly visualize the structure of individual molecules
separately. Among various methods to prepare EMspecimen from the original biological macromolecules,
ice-embedding by quick-freezing is naturally the most ideal
technique to preserve the instantaneous structure of raw
material without staining. Together with the powerful

ARTICLE IN PRESS
26

Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39‘‘single-particle analysis’’ method, this cryo-EM techniquehave already taken an invaluable part to capture threedimensional structure of large protein assembly [2–6]. In
order to automatically pick up extremely low contrast
images of protein particles from the background, various
information theories were successfully applied to morereliable procedure [7,8] than before.
The most serious weakness of such near-ideal method,
however, is the fragility of the protein molecule against theelectron beam, which limits the dose of the available
irradiation. To earn enough information for structural
analyses, one should collect a huge number of images ofhighly puriﬁed protein molecules or supramolecularassemblies, classify their images according to the view
angles (and sometimes the conformational difference), thentake the average within each class. There, classiﬁcation is
the most crucial step for the success, sometimes resulting indifferent structures for the same molecule [9]. The situation
is even more problematic if the target molecule is ﬂexible
and takes multiple or contiguous conﬁgurations, such as in
the case for functioning motor-proteins. Each image might
be too noisy for the unambiguous classiﬁcation, and theessential structural information related to the function ofmolecular machines might be permanently lost if averaging
is done in an inappropriate manner. Some method tovisualize individual molecule with sufﬁcient S/N ratio is theabsolute requirement, just as the introduction of highsensitivity ﬂuorescence detector was previously so in single
molecule physiology.
Quick-freeze deep-etch replica EM technique [10] might
be one of the few strong candidates to satisfy suchconditions. Since freeze-replica specimens consist of heavy
metal and carbon, they are not only tolerant to extremely
high electron dose, but also give highly contrasted imagesof single protein molecules that can be easily recognizedeven by inexperienced observers. Considering that theimage contrast of this specimen is given in a similar mannerto conventional low-angle rotary-shadowing, it might benatural to assume that the ﬁnal images would bedeteriorated by the granularity of platinum/carbon coating. However, the grain size of the freeze-replica image isextremely ﬁne and is almost undetectable even by
examination of highly magniﬁed views of the original
micrographs. Usukura et al. [11] noted that conventionalrotary-shadowing can also give the images with substantially reduced grain-size, merely by shadowing performed
at low temperature. This supports the view that granularappearance of the background by conventional shadowing
at room temperature might be, at least partly, due to the
movement of original ﬁne metal particles to coagulate intolarge islets to deteriorate the apparent resolution [12].
Eskandari et al. [13] and Walz et al. [14] independently
estimated that freeze-replica specimen includes the structural information up to 1 nm or even better, in accordancewith our data. It was also shown by experimental andsimulation studies, that rotary shadowing provides much
more uniform metal coating [15] with higher resolution [12]than unidirectional shadowing. Thus, the spatial resolutionof quick-freeze replica samples is certainly much higher
than presumed from the granular impression of conventional low-angle shadowing.
By careful observation of such freeze-replicated imagesof the protein assembly, one could notice the presence ofsubtle light and dark gray-scale patterns on the surface of
the individual molecules (Fig. 1(a), [16]). Considering thatthe image contrast due to heavy metal shadowing, it iseasily supposed that the pattern might somehow reﬂect thesurface proﬁles of the protein covered with thin metal
deposit according to the incident beam angle of the metal
vapor from distant evaporation source. When the particles
like protein assembly were shadowed while rotating, theamount of the metal deposit at given position would be thefunction of the maximum inclination at that position along
the surface of the target. The image contrast of nearly
untilted views of the replicated specimen might arise mostly
according to the density gradation projected onto thehorizontal plane normal to the incident electron beam.
Thus, the grooves and hills having less and more metal
deposit would appear brighter and darker than thesurrounding area, respectively. If the above assumption iscorrect, we might be able to determine the approximate
view-angle of each single molecule in the EM ﬁeld, together
with the conformational type of the target, by comparison
with appropriately surface-rendered atomic model. As amatter of fact, we realized empirically that the delicate
surface pattern observed in freeze-replicated protein
molecules could be surprisingly well-presented by a lightrendering software if the shadowing conditions were
imitated as the illumination from multiple directions
around the target molecules [17] (Fig. 1(b)). A roughorientation of a particle was deﬁned by similarity
comparison with a movable model image of the target by
an interactive visualization tool, as the example shown in
Figs. 1(c) and (d) (Katayama, unpublished result; see also
Refs. [16,17]). We examined such recognition process by
self-analysis and found that a human brain judges theidentity of the image and the atomic model, not by a singletype feature but by multi-step procedure to discriminate
different kinds of features at the same time. We reached theidea that the recognition of outline shape, together with adelicate pattern visible on the molecule’s surface might be
the essential steps of the identiﬁcation process, that
prompted us to incorporate these steps into computerassisted recognition procedure. Our ﬁnal goal is toestablish the new methodology to automatically pick up
the images of individual protein molecules, whether single
or in complex, from replica EM ﬁeld and precisely
determine its conformational type and view angle, bycomparison with the list of conformers registered in thedata-base. Were the degree of matching not enough, we
might be able to partly modify the atomic model for better
matching. Along that way, the conﬁguration of eachprotein molecule under various functional states could
become within the reach of our analysis, if the proper

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

27

Fig. 1. (a) An example of quick-freeze deep-etch replica image of myosin head attached to actin-ﬁlament under rigor state. Note the presence of delicate
pattern on the surface of the molecule. (b) Schematic presentation of the process to generate the template images by virtual lighting of the solid atomicmodel from eight directions. (c) Surface-rendered model image of most likely conformation (near-rigor state; 1DFK) and the view angle which were
determined manually. The image was produced by homemade visualization tool, with subdomains represented in different colors. (d) The artiﬁcial imageof 1DFK in the same solid angle as above but generated by virtual lighting. Corresponding landmarks in (a), (c) and (d) are indicated by arrowheads.

method to identify the surface feature of the molecules
exists.
In this work, we present a novel algorithm dedicated to
identify the Euler angle and the conformational type ofgiven image of single molecule picked up from the ﬁeld of
quick-freeze replica specimens, taking heavy meromyosin
head structure as an example.
2. Procedure for comparison of real and computer generated
images2.1. Main conception
In order to identify the surface feature of the replica
image to the view from speciﬁc solid angle, we need tocompare the real image with a comprehensive set of
artiﬁcial images that possess the essential factors requested
for the recognition of surface proﬁle features. Since thepurpose of our analysis is merely to deﬁne the conﬁgura-tion and the view-angle of the speciﬁc molecules visualizedby shadowing, the outline shape and the position ofprominent surface features that possibly correspond to
subdomain arrangement are necessary and sufﬁcient
information to execute the following steps.
Fig. 2 shows a ﬂow diagram of our image processing
system that consists of two phases. In archiving phase, acomprehensive set of artiﬁcial images of the atomic model
were generated as templates, from 3D coordinates of theprotein molecule in question, by postulating its replica
images viewed from all the directions. To efﬁciently
prepare many artiﬁcial images, we employed commercial
light-rendering program, as the easiest and the fastest way
to collect the templates including the required structural
information for comparison and matching. More speciﬁcally, a rotational series of images were produced to cover
all the solid angles, assuming a virtual solid model of theprotein molecule as illuminated by a number of surrounding light-sources from the appropriate elevation angles.

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

28
PDB Data

AtomicAtomic Model
Model ofof
TargetTarget Particle
Particle

Generation
Generation ofof
Template
Template Images
ImagesFeatureFeatureExtraction
Extraction

DB

Archiving
Recognition Process

Retrieval
Replica EM
Image

Surface Pattern-based

Image
Image
Segmentation
Segmentation

FeatureFeatureExtraction
Extraction

Outline-based
Outline-based
Matching
Matching

Matching++
Calculation
Calculation ofof
Fitness
Fitness Score
Score

Matching
ResultFig. 2. Flow diagram of proposed procedure. See details in the text.

The outline shape and the surface proﬁle pattern of themolecule were extracted from each template utilizing a new
ﬁlter, as the characteristic features and then stored in the
database, together with the area surrounded by the outline.
In retrieval phase, true replica image was digitized and its
contrast was reversed for comparison with the model
images, because the metal-evaporated area appears darker
than the carbon background (quick-freeze deep-etched
images are conventionally presented after contrast reversal
so that they look like natural objects softly illuminated bymoonlight, to human eyes). The target particle wascropped from the ﬁeld, and the same characteristic features
as above were extracted in a similar manner. Because ofdifferent image-forming mechanism, the global appearances of real and artiﬁcial images were not close enough forthe direct correlation analysis. So, they were pre-processed
for texture-matching as described shortly.
The characteristic features of the target image were
quantitatively compared with those of archived templates
one by one, to ﬁnally determine its best-matched conﬁguration and the orientation in 3D space. Actual examination was carried out in two steps, ﬁrst by the outline, then,the surface feature pattern of the particles. At the sametime, the ﬁtness value of the selected molecule; theoverlapping area of two kinds of images after ﬁtting therespective patterns, was evaluated as the responsive score
for the ﬁnal decision.
Since the ﬁnal aim of the present procedure is todetermine the instantaneous structural states of given
single protein molecules captured in freeze-replica images,
we applied the same approach to assess the best-matching
conﬁguration for a given image as well, among the known
conformers.
2.2. Theoretical basis and development of new imageprocessing filter
In conventional computer-assisted image processing,image signals including some periodical or symmetricalfeatures are usually transformed to a sum of Fouriercomponents as a convenient form for digital computation.
Mathematical morphology is another approach moresuitable for the detection of the size and orientation of
the object, by taking the image as signals with various
geometrical features. In the present work, we designed

two kinds of ﬁlters based on mathematical morphology,
one to segregate the target particle from the background inEM ﬁeld and the other to extract protein molecule’s surface
pattern produced by shadowing the engraved relief proﬁle,especially the subtle details that could not be readily
recognized by non-expertise human eyes.
Below, the concepts of basic morphology operations andthe morphological ﬁlter are described.
2.2.1. Basic morphology operations
Mathematical morphology is based on set-theory concepts of shape [18]. An image can be represented by a set ofpixels. Morphological operations deal with two images
always as a set: an objective image to be analyzed and astructuring element (SE), which is analogous to the kernelof a convolution operation. Each SE has a shape as aparameter of the operation. The shape of the SEs in these
operators is chosen from either of square, circle, diamond,cross and line segment, etc. The shape and the size ofSE play a crucial role in such type of processing andmust be ﬂexibly adapted to the geometric properties ofthe objects to be processed. A flat (binary) SE is thedefault for the morphological algorithm, because of thecomputational advantage. In smoothing process, it is moresuitable than non-flat (gray-scale) SE to preserve someinter-region boundaries without blur. Hence, we concentrated on morphology using ﬂat SEs throughout our
processes.
The two basic morphological operations are dilation anderosion [19]. All other morphological operations aresequential combination of two basic operations; e.g. thetwo popular nonlinear ﬁlters are opening, an erosion
followed by a dilation, and a closing, vise versa,
respectively [20–25]. Haralick and colleagues [19] deﬁned
morphological operations as follows. Dilation and erosion
are calculated as minimum and maximum value given in
Eqs. (1) and (2), respectively. Here, the functions f and g
are a gray-scale image and an SE, respectively.
Dilation:ðf  gÞðX Þ ¼

max

ff ðX  ZÞ þ gðZÞg.

(1)

min

ff ðX þ ZÞ  gðZÞg.

(2)

X Z2F ;Z2G

Erosion:ðf  gÞðX Þ ¼

X þZ2F ;Z2G

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

F and G are the domains of f and g, respectively,
satisfying f: F-E, g: G-E. E is a Euclidian space.
Opening:f g ¼ ðf  gÞ  g.

(3)

Closing:g

f ¼ ðf  gÞ  g.

(4)

The opening operation eliminates upwardly convex parts
smaller than the SE from the objective signal. On thecontrary, the closing operation ﬁlls downwardly convex
parts smaller than the SE. Accordingly, we see that the
upwardly convex features of the objective signal are
extracted by the difference between the original signal
and the result of opening operation. Similarly, downwardly
convex features of the objective signal are extracted by theclosing operation.
2.2.2. Morphological smoothing filter
Image segmentation has been widely used [26,27] in EM
to extract the target particles form the background. Its
general algorithm is based on two properties; similarity anddiscontinuity, of gray-scale value. Since the segmentationof the region of interest (particle region) should be
separated from the rest by the surrounding edge, thealgorithm is to subdivide the image based on rapid changes
in gray level across the periphery. So, pre-processing of theoriginal image by smoothing is important. Appropriate
segmentation requires the edge structure be preserved, andthe brightness change to be smoothed within the particle
region.
Median ﬁlter is commonly used for smoothing. Though
it works effectively to remove shot or salt and pepper typesof noise, while preserving the edges (perfect step edges), it
degrades thin lines and features smaller than half the areaof window size, and is not appropriate for dense noise
removal. On the other hand, non-linear morphologicalsmoothing ﬁlter is useful for noise reduction of gray-scale
images corrupted by dense, low-amplitude, random orpatterned noise [28]. Opening and closing are two
operators commonly used to smooth the image in different
ways. Opening can remove peaks (positive impulse), andclosing can ﬁll pits (negative impulse). In order to remove
the positive and negative impulsive noise in certain images
at the same time, opening and closing are usually used as acascade. Maragos deﬁned open-closing (OC) and closeopening (CO) ﬁlters using the same SE [21]. OC or CO
ﬁlter gives better performance than median ﬁlter atremoving noise while preserving the edges [29,30]. Stevenson and Arce [29] demonstrated from the statistic viewpoint, that the distribution of the output of the CO
operator was biased toward higher values than the input
distribution. The output distribution of OC was analogous
to CO, but biased toward smaller values. Thus, smoothingby biased morphological operators may not be suitable toprocess the images where preservation of intensity levels is
crucial. Since morphological operators are deﬁned in29complementary pairs which are equally and oppositely
biased, one potential solution to the biasing problem is toaverage the complementary operators.
To achieve better smoothing, we employed a morphological smoothing ﬁlter called the LOCO ﬁlter [31] that is alinear combination of OC and CO. It was deﬁned asfollows:
1
1LOCOðf ; gÞ ¼ ðf g Þg þ ðf g Þg .
(5)
2
2
LOCO(f;g) gives an approximate smoothing with respect
to the SE and, unlike its constituent operators, is unbiased
with respect to gray level.
2.2.3. Rotational mathematical morphology
Morphological ﬁlters; in particular, openings and closings, are superior to linear ﬁlters for smoothing, but has acommon weak-point that a unique type of artifact might
appear in the output, depending on the shape of the ﬁlter
kernel (i.e. SE), due to remaining convolution effect of the
SE shape. This artifact could invoke a serious problem inprocessing boundaries. In conventional mathematicalmorphology, the orientation of SE is ﬁxed throughout
the whole process, making the operation of the image withthe complex directional features impossible. This artifact
could inﬂuence the next recognition process. Thus, anoutline along the boundary of a molecule region was usedin this study, as one of the features to be recognized. In theideal case, unnatural artifact might be relaxed, if a perfectcircle SE can be used, but it is not practical to expect that aperfect circle is drawn as a discrete image, nor being
expressed as a square if size of a SE becomes small, andline-segment SE might sometimes be necessary. Accordingly, we devised more universal technique to reduce thisartifact and proposed a new smoothing method utilizing
rotational mathematical morphology [32,33] to improvethe extracting accuracy of the outline around the molecule
region. In this strategy, multiple SEs rotated at an arbitrary
angle were used with bicubic interpolation. This processing
has the absolute advantage to suppress artifactual patterns,
preserving the structural details, regardless of the directionof SE in the image.
We also propose the LOCO ﬁlter involving rotational
mathematical morphology for Opening and Closing. Thismethod utilizes a square or a disk shape as the structureelement for smoothing. Assume that the straight angle(1801) is equally divided into N directions. Let gi denote therotation of a SE g by the degree of 180i/N, where i ¼ 0,
1,y, N1, and by let (k,l) denote 2D coordinates. In
Refs. [34,35], multiple SEs were applied to the image andthe maximum value among the opening operations was
taken as the result at the coordinate in question. The
maximum opening of f by g is deﬁned as
Y op ðk; lÞ ¼ max½Y 0 ðk; lÞ; Y 1 ðk; lÞ; . . . ; Y N1 ðk; lÞ,

(6)

where the image Yi is the image f opened by gi. In thisoperation, the SEs are ﬂoating freely under the surface ofARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

30

an image, after the operation, and any part of the surface
will be removed if none of these SEs ﬁts under the surface.
By the duality between opening and closing, the minimum
closing can be described in a similar way. The minimum
closing of f by g is deﬁned as
Y cl ðk; lÞ ¼ min½Y 0 ðk; lÞ; Y 1 ðk; lÞ; . . . ; Y N1 ðk; lÞ,

(7)

where the image Yi is the image f closed by gi. In thisoperation, the SEs are ﬂoating freely above the surface ofan image, and any downward part of the surface will be
ﬁlled in, if none of these SEs ﬁts above the surface.
The results of smoothing by the proposed method(LOCO ﬁlter based on rotational mathematical morphology) were compared with those by conventional LOCO
ﬁlter. It is notable that the internal structure and outline
shape of the object were well preserved during noise
reduction process. Fig. 3 shows an example of smoothing
by a LOCO ﬁlter. The original image (Fig. 3(a), 123  123
pixels) has 8-bit gray-scale. The deteriorated image (Fig.
3(b)) was made by adding a random noise with thebrightness from 0 through 255 to the original. Figs. 3(c)and (d) exhibit the results of smoothing with n  n squareshaped SEs of different size (n ¼ 5, 9). Fig. 3(c) shows theresult by conventional LOCO ﬁlter. More noises were
removed by increasing the SE size, but the internal objectof the circle and the diamond were deformed as inﬂuenced
by the shape of SE. On the other hand, smoothingaccording to our proposal gave apparently the better result(Fig. 3(d); N ¼ 8 for this case), considering the balance ofthe computer time and the performance. The structure of
the object in the original image was well-preserved, evenwith the large size of SE.
The recovery from a noisy image was evaluated bynormalized mean square error (NMSE) between a restored
image (f^) of size H  W and the original image (f) of size

H  W, that was deﬁned as
PH PW
2
^
i¼1
j¼1 ½f ði; jÞ  f ði; jÞ
NMSE ¼,
PH PW 2
i¼1
j¼1 f ði; jÞ(8)where i and j are the row and column position within f^ andf. It is obvious from Fig. 3(e) that the proposed method ismuch better than the conventional ones in NMSE. Fig. 3(e)show the NMSE value of smoothing with n  n squareshaped SEs of different size (n ¼ 3, 5, 7, 9, and 11 pixels).
Our new method not only showed a superior performanceto that of conventional one for all SE sizes but also was
effective for SEs of various shapes.
2.2.4. Top-hat transform
We designed a top-hat transform ﬁlter to effectively
extract delicate surface feature pattern of the target protein
molecule, since the isolation of convex gray-scale objects
can be accomplished by this kind of transform [36]. There,
the opening operation was anti-extensive, in the sense thatthe gray level of every pixel in the opened image was not
greater than that in the original image because the objects
with high gray level having supports smaller than that of
the SE should be eliminated by the opening operation. So,
the residual of the opened image from the original image
can be deﬁned as top-hat transform (TH):TH ¼ f  f g .

(9)

We succeeded to remove the artifact as above by the
opening operation with the rotational mathematicalmorphology, Eq. (6).
2.2.5. Segregation of the target particle region from thebackground and extraction of surface feature patternHere, we take a replica image of heavy meromyosin(HMM) as an example and show the actual process to crop

0.04
Conventional
Proposed

0.035
(c)

n=5

0.03

n=9
NMSE

(a)

0.025
0.02
0.015
0.01
0.005
0

(b)

(d)

n=5

n=9

(e)

3

5
7
9
SE Size n (Pixels)

11

Fig. 3. Example of morphological smoothing. Comparison of the effect of smoothing between proposed and conventional methods. (a) Original image(123  123 pixels). The radius of a circle object in the internal area is 15 pixels, and the side length of the diamond in internal area is 15 pixels. (b) The sameimage but corrupted by random noise. (c) Smoothed images by conventional LOCO ﬁlter with square-shaped SE. (d) Smoothed images by proposed
LOCO ﬁlter with square-shaped SE. (e) Artifacts due to ﬁltering with various size of square-shaped SE.

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

the molecule region from the background. HMM (molecular weight about 350 kDa) consists of three portions; two
globular subfragment-1s (S1, molecular weight about
110 kDa); whose carboxyl-terminals converge into rodshaped subfragment-2 (S2) moiety. Each S1 consists of two
regions, i.e. motor-domain (MD) and the lever-arm portion(LA). LA contains alpha-helical extension wrapped by two
kinds of light chains called essential light chain (ELC) andregulatory light chain (RLC), respectively (Fig. 4(a)).
Original replica image of HMM (Fig. 4(b)); details on thematerials and methods are described in Ref. [16], wasdigitized at 0.1 nm/pixel in 8-bit gray-scale and wassmoothed by the morphological ﬁlter. The target particle
was segregated from the background by setting theappropriate threshold as above. Attention was paid toachieve the greatest performance concerning three points;
always to keep the contrast between the region of target
particle and background high enough, to keep the brightness within the region of the target particle as uniform as
possible, and not to destroy the outline shape. Because thecontrast of a replica EM image was very high, distinction
between the region of the target particle and backgroundwas easy. However, areas with relatively low brightness
were sometimes present within the region of the target
particle. If binarization was carried out without appropriate smoothing, they could be regarded as ‘‘holes’’ in the
middle of the target particle. Here, we used the LOCO ﬁlter
based on rotational mathematical morphology in thesmoothing process, to preserve the effectiveness of the
proposed method. Fig. 4(b)(j) shows one example ofthe proposed procedure to extract two S1 regions of single
HMM independently. The original replica image (Fig. 4(b))
was ﬁrst processed with a square-shaped SE of theadequate size, observing above cautions. The side-lengthof SE was set at 21 pixels (i.e. 2.1 nm). Fig. 4(c) represents
the image after smoothing the relief narrower than 2 nm,
indicating that the proposed method to give reasonable
results. There, the foreground particle was well distin-

31

guished from the background while preserving the undulations in the contour shape of the particle, but the potential
cavities within the molecule region were evenly ﬁlled.
Fig. 4(d) shows the result after its binarization. The
foreground particle was appropriately separated from thebackground at certain value determined by Otsu’s method[37] which maximize the discriminant measure variable andenables us to select nonparametric and unbiased thresholdautomatically from the histogram. Then, the size of thearea occupied by each isolated object was calculated. If thesize of certain object was less than that pre-deﬁned, suchobject was discarded either as simple photographic noise,salt deposit called ‘‘eutectic’’ of replica specimens or the S2
region, and the gray level of those area was converted tobackground intensity. Thus, S1s region of the target was
extracted from the original image (Fig. 4(e)). Then, theoriginal image was cropped by a mask (Fig. 4(e)) to givethe protein area (Fig. 4(f)). Each S1 was separated at thehinge, the narrowest and branched position (marked byarrowhead in Fig. 4(f)), by setting the SE to the smallest
size wider than the hinge. All the area smaller than this SE
was eliminated by smoothing (Fig. 4(g)), because theOpening operation substituted the brightness of this region
with the background level (i.e. 0). The head moiety of
HMM was trimmed into each S1 (Fig. 4(i)) by a binarizedmask (Fig. 4(h)).
The surface feature was extracted from S1 images,
again by top-hat transform. Considering the spatial
resolution of the sample, the surface feature pattern,possibly reﬂecting the subdomain arrangement wasdeﬁned as a convex structure within 2 nm. Thus, the sidelength of the square-shaped SE was set as 2 (nm) and thesurface feature was extracted from the internal area ofthe particle, followed by linear contrast stretching(Fig. 4(j)). The hills and the grooves were expressed asbright and dark pattern, respectively. The parameter of therotational morphology (N) was set at 36, in all smoothing
processes.

Fig. 4. Segregation of the target particle from background, and extraction of surface feature pattern from that region. (a) Schematic drawing of HMM
composition. (b) Original replica image. (c) Smoothed image. (d) Binarized image of (c) with a threshold 77. (e) Mask to segregate molecule region.
(f) Extracted S1 region. (g) Smoothed image of (f). (h) Binarized image of (g) with a threshold 49. (i) Finally cropped S1 target image. (j) Surface pattern
extracted from the target.

ARTICLE IN PRESS
32

Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

2.3. Generation of template images and extraction ofcharacteristic features
There are a number of theoretical considerations on themechanism of image formation by rotary-shadowing[38–41]. Though it might be possible to simulate therotary-shadowed replica images by strict application ofabove theoretics, it would apparently take great amount ofcomputation time to generate a large number of template
images we would need. Since the information requested forour purpose is simply the position of subdomains on thesurface of the molecules, we took the advantage of much
faster approach that satisﬁes the minimum requirement.
Thus, we likened the images of metal shadowed molecules
to the atomic models illuminated by surrounding lights, sothat we could generate comprehensive set of artiﬁcial
images most easily by using commercial ray-tracing software. Several studies correlated electron microscopic
images of metal-shadowed samples to the images of thesolid-model illuminated by the lighting source [42,43].
Though the physical process to form two kinds of images
was apparently quite different from each other, preprocessing both of them with morphological ﬁlter enabled us toquantitatively compare the positions of hills and grooves
by matching the image texture. More details on this crucial
point will be discussed separately.
Atomic coordinates of the protein molecule were takenfrom the Protein Data-Bank (http://www.rcsb.org/pdb/
index.html) for 3D model building, and a set of images
representing the imaginary views of rotary-replicatedmolecules were produced utilizing light-rendering software
adapted for this speciﬁc purpose (‘‘Shade 6 professional
Rev. 19’’, e-frontier, Inc. that was supplemented withhomemade plug-in’s). Each atom was expressed as 4 nm
radius sphere with no specular reﬂection; i.e. ideal Lambertsurface. The effect of rotary shadowing was replaced byvirtual lighting of the atomic model from eight (maximum
number as default) equally spaced light sources (Fig. 1(b)),
surrounding the model. The orientation of the view-angle
was implemented as the rotation of the model from theoriginal co-ordinates by three angles (view angles; equivalent to Euler angles). Parameters f, y and c specify rotation
angles around the x-, y- and z-axes; and the projection wasmade parallel to the z-axis, so that the change in f and y
includes the movement along the line of sight, whereas thechange in c represents a planar rotation. Actual rotation of
the single conformer was carried out every 101 in solid
angles to generate comprehensive set of template images. By
this way, structural information of the molecular surface
features like grooves, and hills were automatically implemented into projected images of the solid model andexpressed as light and dark regions, respectively.
Correlation analysis is the standard method for thequantitative comparison among various images. The
mandatory process to obtain reliable results is to matchthe texture of all the images in question as much as
possible. Since the quality of artiﬁcial model-images

apparently differed from that of true replica images, weonce more utilized morphological ﬁlter for texture matching of two kinds of images of distinct entity. The featurepatterns of the artiﬁcial images were extracted in a similarmanner to that for target replica images. Those structural
data were stored in the database for later comparison. The
total number of templates stored in the database was about
15,000 for a single conformer.
2.4. Outline-based matching
The ﬁrst step of the recognition process in the retrieval
phase was the matching between the target particle image
and templates based on the outline shape. Fourier
descriptors or complex autoregressive models might bemost frequently used to represent object’s shapes. However,they request a priori information related to the ordering of
the points along the outline curve. Since the calculation of
the curvature requires the evaluation of second derivatives,
difﬁculties would arise especially when the objects are small
or noisy. Since we needed the descriptor sensitive to therotation of the object, they were not appropriate for ourpresent purpose. Peripheral feature is another descriptor
originally proposed for character recognition [44]. It is arotation variant which has been successfully applied toofﬂine signature veriﬁcation as an effective parameter torepresent the outline shape of object [45]. We employed thisfeature to compare the outline shape and angle of planarrotation in between target particle and the templates.
The peripheral feature of the molecule was deﬁned andprocessed as follows (Fig. 5(a)). Rectangular area circumscribing the whole molecule was taken and the brightnessof the pixels in the molecule region and the backgroundwere set as 1 (white) and 0 (black), respectively. The longside length of that region was normalized to 256 pixels.
Then the total area was expanded into 256  256 pixelsquare while keeping its gravity center.
Then square-shaped binary image was subdivided into
horizontal and vertical strips of single pixel-width. Withineach strip, the length between the edge of the virtual frame
and the ﬁrst black-to-white pixel jump was calculated. This
operation generates a 4  Ns peripheral feature vector (fouredges with Ns strips for each edge, e.g. N s ¼ 256, in thisstudy, for the full resolution). Let the feature vector bedenoted by v ¼ (v1, v2,y,v4  Ns)T where T stands fortranspose.
The similarity of the peripheral feature vector between
the target particle and the set of templates was measured as
normalized cross-correlation coefﬁcient. Let vp deﬁne aperipheral feature vector of the target particle and vt a
peripheral feature vector of a template, both with dimension K. Normalized cross-correlation coefﬁcient ros was
expressed by the formulaPK
i¼1 ðvpi  mp Þðvti  mt Þﬃ,
ros ¼ qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ(10)
ﬃqﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
PK
PK
2
2
ðv

m
Þ
ðv

m
Þ
pi
ti
p
t
i¼1
i¼1

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

Fig. 5. (a) Extraction of peripheral feature. The area in the square but
excluded by the molecule was sliced into strips along four sides, and thesequential length values of the strips were taken as the constituents of thefeature vector. Altogether, a 1024-dimensional feature vector was extracted.
(b) Surface pattern-based matching and the deﬁnition of ﬁtness. (b1) Surface
pattern of the target particle. (b2) Molecule region of the target. (b3) Surface
pattern of the template. (b4) Molecule region of a template. (b5) Correlation
map between the target and a template. Correlation coefﬁcient was expressedas brightness level. In this search window, maximum brightness; 255 was
assigned to maximum correlation coefﬁcient. (b6) Spatial relationship of amolecule region of the target and a template. The ratio of overlapping areain the total area was deﬁned as ‘‘ﬁtness’’.

PPK
where mp ¼ ð1=KÞ K
i¼1 vpi and mt ¼ ð1=KÞ i¼1 vti . Here,dimension K ¼ 4  N s . ros ranges from 1 to +1.
Similarity of the outline shape was evaluated by this score
and the templates with higher similarity were selected as
the candidates to proceed to the next step.
2.5. Surface pattern-based matching and calculation of
fitness score
In the second step, the surface pattern and the area ofmolecule region of selected templates were reloaded andcompared with those of the target, one by one. Searchwindow was created by the surface pattern image of thetarget particle, and its similarity with each template wasestimated as normalized cross-correlation coefﬁcient rsp.
The score, useful to ﬁnd the degree of matching of atemplate It (i,j) of size m  m0 in a search window Ip (i,j) ofsize M  M0 was deﬁned as
P m P m0
0
0
i¼1
j¼1 fI p ði; jÞ  m p gfI t ði; jÞ  m t g
q
ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
rsp ¼ qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ,
Pm Pm0P m P m0
0 2
0 2
i¼1
i¼1
j¼1 fI p ði; jÞ  m p g
j¼1 fI t ði; jÞ  m t g
(11)

33

P Pm 0where, m0 p ¼ ð1=mm0 Þ m
i¼1
j¼1 I p ði; jÞ is the averaged graylevel of corresponding
partof
the search window matrix,
P Pm 0and m0 t ¼ ð1=mm0 Þ mIði;jÞ is the counterpart of the
t
j¼1
i¼1
template matrix. (i,j) are the number of rows and columnsof the template matrix. For all (i,j)AM  M0 . rsp ranges
from 1 to +1.
However, the value of background area of target particle
and template was replaced by the averaged gray levelwithin the particle area.
Though cross-correlation function search is generally themethod of choice to accurately classify the particles
according to their projection angles, malclassiﬁcation could
sometimes happen. In fact, we often encountered the cases
where the appearance of a template resembled to the other,though the actual projection angle was completely different
from each other. This problem is inevitable, as long as onlya single feature was used in the search of view-based
system. So, we introduced the ﬁtness score that evaluates
the degree of overlapping of a molecule region of the target
particle and the template, and it was further integrated witha similarity of a surface feature pattern. The ﬁtness wasdeﬁned as follows (Fig. 5(b)). The molecule region of thetarget particle in the search window and an excluded region
were expressed by the signs P and P̄ (Fig. 5(b2)),
respectively. The region occupied by the molecule andthe excluded region for the template were expressed by
the signs T and T̄ (Fig. 5(b4)), respectively. Then theﬁtness wasFitness ¼ 1 

ðP̄ \ TÞ [ ðP \ T̄Þ
.
P

(12)

When matching procedure was ﬁnished, the ﬁtness at the
point should be unity if following conditions were satisﬁed.
P̄ \ T ¼ 0,

(13)

P \ T̄ ¼ 0.

(14)

Condition (13) means that a molecule region of atemplate must not protrude from a molecule region of
the target particle. Condition (14) deﬁnes that all of amolecule region of target being covered by a molecule
region of a template. Figs. 5(b1) and (b2) exhibit theexample of surface pattern of the target particle and theone within the molecule region. Fig. 5(b3) shows one of thetemplates, and Fig. 5(b4), its image in the molecule region.
A template window was shifted pixel by pixel across alarger search window, and in each position, the crosscorrelation coefﬁcient rsp between the template window
and the corresponding part of the search window wascomputed according to Eq. (11). Then, the ﬁtness wascalculated at the position where maximum cross correlation coefﬁcient (rsp_max) was obtained. Fig. 5(b5) indicatesthe map of correlation coefﬁcient between a search window
created by the target particle (Fig. 5(b1)) and a template(Fig. 5(b3)). The actual overlap of two molecule regions at
the best-matching position is shown in Fig. 5(b6). For
similarity retrieval of the target, each template was rated

ARTICLE IN PRESS
34

Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

according to the value of ﬁtness and the template with amaximum ﬁtness was selected as the best-matching
candidate for the view-angle of the target.
2.6. Determination of the conformational type of target
particle
Further attempt was made to determine the most likely
conformation of the target particle. The best candidate wasselected from each database generated for all available
conformers. Since they were determined according to theﬁtness, the outline shape and the size of the selected
templates should be similar to the target particle. On theother hand, the ﬁtting of the surface pattern to that outline is
an independent matter and should be evaluated separately
for more effective discrimination. According to this idea, we
employed a norm of the ﬁtness and the correlation coefﬁcientof the surface pattern, as an index for the judgment of the
conﬁguration. This coefﬁcient rconf was deﬁned as follows.
qﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ(15)
rconf ¼ ðrsp_max Þ2 þ ðFitnessÞ2 ,and FitnessX0. rconf
where, rconf is deﬁned in rsp_max
pﬃﬃX0
ﬃ
ranges from 0 to the maximum 2 when rsp_max and Fitness
are equal to unity.

3. Application and evaluation of proposed procedure
We applied the proposed method to quick-freeze deepetch replica images of HMM heads in putative transition
state as stabilized by simultaneous binding of ADP andinorganic vanadate. Since the structure of such complex
has been well characterized both by X-ray crystallography[46] and EM [16], it might give a good measure to examine
the feasibility of the novel procedure. Some of the original
images of sampled particles (Fig. 6(a)) [16] indicate
characteristic strongly kinked conﬁguration as reported,but the others (numbered 4, 6 and 8) not. The second row(Fig. 6(b)) shows the surface patterns extracted fromrespective images. Now, we will pick up the ﬁrst particle(Fig. 6(a1)) as an example and show the typical ﬂow along
the actual application procedure. At ﬁrst, a comprehensive
set of template images were generated with an angular stepof 101 from the atomic coordinates of the scallop myosin
S1 that was downloaded from the Protein Data-Bank(ID:1DFL). Then, the characteristic features were extracted from each image and stored for later comparison
with the target image. Next, similar features were extracted
from the target image and were subjected to the examination. Fig. 7(a) shows the result of similarity retrieval by theoutline-based matching. Templates were rated according to

Fig. 6. Electron microscopic images of quick-freeze deep-etch replica of heavy meromyosin in ADP/inorganic vanadate transition state, and the bestmatched templates for each conﬁguration as determined by the present procedure. Only one of two heads for each molecule is indicated. (a) Original
images. (b) Extracted surface pattern of the target particles. (c–e) Best-matched templates for three crystallographic conﬁgurations. (c) Vanadate-transition
state (d) Near-rigor state. (e) ADP state. According to the computer analysis, the right conformation was the vanadate-type for all of them and the Euler
angles of these templates were; no. 1: (14, 23, 6), no. 2: (3, 23, 54), no. 3: (13, 13, 134), no. 4: (56, 27, 59), no. 5: (8, 7, 64), no. 6: (121, 7, 19),
no. 7: (7, 3, 74), no. 8: (43, 15, 161).

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

the similarity to the target particle. Maximum distance wascalculated from maximum and minimum value of asimilarity and the normalized distance (ND) was deﬁned
as the difference between them.
ND ¼ ðmax_valueÞ  ðmin_valueÞ.

(16)

Tentative ﬁrst-stage candidates were selected from thetemplates in the database that had less than top 10%

35

threshold value of ND. In this case, 46 templates with asimilarity score higher than threshold ( ¼ 0.84) survived.
Then, the candidates were subjected to the second-stage
retrieval by the similarity of surface pattern-based matching. ND was calculated in the same way as before to set theadequate threshold for the ﬁnal selection. Fig. 7(b) shows
the distribution of view angles of the templates qualiﬁed tohave less than 5% (ND0–5%), 5%–10% (ND5–10%),

Fig. 7. Example of the results obtained in each step of our procedure. See text for the details. (a) Ranking of the templates according to outline-basedmatching for target particle No. 1. ND measured in ros is shown in a vertical axis. (b, d) Distribution of view angles of the templates selected by matchingof the surface pattern for no. 1 particle. (c, e) Distribution of view angles by surface pattern-based matching combined with the ﬁtness. (f) Finally selected
templates by ﬁner search. (g) Distribution of view angles of eight samples in Fig. 6(a). (h) Discrimination of the conformational type by rconf.

ARTICLE IN PRESS
36

Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

10%–20% (ND10–20%), and 20% or more (ND20% or more)of ND. We ﬁrst expected that the view-angles corresponding to highly scored templates would distribute within the
narrow range of projection angles, naturally because theclose view direction should give the analogous appearanceof the object. However, the selected angles were rather
randomly scattered and, at ﬁrst, we could not grasp thereason for such consequence. Fig. 7(d) shows the actual
template images that were rated as the top 8. The
appearance of the top-ranked candidates differed greatly
from each other and we could not recognize the similarity
among them, at a glance. We further sought for the reason
for the false similarity and found that the erroneous results
might be attributable mainly to the presence of highly
contrasted region that matched locally. This is exactly thereason why we introduced the new parameter ‘‘ﬁtness’’. We
reexamined the performance of the above procedure afterthat improvement, it is notable that, this time, thetemplates having higher similarity (less than 10% of total)
to the target particle formed a clear cluster; indicated bygray hatching (Fig. 7(c)), justifying the ﬁnal selection being
highly likely.
Though cluster-formation might be considered as anatural consequence of the closeness in the projection angle
in space, it turned out to be a good measure for theconsistency of the selection. Thus, the reliability for theﬁnal decision dramatically improved by the combination of
surface pattern matching and the ﬁtness scoring.
After conﬁrming the consistency of the approximate
view-angles of templates by clustering, ﬁner search was
carried out for more precise determination of the projection angle around the cluster (Fig. 7(c)). Templates wereprepared with 11 angular increment within a range of 7201
of the best-matched candidate in the preceding selection.
Fig. 7(f) exhibits ﬁnal top 8 templates and it is obvious thatthat we might reach very accurate results while saving thecomputation time, by the use of coarse-to-ﬁne matching
method.
We applied the same procedure to determine theprojection angle of the other particles (Fig. 6(a)) that were
picked up from the same replica specimens. Fig. 7(g) shows
the projection angles for eight particles with various
projection angles (Fig. 6(a)) all clustered around the most
likely view for each target. The gallery of best-matched
templates is exhibited in Fig. 6(c).
Analogous search was conducted to deﬁne the bestmatching conformation of the target. Scallop myosin S1 isthe only species whose crystallographic structures withvarious nucleotides were systematically determined as indifferent conﬁgurations. We took three structures; i.e. nearrigor state (1DFK) and ADP state (1B7T) and vanadatetransition state (1DFL), from the data-base and archived
their templates. Figs. 6(d) and (e) exhibit the best-rated
candidates for Rigor and ADP-type templates according tothe ﬁtness. Norm scores were distinctively high whentemplate’s conﬁguration matched with the true structure of
the target (Fig. 7(h)). Among 72 target particles examined,

the mean7SD (standard deviation) of rconf for various
conformations were as follows: 0.9570.048 for Vi/Vi was,
0.7370.098 for Vi/Rigor and 0.7470.089 for Vi/ADP.
Thus, all the samples were correctly determined as to be invanadate-transition state. Preliminary attempts with theother replica images containing different conformers
conﬁrmed the reliability of the ﬁnal judgment by ourprocedure. Thus, our novel methodology enables a correct
determination of two structural information; the conﬁguration and the view angle of the particles observed inquick-freeze replica images.
4. Discussion
4.1. Difference in the texture of replica and model images
and the crucial role of morphology filter
We conveniently utilized commercial light-rendering
software to generate a huge number of templates fromthe atomic models of the protein assembly, and succeeded
to identify the best- matching candidates corresponding togiven replica images. Though both processes might bear
nominally common process ‘‘shadow-formation’’, it isapparent that they cannot be related in a straightforwardmanner, since physical process of image formation is quite
different from each other. Rotary-shadowed replica specimens are prepared in dedicated vacuum-apparatus and areobserved under transmission electron microscope. As aresult of heavy-metal evaporation from certain elevation
angle, the thickness of the metal coat on the surface of thetarget particle, D, changes according to the incident angle
Y of the metal beam from the gun. This is expressed as
D ¼ D0 cos Y [47] where, D0 is the maximum thickness
obtained for Y ¼ 0. The observed density at given point of
the replica specimen depends on the path length along thetransmission electron beam. Since the image contrast
comes from the differential density of the point from thesurrounding area, the grooves ﬂanked by two opposing
slopes of large subdomain hills give high contrast line
proﬁle along globally globular surface of the moleculararchitecture. Next, suppose that we have the atomic modelof the target protein assembly, which is illuminated fromsurrounding light sources. If the size of the atom istactically set so that subdomain structures appear asclusters of spherical atoms that closely overlap to makeglobally globular structure, we would be able to generate
the model image, in which only the thicker grooves
between large clusters appear dark enough to be caught
by human eye, while the ﬁner ones do not. When such
images were adequately smoothed, only the large globular
subdomains surrounded by relatively thick grooves would
become prominent exhibiting characteristic subdomain
arrangement separated by dark streak pattern on thesurface of the atomic model. Thus, the position of the topsof the hills and the grooves in between would nicely matchwith those observed in replica images. However, there aredeﬁnitely some differences in the image properties because

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39of the different physical processes that might hamper theirstraightforward comparison. For instance, rotary-shadowed particles under transmission EM appear very brightat the edges of the target because of the long electron pathalong the standing metal coat. Though the surface patternof extreme edge is impossible to evaluate, structural
information along the edge is included as overall peripheral
shape. On the other hand, the delicate pattern of hills andgrooves in the major area of the molecule could easily bepicked up for comparison with artiﬁcial surface image ofthe model. Since the ﬁlter we devised can effectively
enhance the features of predetermined shape and scale,global difference of the texture of two kinds of images wereeffectively cancelled, enabling us to examine the precise
positions of surface features.
4.2. Robustness of our procedure against various
perturbations
Though the replica specimen by itself gives high-contrast
low-noise images, there are still remaining sources todeteriorate the quality of the ﬁnal target images. They aresalt and pepper noise or slight blurring in some of theimages, which are almost unavoidable during photographic
procedure, because we need extremely high-magniﬁcation
view of the target particles picked up from the original
negatives that had been already taken with very high direct
magniﬁcation. To evaluate the robustness of our method
against various perturbations, we examined how thebackground noise affects the ﬁnal decision by the37

computer. Instead of real target images, we employedartiﬁcial model images to which two kinds of perturbations
were added to various extents. Actually, six model images
were prepared as target, a control (T0, Te0 and Tg0, in
Fig. 8) and ﬁve others by addition of random noise plus
Gaussian blur (T1–T5, Te1–Te5 and Tg1–Tg5). Effects ofperturbations were examined at ﬁrst for view-angle determination. Templates were prepared by rotation of the atomicmodel with 5-degree increment around y-axis from theoriginal position (1801pyp1801). Fitness was scored foreach template to ﬁnd the best-matching one (Fig. 8(a) andenlarged view of its central part). Though the absolute ﬁtness
score declined according to the noise level, the ﬁnal answer tothe best-matching angle remained the same.
The validity of conformational search was assessed in asimilar manner but assuming the different views of thesame target structure (Vi-type) showing two extreme
appearances (‘‘extended’’, Te and ‘‘globular’’, Tg, inFig. 8(b)). Again substantial level of perturbation intentionally added to the target image did not affect the
ranking of the likely conformation, further conﬁrming therobustness of our procedure.
4.3. Upcoming application of our procedure to the othermaterials and occasions
In this paper, we proposed a novel template-matching
method to determine the view angle of protein image
observed by a quick-freeze deep-etch replica EM. Its
practical utility was conﬁrmed with the images of myosin

Fig. 8. Inﬂuence of added noise to ﬁnal recognition of view-angle and conformational state. An artiﬁcial image of Vi-type structure was used as a target.
(a) Fitness scores when target image was deteriorated by the addition of noise, so that peak signal-to-noise ratio (PSNR) became T1: 26.62 [dB], T2: 21.61[dB], T3: 18.38 [dB], T4: 15.94 [dB], and T5: 14.10 [dB], respectively. (b) Norm scores (rconf) of two different views when target was deteriorated so that
PSNR reduced to Te1: 26.39 [dB], Te2: 21.57 [dB], Te3: 18.33 [dB], Te4: 15.88 [dB], Te5: 14.14 [dB], and PSNR by degradation of Tg0; Tg1: 27.13 [dB], Tg2:
21.67 [dB], Tg3: 18.32 [dB], Tg4: 15.67 [dB], Tg5: 13.92 [dB].

ARTICLE IN PRESS
38

Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39

heads as test-materials. A great advantage of our procedureis that even a single view of the target particle can be thesubject of analysis.
Though template-matching is a common procedure
widely used in various ﬁelds of technology, its use in EM
seems to be rather limited. During the course of 3D imagereconstruction of the live cells by a tomographic approach,Baumeister’s team attempted to automatically classify the
species of macromolecular assemblies that co-existed, bytemplate-matching [48,49]. There, the target and thetemplates were both 3D structures reconstructed either bytomography or single-particle analysis, and its practicalapplication to general objects might not be as easy as in our
procedure dealing with 2D images. As a matter of fact, we
imagine that the extraction of the characteristic features of
the target and template particles according to ourprocedure might work effectively, even in their studies.
Since quick-freeze deep-etch replication can equallyvisualize the instantaneous structure of protein assemblies
in solution or in live cells [50,51], the proposed image
analysis procedure should be generally applicable todetermine the surface features of various protein assemblies
there. Such structural information might be a fruitfulsource to give us a hint of unknown molecular mechanismsof various biological events. Current procedure worked
almost perfectly, when the target particles were wellisolated from the others, but may not be so where two or
more particles intimately collide to form tightly fused
images. We are currently upgrading the algorithm so that
sufﬁciently reliable structure can be available even if a partof the total structure is occluded by the interaction with theother components. We expect that our procedure would
ﬁnd even larger ﬁeld of applicability after such improvement.
5. Conclusion
We proposed a procedure to deﬁne the delicate surface
pattern of the protein molecules captured by quick-freeze
deep-etch replica EM. We applied the procedure to thereplica image of HMM heads as test-materials. The system
successfully recognized the most likely projection angle andthe conformation of the sampled particles. In future, wemight directly analyze the gross structure of macromolecular assembly captured under functional states, wishfullyeven in live cells in situ.
Acknowledgments
We thank Dr M. Fujio for his kind reading andcomments on the manuscript and to Ms. T. Shiraishi forher expert technical assistance especially in replica preparation and photography. We are greatly indebted to Dr.
T. Kodama for his warm encouragement and useful advice
on this study. This work was partly supported by Grantsin-Aid for Scientiﬁc Research B and Grants-in-Aid forSpeciﬁc Areas to E.K. from the Ministry of Education,

Culture, Sports, Science and Technology of Japan. This
work was also supported by Technology Development
Program for Advanced Measurement and Analysis(Program-T) from Japan Science and Technology Agency
to E.K.
References[1] Y. Harada, T. Funatsu, M. Tokunaga, K. Saito, H. Higuchi, Y. Ishii,
T. Yanagida, Methods Cell. Biol. 55 (1998) 117.
[2] A. Abbott, Nature 417 (2002) 894.
[3] J. Frank, J. Zhu, P. Penczek, Y. Li, S. Srivastava, A. Verschoor,
M. Radermacher, R. Grassucci, R.K. Lata, R.K. Agrawal, Nature
376 (1995) 441.
[4] J. Frank, P. Penczek, R.K. Agrawal, R.A. Grassucci, A.B. Heagle,
Methods Enzymol. 317 (2000) 276.
[5] I.S. Gabashvili, R.K. Agrawal, C.M.T. Spahn, R.A. Grassucci,
D.I. Svergun, J. Frank, P. Pnczek, Cell 100 (2000) 537.
[6] M. van Heel, B. Gowen, R. Matadeen, E.V. Orlova, R. Finn,
T. Pape, D. Cohen, H. Stark, R. Schmidt, M. Schatz, A. Patwardhan,
Q. Rev. Biophys. 33 (2000) 307.
[7] T. Ogura, C. Sato, J. Struct. Biol. 136 (2001) 227.
[8] K. Mio, T. Ogura, Y. Hara, Y. Mori, C. Sato, Biochem. Biophys.
Res. Commun. 333 (2005) 768.
[9] C.W. Taylor, P.C.A. da Fonseca, E.P. Morris, Trends Biochem. Sci.
29 (2004) 210.
[10] J.E. Heuser, J. Mol. Biol. 169 (1983) 155.
[11] J. Usukura, Y. Nishizawa, A. Shimomura, K. Kobayashi,
T. Nagatsu, M. Hagiwara, Genes Cells 5 (2000) 515.
[12] W. Colquhoun, R. Sokol, J. Electron. Microsc. Tech. 3 (1986) 169.
[13] S. Eskandari, M. Kreman, M.P. Kavanaugh, E.M. Wright,
G.A. Zampighi, Proc. Natl. Acad. Sci. USA 97 (2000) 8641.
[14] T. Walz, P. Tittmann, K.H. Fuchs, D.J. Müller, B.L. Smith, P. Agre,H. Gross, A. Engel, J. Mol. Biol. 264 (1996) 907.
[15] H. Winkler, H. Gross, Scanning Microsc. Suppl. 2 (1988) 379.
[16] E. Katayama, J. Mol. Biol. 278 (1998) 349.
[17] E. Katayama, N. Ichise, N. Yaeguchi, T. Yoshizawa, S. Maruta,
N. Baba, Adv. Exp. Med. Biol. 538 (2003) 295.
[18] J. Serra, Image Analysis and Mathematical Morphology, Academic
Press, London, 1982.
[19] R.M. Haralick, S.R. Sternberg, X. Zhuang, IEEE Trans. PatternAnal. Mach. Intell. 9 (1987) 532.
[20] C.R. Giardina, E.R. Dougherty, Morphological Methods in Imageand Signal Processing, Prentice-Hall, Englewood Cliffs, NJ, 1988.
[21] P. Maragos, R.W. Schafer, IEEE Trans. Acoust. Speech Signal
Process. 35 (1987) 1153.
[22] P. Maragos, R.W. Schafer, IEEE Trans. Acoust. Speech Signal
Process. 35 (1987) 1170.
[23] P. Maragos, R.W. Schafer, Proc. IEEE 78 (1990) 690.
[24] J. Serra, Comp. Vision Graph. Image Process. 35 (1986) 283.
[25] S.R. Sternberg, Comp. Vis. Graph. Image Process. 35 (1986) 333.
[26] P.S. Adiga, R. Malladi, W. Baxter, R.M. Glaeser, J. Struct. Biol. 145(2004) 142.
[27] A.S. Frangakis, R. Hegerl, J. Struct. Biol. 138 (2002) 105.
[28] R.A. Peters, IEEE Trans. Image Process. 4 (1995) 554.
[29] R.L. Stevenson, G.R. Arce, IEEE Trans. Circuits Syst. 34 (1987)
1292.
[30] D. Schonfeld, J. Goutsias, IEEE Trans. Pattern Anal. Mac. Intell. 13(1991) 14.
[31] M.A. Schulze, J.A. Pearce, in: Proceedings of the IEEE International
conf. Acoust. Speech Signal Process, 1993, Vol. 5, p. 57.
[32] S.C. Pei, C.L. Lai, F.Y. Shih, Pattern Recogn. 31 (1998) 1127.
[33] F.Y. Shih, Y.T. Wu, Pattern Recogn. 37 (2004) 79.
[34] J. Song, E.J. Delp, Comp. Vis. Graph. Image Process. 50 (1990) 308.
[35] H. Kobatake, Y. Yoshinaga, IEEE Trans. Med. Imaging 15 (1996)
235.

ARTICLE IN PRESS
Y. Kimori et al. / Ultramicroscopy 107 (2007) 25–39[36] F. Meyer, Contrast feature extraction, in: J.-L. Chermant (Ed.),
Quantitative Analysis of A Microstructures in Material Sciences,
Biology and Medicine, Special Issue of Practical Metallography,
Riederer Verlag, Stuttgart, 1978, p. 374.
[37] N. Otsu, IEEE Trans. Syst. Man Cybern. 9 (1979) 62.
[38] P.R. Smith, J. Kistler, J. Ultrastruct. Res. 61 (1977) 124.
[39] L. Landman, J. Roth, J. Microsc. 139 (1985) 221.
[40] J.P. Chalcroft, Ultramicroscopy 16 (1985) 371.
[41] R. Guckenberger, Ultramicroscopy 16 (1985) 357.
[42] V.D. Vasiliev, O.M. Selivanova, S.N. Ryazantsev, J. Mol. Biol. 171(1983) 561.
[43] B. Tesche, H. Schmiady, Ultramicroscopy 16 (1985) 423.
[44] M. Umeda, IEICE Trans. Inf. syst. E 79D (1996) 401.

39[45] B. Fang, C.H. Leung, Y.Y. Tang, P.C.K. Kwok, K.W. Tse, Y.K.
Wong, IEE Proc. Vis. Image Signal Process. 149 (2002) 85.
[46] C.A. Smith, I. Rayment, Biochemistry 35 (1996) 5404.
[47] R. Krbecek, C. Gebhardt, H. Gruler, E. Sackmann, Biochim.
Biophys. Acta 554 (1979) 1.
[48] J. Walz, D. Typke, M. Nitsch, A.J. Koster, R. Hegerl,
W. Baumeister, J. Struct. Biol. 120 (1997) 387.
[49] J. Böhm, A.S. Frangakis, R. Hegerl, S. Nickell, D. Typke,
W. Baumeister, Proc. Natl. Acad. Sci. USA 97 (2000) 14245.
[50] E. Katayama, H. Funahashi, T. Michikawa, T. Shiraishi, T. Ikemoto,
M. Iino, K. Mikoshiba, EMBO J. 15 (1996) 4844.
[51] E. Katayama, T. Shiraishi, K. Oosawa, N. Baba, S.-I. Aizawa,J. Mol. Biol. 255 (1996) 458.