1

Integrated Computer-Aided Engineering (2020) 1–15
IOS Press

Image-based Textile Decoding
Siqiang Chen a,b , Masahiro Toyoura b,∗ , Takamasa Terada b , Xiaoyang Mao a,b , Gang Xu aaSchool of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou 310018, Zhejiang,
Chinab
Department of Computer Science and Engineering, University of Yamanashi, Takeda 4-3-11, Kofu,
Yamanashi, Japan

Abstract. A textile fabric consists of countless parallel vertical yarns (warps) and horizontal yarns (wefts). Whilecommon looms can weave repetitive patterns, Jacquard looms can weave the patterns without repetition restrictions. A pattern in which the warps and wefts cross on a grid is defined in a binary matrix. The binary matrix candefine which warp and weft is on top at each grid point of the Jacquard fabric. The process can be regarded as
encoding from pattern to textile. In this work, we propose a decoding method that generates a binary pattern froma textile fabric that has been already woven. We could not use a deep neural network to learn the process based
solely on the training set of patterns and observed fabric images. The crossing points in the observed image were
not completely located on the grid points, so it was difficult to take a direct correspondence between the fabricimages and the pattern represented by the matrix in the framework of deep learning. Therefore, we propose amethod that can apply the framework of deep learning via the intermediate representation of patterns and images.
We show how to convert a pattern into an intermediate representation and how to reconvert the output into apattern and confirm its effectiveness. In this experiment, we confirmed that 93% of correct pattern was obtainedby decoding the pattern from the actual fabric images and weaving them again.
Keywords: Textile, fabrication, intermediate representation, pattern decoding, Jacquard fabric

1. Introduction
With the rapid development of modern production techniques, the textile industry has undergone rapid changes. Intelligent weave machines
have been introduced to make textiles convenient
to produce and diversified in style and pattern.
In addition, we have a clear need for personalized
customization and on the whole, are not satisfied
with traditional, uniform styles. Many of the an* Publised

in Integrated Computer-Aided Engineering. The final publication is available at IOS Press
through http://dx.doi.org/10.3233/ICA-200647. Corresponding authors: Masahiro Toyoura, Department ofComputer Science and Engineering, University of Yamanashi, Takeda 4-3-11, Kofu, Yamanashi, Japan. Email: mtoyoura@yamanashi.ac.jp. Gang Xu, School ofComputer Science and Technology, Hangzhou Dianzi
University, Hangzhou 310018, Zhejiang, China. E-mail:gxu@hdu.edu.cn.

cient textiles exist as real objects only, and thepatterns are still valuable and not outdated. If wewant to reproduce such ancient fabrics, we needto analyze the patterns. Technicians can analyze
the pattern by observing the fabric using a microscope and recording the crossing state of each yarn
by disassembling them, but this is time consuming
and tedious. At the same time, the original textile will be destroyed. Therefore, it is not a good
way to solve this problem and we need to find anovel technique for automatically extracting thepattern. This would be useful for both reproduction and generating new patterns.
In this work, we focused on Jacquard fabric in
which the crossing of warp and weft was specified
by a binary matrix pattern. In contrast to ordinary looms, which only weave repetitive patterns,
Jacquard looms can weave free patterns without
repetition restrictions. With Jacquard fabric, thewarp and weft can be defined for each crossing

0000-0000/20/$00.00 c 2020 – IOS Press and the authors. All rights reserved

2

S. Chen et al. / Image-based Textile Decoding

point; the resulting fabric is not composed of uniform pattern regions. If the fabric is made withsmall repetitive patterns, the analysis can be done
manually without much effort, but for Jacquard
fabrics, the analysis of large patterns that makeup the entire fabric is required, and manual analysis requires much effort. The segmentation basedon pattern uniformity does not work for analyzing
Jacquard textile patterns.
Jacquard textiles are often used in valuable fabrics such as traditional costumes and neckties, andthere is high industrial value for the encoding anddecoding of the patterns. Therefore, in this work,
we focused on Jacquard fabric, in which the crossing of the warp and weft was specified by a binary
matrix pattern.
The warp and weft yarns were dyed different colors and the pattern defined how the yarns crossed.
The binary matrix pattern was defined as a binary image. For example, when the black yarn
was above the white yarn, the crossing point was
displayed in either black or white. We input thebinary pattern into the weaving machine to produce the fabric. The Jacquard loom was able toaccept the over-under relationship for the individual crossing points. Modern Jacquard looms canweave a coded pattern by loading a file of the pattern [1].
The appearances of images are different fromeach other, even at the same crossing point wherea weft overlaps a warp. Fig. 1 shows zoomed-in images of the crossing point area in an observational
image. Conventional template matching was notable to estimate the exact positions of the crossing
points.

Fig. 1. The variety of crossing points makes the problem
difficult. Conventional template matching cannot deal withthe crossing points.

Where there are many pairs of fabric images andtheir corresponding binary patterns, deep neuralnetworks (DNNs) can decode unknown binary patterns from a corresponding fabric image. However,
DNNs cannot directly output a binary patternwith thousands of crossing points from a fabric image with millions of pixels. To solve the problem,
we introduced an intermediate representation thatbridged the pair and enabled us to output the intermediate representational patterns by a DNN. In
addition, to convert a fabric image into the intermediate representation, and the intermediate representation into a binary pattern, we built a practical method for the conversion. We introduced intermediate representation images because we expected the images to directly represent the likelihoods which are the crossing points. In addition,
due to insufficient sample size, complex networks
that require a large amount of training data are
not likely to be trained well. A more complex network, like an end-to-end network which directly
answers the positions of crossing points, trained bynumerous samples could provide accurate results.
We believe that the objective of this study, textile pattern decoding, has not been done to thebest of the authors’ knowledge in previous work.
Therefore, we have not found any previous methodthat allows direct comparison of accuracy. Instead,by changing the parameters and kinds of filters invarious ways, we confirmed the best possible settings for the current situation.
The contributions of this paper are as follows:
1. Introduction of an intermediate representation for textile decoding tasks;
2. Interface design for manual tagging of crosspoints on an observed textile image (Section
3.1.1);
3. Proposal of a pre-process for converting afabric image to an intermediate representation (Section 3.1.3); and4. Proposal of a post-process for converting an
output intermediate representation pattern
into a regular binary matrix pattern (Section
3.3).
Fig. 2 shows the overview of our proposed
method. Section 2 introduces the background andrelated work, and Section 3 shows a method of converting a fabric image into an intermediate representation pattern as well as an intermediate representation pattern into a binary pattern. We also

S. Chen et al. / Image-based Textile Decoding

3
Intermediate
representation

Image
representation

Pattern
representation(2) Manual labeling(3) Integration(4) Training
Fabric image

Input image
Input

Preprocessing

Neuralnetwork

Labeled image

Input pattern

Output image

Result pattern

Output(5) Execution

Fabric observation
Fabric image

Input image(1) Pre- processing(6) Post- processing

Fig. 2. The overview of our proposed method including the steps of (1) Pre-processing; (2) Manual labeling; (3) Integration;(4) Training; (5) Execution; and (6) Post-processing.

describe the details of DNN configurations. In Section 4 we present experimental results and in Section 5, a summary.

2. Related Work2.1. Pattern Creation by Computer Support
Depending on the local density of the crossing points as well as the colors of yarn appearing
on the top, a textile pattern brings different levels of brightness. Inappropriate patterns result in
misaligned grid points and partial fabric stiffness.
Considering many conditions, and through trial
and error, textile patterns have been manually created since ancient times. In recent years, systems
for designing textile patterns with the support ofcomputers have been proposed and it has become
possible to create more complex patterns [2,3,4].
Toyoura et al. [5] proposed a dithering method
for reproducing smoothly changing tones and finedetails of natural images on woven fabric, focusing on representing gray scale images by using twocolors of warp and weft yarns. The weaving pattern is generated by binarizing the input image using dither masks. The step dithering method alternately places values of 0 and 255 at given intervals
in each row of the dither mask such that there is atleast one cross point of warp and weft yarns in thespacing in the resulting fabric. Within these intervals, the threshold increases from 0 to 255 or from255 to 0. This forms a stepping structure up anddown. The resulting binary image reproduces thebrightness of the input image while limiting thenumber of crossing points in the weave pattern.
By modeling and rendering 3D CG (computer
graphics) from a textile pattern, the woven result
can also be predicted. By adding a physical collision detection, it is possible to reproduce the appearance of fabrics by CG. Users will be able tomodify the pattern without actually weaving it.
There have been many studies on computer
graphics to generate photo-realistic fabric images
from defined patterns [6,7,8,9]. The images obtained by observing yarns, fabrics, and 3D yarn
data obtained by CT are used to generate realistic
images. 3D fabric models are constructed from thedata. These are the opposite of directional studies,as our proposed method restores patterns from actual fabrics. In this work, we aimed to output apattern of weaving, which would be a complementary work.
2.2. Pattern Analysis for Fabric Images
Due to industrial demand, the pattern analysis by DNNs has been introduced for contentbased image retrieval [10], noisy image recognition [11], 3D medical image super-resolution [12],video surveillance [13], foreground detection [14],multi-object tracking [15], explosive device detection [16], pupil detection [17], online data streaming [18], airport baggage handling [19], and manyother objectives. Computational photography [20]

4

S. Chen et al. / Image-based Textile Decoding

and purpose-specific machine learning [21] havebeen also employed to solve industrial problems.
Many studies have been conducted to detect defects from fabric images. Huangpeng et al. [22]constructed a weighted, low-rank, representationalmodel of textures and detected defects. Ren et al.
[23] realized the detection from a small sample set
using pre-trained DNNs, and Wei et al. [24] also
employed DNN for the classification of fabric defects using a small number of samples. Jeyaraj
et al. [25] improved the accuracy of defect detection by introducing the characteristic of texture
in the training of DNN. Jing et al. [26] analyzed
the detailed parameter settings of pre-trained networks, image patch sizes, the number of layers, andso on for detecting defects in repetitive patterns.
Li et al. [27] proposed a method for detecting defects by Fisher criterion-based stacked denoising
autoencoders. Liu et al. focused on information
entropy and frequency domain saliency [28]. The
ability to detect defects is valuable for the industry [29,30,31] and patents for defect detection havebeen published [32]. On the other hand, defectdetection is a task of anomaly detection or classification; however the pattern analysis of interestin this work is the binarization of the regularlyaligned crossing points of warp and weft yarns.
In order to measure the density of the weft
for the woven fabric, Schneider et al. detected
the intensity minima of brightness and estimatedthe positions for each [33]. Compared with thesimple Fourier transform, the change in the density of the weft can be obtained more accurately.
Luo and Li [34] employed image processing techniques for quantifying fiber distribution uniformity
in blended yarns based on cell counting and dilation area estimation. Meng et al. [35] realized theirgoal of estimating the yarn density of multi-color
woven fabrics by deep learning. In this method,
assuming that the whole fabric is woven in a periodical pattern, the intersection of the warp andthe weft arranged on a regular grid is detected.
Near-infrared spectroscopy revealed the materialof yarns [36]. Machine learning also contributed toestimate the kinematic behavior of yarns [37]. In
this paper, we aimed to analyze different patterns
in a fabric. The grid formed by the warp and theweft may be greatly collapsed.
Zheng et al. [38] proposed a method for recognizing how predefined patterns are arranged
on a piece of fabric. Loke and Cheong used co-occurrence matrices for the recognition [39]. Thesemethods can be applied when the fabric is composed of a small number of known patterns, but
this cannot be assumed for woven fabrics in which
the pattern is created manually. Therefore, we
aimed to determine the intersection of each pointon the grid.
3. Textile Pattern Decoding from Observed Image
Here, we describe a method for decoding binary textile patterns from observed images. The
method consists of three parts; pre-processing forconverting a fabric image to an intermediate representation (Section 3.1); using a DNN for generating a label image (Section 3.2); and postprocessing for converting the intermediate representation into a binary matrix textile pattern (Section 3.3). The training data is a set of observed images and a corresponding set of intermediate representations of these images with manually labeled
cross point positions. At runtime, the final binary
pattern is output from an observed image only.
Fig. 2 shows an overview of the process.
3.1. Pre-process for converting a fabric image toan intermediate representation
3.1.1. Manual tagging of crossing points
Analysis of the weaving pattern can be done
manually by observing the fabric using a microscope and recording the cross state of each yarn
by disassembling it. This method is very time consuming, laborious, inefficient, and costly. In addition, it is also undesirable to perform such destructive inspections on textiles of high historical value.
As is shown in Fig. 3, we estimated fabric patterns
from captured images and interactively modified
fabric patterns to analyze weaving patterns. Initial estimation results of the crossing point positions of warp and weft yarns were given by image
analysis. The positions were interactively modified
with a GUI so that the pattern could be obtainedin a short time without destroying the actual fabric. Fig. 3 outlines the detection of the positions ofweft yarns by filtering the image with a Laplacianof Gaussian (LOG) filter. A LOG filter is used to
extract edges, such as regional boundaries, whileremoving small noise. We expected to ignore thefine threads and to output the boundaries between
warp and weft yarns. Where the boundary pixels

S. Chen et al. / Image-based Textile Decoding

5...

Smoothing

Local minima correspondthe center position of yarns(a) Observed image(b) LOG filtered image(c) Yarn position estimation

Fig. 3. Estimation of warp and weft positions. For the input image shown in (a), the LOG filter image shown in (b) is
calculated (σ=11 for Gaussian filter). In order to find the position of wefts, the sum of the pixel values in the LOG image iscalculated row by row. (c) For a 1D sequence of sums, we smooth the values by finding the average of a certain value and thefive neighboring values before and after the value. The local minima of the sequence indicate the positions where the powerof edges is low, so they are output as the center positions of wefts. To estimate the positions of warps, the local minima of
the sequence of vertical sums.

are few, it can be assumed that it is the center ofthe crossing point of warp and weft yarns. Since
the edge pixels could be observed at the edge of theyarn, the center of the yarn was found by picking
up the local minima with fewer edge pixels [38].
When the pixel values were integrated in thehorizontal direction (X direction), the intensity ofthe edge component in each row was obtained. The
position where the edge intensity reached the maximum was the end of the row with many edge elements, and the row whose edge intensity reachedthe local minimum was the center of the yarn withrelatively few edges. Applying the same process to
the warp yarns, the crossing points of the warp
and weft yarns could be obtained as the initial positions of crossing points. Although this method
can analyze patterns quickly, it is not automatedenough to perform large-scale analysis. Therefore,
we used this method only to prepare training data.
Once the initial positions of the warps and wefts
were determined from the input image, we computed which of the warps or wefts was higher foreach of the grid points. By giving two representative colors of warp and weft, we could see whichcolor was closer at a grid point, and then give thestate of the grid at that point as 0 or 1.
In the developed GUI, a user can add, remove,and move yarns by clicking and dragging on thescreen. The operational mode was defined as move,
add, or delete by key presses: The operation withthe shift key pressed was for the warp; the operation with the ctrl key pressed was for the weft,
and the operation without pressing any key was forthe crossing point. When the weft and warp positions were updated, the initial intersection (basedon the new warp and weft positions) could be obtained by pressing the C key. When the F key waspressed, the vertical direction of the weft and warp
at the crossing point closest to the clicked mouseposition was reversed.
3.1.2. Noise removal of fine fibers
In the observed image, even fine fibers are observed, as shown in Fig. 4(a). The fine fibers produce intensity edges which result in high values
in the LOG-filtered image. This problem can beeasily solved by the image processing of erosionand dilation, resulting in the image shown in Fig.
4(b). Erosion and dilation are used to remove theregions with little area. In our experiments, weused three functions in MATLAB; strel, imerode,and imdilate. To obtain the results, the image waseroded with a 5.5 pixel radius setting and then expanded with the same setting. The radius setting
was determined by checking to see if the thin fibers
disappeared in the sample image.
Fig. 4(c) shows the image with the manuallylabeled crossing points. The red and blue points
are central with the red point indicating that thewarp is on top at the point. The blue point indicates that the weft is on top at the point. Fig.
4(d) shows the image with white pixels for the
point of the warp on the weft; black pixels for the
point of weft on the warp; and gray pixels for otherthan the crossing points in the image. For machinelearning, the pre-processed image in Fig. 4(b) andthe intermediate representation image in Fig. 4(d)are provided as a pair; the DNN receives the pre-6

S. Chen et al. / Image-based Textile Decoding(a) Observed image.

(b) Pre-processed image.

(c) Manually tagged crossing
points.

(d) Filtered labeled image.

Fig. 4. Converting processes of input image into labeled image. The observed image was captured with a camera with amacro lens. The image was smoothed in the pre-process and the positions of the crossing points were manually tagged. By
filtering the crossing points, we obtained the final, labeled image.

(a) Impulse peak pattern.
I0 , defined by Eq. (1).

(b) Gaussian-filtered peak
pattern. IG , defined by
Eq. (2).

(c) Box-filtered peak pattern.
IB , defined by Eq. (3).

(d) Binary pattern.

Fig. 5. Impulse peak pattern and three kinds of labeled images.

processed image and outputs an intermediate representational image.
3.1.3. Converting intermediate representation
imagesHere, we considered how to make an intermediate representational image. If we were to use theresult of manual labeling in its present form, we
would get an image I0 with impulses at the crossing points, as shown in Fig. 5(a) and Eq. (1). At acrossing point, it would take a value of 0 or 1 andat the very next pixel, it would take a gray valueof 0.5.
 1 if wef t on warp at (x, y),
I0 (x, y) = 0 if warp on wef t at (x, y),(1)
0.5 otherwise.

In the input image, there is no significant change
in the immediate vicinity of the crossing position,
so it is expected that DNN would have difficulty
producing such an image. The image IG with theGaussian blur shown in Fig. 5(b) can represent
the situation in which the likelihood peaks at thecrossing point and the likelihood gradually decrease. It is expected to be easier to handle with

DNN than the impulses. I0 can be reproduced byfinding the minimum and maximum points of eachpeak. Furthermore, by trial and error, we foundthat the image with the box filter as shown in Fig.
5(c) has better accuracy for the resulting images.
The results of using I0 , IG defined by Eq. (2), andIB defined by Eq. (3) for the training data, respectively, are presented in the Experiment section.
IG = LOG(I0 ),(2) 1 max(s,t)∈N (x,y) I0 (s, t) = 1,
IB (x, y) = 0 max(s,t)∈N (x,y) I0 (s, t) = 0,(3)
0.5 otherwise.

Note that N (x, y) represents the set of neighborhood pixels of (x, y). In Fig. 2 and Fig. 4(b), we
used a window of 9 × 9 pixels as N (x, y). Different
sizes of windows were also tried and examined in
the experiment.
In Fig. 5(d), the final binary pattern is transformed into an image, but the size of the image ismuch smaller than the input image. Additionally,
the positions of the crossing points are different
from those of the input image. To obtain such animage by DNN is difficult because DNN tries to de-

S. Chen et al. / Image-based Textile Decoding

3.3. Post-process for converting intermediate
representation into binary matrix textile
pattern
DNN outputs an intermediate representation
image; however, a non-trivial post-processing is required to obtain the final binary pattern. This oc-

320x512x3

160x256x144

Fabric image

320x512x99

40x64x144

40x64x48

80x128x48

160x256x48

20x32x48
10x16x48
20x32x96

80x128x144

We used a DNN to output a label image fromthe input image. It was necessary to determine
whether a pixel corresponded to a crossing pointby looking at the surroundings of the pixel without
losing the accuracy of the position. We employeda DNN with a U-net structure [40] to solve thistask in which a network with this structure cantake into account the surroundings by a networkof autoencoders and maintain the resolution by
jumping paths.
The model structure is shown in Fig. 6. The
input to the DNN is a pre-processed image, which
is an image with fine noise removed, and the output from the DNN is an intermediate representation of the image, which shows the likelihood of
the existence of the crossing points on the image.
The model has a total of six down-roll multi-layerand six upper convolution layers; the final output
size is the same as the input size. When the preprocessed observation images were input, the network was trained so that the label images were
output. The input image size of 320×512 was chosen because it is difficult to train a network withtoo many neurons due to the limited number ofsamples, and because it is easy to work with whentagging manually. It does not necessarily have tobe this size; other sizes may be used with confidence.
In the learning phase, we gave a pair of observed
images and a label image that were manually generated. For the loss function, the L1 norm of pixel
values was employed. The number of neurons ineach layer is also shown in the figure. PyTorch wasused for implementation. A more detailed implementation environment is described at the beginning of the experimental section.

Label image

320x512x3

termine whether a point is a crossing point basedon the information of a pixel and its neighbors.
In that situation, DNN cannot produce an image
like Fig. 5(d) in which the position is completely
misaligned.
3.2. DNN model for generating label image

7

Fig. 6. Neural network model structure for converting labeled intermediate representation image.

Deep Neural
Network

1)

Tri-valued image

Output image
2)

Region-integrated
image

Gravity points ofindividual regions

3)

4)

Approximation ofyarn positions

Resulting binary
patternFig. 7. Post processing for the conversion of output image
from a deep neural network.

curs in several steps: 1) the intermediate representation image is converted into a tri-valued imageof 0, 0.5, and 1; 2) each peak region is merged into
one; 3) the approximate horizontal and vertical positions of the warp and weft yarns are found; and4) a determination is made as to which of warp
and weft are over at each grid point. Finally, we
get a binary pattern with 0s and 1s on the grid
points, as shown in Fig. 7.

8

S. Chen et al. / Image-based Textile Decoding

3.3.1. Converting the intermediate representation
image into a tri-valued image
The intermediate representation image output
from the DNN had continuous values in the rangeof (0, 1). In order to find the crossing points bysegmenting the image, each pixel must have a discrete value. We replaced the value of each pixel
with the closest of the three values, in which thecrossing point of the warp was 0, the crossing pointof the weft was 1, and 0.5 otherwise. This process
is equivalent to thresholding by two values of 0.25
and 0.75. An image is converted into a tri-valued
image only with white, black, and gray pixels.
3.3.2. Integrating regions of multiple values
In the image obtained by the process describedin 3.3.1, based on the gray background, a region
consisting of white pixels only, a region consistingof black pixels only, and a region containing bothwhite and black pixels appeared. Since the crossing
points were never adjacent to each other in realfabrics, no region ever contained both white andblack pixels.
We aim to avoid the coexistence of white andblack pixels in each region that represents a crossing point. The region containing both white andblack pixels was merged into the region with thelarger number of white or black pixels. The pseudocode for the process is described in Algorithm 1.
The regions were labeled by segmentation andthe adjacent regions for each region were searched.
The connected components labeling (CCL) gave
the individual, connected regions of a binary image drawn by identical numbers. A region containing both white and black pixels could be detected from the adjacency matrix. Ignoring thebackground gray region, we first flagged the adjacent regions. We then flagged the adjacent regionsof the first set of adjacent regions and repeatedthe process until no more flags appeared. Finally,
the black and white regions included in each independent region were recognized. The number ofwhite and black pixels belonging to each region
were counted and the region was merged by thehigher color.
After we merged the regions with only white
pixels and black pixels, we next found the representative points of each region. Since the representative point should be located in the center ofthe crossing point region, the pixel at the grav-

Algorithm 1 Integrating regions with multiple values with connected component labeling (CCL)
Input: Tri-valued image
1: Pixel value v(p) ∈ {0, 0.5, 1} in tri-valued image
2: 4-neighboring pixel of p is N (p)3: Function BW (p, q) = True only when (v(p) = 0 ∧v(q) = 1) ∨ (v(p) = 1 ∧ v(q) = 0)
Output: Tri-values image without 0-1 mixed regions
4:⊲ CCL segmentation
5: CCL segments regions with unique values
6: Identify regions {r} by IDs {i}7:⊲ Merge region IDs
8: flag ← True
9: while flag = True do
10:flag ← False
11:
foreach Region ri do
12:
foreach Pixel p ∈ ri do
13:
if ∃q ∈ N (p), BW (p, q) = T rue then14:
if p ∈ ri and q ∈ rj and i < j then
15:
j←i
16:
flag ← True17:
end if18:
end if
19:
end foreach20:
end foreach
21: end while22:⊲ Unify pixel values in individual regions
23: foreach Region ri do
24:
if 0-valued pixels are more than 1-valued pixels
in ri then25:
All pixels in ri ← 0
26:
else
27:
All pixels in ri ← 1
28:
end if
29: end foreachity point of each segmented region was to be therepresentative point of that region.
3.3.3. Finding approximate horizontal andvertical positions
Regional representative points indicating crossing points included duplicated and missing points.
The crossing points were constraints that existed
in a grid along the yarns. The regional representative points satisfying this constraint were pickedup and converted into binary matrix patterns.
First, by detecting the positions of the warp
and weft yarns, the approximate positions of thegrid points were estimated. The distance transformation image for the obtained candidate crossing

S. Chen et al. / Image-based Textile Decoding

points were able to show the likelihood of a warp
or weft passing through its position. The positionsof warp and weft yarns were estimated in the same
way as in the pre-processing, using the distancetransformation image as the target.
The crossing point pixels were replaced by 1 as
the object region and the other pixels were replaced by 0 as the background region. The valueof each pixel in the distance transformation image
indicated the distance from that pixel to the nearest pixel in the object region. The distance was 0
for pixels in the object region; the further away
the pixel was, the larger the value was.
We integrated the distance transformation values of pixels in a column. Along a column, themore pixels that were in the object region or close
to it, the smaller the integral became while thelikelihood of being the center of the warp or weftyarn increased. As described in Section 3.A, thepositions of warp and weft yarns were estimated bydetermining the likelihood according to the powerof intensity edges. The integral was smoothed in
the same way to reduce the effect of local noise.
By way of this process, the positions of the warp
and weft yarns were estimated.
3.3.4. Determining value at each grid point
As the positions of the warp and weft yarns were
estimated, the positions of the grid points werealso estimated automatically. By assigning a valueof 0 or 1 to each grid point, we were able to obtainthe final binary pattern in a matrix form. In Fig. 7,
the blue lines indicate the estimated location of thewarps while the red lines indicate the estimatedlocation of the wefts. The white and black points
are representative points with the values of 1 or 0.
For each grid point, we extracted the best candidate point within distance s and let the value of
0 or 1 of the candidate points be the value of thegrid points. If there was more than one candidate
point, we adopted the closest one to the grid point;
if there was no point, we adopted the closer colorof which the warp or weft colors from the colorof the grid point itself. s allowed for being set arbitrarily; a smaller value meant that we adopted
only candidates strictly close to the grid point. In
the experiment, we confirmed the accuracy of pattern reproduction according to the variety of s.

9

4. Experiment
Textile samples were observed with an SLR
camera Canon EOS M for consumer use; we attached a fixed focus macro lens with LED illumination Canon EF-S 35mm f/2.8 Macro IS STM.
High-resolution images were divided into small,partial images. The images were made by observing samples represented in reference [5]. The samples were woven with black and white yarns only,
and the patterns were generated from natural images. We obtained 176 images; each had a resolution of 512 × 320 pixels. We then manually tagged
all the images. The number of images is to be
increased to get more accurate results, but it requires more effort. The training images were augmented by flipping them vertically and horizontally and rotating them 180◦ ; then, 704 examples were used for the training. With the horizontal mirror images, the vertical mirror images, theimages flipped 180◦ , and the original images, there
were a total of 176 × 4 = 704 images. For each
image, we performed the manual cross point extraction as described in Section 3.1. The observed
images and manually added tags are shared athttps://github.com/toyoura/TextileDecoding.
PyTorch was employed as a deep learning framework and GPU GeForce2080Ti 11GB was used fortraining and validation. It took about 2.4 hours
to train. The batch size was set to 4 and the
maximum epoch was set to 400. To complete theprocess of one image at runtime it took about
1.21 seconds, which included 0.24 second for preprocessing, 0.08 second for executing DNN, and0.89 second for post-processing.
4.1. Comparison of intermediate representations
DNNs were respectively trained by the threeintermediate representational images describedin Section 3.2. We performed an 11-fold crossvalidation procedure to ensure the robustness ofthe method. The 176 images were divided into 11
groups of 16 images. After training the networkwith 160 images, we verified the accuracy of theremaining 16 images and repeated this process 11
times. The output images obtained by each training are shown in Fig. 8. For the results, no postprocessing was done.
First, we can see that impulse peak patterntraining does not provide good results. Many re-

10

S. Chen et al. / Image-based Textile Decoding(a) Pre-processed input image.

(b) Result image by impulse peak pattern training.

(c) Result image by Gaussian-filtered peak image training.

(d) Result image by box-filtered peak image training.

(e) ROC-like curve.
Fig. 8. Comparison of the different kinds of intermediate representations for DNN training. The results shown in (c) are given
with σ = 5 for Gaussian filter, and (d) are given with N (x, y) as a window of 9 × 9 pixels.

gions of the white yarns were detected as cross-the edges formed by the yarn contours. When con-ing points, and candidate points also appeared atsidering the segmentation of post-processing, theS. Chen et al. / Image-based Textile Decoding

noisy clutters did not result in the correct detection of candidate points.
Although much better results were provided
than those by impulse peak pattern training,
Gaussian-filtered peak pattern training generated
blurry images. The post-processing of the segmentation was not successful because each of the regions became one continuous region of adjacent,segmented regions. This trend did not improve,even after changing the variance of the Gaussian
peak. When the variance was set too small, the results were similar to those induced by the impulse
peak pattern.
Finally, the results trained by the box-filteredpeak pattern were better than the two previous
results. The region indicating each crossing point
was less likely to be integrated with its adjacent regions. The extraction of the candidate representative points by post-processing also showed good results in appearance. The correct rate for all groups
was 83.25% on average, and the minimum was
74.13%, the maximum was 99.17%, and the standard deviation was 8.40%.
4.2. Performance of cross point detection
The resulting binary patterns obtained in theform of matrices could not be evaluated directly
because the same number of crossing points ineach row and column could not be aligned in theobserved image. Therefore, the observed crossing
points could not be represented in a matrix manner. Instead of the direct evaluation of the binary
patterns, we verified whether the label images werecorrectly obtained. The correct label image could
be given by manual labeling. The accuracy of theposition of the crossing points could also be examined.
Here, we evaluated the representative points ofthe crossing point regions obtained from the resulting label image. Many pixels took a value of
0.5 and a few took 0 or 1 to indicate that the pixels were representative crossing points. Similarly,
in the manually labeled images, a small numberof crossing points were placed in a background ofmany pixels that took a value of 0.5. In order toverify the match between them, we set a thresholds for Euclidean distance. For a crossing point in
the manually tagged image, if the extracted candidate crossing points were within the thresholddistance, we considered the crossing point to be

11

correctly detected; if there were multiple intersections within s, we considered the crossing pointwith the smallest distance to be the corresponding
point as shown in Fig. 9.
Correct: Correct 0 crossing point: Correct 1 crossing point

Correct: Detected 0 crossing point
Error

Missed: Detected 1 crossing point

Threshold sFig. 9. Threshold s for quantitative performance assessment.

For quantitative evaluation, for each representative crossing point in the estimated binary pattern, the rate of correctly detected crossing points(Correct), the rate of incorrectly detected crossing
points (Error), and the rate of points with no candidate points in distance s (Missed) were calculated. These rates varied depending on the valueof s and were represented by ROC-like curves.
Fig. 8(e) shows the resulting label images andROC-like curves. Note that we omitted the quantitative analysis for impulse peak pattern training.
There were not enough extracted crossing points
from impulse peak pattern training because resulting output images had few large white and black
regions after post-processing.
After extensive trial and error, we found a DNN
with 12 layers, a 9 × 9 box-filter, and a distancethreshold s = 10 indicated good performance. In
the following, we show the experimental resultsfor different filters, the window size, the numberof layers, and the distance threshold s. From theimage set used for the 11-fold cross-validation, we
employed the worst group (160 images for training
and 16 images for test) for the comparison experiment.
Number of DNN layers To confirm the optimal
structure of the DNN, we designed comparative
experiments, reducing and increasing the numberof neural network layers. The number of layers
should be even because the same number of layers is required for encoder and decoder parts. The
deepest layers near the bottleneck were added andremoved for changing the number of layers. The
smallest layer of 14-layer network is 5 × 8 × 48,

12

S. Chen et al. / Image-based Textile Decoding(a) Number of layer.

(b) Box filter size.

(c) Gaussian filter size.

(d) s in Box filter.

(e) s in Gaussian filter.

Fig. 10. Comparison experiment for different number of layers, filters, and distance threshold s.

(a) Input image.

(b) Estimated binary pat- (c) Binary pattern valida- (d) Reproduced woven fabtern.
tion.
ric of (b).
Fig. 11. Reproduced woven fabric by final decoded binary patterns.

then we cannot add any more layers to 14-layer
network. Fig. 10(a) shows that the 12-layer neural network had an accuracy level of 74.13%. Under the same circumstances, the 10-layer neuralnetwork had an accuracy level of 74.88%, while
the accuracy of the 14-layer neural network was
78.92%. For the size of the dataset and the degreeof difficulty of the task, the 14-layer network had
the best performance.
Filter parameters
For the intermediate representation, we controlled other unchanged variables, and compared
their results by changing the size of the filter. Although there were a number of values from which

S. Chen et al. / Image-based Textile Decoding

to choose, considering the feasibility of the experiment, as shown in Figs. 10(b) and 10(c), we compared the Box methods with filter sizes of 5, 7, 9,
and 11; we compared the Gaussian methods withfilter sizes of 3, 5, 7, and 9. When the filter size was
9, the Box-filter method had the highest accuracyof 78.92%, 7 and 11 had accuracy levels of 72.21%
and 73.99%, respectively. When the filter size was5, the Gaussian-filter method had the highest accuracy of about 71.69%; 7 and 9 had accuracy levels of about 60.77% and 41.00%, respectively. For
Gaussian size 3 and Box size 5, we could not extract the binary pattern, so their accuracy was 0.
By comparison, when the filter size of the Boxfilter method was 9, that yielded the best result,and that was the scheme adopted for our method.
Distance threshold s
Each resulting label image showed the same
trend, as shown in Figs. 10(d) and 10(e), there
were 47.47% of correct crossing points at s = 5;74.13% of correct crossing points at s = 10; a
6.81% error crossing points at s = 10; and only
0.67% of missed crossing points when s was increased. This indicated that 74.13% of the crossing points were detected correctly within the distance s, while 7.48% of the warp and weft crossing points were detected incorrectly. Statistically,
the rate of correct crossing points was greatest at
s = 15, which was 74.44%. Considering the total
rate of correct, error, and missed, the performanceis the best at s = 10.
4.3. Validation of decoded binary patterns
Fig. 11 shows the results that were woven by
the obtained binary pattern, together with theobserved image. Since the input image contained
crossing points near the image boundary that
could not be expressed in the form of a matrix, we
ignored the intersections near the boundary andevaluated them. In addition, we added binary pattern validation in Fig. 11(c) to show the analysis
results. We evaluated whether the values of 0 and
1, given the grid points in the estimated binary
image matched the values in the ground truth image. Each grid point corresponded to a pixel of theobserved image, so we could check the match between it and the binary value of the nearest crossing point in the ground truth image. In Fig. 11(c),a red box shows that 0 was wrongly estimated as

13

1, and a blue box shows that 1 was wrongly estimated as 0 for each grid point. In the patterns,
we were able to obtain results that were close inappearance. Since some of the patterns were extracted that are not common as woven patterns,
such as too many or too few crossing, there wasroom for the pattern to be improved by converting
it with an uncommon pattern such as a restriction.
Although the error in which 0 was wrongly set to
1 and vice-versa is different from a false positiveor a false negative in a general sense, the accuracy
and F-measure of the statistical metric can be applied to the data. The accuracy for 176 images was
0.930 and the F-measure was 0.929, on average.
Although the obtained pattern was not perfect,
it can be said that the same pattern was approximately obtained automatically.

5. Conclusions
In this paper, we proposed a method for decoding the binary patterns that define the weavingof fabric. By developing intermediate representations, we were able to accomplish the task by deeplearning. The pre-processing and post-processing
allowed us to bridge the intermediate representation image and the binary pattern. The experimental results showed that our method allowed
for correctly extracting 93% of the crossing points,
and the reproduced textiles were close to the original one in appearance. Although the box filter
may not be optimal, it gives better results than
the isotropic Gaussian filter. We consider this tobe due to the grid arrangement of the yarns.
Only black and white yarn images were discussed in this paper. If the warp and weft are of
one color each, the same processing is considered
possible by converting the image according to theproximity of each color. Multiple colors may be
used for weft yarns, and this will be handled byconverting the image by the color distance fromthat of warp yarn; however we have not confirmed
whether this will work well. That is a topic for
future studies.
It is necessary to ensure that each of the piecesof yarn are captured one by one in the observed
image, which limits the scope of the observation.
We need a method to integrate the resulting patterns observed at multiple locations. The resulting patterns are partially incomplete, which makes

14

S. Chen et al. / Image-based Textile Decoding

the problem more difficult. Various image stitching
techniques are helpful. Although we used LOGfiltered images for detecting the positions of theyarns, there is the possibility of improving the accuracy by introducing the structural tensor. In thecase of textiles, the yarns are made of even fineryarns twisted together, so the fine edges can interfere with calculating the correct tensor. We would
like to address these issues in the future.
We will also leave the further tasks of the optimization of network configuration and the selection of the backbone network to future work.
Recent powerful machine learning techniques [41,42,43,44] will contribute to improve the accuracyof pattern decoding even when the dataset is notlarge enough.

[7] Leaf J, Wu R, Schweickart E, James DL, Marschner S.
Interactive design of yarn-level cloth patterns. ACM
Transactions on Graphics. 2018;37(6):article 202.
[8] Sadeghi I, Bisker O, Deken JD, Jensen HW. A
practical microcylinder appearance model for cloth
rendering. ACM Transactions on Graphics (TOG).
2013;32(2):article 14.
[9] Zhao S, Luan F, Bala K. Fitting procedural yarn models for realistic cloth rendering. ACM Transactions on
Graphics (TOG). 2016;35(4):article 51.
[10] Hamreras S, Boucheham B, Molina-Cabello MA,
Benı́tez-Rochel R, López-Rubio E. Content based image retrieval by ensembles of deep learning object
classifiers. Integrated Computer-Aided Engineering.
2020;27:317-31.
[11] Koziarski M, Cyganek B. Image recognition with deepneural networks in presence of noise – Dealing with andtaking advantage of distortions. Integrated ComputerAided Engineering. 2017;24:337-49.
[12] Thurnhofer-Hemsi K, López-Rubio E, Roé-Vellvé N,
Molina-Cabello MA. Multiobjective optimization ofdeep neural networks with combinations of Lp-norm
Acknowledgement
cost functions for 3D medical image super-resolution.
Integrated Computer-Aided Engineering. 2020;27:233The authors appreciate the joint support for51.
this project by the JSPS KAKENHI (Grant Nos.
[13] Benito-Picazo J, Domı́nguez E, Palomo EJ, LópezRubio E. Deep learning-based video surveillance sysJP16H05867, JP20H04472), the NSFC-Zhejiang
tem managed by low cost hardware and panoramic
Joint Fund for the Integration of Industrializacameras. Integrated Computer-Aided Engineering.
tion and Informatization (Grant No. U1909210),
2020;27:373-87.
the National Natural Science Foundation of China[14] Garcı́a-González J, Ortiz-de-Lazcano-Lobato JM,(Grant Nos. 61761136010, 61772163), the NatLuque-Baena RM, López-Rubio E. Background subtraction by probabilistic modeling of patch featuresural Science Foundation of Zhejiang Province
learned by deep autoencoders. Integrated Computer(No.LY18F020016), and Zhejiang Lab Tianshu
Aided Engineering. 2020;27:253-65.
Open Source AI Platform (No.111007-AD1901).
[15] Yang T, Cappelle C, Ruichek Y, El Bagdouri M.
Multi-object tracking with discriminant correlation filter based deep learning tracker. Integrated ComputerReferences
Aided Engineering. 2019;26:273-84.
[16] Colreavy-Donnelly S, Caraffini F, Kuhn S, Gongora
M, Florez-Lozano J, Parra C. Shallow buried impro[1] TSUDACOMA Corp., ZAX9200i AIR JET LOOM,
vised explosive device detection via convolutional neuwww.tsudakoma.co.jp/english/business/textile/product/air
ral networks. Integrated Computer-Aided Engineering.
jet loom/zax9200i/index.html (Last access: 2020-102020;27:403-16.
06).
[17] Vera-Olmos FJ, Pardo E, Melero H, Malpica N. Deep[2] Igarashi T, Toyoura M, Mao X. Dithering method for
Eye: Deep convolutional network for pupil detection in
reproducing smoothly changing tones and fine details
real environments. Integrated Computer-Aided Engiof natural images on woven fabric. Textile Researchneering. 2019;26:85-95.
Journal. 2018;88(24):2782–99.
[18] Lara-Benı́tez P, Carranza-Garcı́a M, Garcı́a-Gutiérrez[3] Ng F, Kim KR, Hu J, Zhou J. Patterning technique for
J, Riquelme JC. Asynchronous dual-pipeline deep
expanding color variety of Jacquard fabrics in alignlearning framework for online data stream clasment with shaded weave structures. Textile Researchsification. Integrated Computer-Aided Engineering.
Journal. 2014 Apr;84(17):1820-8.
2020;27:101-19.
[4] Zhang A, Zhou J. Hierarchical combination design of[19] Sørensen RA, Nielsen M, Karstoft H. Routing in conshaded-weave database for digital jacquard fabric. The
gested baggage handling systems using deep reinforcejournal of the Textile Institute. 2019;110(3):405-11.
ment learning. Integrated Computer-Aided Engineer[5] Toyoura M, Igarashi T, Mao X. Generating Jacquard
ing. 2020;27:139-52.
Fabric Pattern With Visual Impressions. IEEE Trans[20] Wu S, Zhang G, Neri F, Zhu M, Jiang T, Kuhnert
actions on Industrial Informatics. 2019;15(8):4536-44.
K-D. A multi-aperture optical flow estimation method[6] Dobashi Y, Iwasaki K, Okabe M, Ijiri T, Todo H. Infor an artificial compound eye. Integrated Computerverse appearance modeling of interwoven cloth. The
Aided Engineering. 2019;26:139-57.
Visual Computer. 2019;35(2):175-90.

S. Chen et al. / Image-based Textile Decoding[21] Rostami S, Neri F, Epitropakis M. Progressive
preference articulation for decision making in
multi-objective optimisation problems. Integrated
Computer-Aided Engineering. 2017;24:315-35.
[22] Huangpeng Q, Zhang H, Zeng X, Huang W. Automatic
visual defect detection using texture prior and lowrank representation. IEEE Access. 2018;6:37965-76.
[23] Ren R, Hung T, Tan KC. A generic deep-learningbased approach for automated surface inspection.
IEEE transactions on cybernetics. 2017;48(3):929-40.
[24] Wei B, Hao K, Tang X-s, Ding Y. A new method
using the convolutional neural network with compressive sensing for fabric defect classification basedon small sample sizes. Textile Research Journal.
2019;89(17):3539-55.
[25] Jeyaraj PR, Nadar ERS. Computer vision for automatic detection and classification of fabric defect employing deep learning algorithm. International Journalof Clothing Science and Technology. 2019.
[26] Jing JF, Ma H, Zhang HH. Automatic fabric defectdetection using a deep convolutional neural network.
Coloration Technology. 2019;135(3):213-23.
[27] Li Y, Zhao W, Pan J. Deformable patterned fabric defect detection with fisher criterion-based deep learning. IEEE Transactions on Automation Science andEngineering. 2016;14(2):1256-64.
[28] Liu G, Zheng X. Fabric defect detection based on information entropy and frequency domain saliency. The
Visual Computer. 2020;1-14.
[29] AI
qualitycontrol
continues
textile’shistoryofautomation,
The
Engineer,https://www.theengineer.co.uk/ai-textileautomation/. (Last access: September 9, 2020)[30] V. Tirronen, F. Neri, T. Kärkkäinen, K. Majava, T.
Rossi, An Enhanced Memetic Differential Evolution in
Filter Design for Defect Detection in Paper Production, Evolutionary Computation Journal, MIT Press,
Volume 16, Issue 4, pages 529-555, 2008.
[31] Kwak C, Ventura JA, Tofang-Sazi K. Automated defect inspection and classification of leather fabric. Intelligent Data Analysis. 2001;5:355-70.
[32] Complicated jacquard defect detecting method basedon deep learning, China Patent, CN105654121A, 2016.
[33] Schneider D, Gloy Y, Merhof D. Vision-based onloom measurement of yarn densities in woven fabrics.

15

IEEE transactions on instrumentation and measurement. 2014;64(4):1063-1074.
[34] Luo X, Li L. Design of quantification methods for cross-sectional uniformity of functional
fiber blended yarns. Textile Research Journal.
2020;0040517520901700.
[35] Meng S, Pan R, Gao W, Zhou J, Wang J, He W.
Woven fabric density measurement by using multiscale convolutional neural networks. IEEE Access.
2019;7:75810-21.
[36] Sun X, Zhou M, Sun Y. Classification of textile fabrics by use of spectroscopy-based pattern recognition
methods. Spectroscopy Letters. 2016;49(2):96-102.
[37] Ribeiro R, Pilastri A, Moura C, Rodrigues F, Rocha R,
Cortez P, editors. Predicting the Tear Strength of Woven Fabrics Via Automated Machine Learning: An Application of the CRISP-DM Methodology. ICEIS (1);2020.
[38] Zheng D, Wang P, Zhou J, Ho K. Color pattern recognition for yarn-dyed fabrics. Color Research & Application. 2019;44(1):88-97.
[39] Loke K-S, Cheong M, editors. Efficient textile recognition via decomposition of co-occurrence matrices.
IEEE International Conference on Signal and Image
Processing Applications; 2009.
[40] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation.
International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015;234-41.
[41] Ahmadlou M, Adeli H. Enhanced probabilistic neuralnetwork with local decision circles: A robust classifier.
Integrated Computer-Aided Engineering. 2010;17:197210.
[42] Rafiei MH, Adeli H. A New Neural Dynamic Classification Algorithm. IEEE Transactions on Neural Networks and Learning Systems. 2017;28(12):3074-83.
[43] Pereira DR, Piteri MA, Souza AN, Papa JP, Adeli H.
FEMa: a finite element machine for fast learning. Neural Computing and Applications. 2020;32(10):6393404.
[44] Alam KMR, Siddique N, Adeli H. A dynamic ensemble
learning algorithm for neural networks. Neural Computing and Applications. 2020;32(12):8675-90.