Patent Description:
On knitting machines, plural yarn cones which are wound by yarns are arranged on the knitting machines or at the periphery of them. For knitting knitted fabrics (for example, knitted products), various yarn cones are used. The operators of knitting machines exchange yarn cones having yarns which are different in colors and thicknesses according to the knitted fabrics to be knitted. When yarn cones become empty, the knitting machines have to stop, and therefore, the operators have to monitor the residual amounts of yarns of yarn cones manually. Yarns fed from yarn cones are connected to and pass through plural tensioning devices which are arranged on the upper part of knitting machines and then fed to needles with proper tension. The operators of knitting machines have to connect correctly respective yarns fed from respective yarn cones to predetermined tensioning devices. However, the connection between the yarn cones and the tensioning devices via the yarns is sometimes wrong.

Various devices have been developed to detect the residual amounts of yarns of yarn cones. For example, Patent Document <NUM> proposes to irradiate light towards the sides of yarn cones and receive the reflected light thereon to detect the residual amounts of yarns. Patent Document <NUM> proposes to irradiate light to the bottoms of yarn cones in order to receive light transmitting through the sides of them by a photo-receiving sensor for detecting the residual amounts of yarns when the outer diameters of the yarn cones become smaller. Patent Document <NUM> relates to a collimated laser beam-based fabric weaving online defect detection method. Patent Document <NUM> relates a method and apparatus of detecting residual yarn on spinning cop tubes, wherein a spinning cop tube is exposed to a suitable light source; the light reflected by the spinning cop tube is detected by a suitable means, e.g., via a camera or other picture-taking device, and two-dimensional picture signals of the spinning cop tube are generated thereby on the basis of the reflected light detected in order to generate a picture matrix comprising picture data in digital form; and the picture signals thusly generated are evaluated by means of performing an edge filtering of the digital picture data in order to segment image edges corresponding to brightness transitions in the picture matrix, from which the presence of residual yarn on the spinning cop tube can be assessed based on the edge filtering. Patent Document <NUM> relates to a method for recognizing the amount of yarns on a yarn pipe in a machine vision system in the textile field. The method comprises the following steps: photographing an image of a yarn pipe object, and converting the photographed yarn pipe image into a grey-scale map; cutting the grey-scale map according to the size of the yarn pipe, so as to acquire an area-of-interest ROI; filtering the background part of the ROI, and extracting the maximum gray value of the ROI; carrying out binaryzation on the ROI according to a preset threshold, so as to obtain a binary picture in which yarns and a light reflection region are overlapped together; processing the binary picture to eliminate light reflection parts in the ROI, so as to obtain a main effect picture of the yarns; and recognizing the yarns according to the main effect picture of the yarns, and determining the proportion of the yarns on the yarn pipe. Patent Document <NUM> relates to an automatic and continuous measurement and control method of the tension and/or running of the yarns fed into knitting machines.

The detection devices of residual yarn amounts proposed by Patent Documents <NUM> and <NUM> need optical elements such as photo-receiving sensors and reflective mirrors provided in the vicinity of yarn cones. Therefore, many optical elements are needed. When yarn cones are arranged in a dense manner, the space for the optical elements is restricted and the adjustment of the light pass becomes difficult when assembling the knitting machine. Further, the relative positions of yarn cones to the optical elements are uniquely predetermined. Therefore, when yarn cones are arranged out of the predetermined positions, the yarn cones cannot be detected. Further, according to Patent Documents <NUM> and <NUM>, the connection between the yarn cones and the tensioning devices cannot be detected.

The object of the invention is to provide an image processing device that can detect simply the status of an arbitrary number of yarn cones, arranged on the upper part of knitting machine or at the periphery of it, regarding the residual amounts of yarns, the connections to the tensioning devices, and their positions, and have high workability and high versatility.

The present invention is defined by the appended independent claim. Further embodiments of the present invention are defined by the appended dependent claims. Advantageous Effects of the Invention.

According to the image processing device (4A to 4C) of the present invention, the statuses of an arbitrary number of yarn cones in the captured image, for example, the residual amounts of yarns of the yarn cones, the connection relation between the yarn cones and the tensioning devices, the positions of yarn cones, is detectable. Therefore, optical elements for respective yarn cones are not needed. The statuses of yarn cones are detectable without restrictions due to the number and the positions of yarn cones, arranged on the knitting machine or at the periphery of it. Thus, the statuses of yarn cones are simply detectable, and therefore, the workability and versatility are improved.

In addition, according to the machine-learning device (6A, 6B1, 6B2, and 6C) of the present invention, the machine-learned models (5A, 5B1, 5B2, and 5C) are generated such that they can accurately detect (infer) the statuses of the arbitrary number of yarn cones in the captured image (for example, the residual amounts of yarns of yarn cones, the connection between the yarn cones and the tensioning devices, the arranged positions of yarn cones).

Embodiments for carrying out the present invention will be described below with reference to the drawings. The following will schematically describe those necessary for achieving the objects of the invention. The description will sometimes be limited to the above, and other portions can be carried out with reference to relevant conventional arts.

<FIG> is a perspective view of a flat knitting machine <NUM> according to the first embodiment.

The flat knitting machine <NUM> comprises: a lower frame 11A that is installed on a floor; a pair of needle beds <NUM> arranged on the lower frame 11A which support a plurality of needles <NUM> in an advanceable and retractable manner and face each other in a front to back relation; a carriage <NUM> that moves along the needle beds <NUM> and makes the needles <NUM> advanced, and retracted; and a top plate <NUM> located over both of the needle beds <NUM> and the carriage <NUM> and supporting one or more yarn cones <NUM> on its upper surface. The flat knitting machine <NUM> may be provided with a creel stand around it that supports any number of yarn cones <NUM>.

The flat knitting machine <NUM> supplies yarns <NUM> from the yarn cones <NUM> on the top plate <NUM> to the needles <NUM>. The flat knitting machine <NUM> is further provided with a plurality of tensioning devices <NUM> attached to the upper frame 11B and positioned over the top plate <NUM>; and a side tensioning devices <NUM>, located, for example, on both sides of the lower frame 11A; and a plurality of yarn feeders <NUM> which is selectively caught and moved by the carriage <NUM>. The yarns <NUM> from the yarn cones <NUM> are fed to the needle <NUM> via the tensioning devices <NUM>, the side tensioning devices <NUM>, and the yarn feeders <NUM>.

The flat knitting machine <NUM> is further equipped with a system for detecting the statuses of the yarn cones <NUM> on the top plate <NUM>: two cameras <NUM> are mounted on both left and right sides of the upper frame 11B; and an image processing device 4A is accommodated in the lower frame 11A.

The cameras <NUM> are provided with image sensors such as CMOS sensors and CCD sensors and capture one or plural images of the top plate <NUM> including the yarn cones from the above. The cameras <NUM> are adjusted such that the yarn cones <NUM> on the top plate <NUM> are within the camera-captured image. The cameras <NUM> may be equipped with functions for pan, tilt, and zoom, and the number of cameras <NUM> is not limited to two, but may be one, three or more. The mounting positions of the cameras <NUM> can be modified; for example, they may be attached to the ceiling over the flat knitting machine <NUM>.

<FIG> shows an example of a camera-captured image <NUM> captured by camera <NUM>.

The camera-captured images <NUM> include yarn cones <NUM> placed on the top plate <NUM>. In the example in <FIG>, five yarn cones 2A-2E are included. The yarn cone <NUM> has a conical core <NUM> with a yarn <NUM> wound around it. The yarn cones can be placed at any position on the top surface of the top plate <NUM>. Therefore, the number of yarn cones <NUM> included in the camera-captured image <NUM> is not fixed but is arbitrary. The camera-captured image <NUM> may include not only the yarn cones <NUM> on the top plate <NUM> but also yarn cones <NUM> that are positioned on creel stands or the like; they are provided in the vicinity of the flat knitting machine <NUM>.

The imaged statuses of the yarn cones <NUM> (e.g., external shape and external size), included in the camera-captured images <NUM>, depends on the outer diameter of the core <NUM>, the thickness and the number of turns of yarn <NUM>, and also the status of the use of yarn <NUM>. Furthermore, the imaged statuses of yarn cones <NUM> also vary according to the relative positions of yarn cones <NUM> and cameras <NUM>. Furthermore, in camera-captured images <NUM>, a portion of yarn cone <NUM> at the back may be hidden by another yarn cone <NUM> at the front. In addition to yarn cones <NUM>, the camera-captured images <NUM> include yarns <NUM> that are unrolled from yarn cones <NUM> and connected to the tensioning devices <NUM>. The camera-captured image <NUM> may include objects other than the yarn cones <NUM>, such as the tensioning devices <NUM>.

In the five yarn cones 2A-2E included in the camera-captured image <NUM> shown in <FIG>, the yarn cone 2A is in an unused status in which the yarn cone 2A was installed and not used. As the yarn <NUM> is gradually used from the unused status, as in yarn cones 2B and 2C, the outer diameter of the yarn cones decreases and the amounts of yarns remaining also decreases. In the yarn cone 2D, part of the core <NUM> is exposed through a clearance in the yarn <NUM>. The yarn cone eventually becomes "without yarn" as the yarn cone 2E is and the entire core <NUM> is exposed. Cones 2A-2D are in "with yarn" status, and the remaining amounts of yarns are, for example, "<NUM>% remaining," "<NUM>% remaining," "<NUM>% remaining," and "<NUM>% remaining.

<FIG> is a schematic block diagram of the flat knitting machine <NUM> provided with the image processing device 4A according to the first embodiment.

The flat knitting machine <NUM> includes, as the electrical components: one or plural cameras <NUM>; an image processing device 4A; a control panel <NUM> including a micro-controller, etc.; and an operation display panel <NUM> equipped with a touch panel, switches, etc. The control panel <NUM> is connected to actuators such as motors and sensors (both not shown) and controls various parts of the flat knitting machine <NUM> such that it controls the knitting operation by the flat knitting machine <NUM>. The operation display panel <NUM> accepts the instructions by the operator and outputs various information in the display or as sounds.

Image processing device 4A comprises a general-purpose or dedicated computer or the like, and comprises: for example, a processor such as a CPU or a GPU; a memory such as ROM or RAM; a communication device for communicating with outer networks or other devices via a wire or wirelessly; and a bus for connecting these parts. The image processing device 4A may be incorporated into the control panel <NUM> as a part of the control panel <NUM> and may or may not be a stand-alone device. The image processing device 4A may consist of a program to cause a processor to perform the operations described below, or a non-transitory computer-readable medium containing one or more instructions for causing the operations to be performed.

The image processing device 4A further comprises, as shown in <FIG>: an acquisition unit <NUM>; and a yarn cone status detection unit <NUM>; a memory for learned model <NUM>; and an output processor <NUM>. The acquisition unit <NUM>, the yarn cone status detection unit <NUM>, and the output processor <NUM> comprise the above processing means, and the memory for learned model <NUM> comprises the above memory.

The acquisition unit <NUM> is connected to the one or plural cameras <NUM> and functions as an interface for acquiring the image captured by the cameras <NUM>. The acquisition unit <NUM> acquires the camera-captured image <NUM>, including yarn cones <NUM>, on or around the flat knitting machine <NUM> (positioned on the top plate <NUM>, in this embodiment). The connection method and communication method for the connection of the image processing device 4A to the cameras <NUM> via the acquisition unit <NUM> is optional.

The yarn cone status detection unit <NUM> performs image processing to the camera-captured image <NUM> acquired by the acquisition unit <NUM> and detects the statuses of cones <NUM> included in the camera-captured image <NUM>. The statuses of cones <NUM> include, for example, the amounts of yarns remaining in cones <NUM>, the position, external shape, and external sizes of cones <NUM>, the type (color and thickness) of yarn <NUM>, and the connections between cones <NUM> and tensioning devices <NUM>. The yarn cone status detection unit <NUM> may detect one or more regarding the statuses of cones <NUM>.

The yarn cone status detection unit <NUM> detects the amounts of yarns remaining in cones <NUM> as the statuses of the yarn cones <NUM>, in this embodiment. The yarn cone status detection unit <NUM> inputs the camera-captured image <NUM> to the learned model 5A as the input data as image processing for the camera-captured image <NUM>. The learned model 5A outputs the inference result (remaining amounts of yarns in cones <NUM>). The relevant processes will be described.

The learned model 5A is generated by machine-learning using the machine-learning device 6A and the machine-learning method. The learned model 5A performs machine-learning of the correlation between the camera-captured image <NUM> containing one or plural cones and the remaining amounts of yarns of the cones <NUM> in the camera-captured image <NUM>.

The yarn cone status detection unit <NUM> performs inference using the learned model 5A and applies predetermined image adjustments (e.g., image format, image size, image filter, image mask, etc.) to the camera-captured image <NUM> as the input data acquired by the acquisition unit <NUM> as the pre-processing for inference. As a post-processing step of inference, the yarn cone status detection unit <NUM> applies, for example, a predetermined arithmetic or logical formula to the output data (inference result) outputted by the learned model 5A to determine the amounts of yarns remaining in the cones <NUM>. The inference results by yarn cone status detection unit <NUM> may be stored in a memory not shown or may be used for online machine-learning or re-learning, for example, to improve the inference accuracy of the learned model 5A.

The output processor <NUM> outputs the detection results detected by the yarn cone status detection unit <NUM>, i.e., the statuses of the yarn cones <NUM>. Various means can be employed as specific output means. For example, the detection results may be reported to the operator by display or sound via the operation display panel <NUM>. It can be sent and stored in the memory of the control panel <NUM> as the operation history of the flat knitting machine <NUM>. It may be transmitted to a higher-level production control system of the flat knitting machine <NUM>. In doing so, the output processor <NUM> may acquire the necessary data (e.g., knitting data, etc.) from the flat knitting machine <NUM>.

The image processing method by the image processing device 4A will be described with reference to <FIG> is a flowchart showing the image processing method by the image processing device 4A in accordance with the first embodiment. The series of image processing processes shown in <FIG> are repeatedly executed by the image processing device 4A at predetermined timings. The predetermined timing may be, for example, at every predetermined time interval or every production number, or it may be when a predetermined event occurs (e.g., when an operator operates, sets up, or stops the machine). In the following, the image processing process is performed while the flat knitting machine <NUM> is knitting.

When the flat knitting machine <NUM> starts knitting, yarns <NUM> are accordingly supplied from the yarn cones <NUM> placed on the top plate <NUM>, and the knitted fabric is knitted. Next, at a predetermined timing, the acquisition unit <NUM> acquires camera-captured image <NUM> of the yarn cones <NUM> on the top plate <NUM> by the cameras <NUM> (step S100).

Then, the yarn cone status detection unit <NUM> refers to the learned model 5A stored in the memory for learned model <NUM> and performs inference using this learned model 5A (step S110). In this case, it is preferable that the learned model 5A to be used for inference is identified in advance based on a predetermined program or worker operation.

Specifically, the yarn cone status detection unit <NUM> pre-processes the camera-captured image <NUM> acquired by the acquisition unit <NUM> and inputs it to the learned model 5A. Post-processing is applied to the output data from the learned model 5A by the yarn cone status detection unit <NUM>. In this way, the remaining amounts of yarns of the cones <NUM> in the camera-captured image <NUM> are detected as a result of the inference by the learned model 5A. In the inference result of step S110 shown in <FIG>, five cones in five cone regions A-E (lower left and upper right coordinates are indicated) are detected from the camera-captured image <NUM>. The remaining amounts of yarns in the five cones <NUM> are then detected in a multi-level manner. Specifically, the amounts of yarns remaining in the four cone regions A - D are "<NUM>% remaining," "<NUM>% remaining," "<NUM>% remaining," and "<NUM>% remaining," while the amount of yarn remaining in cone region E is "no yarn.

In step S110, the yarn cone status detection unit <NUM> infers the amounts of yarns remaining. Based on this result, the output processor <NUM> determines whether the remaining amounts of yarns in cones <NUM> are "no yarn" or not (step S120). If the output processor <NUM> makes a positive judgment in step S120 ("Yes" in step S120), it informs the operator that the result of the inference is "no yarn", for example, on the operation display panel <NUM> (step S130). In other words, the operator is informed that the flat knitting machine <NUM> shall stop. In <FIG>, the cone <NUM> in the cone region E is reported as "no yarn". At that time, the output processor <NUM> may notify the control panel <NUM> that the detection result is "no yarn" and stop the flat knitting machine <NUM>.

If all cones <NUM> are not "no yarn" ("NO" in step S120), the output processor <NUM> determines (step S140) whether the amount of yarn remaining in any cone <NUM> is less than the alert level that indicates the yarn shall become soon empty (e.g., <NUM>% or less). If the output processor <NUM> determines positively in step S140 ("Yes" in step S140), it informs the operator that the result of the inference is a "no yarn alert", namely, the yarn shall soon become empty and that the flat knitting machine <NUM> shall soon stop (step S150). At that time, the output processor <NUM> may notify the "no yarn alert" on the control panel <NUM>.

If the detection result of the amounts of yarns remaining for all cones <NUM> are not less than the alert level and the output processor <NUM> judges negatively in step S140 ("NO" in step S140), the image processing process ends without notifying the operator. The output processor <NUM> may output that the detection result is not "no yarn" and "no yarn alert"#, i.e., "with yarn", and notify on the control panel <NUM>, for example.

The output processor <NUM> is configured to output "no yarn" and "no yarn alert" as the detection results. However, the output processor <NUM> may output only one of them. In that case, any of steps S120, S130, S140, and S150 may be omitted. If the tensioning device <NUM> has a function to detect "no yarn," the tensioning device <NUM> may detect "no yarn" and the output processor <NUM> may output "no yarn alert. " In that case, steps S120 and S130 may be omitted. Furthermore, if the remaining amounts of yarns are detected in a binary value, for example, if the detection result of the remaining amounts of yarns in the four cones <NUM> in cone regions A-D is "with yarn" and the detection result of the remaining amount of yarn in cone region E is "no yarn," steps S140 and S150 may be omitted.

As described above, the image processing device 4A and the image processing method according to this embodiment can detect the statuses (in this embodiment, the remaining amounts of yarns) of the yarn cones <NUM> in the camera-captured image <NUM>. Therefore, there is no need to install optical components for respective cones <NUM>. The statuses of cones <NUM> can be detected without restriction by conditions such as the number or positions of cones <NUM>. Therefore, the statuses of cones <NUM> can be easily detected, with improved workability and versatility.

A machine-learning device 6A for detecting the amounts of yarns remaining in one or more cones <NUM> located on the top panel of the flat knitting machine <NUM> or in the vicinity of the flat knitting machine will be described with reference to <FIG> is a schematic block diagram of the machine-learning device 6A for the first embodiment.

The machine-learning device 6A generates a learned model ("learned model 5A") which can detect (estimate) the amounts of yarns remaining in the yarn cones <NUM> based on a yarn cone image <NUM> that includes an arbitrary number of yarns cones <NUM>. The learned model 5A generated by the machine-learning device 6A is provided to the image processing device 4A and stored in the memory for learned model <NUM>. The learned model 5A is thus used for inference by the yarn cone status detection unit <NUM>. Here, the image processing device 4A and the machine-learning unit 6A are shown as separate devices. However, the image processing device 4A and the machine-learning unit 6A can be single devices.

The machine-learning device 6A comprises a general-purpose or dedicated computer or the like. The machine-learning device 6A may comprise, for example, a processor such as CPU, GPU, etc.; a memory unit comprising ROM, RAM, etc.; and a communication device for communicating with a network or other devices by wired or wireless means; and a bus connecting these parts. The machine-learning device 6A may be provided as a stand-alone device. The machine-learning device may be provided in the form of a program for causing a suitable processor to perform predetermined operations, or a non-transitory storage medium containing instructions for performing these operations.

The machine-learning device 6A has a learning data set acquisition unit <NUM>, a memory for learning data set <NUM>, a machine-learning unit <NUM>, and a memory for a learned model <NUM>, as shown in <FIG>. The learning data set acquisition unit <NUM> and the machine-learning unit <NUM> consists of the processor. The memory for learning data set <NUM> and the memory for learned model <NUM> consist of the above memory.

The data set acquisition unit <NUM> comprises an interface unit, for example, to acquire a plurality of data to form a learning data set (for training) from various devices connected via a communication device. Here, the various devices connected to the learning data set acquisition unit <NUM> are, for example, an external device <NUM> or a terminal device <NUM> used by a worker. In <FIG>, the machine-learning device 6A, the external device <NUM>, and the terminal device <NUM> are shown as separate devices. However, these devices may be combined as appropriate. For example, the external device <NUM> and the terminal device <NUM> may be a single device. The entire machine-learning device 6A, external device <NUM>, and terminal device <NUM> may also be a single device.

The learning data set acquisition unit <NUM> acquires the yarn cone image <NUM> as the input data, for example, from the external device <NUM>. The learning data set acquisition unit <NUM> acquires the remaining amounts of yarns of the yarn cones <NUM> in the yarn cone image <NUM> as the output data, for example, from the terminal device <NUM>. By mapping these input and output data, a single learning data set is formed.

The yarn cone image <NUM> including an arbitrary number of cones is an image containing the arbitrary number of cones <NUM>, as shown in <FIG>. The yarn cone image <NUM> may be an image of the yarn cones <NUM> on or around the flat knitting machine <NUM> (in this embodiment, on the top plate <NUM>) taken by the camera <NUM>. It may also be, for example, an image of the yarn cones <NUM> placed on a simulated table that simulates the top plate <NUM> by a camera for data preparation.

The amounts of yarns remaining in cones <NUM> are the information indicating the amounts of yarns <NUM> remaining wound around the core <NUM> of cones <NUM>. Various data formats can be used for this data. For example, the amounts of yarns remaining in cones <NUM> comprises information indicating "no yarn" or "yarn available". In this case, the amount of yarn remaining is classified into two values, for example, "<NUM>" for "no yarn" and "<NUM>" for "with yarn. The value of the remaining amount of yarn corresponds to the value that the operator, using the terminal device <NUM>, has assigned to the input data. The amounts of yarns remaining in cones <NUM> can be indicated in several steps. In this case, the amount of yarn remaining has many values. For example, when expressing the amount of yarn remaining in eleven steps, the value "<NUM>" indicates that there is "no yarn", "<NUM>% remaining", "<NUM>% remaining",. , "<NUM>% remaining", "<NUM>", "<NUM>",.

The memory for learning data set <NUM> for learning is a database and stores the learning data set obtained by the data set acquisition unit <NUM>. The structure of the database forming the memory for learning data set <NUM> can be modified as needed. For example, in <FIG>, the memory for learning data set <NUM> and the memory for learned model <NUM> are shown as separate storage units. However, they can be a single storage unit (database).

The machine-learning unit <NUM> performs machine-learning using a plurality of data sets for learning stored in the memory <NUM>. This generates a learned model ("learned model 5A") so that the correlation between the input data and the output data in the plurality of data sets for learning can be inferred. In the embodiment, a convolutional neural network (CNN (Convolutional Neural Network)) which will be described in detail is employed as the specific learning method for machine-learning.

The memory for learned model <NUM> is a database for storing the learned model 5A generated by the machine-learning unit <NUM>. The learned model 5A stored in the memory for learned model <NUM> are applied to the real system (image processing device 4A) via communication lines including the Internet or storage media upon request.

Next, the learning method in the machine-learning unit <NUM> using the multiple acquired data sets for learning will be described with reference to <FIG> shows a CNN-based inference model 50A used in the machine-learning device 6A of the first embodiment.

The inference model 50A has an input layer <NUM>, an intermediate layer <NUM>, and an output layer <NUM>. The inference model 50A detects objects in the image (object detection) and detects the status of the detected objects (status detection). When an image contains multiple objects, object detection and status detection are performed for each object.

The input layer <NUM> has a number of neurons corresponding to pixels in the image as the input data, and the pixel value of each pixel is inputted to each neuron.

The intermediate layer <NUM> comprises convolution layers <NUM>, pooling layers <NUM> and full connection layer <NUM>. The convolution layer <NUM> and the pooling layer <NUM> are, for example, provided in alternating layers. The convolution layers <NUM> and the pooling layers <NUM> extract features from the image input via the input layer <NUM>. The full connection layer <NUM> transforms the features extracted from the image by the convolution layers <NUM> and the pooling layers <NUM> using, for example, an activation function into characteristic vector. The full connection layer <NUM> may have multiple layers.

The output layer <NUM> outputs output data including the object detection results and the status detection results, as the inference results based on the characteristic vector outputted from the full connection layer <NUM>. The object detection result is, for example, information that identifies the region (e.g., rectangle, circle, etc.) that includes the detected object. In the case of a rectangle, the lower left and upper right coordinates may be specified, and the center coordinates, width and height may be specified alternatively. The status detection result is, for example, information indicating the status of the detected object, and may be discrete or continuous values. In addition to them, the output data may include, for example, a score indicating the certainty of the estimation results.

Between the layers of the inference model 50A, there are synapses connecting the neurons in the layers. Synapses in the convolutional layer <NUM> and the full connection layer <NUM> of the intermediate layer <NUM> have weights associated with them.

The machine-learning unit <NUM> inputs the learning data set to the inference model 50A and learns the correlation between the yarn cone image <NUM> and the remaining amounts of yarns of cones <NUM> in yarn cone image <NUM>. Specifically, the machine-learning unit <NUM> inputs the yarn cone image <NUM> which is part of the learning data set, as the input data to the input layer <NUM> of the inference model 50A. As a pre-processing step in inputting the yarn cone image <NUM> to the input layer <NUM>, the machine-learning unit <NUM> may apply image adjustments (e.g., image format, image size, image filter, image mask, etc.) to the yarn cone image <NUM>.

The machine-learning unit <NUM> iteratively adjusts the weights associated with each synapse (back-propagation) so that the value evaluated by the error function is smaller. To evaluate the error, an error function is used that compares the remaining amount of yarn (status detection result) indicated by the output data outputted from the output layer <NUM> as a result of inference with the remaining amount of yarn (teacher data) in this learning data set.

When the series of steps have been performed a predetermined number of times, or when the evaluated value of the error function is smaller than the allowable value, the machine-learning unit <NUM> terminates the learning, with assuming that the learning termination conditions have been met. The inference model 50A (all weights associated with each synapse) is then stored in the memory for learned model <NUM> as the learned model 5A.

A machine-learning method corresponding to the machine-learning device 6A will be described with reference to <FIG> is a flowchart showing an example of the machine-learning method for the first embodiment.

The machine-learning method shown below is executed by the machine-learning device 6A, for example. First, a desired number of data sets for learning are prepared as the preliminary preparation for starting the machine-learning. The prepared plurality of data sets for learning are stored in the memory <NUM> (step S200). The number of data sets for learning to be prepared is determined by considering the inference accuracy required for the final learned model 5A.

Several methods can be employed to prepare the data sets for learning. For example, during operation of the flat knitting machine <NUM>, the cameras <NUM> are used to capture the image of the yarn cones <NUM> in various statuses. The operator inputs the remaining amounts of yarns in the cones <NUM> in the yarn cone image <NUM> using a terminal device <NUM> or the like, so as to associate them with the captured cone image <NUM>. By repeating this process, the desired number of data sets for learning are prepared. In addition to this method, a learning data set may be obtained, for example, by using a camera for the data preparation to capture the image of the yarn cones <NUM> placed on a stand that simulates the top plate <NUM>. In doing so, yarn cone image <NUM> are prepared for the learning data set, in various statuses of the remaining amounts of yarns such as with and without yarn. Preferably, the yarn cone image <NUM> are prepared including various yarn cones <NUM> with various positions, various outline shapes, and various outline sizes.

Next, an inference model 50A before learning is prepared (S210) to begin the learning in the machine-learning unit <NUM>. The pre-training inference model 50A prepared here, for example, has the structure shown in <FIG>, and the weight of each synapse is set to an initial value. From the plurality of data sets for learning stored in the memory <NUM>, one learning data set is selected at random, for example (step S220). Input data in the selected learning data set is inputted to the inference model 50A before learning (step S230). In the learning data set shown in <FIG>, the input data is the yarn cone image <NUM> that contains five cones <NUM> in the region identified by the five cone regions A to E (lower left and upper right coordinates), and the output data is the information regarding the five cones <NUM> in the five cone regions A to E with yarn of "<NUM>% remaining", "<NUM>% remaining %", "<NUM>% remaining", "<NUM>% remaining", and "no yarn".

In step S230, the neural network model before training infers the amount of yarn remaining (the result of the status detection). In most cases, the correct information is not inferred regarding the amounts of yarns remaining in the yarn cones <NUM>. Therefore, machine-learning is performed (step S240) using the teacher data (remaining amount of yarn) in the learning data set obtained in step S220 and the remaining amount of yarn output in step S230.

The machine-learning here is back-propagation. For example, the process of comparing the remaining amount of yarn in the teacher data with the amount of yarn outputted from the output layer <NUM> and adjusting the weights assigned to each synapse in the inference model 50A to obtain a preferred inference model 50A. The format of the output data in the output layer <NUM> of the inference model 50A before learning is in the format similar to that of the teacher data.

In the inference result shown in <FIG>, five cones <NUM> in five cone regions A-E (lower left and upper right coordinates) are detected in the yarn cone image <NUM>. The amounts of yarns remaining in the five cone regions A-E are "<NUM>% remaining", "<NUM>% remaining", "<NUM>% remaining", "<NUM>% remaining" and "no yarn". The following are the remaining yarn amounts for the five cones in cone region A-E. There is an error with the teacher data for the amounts of yarns remaining in cones <NUM> in cone region B. Based on this error, the weights of the inference model 50A are adjusted.

Once the machine-learning is performed in step S240, a decision is made as to whether or not to continue machine-learning, e.g., based on the number of remaining untrained data sets for learning (step S250). The learning data set is stored in the memory <NUM>. If the learning end condition is not met and machine-learning is to continue ("NO" in step S250), return to step S220. If the learning end condition is satisfied and machine-learning is terminated ("Yes" in step S250), move to step S260. If machine-learning is to continue, steps S220-S240 are performed multiple times on the inference model 50A being trained, using the untrained learning data set. The accuracy of the learned model 5A generated increases with this number of times.

When the machine-learning process is completed ("Yes" in step S250), the generated inference model 50A is stored in the memory for learned model <NUM> as the learned model 5A (step S260). In this way, the series of machine-learning processes are completed.

In the learning method and the machine-learning method of the machine-learning device 6A, the machine-learning process is repeatedly performed on one (pre-trained) inference model 50A to generate one learned model 5A. This improves the accuracy and generates a learned model 5A that can be applied to the image processing device 4A. However, the invention is not limited to this acquisition method. For example, a plurality of learned model 5A that has been subjected to the machine-learning a predetermined number of times are stored in advance in the learned model memory <NUM> as one candidate. For example, a data set for the determination of validity may be inputted to the multiple learning data set to generate values for the neurons in the output layer. Then, the accuracy of the values identified in the output layer are compared and considered, and the best learned model 5A to be applied to the image processing device 4A may be selected. The data set for the validation may comprise a data set similar to the learning data set and not having been used for training.

As described above, the machine-learning device 6A and the machine-learning method according to this embodiment can generate a learned model 5A that can accurately detect (estimate) the statuses (remaining amounts of yarns) of the yarn cones <NUM> from the camera-captured image <NUM> including one or plural cones <NUM>.

In the first embodiment, the machine-learning device 6A generates one learned model 5A. When the image processing device 4A inputs the camera-captured image <NUM> to the learned model 5A, the learned model 5A detects the remaining amounts of yarns of cones <NUM> in the camera-captured image <NUM>. In contrast, in the second embodiment, the first and second machine-learning devices 6B1 and 6B2 generate the first and second learned models 5B1 and 5B2. In the image processing device 4B, the models SB1 and 5B2 are connected in series such that the first learned model 5B1 is the first stage and the second learned model 5B2 is the second stage. When a camera-captured image <NUM> is inputted to the first learned model 5B1, it detects the remaining amounts of yarns of cones <NUM> in the camera-captured image <NUM>. The other basic structure and operation are the same as in the first embodiment. The characteristic parts of the second embodiment are described. The first and second machine-learning devices 6B1 and 6B2 are described as separate devices. However, the first and second machine-learning devices 6B1 and 6B2 may be a single device.

<FIG> is a schematic block diagram of a flat knitting machine <NUM> equipped with an image processing device 4B according to the second embodiment. A first learned model 5B1 and a second learned model 5B2 are stored in the memory for learned model <NUM> of the image processing device 4B.

The first learned model 5B1 is generated by the first machine-learning device 6B1 and the machine-learning method. The first learned model 5B1 has machine-learned to correlate a cone image <NUM> containing an arbitrary number of cones <NUM> with regions in the yarn cone image <NUM> that contains only one yarn cone <NUM>.

The second learned model 5B2 is generated by the second machine-learning device 6B2 and the machine-learning method. The second learned model 5B2 learns correlation between a single cone image <NUM> containing one yarn cone <NUM> and the remaining amount of yarn of the yarn cone <NUM> in the single cone image <NUM>.

<FIG> is a flowchart, showing the image processing method by the image processing device 4B for the second embodiment.

At a predetermined timing, the acquisition unit <NUM> acquires a camera-captured image <NUM> in which the yarn cones <NUM> on the top plate <NUM> are captured by the camera <NUM> (step S100).

The yarn cone status detection unit <NUM> performs inference using the first learned model 5B1 stored in the memory for learned model <NUM> (step S111). Specifically, the yarn cone status detection unit <NUM> inputs the camera-captured image <NUM> acquired by the acquisition unit <NUM> into the first learned model 5B1. The first learned model 5B1 identifies one or more regions, each containing one yarn cone <NUM> in the camera-captured image <NUM>. In the inference result of step S111 shown in <FIG>, five cones <NUM> are detected in the camera-captured image <NUM>; cone regions A-E containing the five cones <NUM> are identified with two coordinates, the lower left coordinate and the upper right coordinate.

The yarn cone status detection unit <NUM> performs inference (step S112) using the second learned model 5B2 stored in the memory for learned model <NUM>. Specifically, the yarn cone status detection unit <NUM> inputs to the second learned model 5B2 a region image <NUM> that includes one yarn cone <NUM> of the camera-captured image <NUM> acquired by the acquisition unit <NUM>. By doing so, the second learned model detects the remaining amount of yarn in the one yarn cone <NUM> included in the region image <NUM>. The inference result of step S112 shown in <FIG> estimates the amounts of yarns remaining in the five cones from the five regional image <NUM> corresponding to the five cone regions A-E (identified by the lower left and upper right coordinates). The amounts of yarns remaining in the four cones <NUM> in cone regions A-D (lower left and upper right coordinates) are "<NUM>% remaining," "<NUM>% remaining," "<NUM>% remaining," and "<NUM>% remaining"; the amount of yarn remaining in cone region E is "no yarn.

In steps S111 and S112, the yarn cone status detection unit <NUM> makes inferences. If the detection result of the amount of yarn remaining for any of cones <NUM> is "no yarn" ("Yes" in step S120), the output processor <NUM> reports "no yarn" (step S130). If the detection result of the amount of yarn remaining for any of the <NUM> cones is less than or equal to the alert level ("Yes" in step S140), the output processor <NUM> reports "no yarn alert" (step S150). If the detection result of the amounts of yarns remaining for all cones <NUM> is neither "no yarn" nor "no yarn alert", i.e., the detection result of the amounts of yarns remaining for all cones <NUM> is "with yarn" ("NO" in step S120 and "NO" in step S140), the image processing process is completed.

Similar to the image processing device 4A and the image processing method, the image processing device 4B and the image processing method according to the embodiment can detect the statuses of one or more cones <NUM> in the camera-captured image <NUM> (the remaining amount of yarn in this embodiment). Therefore, there is no need to install optical components for respective cones <NUM>. The statuses of cones <NUM> can be detected without being restricted by the number or positions of cones <NUM> placed on or around the flat knitting machine. Thus, the statuses of cones <NUM> can be easily detected, with improved workability and versatility.

The first machine-learning device 6B1 detects the statuses of one or more cones <NUM> positioned on or around the top plate of the flat knitting machine <NUM> and will be described with reference to <FIG> and <FIG>. <FIG> is a schematic block diagram of the first machine-learning device 6B1 for the second embodiment. <FIG> shows an example of a first inference model 50B1 based on a CNN used in the first machine-learning device 6B1 for the second embodiment.

The first machine-learning device 6B1 generates a learned model (first learned model 5B1) that can identify (estimate) an arbitrary number of regions enclosing one yarn cone <NUM> respectively in the cone image <NUM>, based on the cone image <NUM> that includes one or more yarn cones <NUM>.

The learning data set acquisition unit <NUM> obtains the yarn cone image <NUM> as the input data, for example, from an external device <NUM>. The regions containing one yarn cone <NUM> respectively in the yarn cone image <NUM> are acquired as the output data, e.g., from the terminal device <NUM>. By mapping these input and output data, a learning data set is formed and stored in the memory <NUM>. In the learning data set shown in <FIG>, the input data is the cone image <NUM> which contains five cones <NUM> in five cone regions A-E (identified by the lower left and upper right coordinates). The output data is information identifying cone regions A-E (identified by lower left and upper right coordinates) that contain one yarn cone <NUM> each. The input data which is part of the learning data set, may be a single cone image <NUM> containing one yarn cone <NUM>.

The machine-learning unit <NUM> performs the machine-learning using the plurality of data sets for learning stored in the memory <NUM>. This generates the learned model (first learned model 5B1) that infers correlation between input data and output data contained in the plurality of data sets for learning.

Specifically, the machine-learning unit <NUM> performs the machine-learning with learning the yarn cone image <NUM> which is part of the learning data set, and the yarn cone image <NUM> is inputted to the input layer <NUM> of the first inference model 50B1 shown in <FIG>. The regions outputted from the output layer <NUM> as the inference result (object detection result) is compared with the regions where the yarn cones are present (the teacher data) that is part of the learning data set. For example, in the inference result shown in <FIG>, five cones <NUM> are detected in five cone regions A-E (lower left and upper right coordinates are specified) from the cone image <NUM>. The cone regions A-E, which contain five cones, are identified by their two points, namely, the lower left and upper right coordinates. If there is an error between the inference results and the teacher data for the cone regions A-E, the weights in the first inference model S0B1 are adjusted based on the error.

When the learning end condition is met and the machine-learning is completed, the machine-learning unit <NUM> stores the first inference model 50B1 as the first learned model 5B1 in the memory for learned model <NUM>.

The second machine-learning device 6B2 detects the remaining amounts of yarns in one or more cones <NUM> located on or around the flat knitting machine <NUM> and will be described with reference to <FIG> and <FIG>. <FIG> is a schematic block diagram of the second machine-learning device 6B2. <FIG> shows one example of a second inference model 50B2 based on a CNN used in the second machine-learning device 6B2.

The second machine-learning device 6B2 generates the learned model (second learned model 5B2) based on a single cone image <NUM> containing one yarn cone <NUM> and detects (estimates) the remaining amount of yarn of the one yarn cone <NUM> in the yarn cone image <NUM>.

The learning data set acquisition unit <NUM> acquires the single cone image <NUM>, as the input data, for example, from an external device <NUM>. It also acquires output data indicating the amount of yarn remaining in the yarn cone <NUM> within the single cone image <NUM>, for example, from the terminal device <NUM>. By making these input data and output data correspond with each other, one learning data set is formed. It is stored in the memory for learning data set <NUM>. In the learning data set illustrated in <FIG>, the input data is the single cone image <NUM> containing only one yarn cone <NUM>, and the output data is the information indicating that the amount of yarn remaining in the cone <NUM> is "<NUM>%.

The machine-learning unit <NUM> performs the machine-learning, using a plurality of learning data set stored in the memory <NUM>. The machine-learning generates a learned model (second learned model 5B2) that infers correlation between the input data and the output data, both included in the plurality of learning data set.

Specifically, the machine-learning unit <NUM> inputs the single cone image <NUM> that forms the learning data set into the input layer <NUM> of the second inference model 50B2 shown in <FIG>. Then, the output of the remaining amount of yarn (status detection result) outputted from the output layer <NUM> as the inference result is compared with the actual remaining amount of yarn (teacher data) in the learning data set. In this way, the machine-learning is conducted. The inference result illustrated in <FIG> shows that the yarn remaining amount of cone <NUM> in the single cone image <NUM> is inferred as "<NUM>%. " Since there is an error between the teacher data regarding the remaining amount of yarn in cone <NUM>, the weights of the second inference model 50B2 are adjusted based on the error.

When the learning end condition is satisfied, and machine-learning is finished, the machine-learning unit <NUM> stores the second inference model 50B2 as the second learned model 5B2 in the memory for learned data set <NUM>.

The first and second machine-learning devices 6B1 and 6B2 and the machine-learning method of the present embodiment generate the first and second learned models 5B1 and 5B2 in the same manner as the first machine-learning device 6A and the first machine-learning method. The first and second learned models 5B1 and 5B2 and the machine-learning method can accurately detect (estimate), from the camera-captured image <NUM> including the arbitrary number of yarn cones <NUM>, the status (in this embodiment, the yarn residue) of the yarn cones <NUM> in the image <NUM>.

In particular, the first machine-learning device 6B1 generates a first learned model 5B1 such that it identifies regions surrounding one yarn cone <NUM> respectively in the camera-captured image <NUM>. The second machine-learning device 6B2 generates a second learned model 5B2 for detecting the statuses of the respective cones <NUM>. Therefore, there is no need to prepare a large number of images with various combinations of conditions as the necessary learning data set. A learning data set that is suitable for each of the first and second learned models 5B1 and 5B is thus simply prepared. This reduces the time and effort required to prepare the learning data set and improves the inference accuracy by the first and second learned models 5B1 and 5B2.

In the first embodiment, the machine-learning device 6A generates the learned model 5A, and the image processing device 4A detects the remaining amount of yarn in cone <NUM> in the camera-captured image <NUM>, with inputting the camera-captured image <NUM> to the learned model 5A. In the third embodiment, the machine-learning device 6C generates a learned model 5C, and the image processing device 4C inputs the camera-captured image <NUM> to the learned model 5C. The learned model 5C detects the connections between the yarn cones <NUM> and the tensioning devices <NUM> in the camera-captured image <NUM>. Other points are the same as in the first embodiment. In the following, the characteristics of the third embodiment will be described.

<FIG> is a schematic block diagram showing a flat knitting machine <NUM> equipped with the image processing device 4C of the third embodiment. The learned model 5C is stored in the memory for learned model <NUM> of the image processing device 4C.

The learned model 5C has performed the machine-learning, using the machine-learning device 6C and the corresponding machine-learning method, and is capable of correlating the cone image <NUM> including one or more cones <NUM> and the connections between the one or more cones <NUM> and the tensioning devices <NUM>. Specifically, the yarn cone image <NUM> includes one or more cones <NUM> and the yarns <NUM> that are unrolled from the cones <NUM> and connected to the tensioning devices <NUM>, as shown in <FIG>. The learned model 5C extracts, from the yarn cone image <NUM>, the position of each cone <NUM>, the direction in which the yarn <NUM> is unrolled from each cone <NUM> and extends, and the characteristic quantities that exist in the direction where the yarn <NUM> extends and learns them, for identifying the connections between the yarn cones <NUM> and the tensioning devices <NUM>. The aforementioned characteristic quantities are obtained in a range till the yarn reaches the tensioning device <NUM>.

<FIG> is a flowchart showing the image processing method according to the third embodiment of the image processing device 4C. It is assumed that there are n-tensioning devices <NUM> and that the devices are numbered "<NUM>, <NUM>, <NUM>,. , n" in order from the left to the right, for example.

At a predetermined timing, the acquisition unit <NUM> acquires a camera-captured image <NUM> in which cones <NUM> on the top plate <NUM> are captured by the camera <NUM> (step S100).

The yarn cone status detection unit <NUM> refers to the learned model 5C stored in the memory for learned model <NUM> and performs the inference process using the learned model 5C (step S113). The yarn cone status detection unit <NUM> inputs the camera-captured image <NUM> acquired by the acquisition unit <NUM> to the learned model 5C. Then, the positions of one or more cones <NUM>, the directions in which the yarns <NUM> are unrolled from cones <NUM> and extend, and the characteristic quantities that exist in the direction in which the yarns <NUM> extend are extracted in a range till the tensioning devices <NUM>. These data are used to detect the connections between the yarn cones <NUM> and the tensioning devices <NUM> in the camera-captured image <NUM>.

In the inference result of step S113, shown in <FIG>, five cones <NUM> in five cone regions A-E are detected from the camera-captured image <NUM>. The regions are indicated by the lower left and upper right coordinates of the square regions. The yarn <NUM> extending from the cone <NUM> in the cone region A is connected to the first tensioning device <NUM>, the yarn <NUM> extending from the cone <NUM> in the cone region C is connected to the fourth tensioning device <NUM>, the yarn <NUM> extending from the cone <NUM> in the cone region D is connected to the 10th tensioning device <NUM>, and the yarn <NUM> extending from the cone <NUM> in the cone region B is connected to the 17th tensioning device <NUM>. The yarn cone <NUM> in the cone region E has no yarn and is detected as not connected to a tensioning device <NUM>.

The output processor <NUM> determines whether the connection between the yarn cones <NUM> and the tensioning devices <NUM> detected by the yarn cone status detection unit <NUM> in step S113 matches the connection in the knitting data of the flat knitting machine <NUM>. If the connections do not match between them ("Yes" in step S121), the operator is informed that a "work error" has occurred (step S131). On the other hand, if the output processor <NUM> determines that the connections are correct ("NO" in step S121), the image processing process is finished.

According to the image processing device 4C and the image processing method according to the embodiment, the statuses of one or more cones <NUM> in the camera-captured image <NUM> can be detected. In particular, the connections between the cones <NUM> and the tensioning devices <NUM> can be detected. Therefore, there is no need to provide optical components for respective cones <NUM>. The statuses of the cones <NUM> can be detected without restrictions such as the number and positions of the, cones <NUM> on or around the knitting machine. Therefore, the statuses of the cones <NUM> can be easily detected, with improved workability and versatility.

The machine-learning device 6C detects the connection between the yarn cones located on or around the flat knitting machine <NUM> and the tensioning devices and will be described with reference to <FIG> and <FIG>. <FIG> is a schematic block diagram of the machine-learning device 6C for the third embodiment. <FIG> shows an example of a CNN-based inference model 50C used in the machine-learning device 6C according to the third embodiment.

The machine-learning apparatus 6C uses, as the input data, the cone image <NUM> including an arbitrary number of cones <NUM>. The machine-learning device 6C generates a learned model (learned model 5C) that is capable of detecting (estimation) the correlation between the yarn cones <NUM> in the image <NUM> and their connections to the tensioning devices <NUM>.

The learning data set acquisition unit <NUM> acquires the yarn cone image <NUM> as the input data, for example, from the external device <NUM>, and acquires the connections between the yarn cones <NUM> and the tensioning devices <NUM> in the yarn cone image <NUM> as the output data, for example, from the terminal device <NUM>. Then, by making these input data and output data corresponding, the learning data set is formed and stored in the memory <NUM>. In the learning data set illustrated in <FIG>, the input data is the yarn cone image <NUM>. Three cones <NUM> are included in regions defined by the three cone regions A to C (designated by the lower left and upper right coordinates of squares), and the output data is that the yarn cone <NUM> in the cone region C is connected to the "2nd" tensioning device <NUM>, the yarn cone <NUM> in the cone region B is connected to the "4th" tensioning device <NUM>, and the yarn cone <NUM> in the cone region A is connected to the "15th" tensioning device <NUM>.

The machine-learning unit <NUM> learns a plurality of learning data set stored in the memory <NUM> for machine-learning and generates the learned model (learned model 5C) that infers the correlation between the input data and the output data.

Specifically, in the machine-learning unit <NUM>, the yarn cone image or cone image <NUM> in the learning data set are inputted into the input layer <NUM> of the inference model 50C shown in <FIG>. The inferred connection relation (the detection result of the relevant status) is outputted from the output layer <NUM> as the inference result. The output is compared with the correct connection data in the learning data set (the teacher data). Thus, the machine-learning is performed. The inference result illustrated in <FIG> shows that the three cones <NUM> are present in the three cone regions A to C in the yarn cone image <NUM>: the yarn cone <NUM> in the cone region C is connected to the 2nd tensioning device, the yarn cone <NUM> in the cone region B is connected to the third tensioning device, and the yarn cone <NUM> in the cone region A is connected to the fifteenth tensioning device. The regions are indicated by the lower left and upper right coordinates of squares. There is an inconsistency with the teacher data regarding the connection of the cone <NUM> in the cone region B, and the weights of inference model 50C are adjusted based on this error.

When the machine-learning is finished due to the satisfaction of learning end conditions, the machine-learning unit <NUM> stores the inference model 50C in the memory for learned model <NUM> as the learned model 5C.

In this embodiment, the output layer <NUM> of the inference model 50C makes the n-tensioning devices <NUM> correspond to the yarn cone regions by assigning the serial numbers (identification information) of the tensioning devices <NUM> to the yarn cone regions. However, the output layer <NUM> may assign the regions of cone <NUM> with the serial numbers of tensioning devices <NUM> connected to cones <NUM>. In such cases, the format of the output data in the learning data set is changed according to the data format of the output layer <NUM>.

As described above, according to the machine-learning device 6C and the machine-learning method according to this embodiment, from the camera-captured image <NUM> containing an arbitrary number of cones <NUM>, the learned model 5C is generated such it is capable of accurately detecting (estimating) the statuses of the arbitrary number of cones <NUM> included in a camera-captured image <NUM>, in particular in this embodiment, the connection between the yarn cones <NUM> and the tensioning devices <NUM>.

For example, the present invention can be carried out using the inference devices 5A, 5B1, 5B2, and 5C, instead of using the image processing devices 4A-4C. In that case, the inference devices 5A, 5B1, 5B2, and 5C include a memory and at least one processor, and the processor executes the series of processes. The series of processes include the process where a camera-captured image <NUM> including one or plural cones is captured, and the process where, upon input of the camera-captured image <NUM>, the statuses of the yarn cones <NUM> in the camera-captured image <NUM> is inferred. For example, the remaining yarn amounts in the yarn cones, the connections between the yarn cones and the tensioning devices, and the positions where the yarn cones are located, etc. are inferred. The inference devices 5A, 5B1, 5B2, and 5C described above can be more easily implemented to various devices than the image processing devices 4A-4C. When inferring the statuses of cones <NUM>, the learned models 5A, 5B1, 5B2, and 5C, which have been machine-learned by the machine-learning device 6A, 6B1, 6B2, 6A and the machine-learning method, are usable, and the inference method used by the yarn cone status detection unit <NUM> in the image processing devices 4A-4C is usable.

In the above embodiments, the image processing apparatuses 4A-4C are described as implemented in the flat knitting machines 1A-1C. However, the image processing devices 4A-4C may be implemented in any type of knitting machine, such as a circular knitting machine and a warp knitting machine. The image processing devices 4A-4C may, for example, be installed on the knitting machines at the factory where the machine is manufactured, or may be fitted to the knitting machine after shipment from the factory. Further, the functions of the image processing devices 4A-4C may be implemented to the control panels <NUM> of the knitting machines after shipment from the factory.

In the above embodiments, the image processing devices 4A-4C are described as performing the inference using the learning data set as the image processing for the camera-captured image <NUM>. However, the image processing devices 4A-4C may perform pattern recognitions, such as pattern matching, using recognition parameters indicating the features of cone <NUM> in the camera-captured image <NUM>.

The above embodiments use the inference models 50A, 50B1, 50B2, and 50C based on CNN, as the learning method for machine-learning. However, other learning methods can be used. For example, neural networks, ensemble learning (random forest, boosting, etc.), R-CNN (Region-based Convolutional Neural Network), a type of CNN, Fast-R-CNN, Faster-R-CNN, YOLO (You Only Look Once), SSD (Single Shot multi-box Detector) may be used. In this case, the data format of the input data for the input layer <NUM> and the output data from the output layer <NUM> may be changed as appropriate.

Claim 1:
An image processing device configured to detect respective statuses of plural yarn cones (<NUM>, 2A-2E), arranged on a knitting machine main body or at a periphery of the knitting machine main body, characterized by
an acquisition unit (<NUM>) for acquiring a captured image (<NUM>, <NUM>) including plural yarn cones (<NUM>, 2A-2E), arranged on the knitting machine main body or at a periphery of the knitting machine main body; and
a yarn cone status detection unit (<NUM>) for detecting respective statuses of the plural yarn cones (<NUM>, 2A-2E) included in the captured image (<NUM>, <NUM>) by applying image processing to the captured image (<NUM>, <NUM>) acquired by the acquisition unit (<NUM>),
wherein said yarn cone status detection unit (<NUM>) is configured to detect respective residual amounts of yarns of the plural yarn cones (<NUM>, 2A-2E) in the captured image (<NUM>, <NUM>) by applying said image processing to the captured image (<NUM>, <NUM>), and
wherein said yarn cone status detection unit (<NUM>) is configured to:
identify plural regions (A-E) enclosing one yarn cone respectively, by inputting said captured image (<NUM>, <NUM>) acquired by the acquisition unit (<NUM>) to a first learned model (5B1) which has machine-learned the correlation between a cone image (<NUM>) including one yarn cone (<NUM>) and the regions (A-E) respectively enclosing one yarn cone of the plural yarn cones (<NUM>, 2A-2E) in the captured image (<NUM>, <NUM>), and
detect the respective residual amounts of yarns of said plural yarn cones (<NUM>, 2A-2E), one by one, in the captured image (<NUM>, <NUM>), by inputting the cone image (<NUM>) including said one yarn cone (<NUM>) to a second learned model (5A, 5B2) which has machine-learned the correlation between the cone image (<NUM>) including one yarn cone (<NUM>) and the residual amount of yarn of the yarn cone in the cone image (<NUM>).