Patent Description:
In recent years, there has been demand to reuse waste such as scrap as recyclable resources for effective resource utilization. For reuse of waste, it is necessary to determine recyclable resources. Waste determination methods not relying on human power have been conventionally proposed (for example, <CIT> (PTL <NUM>)). Further, PTL <NUM> discloses a platform for scrapping metal which is based on artificial intelligence. Yet further, PTL <NUM> discloses a general concept for recognizing pattern in input data.

PTL <NUM>: <CIT>; PTL <NUM>: <CIT>; PTL <NUM>: <CIT>.

The technique described in PTL <NUM> is intended for determination of waste such as dismantled houses and disaster debris, and methods of efficiently determining scrap such as metal are not studied. For example, iron scrap is distributed in the market as a reusable resource related to iron, and is recycled into iron using electric heating furnaces or the like. Conventionally, grades of scrap are determined visually by workers at iron scrap processing sites. This is because scrap metal pieces after crushing have various scales and the shape of each scrap is different and therefore the whole needs to be visually inspected in order to make grade determination. It is thus difficult to automate scrap determination. Meanwhile, in visual determination works by workers, determination results vary depending on the skill of the workers. There is also a problem of aging of workers and difficulty in securing personnel. Hence, scrap determination techniques have room for improvement.

It could therefore be helpful to provide a scrap determination system and a scrap determination method that can improve scrap determination techniques.

The mentioned objects are met by the subject matter of the independent claims.

A corresponding scrap determination system is according to claim <NUM>, and a corresponding scrap determination method is according to claim <NUM>.

It is thus possible to improve scrap determination techniques.

Some of the disclosed embodiments will be described below, with reference to the drawings.

In the drawings, the same or corresponding parts are given the same reference signs. In the following description of embodiments, the description of the same or corresponding parts is omitted or simplified as appropriate.

<FIG> is a schematic diagram illustrating the overall structure of a scrap determination system <NUM> according to one of the disclosed embodiments. This embodiment describes the case where scrap to be determined is iron scrap, but scrap to be determined is not limited to iron scrap and may be, for example, other metal scrap.

Iron scrap can be roughly divided into two types based on its source. One is process scrap (also called mill scrap) generated in the production stage in the manufacturing industry. Process scrap is recovered by recovery companies, and then distributed under different names such as new scrap (shindachi), steel turnings, and pig scrap. Most of process scrap are taken by steelmakers without undergoing processing (intermediate processing). Process scrap is iron scrap of clear history, and is considered to be useful as with return scrap in terms of quality. Moreover, there is little possibility of foreign matter being mixed in during the generation, recovery, and transportation stages.

The other scrap is obsolete scrap generated as a result of aging of steel structures. Obsolete scrap includes scrap generated during the repair or damage stage. Obsolete scrap is generated in various scenes such as building dismantling, machinery renewal, used automobiles, containers, etc., and in various shapes. Therefore, recovered obsolete scrap is subjected to processing such as sizing, crushing, and volume reduction in order to increase the feed efficiency in steelmaking, and then treated as heavy scrap. Steel plate products such as home appliances, automobile bodies, and vending machines are reduced in volume mainly by crushing, and then subjected to magnetic separation to select only iron. Since such obsolete scrap becomes diverse in each of the generation, recovery, and processing stages, grade determination is performed after the processing. Grade determination for obsolete scrap is made based on the shape, i.e. the thickness, width, length, etc., of the scrap. Currently, in Japan, the Uniform Standards of Ferrous Scraps established by the Japan Ferrous Raw Materials Association in <NUM> is widely used.

As mentioned earlier, grades of scrap are conventionally determined visually by workers at iron scrap processing sites. Such visual determination works by workers have problems such as variation in determination results due to different skill levels of the workers. In view of this, the scrap determination system <NUM> according to this embodiment performs scrap determination based on camera images of iron scrap instead of visual determination by workers.

This embodiment describes an example of determining six types of grades, namely, HS, H1, H2, and H3 which are typical among scraps and L1 and L2 of low iron quality such as rusted galvanized iron plates, although the grades to be determined are not limited to such. The grades to be determined may include new scrap (shear chips), turnings (cutting chips), and the like. Thus, the scrap grades to be determined in this embodiment may include any shape of scrap grades according to the needs of manufacturing sites.

As illustrated in <FIG>, the scrap determination system <NUM> according to this embodiment includes a plurality of cameras <NUM> and an information processing device <NUM>. The plurality of cameras <NUM> and the information processing device <NUM> are connected via a network <NUM>. The plurality of cameras <NUM> are, for example, network cameras, and transmit captured camera images to the information processing device <NUM> via the network <NUM>. Although the number of cameras <NUM> is four in <FIG>, the number of cameras <NUM> is not limited to such. The number of cameras <NUM> may be less than four, and may be one. The number of cameras <NUM> may be more than four. A camera image is an image of iron scrap taken when the iron scrap has been moved to a yard after being transported by a truck. <FIG> illustrates specific examples of camera images taken by the cameras <NUM>. As illustrated in <FIG>, scrap included in each camera image is a mixture of iron scrap of a plurality of grades. The information processing device <NUM> determines the grades of iron scrap in the camera image and the ratio of each grade, based on a plurality of models generated by machine learning. In actual operation, scrap is traded based on the weight ratio of each grade and the total weight of the scrap. This embodiment describes an example in which the information processing device <NUM> determines the ratio relating to the weight of scrap of each grade in a camera image.

The information processing device <NUM> includes a controller <NUM>, a storage <NUM>, an acquisition section <NUM>, and an output section <NUM>.

The controller <NUM> includes at least one processor, at least one dedicated circuit, or a combination thereof. Examples of the processor include a general-purpose processor such as a central processing unit (CPU), and a dedicated processor specialized in a specific process. Examples of the dedicated circuit include a field-programmable gate array (FPGA) and an application specific integrated circuit (ASIC). The controller <NUM> executes a process relating to the operation of the information processing device <NUM> while controlling each component in the information processing device <NUM>.

The storage <NUM> includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of two or more thereof. Examples of the semiconductor memory include a random access memory (RAM) and a read only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read only memory (EEPROM). For example, the storage <NUM> functions as a main storage device, an auxiliary storage device, or a cache memory. The storage <NUM> stores data used for the operation of the information processing device <NUM> and data obtained as a result of the operation of the information processing device <NUM>. For example, the storage <NUM> stores a first scrap determination model <NUM>, a second scrap determination model <NUM>, and a selection model <NUM>.

The first scrap determination model <NUM> is a learning model for determining each grade of scrap included in a camera image and the ratio of the grade based on the camera image. The first scrap determination model <NUM> is generated based on teaching data including first learning images. Each first learning image is an image of single-grade iron scrap. In detail, the first scrap determination model <NUM> is generated by machine learning using a machine learning algorithm such as neural network, based on teaching data including first learning images and track record data of determination for the first learning images. <FIG> schematically illustrates a learning process by the first scrap determination model <NUM>. As illustrated in <FIG>, first learning images which are each an image of single-grade iron scrap are input to an input layer of the first scrap determination model <NUM>. Each image of single-grade iron scrap is associated with track record data of a grade determined by an operator. The weight coefficients between neurons are adjusted using the teaching data, and the learning process by the first scrap determination model <NUM> is performed. Although the respective images are associated with HS, H1, H2, H3, L1, and L2 in <FIG>, the grades are not limited to such. An iron scrap image of any other grade may be used as an image of single-grade iron scrap. Although the first scrap determination model <NUM> is generated based on a multi-layer perceptron composed of an input layer, a hidden layer, and an output layer in <FIG>, the first scrap determination model <NUM> is not limited to such. The first scrap determination model <NUM> may be generated by any other machine learning algorithm. For example, the first scrap determination model <NUM> may be generated based on a machine learning algorithm such as convolutional neural network (CNN) or deep learning.

When determining each grade of scrap included in a camera image and the ratio of the grade, the controller <NUM> determines, based on the area ratio of scrap of each grade in the camera image, the ratio of the scrap, using the first scrap determination model <NUM>. <FIG> schematically illustrates a determination process by the first scrap determination model <NUM>. As illustrated in <FIG>, the controller <NUM> inputs an image of each part of a camera image divided in a grid pattern to the first scrap determination model <NUM>, and determines the grade of scrap in the partial image. The controller <NUM> randomly extracts partial images of the camera image divided in a grid pattern and determines the grade of scrap in each partial image, and calculates the area ratio of each grade of scrap in the image. Further, the controller <NUM> converts the area ratio into the weight ratio based on the bulk density of each scrap. Thus, the controller <NUM> calculates each grade of scrap included in the camera image and the ratio of the grade. Although an example in which the controller <NUM> randomly extracts partial images of the camera image and calculates the area ratio is described here, the determination process is not limited to such. For example, the controller <NUM> may calculate the area ratio based on all partial images constituting the camera image.

The second scrap determination model <NUM> is a learning model for determining each grade of scrap included in a camera image and the ratio of the grade based on the camera image. The second scrap determination model <NUM> is generated based on teaching data including second learning images different from the first learning images. Each second learning image is an image of mixed-grade iron scrap. Mixed-grade iron scrap is iron scrap containing iron scraps of a plurality of grades. In detail, the second scrap determination model <NUM> is generated by machine learning using a machine learning algorithm such as neural network, based on teaching data including second learning images and track record data of determination for the second learning images. <FIG> schematically illustrates a learning process by the second scrap determination model <NUM>. As illustrated in <FIG>, second learning images which are each an image of mixed-grade iron scrap are input to an input layer of the second scrap determination model <NUM>. Each image of mixed-grade iron scrap is associated with track record data of a grade and the ratio of the grade determined by an operator. The weight coefficients between neurons are adjusted using the teaching data, and the learning process by the model is performed. Although the second scrap determination model <NUM> is generated based on a multi-layer perceptron composed of an input layer, a hidden layer, and an output layer in <FIG>, the second scrap determination model <NUM> is not limited to such. The second scrap determination model <NUM> may be generated by any other machine learning algorithm. For example, the second scrap determination model <NUM> may be generated based on a machine learning algorithm such as convolutional neural network (CNN) or deep learning. When determining each grade of scrap included in a camera image and the ratio of the grade using the second scrap determination model <NUM>, the controller <NUM> randomly extracts partial images of the camera image divided and determines the grade of scrap in each partial image, and calculates the weight ratio of each grade of scrap in the image, as in the first scrap determination model <NUM>. This is because the determination accuracy can be improved by dividing an image and determining randomly selected images many times. Although an example in which the controller <NUM> randomly extracts partial images of the camera image and calculates the weight ratio of each grade of scrap is described here, the determination process is not limited to such, as in the first scrap determination model <NUM>. For example, the controller <NUM> may calculate the weight ratio based on all partial images constituting the camera image.

The selection model <NUM> is a model for estimating, based on a camera image, which of the first scrap determination model <NUM> and the second scrap determination model <NUM> outputs a more probable solution when determining each grade of scrap included in the camera image and the ratio of the grade. The selection model <NUM> selects the model that outputs a more probable solution, based on the estimation result. The controller <NUM> determines each grade of scrap and the ratio of the grade based on the camera image, using the model selected by the selection model <NUM>. In other words, the selection model <NUM> determines which of the first scrap determination model <NUM> and the second scrap determination model <NUM> is to be used for scrap grade determination, based on the camera image. Teaching data for the selection model <NUM> includes a camera image of scrap acquired from each camera <NUM> via the network <NUM>, each grade of scrap and the ratio of the grade estimated by the first scrap determination model <NUM>, each grade of scrap and the ratio of the grade estimated by the second scrap determination model <NUM>, and track record data of each grade and the ratio of the grade determined by an operator. The track record data for model selection is based on the determination result when inputting the camera image to each of the first scrap determination model <NUM> and the second scrap determination model <NUM> and the result of the operator determining the grades and the ratio of each grade for the camera image. The selection model <NUM> is an estimation model generated by machine learning using a machine learning algorithm such as neural network, based on such teaching data. For example, the selection model <NUM> is generated based on a machine learning algorithm such as multi-layer perceptron, convolutional neural network (CNN), or deep learning.

The acquisition section <NUM> acquires a camera image including scrap from each camera <NUM> via the network <NUM>. The acquisition section <NUM> includes at least one communication interface. Examples of the communication interface include a LAN interface, a WAN interface, an interface that complies with a mobile communication standard such as Long Term Evolution (LTE), <NUM> (4th generation), or <NUM> (5th generation), and an interface that complies with short-range wireless communication such as Bluetooth® (Bluetooth is a registered trademark in Japan, other countries, or both). The acquisition section <NUM> receives data used for the operation of the information processing device <NUM>, and transmits data obtained as a result of the operation of the information processing device <NUM>.

The output section <NUM> includes at least one output interface. An example of the output interface is a display. The display is, for example, a liquid crystal display (LCD) or an organic electroluminescent (EL) display. The output section <NUM> outputs data obtained as a result of the operation of the information processing device <NUM>. The output section <NUM> may be connected to the information processing device <NUM> as an external output device, instead of being included in the information processing device <NUM>. As a connection method, any method such as USB, HDMI®, or Bluetooth® may be used.

The functions of the information processing device <NUM> are implemented by a program according to this embodiment being executed by a processor corresponding to the controller <NUM>. That is, the functions of the information processing device <NUM> are implemented by software. The program causes a computer to execute the operation of the information processing device <NUM>, thus causing the computer to function as the information processing device <NUM>. In other words, the computer executes the operation of the information processing device <NUM> according to the program to thus function as the information processing device <NUM>.

In this embodiment, the program can be recorded in a computer-readable recording medium. The computer-readable recording medium includes a non-transitory computer-readable medium, such as a magnetic recording device, an optical disc, a magneto-optical recording medium, or a semiconductor memory. The program is distributed, for example, by selling, transferring, or renting a portable recording medium having the program recorded therein, such as a digital versatile disc (DVD) or a compact disc read only memory (CD-ROM). The program may be distributed by storing the program in a storage of a server and transmitting the program from the server to another computer. The program may be provided as a program product.

In this embodiment, for example, the computer once stores, in the main storage device, the program recorded in the portable recording medium or the program transmitted from the server, and then reads the program stored in the main storage device by the processor and executes a process according to the read program by the processor. The computer may read the program directly from the portable recording medium and execute the process according to the program. Each time the computer receives the program from the server, the computer may execute the process according to the received program. Without the program being transmitted from the server to the computer, the process may be executed by application service provider (ASP) services that implement the functions by only execution instruction and result acquisition. The program includes information that is to be processed by an electronic computer and is equivalent to a program. For example, data that is not a direct command to the computer but has the property of defining a process of the computer is "equivalent to a program".

All or part of the functions of the information processing device <NUM> may be implemented by a dedicated circuit corresponding to the controller <NUM>. That is, all or part of the functions of the information processing device <NUM> may be implemented by hardware.

A scrap determination method executed by the scrap determination system <NUM> according to one of the disclosed embodiments will be described below. <FIG> is a flowchart illustrating a scrap determination method according to one of the disclosed embodiments.

First, each camera <NUM> in the scrap determination system <NUM> takes a camera image including scrap (step S10). The camera <NUM> then transmits the camera image to the information processing device <NUM> via the network <NUM>. The acquisition section <NUM> in the information processing device <NUM> acquires the camera image via the network <NUM> (step S20).

Following this, the controller <NUM> determines, based on the acquired camera image, which of the first scrap determination model <NUM> and the second scrap determination model <NUM> is to be used, using the selection model <NUM> (step S30).

The controller <NUM> then determines each grade of scrap included in the camera image and the ratio of the grade, using the model selected by the selection model <NUM> out of the first scrap determination model <NUM> and the second scrap determination model <NUM> (step S40).

The controller <NUM> then instructs the output section <NUM> to output the grade of scrap and the ratio determined in step S40. The output section <NUM> outputs the grade of scrap and the ratio determined in step S40 (step S50).

Thus, the scrap determination system <NUM> according to one of the disclosed embodiments can automatically determine scrap grades and their ratios from a camera image of scrap taken by each camera <NUM>, using the first scrap determination model <NUM> or the second scrap determination model <NUM>. Here, which of the first scrap determination model <NUM> and the second scrap determination model <NUM> is to be used is selected by the selection model <NUM>, so that a more appropriate model is automatically selected. In other words, the scrap determination system <NUM> according to one of the disclosed embodiments can determine and output scrap grades and their ratios without manual intervention. The scrap determination system <NUM> according to one of the disclosed embodiments can therefore improve scrap determination techniques.

While the presently disclosed techniques have been described by way of the drawings and embodiments, various changes and modifications may be easily made by those of ordinary skill in the art based on the present disclosure. Such changes and modifications are therefore included in the scope of the present disclosure. For example, the functions included in the components, steps, etc. may be rearranged without logical inconsistency, and a plurality of components, steps, etc. may be combined into one component, step, etc. and a component, step, etc. may be divided into a plurality of components, steps, etc..

The scope of protection of the present invention is defined by the appended claims.

For example, in the learning process and the determination process of each of the first scrap determination model <NUM>, the second scrap determination model <NUM>, and the selection model <NUM>, the controller <NUM> may use zoom information corresponding to each image. In the case of using zoom information, each camera <NUM> transmits, together with a camera image, zoom information of open network video interface forum (ONVIF) data corresponding to the camera image, to the information processing device <NUM> via the network <NUM>. For example, the first learning images, the second learning images, and the camera images may each be normalized based on zoom information corresponding to the image. In detail, the controller <NUM> normalizes each of the first learning images, the second learning images, and the camera images to a predetermined magnification factor based on the zoom information corresponding to the image. The controller <NUM> then performs the learning process using the normalized first learning images or second learning images, and performs the determination process based on the camera images. As a result of normalizing each image by such a normalization process, the determination accuracy of the scrap determination system <NUM> can be enhanced.

In the case of normalizing each image to a predetermined magnification factor based on the corresponding zoom information, the controller <NUM> may classify images into groups and perform normalization to a different magnification factor for each group based on zoom information. <FIG> is a conceptual diagram of normalization for each group of camera images. In <FIG>, each camera image is classified depending on the magnification factor. Specifically, in the case where the magnification factor of the camera image is in a range R<NUM> of x<NUM> or more and less than x<NUM> (hereafter also referred to as "first range R<NUM>"), the camera image is classified into a first group. In the case where the magnification factor of the camera image is in a range R<NUM> of x<NUM> or more and less than x<NUM> (hereafter also referred to as "second range R<NUM>"), the camera image is classified into a second group. In the case where the magnification factor of the camera image is in a range R<NUM> of x<NUM> or more and less than x<NUM> (hereafter also referred to as "third range R<NUM>"), the camera image is classified into a third group. In the case where the magnification factor of the camera image is in a range R<NUM> of x<NUM> or more and less than x<NUM> (hereafter also referred to as "fourth range R<NUM>"), the camera image is classified into a fourth group. In the case where the magnification factor of the camera image is in a range R<NUM> of x<NUM> or more and less than x<NUM> (hereafter also referred to as "fifth range R<NUM>"), the camera image is classified into a fifth group. The camera images in the first range R<NUM>, the second range R<NUM>, the third range R<NUM>, the fourth range R<NUM>, and the fifth range R<NUM> are respectively normalized to magnification factors X<NUM>, X<NUM>, X<NUM>, X<NUM>, and X<NUM> each as a normal magnification factor of the corresponding range. That is, the first learning images, the second learning images, and the camera images are each normalized to a magnification factor that differs depending on the zoom information corresponding to the image. In other words, the controller <NUM> normalizes each camera image to one of a plurality of normal magnification factors that is selected based on the zoom information of the camera image. In this way, variation in image resolution caused by excessive enlargement or reduction of camera images can be suppressed, and the determination accuracy of the scrap determination system <NUM> can be enhanced. Although the controller <NUM> classifies images into five groups and normalizes each image to the corresponding one of the five magnification factors in <FIG>, the classification is not limited to such. For example, the number of groups into which images are classified may be four or less or six or more, and the controller <NUM> may normalize each image to a magnification factor that differs depending on the group.

Although an example in which zoom information of camera images is used in each of the learning process and the determination process is described above, the information used is not limited to such. For example, the scrap determination system <NUM> may use at least part of ONVIF data obtained from each camera <NUM>, in each of the learning process and the determination process. The ONVIF data includes pan, tilt, and zoom information. That is, the scrap determination system <NUM> may perform the learning process and the determination process using at least one of pan, tilt, and zoom information.

For example, in the learning process and the determination process of each of the first scrap determination model <NUM>, the second scrap determination model <NUM>, and the selection model <NUM>, the controller <NUM> may use information about a carry-in company that carries scrap in. This makes it possible to perform determination in consideration of the tendency of scrap carried in by each carry-in company, so that the determination accuracy of the scrap determination system <NUM> can be improved.

For example, the scrap determination system <NUM> and scrap determination system <NUM> may further accumulate each camera image used in the determination process as new teaching data after the determination. The controller <NUM> may then relearn, based on the camera image, the first scrap determination model <NUM>, the second scrap determination model <NUM>, and the selection model <NUM>, using the result of the operator determining each grade and the ratio of the grade. For example, if there is a problem with the output result (determination result), the output information having the problem, the camera image and the track record data corresponding to the information may be used as teaching data to perform relearning of at least one of the first scrap determination model <NUM>, the second scrap determination model <NUM>, and the selection model <NUM>. This can improve the determination accuracy and speed of the first scrap determination model <NUM>, the second scrap determination model <NUM>, and the selection model <NUM>.

Claim 1:
A scrap determination system (<NUM>) comprising:
an acquisition section (<NUM>) configured to acquire a camera image including scrap;
a first scrap determination model (<NUM>) generated using teaching data including first learning images, and configured to determine, based on the camera image, each grade of the scrap included in the camera image and a ratio of the grade;
a second scrap determination model (<NUM>) generated using teaching data including second learning images different from the first learning images, and configured to determine, based on the camera image, each grade of the scrap included in the camera image and a ratio of the grade;
a selection model (<NUM>) configured to determine which of the first scrap determination model (<NUM>) and the second scrap determination model (<NUM>) is to be used, based on the camera image; and
an output section (<NUM>) configured to output information of each grade of the scrap and a ratio of the grade determined based on the camera image using a model selected by the selection model (<NUM>) out of the first scrap determination model (<NUM>) and the second scrap determination model (<NUM>), wherein the first learning images are each an image of single-grade metal scrap, and
when determining each grade of the scrap included in the camera image and a ratio of the grade using the first scrap determination model (<NUM>), the ratio of the grade is determined based on an area ratio of scrap of the grade in the camera image; the second learning images being each an image of mixed-grade metal scrap; and
the ratio of each grade being a weight ratio to the total weight of scrap included in the camera image.