Material classification using specular gloss

Gloss-based material classification of an object fabricated from an unknown material, particularly where the unknown material is one from a limited set of predetermined materials. The object is illuminated with an area light source such that the object is illuminated from multiple angles. An image of the object is obtained, and specular reflections from the object are measured by analyzing the image. The object material is classified based on a number of high-intensity specular reflections.

FIELD

The present disclosure relates to material classification, and in particular relates to classification of an object fabricated from an unknown material.

BACKGROUND

In the field of material classification it is common to use a classification pipeline, in which materials are classified according to color. For example, in a recycling pipeline, spectral colors of different objects can be used to sort the objects for different types of recycling (e.g., paper vs. aluminum).

SUMMARY

One difficulty with the above arrangement is that some objects are difficult to classify using spectral color or other conventional means. In particular, objects composed of different materials that have similar color properties are difficult to distinguish.

The foregoing situation is addressed by classifying an object fabricated from an unknown material based on specular reflections measured when the object is subjected to illumination by an area light source (such as an array of multiple light sources), which illuminates the object from several angles.

Thus, an example embodiment described herein performs material classification of an object fabricated from an unknown material. The object is illuminated with an area light source such as an array of multiple light sources, so that the object is illuminated from multiple angles. A grayscale image of the object is obtained. Specular reflections from the object are measured by analyzing the grayscale image. The object is classified based on the measured intensity of the specular reflections, commonly referred to as glossiness.

By classifying objects based on specular reflections measured when the object is subjected to illumination by an area light source which illuminates the object from several angles, it is ordinarily possible to classify objects composed of different materials that have similar color properties that are difficult to distinguish, and which otherwise might be left unknown. Moreover, by illuminating the object with an area light source, the object is illuminated from multiple different angles such that if the object is glossy and is likely to emit specular reflections, the number and intensity of specular reflections is enhanced.

In one example aspect, the classification is based on whether the number of high intensity specular reflections is large or the number of high intensity specular reflections is small.

In another example aspect, the classifying includes generating a histogram of intensity values of the specular reflections, and differentiating the object from another object by comparing a value of a statistical metric of the object against a value of the statistical metric of another object. In example aspects described herein, the statistical metric is kurtosis, skewness, or an average intensity of a percentage of pixel values of the histogram. In another aspect, the grayscale image is processed to increase contrast prior to generating the histogram.

In still another aspect, the area light source includes an array of multiple light sources.

In yet another aspect, differentiation is performed between objects fabricated from materials which have similar spectral signatures but are formed from different materials. For example, differentiation is performed between similarly-colored objects such as black-colored objects.

In another aspect, an earlier stage of a multi-stage classification process performs material classification based on spectral signatures, and indeterminate objects are delivered for classification according to the embodiment.

In still another aspect, multiple objects are simultaneously classified.

This brief summary has been provided so that the nature of this disclosure may be understood quickly. A more complete understanding can be obtained by reference to the following detailed description and to the attached drawings.

DETAILED DESCRIPTION

FIG. 1is an example embodiment of a two-stage classification system according to the description herein, in the form of a recycling system10in which objects to be recycled are classified according to the materials from which the objects are fabricated, and the classified objects are sorted for recycling according to their material classification. As shown inFIG. 1, recycling system10includes a first stage classification system10aincluding first stage classification and sorting controller100, and a second stage classification system10bincluding second stage classification and sorting controller200.

Both of the first and second stage classification systems10aand10boperate to perform material classification of an object fabricated from an unknown material. The first stage classification system of this example embodiment performs material classification using spectral reflection characteristics of the object. The objects are sorted based on material classification into multiple different classification categories, such as classification A shown at41a, classification B shown at41b, and so on through classification N shown at41n. Such classifications might, for example, include classification into different metals such as classification into brass, copper and aluminum, and/or might include classification into differently-colored papers and/or differently colored plastics and glass.

At least in part because the first stage classification system10aperforms material classification using spectral reflectance, it may be difficult for the first stage classification system to differentiate between different materials having similar spectral reflectance signatures. As one example, it may be difficult to classify as between different materials that are colored black, due in part because black-colored materials have similar spectral reflectance signatures. Such objects are sorted by the first stage classification system10ainto an indeterminate category at41z, for further classification by the second stage classification system10b.

The second stage classification system10bperforms material classification of an object fabricated from an unknown material, in which the objects tend to be similarly-colored or have similar spectral reflectance signatures. Briefly, the objects are subjected to illumination by an area light source such as an array of multiple light sources, and specular reflections from the objects are measured. Material classification is effected based on whether the number of high-intensity specular reflections is large or the number of high-intensity specular reflections is small. In that regard, the number of high-intensity specular reflections gives an indication of glossiness. In one example, histograms may be formed for pixel intensities of captured images, and statistical analysis of the histograms may be used for differentiation between materials. The use of area illumination ensures that glossy objects, which are more likely to form specular reflections than matte objects, are illuminated from a sufficiently large number of angles so as to enhance the occurrence of specular reflections. As such, the second stage classification system10bis effective at differentiating between different materials having similar spectral reflectance signatures but different glossiness properties, such as black objects fabricated from black rubber on the one hand, and black objects fabricated from HIPS plastic on the other. Such objects may be sorted into bins241aand241bas shown inFIG. 1, with other objects being sorted into further bins such as at241c.

Generally, the first stage classification system10amay be formed in accordance with the description provided in U.S. application Ser. No. 14/092,492, “Material Classification Using Spectral BRDF Slices”, filed Nov. 27, 2013 , which is incorporated herein by reference as if set forth in full.

As generally described in the aforementioned application, objects11a,11b, etc. are conveyed on a first stage conveyor mechanism12to a first stage classification station20, where the objects are classified according to their material, and thence to a first stage sorting station30, where the objects are sorted according to their material classification. First stage classification station20includes plural light sources21and22, together with a first stage camera24for capturing images of objects positioned at first stage classification station20. The object at the first stage classification station is illuminated individually by each of the plural light sources under control of first stage classification and sorting controller100, and first stage camera24captures one or more images for each individual illumination. Under control of the first stage classification and sorting controller100, a classification is made for the material from which the object is fabricated.

First stage conveyor mechanism12continues to convey the object to first stage sorting station30, where first stage sorting actuator31sorts the objects according to the material classification. Sorting is controlled by first stage classification and sorting controller100, which commands first stage actuator mechanism31to sort the classified objects into multiple receptacles41a,41band41band so on through receptacle41n.

Indeterminate materials that the first stage classification system is unable to differentiate and/or to classify are sorted into bin41z, and thence delivered to the second stage classification system10b.

As shown inFIG. 1, in the second stage, objects211a,211band211care conveyed on a second stage conveyor mechanism212to a second stage classification station220, where the objects are classified according to their material, and thence to a second stage sorting station230, where the objects are sorted according to their material classification. Second stage classification station220includes an area light source221such as multiple light sources arranged in an array, together with a second stage camera224for capturing images of objects positioned at second classification station220. The object at the second stage classification station is illuminated by the area light source221under control of second stage classification and sorting controller200, and the second stage camera224captures one or more images of the illuminated object. Under the control of the second stage classification and sorting controller200, a classification is made of the material from which the object is fabricated.

Second stage conveyor mechanism212continues to convey the object to second stage sorting station230, where second stage sorting actuator231sorts the object according to the material classification. Sorting is controlled by second stage classification and sorting controller200, which commands second stage actuator mechanism231to sort the classified objects into multiple receptacles241a,241band241c.

Thus, as described herein, an earlier stage of a multi-stage classification process performs material classification based on spectral signatures, and indeterminate objects are delivered for classification according to the embodiment. In the second stage, for example, differentiation can be performed between objects fabricated from materials which have similar spectral signatures but are formed from different materials. In one specific example, differentiation is performed between black-colored objects.

FIG. 2is a more detailed view of an object on conveyor mechanism212at second stage classification station220. In this figure, object211bis subjected to illumination by area illumination light source221, which in this embodiment is an array of multiple light sources, and an image of the object is captured by camera224. In one example, the area illumination is broad-band visible light illumination, and camera224is an RGB camera which captures color or grayscale images in the visible light spectrum, with RGB and/or grayscale values for each pixel of the image. Meanwhile, the multiple light sources of the area light source221ordinarily ensure that the object is illuminated from a sufficiently large number of angles, so that camera224is able to capture sufficient specular reflections to classify the object, even from arbitrarily shaped objects.

In that regard, camera224may capture a grayscale image directly, whereas in other examples, camera224may convert or transform a color image to a grayscale image. Camera224may also transform or convert the RGB image to another color space indicative of intensity, such as Hue-Saturation-Luminance (HSV) or a LAB color space. In addition, camera224may capture a color image in order to segment multiple objects being examined simultaneously, and then may convert the segmented images of each object into grayscale images to classify each object, as described below with respect toFIG. 4.

FIG. 3is a view for explaining the architecture of second-stage classification and sorting controller200.

As shown inFIG. 3, second stage classification and sorting controller200includes central processing unit (CPU)210which interfaces with computer bus214. Also interfacing with computer bus214are non-volatile memory256(e.g., a hard disk or other nonvolatile, non-transitory storage medium), network interface211, keyboard interface212, camera interface213, random access memory (RAM)216for use as a main run-time transient memory, read only memory (ROM)216a, and display interface217for a display screen or other output.

RAM216interfaces with computer bus214so as to provide information stored in RAM216to CPU210during execution of the instructions in software programs, such as an operating system, application programs, image processing modules, and device drivers. More specifically, CPU210first loads computer-executable process steps from non-volatile memory256, or another storage device into a region of RAM216. CPU210can then execute the stored process steps from RAM216in order to execute the loaded computer-executable process steps. Data also can be stored in RAM216so that the data can be accessed by CPU210during the execution of the computer-executable software programs, to the extent that such software programs have a need to access and/or modify the data.

As also shown inFIG. 3, non-volatile memory256contains computer-executable process steps for operating system218, and application programs219, such as programs for monitoring status and for display of status of the second stage. Non-volatile memory256also contains computer-executable process steps for device drivers for software interface to devices, such as input device drivers220, output device drivers221, and other device drivers222.

Non-volatile memory256also stores a module for second stage classification and sorting controller module275. The module comprises computer-executable process steps stored on a non-transitory computer readable storage medium such as non-volatile memory256, wherein the process steps operate to control the area light source221, camera224and sorting actuator231. According to such control, objects fabricated from an unknown material are subjected to an illumination by area light source221, and specular reflections from the object are measured using camera224. Material classification is effected based on whether the number of high-intensity specular reflections is large or the number of high-intensity specular reflections is small. For this purpose and other purposes, a module may be provided for histogram analysis, whereby intensity values of image pixels captured by camera224are formed into histograms, and statistical metrics such as kurtosis and skewness are applied against the histogram. For this and other purposes, a classification machine may be trained so as to differentiate between materials based on the captured images, including for example the histograms and the statistical metrics derived from the histograms.

The computer-executable process steps for these modules may be configured as part of operating system218, as part of an output device driver in output device drivers221, or as a stand-alone application program. These modules may also be configured as a plug-in or dynamic link library (DLL) to the operating system, device driver or application program. It can be appreciated that the present disclosure is not limited to these embodiments and that the disclosed modules may be used in other environments.

FIG. 4is a flow diagram illustrating the general flow of processing performed by the second stage classification and sorting controller.

Briefly, inFIG. 4, material classification is performed, of an object fabricated from an unknown material. The object is illuminated with an area light source comprising multiple light sources, and the object is illuminated from multiple angles. A grayscale image of the object is captured. Specular reflections from the object are measured by analyzing the grayscale image. The object is classified based on the measured specular reflections.

In this regard,FIG. 4describes material classification in the context of classifying several objects in parallel, as described more fully below with respect toFIG. 7. Nevertheless, it should be understood that the material classification described below could also be applied to a situation where objects are classified alone, or where multiple objects are classified one at a time, in series (e.g., by capturing a separate image of each object).

In step S401, first stage classification and sorting is performed, as described above with respect toFIG. 1. In particular, first stage classification and sorting may perform material classification using spectral reflection characteristics of the object, such as spectral signatures.

As a result of step S401, one or more objects may be classified as first stage classified objects in step S402. In particular, some objects can be easily classified by spectral reflection characteristics, and as a result do not require further second stage classification. For example, different metals such as brass, copper and aluminum, differently-colored papers and/or differently colored plastics and glass might be adequately classified from spectral reflection characteristics alone. Nevertheless, other objects, such as black-colored objects, may be difficult or impossible to classify using spectral reflection characteristics, and for these objects, the process proceeds to step S403.

In step S403, indeterminate objects, i.e., those that could not be classified during the first stage, are delivered for the second stage of processing. For example, as discussed above with respect toFIG. 1, objects which have been determined to be indiscriminate might be conveyed by second stage conveyor mechanism212to a second stage sorting station230.

In step S404, the one or more indeterminate objects consisting of, for example, glossy and matte materials such as black rubber and black plastic, are illuminated by multiple light sources from several angles. For example, the object(s) may be illuminated from multiple angles by an area light source including an array of multiple light sources. By virtue of this arrangement, it is ordinarily possible to ensure that the camera (e.g., camera224) captures several specular reflections from each object, even if the object is arbitrarily shaped.

In that regard, an example of such illumination is illustrated inFIG. 2, described above. In the example shown inFIG. 2, object211bis subjected to illumination by area illumination light source221, which in this embodiment is an array of multiple light sources from multiple angles. In one example, the area illumination is broad-band visible light illumination.

In step S405, a color image of the objects is captured by the camera. As described below, classification in the second stage is performed using a grayscale or other intensity image. Nevertheless, in the context of classifying multiple objects at once, a color image can be used to segment the image into images of each object, which are then converted to individual grayscale images.

Thus, in step S406, the image is segmented based on objects. Put another way, the color image is used to segment the image of multiple objects into respective separate images for each object. For example, image processing techniques such as chroma-keying can be used to segment the color image into respective images for each object.

In step S407, each respective image of each object is processed to improve classification accuracy. In particular, a grayscale image of each object may be obtained by converting or transforming the RGB image of each object, although a grayscale image may also be captured directly by, for example, capturing an image in a black-and-white mode. At any rate, each object can be classified at the second stage by using its grayscale image (or another intensity image).

Taking a grayscale image of the object, a reflectively flat material such as black rubber will appear as an evenly black object, whereas a glossy material such as black high impact polystyrene (HIPS) will contain black regions where no specular reflections are captured, mixed with white regions where specular reflections of the light source create intensely illuminated regions.

In this regard,FIG. 5is a view for explaining differences in specular reflection for objects having similar reflection characteristics, such as differences in specular reflection between an object formed of black rubber and an object formed of black high impact polystyrene (HIPS).

As shown inFIG. 5, rubber object251appears evenly black in the grayscale image, whereas HIPS object252is generally black, but includes white regions such as253and253bwhere specular reflections of the light source create intensely illuminated regions.

Returning toFIG. 4, in step S407, the raw grayscale images can be processed using techniques such as equalization to, for example, increase contrast for better classification accuracy. Thus, in this example, a grayscale image is processed to increase contrast prior to generating a feature vector such as a histogram.

In step S408, a feature vector, such as an intensity histogram, is created from the object pixels of the grayscale image. An intensity histogram is one helpful way of differentiating lower-intensity specular reflections from higher-intensity specular reflections, thereby to determine the number of high-intensity specular reflections.

In particular,FIG. 6is an example of histograms of intensity values of imaged pixels, comparing histogram values for a black rubber and histogram values for a HIPS plastic. As shown inFIG. 6, the black rubber histogram should be heavily concentrated on the black side due to the lack of specular highlights, while the HIPS histogram should be relatively bright due to its glossiness. In such an example, therefore, the classification can be based on whether the number of high intensity specular reflections is large or the number of high intensity specular reflections is small, as described below.

In step S409, a classifier is calculated based on the feature vector, such as the average intensity of the top 10% of the histogram mass. In particular, material classification can be performed by several methods, using the shape of the histogram for each object. For example, statistical metrics such as kurtosis and skewness are possible candidates for differentiation. As mentioned in step409, taking the average intensity of the top 10% of the pixel values can also produce robust contrast between the two materials. The percentage value can be adjusted for optimal performance.

Thus, in this example, the classification includes generating a histogram of intensity values of the specular reflections, and differentiating the object from another object by comparing a value of a statistical metric of the object against a value of the statistical metric of another object.

In step S410, the feature vectors are compared using the classifier, by comparing to a threshold to determine the material type of each object. For purposes of conciseness, the following description refers to the statistical metric as an average intensity of a percentage of pixel values of the histogram, as described above with respect to step409. As mentioned above, a black rubber histogram should be heavily concentrated on the black side due to the lack of specular highlights, while the HIPS histogram should be relatively bright due to its glossiness. A learning process or separate input can be used to determine an appropriate threshold. For example, an average intensity of the top 10% of the pixel values of different objects can be determined by a learning process, and a threshold to differentiate between the two can be obtained therefrom to be applied to histograms of objects to be sorted. Of course, this is simply an example, and the value and type of threshold will depend on the statistical measure being used.

In step S411, if the average intensity of the top 10% of the histogram mass is below the threshold, the object is classified as a matte material such as black rubber. Meanwhile, in step S412, if the average intensity of the top 10% of the histogram mass is above the threshold, the object is classified as a glossy material such as HIPS plastic.

FIG. 7is an example of another embodiment in which multiple objects are classified simultaneously by simultaneous illumination at a classification station.

In particular, as shown inFIG. 7, an imaging surface312includes multiple objects placed thereon, such as objects311a,311band311c. Camera324captures an image of the set of objects, which are simultaneously illuminated from multiple angles by illumination sources321.

According to other embodiments contemplated by the present disclosure, example embodiments may include a computer processor such as a single core or multi-core central processing unit (CPU) or micro-processing unit (MPU), which is constructed to realize the functionality described above. The computer processor might be incorporated in a stand-alone apparatus or in a multi-component apparatus, or might comprise multiple computer processors which are constructed to work together to realize such functionality. The computer processor or processors execute a computer-executable program (sometimes referred to as computer-executable instructions or computer-executable code) to perform some or all of the above-described functions. The computer-executable program may be pre-stored in the computer processor(s), or the computer processor(s) may be functionally connected for access to a non-transitory computer-readable storage medium on which the computer-executable program or program steps are stored. For these purposes, access to the non-transitory computer-readable storage medium may be a local access such as by access via a local memory bus structure, or may be a remote access such as by access via a wired or wireless network or Internet. The computer processor(s) may thereafter be operated to execute the computer-executable program or program steps to perform functions of the above-described embodiments.

According to still further embodiments contemplated by the present disclosure, example embodiments may include methods in which the functionality described above is performed by a computer processor such as a single core or multi-core central processing unit (CPU) or micro-processing unit (MPU). As explained above, the computer processor might be incorporated in a stand-alone apparatus or in a multi-component apparatus, or might comprise multiple computer processors which work together to perform such functionality. The computer processor or processors execute a computer-executable program (sometimes referred to as computer-executable instructions or computer-executable code) to perform some or all of the above-described functions. The computer-executable program may be pre-stored in the computer processor(s), or the computer processor(s) may be functionally connected for access to a non-transitory computer-readable storage medium on which the computer-executable program or program steps are stored. Access to the non-transitory computer-readable storage medium may form part of the method of the embodiment. For these purposes, access to the non-transitory computer-readable storage medium may be a local access such as by access via a local memory bus structure, or may be a remote access such as by access via a wired or wireless network or Internet. The computer processor(s) is/are thereafter operated to execute the computer-executable program or program steps to perform functions of the above-described embodiments.

The non-transitory computer-readable storage medium on which a computer-executable program or program steps are stored may be any of a wide variety of tangible storage devices which are constructed to retrievably store data, including, for example, any of a flexible disk (floppy disk), a hard disk, an optical disk, a magneto-optical disk, a compact disc (CD), a digital versatile disc (DVD), micro-drive, a read only memory (ROM), random access memory (RAM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), dynamic random access memory (DRAM), video RAM (VRAM), a magnetic tape or card, optical card, nanosystem, molecular memory integrated circuit, redundant array of independent disks (RAID), a nonvolatile memory card, a flash memory device, a storage of distributed computing systems and the like. The storage medium may be a function expansion unit removably inserted in and/or remotely accessed by the apparatus or system for use with the computer processor(s).

This disclosure has provided a detailed description with respect to particular representative embodiments. It is understood that the scope of the appended claims is not limited to the above-described embodiments and that various changes and modifications may be made without departing from the scope of the claims.