Inspection device controlled processing line system

An optical inspector with feedback capability includes an optical device that captures an image when a sample is within the field of view of the optical device, a storage device that stores the captured image, a processor that determines a quality characteristic value of the sample based on the captured image, and an interface circuit that outputs inspection data or a command based on the quality characteristic value. A method of controlling a sample processing line is also disclosed, the method including capturing an image of a sample traversing the processing line, determining a quality characteristic of the sample based at least in part on the captured image, and causing the operation of a device included in the processing line to be adjusted based at least in part on the quality characteristic value. In one example, the optical inspector is an in-flight 3D inspector located in the processing line.

TECHNICAL FIELD

The described embodiments relate generally to capturing multiple images of an object at a single moment while the object is in-flight, and more particularly to use the captured images to generate a three-dimensional image of the sample to aid in the detection of sample defects.

BACKGROUND INFORMATION

Detection of defects present on various objects such as tree nuts and tablets is currently performed by human inspection. A human inspector visually scans multiple objects and looks for defects in each of the objects viewed. The human inspector then generates a report as to the quality of the objects viewed. The quality report is used to grade the quality of the objects and determine the price of the objects. Human inspection takes a great amount of time and cost. Human inspection also results in inconsistent quality reports between different human inspectors. A faster, less expensive and more repeatable inspection solution is needed.

SUMMARY

In a first novel aspect, an in-flight 3D inspector includes a first camera and a second camera, a trigger, a light source, a storage device, a sample input funnel, a sample chute, a collector bin, and a computer system. The sample chute receives a sample from the sample input funnel and directs the sample toward a focal plane. The trigger detects the presence of a sample and generates a trigger signal. The first and second cameras are each focused on the focal plane and are adapted to capture an image in response to receiving a trigger signal. The light source is adapted to illuminate the sample when it travels through the focal plane. The in-flight 3D inspector is configured such that the sample is not in contact with any surface while traveling through the focal plane. A storage device adapted to store images captured by the first and second cameras. The collector bin is adapted to receive the sample after the sample travels through the focal plane. The computer system adapted to generate a three dimensional image of the sample based on the images captured by the first and second cameras.

In a second novel aspect, in a first step a sample is propelled toward a focal plane. In a second step, it is determined when the sample will travel through the focal plane. In a third step, the sample is illuminated as it travels through the focal plane. In a fourth step, two or more images of the sample are captured while the sample is traveling through the focal plane. Each image is captured from a different angle, and the sample is not in contact with any surface as the sample travels through the focal plane. In a fifth step, the two or more images are stored in a storage device. In a sixth step, a three dimensional image of the sample based on the two or more captured images is generated. In a seventh step, one or more characteristics of the sample are determined based on the three dimensional image.

Further details and embodiments and techniques are described in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims.

DETAILED DESCRIPTION

Reference will now be made in detail to background examples and some embodiments of the invention, examples of which are illustrated in the accompanying drawings. In the description and claims below, relational terms such as “top”, “down”, “upper”, “lower”, “top”, “bottom”, “left” and “right” may be used to describe relative orientations between different parts of a structure being described, and it is to be understood that the overall structure being described can actually be oriented in any way in three-dimensional space.

Due to the drawbacks of human visual inspection, an automated inspector is needed to quickly, inexpensively and accurately detect defects present in objects such as tree nuts, tablets, screws and many other types of objects. Some of the most important features of such an automatic inspector include: cost, number of objects inspected per minute, accuracy of defect detection, reliability of defect detection and ease of use with minimal user training.

FIG. 1is a first diagram of the in-flight 3D inspector1view from a first perspective. The in-flight 3D inspector1includes a display2, a display support arm3, a sample input funnel4, a power switch5, an optical system mounting frame6, an axial fan7, a first light source9, a second light source8, an RJ-45 connector10, a collector bin11, and a computer system12. The display2outputs information from the computer system12to a human user looking at the display. The display support arm3attaches the display2to the in-flight 3D inspector1. In one example, the display support arm is adjustable with two hinges as shown inFIG. 1. In another example, the display support arm3is adjustable in additional dimensions (not shown inFIG. 1). The sample input funnel4is where samples are input to the in-flight 3D inspector. Power switch5is used by a human user to turn on (or off) the in-flight 3D inspector. The light sources are mounted to the optical system mounting frame6. The axial fan7is used to create positive pressure in a camera enclosure (not shown inFIG. 1). In one example, the axial fan7is coupled to a first hose that directs air flow to a first camera enclosure and is coupled to a second hose that directs air flow to a second camera enclosure (not shown). The hoses can be fixed or flexible hoses made of various materials including various plastics, fiberglass and metal materials. In this fashion, positive pressure in each camera enclosure is created. The positive pressure prevents debris from entering the camera enclosures and settling on any of the cameras. RJ-45 connector10is configured to receive an RJ-45 cable connected to a local network and electrically connect the RJ-45 cable to a network input port included on the computer system12. The RJ-45 cable may be an ethernet cable (not shown). Via the RJ-45 connector10and a RJ-45 ethernet cable, the computer system12can connect to a local network or the public Internet. The computer system12may also include a wireless networking card (not shown) that allows computer system12to wirelessly communicate (i.e. WiFi or cellular connection) with a network without the need for a wired connection. The collector bin11is configured to collect samples that have completed their path through the in-flight 3D inspector.

FIG. 2is a second diagram of the in-flight 3D inspector1view from a second perspective.FIG. 2illustrates a sample chute13that is configured to guide a sample from the sample input funnel4. The location of a power management module14is also shown inFIG. 14. The power management module14receives input power from the local power grid and generates power signals for the various electrical components operating within the in-flight 3D inspector1. For example, the power management module14generates a power signal that is used to power the various light sources, the various cameras (not shown), the axial fan, the display and the computer system. In one example, the power management module14includes a battery which can be used to operate the in-flight 3D inspector when power from the local power grid is lost.

FIG. 3is a third diagram of the in-flight 3D inspector1view from a right side view.FIG. 3shows a first camera pair18and a second camera pair19.FIG. 3also illustrates that sample chute13is aligned with the midpoint between the first camera pair18and the second camera pair19. The physical arrangement of the first camera pair18and the second camera pair19is illustrated inFIG. 6.FIG. 6illustrates that the first camera pair18includes a first camera21and a second camera22. The second camera pair19includes a third camera23and a fourth camera24. All four cameras are focused on the same focal plane. The focal plane is located at the midpoint between the first camera pair18and the second camera pair19. As discussed above regardingFIG. 3, the chute is also aligned with the midpoint between the first camera pair18and the second camera pair19.

Camera Positioning

The four cameras are positioned such that each camera is focused on the focal plane. Each camera utilizes a lens to focus on the focal plane. In one example, wide angle lenses are used by each camera. One example of a wide angle lens is FL-BC1618-9M Ricoh lens. This wide angle lens has a format size of 1″ format, a focal length of sixteen millimeters, a maximum aperture ratio of 1:1.8, an iris range of 1.8 to 16, and a resolution of nine mega-pixels. Other types of lenses may be used to achieve the necessary focus of each camera on the focal plane.

FIG. 4is a fourth diagram of the in-flight 3D inspector1view from a left side view.FIG. 4illustrates that a third light source15and a fourth light source16are also included in the in-flight 3D inspector1. In one example, the first, second, third and fourth light sources are mounted to the optical system mounting frame6. In another example, the light sources are mounted directly to outer frame of the in-flight 3D inspector1(not shown). After reading of the present disclosure, one skilled in the art will readily appreciate the various ways that light sources and cameras can be physically mounted within the in-flight 3D inspector1.

FIG. 5is a diagram of the in-flight 3D inspector1illustrating the path a sample travels through the in-flight 3D inspector1. First, a sample17is placed into the sample input funnel4. The sample input funnel4directs the sample17to sample chute13. In one example, the sample input funnel4is configured to vibrate such that sample17is directed toward sample chute13. Sample chute13directs the sample17to a focal plane where the first camera pair18and the second camera pair19are both focused. In-flight 3D inspector1may be used to generate images of various types of samples, such as tree nuts, a peanuts, tablets, screws and washers.

Triggering System

Before the sample17reaches the focal plane, a trigger senses the presence of the sample17near the sample chute13and generates a trigger signal. In one example, the trigger is attached to the sample chute13and includes an optical transmitter and an optical receiver. In operation, the sample17interferes with the light traveling between the optical transmitter and the optical receiver as sample17travels along sample chute13. This interference in received light is sensed by the optical receiver when the transmitted light does not irradiate the optical receiver. In response to detecting the interference in received light, the trigger generates a trigger signal. The trigger signal can be an electric signal that propagates along a conductor, or the trigger signal can be an electro-magnetic signal that propagates across free space to a receiving terminal. The duration between the time when the trigger signal is generated and the time when the sample17intersects the focal plane is based on where the trigger is located relative to the focal plane of the camera pairs. Once the trigger location is selected the duration between the time when the trigger signal is generated and the time when the sample17intersects the focal plane can be empirically measured or calculated. Once the duration between when the trigger signal is generated and the time when the sample17intersects the focal plane has been determined, the trigger signal can be used to determine the future time when the sample17will intersect the focal plane. This timing information can be used to properly control the various light sources and cameras in the in-flight 3D inspector.

The trigger is not shown inFIG. 5. However, a system diagram of the triggering system is illustrated inFIG. 6.FIG. 6is a diagram of a double stereo camera system configuration with triggering. The triggering system includes trigger30, controller31and/or computer system12, cameras21-24and light sources8-9and15-16. In one example, the trigger signal32(i) causes light sources8,9,15, and16to turn on, and (ii) causes the first camera pair18and the camera pair19to capture an image when the sample17intersects in the focal plane. In another example, light sources8,9,15and16are already on and the trigger signal32only causes the first camera pair18and the camera pair19to capture an image when the sample17intersects in the focal plane.

In a first embodiment, the trigger signal is communicated from the trigger30to a controller31that controls when the first camera pair18and the second camera pair19capture images. In a second embodiment, the trigger signal32is communicated from the trigger30directly to the first camera pair18and the second camera pair19and causes the camera pairs18and19to capture images. In a third embodiment, the trigger signal32is communicated from the trigger30to computer system12that controls when the first camera pair18and the second camera pair19capture images.

In a fourth embodiment, the trigger signal is communicated from the trigger30to a controller31that controls when the light sources8-9and15-16are turned on. The controller31acts as a switch that connects an output power terminal of a power supply included in power management module14to a power input terminal of each light source8-9and15-16. The controller switch turns ON the light sources in response to receiving the trigger signal. After the sample has passed though the focal plane, the controller turns OFF the light sources by disconnecting the output power terminal of the power supply from the power input terminal of each light source.

In a fifth embodiment, the trigger signal32is communicated from the trigger30directly to the light sources8-9and15-16and causes the light sources8-9and15-16to turn ON. In this embodiment, each light source8-9and15-16is configured to receive a power signal and an ON/OFF signal. The ON/OFF signal is controlled by the trigger signal. The light sources may include a timer circuit that is used to turn OFF the light sources after the sample has passed through the focal plane.

In a sixth embodiment, the trigger signal32is communicated from the trigger30to computer system12that controls when the light sources8-9and15-16are turn on. In this embodiment, each light source8-9and15-16is configured to receive a power signal and an ON/OFF signal. The ON/OFF signal is output by the computer system12in response to receiving the trigger signal from the trigger.

The light sources may be controlled such that the light sources turn on after the camera shutters are opened and turn off before the camera shutters are closed.

Controller31may be configured to communicate with computer system12via an RS232 communication link, an ethernet communication link, an Universal Serial Bus (USB) communication link, or any other available data communication links.

When the sample17travels through the focal plane, sample17is not contacting any surface. At this point in time, the light sources8-9and15-16are turned on and the first camera pair18and the second camera pair19capture at least one image of the sample. Each camera captures an image from a unique angle at the same moment in time as the sample travels through the focal plane.FIG. 7is an image captured by a first camera of the double stereo camera system.FIG. 8is an image captured by a second camera of the double stereo camera system.FIG. 9is an image captured by a third camera of the double stereo camera system.FIG. 10is an image captured by a fourth camera of the double stereo camera system. Each of these images is stored on a memory device located on the in-flight 3D inspector. On one example, the memory device is located within the computer system12. It is noted that the captured images may only be temporarily stored on a memory device within the in-flight 3D inspector before being communicated across a network to another storage device located outside of the in-flight 3D inspector. For example, the captured images stored on a storage device within the computer system12may be communicated across RJ-45 connector10and a local network to another storage device not included in the in-flight 3D inspector. In this fashion, multiple images of the sample17are captured from four different angles at the same moment while the sample17is traveling through the focal plane while not in contact with any surface.

Capturing of images while the sample is not contacting any surface provides a great benefit. When the sample is not contacting any surface, images of each surface of the sample can be collected at the same moment in time. This is not possible in other image capturing systems. For example, when a sample is moved along a conveyer belt image of only one side of the sample may be captured at any one moment in time. View of the other side of the sample is blocked by the conveyer belt and therefore cannot be captured at the same moment in time. Capturing images of all surfaces of the sample at the same moment in time allows for generation of high quality 3D images of the sample. When images of various surfaces of the sample are taken at different moments in time, proper alignment of images is very difficult, requires additional processing and result in 3D images with lower quality due to introduced error.

The cameras communicate the captured images to the controller31or computer system12via bus. In one example, the bus is an Universal Serial Bus (USB). In another example, the bus is an IEEE 1394 “FireWire” bus.

In one example, the cameras are Charged Coupled Device (CCD) cameras. In another example, the cameras are Complementary Metal-Oxide Semiconductor (CMOS) cameras. In yet another example, the cameras are Indium Gallium Arsenide (InGaAs) cameras that are capable of measuring Short Wave Infra Red (SWIR) light.

Either line scan cameras and area scan cameras can be used to implement an in-flight 3D inspector. A line scan cameras contain a single row of pixels used to capture data very quickly. As the object moves past the camera, a complete image can be reconstructed in software line by line. Area scan cameras contain a matrix of pixels that capture an image of a given scene. They are more general purpose than line scan cameras, and offer easier setup and alignment.

It is noted herein that the light sources may each include a separate power source that drives the light when a control signal is received. Alternatively, each light source may be configured in an always on state where the power input terminal on each light source is coupled to an output terminal of a power supply where the output of the power supply is controlled by a control signal.

It is noted that the sample chute13is only one example how the sample can be directed to the focal plane. In a first alternative embodiment, the sample can be directed to the focal plane by use of a conveyer belt. In this first alternative embodiment, the sample would be directed from the sample input funnel to the conveyer belt, which in turn would propel the sample off the edge of the conveyer belt toward the focal plane. In a second alternative embodiment, the sample can be directed to the focal plane by use of an airburst. In this second alternative embodiment, the sample would be directed proximate to an airburst source, which in turn would propel the sample toward the focal plane. One example of an airburst source is a highly pressurized air tank connected to an electronically controlled valve, which outputs an airburst momentarily while the valve is open.

Once the sample17passes the focal plane, the sample17falls into collector bin11. In one example, a collector bucket20is placed in collector bin11. In this example, the sample17falls into the collector bucket20. Additional samples placed into sample input funnel4make their way through the in-flight 3D inspector and eventually also fall into collector bucket20. Once all samples have passed through the in-flight 3D inspector, a user can remove all samples by removing the collector bucket20from the collector bin11.

In another example, a collector bucket20is not placed in collector bin11. Rather, collector bin11is coupled to a sample sorting machine (not shown). In this example, the samples that pass through the in-flight 3D inspector are routed into different bins. The bin each sample is routed into is based on the images captured of the sample. In the event that the images of the sample indicate that the sample has a first type of defect, then the sample is routed to a first bin. In the event that the images of the sample indicate that the sample has a second type of defect, then the sample is routed into a second bin. Alternatively, in the event that the images of the sample indicate that the sample does not have any defects, then the sample is routed to a third bin. The sorting machine can route the samples using various different methods. A first method of routing includes using a burst of air to redirect the trajectory of a sample as it falls into the collector bin. A second method of routing includes using a mechanically controlled flap to redirect the trajectory of a sample as it falls into the collector bin.

3D Image Generation

Once the images are captured from each of the cameras, a 3D image of the sample can be created. In one example, the 3D image is generated by the computer system12included in the in-flight 3D inspector. In another example, the 3D image is generated by another computer system not included in the in-flight 3D inspector after the images are communicated across a network from the in-flight 3D inspector to the computer system not included in the in-flight 3D inspector.

The images captured by the first camera pair18are used to create a 3D image of a first side of the sample. The images captured by the second camera pair19are used to create a 3D image of the second side of the sample. In one example, data included in the captured 2D images are combined into a new dataset and missing information is added to complete the 3D information of the object: depth (distance). By using triangulation on matching pixels of the multiple 2D images captured by the in-flight 3D inspector, the depth component is derived and added to the dataset. This new dataset describes the object in 3D. This dataset is then used by advanced mathematical algorithms to describe the characteristics of the objects. The 3D images of the first and second sides of the sample are combined to create a 3D image of the entire sample. Once the 3D image of the entire sample is constructed, the 3D image can be analyzed to determine if various types of defects are present on the sample. For example, if the 3D image does not match a predetermined shape within a specified tolerance, then the sample is determined to be defective with respect to shape. In another example, if the 3D image shows a flat surface greater than a specified area, then the sample is determined to be defective with respect to surface contour.

Once the defect information is determined based on the 3D image of the sample, the defect information is stored with the 3D image. The defect information can be displayed on display2to a user of the in-flight 3D inspector. The defect information can also be used to generate a report indicating the number of defects detected across a multiple samples that have been inspected. The defect information for each sample can be used by a sorting machine attached to the collector bin11of the in-flight 3D inspector to determine how the sample is to be routed. The defect information for multiple samples can be used to generate a quality report indicating the quality grade of the multiple samples.

Various calibrations of the cameras may be performed. An internal calibration may be performed for each camera. Internal calibration includes calibration of principle points, focal lengths, pixel size ratios, and radial parameters. A stereo calibration may be performed as well. A stereo calibration addresses the external 3D rotation and translation between individual cameras of a stereo system. An inter-stereo calibration may also be performed to address the external 3D rotation and translation between the two stereo systems. In an inter-stereo calibration, a transformation is performed that stitches two different side reconstructions into one 3D model.

Capturing Images of Multiple samples in a Single Image

The single sample chute13illustrated inFIG. 5illustrates one embodiment of the present invention. In another embodiment (not shown inFIG. 5) the sample chute may be configured to direct multiple samples through the focal plane at the same moment in time. In this embodiment, the sample chute would cause multiple samples to fall through the focal plane along a single axis at the same time. Aligning the samples along a single axis prevents one sample from blocking a camera's view of another sample. The first and second camera pairs would then capture an image including multiple samples instead of just one. Said another way, a single image would include multiple samples instead of just one. Once the images of the multiple samples are captured, the computer system12would (i) determine which portions of each image are of each sample, and (ii) only use the portions of each image that are of the same sample to generate the 3D image of the sample.

This configuration would greatly accelerate the rate at which the in-flight 3D inspector can capture images of multiple samples. For example, if the sample chute directed ten samples through the focal plane as the same time instead of only one sample, then the in-flight 3D inspector would be able to collect images of samples ten times faster. Said another way, the in-flight 3D inspector would only require one-tenth the amount of time to collect images of a set of samples.

FIG. 11is a flowchart200of an in-flight 3D inspector. In step201, a sample is propelled through a focal plane of a dual stereo camera system. In step202, a trigger signal is generated. The trigger signal indicates when the sample will travel through the focal plane of the stereo camera system. In step203, a predetermined amount of time after the trigger signal is generated, an image of the sample is captured by each camera included in the dual stereo camera system. The sample is illuminated by a light source while the image of the sample is captured. In step204, the sample is collected in a collector bin and the captured images are stored in a memory device.

FIG. 12is a flowchart300of an in-flight 3D inspector with defect processing. In step301, a sample is propelled through a focal plane of a dual stereo camera system. In step302, a trigger signal is generated. The trigger signal indicates when the sample will travel through the focal plane of the stereo camera system. In step303, a predetermined amount of time after the trigger signal is generated, an image of the sample is captured by each camera included in the dual stereo camera system. The sample is illuminated by a light source while the image of the sample is captured. In step304, the sample is collected in a collector bin and the captured images are stored in a memory device. In step305, the captured images are stitched together to generate a 3D image of the sample. In step306, the 3D image of the sample is used to determine one or more characteristics of the sample.

Various Numbers of Cameras Can Be Used

The two pairs of cameras18-19discussed above are used in a first embodiment of the present invention. In other embodiments, various other numbers of cameras may be used. For example, in another embodiment, the in-flight 3D inspector may include only one pair of stereo cameras that capture two images of the sample and the images are used to construct a 3D image of the sample from only one point of view. In another embodiment, three pairs of stereo cameras can be used to capture six images of the sample and the images are used to construct a 3D image of the sample from three points of view. After review of this disclosure, the reader will appreciate that additional cameras will provide additional accuracy of the 3D image created by the in-flight 3D inspector.

Inspection Device Controlled Processing Line System

FIG. 13is a diagram of an inspection device400. Inspection device400includes a processor401, a storage device402, an interface circuit403, an optical device404and/or other sensors405. The various parts of inspection device400communicate with each other across a bus406. On skilled in the art will note that various known bus architectures can be used to implement inspection device400. One example of a bus architecture is Peripheral Component Interconnect Express (PCIe), which provides standardized communication between various device components. However, many other possible options exist, such as: Ethernet for Control Automation Technology (EtherCAT), Ethernet Industrial Protocol (EtherNet/IP), Process Field Net (PROFINET), Ethernet Powerlink, Third Generation of the Sercos Interface (SERCOS III), Control and Communication Link (CC-Link IE), and Modbus/TCP, Modbus, Sinec H1, Process Field Bus (Profibus), Controller Area Network Protocol (CANopen), DeviceNet, and FOUNDATION Fieldbus. One example of a processor is an intel x86 processor. One example of a storage device is a NAND flash based solid state drive. One example of an interface circuit is a Network Interface Card (NIC) that communicates across a physically connected cable to a network switch or router. Another example of an interface circuit is a Wireless Network Interface Controller (WNIC) that communicates across standards such as WiFi (802.11 protocols), Bluetooth and other such protocols. Another example of an interface circuit is a cellular communication device that communicates across cellular networks that use protocols such as GSM, WCDMA, CDMA2000, LTE, etc. An example of an optical device is a high shutter speed, high resolution digital camera that is controllable by a computer across a standardized data port, such as USB. Other examples of optical devices include, but are not limited to, millimeter wave cameras, Near-Infr-Red (NIR) cameras, hyper-spectral cameras, and x-ray cameras. Other sensors405may include audio, electromagnetic, and odor sensors that are controllable by a computer across a standardized bus, such as USB. Other examples of sensors include, but are not limited to weight scale sensors, proximity sensors, temperature sensors, humidity sensors, texture sensors, and moisture sensors.

FIG. 14illustrates an inspection data communication system. The inspection data communication can be between inspection device412and upstream slave device411or between inspection device412and downstream slave device413. The terms upstream indicates that sample pass through the slave device before passing through the inspection device412. The term downstream indicates that samples pass through the inspection device412before passing through the salve device.

It is noted herein, that a slave device is any device located along the sample processing line. Examples of a slave devices includes, but is not limited to: a sorting device, a mixing device, a display device, a sizing device, a blanching device, a feeding device, a cutting, a slicing device, a baking device, a drying device, a freezing device, a coating device, a washing device.

In one scenario, a sample passes through the slave device411and then passes through the inspection device412. Within the inspection device412, the optical device404of the inspection device400is triggered by the processor401to capture an image. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image captured by the optical device404is then stored into storage device402. The processor401then processes the captured image and determines one or more quality characteristics of the sample in the captured image. Many different quality characteristics may be determined from the captured image. Some examples of possible quality characteristics includes, but are not limited to: shape quality (based on matching a predetermined shape within a specified tolerance, then the sample is determined to be defective with respect to shape), surface contour quality (when a flat surface is greater than a specified area, then the sample is determined to be defective with respect to surface contour), hole quality (presence of holes in the sample), pest quality (presence of insects in/or on the sample), color quality (irregular color of the sample), size quality (irregular size of the sample), moisture level, oil content, fat content, and mycotoxin content. In one example, a group of quality characteristics are referred to as inspection data415.FIG. 18,FIG. 19, andFIG. 20illustrate various examples of inspection data. Communication medium417can be a wired medium such as ethernet or RS-232. Alternatively, communication medium417can be wireless medium such as WiFi (802.11) or a cellular link. The inspection data415is then communicated to slave device411. In this fashion, the slave device411can then analyze the inspection data and adjust the operation of slave device411such that more desirable samples are output from slave device411. This scenario requires that slave device411include some local knowledge and processing capability to analyze the received inspection data and to adjust the operations of the slave device411based on the analysis.

It is noted herein, the inspection device400illustrated inFIG. 13is only one example of an inspection device. Another example of an inspection device is the in-flight 3D inspector1illustrated inFIGS. 1-5.

It is also noted herein, that multiple samples may be within the field of view of the optical device404when an image is captured and therefore quality characteristics of multiple samples may be determined using a single captured image.

In another scenario, a sample passes through the inspection device412and then passes through the slave device413. Within the inspection device412, the optical device404of the inspection device400is triggered by the processor401to capture an image. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image captured by the optical device404is then stored into storage device402. The processor401then processes the captured image and determines one or more quality characteristics of the sample in the captured image. Many different quality characteristics may be determined from the captured image. In one example, multiple quality characteristics are referred to as inspection data415. The inspection data415is then communicated to slave device413via communication medium417. Communication medium417can be a wired medium such as ethernet or RS-232. Alternatively, communication medium can be wireless medium such as WiFi (802.11) or cellular link In this fashion, the slave device413can then analyze the inspection data and adjust the operation of slave device413such that more desirable samples are output from slave device413. This scenario requires that slave device413include some local knowledge and processing capability to analyze the received inspection data and to adjust the operations of the slave device413based on the analysis.

While the scenario illustrated inFIG. 14provides the slave devices411and413with the most control over how they operate, in many instances slave devices411and413will not have the necessary knowledge and processing power to analyze the inspection data generated by the inspection device412. This problem is addressed by moving the processing of the inspection data to the inspection device412. This solution is illustrated inFIG. 15.

FIG. 15illustrates a command communication system. The terms upstream indicates that sample pass through the slave device before passing through the inspection device422. The term downstream indicates that samples pass through the inspection device422before passing through the slave device. In this system, a sample passes through the slave device411and then passes through the inspection device412. Within the inspection device412, the optical device404of the inspection device400is triggered by the processor401to capture an image. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image captured by the optical device404is then stored into storage device402. The processor401then processes the captured image and determines one or more quality characteristics of the sample in the captured image. In one example, multiple quality characteristics are referred to as inspection data. Instead of communicating the raw inspection data to the slave device421, the inspection device422performs the analysis of the inspection data and generates a command425to adjust the operation of slave device421.FIG. 21,FIG. 22, andFIG. 23illustrate various examples of commands that are generated based on inspection data. For example, a command may be to set a threshold value to be used by a slave device. In another example, a command may be to set a mixing ratio value in a slave device. In yet another example, the command may be to adjust a set-point value in a slave device. The command425is then communicated to slave device421via communication medium427. Slave device421then adjusts operation as commanded such that more desirable samples are output from slave device421. This scenario does not require that slave device421include some local knowledge and processing capability to analyze inspection data and to adjust the operations of the slave device421based on the analysis. Rather, this scenario does not require any local knowledge or processing capability to be present on the slave device421, because all the necessary analysis is performed by the inspection device422. Slave device421can operate as a “dumb” terminal that simply adjusts operation based on received commands from the inspection device422. This solution may be very valuable as it reduces the number of devices that are required to have local processing capability and knowledge, which in turn reduces the cost of the overall system.

In another scenario, a sample passes through the inspection device422and then passes through the slave device423. Within the inspection device412, the optical device404of the inspection device400is triggered by the processor401to capture an image. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image captured by the optical device404is then stored into storage device402. The processor401then processes the captured image and determines one or more quality characteristics of the sample in the captured image. In one example, multiple quality characteristics are referred to as inspection data. Instead of communicating the raw inspection data to the slave device423, the inspection device422performs the analysis of the inspection data and generates a command426to adjust the operation of slave device423. The command426is then communicated to slave device423via a communication medium. Slave device423then adjusts operation as commanded such that more desirable samples are output from slave device423. This scenario does not require that slave device423include some local knowledge and processing capability to analyze inspection data and to adjust the operations of the slave device423based on the analysis. Rather, this scenario does not require any local knowledge or processing capability to be present on the slave device423, because all the necessary analysis is performed by the inspection device422. Slave device423can operate as a “dumb” terminal that simply adjusts operation based on received commands from the inspection device422. This solution may be very valuable as it reduces the number of devices that are required to have local processing capability and knowledge, which in turn reduces the cost of the overall system.

While the scenario illustrated inFIG. 15provides cost saving by only requiring a single device in the system to have the necessary knowledge and processing power, it may be even more advantageous if the none of the devices in the system are required to have local processing capability and knowledge to analyze the captured images.FIG. 16illustrates an inspection data control system using a remote computing device.

FIG. 16illustrates an inspection data control system using a remote computing device. The terms upstream indicates that sample pass through the slave device before passing through the inspection device432. The term downstream indicates that samples pass through the inspection device432before passing through the salve device. In this system, a sample passes through the slave device431and then passes through the inspection device432. Within the inspection device432, the optical device404of the inspection device400is triggered by the processor401to capture an image438. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image438captured by the optical device404is then stored into storage device402. The processor401does not process the captured image438to determine one or more quality characteristics of the sample in the captured image438. Rather, the inspection device432communicates the captured image438to a remote computing device434. In one example, remote computing device434is a remote computer or server that is not part of any machine through which the sample flows. In response to receiving the captured image438, the remote computing device434performs the analysis of the captured image438and generates a command436to adjust the operation of slave device431. The command436is then communicated to slave device431via communication medium437. Slave device431then adjusts operation as commanded such that more desirable samples are output from slave device431. This scenario does not require any local knowledge or processing capability to be present on the slave device431, because all the necessary analysis is performed by the remote computing device434Likewise, this scenario does not require any local knowledge or processing capability to be present on the inspection device432, because all the necessary analysis is performed by the remote computing device434. Both slave device431and inspection device432can operate as “dumb” terminals that simply adjust operation based on received commands from the remote computing device434. This solution may be very valuable as it does not require any devices through which the sample passes to have local processing capability and knowledge, which in turn reduces the cost of the overall system.

In another scenario, a sample passes through the inspection device432and then passes through the slave device433. Within the inspection device432, the optical device404of the inspection device400is triggered by the processor401to capture an image438. The triggering by the processor401is executed when a sample is within the field of view of the optical device404. The image438captured by the optical device404is then stored into storage device402. The processor401does not process the captured image438to determine one or more quality characteristics of the sample in the captured image. Rather, the inspection device432communicates the captured image438to a remote computing device434. In one example, remote computing device434is a remote computer or server that is not part of any machine through which the sample flows. In response to receiving the captured image438, the remote computing device434performs the analysis of the captured image438and generates a command437to adjust the operation of slave device433. The command437is then communicated to slave device433via communication medium. Slave device433then adjusts operation as commanded such that more desirable samples are output from slave device433. This scenario does not require any local knowledge or processing capability to be present on the slave device433, because all the necessary analysis is performed by the remote computing device434. Likewise, this scenario does not require any local knowledge or processing capability to be present on the inspection device432, because all the necessary analysis is performed by the remote computing device434. Both slave device433and inspection device432can operate as “dumb” terminals that simply adjust operation based on received commands from the remote computing device434. This solution may be very valuable as it does not require any devices through which the sample passes to have local processing capability and knowledge, which in turn reduces the cost of the overall system.

In another example, captured image438is not communicated from the inspection device432to the remote computing device434, but rather inspection data435is communicated from the inspection device432to remote computing device434. In this scenario, the inspection device432captures an image of the sample and from the captured image determines quality characteristic(s) of the sample. The inspection data (grouping of quality characteristics) is then communicated to the remote computing device434. In response to receiving the inspection data, the remote computing device434generates one or more commands to adjust one or more slave devices. In this example, the inspection device432requires the processing capability to determine the quality characteristics, but does not require the capability to determine commands for adjusting slave devices.

While the scenario illustrated inFIG. 16a great improvement, a remote computing device can be used in an even more beneficial way. This improved use is illustrated inFIG. 17.

FIG. 17illustrates an inspection data control system of multiple processing lines using a remote computing device. Each processing line441-446includes at least one inspection device that is capable of capturing an image and sending the capture image and/or inspection data based on the captured image to a remote computing device440.

The in response to receiving only the captured image data448, the remote computing device440determines quality characteristics and then based on those quality characteristics (“inspection data”) the remote computing device440generates command(s) to adjust the operation of slave device(s) in the processing line from which the image was captured.

In response to receiving the inspection data447, the remote computing device440generates command(s) to adjust the operation of slave device(s) in the processing line from which the image was captured.

This scenario also reduces the complication of managing multiple sample processing lines. A single remote computing device440could receive inspection data from various inspection devices included in various processing lines441-446. In this fashion, the single remote computing device440could monitor and adjust all the various slave devices in processing lines441-446. This scenario can also provide for advanced learning because all inspection data from all processing lines441-446are received by the remote computing device440, which in turn allows for improved artificial intelligence learning by way of access to larger sets of relevant inspection data.

This scenario also allows for real-time monitoring and adjusting of multiple processing lines located at various locations around the world.

FIG. 24is a flowchart300of an inspection data communication system. In step301, an image of a sample is captured by an inspection device as the sample travels along a processing line. In step302, the captured image is processed with respect to quality characteristic(s) and inspection data is generated. In step303, the inspection data is communicated from the inspection device to another device located along the sample processing line. In step304, the device receives the inspection data and in response adjusts the operation of the device based at least in part on the inspection data received.

FIG. 25is a flowchart400of a command communication system. In step401, an image of a sample is captured by an inspection device as the sample travels along a processing line. In step402, the captured image is processed with respect to quality characteristic(s) and inspection data is generated. In step403, a command is generated based at least in part on the inspection data. In step404, the command is then communicated from the inspection device to another device located along the sample processing line. In step405, the device receives the command and in response adjusts the operation of the device based at least in part on the command received.

FIG. 26is a flowchart500of an inspection data control system using a remote computing device. In step501, an image of a sample is captured by the inspection device as the sample travels along a processing line. In step502, the captured image is communicated to the remote computing device. In step503, in response to receiving the captured image, the remote computing device determines quality characteristic(s) and generates inspection data. In step504, the remote computing device processes the inspection data and generates a command. In step505, the command is communicated from the remote computing device to a device located along the sample processing line. In step506, the device receives the command and in response adjusts the operation of the device based at least in part on the command received.

FIG. 27is a flowchart600of an inspection data control system using a remote computing device. In step601, an image of a sample is captured by the inspection device as the sample travels along the processing line. In step602, the captured image is processed with respect to quality characteristic(s) and inspection data is generated. In step603, the inspection data is communicated from the inspection device to a remote computing device and in response to receiving the inspection data, the remote computing device processes the inspection data and generates a command. In step604, the command is communicated from the remote computing device to another device located along the sample processing line. In step605, the device receives the command and in response adjusts the operation of the device based at least in part on the command received.