Assembly line with integrated electronic visual inspection

Methods and systems are disclosed for obtaining a first image of a tray, determining a presence or absence of one or more first patterns in the first image, determining a rotation of each the one or more first patterns in the first image, and performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image.

BACKGROUND

Assembly of products that contain multiple items, such as pharmaceutical packaging, is a complex task. The assembly can proceed in one or more stages with items being placed into the product at each stage. Errors can be introduced at each stage by failing to place a correct item in the product, placing too many of the correct item in the product, and/or placing an incorrect item in the product. Products that are ultimately shipped with errors result in lost revenue, increased customer complaints, and lost time in addressing the customer complaints. In the case of a pharmaceutical product package, one unintended result of improper packaging is that clinicians or patients may be unwilling to use a pharmaceutical product contained within an improperly assembled package. This can be particularly true for pharmaceutical products that are administered parenterally, e.g., subcutaneously, intramuscularly, intravenously, intra-ocularly, or by inhalation. Even if an improperly assembled package is returned to the manufacturer by a clinician or a patient, a regulatory agency, such as the U.S. Food and Drug Administration, will not allow the pharmaceutical product to be repackaged, resulting in a Notice of Event (NOE). Such NOE's trigger investigations, added expense, and potentially result in an impaired competitive.

It would be desirable, therefore, to develop new technologies for product assembly, that overcomes these and other limitations of the prior art, and enhances it by reducing errors and increasing efficiency of package assembly.

SUMMARY

It is to be understood that both the following general description and the following detailed description are exemplary and explanatory only and are not restrictive. Methods and systems are disclosed for obtaining a first image of a tray, determining a presence or absence of one or more first patterns in the first image, determining a rotation of each the one or more first patterns in the first image, and performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image.

DETAILED DESCRIPTION

The present disclosure relates to improvements in computer functionality related to manufacturing and product assembly.

FIG. 1is a block diagram illustrating various aspects of an exemplary system100in which the present methods and systems can operate. One skilled in the art will appreciate that provided herein is a functional description and that the respective functions can be performed by software, hardware, or a combination of software and hardware.

In one aspect, the system100can comprise a conveyor belt101. The conveyor belt101can comprise one or more cleats102. The one or more cleats102can be made of rubber or similar material for attachment to the conveyor belt101. The one or more cleats102can be raised or otherwise extend above the surface of the conveyor belt101. The one or more cleats102can comprise a leading cleat and a trailing cleat based on a direction of travel103. The leading cleat and the trailing cleat can be relative to an object placed on the belt, such that the leading cleat is in front of the object relative to the direction of travel103and the trailing cleat is behind the object relative to the direction of travel103. Accordingly, a leading cleat for a first object can also be a trailing cleat for a second object that is ahead of the first object and so on. One or more objects104can be placed on the conveyor belt101. In an aspect, the one or more objects104can comprise a product in one or more states of assembly. For example, the one or more objects104can comprise a tray. The tray can be configured to hold one or more items. The one or more items can be related to a medical treatment. For example, the one or more items can comprise one or more syringes, auto injectors, one or more syringe needles, one or more containers of a medicament, one or more pamphlets or sets of written instructions, combinations thereof, and the like.

In one aspect, the set of written instructions sets forth information about how to use and administer a medicament. In another aspect, the written instructions are a medication label approved by a regulatory agency, such as the U.S. Food and Drug Administration.

In one aspect, the medicament is a solid formulation. In another aspect the medicament is a liquid formulation. In another aspect the medicament is a gel formulation.

In one aspect, the medicament is formulated for oral administration. In another aspect the medicament is formulated for parenteral administration. In another aspect the medicament is formulated for subcutaneous administration. In another aspect the medicament is formulated for intramuscular administration. In another aspect the medicament is formulated for intravenous administration. In another aspect the medicament is formulated for inhalation administration. In another aspect the medicament is formulated for intraocular administration.

In one aspect, the medicament comprises a small molecule active ingredient. In another aspect, the medicament comprises a biologic. In another aspect, the medicament comprises a peptide or polypeptide active ingredient.

In one aspect, the medicament comprises a vascular endothelial growth factor (VEGF) derivative active ingredient. In another aspect, the medicament comprises aflibercept, which is described in one or more of U.S. Pat. Nos. 7,070,959; 7,303,746; 7,303,747; 7,306,799; 7,374,757; 7,374,758; 7,531,173; 7,608,261; 7,972,598; 8,029,791; 8,092,803; 8,343,737; 8,647,842, each of which is incorporated by reference in its entirety.

The conveyor belt101can pass over a drive roll which can be driven by a stepper motor105. The use of the stepper motor105enables precise positioning of the one or more objects104relative to a camera106, a camera107, and a camera108. The length of each of the one or more objects104can be represented as a precise number of motor steps. The conveyor belt101can be precisely advanced or reversed to cause each of the one or more objects104to be moved into a field of view109, a field of view110, and a field of view111, associated with the camera106, the camera107, and the camera108, respectively. A programmable logic controller (PLC)112(the PLC112can comprise a computing device, a PLC, or other controller/processor) can be configured to cause the stepper motor105to execute any number of steps in either direction to cause the one or more objects104to be moved into the field of view109, the field of view110, and the field of view111.

In an aspect, the camera106, the camera107, and/or the camera108can be configured for scanning, decoding, reading, sensing, imaging, capturing, and/or interpreting visual codes. In some aspects, the camera106, the camera107, and/or the camera108can be configured to process laser, linear, or area imaging. For example, in one aspect, the camera106, the camera107, and/or the camera108may include an imager for scanning, reading, and decoding one-dimensional or two-dimensional barcodes. The camera106, the camera107, and/or the camera108can include any imager, barcode scanner, or visual code scanner capable of extracting information from visual codes consistent with the disclosed embodiments. In certain aspects, the camera106, the camera107, and/or the camera108can be configured to process scanned barcodes, images, and other data. The camera106, the camera107, and/or the camera108can include one or more depth cameras for capturing, processing, sensing, observing, modeling, detecting, and interacting with three-dimensional environments. In certain aspects, the camera106, the camera107, and/or the camera108can recognize and detect depths and colors of objects in the field of view109, the field of view110, and the field of view111, respectively. The camera106, the camera107, and/or the camera108can also provide other camera and video recorder functionalities, such as taking pictures, recording videos, streaming images or other data, storing data in image buffers, etc. These functionalities may or may not include depth information. In connection with hardware and/or software processes consistent with the disclosed embodiments, the camera106, the camera107, and/or the camera108can determine sizes, orientations, and visual properties of the one or more objects104. The camera106, the camera107, and/or the camera108can include or embody any camera known to one of ordinary skill in the art capable of handling the processes disclosed herein. The camera106, the camera107, and/or the camera108can include appropriate hardware and software components (e.g., circuitry, software instructions, etc.) for transmitting signals and information to and from a pass/fail controller113to conduct processes consistent with the disclosed embodiments. The pass/fail controller can113comprise a computing device, a PLC, or other controller/processor. The camera106, the camera107, and/or the camera108can output an image and/or one or more notifications to a monitor114, a monitor115, and a monitor116, respectively.

Positioning of the one or more objects104into the field of view109, the field of view110, and the field of view111can occur at a start-up of the system100and can be adjusted during use of the system100. One or more of the camera106, the camera107, and/or the camera108can be used to ensure proper positioning of the conveyor belt101. For example, the camera107can be configured to generate an image of the area within the field of view110. The camera107can determine a location of the one or more cleats102in the image. In an aspect, the camera107can determine the location of the leading cleat. The camera107can compare the determined location of the one or more cleats102in the image to a reference location. If the determined location is equal to the reference location then no adjustment is necessary to the conveyor belt101. If the determined location is not equal to the reference location, the camera107can determine an offset based on the difference between the determined location and the reference location. The offset can be determined in a measure of distance, for example, millimeters, centimeters, inches, and the like and/or the offset can be determined as a number of steps. The camera107can transmit a signal to the PLC112to advance or reverse the conveyor belt101by the offset by engaging the stepper motor105.

In operation, the system100can be configured to assess a current state of assembly of the one or more objects104and take one or more actions based on the current state of assembly. As each of the one or more objects104is advanced by the conveyor belt101, the one or more objects104will each be placed in the field of view109, the field of view110, and the field of view111of the camera106, the camera107, and/or the camera108, respectively. WhileFIG. 1illustrates only three cameras, it is specifically contemplated that less than three or more than three cameras can be used. It is further contemplated that the conveyor belt101can be configured to have more than the illustrated three objects104disposed thereon, regardless of the number of cameras. As the one or more objects104progress along the conveyor belt101, one or more items can be assembled into the one or more objects104by a human operator or a robot.

When each of the one or more objects104is within a field of view of one of the cameras, the camera can generate an image of the object104within the field of view associated with that camera. For example, the camera106can generate an image of the area within the field of view109, the camera107can generate an image of the area within the field of view110, and the camera108can generate an image of the area within the field of view111. Each of the camera106, the camera107, and/or the camera108can analyze their respective images. The analysis of an image can comprise determining a presence or absence of one or more patterns. The one or more patterns can comprise a text pattern, a numeric pattern, a symbol pattern, and combinations thereof. For example, a text pattern can comprise any sequence of characters such as, “FILTER NEEDLE”. A numeric pattern can comprise any sequence of numbers such as, “6941518”. The symbol pattern can comprise any sequence of symbols such as, “●□□♦”. In an aspect, the camera106, the camera107, and/or the camera108can utilize optical character recognition (OCR) to “read” the one or more patterns. In another aspect, the camera106, the camera107, and/or the camera108can be configured to not utilize OCR, but rather can be configured to merely recognize the one or more patterns as a specific pattern.

In an aspect, the one or more patterns can be embodied on the one or more items to be assembled into the one or more objects104. In an aspect, at least a portion of the one or more items can comprise one or more associated patterns. Thus, in the event the camera106, the camera107, and/or the camera108determines the presence of the one or more patterns, the presence of the one or more patterns indicates a presence of the item associated with a specific pattern. For example, if the camera106determines the presence of “FILTER NEEDLE” in the image taken of the area within the field of view109, then a conclusion can be drawn that an item associated with the pattern “FILTER NEEDLE” is present in the object104within the field of view109. The camera106, the camera107, and/or the camera108can be configured to determine the presence or absence of a plurality of patterns within a single image. For example, the camera106can determine the presence of “FILTER NEEDLE” and “FILTER NEEDLE” in the image taken of the area within the field of view109. The presence of both patterns can indicate that an item associated with two occurrences of the pattern “FILTER NEEDLE” is present in the object104within the field of view109.

Each of the items that can be assembled into the one or more objects104can be associated with one or more patterns that indicate a presence or absence of a specific number of the item. For example, an item may only be embodied with one occurrence of a specific pattern. If the camera106, the camera107, and/or the camera108determine that the specific pattern only occurs once then the conclusion can be drawn that only one of the item is present. However, if the camera106, the camera107, and/or the camera108determine that the specific pattern occurs two or more times then the conclusion can be drawn that more than one of the item is present. In another example, an item may be embodied with two occurrences of a specific pattern. If the camera106, the camera107, and/or the camera108determine that the specific pattern only occurs twice then the conclusion can be drawn that only one of the item is present. However, if the camera106, the camera107, and/or the camera108determine that the specific pattern occurs one or three or more times then the conclusion can be drawn that more than one of the item is present. In a further example, an item may be embodied with a range of specific patterns. For example, the item may be embodied with one to two occurrences of the specific pattern. If the camera106, the camera107, and/or the camera108determine that the specific pattern occurs once or twice then the conclusion can be drawn that only one of the item is present. However, if the camera106, the camera107, and/or the camera108determine that the specific pattern occurs three or more times then the conclusion can be drawn that more than one of the item is present.

Each of the camera106, the camera107, and/or the camera108can be configured to analyze an entire image or one or more specific regions of an image.FIG. 2illustrates an example image200of an object104. The object104can comprise a tray201configured for storing one or more items. The one or more items can be assembled into the tray201such that at least a portion of the one or more items is present in one or more specific regions. The tray201can comprise one or more regions, for example, a region202, a region203, and a region204. Each of the region202, the region203, and the region204can be associated with an area where the one or more patterns should be present if the item is present in the tray201. For example, the region202can be associated with a location of a vial cap of a vial when assembled into the tray201, the region203can be associated with a location of one or more syringes and/or one or more needles when assembled into the tray201, and the region204can be associated with a location of one or more pamphlets when assembled into the tray201. Each of the camera106, the camera107, and/or the camera108can be configured to analyze one or more assigned regions of the image200. For example, the camera106can be assigned to analyze the region202and the region203, the camera107can be assigned to analyze the region203, and the camera108can be assigned to analyze the region203and the region204. Any combination of assigned regions is contemplated. Furthermore, each of the camera106, the camera107, and/or the camera108can be configured to determine presence or absence of one or more assigned patterns in the assigned regions. For example, the camera106can be assigned to determine presence or absence of a vial cap in the region202and presence or absence of a first pattern (including a number of occurrences of the first pattern) in the region203, the camera107can be assigned to determine presence or absence of a second pattern (including a number of occurrences of the second pattern) in the region203, and the camera108can be assigned to determine presence or absence of a third pattern (including a number of occurrences of the third pattern) in the region203and presence or absence of a fourth pattern (including a number of occurrences of the fourth pattern) in the region204. Any combination of assigned patterns and assigned regions is contemplated.

Returning toFIG. 1, each of the one or more objects104can be configured to contain a specific number of each of the one or more items. The presence of the specific number of each item indicates that the one or more objects104is correctly assembled. The presence of anything other than the specific number of each item indicates that the one more objects104is incorrectly assembled. Each of the camera106, the camera107, and/or the camera108can be configured to make an independent assessment of the object104within the respective field of view. If a camera determines that the specific number of items the camera is configured to detect is present, the camera can issue a PASS signal to the pass/fail controller113. If a camera determines that the specific number of items the camera is configured to detect is not present, the camera can issue a FAIL signal to the pass/fail controller113. If each of the camera106, the camera107, and/or the camera108issues a PASS signal to the pass/fail controller113, then the pass/fail controller113can provide a signal to the PLC112to cause the stepper motor105to advance the conveyor belt101to advance the one or more objects104to be positioned under the field of view of the next camera. The pass/fail controller113can further transmit a notification to each of the monitors114-116to display a PASS notification. If one or more of the camera106, the camera107, and/or the camera108issues a FAIL signal to the pass/fail controller113, the pass/fail controller113will not provide a signal to the PLC112to cause the stepper motor105to advance. The pass/fail controller113can further transmit a notification to the monitors114-116associated with the camera(s) issuing the FAIL signal to display a FAIL notification. An operator (e.g., a human or a robot) positioned at the monitors114-116displaying the FAIL notification can take corrective action to remedy the FAIL status. For example, if the FAIL signal was issued as a result of a missing item, the operator can replace the missing item whereupon the camera that made the prior FAIL determination can re-generate and re-analyze an image to determine that the item is now present and issue a PASS signal to the pass/fail controller113. In another example, if the FAIL signal was issued as a result of one or more extra items, the operator can remove the one or more extra items whereupon the camera that made the prior FAIL determination can re-generate and re-analyze an image to determine that the required number of items is now present and issue a PASS signal to the pass/fail controller113.

In a further aspect, the analysis of an image by the camera106, the camera107, and/or the camera108can comprise not only determining the presence of absence of the one or more patterns, but also determining a rotation of two or more patterns. In an aspect, the two or more patterns can be embodied on the one or more items to be assembled into the one or more objects104along a specific axis. In an aspect, at least a portion of the one or more items can comprise two or more associated patterns along a specific axis. Thus, in the event the camera106, the camera107, and/or the camera108determines the presence of the two or more patterns along the specific axis, the presence of the two or more patterns along the specific axis indicates a presence of the item associated with a specific pattern along the specific axis. For example, if the camera106determines the presence of “FILTER NEEDLE” and “FILTER NEEDLE” along the same axis (e.g., 30°, 60°, 90°, 120°, 180°, and the like) in the image taken of the area within the field of view109, then a conclusion can be drawn that an item associated with the pattern “FILTER NEEDLE” and “FILTER NEEDLE” along the same axis is present in the object104within the field of view109. The camera106, the camera107, and/or the camera108can be configured to determine the rotation of a plurality of patterns within a single image. For example, the camera106can determine the presence of “FILTER NEEDLE” and “FILTER NEEDLE” along a first axis and the presence of “SYRINGE NEEDLE” and “SYRINGE NEEDLE” along a second axis in the image taken of the area within the field of view109. The presence of both patterns along two different axes can indicate that an item associated with two occurrences of the pattern “FILTER NEEDLE” along the first axis is present in the object104and an item associated with two occurrences of the pattern “SYRINGE NEEDLE” along the second axis is also present in the object104. By way of further example, the camera106can determine the presence of “FILTER NEEDLE” and “FILTER NEEDLE” along a first axis and the presence of “FILTER NEEDLE” along a second axis in the image taken of the area within the field of view109. The presence of both patterns along two different axes can indicate that two occurrences of an item associated with the pattern “FILTER NEEDLE” are present in the object104.

Each of the items that can be assembled into the one or more objects104can be associated with one or more patterns that are embodied along a specific axis that indicate a presence or absence of a specific number of the item. For example, an item may be embodied with two occurrences of a specific pattern along a specific axis. If the camera106, the camera107, and/or the camera108determine that the specific pattern only occurs twice along the specific axis then the conclusion can be drawn that only one of the item is present. However, if the camera106, the camera107, and/or the camera108determine that the specific pattern occurs along more than one axis then the conclusion can be drawn that more than one of the item is present.

FIG. 3AandFIG. 3Billustrate an example image300and303of a tray201that comprises an item301and an item302. The item301can be a vial and the item302can be a filter needle, for example. Whichever of the camera106, the camera107, and/or the camera108that generates the image300can determine that a vial cap is present in the region202. The presence of a single vial cap indicates that the item301is present. The camera106, the camera107, and/or the camera108that generates the image300can determine that, in the region203, two occurrences of a pattern are present, “TEXT A”. In an aspect, the two occurrences of the pattern, “TEXT A”, can indicate that a one or more than one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item302(e.g., a single instance of the item302can have either a single occurrence of “TEXT A” or a double occurrence of “TEXT A”) the camera106, the camera107, and/or the camera108can determine whether “TEXT A” and “TEXT A” appear on the same axis. If “TEXT A” and “TEXT A” appear on the same axis then the camera106, the camera107, and/or the camera108can determine that a single instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. If “TEXT A” and “TEXT A” appear on different axes then the camera106, the camera107, and/or the camera108can determine that a more than one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item302are present and generate a PASS or a FAIL signal based on the expected number of instances of the item302versus the determined number of instances of the item302.

FIG. 4Aillustrates an example image400of the tray201that comprises the item301and two instances of the item302. The camera106, the camera107, and/or the camera108that generates the image400can determine that, in the region203, three occurrences of a pattern are present, (“TEXT A”). In an aspect, the three occurrences of the pattern, “TEXT A” can indicate that one or more than one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item302(e.g., a single instance of the item302can have a single occurrence of “TEXT A”, a double occurrence of “TEXT A”, or a triple occurrence of “TEXT A”) the camera106, the camera107, and/or the camera108can determine whether the three occurrences of “TEXT A” appear on the same axis. As shown inFIG. 4A, two occurrences of “TEXT A” appear on the same axis and one occurrence of “TEXT A” appears on a different axis. Accordingly, the camera106, the camera107, and/or the camera108can determine that more than one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item302are present and generate a PASS or a FAIL signal based on the expected number of instances of the item302versus the determined number of instances of the item302.

FIG. 4Billustrates an example image401of the tray201that comprises the item301, one instance of the item302, and one instance of an item402. In one aspect, the camera106, the camera107, and/or the camera108that generates the image400can determine that, in the region203, two occurrences of a first pattern are present, (“TEXT A”) and one occurrence of a second pattern is present, (“TEXT B”). In an aspect, the two occurrences of the pattern, “TEXT A” can indicate that one or more than one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item302(e.g., a single instance of the item302can have a single occurrence of “TEXT A”, a double occurrence of “TEXT A”, or a triple occurrence of “TEXT A”) the camera106, the camera107, and/or the camera108can determine whether the two occurrences of “TEXT A” appear on the same axis. As shown inFIG. 4B, the two occurrences of “TEXT A” appear on the same axis. Accordingly, the camera106, the camera107, and/or the camera108can determine that more one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. However, the one occurrence of the pattern “TEXT B” can indicate that an item has been placed in the tray201that should not be in the tray201at this stage in the assembly process. Accordingly, the camera106, the camera107, and/or the camera108can generate a FAIL signal based on the presence of a pattern that is not intended to be present.

In another aspect, the camera106, the camera107, and/or the camera108that generates the image400can determine that the pattern “TEXT B” is present and can ignore the presence of the pattern “TEXT A” (or any other pattern as required). In an aspect, the one occurrence of the pattern, “TEXT B” can indicate that one instance of the item302is present and the camera106, the camera107, and/or the camera108can generate a PASS signal.

FIG. 5Aillustrates an example image500of the tray201that comprises the item301, the item302, and a single instance of an item501. The camera106, the camera107, and/or the camera108that generates the image500can be configured to ignore the vial cap in the region202and to ignore the presence of the pattern “TEXT A” in the region203. Instead, the camera106, the camera107, and/or the camera108that generates the image400can determine that, in the region203, two occurrences of another pattern are present, (“TEXT B”). In an aspect, the two occurrences of the pattern, “TEXT B” can indicate that either one or more than one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item501(e.g., a single instance of the item501can have a single occurrence of “TEXT B”, a double occurrence of “TEXT B”, or a triple occurrence of “TEXT B”) the camera106, the camera107, and/or the camera108can determine whether the two occurrences of “TEXT B” appear on the same axis. As shown inFIG. 5A, the two occurrences of “TEXT B” appear on the same axis. Accordingly, the camera106, the camera107, and/or the camera108can determine that one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item501are present and generate a PASS or a FAIL signal based on the expected number of instances of the item501versus the determined number of instances of the item501.FIG. 5Billustrates an example image503of the tray201that comprises the item301, the item302, and a single instance of the item501.FIG. 5Bis similar toFIG. 5Awith the exception thatFIG. 5Billustrates that the pattern “TEXT B” occurs twice along the same axis, however at a different angle than the axis inFIG. 5A.

FIG. 6Aillustrates an example image600of the tray201that comprises the item301and two instances of the item501. The camera106, the camera107, and/or the camera108that generates the image600can determine that, in the region203, four occurrences of a pattern are present, “TEXT B”. In an aspect, the four occurrences of the pattern, “TEXT B” can indicate that one or more than one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item501(e.g., a single instance of the item501can have a single occurrence of “TEXT B”, a double occurrence of “TEXT B”, a triple occurrence of “TEXT B”, or a quadruple occurrence of “TEXT B”) the camera106, the camera107, and/or the camera108can determine the axes upon which the four occurrences of “TEXT B” appear. As shown inFIG. 5A, two occurrences of “TEXT B” appear on a first axis and the other two occurrences of “TEXT B” appear on a second axis. Accordingly, as the two sets of “TEXT B” appear on different axes, the camera106, the camera107, and/or the camera108can determine that more than one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item501are present and generate a PASS or a FAIL signal based on the expected number of instances of the item501versus the determined number of instances of the item501.

FIG. 6Billustrates an example image601of the tray201that comprises the item301, the item302, and two instances of the item501. The camera106, the camera107, and/or the camera108that generates the image601can determine that, in the region203, three occurrences of a pattern are present, “TEXT B”. In an aspect, the three occurrences of the pattern, “TEXT B” can indicate that one or more than one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item501(e.g., a single instance of the item501can have a single occurrence of “TEXT B”, a double occurrence of “TEXT B”, a triple occurrence of “TEXT B”, or a quadruple occurrence of “TEXT B”) the camera106, the camera107, and/or the camera108can determine the axes upon which the three occurrences of “TEXT B” appear. As shown inFIG. 6B, two occurrences of “TEXT B” appear on a first axis and the one occurrence of “TEXT B” appears on a second axis. Accordingly, as the two sets of “TEXT B” appear on different axes, the camera106, the camera107, and/or the camera108can determine that more than one instance of the item501is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item501are present and generate a PASS or a FAIL signal based on the expected number of instances of the item501versus the determined number of instances of the item501.

FIG. 7Aillustrates an example image700of the tray201that comprises the item301, the item302, the item501, a single instance of an item701, and a single instance of an item702. The camera106, the camera107, and/or the camera108that generates the image700can be configured to ignore the vial cap in the region202and to ignore the presence of the patterns “TEXT A” and “TEXT B” in the region203. Instead, the camera106, the camera107, and/or the camera108that generates the image700can determine that, in the region203, two occurrences of another pattern are present, (“TEXT D”). In an aspect, the two occurrences of the pattern, “TEXT D” can indicate that either one or more than one instance of the item701is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item701(e.g., a single instance of the item701can have a single occurrence of “TEXT D”, a double occurrence of “TEXT D”, or a triple occurrence of “TEXT D”) the camera106, the camera107, and/or the camera108can determine whether the two occurrences of “TEXT D” appear on the same axis. As shown inFIG. 7A, the two occurrences of “TEXT D” appear on the same axis. Accordingly, the camera106, the camera107, and/or the camera108can determine that one instance of the item701is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item701are present and generate a PASS or a FAIL signal based on the expected number of instances of the item701versus the determined number of instances of the item701. In the same image700, the camera106, the camera107, and/or the camera108can determine that, in the region204, two occurrences of another pattern are present, (“TEXT C”). In an aspect, the two occurrences of the pattern, “TEXT C” can indicate that either one or more than one instance of the item702is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item702(e.g., a single instance of the item702can have a single occurrence of “TEXT C”, a double occurrence of “TEXT C”, or a triple occurrence of “TEXT C”) the camera106, the camera107, and/or the camera108can determine whether the two occurrences of “TEXT C” appear on the same axis. As shown inFIG. 7A, the two occurrences of “TEXT C” appear on the same axis. Accordingly, the camera106, the camera107, and/or the camera108can determine that one instance of the item702is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item702are present and generate a PASS or a FAIL signal based on the expected number of instances of the item702versus the determined number of instances of the item702.FIG. 7Billustrates an example image703of the tray201that comprises the item301, the item302, the item501, a single instance of the item701, and a single instance of the item702.FIG. 7Bis similar toFIG. 7Awith the exception thatFIG. 7Billustrates that the pattern “TEXT D” occurs twice along the same axis, however at a different angle than the axis inFIG. 7Aand similarly the pattern “TEXT C” occurs twice along the same axis, however at a different angle than the axis inFIG. 7A.

FIG. 8Aillustrates an example image800of the tray201that comprises the item301, the item302, the item501, two instances of the item701, and a single instance of the item702. The camera106, the camera107, and/or the camera108that generates the image800can determine that, in the region203, three occurrences of a pattern are present, “TEXT D”. In an aspect, the three occurrences of the pattern, “TEXT D” can indicate that one or more than one instance of the item701is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In another aspect, depending on pattern configuration on the item701(e.g., a single instance of the item701can have a single occurrence of “TEXT D”, a double occurrence of “TEXT D”, a triple occurrence of “TEXT D”, or a quadruple occurrence of “TEXT D”) the camera106, the camera107, and/or the camera108can determine the axes upon which the three occurrences of “TEXT D” appear. As shown inFIG. 8A, two occurrences of “TEXT D” appear on a first axis and the one occurrence of “TEXT D” appears on a second axis. Accordingly, as the two sets of “TEXT D” appear on different axes, the camera106, the camera107, and/or the camera108can determine that more than one instance of the item701is present and the camera106, the camera107, and/or the camera108can generate a PASS or a FAIL signal as appropriate. In an aspect, the determination of axes can be used to confirm that any number of the item701are present and generate a PASS or a FAIL signal based on the expected number of instances of the item701versus the determined number of instances of the item701.FIG. 8Bis similar toFIG. 8Awith the exception thatFIG. 8Billustrates that the pattern “TEXT D” occurs twice along a first axis and once along a second axis, however the first and second axes are at different angles than the axes inFIG. 8A.

Returning toFIG. 1, each of the camera106, the camera107, and the camera108can independently determine both the presence or the absence of one or more patterns in an image and determine a rotation of each the one or more patterns in the image of an object104. Each of the camera106, the camera107, and the camera108can perform an action based on the presence or absence and the rotation of the one or more patterns in the image. If a camera determines that a correct number of an item is present in the image of an object104based on presence of pattern and rotation of the pattern, the action can comprise transmitting a PASS signal to the pass/fail controller113. If the camera determines that an incorrect number of an item is present in the image of an object104based on presence of pattern and rotation of the pattern, the action can comprise transmitting a FAIL signal to the pass/fail controller113. If each of the camera106, the camera107, and/or the camera108issues a PASS signal to the pass/fail controller113, then the pass/fail controller113can provide a signal to the PLC112to cause the stepper motor105to advance the conveyor belt101to advance the one or more objects104to be positioned under the field of view of the next camera. The pass/fail controller113can further transmit a notification to each of the monitors114-116to display a PASS notification. If one or more of the camera106, the camera107, and/or the camera108issues a FAIL signal to the pass/fail controller113, the pass/fail controller113will not provide a signal to the PLC112to cause the stepper motor105to advance. The pass/fail controller113can further transmit a notification to the monitors114-116associated with the camera(s) issuing the FAIL signal to display a FAIL notification. An operator (e.g., a human or a robot) positioned at the monitors114-116displaying the FAIL notification can take corrective action to remedy the FAIL status.

In another aspect, one or more of the camera106, the camera107, and the camera108can count a number of the one or more objects104. For example, a the one or more objects104pass by one of the camera106, the camera107, and the camera108, the camera can increment a tally of the one or more objects104imaged by the camera. In a further aspect, a number of empty locations can be interspersed between the one or more objects104. For example, in certain scenarios one or more of the camera106, the camera107, and the camera108may not have an object104within a respective field of view. The conveyor belt101can have a pattern (e.g., a “no tray” pattern) embodied thereon in a position where the object104would otherwise be placed. The camera106, the camera107, and the camera108can identify the pattern and issue a PASS signal to contribute to advancement of the conveyor belt101.

FIG. 9illustrates an example embodiment of the system100illustrating positioning of camera106, the camera107, and the camera108relative to the conveyor belt101.FIG. 9further illustrates positioning of the monitors114-116. The stepper motor105is illustrated at one end of the conveyor belt101. One or more of the PLC112and/or the pass/fail controller113can be contained with a housing901. One or more dispensers902can be configured for storing one or more items to be accessed during assembly into the one or more objects104. The system100can comprise one or more emergency stop (“E-Stop”) buttons903. The E-Stop buttons903can be engaged at any point in time to temporarily cease operation of the system100, for any reason. The E-Stop buttons903can be reset, and the system100restarted (e.g., by an operator or technician that has determined that it is safe to do so). The system100can comprise one or more OptoSwitches904. The OptoSwitches904can be actuated (“tripped”) by placing a finger or thumb in the saddle-like structure of the OptoSwitch904. This action breaks an optical signal path, causing a switch condition. The OptoSwitches904can be used to accept a visual inspection during “Manual Trigger” mode, and start/restart the belt motion during “Autonomous” (or “Auto”) mode.

The system100can comprise a key switch mechanism905. The key switch mechanism905can be used to toggle between an “Autonomous” Mode and “Manual Trigger” Mode. Under normal operation, regardless of mode, a first operator station can comprise an operator loading trays onto the conveyor belt101. In an aspect, these trays can be pre-fitted with a prefilled capped vial. In manual trigger mode, at a second operator station, an operator can load a filter needle tip into the tray. After this operation, the camera106inspects the tray for the appropriate items. At a third operator station, an injection needle tip can be added to the tray. Then, the camera107inspects the tray for appropriate items. At a fourth operator station, an operator loads an empty blister-packed syringe into the tray. Afterwards, a fifth operator loads a Physician Insert (PI) into the tray. After the PI is loaded, the camera108inspects the tray for completed loading. Once the tray passes this last station, the fully populated tray exits the conveyor belt101for boxing.

In automated mode, trays are moved down the conveyor belt101automatically. The system100can maintain a dwell time (e.g., 1-5 seconds) before the conveyor belt101shifts to the next position. The shift occurs only when all three inspection cameras (e.g., the camera106, the camera107, and the camera108) clear the tray (“Pass”) that is being inspected by a respective camera. An issue at any inspection station can result in a the conveyor belt101and a “red light” condition (“Fail”), at which point an operator can correct the issue or pull the tray from the conveyor belt101(each camera can allow the conveyor belt101to advance when there is no tray in its field of view). The advancement of the conveyor belt101can be dependent on all cameras detecting a “passing” tray configuration. A display screen (e.g., the monitors114-116) at each camera station can display the associated camera's video stream, with overlaid “Pass”, “Fail”, or “No Job” statuses depending on the inspection results. Camera online status can be reset from the monitors114-116if required during operation.

In an aspect, illustrated inFIG. 10, a method1000is disclosed comprising obtaining a first image of a tray at1010. The method1000can comprise determining a presence or absence of one or more first patterns in the first image at1020. The one or more first patterns can comprise text patterns, numeric patterns, symbol patterns, and combinations thereof. The method1000can comprise determining a rotation of each the one or more first patterns in the first image at1030. The method1000can comprise performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image at1040. In an aspect, each step of the method1000can be performed by a computing device, a camera (with processing functionality), or a combination thereof. In some aspect, multiple computing devices and/or cameras can be employed to perform the method1000. For example, multiple cameras can be used wherein a first camera can perform steps1010,1020, and step1030while a second camera performs step1040. In another aspect, the method1000can be repeated at each of several cameras and/or computing devices as a tray proceeds along an assembly line. For example, steps1010,1020,1030, and1040can be performed by a first camera for a specific pattern(s), then steps1010,1020,1030, and1040can be performed again by a second camera for another specific pattern(s). Still further, one or more sub-steps described herein can be performed by a designated camera and/or computing device.

Determining a presence or absence of one or more first patterns in the first image can comprise determining presence of one or two of the one or more first patterns and wherein determining a rotation of each the one or more first patterns in the first image can comprise determining that the one or two of the one or more first patterns are on a first axis. Performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image can comprise generating a pass inspection signal and advancing a belt having the tray disposed thereon. Determining a presence or absence of one or more first patterns in the first image can comprise determining presence of three or more of the one or more first patterns. Performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image can comprise generating a fail inspection signal and notifying an operator that a first item associated with the one or more first patterns should be removed from the tray. Determining a presence or absence of one or more first patterns in the first image can comprise determining presence of two of the one or more first patterns and wherein determining a rotation of each the one or more first patterns in the first image can comprise determining that the two of the one or more first patterns are not on a same axis. Performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image can comprise generating a fail inspection signal and notifying an operator that a first item associated with the one or more first patterns should be removed from the tray.

The method1000can further comprise obtaining a second image of the tray, determining a presence or absence of one or more second patterns in the second image, determining a rotation of each the one or more second patterns in the second image, and performing an action based on the presence or absence and the rotation of the one or more second patterns in the second image. The one or more second patterns can comprise text patterns, numeric patterns, symbol patterns, and combinations thereof. Determining a presence or absence of one or more second patterns in the second image can comprise determining presence of one or two of the one or more second patterns and wherein determining a rotation of each the one or more second patterns in the second image can comprise determining that the one or two of the one or more second patterns are on a second axis. Performing an action based on the presence or absence and the rotation of the one or more second patterns in the second image can comprise generating a pass inspection signal and advancing a belt having the tray disposed thereon. Determining a presence or absence of one or more second patterns in the second image can comprise determining presence of three or more of the one or more second patterns. Performing an action based on the presence or absence and the rotation of the one or more second patterns in the second image can comprise generating a fail inspection signal and notifying an operator that a second item associated with the one or more second patterns should be removed from the tray. Determining a presence or absence of one or more second patterns in the second image can comprise determining presence of two of the one or more second patterns and wherein determining a rotation of each the one or more second patterns in the second image can comprise determining that the two of the one or more second patterns are not on a same axis. Performing an action based on the presence or absence and the rotation of the one or more second patterns in the second image can comprise generating a fail inspection signal and notifying an operator that a second item associated with the one or more second patterns should be removed from the tray.

The method1000can further comprise determining a location of a cleat in the first image, comparing the determined location of the cleat in the first image to a reference location, determining that the determined location is different from the reference location, determining an offset based on the difference between the determined location and the reference location, and transmitting a signal to a belt controller to adjust a distance to advance a belt having the tray disposed thereon by the offset. The offset can be one of a negative value, a positive value, or a zero value. In an aspect, determining the offset based on the difference between the determined location and the reference location, and transmitting the signal to the belt controller to adjust the distance to advance the belt having the tray disposed thereon by the offset can be performed by one or more cameras. For example, a single camera can be designated to determine the offset. The offset determination can be made after each movement of the belt.

The method1000can further comprise repeatedly obtaining a first image of a tray, determining a presence or absence of one or more first patterns in the first image, determining a rotation of each the one or more first patterns in the first image, and performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image for each of a plurality of trays.

The method1000can further comprise counting a number of the plurality of trays, wherein a number of empty tray locations are interspersed between the plurality of trays. The method1000can further comprise counting a number of the empty tray locations. Determining the presence or absence of one or more first patterns in the first image can comprise determining a no tray pattern. Performing the action based on the presence or absence and the rotation of the one or more second patterns in the first image can comprise advancing a belt having the no tray pattern disposed thereon.

In an exemplary aspect, the methods and systems can be implemented on a computer1101as illustrated inFIG. 11and described below. By way of example, the camera106, the camera107, the camera108, the PLC112, and/or the pass/fail controller113(or a component thereof) ofFIG. 1can be a computer1101as illustrated inFIG. 11. Similarly, the methods and systems disclosed can utilize one or more computers to perform one or more functions in one or more locations.FIG. 2is a block diagram illustrating an exemplary operating environment1100for performing the disclosed methods. This exemplary operating environment1100is only an example of an operating environment and is not intended to suggest any limitation as to the scope of use or functionality of operating environment architecture. Neither should the operating environment1100be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment1100.

Further, one skilled in the art will appreciate that the systems and methods disclosed herein can be implemented via a general-purpose computing device in the form of a computer1101. The computer1101can comprise one or more components, such as one or more processors1103, a system memory1112, and a bus1113that couples various components of the computer1101including the one or more processors1103to the system memory1112. In the case of multiple processors1103, the system can utilize parallel computing.

The bus1113can comprise one or more of several possible types of bus structures, such as a memory bus, memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. The bus1113, and all buses specified in this description can also be implemented over a wired or wireless network connection.

The computer1101typically comprises a variety of computer readable media. Exemplary readable media can be any available media that is accessible by the computer1101and comprises, for example and not meant to be limiting, both volatile and non-volatile media, removable and non-removable media. The system memory1112can comprise computer readable media in the form of volatile memory, such as random access memory (RAM), and/or non-volatile memory, such as read only memory (ROM). The system memory1112typically can comprise data such as image analysis data1107and/or program modules such as operating system1105and image analysis software1106that are accessible to and/or are operated on by the one or more processors1103.

In another aspect, the computer1101can also comprise other removable/non-removable, volatile/non-volatile computer storage media. The mass storage device1104can provide non-volatile storage of computer code, computer readable instructions, data structures, program modules, and other data for the computer1101. For example, a mass storage device1104can be a hard disk, a removable magnetic disk, a removable optical disk, magnetic cassettes or other magnetic storage devices, flash memory cards, CD-ROM, digital versatile disks (DVD) or other optical storage, random access memories (RAM), read only memories (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

Optionally, any number of program modules can be stored on the mass storage device1104, including by way of example, an operating system1105and image analysis software1106. One or more of the operating system1105and image analysis software1106(or some combination thereof) can comprise elements of the programming and the image analysis software1106. Image analysis data1107can also be stored on the mass storage device1104. Image analysis data1107can be stored in any of one or more databases known in the art. Examples of such databases comprise, DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, and the like. The databases can be centralized or distributed across multiple locations within the network1115.

In another aspect, the user can enter commands and information into the computer1101via an input device (not shown). Examples of such input devices comprise, but are not limited to, a keyboard, pointing device (e.g., a computer mouse, remote control), a microphone, a joystick, a scanner, touch-enabled devices such as a touchscreen, tactile input devices such as gloves and other body coverings, motion sensors, and the like. These and other input devices can be connected to the one or more processors1103via a human machine interface1102that is coupled to the bus1113, but can be connected by other interface and bus structures, such as, but not limited to, a parallel port, game port, an IEEE 1394 Port (also known as a Firewire port), a serial port, network adapter1108, and/or a universal serial bus (USB).

In yet another aspect, a display device1111can also be connected to the bus1113via an interface, such as a display adapter1109. It is contemplated that the computer1101can have more than one display adapter1109and the computer1101can have more than one display device1111. For example, a display device1111can be a monitor, an LCD (Liquid Crystal Display), light emitting diode (LED) display, television, smart lens, smart glass, and/or a projector. In addition to the display device1111, other output peripheral devices can comprise components such as speakers (not shown) and a printer (not shown) which can be connected to the computer1101via Input/Output Interface1110. Any step and/or result of the methods can be output in any form to an output device. Such output can be any form of visual representation, including, but not limited to, textual, graphical, animation, audio, tactile, and the like. The display1111and computer1101can be part of one device, or separate devices.

In an aspect, the computer1101can be coupled to the system100via the Input/Output Interface1110. The computer1101can be configured to monitor and store data. The computer1101can be configured to store images acquired by cameras connected to the system100, store data related to pass/fail statistics generated during system-generated inspections, etc. The computer1101can also be used as a programming interface to one or more smart devices (e.g., smart cameras) and/or embedded logic controllers that require customized firmware to operate. The computer1101can be used to generate, troubleshoot, upload, and store iterations of this software or firmware.

The computer1101can operate in a networked environment using logical connections to one or more remote computing devices1114a,b,c. By way of example, a remote computing device1114a,b,ccan be a personal computer, computing station (e.g., workstation), portable computer (e.g., laptop, mobile phone, tablet device), smart device (e.g., smartphone, smart watch, activity tracker, smart apparel, smart accessory), security and/or monitoring device, a server, a router, a network computer, a peer device, edge device or other common network node, and so on. Logical connections between the computer1101and a remote computing device1114a,b,ccan be made via a network1115, such as a local area network (LAN) and/or a general wide area network (WAN). Such network connections can be through a network adapter1108. A network adapter1108can be implemented in both wired and wireless environments. Such networking environments are conventional and commonplace in dwellings, offices, enterprise-wide computer networks, intranets, and the Internet. In an aspect, the network adapter1108can be configured to provide power to one or more connected devices (e.g., a camera). For example, the network adapter1108can adhere to the Power-over-Ethernet (PoE) standard or the like.

The disclosed methods and systems were implemented, tested, and results compared with a standard manual-only, operator-driven assembly line process. The following table indicates the disclosed methods and systems outperform the standard manual-only, operator-driven assembly line process: