Patent ID: 12217413

In the drawings, reference numbers may be reused to identify similar and/or identical elements.

DETAILED DESCRIPTION

With reference toFIG.1, an inspection system100is presented. The inspection system100provides a workstation including a robot104, a vision unit108, a working table116having zones120, light surface units118having light surfaces124, and a human machine interface HMI that allows an operator to interface with the inspection system100. The inspection system100works in collaboration with the operator to inspect and inventory surgical kits. The inspection system100may also be referred to as a semi-automated inspection system. The inspection system100may be employed by vendors that supply the surgical kits to users of the surgical kits (e.g., hospitals, etc.) or the inspection system100may be employed by the users themselves.

The inspection system100receives used surgical kits and inspects the used surgical kits in conjunction with the operator based on an electronic kit inspection recipe126to identify missing, damaged, worn, and/or misplaced parts. The kit inspection recipe126may be stored in a database DB, such as local databases, cloud-based databases, or other type of database. The kit inspection recipe126provides information regarding all the parts required for a complete surgical kit and information as to how the robot104, the vision unit108, and/or the light surface units118are to be positioned and/or operated when inspecting each part, and which identification methods are to be employed to inspect each of the parts. Once the inspection procedure is complete, the inspection system100generates and stores output (e.g., an electronic inspection report) indicating which parts are missing, damaged, and/or worn. In some cases, the inspection reports may be electronically transmitted to a surgical kit management system, an enterprise resource planning (ERP) system, and/or a warehouse management system (WMS), to update a bill of materials for each of the surgical kits. This may be done automatically or in response to user input via the human machine interface HMI. The human machine interface HMI and one or more of these systems may be integrated to communicate with each other. Missing, damaged, and/or worn parts can then be replaced, and the surgical kit processed (e.g., sterilized) to be ready for the next surgical procedure. The inspection system100reduces the possibility for operator error and thus provides an improved way of inspecting surgical kits. It should be appreciated that although the inspection system100is described throughout for inspecting surgical kits, it may also be used for inspecting kits other than surgical kits.

The robot104includes a robotic manipulator128having a base130and a robotic arm132extending from the base130. The robotic manipulator128supports and carries the vision unit108to move the vision unit108to a plurality of different poses in view of the each of the parts being inspected for each of the surgical kits. The base130of the robotic manipulator128may be mounted to the working table116at the center of the working table116or at any other suitable position where the robotic manipulator128is able to position the vision unit108in view of each of the zones120. In some versions, the robotic manipulator128may include additional robotic arms132, or any other suitable robotic structure to move the vision unit108. In the version shown, the robotic arm132is a serial robotic arm. Parallel robotic arms, or any other suitable mechatronic structure for moving the vision unit108may be employed.

As shown inFIG.2, a robot controller134controls movement of the robotic manipulator128based on robot positioning instructions embodied in the kit inspection recipe126and based on operator input. The robot controller134may control positioning of the robotic manipulator128, and by extension, the vision unit108, by sensing a current pose (position and orientation) of the robotic manipulator128and/or vision unit108at each processing time step (e.g., frame) and determining a new commanded pose to which to move the robotic manipulator128and/or vision unit108based on the kit inspection recipe126. The current pose and the new commanded pose may be defined with respect to a coordinate reference frame, such as a coordinate reference frame of the robotic manipulator128(e.g., fixed to its base130), a coordinate reference frame of the vision unit108(e.g., associated with a moving tool center point (TCP) of the vision unit108), or a coordinate reference frame of a tray of the surgical kit. Registration and calibration processes can be used to transform coordinates in one coordinate reference frame to another. As described further below, in some versions, the coordinate reference frame of the robotic manipulator128and/or the vision unit108are transformed to the coordinate reference frame of the tray being inspected (or vice versa) so that the robot controller134is able to accurately position the vision unit108relative to the parts in the tray.

The robot controller134determines how to move one or more joints of the robotic manipulator128(via joint motors M) to achieve movement to the new commanded pose. The robotic manipulator128may have position sensors S (e.g., joint and/or motor encoders) at each of the joints to determine the current pose of the robotic manipulator128in the coordinate reference frame of the robotic manipulator128via kinematic data associated with the robotic manipulator128. The robot controller134can thereby also determine the current pose of the vision unit108(e.g., the TCP thereof) attached to the robotic manipulator128in the coordinate reference frame of the robotic manipulator128via a known and stored geometric relationship between the vision unit108(e.g., its TCP) and the robotic manipulator128, which can be determined during manufacturing or via a calibration procedure. The robot controller134may instruct the robotic manipulator128to perform a sequence of movements to position the vision unit108in one or more poses as dictated by the kit inspection recipe126for each surgical kit. The kit inspection recipe126may provide a predefined set (and sequence) of poses to which to move the robotic manipulator128in one of the coordinate reference frames previously mentioned. The predefined set (and sequence) of poses can be determined by teaching the robotic manipulator128the poses (and sequence) via a teach pendant, and/or by programming the poses via other methods. In some versions, the robot104may be a model UR5e robot manufactured by Universal Robots of Denmark.

The robot104may also include one or more sensors to detect operator applied forces and torques. In the version shown, a force-torque sensor F-T is attached to a coupling CP (seeFIG.1) of the robotic arm132. The vision unit108is attached to the coupling CP. The force-torque sensor F-T interconnects the coupling CP and the remaining portions of the robotic arm132so that the force-torque sensor F-T can detect forces and torques applied to the vision unit108by the operator. The force-torque sensor F-T may be a six degree-of-freedom type force-torque sensor to measure forces and torques in six degrees of freedom. Additionally, or alternatively, one or more torque sensors may be located at each joint of the robotic manipulator128to measure external forces and torques. The robotic manipulator128may have six joints to move the vision unit108in six degrees of freedom. When the operator wishes to manually cause movement of the robotic manipulator128, the operator may apply forces and torques on the vision unit108. As a result, the force-torque sensor F-T measures the forces and torques and sends corresponding output to the robot controller134. The robot controller134is then able to evaluate the force and torques to determine a new commanded pose for the vision unit108based on the measurements, thereby effectively causing movement of the robotic manipulator128in a manner expected by the operator. Such collaborative robotic arms are well-known for responding to user-applied forces and torques.

With reference toFIGS.3A through3C, the vision unit108includes one or more camera units to capture images of the parts of the surgical kits. In the version shown, the vision unit108includes a first camera unit136and a second camera unit138. The first camera unit136may be configured to capture images of parts of a first type under dynamic conditions such as when lighting conditions are changing or working distances are changing. The first camera unit136may include one or more cameras (e.g., machine vision cameras) and one or more lenses, including fixed focal length lenses, and lenses for automated adjustment of focal length and aperture. In some versions, referring toFIG.3B, the first camera unit136includes a first camera140, a first imaging lens142having a fixed focal length (e.g., 16 mm), and a first liquid lens144having a variable focus. In some examples, the first camera unit136may include camera model: CAM-CIC-5000-17-GC, manufactured by Cognex Corporation of Natick, MA with an ENMT lens manufactured by Opto-Engineering of Mantova, Italy. In some versions, the first camera unit136may include camera model: Alvium 1800 U-129C, manufactured by Allied Vision Technologies GmbH of Stadtroda, Germany with a 16 mm imaging lens, model: C-series #59-870 and a liquid lens, model: EL-10-30-Ci-VIS-LD-MV from Optotune Switzerland AG of Dietikon, Switzerland.

The second camera unit138may be configured to capture or read text, symbols, or other characters associated with parts of a second type. For example, the second camera unit138may be configured to read etched or laser marked part numbers on implants, such as screws. The second camera unit138may include one or more cameras (e.g., machine vision cameras) and one or more lenses, including fixed focal length lenses, and lenses for automated adjustment of focal length and aperture. In some versions, the second camera unit138includes a second camera146, a second imaging lens148having a fixed focal length (e.g., 35 mm), different than the first imaging lens142, and a second liquid lens150having a variable focus. The second camera unit138may also include a specialty lens configured to allow for magnification up to a predetermined factor. The predetermined factor may be equal to 1.5 times or any other suitable factor. In some examples, the second camera unit138may include camera model: CAM-CIC-5000-17-GC manufactured by Cognex Corporation of Natick, MA with a MC150X lens manufactured by Opto-Engineering of Mantova, Italy. In some versions, the second camera unit138may include camera model: Alvium 1800 U-129, manufactured by Allied Vision Technologies GmbH of Stadtroda, Germany with a 35 mm imaging lens, model: C-series #59-872 and a liquid lens, model: EL-10-30-Ci-VIS-LD-MV from Optotune Switzerland AG of Dietikon, Switzerland.

In some versions, the vision unit108may include one or more light sources to illuminate the parts of the surgical kits to improve imaging of the parts. For example, a first light152may be mounted to the robotic manipulator128as part of the first camera unit136. The first light152may be model: DL194-WHI-I3S, manufactured by Advanced Illumination of Rochester, VT to generate diffuse, white, strobed light when capturing images with the first camera140. A second light154may be mounted to the robotic manipulator128as part of the second camera unit138. The second light154may be model: DL2230-WHI-I3S, manufactured by Advanced Illumination of Rochester, VT to generate diffuse, white, strobed light when capturing images with the second camera146.

The vision unit108includes one or more camera controllers to control operation of the vision unit108. In the version shown, the vision unit108includes a first camera controller156to control the first camera unit136and a second camera controller158to control the second camera unit138. The camera controllers156,158are coupled to their respective cameras140,146, liquid lenses144,150, and lights152,154to control operation thereof in accordance with the kit inspection recipe126.

As shown inFIG.3C, the vision unit108has a support structure155with a mounting plate157that is mounted to the distal end of the robotic arm132of the manipulator128. The support structure155, in the embodiment shown, has a plurality of mounting plates and brackets to which the components of the camera units136,138are mounted. The camera units135,138are thus supported by the support structure to move with the manipulator128.

The working table116has one or more working surfaces to support the surgical kits during inspection. The working table may include lockable casters, storage, and guides to help orient trays on the working surface. In some versions, the working table116and/or other portions of the workstation may be formed at least partially of Formica. This may be helpful, for example, to avoid interference with RFID readers that could be placed all around the workstation (at the sides/above/below) to automatically identify tagged parts (e.g., with RFID tags) placed on the working table116at the workstation. The tagged parts could be set on the working surface of the working table116(either directly or in a container) and then automatically counted. In this case, the inspection system100would be able to identify the tagged parts that are present and compare to a list of tagged parts that should be present in the surgical kit. In some versions, the working table116is formed primarily of stainless steel and the inspection system100operates without reading any tagged parts.

The light surface units118with light surfaces124are provided to assist in inspecting loose parts PRTS from each of the surgical kits. As shown inFIG.1, the light surface units118may include a first light surface unit118-1having a first light surface124-1and a second light surface unit118-2having a second light surface124-2. The first light surface124-1and the second light surface124-2may be arranged at opposing sides of the workstation. The first light surface124-1may be arranged between a first zone120-1and a third zone120-3while the second light surface124-2may be arranged between a second zone120-2and a fourth zone120-4; however, other configurations are contemplated. The light surfaces124may form an upper surface of the light surface units118shaped to receive the loose parts PRTS and contain the loose parts PRTS to keep them on the light surface124. The light surface units118may also be referred to as light tables, light containers, or light boxes.

Referring toFIGS.4A and4B, each light surface unit118may include a light source160, such as one or more white LED backlights and/or any other suitable light(s), configured to illuminate the light surfaces124and the loose parts PRTS from beneath the light surfaces124for enhanced imaging of the loose parts PRTS placed on the light surfaces124.FIGS.4A and4Bshow a light surface124in a first state S1(FIG.4A) in which the light surface124is not illuminated by the light source160and in a second state (FIG.4B) in which the light surface124is illuminated by the light source160to allow for better imaging (e.g., better contrast) to make identification via the inspection system100easier owing to the improved images captured by the vision unit108due to the light source160.

The light surface units118include one or more light surface controllers to control operation of the light surface units118. In the version shown, each light surface unit118has a separate light surface controller162, but a single light surface controller could control all the light surface units118. The light surface controllers162are coupled to their respective light sources160to control operation thereof in accordance with the kit inspection recipes126. As described in greater detail below, each kit inspection recipe126may include instructions for relevant settings/states for the light sources160of the light surface units118(e.g., active, inactive, etc.). In some cases, the operator may control the light surface units118via input received by the light surface controllers162.

Referring toFIG.2, a block diagram of a control system of the inspection system100is depicted. The human machine interface HMI may use software or firmware to implement the techniques and/or methods introduced herein. The software or firmware may be stored on a non-transitory computer readable medium or memory164. The human machine interface HMI includes a system controller166to control operation of the human machine interface HMI and the inspection system100. The system controller166may be embodied in a computer system that runs the software for the robot104, the vision unit108, and/or the light surface units118. The system controller166includes one or more processors168to execute one or more software modules/programs, such as an inspection module170. In some versions, the system controller166may employ a learning module171, which may be realized as a part of the inspection module170or may otherwise communicate with the inspection module170or other portions of the system controller166. These software modules/programs collaborate to receive and or transmit data to the robot controller134, camera controllers156,158, and/or light surface controllers162, via any suitable communication protocol. Each of the robot controller134, camera controllers156,158, and light surface controllers162may have their own memory164and one or more processors168and may coordinate with the system controller166to implement the techniques and/or methods described herein. More or fewer controllers may be used in the control system. In some versions, the system controller166controls and operates the robot104, the vision unit108, and/or the light surface units118. In some versions, collectively, the system controller166, robot controller134, camera controllers156,158, and/or light surface controllers162may be embodied in one or more computers.

The inspection module170may be configured to provide inspection instructions by transmitting associated instructions based on the kit inspection recipes126to the robot controller134, camera controllers156,158, and/or the light surface controllers162. Each of the kit inspection recipes includes a unique set of instructions for the robot controller134, camera controllers156,158, and/or the light surface controllers162to control the robotic manipulator128, camera units136,138, and light surface units118during inspection of surgical kits. For instance, each surgical kit has a unique kit inspection recipe126that includes various instructions for controlling the robotic manipulator128and the camera units136,138based on a layout of the surgical kits. Once inspection of each of the surgical kits is complete, the inspection module170may receive kit inspection results (discussed in greater detail with respect toFIG.8B) back from the robot controller134and/or camera controllers156,158and store the kit inspection results in the one or more databases DB. The inspection module170may be configured to maintain the database DB of records pertaining to surgical kits including respective kit inspection recipes126for each surgical kit and an inspection history for each surgical kit including kit inspection results. Images associated with parts that were improperly identified and therefore resulted in a need for correction by the operator may be flagged. These images may be used as training images. The inspection module170and/or the learning module171, as discussed in greater detail below, may be configured to learn from the flagged training images using one or more machine learning algorithms or models (e.g., an automated supervised learner model or a reinforcement learner model) to increase a rate at which the inspection system100is correctly identifying parts within surgical kits, as well as to otherwise optimize control of the robotic manipulator128, camera units136,138, and light surface units118during inspection of surgical kits.

The operator is configured to collaborate with the robot104, the vision unit108, and/or the light surface units118via a user interface UI coupled to the system controller166. In some versions, separate user interfaces UI may be coupled to each of the system controller166, robot controller134, camera controllers156,158, and/or light surface controllers162. The user interfaces UI may each include one or more displays172(e.g., flat panel LED display, OLED display, etc.) and one or more user input devices174(e.g., touchscreen, keyboard, computer mouse, pushbuttons, foot pedals, sensors, gesture control, voice control, etc.) to facilitate interaction with the operator. For example, the inspection module170, via a GUI on the display172, may prompt the operator for certain information, as discussed in greater detail below, prior to, during, and subsequent to inspection of the surgical kits. The operator may provide input to the inspection module170via the GUI shown on the display172.

A reader device176(e.g., barcode scanner, RFID tag reader) may be coupled to the system controller166. The reader device176may include an optical scanner, one or more radio frequency antennas, etc. that can read and decode the barcodes, RFID tags, etc. from the surgical kits and provide identification information to the inspection module170.

The user interface UI coupled to the system controller166, and the reader device176, may be slidably mounted to the working table116via a slider177. A rail179fixed to the working table116slidably supports the slider177(seeFIG.3A). The user interface UI and the reader device176are mounted to and supported by the slider177such that the operator may slide the user interface UI and the reader device176as desired. Any suitable mounting system may be used. A second, slidable user interface UI and/or reader device176may be included on an opposite side of the working table116, such as for processing multiple kits at the working table116and/or for multiple operators. In some cases, the same operator may process multiple kits at the working table116with multiple robots104and vision units108(not shown).

With reference toFIG.5, an example surgical kit178is shown. The surgical kit178includes a first tray180with surgical instruments INST, a second tray182with surgical implants IMP, and loose parts PRTS. While, in the example provided, the surgical kit178includes the two trays,180,182and loose parts PRTS, the surgical kit178may include any number of trays and any number of loose parts. In surgical kits where there is only one tray and no loose parts, a surgical tray and a surgical kit are the same for practical purposes. The first tray180may include a first unique identifier id1(e.g., embodied in a bar code, RFID tag, etc.) and the second tray182may include a second unique identifier id2(e.g., embodied in a bar code, RFID tag, etc.). The first and/or second unique identifiers id1, id2may be scanned by the reader device176and used by the inspection module170to retrieve a matching surgical kit inspection recipe126and associated instructions. The first unique identifier id1and the second unique identifier id2may be the same to identify the surgical kit, may be different to uniquely identify each tray, and/or may uniquely identify the tray and the surgical kit.

The inspection module170retrieves and provides the inspection system100with the kit inspection recipe126in response to the human machine interface HMI receiving the unique identifier id1and/or id2associated with the surgical kit. For example, the operator may scan into the human machine interface HMI, a barcode (or another suitable unique identifier) associated with the surgical kit via the reader device176. The human machine interface HMI uses the scanned barcode to determine the associated id and the inspection module170retrieves the kit inspection recipe126associated with the surgical kit. In some configurations, the surgical kit may include a barcode that is separate from the barcodes of any trays in the surgical kit. For the surgical kit178, for example, the surgical kit may have an id that is separate from the unique identifiers id1, id2of the first and second trays180,182and that is separately scanned by the reader device176and used to retrieve the kit inspection recipe126. In other configurations, each tray of the surgical kit178may have the same kit id (e.g., id1and id2are the same, or both identify the same surgical kit) and the human machine interface HMI may be configured to retrieve the kit inspection recipe126based on the first and/or second identifiers id1, id2of the first and second trays180,182.

With reference toFIG.6, the zones at which surgical trays may be inspected include the first zone120-1, the second zone120-2, the third zone120-3, and the fourth zone120-4. The first zone120-1may correspond to “Zone A”, the second zone may correspond to “Zone B”, the third zone may correspond to “Zone C”, and the fourth zone120-4may correspond to “Zone D.” Each of the zones120may include one or more bins122. For example, the first zone120-1may include a first bin122-1and a second bin122-2, the second zone120-2may include a third bin122-3and a fourth bin122-4, the third zone120-3may include a fifth bin122-5and a sixth bin122-6, and the fourth zone120-4may include a seventh bin122-7and an eighth bin122-8.

The operator may load the one or more trays of the surgical kit into any of the zones and any loose parts PRTS can be placed onto one or more of the light surfaces124-1,124-2. For example, with respect to the surgical kit178, the first tray180and the second tray182are in the seventh bin122-7and the eighth bin122-8, respectively, in the fourth zone120-4(i.e., “Zone D”) and the loose parts PRTS are set up on the second light surface124-2. While the robot104is inspecting the surgical kit178, the operator may set up additional surgical kits in the remaining zones120-1,120-2,120-3, and on the other light surface124-1.

During inspection, the inspection system100captures one or more images of the parts in the surgical kit to determine one or more of the following for each part: (i) is the part present or missing; (ii) is the part in the proper location in the surgical kit (if there is a specific location at which the part is to be located); (iii) is the part damaged; (iv) is the part worn; and (v) how many times has the part been in the surgical kit. Any other characteristics of the parts can be determined by identifying the parts in the surgical kit.

Example images taken of the parts from the surgical kit178are shown inFIGS.7A through7D. InFIG.7A, the first camera unit136was placed at a predefined pose relative to the first tray180based on the kit inspection recipe126and then employed to capture an image in which a part no. of a part (e.g.,703882) could be seen and electronically translated via optical character recognition (OCR). The inspection module170can compare the characters found in the image and translated via OCR to characters that the kit inspection recipe126indicates should be seen in the image captured at that pose to determine if there is a match. If there is a match, then the inspection system100indicates that the part is present. If there is no match, then the inspection system100indicates that the part is missing. In some cases, there may be a match, but the characters are found in a position and/or orientation not expected by the inspection module170, i.e., the part is misplaced. The inspection system100may report the misplacement to the operator and once the inspection is complete, the operator may place all misplaced parts in their proper location.

InFIG.7B, the second camera unit138was placed at a predefined pose relative to the second tray182based on the kit inspection recipe126and then employed to capture an image in which a part no. and/or lot code of a part (e.g., 657318 or v07603) could be seen and electronically translated via optical character recognition (OCR). In this example, the second camera unit138is used to provide high resolution images of small parts (e.g., a head of a 3.5 mm screw is shown). The inspection module170can compare the characters found in the image and translated via OCR to characters that the kit inspection recipe126indicates should be seen in the image captured at that pose to determine if there is a match. If there is a match, then the inspection system100indicates that the part is present. If there is no match, then the inspection system100indicates that the part is missing or misplaced.

InFIG.7C, the loose parts PRTS are shown on the light surface124being back illuminated and the first camera unit136has captured an image of all the loose parts PRTS to count/identify which parts are present/missing. In this case, pattern recognition algorithms may again be used. For example, the image shown inFIG.7Cshows two types of parts, those with enlarged portions and those without. The kit inspection recipe126may provide one or more patterns expected to be seen in the image of the loose parts PRTS, including different patterns for different parts. For example, the kit inspection recipe126for the surgical kit178may provide a geometric pattern for the enlarged portion. The inspection system100can then count the number of enlarged portions that can be seen in the image via pattern matching the geometric pattern associated with the enlarged portion to the image and then counting how many times the pattern is found—five times in the example ofFIG.7C(as indicated by the pattern matching indicator boxes).

InFIG.7D, the first camera unit136was placed at a predefined pose relative to the first tray180based on the kit inspection recipe126and then employed to capture an image of one of the instruments INST. The kit inspection recipe126provides one or more patterns (e.g., unique geometric shapes, etc.) expected to be seen in the image captured. The image is then processed via the inspection module170using pattern recognition algorithms to determine if any of the one or more patterns are found in the image. If the one or more patterns are found (matched) in the image, then the inspection system100indicates that the part is present. If no matches are found, then the inspection system100indicates that the part is missing. In some cases, there may be a match, but the pattern is found in a position and/or orientation not expected by the inspection module170, i.e., the part is misplaced. The inspection system100may report the misplacement to the operator and once the inspection is complete, the operator may place all misplaced parts in their proper location. InFIG.7D, the pattern matching indicator box indicates that the pattern was found in the image.

The kit inspection recipe126can dictate which camera unit136,138to use to inspect each part, and the type of part identification used for each part (e.g., pattern recognition, OCR, and the like). In addition to identifying whether the part is present or missing, the inspection module170may also determine if the part is damaged (e.g., has one or more defects) or is worn. This may be accomplished by comparing the captured images to images of undamaged or unworn parts to find differences. This can also be accomplished by the one or more of the controllers134,156,158,162,166employing deep learning algorithms as an identification method to further inspect the parts in the surgical kit, such as may be facilitated by the learning module171or other parts of the system controller166. In some versions, these types of deep learning algorithms may rely on training of neural networks using images of undamaged/unworn parts and/or images of damaged/worn parts, including parts that have nicks, scratches, etc. Accordingly, in some versions, one or more of the controllers134,156,158,162,166can analyze the parts to detect defects and can classify those defects (e.g., as a “scratch” or “nick”, etc.). Both the first camera unit136and the second camera unit138can capture images that allow the inspection system100to identify such defects. The inspection system100can then utilize deep learning algorithms that are sufficiently trained to determine whether each of the parts is damaged, worn, etc. Furthermore, deep learning algorithms and methods may be employed to promote improved inspection of surgical kits by, among other things, optimizing control of the robotic manipulator128, camera units136,138, and/or light surface units118during inspection of specific parts. Deep learning algorithms and methods that may be employed by the inspection system100include those found in VisionPro® ViDi™ deep learning-based image analysis software from Cognex Corporation of Natick, MA, including those utilized in VisionPro® ViDi™ v 3.1 and Vidi2. Deep learning algorithms and methods that may be employed by the inspection system100for pattern matching, string matching, detecting defects, and the like may include those described in U.S. Patent application Pub. No. 2020/0005069 to Wang et al., entitled “System And Method For Finding And Classifying Patterns In An Image With A Vision System,” filed on Jun. 6, 2019, which is hereby incorporated herein by reference.

As will be described further below, each kit inspection recipe126provides information to be transmitted to one or more of the controllers134,156,158,162,166as to how the robot104, vision unit108, and/or light surface units118are to move and/or operate to inspect each of the parts, such as each of instruments INST in tray180, each of the implants IMP in tray182, and each of the loose parts PRTS. Such information includes, for example: (i) the pose (i.e., coordinates x, y, z, u, v, w) to which the robotic manipulator128should move to capture one or more images of the part; (ii) which camera unit136,138(e.g., one or both) should be operated at each pose to capture the one or more images of the part; (iii) one or more lens settings and/or lighting settings for the camera unit136,138being operated; (iv) settings for the light surface unit118, if used; and/or (iv) the identification method used to identify the part in the one or more images.

With reference toFIG.8A, example block diagrams of various classes (i.e., program code templates) are shown for creating objects associated with the kit inspection recipe126. The kit inspection recipe126may be created in advance based on operator input and is unique to each surgical kit. For surgical kits that are different, a separate kit inspection recipe126is provided. Once created, the kit inspection recipes126may be stored in the database DB or any suitable location for later retrieval by the inspection system100. To start the inspection process, as described further below, the inspection system100retrieves the kit inspection recipe126based on the identification of the surgical kit being inspected. If there is no kit inspection recipe126for a surgical kit, then the operator must create a new kit inspection recipe126.

An example block diagram of a kitRecipe class212is shown for creating the kit inspection recipe126(i.e., the kitRecipe class212is instantiated to create the kitRecipe object). The kitRecipe class212may include one or more variables such as id, recipeVersion , recipeVersionDate, kitId, kitDisplayName, and multipleTrays. The id variable may be assigned a unique identifier for each of the surgical kits (e.g., such as the unique identifiers id1, id2for the surgical kit178). The recipeVersion variable may be assigned a version of the kit inspection recipe126, for example, “first version,” “second version,” etc. The kitId variable may be assigned a name indicative of the type of surgical kit, or may correlate to an existing part number for the kit, etc. The kitDisplayName variable may be assigned a name indicative of the type of surgical kit, which is to be displayed on the GUI of the inspection module170. The kitId and the kitDisplayName may be the same in some cases. The multipleTrays variable may be assigned a boolean value, for example, with true indicating that multiple trays are present in the surgical kit and false indicating that there is just a single tray or only loose parts PRTS for a particular surgical kit. For example, the kit inspection recipe126for the surgical kit178would have this variable being assigned a true value since there are two trays180,182and loose parts PRTS (also considered a “tray” in the kit inspection recipe126.

The kitRecipe class212may also include a trayRecipe subclass216. The inspection module170may use the trayRecipe subclass216to create a trayRecipeObject for each tray in the surgical kit and for the loose parts PRTS. For example, for the surgical kit178, the trayRecipe subclass216would be instantiated to create three trayRecipeObjects for the kitRecipe object—one for the first tray180, one for the second tray182, and one for the loose parts PRTS.

The trayRecipe subclass216may include one or more variables such as trayID, trayDisplayName, and trayLayoutImage. The trayID variable may be assigned a unique identifier for each tray of the surgical kit (e.g., such as the unique identifiers id1, id2for the surgical kit178). The trayDisplayName may be assigned a name to be displayed for each tray on the GUI of the inspection module170, for example for the first tray180the name shown on the GUI may correspond to “Instrument Tray.” The trayLayoutImage may be assigned an image of a tray, for example an image of the first tray180, second tray182, etc. The trayRecipe subclass216may also include a subclass such as a partRecipe subclass220.

The inspection module170may use the partRecipe subclass220to create a partrecipeObject for each part of a tray, for example, for each instrument INST in the first surgical tray180, for each implant IMP in the second surgical tray182, and collectively for the loose parts PRTS. Each partrecipeObject may include a unique set of instructions that the robot controller134, the camera controllers156,158, the light surface controllers162, and/or the system controller166use to control the robot104and the vision unit108to inspect each part in the surgical kit. The partRecipe subclass220may include variables such as partRecipeId, partRecipeVersion, partRecipeVersionDate, partNumber, partDisplayName, and partRecipeId. The partRecipeId may be assigned a unique identifier for each part of the surgical kit. The partRecipeVersion may be assigned a recipe version, for example, a first version, a second version, etc. The partRecipeVersionDate may be assigned a date and/or time that the corresponding version of the part recipe was created or generated. The partNumber variable may be assigned an existing part number of a part (e.g., serial number, etc.). The partDisplayName variable may be assigned a name of a part to be displayed on the GUI of the inspection module170.

The partRecipe subclass220may also include one or more subclasses such as a visionTool subclass224, an opticalSettings subclass228, and a coordinates subclass232. The coordinates subclass232may include variables x, y, z, u, v, and w that correspond to robot coordinates, including position (x, y, z) and orientation (u, v, w) for a coordinate reference frame, such as a coordinate reference frame associated with the tray in which the part resides. The inspection module170may use the coordinates subclass232to create a coordinatesObject for each part in a tray, for example, each instrument INST in the first tray180or for each implant IMP in the second tray182, or may create one coordinatesObject for the loose parts PRTS. The robot controller134may position the robot104/vision unit108at the coordinates associated with each part as determined by the coordinatesObject for each part.

The inspection module170may also be configured to explore the system state-space (i.e., the six locomotive degrees of freedom combined with the focal depth of the liquid lens and integration time of the image sensor sensor) via policy gradient algorithms optimize control policies for locating each part. In some versions, the control policy may be defined as the state vector which maximizes the confidence score returned by the inspection module170. For example, when a part is not located at the specified location in the kit inspection recipe126, the inspection module170may instruct the robot104to explore the state-space using the policy gradient algorithms to locate the part within the tray while maximizing the confidence score. The learning module171may cooperate with the inspection module170to improve optimized identification and handling of such scenarios.

The visionTool subclass224may include variables such as type, region, matchPattern, and matchString. The type variable may be assigned a value associated with either the first camera unit136, the second camera unit138, or both camera units136,138. The region variable may be assigned coordinates and/or dimensions for a rectangle that dictates how large of a region should be imaged, for example. The matchPattern may be assigned an image associated with a pattern that is to be found in the image captured by the vision unit108to verify the presence of the part. For example, the matchPattern image may be a pattern that is to be matched in the one or more images captured by the first camera unit136, the second camera unit138, or both the camera units136,138. During inspection, if there is a match in the captured image to this stored image, this indicates that the part is present. If there is no match, then the part is determined by the inspection module170to be missing. Matching of the images may be determined by known pattern recognition algorithms, including deep learning-based pattern matching algorithms such as those described in U.S. patent application Pub. No. 2020/0005069 to Wang et al., incorporated herein by reference. The matchString variable may be assigned a particular string of characters that need to be matched in one or more images captured by the first camera unit136, the second camera unit138, or both the camera units136,138to determine if the part is present. During inspection, the one or more images captured may be processed using optical character recognition (OCR), including deep learning-based OCR, to determine the characters or strings of characters present in the one or more images. These characters or strings of characters can then be compared to the matchString variable to see if there is a match. If the characters match, then the part is determined to be present. In some versions, the first camera unit136is used to capture images for parts that are identified via matching patterns and the second camera unit138is used to capture images for parts that are identified via matching strings. The inspection module170may use the visionTool subclass224to create a visionToolsObject for each part of a tray.

The opticalSettings subclass228may include one or more variables for adjusting a setting of one or both of the first camera unit136and/or the second camera unit138including a strobe/flash length, an exposure length, and a focal power adjustment. The strobe/flash variable determines the duration of the flash or strobe. The exposure length (i.e., shutter speed) determines a duration that an image sensor inside the camera unit is exposed (i.e., open) to light. The focal power adjustment variables determine the focal length (i.e., the distance between an optical center of a lens and the image sensor when the subject is in focus).

The above-mentioned classes and subclasses are shown in an example overall kit inspection recipe class schema236. These classes and subclasses are instantiated to create corresponding objects for each kit inspection recipe126. For simplicity of illustration purposes, the overall class schema236does not show multiple trays or multiple parts per tray; however, it is contemplated that the kit inspection recipe126may contain any number of trayRecipe objects and any number of partRecipe objects. Once the kit inspection recipe126is created, it may be saved to the database DB and retrieved at a later time for inspection of the associated surgical kit.

As previously discussed, images associated with parts that were improperly identified and therefore resulted in a need for correction by the operator may be flagged and used as training images. Periodically, the inspection module170and/or the learning module171may use the training images to optimize processes, such as to update the kit inspection recipes, update search policies for specific parts, and the like. The inspection module170may update any portion of the kit inspection recipes such as any value for any variable associated with the partRecipeObject, the visionToolObject, theCoordinatesObject, opticalSettingObject, the kitRecipeObject and/or the trayRecipeObject. In some versions, the inspection module170may update the opticalSettingObject to refine the focal length variable of the opticalSettingObject based on the training images. In some versions, the inspection module170may update one or more of the variables of the coordinate object to refine the position in which the robot controller134positions the robot104/vision unit108when inspecting a particular part.

Once the kit inspection recipe126has been updated, the kit inspection recipe126may be tested. When the number of tests exceeds a certain threshold, an operator can review the inspection history of the robot104and determine whether or not performance was improved when compared to the performance of a prior inspection recipe. In reviewing performance, the operator van evaluate whether the rate at which parts were correctly identified improves compared to the prior inspection recipe. When the operator deems that the kit inspection recipe126is producing satisfactory results and performance has improved when compared to the prior kit inspection recipe, the operator may confirm that the kit inspection recipe126is ready for use. In the event that an update to the kit inspection recipe126leads to a less desirable outcome, the inspection module may restore the previous kit inspection recipe or restore a default kit inspection recipe.

Referring toFIG.8B, the software operated by the inspection system100also provides inspection results for viewing and storing. The results are compiled in objects that are instances of inspection classes set forth inFIG.8B. A kitInspection class240may be used to create a kit inspection result260(i.e., kit inspection results object). The kitInspection class240may include variables such as id, systemid, inspectionStartTime, inspectionEndTime, kitInspectionComplete, and kitDisplayName. The id variable may be assigned the unique identifier associated with the surgical kit. The systemId variable may be assigned a name or a unique identifier associated with the inspection system100used to inspect the surgical kit. The inspectionStartTime variable may be assigned a time corresponding to the time that the robot104and/or the vision unit108began inspection of the first part, the time that a kitInspection object was created, or the like. The inspectionEndTime may be assigned a time corresponding to when the robot104and/or the vision unit108finished inspection of the last part, the time the robot104was set to idle after the last tray was inspected, the time the inspection results were submitted to the inspection module170, or the like. The kitInspectionComplete may be assigned a Boolean value, for example, true or false, with true indicating that the robot104and/or the vision unit108finished inspecting the surgical kit and false indicating that the robot104and/or vision unit108did not finish inspecting the surgical kit.

The kitInspection class240may also include a trayInspection subclass244. The trayInspection subclass244may be used to create a trayInspectionObject for each tray in the surgical kit, for example, a trayInspectionObject for the first tray180, a trayInspectionObject for the second tray182, and a trayInspectionObject for the loose parts PRTS. The trayInspection subclass244may include one or more variables such as trayInspectionStartTime, trayInspectionEndTime, trayInspectionComplete, and trayDisplayName.

The trayInspectionStartTime variable may be assigned a time corresponding to when the robot104and/or the vision unit108started inspection of the first part in the tray. The trayInspectionEndTime variable may be assigned a time corresponding to when the robot104and/or the vision unit108finished inspection of the last part in the tray. The trayInspectionComplete variable may be assigned a Boolean value with true indicating that the robot104and/or the vision unit108finished inspecting the tray and false indicating that the robot104and/or the vision unit108did not finish inspecting the tray.

The trayInspection subclass244may also include a partInspection subclass248. The partInspection subclass248may be used to create a partInspectionObject for each part of the tray. The partInspection subclass248may include one or more variables such as partNumber, partDisplayName, partInspectionResult, foundQuantity, algorithmId, and image. The partNumber variable, as previously discussed, may be assigned an existing part number of a part (e.g., serial number, etc.). The partDisplayName variable may be assigned a name of a part to be displayed for the part on the GUI of the inspection module170. The partInspectionResult may be assigned a partInspectionResultType as shown in252. The partInspectionResultType may include one or more predetermined values such as “Pass”, “Fail”, or “Operator Determination Required.” The foundQuantity variable may include a number of parts found that have the same partNumber. In some cases, the foundQuantity may be either “1” (part present) or “0” (part missing). The partInspectionResultType may also include a confidence score associated with “Pass” and “Fail” which represents the degree of confidence that the correct result was returned. The algorithmId variable may correspond to a type of algorithm (identification method) used to process the one or more images captured by the first camera unit136and/or the second camera unit138for that part to determine if the part is present or missing (e.g., pattern recognition, OCR, etc.). The image variable may be assigned an image captured of the part.

The above-mentioned classes are shown in an example overall kitInspection Results class schema256. These classes and subclasses are instantiated to create corresponding objects for each kit inspection result260. For illustration purposes, the overall kitInspection Results class schema256does not show classes for multiple trays or multiple parts per tray; however, it is contemplated that the kit inspection results260may contain any number of trayInspectionObjects and any number of partInspectionObjects. Once the kit inspection result260is completed, it may be saved to the database DB and retrieved by the inspection module170at a later time. The kit inspection result260may be retrieved to determine how to replenish the surgical kit, for billing purposes, etc. The kit inspection result260may also be retrieved in order to determine if any adjustments need to be made to the kit inspection recipes126to increase a rate at which the inspection system100is correctly identifying parts. The kit inspection result260may also include a confidence score for each part, representing the degree of confidence in the inspection system correctly identifying each of the surgical parts.

FIG.8Cshows a high-level block diagram of steps carried out by the inspection system100. At270, the operator initially scans in one or more trays of a surgical kit using the reader device176. The system controller166then determines the id of the surgical kit based on the scan. At272, the system controller166then determines if a kit inspection recipe126exists by scanning the database DB storing the kit inspection recipes to see if any are associated with the id of the surgical kit. If a kit inspection recipe126exists that is associated with the id, the system controller166retrieves the associated kit inspection recipe126from the database DB. If there is no associated kit inspection recipe126, the operator is prompted to find the kit inspection recipe126from a list of kit inspection recipes or to create a new kit inspection recipe using the classes/objects previously described with respect toFIG.8A. Once the kit inspection recipe126is retrieved or created, then the robot104and the vision unit108are activated to execute the kit inspection recipe126at274. The results of the kit inspection are displayed to the operator at276and then submitted to one or more locations at278, e.g., stored in the database DB, submitted to a replenishing system, billing system, etc.

With reference toFIGS.9-14, an example method300of using the inspection system100is shown. As will be appreciated from the subsequent description below, this method300merely represents an example and non-limiting sequence of blocks to describe operation of one or more of the controllers134,156,158,162,166in executing instructions from the inspection module170and is in no way intended to serve as a complete functional block diagram of the control system.

The method300may begin at302, where the system controller166receives a scanned tray barcode from the reader device176and parses a set-type from the scanned tray barcode to determine the id (also referred to as Set-ID) of the surgical kit. At304, the system controller166queries the database DB storing the kit inspection recipes126based on the id of the surgical kit. At306, the system controller166determines whether a kit inspection recipe126was found. If so, the method300continues at308; otherwise, the method300continues at A ofFIG.10.

At346ofFIG.10, the system controller166displays an error message on the display172indicating that no kit inspection recipe126was found. At348, the system controller166displays a list of known kit inspection recipes126from which the operator can manually select one of the kit inspection recipes126. At350, the system controller166determines whether the operator canceled since no kit inspection recipe126existed. If the operator canceled, the method300may end since no kit inspection recipe126was found and one will need to be created; otherwise, the method300may continue at351. At351, the system controller receives a selection of a correct kit inspection recipe126(e.g., via the user interface UI). At352, the system controller166receives the id entered by the operator via the user interface UI and the method300continues at X ofFIG.9.

At308ofFIG.9, the system controller166queries the database DB that stores local inspection results to determine if there are already inspection results associated with the surgical kit. At310, the system controller166determines whether results were found. If so, the method300continues at312; otherwise, the method300continues at319. At312the results are displayed to the operator and at314, the system controller166determines whether the results are complete. If the results are complete, the method300continues at316; otherwise, the method300continues at319. At316, the system controller166determines whether the results were submitted. If so, the method300may end; otherwise, the method300may continue at318where the system controller166displays the kit inspection results so that the operator may review the results.

At319, the system controller166determines whether the scanned kit is the active kit. If so, the method300continues at G ofFIG.11, otherwise, the method300continues at320. At320, the system controller166determines whether the active kit is complete (i.e., the recently scanned kit is not yet active). If so, the method300may continue at336; otherwise, the method300may continue at324. At324, the system controller166prompts the operator via the display172that the most recent scanned kit will become active. At328, the system controller166determines whether operator confirmation was received. If so, the method300continues at332; otherwise, the method300may end.

At332, the system controller166stores active kit results to the database DB. At336, the system controller166may load the kit inspection recipe126for the most recently scanned kit. At340, the system controller166determines whether inspection results were found for the newly loaded kit inspection recipe. If so, the method300may continue at344; otherwise, the method300may continue at B ofFIG.11. At344, the system controller166loads the kit inspection results that were found (i.e., the saved kit inspection results object).

At354ofFIG.11, the system controller166creates a kit inspection results object to be ready for inspection. At356, the system controller166prompts the operator via the display172to place a tray in one of the zones (e.g., A, B, C, or D). At358, the system controller166receives confirmation of the tray being present at one of the zones (A, B, C, or D) and receives the trayId, which can be manually entered by the operator via the user interface UI or can be determined through sensing of the tray by one or more sensors at the zones (A, B, C, D) and/or by imaging the trays and identifying unique identifiers in the images (e.g., landmarks, codes, patterns, etc.). At360, the system controller166adds the trayId and zone to a work queue for the robot104and the method300continues at C ofFIG.12.

At362ofFIG.12, the robot controller134and/or the system controller166sets the robotic manipulator128to an active state and moves the robot104and the vision unit108to a macro image position configured to capture an image of the entire tray of parts. At364, one or more of the camera controllers156,158and/or the system controller166load macro image optical settings to ready one of the camera units136,138to capture the macro image. At366, the one of the camera units136,138captures the macro image. At368, the system controller166determines whether there are multiple trays in the surgical kit. If so, the method300continues at D ofFIG.14; otherwise, the method300continues at370.

At408ofFIG.14, the system controller166determines which tray of the surgical kit the macro image is associated with. At410, the system controller166determines whether the tray was identified (e.g., manually by the operator, automatically using sensors, or automatically by imaging and identifying unique identifiers in the images, etc.). If the tray was identified, the method300continues at416; otherwise, the method300continues at412. At412, the system controller166displays the list of trays associated with the kit inspection recipe126being used to inspect the surgical kit. At414, the system controller166receives an operator selection of an active tray and control continues back atFIG.12, letter E. At416, the system controller determines whether tray inspection is complete. If so, the method300continues at418; otherwise, the method300continues back atFIG.12, letter E. At418, the system controller166prompts a message to the operator indicating that the tray inspection was previously completed.

At420, the system controller166determines whether the operator selected cancel via the user interface UI. If so, the method300continues at424; otherwise, the method300continues at422. At422, the system controller166determines whether the operator selected override. If so, the method300continues back atFIG.12, letter E; otherwise, the method300may continue to424. At424, the system controller166removes the current index from the work queue of the robot104and the method300continues back atFIG.12, letter P.

At370ofFIG.12, the robot controller134, camera controllers156,158, and/or system controller166registers a location of the tray. This may include determining where one or more reference features of the tray (e.g., points, lines, contours, etc.) are in the coordinate reference frame of the robot104and/or the coordinate reference frame of the vision unit108. Note that the reference features used for registration may be the same features as the unique identifiers used to determine the identification of the tray. Such registration can include, for example, evaluating the macro image captured at366to determine a location of the one or more reference features of the tray in the coordinate reference frame of the robot104and/or the coordinate reference frame of the vision unit108, and by extension, determining a pose of the tray reference frame in the coordinate reference frame of the robot104and/or in the coordinate reference frame of the vision unit108. The robot controller134, camera controllers156,158, and/or system controller166may be configured to identify the one or more reference features of the tray and their associated coordinates with respect to two or three axes (x, y, or x, y, z) to determine the tray's position and/or orientation.

In some versions, the macro image is compared to a stored image of the tray and the macro image transformed to align with the stored image using, for example, feature-based registration algorithms (e.g., finding correspondence between the reference features). By understanding the correspondence between the reference features in the images, a geometrical transformation can then be used to map the stored image and its associated tray reference frame to the macro image. In some versions, the vision unit108may utilize one of one or more of the following vision capabilities to determine the reference features of the tray in three dimensions: laser triangulation; stereo vision; time-of-flight; and structured light. Once registration is complete, the coordinate reference frame of the robot and/or the coordinate reference frame of the vision unit108can then be translated to the tray reference frame (or vice versa) since the coordinates stored in the kit inspection recipe126are defined with respect to the tray reference frame. By virtue of such translation, the robot controller134and/or the system controller166can now position the robotic manipulator128and the vision unit108at specific poses relative to the tray.

At372, the robot controller134and/or the system controller166positions the robotic manipulator128at a pose relative to the tray in accordance with instructions from the kit inspection recipe126(e.g., based on the coordinates given for a part in the surgical kit). At374, the system controller166determines whether the light surface unit118is being used. If so, the method300continues at376; otherwise, the method300continues at378. At378, the camera controllers156,158and/or the system controller166adjust optical/lighting settings based on the kit inspection recipe126to prepare for capturing an image. At380, one or both of the camera units136,138capture an image at a pose associated with instructions from the kit inspection recipe126(e.g., of a single part or of the light surface124if the light surface unit118is being used). At382, the camera controllers156,158and/or the system controller166execute vision tools, which include selecting the identification method being used to determine if the part (or parts) is present or missing (e.g., pattern matching, string matching (OCR), etc).

At384, the system controller166determines whether there was inspection uncertainty based on, for example, a best fit analysis used in the pattern matching or string matching. If there is inspection uncertainty, the method300continues at390; otherwise, the method300continues at386. At386, the system controller166updates the kit inspection result object with the analysis performed at382to determine if the part was present or missing. At390, the system controller166sets the partInspectionResult variable for that part (or parts) to “Operator Determination Required.” At392, the camera controllers156,158and/or the system controller166stores the image associated with the current part (or parts) and the method may continue to388. At388, the system controller166determines whether the tray inspections associated with the kit inspection recipe126are complete, i.e., has the last part of the kit inspection recipe126been inspected or not. If so, the method300may continue at letter F ofFIG.13; otherwise, the method300continues back at J of the current figure to inspect the next part in the predefined sequence established by the kit inspection recipe126.

At394, the system controller166determines whether the tray results are complete, i.e., has a determination been made for each part regarding whether the part is present or missing, or are there any parts that require operator determination (see390inFIG.12). If the tray results are complete, the method300continues at396; otherwise, the method300continues at398. At396, the robot controller134and/or the system controller166set the robot state to idle. At398, the system controller166displays one or more saved images from the incomplete tray results and prompts the operator for a determination for each of the images. At400, the system controller166receives an operator determination pertaining to the incomplete result. For instance, steps398and400iterate through all images for the parts where a determination of present/missing has not yet been made until the operator has made a determination for all the parts. The operator may also make determinations regarding whether each of the parts that are present are damaged/defective. Notably, the inspection system100can feed these additional results determined by the operator (and the associated images) into the inspection system100to enhance the deep-learning algorithms that are used to inspect the parts thereby increasing the ability of the inspection system100to accurately determine whether parts are present/absent, damaged/defective, etc. In other words, each time the operator views an image and determines that a part is present or absent, damaged/defective, etc., the inspection system100now has another data set (image) that can be added to its training to make a future determination of whether the part is present/absent, damaged, defective, etc. At402, the system controller166determines whether the kit inspection recipe126is complete (i.e., have all parts been inspected for the surgical kit). If so, the method300continues at406; otherwise, the method300continues at404. At404, the robot controller134and/or the system controller166determine whether the work queue for the robot104is empty. If so, the method300continues to letter G ofFIG.11; otherwise, the method300continues to letter C ofFIG.12. At406, the system controller166may display the kit inspection result on the display172, and the method300may end. While the example method300is shown as “starting” and “ending” inFIGS.9-14for illustrative purposes, it will be appreciated that the method300may instead return to302. Furthermore, as noted above, the method300described above and depicted inFIGS.9-14is in no way intended to serve as a complete functional block diagram of the control system, and other configurations are contemplated.

The foregoing description is merely illustrative in nature and is not intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms or ways. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order without altering the principles of the present disclosure. Further, although each of the examples is described above as having certain features, any one or more of those features described with respect to any example of the disclosure can be implemented in and/or combined with features of any of the other examples, even if that combination is not explicitly described. In other words, the described examples are not mutually exclusive, and permutations of one or more examples with one another remain within the scope of this disclosure.

Spatial and/or functional relationships between elements (for example, between controllers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “next to,” “on top of,” “above,” “below,” “adjacent,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.

As may be used herein throughout the disclosure, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.” The term subset does not necessarily require a proper subset. In other words, a first subset of a first set may be coextensive with (equal to) the first set.

In the FIGURES, the direction of an arrow, as indicated by the arrowhead, may generally demonstrate the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgements of, the information to element A.

In this application as may be used, including the definitions below, the term “controller” may be replaced with the term “circuit.” The term “controller” may refer to, be part of, or include the following: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

The controller may include one or more circuits, such as interface circuits. In some examples, the interface circuit(s) may implement wired or wireless (WIFI) interfaces that connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of a LAN are Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2016 (also known as the WIFI wireless networking standard) and IEEE Standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of a WPAN are the BLUETOOTH wireless networking standard from the Bluetooth Special Interest Group and IEEE Standard 802.15.4.

Each controller may communicate with one or more other controllers using the interface circuit(s). Although the controller may be depicted in the present disclosure as logically communicating directly with other controllers, in various configurations the controller may communicate via a communications system. The communications system includes physical and/or virtual networking equipment such as hubs, switches, routers, and gateways. In some configurations, the communications system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communications system may include multiple LANs connected to each other over the Internet or point-to-point leased lines using technologies including Multiprotocol Label Switching (MPLS) and virtual private networks (VPNs).

In various configurations, the functionality of the controller may be distributed among multiple controllers that are connected via the communications system. For example, multiple controllers may implement the same functionality in a distributed manner. In a further example, the functionality of the controller may be split between a server (also known as remote, or cloud) controller and a client (or, user) controller.

Some or all hardware features of a controller may be defined using a language for hardware description, such as IEEE Standard 1364-2005 (commonly called “Verilog”) and IEEE Standard 10182-2008 (commonly called “VHDL”). The hardware description language may be used to manufacture and/or program a hardware circuit. In some configurations, some or all features of a controller may be defined by a language, such as IEEE 1666-2005 (commonly called “SystemC”), that encompasses both code, as described below, and hardware description.

The term code, as may be used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple controllers. The term group processor circuit, as may be used above, encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more controllers. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple controllers. The term group memory circuit, as may be used above, encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more controllers.

The term memory circuit, as may be used above, is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks and flowchart elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.

The computer programs include processor-executable instructions that are stored on at least one non-transitory computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

The computer application and/or programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C #, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SSENSORLINK, and Python®.

The present disclosure also comprises the following clauses, with specific features laid out in dependent clauses, that may specifically be implemented as described in greater detail with reference to the configurations and drawings above.I. A method for inspecting a kit having parts of different types using a robot and a vision unit supported by the robot, the vision unit having a first camera unit and a second camera unit, the method comprising the steps of: scanning the kit to identify the kit and obtain unique inspection instructions for the kit to control inspection of the kit; placing the kit with parts of a first type and a second type on a working table; placing loose parts on a light surface illuminated by a light source from beneath the light surface; and activating the robot such that the robot is positioned at a plurality of poses relative to each of the kits, and in a predefined sequence, in accordance with the unique inspection instructions, wherein the first camera unit and the second camera unit move relative to each of the kits during the positioning of the robot at the plurality of poses, and wherein one or both of the first camera unit and the second camera unit operate at each of the plurality of poses to capture images at each of the plurality of poses.II. The method of clause I, comprising applying forces and torques to the vision unit to control movement of the robot.III. An inspection system for inspecting kits having parts of different types, the inspection system comprising: one or more camera units configured to obtain images of parts of a first type and parts of a second type; a light source; a light surface to receive loose parts from the kits, the one or more camera units being configured to capture one or more images of the loose parts while the light source illuminates the light surface from beneath the light surface; a robotic manipulator supporting the one or more camera units such that the one or more camera units are capable of being moved by the robotic manipulator relative to the kits; and one or more controllers being configured to: obtain unique inspection instructions for each of the kits to control inspection of each of the kits; operate the one or more camera units to capture images; detect parts in the images captured by the one or more camera units; compare the detected parts for each of the kits to a predefined list of parts for each of the kits; and generate output indicating inspection results for each of the surgical kits.IV. The inspection system of clause III, wherein: the one or more camera units includes a first camera unit and a second camera unit; the first camera unit configured to obtain images including unique geometric features of the parts of the first type; and the second camera unit configured to obtain images of characters on the parts of the second type.V. The inspection system of clause IV, wherein the one or more controllers are configured to operate the first camera unit and the second camera unit independently of each other.