Patent Description:
This disclosure relates to methods and apparatus for characterizing a specimen container (and specimen therein) in an automated diagnostic analysis system.

Automated diagnostic analysis systems may conduct assays or clinical analyses using one or more reagents to identify an analyte or other constituent in a specimen such as urine, blood serum, blood plasma, interstitial liquid, cerebrospinal liquid, and the like. Such specimens are usually contained within specimen containers (e.g., specimen collection tubes). The testing reactions generate various changes that may be read and/or manipulated to determine a concentration of an analyte or other constituent in the specimen.

Improvements in automated testing technology have been accompanied by corresponding advances in pre-analytical specimen preparation and handling operations such as sorting, batch preparation, centrifuging of specimen containers to separate specimen components, cap removal to facilitate fluid access, pre-screening for HILN (Hemolysis, Icterus, and/or Lipemia, or Normal), and the like by automated specimen preparation systems called Laboratory Automation Systems (LASs). LASs may also automatically transport a specimen in a specimen container to a number of specimen processing stations so various operations (e.g., pre-analytical or analytical testing) can be performed thereon.

LASs may handle a number of different specimens contained in standard, barcode-labeled specimen containers, which may be of different sizes (e.g., diameters and heights). The barcode label may contain an accession number that may contain or be correlated to patient information and other information that may have been entered into a hospital's Laboratory Information System (LIS) along with test orders. An operator may place the labeled specimen containers onto the LAS system, which may automatically route the specimen containers for pre-analytical operations such as centrifugation, de-capping, and/or aliquot preparation before the specimen is subjected to clinical analysis or assaying by one or more analyzers (e.g., clinical chemistry or assaying instruments) that may also be part of the LAS.

For certain tests, a biological liquid such as a serum or plasma portion (obtained from whole blood by centrifugation) may be analyzed. Where the specimen is whole blood, a gel separator may be added to the specimen container to aid in the separation of a settled blood portion from the serum or plasma portion. After pre-processing, the specimen container may be transported to an appropriate analyzer that may extract a portion of the biological fluid (e.g., serum or plasma portion) from the specimen container and combine the fluid with one or more reagents and possibly other materials in a reaction vessel (e.g., a cuvette). Analytical measurements may then be performed via photometric or fluorometric absorption readings by using a beam of interrogating radiation or the like. The measurements allow determination of end-point rate or other values, from which an amount of an analyte or other constituent in the biological fluid is determined using well-known techniques.

However, the presence of any interferent (e.g., Hemolysis, Icterus, and/or Lipemia) in the specimen, which may result from a patient condition or sample processing, may adversely affect test results of the analyte or constituent measurement obtained from the one or more analyzers. For example, the presence of hemolysis (H) in the specimen, which may be unrelated to a patient's disease state, may cause a different interpretation of the disease condition of the patient. Moreover, the presence of icterus (I) and/or lipemia (L) in the specimen may also cause a different interpretation of the disease condition of the patient.

In some systems, a skilled laboratory technician may visually inspect and rate the integrity of the serum or plasma portion of the specimen as either normal (N) or as having a degree of H, I, and/or L (e.g., by assigning an index). This may involve a review of the color of the serum or plasma portion against known standards. However, such manual visual inspection is very subjective, labor intensive, and fraught with possible human error.

Because manual inspection may be problematic, efforts have been made to evaluate specimen integrity without the use of visual inspection by a laboratory technician, but rather by using an automated machine-vision inspection apparatus, wherein such evaluation takes place during pre-analytical testing (hereinafter "pre-screening"). The pre-screening involves automated detection of an interferent, such as H, I, and/or L, in a serum or plasma portion obtained from whole blood by fractionation (e.g., by centrifugation).

However, in some instances, one or more of the above-described barcode-labels may be affixed directly on the specimen container. Such labels may partially occlude and obscure certain lateral viewpoints of the specimen, so that there may be some orientations that do not provide a clear opportunity to visually observe the serum or plasma portion. Thus, automation of such pre-screening has included, for example, rotationally orienting the specimen in such a way that allows for automated pre-screening for H, I, and/or L or N (see e.g., <CIT>). In other systems, the specimen container and specimen are imaged from multiple viewpoints and processed with model-based systems so that rotation of the specimen container is not needed (see, e.g., <CIT>).

<CIT> discloses a method of characterizing a specimen for HILN (H, I, and/or L, or N), which method includes capturing images of the specimen at multiple different viewpoints, processing the images to provide segmentation information for each viewpoint, generating a semantic map from the segmentation information, selecting a synthetic viewpoint, identifying front view semantic data and back view semantic data for the synthetic viewpoint, and determining HILN of the serum or plasma portion based on the front view semantic data with an HILN classifier, while taking into account back view semantic data.

In some instances, only a small portion of the serum or plasma portion may be visible, so that any H, I, and/or L, or N reading taken on the serum or plasma portion may not involve a high level of confidence. Moreover, such systems may be complicated and processing of the image data may be computationally burdensome.

Accordingly, there is an unmet need for a robust and efficient method and apparatus for characterizing a serum or plasma portion of a specimen in order to determine a presence of hemolysis (H), icterus (I), and/or lipemia (L), or whether the serum or plasma portion of the specimen is normal (N). More particularly, there is an unmet need for improved methods and apparatus for determining if a specimen includes H, I, and/or L or is normal (N).

According to a first aspect, a method of characterizing a specimen container is provided. The method includes capturing multiple images of the specimen container from multiple viewpoints wherein the specimen container includes a serum or plasma portion of a specimen therein; inputting image data from the multiple images to a segmentation convolutional neural network and processing the image data with the segmentation convolutional neural network to simultaneously output multiple label maps; inputting the multiple label maps to a classification convolutional neural network and processing the multiple label maps with the classification convolutional neural network; and outputting from the classification convolutional neural network a classification of the serum or plasma portion as being one or more of hemolytic, icteric, lipemic, or normal. The method further comprises stacking together a respective captured image from each viewpoint into a single image with additional channels corresponding to the number of viewpoints, wherein the image data from the multiple images comprises consolidated pixel or patch data from multiple exposures, and wherein the number of simultaneously output multiple label maps corresponds to the number of multiple viewpoints.

According to another aspect, a quality check module is provided. The quality check module includes a plurality of image capture devices configured to capture multiple images from multiple viewpoints of a specimen container containing a serum or plasma portion of a specimen therein, and a computer coupled to the plurality of image capture devices. The computer is configured and operative to: input image data from the multiple images to a segmentation convolutional neural network and process the image data with the segmentation convolutional neural network to simultaneously output multiple label maps, input the multiple label maps to a classification convolutional neural network and process the multiple label maps with the classification convolutional neural network, and output from the classification convolutional neural network a classification of the serum or plasma portion as being one or more of hemolytic, icteric, lipemic, or normal. The computer is further configured to stack together a respective captured image from each viewpoint into a single image with additional channels corresponding to the number of viewpoints, wherein the image data from the multiple images comprises consolidated pixel or patch data from multiple exposures, and wherein the number of simultaneously output multiple label maps corresponds to the number of multiple viewpoints.

Still other aspects, features, and advantages of this disclosure may be readily apparent from the following description by illustrating a number of example embodiments and implementations, including the best mode contemplated for carrying out the invention. This disclosure may also be capable of other and different embodiments, and its several details may be modified in various respects, all without departing from the scope of the invention, as defined by the appended claims (see further below).

The drawings, described below, are for illustrative purposes and are not necessarily drawn to scale. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive. The drawings are not intended to limit the scope of the invention in any way.

During pre-screening of a specimen contained in a specimen container, such as at a quality check module described further below, a method is provided in accordance with embodiments that determines the presence of an interferent in the serum or plasma portion of the specimen. The serum or plasma portion may be the liquid component of blood and may be found above the settled blood portion after fractionation (e.g., by centrifugation). The settled blood portion may be a packed semi-solid made up of blood cells such as white blood cells (leukocytes), red blood cells (erythrocytes), and platelets (thrombocytes). Plasma and serum may differ from each other in the content of coagulating components, primarily fibrinogen. Plasma may be the un-clotted liquid, whereas serum may refer to blood plasma that has been allowed to clot either under the influence of endogenous enzymes or exogenous components.

An interferent, such as H, I, and/or L, or a determination of normal (N) (hereinafter "HILN"), as used herein refers to the presence of at least one of hemolysis (H), icterus (I), or lipemia (L) in the serum or plasma portion of the specimen. "N" refers to "normal," which may be defined as a serum or plasma portion that includes acceptably low amounts of H, I, and L. Hemolysis may be defined as a condition in the serum or plasma portion wherein red blood cells are destroyed during processing, which leads to the release of hemoglobin from the red blood cells into the serum or plasma portion such that the serum or plasma portion takes on a reddish hue. The degree of hemolysis may be quantified by assigning a Hemolytic Index. Icterus may be defined as a condition of the blood where the serum or plasma portion is discolored dark yellow, caused by an accumulation of bile pigment (bilirubin). The degree of icterus may be quantified by assigning an Icteric Index. Lipemia may be defined as a presence in the blood of an abnormally high concentration of emulsified fat, such that the serum or plasma portion has a whitish or milky appearance. The degree of lipemia may be quantified by assigning a Lipemic Index.

The method in accordance with embodiments may determine just HILN or N-Class H (e.g., H1, H2, H3, or more), N-Class I (e.g., I1, I2, I3, or more), and/or N-Class L (e.g., L1, L2, L3, or more), or N. In addition, the method may classify (or "segment") various regions of the specimen container and specimen, such as serum or plasma portion, settled blood portion, gel separator (if used), air, label, type of specimen container (indicating, e.g., height and width/diameter), and/or type and/or color of a specimen container cap. A specimen container holder or background may also be classified. Differentiation of the serum and plasma portion from the region comprising one or more labels on the specimen container is a particularly vexing problem, because the one or more labels may wrap around the specimen container to various degrees. Thus, the one or more labels may obscure one or more views, such that a clear view of the serum or plasma portion may be difficult to obtain.

Thus, classification of the serum or plasma portion may be challenging due to interference from the one or more labels, whose placement may vary substantially from one specimen container to the next. In particular, the obstruction caused by the one or more labels may heavily influence the spectral responses, such as from various viewpoints, given that the one or more labels may appear on a back side and thus may affect light transmission received at a front side.

Moreover, the quality check module of an automated diagnostic analysis system and associated characterization method performed therein should be computationally efficient. Accordingly, given the challenges described above, in a first broad aspect, embodiments of this disclosure provide methods and apparatus configured to determine the presence of HILN using a single semantic segmentation convolutional neural network (SCNN) whose output is coupled as input to a classification convolutional neural network (CCNN), which are collectively referred to herein as a single deep neural network (SDNN). The SDNN may include a large number of operational layers (e.g., <NUM> - <NUM>; other numbers of operational layers are possible), described further below.

In some embodiments, the input to the SCNN may be multi-spectral, multi-exposure image data, which may be consolidated and normalized, and obtained from a plurality of image capture devices. An image capture device may be any device capable of capturing a pixelated image (e.g., digital image) for analysis, such as a digital camera, a CCD (charge-coupled device), one or more CMOS (complementary metal-oxide semiconductor) sensors, an array of sensors, or the like. The plurality of image capture devices may be arranged and configured to capture images from multiple viewpoints (e.g., three viewpoints; other numbers of viewpoints are possible). The methods described herein may use high dynamic range (HDR) image processing of the specimen container and serum or plasma portion as an input to the SCNN. HDR imaging may involve capturing multiple exposures while using multiple spectral illuminations. In some embodiments, the SCNN is trained to recognize regions occluded by one or more labels on the specimen container so that the SCNN can better account for the presence of labels on the back side of the specimen container from any viewpoint in characterizing HILN.

As a result, more effective classification of the serum or plasma region may be available in cases where label obstruction is present, and the confidence in the intensity readings for those regions of the serum or plasma portion that are occluded by a label can be improved. Thus, an improved determination of HILN and/or the extent of HIL can be output from the SDNN.

The specimen may be collected in a specimen container, such as a blood collection tube and may include a settled blood portion and a serum and plasma portion after fractionation (e.g., separation by centrifugation). In some specimen containers, a gel separator may be used, which positions itself between the settled blood portion and the serum or plasma portion during centrifugation. The gel separator serves as a physical barrier between the two portions (liquid and semi-solid, settled blood cells), and may minimize remixing thereof. The specimen containers may be of different sizes and thus may be supplied for pre-screening and to the analyzers in a number of different configurations. For example, the specimen containers may have sizes such as <NUM> x <NUM>, <NUM> x <NUM>, <NUM> x <NUM>, and <NUM> x <NUM>. Other suitable sizes may be used.

In accordance with one aspect, the characterization method may be carried out by a quality check module, and in specimen testing systems, each including the SDNN. The SDNN may include operational layers including, e.g., BatchNorm, ReLU activation, convolution (e.g., 2D), dropout, and deconvolution (e.g., 2D) layers to extract features, such as simple edges, texture, and parts of the serum or plasma portion and label-containing regions. Top layers, such as fully convolutional layers, may be used to provide correlation between parts. The output of the layer may be fed to a SoftMax layer, which produces an output on a per pixel (or per patch - including n x n pixels) basis concerning whether each pixel or patch includes HILN. In some embodiments, only an output of HILN is provided from the CCNN. In other embodiments, the output of the CCNN may be fine-grained HILN, such as H1, H2, H3, I1, I2, I3, L1, L2, L3, or N, so that for each interferent present an estimate of the level (index) of the interferent is also obtained.

Should the specimen be found to contain one or more of H, I, and L, a suitable notice may be provided to the operator, and/or the specimen container may be taken off line (<NUM>) to perform a remediation to rectify the one or more of the H, I, or L, (<NUM>) to more accurately measure an extent of the interferent present, (<NUM>) to redraw the specimen, or (<NUM>) to perform other processing. Thus, the ability to pre-screen for HILN, such as at the first possible instance after centrifugation, and before analysis by one or more analyzers, may advantageously (a) minimize time wasted analyzing specimens that are not of the proper quality for analysis, (b) may avoid or minimize erroneous test results, (c) may minimize patient test result delay, and/or (d) may avoid wasting of patient specimen.

In some embodiments, combinations of segmentation output and HILN output may be provided. The outputs may result from multiple branches of the SDNN. The branches may include separate convolutional layers and deconvolution and SoftMax layers, wherein one branch may be dedicated to segmentation and the other to HILN detection. Multi-branch embodiments including HILN, segmentation, specimen container type detection, and/or cap type detection may also be provided.

Further details of inventive characterization methods, quality check modules configured to carry out the characterization methods, and specimen testing apparatus including one or more quality check modules will be further described with reference to <FIG> herein.

<FIG> illustrates a specimen testing apparatus <NUM> capable of automatically processing multiple specimen containers <NUM> containing specimens <NUM> (see, e.g., <FIG>). The specimen containers <NUM> may be provided in one or more racks <NUM> at a loading area <NUM> prior to transportation to, and analysis by, one or more analyzers (e.g., first, second, and third analyzer <NUM>, <NUM>, and/or <NUM>, respectively) arranged about the specimen testing apparatus <NUM>. More or less numbers of analyzers can be used. The analyzers may be any combination of clinical chemistry analyzers and/or assaying instruments, or the like. The specimen containers <NUM> may be any suitably transparent or translucent container, such as a blood collection tube, test tube, sample cup, cuvette, or other clear or opaque glass or plastic container capable of containing and allowing imaging of the specimen <NUM> contained therein. The specimen containers <NUM> may be varied in size.

Specimens <NUM> (<FIG>) may be provided to the specimen testing apparatus <NUM> in the specimen containers <NUM>, which may be capped with a cap <NUM>. The caps <NUM> may be of different types and/or colors (e.g., red, royal blue, light blue, green, grey, tan, yellow, or color combinations), which may have meaning in terms of what test the specimen container <NUM> is used for, the type of additive included therein, whether the container includes a gel separator, or the like. Other colors may be used. In one embodiment, the cap type may be determined by the characterization method described herein.

Each of the specimen containers <NUM> may be provided with a label <NUM>, which may include identification information 218i (i.e., indicia) thereon, such as a barcode, alphabetic, numeric, or combination thereof. The identification information 218i may be machine readable at various locations about the specimen testing apparatus <NUM>. The machine readable information may be darker (e.g., black) than the label material (e.g., white paper) so that it can be readily imaged. The identification information 218i may indicate, or may otherwise be correlated, via a Laboratory Information System (LIS) <NUM>, to a patient's identification as well as tests to be performed on the specimen <NUM>. The identification information 218i may indicate other or additional information. Such identification information 218i may be provided on the label <NUM>, which may be adhered to or otherwise provided on an outside surface of the tube <NUM>. As shown in <FIG>, the label <NUM> may not extend all the way around the specimen container <NUM> or all along a length of the specimen container <NUM> such that from the particular front viewpoint shown, a large part of the serum or plasma portion 212SP is viewable (the part shown dotted) and unobstructed by the label <NUM>.

However, in some embodiments, multiple labels <NUM> may have been provided (such as from multiple facilities that have handled the specimen container <NUM>), and they may overlap each other to some extent. For example, two labels (e.g., a manufacturer's label and a barcode label) may be provided and may be overlapping and may occlude (obstruct) some or all of one or more viewpoints.

Thus, it should be understood that in some embodiments, although the label(s) <NUM> may occlude some portion of the specimen <NUM> (an occluded portion), some portion of the specimen <NUM> and serum and plasma portion 212SP may still be viewable from at least one viewpoint (an un-occluded portion). Thus, in accordance with another aspect of the disclosure, embodiments of the SDNN configured to carry out the characterization method can be trained to recognize the occluded and un-occluded portions, such that improved HILN detection may be provided.

Again referring to <FIG>, the specimen <NUM> may include the serum or plasma portion 212SP and a settled blood portion 212SB contained within the tube <NUM>. Air <NUM> may be provided above the serum and plasma portion 212SP and a line of demarcation between them is defined as the liquid-air interface (LA). The line of demarcation between the serum or plasma portion 212SP and the settled blood portion 212SB is defined as a serum-blood interface (SB). An interface between the air <NUM> and cap <NUM> is defined as a tube-cap interface (TC). The height of the tube (HT) is defined as a height from a bottom-most part of the tube <NUM> to a bottom of the cap <NUM>, and may be used for determining tube size. A height of the serum or plasma portion 212SP is (HSP) and is defined as a height from a top of the serum or plasma portion 212SP to a top of the settled blood portion 212SB. A height of the settled blood portion 212SB is (HSB) and is defined as a height from the bottom of the settled blood portion 212SB to a top of the settled blood portion 212SB at SB. HTOT is a total height of the specimen <NUM> and equals HSP plus HSB.

In cases where a gel separator <NUM> is used (<FIG>), the height of the serum or plasma portion 212SP is (HSP) and is defined as a height from the top of the serum or plasma portion 212SP at LA to the top of the gel separator <NUM> at SG, wherein SG is an interface between the serum or plasma portion 212SP and the gel separator <NUM>. A height of the settled blood portion 212SB is (HSB) and is defined as a height from the bottom of the settled blood portion 212SB to the bottom of the gel separator <NUM> at BG, wherein BG is an interface between the settled blood portion 212SB and the gel separator <NUM>. HTOT is the total height of the specimen <NUM> and equals HSP plus HSB plus height of the gel separator <NUM>. In each case, Tw is a wall thickness, W is an outer width, which may also be used for determining the size of the specimen container <NUM>, and Wi is an inner width of the specimen container <NUM>.

In more detail, specimen testing apparatus <NUM> may include a base <NUM> (<FIG>) (e.g., a frame, floor, or other structure) upon which a track <NUM> may be mounted. The track <NUM> may be a railed track (e.g., a mono rail or a multiple rail), a collection of conveyor belts, conveyor chains, moveable platforms, or any other suitable type of conveyance mechanism. Track <NUM> may be circular or any other suitable shape and may be a closed track (e.g., endless track) in some embodiments. Track <NUM> may, in operation, transport individual ones of the specimen containers <NUM> to various locations spaced about the track <NUM> in carriers <NUM>.

Carriers <NUM> may be passive, non-motored pucks that may be configured to carry a single specimen container <NUM> on the track <NUM>, or optionally, an automated carrier including an onboard drive motor, such as a linear motor that is programmed to move about the track <NUM> and stop at pre-programmed locations. Other configurations of carrier <NUM> may be used. Carriers <NUM> may each include a holder <NUM> (<FIG>) configured to hold the specimen container <NUM> in a defined upright position and orientation. The holder <NUM> may include a plurality of fingers or leaf springs that secure the specimen container <NUM> on the carrier <NUM>, but some may be moveable or flexible to accommodate different sizes of the specimen containers <NUM>. In some embodiments, carriers <NUM> may leave from the loading area <NUM> after being offloaded from the one or more racks <NUM>. The loading area <NUM> may serve a dual function of also allowing reloading of the specimen containers <NUM> from the carriers <NUM> to the loading area <NUM> after pre-screening and/or analysis is completed.

A robot <NUM> may be provided at the loading area <NUM> and may be configured to grasp the specimen containers <NUM> from the one or more racks <NUM> and load the specimen containers <NUM> onto the carriers <NUM>, such as onto an input lane of the track <NUM>. Robot <NUM> may also be configured to reload specimen containers <NUM> from the carriers <NUM> to the one or more racks <NUM>. The robot <NUM> may include one or more (e.g., least two) robot arms or components capable of X (lateral) and Z (vertical - out of the paper, as shown), Y and Z, X, Y, and Z, or r (radial) and theta (rotational) motion. Robot <NUM> may be a gantry robot, an articulated robot, an R-theta robot, or other suitable robot wherein the robot <NUM> may be equipped with robotic gripper fingers oriented, sized, and configured to pick up and place the specimen containers <NUM>.

Upon being loaded onto track <NUM>, the specimen containers <NUM> carried by carriers <NUM> may progress to a first pre-processing station <NUM>. For example, the first pre-processing station <NUM> may be an automated centrifuge configured to carry out fractionation of the specimen <NUM>. Carriers <NUM> carrying specimen containers <NUM> may be diverted to the first pre-processing station <NUM> by inflow lane or other suitable robot. After being centrifuged, the specimen containers <NUM> may exit on outflow lane, or otherwise be removed by a robot, and continue along the track <NUM>. In the depicted embodiment, the specimen container <NUM> in carrier <NUM> may next be transported to a quality check module <NUM> to carry out pre-screening, as will be further described herein with reference to <FIG> herein.

The quality check module <NUM> is configured to pre-screen and carry out the characterization methods described herein, and is configured to automatically determine a presence of, and possibly an extent of H, I, and/or L contained in a specimen <NUM> or whether the specimen is normal (N). If found to contain effectively-low amounts of H, I and/or L, so as to be considered normal (N), the specimen <NUM> may continue on the track <NUM> and then may be analyzed by the one or more analyzers (e.g., first, second, and/or third analyzers <NUM>, <NUM>, and/or <NUM>). Thereafter, the specimen container <NUM> may be returned to the loading area <NUM> for reloading to the one or more racks <NUM>.

In some embodiments, in addition to detection of HILN, segmentation of the specimen container <NUM> and specimen <NUM> may take place. From the segmentation data, post processing may be used for quantification of the specimen <NUM> (i.e., determination of HSP, HSB, HTOT, and determination of location of SB or SG, and LA). In some embodiments, characterization of the physical attributes (e.g., size - height and width/diameter) of the specimen container <NUM> may take place at the quality check module <NUM>. Such characterization may include determining HT and W, and possibly TC, and/or Wi. From this characterization, the size of the specimen container <NUM> may be extracted. Moreover, in some embodiments, the quality check module <NUM> may also determine cap type, which may be used as a safety check and may catch whether a wrong tube type has been used for the test ordered.

In some embodiments, a remote station <NUM> may be provided on the specimen testing apparatus <NUM> that is not directly linked to the track <NUM>. For instance, an independent robot <NUM> (shown dotted) may carry specimen containers <NUM> containing specimens <NUM> to the remote station <NUM> and return them after testing/pre-processing. Optionally, the specimen containers <NUM> may be manually removed and returned. Remote station <NUM> may be used to test for certain constituents, such as a hemolysis level, or may be used for further processing, such as to lower a lipemia level through one or more additions and/or through additional processing, or to remove a clot, bubble or foam, for example. Other pre-screening using the HILN detection methods described herein may be accomplished at remote station <NUM>.

Additional station(s) may be provided at one or more locations on or along the track <NUM>. The additional station(s) may include a de-capping station, aliquoting station, one or more additional quality check modules <NUM>, and the like.

The specimen testing apparatus <NUM> may include a number of sensors <NUM> at one or more locations around the track <NUM>. Sensors <NUM> may be used to detect a location of specimen containers <NUM> on the track <NUM> by means of reading the identification information 218i, or like information (not shown) provided on each carrier <NUM>. Any suitable means for tracking the location may be used, such as proximity sensors. All of the sensors <NUM> may interface with the computer <NUM>, so that the location of each specimen container <NUM> may be known at all times.

The pre-processing stations and the analyzers <NUM>, <NUM>, <NUM> may be equipped with robotic mechanisms and/or inflow lanes configured to remove carriers <NUM> from the track <NUM>, and with robotic mechanisms and/or outflow lanes configured to reenter carriers <NUM> to the track <NUM>.

Specimen testing apparatus <NUM> may be controlled by the computer <NUM>, which may be a microprocessor-based central processing unit CPU, having a suitable memory and suitable conditioning electronics and drivers for operating the various system components. Computer <NUM> may be housed as part of, or separate from, the base <NUM> of the specimen testing apparatus <NUM>. The computer <NUM> may operate to control movement of the carriers <NUM> to and from the loading area <NUM>, motion about the track <NUM>, motion to and from the first pre-processing station <NUM> as well as operation of the first pre-processing station <NUM> (e.g., centrifuge), motion to and from the quality check module <NUM> as well as operation of the quality check module <NUM>, and motion to and from each analyzer <NUM>, <NUM>, <NUM> as well as operation of each analyzer <NUM>, <NUM>, <NUM> for carrying out the various types of testing (e.g., assay or clinical chemistry).

For all but the quality check module <NUM>, the computer <NUM> may control the specimen testing apparatus <NUM> according to software, firmware, and/or hardware commands or circuits such as those used on the Dimension® clinical chemistry analyzer sold by Siemens Healthcare Diagnostics Inc. of Tarrytown, New York, and such control is typical to those skilled in the art of computer-based electromechanical control programming and will not be further described herein. However, other suitable systems for controlling the specimen testing apparatus <NUM> may be used. The control of the quality check module <NUM> may also be provided by the computer <NUM>, but in accordance with the inventive characterization methods described in detail herein.

The computer <NUM> used for image processing and to carry out the characterization methods described herein may include a CPU or GPU, sufficient processing capability and RAM, and suitable storage. In one example, the computer <NUM> may be a multi-processor-equipped PC with one or more GPUs, <NUM> GB Ram or more, and a Terabyte or more of storage. In another example, the computer <NUM> may be a GPU-equipped PC, or optionally a CPU-equipped PC operated in a parallelized mode. MKL could be used as well, <NUM> GB RAM or more, and suitable storage.

Embodiments of the disclosure may be implemented using a computer interface module (CIM) <NUM> that allows a user to easily and quickly access a variety of control and status display screens. These control and status display screens may display and enable control of some or all aspects of a plurality of interrelated automated devices used for preparation and analysis of specimens <NUM>. The CIM <NUM> may be employed to provide information about the operational status of a plurality of interrelated automated devices as well as information describing the location of any specimen <NUM>, as well as a status of tests to be performed on, or being performed on, the specimen <NUM>. The CIM <NUM> is thus adapted to facilitate interactions between an operator and the specimen testing apparatus <NUM>. The CIM <NUM> may include a display screen operative to display a menu including icons, scroll bars, boxes, and buttons through which the operator may interface with the specimen testing apparatus <NUM>. The menu may comprise a number of function elements programmed to display and/or operate functional aspects of the specimen testing apparatus <NUM>.

<FIG> show a first embodiment of a quality check module <NUM> configured to carry out the characterization methods as shown and described herein. Quality check module <NUM> may be configured to pre-screen for presence of an interferent (e.g., H, I, and/or L) in a specimen <NUM> (e.g., in a serum or plasma portion 212SP thereof) prior to analysis by the one or more analyzers <NUM>, <NUM>, <NUM>. Pre-screening in this manner allows for additional processing, additional quantification or characterization, and/or discarding and/or redrawing of a specimen <NUM> without wasting valuable analyzer resources or possibly having the presence of an interferent affect the veracity of the test results.

In addition to the interferent detection methods described herein, other detection methods may take place on the specimen <NUM> contained in the specimen container <NUM> at the quality check module <NUM>. For example, a method may be carried out at the quality check module <NUM> to provide segmentation as an output from the SDNN. The segmentation data may be used in a post processing step to quantify the specimen <NUM>, i.e., determine certain physical dimensional characteristics of the specimen <NUM> (e.g., LA and SB, and/or determination of HSP, HSB, and/or HTOT). Quantification may also involve estimating, e.g., a volume of the serum or plasma portion (VSP) and/or a volume of the settled blood portion (VSB). Other quantifiable geometrical features may also be determined.

Furthermore, the quality check module <NUM> may be used to quantify geometry of the specimen container <NUM>, i.e., quantify certain physical dimensional characteristics of the specimen container <NUM>, such as the location of TC, HT, and/or W or Wi of the specimen container <NUM>.

Referring to <FIG>, <FIG>, the quality check module <NUM> includes multiple image capture devices 440A-440C. Three image capture devices 440A-440C are shown and are preferred, but optionally two or four or more can be used. Image capture devices 440A-440C may be any suitable device for capturing well-defined digital images, such as conventional digital cameras capable of capturing a pixelated image, charged coupled devices (CCD), an array of photodetectors, one or more CMOS sensors, or the like. For example, the three image capture devices 440A, 440B, 440C are illustrated in <FIG> and are configured to capture images from three different lateral viewpoints (viewpoints labeled <NUM>, <NUM>, and <NUM>). The captured image size may be, e.g., about <NUM> x <NUM> pixels. In another embodiment, the image capture devices 440A, 440B, 440C may capture an image size that may be about <NUM> x <NUM> pixels, for example. Other image sizes and pixel densities may be used.

Each of the image capture devices 440A, 440B, 440C is configured and operable to capture lateral images of at least a portion of the specimen container <NUM>, and at least a portion of the specimen <NUM>. For example, the image capture devices 440A-440C may capture a part of the label <NUM> and part or all of the serum or plasma portion 212SP. In some instances, e.g., part of a viewpoint <NUM>-<NUM> may be partially occluded by label <NUM>. In some embodiments, one or more of the viewpoints <NUM>-<NUM> may be fully occluded, i.e., no clear view of the serum or plasma portion 212SP may be possible. However, even in cases where a side (front side or back side) of a viewpoint <NUM>-<NUM> is fully occluded by one or more labels <NUM>, the characterization method may still be able to distinguish the boundaries of the serum or plasma portion 212SP through the one or more occluding labels <NUM>.

In the embodiment shown, the plurality of image capture devices 440A, 440B, 440C are configured to capture lateral images of the specimen container <NUM> and specimen <NUM> at an imaging location <NUM> from the multiple viewpoints <NUM>-<NUM>. The viewpoints <NUM>-<NUM> may be arranged so that they are approximately equally spaced from one another, such as about <NUM>° from one another, as shown. As depicted, the image capture devices 440A, 440B, 440C may be arranged around the track <NUM>. Other arrangements of the plurality of image capture devices 440A, 440B, 440C may be used. In this way, the images of the specimen <NUM> in the specimen container <NUM> may be taken while the specimen container <NUM> is residing in the carrier <NUM> at the imaging location <NUM>. The field of view of the multiple images obtained by the image capture devices 440A, 440B, 440C may overlap slightly in a circumferential extent.

In one or more embodiments, the carriers <NUM> may be stopped at a pre-determined location in the quality check module <NUM>, such as at the imaging location <NUM>, i.e., such as at a point where normal vectors from each of the image capture devices 440A, 440B, 440C intersect each other. A gate or the linear motor of the carrier <NUM> may be provided to stop the carriers <NUM> at the imaging location <NUM>, so that multiple quality images may be captured thereat. In an embodiment where there is a gate at the quality check module <NUM>, one or more sensors (like sensors <NUM>) may be used to determine the presence of a carrier <NUM> at the quality check module <NUM>.

The image capture devices 440A, 440B, 440C may be provided in close proximity to and trained or focused to capture an image window at the imaging location <NUM>, wherein the image window is an area including an expected location of the specimen container <NUM>. Thus, the specimen container <NUM> may be stopped so that it is approximately located in a center of the view window in some embodiments. Within the images captured, one or more reference datum may be present.

In operation of the quality check module <NUM>, each image may be triggered and captured responsive to a triggering signal provided in communication lines 443A, 443B, 443C that may be sent by the computer <NUM>. Each of the captured images may be processed by the computer <NUM> according to one or more embodiments. In one particularly effective method, high dynamic range (HDR) processing may be used to capture and process the image data from the captured images. In more detail, multiple images are captured of the specimen <NUM> at the quality check module <NUM> at multiple different exposures (e.g., at different exposure times), while being sequentially illuminated at one or more different spectra. For example, each image capture device 440A, 440B, 440C may take <NUM>-<NUM> images of the specimen container <NUM> including the serum or plasma portion 212SP at different exposure times at each of multiple spectra. For example, <NUM>-<NUM> images may be taken by image capture device 440A at viewpoint <NUM> while the specimen <NUM> is backlit illuminated with light source 444A that has a red spectrum. Additional like images may be taken sequentially at viewpoints <NUM> and <NUM>.

In some embodiments, the multiple spectral images may be accomplished using different light sources 444A-444C emitting different spectral illumination. The light sources 444A-444C may back light the specimen container <NUM> (as shown). A light diffuser may be used in conjunction with the light sources 444A-444C in some embodiments. The multiple different spectra light sources 444A-444C may be RGB light sources, such as LEDs emitting nominal wavelengths of <NUM> +/- <NUM> (Red), <NUM> +/- <NUM> (Green), and <NUM> +/- <NUM> (Blue). In other embodiments, the light sources 444A-444C may be white light sources. In cases where the label <NUM> obscures multiple viewpoints, IR backlighting or NIR backlighting may be used. Furthermore, RGB light sources may be used in some instances even when label occlusion is present. In other embodiments, the light sources 444A-444C may emit one or more spectra having a nominal wavelength between about <NUM> and about <NUM>.

By way of a non-limiting example, to capture images at a first wavelength, three red light sources 444A-444C (wavelength of about <NUM> +/- <NUM>) may be used to sequentially illuminate the specimen <NUM> from three lateral locations. The red illumination by the light sources 444A-444C may occur as the multiple images (e.g., <NUM>-<NUM> images or more) at different exposure times are captured by each image capture device 440A-440C from each viewpoint <NUM>-<NUM>. In some embodiments, the exposure times may be between about <NUM> and <NUM>. Other exposure times may be used. In some embodiments, each of the respective images for each image capture device 440A-440C may be taken sequentially, for example. Thus, for each viewpoint <NUM>-<NUM>, a group of images are sequentially obtained that have red spectral backlit illumination and multiple (e.g., <NUM>-<NUM> exposures, such as different exposure times). The images may be taken in a round robin fashion, for example, where all images from viewpoint <NUM> are taken followed sequentially by viewpoints <NUM> and <NUM>.

In each embodiment, the quality check module <NUM> may include a housing <NUM> that may at least partially surround or cover the track <NUM> to minimize outside lighting influences. The specimen container <NUM> may be located inside the housing <NUM> during the image-taking sequences. Housing <NUM> may include one or more doors 446D to allow the carriers <NUM> to enter into and/or exit from the housing <NUM>. In some embodiments, the ceiling may include an opening 446O to allow a specimen container <NUM> to be loaded into the carrier <NUM> by a robot including moveable robot fingers from above.

Once the red illuminated images are captured in the embodiment of <FIG>, another spectra of light, for example, green spectral light sources 444A-444C may be turned on (nominal wavelength of about <NUM> with a bandwidth of about +/- <NUM>), and multiple images (e.g., <NUM>-<NUM> or more images) at different exposure times may be sequentially captured by each image capture device 440A, 440B, 440C. This may be repeated with blue spectral light sources 444A-444C (nominal wavelength of about <NUM> with a bandwidth of about +/- <NUM>) for each image capture devices 440A, 440B, 440C. The different nominal wavelength spectral light sources 444A-444C may be accomplished by light panels including banks of different desired spectral light sources (e.g., R, G, B, W, IR, and/or NIR) that can be selectively turned on and off, for example. Other means for backlighting may be used.

The multiple images taken at multiple exposures (e.g., exposure times) for each respective wavelength spectra may be obtained in rapid succession, such that the entire collection of backlit images for the specimen container <NUM> and specimen <NUM> from multiple viewpoints <NUM>-<NUM> may be obtained in less than a few seconds, for example. In one example, four different exposure images for each wavelength at three viewpoints <NUM>-<NUM> using the image capture devices 440A, 440B, 440C and back lighting with RGB light sources 444A-444C will result in <NUM> images x <NUM> spectra x <NUM> image capture devices = <NUM> images. In another example, <NUM> different exposure images for each wavelength at three viewpoints using the image capture devices 440A, 440B, 440C and back lighting with R, G, B, W, IR, and NIR light sources 444A-444C will result in <NUM> images x <NUM> spectra x <NUM> cameras = <NUM> images.

According to embodiments of the characterization methods, the processing of the image data may involve a preprocessing step including, for example, selection of optimally-exposed pixels from the multiple captured images at the different exposure times at each wavelength spectrum and for each image capture device 440A-440C, so as to generate optimally-exposed image data for each spectrum and for each viewpoint <NUM>-<NUM>. This is referred to as "image consolidation" herein.

For each corresponding pixel (or patch), for each of the images from each image capture device 440A-440C, pixels (or patches) exhibiting optimal image intensity may be selected from each of the different exposure images for each viewpoint <NUM>-<NUM>. In one embodiment, optimal image intensity may be pixels (or patches) that fall within a predetermined range of intensities (e.g., between <NUM>-<NUM> on a scale of <NUM>-<NUM>), for example. In another embodiment, optimal image intensity may be between <NUM>-<NUM> on a scale of <NUM>-<NUM>), for example. If more than one pixel (or patch) in the corresponding pixel (or patch) locations of two exposure images is determined to be optimally exposed, the higher of the two is selected.

The selected pixels (or patches) exhibiting optimal image intensity may be normalized by their respective exposure times. The result is a plurality of normalized and consolidated spectral image data sets for the illumination spectra (e.g., R, G, B, white light, IR, and/or IR - depending on the combination used) and for each image capture device 440A-440C where all of the pixels (or patches) are optimally exposed (e.g., one image data set per spectrum) and normalized. In other words, for each viewpoint <NUM>-<NUM>, the data pre-processing carried out by the computer <NUM> results in a plurality of optimally-exposed and normalized image data sets, one for each illumination spectra employed.

<FIG> shows apparatus <NUM> that includes functional components configured carry out the HILN characterization method described herein. Apparatus <NUM> may be embodied as a quality check module <NUM> controlled by the computer <NUM>. As discussed above, the specimen container <NUM> may be provided at the imaging location <NUM> (<FIG>) of the quality check module <NUM> in functional block <NUM>. The multi-view images are captured in functional block <NUM> by the plurality of image capture devices 440A-440C. The image data for each of the multi-view, multi-spectral, multi-exposure images may be pre-processed in functional block <NUM> as discussed above to provide a plurality of optimally-exposed and normalized image data sets (hereinafter "image data sets"). Moreover, respective image data from each of the plurality of image capture devices 440A-440C are stacked as a single input with additional channels corresponding to the number of image capture devices (e.g., three times more channels corresponding to the three image capture devices 440A-440C). For example, images from the three image capture devices 440A-440C may be stacked along the channel dimension, wherein each image capture device may generate a hyper-spectrum image having a dimension of 1272x360x6. The resulting stacked image input may then have a dimension of 1272x360x18, wherein the first <NUM> channels belong to the image capture device 440A, the second <NUM> channels belong to the image capture devices 440B, and the third <NUM> channels belong to the image capture device 440C. This stacked image input may be provided to a single deep convolutional neural network (SDNN) <NUM>.

The SDNN <NUM> is advantageous over known techniques that separately process images from each image capture device via a respective convolutional neural network. That is, if three image capture devices were used, known techniques included three separate convolutional neural networks and three separate statistical analyses to determine HILN. Such techniques are not memory efficient or computationally efficient. In contrast, by stacking the images from the plurality of image capture devices 440A-440C into a single stacked image input and by processing the stacked image inputs with the SDNN <NUM> as described herein, higher memory and computational efficiency is achieved.

The SDNN <NUM> includes a segmentation convolutional neural network (SCNN) <NUM> that receives the stacked image data and simultaneously outputs multiple pixel label maps <NUM>, wherein the number of pixel label maps <NUM> corresponds to the number of image capture devices (e.g., three, corresponding to the three image capture devices 440A-440C). The SDNN <NUM> may also include a classification convolutional neural network (CCNN) <NUM> that receives the multiple pixel label maps <NUM> as input and outputs a determination of HILN <NUM>, 540I, <NUM>, 540N. Optionally, the SCNN <NUM> may output serum segmentation information <NUM> and/or specimen container/cap type information <NUM>.

Prior to receiving image data from image capture devices 440A-440C for determining HILN (and/or optionally segmentation and/or cap type information), the SDNN <NUM> may have been previously trained to recognize HILN and optionally serum segmentation and/or specimen container/cap type. In some embodiments, the SCNN <NUM> may first be trained without the CCNN <NUM>. Multiple sets of training examples may be used to train the SCNN <NUM>. The SCNN <NUM> may be trained by imaging with the quality check module <NUM> a multitude of samples of specimen containers <NUM> containing specimen <NUM> by graphically outlining various regions of a multitude of examples of specimens <NUM> having various specimen HILN conditions, outlining the various regions of occlusion by label <NUM>, levels of serum or plasma portion 212SP, and the like. Along with the graphical outlines, class characterization information for each area may be provided. As many as <NUM> or more, <NUM> or more, <NUM>,<NUM> or more, or even <NUM>,<NUM> or more images may be used for training the SCNN <NUM>. Each training image may have at least the serum or plasma portion 212SP, its H, I. L, or N identified, various index levels (if output), and the label <NUM> outlined manually to identify and teach the SCNN <NUM> the areas that belong to each class that will be a possible output. The SCNN <NUM> may be tested intermittently with a sample specimen container to see if the SCNN <NUM> is operating at a sufficiently high level of confidence. If not operating at <NUM>% (e.g., <NUM>% confidence level or more) in determining the correct HILN configuration as an output, then more training samples may be imaged and input along with associated characterization information. In embodiments where segmentation and/or cap type is also provided, the training may involve outlining the segmented classes and/or cap types outputted and including as input class identification information. After training of SCNN <NUM> alone, CCNN <NUM> may be added to SCNN <NUM> and both networks may be additionally trained end-to-end, wherein any segmentation and classification losses can be combined at the end. That is, the loss from the CCNN <NUM> can be back-propagated to the SCNN <NUM>.

In some embodiments of apparatus <NUM>, the output of the SDNN <NUM> may be N-class hemolytic <NUM>, N-class icteric 540I, N-class lipemic <NUM>, or normal 540N, wherein N-class is the number (N) of class options in that interferent class. As before, stacked multi-view, multi-spectral, multi-exposure consolidated and normalized image data sets may be input into the SDNN <NUM> and the image data sets may be operated upon and processed by the SCNN <NUM> and CCNN <NUM>. The output of the processing by the SDNN <NUM> may be multiple output possibilities (N-classes) for each of HIL, and/or for each viewpoint. For example, N may equal three, wherein the outputs may include H1, H2, and H3 at <NUM>; <NUM>, I2, and I3 at 540I; and L1, L2, L3 at <NUM>.

<FIG> illustrates an architecture <NUM> of SCNN <NUM> in accordance with one or more embodiments. SCNN <NUM> may be coded using any suitable scientific computing framework, program, or toolbox, such as, for example, Caffe available from Berkley Vision and Learning Center (BVLC), Theano, a Python framework for fast computation of mathematical expressions, TensorFlow, Torch, and the like. Architecture <NUM> may include the following operational layers: two convolutional layers (CONV1 and CONV2) <NUM> and <NUM>; five dense block layers (DB1-DB5) <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>; four concatenation layers (C1-C4) <NUM>, <NUM>, <NUM>, and <NUM>; two transition down layers (TD1 and TD2) <NUM> and <NUM>; and two transition up layers (TU1 and TU2), arranged as shown in <FIG> wherein multiple pixel label maps <NUM> are output. Note that the input to each dense block layer <NUM> and <NUM> is concatenated (at C1 and C2, respectively) with its output, which may result in a linear growth of the number of pixel label maps. Each dense block layer <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may include multiple layers (e.g., <NUM> or <NUM>), each including a batch normalization operation, a ReLu layer, and a 3x3 convolutional layer with dropout p = <NUM>. A first layer receives an input and outputs a number of pixel label maps, which are concatenated to the input. A second layer then receives the concatenated output as its input and outputs a number of pixel label maps, which are again concatenated to the previous pixel label maps. This is repeated for each layer in the dense label block. Each transition down layer <NUM> and <NUM> may include a batch normalization operation, followed by a ReLu layer, followed by a 1x1 convolutional layer with dropout p = <NUM>, followed by a 2x2 max pooling layer. Each transition up layer <NUM> and <NUM> may include a 3x3 transposed convolutional layer with stride <NUM>.

<FIG> illustrates an architecture <NUM> of CCNN <NUM> in accordance with one or more embodiments. CCNN <NUM> may be coded using any suitable scientific computing framework, program, or toolbox, such as, for example, Caffe available from Berkley Vision and Learning Center (BVLC), Theano, a Python framework for fast computation of mathematical expressions, TensorFlow, Torch, and the like. Architecture <NUM> may include the following operational layers: five sets of convolutional layers (CONV1-CONV5) <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> and max pooling layers (POOL1-POOL5) <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, followed by two fully-connected layers (FC1 and FC2) <NUM> and <NUM>, followed by a softmax layer <NUM>. Convolutional layer <NUM> receives as input multiple pixel label maps <NUM>, which may be multiple pixel label maps <NUM> or <NUM> (<FIG> and <FIG>, respectively). Convolutional layer <NUM> may be one or more 3x3 convolutional layers of depth of <NUM> (i.e., <NUM> filters). Convolutional layer <NUM> may be one or more 3x3 convolutional layers of depth of <NUM> (i.e., <NUM> filters). Convolutional layer <NUM> may be one or more 3x3 convolutional layers of depth of <NUM> (i.e., <NUM> filters). Convolutional layer <NUM> may be one or more 3x3 convolutional layers of depth of <NUM> (i.e., <NUM> filters). And convolutional layer <NUM> may be one or more 3x3 convolutional layers also of depth of <NUM> (i.e., <NUM> filters). Fully-connected layers <NUM> and <NUM> may each be of size <NUM>, while softmax layer may be of size <NUM>.

As used herein, a convolution layer is a processing step that may apply a filter (also referred to as a kernel) to input image data (e.g., pixel intensity values) to output an activation map that may indicate detection of some specific type of feature (e.g., from a simple curve after application of a first convolution layer to somewhat more complex features after application of several convolution layers) at some spatial position in the input image data.

A max pooling layer is a processing step that may apply a filter to generate output activation maps having maximum pixel values appearing in the one or more activation maps received from a convolutional layer.

A ReLU (rectified linear unit) layer is a processing step that may apply a nonlinear function to all values in a received activation map resulting in, e.g., all negative activation values being assigned a value of zero.

A fully connected layer is a processing step that aggregates previous activation maps (each of which may indicate detection of lower level features) to indicate detection of higher-level features.

A softmax layer is typically a final processing step that outputs a probability distribution highlighting or identifying the most likely feature of one or more images from a class of image features.

By using a single deep convolutional neural network such as SDNN <NUM> for receiving image data from all the image capture devices 440A-440C, instead of using a respective convolutional neural network for each image capture device 440A-440C as in some known apparatus, and by attaching a classification convolutional neural network (e.g., CCNN <NUM> and architecture <NUM>) to the segmentation convolutional neural network (e.g., SCNN <NUM> and architecture <NUM>), the single deep convolutional neural network design has higher memory and computational efficiency as well as increased system performance as compared to other known convolutional neural networks.

<FIG> illustrates a flowchart of a characterization method <NUM> according to embodiments of the disclosure. The characterization method <NUM> may be carried out by quality check module <NUM> as described herein. In particular, the characterization method <NUM> may determine a presence of an interferent in a specimen <NUM> according to one or more embodiments. The characterization method <NUM> includes, in process block <NUM>, capturing multiple images of a specimen container (e.g., specimen container <NUM>) including a serum or plasma portion (e.g., serum or plasma portion 212SP) of a specimen (e.g., specimen <NUM>) from multiple viewpoints (e.g., viewpoints <NUM>, <NUM>, and <NUM>). Moreover, the specimen container <NUM> may include one or more labels (e.g., label <NUM>) thereon. The one or more images may be digital, pixelated images captured using one or more image capture devices (e.g., image capture devices 440A-440C).

The characterization method <NUM> further includes, in process block <NUM>, inputting image data (e.g., stacked, consolidated, and normalized image data sets) from the multiple images to a segmentation convolutional neural network (e.g., SCNN <NUM>) and processing the image data with the SCNN to simultaneously output multiple label maps. The processing may be accomplished by the computer <NUM> described herein after suitable training of the SCNN <NUM>.

In process block <NUM>, the characterization method <NUM> includes inputting the multiple label maps to a classification convolutional neural network (e.g., CCNN <NUM>) and processing the multiple label maps with the CCNN. The processing may be accomplished by the computer <NUM> described herein after suitable training of the SCNN <NUM> and the CCNN <NUM>.

The characterization method <NUM> further includes, in process block <NUM>, outputting from the classification convolutional neural network (e.g., CCNN <NUM>) a classification of the serum or plasma portion as being one or more of hemolytic, icteric, lipemic, and normal (i.e., H, I, L, H and I, H and L, I and L, H,I, and L, or N).

The multiple images from the multiple viewpoints may be captured at different exposure times and/or at a different spectral illumination (e.g., R, G, B, white light, IR, and/or near IR). For example, there may be <NUM>-<NUM> different exposures or more taken at different exposure times for each viewpoint under the different spectral illumination conditions.

In an optional aspect, in addition to the HILN determination, a segmentation of the image data sets may be obtained. The method <NUM> may, in process block <NUM>, output from the SCNN (e.g. SCNN <NUM>) a segmentation of the specimen container <NUM> and specimen <NUM>. The image data may be segmented into N'-classes (e.g., <NUM> classes), such as (<NUM>) Tube, (<NUM>) Gel Separator, (<NUM>) Cap, (<NUM>) Air, (<NUM>) Label, (<NUM>) Settled Blood Portion, and/or (<NUM>) Serum or Plasma Portion. Other numbers of classes may be used.

The characterization method <NUM> may also optionally include, in process block <NUM>, outputting from the SCNN (e.g., SCNN <NUM>) a cap type (<NUM>), which may be a specific cap shape or cap color that was pre-trained into the SCNN <NUM> and the CCNN <NUM>.

Accordingly, based on the foregoing it should be apparent that an improved characterization method <NUM> is provided that better characterizes the serum or plasma portion 212SP by accounting for labels that may occlude the one or more viewpoints. The improved characterization may be used to provide a rapid and robust characterization of a presence of HILN in the specimen <NUM>, and in some embodiments, an interferent level (H1, H2, H3, I1, I2, I3, L1, L2, L3) may be assessed and output from the CCNN <NUM>.

As should be apparent, the above characterization methods may be carried out using a quality check module (e.g., quality check module <NUM>), comprising a plurality of image capture devices (e.g., image capture devices) 440A-440C arranged around an imaging location (e.g., imaging location <NUM>), and configured to capture multiple images from multiple viewpoints (e.g., multiple viewpoints <NUM>-<NUM>) of a specimen container <NUM> including one or more labels <NUM> and containing a serum or plasma portion 212SP of a specimen <NUM>, and a computer (e.g., computer <NUM>) coupled to the plurality of image capture devices and configured to process image data of the multiple images. The computer (e.g., computer <NUM>) may be configured and capable of being operated to process and stack the multiple images from the multiple viewpoints (e.g., viewpoints <NUM>-<NUM>) to provide HILN determination or HILN determination in combination with segmentation for each of the multiple viewpoints.

Claim 1:
A method (<NUM>) of characterizing a specimen container (<NUM>), comprising:
capturing (<NUM>) multiple images of the specimen container from multiple viewpoints, the specimen container including a serum or plasma portion (212SP) of a specimen (<NUM>) therein;
inputting (<NUM>) image data (<NUM>) from the multiple images to a segmentation convolutional neural network (<NUM>) and processing the image data with the segmentation convolutional neural network to simultaneously output multiple label maps (<NUM>, <NUM>, <NUM>);
inputting (<NUM>) the multiple label maps to a classification convolutional neural network (<NUM>) and processing the multiple label maps with the classification convolutional neural network; and
outputting (<NUM>) from the classification convolutional neural network a classification of the serum or plasma portion as being one or more of hemolytic, icteric, lipemic, or normal (<NUM>, <NUM>, <NUM>, 540N), characterized in that the method further comprises stacking together a respective captured image from each viewpoint into a single image with additional channels corresponding to the number of viewpoints, wherein the image data (<NUM>) from the multiple images comprises consolidated pixel or patch data from multiple exposures, and wherein the number of simultaneously output multiple label maps (<NUM>, <NUM>, <NUM>) corresponds to the number of multiple viewpoints.