Patent Description:
Analyte measurement systems that are known to the art enable the analysis of a bodily fluid dose provided by a user to identify the level of one or more analytes in the body of the user using an electronic device and one or more electrochemical reactions. These analyte measurement systems provide significant benefits for the accurate measurement of analytes in fluidic samples (i.e., biological or environmental) for individual users. Some analyte measurement systems employ a test strip that bears a chemical reagent. Upon receiving a fluid dose containing the analyte, a chemical reaction between the reagent and the analyte changes the color of the reagent, where the color change varies based on the concentration of the analyte, which in turn provides a measurement of the analyte. While many analytes are measured in this manner, one specific example of an analyte that is measured in a fluid dose is glucose, which is measured in a bodily fluid dose as part of monitoring and treatment of diabetes mellitus.

Older test strip systems that change color have relied upon a human observer to judge the analyte measurement by observing the change in color in the reagent, often with the assistance of printed color-matching guide. Such manual systems may present problems with reduced accuracy and inconsistent measurements based on the perceptions of different human observers. More recently, automated analyte measurement devices that use cameras to observe the reagent have been developed to provide improvements to analyte measurement accuracy. For example, widely available smartphones include optical sensors and digital image processing hardware that enables the smartphones to generate measurements of analytes in test strips when the smartphone executes a specifically configured analyte measurement software application.

While the use of specifically configured optical measurement devices improves the analyte measurement, challenges remain in ensuring the accuracy of the measurement process. One such challenge occurs in ensuring that optical measurements of a test strip are taken at the appropriate time after the test strip receives a dose of the fluid dose. An optical measurement that is taken too early may be inaccurate because the reagent has not had sufficient time to complete chemical reactions with the analyte, but if the optical measurement is taken too late then the reagent may have experienced drying or bleaching that affect the color of the reagent. Either situation may lead to inaccurate analyte measurement results even if the test strip and analyte measurement device are fully operable. Consequently, improvements to optical analyte measurement systems that overcome these challenges would be beneficial.

<CIT> discloses a system for guiding collection of a hazardous contaminant sample comprising a substrate having a test area for the collection of the hazardous contaminant sample and a developing region, wherein one or more processors are configured to determine the time the liquid is applied to the acquisition pad using images captured by an imaging device and the developing region is scanned after a predetermined time after said application time.

<CIT> discloses a system and method for automated camera-based optical assessment involving color assessment of a physical object, involving use of a specially-configured test card comprising a reagent pad configured to change to an expected color in response to an enzymatic reaction, a colored field with an the expected color and positioned adjacent to the reagent pad, and a QR code as an alignment as well as for verification and/or authentication.

The current invention concerns a method for measuring an analyte according to claim <NUM>. The method includes identifying, with a processor, a test strip in a video stream generated by a camera based on at least one registration mark associated with the test strip depicted in the video stream, identifying, with the processor, application of a fluid dose to a deposit site formed on the test strip based on the video stream, activating, with the processor, a timer in response to the identification of the application of the fluid dose, generating, with an optical sensor, at least one optical measurement of a reagent located at a measurement site on the test strip, and generating, with the processor, a measurement of an analyte in the fluid dose based on the at least one optical measurement of the reagent only in response to the at least one optical measurement being generated after a predetermined minimum time period has elapsed subsequent to the activating of the timer and prior to a predetermined maximum time period elapsing subsequent to the activating of the timer.

According to the current invention, the method includes identifying, with the processor, a vial in a video stream generated by a camera based on at least one of an outline shape of the test vial or at least one registration mark located on the vial depicted in the video stream, identifying, with the processor, an opening of the vial in the video stream based on at least one registration mark located on a lid of the vial, and identifying, with a processor, extraction of the test strip from the vial after the identifying of the opening of the vial in the video stream based on the at least one registration mark associated with the test strip depicted in the video stream.

In a further embodiment of the method, the at least one registration mark associated with the vial further includes an indicator formed on a label of the vial.

In a further embodiment of the method, the at least one registration mark located on the lid of the vial further includes a color marking formed on an interior surface of the lid.

In a further embodiment of the method, the at least one registration mark associated with the test strip further includes a printed mark formed on a side of the test strip at a predetermined position relative to the deposit site.

In a further embodiment, the method includes identifying, with the processor, that a reverse side of the test strip is exposed in the video stream based on an absence of the printed mark formed on the side of the test strip and generating, with the processor and an output device, an output message indicating that the test strip should be rotated to expose the side of the test strip bearing the printed mark.

In a further embodiment of the method, the at least one registration mark associated with the test strip further includes an indicator formed on a rear surface of a color card that holds the test strip.

In a further embodiment, the method of identifying the application of the dose includes identifying, with the processor, a finger of a user in the video stream, and identifying, with the processor, contact between the finger and the deposit site in the video stream, and identifying, with the processor, the application of the dose in response to a change in an optical property of the deposit site in the video stream after the contact between the finger and the deposit site. In a further embodiment, the method of identifying the application of the dose further includes identifying, with the processor, the application of the dose in response to a change in an optical property of the deposit site in the video stream.

In a further embodiment, the method of identifying the application of the dose further includes identifying, with the processor, the application of the dose in response to contact between the finger and the deposit site in the video stream.

In a further embodiment, the method includes generating, with the processor and an output device, an output message informing the user that the measurement of the analyte in the fluid dose cannot be completed in response to no optical measurement of the measurement site being generated after the predetermined minimum time period has elapsed and prior to the predetermined maximum time period elapsing.

In a further embodiment of the method, the optical sensor that generates the measurement is the camera that generates the video stream.

In a further embodiment of the method, the optical sensor that generates the measurement is a camera that is different than the camera that generates the video stream.

In a further embodiment of the method, the camera is incorporated in a wearable electronic device and the optical sensor is incorporated in a mobile electronic device.

The current invention further concerns system for measurement of an analyte according to claim <NUM>. The system includes a wearable electronic device and a mobile electronic device. The wearable electronic device includes a camera configured to generate the video stream and a transmitter configured to transmit the video stream to the mobile electronic device. The mobile electronic device includes a receiver configured to receive the video stream transmitted from the wearable electronic device, an optical sensor configured to generate optical measurements, a memory configured to store program instructions, and a processor operatively connected to the receiver, the optical sensor, and the memory. The processor is configured to execute the program instructions to identify a test strip in the video stream based on at least one registration mark associated with the test strip depicted in the video stream, identify application of a fluid dose to a deposit site formed on the test strip based on the video stream, activate a timer in response to the application of the fluid dose, generate, with the optical sensor, at least one optical measurement of a reagent located at a measurement site on the test strip, and generate a measurement of an analyte in the fluid dose based on the at least one optical measurement of the reagent only in response to the at least one optical measurement being generated after a predetermined minimum time period has elapsed subsequent to the activating of the timer and prior to a predetermined maximum time period elapsing subsequent to the activating of the timer.

According to the current invention, the processor is further configured to identify a vial in the video stream generated by a camera based on at least one registration mark located on the vial depicted in the video stream, identify an opening of the vial in the video stream based on at least one of an outline shape of the test vial or at least one registration mark located on a lid of the vial, and identify extraction of the test strip from the vial after the identification of the opening of the vial in the video stream based on the at least one registration mark associated with the test strip depicted in the video stream.

In a further embodiment of the system, the at least one registration mark associated with the vial further includes an indicator formed on a label of the vial.

In a further embodiment of the system, the at least one registration mark located on the lid of the vial further includes a color marking formed on an interior surface of the lid.

In a further embodiment of the system, the at least one registration mark associated with the test strip further includes an indicator formed on a surface of the test strip at a predetermined position relative to the deposit site.

In a further embodiment, the processor is configured to identify that a reverse side of the test strip is exposed in the video stream based on an absence of the indicator formed on the surface of the test strip, and generate, with an output device, an output message indicating that the test strip should be rotated to expose the surface of the test strip bearing the indicator.

In a further embodiment of the system, the at least one registration mark associated with the test strip further includes an indicator formed on a rear surface of a color card that holds the test strip.

In a further embodiment, the processor is configured to identify a finger of a user in the video stream, identify contact between the finger and the deposit site in the video stream, and identify the application of the dose in response to a change in an optical property of the deposit site in the video stream after the contact between the finger and the deposit site.

In a further embodiment, the processor is configured to identify the application of the dose in response to a change in an optical property of the deposit site in the video stream.

In a further embodiment, the processor is configured to identify the application of the dose in response to contact between the finger and the deposit site in the video stream.

In a further embodiment, the system includes an output device in at least one of the wearable electronic device or the mobile electronic device, the processor being operatively connected to the output device and further configured to generate an output message informing the user that the measurement of the analyte in the fluid dose cannot be completed in response to no optical measurement of the measurement site being generated after the predetermined minimum time period has elapsed and prior to the predetermined maximum time period elapsing.

The advantages, effects, features and objects other than those set forth above will become more readily apparent when consideration is given to the detailed description below. Such detailed description makes reference to the following drawings, wherein:.

These and other advantages, effects, features and objects are better understood from the following description. In the description, reference is made to the accompanying drawings, which form a part hereof and in which there is shown by way of illustration, not limitation, embodiments of the inventive concept. Corresponding reference numbers indicate corresponding parts throughout the several views of the drawings.

While the inventive concept is susceptible to various modifications and alternative forms, exemplary embodiments thereof are shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the description of exemplary embodiments that follows is not intended to limit the inventive concept to the particular forms disclosed, but on the contrary, the intention is to cover all advantages, effects, and features falling within the scope of the invention as defined by the appended claims.

As such, it should be noted that the embodiments described herein may have advantages, effects, and features useful in solving other problems.

The devices, systems and methods now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the inventive concept are shown. Indeed, the devices, systems and methods may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements.

Likewise, many modifications and other embodiments of the devices, systems and methods described herein will come to mind to one of skill in the art to which the disclosure pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the devices, systems and methods are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the embodiments. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of skill in the art to which the disclosure pertains. Although any methods and materials similar to or equivalent to those described herein can be used in the practice or testing of the methods, the preferred methods and materials are described herein. Moreover, reference to an element by the indefinite article "a" or "an" does not exclude the possibility that more than one element is present, unless the context clearly requires that there be one and only one element. The indefinite article "a" or "an" thus usually means "at least one. " Likewise, the terms "have," "comprise" or "include" or any arbitrary grammatical variations thereof are used in a non-exclusive way. For example, the expressions "A has B," "A comprises B" and "A includes B" may refer both to a situation in which, besides B, no other element is present in A (i.e., a situation in which A solely and exclusively consists of B) or to a situation in which, besides B, one or more further elements are present in A, such as element C, elements C and D, or even further elements.

As used herein, the term "mobile electronic device" refers to a portable computing device that provides a user one or more of each of the following components: an output device, an input device, a memory, and a wireless communication device that are controlled by one or more processors in the mobile electronic device. As used herein, the term "wearable electronic device" refers to a type of mobile electronic device that is further adapted to be worn by a human user in a similar manner to glasses, clothing, watches, or jewelry. Examples of output devices include, but are not limited to, liquid crystal display (LCD) displays, organic or inorganic light emitting diode (LED) displays, and other forms of graphical display device, audio speakers, and haptic feedback devices. Examples of input devices include, but are not limited to buttons, keyboards, touchscreens, and audio microphones. Examples of memory include, but are not limited to, both volatile data storage devices such as random-access memory (RAM) and non-volatile data storage devices such as magnetic disks, optical disks, and solid-state storage devices including EEPROMs, NAND flash, or other forms of solid-state data storage devices. Examples of wireless communication devices include, but are not limited to, radio transceivers that operate with the Near Field Communication (NFC) protocol, the Bluetooth protocol family, including Bluetooth Low Energy (BLE), the IEEE <NUM> protocol family ("Wi-Fi"), and cellular data transmission standards ("<NUM>," "<NUM>," or the like). Examples of the processors include digital logic devices that implement one or more central processing units (CPUs), graphics processing units (GPUs), neural network processors (NPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and any other suitable digital logic devices in an integrated device or as a combination of devices that operate together to implement the processor. Common examples of mobile electronic devices include, but are not limited to, smartphones, tablet computing devices, and notebook computers. Common examples of wearable electronic devices include, but are not limited to, smart watches and smart glasses.

<FIG> depicts an analyte measurement system <NUM> that includes a wearable electronic device <NUM> and a mobile electronic device <NUM>. During operation, the wearable electronic device <NUM> generates a video stream that enables the mobile electronic device <NUM> to track a vial <NUM> that holds one or more test strips <NUM> to identify when the test strip <NUM> receives a fluid dose, such as a blood dose from a finger <NUM> of human test subject. As described in further detail below, the system <NUM> starts a timer upon detection of application of a fluid dose to a deposit site <NUM> on one side of the test strip <NUM> to enable an optical sensor <NUM> in the mobile electronic device <NUM> to generate optical measurements of a measurement site <NUM> on a test strip reverse side <NUM>' after a predetermined minimum time has elapsed and prior to a predetermined maximum time elapsing.

In <FIG>, the test strip <NUM> includes a deposit site <NUM> where a user provides a liquid blood sample. The test strip <NUM> also includes registration marks <NUM>, which are depicted as printed marks in the form of indicator arrows on the surface of the test strip <NUM> at a predetermined position relative to the deposit site <NUM>. A hole <NUM> is also formed through one end of the test strip <NUM>. The registration marks <NUM> enable efficient identification and tracking of the test strip <NUM>, including identification of which side of the test strip <NUM> faces the camera in the video stream. In the illustrative example of <FIG>, only one side of the test strip <NUM> is configured to receive the blood sample and the registration marks <NUM> are formed only on this side of the test strip <NUM>, although in alternative embodiments a test strip may be configured to receive a blood dose on either side of the test strip. The test strip reverse side <NUM>' refers to the same test strip <NUM>, but with the reverse side of the test strip in view including the measurement site <NUM> and the hole <NUM>. Neither the registration mark arrow indicators <NUM> nor the deposit site <NUM> are visible to the camera <NUM> when the test strip reverse side <NUM>' is in view. In some embodiments, the system <NUM> can detect that the test strip reverse side <NUM>' is exposed to the camera <NUM>. In the test strip <NUM>, the deposit site <NUM> provides a fluid inlet that enables a fluid dose to permeate through one or more internal layers in the test strip to enable chemical reactions with one or more reagents in the test strip. Examples of internal layers include, for example, filters and different layers of chemical reagents that react to one or more analytes in the fluid dose. The measurement site <NUM> is an optically exposed region formed on the test strip reverse side <NUM>' that changes color in response to the level of analyte in the fluid dose that permeates through the test strip from the deposit site <NUM>. In one configuration, the reagent is exposed at the measurement site <NUM> directly, while in another configuration an optically transparent layer, such as a film, covers the reagent while providing an optical aperture. Both configurations enable generation of optical measurements of the reagent to detect changes in the color of the reagent due to reaction with the analyte in the fluid dose.

Referring to <FIG> and to a schematic diagram in <FIG>, in one embodiment of the system <NUM> the wearable electronic device <NUM> is embodied as a pair of eyeglasses, also referred to as "smart glasses", although other forms of wearable electronic devices including smart watches may be used in alternative configurations. The wearable electronic device <NUM> includes a frame and optional lenses that are similar to a conventional pair of eyeglasses. The wearable electronic device <NUM> further includes a camera <NUM>, positional sensors <NUM>, and a head-up display (HUD) <NUM>, that are each operatively connected to an electronic control unit <NUM>. The camera <NUM> is, for example, a CMOS or other suitable digital imaging device that generates images and video streams of the region in front of the wearable electronic device <NUM> that corresponds to the view of the person wearing the wearable electronic device <NUM>. In some embodiments, a single monochrome or color camera generates the video stream as a two-dimensional video stream. In other embodiments, the camera <NUM> is further configured to generate a video stream that provides three-dimensional object data. In one configuration, the camera <NUM> further incorporates two or more cameras that provide stereoscopic video, and, in another configuration, the camera <NUM> includes a depth sensor that provides three-dimensional depth information corresponding to objects in the video stream. The positional sensors <NUM> include, for example, a microelectromechanical (MEMs) three-axis gyroscope and one or more accelerometers that provide data to identify the spatial orientation of the wearable electronic device <NUM> during operation. The HUD <NUM> provides visual outputs to the wearer without requiring the wearer to change his or her gaze to a particular display device. While <FIG> depicts a HUD <NUM> that is separate from the glass lenses in the wearable electronic device <NUM>, alternative configurations provide one or more visual display devices that are integrated into the lenses or that project graphical outputs onto the lenses. While not depicted in further detail, the wearable electronic device <NUM> optionally includes audio output devices as well.

The electronic control unit <NUM> houses at least one wearable electronic device processor <NUM> that is operatively connected to the camera <NUM>, the positional sensors <NUM>, and the HUD <NUM>. The electronic control unit <NUM> further houses a memory <NUM> and a communications transceiver <NUM> that are operatively connected to the wearable electronic device processor <NUM>. In the embodiment of <FIG>, the memory <NUM> stores firmware instructions <NUM> that control the operation of the wearable electronic device <NUM>. The communications transceiver <NUM> includes a transmitter that enables transmission of data, including a video stream, to a corresponding communications transceiver <NUM> in the mobile electronic device <NUM>. The communications transceiver <NUM> further includes a receiver that enables the wearable electronic device <NUM> to receive data from the mobile electronic device <NUM>, and, in particular, to receive messages from the mobile electronic device <NUM> for display to a user via the HUD <NUM>. In the illustrative example of <FIG>, the communications transceiver <NUM> is a Bluetooth or Bluetooth low-energy wireless data communications transceiver, although alternative configurations may use a different wireless communication standard or may employ a wired connection interface such as Universal Serial Bus (USB).

Referring to <FIG> and to a schematic diagram in <FIG>, in one embodiment of the system <NUM> the mobile electronic device <NUM> further includes a mobile electronic device processor <NUM> that is operatively connected to a timer <NUM>, a memory <NUM>, a communication transceiver <NUM>, the optical sensor <NUM>, and one or more display and user input/output (I/O) devices <NUM>. The mobile electronic device <NUM> is operatively connected to the wearable electronic device <NUM> using the transceiver <NUM> to communicate with the corresponding transceiver <NUM> in the wearable electronic device <NUM>.

In the mobile electronic device <NUM>, the optical sensor <NUM> is, for example, a digital camera that generates still images or video of the test strip <NUM>, including the measurement site <NUM> located on the test strip reverse side <NUM>', and, optionally, the color card <NUM> to generate at least one optical measurement for analysis to measure the analyte level in the fluid dose that is applied to the test strip. While the optical sensor <NUM> and the camera <NUM> in the wearable electronic device <NUM> may be configured with similar hardware in some embodiments, in the configuration of <FIG> the camera <NUM> is configured for the generation of a video stream that captures the overall scene including the vial <NUM>, the test strip <NUM> and the deposit site <NUM>, optionally the color card <NUM>, and the finger <NUM> for identification of the time at which the test strip <NUM> receives a fluid dose. The optical sensor <NUM> is configured to generate one or more optical measurements of the measurement site <NUM> on the test strip reverse side <NUM>' at the appropriate time after application of the fluid dose to provide an input to an analyte measurement process. The optical measurement is, for example, a digital photograph that includes the measurement site <NUM> on the test strip reverse side <NUM>'. As such, the camera <NUM> provides a broader view of multiple elements employed in the analyte testing process, while the optical sensor <NUM> provides more detailed digital images or video of the reagent on the test strip <NUM> and optionally the calibration data that are provided on the color card <NUM>. In an alternative configuration, a single camera performs both the functions of the camera <NUM> in the wearable electronic device <NUM> and the optical sensor <NUM> in the mobile electronic device <NUM>. For example, a single camera may be reconfigurable to generate a video stream with a reduced resolution for identification and tracking of the vial <NUM>, the test strip <NUM>, the color card <NUM>, and the finger <NUM> to enable the mobile electronic device <NUM> to identify the time at which the test strip <NUM> receives the fluid dose. Subsequently, the single camera may operate at a higher resolution to capture one or more high-fidelity images of the test strip reverse side <NUM>' either alone or in conjunction with the color card <NUM> to provide an input for the analyte measurement process.

In the mobile electronic device <NUM>, the user input/output (I/O) devices <NUM> include a touchscreen display device that provides a graphical output to a user and receives touch inputs to control the operation of the mobile electronic device <NUM> and, more particularly, to provide input to the analyte measurement process. Other examples of I/O devices include microphones for speech input and speakers for audio output, mechanical buttons, and the like. In some configurations, the wearable electronic device <NUM> implements user I/O devices such as an audio input device or gesture tracking input device that uses the camera <NUM> to record inputs from the user that the camera <NUM> transmits to the mobile electronic device <NUM>. The wearable electronic device <NUM> may further receive output data from the mobile electronic device <NUM> for display to the user via the HUD <NUM>.

In the mobile electronic device <NUM>, the timer <NUM> enables the mobile electronic device processor <NUM> to maintain a count of elapsed time during operation, which includes counting an elapsed time starting at when the test strip <NUM> receives a fluid dose to ensure that optical measurements of the reagent for analyte measurement occur after a predetermined minimum time has elapsed and before a predetermined maximum time has elapsed. While the timer <NUM> is depicted as a discrete component for illustrative purposes, in many practical embodiments the timer <NUM> is integrated into the mobile electronic device processor <NUM> as a timer circuit or is implemented as a software timer.

In the mobile electronic device <NUM>, the memory <NUM> includes one or more non-volatile and volatile data storage devices. In the configuration of <FIG>, the memory <NUM> stores application software <NUM> and operating system software <NUM> that both contain instructions for execution by the mobile electronic device processor <NUM>. The application software <NUM> includes instructions that implement a user interface and an analyte analysis program to perform the analyte measurement process based on an image analysis of one or more optical measurements of the reagent on the test strip <NUM>. The application software <NUM> also stores predetermined minimum and maximum elapsed time thresholds to ensure that the optical measurements are generated after the fluid dose has had sufficient time to react with the reagent in the test strip <NUM> but before a maximum useful time period for measuring the analyte has elapsed. As described in further detail below, part of the analyte measurement process includes the identification of the vial <NUM>, removal of the test strip <NUM>, and the identification of contact between the finger <NUM> and the test strip <NUM> to apply the fluid dose to the deposit site <NUM>. The application software <NUM> further includes object recognition data <NUM> that enable the mobile electronic device processor <NUM> to perform automated object identification and tracking of the vial <NUM> and test strip <NUM> in the video stream that is received from the wearable electronic device <NUM>. The object recognition data <NUM> are generated through a training process that occurs prior to distribution of the application software <NUM>. In particular, the training process utilizes the predetermined shapes, colors, and patterns of registration marks formed on the vial <NUM>, test strip <NUM>, test strip reverse side <NUM>', and the color card <NUM> to enable automated identification and tracking of these components in a video stream. Examples of the object recognition data <NUM> include image classifiers such as neural networks, particularly convolutional neural networks, support vector machines, hidden Markov models, one-dimensional and two-dimensional barcode scanning engines, and the like. Additionally, the object recognition data <NUM> may include filters for color detection and edge detection along with other image processing data needed for tasks such as object detection and image segmentation to enable tracking of objects such at the vial <NUM>, test strip <NUM>, and color card <NUM> in the video stream. The operating system (OS) software <NUM> includes the software kernel, drivers, libraries, and other system software that are associated with a standard commercially-available operating system. The OS software <NUM> provides standardized services such as network and graphics stacks, file systems for data storage and management, software access to the optical sensor <NUM>, display and I/O devices <NUM>, timer <NUM>, communications transceiver <NUM>, and other components in the mobile electronic device <NUM>. In the mobile electronic device <NUM>, the communications transceiver <NUM> includes a transmitter that enables transmission of data, including command data and output message data, to the corresponding transceiver <NUM> in the wearable electronic device <NUM>. The communications transceiver <NUM> further includes a receiver that enables the mobile electronic device <NUM> to receive data from the mobile electronic device <NUM>, and, in particular, to receive a video stream from the camera <NUM> in the wearable electronic device <NUM>. In the illustrative example of <FIG>, the communications transceiver <NUM> is a Bluetooth or Bluetooth low-energy wireless data communications transceiver, although alternative configurations may use a different wireless communication standard or may employ a wired connection interface such as USB.

<FIG> further depicts a vial <NUM> and a color card <NUM>. The vial <NUM> stores one or more of the test strips <NUM>. In addition to providing storage, the vial <NUM> protects the test strips from contamination in the environment, which includes preventing the reagents in the test strips <NUM> from absorbing excessive amounts of moisture and light from the ambient environment. The vial <NUM> includes a printed label <NUM> that further bears one or more registration marks, which are depicted as the dashed line indicator registration marks <NUM> that are printed along one or more edges of the label <NUM> in <FIG>, or another suitable surface of the vial <NUM> in another embodiment. The outline shape of the vial <NUM> and the registration marks <NUM> form a simple visual indicator that enable efficient identification and tracking of the vial <NUM> in a video stream that the wearable electronic device <NUM> generates and transmits to the mobile electronic device <NUM>. The registration marks <NUM> are widely distributed on the exterior of the vial <NUM> to enable identification and tracking of the vial <NUM> from a wide range of viewing angles and when the hand of a user holds the vial <NUM>. Alternative embodiments of registration marks for the vial <NUM> include, for example, printed pattern indicators including barcodes, or engraved or embossed geometric shapes that are formed on the exterior of the vial <NUM> that assist in automated identification and tracking of the vial <NUM>. In the embodiment of <FIG>, a lid <NUM> provides access to the interior of the vial <NUM>. The lid <NUM> may be fully removable or may remain attached to the body of the vial <NUM> while opened. In either configuration, a second registration mark <NUM> is formed on the interior surface of the vial lid <NUM>. The second registration mark <NUM> is, for example, a circle or other geometric shape formed with a predetermined color that contrasts with the color of the vial <NUM> to provide a clear indication that the vial <NUM> has been opened in the video stream that the wearable electronic device <NUM> generates during use of the vial <NUM>. In alternative embodiments, the registration mark <NUM> is a one or two dimensional barcode or other registration mark that is identifiable to automated vision algorithms. The registration marks <NUM> and <NUM> enable accurate identification of both the vial <NUM> and a determination of when the vial <NUM> is closed or opened.

During operation, the optical sensor <NUM> detects the color change in the reagent that is visible to the measurement site <NUM> on the test strip reverse side <NUM>' in response to one or more chemical reactions with the analyte in the fluid dose. In the illustrative example of <FIG>, the change in color of the reagent located at the measurement site <NUM> indicates a level of glucose analyte in the blood sample. As described above, the system <NUM> identifies when the deposit site <NUM> receives the fluid dose and uses a timer to determine when an optical sensor should generate subsequent optical measurements of the measurement site <NUM> to ensure an accurate measurement of the blood glucose level. While the system <NUM> depicts a test strip that includes a separate deposit site <NUM> and measurement site <NUM> for illustrative purposes, those of skill in the art will recognize that alternative test strips provide a single deposit site and measurement site with the reagent that are co-located on the same region of the test strip. As such, in some embodiments the deposit site and reagent occupy separate locations on the test strip while in other embodiments the deposit site and the reagent refer to a single location of the test strip. In <FIG>, the color card <NUM> is an optional component that has a rear side depicted in <FIG> that holds the test strip <NUM> in place prior to dosing. The color card <NUM> also has a front side (not shown) that includes a predetermined arrangement of colors and other fiducial markings that assist in calibrating images from the optical sensor <NUM> for accurate color measurement of the exposed measurement site <NUM>. An aperture <NUM> in the color card <NUM> enables the measurement site <NUM> on the test strip reverse side <NUM>' to be measured by the optical sensor within the color pattern of the color card <NUM>. The rear side of the color card <NUM> includes registration marks <NUM> and <NUM>, which are depicted as arrow indicators that are printed on the rear side of the color card <NUM> in the illustrative embodiment of <FIG>. The registration marks <NUM> and <NUM> are associated with the test strip <NUM> and further assist in identifying and tracking the test strip <NUM> in the video stream to detect when the test strip <NUM> receives the fluid dose. The color card <NUM> is optional, and the system <NUM> is configured to generate optical measurements of the reagent at the measurement site <NUM> in the test strip reverse side <NUM>' to measure the glucose analyte in a blood sample using the test strip reverse side <NUM>' in isolation or in conjunction with the color card <NUM>.

<FIG> depicts a process <NUM> for operation of the system <NUM> to perform an analyte testing operation with automated detection of when a fluid dose is applied to the deposit site of a test strip and automated timing of when the system <NUM> should generate one or more optical measurements of the measurement site to measure a level of analyte in the fluid dose. In the description of the process <NUM>, a reference to the process performing a function or action refers to the operation of one or more digital processors, such as the processors in the wearable electronic device <NUM> and the mobile electronic device <NUM>, to execute stored program instructions to perform the function or action in conjunction with other components in the system <NUM>.

The process <NUM> begins with activation of a camera, such as the camera <NUM> in the wearable electronic device <NUM>, to generate a video stream of a scene in front of a user at the beginning of an analyte testing process (block <NUM>). In the system <NUM>, the user begins execution of the application software <NUM>, and the mobile electronic device <NUM> transmits a command to the wearable electronic device <NUM> to activate the camera <NUM>. In the embodiment of <FIG>, the wearable electronic device processor <NUM> activates the camera <NUM> and uses the transceiver <NUM> to transmit a video stream from the camera <NUM> to the mobile electronic device <NUM>, which enables the mobile electronic device processor <NUM> to receive the video stream for further processing using the corresponding transceiver <NUM>. As is generally known in the art, the video stream includes a series of frames of image data that depict the view from the camera <NUM> over time during the analyte testing process.

The process <NUM> continues as the mobile electronic device processor <NUM> identifies the vial <NUM> in the video stream generated by the wearable electronic device <NUM> (block <NUM>). While numerous digital image processing techniques may be used to identify an object, such as the vial <NUM> or other objects that are detected in the video stream during the process <NUM>, a nonlimiting example of the preferred technique is described in further detail herein. The identification process for the vial <NUM> further includes an object tracking operation that segments different portions of frames in the video stream that contain objects and an object identification operation that uses an image classifier to identify the tracked objects.

In the object tracking operation, the mobile electronic device processor <NUM> in the mobile electronic device <NUM> identifies and tracks one or more objects that are depicted in the video stream. To track objects, the mobile electronic device processor <NUM> performs a contour detection operation that identifies the boundaries of various objects in the video stream that have similar image intensity values, including the boundaries of the vial <NUM>. In particular, each frame of the video stream is formed as a two-dimensional array of pixels, and the mobile electronic device processor <NUM> identifies contours based on contiguous regions of pixels with the same or similar numeric pixel values in either color data (e.g. red/green/blue) or monochrome image data (e.g. gray scale values). In some configurations, the mobile electronic device processor <NUM> performs image pre-processing operations such as converting a color video stream to grayscale, thresholding of the grayscale pixels, and performing an edge detection processing to improve the accuracy of the contour detection process. The mobile electronic device processor <NUM> segments the original image using, for example, rectangular bounding boxes that surround the detected contour areas, and the mobile electronic device processor <NUM> performs the contour detection process over a series of video frames to track the movement of the object, such as when a user moves the vial <NUM>. For example, as depicted in view <NUM> of <FIG>, the video stream depicts the vial <NUM> and the mobile electronic device processor <NUM> generates a rectangular bounding box <NUM> segment in a frame of the video stream that contains the detected contour of vial <NUM>. While view <NUM> depicts the vial <NUM> in isolation, some frames in the video stream contain more than one object and the contour detection process described above enables tracking of multiple objects in the video stream.

Upon completion of the tracking operation, the mobile electronic device processor <NUM> has access to one or more image segments that contain objects, but has not yet determined the identity of specific objects. For example, the mobile electronic device processor <NUM> has tracked an object in the image segment <NUM> but has not yet identified that the object is the vial <NUM> or some other object. The object tracking process produces multiple image segments that can improve the accuracy of the image classifier for the detection of multiple relevant objects that may occupy different portions of a frame in the video stream. To complete the object identification process, the mobile electronic device processor <NUM> provides the segmented portion of the image containing the tracked object as an input to a trained image classifier that is stored with the object recognition data <NUM> in the memory <NUM>. The image classifier is, for example, a trained convolutional neural network (CNN) or other suitable image classifier that is trained to identify a predetermined set of objects, such as the vial <NUM> and interior of the vial lid <NUM>, either side of the test strip <NUM>/<NUM>', the color card <NUM>, or a finger <NUM>. The training process for the image classifier occurs prior to the process <NUM> and uses a set of training images that include multiple examples of the objects to be identified in various expected situations that would occur during an analyte testing process. The image classifier is trained using, for example, a gradient descent training process that is otherwise known to the art. The image classifier is trained to recognize, either expressly or implicitly, some or all of the outline shape of the vial <NUM>, registration mark features <NUM> that are formed on the vial <NUM>, the interior of the vial lid <NUM>, on either side of the test strip <NUM>/<NUM>', and on the color card <NUM> to improve the accuracy of identifying the predetermined objects. Additionally, the training process can include training examples that occur when the registration marks are only partially visible to the camera <NUM>, such as when a user holds the vial <NUM> in hand, which may occlude some of the registration marks <NUM>. The mobile electronic device processor <NUM> optionally performs additional pre-processing of the image data, which may include resizing the image data to a predetermined resolution, or performing a rotational transformation of the image based on metadata that are received from the positional sensors <NUM> in the wearable electronic device <NUM> that identifies the angular orientation of the camera <NUM> at the time each frame of the video stream is generated to improve the accuracy of the image classifier. In some configurations, the image classifier is trained using monochrome image data, but in other configurations, a color image is preferred, including configurations in which registration marks are formed using predetermined colors that assist in image classification to identify an object. The classifier also rejects extraneous objects that may be present in the video stream as non-relevant. Additionally, because the video stream includes a series of frames, the mobile electronic device <NUM> can recognize the vial <NUM> in one or more frames of the video stream even if the tracking and identification process is not successful in a portion of the video stream frames. One example of a software framework that enables the image processing operations described above in the application software <NUM> is the Open Computer Vision (OpenCV) project that is available at https://opencv. The process described above for identification of the vial <NUM> is substantially the same as the processes described below for the identification of other objects in the video stream during the process <NUM>.

During the vial identification process, the mobile electronic device <NUM> optionally transmits a graphic, such as an icon or animation, to the wearable electronic device <NUM> to assist the user in identifying the next step in the process for performing the test analysis. For example, the mobile electronic device <NUM> transmits a graphical icon that corresponds to the shape of the vial <NUM> to the wearable electronic device <NUM>, and the wearable electronic device processor <NUM> generates a graphical display of the icon using the HUD <NUM> to alert the user to retrieve the vial <NUM> and place it in view of the camera <NUM> until successful identification of the vial <NUM> in the video stream. In <FIG>, the view <NUM> depicts the icon <NUM> that the HUD <NUM> superimposes over the scene recorded by the camera <NUM> to prompt the user to retrieve the vial <NUM>.

Referring again to <FIG>, the process <NUM> continues as the mobile electronic device processor <NUM> identifies that the vial <NUM> is opened in response to detecting the registration mark formed <NUM> that is formed on the interior of the lid <NUM> in the vial <NUM> (block <NUM>). Referring to <FIG>, view <NUM> depicts the lid <NUM> removed from the vial <NUM> with the registration mark <NUM> visible on the interior of the lid <NUM>. The mobile electronic device processor <NUM> tracks and identifies the lid <NUM> within the image segment <NUM> in the same manner that is described above regarding the vial <NUM>. Additionally, the mobile electronic device <NUM> optionally transmits an icon of the lid to the wearable electronic device <NUM>, and the wearable electronic device <NUM> displays the icon <NUM> in the HUD <NUM> to provide guidance to the user.

Referring again to <FIG> and <FIG>, the process <NUM> continues as the mobile electronic device processor <NUM> identifies that the test strip <NUM> has been removed from the opened vial <NUM> (block <NUM>). The mobile electronic device processor <NUM> tracks and identifies the test strip <NUM> within the image segment <NUM> in the same manner that is described above regarding the vial <NUM> and lid <NUM> as depicted in view <NUM>. Additionally, the mobile electronic device <NUM> optionally transmits an icon of the test strip to the wearable electronic device <NUM>, and the wearable electronic device <NUM> displays the icon <NUM> in the HUD <NUM> to provide guidance to the user. As depicted in <FIG>, in some instances the user removes the test strip with the reverse side <NUM>' visible to the camera <NUM>. The mobile electronic device processor <NUM> tracks and identifies the test strip reverse side <NUM>' in the region <NUM> and optionally generates an output message for the user via the HUD <NUM> or another output device <NUM> to rotate the test strip so that the side of the test strip <NUM> with the registration marks <NUM> and the deposit site <NUM> is visible in the video stream. The absence of the registration marks <NUM> on the test strip reverse side <NUM>' and optionally other different features of the reverse side <NUM>' provide sufficient differences for the image classifier to distinguish the sides <NUM>/<NUM>' of the test strip. This operation is performed at subsequent stages of the process <NUM> if the test strip <NUM> is flipped to expose the reverse side <NUM>' prior to the test strip <NUM> receiving the fluid dose. While view <NUM> depicts the vial <NUM>, interior of the lid <NUM>, and the test strip <NUM> concurrently for illustrative purposes, the detection of removal of the test strip <NUM> from the vial <NUM> only requires a sequence of detecting the vial <NUM>, lid interior <NUM>, and the test strip <NUM> within the same video stream within a comparatively short period of time, such as a <NUM> second, <NUM> second, or <NUM> second time window. As such, the vial <NUM>, lid <NUM>, and test strip <NUM> do not need to be identified simultaneously in the video stream for the process <NUM> to identify that the test strip <NUM> has been removed from the vial <NUM>.

During the process <NUM>, if the mobile electronic device processor <NUM> fails to identify the sequence of the vial <NUM>, the interior of the lid <NUM> indicating that the vial <NUM> is opened, or the removal of the test strip <NUM> within a predetermined period of time (block <NUM>) then the process <NUM> returns to the processing described above with reference to block <NUM> to enable the user to repeat the process. Upon successfully identification that the test strip <NUM> has been removed from the vial <NUM> (block <NUM>) the process <NUM> continues as the mobile electronic device processor <NUM> continues to track the test strip <NUM> that has been identified in the video stream (block <NUM>). In one configuration, the mobile electronic device processor <NUM> tracks the test strip <NUM> in isolation as depicted in view <NUM> of <FIG>. In another configuration that utilizes the color card <NUM>, the mobile electronic device processor <NUM> identifies the color card <NUM> based at least in part on the registration mark arrows <NUM> and <NUM> and tracks the insertion of the test strip <NUM> into the color card <NUM> as depicted in view <NUM> of <FIG>. After identification of the test strip <NUM> and prior to the test strip <NUM> receiving the fluid dose, the mobile electronic device processor <NUM> stores at least one image of the test strip, including the deposit site <NUM>, in the memory <NUM>. As described in further detail below, at least one optical property of the deposit site <NUM> changes after the deposit site <NUM> receives the fluid dose, and the change in optical property enables detection of the time at which the test strip <NUM> receives the fluid dose.

The process <NUM> continues as the mobile electronic device processor <NUM> identifies the application of the fluid dose to the deposit site <NUM> on the test strip <NUM> based on the video stream, and starts the timer <NUM> upon identification of the fluid dose application (block <NUM>). In one configuration, the mobile electronic device processor <NUM> identifies the finger <NUM> of the user in the video stream using the same procedure described above for identification of the vial <NUM>, lid <NUM>, test strip <NUM>, and color card <NUM>. The mobile electronic device processor <NUM> further identifies contact between the finger and the deposit site <NUM> in the video stream. For example, contact is identified in response to the finger <NUM> occluding the test strip <NUM> in the video stream as depicted in view <NUM> of <FIG> and view <NUM> of <FIG>. After identification of the contact, the mobile electronic device processor <NUM> identifies that the fluid dose has been applied to the deposit side <NUM> based on a change in at least one optical property of the deposit site <NUM> in the video stream relative to the previously recorded images of the deposit site <NUM> as depicted in view <NUM> of <FIG> and view <NUM> of <FIG>. Examples of optical properties of the deposit site <NUM> that change after the deposit site <NUM> receives the fluid dose include changes in one or more of color, contrast, and brightness of the deposit site <NUM> that occur due to the application of the fluid dose. In another configuration, the mobile electronic device processor <NUM> omits the identification of the finger <NUM> and contact between the finger <NUM> and the test strip <NUM> in the video stream. In this simplified configuration, the mobile electronic device processor <NUM> continues to track the test strip <NUM> until the detection of the change in the at least one optical property of the deposit site <NUM> to identify that the test strip <NUM> has received the fluid dose. In yet another configuration, the mobile electronic device processor <NUM> omits the identification of a change in the optical characteristic of the deposit site <NUM> and instead detects the dosing of the test strip <NUM> based on detection of contact between the finger <NUM> and the test strip <NUM> in the video stream. In this simplified configuration, the mobile electronic device processor <NUM> identifies contact based on the finger <NUM> occluding the test strip <NUM> in the video stream, such as in views <NUM> and <NUM>, or on close proximity of the finger <NUM> and test strip <NUM>. This configuration enables detection of test strip dosing in situations where poor ambient light conditions make detection of changes in the optical characteristic at the dosing site <NUM> difficult to detect. In all three configurations, the mobile electronic device processor <NUM> activates the timer <NUM> upon detection of the application of the fluid dose to the deposit site <NUM> on the test strip <NUM>.

The process <NUM> continues as the timer <NUM> reaches a predetermined minimum time and the mobile electronic device processor <NUM> optionally generates an output signal to the user that the optical sensor <NUM> should be used to generate one or more optical measurements of the measurement site <NUM> on the test strip reverse side <NUM>' (block <NUM>). The mobile electronic device processor <NUM> generates an output on a display touchscreen <NUM> of the mobile electronic device <NUM> or via the HUD <NUM> of the wearable electronic device <NUM> to indicate that the optical measurement of the test strip should proceed as the optical sensor <NUM> generates one or more optical measurements of the measurement site <NUM> (block <NUM>). The mobile electronic device processor <NUM> optionally generates an output that includes a countdown timer to indicate the amount of time remaining in the predetermined time window after the expiration of the minimum time period and prior to the expiration of the predetermined maximum time period to generate the optical measurements. In one configuration, the mobile electronic device processor <NUM> activates the optical sensor <NUM> only after the timer <NUM> indicates that the predetermined minimum time period has expired, while in another configuration the mobile electronic device processor <NUM> only accepts optical measurements from the optical sensor <NUM> that have a timestamp that falls within the predetermined time window. While the precise predetermined minimum and maximum time periods for generation of the optical measurements may vary between embodiments, in one configuration the minimum time period following the fluid dose is <NUM> seconds and the maximum time period is <NUM> seconds. This provides a <NUM> second time window for the optical sensor <NUM> to produce one or more optical measurements of the measurement site <NUM>.

After the timer <NUM> reaches the expiration of the predetermined maximum time period, the mobile electronic device <NUM> generates an output to indicate that the maximum time period has expired (block <NUM>). If a sufficient number of optical measurements have been generated prior to the expiration of the predetermined maximum time period (block <NUM>), then the mobile electronic device processor <NUM> continues with the analyte measurement process based on the optical measurements (block <NUM>). In another configuration, if the optical sensor <NUM> generates a sufficient number of optical measurements prior to the expiration of the predetermined maximum time period, then the mobile electronic device processor <NUM> optionally commences the measurement process of block <NUM> without waiting for the expiration of the timer <NUM>. While not described in further detail herein, the analyte measurement process analyzes the color and optionally other optical properties of the reagent at the measurement site <NUM> on the test strip to determine the level of analyte in the fluid sample, such as the level of glucose in a blood sample. In configurations that use the color card <NUM>, the mobile electronic device processor <NUM> uses additional optical data from the color card <NUM> to assist with the analyte measurement process. The mobile electronic device <NUM> displays the measurement of the analyte level to the user via the display device <NUM>, the HUD <NUM> in the wearable electronic device <NUM>, or via another output device. The system <NUM> and process <NUM> increase the reliability of the analyte measurement process because all of the optical measurements are generated during the predetermined time window to ensure that the reagents in the test strip <NUM> have sufficient time to complete chemical reactions prior to the generation of the optical measurements but also do not experience dehydration or bleaching before the completion of the optical measurement process.

During the process <NUM>, if the predetermined maximum time period expires prior to the generation of a sufficient number of optical measurements (block <NUM>), then the mobile electronic device processor <NUM> does not continue with the analyte measurement process and the mobile electronic device <NUM> generates an output message indicating that the analyte measurement cannot be completed and instructing the user begin the analyte testing process again using a new test strip via the display device <NUM>, the HUD <NUM> in the wearable electronic device <NUM>, or via another output device (block <NUM>).

As described above, process <NUM> performs object identification that begins with identification of the test strip vial <NUM> and the opening of the lid <NUM> in the video stream, which enables the system <NUM> to verify that the test strip <NUM> was extracted from the vial <NUM> instead of being a loose test strip that may have been outside of the vial <NUM> for a prolonged period of time. Some test strips may be contaminated if left outside of a vial for a prolonged period of time. However, in a simplified configuration of the process <NUM>, the system <NUM> omits the identification of the test strip vial <NUM>, the lid <NUM>, and the extraction of the test strip <NUM> from the vial <NUM>. The simplified configuration begins with the generation of the video stream and the tracking and identification of the test strip <NUM> in the same manner that is described above. In this configuration, the process <NUM> does not verify that the test strip <NUM> was extracted from a vial, which may not be necessary for some analyte testing systems. This simplified configuration of the process <NUM> is otherwise identical to the process described above.

While the embodiments disclosed herein use a separate wearable electronic device <NUM> and mobile electronic device <NUM> for illustrative purposes, those of skill in the art will recognize that a single electronic device could be configured to perform the operations described herein. In particular, while state of the art wearable electronic devices typically interface with a mobile electronic device for complex operations, more capable wearable electronic devices could implement all of the functions described herein. Alternatively, the mobile electronic device <NUM> could be configured to perform all of the functions described herein using the optical sensor <NUM> as a camera to generate the video stream and perform the other processing that is described above. As such, specific references to the operations of a processor refer to the wearable electronic device processor <NUM> and the mobile electronic device processor <NUM> in the description above both individually, in combination, and, alternatively, to the operation of a single processor in configurations that use a single electronic device.

Claim 1:
A method for measuring an analyte comprising:
- identifying, with a processor (<NUM>), test strip (<NUM>) in a video stream generated by a camera (<NUM>) based on at least one registration mark (<NUM>) associated with the test strip depicted in the video stream;
- identifying, with the processor, application of a fluid dose to a deposit site (<NUM>) formed on the test strip based on the video stream;
- activating, with the processor, a timer (<NUM>) in response to the identification of the application of the fluid dose;
- generating, with an optical sensor (<NUM>), at least one optical measurement of a reagent located at a measurement site (<NUM>) on the test strip; and generating, with the processor, a measurement of an analyte in the fluid dose based on the at least one optical measurement of the reagent only in response to the at least one optical measurement being generated after a predetermined minimum time period has elapsed subsequent to the activating of the timer and prior to a predetermined maximum time period elapsing subsequent to the activating of the timer, characterized in that the method further comprises:
- identifying, with the processor, a vial (<NUM>) in a video stream generated by a camera based on at least one of an outline shape of the test vial or at least one registration mark (<NUM>) located on the vial depicted in the video stream;
- identifying, with the processor, an opening of the vial in the video stream based on at least one registration mark (<NUM>) located on a lid (<NUM>) of the vial; and
- identifying, with a processor, extraction of the test strip from the vial after the identifying of the opening of the vial in the video stream based on the at least one registration mark associated with the test strip depicted in the video stream.