Device and method for data capture aiming assistance

A data capture device includes: a display, a primary image sensor having a primary field of view centered on a primary optical axis; an auxiliary image sensor having an auxiliary field of view centered on an auxiliary optical axis, wherein the auxiliary field of view is larger than the primary field of view; a memory storing offset data defining an offset between the primary field of view and the auxiliary field of view; a data capture controller connected to the primary image sensor, the auxiliary image sensor and the memory; wherein the data capture controller is configured to: responsive to activation of an aiming mode, control the auxiliary image sensor to capture a video stream; select, according to the offset data, a portion of the video stream corresponding to the primary field of view; and present the selected portion of the video stream on the display.

BACKGROUND

Data capture devices such as handheld computers may be employed for data capture operations (e.g. barcode scanning) under a variety of conditions. For example, such devices can be deployed to perform barcode scanning at various ranges, under various lighting conditions, and in connection with a variety of objects bearing the codes to be scanned. Such devices may have mechanisms to aid an operator in aiming the device, such as a laser emitter to project a dot on the surface to be scanned. Under certain conditions, however, such mechanisms may fail (e.g. the laser dot mentioned above may not be visible beyond certain distances, or under certain lighting conditions), leading to reduced scan accuracy.

DETAILED DESCRIPTION

Examples disclosed herein are directed to a data capture device, comprising: a housing; a display supported by the housing; a primary image sensor supported by the housing and having a primary field of view centered on a primary optical axis; an auxiliary image sensor supported by the housing and having an auxiliary field of view centered on an auxiliary optical axis, wherein the auxiliary field of view is larger than the primary field of view; a memory storing offset data defining an offset between the primary field of view and the auxiliary field of view; a data capture controller connected to the primary image sensor, the auxiliary image sensor and the memory; wherein the data capture controller is configured to: responsive to activation of an aiming mode, control the auxiliary image sensor to capture a video stream; select, according to the offset data, a portion of the video stream corresponding to the primary field of view; and present the selected portion of the video stream on the display.

Additional examples disclosed herein are directed to a method in a data capture device having (i) a display, (ii) a primary image sensor having a primary field of view centered on a primary optical axis, and (iii) an auxiliary image sensor having an auxiliary field of view centered on an auxiliary optical axis, wherein the auxiliary field of view is larger than the primary field of view, the method comprising: storing offset data in a memory of the data capture device defining an offset between the primary field of view and the auxiliary field of view; at a data capture controller of the data capture device connected to the primary image sensor, the auxiliary image sensor and the memory: responsive to activation of an aiming mode, controlling the auxiliary image sensor to capture a video stream; selecting, according to the offset data, a portion of the video stream corresponding to the primary field of view; and presenting the selected portion of the video stream on the display.

FIGS. 1A and 1Bdepict a data capture device100that may be deployed in a wide variety of environments, including transport and logistics facilities (e.g. warehouses), healthcare facilities, and the like. The data capture device100in the example illustrated inFIG. 1. is a handheld data capture device including a housing defined by a body104and a handle108. The housing supports various other components of the device100, as will be discussed below in greater detail.

As shown inFIG. 1A, which illustrates a front view of the device100, the body104of the housing supports a display112, which may include an integrated touch screen. The body104also supports various inputs, such as a microphone116and a button120. As shown inFIG. 1B, which illustrates a rear perspective view of the device100, the handle108supports additional inputs, including a primary trigger button124and an auxiliary trigger button128. The body104and/or the handle108can support additional inputs in other examples, or the above-mentioned inputs can be omitted in other examples. The body104can also support outputs, such as a speaker130(an additional speaker may be provided on the opposite side of the body104than the side shown inFIG. 1B).

The body104also supports, as shown inFIG. 1B, a primary image sensor132, also referred to herein as an imager132. The imager132has a primary optical axis134extending away from the imager132, on which a primary field of view (FOV) of the imager132is centered. The body104further supports an auxiliary image sensor136, also referred to herein as a camera136. The camera136has an auxiliary optical axis138extending away from the camera136, on which an auxiliary FOV of the camera136is centered. The imager132and the camera136are substantially coplanar (i.e. the imager132and the camera136are located in a common image sensor plane), although as shown inFIG. 1Bthe imager132and the camera136are at different locations within that plane.

The primary image sensor132enables the device100to perform data capture operations such as barcode scanning. In particular, the primary image sensor132is configured to capture one or more images responsive to activation of a primary input (e.g. the primary trigger124), and to detect and decode a machine-readable indicium in such images. A wide variety of indicia can be detected and decoded following capture by the imager132, including 1D and 2D barcodes.

As will be discussed in greater detail below, the imager132has an FOV that is smaller (i.e. narrower) than the FOV of the camera136. As will be apparent to those skilled in the art, to capture images of an indicium on an object, the device100must be oriented (i.e. aimed, by an operator of the device100) such that the indicium falls within the FOV of the imager132. To assist in aiming the device100to capture the indicium, the device100can also include an emitter140such as a laser diode, configured to emit a laser beam coinciding with the primary optical axis134to project a visible dot on an object at which the imager132is aimed. However, under some conditions the above-mentioned dot may not be visible. The device100therefore implements additional functionality to assist in aiming the imager132by using the camera136to simulate the current FOV of the imager132on the display112.

Before discussing the aiming assist functions implemented by the device100, certain internal components of the device100are described in further detail, with reference toFIG. 2.

As shown inFIG. 2, the device100includes a central processing unit (CPU), also referred to as a processor200, interconnected with a non-transitory computer readable storage medium, such as a memory204. The memory204includes a suitable combination of volatile memory (e.g. Random Access Memory (RAM)) and non-volatile memory (e.g. read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash). The processor200and the memory204each comprise one or more integrated circuits (ICs).

The components of the device100shown inFIGS. 1A and 1B(that is, the display112, microphone116, button120, triggers124and128, image sensors132and136, and the emitter140) are interconnected with the processor200via one or more communication buses. The components of the device100are powered by a battery or other power source, over the communication buses or by distinct power buses.

The memory204stores a plurality of applications, each including a plurality of computer readable instructions executable by the processor200. The execution of the above-mentioned instructions by the processor200causes the device100to implement certain functionality, as discussed herein. The applications are therefore said to be configured to perform that functionality in the discussion below. In the present example, the memory204of the device100stores a data capture application216, also referred to herein as the application216. The device100is configured, via execution of the application216by the processor200, to perform data capture operations and implement an aiming mode for such data capture operations under certain conditions. The memory204also stores a repository220containing offset data for use in implementing the aiming mode mentioned above. The contents of the offset data in the repository220, as well as mechanisms for using and updating the offset data, will be discussed in greater detail below.

The processor200, as configured via the execution of the application216, may also be referred to as a data capture controller. In some embodiments, the functionality described herein is implemented by two or more controllers, rather than by the processor200exclusively. For example, in some embodiments the device100includes a scanning controller configured to control the imager132and the emitter140, and to decode data from images captured by the imager132. The scanning controller passes decoded data to the processor200for subsequent processing. The processor200, meanwhile, controls the camera136, the display112and the inputs mentioned above. In such embodiments, the data capture controller is therefore implemented by such a scanning controller and the processor200together.

In further embodiments, the processor and/or the above-mentioned scanning controller are implemented as one or more specifically-configured hardware elements, such as field-programmable gate arrays (FPGAs) and/or application-specific integrated circuits (ASICs).

Turning now toFIG. 3, the operation of the device100will be described in further detail.FIG. 3illustrates a flowchart of a method300of data capture aiming assistance. The performance of the method300will be described in conjunction with its performance by the device100.

At block305, the device100is configured to store the above-mentioned offset data, e.g. in the repository220. The offset data defines an offset between the primary field of view of the imager132and the auxiliary field of view of the camera136. More specifically, as will be described with reference toFIGS. 4A and 4B, the offset data defines an offset between the center of the primary field of view and the center of the auxiliary field of view. In effect, therefore, the offset data also defines an offset between the primary and auxiliary optical axes.

Turning toFIG. 4A, a simplified illustration of the device100is shown, illustrating the relative positions and sizes of a primary field of view400of the imager132and an auxiliary field of view404of the camera136. As is shown inFIG. 4A, and as noted earlier, the primary FOV400is smaller than the auxiliary FOV404. In addition, due to the distinct physical positions of the imager132and the camera136on the device100(shown inFIG. 1B), the FOV400is not centered within the FOV404(that is, the optical axes134and138are not coincident, though they may be parallel to one another). Instead, the center of the FOV400is offset from the center of the FOV404by an offset vector408illustrated inFIG. 4A. The offset data stored at block305defines the offset vector408. As will be apparent, the FOV404of the camera136can be represented as a pixel array having an origin412, in which each position within the FOV404has an X pixel coordinate and a Y pixel coordinate. The offset data may include horizontal and vertical pixel distances defined within the frame of reference defined by the origin412and the X and Y axes indicated inFIG. 4A.

The offset data can include additional parameters beyond the distances noted above. In some examples, the offset data also includes dimensions of the FOV400, expressed according to the frame of reference mentioned above (i.e. in pixel dimensions) and shown inFIG. 4Bas a region416. The region416has the same size as the FOV400shown inFIG. 4A. The offset data can also include, in some examples, dimensions of an intermediate FOV, larger than the FOV400and smaller than the FOV404, indicated inFIG. 4Bas a region420. That is, the offset data as shown graphically inFIG. 4Bcan include three pairs of values: the distances defining the vector408, dimensions defining the region416, and dimensions defining the region420.

In some embodiments, the offset data includes the above-mentioned data (i.e. at least the parameters defining the vector408, and optionally parameters defining one or both of the regions416and420) for each of a plurality of ranges. As will now be apparent, the position of the FOV400within the FOV404changes according to distance from the device100. Turning toFIG. 5, a side view of the device100is shown, along with the FOVs400and404at various distances. In particular, at a first distance (i.e. a first range)500from the imager132and the camera136, a first primary FOV400aand auxiliary FOV404aare illustrated. Further, at a second range504, a second primary FOV400band auxiliary FOV404bare illustrated. As is evident from the FOVs400a,400band404a,404b, the offset between the FOVs400and404varies with range. The repository220may therefore contain subsets508a,508bof offset data, with each subset508corresponding to a distinct range or subset of ranges. The repository220can be stored, for example, as a lookup table in the memory204, with a plurality of entries each corresponding to a given range or subset of ranges.

Returning toFIG. 3, the offset data stored at block305can be obtained for storage in a variety of ways. For example, predetermined offset data can be loaded into the memory204during the manufacturing of the device100, according to specified relative positions of the imager132and the camera136. In other examples, each device100can be calibrated at the manufacturing stage by capturing images of a predetermined object (e.g. a binary-coded image) with both the imager132and the camera136, and registering the captured images to determine the position of the primary FOV within the auxiliary FOV. As will be discussed in greater detail below, the device100can be configured to update the offset data (e.g. to recalibrate) under certain conditions.

At block310, the device100determines whether an aiming mode has been activated. It is assumed that prior to the performance of block310, the device100has entered a data capture mode. In the data capture mode, the processor200awaits the activation of a primary input, such as the primary trigger124. In response to activation of the primary trigger124, the processor200controls the imager132to capture an image of its current FOV, and to detect and decode an indicium in the captured image. When in the data capture mode, the emitter140can be controlled to emit a beam to project an aiming dot onto any objects within the primary FOV400.

Activation of the aiming mode while the data capture mode is active can be initiated by activation of an auxiliary input, such as the auxiliary trigger128. Various other inputs can be employed to activate the aiming mode, however. For example, the auxiliary input can be the microphone116, and an audible command issued by the operator of the device100can be captured by the microphone116and detected by the processor200.

When the aiming mode has not been activated, the device100continues to operate in the data capture mode (without aiming assistance), proceeding to block325as will be discussed below. When the aiming mode has been activated, however (i.e. when the determination at block310is affirmative), the device100proceeds to block315.

At block315, the processor200activates the auxiliary image sensor136, which otherwise remains inactive during the data capture mode (as the primary image sensor132is employed to capture and detect indicia such as barcodes). Activation of the camera136causes the camera136to capture a video stream, which is provided to the processor200.

At block320, the processor200selects a portion of the above-mentioned video stream and controls the display112to present the selected portion, substantially in real-time (i.e. substantially simultaneously with the capture of the video stream). That is, the processor200is configured to select a portion of each frame of the video stream captured by the camera136, and to present the selected portions in sequence on the display.

The portion of the video stream selected for presentation on the display112is selected according to the above-mentioned offset data. In general, the selected portion of the video stream captured by the camera136provides a virtual viewport for the imager132, presenting to the operator of the device100a current representation of the primary FOV of the imager132. However, the viewport is referred to as virtual because it is obtained not via the imager132itself, but via the camera136.

Turning toFIG. 6A, a set of objects, including an object600(e.g. a roll of steel in a warehouse) bearing an indicium604such as a barcode, is shown.FIG. 6Aalso illustrates the primary FOV400of the imager132and the auxiliary FOV404of the camera136following activation of the aiming mode at block315. As seen inFIG. 6A, the indicium604is centered within the auxiliary FOV404, but is not within the primary FOV400. Therefore, if a scan operation were to be initiated, the indicium604would not be detected. By applying the offset data discussed in connection withFIGS. 4A, 4B and 5, the processor200is configured to select a portion608of the FOV404, illustrated inFIG. 6B. When, as discussed in connection withFIG. 5, the offset data includes a plurality of subsets of offset data corresponding to respective imaging ranges, the processor200receives, e.g. from the imager132or the camera136, a detected range indicating the distance from the device100to the objects within the primary FOV400(if the range is received from the imager132) or the auxiliary FOV404(if the range is received from the camera136). The processor200selects a subset of offset data that corresponds to the received range.

The position and size of the portion608is defined by the region420shown inFIG. 4B. As seen inFIG. 6B, the portion608encompasses both the indicium604and the primary FOV400. The processor200also presents on the display112a bounding box612that indicates the size of the primary FOV40.

Stated another way, at block320the processor200controls the camera136to capture an image (e.g. a frame in the above-mentioned video stream), depicting the auxiliary FOV404as shown inFIG. 6A. The processor200then selects the portion608of the captured image having dimensions as specified by the region420shown inFIG. 4B, and centered at a point in the captured image that is offset from the center of the FOV404by the offset vector408. The processor200further generates the bounding box612having the dimensions of the region416as shown inFIG. 4B, and centered at the same point as the portion608.

At block325, the processor200determines whether a scan operation has been initiated, e.g. via activation of the primary trigger124. When the determination is negative, the performance of the method300returns to block310, and the processor200determines whether to continue the generation of the virtual viewport as discussed above via blocks315and320. Turning toFIGS. 7A and 7B, the device100has been reoriented to shift the auxiliary FOV404and the primary FOV400, with the aiming mode remaining activated. The display112is therefore updated by the processor to present a portion708of the image captured by the camera136, along with a bounding box712indicating that the indicium604now falls within the primary FOV400of the imager132.

Returning again toFIG. 3, at a subsequent performance of block325it is assumed that the primary trigger124is activated, and the determination at block325is therefore affirmative. The processor200is configured to proceed to block330, at which the imager132is controlled to capture one or more images. The processor200is configured to detect and decode, from the image captured by the imager132, an indicium (e.g. the indicium604shown inFIGS. 6A-6B and 7A-7B). Various suitable mechanisms for detecting and decoding indicia may be implemented. Such mechanisms are not the subject of the present discussion, and will therefore not be described in detail herein. The processor200obtains, as a result of the performance of block330, at least decoded data (e.g. a string of text or the like) encoded in the indicium604. The processor200may also obtain a location of the indicium604within the primary FOV400(e.g. expressed as pixel coordinates), as well as a decode time elapsed between activation of the primary trigger124and completion of the decoding. The decode time may also be referred to as a “trigger-to-beep” time.

The device100, therefore is enabled to provide a virtual viewport for the imager132via control of the camera136and use of the offset data. In the embodiment illustrated inFIGS. 6A-7B, for example, the virtual viewport provides a representation of the current primary FOV400of the imager132centered on the display112, and also provides additional image data around such a representation, to assist the operator of the device100in determining whether or how to adjust aiming of the device100.

The offset data, as will now be apparent to those skilled in the art, describes the relative physical positions and orientations of the imager132and the camera136. As will now be apparent to those skilled in the art, the relative physical positions and orientations of the imager132and the camera136may change over time for a given device100. For example, dropping a device100may lead to minor physical shifts in components. Further, the predefined offset data provided at manufacturing may not account for deviations from specified component positions. The device100is therefore also enabled, in some embodiments, to update the offset data, as will be discussed below in connection with the remainder of the method300. In other embodiments, the updating of offset data may be omitted, and the performance of the method300may therefore conclude after block330. The memory204, in further embodiments, stores a configurable setting defining whether or not the self-calibration routines discussed below are enabled. The setting can be altered via input data received at the inputs mentioned above, via the touch screen integrated with the display112, or the like.

At block335, the processor200is configured to obtain offset update data. The offset update data, as will be discussed below, includes one or more attributes that can be processed to determine whether to alter the offset data as currently stored in the repository220. Various examples of offset update data are contemplated. In some examples, the offset update data includes a location, within the auxiliary FOV404(i.e. within a frame of the video stream captured by the camera136), of the dot projected by the emitter140. That is, the processor200is configured to detect, in the video stream, the projected dot and when the dot is detected, to determine the location (e.g. in pixel coordinates relative to the origin412) of the dot.

The processor200is configured to detect the dot based on any suitable image attributes. For example, the processor200can detect a region of the image having a predefined color corresponding to the color of the beam emitted by the emitter140. In a further example, the processor200can detect a region of the image having an intensity exceeding a predefined threshold. In further examples, the processor200controls the emitter140to modulate the beam, such that the intensity, color, or both of the dot are modulated over time. The processor200, in such examples, to detect the presence of corresponding modulation over a sequence of image frames in the video stream captured by the camera136. The detection of the above attributes can also be combined to detect the dot.

Referring toFIG. 8, an image804captured via the camera136(i.e. representing the auxiliary FOV404) depicts the object600and indicium604as discussed above, as well as a dot808projected on the object600by the emitter140. The location of the dot808in the image804(represented by the coordinates812) is obtained at block335. At block340, the processor200determines whether to update the offset data, for example by determining whether an offset vector between the location defined by the coordinates812and the center of the image804(defined by the coordinates816) deviates from the offset vector408defined in the repository220(e.g. for the current range). When the determination is negative at block340, the offset data in the repository220is not altered, and performance of the method300ends. When the determination at block340is affirmative, however, the processor200updates the offset data in the repository220at block345, e.g. by replacing the offset vector408with an adjusted offset vector defined between the coordinates816and812. In the present example, the offset vector408is shown alongside an offset vector820generated from the image804. As is evident fromFIG. 8, the offset vectors408and820are identical, and the determination at block340is therefore negative.

Returning to block335, in other embodiments the offset update data can include, in addition to or instead of the dot-based data mentioned above, locations of indicia detected and decoded in a number of performances of block330, as well as the decode times for each indicium. As will be understood by those skilled in the art, the processor200typically captures a stream of image frames via the imager132, and searches each frame in a predefined pattern (e.g. a spiral beginning at the center of the frame). Therefore, low decode times indicate that few frames were captured before the indicium was detected, and/or that minimal searching within a frame was required. This indicates in turn that the device100was accurately aimed when the primary trigger124was activated (i.e. when the decode timer was initiated). In contrast, elevated decode times indicate that a larger number of frames were captured before the indicium was detected and/or that more extensive searching within a frame was required. This indicates that the initial aiming of the device100may have had reduced accuracy.

In such embodiments, the determination performed by the processor at block340can include a determination of whether a sufficient number of decode operations have been performed to adjust the offset data based on the above-mentioned locations and decode times. When the determination at block340is affirmative, at block345the processor200determines an adjustment to the offset data based on the decode locations and decode times.

For example, turning toFIG. 9A, locations and decode times from three example decode operations are shown. In particular, locations900,904and908are illustrated relative to the primary FOV400, and associated decode times are shown with each location. The processor200is configured to select a subset of the decode times (e.g. the lowest 50% of the decode times, all decode times below a configurable threshold, or the like). The processor200then, having selected the subset of decode times, to determine an average location from the corresponding locations. Thus, as shown inFIG. 9A, having selected the decode times associated with the locations900and904the processor generates an average location912from the locations900and904. The processor200then determines an offset adjustment916based on the average location912relative to the center of the primary FOV400.

Referring toFIG. 9B, the processor then applies the adjustment916to the current offset data (defining the offset vector408), e.g. summing the offset vector408and the adjustment916, to generate updated offset data defining an offset vector and regions924and928. The virtual viewport, in other words, has been shifted up and to the left, in the direction where the lowest decode times have been obtained.