Mobile Capture of High-Resolution Data-Bearing Markings

Recent developments in data-bearing print include a number of technologies that enable high-capacity encoding of data via manipulation of high-resolution printed structure (halftones). While it has been shown that data encoded with these methods can be decoded from scanned images, there are challenges associated with achieving the same type of functionality on a mobile imaging device. The image must be captured at a distance that can resolve the required detail. At the same time, it is necessary to locate the desired region of interest quickly, and to determine if the region is in focus. Due to the designs of several common mobile phone APIs, in some applications it is necessary to achieve this result using frames captured at video rate/quality, and without direct control of an imaging device’s focus system. Nonetheless, the proposed approach is capable of extracting information embedded in the halftone structure of 600 dpi prints with a mobile device. Introduction Recent developments in data-bearing print include a number of technologies that enable higher capacity encoding of data than traditional schemes (such as 1-D, DataMatrix, QR, MS Tag, etc.) via manipulation of halftones. Some examples are based on halftone dot orientation [1] or position [2] (see Fig. 1). One of the appeals of these approaches is that they offer greater graphic design flexibility in the creation of data-bearing print. While it has been shown that data encoded with these methods can be decoded from scanned images, there are challenges associated with achieving the same functionality on a mobile imaging device. This work describes these challenges and presents a system designed to overcome them, specifically to allow information to be gleaned from the halftone information. The solution involves the combination of a fast object detector [3] and multi-scale alignment scheme [4]. The object detector has some distinct advantages when it is applied under the Figure 1. A QR code (left) and a design with d halftone structure (right). constraints of the problem, most notably that it decomposes the problem of detecting an object into stages that involve measuring the degree of the presence of simple piecewise-constant structures. Though this tool is capable of detecting complex objects in natural images, the fact that print contains explicit bi-tonal (i.e., piecewiseconstant) structures makes it an appropriate choice for the solution. The approach is tested with several devices in conjunction with steganographic halftones, which require sub-pixel accurate alignment to enable correct decoding. It is shown that this strategy can achieve correct decoding on error-protected payloads embedded in prints that are captured with mobile imaging devices. This paper is organized as follows. The following section discusses practical considerations that help define the problem. Next, the proposed solution is described, and test results are given in the last section. Constraints and Challenges Candidate solutions must accommodate several considerations in order to be practical. First, they must provide imagery with enough visual acuity to resolve information in the high-resolution structures. Second, the solutions must be implementable on a range of devices. Resolving the Target One of the most difficult aspects of creating a mobile reader is that it must be capable of establishing content that is in focus. Stegatones [2] represent one high-resolution method of conveying information where the positions of halftone clustered dots are used to embed data in print. Typically, the halftones used for this purpose are rendered at 400-600 dpi, and in general, these markings pose few problems for decoding from captured scans. An example stegatone print is given in Fig. 2. To be properly resolved and interpreted, the captured print must usually exhibit a resolution that is at least as great as the resolution of the original rendering. Obviously, some devices will not be able to do so depending on the quality of the camera. Nonetheless, on devices that demonstrate the ability to resolve enough relevant details, the problem of capturing an in-focus image must be solved. This issue is illustrated with the following example, based around an iPhone 4S. Let D represent the distance to a target in cm. The resolution of pixels captured by the camera is given by resolution (dpi) = 1565 – 91.7 D. (1) Tests used to establish this relationship also indicate that the objects closer than 8 cm are not in focus. Similarly, objects captured beyond 11 cm will generally not have enough resolution Figure 2. An enlarged view of a captured data-bearing “flip-up” graphic; the capture and subsequent reading process is used to extract information from minute perturbations. to be properly interpreted. The absolute measurements discussed here will vary from device to device; the key point is that the distance-to-target window where a data-bearing print can be correctly interpreted can be relatively small. Fig. 3(b) shows data bearing print examples that were captured in-focus, but too far from the target for proper decoding, and close enough to include all the necessary detail needed for decoding, but out of focus; the iPhone window is illustrated in Fig. 3(b). A mobile reader must be set up to be able to determine the difference between these prints and an ideal candidate for decoding. There are several methods that can be used to address the constraints mentioned above, two of which include (1) guiding the user through the process of capturing an image of the target (common, for instance, in mobile check-cashing tools [5]), and (2) determining the information via constant monitoring of a video stream. While the first option allows a greater degree of control from the algorithm design perspective, and for more efficient (explicit) focus estimation, it does not offer a user experience comparable to that achieved with, for example, QR codes (see Fig. 1). The ubiquity of QR codes in printed media has popularized a mechanism by which users interact with print. The goal is to achieve the same interaction with mobile device and a high resolution marking, ideally without guiding the user through a capture procedure. Furthermore, given the narrow window in which focus can be achieved, any user action to manually begin a capture process might actually distort the achieved focus at any given point. As a result, this work examines solutions that can be automated through monitoring streaming images. Multi-Platform Implementation While proof-of-concept implementations are a good first step towards developing a mobile reading procedure, ideally, a solution would be applicable to a class of devices, as opposed to just one. This constraint suggests limitations to general algorithm complexity, as well as possible focus analysis/estimation schemes. (a)