Patent Publication Number: US-10770114-B2

Title: Video matrix barcode system

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to and the benefit of International Patent Application No. PCT/US2017/031780, filed May 9, 2017, entitled “VIDEO MATRIX BARCODE SYSTEM,” the entire contents of which are incorporated herein by reference. 
     BACKGROUND 
     The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure. 
     Matrix barcodes, such as quick response or QR codes, are symbols that are widely used in packaging, advertising, transportation, and more for graphically storing information. Unlike simple one-dimension barcodes, the two-dimensional techniques used in matrix barcodes allow more data to be conveyed via the symbol. In some embodiments of matrix barcodes, nearly 3,000 characters of data can be stored. However, this is not always enough data to convey the amount of information desired. 
     SUMMARY 
     A data object that is too large to be conveyed in a single matrix barcode may be parsed into individual matrix barcode payloads, each with associated metadata for instructions on reassembling the original data object, the payloads and metadata rendered into individual matrix barcodes. These individual matrix barcodes may then be assembled into a matrix barcode movie. A signboard or other display type may be used to project the matrix barcode movie. A reader, such as a modified smartphone, may be used to capture the matrix barcode movie and extract the individual matrix barcodes that make up the movie. Each frame of the matrix barcode movie may contain payload data and metadata used to reassemble the original content. Color matrix barcodes, now being tested, can further increase the capacity of matrix barcode movies. 
     In environments where it may be insecure, undesirable, or simply inconvenient to broadcast information via a radio signal, the matrix barcode movie offers a method of limiting the dissemination of information to devices in a known position with respect to a matrix barcode movie display device, without any pairing, connecting, or other handshaking between the sender and the recipient. For some, the ability to receive information anonymously without leaving a Bluetooth or WiFi “trail” may be perceived as a particular advantage over radio-based information systems. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The figures depict a preferred embodiment for purposes of illustration only. One skilled in the art may readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. 
         FIG. 1  is a system diagram illustrating system components and data generation in accordance with the current disclosure; 
         FIG. 2  is an illustration of an embodiment of a matrix barcode movie; 
         FIG. 3  is a flowchart of a method of generating a matrix barcode movie; and 
         FIG. 4  is a flowchart of a method of decoding a matrix barcode movie. 
     
    
    
     DETAILED DESCRIPTION 
     Matrix barcodes are used in industry, retail, product markings, web sites, and more for easily relaying information such as a URL or product information. One type of matrix barcode is the well-known QR Code, a registered trademark of Denso Wave Incorporated of Japan. Matrix barcodes are defined in several different formats that support data content sizes from several hundred to several thousand characters or bytes. Matrix barcodes may include codes as the nature of the data, including, but not limited to, a URL type for web addresses, a VCard type for business card data, a text type, email type, and more. All matrix barcodes follow the same basic formats of position, alignment, and timing markers, as well as data rows and columns. While higher density matrix barcodes may convey relatively large amounts of data, they may not always be capable of transferring the amount of data an advertiser or other user may desire. 
       FIG. 1  illustrates one exemplary embodiment of a system  100  that addresses data size limitations of matrix barcodes. The system  100  includes a movie generator  102 , a projector  120  and a reader  128 . The movie generator  102  may be a processor-based machine including a processor  104  and memory  105 . The processor  104  may execute commands stored in the memory  105 , while the memory  105  may also include data and settings as well as executable code. The movie generator  102  may also include modules implemented in hardware or software that process data into videos. 
     In the illustrated embodiment, the movie generator  102  may have functional modules including a parser  108 , a converter  112 , and a movie packager  116 . It will be recognized that the exact functions of each module may be combined or further separated based on the exact needs of the system  100 , such as conversion speed, cost, throughput, etc. As such, some or all of the aspects of these modules  108 ,  112 ,  116  may be implemented in hardware when speed of processing is a factor. Similarly, the modules may be implemented in software when, for example, low cost is desirable and throughput is not a consideration. 
     The projector  120  may include a processor  122  and memory (not depicted) a user interface  124  and a display  126 . The user interface  124  may include a specific input device such as a keyboard, cursor device, touch screen, or other manual interface. In an embodiment, more than one matrix barcode movie may be selectable on the display  126  via the user interface  124 . A further manual interaction may cause the matrix barcode movie to begin playback. In another embodiment, the user interface  124  may include a camera or other sensor that is capable of reading information conveyed by the reader  128 . For example, the user interface  124  may wait for an optical signal from the reader  128  before beginning playback of the movie or the optical signal may indicate the frame rate at which the matrix barcode movie is to be played. Alternatively, cover art for a particular matrix barcode movie may indicate a frame rate at which the movie will play so that the reader  128  may note the frame rate and in response adjust its capture rate accordingly. 
     The reader  128  may include a camera  130 , a video capture module  134 , a data extractor  138 , and an assembler  142 . The reader  128  may also include a processor  146 , a memory  148 , and a display  150 . The camera  130  may be used to record the matrix barcode video played via the display  126  of the projector  120 . A video capture (vidcap) module  134  decomposes the video back into individual matrix barcodes, the assembler  142  extracts the data from the individual matrix barcodes and creates a duplicate data object  144  corresponding to the original data object  106 . 
     In operation, the movie generator receives a data object  106  and creates individual data payloads and associated metadata  110  via the parser module  108 . The converter  112  then generates matrix barcodes  114  representing the separate payload/metadata pairs. In order to correctly parse the data payload and metadata  110  to the appropriate size, the parser  108  must be aware of the particular matrix barcode type that will be used by the converter  112  in order to ensure that adequate data space will be available in the matrix barcodes  114 . A movie packager  116  may then perform a data compression on the individual matrix barcodes  114  using, for example, an H.264 compression scheme. The movie packager  116  may then place the compressed frames into a container to complete the development of the movie  118 . In various embodiments the container may be an MP4, AVI, WMV, or any of numerous known movie container formats. 
     When creating the matrix barcode movie  118  from the individually encoded matrix barcode frames, the compression process may be set to make each frame an independent or I-frame, or may use a standard coding technique that selectively inserts I-frames at intervals. In the latter case, progressive or P-frames are generated between I-frames where each P-frame carries only the differences between frames since the last I-frame. A movie with nothing but I-frames may be larger than that possible using more aggressive compression with P-frames but may be easier to decode during presentation, which in turn may allow a less expensive display process to be used at the projector  120 . 
     In another embodiment, the matrix barcode movie  118  may be transmitted as a movie file to the reader  128  where the reader  128  may decode the movie data directly into barcodes. In such a case, having individual I-frames may reduce the complexity of re-creating the matrix barcodes. 
     The projector  120  may receive the movie  118  via any suitable transport mechanism, including but not limited to, a removable memory, a wireless network, and a wired network. The movie  118  may be viewable via the display  126 , such as product signage or active tags. In an embodiment, the movie  118  may play as a loop or may play responsive to a signal received either manually or from the reader  128 . An indication of frame rate may be transmitted between the projector  120  and the reader  128 . In various embodiments, the projector  120  may adjust the frame rate of the video as played or the reader  128  may adjust the video capture rate of the camera  130  based on the capabilities exchanged between the projector  120  and the reader  128 . For example, some video utilities allow a video file to be replayed at a different frame rate from the original encoding. Particularly since audio is not a factor in the playback, adjusting the video frame rate can be accomplished using several known techniques. 
     A movie copy  132  is received at the reader  128  via the camera  130  and a process is begun to extract the individual frames of the movie  118 . In this process the separate matrix barcodes are captured from the movie  118  and, in an embodiment, may be written out as individual image files of the matrix barcodes  136  via a video capture (vidcap) module  134 . In order to ensure that each frame of the matrix barcode movie is captured, an application on a smartphone may allow adjustments to the frame rate of its camera. The matrix barcode sequence, or matrix barcode movie, may be played at a frame rate below the Nyquist rate of the camera capture speed. For example, a camera capable of recording 30 frames per second could be used to capture each unique matrix barcode if played at a rate below 30 frames per second, for example, at 15 frames per second. 
     The recreated individual matrix barcodes  136  may processed by a data extractor  138  to generate the individual payload data and metadata  140  associated with each matrix barcode. An assembler  142  may use the metadata to reassemble the payload data to duplicate the original data object  106  in a new data object  144 . 
     In various embodiments, the data object  144  may be used in taking an action. For example, in one embodiment, the data object  144  may be a URL and an action would be to open the URL. In another embodiment, the data object  144  may be a multimedia object with text and images, and the action may be to present the information on the display  150 . In another embodiment, the data object  144  may be a digital certificate with a cryptographic key and the action may be to store the certificate in a browser&#39;s certificate store. The action to be taken may be determined in any of several ways. In one embodiment, the data object  144  may have a file type that indicates what action to take, for example, a .pdf file extension may indicate that the file should be opened in a PDF viewer and a .doc file extension may indicate that the file should be opened in Microsoft Word™. In another embodiment, one or more elements of the metadata included in the matrix barcodes  114 ,  136  may indicate what action to take with the data object  144 . Based on local security policies, the action may require intervention from a user or operator of the reader  128 . The reader  128  may also include a processor  146 , a memory  148 , and a display  150  that combine to run executable programs and allow user interactions with the reader  128 . 
       FIG. 2  is an illustration of an embodiment of deploying a matrix barcode movie. In this illustration, a matrix barcode movie  182  is incorporated in another media platform other than as a standalone element. Shown here, the matrix barcode sequence is incorporated in the normal frame sequence of another media, such as a movie. In other embodiments, the matrix barcode movie may be incorporated media platforms including video advertising, how-to media of in-store displays, animated graphic image format (GIF) sequences, video games, etc. In this embodiment, the reader  128  may capture the matrix barcode frames in the same manner described herein. 
       FIG. 3  is a flowchart of a method  200  of creating a matrix barcode movie  118 . At block  202 , a data object  106  may be received at a movie generator  102 . In an embodiment, the data object  106  may be larger than the capacity of a single matrix barcode of a particular type. The choice of matrix barcode type may be based on any of several factors but relevant to this disclosure is only that the selected matrix barcode type is not capable of encoding the entire data object  106  in a single image. At block  204 , the data object  106  may be parsed into individual payload data packets and corresponding metadata generated for each payload data packet at block  206 . The metadata may use a predetermined format to indicate the start and size of the metadata and a start and size of the payload data packet for use in decoding. The metadata may also include a sequence number for its corresponding data payload data packet to allow resequencing the parsed data during reassembly. The process may be repeated via block  208  until the entire data object  106  is parsed into payload data packets and corresponding metadata  110 . 
     A matrix barcode image may be rendered for each of the payload data packets and its corresponding metadata at block  210 . The resulting matrix barcodes  114  can be placed in any sequence although there is a logical rational to place them in the order they were created and will be decoded. The resulting individual matrix barcodes, as images, may be encoded into matrix barcode frames at block  212 . In an embodiment, the barcode images may be encoded into an H.264 format or other similar video compression format. At block  214 , the encoded frames may then be formatted into a matrix barcode movie  118  in any of several well-known containers. These container formats may include MP4, AVI, or WMV formats, among others. In an embodiment, the matrix barcode movie  118  may be named to include a container designator, e.g., filename.mp4. Additionally, the filename may include a framerate of the movie  118 , e.g., filename15.mp4 to indicate an MP4 container with a 15 frames/second frame rate. 
     A flowchart of a method  220  of decoding and acting on a matrix barcode movie is illustrated in  FIG. 4 . At block  222 , a frame rate for video capture may optionally be set for a camera  130  of a reader  128 . An indication of a desired, or in some cases, a minimum frame rate may be transmitted from the projector  120  to the reader  128  via an optical signal directly or by instructing a user of the reader to manually set the frame rate via a user interface of the reader  128 . For example, a cover art shown on the display  126  may indicate the frame rate of the movie or a minimum frame rate setting for the camera  130 . 
     At block  224 , the movie  118  may be presented by the projector  120  and received at the reader  128  by the camera  130  so that a movie file  132  is created at the reader. At block  226 , a video capture module  134  may be used to convert the frames of the matrix barcode movie  132  into the individual matrix barcode images  136  of the original matrix barcodes  114 . After the matrix barcode images  136  have been extracted, at block  228  the payload data packet and corresponding metadata  140  for each barcode may be read. A de-duplication process may eliminate duplicate frames that may be created via oversampling of the movie  118  as projected, either at the matrix barcode stage or at the payload data packet stage. 
     Then, at block  230  the data object  144  may be re-created from the payload data packets using the metadata. Finally, at block  232 , an action may be taken corresponding to the data object  144 . The action may directly involve the data object  144 , for example, playing a video when the data object  144  is a video file. Alternatively, the data object  144  may be an executable file that runs on the processor  146  of the reader  128 . In another embodiment, the data object  144  may simply have instructions that are followed in order to allow a different action to occur, such as unlocking a previously downloaded program or allowing access to a previously hidden website. 
     The technical effects of using a matrix barcode movie  118  to transfer information are the ability to limit distribution of large amounts of data to a well-defined physical space and to permit such a transfer using a fully passive, one-way transfer. Unlike radio frequency data broadcasts that radiate in every direction and require handshaking and leave a device signature, the optical transfer using the matrix barcode movie permits rapid transfer of large amounts of data to an anonymous target (i.e., the reader  128 ). Unlike the limited capacity of matrix barcodes, the matrix barcode movie  118  allows a virtually unlimited data capacity and data types beyond those defined for current matrix barcodes. 
     The use of a matrix barcode movie  118  to transfer a data object  106  between entities benefits the sender of the data by ensuring that the recipient is within a limited range of the projector  120  and presents the data in a format that is universally available to devices such as smartphones and tablets, that is via the device&#39;s camera. There is no need for communicating WiFi passwords or opening up either the sending or receiving devices to attacks via an unprotected radio network. Further, in the case where the video transfer is used to confirm presence, the use of the optical transfer ensures that the recipient is in a given physical proximity to the projector  120 , rather than merely within WiFi range. The use of the matrix barcode movie  118  to transfer data also benefits the recipient in a similar way by allowing anonymous receipt of the information, without the need to log in to a network or otherwise leave an electronic footprint. 
     The figures depict preferred embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein 
     Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for the systems and methods described herein through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the systems and methods disclosed herein without departing from the spirit and scope defined in any appended claims.