Synchronizing a user device and a kiosk interface using a visual code, and applications thereof

A user can scan a QR code on a kiosk with her mobile device. Scanning the QR code will cause the mobile device to display an interface displayed on the kiosk. When a user enters information on the device corresponding information is displayed on the kiosk. In this way, a user can engage with the kiosk interface in a contactless manner.

BACKGROUND

Field

This field is generally related to interfacing with a kiosk computing device.

Related Art

A kiosk computing device is a computer terminal with specialized hardware and software that provides access to information and applications for communication, commerce, entertainment, or education. They are built into a small structure in a public area and are used for providing and collecting information. Examples include communication and videoconferencing kiosks, financial services kiosks (e.g., Automated Teller Machines), photobook kiosks, Internet kiosks, ticketing kiosks, restaurant kiosks, visitor management and security kiosks, building directory and wayfinding kiosks, hospital and medical clinic registration kiosks, self-checkout kiosks, credit card entry kiosks, and information kiosks.

Kiosk computing devices are typically placed in high foot traffic settings such as shops, hotel lobbies or airports. Given their public nature, often kiosks can be used by many people. They typically have inputs such as buttons and touchscreens that require users to contact them to interact with them.

Infectious diseases, such as cold, flu, and COVID-19, are often spread through contact with shared services. When a person touches a service, she may leave behind particles containing bacteria or viruses. When the next person touches the same surface, the particles may spread to her hand, and her hand may further contact her nose or mouth, possibly causing transmission of a respiratory illness.

To prevent transmission of disease, some mobile device applications allow for touchless interaction with an application. These mobile applications are specially designed for interaction with a particular kiosk application. This requires special development to be done for each application.

Improved systems and methods are needed to allow for touchless access to a kiosk that is more flexible without requiring specialized software to be installed on the mobile device.

BRIEF SUMMARY

In an embodiment, a computer-implemented method enables contactless access to a kiosk computing device. In the method, the kiosk computing device displays a first graphical user interface including a first input field. A machine-readable optical label is generated for display on the kiosk computing device. A mobile device captures and decodes the machine-readable optical label to determine a link identifier. The mobile device requests a network resource addressed by the link identifier. In response to the request, a second graphical user interface that includes a second input field corresponding to the first input field is retrieved and presented on the mobile device. The first and second graphical user interfaces are synchronized such that, when the user enters information on the second input field in the second graphical user interface, the information appears on the first input field in the first graphical user interface and vice versa. In this manner any number of input fields can be completed by the mobile device and kept in sync between the two devices.

System, device, and computer program product embodiments are also disclosed.

Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments, are described in detail below with reference to accompanying drawings.

The drawing in which an element first appears is typically indicated by the leftmost digit or digits in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements.

DETAILED DESCRIPTION

In an embodiment, a user can scan a quick response (QR) code on a kiosk with her mobile device using typically pre-installed software such as the operating system camera app. Scanning the QR code will cause the mobile device to retrieve a web page stored on a server to display a similar interface as displayed on the kiosk application. The web page is presented in a browser on the user's device using typically pre-installed browser software. When a user enters information on the mobile device, corresponding information is displayed on the kiosk. In this way, a user can engage with the kiosk interface in a contactless manner using their mobile web browser, thus providing a way to touchlessly interact with a kiosk without having to install any special software. By allowing touchless interaction, embodiments may prevent spread of infectious diseases.

FIG. 1illustrates a system100for enabling contactless access to a kiosk computing device according to an embodiment. System100includes a mobile device102, a kiosk computing device104, a synchronizing server112, and a public web server120coupled by a network110.

Mobile device102is a computer small enough to be portable. Examples include smart phones and tablet computers. They can also include digital media players, digital cameras, wearable computers (such as smart watches and augmented/virtual reality glasses), and drones. To allow interaction with the user, these mobile devices often contain a visual output, such as an LCD or OLED screen interface. They also have user inputs such as a touchscreen or other button interfaces. These devices can typically communicate via various communication interfaces such as Wi-Fi, Bluetooth, or cellular network interfaces.

Mobile device102includes an optical scanner116. Optical scanner116can capture an image and detect whether the image includes a machine-readable optical label such as a QR code. If it does contain a machine-readable optical label, optical scanner116can decode the label to determine any information embedded therein. To capture the image, optical scanner116may be or include an image sensor, that is, a camera, capable of capturing still images and/or video.

Mobile device102further includes a web browser118. Web browser118can retrieve a network resource (such as a webpage) addressed by the link identifier (such as a uniform resource locator, or URL) and present the network resource for display. In particular, web browser118is a software application for accessing information on the World Wide Web. Usually, web browser118makes this request using the hypertext transfer protocol (HTTP or HTTPS). When a user requests a web page from a particular website, the web browser retrieves the necessary content from a web server, interprets and executes the content, and then displays the page on a display on mobile device102shown as client/counterpart web page119. In examples, the content may have HTML and client-side scripting, such as JavaScript. Once displayed, a user can input information and make selections on the page, which can cause web browser118to make further requests.

As mentioned above, mobile device102is connected to a synchronizing server112which is also connected to the kiosk computing device104via a network110. Network110is a group of computers that use a set of common communication protocols over digital interconnections for the purpose of sharing resources located on or provided by the network nodes. Network110can include both public networks, such as the public cellular network, and private networks, such as corporate intranets.

Kiosk computing device104is a computer terminal featuring specialized hardware and software that provides access to information and applications. Kiosk computing device104includes a kiosk application106. Kiosk application106presents kiosk content109for display on kiosk computing device104. For example, kiosk application106may be defined an HTML page, or group of interconnected HTML pages, or a standalone application of screens that define a kiosk content109that kiosk computing device104displays to a user and uses to solicit information from the user. To solicit information, kiosk content109may have various input fields, such as text fields, checkboxes, radio buttons, picture boxes, or clickable buttons etc. A picture box may capture an image from a device's camera, for example, to capture a photograph of a user, driver's license, or medical ID card. The input fields may be formatted with images and text to make an appealing interface for a user. Kiosk application106may be a web application or, alternatively, a native application.

A kiosk touchless library108may be used to support kiosk application106. Kiosk touchless library108may include executable software that provides functions necessary to allow contactless access to kiosk computing device104from mobile device102. Kiosk touchless library108may be implemented in various ways, including as JavaScript libraries or Native APIs or a combination of both for Hybrid applications. In addition, a skilled artisan would recognize that kiosk touchless library108could be implemented as a separate component from kiosk application106.

To allow contactless access, kiosk touchless library108may include software necessary to generate a machine-readable optical label, to generate a page for display on mobile device102, and to transmit and receive data to and from mobile device102. A brief overview of each of these functions is described in turn, before a more detailed discussion of the operation with respect to, for example,FIGS. 2A-E.

To generate the machine-readable optical label, kiosk touchless library108may include a commercially available QR code generator. Using the QR code generator, kiosk touchless library108may generate a touchless QR code107that encodes information needed by mobile device102to locate a network resource and provide contactless access. In other words, kiosk touchless library108may generate a QR code that is a link identifier. The QR code may also encode information needed to help maintain security, such as a security token (not shown). The security token may be a random number or an encrypted random number that could be used to encrypt and decrypt messages sent from the mobile device102to the kiosk computing device104and vice versa by way of the synchronizing server112.

To generate a page for display on mobile device102, kiosk touchless library108may be used to generate a version of the content presented by the kiosk application106that is reformatted for different screen or input capabilities that mobile device102may have with kiosk computing device104. For example, mobile device102may have a different screen size and/or orientation than kiosk computing device104. Kiosk content109may need to be reformatted to account for the different screen size before a counterpart interface is displayed on mobile device102. In addition, mobile device102and kiosk computing device104may have different input capabilities. For example, kiosk computing device104may have physical keyboard, attached payment devices, or button inputs which are other input devices that are not present on mobile device102. Such inputs may need to be emulated on-screen. Kiosk touchless library108may need to alter the kiosk content to be presented in client/counterpart web page119to account for the different input capabilities between mobile device102and kiosk computing device104. The mobile device102may also support features that the kiosk computing device104does not, including ADA-508 input improvements for people with disabilities, such as high-contrast capabilities or voice-over capabilities for the visually impaired. The client/counterpart web page119displayed by mobile device102can be generated by the kiosk touchless library108, pre-generated by the kiosk touchless library108, or separately authored altogether. To transmit events to mobile device102, kiosk touchless library108may provide JavaScript that interacts with a document object model of kiosk application106. The kiosk touchless library108may similarly provide native libraries functions to support a native kiosk application106. For example, kiosk touchless library108may periodically or intermittently check the document object model to determine what data a user has inputted into various input fields on kiosk application106. If new data has been detected, kiosk touchless library108may send a message across network110to mobile device102to update a counterpart field displayed on mobile device102by way of synchronizing server112(below).

Conversely, mobile touchless library121may receive information input into the counterpart fields on mobile device102. When the client/counterpart web page119was generated or authored, it may have included its own client-side touchless library (discussed further below) that includes its own message passing capabilities to send and receive messages from the kiosk computing device104. That client/counterpart web page119will evaluate the document object model on mobile device102to determine whether a user has updated information on the counterpart form. When the updates are made, mobile device102transmits a message to kiosk computing device104by way of the mobile touchless library121and the synchronizing server112via network110. Kiosk touchless library108receives and evaluates the message. In response to message, kiosk touchless library108may update a counterpart field on kiosk application106to include the newly input information.

Public web server120acts to serve a client/counterpart web page119that embodies enough information to setup a secure communication session between the mobile device102and kiosk computing device104via the synchronization server112. Once a secure communication session (described below) is established, kiosk touchless library108and mobile touchless library121will allow point-to-point encrypted messaging.

Client/counterpart web page119may include the necessary UI components, or the UI components may be sent directly from the kiosk application106using kiosk touchless library108and the aforementioned communication session. For example, the kiosk touchless library108could use reflection or other such technology to generate content for the mobile device102and serve by way of the synchronizing server112acting as an ad-hoc web server, limiting the need to duplicate public content authoring (for mobile device102) and private content authoring (for kiosk computing device104).

The synchronizing server112acts to set up a communication session between mobile device102and kiosk computing device104. When touchless QR code107is generated by kiosk touchless library108, it is expected to have enough information to establish a secure, one-to-one, messaging connection between the kiosk computing device104and the mobile device102that scanned the touchless QR code107combined with the information included within the client/counterpart web page119and public web server120resources. A skilled artisan may realize there are a many cryptographic methods including 3DES, RSA+3DES, etc., for securing messages, as well as many communication methods such as websockets, MQTT, webhooks, etc., that can be used to setup a communication session.

As a detailed example, the touchless QR code107could include a security token and an individualized kiosk topic (not shown) that was generated uniquely per session by kiosk touchless library108. The kiosk application106could setup a subscribe mechanism with synchronizing server112to receive messages on this messaging topic. Any messages sent to synchronizing server112to be routed to the unique kiosk topic as registered by kiosk computing device104would be expected to be encrypted with the security token known only to the kiosk computing device104and the user device102scanning the touchless QR code107after loading client/counterpart web page119. The kiosk application106would diligently wait for a mobile device102to attempt a connection. Once a mobile device102scans the touchless QR code107on kiosk computing device104and loads the client/counterpart web page119, a communication handshake with the kiosk computing device104would be attempted. The client/counterpart web page119could use the mobile touchless library121to facilitate a handshake with the kiosk computing device104through security server112so long as the user device102and the client/counterpart web page119together can construct the correct security token by way of the scanned touchless QR code107and any other cryptographic steps. If the handshake succeeds, the kiosk computing device104and the user device102are paired and able to communicate via synchronizing server112. If more than one user device or outside session attempts to connect to the kiosk, the communication session could be torn down altogether, mitigating digital sniffing.

Security server114conducts a number of security checks. For example, one check may be to enforce “one-to-one” exclusivity between a mobile device102and a kiosk computing device104. If more than one mobile device attempts to connect, the security server114could, as a security precaution, tear down the connection the kiosk computing device104set up. Another such check could be to monitor for malicious agents trying to connect over and over and/or failing handshake protocols. The security server could blacklist IP addresses as a security precaution.

The operation of system100is further described with respect toFIGS. 2A-E.

FIG. 2Aillustrates creation of a touchless session. To create the touchless session, kiosk computing device104may display a QR code as part of its interface as illustrated with respect toFIG. 3.

FIG. 3illustrates an interface300for a medical check-in kiosk. Interface300may, for example, be presented in the lobby of a doctor or medical professional's office. Interface300includes a QR code302. QR code302may encode a link identifier and a security token. As described above, the security token may be a random or pseudorandom number generated by kiosk computing device104. The security token may, in one example, be a parameter in the link identifier. For example, QR code302may have a URL, and other parameters including a security token.

Returning toFIG. 2A, when kiosk computing device104generates QR code302, it may send a setup message204to synchronizing server112. The setup message204may indicate to synchronizing server112to expect a connection request. Synchronizing server112may also register the setup with the security server114.

Displaying interface300and QR code302, a user of mobile device102may direct optical scanner116to conduct QR scan202of kiosk computing device104. QR scan202may be an image capture by optical scanner116of QR code302displayed on interface300. Optical scanner116decodes QR code302to identify a link identifier and security token embedded therein. Optical scanner116, having decoded QR code302, may send the link identifier and security token to web browser118, as illustrated inFIG. 2A.

FIG. 2Billustrates how the mobile device102begins loading initial content. The content request is made via an HTTP or HTTPS URL request, and the public web server120responds with at least an initial client/counterpart web page119and mobile touchless library121. Once a communication session between the user device102and the kiosk computing device104is established as described byFIG. 2C, the client/counterpart web page119may need to enriched with additional UI or library content to be provided by the kiosk application106as described byFIG. 2D.

FIG. 2Cillustrates how a communication session is established between the mobile device102and the kiosk computing device104. By this point, the mobile device102has been provisioned with a client/counterpart web page119, a mobile touchless library121, and a security code as gleaned from QR code302. The mobile device102will attempt to connect to the synchronizing server through an encrypted message by way of a communication session request228which is passed through to the kiosk computing device104. After decryption by way of kiosk touchless library108, the kiosk computing device104returns an encrypted verdict to the mobile device102through a communication verdict message229. If the verdict was successful, the mobile device102and the kiosk computing device104now have a secure end-to-end, encrypted communication session to send content, libraries, and messages as described byFIGS. 2D and 2E.

FIG. 2Dillustrates how additional UI and library content presented in kiosk computing device104gets set up on mobile device102. This additional content, coined “shadow page content,” represents the rest of the data necessary for the client/counterpart web page119to “shadow” or shadow the kiosk content as shown in sample applicationFIG. 4B. An initialize request230is sent by the client/counterpart web page119via the mobile touchless library121to the kiosk computing device104through the synchronization server112. The kiosk computing device104will return additional encrypted content and encrypted libraries that may not have been present in the initial client/counterpart web page119such as fields, values, and libraries that are obscured or specialized for this session. It is worth noting that only the mobile device102and kiosk computing device104have the information necessary to encrypt and decrypt these messages, providing point-to-point encryption.

FIG. 2Eshows the mobile device102now has content that has been served from a web browser118combined with content that has been served from the kiosk computing device104. Similar to kiosk touchless library108inFIG. 1, mobile touchless library121may be a JavaScript library. Mobile touchless library121and subsequent shadow page library content from222may include scripting commands that periodically or intermittently evaluate the document object model of the client/counterpart web page119to determine whether an update has been made. When an update has been identified, mobile touchless library121or additional shadow page content libraries from additional content222may send a message to update the corresponding field on kiosk computing device104as illustrated with respect toFIG. 2E.

FIG. 2Eincludes a diagram250that shows messages252and254being passed between mobile device102and kiosk computing device104. Messages252and254may be transmitted over the secure connection between mobile device102and kiosk computing device104. The connection may be a socket-based connection either directly between mobile device102and kiosk computing device104or between two different socket connections to synchronizing server112, which can act as an intermediary between mobile device102and kiosk computing device104.

Messages252and254communicate events and updates to corresponding input fields in kiosk computing device104and mobile device102. For example, looking toFIGS. 4A and 4B, when a user makes an input into input field454on interface452(on the kiosk computing device450), messages254and252communicate that update to the mobile device, which updates field404on interface402to display the updated information. Similarly, when a user makes an input into field404on interface402(on the mobile device), messages254and252communicate that update to the kiosk, which updates field454on interface452to display the updated information.

In one embodiment, messages252and254may communicate updates every time a user makes a keystroke on the respective interfaces. In another embodiment, messages252and254may communicate updates when a user navigates to a new field on the respective interfaces. In yet another embodiment, messages252and254may communicate updates periodically, perhaps every several seconds.

It is possible for collisions to occur when, for example, a user makes a change to a field on one device before an update on the other device has been propagated through to the corresponding field. A skilled artisan would recognize that mobile device102, kiosk computing device104, and synchronizing server112may implement various different collision resolution methods.

One such method for resolving collisions may be to measure time difference between mobile device102and kiosk computing device104during the communication setup phase shown inFIG. 2C. Whenever an update is made, messages252and254include a timestamp indicating when the update was made that can be adjusted appropriately for mobile or kiosk time depending on the receiver. Touchless libraries108and121will receive the timestamp. When any conflicting updates occur, touchless libraries108and121may use the timestamp to enter the earliest update or latest update in the input field, disregarding the other updates.

FIG. 5is a flowchart illustrating a method500for enabling contactless access to a kiosk computing device according to an embodiment.

At step502, a machine-readable optical label is generated for display on the kiosk computing device. In addition to the machine-readable optical label, the kiosk computing device displays a graphical user interface including at least one input field. The at least one input field can be displayed on the same page as the optical label or on a different page.

At step504, a mobile device captures the machine-readable optical label. An application on the mobile device may activate a camera on the mobile device and analyze images coming in from the camera to detect the optical label. When detected, the application may capture the image.

At step506, the mobile device decodes the machine-readable optical label to determine a link identifier. The mobile device may also decode a security token from the optical label.

At step508, the mobile device requests, over a network, a network resource addressed by the link identifier. In response to the request, the network resource (which may be the kiosk or another intermediate server) transmits to the mobile device a page that is able to be rendered and executed on a web browser on the mobile device. The page includes executable code which is such that, when the web browser executes the page, the web browser transmits data from the second input field to appear in the first input file. The page also specifies a graphical user interface that corresponds to the graphical user interface presented on the kiosk computing device. In particular, the page includes input fields that correspond to the respective input fields presented on the kiosk computing device.

At step510, the mobile device presents the graphical user interface specified in the page. In particular, a web browser on the mobile device renders the page and presents it to a user on a display of the mobile device.

At step512, the graphical user interface of the mobile device and the graphical user interface of the kiosk computing device are synchronized. In particular, the interfaces are synchronized such that, when the user enters information on an input field on one of the interfaces, the information appears on the corresponding input field of the other interface. To synchronize the interfaces, a connection may be created between a client on the mobile device and a server on the kiosk computing device. Information entered by the user in the respective fields may be transmitted via the connection. This transmission may occur repeatedly with messages specifying the updated information. The messages may be transmitted as updated information is entered by user.

To avoid collisions, when the connection is set up, time clocks on the mobile device and kiosk computing device may be synchronized. Then, collisions may be resolved using the synchronized first and second time clocks.

The databases disclosed herein may be any stored type of structured memory, including a persistent memory. For example, this database may be implemented as a relational database or file system. Mobile device102, kiosk computing device104, mobile device400, kiosk computing device450, synchronizing server112, and security server114each include processors and memory, including a nonvolatile computer readable medium for storing processing instructions thereon. The memory may tangibly embody the data and program instructions. Software may include one or more applications and an operating system. Hardware can include, but is not limited to, a processor, a memory, and a graphical user interface display. The computing device may also have multiple processors and multiple shared or separate memory components. To carry out its programmed functionality, the devices may have various modules implemented in hardware, software, firmware, or any combination thereof.

Identifiers, such as “(a),” “(b),” “(i),” “(ii),” etc., are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily designate an order for the elements or steps.