Patent Description:
As mobile devices such as smartphones have become increasingly ubiquitous, the creation of dynamic video recordings by amateur videographers has increased accordingly. For various video applications such as video stabilization, stitching, and segmentation, the ability to identify static background portions of a video is important. For dynamic videos with a large portion of the foreground occupied by one or more moving foreground objects and a constantly changing background, existing strategies for static background identification mis-identify the background portions and over-segment the video into more than foreground and background objects.

<CIT> discloses a system and method for markers with digitally encoded geographic coordinate information for use in an augmented reality (AR) system. The method includes automatically matching digital data within an AR system by utilizing a digitally encoded marker (DEM) containing world coordinate information system and mathematical offset of digital data and a viewing device. The method further includes encoding geographic coordinate information into markers (e.g., DEMs) and decoding the coordinate information into an AR system.

The disclosure provides a system and method for generating augmented reality content based on detection of markers placed in a physical environment.

According to an aspect, there is provided a method of an electronic device as set out in claim <NUM>. Optional features are set out in claims <NUM> to <NUM>.

According to another aspect, there is provided an electronic device as set out in claim <NUM>. Optional features are set out in claims <NUM> to <NUM>.

According to another aspect, there is provided a non-transitory computer readable medium as set out in claim <NUM>.

Before undertaking the mode for invention below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The phrase "at least one of,"
when used with a list of items, means that different combinations of one or more of the listed items may be used, and only one item in the list may be needed.

For a more complete understanding of the disclosure and its advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:.

<FIG>, discussed below, and the various embodiments used to describe the principles of the disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the disclosure may be implemented in any suitably arranged wireless communication system.

With the increasing ubiquity of mobile computing devices, including mobile phones and head-mountable devices or head-mounted devices (HMDs), augmented reality (AR) applications are in demand. AR applications allow for the display of content that may be overlaid on a representation of the real world environment surrounding the AR device. For example, a mobile device or HMD may provide AR experiences by displaying, on a display, an image or video feed of the real world as captured by a camera on the device, with content such as labels, pictures, 3D models, or the like overlaid on the representation. Such an embodiment could function through a display of a smart phone or a display of an HMD that encloses a user's eyes but has an external camera. In another example, an HMD may provide a display overlaid on a transparent lens (for example, by using a miniature projection device to project the display onto the transparent lens) such that a user is directly viewing the real world environment through the lens with the content of the display overlaid on the real world environment.

Embodiments of the disclosure contemplate that information used to dynamically generate content for AR applications may be provided to mobile devices by markers placed in the physical environment. Such markers could include, for example, passive markers such as drawings or markings (e.g., a quick response (QR) code) on a surface, or active markers such as devices that include light sources (including infrared or visible light emitting diodes (LEDs)). A camera of the mobile device may capture an image or a video feed of the marker, and may interpret the marker and retrieve the information contained in the marker, which may be used to generate content for display via an AR application. For simplicity, embodiments of the present disclosure will be discussed in the context of an HMD, but it is understood that other types of mobile devices, such as a smart phone, may be used with embodiments of the disclosure.

The information provided by the marker may be used directly by the HMD to generate content for an AR application. For example, the information may include content data that specifies parameters of an image or 3D object to display, and/or metadata that specifies position and size of the content, text or other information to display, or the like. The HMD may then interpret this information to generate content for display. The information provided by the marker may additionally, or alternatively, include information that specifies a third party source of content (e.g., a server IP address, a BLUETOOTH ID, etc.). For example, the information provided by the marker may specify a server IP address, the HMD may connect to the server at the specified address via a wireless communication protocol, and the server may in turn provide the HMD with dynamically generated content, such as an advertisement, streaming video, or the like, to be displayed by the AR application. The third party source of content may, in some embodiments, include an edge computing node that is located geographically near the marker.

In some embodiments, the marker is self-contained. For example, the marker may be a low-power, battery-powered device that is pre-configured with information to be optically transmitted to HMDs via an LED. It is understood that the marker may broadcast this information to any HMDs within range, and that references to a marker transmitting information to an HMD may include a marker broadcasting information that is received by an HMD. The marker may then be positioned within the physical environment and left to operate on its own without further interaction. In other embodiments, the marker may include additional communications interfaces, such as a wired or wireless interface, that may be used by a backend device to reconfigure or update the information to be transmitted by the marker.

In some embodiments, the marker may be capable of two-way communication with an HMD using a wireless interface such as BLUETOOTH, WIFI direct, or the like to receive information from the HMD. For example, the marker could optically transmit to the HMD a request for a device ID, service account credentials, or the like. The optical transmission could include details for connecting to the marker via another communication protocol, and the HMD may use this communication protocol to provide the requested information. The marker could in turn optically transmit to the HMD content for display based on the received information.

<FIG> illustrates an example system <NUM> according to embodiments of the disclosure. The embodiment of the system <NUM> shown in <FIG> is for illustration only. Other embodiments of the system <NUM> could be used without departing from the scope of the disclosure.

The system <NUM> may include a network <NUM> that may facilitate communication between various components in the system <NUM>. For example, network <NUM> may communicate Internet Protocol (IP) packets or other information between network addresses. The network <NUM> includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

The network <NUM> may facilitate communications between a server <NUM> and devices such as mobile devices <NUM> and <NUM>, and markers <NUM>. Server <NUM> may represent one or more servers. Each server <NUM> includes any suitable computing or processing device that may provide computing services for one or more client devices. Each server <NUM> may, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network <NUM>.

Each mobile device <NUM> and <NUM> may represent any suitable computing or processing device that interacts with at least one server or other computing device(s) over the network <NUM>. In this example, the mobile devices <NUM> and <NUM> include an HMD <NUM> and a mobile phone <NUM> (such as a smartphone). However, any other or additional mobile devices could be used in the system <NUM>. For simplicity, examples in the disclosure will refer to the HMD <NUM>, but it may be understood that any suitable mobile device, such as mobile phone <NUM>, could be used with embodiments of the disclosure.

The markers <NUM> may represent any suitable computing or processing device that may be placed in a physical environment and configured to communicate with an HMD <NUM>. For example, a marker <NUM> may be a low-power, battery-powered device that communicates optically (e.g., via an LED) to an HMD <NUM>. For example, a marker <NUM> may communicate with an HMD <NUM> using IEEE <NUM>. <NUM> visible light communication (VLC) protocols, or using IR. In some embodiments, a marker <NUM> is an independent device that operates without any connection to the network <NUM> or server <NUM>. For example, the marker <NUM> may be preconfigured with data to transmit to an HMD <NUM>, and subsequently placed in the environment of the system <NUM> to operate indefinitely (or until the marker <NUM> is removed from the system <NUM>). In such embodiments, the marker <NUM> may make one way transmissions to the HMD <NUM>, and may not be configured to receive return communications from the HMD <NUM>. This conserves power and may allow the marker <NUM> to operate on battery power for long periods of time. In some embodiments, a passive marker <NUM> is placed in the environment of the system <NUM> instead of or in addition to active markers <NUM>. For example, a passive marker <NUM> may include a QR code, a drawing, or an object that is recognized by the HMD <NUM> using computer vision.

In some embodiments, a marker <NUM> may additionally include communications interfaces that allow the marker <NUM> to receive communications from the HMD <NUM>. For example, the marker <NUM> may periodically (or constantly) transmit an optical signal via an LED for receipt by nearby HMDs <NUM>, and may receive transmissions from an HMD <NUM> using a different communication interface such as BLUETOOTH, WIFI direct, or the like. In some embodiments, a marker <NUM> that includes such two-way communications interfaces may be connected to a power source that allows for higher power operation than a battery-powered marker <NUM>.

In this example, HMD <NUM> and suitable markers <NUM> may communicate indirectly with the network <NUM>. For example, an HMD <NUM> or a marker <NUM> may communicate via one or more base stations <NUM>, such as cellular base stations or eNodeBs (eNBs), or via one or more wireless access points <NUM>, such as IEEE <NUM> wireless access points. Note that these are for illustration only and that the HMD <NUM> or marker <NUM> may communicate directly with the network <NUM> or indirectly with the network <NUM> via any suitable intermediate device(s) or network(s).

Although <FIG> illustrates one example of a system <NUM>, various changes may be made to <FIG>. For example, the system <NUM> may include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of the disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document may be used, these features may be used in any other suitable system.

<FIG> illustrates an example electronic device <NUM> according to embodiments of the disclosure. The embodiment of the electronic device <NUM> shown in <FIG> is for illustration only and other embodiments may be used without departing from the scope of the disclosure. The electronic device <NUM> may come in a wide variety of configurations, and <FIG> does not limit the scope of the disclosure to any particular implementation of an electronic device. In certain embodiments, one or more of the devices <NUM> and <NUM> of <FIG> may include the same or similar configuration as electronic device <NUM>.

As shown in <FIG>, the electronic device <NUM> includes an antenna <NUM>, a communication unit <NUM>, a transmit (TX) processing circuitry <NUM>, a microphone <NUM>, and a receive (RX) processing circuitry <NUM>. The communication unit <NUM> may include, for example, a RF transceiver including communication circuitry, a BLUETOOTH transceiver including communication circuitry, a WI-FI transceiver including communication circuitry, ZIGBEE, infrared, and the like. The electronic device <NUM> also includes a speaker <NUM>, a processor <NUM>, an input/output (I/O) interface <NUM>, an input <NUM>, a display <NUM>, a memory <NUM>, one or more cameras <NUM>, and a biometric scanner <NUM>. The memory <NUM> includes an operating system (OS) <NUM> and applications <NUM>.

The communication unit <NUM> receives, from the antenna <NUM>, an incoming RF signal transmitted such as a BLUETOOTH or WI-FI signal from an access point (such as a base station, Wi-Fi router, Bluetooth device) of the network <NUM> (such as a Wi-Fi, Bluetooth, cellular, <NUM>, LTE, LTE-A, WiMAX, or any other type of wireless network). The communication unit <NUM> may down-convert the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry <NUM> that generates a processed baseband signal by filtering, decoding, or digitizing the baseband or intermediate frequency signal, or a combination thereof. The RX processing circuitry <NUM> transmits the processed baseband signal to the speaker <NUM> (such as for voice data) or to the processor <NUM> for further processing (such as for web browsing data and remittance).

The TX processing circuitry <NUM> receives analog or digital voice data from the microphone <NUM> or other outgoing baseband data from the processor <NUM>. The outgoing baseband data may include web data, e-mail, or interactive video game data. The TX processing circuitry <NUM> encodes, multiplexes, digitizes, or a combination thereof, the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The communication unit <NUM> receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry <NUM> and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna <NUM>.

The processor <NUM> may include one or more processors or other processing devices and execute the OS <NUM> stored in the memory <NUM> in order to control the overall operation of the electronic device <NUM>. For example, the processor <NUM> may control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit <NUM>, the RX processing circuitry <NUM>, and the TX processing circuitry <NUM> in accordance with well-known principles. The processor <NUM> may be also capable of executing other applications <NUM> resident in the memory <NUM>. For example, the processor <NUM> may execute applications <NUM> that perform computer vision on images or video received via camera <NUM>. In particular, the processor <NUM> may execute applications <NUM> that detect and interpret data from markers <NUM> and <NUM>, and that generate, based on the interpreted data from markers <NUM> and <NUM>, content for display via display <NUM>.

The processor <NUM> may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor <NUM> may include at least one microprocessor or microcontroller. Example types of processor <NUM> may include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry. The processor <NUM> may be also coupled to the I/O interface <NUM> that provides the electronic device <NUM> with the ability to connect to other devices, such as a marker <NUM>. The I/O interface <NUM> may be the communication path between these accessories and the processor <NUM>.

The processor <NUM> may be also coupled to the input <NUM> and the display <NUM>. The operator of the electronic device <NUM> may use the input <NUM> to enter data or inputs into the electronic device <NUM>. Input <NUM> may be a keyboard, touch screen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user in interact with electronic device <NUM>. For example, the input <NUM> may include voice recognition processing thereby allowing a user to input a voice command via microphone <NUM>. For another example, the input <NUM> may include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel may recognize, for example, a touch input in at least one scheme among a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. Input <NUM> may be associated with a camera <NUM> by providing additional input to processor <NUM>. The camera may be used to capture images, such as a QR code, or video feeds, such as a series of blinks from an LED, to be processed by the electronic device <NUM> or passed on to a server <NUM> on the network. The input <NUM> may also include a control circuit. In the capacitive scheme, the input <NUM> may recognize touch or proximity.

The display <NUM> may be a liquid crystal display (LCD), light-emitting diode (LED) display, optical LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like. The display <NUM> may also be a projector. For example, a miniature projector may be used as the display <NUM> to project an image onto a transparent lens of an HMD for AR applications.

The memory <NUM> may be coupled to the processor <NUM>. Part of the memory <NUM> may include a random access memory (RAM), and another part of the memory <NUM> may include a Flash memory or other read-only memory (ROM). The memory <NUM> may include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory <NUM> may contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc.

Electronic device <NUM> further may include one or more cameras <NUM> that capture images of the physical environment surrounding the electronic device <NUM>. In some embodiments, the camera <NUM> may be an infrared (IR) camera or a multi-spectrum camera. In some embodiments, one camera <NUM> is a multi-spectrum camera used to capture images or video for various applications, and a second camera <NUM> is an IR camera used to capture communications via IR from a marker <NUM>. The camera <NUM> may be a still photography camera or a video camera. In some embodiments, the camera <NUM> may be a photodiode (i.e., a single pixel camera).

In embodiments where camera <NUM> is a still camera, the camera <NUM> may take periodic still photographs. With respect to the capture of an optical transmission from a marker <NUM> (for example, in the form of a blinking LED), as discussed above and below, the period of still photography may be configured to match a known period of optical transmission of the marker <NUM> such that a photo is taken during each time period where the LED of the marker <NUM> is expected to be illuminated. This allows capture of a bitstream from the marker <NUM>, where each photo captures a bit of information from the marker <NUM>, and where the illumination or lack of illumination indicates the value of the bit. In other words, when the LED of the marker <NUM> be turn on or turn off during each time period based on light pattern corresponding to a data, e.g. image data, to be transmitted to the HMDs <NUM>, the camera <NUM> may capture a plurality of images during the each time period. And processor <NUM> may identify the light pattern based on the plurality of images captured by the camera <NUM>, and may identify the data corresponding to the light pattern. With respect to a static marker <NUM> (such as a drawing, QR code, or an object in the environment), computer vision may be applied to locate and interpret markers <NUM> in each photograph that is taken of the environment.

In embodiments where camera <NUM> is a video camera, the camera <NUM> may capture a continuous video feed of the surrounding environment. When a marker <NUM>'s blinking LED is captured, the illumination or lack thereof may be determined and interpreted as a bit stream. Additionally, computer vision may be applied to each frame of a video stream to locate and interpret static markers <NUM> that are captured in the video feed.

Although <FIG> illustrates an example of an electronic device in a computing system, various changes may be made to <FIG>. For example, various components in <FIG> may be combined, further subdivided, or omitted and additional components may be added according to particular needs. As a particular example, the processor <NUM> may be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communication networks, electronic devices may come in a wide variety of configurations, and <FIG> does not limit the disclosure to any particular electronic device.

<FIG> illustrates an example electronic device <NUM> according to embodiments of the disclosure. The electronic device <NUM> may be a marker <NUM> of <FIG>. The electronic device <NUM> may be used to transmit and receive communications with an electronic device <NUM>, such as an HMD <NUM>, as will be further described below.

The electronic device <NUM> includes a bus system <NUM>. The bus system <NUM> may support communication between at least one processor <NUM>, at least one storage device <NUM>, at least one communication interface <NUM>, at least one input/output (I/O) unit <NUM>, and a light emitting unit (e.g., an LED) <NUM>.

The processor <NUM> may execute instructions that may be loaded into a memory <NUM>. For example, the processor <NUM> may execute instructions that convert data stored in a memory <NUM> into a signal to be optically communicated via the light emitting unit <NUM>. The processor <NUM> may include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processors <NUM> may include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discrete circuitry.

The memory <NUM> and a persistent storage <NUM> may be examples of storage devices <NUM>, which represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory <NUM> may represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage <NUM> may contain one or more components or devices supporting longer-term storage of data, such as a read only memory, hard drive, Flash memory, or optical disc. In some embodiments, data to be transmitted from the electronic device <NUM> to an electronic device <NUM> (such as an HMD <NUM>) may be stored in a memory <NUM>.

The communication interface <NUM> may support communications with other systems or devices. For example, the communication interface <NUM> may include a network interface card or a wireless communication unit facilitating communications over a network, such as network <NUM>. The communication interface <NUM> may support communications through any suitable physical or wireless communication link(s).

The I/O unit <NUM> may allow for input and output of data. For example, the I/O unit <NUM> provides a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit <NUM> also may send output to a display, printer, or other suitable output device. The I/O unit <NUM> may be used, for example, to facilitate pre-configuration of the marker <NUM> before it is deployed in a system <NUM>.

The light emitting unit <NUM> may allow for optical transmission of data to electronic devices <NUM>, such as HMDs <NUM>. In some embodiments, the light emitting unit <NUM> may include an infrared or visible spectrum LED and an appropriate controller for the LED, and optical communication may be achieved by switching the LED between "off" and "on" states (e.g., blinking the LED). Each state of the LED may represent a state of a binary bit, and thus the LED may be used to communicate a bitstream. In other words, processor <NUM> of the marker <NUM> control the LED to be turn on or turn off during each time period based on light pattern data to be transmitted to the HMDs <NUM>. The light pattern may be configured to indicate a turn on or turn off during each time period for representing a variety of types of data, e.g. image, video, etc, and the light pattern may be pre-stored in the memory <NUM> of the marker <NUM> or may be received from external device, e.g. server <NUM>. In some embodiments, the processor <NUM> may execute a program stored in a storage device <NUM> in order to transition the state of the light emitting unit <NUM> at a predetermined periodic rate, which is also known by HMDs <NUM> that capture light from the light emitting unit <NUM>.

Although <FIG> illustrates an example electronic device <NUM>, various changes may be made to <FIG>. For example, various components in <FIG> may be combined, further subdivided, or omitted and additional components may be added according to particular needs. As a particular example, the processor <NUM> may be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, computing devices may come in a wide variety of configurations, and <FIG> does not limit the disclosure to any particular system or device.

<FIG> illustrate examples of interpretation of communications from a marker by an HMD <NUM> according to embodiments of the disclosure. The illustrations of <FIG> represent the point of view of an HMD such as HMD <NUM> (i.e., they represent what a user of an HMD <NUM> would see while looking at the display or through the lenses of the HMD <NUM>). For convenience, this and further examples of the disclosure may use an HMD <NUM>, but it is understood that any suitable electronic device <NUM> may be used.

As shown in <FIG>, a marker <NUM> may be placed within an environment. In this example, the marker <NUM> uses a visible light LED, but it is understood that a marker <NUM> using an invisible spectrum LED (such as an IR LED) may be used. Alternatively, a passive marker <NUM> may be used. <FIG> represents a scenario in which the HMD <NUM> does not receive any interpretable data from the marker <NUM>, or alternatively a scenario where a user is not wearing an HMD <NUM> at all (i.e., a scenario in which a person is viewing the environment with the naked eye). Accordingly, the marker <NUM> may be plainly visible with no generated content overlaid on the environment as a result. A similar scenario is illustrated in <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG> with various embodiments of markers <NUM> or <NUM>.

As shown in <FIG>, data may be interpreted from the marker <NUM> and may be used by the HMD <NUM> to generate content, which is displayed as object <NUM> within the environment. In this example the object <NUM> is a static 3D object, but it is understood that any suitable object may be generated based on data transmitted by the marker <NUM>. For example, the object <NUM> may be an animated 2D or 3D object, a video, a text display, or the like. Furthermore, while the object (or content) <NUM> may be displayed directly over the location of the marker <NUM> in this example, it is understood that the data transmitted by the marker <NUM> may include metadata that instructs the HMD <NUM> to overlay the object at a different position within the environment.

In some embodiments, data transmitted by the marker <NUM> to the HMD <NUM> may include one or more of a start code, a third party device identifier or address, image data, or metadata. For example, the start code may be a binary sequence in the communication from the marker <NUM> that indicates to the HMD <NUM> which type of data that is contained in the communication from the marker <NUM>. That is, the start code may indicate that the data is one or more of metadata, image data, or a server address. The start code may also function to indicate to the HMD <NUM> that data begins to be transmitted after the start code. This may be useful if the marker <NUM> is continuously broadcasting the same block of data, as it may allow an HMD <NUM> that begins to receive the transmission even if it moves into view of the marker <NUM> midway through the marker <NUM>'s data transmission.

<FIG> and <FIG> illustrate examples of interpretation of communications from a portable marker that may be configured to attach to a person's clothing according to embodiments of the disclosure. View <NUM> is an exploded view of a portion <NUM> of <FIG> that illustrates a portable marker <NUM> attached to a chest pocket of a shirt. Such a marker <NUM> may be configured to clip or pin onto clothing. In some embodiments, marker <NUM> may be configured to attach to the interior of a piece of clothing, and to use a wavelength of light (such as IR light) that is not significantly attenuated by clothing, thereby providing the functionality of the marker <NUM> without affecting the aesthetic appearance of the clothing.

<FIG> illustrates an application of a portable marker <NUM> that may be attached to a person's clothing. In this example, the marker <NUM> optically may transmit to an HMD <NUM> data that includes image data for a virtual costume <NUM> and metadata indicating the position at which to display the virtual costume <NUM> with respect to the marker <NUM> (e.g., position a center of the virtual costume <NUM> a distance x to the left of the marker <NUM> and a distance y above the marker <NUM>). In response to receiving the data from marker <NUM>, the HMD <NUM> may generate and display the virtual costume <NUM> as specified by the marker <NUM>. In other embodiments, such a marker may be used to perform a privacy function. In particular, the marker <NUM> may instruct an HMD <NUM> to blur out a location corresponding to the face of the owner of the marker <NUM> in any photographs or video feed that the HMD <NUM> takes that include the marker <NUM>.

<FIG> and <FIG> illustrate examples of interpretation of communications from a marker <NUM> that is configured to attach to a wall of a room according to embodiments of the disclosure. View <NUM> is an exploded view of a portion <NUM> of <FIG> that illustrates a portable marker <NUM> attached to a column. In such embodiments, the marker <NUM> may be configured to attach to the column or wall with a one-time-use or reusable adhesive. The marker <NUM> may then be easily removed and repositioned to another location.

<FIG> illustrates an example application of a marker <NUM> that is attached to a wall, column, or other similar location within a room. In this example, marker <NUM> may optically transmit to an HMD <NUM> data that includes a server address for a video con-ferencing service (e.g., an address for a server <NUM>). The HMD <NUM> may interpret the server address, connects to the server, and begins an AR video conference. Based on data received from the server <NUM> and from the marker <NUM>, the HMD <NUM> may generate content <NUM> for display overlaid on the location of the marker <NUM>. In this example, the content <NUM> may represent the face of a person that is participating in the video conference. In other embodiments, the content <NUM> may include any suitable representation of a video conference, such as a video feed of a room at the other end of the video conference, virtual avatars of other participants in the video conference, or the like.

<FIG> illustrate examples of interpretation of communications from a marker that is used to generate a virtual computer desktop display on an HMD according to embodiments of the disclosure. As shown in <FIG>, a marker <NUM> may be presented on a wall or other surface of an environment. An HMD <NUM> recognizes the marker <NUM> and interprets it as indicating that a virtual computer desktop <NUM> may be generated and displayed at the location of the marker <NUM>, as shown in <FIG>. In some embodiments, the marker <NUM> may include metadata that indicates a size and position of the virtual computer desktop <NUM> relative to the marker <NUM>. While a QR code is illustrated as marker <NUM>, it is understood that any other suitable passive marker <NUM> or active marker <NUM> may be used. The content of the virtual computer desktop <NUM> may be generated independently by the HMD <NUM> or may be indicated by the marker <NUM>. Additionally, the marker <NUM> may indicate a server address for a server <NUM>, and the content of the virtual computer desktop <NUM> may be generated by the server <NUM> and transmitted to the HMD <NUM>.

As shown in <FIG>, a marker <NUM> may be presented on a display of an electronic device <NUM>. The electronic device <NUM> may dynamically generate and display the marker <NUM> to communicate with an HMD <NUM>. In this example, the electronic device <NUM> may be docked in a docking station <NUM>. The electronic device <NUM> may generate the marker <NUM> in response to recognize that it is docked in the docking station <NUM>. An HMD <NUM> may recognize the marker <NUM> and interprets it as instructions indicating that a virtual computer desktop <NUM> may be generated and displayed as shown in <FIG>. In the example of <FIG>, the electronic device <NUM> and its docking station <NUM> may be located on the surface of a desk, and the marker <NUM> may include metadata that indicates that the virtual computer desktop <NUM> may be displayed at a position that appears to float above the desk and in front of the electronic device <NUM> and its docking station <NUM>. Accordingly, the HMD <NUM> may generate and display the virtual computer desktop <NUM> at the designated location. In some embodiments, the marker <NUM> may be dynamically modified by the electronic device <NUM> to transmit information that causes the HMD <NUM> to adjust parameters of the virtual computer desktop <NUM>, for example by changing its position or size. In some embodiments, the marker <NUM> may include an address of a server <NUM> or information that specifies a communication channel to the electronic device <NUM> (using, e.g., a BLUETOOTH, WIFI direct, or other communication protocol). The HMD <NUM> may then connect to the server <NUM> or the electronic device <NUM> to receive information used to generate content for display on the virtual computer desktop <NUM>.

<FIG> and <FIG> illustrate examples of a marker that is used to generate a virtual computer desktop display on an HMD, similar to the examples of <FIG> and <FIG>, using an LED <NUM> rather than a passive marker <NUM>. As illustrated in <FIG>, the LED <NUM> may be integrated with (or coupled to) the docking station <NUM> and may perform a function similar to or identical to an active marker <NUM> as described elsewhere in the disclosure. The LED <NUM> may, accordingly, optically broadcast data to an HMD <NUM> that the HMD <NUM> may interpret as instructions to generate and display a virtual computer desktop <NUM> as shown in <FIG>. The parameters used to generate the virtual computer desktop <NUM> of <FIG> may be similar to those used to generate the virtual computer desktop <NUM> of <FIG>.

In some embodiments, the LED <NUM> may be controlled by the electronic device <NUM> via the docking station <NUM>. In other embodiments, the LED <NUM> may be controlled independently by the docking station <NUM>, and the docking station <NUM> may trigger operation of the LED <NUM> based on connection of an electronic device <NUM>. The data broadcast by the LED <NUM> may also include an SSID associated with the docking station <NUM>, thereby providing the HMD <NUM> with information needed to connect to the docking station <NUM> via another communication protocol (e.g., BLUETOOTH, WIFI direct, etc.), and the docking station <NUM> may provide the HMD <NUM> with parameters used by the HMD <NUM> to generate the virtual computer desktop <NUM>, with content for display on the virtual computer desktop <NUM>, or both. As discussed elsewhere in the disclosure, it is understood that the LED <NUM> may be a visible light LED, an IR LED, or any other suitable light source.

<FIG> illustrate examples of interpretation of communications from a marker that is used to generate content for display on an HMD subject to detection, by the HMD, of other events in the environment. <FIG> illustrates a marker <NUM> in an environment, similar to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>. <FIG> illustrates an embodiment similar to that of <FIG>, in the case that the marker <NUM> may transmit information including a third party server ID to the HMD <NUM>. The HMD <NUM> connects to the server (e.g., a server <NUM>) and may receive further data that the HMD <NUM> uses to generate and display content <NUM>, as described elsewhere in the disclosure. In this embodiment, the server <NUM> may additionally provide that a different set of content be generated and displayed if the HMD <NUM> detects one or more particular objects or events in the environment. For example, as illustrated in <FIG>, the information received from the server <NUM> instructs the HMD <NUM> that if the HMD <NUM> detects a hand gesture <NUM> then a virtual computer desktop <NUM> may be displayed instead of the content <NUM>. The information received from the server <NUM> may include more than two alternative cases (e.g., the information may include a number of different pieces of content for display based on a number of different gestures, or based on the presence of other physical objects such as signs or furniture that are placed within the environment).

In some embodiments, more than one marker <NUM> or <NUM> may be present in an environment, and an HMD <NUM> may detect and interpret multiple markers <NUM> or <NUM> simultaneously. For example, as illustrated in <FIG>, multiple active markers <NUM> may be present on a city street, each providing data or metadata that instructs the HMD <NUM> to display text or other content. Additionally, multiple passive markers <NUM> may be present in the environment, and the HMD <NUM> may interpret them to generate additional content for display. In this embodiment, the passive markers <NUM> include objects in the environment such as road lane markings, bike lane markings, and the sky. The HMD <NUM> may interpret images or video captured with one or more cameras of the HMD <NUM> (e.g., a camera <NUM>) and demultiplexed using computer vision techniques.

<FIG> illustrates an example embodiment in which markers communicate with each other to adjust the data that they transmit to an HMD. In the example of <FIG>, a number of vehicles <NUM> on a road are equipped with markers <NUM> that are visible to drivers of other vehicles <NUM>. In this embodiment the markers <NUM> are shown on the rear of each vehicle <NUM>, but any number of markers may be used on different locations on a vehicle <NUM> (such as on the front and sides of the vehicle <NUM>). The markers <NUM> may interface with onboard computing systems of the vehicle <NUM> and communicate with each other using wireless transmission protocols such as IEEE <NUM>, WIFI direct, or other suitable protocols. In some embodiments, the markers <NUM> communicate information pertaining to their vehicle <NUM> to markers <NUM> of other vehicles <NUM>, which may further relay that information to other markers <NUM> of other vehicles <NUM>, thereby building a body of traffic information in each marker <NUM> for all vehicles <NUM> within proximity to each other. This information may be used to either update onboard computing systems of other vehicles <NUM> or to optically transmit the traffic information to HMDs <NUM> of drivers of other vehicles <NUM>.

<FIG> illustrates another example of an embodiment in which markers communicate with each other to adjust the data that they transmit to an HMD. In this embodiment, a number of people are wearing clothing-attached markers <NUM> that communicate with each other using wireless transmission protocols such as IEEE <NUM>, WIFI direct, or other suitable protocols. The markers <NUM> may, for example, transmit to each other information including details of the clothing items to which the markers <NUM> are attached or other information about the owner of the clothing. The markers <NUM> may use that information for various applications, such as social networking (e.g., a marker <NUM> may interface with an HMD <NUM> to provide a user of the HMD <NUM> with information that allows them to compare clothing brands worn by other people wearing markers <NUM>). Additionally or alternatively, each marker <NUM> may optically transmit its information to HMDs <NUM>, thereby providing the HMDs <NUM> with information to display that is related to the clothing (e.g., a brand name of the clothing).

<FIG> illustrates an example method for a marker-based augmented reality system according to the disclosure. The method may be performed using, for example, an HMD <NUM> and markers <NUM> and <NUM>, although it is understood that any suitable electronic devices <NUM> and <NUM> may be used with the method.

Beginning at step <NUM>, an HMD <NUM> may acquire image data from a camera, for example a camera <NUM>. This image data may include still images or videos of a real-world environment around the HMD <NUM>. In particular, the image data may include images of the real-world environment that is visible on a display of the HMD <NUM>.

At step <NUM>, the HMD <NUM> may analyze the image data to identify at least one marker (such as a marker <NUM> or <NUM>). The marker may be associated with a physical location in the real-world environment surrounding the HMD <NUM>. For example, the marker may be a passive marker <NUM> that is written, drawn, or printed on a surface in the real-world environment, and the camera <NUM> of the HMD <NUM> may be an RGB camera. In another example, the marker may be an active marker <NUM> that may be attached to or embedded in a surface of the real-world environment. In this example, the marker <NUM> may use a visible spectrum light source and the camera <NUM> may be an RGB camera, or the marker <NUM> may use an IR spectrum light source and the camera <NUM> may be an IR camera.

At step <NUM>, the HMD <NUM> may determine information represented via the at least one marker. The information may include at least one of content data, metadata, or server data. Content data may include details of content to be generated and displayed on a display of the HMD <NUM>. Metadata may include parameters of the content such as size and location of the content. In some embodiments, the content data and metadata may be combined and interpreted as a whole by the HMD <NUM> to determine the content to display as well as size and location of the content. When the information may include server data (e.g., a server address for a server <NUM>), the HMD <NUM> may connect to the identified server using any appropriate communication protocol, and may receive from the server information that is interpreted to generate content. That is, the HMD <NUM> may receive content data and metadata from the server.

At step <NUM>, the HMD <NUM> may generate content based on at least one of the content data, the metadata, or the server data. For example, the HMD <NUM> may interpret content data to determine the appearance of content (e.g., a 3D object, a 2D object, a text string, a video, etc.). The HMD <NUM> may interpret the metadata to determine other parameters of the content, such as a size of the content or a location relative to the marker (or relative to an object in the real-world physical environment) at which to display the content as an overlay on a displayed representation of the real-world physical environment. As discussed above, the HMD <NUM> may make these interpretations from data provided by a server that the HMD <NUM> identifies from server data.

At step <NUM>, the HMD <NUM> may present, via a display <NUM> of the HMD <NUM>, the content overlaying a displayed representation of the real-world environment surrounding the HMD <NUM>. The content may be presented at one or more locations in the displayed representation of the real-world environment relative to the physical location that is associated with the at least one marker. The presentation location may be determined as discussed in step <NUM>.

At step <NUM>, the HMD <NUM> may analyze the image data to identify a gesture in the image data. In the case that the information determined from the marker includes an indication that content be modified based on a gesture, the HMD <NUM> may analyze the image data for such a gesture. For example, the marker may indicate to the HMD <NUM> that certain content be displayed when an open hand is identified within a certain distance of the camera (e.g., when an open hand occupies a certain amount of an image).

At step <NUM>, the HMD <NUM> may modify, based on the identified gesture, the content overlaying the displayed representation of the real-world environment. For example, when the gesture is identified, the HMD <NUM> may change the content that is presented from a static 2D object to a video, or may connect to a server <NUM> and stream content from the server <NUM>.

At step <NUM>, the HMD <NUM> may generate a message for transmission to the at least one marker, and at step <NUM> the HMD <NUM> may transmit, to the at least one marker, the message. The HMD <NUM> may use any appropriate communication protocol to transmit the message to the marker. For example, the HMD <NUM> may use BLUETOOTH or WIFI direct to transmit the message to the marker. The message may be a response to a request for information (e.g., an identifier of the HMD <NUM>, account information of the HMD <NUM> for a service related to the marker, or the like) from the marker.

Claim 1:
A method of an electronic device (<NUM>) comprising:
acquiring an image including a marker, using a camera (<NUM>) of the electronic device (<NUM>);
identifying first information optically transmitted by the marker, the first information indicating a request to transmit second information related to the electronic device (<NUM>);
transmitting the second information to the marker using a wireless communication interface of the electronic device (<NUM>);
identifying third information optically transmitted by the marker in response to transmission of the second information, the third information including at least one of content data, metadata, or server data;
generating content based on the third information; and
displaying, on a display (<NUM>) of the electronic device (<NUM>), the content overlaying the image,
wherein the content is displayed on at least one portion corresponding to a location in which the marker is located in the image.