Patent Description:
Portable networked devices with displays, such as tablets and smartphones have, within the course of little over a decade, gone from being luxury items to supplanting desktop computers, fax machines and landline telephones as the default device for personal and business communications. The smartphone's dominance as the primary communication interface for millions of people is reflected by the frequency with which many users check their devices and the steady rise in screen time among users.

In addition to the emergence of portable networked devices as a dominant communication medium, the past decade has also witnessed the emergence of new display technologies, including virtual reality (VR) and augmented reality (AR), which harness the possibilities of small form processors and lightweight displays to provide displays which supplant or augment the feed of visual information to viewers' eyes from the physical world.

The ubiquity of portable devices, such as smartphones and the emergence of new display technologies, such as AR and VR present numerous opportunities and technological challenges associated with improving the functionality of smartphones and other portable devices as tools for communication by incorporating AR and VR technologies.

<CIT> relates to discloses an augmented user device having a display for overlaying virtual objects onto objects in a real scene. <CIT> discloses a method for mobile augmented reality personalization.

<CIT> discloses an augmented reality system with visibility and persistence control.

This disclosure has been made to address at least the above problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of this disclosure is to provide an apparatus and a method for augmented reality.

According to the present invention, an augmented reality apparatus and a method for providing a personalized augmented reality (AR) display are provided as defined by the appended set of claims.

Other technical features may be readily apparent to one skilled in the art from the following Figures, descriptions, and claims.

The phrase "associated with, "as well as derivatives thereof, means to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, have a relationship to or with, or the like.

Those of ordinary skill in the art should understand that in many if not most instances, such definitions apply to prior as well as current uses of such defined words and phrases.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:.

<FIG>, discussed below, and the various embodiments used to describe the principles of this disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of this disclosure may be implemented in any suitably arranged wireless communication system.

<FIG> illustrates a non-limiting example of a device for augmented reality-based communication according to some embodiments of this disclosure. The embodiment of device <NUM> illustrated in <FIG> is for illustration only, and other configurations are possible. However, suitable devices come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular implementation of a device. For example, device <NUM> may be implemented, without limitation, as a smartphone, a wearable smart device (such as a smart watch), a tablet computer, or as a head-mounted display.

As shown in the non-limiting example of <FIG>, the device <NUM> includes a communication unit <NUM> that may include, for example, a radio frequency (RF) transceiver, a Bluetooth® transceiver, or WI-FI® transceiver, etc., transmit (TX) processing circuitry <NUM>, a microphone <NUM>, and receive (RX) processing circuitry <NUM>. The device <NUM> also includes a speaker <NUM>, a main processor <NUM>, an input/output (I/O) interface (IF) <NUM>, input/output device(s) <NUM>, and a memory <NUM>. The memory <NUM> includes an operating system (OS) program <NUM> and one or more applications <NUM>.

Applications <NUM> can include games, social media applications, applications for geotagging photographs and other items of digital content, virtual reality (VR) applications, augmented reality (AR) applications, operating systems, device security (e.g., anti-theft and device tracking) applications or any other applications which access resources of device <NUM>, the resources of device <NUM> including, without limitation, speaker <NUM>, microphone <NUM>, input/output devices <NUM>, and additional resources <NUM>. According to some embodiments, applications <NUM> include applications which can consume image data from physical objects in a field of view of a camera of electronic device <NUM> and provide AR or VR content through a display of device <NUM>, or a display of a separate device.

The communication unit <NUM> may receive an incoming RF signal, for example, a near field communication signal such as a Bluetooth® or Wi-Fi™ signal. The communication unit <NUM> can down-convert the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry <NUM>, which generates a processed baseband signal by filtering, decoding, or digitizing the baseband or IF signal. The RX processing circuitry <NUM> transmits the processed baseband signal to the speaker <NUM> (such as for voice data) or to the main processor <NUM> for further processing (such as for web browsing data, online gameplay data, notification data, or other message data). Additionally, communication unit <NUM> may contain a network interface, such as a network card, or a network interface implemented through software.

The TX processing circuitry <NUM> receives analog or digital voice data from the microphone <NUM> or other outgoing baseband data (such as web data, e-mail, or interactive video game data) from the main processor <NUM>. The TX processing circuitry <NUM> encodes, multiplexes, or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The communication unit <NUM> receives the outgoing processed baseband or IF signal from the TX processing circuitry <NUM> and up-converts the baseband or IF signal to an RF signal for transmission.

The main processor <NUM> can include one or more processors or other processing devices and execute the OS program <NUM> stored in the memory <NUM> in order to control the overall operation of the device <NUM>. For example, the main processor <NUM> could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit <NUM>, the RX processing circuitry <NUM>, and the TX processing circuitry <NUM> in accordance with well-known principles. In some embodiments, the main processor <NUM> includes at least one microprocessor or microcontroller.

The main processor <NUM> is also capable of executing other processes and programs resident in the memory <NUM>. The main processor <NUM> can move data into or out of the memory <NUM> as required by an executing process. In some embodiments, the main processor <NUM> is conFigured to execute the applications <NUM> based on the OS program <NUM> or in response to inputs from a user or applications <NUM>. Applications <NUM> can include applications specifically developed for the platform of device <NUM>, or legacy applications developed for earlier platforms. Additionally, main processor <NUM> can be manufactured to include program logic for implementing methods for monitoring suspicious application access according to certain embodiments of the present disclosure. The main processor <NUM> is also coupled to the I/O interface <NUM>, which provides the device <NUM> with the ability to connect to other devices such as laptop computers and handheld computers. The I/O interface <NUM> is the communication path between these accessories and the main processor <NUM>.

The main processor <NUM> is also coupled to the input/output device(s) <NUM>. The operator of the device <NUM> can use the input/output device(s) <NUM> to enter data into the device <NUM>. Input/output device(s) <NUM> can include keyboards, head mounted displays (HMD), touch screens, mouse(s), track balls or other devices capable of acting as a user interface to allow a user to interact with electronic device <NUM>. In some embodiments, input/output device(s) <NUM> can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device.

Input/output device(s) <NUM> can include one or more screens, which can be a liquid crystal display, light-emitting diode (LED) display, an optical LED (OLED), an active matrix OLED (AMOLED), or other screens capable of rendering graphics.

The memory <NUM> is coupled to the main processor <NUM>. According to certain embodiments, part of the memory <NUM> includes a random access memory (RAM), and another part of the memory <NUM> includes a Flash memory or other read-only memory (ROM). Although <FIG> illustrates one example of a device <NUM>. Various changes can be made to <FIG>.

For example, according to certain embodiments, device <NUM> can further include a separate graphics processing unit (GPU) <NUM>.

According to certain embodiments, electronic device <NUM> includes a variety of additional resources <NUM> which can, if permitted, be accessed by applications <NUM>. According to certain embodiments, additional resources <NUM> include an accelerometer or inertial motion unit <NUM>, which can detect movements of the electronic device along one or more degrees of freedom. Additional resources <NUM> include, in some embodiments, a dynamic vision sensor (DVS) <NUM>, one or more cameras <NUM> of electronic device <NUM>.

Although <FIG> illustrates one example of a device <NUM> for performing semi-dense depth estimation, various changes may be made to <FIG>. For example, the device <NUM> could include any number of components in any suitable arrangement. In general, devices including computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example of an architecture <NUM> incorporating a personalized communication augmented reality (PCAR) framework according to some embodiments of this disclosure. The embodiment of the architecture <NUM> shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

For ease of explanation, in the non-limiting example of <FIG>, architecture <NUM> adheres to a client-host model between host platform <NUM> and client platform <NUM>, with the components of a PCAR framework <NUM> implemented as software on a logical layer between an AR application layer <NUM> and an OS/device layer <NUM>. Other embodiments, such as embodiments utilizing a peer-to-peer relationship (as opposed to a host-client) between entities, or where the components of PCAR framework <NUM> are implemented as hardware, or a combination of hardware and software, are possible and within the contemplated scope of this disclosure.

According to certain embodiments, host platform <NUM> comprises a database server, which is communicatively connected (for example, via the internet, a wireless network, or an intranet) to client platform <NUM>. As one non-limiting example, host platform <NUM> comprises networked computer, such as a database server running an instance of MySQL to host a PCAR database <NUM>.

According to various embodiments, PCAR database <NUM> comprises, at a minimum, a repository of information associating items of augmented reality (AR) content with identifiers of objects. According to some embodiments, PCAR database <NUM> also includes data corresponding to a schema comprising object signatures (for example, identifiers of a particular object - for example, a chair at a known location), a user identifier (for example, an ID of a user who created an association between item(s) of AR content and an object, visibility data (for example, data specifying the permissions of other user to view the AR content whose association was created by a user identified in the user identifier field, object location, and the expiration time of the association between the item of AR content and the object.

In certain embodiments, the associations between item(s) of AR content and objects are determined by a user (for example, as shown in <FIG> of this disclosure). In various embodiments, the logic for creating associations between objects and item(s) of content reside within the AR apparatus itself. As a non-limiting example, a PCAR service running on client platform <NUM> could use information of the client platform itself (for example, an international mobile equipment identifier (IMEI) associated with client platform, or system information, such as an identifier of the operating system) to create associations between objects (for example, a charger for the device) and item(s) of AR content. As one example, a PCAR service for client platform <NUM> may receive from a PCAR database <NUM>, information associating an item of AR content provided by a mobile carrier for client platform <NUM> with an object. As an example, an item of AR content comprising an advertisement for an upgraded mobile service plan may be presented when the client platform <NUM> receives image data associated with the charger or power supply for client platform <NUM>. In this example, the itemAR content is selected to be associated with the object based on a piece of information regarding the client platform, such as the identity of the wireless carrier to which the client platform is a subscriber.

In various embodiments according to this disclosure, the associations between items of AR content and objects are driven by, and pushed out, by host platform <NUM> in conjunction with PCAR database <NUM>. Thus, certain embodiments according to this disclosure support communications between client devices (for example, as shown in <FIG>), as well as end-to-end communications between a host platform <NUM> and <NUM>. In some embodiments according to this disclosure, host platform <NUM> receives information (for example, from network access points, such as Wi-Fi routers or ZigBee transceivers) regarding the identity of devices within visual range of an object which is an augmentation target. Based on this information, host platform <NUM> pushes out association information maintained in PCAR database <NUM> to some or all of the identified devices. As an illustrative example, the augmentation shown to users in <FIG> of this disclosure may be selected on device information obtained by host platform <NUM>. As one example, the information regarding the television set shown in <FIG> might, based on device identity information, only be pushed out to devices manufactured by the maker of the television.

As shown in the non-limiting example of <FIG>, client platform <NUM> comprises one or more electronic devices (for example, a head mounted display and a smartphone, or electronic device <NUM> of <FIG>) which are communicatively connected to host platform <NUM>. In this illustrative example, on the client side, architecture <NUM> comprises three logical layers - a device / operating system (OS) layer <NUM>, a middle layer comprising PCAR framework <NUM>, and an AR application layer <NUM>.

In certain embodiments according to this disclosure, device / OS layer <NUM> comprises the hardware, software or combination of hardware and software of client platform <NUM> which obtains image data <NUM> and location data <NUM>.

In some embodiments, image data <NUM> comprises data output by from one or more externally-oriented sensors (for example, a CMOS video camera or a dynamic vision sensor (DVS), conFigured to produce a comparatively less data-intensive representation of a field of view by capturing changes in the intensity of received light at the pixels of the sensor). As used in this disclosure, the term "externally-oriented"encompasses directions away from a direction of projection of a display for presenting AR content in an AR apparatus. As a non-limiting example, for an AR device to be worn over a user's eyes, a display in the AR device projecting content towards the user's eyes would be "internally-oriented" or "internally facing," while a camera of the device pointed at the world in front of the user, would be "externally oriented. " In some embodiments, the one or more sensors generating image data <NUM> may be components of an augmented reality apparatus (for example, a headset, "smart glasses" or a smartphone).

In some embodiments, location data <NUM> comprises positioning data (for example, data assigning a specific geospatial coordinate to a sensor), as well as metadata of a specific location (for example, data associating a particular geospatial location and altitude with a particular room in a building, as well as attributes of the location, such as the presence of IoT devices or other devices with which an AR apparatus can communicate). In the non-limiting example of <FIG>, location data <NUM> may be obtained from a GPS sensor, an IMU (for example, IMU <NUM> in <FIG>), and a communication unit (for example, communication unit <NUM> in <FIG>) for communicating with networked devices at a particular location.

According to various embodiments, data from device / OS layer <NUM>, such as image data <NUM> and location data <NUM> are provided to PCAR framework <NUM>.

According to various embodiments, PCAR framework <NUM> catalyzes improvements in the functionality of certain electronic devices, such as smartphones or devices with head mounted displays, such as "smart glasses," by allowing users to communicate by assigning items of virtual reality or augmented reality content to locations and objects (including, without limitation, native objects)within the physical world. As used in this disclosure, the term "native object" encompasses an object whose form or appearance has not been modified, for example, by affixing a sticker with a QR code, to enhance its recognition and identification in a set of digital image data. As discussed elsewhere herein, embodiments according to the present disclosure facilitate the creation, assignment and management of AR and VR content by AR apparatus users, thereby allowing commonly-used devices, such as smartphones, to operate as platforms for managed AR and VR based communication between users. As such, embodiments as disclosed and claimed, extent the functionality of AR from being a technology merely allowing users of AR apparatus to consume content (for example, by seeing AR content created and selected for a general audience, such as AR content associated with a landmark viewed through an AR apparatus), to being a technology which allows users of AR apparatus to consume, produce, and manage content, thereby allowing AR apparatus to function as a tool for communication.

Referring to the non-limiting example of <FIG>, PCAR framework <NUM> comprises signature engine <NUM>, object recognition engine <NUM>, and augmentation engine <NUM>. In this non-limiting example, each of signature engine <NUM>, object recognition engine <NUM> and augmentation engine <NUM> are implemented as program code executed by a processor of client platform <NUM>. In certain embodiments, the components of PCAR framework may be implemented as hardware (for example, by an image recognition processor) or across multiple computing platforms (for example, a cloud based platform connected by an API to the client platform <NUM>).

According to certain embodiments, object recognition engine <NUM> obtains image data from covering a field of view of an externally-oriented camera or other visual sensor (for example, a DVS sensor, such as DVS sensor <NUM> in <FIG>) trained on a field of view in which objects which can be recognized as augmentation objects may be found. In certain embodiments, object recognition engine <NUM> processes the image data to detect features in the image data. According to various embodiments, the object recognition engine <NUM> performs object recognition using a binary descriptor based object recognition technique, including, for example, the binary robust independent elementary features (BRIEF), binary robust invariant scalable keypoints (BRISK), fast retina keypoints (FREAK), or oriented fast and rotated BRIEF (ORB). According to various embodiments, object recognition engine <NUM> identifies one or more objects within the field of view of an externally facing sensor (such as an CMOS camera or DVS sensor) and provides a descriptor or other data associated with the identification of the first object to augmentation engine <NUM> or signature engine <NUM>.

According to various embodiments, signature engine <NUM> receives descriptors of recognized objects in a field of view (for example, a camera field of view of an AR apparatus operating as client platform <NUM>) from object recognition engine <NUM>. In certain embodiments, object recognition engine <NUM> operates continuously, and continuously scans image data from a field of view for objects to recognize. According to some embodiments, object recognition engine <NUM> may, to conserve battery or other resources, suspend or reduce its operation when a threshold condition (for example, it stops recognizing unique objects at a given location) is met. Additionally, in various embodiments, signature engine <NUM> also receives location data <NUM> from one or more entities (such as a GPS sensor, or wireless communication unit). According to various embodiments, signature engine <NUM> generates, for each object associated with a descriptor generated by object recognition engine <NUM>, and location data <NUM>, a signature for the object. In certain embodiments, signature engine <NUM> generates the signature based on just the descriptor generated by object recognition engine <NUM>. According to various embodiments, the signature may be unique. According to some embodiments, a signature consists of a hash of the descriptor and location. In the non-limiting example of <FIG>, signature engine <NUM> periodically performs comparisons of generated signatures against signatures maintained in a PCAR database <NUM> maintained on host platform <NUM>. According to various embodiments, when a match between a signature at client platform <NUM> and host platform <NUM> is determined, signature engine <NUM> receives augmentation data. According to various embodiments, augmentation data encompasses, at a minimum, data associated with an AR display, the AR display being associated with the object for which a signature match was determined.

In various embodiments according to this disclosure, PCAR framework <NUM> includes augmentation engine <NUM>. In certain embodiments, augmentation engine <NUM> supports at least two modes of operation comprising a normal mode, and an augmentation mode. In some embodiments, PCAR framework <NUM> allows the client platform <NUM> to simultaneously operate in both normal mode and augmentation mode. In certain embodiments, client platform <NUM> may operate exclusively in either a normal mode or an augmentation mode. According to the non-limiting example of <FIG>, in the normal mode, augmentation engine <NUM> receives augmentation data for objects which have been recognized, and whose signature matches a signature associated with an AR augmentation in PCAR database <NUM>. In certain embodiments, augmentation engine <NUM> receives outputs from object recognition engine <NUM>, including, for example, data indicating the location of features of the object (for example, corners, legs, etc.) In various embodiments, augmentation engine <NUM> provides an AR application data comprising, or based on, the augmentation data and the data from object recognition engine <NUM>. In various embodiments according to this disclosure, in normal mode, augmentation engine <NUM> also receives certain inputs provided through AR application (for example, an input indicating that an item of AR content has been viewed), which are passed through signature engine <NUM> to PCAR database <NUM>.

Referring to the non-limiting example of <FIG>, in an augmentation mode, augmentation engine <NUM> receives inputs from AR application <NUM> for associating an item of AR content with a recognized object in a field of view of a sensor communicatively connected to client platform <NUM>. In certain embodiments, the augmentation engine formats the received inputs according to a schema of PCAR database <NUM> into augmentation data, which is further associated with an object signature for the recognized object (for example, a signature generated by signature engine <NUM>) and passed to PCAR database <NUM>.

As shown in the non-limiting example of <FIG>, in architecture <NUM>, client platform <NUM> also comprises an AR application <NUM>. According to certain embodiments, AR application <NUM> provides AR content for display on an internally-facing display of client platform <NUM> (for example, an AR apparatus communicatively connected to host platform <NUM>). In some embodiments, the internally-facing display of client platform <NUM> provides an AR field of view, which comprises a portion of a camera field of view of a camera or other sensor providing image data <NUM>. In certain embodiments, the AR field of view is a subset of the camera field of view. In certain embodiments, the AR field of view comprises an area of overlap with the camera field of view. In various embodiments, the AR field of view is co-extensive with the camera field of view. In some embodiments, the camera field of view is a subset of the AR field of view. In certain embodiments, AR application <NUM> displays both the AR content and a visual identification (for example, a box or callout) around the recognized object that is the augmentation target with which the AR content is associated. According to various embodiments, AR application <NUM> provides a user interface for assigning AR content to the augmentation target.

<FIG> illustrates an example of an augmented reality (AR) apparatus <NUM> according to one or more embodiments of this disclosure. According to various embodiments, AR apparatus may function as a client device (for example, client platform <NUM> in <FIG>) in communicative contact with a host system (for example, host platform <NUM> in <FIG>) which maintains one or more databases of object signatures and augmentation data associated with object signatures. According to some embodiments, AR apparatus <NUM> may operate as an accessory device to another device (for example, a smartphone), which in turn acts as a client device to a host system maintaining associations between object signatures and augmentation data. In certain embodiments, associations between object signatures and augmentation data may be maintained across a variety of peer devices, rather than according to a host-client architecture.

Referring to the non-limiting example of <FIG>, AR apparatus <NUM> includes an externally oriented camera <NUM>. For the purposes of explaining this non-limiting example, the arrow <NUM> is provided. Arrow <NUM> points externally, towards a field of view away from the direction of projection of an internal-facing display of AR apparatus <NUM>. According to various embodiments, externally oriented camera <NUM> is an RGB digital video camera (for example, a camera using a CMOS sensor). According to some embodiments, externally oriented camera <NUM> is a camera capable of detecting light at wavelengths outside the visible range of the human eye (for example, infrared). In certain embodiments, externally oriented camera <NUM> is a dynamic vision sensor (DVS), which provides an event stream of changes in the intensity of light received at pixels of a sensor of the DVS. In this non-limiting example, externally-oriented camera <NUM> generates image data, either as an event stream or as discrete image frames, which are passed to a PCAR framework (for example, PCAR framework <NUM> in <FIG>).

Referring to the non-limiting example of <FIG>, AR apparatus <NUM> includes an externally oriented camera <NUM>. For the purposes of explaining this non-limiting example, the arrow <NUM> is provided. Arrow <NUM> points externally, towards a field of view away from the direction of projection of an internal-facing display of AR apparatus <NUM>. According to various embodiments, externally oriented camera <NUM> is an RGB digital video camera (for example, a camera usinga CMOS sensor). According to some embodiments, externally oriented camera <NUM> is a camera capable of detecting light at wavelengths outside the visible range of the human eye (for example, infrared). In certain embodiments, externally oriented camera 305is a dynamic vision sensor (DVS), which provides an event stream of changes in the intensity of light received at pixels of a sensor of the DVS. In this non-limiting example, externally-oriented camera <NUM> generates image data, either as an event stream or as discrete image frames, which are passed to a PCAR framework (for example, PCAR framework <NUM> in <FIG>).

According to various embodiments, AR apparatus <NUM> includes second camera <NUM>. In some embodiments, second camera <NUM> is an externally-oriented camera of the same type as externally-oriented camera <NUM>, thereby forming a stereoscopic pair which can generate image data comprising depth estimation. In certain embodiments, second camera <NUM> is an externally-oriented camera with a different sensor type than externally-oriented camera <NUM>. For example, in some embodiments, to extend battery life and minimize processor usage, externally-oriented camera <NUM> is a DVS sensor, and second camera <NUM> is a CMOS type camera, which, while less efficient than a DVS sensor, can provide additional image data (for example, data regarding colors and elements of a scene whose brightness may not change at a level detectable by a DVS sensor) that is useful for object recognition. According to various embodiments second camera <NUM> is an internally-facing camera, which tracks the motion of a user's eyes, and by implication, the direction of the user's gaze. Gaze tracking can be used to support foveal rendering of items of AR content, which can conserve battery and processor resources by rendering items of AR content away from a viewer's gaze at lower resolutions.

According to certain embodiments, AR apparatus <NUM> includes processor <NUM> and memory <NUM>. In certain embodiments, memory <NUM> contains program code, which when executed by processor <NUM>, causes AR apparatus <NUM> to execute an AR application (for example, AR application <NUM> in <FIG>), or one or more components of a PCAR framework (for example, PCAR framework <NUM> in <FIG>).

Referring to the non-limiting example of <FIG>, AR apparatus <NUM> includes an inertial measurement unit <NUM>, which generates location data associated with the motion of AR apparatus <NUM> along one or more degrees of freedom. In certain embodiments, data output from IMU <NUM> may be used for positioning (for example, to confirm a geospatial position of AR apparatus <NUM>), or to obtain image stabilization data (for example, data indicating the direction and periodicity of a camera shake) to facilitate object recognition.

In some embodiments, AR apparatus <NUM> includes input/output interface <NUM>. According to various embodiments, I/O interface <NUM> provides communicative connectivity between AR apparatus <NUM> and at least one other electronic device, such as a smartphone, or computer to which AR apparatus <NUM> is an accessory device. According to certain embodiments, I/O interface <NUM> connects AR apparatus over a network to a host platform providing a PCAR database. I/O interface is, in certain embodiments, a wireless communication interface, such as a Bluetooth transceiver, or communication hardware supporting communications over one or more longer range wireless systems (for example, communication unit <NUM> in <FIG>).

<FIG> illustrates an example of fields of view at an AR apparatus according to some embodiments of this disclosure. The example of fields of view shown in <FIG> is for illustration only and other examples could be depicted without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, an AR apparatus <NUM> is shown. AR apparatus <NUM> comprises an externally-oriented camera <NUM>, which is a camera trained on a field of view (e.g., a "camera field of view") comprising a set of viewing angles which have a component which points away from the direction from an internally-facing display of the AR apparatus and an intended viewing location <NUM> (represented in this example by a viewer's eye). In some embodiments, externally-oriented camera <NUM> gathers image data from the camera field of view. AR apparatus <NUM> recognizes objects in the camera field of view, and a PCAR framework running on AR apparatus provides items of AR content (represented by arrow <NUM>) associated with the recognized objects to internally-facing display <NUM> to be presented towards intended viewing location <NUM>.

According to various embodiments, internally-facing display <NUM> is at least partially transparent, thereby allowing light from external objects located along viewing angles within a field of view in which AR content can be displayed (e.g., an "AR field of view") to pass through internally-facing display <NUM> to intended viewing location <NUM>. In this non-limiting example, light from tree <NUM> (represented by arrow <NUM>) passes through display <NUM> to intended viewing location. Thus, the view at intended viewing location includes both light from tree <NUM>, as well as one or more items of AR content associated with tree <NUM>.

<FIG> illustrates an example of fields of view at an AR apparatus according to various embodiments of this disclosure. The example of fields of view shown in <FIG> is for illustration only and other examples could be depicted without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, a scene <NUM> is shown from a viewing point along an internally-facing direction relative to an AR apparatus <NUM>. In this illustrative example, scene <NUM> includes first tree 501a, second tree 501b, house <NUM> and stream <NUM>.

According to certain embodiments, AR apparatus <NUM> includes an externally-oriented camera covering a range of view angles defining a camera field of view <NUM>, which includes first tree 501a, house <NUM> and part of stream <NUM>. Further, AR apparatus <NUM> includes an internally-facing display <NUM>. In this non-limiting example, internally-facing display <NUM> is at least partially transparent, and permits light from first tree 501a and house <NUM> to pass to the viewing point. According to some embodiments, internally-facing display <NUM> comprises one or more regions in which items of AR content (for example, notification <NUM> and object frame <NUM>) can be displayed. In the non-limiting example of <FIG>, the portions of internally-facing display <NUM> cover a range of viewing angles which defines an AR field of view <NUM>.

Referring to the non-limiting example of <FIG>, a PCAR framework operating on AR apparatus <NUM> recognizes house <NUM> from image data of camera field of view <NUM>, determines (for example, based on a signature comparison) that house <NUM> is an augmentation target, and responsive to determining that house <NUM> is an augmentation target, displays AR content comprising object frame <NUM>, which identifies house <NUM> as a recognized object, and notification <NUM>, which is associated with house <NUM>.

Note that, depending on the design objectives, the relative proportions of camera field of view <NUM> and AR field of view <NUM> may change. Additionally, depending on applications, the proportions of AR field of view relative to internally-facing display <NUM> might change. In certain embodiments, such as, for example, an AR apparatus to be used in connection with active pursuits, such as skiing or mountain biking, it may makes sense for camera field of view <NUM> to extend significantly beyond AR field of view <NUM>. In such embodiments, an externally-oriented camera may be able to identify hazards in a user's peripheral vision and provide AR content (for example, a warning regarding a vehicle rapidly approaching from the side). In some embodiments, it may be appropriate for AR field of view <NUM> to comprise a relatively small portion of display <NUM>, so as to avoid distractions. In various embodiments, it may be appropriate for camera field of view <NUM> to comprise a subset of AR field of view <NUM>. For example, in embodiments where the AR apparatus is a headset intended for use in applications involving small, near field details (for example, surgery), the externally-oriented camera may provide a magnified view of a small area, which is surrounded by an AR field of view in which multiple pieces of AR content (for example, a patient's vital signs) can be presented.

<FIG> illustrates an example of a PCAR database schema <NUM> according to certain embodiments of this disclosure. The embodiment of the PCAR database schema <NUM> shown in <FIG> is for illustration only and other examples could be depicted without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, in certain embodiments, associations between objects which can be identified based on image data from AR apparatus and augmentation data used to generate items of AR content, as well as user-managed attributes of items of AR content are maintained in a PCAR database (for example, PCAR database <NUM> in <FIG>) on a host platform. In some embodiments, a PCAR database is a relational database utilizing a schema, such as schema <NUM>. In certain embodiments, PCAR database is a columnar database, or other suitable structure for storing and managing associations between objects, augmentation data, and items of AR content.

According to certain embodiments, values of an object signature <NUM> field serve as a primary key (PK) of database schema <NUM>. In some embodiments, an object signature comprises a unique identifier of an object associated with one or more items of augmentation data used to generate items of AR content. In some embodiments, an object signature is a unique identifier of a particular instance of an object (for example, a refrigerator in a particular location, such as a house). In certain embodiments, the object signature comprises a unique identifier of an object for which there may be multiple instances (for example, a particular model of television, or a model of car). In the illustrative example of <FIG>, the object signatures are assigned numbers for objects (for example, signature <NUM>, which corresponds to the assigned number "<NUM>"). In certain embodiments, to enhance security, signatures may correspond to an alphanumeric string (for example, a hash of descriptors of an object and other associated data), or an encryption thereof.

In the non-limiting example of <FIG>, schema <NUM> includes a "User ID" field <NUM>, the values of which correspond to identifiers of a user who created an association between an object having an object signature and augmentation data. For example, the User ID "Martha"is associated with the object signature "<NUM>," identifying Martha as the creator and manager of an item of an item of AR content, the item of AR content being associated with "Virtual Object" augmentation data <NUM>.

According to various embodiments, schema <NUM> includes an Augmentation Data field <NUM>. In certain embodiments, the values of augmentation data field <NUM> comprise pointers to locations where data for an item of AR content are stored. For example, the "Text and Web Content Item" augmentation data <NUM> associated with object signature <NUM> may correspond to images and text stored at location at an edge caching server, the address ofwhich is provided to an AR application (for example, AR application <NUM> in <FIG>), and which the AR application accesses to assemble an item of AR content for display on the AR apparatus. In some embodiments, augmentation data may comprise an actual item of AR content (for example, a video to be played at an AR apparatus).

In certain embodiments according to this disclosure, schema <NUM> includes a visibility field <NUM>, which is an attribute of the association between an object (as identified by its object signature) and one or more items of augmentation data. As shown in the non-limiting example of <FIG>, values in visibility field <NUM> correspond to identifiers of users who can access the augmentation data associated with a particular object signature. For example, visibility value <NUM> specifies that "Martha" is the user with permission to see item(s) of AR content based on augmentation data <NUM>, in response to an AR apparatus associated with Martha recognizing, based on obtained image data, an object having object signature <NUM>.

As shown in the non-limiting example of <FIG>, schema <NUM> also includes object location field <NUM>, whose values specify a location, or set of locations associated with an object having a particular object signature value. According to various embodiments, object location data can be used to distinguish between objects which are generic augmentation targets (for example, any instance of a particular model of automobile), and objects which are specific augmentation targets (for example, a specific instance of a recognized refrigerator at a particular location, which is a user's or viewer's "home fridge. For example, object location value <NUM> indicatesthat the object associated with object signature <NUM> is located indoors at a specific set of GPS coordinates. In this way, a PCAR database or a PCAR framework utilize object location <NUM> in determining whether to present an item of AR content based on augmentation data <NUM> in response to recognizing an object having object signature <NUM>. If the PCAR database or PCAR framework determines that the recognized object is outside or at a different GPS location, the item of AR content will not be presented. As another example, object location value <NUM> is equal to "anywhere," indicating that there are no location-based constraints on presenting an item of AR content based on augmentation data <NUM>.

In various embodiments, schema <NUM> includes expiration time field <NUM>, the values of which specify the duration an item of AR content associated with the augmentation target having a given object signature is to be available for viewing. As an example, expiration time value <NUM> specifies that item(s) of AR content based on augmentation data <NUM> are to be available until read by a user specified in visibility field <NUM>. As another example, expiration time value <NUM> is "indefinite", indicating that AR content based on augmentation data <NUM> will be persistently available to "John" and "Ted" in response to recognizing objects having object signature.

While not shown in the illustrative example of <FIG>, schema <NUM> can include other fields, such as fields indicating a relationship (for example, a hierarchical, or parent-child relationship) between object signatures. For example, a particular chest of drawers may be associated with a first object signature. At the same time, a drawer, or other sub-component of the chest of drawers, may be associated with a second object signature. A relationship field of schema <NUM> may include values specifying a parent-child relationship between the chest of drawers, and a particular dresser. In this way, AR communications utilizing the chest of drawers as an augmentation target may be more nuanced and effective. For example, a first user may associate an AR item with the chest of drawers, the AR item comprising a note saying that a second user has clean laundry, which is in a particular drawer within the chest of drawers. Further, the database entry associating the item of AR content with the chest of drawers may include a value in a relationship field specifying a particular drawer, which may also be associated with a second item of AR content.

<FIG> illustrates aspects of image recognition using binary image descriptors, according to various embodiments of the present disclosure. The example of the image recognition shown in <FIG> is for illustration only and other examples could be depicted without departing from the scope of the present disclosure.

Certain embodiments according to this disclosure recognize objects from image data obtained over a camera field of view by recognizing three dimensional feature points (also referred to as "smart feature points") within the image data of a scene. According to certain embodiments, three dimensional feature points comprise characteristic features (for example, corners and relationships between corners) of objects which can be reliably found within image data obtained at an AR apparatus. Various embodiments according to this disclosure recognize three dimensional feature points of objects (including objects which are augmentation targets) by using binary image descriptors.

Referring to the non-limiting example of <FIG>, a three dimensional feature space <NUM> is shown. According to various embodiments, three dimensional feature space <NUM> is a three dimensional section of the physical world covered by the field of view of an externally-facing camera of an AR apparatus. According to various embodiments, a recognition engine (for example, object recognition engine <NUM> in <FIG>) or other image analysis process, obtains image data comprising patches associated with points in the feature space, such as points 701a, 701b and 701c shown in <FIG>. In certain embodiments, the image patches are selected based on a visual scanning / search algorithm of the recognition engine. In various embodiments, the image patches are selected based on existing knowledge regarding the feature space, such as a map of known features of the space based on previously obtained image data.

As shown in the non-limiting example of <FIG>, image patch <NUM> is obtained from image data obtained at point 701a. According to various embodiments, a binary representation of image patch <NUM> is obtained by image intensity data at points of a sampling pattern. As shown by graphic <NUM>, in this illustrative example, the sampling pattern comprises a 3x3 grid which includes sampling points 715a, 715b and 715c. According to various embodiments, a representative slice of the visual information contained in patch <NUM> can be encoded as a binary string or binary vector, by comparing the intensity value of a predetermined set of pairs of sampling points within the sampling pattern. In the non-limiting example of <FIG>, in the predetermined set of pairs of sampling points is represented through lines connecting a given sampling point to certain other sampling points. In certain embodiments, where a given sampling point of a sampling point pair in a patch has the higher intensity value of the sampling point pair, a "<NUM>" is written to the binary string. Similarly, if the sampling point has a lower intensity value than a conjugate sampling point of the sampling point pair, a "<NUM>" is written to the binary string of binary vector. By repeating this comparison across the predetermined set of sampling vectors, a binary string or binary vector <NUM> representing the patch is generated.

According to certain embodiments, binary vector <NUM> can be rapidly compared against other binary vectors describing features of objects to be recognized (also known as "binary descriptors") by calculating the Hamming distance (for example, a value representing the number of divergent or similar values between two binary vectors) or by applying an exclusive OR ("XOR") operation to values of a binary vector encoding information from an image patch (for example, binary vector <NUM>) and a binary vector representing a feature of a recognizable object. Where a hamming distance or value of an XOR operation is within an acceptable range, the image patch is recognized as a feature of an object. In certain embodiments, by iteratively encoding and comparing patches of image data obtained across feature space <NUM>, objects within the space can be recognized the purposes of AR based communication.

<FIG> illustrate aspects of augmented reality-based communication according to certain embodiments of this disclosure. In the illustrative examples of <FIG>, a method of communication according to this disclosure is shown from the perspective of an AR field of view of an apparatus (for example, AR apparatus <NUM> in <FIG>) implementing a PCAR framework (for example, PCAR framework <NUM> in <FIG>).

Referring to the non-limiting example of <FIG>, the portion of a scene <NUM> visible within the AR field view of an AR apparatus is shown at an initial time, such as immediately after the wearer of the AR apparatus enters the room containing the elements of scene <NUM>. According to certain embodiments, the elements of the scene, such as refrigerator <NUM> are visible from light within the scene passing through a clear display (for example, display <NUM> in <FIG>) of the AR apparatus. While not depicted in the non-limiting example of <FIG>, in addition to allowing light from the scene to pass through the display, the AR apparatus is also obtaining image data of a camera field of view of one or more externally-oriented cameras and processing the image data to identify objects within the camera field of view.

Referring to the non-limiting example of <FIG>, elements of an AR field of view of scene <NUM> are illustrated. In this non-limiting example, two items of AR content - first highlighting frame 810a and second highlighting frame 810b are depicted in a portion of the AR field of view at locations of the display coinciding with the position of refrigerator <NUM> and second refrigerator <NUM>. According to certain embodiments, the AR apparatus, or a PCAR framework implemented on a device connected to the AR apparatus, has identified, based on binary descriptors of the refrigerators in image data from an externally-oriented apparatus, the refrigerators. Further, having identified the refrigerators, the AR apparatus has performed a calculation to map the location of refrigerator <NUM> to a location, or a coordinate representing a location, in the AR field of view. Additionally, according to certain embodiments, the AR apparatus is in an augmentation mode, and has determined that refrigerator <NUM> and second refrigerator <NUM> are augmentation targets, which can be associated with an item of AR content (for example, a note or message) to communicate with other AR apparatus users.

In certain embodiments, the AR apparatus's identification of objects within scene <NUM> can be assisted by data provided by the PCAR framework and/or the objects themselves. In one non-limiting example, the PCAR framework can provide an AR apparatus with additional object information (for example, an identification of objects near the current location of the AR apparatus). This additional object information can be used to provide notifications to a user of the AR apparatus when a predetermined condition (for example, a proximity value to the object) is satisfied. Thus, in one example, a user of an AR apparatus may, upon entering the room of scene <NUM>, receive a message notifying him of refrigerator <NUM> being an augmentation target. Additionally, in instances where an item of AR content has already been associated with refrigerator <NUM>, the additional object notification may be more specific, such as a message of the form "Message for you on the refrigerator. " Through the use of such additional object information, computing and battery resources of an apparatus may be conserved, potentially less image data needs to be received to recognize an object if a user is given an indication of the object to be recognized.

Additionally, in certain embodiments, objects within scene <NUM> themselves may broadcast or otherwise provide an AR apparatus of their identity, as well as additional object information. For example, refrigerator <NUM>, may, in some embodiments, include a Bluetooth Low Energy transmitter, or LED beacon which emits a signal informing AR apparatus that it is an augmentation target, or that a piece of AR content is associated with refrigerator <NUM>. Here, the object itself provides additional assistance as to the presence of augmentation targets and/or associations between items of AR content and the object, thereby potentially reducing the amount of image data an AR apparatus must process to find augmentation targets within scene <NUM>.

Turning to the non-limiting example of <FIG>, which illustrates elements of scene <NUM> within an AR field of view. In this illustrative example, a user (identified as "John") has provided an input (for example, a verbal command or eye gesture) initiating the process of creating an association between an augmentation target (in this case, refrigerator <NUM>) and one or more items of AR content. According to certain embodiments, John's input triggers the display of option menu <NUM>, which presents various user-selectable parameters of an association between the augmentation target and one or more items of AR content. As shown in this non-limiting example, the user-selectable parameters comprise a type <NUM> of AR content to be associated with the augmentation target, users <NUM> to whom the selected AR item(s) of AR content will be visible, and an expiration or duration time <NUM> for the association between the augmentation target and the item(s) of AR content. In certain embodiments, the user-selectable parameters of option menu <NUM> may map to one or more fields of a PCAR database schema (for example, schema <NUM> shown in <FIG>).

Turning to the non-limiting example of <FIG>, which illustrates an input window <NUM> presented to the user in response to a selection of parameters further defining the association between the augmentation target and an item of AR content. In this illustrative example, the user "John" has selected the values "TEXT" from type <NUM>, "JEN" from users <NUM>, and "UNTIL READ" from time <NUM>, and has been presented with input window <NUM>, which allows a user to input textual data to be provided as part of an item of AR content associated with an augmentation target. In this non-limiting example, the user "John" inputs text describing food which has been left for a user specified by a value of users <NUM> in option menu820. In various embodiments according to this disclosure, after the user creating the association between an augmentation target and an item of AR content (in this case, a message to be seen by certain users who look at refrigerator <NUM> through AR apparatus connected to a PCAR database), the text inputted through input window <NUM>, along with values for the parameters inputted via option menu <NUM>, as well as location, and user profile information, is transmitted to a platform hosting a database (for example, PCAR database <NUM> in <FIG>) of associations between augmentation targets, items of AR content, and parameters of the associations.

<FIG> illustrates elements of scene <NUM> in the AR field of view of an AR apparatus associated with the user "Jen" selected by the user "John" via option menu <NUM>. In this non-limiting example, the PCAR framework of Jen's AR apparatus is operating in a "normal" mode, and has identified refrigerator <NUM> as an augmentation target. According to various embodiments, the PCAR framework on Jen's device recognizes refrigerator <NUM> as an augmentation target using the same object recognition technique as a PCAR framework on John's device (for example, object recognition based on comparing binary descriptors of patches of image data against binary descriptors of features). According to certain embodiments, Jen's device may recognize refrigerator <NUM> as an augmentation target using a different method, such as by recognizing a visually coded link to a reference in a PCAR database (for example, a QR code) on the refrigerator, or detecting an identifying signal from the refrigerator (for example, an IR beacon on the refrigerator transmitting an identification code). Having determined that refrigerator <NUM> is an augmentation target, the PCAR framework on Jen's device generates an object signature associated with refrigerator <NUM>.

In this illustrative example, the object signature generated at Jen's device matches an object signature for refrigerator <NUM> maintained at a PCAR database. Further, Jen's user profile matches a value associated with a permission to view the AR content associated with the object signature. As such, Jen receives data associated with an item of AR content (for example, the text of John's message, as well as information specifying how long the image is to be presented. Based on the received data, an item of AR content <NUM> is presented on a display of Jen's AR apparatus. In this non-limiting example, Jen is presented with a second option window <NUM>, which allows her to choose between either marking John's message as having been read, or responding to John's original message.

<FIG> illustrates elements of scene <NUM> within the AR field of view of the AR apparatus associated with the user "Jen". In this non-limiting example, Jen selected "Respond" in second option window <NUM>, and has composed a response message <NUM> (for example, an email or text message) to be sent to John in response. As shown above, certain embodiments according to the present disclosure enhance the functionality of smartphones and other AR apparatus by allowing AR to be used as either a standalone communication tool, or as a complement to other communication tools, such as text and email.

<FIG> illustrate aspects of augmented reality-based communication according to various embodiments of this disclosure. The embodiments of the augmented reality-based communication shown in <FIG> are for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

The non-limiting example of <FIG> illustrates elements of a scene <NUM> visible in the AR field of view of an AR apparatus associated with a first user, Mary. In this non-limiting example, the elements of scene <NUM> visible in the AR field of view include television <NUM>.

The non-limiting example of <FIG> illustrates elements of scene <NUM> visible in the AR field of view of the AR apparatus associated with the user Mary. In this non-limiting example, Mary's AR apparatus has, based on image data of an externally-oriented camera of the AR apparatus, identified television <NUM> as both a television and an augmentation target, and having mapped the position of television <NUM> relative to the AR field of view, displays, highlighting frame <NUM> around television <NUM> on an internally-facing display of Mary's AR apparatus.

<FIG> illustrates elements of scene <NUM> visible in the AR field of view of Mary's apparatus. In this non-limiting example, in response to an input from Mary (for example, an affirmative input, such as a voice command, or an implied input, such as gaze tracking data indicating showing Mary looking at the augmentation target in a way that satisfies a specified criteria, such as a temporal threshold), a second item of AR content 915a is presented on an internally facing display (for example, a display like display <NUM> in <FIG>). According to certain embodiments, the second item of AR content 915a may be based, at least in part, on data (for example, data maintained in a PCAR database) provided by a PCAR database in response to a match between an object signature of television <NUM> and descriptors of the television maintained in the PCAR database. In this non-limiting example, the second item of AR content 915a is based, at least in part, on a user profile data accessible to an AR application (for example, AR application <NUM> in <FIG>) operating on Mary's AR apparatus. In various embodiments, such user profile data defines one or more presentation parameters (for example, a choice of language and personalization parameters, such as a name or photograph) of the second item of AR content <NUM>. In the illustrative example of <FIG>, the presentation parameters drawn from a user profile associated with Mary include her name <NUM> and the use of English as the language for presenting the message regarding the television set in her AR field of view.

<FIG> illustrates elements of scene <NUM> visible in the AR field of view of a second user, "Lars". As illustrated by this non-limiting example, creating associations between augmentation targets and items of AR content according to embodiments of the present disclosure can provide an efficient and easily extensible method of personalized communications. In this non-limiting example, an AR apparatus associated with second user Lars, has, identified television <NUM> as an augmentation target, and presented a first item of AR content associated with the augmentation target on a internally-facing display of Lars's AR apparatus. Based on Lars's response to the first item of AR content, a second item of AR content 915b is displayed. According to various embodiments, second item of AR content 915b is based on the same data received from a PCAR database as second item of AR content 915a in <FIG>. Using the same information from a PCAR database, an AR application on Lars's AR apparatus can use information in Lars's user profile to apply different presentation parameters to substantially change the appearance of second item of AR content 915b relative to second item of AR content 915a. In this non-limiting example, information in Lars's user profile provides presentation parameters changing both the addressee <NUM> of the message, but also switching the language of the message text to German.

<FIG> illustrates operations of a method <NUM> of operating a PCAR framework in a normal mode of operation. According to certain embodiments, a "normal mode of operation" of a PCAR encompasses a mode of operation in which the PCAR is "listening" or "looking" for items of AR content associated with identified augmentation targets in image data obtained from a camera field of view of an apparatus. In some embodiments, a "normal" mode of operation operates in parallel with an "augmentation mode," in which a PCAR frameworkis creating or updating an association between an augmentation target and an item of AR content. In certain embodiments, a "normal" mode of operation operates as an alternative to an "augmentation mode. " While the flow chart depicts a series of sequentialsteps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process depicted in the example depicted is implemented by a processor in, for example, an electronic device.

Referring to the non-limiting example of <FIG>, method <NUM> includes operation <NUM>, wherein a PCAR framework (for example, PCAR framework <NUM> in <FIG>) receives location data and image data from sensors connected to an AR apparatus (for example, image data from externally-oriented camera <NUM> in <FIG> and location data from a GPS sensor or IMU <NUM> in <FIG>). According to various embodiments, image data is received by a camera field of view object recognition engine (for example object recognition engine <NUM> in <FIG>) and location data is received at the PCAR framework by a signature engine (for example, signature engine <NUM> in <FIG>).

According to various embodiments, at operation <NUM>, the PCAR framework performs an initial determination of whether there are any augmentations associated with the user device at a current device location (as determined from location data received in operation <NUM>). According to certain embodiments, the determination performed at operation <NUM> comprises sending a query to a host platform (for example host platform <NUM> in <FIG>) which maintains an index or other data structure associating locations, user devices and items of AR content. In certain embodiments, if the outcome of the determination at operation <NUM> is negative, method <NUM> loops back to operation <NUM>. If, however, the PCAR determines that there is an item of AR content associated with an augmentation target and visible to viewing devices with the user's profile and current device location, method <NUM> proceeds to operation <NUM>.

In certain embodiments according to the present disclosure, at operation <NUM>, one or more components (for example, an augmentation engine, such as augmentation engine <NUM> in <FIG> and an object recognition engine, such as object recognition engine <NUM> in <FIG>) determines whether there are objects within a given sensor range (for example, the field of view of an externally-oriented camera of an AR apparatus) which comprise augmentation targets. According to various embodiments, the determination of whether there are augmentation targets, and in particular, augmentation targets associated with items of AR content for the present user is performed by applying one or more object recognition algorithms to image data from the externally-oriented camera. In certain embodiments, the determination of whether there are augmentation targets associated with AR content for a present user may be assisted by the augmentation target itself. For example, the augmentation target may have one or more surfaces with a coded medium (for example, a barcode or QR code), bypassing the need for recognition of the object as the augmentation target. As another example, the augmentation target may also have a beacon (for example, a flashing infrared beacon) advertising itself as an augmentation target.

In the non-limiting example of <FIG>, if, at operation <NUM>, the PCAR cannot find objects in the relevant sensor range(s) which are augmentation targets, method <NUM> reverts back to operation <NUM>. When the PCAR, or a module thereof, such as an augmentation engine, finds objects within the relevant sensor range(s) which are augmentation targets, method <NUM> proceeds to operation <NUM>.

In various embodiments according to the present disclosure, at operation <NUM>, responsive to finding object(s) within the relevant sensor range(s) (for example, the range of an IR beacon, or within a camera field of view of an externally-oriented camera of an AR apparatus), the PCAR interacts with one or more of a PCAR database (or other repository or data structure maintaining data storing associations between augmentation targets and items of AR content) and an AR application (for example, AR application <NUM> in <FIG>) to display one or more items of AR content associated with an augmentation target found in operation <NUM>. According to some embodiments, the item of AR content is a highlighting frame (for example, first highlighting frame 810a in <FIG>). In various embodiments, the item of AR content displayed at operation <NUM> comprises a communicative piece of content (for example, second item of AR content 915a in <FIG>).

Referring to the non-limiting example of <FIG>, at operation <NUM>, the PCAR framework provides information regarding the present user's interaction to the device(s) maintaining the PCAR database to update the data defining the association between the augmentation target and the item(s) of AR content. For example, the PCAR framework may provide information indicating that a particular item of AR content (for example, item of AR content <NUM> in <FIG>) has been read or responded to. Based on the provided information, and the specified parameters of the association between augmentation target and item of AR content (for example, display settings specifying that the item of content is to be displayed until read) the PCAR database may be updated (for example, by canceling or deleting the association between the augmentation target and item of AR content).

<FIG> illustrates operations of a method <NUM> of operating a PCAR framework in an augmentation mode, according to at least one embodiment of this disclosure. While the flow chart depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process depicted in the example depicted is implemented by a processor in, for example, an electronic device. Further, while the operations of method <NUM> are described with reference to an exemplary embodiment in which the operations are carried out by a PCAR framework on an electronic device, in certain embodiments, performance of the operations of method <NUM> may be distributed across multiple computing platforms.

Referring to the non-limiting example of <FIG>, at operation <NUM>, an augmentation mode of a PCAR framework is activated, and an object recognition engine (for example, object recognition engine <NUM> in <FIG>) is activated. Such activation may be in responses to a received input (for example, a user input). In this way, the PCAR framework is conFigured to recognize augmentation targets within a relevant sensor range (for example, the field of view of an externally-oriented camera, or the transmission range of a BLUETOOTH® Low Energy ("BLE") beacon on a predetermined augmentation target).

According to various embodiments, at operation <NUM>, the PCAR framework recognizes one or more objects as augmentation targets (for example, through comparisons of binary representations of image data patches against binary descriptors of defined features) based on image or other data provided to the PCAR. Responsive to recognizing objects that are augmentation targets, the PCAR maps the objects' location in the image data to positions with an AR field of view (for example, AR field of view <NUM> in <FIG>) and displays one or more items of AR content comprising selectable overlays of the augmentation targets, and method <NUM> proceeds to operation <NUM>. If, however, the PCAR framework does not recognize any objects, in certain embodiments, method <NUM> reverts back to operation <NUM>. In various embodiments, when reverting to operation <NUM>, the PCAR framework switches back to "normal" mode, until a fresh input enabling the augmentation mode is received.

In the non-limiting example of <FIG>, at operation <NUM>, a user may select.

In the non-limiting example of <FIG>, at operation <NUM>, a selection input is received. According to certain embodiments, the selection input selects an object associated with an overlay marker as an augmentation target to be associated with one or more pieces of AR content. According to various embodiments, selection input may be provided by a user, such as through a verbal input processed by a speech recognition algorithm implemented on an AR apparatus or device (for example, a smartphone) communicatively connected thereto. In certain embodiments, the selection input may be provided through a gaze or gesture (for example, a nod) detected by one or more sensors (for example, an internally-facing camera) of an AR apparatus. If a selection input is received by the PCAR framework (for example, within a specified period of time), method <NUM> proceeds to operation <NUM>. If a user selection is not received, or not received within a specified period of time, method <NUM> reverts to operation <NUM>.

In certain embodiments, at operation <NUM>, the PCAR framework obtains (for example, by generating a new signature, or retrieving a previously stored signature) an object signature for the object selected in operation <NUM> as an augmentation target to be associated with one or more items of AR content. In various embodiments, at operation <NUM>, the PCAR framework also obtains (for example, by receiving user inputs, such as via option menu <NUM> in <FIG>, or from a user profile) parameters (for example, text or images to be displayed) further defining the association between the selected augmentation target and the item of AR content. At operation <NUM>, the object signature, and the parameters defining the association between the augmentation target and the item of AR content are provided to a PCAR management entity (for example, host platform <NUM> in <FIG>).

Referring to the non-limiting example of <FIG>, at operation <NUM>, the PCAR management entity (for example, a server maintaining a PCAR database) updates a PCAR database to include the association between the selected augmentation target and an item of AR content, and the parameters thereof, created at operation <NUM>.

<FIG> illustrates operations of a method <NUM> for performing augmented reality-based communications according to the invention.

The process depicted in the example depicted is implemented by a processor in, for example, an electronic device.

Referring to <FIG>, at operation <NUM>, a processor (for example, processor <NUM> in <FIG>) obtains image data of a camera field of view (for example, camera field of view <NUM> in <FIG>) from an externally-oriented camera (for example, externally-oriented camera <NUM> in <FIG>).

According to operation <NUM>, an object in the camera field of view (for example, house <NUM> in <FIG>) is identified (for example, based on an analysis of binary descriptors of patches of image data, as described with reference to <FIG>) based on the image data obtained at operation <NUM>.

As shown in <FIG>, at operation <NUM>, the position of the object identified in operation <NUM> is mapped to a position of an AR field of view (for example, the location of object frame <NUM> in Figure within AR field of view <NUM> is determined based on a mapping of camera field of view <NUM> to AR field of view <NUM>).

At operation <NUM>, a determination of whether the object identified at operation <NUM> is an augmentation target is performed. In certain embodiments, this determination is performed by comparing an object signature of the object identified at operation <NUM> against a set of object signatures (for example, object signatures maintained in a PCAR database utilizing database schema <NUM> in <FIG>). In various embodiments, the determination of whether an object comprises an augmentation target is based on whether the object belongs to a defined class of objects which can be augmentation targets (for example, non-transitory items, such as buildings or pieces of furniture).

Referring to <FIG>, at operation <NUM>, responsive to determining that the object identified at operation <NUM> comprises an augmentation target, an item of AR content associated with the augmentation target is displayed on an internally-facing display in the AR field of view of the AR apparatus. For example, the item of AR content may be a frame or overlay which can be selected (for example, first highlighting frame 810a in <FIG>). In certain embodiments, the item of AR content may be a pre-associated item of AR content (for example, item of content <NUM> in <FIG>).

<FIG> illustrates operations <NUM> of methods for performing augmented reality-based communications according to various embodiments of this disclosure. Depending on embodiments, operations within operations <NUM> may be performed in addition to the operations of method <NUM> in <FIG>. The process depicted in the example depicted is implemented by a processor in, for example, an electronic device.

Referring to the non-limiting example of <FIG>, at operation <NUM>, a PCAR framework (for example, PCAR framework <NUM> in <FIG>) or module thereof identifies, a parent object to another object in the camera field of view based on image data received from an externally-oriented camera of an AR apparatus. According to various embodiments, the PCAR framework recognizes the first object separately from its parent object (for example, the PCAR recognizes a distinctive car wheel and a car), and derives the parent-child relationship from additional information provided in a PCAR database or object descriptor database (for example, data associating a particular kind of wheel with a specific model car). According to some embodiments, the PCAR framework recognizes the first object as a feature of the parent object and based on this recognition, identifies the parent-child relationship between the objects.

In certain embodiments, the PCAR frame identifies a first object based on a comparison of one or more binary descriptors of image data representing the first object (for example, image data patch <NUM> in <FIG>) against binary descriptors of features of the first object.

According to various embodiments, at operation <NUM>, an item of AR content is displayed at a location in the AR field of view, which has been determined not to obstruct either a view of an object which is an augmentation target (for example, house <NUM> in <FIG>), or another item of AR content (for example, notification <NUM> in <FIG>). In certain embodiments, the determination of non-obstructing locations within the AR field of view is based on a mapping of the locations of objects in a camera field of view to an AR field of view (for example, as performed by operation <NUM> in <FIG>). According to the invention, the placement of items of AR content within an AR field of view is determined based on an analysis of the image data from the camera field of view, as well as stored contextual information regarding objects in the camera field of view. According to certain embodiments, the stored contextual information regarding objects includes information identifying the location and size of essential features of an object. In this way, items of AR content can be displayed in a way in which essential features of a scene are not occluded by items of AR content. As one non-limiting example, a movie poster which is an augmentation target may comprise the images of the actors, the film's title, and fine print identifying the studio, producers and other stakeholders in the project. The stored contextual information may include rules regarding the placement of AR content regarding the elements of the augmentation target, including, without limitation, proximity requirements for items of AR content associated with elements of the augmentation target. Additionally, the stored contextual information may include rules about feature visibility (for example, rules specifying that covering faces with AR content is impermissible, but covering text is permitted). In this way, certain embodiments according to this disclosure can present AR content associated with augmentation targets in a way that is contextually aware, and uncluttered.

In some embodiments according to this disclosure, at operation <NUM>, the position of an item of AR content is displayed on an internally-facing display of an AR apparatus at a location based on the mapped position of the first object relative to the AR field of view (for example highlighting frame 810b in <FIG> is displayed at a location based on the mapped position of second refrigerator <NUM> in the AR field of view).

In the non-limiting example of <FIG>, at operation <NUM>, the item of AR content associated with the augmentation target to be displayed is selected, at least in part, based on information in a user profile of an AR apparatus (for example, second item of AR content 915b in <FIG> is, in certain embodiments, selected for display based on information in a user profile identifying a present user of an AR apparatus as a German speaker named Lars ").

Claim 1:
A method for providing a personalized augmented reality, AR, display, the method comprising:
obtaining (<NUM>), at an augmented reality, AR, apparatus (<NUM>, <NUM>), image data of a camera field of view, the camera field of view comprising a field of view of an externally oriented camera (<NUM>, <NUM>) of the augmented reality apparatus (<NUM>, <NUM>);
identifying (<NUM>) a first object in the camera field of view based on the image data;
mapping (<NUM>) a position of the first object relative to an augmented reality, AR, field of view, the AR field of view comprising a portion of the camera field of view in which augmented reality, AR, content can be displayed on an internally-facing display (<NUM>, <NUM>) of the augmented reality apparatus (<NUM>, <NUM>);
receiving, by a signature engine (<NUM>) of a personalized communication augmented reality, PCAR, framework (<NUM>) implemented on the augmented reality apparatus (<NUM>, <NUM>), location data from a sensor connected to the AR apparatus, and a descriptor of the first object from an object recognition engine (<NUM>) of the PCAR framework (<NUM>);
generating, by the signature engine (<NUM>), a signature for the first object based on the descriptor and the location data, wherein the signature is unique and consists of a hash of the descriptor and the location data;
comparing, by the signature engine (<NUM>), the generated signature with a plurality of signatures stored in a personalized communication augmented reality, PCAR, database (<NUM>) maintained on a host platform (<NUM>);
when there is a match between the generated signature and one of the plurality of stored signatures, determining (<NUM>), by the signature engine (<NUM>), whether the first object comprises an augmentation target and receiving, by the signature engine (<NUM>), augmentation data, wherein the augmentation data is associated with an item of AR content; and
responsive to determining that the first object comprises the augmentation target, displaying (<NUM>), on the internally-facing display, the item of AR content associated with the augmentation target in the AR field of view, the item placement being based on an analysis of the image data and stored contextual information regarding objects in the camera field of view.