Displaying 3D content shared from other devices

One exemplary implementation involves performing operations at a device with one or more processors, a camera, and a computer-readable storage medium, such as a desktop computer, laptop computer, tablet, or mobile phone. The device receives a data object corresponding to three dimensional (3D) content from a separate device. The device receives input corresponding to a user selection to view the 3D content in a computer generated reality (CGR) environment, and in response, displays the CGR environment at the device. To display the CGR environment the device uses the camera to capture images and constructs the CGR environment using the data object and the captured images.

TECHNICAL FIELD

The present disclosure generally relates to three dimensional (3D) content, and in particular, to systems, methods, and devices for sharing and displaying 3D content as part of computer generated reality (CGR) environments.

BACKGROUND

Existing computing systems and applications do not adequately facilitate the sharing and use of 3D content to provide and use CGR environments on electronic devices.

SUMMARY

Various implementations disclosed herein include devices, systems, and methods that display CGR environments using 3D content shared from other devices. Some implementations involve performing operations at a device with one or more processors, a camera, and a computer-readable storage medium. The device receives a data object corresponding to three dimensional (3D) content from a separate device. The device receives input corresponding to a user selection to view the 3D content in a CGR environment, and in response, displays the CGR environment at the device. To display the CGR environment, the device uses the camera to capture images and constructs the CGR environment using the data object and the captured images. The user of the device is thus able to simply and easily receive, view, and use 3D content in a CGR environment without necessarily needing to move the 3D content to a particular storage location on the device, identify that the received data object has 3D content that can be experienced in a CGR environment, identify an app to provide the CGR experience, launch such an app, or import or add the received 3D content to the CGR environment provided by such an app. The user experience is a more efficient, effective, and intuitive.

The devices, systems, and methods disclosed herein enable the display of 3D content corresponding to received data objects in CGR environments based on user input. The devices, systems, and methods improve the ability of users to share 3D content to be experienced in CGR environments. For example, a first user, who is at home using a first device, can receive a data object corresponding to a couch from a second user who is in a retailer store looking at the couch. In this example, the second user uses a second device to create or identify a data object corresponding to the couch, e.g., using a camera of the second device to create a file that includes a 3D model of the couch or identifying a file or data storage address of a file that includes a 3D model of the couch. The second user then uses the second device to send a communication to the first user that includes or provides access to the data object. As examples, the second device can send a text message with the data object (e.g., file) attached, an e-mail message with the data object attached, or any other form of message attaching or providing a link or data storage address to obtain the data object.

Based on receiving the communication from the second device, the first device enables viewing or use of the data object. The first device receives input corresponding to a first user selection to view the content corresponding to the data object in a CGR environment. As examples, the first user may have clicked, double clicked, or tapped on a thumbnail image, link, icon, or button representing the data object within a text dialog window, an e-mail message viewer, or a social media message viewing website. Responsive to detecting this input, the first device displays a CGR environment by using the camera to capture images (e.g., video) and constructing the CGR environment using the data object and the captured images. In some implementations, the first device overlays the 3D content corresponding to the data object on the captured images. In some implementations, the first device automatically detects a file type of the data object and, based on the detected file type, identifies and launches a viewer to provide the CGR environment, including the 3D content corresponding to the data object. In some implementations, the first device accesses a plugin to launch the viewer within the user interface of the same app in which the thumbnail image, link, icon, or button representing the data object was displayed.

In some implementations, the CGR environment is configured to respond to user input interacting with the additional content or changing the user's viewpoint. For example, the first user may reposition the couch relative to the real world tables in the captured images and then physically move the device around the room to view the couch from different viewpoints within the room.

DESCRIPTION

As used herein, the phrase “physical environment” refers to a physical world that people can sense or interact with without aid of electronic systems. Physical environments, such as a physical park, include physical articles, such as physical trees, physical buildings, and physical people. People can directly sense or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

As used herein, the phrase “computer-generated reality” refers to a wholly or partially simulated environment that people sense or interact with via an electronic system. In CGR, a subset of a person's physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that comports with at least one law of physics. For example, a CGR system may detect a person's head turning and, in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some situations (e.g., for accessibility reasons), adjustments to characteristic(s) of virtual object(s) in a CGR environment may be made in response to representations of physical motions (e.g., vocal commands).

A person may sense or interact with a CGR object using any one of their senses, including sight, sound, touch, taste, and smell. For example, a person may sense or interact with audio objects that create 3D or spatial audio environment that provides the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, which selectively incorporates ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, a person may sense or interact only with audio objects.

Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment that is designed to be based entirely on computer-generated sensory inputs for one or more senses. A VR environment comprises a plurality of virtual objects with which a person may sense or interact. For example, computer-generated imagery of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense or interact with virtual objects in the VR environment through a simulation of the person's presence within the computer-generated environment, or through a simulation of a subset of the person's physical movements within the computer-generated environment.

In some MR environments, computer-generated sensory inputs may respond to changes in sensory inputs from the physical environment. Also, some electronic systems for presenting an MR environment may track location or orientation with respect to the physical environment to enable virtual objects to interact with real objects (that is, physical articles from the physical environment or representations thereof). For example, a system may account for movements so that a virtual tree appears stationery with respect to the physical ground.

Referring toFIG. 1, an example operating environment100for implementing aspects of the present invention is illustrated. In general, operating environment100represents two devices105,130involved in the sharing of a data object representing 3D content for display as part of a CGR experience. As depicted in the example ofFIG. 1, the operating environment100includes a first device105being used by a first user110in a physical environment (first real world scene115) and a second device130being used by a second user135in a physical environment (second real world scene140). In this example, the first real world scene115includes end tables120a,120band wall picture125. The second real world scene140includes a sofa145. In the example illustrated inFIG. 1, the first real world scene115and the second real world scene140are separate from one. However, the devices, systems, and methods described herein can be practiced in circumstances in which the real world scenes115,140are the same or in which some or all of the real world objects are in both of the scenes115,140. Moreover, the devices, systems, and methods can involve the second user135sharing data objects that are unrelated to the second real world scene140in which the second user135is located. For example, the second user135could create a virtual object on the second device130using a computer-aid design (CAD) software application and send a corresponding data object to the first device110for viewing and use.

In the example ofFIG. 1, the second user135observes the couch145in the second real world scene140and would like to know if the couch would be a good (e.g., aesthetically pleasing) match with the real world objects in the first real world scene115. For example, the second user135may be walking through through a retail furniture store looking at multiple couches and find couch145. The devices, systems, and methods enable the second user140to send a data object corresponding to three dimensional (3D) content representing the couch145to the first device105, and enable the first user110to receive the data object, provide input (e.g., a click on an icon representing the data object), and then experience a CGR environment that combines the real world (e.g., via locally captured images of the first real world scene115) and the 3D content corresponding to the data object (e.g., an image based on the 3D appearance of the couch).

In some implementations, the second device130creates or stores the data object corresponding to the couch that will be provided to the first device105. The second device130may include one or more cameras that capture images of the second real world scene140and the second device may be configured to generate a 3D model of the desired 3D content based on the captured images. In some implementations, the second device generates a model of multiple objects (or all objects) within the second real world scene140and the second user135provides input to individually select one or more of those objects to be individually stored as data objects and shared with one or more other users or user devices.

In some implementations, the second device130obtains the data object corresponding to the couch that will be provided to the first device105. The data object may be obtained from a separate data object storage device.FIGS. 2 and 3illustrate exemplary techniques for sharing a data object that is stored at a separate data object storage device.

FIG. 2is a flow chart illustrating a flow of communications200involved in sharing a data object according to some implementations. InFIG. 2, the second device130obtains a data object by sending a request220for the data object to a data object storage device215and receiving a response230that includes the data object. The second device130then uses the received data object to send a message240to the first device105. In some implementations, the second device130identifies which data object to request based on information in the first real world scene140. For example, a camera on the second device130can capture one or more images of the couch145and a data object can be identified by matching the captured image or images with a corresponding 3D model of the couch in a database of 3D models of couches maintained at the data object storage device215. Alternative or additional information can be used to identify the data object including, but not limited to, information from bar codes on or near the couch145, RFID tags on or near the couch145, user entered object identification data, or location data (e.g., identifying the store, row, floorplan position, etc. based on device on GPS, NFC-based tracking, or other location detection technology).

FIG. 3is a flow chart illustrating an alternative flow of communications300involved in sharing a data object according to some implementations. InFIG. 3, the second device130identifies a data object corresponding to the couch, for example, using one or more of the techniques discussed with reference toFIG. 2. The second device identifies a storage location of the data object and identifies a link that provides access to the data object at the storage location. The second device130then sends a message305with the link to the data object to the first device105. Based on user input accessing the data object on the first device105, the first device105uses the link to send a request310for the data object to the data object storage device205. The data object storage device205responds by providing the data object315corresponding to the couch.

FIG. 4illustrates a CGR environment400being provided on the first device105in the first real world scene115of the environment100ofFIG. 1. In this example the first device105has received the data object corresponding to the couch (i.e., couch145ofFIG. 1). The first device105may have presented a representation of the content of the received data object such as a thumbnail image, icon, link, or button representing or corresponding to the received data object. In the case of an icon or thumbnail image, for example, the icon or thumbnail image may include a two dimensional (2D) image of the content of the 3D object from a single or standard viewpoint. Thumbnail images, icons, links, and buttons are examples of graphical features that can received input (e.g., clicks, touches, etc.) corresponding to a user selection to view the 3D content in a CGR environment. For example, the first user could provide input selecting to view the 3D content of the couch in a CGR environment by clicking on a thumbnail image corresponding to the data object.

Responsive to detecting input, the first device105displays a CGR environment400. To display the CGR environment400, the first device105controls one or more cameras on the first device105to capture images of the first real world scene105and constructs the CGR environment400using the data object and the captured images. In some implementations, the first device105includes a suitable combination of software, firmware, or hardware to provide the CGR experience to the first user110. In other implementations, the first device105interacts with one or more other device (local or remote) to provide the CGR environment400, e.g., the first device105may communicate with a separate controller device (not shown) that performs some or all of the processing and storage required to provide the CGR environment400. According to some implementations, the first device105presents the CGR environment400to the first user110while the first user110is physically present within the first real world scene105. In some implementations, the first device105is configured to provide the CGR environment400using optical see-through of the first real world scene115. In some implementations, the first device105is configured to provide the CGR environment400using video pass-through of the first real world scene115.

In some implementations, the first device105is a head-mounted device (HMD) that the first user110wears. An HMD may enclose the field-of-view of the first user110. The HMD includes one or more CGR screens or other displays configured to display the CGR environment400. In some implementations, an HMD includes a screen or other display to display the CGR environment400in a field-of-view of the first user110. In some implementations, the HMD is worn is a way that a screen is positioned to display the CGR environment400in a field-of-view of the first user110. In some implementations, the first device105is a handheld electronic device (e.g., a smartphone or a tablet) configured to present the CGR environment400to the first user110. In some implementations, the first device105is a CGR chamber, enclosure, or room configured to present an CGR environment in which the first user110does not wear or hold the first device105.

The first device105is configured to use images or other real world information detected based on a camera or other sensor on the first device105. In some implementations, to provide the CGR environment400, the first device105uses at least a portion of one or more camera images captured by a camera. In the example ofFIG. 4, the CGR environment400includes depictions of items captured by a camera of the first device105. The CGR environment400depicts a wall picture425corresponding to wall picture125, a portion an end table420acorresponding to a portion of end table120a, i.e., only a portion of a flower on a vase in the end table120ais visible, and end table420bcorresponding to end table120b. The CGR environment400also depicts couch445corresponding to the data object and thus to the couch145in the second real world scene140(FIG. 1).

In some implementations, the first device105enables the first user110to change the viewpoint or otherwise modify or interact with the CGR environment400. In some implementations, a first device105is configured to receive user input that repositions received 3D content such as the couch445relative to the real world items depictions (e.g., wall picture425, end tables420a,420b) depicted in the CGR environment400.

FIG. 5illustrates the CGR environment400ofFIG. 4after being modified based on user input. For example, the first user110may have provided touch input on a screen of device105to drag the depiction of the couch445in between the depictions of the end tables420a,420b. Similarly, the first user110may have provided input to rotate, resize, or otherwise modify the depiction of the couch445relative to the depictions of the end tables420a,420b.

The positional relationship between received 3D object, such as the depiction of the couch445, and real world objects, such as the end tables420a,420b, can be maintained as the viewpoint used to provide a view of the CGR environment400on the first device105changes. For example, the viewpoint used for the CGR environment may change as the first user110walks around the first real world scene115. As the first user110walks around, additional camera images of the first real world scene115are captured and used to provide an updated view of the CGR environment400. During such movements and corresponding changes in viewpoint, the positional relationship between the depiction of the couch445and end tables420a,420bcan be maintained. In the example ofFIG. 5, the depiction of the couch445would remain between the depictions of the end tables420a,420bas the viewpoint changes, e.g., regardless of whether the user is looking at it from the left side or the right side of the first real world scene115.

Examples of 3D content corresponding to data objects include, but are not limited to, a table, a floor, a wall, a desk, a book, a body of water, a mountain, a field, a vehicle, a counter, a human face, a human hand, human hair, another human body part, an entire human body, an animal or other living organism, clothing, a sheet of paper, a magazine, a book, a vehicle, a machine or other man-made object, and any other 3D item or group of items that can be identified and represented. 3D content can additionally or alternatively include created content that may or may not correspond to real world content including, but not limited to, aliens, wizards, spaceships, unicorns, and computer-generated graphics and models.

FIG. 6is a block diagram illustrating device components of first device105according to some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the first device105includes one or more processing units602(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, or the like), one or more input/output (I/O) devices and sensors606, one or more communication interfaces608(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or the like type interface), one or more programming (e.g., I/O) interfaces610, one or more displays612, one or more interior or exterior facing image sensor systems614, a memory620, and one or more communication buses604for interconnecting these and various other components.

In some implementations, the one or more communication buses604include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I/O devices and sensors606include at least one of a touch screen, a softkey, a keyboard, a virtual keyboard, a button, a knob, a joystick, a switch, a dial, an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, one or more depth sensors (e.g., a structured light, a time-of-flight, or the like), or the like. In some implementations, movement, rotation, or position of the first device105detected by the one or more I/O devices and sensors606provides input to the first device105.

In some implementations, the one or more displays612are configured to present the CGR environment. In some implementations, the one or more displays612correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS), or the like display types. In some implementations, the one or more displays612correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. In one example, the first device105includes a single display. In another example, the first device105includes a display for each eye. In some implementations, the one or more displays612are capable of presenting CGR content.

In some implementations, the one or more image sensor systems614are configured to obtain image data that corresponds to at least a portion of a scene local to the first device105. The one or more image sensor systems614can include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome camera, IR camera, event-based camera, or the like. In various implementations, the one or more image sensor systems614further include illumination sources that emit light, such as a flash.

The memory620includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory620includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory620optionally includes one or more storage devices remotely located from the one or more processing units602. The memory620comprises a non-transitory computer readable storage medium. In some implementations, the memory620or the non-transitory computer readable storage medium of the memory620stores the following programs, modules and data structures, or a subset thereof including an optional operating system630and one or more applications640.

The operating system630includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the operating system630includes an CGR viewer unit632that is configured to be called from the one or more applications640to display a CGR environment within a user interface provided by each of the one or more applications640.

In some implementations, each of the one or more applications640is configured to provide a user interface that allows the user to send and receive communications and to display 3D content corresponding to a data object included in or accessed via a received communication. To that end, in various implementations, the one or more applications640each includes a communications unit642, a communications user interface unit644, and a CGR experience unit646. In some implementations, the communications unit642is configured to send and receive communications including but not limited to SMS messages, MMS messages, text messages, e-mails, social media messages, and the like. In some implementations, the communications interface unit is configured to provide the user interface for displaying received communications or composing and sending communications to other devices, other accounts, and other users. In some implementations, the CGR experience unit646is configured to provide a CGR experience. For example, the CGR experience unit646may display 3D content corresponding to a received data object in the communications user interface. In some implementations, the CGR experience unit646includes a plugin that launches the CGR viewer unit632to display 3D content corresponding to a received data object within the communications user interface.

In some implementations, the first device105is a head-mounted device. Such a head-mounted device can include a housing (or enclosure) that houses various components of the head-mounted device. The housing can include (or be coupled to) an eye pad disposed at a proximal (to the user) end of the housing. In some implementations, the eye pad is a plastic or rubber piece that comfortably and snugly keeps the head-mounted device in the proper position on the face of the user (e.g., surrounding the eye of the user). The housing can house a display that displays an image, emitting light towards one or both of the eyes of a user.

FIG. 7is a flowchart representation of a method700for displaying a CGR environment using 3D content shared from another device in accordance with some implementations. In some implementations, the method700is performed by a device (e.g., first device105ofFIGS. 1-6). The method700can be performed at a mobile device, desktop, laptop, or server device. The method700can be performed on a head-mounted device that has a screen for displaying 2D images or a screen for viewing stereoscopic images. In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block710, the method700receives a data object corresponding to 3D content. The data object is received from a separate device such as from a device used by another user to send the data object or from a server device from which the data object is retrieved. In some implementations, the data object is received in a text message. In some implementations, the data object is received in an e-mail message that attaches the data object. In some implementations, the data object is received based on user input. For example, the device may receive a message that has a link (e.g., a text message, e-mail message, or social media message having the link), receive input accessing the link (e.g., touching or clicking on the link), and responsive to detecting the input, access the link to download the data object.

At block720, the method700receives input corresponding to a user selection to view the 3D content in a CGR environment. In some implementations, the method700presents an icon, a graphic, or text representing 3D content corresponding to the data object on a user interface that is used for communications (e.g., sending/receiving messages, e-mails, and other communications) and receives input corresponds to a user selection of the icon, the graphic, or the text. In some implementations, a user interface displays an icon, graphic, or text representing such received 3D content and also displays an indicator (text, graphic, etc.) that indicates that a selection of the icon, graphic, or text will launch a viewer for viewing the received 3D content in a CGR viewing mode.

Blocks730and740are performed responsive to detecting the input to display a CGR environment at the device. At block730, the method700uses a camera to capture images. The images captured by the camera depict real world content at the scene of the device that can be included in the CGR environment.

At block740, the method700constructs the CGR environment using the data object and the captured images. In some implementations, constructing the CGR environment is performed by overlaying the 3D content on the captured images. In some implementations, constructing the CGR environment is performed by detecting one or more planar surfaces in the real world content depicted in the images and positioning the 3D content corresponding to the received data object in the CGR environment based on the detected planar surfaces. For example, at a first instant in time, an image of the received 3D content may be positioned over the most recently captured image of the captured images, at a second instant in time after the capture of an additional captured image, an image (the same or different) of the received 3D content may be positioned over the new recently captured image, etc. In some implementations, constructing the CGR environment is performed by constructing a 3D model corresponding to some or all of the real world content depicted in the images and adding the 3D content corresponding to the received data object to the model and then creating an image of the combined content from a particular viewpoint.

Once the CGR environment is displayed, the method700can involve changing the CGR environment based on user input. In some implementations, this involves receiving input to change position or rotation of the 3D content and, responsive to the input, changing the position or the rotation of the 3D content in the CGR environment. For example, the user may move a depiction of a couch to another location in the scene.

In some implementations, the change involves a change of viewpoint. For example, this can involve receiving a movement or rotation of the device and updating the displaying of the CGR environment based on the movement. As the user moves his/her head, moves around the room, jumps up and down, etc., the viewpoint changes. However, the position and rotation of the 3D content relative to real world objects depicted in the CGR environment remain constant. The couch remains in its position relative to the floor and other real world objects depicted in the scene. To maintain constant relative positioning planar surfaces or features are identified in the images and used to maintain the relative position of the received 3D content.

FIG. 8is a flowchart representation of a method800for displaying an CGR environment with received 3D content by launching a viewer from within an application in accordance with some implementations. In some implementations, the method800is performed by a device (e.g., first device105ofFIGS. 1-6). The method800can be performed at a mobile device, desktop, laptop, or server device. The method800can be performed on a head-mounted device that has a screen for displaying 2D images or a screen for viewing stereoscopic images. In some implementations, the method800is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method800is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block810, the method800presents, within an app, an icon graphic or text representing 3D content corresponding to a received data object. At block820, the method800receives input selecting the icon, graphic or text. Responsive to detecting the input, the method800provides a CGR environment via a technique depicted in blocks830,840,850.

At block830, the method800detects the file type of the data object. In some implementations, the file type is detected based on an extension of a file of the data object (e.g., .AC, .max, 0.3ds, 0.3dm, 0.3dmf, .dwg, .blend, .cob, .dae, 0.3dxml, .off, .x, .dxf, .x3d, .fmz, etc.). In some implementations, the file type is detected by inspecting the content of the data object.

At block840, the method800identifies a viewer based on the file type of the data object. In some implementations, the viewer is identified using a table that associates viewer applications with different respective file types or file extensions. For example, based on identifying that couch.AC has the file extension “.AC”, the method800can look up in a table on the device to identify that files having that extension require using a particular viewer. A plugin of the app may include functionality access such a table or otherwise identify the viewer based on the file type. In such instances, the plugin is a subset of code of the app that is added during development of the app to enable display of CGR environments within the app.

At block850, the method800launches the viewer within the app using the data object to provide an CGR environment that includes the 3D content. In some implementations, a plugin added to the app accesses a CGR viewer unit (e.g., CGR viewer unit632ofFIG. 6) that is part of an operating system on the device or that is otherwise process isolated from the app. In response, the CGR viewer unit launches executable code that provides a viewing/using interface for viewing or using an CGR environment within the user interface of the app. For example, the user interface of the app may display a popup window that presents the CGR environment and UI controls for controlling the viewing or use of the CGR environment. In some implementations, an app that provides a user interface for communications (e.g., sending/receiving messages and other communications) displays an embedded viewer interface provided by a separate, operating-system level CGR viewer unit. In some implementations, such a single CGR viewer unit is accessed and used by multiple, different applications on the device, for example, with in a text messaging app, an e-mail app, an web-browsing app, a social media app, a game, etc. to view 3D content having a one or more particular types (e.g., all data objects having files with extension .AC or .max, etc.).

FIG. 9is a flowchart representation of a method900for displaying an CGR environment with received 3D content based on planar surface detection in accordance with some implementations. In some implementations, the method900is performed by a device (e.g., first device105ofFIGS. 1-6). The method900can be performed at a mobile device, desktop, laptop, or server device. The method900can be performed on a head-mounted device that has a screen for displaying 2D images or a screen for viewing stereoscopic images. In some implementations, the method900is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method900is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block910, the method900detects a planar surface in a real world environment of captured images. In some implementations, this involves using a surface detection algorithm or neural network/machine learning procedure to analyze the color of pixels of the captured images to identify the surfaces. Detecting a planar surface can involve identifying feature points and corresponding planar surfaces in each of multiple images/frames of captured video images/frames. Detecting planar surfaces can additionally or alternatively involve the use of received IR or other sensor data identifying the distances corresponding to different portions of the captured images. For example, the captured images can be RGB-D images that combine RGB images from an RGB camera and depth images from a depth sensor.

At block920, the method900provides an CGR environment with 3D content corresponding to a received data object positioned on the planar surface. In some implementations, a planar surface is identified and the 3D content is positioned with a surface of the 3D content adjacent to the planar surface. For example, if a planar surface corresponding to a floor is detected, a couch may be positioned with its bottom surface adjacent or otherwise aligned with (e.g., parallel to, etc.) that planar surface. If the couch, table, or other depiction of a 3D object has legs, feet, wheels, or other supporting structures extending from its bottom, the bottoms of those components may be identified and positioned on the planar surface.

At block930, the method900receives input to change the position of the 3D content. In some implementations, the 3D content is depicted as moving while it is dragged via mouse-based or touch-based input. At block940, the method900repositions the 3D content on the planar surface in the CGR environment. In some implementations, the 3D content is depicted as moving while it is dragged during the input and then automatically moved to be adjacent to the 3D surface following the drag input. For example, a couch may be dragged in the air above a planar surface corresponding to a floor and then automatically moved to be adjacent to or otherwise aligned with the floor at the conclusion of the drag input.

FIG. 10is a flowchart representation of a method1000for displaying an CGR environment with received 3D content based on viewpoint changes in accordance with some implementations. In some implementations, the method1000is performed by a device (e.g., first device105ofFIGS. 1-6). The method1000can be performed at a mobile device, desktop, laptop, or server device. The method1000can be performed on a head-mounted device that has a screen for displaying 2D images or a screen for viewing stereoscopic images. In some implementations, the method1000is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method1000is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block1010, the method1000provides an CGR environment with 3D content corresponding to a received data object and captured images of a real world environment. At block1020, the method1000receives input to change the position of the 3D content and, at block1030, the method1000repositions the 3D content in the CGR environment based on the input. For example, in response to input, the method may reposition a depiction of a couch (3D object) between two end tables in a depiction of the real world scene.

At block1040, the method1000receives a change to a viewpoint associated with the CGR environment. In some implementations, the input to change the viewpoint is a rotation or translation of the device in the real world scene. At block1050, the method1000updates the CGR environment based on the change to the viewpoint using a constant relationship between the 3D content and real world content (e.g., real world objects, surfaces, features, etc.) from the captured images.

In some implementations, a device used to display an CGR environment having 3D content corresponding to a received data object is further configured to switch between a mixed reality (MR) viewing mode and a virtual reality (VR) viewing mode. For example, a user viewing the CGR environment400ofFIG. 5may switch to a VR viewing mode in which only the couch is displayed, i.e., the real world content is not included. In some implementations, this involves receiving input to switch to a VR viewing mode and, responsive to detecting the input, discontinuing display of an MR environment at the device and initiating a VR display of the 3D content without the real world scene. In some implementations, the capturing of images of the real world scene by the camera is discontinued. However, in some implementations, the device continues to capture images of the real world scene by the camera while in VR viewing mode. By doing so, the device is enable to switch back to the MR viewing mode more quickly. For example, to enable quicker switch from VR viewing mode to MR viewing mode, the system is configured to continue plane detection in the background using the one or more images while in VR viewing mode. Note that use of an operating system level CGR viewer may provide better control over the camera of the device and thus enable the use of the camera in VR viewing mode to facilitate the continued image capture and plane detection that enables faster mode transitioning.

In some implementations, a viewer is configured to toggle between VR viewing mode and MR viewing mode and to provide different features in each of the different viewing modes. In some implementations, a viewer is configured with VR mode functionality that enables 3D content to be rotated and zoomed to easily view different sides and characteristics of the 3D content and with AR functionality that positions the 3D content adjacent to or aligned with a real world 3D surface and that maintains the 3D content in a constant position/rotation relative to the real world scene.

In some implementations, a viewer is configured to transition between VR viewing mode and MR viewing mode and vice versa by moving 3D content from one coordinate system into another coordinate system. In some implementations, the viewer is configured to make the viewing mode transition appear smooth and non-jarring so that the 3D content does not appear to jerk to a new position during the transition. In some implementations, the 3D content is positioned on a planar surface at a position that reduces the amount of apparent movement of the object. In some implementations this involves shooting a ray through the screen (perpendicular to the plane of the screen) to the base of the 3D content (e.g., a line corresponding the user's line of sight) to identify a position in the real world scene for the 3D content. This minimizes the apparent translation of the 3D content to the user. In some implementations, the transition involves creating an animation path along the ray that provides a smooth transition. The 3D content is moved over time along such an animation path rather than all at once to avoid a jerky appearance.