English:
[0001] Virtual reality (VR) is a form of computer generated environment (i.e., a virtual world) with which a user can interact in ways that may be similar to or different from the ways that the user interacts with the physical environment. A VR system produces video images that present the virtual world to the system user. A VR system may include various components, such as a headset to display the video images to the user and a controller to receive control input from the user.
[0003] FIG. 1 shows a virtual reality (VR) system with object detection in accordance with various examples;
[0004] FIG. 2 shows another implementation of a virtual reality (VR) system with object detection in accordance with various examples;
[0005] FIG. 3 shows a block diagram of a monitoring station for object detection in a VR system in accordance with various examples;
[0006] FIG. 4 shows a block diagram of another implementation of a monitoring station for object detection in a VR system in accordance with various examples;
[0007] FIG. 5 shows a block diagram of an image processing device for object detection in a VR system in accordance with various examples;
[0008] FIGs. 6-8 show displays of object detection information in a VR headset in accordance with various examples;
[0009] FIG. 9 shows a flow diagram for a method for object detection in a VR system in accordance with various examples; and
[0010] FIG. 10 shows a flow diagram for another implementation of a method for object detection in a VR system in accordance with various examples.
[0011] Virtual reality (VR) systems allow a user to interact with an environment that is generally unavailable to the user in the physical world. One goal of VR systems is to provide the user with an experience that is a close analog to the user's experience and interaction with the physical world. To achieve a convincing experience, the VR system generally visually isolates the user from the physical environment by blocking the user's vision with respect to the physical environment. Some systems may further enhance experience of the virtual world by isolating the user's hearing to the virtual world. Thus, the user's experience in VR may be limited to only the video and audio signals generated by the VR system. While such visual and auditory isolation may enhance the user's experience in the virtual world, the user remains subject to forces present in the physical world.
[0012] Because the user of a VR system is sensorially isolated from the physical environment, the user may be vulnerable, or feel vulnerable, to conditions present in the physical environment. For example, objects that move in the physical environment proximate the VR system user will generally be invisible to the user and a collision between the object and the user may occur. Such objects may include other users of the VR system, human beings or animals generally, and other objects subject to change of proximity to the user caused by movement of the user or movement of the object.
[0013] The VR system disclosed herein monitors the physical environment in which a VR system user operates to identify objects with which the user may collide. The VR system includes a plurality of monitoring stations that capture video images of the physical environment. The monitoring stations may be disposed at the periphery of the VR system operational environment to capture images of the environment from different angles. The images captured by the monitoring stations may be provided to an image processing device, such as a computer, that identifies objects in the captured images that present potential collision hazards for the VR system user. On identification of an object that presents a potential collision hazard, the image processing device may transmit information to the VR headset of the user that informs the user of the presence of the identified object. For example, on receipt of the information from the image processing system, the VR headset may display a representation of the object at location relative the user that approximates the location of the object relative the user in the physical environment. Accordingly, the VR system disclosed herein may apprise the user of the VR system of the presence of an object in the physical environment and allow the user to avoid a collision with object.
[0014] FIG. 1 shows a VR system 100 with object detection in accordance with various examples. The VR system 100 includes a monitoring stations 102-1 and 102-2 (collectively referred to as monitoring stations 102), a VR headset 108, and an image processing device 106. While two monitoring stations 102 are shown in the VR system 100, some examples of the VR system 100 may include more or less than two monitoring stations 102. Similarly, although FIG. 1 depicts the image processing device 106 as separate from the VR headset 108 and the monitoring stations 102, in some implementations of the VR system 100 the image processing device 106 may a component of or integrated with the VR headset 108 or one of the monitoring stations 102.
[0015] Each of the monitoring stations 102 includes a distinct image sensor. For example, monitoring station 102-1 may include an imaging sensor 104-1 . Monitoring station 102-2 may include an imaging sensor 104-2. Imaging sensors 104-1 and 104-2 may be collectively referred to as imaging sensor 104. The image sensor 104 may be an infra-red (IR) depth sensing camera, a camera that captures luma and/or chroma information, or other image sensor suitable for capturing images of objects in a VR operating environment. In FIG. 1 , monitoring stations 102 are disposed to capture images of the VR operating environment 1 14.
[0016] The VR operating environment 1 14 is the physical area in which a user of the VR system 100, i.e., a user of the VR headset 108, interacts with the virtual world presented to the user via the VR headset 108. For example, the VR headset 108 includes video display technology that displays images of the virtual world to the user of the VR headset 108. The monitoring stations 102 may be disposed at different locations in the VR operating environment 1 14 to capture images of the VR operating environment 1 14 from different angles.
[0017] The image processing device 106 is communicatively coupled to the monitoring stations 102. For example, the image processing device 106 may be communicatively coupled to the monitoring stations 102 via a wired or wireless communication link, such as a IEEE 802.1 1 wireless network, an IEEE 802.3 wired network, or any other wired or wireless communication technology that allows the images of the VR operating environment 1 14 captured by the monitoring stations 102 to be transferred from the monitoring stations 102 to the image processing device 106. The spatial relationship of the monitoring stations 102 may be known to the image processing device 106. For example, the relative angle(s) of the optical center lines of the image sensors 104 of the monitoring stations 102 may be known to the image processing device 106. That is, the monitoring stations 102 may be disposed to view the VR operating environment 1 14 at a predetermined relative angle, or the angle at which the monitoring stations 102 are disposed relative to one another may be communicated to the image processing device 106.
[0019] When the image processing device 106 detects an object in the VR operating environment, the image processing device 106 communicates to the VR headset 108 information regarding the detected object. The image processing device 106 is communicatively connected to the VR headset 108. For example, the image processing device 106 may be communicatively coupled to the VR headset 108 via a wired or wireless communication link, such as a IEEE 802.1 1 wireless network, an IEEE 802.3 wired network, or any other wired or wireless communication technology that allows information to be transferred from the image processing device 106 to the VR headset 108.
[0020] The image processing device 106 may determine the location (in three- dimensional space) of an object detected as moving (and pixels representing points of the object) in the images captured by each of the monitoring stations 102 at a given time. Using the relative locations of the monitoring stations 102 known by the image processing device 106, the image processing device 106 can determine the location of the detected object (e.g., determine the location of a point of the object identified in images captured by different monitoring stations 102) by applying triangulation. That is, the image processing device 106 may determine the location of a detected object in three-dimensional space by triangulating a common point found in multiple two-dimensional images captured by different monitoring stations 102 (e.g., monitoring station 102-1 and 102-2). The image processing device 106 may transmit location information for a detected object to the VR headset 108.
[0021] The image processing device 106 may also determine the identity of an object detected in the VR operating environment 1 14. For example, on detection of an object in the VR operating environment 1 14, the image processing device 106 may apply a Viola-Jones object detection framework to identify the object based on the features of the object captured in an image. The image processing device 106 may transmit identity information (e.g., person, animal, etc.) for a detected object to the VR headset 108.
[0022] The VR headset 108 displays the information provided by the image processing device 106 for communication to a user. If the image processing device 106 renders the video displayed by the VR headset 108, then the image processing device 106 may include the information identifying a detected object in the video frames transmitted to the VR headset 108. If the VR headset 108 itself produces the video data displayed by the VR headset 108, then the VR headset 108 may receive the information about the detected object from the image processing device 106 and integrate the information in the video generated by the VR headset 108. Having been made aware of the presence of detected object, and optionally the objects location and/or identity, by the information presented via the VR headset 108, the user will not be surprised by the presence of the detected object in the VR operating environment and can avoid collision with the detected object.
[0023] FIG. 2 shows additional detail of the VR system 100 in accordance with various examples. The VR system 100 may also include one or more controllers 1 12. The controller 1 12 is an input device that the user of the VR headset 108 manipulates to interact with the virtual world. The controller 1 12 may be, for example, a handheld device that the user operates to digitally interact with virtual objects in the virtual world. To identify objects in the VR operating environment 1 14 that may interfere with the user of the VR headset 108, the image processing device 106 identifies an area about the VR headset 108 and the controller 1 12 as corresponding to the user. For example, the image processing device 106 may define an area 1 16 extending from above the VR headset 108 to the floor and to a maximum or predetermined extension of the controller 1 12 about a central axis corresponding to the user as representing the area occupied by the user of the VR system. The image processing device 106 may disregard motion detected within the area 1 16 when detecting objects in the VR operating environment 1 14. Thus, the image processing device 106 may detect motion outside the area 1 16 as corresponding to an object that may be of interest to the user of the VR system 100 while ignoring motion inside the area 1 16 with respect to object detection.
[0024] Some implementations of the image processing device 106 may dynamically map the body and limbs of a user of the VR headset 108 to define an area within which motion is disregarded. For example, the image processing device 106 may determine the location of the controller(s) 1 12 knowing that the distal end of the user's arm is attached to a controller 1 12. The VR headset 108 identifies the location of the user's body and the user's legs are connected to the body. Using this information, the image processing device 106 may determine a dynamic amorphous boundary that defines an area within which motion is disregarded.
[0025] FIG. 3 shows a block diagram of a monitoring station 102 for object detection in a VR system in accordance with various examples. The monitoring station 102 includes an image sensor 304 and a transmitter 302. The image sensor 304 may be a red-green-blue light sensor of any resolution suitable for detection of movement in the VR operating environment 1 14. In some implementations of the monitoring station 102, the image sensor 304 may be an IR depth sensor that includes an IR projector and an IR camera. In the IR depth sensor, the IR projector may project a pattern of IR points and the IR camera may capture images of the points reflected from objects in the VR operating environment that can processed to determine distance.
[0026] The image sensor 304 is communicatively coupled to the transmitter 302. Images captured by image sensor 304 are transferred to the transmitter 302, which transmits the images to the image processing device 106. The transmitter 302 may include circuitry to transmit the images via wired or wireless media. The monitoring station 102 may include additional components that have been omitted from FIG. 3 in the interest of clarity. For example, the monitoring station 102 may include a processor coupled to the image sensor 304 and the transmitter 302, where the processor provides control and image transfer functionality in the monitoring station 102.
[0027] FIG. 4 shows a block diagram of a monitoring station 102 for object detection in a VR system in accordance with various examples. The monitoring station 102 of FIG. 4 is similar to that shown in FIG. 3 and further includes headset and controller transducers 402. The headset and controller transducers 402 may generate optical timing signals that allow the location and orientation of the VR headset 108 and each VR controller 1 12 to be determined with sufficient specificity to facilitate accurate determination of the position of the VR headset 108 and the VR controller 1 12 in the VR operating environment 1 14. The headset and controller transducers 402 may include IR emitters that generate timed signals (e.g., a reference pulse and swept plane) for reception by sensors on the VR headset 108 and the VR controller 1 12. The location and orientation of the VR headset 108 and controller 1 12 can be determined based on which sensors of the VR headset 108 and controller 1 12 detect signals generated by the transducers 402 and relative timing of pulse and sweep detection.
[0028] FIG. 5 shows a block diagram of an image processing device 106 for object detection in a VR system in accordance with various examples. The image processing device 106 includes a transceiver 502, a processor 504, and storage 506. The image processing device 106 may also include various components and systems that have been omitted from FIG. 5 in the interest of clarity. For example, the image processing device 106 may include display systems, user interfaces, etc. The image processing device 106 may be implemented in a computer as known in the art. For example, the image processing device 106 may be implemented using a desktop computer, a notebook computer, rack-mounted computer, a tablet computer or any other computing device suitable for performing the image processing operations described herein for detection of objects in the VR operating environment 1 14.
[0029] The transceiver 502 communicatively couples the image processing device 106 to the monitoring stations 102 and the VR headset 108. For example, the transceiver 502 may include a network adapter that connects the image processing device 106 to a wireless or wired network that provides communication between the monitoring stations 102 and the image processing device 106. Such a network may be based on any of a variety of networking standards (e.g., IEEE 802) or be proprietary to communication between devices of the VR system 100.
[0030] The processor 504 is coupled to the transceiver 502. The processor 504 may include a general-purpose microprocessor, a digital signal processor, a microcontroller, a graphics processor, or other device capable of executing instructions retrieved from a computer-readable storage medium. Processor architectures generally include execution units (e.g., fixed point, floating point, integer, etc.), storage (e.g., registers, memory, etc.), instruction decoding, instruction and data fetching logic, peripherals (e.g., interrupt controllers, timers, direct memory access controllers, etc.), input/output systems (e.g., serial ports, parallel ports, etc.) and various other components and sub-systems.
[0031] The storage 506 is a computer-readable medium that stores instructions and data for access and use by the processor 504. The storage 506 may include any of volatile storage such as random access memory, non-volatile storage (e.g., a hard drive, an optical storage device (e.g., CD or DVD), FLASH storage, read-only-memory), or combinations thereof. The storage 506 includes object detection 508, object identification 512, object location 514, object prioritization 516, and images 518. The images 518 include images of the VR operating environment 1 14 captured by the monitoring stations 102 and transferred to the image processing device 106 via the transceiver 502. The object detection 508, object identification 512, object location 514, and object prioritization 516 include instructions that are executed by the processor 504 to process the images 518.
[0032] The objection detection 508 includes instructions that are executable by the processor 504 to detect objects in the VR operating environment 1 14. The objection detection 508 may include motion detection 510. The motion detection 510 includes instructions that are executable by the processor 504 to detect motion in the VR operating environment 1 14. The motion detection 510 may detect motion by identifying image to image changes in images 518, by identifying a difference between a reference image and a given one of the images 518, or by other motion detection processing. In some implementations, the object detection 508 may ignore motion detected within the area 1 16 corresponding to a user of the VR headset 108. The object detection 508 may identify points associated with a detected object using edge detection. The presence of a detected object may be communicated to the VR headset 108 by the image processing device 106.
[0033] The object location 514 includes instructions that are executable by the processor 504 to determine the three dimensional location of an object detected by the objection detection 508. For example, if the object detection 508 identifies an object in images 518 captured by different monitoring stations 102, then the object location 514 can determine the location of the object in the three dimensional VR operating environment 1 14 by triangulation based on the known locations of the monitoring stations 102 and the known angles between the optical centerlines of the image sensors 304 of the different monitoring stations 102. The location of the object may be communicated to the VR headset 108 by the image processing device 106.
[0034] The object identification 512 includes instructions that are executable by the processor 504 to determine the identity of an object detected by the objection detection 508. For example, the object identification 512 may include a Viola-Jones Framework trained to identify a variety of objects that may be present in the VR operating environment 1 14, such as people and pets. The pixels corresponding to the detected object may be processed by the Viola- Jones Framework to identify the object. The identity of the object may be communicated to the VR headset 108 by the image processing device 106.
[0035] The object prioritization 516 includes instructions that are executable by the processor 504 to prioritize the object-related information provided to the VR headset 108 by the image processing device 106. Prioritization may be based on object size, distance between the object and the VR headset 108, object elevation, and/or other factors indicative of collision between the detected object and a user of the VR headset 108. In some implementations, the object prioritization 516 may prioritize a plurality of detected objects according to a risk of collision between the object and the user of the VR headset 108 and communicate information concerning the objects to the VR headset 108 in order of highest determined collision risk.
[0036] The storage 506 may include additional logic that has been omitted from FIG. 5 in the interest of clarity. For example, the storage 506 may include VR headset communication that includes instructions that are executable by the processor 504 to transfer information concerning a detected object (e.g., object presence, location, and/or identity) to the VR headset 108. In some implementations, the image processing device 106 may produce the video frames displayed by the VR headset 108. In such implementations, the image processing device 106 may include video rendering instructions that are executable by the processor 504 to generate the video frames displayed by the VR headset 108, and may include in the video frames information concerning a detected object.
[0037] FIGs. 6-8 show display of object detection information in the VR headset 108 in accordance with various examples. The displays of FIGs. 6-8 may be stereoscopic in practice, but are shown as monoscopic in FIGs. 6-8 to promote clarity. In FIG. 6, the VR headset 108 displays an object detection warning as a text field 602. The text field 602 may specify the location and identity of the object in addition to the presence of the object.
[0038] In FIG. 7, the VR headset 108 displays an object detection warning as an outline 702 or silhouette of a detected object. The outline 702 of the object may correspond to the identity of the object determined by the object identification 512, and the location of the object in the display of the VR headset 108 may correspond to the location of the object determined by the object location 514.
[0039] In FIG. 8, the VR headset 108 displays a warning symbol 802 indicating that an object has been detected and is to the left of the viewing field of the VR headset 108. Some implementations may combine the object information displays of FIGs. 6-8. For example, an object outside of the viewing area of the VR headset 108 may be indicated as per FIG. 8, and as the VR headset 108 is turned towards the object an outline 702 or silhouette of the object may be displayed as per FIG. 7. Similarly, text display 602 may be combined with either the warning symbol 802 or the object outline 702 to provide additional information to the user of the VR headset 108.
[0040] FIG. 9 shows a flow diagram for a method 900 for object detection in a VR system in accordance with various examples. Though depicted sequentially as a matter of convenience, at least some of the actions shown can be performed in a different order and/or performed in parallel. Additionally, some implementations may perform only some of the actions shown. In some implementations, at least some of the operations of the method 900 can be implemented as instructions stored in a storage device and executed by one or more processors.
[0041] In block 902, the monitoring stations 102 capture images of the VR operating environment 1 14. The monitoring stations 102 are disposed at different locations along the perimeter of the VR operating environment 1 14. The locations of the monitoring stations 102 and the angles of intersection of the optical centerlines of the image sensors 104 of the monitoring stations 102 may be known to the image processing device 106. The images captured by the monitoring stations 102 may include color information (chroma) and brightness information (luma), or may include only luma. The monitoring stations may capture images as a video stream. For example, the monitoring station 102 may capture images at a rate of 10, 20, 30, etc. images per second. The monitoring stations 102 transfer the captured images to the image processing device 106 for use in object detection.
[0042] In block 904, the image processing device 106 processes the images captured by the monitoring stations 102 to detect objects in the VR operating environment 1 14. The processing may include application of motion detection to identify an object in the VR operating environment 1 14. For example, the image processing device 106 may detect motion by comparing two images and identifying differences between the two images. The two images may be images captured by the same monitoring station 102 (e.g., monitoring station 102-1 ) at different times. One of the images may be a reference image. Further, as explained with regard to FIG. 2, the image processing device 106 may identify the VR headset 108 and/or a VR controller 1 12 in the images and ignore differences in the images (i.e., ignore motion) in a predetermined area
[0043] In block 906, the image processing device 106 has detected an object in the VR operating environment 1 14 and transmits information concerning the detected object to the VR headset 108. The information may include a notification of the objects presence in the VR operating environment 1 14. The information may be embedded in a video stream provided by the image processing device 106 for display by the VR headset 108, or provided separately from any video information transferred to the VR headset 108. The information may be transferred via a wired or wireless communication channel provided between the image processing device 106 and the VR headset 108. The communication channel may include a wired or wireless network in accordance with various networking standards.
[0044] In block 908, the VR headset 108 displays the information received from the image processing device 106 concerning the detected object. The information may be displayed in conjunction with a VR scene displayed by the VR headset 108.
[0045] FIG. 10 shows a flow diagram for a method 1000 for object detection in a VR system in accordance with various examples. Though depicted sequentially as a matter of convenience, at least some of the actions shown can be performed in a different order and/or performed in parallel. Additionally, some implementations may perform only some of the actions shown. In some implementations, at least some of the operations of the method 1000 can be implemented as instructions stored in a storage device and executed by one or more processors.
[0046] In block 1002, the monitoring stations 102 capture images of the VR operating environment 1 14. The monitoring stations 102 are disposed at different locations at the perimeter of the VR operating environment 1 14. The locations of the monitoring stations 102 and the angles of intersection of the optical centerlines of the image sensors 104 of the monitoring stations 102 may be known to the image processing device 106. The images captured by the monitoring stations 102 may include color information (chroma) and brightness information (luma), or may include only luma. The monitoring stations may capture images as a video stream. For example, the monitoring station 102 may capture images at a rate of 10, 20, 30, etc. images per second. The monitoring stations 102 transfer the captured images to the image processing device 106 for use in object detection.
[0047] In block 1004, the image processing device 106 processes the images captured by the monitoring stations 102 to detect objects in the VR operating environment 1 14. The processing may include application of motion detection to identify an object in the VR operating environment 1 14. For example, the image processing device 106 may detect motion by comparing two images and identifying differences between the two images. The two images may be images captured by the same monitoring station 102 at different times. One of the images may be a reference image. The image processing device106 may identify the VR headset 108 and/or a VR controller 1 12 in the images and ignore differences in the images (i.e., ignore motion) in a predetermined area 1 16 about the VR headset 108 and/or a VR controller 1 12 as attributable to movement of a user of the VR headset 108.
[0048] In block 1006, the image processing device 106 processes the images captured by the monitoring stations 102 to determine the location of any detect objects in the VR operating environment 1 14. The location determination processing may include identifying the detected object in time coincident images captured by different monitoring stations 102, and applying triangulation to points of the detected object locate the object in three dimensions.
[0049] In block 1008, the image processing device 106 processes the images captured by the monitoring stations 102 to determine the identity of any objects detected in the VR operating environment 1 14. The identify determination processing may include extracting and/or isolating an image of a detected object from an image captured by a monitoring station 102, and providing the image of the detected object to a Viola-Jones Framework. The Viola-Jones Framework may extract features from the image and apply one or more classifiers to identify the object. Various other object identification algorithms may be used in some implementations of the identity determination processing.
[0050] In block 1010, the image processing device 106 prioritizes detected objects for presentation to the VR headset 108. Prioritization may be based on object size, distance between the object and the VR headset 108, object elevation, rate of movement towards the user, and/or other factors indicative of collision between the detected object and a user of the VR headset 108. Some implementations of the object prioritization processing may prioritize a plurality of detected objects according to a risk of collision between the object and the user of the VR headset 108 and communicate information concerning the objects to the VR headset 108 in order of highest determined collision risk.
[0051] In block 1012, the image processing device 106 has detected an object in the VR operating environment 1 14 and transmits information concerning the detected object to the VR headset 108. The information may include one or more notifications of the objects present in the VR operating environment 1 14, location of the detected objects in the VR operating environment 1 14, and identity of the detected objects. The information may be embedded in a video stream provided by the image processing device 106 for display by the VR headset 108, or provided separately from any video information transferred to the VR headset 108. The information may be transferred via a wired or wireless communication channel provided between the image processing device 106 and the VR headset 108. The communication channel may include a wired or wireless network in accordance with various networking standards.
[0052] In block 1014, the VR headset 108 displays the information received from the image processing device 106 concerning the detected object to allow a user of the VR headset 108 to become aware of the detected object, and potentially avoid a collision with the detected object. The information may be displayed in conjunction with a VR scene displayed by the VR headset 108.
[0053] The above discussion is meant to be illustrative of the principles and various examples of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

中文:
[0001]虚拟现实(VR)是一种形式的计算机生成的环境(即虚拟世界),利用所述环境,用户能以可同用户与物理环境交互的方式类似或不同的方式交互。VR系统产生视频图像,所述视频图像向系统用户呈现虚拟世界。VR系统可以包括各种组件,诸如用于向用户显示视频图像的头戴式显示器(headset)以及用于从用户接收控制输入的控制器。
[0003]图1示出了根据各种示例的、具有对象检测的虚拟现实(VR)系统;
[0004]图2示出了根据各种示例的、具有对象检测的虚拟现实(VR)系统的另一实现方式;
[0005]图3示出了根据各种示例的用于VR系统中的对象检测的监控站的框图;
[0006]图4示出了根据各种示例的用于VR系统中的对象检测的监控站的另一实现方式的框图;
[0007]图5示出了根据各种示例的用于VR系统中的对象检测的图像处理设备的框图;
[0008]图6-8示出了根据各种示例的VR头戴式显示器中的对象检测信息的显示;
[0009]图9示出了根据各种示例的用于VR系统中的对象检测的方法的流程图;并且
[0010]图10示出了根据各种示例的用于VR系统中的对象检测的方法的另一实现方式的流程图。
[0011]虚拟现实(VR)系统允许用户与一般在物理世界中对于用户而言不可得的环境交互。VR系统的一个目标是为用户提供如下体验:所述体验是与用户在物理世界情况下的体验和同物理世界的交互的接近的模拟。为了实现令人信服的体验,VR系统一般通过阻断用户关于物理环境的视觉来在视觉上隔离用户与物理环境。一些系统可以通过将用户的听觉隔离至虚拟世界来进一步增强虚拟世界的体验。因而,用户在VR中的体验可以被限制于仅仅是通过VR系统所生成的视频和音频信号。虽然这样的视觉和听觉隔离可以增强用户在虚拟世界中的体验,但是用户保持经受在物理世界中存在的力。
[0012]由于VR系统的用户在感觉上与物理世界隔离,所以用户可易受或感觉易受物理环境中存在的条件所伤害。例如,在物理环境中、邻近于VR系统用户而移动的对象对于用户而言将一般不可见,并且在对象与用户之间的碰撞可发生。这样的对象可以包括VR系统的其它用户,一般是人类或动物,以及其它对象,其经受由用户的移动或对象的移动所引起的相对于用户的邻近度的改变。
[0013]本文中所公开的VR系统监控物理环境,其中VR系统用户操作以标识用户可与之碰撞的对象。所述VR系统包括多个监控站,所述监控站捕获物理环境的视频图像。所述监控站可以被布置在VR系统操作环境的外围,用于从不同角度捕获环境的图像。由监控站所捕获的图像可以被提供到图像处理设备、诸如计算机,所述图像处理设备在所捕获的图像中标识呈现针对VR系统用户的潜在碰撞危险的对象。在标识到呈现潜在碰撞危险的对象时,图像处理设备可以将信息传输到用户的VR头戴式显示器,其向用户告知所标识的对象的存在。例如,在从图像处理系统接收到信息时,VR头戴式显示器可以在相对于用户的位置处显示对象的表示,所述位置与物理环境中对象相对于用户的位置近似。因此,本文中所公开的VR系统可以向VR系统的用户通知物理环境中对象的存在并且允许用户避免与对象碰撞。
[0014]图1示出了根据各种示例的、具有对象检测的VR系统100。VR系统100包括监控站102-1和102-2(共同被称为监控站102)、VR头戴式显示器108、以及图像处理设备106。虽然在VR系统100中示出了两个监控站102,但是VR系统100的一些示例可以包括多于或少于两个监控站102。类似地,尽管图1将图像处理设备106描绘为与VR头戴式显示器108和监控站102分离,但是在VR系统100的一些实现方式中,图像处理设备106可以是监控站102之一或者VR头戴式显示器108的组件或与其集成。
[0015]监控站102中的每一个包括不同的图像传感器。例如,监控站102-1可以包括成像传感器104-1。监控站102-2可以包括成像传感器104-2。成像传感器104-1和104-2可以共同被称为成像传感器104。图像传感器104可以是红外(IR)深度感测相机、捕获亮度和/或色度信息的相机、或者适合用于捕获VR操作环境中的对象的图像的其它图像传感器。在图1中,监控站102被布置成捕获VR操作环境114的图像。
[0016]VR操作环境114是如下物理区域:在所述物理区域中,VR系统100的用户、即VR头戴式显示器108的用户与经由VR头戴式显示器108被呈现给用户的虚拟世界交互。例如,VR头戴式显示器108包括视频显示技术,所述视频显示技术向VR头戴式显示器108的用户显示虚拟世界的图像。监控站102可以被布置在VR操作环境114中的不同位置处,用于从不同角度捕获VR操作环境114的图像。
[0017]图像处理设备106被通信地耦合到监控站102。例如,图像处理设备106可以经由以下各项来被通信地耦合到监控站102:有线或无线通信链路、诸如IEEE 802.11无线网络、IEEE 802.3有线网络、或者允许通过监控站102所捕获的VR操作环境114的图像从监控站102被传递到图像处理设备106的任何其它有线或无线通信技术。监控站102的空间关系对于图像处理设备106而言可以是已知的。例如,监控站102的图像传感器104的光学中心线的(多个)相对角度对于图像处理设备106而言可以是已知的。也就是说,监控站102可以被布置成以预定的相对角度或监控站102相对于彼此被布置所用的角度来查看VR操作环境114,可以被传送到图像处理设备106。
[0019]当图像处理设备106在VR操作环境中检测到对象的时候,图像处理设备106向VR头戴式显示器108传送关于所检测到的对象的信息。图像处理设备106被通信地连接到VR头戴式显示器108。例如,图像处理设备106可以经由以下各项来被通信地耦合到VR头戴式显示器108:有线或无线通信链路、诸如IEEE 802.11无线网络、IEEE 802.3有线网络、或者允许信息从图像处理设备106被传递到VR头戴式显示器108的任何其它有线或无线通信技术。
[0020]图像处理设备106可以确定被检测到为在通过监控站102中的每一个在给定时间所捕获的图像中移动的对象的(在三维空间中的)位置(以及表示对象的点的像素)。通过使用图像处理设备106所已知的监控站102的相对位置,图像处理设备106可以通过应用三角测量来确定所检测到的对象的位置(例如确定在通过不同的监控站102所捕获的图像中所标识的对象的点的位置)。也就是说,图像处理设备106可以通过对在通过不同的监控站102(例如监控站102-1和102-2)所捕获的多个二维图像中所发现的公共点进行三角测量来确定所检测到的对象在三维空间中的位置。图像处理设备106可以将针对所检测到的对象的位置信息传输到VR头戴式显示器108。
[0021]图像处理设备106还可以确定在VR操作环境114中所检测到的对象的标识。例如,在VR操作环境114中检测到对象时,图像处理设备106可以应用Viola-Jones对象检测框架来基于图像中所捕获的对象的特征来标识对象。图像处理设备106可以将针对所检测到的对象的标识信息(例如人、动物等等)传输到VR头戴式显示器108。
[0022]VR头戴式显示器108显示由图像处理设备106所提供的信息,以用于传送给用户。如果图像处理设备106渲染由VR头戴式显示器108所显示的视频,那么图像处理设备106可以包括如下信息:所述信息标识被传输到VR头戴式显示器108的视频帧中的所检测到的对象。如果VR头戴式显示器108本身产生通过VR头戴式显示器108所显示的视频数据,那么VR头戴式显示器108可以从图像处理设备106接收与所检测到的对象相关的信息,并且将所述信息集成在通过VR头戴式显示器108所生成的视频中。已经通过经由VR头戴式显示器108所呈现的信息而被使得知晓了所检测到的对象的存在、以及可选地对象位置和/或标识,用户将不被所检测到的对象在VR操作环境中的存在所惊讶到,并且能避免与所检测到的对象的碰撞。
[0023]图2示出了根据各种示例的VR系统100的附加细节。VR系统100还可以包括一个或多个控制器112。控制器112是输入设备,VR头戴式显示器108的用户操纵所述输入设备来与虚拟世界交互。控制器112可以是例如手持式设备,用户操作所述手持式设备来与虚拟世界中的虚拟对象进行数字交互。为了标识VR操作环境114中可干扰VR头戴式显示器108的用户的对象,图像处理设备106将VR头戴式显示器108和控制器112周围的区域标识为对应于用户。例如,图像处理设备106可以将区域116限定为表示由VR系统的用户所占据的区域,所述区域116从VR头戴式显示器108上方延伸到地面以及延伸到控制器112关于同用户对应的中心轴的最大或预定延伸。当在VR操作环境114中检测对象的时候,图像处理设备106可无视在区域116内所检测到的运动。因而,图像处理设备106可以将区域116外部的运动检测为对应于可令VR系统100的用户感兴趣的对象,而忽视关于对象检测的区域116内部的运动。
[0024]图像处理设备106的一些实现方式可以动态地映射VR头戴式显示器108的用户的身体和四肢,以限定在其内运动被无视的区域。例如,已知用户的手臂的远端被附连到控制器112,图像处理设备106可以确定(多个)控制器112的位置。VR头戴式显示器108标识用户的身体的位置,并且用户的腿连接到身体。通过使用该信息,图像处理设备106可以确定动态无定形边界,所述动态无定形边界限定在其内运动被无视的区域。
[0025]图3示出了根据各种示例的用于VR系统中的对象检测的监控站102的框图。监控站102包括图像传感器304和传输器302。图像传感器304可以是适合用于检测VR操作环境114中的移动的、具有任何分辨率的红绿蓝光传感器。在监控站102的一些实现方式中,图像传感器304可以是IR深度传感器,其包括IR投影仪和IR相机。在IR深度传感器中,IR投影仪可以投影IR点的图案,并且IR相机可以捕获从VR操作环境中的对象反射的点的图像,其可以被处理以确定距离。
[0026]图像传感器304被通信地耦合到传输器302。由图像传感器304所捕获的图像被传递到传输器302,其将图像传输到图像处理设备106。传输器302可以包括用于经由有线或无线介质来传输图像的电路。监控站102可以包括已经为了清楚而从图3被省略的附加组件。例如,监控站102可以包括被耦合到图像传感器304和传输器302的处理器,其中处理器提供在监控站102中的控制和图像传递功能性。
[0027]图4示出了根据各种示例的用于VR系统中的对象检测的监控站102的框图。图4的监控站102类似于图3中所示的那个,并且此外包括头戴式显示器和控制器换能器402。头戴式显示器和控制器换能器402可以生成光学定时信号,所述光学定时信号允许VR头戴式显示器108和每个VR控制器112的位置和定向以充足的特异性来被确定,以促进准确地确定VR头戴式显示器108和VR控制器112在VR操作环境114中的定位。头戴式显示器和控制器换能器402可以包括IR发射器,所述IR发射器生成定时信号(例如参考脉冲和扫掠平面),以用于由VR头戴式显示器108和VR控制器112上的传感器接收。可以基于VR头戴式显示器108和控制器112的哪些传感器检测到由换能器402生成的信号以及脉冲和扫掠检测的相对定时来确定VR头戴式显示器108和控制器112的位置和定向。
[0028]图5示出了根据各种示例的用于VR系统中的对象检测的图像处理设备106的框图。图像处理设备106包括收发器502、处理器504和存储装置506。图像处理设备106还可以包括已经为了清楚而从图5被省略的各种组件和系统。例如,图像处理设备106可以包括显示系统、用户接口等等。图像处理设备106可以被实现在计算机中,如本领域中所已知的。例如,可以通过使用台式计算机、笔记本计算机、机架装配的计算机、平板计算机、或者适合用于执行本文中所述的图像处理操作以用于VR操作环境114中的对象检测的任何其它计算设备来实现图像处理设备106。
[0029]收发器502将图像处理设备106通信地耦合到监控站102和VR头戴式显示器108。例如,收发器502可以包括网络适配器,所述网络适配器将图像处理设备106连接到无线或有线网络,所述无线或有线网络提供在监控站102与图像处理设备106之间的通信。这样的网络可以基于各种联网标准(例如IEEE 802)中的任何或专属于在VR系统100的设备之间的通信。
[0030]处理器504被耦合到收发器502。处理器504可以包括通用微处理器、数字信号处理器、微控制器、图形处理器、或能够执行从计算机可读存储介质所检索的指令的其它设备。处理器架构一般包括执行单元(例如定点、浮点、整数等等)、存储装置(例如寄存器、存储器等等)、指令解码、指令和数据取出逻辑、外围设备(例如中断控制器、定时器、直接存储器访问控制器等等)、输入/输出系统(例如串行端口、并行端口等等)以及各种其它组件和子系统。
[0031]存储装置506是计算机可读介质,其存储指令和数据以供处理器504访问和使用。存储装置506可以包括任何的易失性存储装置、诸如随机访问存储器、非易失性存储装置(例如硬驱动器、光学存储设备(例如CD或DVD)、闪速存储装置、只读存储器)或其组合。存储装置506包括对象检测508、对象标识512、对象定位514、对象优先化516和图像518。图像518包括VR操作环境114的图像,其通过监控站102被捕获并且经由收发器502被传递到图像处理设备106。对象检测508、对象标识512、对象定位514和对象优先化516包括由处理器504执行以处理图像518的指令。
[0032]对象检测508包括可由处理器504执行以检测VR操作环境114中的对象的指令。对象检测508可以包括运动检测510。运动检测510包括可由处理器504执行以检测VR操作环境114中的运动的指令。运动检测510可以通过标识图像518中的图像至图像改变、通过标识参考图像与图像518中给定一个图像之间的差异、或通过其它运动检测处理来检测运动。在一些实现方式中,对象检测508可以忽视在区域116内所检测到的、与VR头戴式显示器108的用户相对应的运动。对象检测508可以通过使用边缘检测来标识与所检测到的对象相关联的点。所检测到的对象的存在可以通过图像处理设备106被传送到VR头戴式显示器108。
[0033]对象定位514包括如下指令:所述指令可由处理器504执行以确定通过对象检测508所检测到的对象的三维位置。例如,如果对象检测508在由不同的监控站102所捕获的图像518中标识对象,那么对象定位514可以通过基于监控站102的已知位置以及在不同监控站102的图像传感器304的光学中心线之间的已知角度进行三角测量来确定对象在三维VR操作环境114中的位置。对象的位置可以通过图像处理设备106被传送到VR头戴式显示器108。
[0034]对象标识512包括如下指令:所述指令可由处理器504执行以确定通过对象检测508所检测到的对象的标识。例如,对象标识512可以包括Viola-Jones框架,所述Viola-Jones框架被训练以标识可存在于VR操作环境114中的各种对象、诸如人和宠物。与所检测到的对象相对应的像素可以通过Viola-Jones框架被处理以标识对象。对象的标识可以通过图像处理设备106被传送到VR头戴式显示器108。
[0035]对象优先化516包括如下指令:所述指令可由处理器504执行以优先化通过图像处理设备106被提供到VR头戴式显示器108的对象有关的信息。优先化可以基于对象大小、在对象与VR头戴式显示器108之间的距离、对象海拔、和/或对在所检测到的对象与VR头戴式显示器108的用户之间的碰撞进行指示的其它因素。在一些实现方式中,对象优先化516可以根据在对象与VR头戴式显示器108的用户之间的碰撞的风险来优先化多个所检测到的对象,并且以所确定的最高碰撞风险的次序将关于对象的信息传送到VR头戴式显示器108。
[0036]存储装置506可以包括已经为了清楚而从图5被省略的附加逻辑。例如,存储装置506可以包括VR头戴式显示器通信,所述VR头戴式显示器通信包括如下指令:所述指令可由处理器504执行以将关于所检测到的对象的信息(例如对象存在、位置和/或标识)传递到VR头戴式显示器108。在一些实现方式中,图像处理设备106可以产生由VR头戴式显示器108所显示的视频帧。在这样的实现方式中,图像处理设备106可以包括视频渲染指令,所述视频渲染指令可由处理器504执行以生成由VR头戴式显示器108所显示的视频帧,并且可以在视频帧中包括关于所检测到的对象的信息。
[0037]图6-8示出了根据各种示例的VR头戴式显示器108中的对象检测信息的显示。在实践中,图6-8的显示可以是立体的,但是在图6-8中被示出为单视场的,以促进清楚性。在图6中,VR头戴式显示器108将对象检测告警显示为文本字段602。除了对象的存在之外,文本字段602可以指定对象的位置和标识。
[0038]在图7中,VR头戴式显示器108将对象检测告警显示为所检测到的对象的轮廓702或剪影。对象的轮廓702可以对应于通过对象标识512所确定的对象的标识,并且对象在VR头戴式显示器108的显示中的位置可以对应于通过对象定位514所确定的对象的位置。
[0039]在图8中,VR头戴式显示器108显示告警符号802,所述告警符号802指示:对象已经被检测到并且在VR头戴式显示器108的视场的左边。一些实现方式可以组合图6-8的对象信息显示。例如,在VR头戴式显示器108的查看区域外部的对象可以按照图8被指示,并且在VR头戴式显示器108被转向对象时,对象的轮廓702或剪影可以按照图7来被显示。类似地,文本显示602可以与告警符号802或对象轮廓702相组合以向VR头戴式显示器108的用户提供附加信息。
[0040]图9示出了根据各种示例的用于VR系统中的对象检测的方法900的流程图。尽管为了方便起见被顺序地描绘,但是所示出的动作中的至少一些可以用不同的次序被执行和/或并行地被执行。另外,一些实现方式可以执行所示出的动作中的仅仅一些。在一些实现方式中,方法900的操作中的至少一些可以被实现为被存储在存储设备中并且由一个或多个处理器执行的指令。
[0041]在框902中,监控站102捕获VR操作环境114的图像。监控站102被布置在沿着VR操作环境114的周界的不同位置处。监控站102的位置以及监控站102的图像传感器104的光学中心线的相交的角度对于图像处理设备106而言可以是已知的。由监控站102所捕获的图像可以包括颜色信息(色度)和亮度信息(亮度),或可以仅仅包括亮度。监控站可以捕获图像,作为视频流。例如,监控站102可以以每秒10、20、30等等幅图像的速率来捕获图像。监控站102将所捕获的图像传递到图像处理设备106以用于使用在对象检测中。
[0042]在框904中,图像处理设备106处理由监控站102所捕获的图像以检测VR操作环境114中的对象。处理可以包括应用运动检测来标识VR操作环境114中的对象。例如,图像处理设备106可以通过比较两幅图像并且标识所述两幅图像之间的差异来检测运动。所述两幅图像可以是由相同的监控站102(例如监控站102-1)在不同时间所捕获的图像。图像之一可以是参考图像。此外,如关于图2所解释的,图像处理设备106可以在图像中标识VR头戴式显示器108和/或VR控制器112,并且忽视在图像中、在VR头戴式显示器108和/或VR控制器112周围的预定区域116中的如可归因于VR头戴式显示器108的用户的移动的差异(即忽视运动)。
[0043]在框906中,图像处理设备106已经在VR操作环境114中检测到对象,并且将关于所检测到的对象的信息传输到VR头戴式显示器108。所述信息可以包括VR操作环境114中对象存在的通知。所述信息可以被嵌入在如下视频流中:所述视频流由图像处理设备106提供以供VR头戴式显示器108显示,或与被传递到VR头戴式显示器108的任何视频信息分离地被提供。所述信息可以经由在图像处理设备106与VR头戴式显示器108之间所提供的有线或无线通信信道来被传递。所述通信信道可以包括根据各种联网标准的有线或无线网络。
[0044]在框908中,VR头戴式显示器108显示从图像处理设备106所接收的、关于所检测到的对象的信息。所述信息可以与由VR头戴式显示器108所显示的VR场景相结合地被显示。
[0045]图10示出了根据各种示例的用于VR系统中的对象检测的方法1000的流程图。尽管为了方便起见被顺序地描绘,但是所示出的动作中的至少一些可以用不同的次序被执行和/或并行地被执行。另外,一些实现方式可以执行所示出的动作中的仅仅一些。在一些实现方式中,方法1000的操作中的至少一些可以被实现为被存储在存储设备中并且由一个或多个处理器执行的指令。
[0046]在框1002中,监控站102捕获VR操作环境114的图像。监控站102被布置在VR操作环境114的周界上的不同位置处。监控站102的位置以及监控站102的图像传感器104的光学中心线的相交的角度对于图像处理设备106而言可以是已知的。由监控站102所捕获的图像可以包括颜色信息(色度)和亮度信息(亮度),或可以仅仅包括亮度。监控站可以捕获图像,作为视频流。例如,监控站102可以以每秒10、20、30等等幅图像的速率来捕获图像。监控站102将所捕获的图像传递到图像处理设备106以用于使用在对象检测中。
[0047]在框1004中,图像处理设备106处理由监控站102所捕获的图像以检测VR操作环境114中的对象。处理可以包括应用运动检测来标识VR操作环境114中的对象。例如,图像处理设备106可以通过比较两幅图像并且标识所述两幅图像之间的差异来检测运动。所述两幅图像可以是由相同的监控站102在不同时间所捕获的图像。图像之一可以是参考图像。图像处理设备106可以在图像中标识VR头戴式显示器108和/或VR控制器112,并且忽视在图像中、在VR头戴式显示器108和/或VR控制器112周围的预定区域116中的如可归因于VR头戴式显示器108的用户的移动的差异(即忽视运动)。
[0048]在框1006中,图像处理设备106处理由监控站102所捕获的图像以确定在VR操作环境114中的任何检测对象的位置。位置确定处理可以包括:在由不同的监控站102所捕获的时间重合的图像中标识所检测到的对象,以及将三角测量应用于所检测到的对象的点以在三维中定位对象。
[0049]在框1008中,图像处理设备106处理由监控站102所捕获的图像以确定在VR操作环境114中所检测到的任何对象的标识。标识确定处理可以包括:从由监控站102所捕获的图像中提取和/或隔离所检测到的对象的图像,以及将所检测到的对象的图像提供到Viola-Jones框架。Viola-Jones框架可以从图像中提取特征并且应用一个或多个分类器来标识对象。在标识确定处理的一些实现方式中可以使用各种其它对象标识算法。
[0050]在框1010中,图像处理设备106优先化所检测到的对象以用于呈现给VR头戴式显示器108。优先化可以基于对象大小、在对象与VR头戴式显示器108之间的距离、对象海拔、朝向用户的移动速率、和/或对在所检测到的对象与VR头戴式显示器108的用户之间的碰撞进行指示的其它因素。对象优先化处理的一些实现方式可以根据在对象与VR头戴式显示器108的用户之间的碰撞的风险来优先化多个所检测到的对象,并且以所确定的最高碰撞风险的次序将关于对象的信息传送到VR头戴式显示器108。
[0051]在框1012中,图像处理设备106已经在VR操作环境114中检测到对象,并且将关于所检测到的对象的信息传输到VR头戴式显示器108。所述信息可以包括VR操作环境114中存在对象的一个或多个通知、所检测到的对象在VR操作环境114中的位置、以及所检测到的对象的标识。所述信息可以被嵌入在如下视频流中:所述视频流由图像处理设备106提供以供VR头戴式显示器108显示,或与被传递到VR头戴式显示器108的任何视频信息分离地被提供。所述信息可以经由在图像处理设备106与VR头戴式显示器108之间所提供的有线或无线通信信道来被传递。所述通信信道可以包括根据各种联网标准的有线或无线网络。
[0052]在框1014中,VR头戴式显示器108显示从图像处理设备106所接收的、关于所检测到的对象的信息,以允许VR头戴式显示器108的用户变得知晓所检测到的对象,并且潜在地避免与所检测到的对象的碰撞。所述信息可以与由VR头戴式显示器108所显示的VR场景相结合地被显示。
[0053]以上讨论意图说明本发明的原理和各种示例。一旦以上公开内容被充分领会,众多变型和修改将变得对于本领域技术人员而言是显然的。所意图的是,所附权利要求被解释为包括所有这样的变型和修改。