Patent ID: 12236527

DESCRIPTION OF THE EMBODIMENTS

FIG.1illustrates a schematic diagram of a remote-control system10according to an embodiment of the present invention, wherein the remote-control system10may be implemented in an extended reality (XR) system (e.g., virtual reality (VR) system, augmented reality (AR) system, or mixed reality (MR) system). The remote-control system10may include a computing device100, a remote controller200, and one or more displays. AlthoughFIG.1only shows only two displays, display310and display320, the number of the displays may be more than two. In one embodiment, the remote-control system10may further include a head mounted display400. The computing device100may communicatively connect to the remote-controller200, the display310, the display320, or the head mounted display400. The remote-controller200may communicatively connect to the display310, the display320, or the head mounted display400.

The computing device100may be a computer, a server, a client terminal, a desktop computer, a laptop computer, a work station, a personal computer (PC), a tablet, or a smart phone, the disclosure is not limited thereto. The computing device100may include necessary components to operate the computing device100, such as a processing unit (e.g., a processor), a communication unit (e.g., a communication chip, a mobile communication chip, a Bluetooth chip, or a Wi-Fi chip), and a storage unit (e.g., a random-access memory (RAM), a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD)).

The remote-controller200may be used for controlling the display310, the display320, or the head mounted display400. The remote-controller200may include an image capture device210, an inertial measurement unit (IMU)220, an input device230, and a processor240, wherein the processor240may be coupled to the image capture device210, the IMU220, or the input device230. The image capture device210or the IMU220may be control by the processing unit coupled to the image capture device210or the IMU200. In addition, the remote-controller20may further include necessary components to operate the remote-controller200, such as a communication unit and a storage unit.

The image capture device210may be a camera or a photographic device for capturing images. The image capture device210may include an image sensor such as a complementary metal oxide semiconductor (COMS) sensor, or a charge coupled device (CCD) sensor. The IMU220may include an accelerometer, a gyroscope, or a magnetometer. The input device230may be a button, a keyboard, or a touch screen.

The display310(or display320) may be used for playing video data or image data. The display310may include necessary components to operate the display310, such as a processing unit, a communication unit, and a storage unit. The display310may be a liquid-crystal display (LCD) or an organic light-emitting diode (OLED) display, the disclosure is not limited thereto.

The head mounted display400may be wear by a user for exploring extended reality scenes. The head mounted display400may include necessary components to operate the head mounted display400, such as a processing unit, a communication unit, and a storage unit. In addition, the head mounted display400may further include an IMU410. The IMU410may be control by the processing unit coupled to the IMU410. The IMU410may include an accelerometer, a gyroscope, or a magnetometer.

In one embodiment, the computing device100may be embedded into the remote-controller200, the display310(or display320), or the head mounted display400. In one embodiment, the computing device100is embedded into the remote-controller200and electrically coupled to the processor240.

FIG.2illustrates a signaling diagram of the synchronization between the remote-controller200and the head mounted display400according to an embodiment of the present invention. Assuming the head mounted display400has obtained map for the extended reality scene, image data for the extended reality scene, and location information of the head mounted display400, wherein the location information is corresponded to the coordinate system of the map. In step S201, the head mounted display400may transmit the map for the extended reality scene, the image data for the extended reality scene, and the location information of the head mounted display400to the remote-controller200via a communication protocol such as Bluetooth protocol. In step S202, the processor240of the remote-controller200may align the map for locating the head mounted display400. Step S203and step S204are corresponded to a repeat processing. In step S203, the processor240of the remote-controller200may detect pose data of the head mounted display400is transmitted from the head mounted display400to the remote-controller200, wherein the pose data of the head mounted display400may be generated by the IMU410. The remote-controller200may perform synchronization with the head mounted display400according to the pose data of the head mounted display400. In step S204, the head mounted display400may detect pose data of the remote-controller200is transmitted from the remote-controller200to the head mounted display400, wherein the pose data of the remote-controller200may be generated by the IMU220. The head mounted display400may perform synchronization with the remote-controller200according to the pose data of the remote-controller200.

FIG.3illustrates a signaling diagram of the remote-control system10according to an embodiment of the present invention. In step S301, the computing device100and the remote-controller200may communicatively connect to each other via communication protocol such as Bluetooth protocol.

In step S302, the computing device100may transmit a command to the display310(and/or display320) to ask the display310(and/or display320) to show a positioning marker. In step S303, the display310(and/or display320) may display a positioning marker according to the command from the computing device100.FIG.4illustrates a schematic diagram of the positioning marker311according to an embodiment of the present invention. The display310(and/or display320) may display one or more positioning markers311as shown inFIG.4, wherein the positioning marker may include an Aruco marker. In one embodiment, the display310and the display320may display positioning markers respectively in the same time according to the command form the computing device100.

Referring toFIG.3, in step S304, the computing device100may notify the remote-controller200that the positioning marker311is being displayed by the display310(and/or display320). In step S305, the processor240of the remote-controller200may guide the user to use the remote-controller200to aim at the display310(and/or display320). In step S306, the processor240of the remote-controller200may detect the positioning marker311. Specifically, the image capture device210of the remote-controller200may obtain environmental image data. When the user aims the remote-controller200at the display310(and/or display320), the environmental image data obtained by the image capture device210may include positioning marker311displayed by the display310(and/or display320). The processor240of the remote-controller200may detect the positioning marker311in the environmental image data based on, for example, object detection technology.

In step S307, the processor240of the remote-controller200may build a map according to the environmental image data based on simultaneous localization and mapping (SLAM) algorithm, wherein the map may be corresponded to a Cartesian coordinate system of an extended reality environment. The processor240of the remote-controller200may further obtain, from the environmental image data, the location information of the display310(and/or display320) according to the positioning marker311, wherein the location information of the display310(and/or display320) may include a coordinate of the display310(and/or display320) corresponding to the Cartesian coordinate system.

In step S308, the computing device100may receive the location information of the display310(and/or display320) from the remote-controller200. In step S309, the computing device100may control the head mounted display400(not shown inFIG.3) or the display310(and/or display320) according to the location information of the display310(and/or display320). Specifically, the computing device100may configure the image data outputted by the head mounted display400or the display310(and/or display320) according to the location information of the display310(and/or display320).

In one embodiment, the computing device100may control the head mounted display400to output an image data corresponding to the display310(and/or display320) according to the location information of the display310(and/or display320) and/or the location information of the head mounted display400. The processor240of the remote-controller200may obtain the location information of the head mounted display400from the head mounted display400and may forward the location information of the head mounted display400to the computing device100. The computing device100may control the head mounted display400to output image data corresponding to the display310(and/or display320) according to the location information of the display310(and/or display320) and/or the location information of the head mounted display400, wherein the location information of the head mounted display400may include the pose data generated by the IMU410.

For example, the computing device100may configure, according to the location information of the head mounted display400and/or the location information of the display310(and/or display320), the head mounted display400to output the image data corresponding to the display310(and/or display320), wherein the image data may represent the location or the shape of the display310(and/or display320) such that the user of the head mounted display400may easily find the display310(and/or display320) in the extended reality scene. Furthermore, since the content displayed by the display310(and/or display320) may be configured by the computing device100, the computing device100may configure the head mounted display400to output the image data including the same content which is being displayed by the display310(and/or display320). Accordingly, even if the head mounted display400is outputting a virtual reality scene which does not show the real display310(and/or display320) to the user, the user exploring the virtual reality environment may still watch the content shown by the display310(and/or display320) in the virtual reality scene. In other words, the head mounted display400may show the user a virtual display310(and/or display320) in the virtual reality scene, wherein the virtual display310(and/or display320) can display the content as same as the display310(and/or display320) in the real world.

In step S310, the processor240of the remote-controller200may determine whether the remote-controller200is being pointed to the display310(or display320). Specifically, the IMU220may measure pose data of the remote-controller200. The processor240of the remote-controller200may determine whether the remote-controller200is being pointed to the display310(or display320) according to the pose data of the remote-controller200and the location information of the display310(or display320). If the remote-controller200is being pointed to the display310(or display320), the processor240of the remote-controller200may determine a position of a cursor on the display310(or display320) according to the pose data of the remote-controller200and the location information of the display310(or display320). Accordingly, the user of the head mounted display400may interact with multiple displays in the same time through the remote-controller200.

In step S311, the processor240of the remote-controller200may transmit the position of the cursor to the computing device100such that the processor240of the remote-controller200may control the display310(or display320) to show the cursor via the computing device100. In step S312, the computing device100may send a command to the display310(or display320) according to the position of the cursor to control the display310(or display320) to show the cursor. In step S313, the display310(or display320) may display the cursor according to the command from the computing device100.FIG.5illustrates a schematic diagram of the cursor50shown on the display310according to an embodiment of the present invention. When the remote-controller200aims to the display310, the corresponding cursor50may be shown on the display310. When the remote-controller200aims to the display320, the corresponding cursor50may be shown on the display320. That is, the cursor50corresponding to the remote-controller200may move between the display310and the display320.

Referring toFIG.3, in step S314, the processor240of the remote-controller200may determine whether the input device230is being operated by the user. If the input device230is being operated by the user, the processor240of the remote-controller200may send a command to the computing device100. In step S315, the computing device100may process a task according to the command. In one embodiment, the command is corresponded to the cursor50. The computing device100may process a task according to the cursor50in response to receiving the command from the remote-controller200. The computing device100may control the display310according to the command corresponding to the cursor50. For example, the computing device100may control the display310to show a special effect at the position of the cursor50according to the command corresponding to the cursor50.

In one embodiment, the access right of the display310(and/or display320) may be managed by the computing device100. The computing device100may determine whether to allow the remote-controller200to control the display310(and/or display320) via the computing device100. If the computing device100allow the remote-controller200to control the display310(and/or display320), the processor240of the remote-controller200may control the display310(and/or display320) via a communication link between the computing device100and the remote-controller200.

FIG.6illustrates a flowchart of a remote-control method according to an embodiment of the present invention, wherein the remote-control method may be implemented by the remote-control system10as shown inFIG.1. In step S601, obtaining environmental image data by an image capture device of a remote-controller. In step S602, building a map according to the environmental image data based on simultaneous localization and mapping (SLAM) algorithm, and obtaining first location information of a first display in the map according to the environmental image data by the remote-controller. In step S603, receiving the first location information from the remote-controller and controlling the first display according to the first location information by a computing device.

In summary, the remote-control system of the present invention may control the display and the head mounted display by the same remote-controller. The remote-controller may obtain image data by the image capture device and build a coordinate system according to the image data and the SLAM algorithm, wherein the coordinate of the display may be known by the remote-control system. Accordingly, the remote-control system may show the position of the display to the user wearing head mounted display, and the user may interact with the display and the XR scene by using the same remote-controller.

It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the disclosure covers modifications and variations provided that they fall within the scope of the following claims and their equivalents.