Controlling data processing

A control apparatus includes two or more user control devices, each user control device having an inertial detector having a wireless communications module to communicate inertial detection results, and an illuminated portion; and a housing to house the two or more control devices so that the two or more control devices are constrained to move with one another but having an obscured portion so as to obscure all but one of the illuminated portions of the two or more control devices.

BACKGROUND

Field of the Disclosure

This disclosure relates to controlling data processing.

Description of the Prior Art

Some data processing activities may be controlled by a detection of a trackable device, for example.

An example arrangement involves a games machine, in which movements of a device such as a head mountable display (HMD) and/or a hand-holdable controller such as a Sony® Move® Controller, can be tracked by one or more cameras.

In a so-called mixed reality system (for example, being capable of combining features of real and virtual environments in a display presented to the user), it may be that multiple cameras are in use, for example a camera to obtain images of the user, for example in order to track the user's position and/or activity, and one or more cameras to capture images of the user's real environment. There is a need to be able to spatially align the frames of reference of the captured images obtained from the various cameras.

Various aspects and features of the present disclosure are defined in the appended claims and within the text of the accompanying description and include at least a head mountable apparatus such as a display and a method of operating a head-mountable apparatus as well as a computer program.

DESCRIPTION OF THE EMBODIMENTS

Referring to the drawings,FIG. 1schematically illustrates a user wearing a head-mountable display (HMD) connected to a Sony® PlayStation® games console300as an example of a base device. The games console300is connected to a mains power supply310and (optionally) to a main display screen (not shown). A cable82, acting as power supply and signal cables, links the HMD20to the games console300and is, for example, plugged into a USB socket320on the console300.

InFIG. 1, the user is also shown holding a hand-held controller330which may be, for example, a Sony® Move® controller which communicates wirelessly with the games console300to control (or to contribute to the control of) operations relating to a currently executed program at the games console.

A camera305is associated with the console300to capture images of the user10and/or the controller330.

The Move controller comprises a handle portion100and an illuminated end portion110. The handle portion100can carry one or more control buttons and houses a so-called inertial measurement unit (IMU) which will be described in more detail below. The illuminated end portion110comprises one or more light emitting diodes (LEDs) inside a translucent spherical shell and which are capable of being illuminated, for example under the control of an apparatus such as the games console300.

The Move controller330provides an example of a control device comprising an elongate handle portion100which houses the inertial detector332,334and an illuminated end portion110at an end of the handle portion.

In use, the IMU transmits inertial detections to the games console300and the games console300also tracks, using images captured by the camera305, the location of the illuminated end portion110.

A pair of video displays in the HMD20are arranged to display images provided via the games console300, and a pair of earpieces60in the HMD20are arranged to reproduce audio signals generated by the games console300. The games console may be in communication with a video server. The USB connection from the games console300also provides power to the HMD20, according to the USB standard.

FIG. 2schematically illustrates a similar arrangement in which the games console is connected (by a wired or wireless link) to a so-called “break out box” acting as a base or intermediate device350, to which the HMD20is connected by a cabled link82. The breakout box has various functions in this regard. One function is to provide a location, near to the user, for some user controls relating to the operation of the HMD, such as (for example) one or more of a power control, a brightness control, an input source selector, a volume control and the like. Another function is to provide a local power supply for the HMD (if one is needed according to the embodiment being discussed). Another function is to provide a local cable anchoring point. In this last function, it is not envisaged that the break-out box350is fixed to the ground or to a piece of furniture, but rather than having a very long trailing cable from the games console300, the break-out box provides a locally weighted point so that the cable82linking the HMD20to the break-out box will tend to move around the position of the break-out box. This can improve user safety and comfort by avoiding the use of very long trailing cables.

FIG. 3schematically illustrates aspects of the operation of the Move controller330in more detail.

As mentioned above, the Move controller330houses an Inertial Measurement Unit (IMU)332. Inertial measurements are considered, for the purposes of this description, to encompass one or more of: accelerometer detections; and gyroscopic detection. The games console300is connected to the camera305which, for the purposes of this explanation, is assumed to be directed towards the Move controller330.

In operation, the camera305captures images including images of the illuminated end portion110of the Move controller330. These are processed by an image processor312to identify a location, with respects to the frame of reference of the captured images, of the illuminated end portion110.

An IMU data receiver314receives wireless telemetry316from the IMU332, for example via a Bluetooth® wireless link provided by a wireless interface334forming part of the IMU. The telemetry signals316indicate inertial measurements performed by the IMU332, for example accelerometer and gyroscopic measurements indicating an orientation of the Move controller330or at least changes in its orientation.

Data provided by the image processor312indicative of the location of the illuminated end portion110in the captured images and data provided by the IMU data receiver314are processed by a location and orientation detector318which provides an output signal332indicative of any current location and orientation of the Move controller320, or at least changes in the location and orientation. Note that these changes can be used in the arrangement ofFIG. 3to control game play, for example. In the camera frame calibration process discussed below, changes in location are relevant, but not necessarily changes in orientation.

This information provided by the signal322can be used in various different ways as a generic control input to a data processing operation. Purely by way of example, in the arrangement shown schematically inFIG. 3, the signal322is provided to a control function generator324which generates control information326to control aspects of the operation of a game engine328executing a computer game. As part of its operation, the game engine may generate display signals to drive one or more displays332such as the one or more displays in the HMD20and/or a further display such as so-called social screen for viewing by users who are not currently wearing the HMD20.

FIG. 4schematically illustrates an issue which can arise in, for example, a so-called mixed reality arrangement such as a video game arrangement in which the games console300is configured to generate a mixed reality video signal for display to a user10(for example via the HMD20) in dependence upon images captured by plural video cameras and images (which may be generated by the games console300) indicative of a virtual environment.

In executing such a mixed reality operation, more than one video camera is generally required because it is necessary to capture the real environment around the user from multiple different angles in order to generate the mixed reality environment for display to the user. InFIG. 4, two video cameras400,410are shown, each associated with a respective games console405,415which includes the features shown as part of the games console300ofFIG. 3and described above.

A Move controller420is provided. While it is possible for both cameras to track the illuminated end portion422of the Move controller420, system restriction prevent the IMU424of the Move controller420(via its wireless link426) communicating with both the games console405and the games console415.

Having both of the games consoles405,415track the Move controller420can be particularly useful not only during potential gameplay situations in a mixed reality environment for example, but also during a calibration fade in which the relative locations of the cameras400,410are determined by using both cameras400,410to track a common object (the Move controller420).

However, for the reasons discussed above, this is not possible using a single Move controller.

One possibility to address this would be to register a second Move controller (with that one of the games consoles405,415not tracking the Move controller420) and keep it out of sight so that it is not tracked by the respective video camera. However, then the motion capture results between the two games consoles405,415would not correlate. This is because the tracking system used to detect the location of the Move controller420is driven, and discussed above, using a combination of IMU data and captured images of the illuminated end portion422.

So, a solution is not necessarily provided by registering a second Move controller but keeping it out of sight.

FIG. 5andFIG. 6schematically illustrate an arrangement which can alleviate this problem.

FIG. 5schematically illustrates a housing500to house a pair of Move controllers andFIG. 6schematically illustrates the housing500populated with a first600and second610Move controller. It can be seen that housing500has the form (in this example) of a tube such that the Move controllers600,610may be fitted inside the tube. The tube is made (at least in part) of a non-transparent, or translucent, material such that the illuminated end portion605of the Move controller600is obscured from view by either of the cameras400,410. However, the illuminated end portion615of the Move controller610protrudes from the housing500(or alternatively is contained within a transparent end portion505) so that it may be seen by the video cameras400,410.

FIG. 6therefore provides an example of a control apparatus comprising:

two or more user control devices600,610, each user control device having an inertial detector424having a wireless communications module426to communicate inertial detection results, and an illuminated portion605,615; and

a housing500to house the two or more control devices so that the two or more control devices are constrained to move with one another but having an obscured portion so as to obscure all (605) but one (615) of the illuminated portions of the two or more control devices600,610.

Note that it is not necessarily a requirement that the whole of the housing500is obscured; a portion505surrounding the illuminated portion605could be obscured, and the remainder not obscured (for example, being transparent or even an open frame structure), for example.

The tubular housing500ofFIG. 5provides an example of a generally tubular housing inside which the two or more control devices are mountable adjacently in a lengthwise direction. In some examples, the housing is formed of a non-transparent or translucent material such as a plastics material.

FIG. 7schematically illustrates the operation of such an arrangement, in which the end portion615is tracked by the video camera400and the video camera410.

In terms of the IMU data, one of the Move controllers such as the Move controller610provides IMU data to the games console405and the other Move controller600provides IMU data to the games console415.

Using this arrangement, at least in a calibration phase during which the relative location of plural video cameras is determined, the housing500carrying the multiple Move controllers600,610but allowing the illuminated end portion615to be visible can be tracked by both of the cameras and games consoles ofFIG. 7. This arrangement can be used to capture multiple points in the space of each camera which can be correlated to one another by the respective location detectors318detecting their location. This data can be used to create a mapping between locations as viewed by each of the cameras so that when images captured by the multiple cameras are combined together in a mixed reality display, for example, features at corresponding locations within the view of the multiple cameras are correctly associated with one another.

In principle a housing similar to the housing500could be provided which encompasses more than two Move controllers, still with just one illuminated end portion protruding or otherwise visible, so that more than two cameras may be mapped together in this way. However, another option where more than two cameras are in use is to generate mappings between them on a pair-wise basis, in other words two at a time.

FIG. 7therefore provides an example of a data processing system comprising:

a control apparatus700;

two or more data processing devices405,415, a respective data processing device being associated with each of the two or more user control devices600,610so as to receive inertial detection results from that user control device;

two or more video cameras400,410, a respective video camera being associated with each data processing device and each being configured to capture images of the non-obscured illuminated portion;

in which each data processing device is configured to detect a location of the control apparatus from images captured by the respective camera of the non-obscured illuminated portion and from inertial detection results received from the respective control device.

By comparing the location data at particular physical positions of the control apparatus, the system ofFIG. 7(for example, one of the devices405,415or another device (not shown)) can be configured to detect a mapping between respective frames of reference of the two or more video cameras by correlating the location of the control apparatus as detected by each data processing device.

FIGS. 8 and 9schematically illustrate further possible refinements of the arrangements discussed above.

Referring toFIG. 8, the housing800and the location810,820of the two IMUs are illustrated in a vertical orientation. If the housing is rotated to a non-vertical orientation shown by the representation830, the movement812,822experienced by the two IMUs is different. Since the two IMUs are being tracked by different respective games consoles405,415in the arrangement ofFIG. 7, this could lead to discrepancies in the detected results and potential errors in the cross-mapping of reference of the multiple cameras400,410.

To address this,FIG. 9schematically illustrates the housing900mounted with respects to a stabilising apparatus910such as a camera stabiliser used to mount a video camera for hand-held or shoulder-mounted operation, and which includes weighting, gyroscopes and the like tends to maintain the housing900in an upright orientation in use. Since this arrangement may be in operation only during a calibration phase, it can be acceptable to use such a stabilisation method even though to do so during gameplay may be onerous for the player.

FIG. 9therefore provides an example of the use of stabiliser apparatus associated with the housing to inhibit rotation of the long axis of the housing.

FIG. 10schematically illustrates a data processing apparatus or device such as the games console300. The apparatus1500comprises one or more processing elements such as a central processing unit1510(CPU), a non-volatile memory1520such as a read only memory (ROM), a flash memory, a magnetic or optical disk or the like, a random access memory (RAM)1530, a user interface1540, a network interface1550to provide connectivity with different apparatus in the system, and (at least for the control apparatuses) a camera interface1560. All of these elements are interconnected by a bus structure1570. In operation, program code is stored in the non-volatile memory1520and transferred to the random access memory1530for execution i.e. CPU1510to perform the functionality discussed above.

Note that although different apparatuses have been provided in the examples given above for the data processing apparatus1020and the earlier control apparatuses, these functions could be shared by a common apparatus executing at various different tasks, for example.

FIG. 11is a schematic flowchart illustrating a method comprising:

moving (at a step1100) a control apparatus according to claim1to a plurality of locations within view of two or more video cameras;

deriving (at a step1110) two or more sets of location data by capturing images of the control apparatus using the two or more video cameras, and in respect of each video camera, detecting inertial detection results received from a respective one of the control devices; and

detecting (at a step1120) a mapping between respective frames of reference of the two or more video cameras by correlating the two or more sets of location data.

It will be appreciated that in example embodiments the techniques discussed above, including the method ofFIG. 11, can be implemented by computer software operating on a general purpose computing system such as one or more games machines which may have the form of the apparatus ofFIG. 10. In these examples, computer software, which when executed by a computer, causes the computer to carry out any of the methods discussed above is considered as an embodiment of the present disclosure. Similarly, embodiments of the disclosure are provided by a non-transitory, machine-readable storage medium (such as the medium1520) which stores such computer software.

It will also be apparent that numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the disclosure may be practised otherwise than as specifically described herein.