Patent Description:
Controlling the aiming direction of a camera has been accomplished in a variety of prior systems. It is useful when the camera is immersed in an environment wider than what the camera can capture. Such systems are used for TV shows recording or for security camera remote control for instance.

<CIT> discloses a method and apparatus for widening a viewing angle in a video conferencing system are provided. The apparatus for widening a viewing angle in a video conferencing system includes: generating reference data from images of a video conference participant captured by a camera included in the video conferencing system; generating movement data based on the video conference participant's movements sensed by the camera, extracting a first control parameter by comparing the reference data with the movement data; transmitting the first control parameter to the other end of the conference; receiving a second control parameter generated at the other end of the conference; and controlling the camera by the second control parameter.

Virtual cameras in games or in immersive content renderers can also be considered as equipped with an aiming direction control system.

When controlling the aiming direction of a camera, the user can watch a 4π steradians environment through camera rotations. If such a feature may appear as a real improvement in terms of immersion in the content, as the user is watching at only a part of the environment, he/she may not look to the direction he/she should look to at a given moment. Indeed, as the user can gaze all around as he/she was in place of the camera, he/she may miss some important events, like highlights of the narration, because he/she is watching another part of the content at the moment the event happens.

According to the background, it is known that forcing a camera panning in order to make the user look toward a reference direction is a very efficient solution. However, it is well known that this solution has drawbacks. For instance, it will make most of people lose their visual cues or make them sick and, as a consequence, it will deteriorate the user's quality of experience.

The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description, the description making reference to the annexed drawings wherein:.

The subject matter is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter. It is understood that subject matter embodiments can be practiced without these specific details.

For the sake of clarity, <FIG> illustrate examples in two dimensions and refer to the only "yaw" angle (i.e. a rotation around the Z-axis). It is understood that the present principles are extendable to a third dimension and to the "pitch" angle (i.e. a rotation around the Y-axis) and to the "roll" angle (i.e. a rotation around the X-axis).

A camera (real or virtual) is located in a place (real or virtual) and surrounded with objects that the camera may film. This constitutes the environment of the camera with which a frame of reference is associated in order to locate objects.

A camera (real or virtual) is associated to a set of parameters relative to the environment the camera is located in. The location of the camera is expressed in the frame of reference associated with the environment. A camera is filming in a given direction that is called the aiming direction of the camera herein after.

<FIG> illustrates a controlled camera system <NUM> at a determined time t (e.g. the initialization time of the system). A camera <NUM> associated with a camera aiming direction controller <NUM>. On <FIG>, the camera controller <NUM> is represented as a head-mounted display (HMD). Indeed, a HMD may be considered as a camera controller. In an embodiment, a HMD is equipped with a head pose estimation system, for instance an integrated inertial measurement unit (IMU) that may include accelerometers and/or gyroscopes or an external positional tracking system using infra-red cameras for example. When the user is moving, the detected position of his/her head is used to orientate the aiming direction of a camera. The camera <NUM> is a motorized device placed in a remote location. In a variant the camera <NUM> is a virtual camera placed in a three-dimensional virtual world.

The camera <NUM> and the camera controller <NUM> share a zero direction <NUM> that is set at the starting of the system. For the sake of clarity, on <FIG>, the camera <NUM> and the camera controller <NUM> are drawn at the same place. In any embodiment, they belong to separated environment. For example, in a security system, cameras are located outside while joysticks for controlling their aiming direction are located in a control station. In the case of a HMD, the IMU belongs to the real world while the camera belongs to a virtual world. The zero direction may be reset to a common value from time to time as it is well known that a drift may progressively appear between the zero direction used on the camera side and its equivalent on the camera controller side. In a first person view application, there is a one to one correspondence between the HMD rotation angle and the camera aiming direction. When the HMD is rotating through a given angle, the camera rotates through the same given angle. In another embodiment, the camera controller is a joystick or a set of keys of a keyboard or a smartphone. In some embodiments, the display on which is rendered what the camera is filming is a TV-set or a computer screen or a mobile device screen like a smartphone or a tablet.

A set of at least one reference direction <NUM> is obtained. A reference direction correspond to a direction in which the user should look to, for instance, if a highlight of the narration is happening in this direction. Reference directions are provided as metadata of the content and received within the content stream or read from the same file. In a variant, the set of reference directions is obtained from a different source than the content. Both data has to be synchronized. In another variant, the reference directions are obtained thanks to the processing of the images of the content, for example using saliency map to detect regions of interest, a reference direction being associated with each region of interest for example. As reference directions depend on the content, their number and the reference directions themselves may vary over time.

According to an embodiment, the present principles aim at inciting the user to look toward such a reference direction or, as a complementary effect, at preventing the user to look too far away from such a reference direction. In a variant in which the user is exploring a three dimensions (3D) modelled scene as in a video game, the scene may not be modelled in every direction. Indeed, for cost or time reasons, as for cinema stages, only a part of the <NUM>° space may be modelled. In such a case, the producer may want to prevent the user to look to the non-modelled part of the 3D scene or to the technical zone of the cinema stage. In this variant, a reference direction corresponds to a direction distant of the non-modelled direction or the technical zone of the cinema stage. Several reference directions may be obtained at the same time. For example, if the narration includes a dialog between two actors, both of them constitute a highlight of the narration. A reference direction may change over the time. On <FIG>, the reference direction <NUM> may follow the plane while moving. In another example in which the camera is filming a tennis game, two reference directions may follow the players and one reference direction may follow the ball.

<FIG> illustrates the controlled camera system <NUM> of <FIG> at a time t after the initialization of the system. On <FIG>, the camera controller <NUM> is represented as a head mounted display (HMD). The user has rotated his/her head toward a direction <NUM> (to the right) in the reference frame formed by the center of the camera <NUM> and the zero direction <NUM>. In the example of the <FIG>, a reference direction <NUM> has been obtained (at the left of the zero direction). The camera <NUM> has been rotated toward a direction <NUM>. On <FIG>, the camera <NUM> has been rotated through an angle smaller than the angle formed between the zero direction <NUM> and the camera controller direction <NUM>. Indeed, as explained by the <FIG>, <FIG>, as the angle between the reference direction <NUM> and the direction obtained through the parameters of the camera controller <NUM> has increased, the camera aiming direction is computed to create a pseudo-haptic effect. In the example of <FIG>, the user has rotated his/her head toward the right up to the direction <NUM> but he/she is seeing what is filmed in the direction <NUM>, less at the right-hand in the environment of the camera. As a consequence of this discrepancy between his/her real movement and the visual feedback from the camera, the user feels a pseudo-haptic resistance. In another embodiment, the camera controller <NUM> is a mouse device or a joystick and the display is a TV-set or a computer screen. The same pseudo-haptic effect is created thanks to the discrepancy between the camera aiming direction and the direction the user expects to look to according to his/her commands on the camera controller.

<FIG> illustrates an example diagram of the discrepancy between the camera aiming direction and the direction associated with the parameters of the camera controller. The angle value φ on <FIG> corresponds to the angle between the zero direction <NUM> and a reference direction <NUM> of <FIG> and <FIG>. The curve <NUM> represents the angle θcam between the zero direction <NUM> and the camera aiming direction <NUM> of <FIG> according to the angle θcontrol between the zero direction <NUM> and the camera controller direction <NUM> of <FIG>. Both domains are circular: values go from φ-π radian to φ+π radian (and φ-π is the same angle than φ-π). The line <NUM> corresponds to a first person view application: the value of θcam always equals the value of θcontrol. The curve <NUM> shows an embodiment of the discrepancy function: the more the user is driving its camera controller away from the reference direction, the less the camera is rotating, up to no longer rotate at all. After this point, the curve <NUM> is flat. Advantageously, as illustrated on the <FIG>, the discrepancy function is a sigmoid-like function: its slope equals to <NUM> at φ and approaches to a limit. In a variant, the discrepancy function is piecewise linear: its slope is equals to <NUM> around φ and equals to <NUM> beyond a given threshold. Noticeably, the use of such a discrepancy function breaks the circular property of angle domains. Indeed, when the user commands to the camera controller to rotate through π radians (<NUM>°), the camera rotates through less tha n π radians and, so, does not face the opposite direction. A consequence of this fact is that the angle φ-π is now different from the angle φ+π and the domain of the discrepancy function is extended over these limits. This is illustrated on <FIG> and <FIG> by the fact that the curve of the discrepancy functions are drawn over the dashed square.

<FIG> illustrates settings that the method may use to compute a discrepancy function when the set of reference directions is changing. On <FIG>, the discrepancy function is a sigmoid-like function. Its slope is constrained to <NUM> at the value φ. The function is computed according to at least two setting values <NUM> and <NUM> which are the values the function reaches when the value of θcontrol respectively equals φ+π and φ-π radians. Additional settings may rule the derivative of the function, i.e. the slopes S1 and S2 that the function respectively has at points <NUM> and <NUM> (<NUM> ≤ S1 ≤ <NUM>; <NUM> ≤ S2 ≤ <NUM>) and the speed at which the slope of the function decreases (from <NUM> at the value φ, to S1 φ+π or to S2 at φ-π). In a variant, the discrepancy function is piecewise linear. Settings <NUM> and <NUM> are useful for this kind of discrepancy function too. A list of values between φ-π and φ+π may be used in addition to indicate thresholds for which the slope of the function changes.

Settings data are set to configure the haptic effect. For example, on <FIG>, the closer values <NUM> and <NUM> are to φ, the more limited the visible part of the scene is. The same way, settings ruling the local slopes of the discrepancy function adjust the pseudo-haptic resistance the user feels through the camera controller when he/she tries to rotate the camera.

A discrepancy function is determined (i.e. computed or calculated for instance) when a change in the reference directions set is detected. It may happen that the user does not look to a direction that belongs to the computed discrepancy function at the moment this function is computed (in particular at the starting of the present method). <FIG> illustrates an iterative computation of discrepancy functions. On the example of <FIG>, at the initialization time, the set of reference directions is empty. The user uses the camera controller to make the camera aim at the direction <NUM> (θcam = θcontrol = θ). A reference direction φ is obtained and a discrepancy function <NUM> is computed according to the settings. The point <NUM> does not belong to the curve of the discrepancy function <NUM>. In order to avoid a sudden shift of the camera's aiming direction, a first discrepancy function <NUM>, which passes through the point <NUM>, is computed. The function <NUM> is computed under the constraint not to increase the distance with the discrepancy function <NUM> and to decrease it when tending toward φ. When the user uses the camera controller to rotate the camera toward the direction φ, the camera rotation is facilitated. On the contrary, the rotation is made hard in the opposite direction. When a change of the camera controller is detected, the aiming direction of the camera follows the curve of the function <NUM> and a second discrepancy function is computed. On the example of <FIG>, the camera controller reaches the point <NUM>, closer to the reference direction and a second discrepancy function <NUM> is computed under the same constraints than the function <NUM>. The second discrepancy function is renamed first discrepancy function and the operation is iterated. Because of the computing constraints, the second discrepancy function is getting closer and closer to the discrepancy function <NUM> and, as a consequence, the wanted pseudo-haptic effect is reached without sudden shift in the aiming direction of the camera.

<FIG> illustrates a discrepancy function computed according to a pair of reference directions. Two reference directions have been obtained. For these reference directions, the camera has to aim at the said reference direction. This is illustrated by the points <NUM> and <NUM> on the <FIG>. According to the example of <FIG>, a discrepancy function <NUM> is calculated under the following constraints:.

As described herein above, the domain of θcontrol is meant to be circular. When the method manages a unique reference direction, it is possible to break this circular property of the domain without disobeying the constraint of continuity, centering the discrepancy function on the reference direction angle value. When there are at least two reference directions, the circular property may be broken only once between two reference direction angle values. On <FIG>, the choice has been made to keep the continuity in the interval from point <NUM> to point <NUM> and to break it in the interval from point <NUM> to point <NUM>. As a consequence, a pseudo-haptic "magnet effect" is observed in the angular interval from point <NUM> to point <NUM> and a pseudo-haptic "resistance effect" is observed over. In a variant, the choice is made to keep the continuity in the interval from point <NUM> to point <NUM> and to break it in the interval from point <NUM> to point <NUM>. In another variant, the choice is made to keep the circular property of the domain of θcontrol, introducing a double pseudo-haptic magnet effect.

<FIG> shows a hardware embodiment of an apparatus <NUM> configured to process an aiming direction of a camera. In this example, the device <NUM> comprises the following elements, connected to each other by a bus <NUM> of addresses and data that also transports a clock signal:.

The device <NUM> is connected to a camera controller <NUM>. In an embodiment, the camera controller is a joystick, a keyboard or a remote control. In another embodiment, the camera controller is an inertial measurement unit comprising accelerometers and/or gyroscopes for example.

The device <NUM> is connected to a camera <NUM> that is equipped to change its aiming direction, i.e. a real camera is motorized and a virtual camera is associated with a program or a script configured to control the camera aiming direction.

Advantageously, the device <NUM> is connected to one or more display devices <NUM> of display screen type directly to the graphics card <NUM> to display images calculated in the graphics card. In a variant, the one or more display device <NUM> is connected to the graphic card <NUM> via the bus <NUM>. In a particular embodiment, the camera controller <NUM> and/or the one or more display device <NUM> are integrated to the device <NUM> such as for Head Mounted Devices.

It is noted that the word "register" used in the description of memories <NUM> and <NUM> designates in each of the memories mentioned, both a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed).

When switched-on, the microprocessor <NUM>, according to the program in the register <NUM> of the ROM <NUM> loads and executes the instructions of the program in the RAM <NUM>.

The random access memory <NUM> notably comprises:.

According to one particular embodiment, the algorithms implementing the steps of the method specific to the present disclosure and described hereafter are advantageously stored in a memory GRAM of the graphics card <NUM> associated with the device <NUM> implementing these steps.

According to a variant, the power supply <NUM> is external to the device <NUM>.

<FIG> diagrammatically shows an embodiment of a method <NUM> as implemented in a processing device such as the device <NUM> according to a non-restrictive advantageous embodiment.

In an initialization step <NUM>, the device <NUM> obtains the settings of the method and a Zero Direction. It should also be noted that a step of obtaining an information in the present document can be viewed either as a step of reading such an information in a memory unit of an electronic device or as a step of receiving such an information from another electronic device via communication means (e.g. via a wired or a wireless connection or by contact connection). Obtained information are stored in register <NUM> of the random access memory <NUM> of the device <NUM>.

A step <NUM> consists in obtaining data representative of a set of reference directions. In a first embodiment, the set of reference directions is received from another device via communications means. These data may be associated with the video content or may be provided by a dedicated server. In a variant, reference direction data are read from a file on a storage medium associated with the device <NUM>. In another embodiment, the set of reference directions is obtained by image processing the video content. For instance, the processing of saliency maps of the images of the video content allow to detect highly salient regions. A point of such a region, for example the barycentre or the pixel with the highest saliency, may be used to determine a reference direction. In another embodiment, some objects of the scene that the camera is filming are associated with positioning device. Reference directions are set according to the position of these objects and the position of the camera. When any of these object is moving and/or when the camera is moving, the reference directions are modified.

When a change is detected in the set of known reference directions (even when created by the initialization step <NUM>), a step <NUM> is executed that computes a discrepancy function. The discrepancy function associate an angle value managed by the camera controller with an angle value corresponding to the aiming direction of the camera. The use of such a function generates a pseudo-haptic effect when using the camera controller as the camera does not react as the user expects. The discrepancy function is computed according to setting data which rule the pseudo-haptic effects. In a variant, additional reference parameters are associated with a reference direction in order to adapt the pseudo-haptic effect to the reference direction. Two occurrences of a similar reference direction may generate different discrepancy functions.

A step <NUM> consists in detecting changes in the parameters of the camera controller. An angle value, called θcontrol in this document, is updated according to the detected change in parameters. This angle is representative of the direction the user would like the camera to aim. A next step <NUM> is executed when θcontrol is updated or when a new discrepancy function has been computed at step <NUM>. In a variant, a timer is associated with the step <NUM> and a step <NUM> is executed once a duration value is over even if no change has been detected in the parameters of the step controller or in the set of reference directions at step <NUM>.

The step <NUM> consists in applying the discrepancy function on θcontrol. The result of this application is an aiming direction for the camera.

An optional step <NUM> consists in transmitting the computed aiming direction to the camera. In a variant, the aiming direction is transmitted only if it differs from the actual aiming direction of the camera for at least a threshold value (e.g. <NUM>°or <NUM>°or <NUM>°). In another variant, t he aiming direction is repeatedly transmitted to the camera even if no new aiming direction has been calculated at step <NUM>.

The method is activated at step <NUM> if a change of the set of reference directions is detected or at step <NUM> if a change in of the parameters of the camera controller is detected. In a variant, the method is activated by the running out of a timer.

Naturally, the present disclosure is not limited to the embodiments previously described. In particular, the present disclosure is not limited to a method of determining an aiming position command to a motorized camera but also extends to a method of transmitting an aiming direction to a camera and to a method of controlling the aiming direction of a motorized camera. The implementation of calculations necessary to compute the aiming position are not limited to an implementation in a CPU but also extends to an implementation in any program type, for example programs that can be executed by a GPU type microprocessor.

The implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method or an apparatus), the implementation of features discussed may also be implemented in other forms (for example a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, smartphones, tablets, computers, mobile phones, portable/personal digital assistants ("PDAs"), and other devices.

Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing, and other processing of images and related texture information and/or depth information. Examples of such equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette ("CD"), an optical disc (such as, for example, a DVD, often referred to as a digital versatile disc or a digital video disc), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two. A processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.

Claim 1:
A method of determining an aiming direction (<NUM>) of a virtual camera (<NUM>) of a head mounted display device when rendering an immersive content in the head mounted display device, the method comprising:
obtaining (<NUM>) at least one reference direction (<NUM>, <NUM>, <NUM>) pointing to a region of interest of the immersive content;
obtaining (<NUM>) an orientation of the head mounted display device (<NUM>, <NUM>, <NUM>);
determining (<NUM>) parameters representative of a function (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), according to the at least one reference direction (<NUM>, <NUM>, <NUM>), the function representing a discrepancy between an aiming direction of the virtual camera and the orientation of the head mounted display device; and
when the reference direction and the orientation of the head mounted display device are different, determining (<NUM>) a new aiming direction of the virtual camera by applying the function on the orientation of the head mounted display device to create a difference between the orientation of the head mounted display device and the new aiming direction of the virtual camera wherein the new aiming direction is between the reference direction and the orientation of the head mounted display device; and
rotating the aiming direction of the virtual camera to the new aiming direction to display visual feedback including an image from the virtual camera at the new aiming direction.