Patent Description:
Mobile devices are prone to various movements that may turn the mobile device such that the visual content rendered on the display is no longer suitable in terms of user's field of view. Various sensors may be used to determine for example the orientation of the device and based on information obtained from the sensors, orientation of the rendered visual content may be updated. Yet, the updated orientation of the rendered visual content may not always result in an optimal user experience.

Publication <CIT> discloses methods and apparatuses that may be implemented in a mobile device to allow for display mode selection based, at least in part, on a motion direction with respect to an orientation of a display of the mobile device. Publication <CIT> discloses techniques by which visual data is mapped to one or more display devices as a function of the positions of the display devices. The invention may support mapping of visual data to any number of display devices, which may cooperate to display the visual data as if the display devices were a single, large display device. In addition, the invention supports dynamic monitoring of the positions of display devices, with the visual data being mapped to a display device changing as the position of each display device changes.

The scope of protection sought for various embodiments is set out by the independent claims. Dependent claims define further embodiments included in the scope of protection. The exemplary embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

As used herein, the term "determining" (and grammatical variants thereof) can include, not least: calculating, computing, processing, deriving, measuring, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), obtaining and the like. Also, "determining" can include resolving, selecting, choosing, establishing, and the like.

Mobile devices, such as mobile phones, wearable devices, cameras or tablet computers, are often carried by a user in various activities during which the orientation of the device may change rather often. For example, if the user is running, on a train or riding a bicycle, the user may at the same time interact with the mobile device and the mobile device consequently moves along the movement of the user. To ensure a good user experience, the mobile device may utilize input from one or more sensors, such as a gyroscope and/or an accelerometer, it comprises and/or is connected to, to determine an optimal orientation for visual content rendered on the display of the device. If the visual content is deemed to be upside down, for example, by the user, the user experience suffers which is not desirable. Orientation of visual content rendered on a display of a mobile device may be determined based on for example the direction of a gravity such that the lower part of the visual content is aligned to be towards the direction of the gravity. The direction of the gravity may be obtained for example based on input received from an accelerometer. However, if the mobile device is in a flat orientation, the direction of the gravity may not be useful anymore with regard to determining the orientation of the visual content to be rendered.

If the mobile device has a flat orientation, which may be understood as the display of the mobile device to be facing upwards or downwards such that the device is at least substantially orthogonal to the gravity, then a camera may be used to detect the face of the user. Based on the face of the user, the orientation of the visual content to be rendered on the display of the mobile device may be determined such the user experience is optimal. Yet, if the user is lying down or there are multiple faces recognized, the mobile device may not be able to recognize the face of the user such that it can determine the optimal orientation for the visual content that is to be rendered. Further, the face of the user may not be visible in a dark environment, the face of the user may not be recognized by the mobile device if the user is wearing a face mask, etc..

If a mobile device therefore is oriented in a flat orientation, the optimal orientation for the visual content to be rendered on a display of the mobile device may need to be determined based on other input. In some exemplary embodiments, if it is determined, based for example input from one or more sensors, that the mobile device is oriented in a flat orientation, movement of the mobile device may also be determined and, based at least partly on the determined movement of the mobile device, orientation of the visual content to be rendered on the display of the mobile device is determined. Movement of the mobile device may be understood to cover movements and trajectories of the mobile device and/or indication regarding if the mobile device is still. Movement and trajectories may be determined, for example, by using methods such as Global Positioning System (GPS), High Accuracy Indoor Positioning (HAIP), Ultra WideBand (UWB) based positioning, positioning systems based on one or more cameras, accelerometers etc. or any combination thereof. For example, if it is determined that the orientation of the mobile device is a flat orientation, then the movement of the mobile device may be determined and visual content may be rendered on the display of the mobile device may be oriented based, at least partly, on the determined movement of the mobile device. For example, the upper part of the visual content may be aligned with the direction of the movement of the mobile device. Additionally, a context of the mobile device may further be determined, and the orientation of the visual content rendered on the display of the terminal device may further be based on the determined context of the mobile device. The context of the mobile device may be understood as activity of the mobile device and/or activity in which the mobile device is involved. For example, an activity of the mobile device may be understood as an application being active and/or the mobile device having one or more active connections to adjacent devices. Further, activity in which the mobile device is involved may be understood for example as the user performing an activity such as passing on the device to another person, running or cycling, and the activity of the user is detected by the terminal device.

<FIG> illustrate an exemplary embodiment of aligning the orientation of the visual content rendered on the display of the mobile device. In <FIG>, a user <NUM> is running and is also using his mobile device <NUM> at the same time. Thus, in this exemplary embodiment, the context of the mobile device <NUM> is an activity of the user, that is, the movement of the user. The direction of the movement of the mobile device, which in this exemplary embodiment corresponds to the direction of the movement of the user <NUM>, is illustrated by the arrow <NUM>. In this exemplary embodiment the activity of the user, that is, the context of the mobile device <NUM>, is determined by the mobile device based, at least partly, on input received from one or more accelerometers comprised in the mobile device. Additionally or alternatively, input from other sensors may be utilized as well to determine the activity of the user. Further additionally or alternatively, a user activity setting in an application such as an activity tracking application may be utilized to determine the activity of the user. The determined activity may further be determined to be such that the user <NUM> is expected to keep focus towards the front direction of the user, which is also the direction of the movement of the mobile device <NUM>. As the orientation of the mobile device is further determined to be a flat orientation, the orientation of the content rendered <NUM> on the display of the mobile device is aligned with the direction of the movement of the mobile device <NUM>. The aligned orientation of the content rendered <NUM> on the display of the mobile device is, in this exemplary embodiment, such that the upper part of the visual content rendered is in the direction of the movement of the mobile device <NUM>. This aligned orientation is then fixed for the duration of the activity. In other words, the orientation of the visual content rendered on the display of the mobile device is kept fixed for as long as the context is determined to be valid. In other words, for as long as the activity of the user continues. Once the activity of the user is determined to be different, it may be determined that the context of the mobile device is no longer valid. Alternatively or additionally, the context may be determined not to be valid if the orientation of the device is determined to be other than a flat orientation.

<FIG> illustrates a flow chart according to the exemplary embodiment of <FIG>. First in S1 an orientation of a mobile device is determined to be a flat orientation. Then in S2 the mobile device determines its context. In this exemplary embodiment, the context is determined based on detecting an activity of the user. In this exemplary embodiment, the activity of the user comprises movement of the user. Next, in S3, it is determined if the activity of the user corresponds to keeping the orientation aligned with the movement of the device. If yes, then in S4 the orientation of the content rendered on the display is kept aligned with the movement of the mobile device.

<FIG> illustrate another exemplary embodiment of determining an orientation of visual content rendered on a display of a mobile device <NUM>. In this exemplary embodiment, a user <NUM> is using a mobile device <NUM> and activating an application that requires the user to point the mobile device <NUM> towards an object that in this exemplary embodiment is a flower but in some other exemplary embodiments the object could be a person or a bar code for example. For example, the user <NUM> sees the flower and decides to take a picture of the flower. In this exemplary embodiment, the flower is knee high so the user <NUM> has the mobile device <NUM> such that its orientation is a flat orientation. The mobile device <NUM> determines that its orientation is flat orientation. As the user <NUM> decides to take a picture of the flower, the user activates the camera application of the mobile device <NUM>. In other words, the context of the mobile device <NUM> may be determined as the mobile device being involved in a user activity based, at least partly, on the application activated. As the user <NUM> activates the camera application, the user <NUM> then points the mobile device <NUM> towards the flower and towards the end of the pointing movement the user <NUM> decides whether the picture is to be a portrait or a landscape picture. For the visual content rendered on the display of the mobile device <NUM> as the user decides if the picture is to be a landscape or a portrait picture the orientation should be kept fixed such that it is not aligned with the movement of the mobile device <NUM> after the pointing movement as such is completed. In other words, once the orientation of the mobile device <NUM> is a flat orientation.

Thus, once it is determined that the context of the mobile device <NUM> is such that it is involved in an activity of a user and once it is determined that the orientation of the mobile device <NUM> is a flat orientation, the visual content rendered on the display of the mobile device <NUM> is oriented such that it remains fixed despite the movement of the mobile device <NUM>. In this exemplary embodiment, direction of the movement of the mobile device <NUM> is illustrated by the arrows <NUM>, <NUM> and <NUM>. Yet, the orientation of the visual content rendered on the display of the mobile device <NUM> is optimally rendered when it is fixed despite the movement of the mobile device.

Although camera was mentioned in this exemplary embodiment, it is to be noted that the orientation of the visual content rendered on the display of the mobile device <NUM>, once the mobile device <NUM> is in a flat orientation, is determined to be fixed despite the movement of the mobile device <NUM>, this approach could also be used with other applications such as meeting audio recorder, journalist interview audio recorder or bar code reader. In other words, orientation of the visual content rendered on the display of the mobile device <NUM> is determined based on the context of the mobile device and the flat orientation of the mobile device <NUM>. The movement of the mobile device <NUM> may be considered to also be a factor in a sense that the orientation is to be aligned along the end point of the movement of the mobile device <NUM> that took place prior to detecting the flat orientation of the mobile device <NUM>, after which the orientation of the visual content rendered on the display of the mobile device <NUM> is kept fixed until the orientation of the display device is no longer flat and/or the context of the mobile device <NUM> is no longer valid. That is, until the application is no longer active.

<FIG> illustrates a flow chart according to the exemplary embodiment of <FIG>. In this exemplary embodiment, in S1, it is determined that the orientation of the mobile device is a flat orientation. Next, in S2 the context of the mobile device is determined, which comprises according to the claimed invention determining that an application is activated and that the application is such that the mobile device is determined to be involved in user activity. Next, in S3, movement of the device, which in this exemplary embodiment is the movement of the device right after the application is activated that may be the first detected movement after the application is activated, is determined and orientation of visual content rendered on the display of the device is aligned with the movement of the mobile device. Alternatively, if no clear movement is detected after the application has been activated, the movement of the mobile device may be determined to be end of the movement prior to activating the application. That is the end of the last movement before the application was activated. The orientation of the visual content rendered on the display is then aligned with the determined direction of the movement of the terminal device. Aligning the visual content may be understood as the upper part of the visual content being in the direction of the movement of the mobile device. Finally, in S4 the aligned orientation of the visual content rendered on the display of the mobile device is kept fixed until the orientation of the mobile device is no longer flat and/or the application is no longer active.

In general, applications such as camera, bar code reader, QR code reader, matrix barcode reader, meeting audio recorder and/or interview audio recorder may require a user to point a mobile device towards an object. In these applications, the user may activate the application after which he makes a move towards the object/person. The movement right after application has been activated may be detected based on for example accelerometers. Since the first movement may take the mobile device from a view direction of the user towards the direction of the object, the movement may be interpreted as revealing the optimal display rotation while the user is using the application.

<FIG> illustrate yet another exemplary embodiment of determining an optimal orientation of visual content rendered on a display of a mobile device. In the exemplary embodiment of <FIG> a user <NUM> moves such that his movement differs from the movement of a mobile device <NUM>. In this exemplary embodiment, the movement of the mobile device <NUM> is detected using at least an accelerometer. The movement of the user <NUM> is determined based on input from one or more of the following: earbuds, smart glasses, virtual reality (VR) glasses, augmented reality (AR) glasses and/or health tracking wearables etc. with accelerometers or other sensors. In this exemplary embodiment, it is determined that as the movement of the user <NUM> and the movement of the mobile device <NUM> differ, the context of the mobile device may be one of the following: the user <NUM> moved the mobile device <NUM> away from him, the user <NUM> rotated the mobile device <NUM> or the user <NUM> rotated his head. Yet, in these options the context of the mobile device <NUM> is such that the mobile device <NUM> is involved in an activity of the user <NUM>. In this exemplary embodiment, it is also determined that the orientation of the mobile device <NUM> is a flat orientation.

The difference determined between the movement of the user <NUM> and the movement of the mobile device <NUM> may result in various orientations of visual content rendered on the display of the mobile device <NUM> to be considered as an optimal rotation. For example, if the movement <NUM> of the mobile device <NUM> is towards right of the trajectory of the movement <NUM> of the user, then the orientation of the visual content rendered on the display of the mobile device <NUM> is rotated clockwise. Alternatively, if the movement of the mobile device <NUM> is towards left of the trajectory of the movement <NUM> of the user, then the orientation of the visual content rendered on the display of the mobile device <NUM> is rotated counter-clockwise.

On the other hand, if the movement <NUM> of the mobile device <NUM> differs significantly of the movement of the user <NUM>, for example the mobile device <NUM> has clockwise rotation of <NUM>° while the user <NUM> has no rotation, then the orientation of the visual content rendered on the display of the mobile device <NUM> is rotated counter-clockwise for example by <NUM>°. Further, if the user <NUM> rotates the mobile device <NUM>, but the user <NUM> does not rotate, then the visual content rendered on the display of the mobile device <NUM> may rotated opposite to the rotation of the mobile device <NUM>. On the other hand, if the user <NUM> rotates his head, but the mobile device <NUM> then the visual content rendered on the display of the mobile device <NUM> is not rotated.

<FIG> illustrates a flow chart according to the exemplary embodiment of <FIG>. In S1, it is determined that an orientation of a mobile device is a flat orientation. Next, in S2 the context of the mobile device is determined. In this exemplary embodiment the context is such that the mobile device is involved in an activity of the user as the movement of the mobile device and the movement of the user are determined. Based on a difference between the movement of the user and the movement of the terminal device, in S3 an orientation of visual content rendered on the display of the mobile device is then determined. In S4, it is continued to determine the orientation of the visual content to be rendered on the display of the mobile device based on the difference between the movement of the user and the movement of the mobile device until it is determined that the orientation of the mobile device is no longer a flat orientation.

<FIG> illustrate another exemplary embodiment in which the orientation of the visual content rendered on a display of a mobile device <NUM> is determined. In the exemplary embodiment of <FIG>, the context of the mobile device <NUM> is such that the mobile device <NUM> is involved in an activity of a user <NUM>. In this exemplary embodiment, an orientation of the mobile device <NUM> is determined to be a flat orientation. It is then determined that the context of the mobile device <NUM> is such that the user <NUM> moves the mobile device <NUM>. This may be for example the case when the user <NUM> gives the mobile device <NUM> to another user <NUM>. As the mobile device <NUM> determines the context, which in this exemplary embodiment is such that the user <NUM> gives the mobile device <NUM> to another user <NUM>, the orientation of the visual content rendered on the display of the mobile device <NUM> is determined to be fixed, in other words, not rotated, or alternatively, the orientation is rotated <NUM>°. In this manner the same user experience is enabled for the other user <NUM> than to the user <NUM>. If the orientation of the visual content rendered on the display of the mobile device <NUM> is not rotated, in other words, is kept fixed, the other user <NUM> is provided with the same user experience than the user <NUM>. The user experience may be dependent on for example speaker locations on the mobile device <NUM> with respect to the display of the mobile device <NUM>. Further, in this manner the user <NUM> can be sure that the other user <NUM> is provided with the same user experience as the user <NUM> did by just looking if the other user <NUM> is holding the device the same way as the user <NUM> did. On the other hand, if the orientation is rotated such that the rotation is limited to <NUM>°, a benefit of portrait or landscape rotation does not disturb the user experience may be achieved.

The context of the mobile device <NUM> may be determined in various manners. For example, the mobile device <NUM> may detect that the user <NUM> is giving the mobile device <NUM> to the other user <NUM> by recognizing a sequence of events in which the face of the user <NUM> is first detected using a camera comprised in the mobile device <NUM> and wherein the camera is on the same side as the display of the mobile device <NUM>, after which movement of the mobile device <NUM> is determined based on input received from one or more accelerometers and then a face of the user <NUM> is detected using the camera. Once the movement of the mobile device <NUM> is completed, or during the movement of the mobile device <NUM>, the visual content rendered on the display for the mobile device <NUM> may be rotated back to the same rotation it had when it was viewed by the user <NUM> or, alternatively, rotation may be disabled, if the face of the user <NUM> is not recognized from the image obtained from the camera.

Alternatively or additionally, the context of the mobile device <NUM> may be determined based on its distance to other devices such as wearable devices worn by the users <NUM> and <NUM>. For example, if the user <NUM> wears a smart watch and the user <NUM> has headphones that are paired with the mobile device <NUM>, the mobile device <NUM> may detect its movement based on the changing distances to the wearable devices. For example, the mobile device <NUM> may detect that it is moving further away from the smart watch and closer to the headphones. If it is further determined that the orientation of the mobile device <NUM> is a flat orientation, then the orientation of the visual content rendered on the display of the mobile device <NUM> may further be determined based on the context of the mobile device <NUM> and the movement of the mobile device <NUM>.

<FIG> illustrates a flow chart according to the exemplary embodiment of <FIG>. First, in S1, it is determined that an orientation of the mobile device is a flat orientation. Then in S2 the context of the mobile device is determined. In this exemplary embodiment the context is an activity of a user in which the mobile device is involved. The user in this exemplary embodiment passes the mobile device to another user. The mobile device also determines its movement. Then, in S3, based on the context of the mobile device and the movement of the mobile device, the orientation of visual content rendered on the display of the mobile device is determined and the visual content is rendered accordingly.

<FIG> illustrate yet another exemplary embodiment in which orientation of visual content rendered on a display of a mobile device is determined. In the exemplary embodiment of <FIG>, a user <NUM> passes the mobile device <NUM> to a user <NUM>. The user <NUM> is wearing headphones that are paired with the mobile device <NUM> and the user <NUM> is also wearing headphones that are paired with the mobile device <NUM>. The mobile device <NUM> determines that its orientation is flat. The mobile device <NUM> then determines that its context is that it has at least one active connection to another device, that is, to the headphones. The mobile device <NUM> also determines that its movement is towards a device to which it has an active connection. At the end of the movement, the mobile device <NUM> determines a direction in which the closest device, to which it has an active connection to is. Based on this direction, the orientation of the visual content to be rendered on the display of the mobile device <NUM> is determined and rendered accordingly. Additionally, the mobile device may restrict rotation of the orientation to <NUM>° angles to keep the user experience the same instead of switching between landscape and portrait.

The determining of the closest device to which the mobile device <NUM> has an active connection to may be achieved in various manners. For example, Ultra Wide Band (UWB) or Bluetooth LE may be utilized determine the closest device based on signal strength and also its direction when Bluetooth LE antenna array is used. It is to be noted that the mobile device <NUM> may have multiple active connections to other devices at the same time. These multiple connections may be for example between the mobile device <NUM> and different wearable devices, such as earbuds, heart rate meters and/or electrocardiogram (EKG) meters.

<FIG> illustrates a flow chart according to the exemplary embodiment of <FIG>. First, in S1, it is determined that an orientation of the mobile device is a flat orientation. Then in S2 the context of the mobile device is determined. In this exemplary embodiment the context is an active connection to at least one other device. The user in this exemplary embodiment passes the mobile device to another user. The mobile device also determines its movement. Then, in S3, based on the context of the mobile device and the movement of the mobile device, the orientation of visual content rendered on the display of the mobile device is determined and the visual content is rendered accordingly.

<FIG> illustrates an apparatus <NUM>, which may be a computing apparatus, according to an example embodiment. The apparatus <NUM> may be, or may be comprised in, a mobile device such as those described in the exemplary embodiments above. The apparatus <NUM> comprises a processor <NUM>. The processor <NUM> interprets computer program instructions and process data. The processor <NUM> may comprise one or more programmable processors. The processor <NUM> may comprise programmable hardware with embedded firmware and may, alternatively or additionally, comprise one or more application specific integrated circuits, ASICs.

The processor <NUM> is coupled to a memory <NUM>. The processor is configured to read and write data to and from the memory <NUM>. The memory <NUM> may comprise one or more memory units. The memory units may be volatile or non-volatile. It is to be noted that in some example embodiments there may be one or more units of non-volatile memory and one or more units of volatile memory or, alternatively, one or more units of non-volatile memory, or, alternatively, one or more units of volatile memory. Volatile memory may be for example RAM, DRAM or SDRAM. Non-volatile memory may be for example ROM, PROM, EEPROM, flash memory, optical storage or magnetic storage. In general, memories may be referred to as non-transitory computer readable media. The memory <NUM> stores computer readable instructions that are execute by the processor <NUM>. For example, non-volatile memory stores the computer readable instructions and the processor <NUM> executes the instructions using volatile memory for temporary storage of data and/or instructions.

The computer readable instructions may have been pre-stored to the memory <NUM> or, alternatively or additionally, they may be received, by the apparatus, via electromagnetic carrier signal and/or may be copied from a physical entity such as computer program product. Execution of the computer readable instructions causes the apparatus <NUM> to perform functionality described above.

The apparatus <NUM> further comprises, or is connected to, an input unit <NUM>. The input unit <NUM> comprises one or more interfaces for receiving a user input. The one or more interfaces may comprise for example one or more motion and/or orientation sensors, one or more cameras, one or more accelerometers, one or more microphones, one or more buttons and one or more touch detection units. Further, the input unit <NUM> may comprise an interface to which external devices may connect to.

The apparatus <NUM> also comprises an output unit <NUM>. The output unit comprises or is connected to one or more displays capable of rendering visual content such as a light emitting diode (LED) display, a liquid crystal display (LCD) and/or a liquid crystal on silicon (LCoS) display. The output unit <NUM> may comprise two displays to render stereoscopic visual content. One display to render content to the left eye and the other display to render content to the right eye. The output unit <NUM> may further comprise a transmission unit, such as one or more waveguides or one or more lenses, to transfer the rendered visual content to the user's field of view. The output unit <NUM> may further comprise one or more audio outputs. The one or more audio outputs may be for example loudspeakers or a set of headphones.

The apparatus <NUM> may further comprise a connectivity unit <NUM>. The connectivity unit <NUM> enables wired and/or wireless connectivity to external networks. The connectivity unit <NUM> may comprise one or more antennas and one or more receivers that may be integrated to the apparatus <NUM> or the apparatus <NUM> may be connected to. The connectivity unit <NUM> may comprise an integrated circuit or a set of integrated circuits that provide the wireless communication capability for the apparatus <NUM>. Alternatively, the wireless connectivity may be a hardwired application specific integrated circuit (ASIC).

It is to be noted that the apparatus <NUM> may further comprise various components not illustrated in the <FIG>. The various components may be hardware component and/or software components.

Claim 1:
An apparatus (<NUM>) comprising means for:
determining that an orientation of a mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is a flat orientation, in which a display of the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is facing upwards or downwards and is at least substantially orthogonal to the gravity;
determining a context of the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), wherein determining the context comprises determining that an application of the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is activated and determining that the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is involved in an activity of a user (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) based on the activated application;
determining movement (<NUM>, <NUM>) of the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>); and
based on the orientation, context and movement (<NUM>, <NUM>) of the device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), determining orientation of visual content to be rendered on the display of the mobile device (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>).