Orientation of visual content rendered on a display of a mobile device

A method comprising determining that an orientation of a mobile device is a flat orientation, determining a context of the mobile device, determining movement of the mobile device, and based on the orientation, context and movement of the device, determining orientation of visual content rendered on a display of the mobile device.

FIELD

The present application relates to rendering visual content on a display of a mobile device.

BACKGROUND

Mobile devices are prone to various movements that may turn the mobile device such that the visual content rendered on the display is no longer suitable in terms of user's field of view. Various sensors may be used to determine for example the orientation of the device and based on information obtained from the sensors, orientation of the rendered visual content may be updated. Yet, the updated orientation of the rendered visual content may not always result in an optimal user experience.

BRIEF DESCRIPTION

According to an aspect there is provided an apparatus comprising means for determining that an orientation of a mobile device is a flat orientation, determining a context of the mobile device, determining movement of the mobile device, and based on the orientation, context and movement of the device, determining orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided an apparatus comprising at least one processor, and at least one memory including a computer program code, wherein the at least one memory and the computer program code are configured, with the at least one processor, to cause the apparatus to determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided a method comprising determining that an orientation of a mobile device is a flat orientation, determining a context of the mobile device, determining movement of the mobile device, and based on the orientation, context and movement of the device, determining orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided a computer program product comprising instructions for causing an apparatus to perform at least the following: determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided a computer program comprising instructions stored thereon for performing at least the following: determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided a non-transitory computer readable medium comprising program instructions for causing an apparatus to perform at least the following: determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

According to another aspect there is provided non-transitory computer readable medium comprising program instructions stored thereon for performing at least the following: determine that an orientation of a mobile device is a flat orientation, determine a context of the mobile device, determine movement of the mobile device, and based on the orientation, context and movement of the device, determine orientation of visual content rendered on a display of the mobile device.

DETAILED DESCRIPTION

Mobile devices, such as mobile phones, wearable devices, cameras or tablet computers, are often carried by a user in various activities during which the orientation of the device may change rather often. For example, if the user is running, on a train or riding a bicycle, the user may at the same time interact with the mobile device and the mobile device consequently moves along the movement of the user. To ensure a good user experience, the mobile device may utilize input from one or more sensors, such as a gyroscope and/or an accelerometer, it comprises and/or is connected to, to determine an optimal orientation for visual content rendered on the display of the device. If the visual content is deemed to be upside down, for example, by the user, the user experience suffers which is not desirable. Orientation of visual content rendered on a display of a mobile device may be determined based on for example the direction of a gravity such that the lower part of the visual content is aligned to be towards the direction of the gravity. The direction of the gravity may be obtained for example based on input received from an accelerometer. However, if the mobile device is in a flat orientation, the direction of the gravity may not be useful anymore with regard to determining the orientation of the visual content to be rendered.

If the mobile device has a flat orientation, which may be understood as the display of the mobile device to be facing upwards or downwards such that the device is at least substantially orthogonal to the gravity, then a camera may be used to detect the face of the user. Based on the face of the user, the orientation of the visual content to be rendered on the display of the mobile device may be determined such the user experience is optimal. Yet, if the user is lying down or there are multiple faces recognized, the mobile device may not be able to recognize the face of the user such that it can determine the optimal orientation for the visual content that is to be rendered. Further, the face of the user may not be visible in a dark environment, the face of the user may not be recognized by the mobile device if the user is wearing a face mask, etc.

If a mobile device therefore is oriented in a flat orientation, the optimal orientation for the visual content to be rendered on a display of the mobile device may need to be determined based on other input. In some exemplary embodiments, if it is determined, based for example input from one or more sensors, that the mobile device is oriented in a flat orientation, movement of the mobile device may also be determined and, based at least partly on the determined movement of the mobile device, orientation of the visual content to be rendered on the display of the mobile device is determined. Movement of the mobile device may be understood to cover movements and trajectories of the mobile device and/or indication regarding if the mobile device is still. Movement and trajectories may be determined, for example, by using methods such as Global Positioning System (GPS), High Accuracy Indoor Positioning (HAIP), Ultra WideBand (UWB) based positioning, positioning systems based on one or more cameras, accelerometers etc. or any combination thereof. For example, if it is determined that the orientation of the mobile device is a flat orientation, then the movement of the mobile device may be determined and visual content may be rendered on the display of the mobile device may be oriented based, at least partly, on the determined movement of the mobile device. For example, the upper part of the visual content may be aligned with the direction of the movement of the mobile device. Additionally, a context of the mobile device may further be determined, and the orientation of the visual content rendered on the display of the terminal device may further be based on the determined context of the mobile device. The context of the mobile device may be understood as activity of the mobile device and/or activity in which the mobile device is involved. For example, an activity of the mobile device may be understood as an application being active and/or the mobile device having one or more active connections to adjacent devices. Further, activity in which the mobile device is involved may be understood for example as the user performing an activity such as passing on the device to another person, running or cycling, and the activity of the user is detected by the terminal device.

FIGS.1A and1Billustrate an exemplary embodiment of aligning the orientation of the visual content rendered on the display of the mobile device. InFIG.1A, a user110is running and is also using his mobile device120at the same time. Thus, in this exemplary embodiment, the context of the mobile device120is an activity of the user, that is, the movement of the user. The direction of the movement of the mobile device, which in this exemplary embodiment corresponds to the direction of the movement of the user110, is illustrated by the arrow130. In this exemplary embodiment the activity of the user, that is, the context of the mobile device120, is determined by the mobile device based, at least partly, on input received from one or more accelerometers comprised in the mobile device. Additionally or alternatively, input from other sensors may be utilized as well to determine the activity of the user. Further additionally or alternatively, a user activity setting in an application such as an activity tracking application may be utilized to determine the activity of the user. The determined activity may further be determined to be such that the user110is expected to keep focus towards the front direction of the user, which is also the direction of the movement of the mobile device120. As the orientation of the mobile device is further determined to be a flat orientation, the orientation of the content rendered135on the display of the mobile device is aligned with the direction of the movement of the mobile device120. The aligned orientation of the content rendered135on the display of the mobile device is, in this exemplary embodiment, such that the upper part of the visual content rendered is in the direction of the movement of the mobile device130. This aligned orientation is then fixed for the duration of the activity. In other words, the orientation of the visual content rendered on the display of the mobile device is kept fixed for as long as the context is determined to be valid. In other words, for as long as the activity of the user continues. Once the activity of the user is determined to be different, it may be determined that the context of the mobile device is no longer valid. Alternatively or additionally, the context may be determined not to be valid if the orientation of the device is determined to be other than a flat orientation.

FIG.1Billustrates a flow chart according to the exemplary embodiment ofFIG.1A. First in S1an orientation of a mobile device is determined to be a flat orientation. Then in S2the mobile device determines its context. In this exemplary embodiment, the context is determined based on detecting an activity of the user. In this exemplary embodiment, the activity of the user comprises movement of the user. Next, in S3, it is determined if the activity of the user corresponds to keeping the orientation aligned with the movement of the device. If yes, then in S4the orientation of the content rendered on the display is kept aligned with the movement of the mobile device.

FIGS.2A and2Billustrate another exemplary embodiment of determining an orientation of visual content rendered on a display of a mobile device220. In this exemplary embodiment, a user210is using a mobile device220and activating an application that requires the user to point the mobile device220towards an object that in this exemplary embodiment is a flower but in some other exemplary embodiments the object could be a person or a bar code for example. For example, the user210sees the flower and decides to take a picture of the flower. In this exemplary embodiment, the flower is knee high so the user210has the mobile device220such that its orientation is a flat orientation. The mobile device220determines that its orientation is flat orientation. As the user210decides to take a picture of the flower, the user activates the camera application of the mobile device220. In other words, the context of the mobile device220may be determined as the mobile device being involved in a user activity based, at least partly, on the application activated. As the user210activates the camera application, the user210then points the mobile device220towards the flower and towards the end of the pointing movement the user210decides whether the picture is to be a portrait or a landscape picture. For the visual content rendered on the display of the mobile device220as the user decides if the picture is to be a landscape or a portrait picture the orientation should be kept fixed such that it is not aligned with the movement of the mobile device220after the pointing movement as such is completed. In other words, once the orientation of the mobile device220is a flat orientation.

Thus, once it is determined that the context of the mobile device220is such that it is involved in an activity of a user and once it is determined that the orientation of the mobile device220is a flat orientation, the visual content rendered on the display of the mobile device220is oriented such that it remains fixed despite the movement of the mobile device220. In this exemplary embodiment, direction of the movement of the mobile device220is illustrated by the arrows222,224and226. Yet, the orientation of the visual content rendered on the display of the mobile device220is optimally rendered when it is fixed despite the movement of the mobile device.

Although camera was mentioned in this exemplary embodiment, it is to be noted that the orientation of the visual content rendered on the display of the mobile device220, once the mobile device220is in a flat orientation, is determined to be fixed despite the movement of the mobile device220, this approach could also be used with other applications such as meeting audio recorder, journalist interview audio recorder or bar code reader. In other words, orientation of the visual content rendered on the display of the mobile device220is determined based on the context of the mobile device and the flat orientation of the mobile device220. The movement of the mobile device220may be considered to also be a factor in a sense that the orientation is to be aligned along the end point of the movement of the mobile device220that took place prior to detecting the flat orientation of the mobile device220, after which the orientation of the visual content rendered on the display of the mobile device220is kept fixed until the orientation of the display device is no longer flat and/or the context of the mobile device220is no longer valid. That is, until the application is no longer active.

FIG.2Billustrates a flow chart according to the exemplary embodiment ofFIG.2A. In this exemplary embodiment, in S1, it is determined that the orientation of the mobile device is a flat orientation. Next, in S2the context of the mobile device is determined, which comprises determining that an application is activated and that the application is such that the mobile device is determined to be involved in user activity. Next, in S3, movement of the device, which in this exemplary embodiment is the movement of the device right after the application is activated that may be the first detected movement after the application is activated, is determined and orientation of visual content rendered on the display of the device is aligned with the movement of the mobile device. Alternatively, if no clear movement is detected after the application has been activated, the movement of the mobile device may be determined to be end of the movement prior to activating the application. That is the end of the last movement before the application was activated. The orientation of the visual content rendered on the display is then aligned with the determined direction of the movement of the terminal device. Aligning the visual content may be understood as the upper part of the visual content being in the direction of the movement of the mobile device. Finally, in S4the aligned orientation of the visual content rendered on the display of the mobile device is kept fixed until the orientation of the mobile device is no longer flat and/or the application is no longer active.

In general, applications such as camera, bar code reader, QR code reader, matrix barcode reader, meeting audio recorder and/or interview audio recorder may require a user to point a mobile device towards an object. In these applications, the user may activate the application after which he makes a move towards the object/person. The movement right after application has been activated may be detected based on for example accelerometers. Since the first movement may take the mobile device from a view direction of the user towards the direction of the object, the movement may be interpreted as revealing the optimal display rotation while the user is using the application.

FIGS.3A and3Billustrate yet another exemplary embodiment of determining an optimal orientation of visual content rendered on a display of a mobile device. In the exemplary embodiment ofFIG.3Aa user310moves such that his movement differs from the movement of a mobile device320. In this exemplary embodiment, the movement of the mobile device320is detected using at least an accelerometer. The movement of the user310is determined based on input from one or more of the following: earbuds, smart glasses, virtual reality (VR) glasses, augmented reality (AR) glasses and/or health tracking wearables etc. with accelerometers or other sensors. In this exemplary embodiment, it is determined that as the movement of the user310and the movement of the mobile device320differ, the context of the mobile device may be one of the following: the user310moved the mobile device320away from him, the user310rotated the mobile device320or the user310rotated his head. Yet, in these options the context of the mobile device320is such that the mobile device320is involved in an activity of the user310. In this exemplary embodiment, it is also determined that the orientation of the mobile device320is a flat orientation.

The difference determined between the movement of the user310and the movement of the mobile device320may result in various orientations of visual content rendered on the display of the mobile device320to be considered as an optimal rotation. For example, if the movement332of the mobile device320is towards right of the trajectory of the movement330of the user, then the orientation of the visual content rendered on the display of the mobile device320is rotated clockwise. Alternatively, if the movement of the mobile device320is towards left of the trajectory of the movement330of the user, then the orientation of the visual content rendered on the display of the mobile device320is rotated counter-clockwise.

On the other hand, if the movement334of the mobile device320differs significantly of the movement of the user310, for example the mobile device320has clockwise rotation of 90° while the user310has no rotation, then the orientation of the visual content rendered on the display of the mobile device320is rotated counter-clockwise for example by 90°. Further, if the user310rotates the mobile device320, but the user310does not rotate, then the visual content rendered on the display of the mobile device320may rotated opposite to the rotation of the mobile device320. On the other hand, if the user310rotates his head, but the mobile device320then the visual content rendered on the display of the mobile device320is not rotated.

FIG.3Billustrates a flow chart according to the exemplary embodiment ofFIG.3A. In S1, it is determined that an orientation of a mobile device is a flat orientation. Next, in S2the context of the mobile device is determined. In this exemplary embodiment the context is such that the mobile device is involved in an activity of the user as the movement of the mobile device and the movement of the user are determined. Based on a difference between the movement of the user and the movement of the terminal device, in S3an orientation of visual content rendered on the display of the mobile device is then determined. In S4, it is continued to determine the orientation of the visual content to be rendered on the display of the mobile device based on the difference between the movement of the user and the movement of the mobile device until it is determined that the orientation of the mobile device is no longer a flat orientation.

FIGS.4A and4Billustrate another exemplary embodiment in which the orientation of the visual content rendered on a display of a mobile device420is determined. In the exemplary embodiment ofFIG.4A, the context of the mobile device420is such that the mobile device420is involved in an activity of a user410. In this exemplary embodiment, an orientation of the mobile device420is determined to be a flat orientation. It is then determined that the context of the mobile device420is such that the user410moves the mobile device420. This may be for example the case when the user410gives the mobile device420to another user430. As the mobile device420determines the context, which in this exemplary embodiment is such that the user410gives the mobile device420to another user430, the orientation of the visual content rendered on the display of the mobile device420is determined to be fixed, in other words, not rotated, or alternatively, the orientation is rotated 180°. In this manner the same user experience is enabled for the other user430than to the user410. If the orientation of the visual content rendered on the display of the mobile device420is not rotated, in other words, is kept fixed, the other user430is provided with the same user experience than the user410. The user experience may be dependent on for example speaker locations on the mobile device420with respect to the display of the mobile device420. Further, in this manner the user410can be sure that the other user430is provided with the same user experience as the user410did by just looking if the other user430is holding the device the same way as the user410did. On the other hand, if the orientation is rotated such that the rotation is limited to 180°, a benefit of portrait or landscape rotation does not disturb the user experience may be achieved.

The context of the mobile device420may be determined in various manners. For example, the mobile device420may detect that the user410is giving the mobile device420to the other user430by recognizing a sequence of events in which the face of the user410is first detected using a camera comprised in the mobile device420and wherein the camera is on the same side as the display of the mobile device420, after which movement of the mobile device420is determined based on input received from one or more accelerometers and then a face of the user430is detected using the camera. Once the movement of the mobile device420is completed, or during the movement of the mobile device420, the visual content rendered on the display for the mobile device420may be rotated back to the same rotation it had when it was viewed by the user410or, alternatively, rotation may be disabled, if the face of the user410is not recognized from the image obtained from the camera.

Alternatively or additionally, the context of the mobile device420may be determined based on its distance to other devices such as wearable devices worn by the users410and430. For example, if the user410wears a smart watch and the user430has headphones that are paired with the mobile device420, the mobile device420may detect its movement based on the changing distances to the wearable devices. For example, the mobile device420may detect that it is moving further away from the smart watch and closer to the headphones. If it is further determined that the orientation of the mobile device420is a flat orientation, then the orientation of the visual content rendered on the display of the mobile device420may further be determined based on the context of the mobile device420and the movement of the mobile device420.

FIG.4Billustrates a flow chart according to the exemplary embodiment ofFIG.4A. First, in S1, it is determined that an orientation of the mobile device is a flat orientation. Then in S2the context of the mobile device is determined. In this exemplary embodiment the context is an activity of a user in which the mobile device is involved. The user in this exemplary embodiment passes the mobile device to another user. The mobile device also determines its movement. Then, in S3, based on the context of the mobile device and the movement of the mobile device, the orientation of visual content rendered on the display of the mobile device is determined and the visual content is rendered accordingly.

FIGS.5A and5Billustrate yet another exemplary embodiment in which orientation of visual content rendered on a display of a mobile device is determined. In the exemplary embodiment ofFIG.5A, a user510passes the mobile device520to a user530. The user510is wearing headphones that are paired with the mobile device520and the user530is also wearing headphones that are paired with the mobile device520. The mobile device520determines that its orientation is flat. The mobile device520then determines that its context is that it has at least one active connection to another device, that is, to the headphones. The mobile device520also determines that its movement is towards a device to which it has an active connection. At the end of the movement, the mobile device520determines a direction in which the closest device, to which it has an active connection to is. Based on this direction, the orientation of the visual content to be rendered on the display of the mobile device520is determined and rendered accordingly. Additionally, the mobile device may restrict rotation of the orientation to 180° angles to keep the user experience the same instead of switching between landscape and portrait.

The determining of the closest device to which the mobile device520has an active connection to may be achieved in various manners. For example, Ultra Wide Band (UWB) or Bluetooth LE may be utilized determine the closest device based on signal strength and also its direction when Bluetooth LE antenna array is used. It is to be noted that the mobile device520may have multiple active connections to other devices at the same time. These multiple connections may be for example between the mobile device520and different wearable devices, such as earbuds, heart rate meters and/or electrocardiogram (EKG) meters.

FIG.5Billustrates a flow chart according to the exemplary embodiment ofFIG.5A. First, in S1, it is determined that an orientation of the mobile device is a flat orientation. Then in S2the context of the mobile device is determined. In this exemplary embodiment the context is an active connection to at least one other device. The user in this exemplary embodiment passes the mobile device to another user. The mobile device also determines its movement. Then, in S3, based on the context of the mobile device and the movement of the mobile device, the orientation of visual content rendered on the display of the mobile device is determined and the visual content is rendered accordingly.

FIG.6illustrates an apparatus600, which may be a computing apparatus, according to an example embodiment. The apparatus600may be, or may be comprised in, a mobile device such as those described in the exemplary embodiments above. The apparatus600comprises a processor610. The processor610interprets computer program instructions and process data. The processor610may comprise one or more programmable processors. The processor610may comprise programmable hardware with embedded firmware and may, alternatively or additionally, comprise one or more application specific integrated circuits, ASICs.

The processor610is coupled to a memory620. The processor is configured to read and write data to and from the memory620. The memory620may comprise one or more memory units. The memory units may be volatile or non-volatile. It is to be noted that in some example embodiments there may be one or more units of non-volatile memory and one or more units of volatile memory or, alternatively, one or more units of non-volatile memory, or, alternatively, one or more units of volatile memory. Volatile memory may be for example RAM, DRAM or SDRAM. Non-volatile memory may be for example ROM, PROM, EEPROM, flash memory, optical storage or magnetic storage. In general, memories may be referred to as non-transitory computer readable media. The memory620stores computer readable instructions that are execute by the processor610. For example, non-volatile memory stores the computer readable instructions and the processor610executes the instructions using volatile memory for temporary storage of data and/or instructions.

The computer readable instructions may have been pre-stored to the memory620or, alternatively or additionally, they may be received, by the apparatus, via electromagnetic carrier signal and/or may be copied from a physical entity such as computer program product. Execution of the computer readable instructions causes the apparatus600to perform functionality described above.

In the context of this document, a “memory” or “computer-readable media” may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.

The apparatus600further comprises, or is connected to, an input unit630. The input unit630comprises one or more interfaces for receiving a user input. The one or more interfaces may comprise for example one or more motion and/or orientation sensors, one or more cameras, one or more accelerometers, one or more microphones, one or more buttons and one or more touch detection units. Further, the input unit630may comprise an interface to which external devices may connect to.

The apparatus600also comprises an output unit640. The output unit comprises or is connected to one or more displays capable of rendering visual content such as a light emitting diode (LED) display, a liquid crystal display (LCD) and/or a liquid crystal on silicon (LCoS) display. The output unit640may comprise two displays to render stereoscopic visual content. One display to render content to the left eye and the other display to render content to the right eye. The output unit640may further comprise a transmission unit, such as one or more waveguides or one or more lenses, to transfer the rendered visual content to the user's field of view. The output unit640may further comprise one or more audio outputs. The one or more audio outputs may be for example loudspeakers or a set of headphones.

The apparatus600may further comprise a connectivity unit650. The connectivity unit650enables wired and/or wireless connectivity to external networks. The connectivity unit650may comprise one or more antennas and one or more receivers that may be integrated to the apparatus600or the apparatus600may be connected to. The connectivity unit650may comprise an integrated circuit or a set of integrated circuits that provide the wireless communication capability for the apparatus600. Alternatively, the wireless connectivity may be a hardwired application specific integrated circuit (ASIC).

It is to be noted that the apparatus600may further comprise various components not illustrated in theFIG.6. The various components may be hardware component and/or software components.

Example embodiments described herein may be implemented using software, hardware, application logic or a combination of them. Also, if desired, different functionalities discussed herein may be performed in a different order, some functionalities may be performed concurrently, and, if desired, some of the above-mentioned functionalities may be combined. Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or dependent claims with features of the independent claims and not solely the combinations explicitly set out in the claims.

It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.

Moreover, the disclosure of the present application should be understood to include any novel features or any novel combination of features either explicitly or implicitly disclosed herein or any generalization thereof and during the prosecution of the present application or of any application derived therefrom, new claims may be formulated to cover any such features and/or combination of such features.