Patent Description:
When providing video feed, that may be provided in real-time, situations change. If the video feed is then transmitted to other users, it may be that the other users are disturbed if a situation changes or they may be paying attention to unwanted aspects of the video feed. It may be beneficial therefore to help the recipients of the video feed to pay attention to aspects of the video that best convey the message that is intended to be conveyed.

The scope of protection sought for various embodiments is set out by the independent claims. The exemplary embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

As used herein, the term "determining" (and grammatical variants thereof) can include, not least: calculating, computing, processing, deriving, measuring, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), obtaining and the like. Also, "determining" can include resolving, selecting, choosing, establishing, and the like.

For people to meet they do not need to be physically in the same location. With the help of video feeds obtained from one or more cameras, video calls and video meetings taking place online people may have meetings in which they see each other's reactions and presence even though they are physically not in the same location. <FIG> illustrates an exemplary system that enables a video feed to be transmitted such that an online video meeting with multiple participants or a video call to another participant may take place. A device <NUM>, that may be a computing device such as a laptop computer, a mobile phone or a tablet computer, is capable of connecting to an online meeting that may also be understood as a video call. The device <NUM> further comprises, or is connected to, at least one display on which the meeting is visually rendered. The visual rendering may comprise one or more other participants and optionally also a video feed of the user <NUM> that uses the device <NUM> to connect to the online meeting. The video feed is enabled by one or more cameras <NUM> that may be comprised in the device <NUM> or the device <NUM> may be connected to those. The device <NUM> further comprises or is connected to at least one microphone (not shown) that is used to capture audio from the user <NUM> to the video feed and the device <NUM> is also connected to, or it comprises, one or more loudspeakers (not shown) that render the audio from others in the online meeting.

The one or more cameras <NUM> may comprise various types of cameras. For example, a regular camera such as a webcam and/or a digital single lens reflex, DSLR or mirrorless system camera. DSLRs and mirrorless cameras may have a benefit of better image quality due to larger high-resolution image sensors and the use of variety of different lenses compared to for example a camera comprised in the device <NUM>.

If an online meeting comprises video feed from one of more participants, a participant may find it difficult to get attention from other participants when trying to speak due to for example latency in audio, which may cause multiple participants ending up speaking simultaneously. Latency may be caused for example by a variety of different techniques and devices used to connect to the online video meeting. Yet, if participants are provided with a visual indication that a participant is about to speak, that may help to avoid problems such as participants speaking on top of each other.

It may also happen that when a participant is attending an online video meeting and transmitting a video feed, there may situations in which the participant is to speak such that the participants in the online video meeting do not hear him speak. If for example the participant needs to talk to someone in a same room, then the participant may not wish to unmute his microphone. Yet, others may recognize from the video feed that the participant speaks and may think that he just forgot to unmute his microphone. Thus, a visual indication regarding that the microphone is muted on purpose could be helpful. Also, if another person accidentally moves to the field of view of the camera providing the video feed, it may be awkward and it might be useful to have the video feed edited such that the other person becomes less visible for example.

<FIG> illustrates an exemplary embodiment of modifying a video feed transmitted to an online video meeting when a user unmutes the microphone that is used to capture audio to the video feed. In this exemplary embodiment a user <NUM> is captured to a video feed <NUM> that is then transmitted to the online video meeting. The system used to join the online video meeting and to transmit the video feed to the online video meeting may be any suitable system, such as the system described in the exemplary embodiment of <FIG>. In this exemplary embodiment the video feed <NUM> captures the real physical environment in which the user <NUM> is physically located. At first the user <NUM> has muted the microphone. As the user <NUM> wishes to say something to the other participants in the online video meeting, the user unmutes the microphone. In other words, the status of the video feed transmitted to the online video meeting changes as the microphone is unmuted as is illustrated in <NUM>. The status of the online video meeting may thus be understood as the status of the microphone in this exemplary embodiment. It is to be noted that the status of the video feed may also be understood as number of people recognized in the video feed in some other exemplary embodiments. As the status of the video feed changes, that is the status of the microphone changes from muted to unmuted, due to various reasons there may be latency in the other participants actually hearing the user <NUM> speak, which may lead to a situation in which the other participants hardly notice the user <NUM> is trying to speak.

However, as it is determined that the status of the video feed changes, the video feed is visually modified. For example, focus and/or cropping may be applied to the video feed. In this exemplary embodiment, the focus in the video feed is now on the user <NUM> in the visually modified video feed <NUM> that is then transmitted to the online video meeting. According to the invention, the environment around the user is blurred, that is, reducing focus from the background. Further, additionally, the video feed may be cropped such that the part of the environment of the user is no longer part of the video feed. Further alternatively or additionally, the aspect ratio between the user <NUM> and the background in the video feed may be changed to help the other participants to focus on the user who is about to speak. These visual modifications may have the benefit of naturally drawing the attention of the other participants to the user <NUM> who is about to speak although due to latency his voice is not yet heard, which may offer a better user experience making it easier to follow the upcoming speaker. The other participants may, in some exemplary embodiments, see this change in the transmitted video feed already before the user starts to speak. Furthermore, the blurring the background may help to draw full attention to the user <NUM> who is about to speak. The visual modifications to the video feed may be achieved using one or more suitable software algorithms designed for image processing and/or parameters regarding the one or more cameras and/or lenses capturing the video feed may be modified. For example, lenses may offer, e.g., bokeh of the area that are not in focus. Further, settings such as aperture and focus point may be modified.

In some exemplary embodiments the video feed may be manipulated by performing a slower zoom-in towards the user to prolong the transition. In such exemplary embodiment, the user may not wish to be focused to while unmuting. Thus, while unmuting, the user may provide a user input, received by the device of the system, that is interpreted to trigger a zooming mode, which in this exemplary embodiment is a slower zoom-in than a default zoom-in. The user input may be any suitable type, for example double pressing the unmute button.

In the exemplary embodiment of <FIG> the user had a real-life background. The modifications to the video feed discussed in that exemplary embodiment are also applicable to artificial backgrounds used in a video feed transmitted to an online video meeting. <FIG> illustrates and exemplary embodiment in which the background of the video feed <NUM> comprises an artificial background. The artificial background may be for example such that the user is perceived to be in front of a landscape that may be obtained from a picture.

In the exemplary embodiment of <FIG>, a user <NUM> participates in an online video meeting by using any suitable system, such as the system described in the exemplary embodiment of <FIG>. The user <NUM> is, in this exemplary embodiment, first muted. In other words, the status of the video feed <NUM> transmitted to the online video meeting is the status of the microphone that provides audio to the video feed. It is to be noted that in some exemplary embodiments, there may be more than one microphones providing audio to the video feed. In this exemplary embodiment, the video feed comprises the user <NUM> rendered in front of an artificial background. As the user decides to speak, the user changes the status of the video feed <NUM> by unmuting the microphone as illustrated in <NUM>. After the status of the video feed has been changed, a modified video feed <NUM> is transmitted to the online video meeting. In the modified video feed, that background is blurred and the user <NUM> is in focus. Additionally, also cropping of the image captured to the modified video feed <NUM> may be applied. Parameters regarding one or more cameras used to capture the video feed transmitted to the online video meeting and/or image processing parameters, may be changed such that the modified video feed <NUM> is obtained. For example, a camera parameter change may be synthesized, or algorithm settings may be applied to the artificial background image to create similar experience as with a live background like in the previous exemplary embodiment.

In the exemplary embodiments of <FIG> and <FIG>, it is to be noted that if the status of the modified video feed changes, for example the status of the microphone changes back to the muted status, then the modifications applied to the modified video feed may no longer be applied.

<FIG> illustrates an exemplary embodiment in which the status of the video feed changes. In this exemplary embodiment, the user <NUM> participates in an online video meeting using a system as described above. The user <NUM> is rendered in a video feed <NUM> and the status of a microphone providing the audio to the video feed <NUM> is muted as illustrated in <NUM>. In this exemplary embodiment, the status of the microphone does not change. Yet, the status of the video feed changes. The status of the video feed in this exemplary embodiment comprises the status of the user <NUM>. In this exemplary embodiment, the status of the user <NUM> changes as the user speaks or stays quiet. It may be recognized, by suitable software algorithms, that the user speaks, but that the status of the microphone is muted. If the other participants of the online video meeting can see the user <NUM> speaking, but do not hear anything, that may cause a confusing situation and disturb the meeting while the user is intentionally keeping the microphone muted as he talks to someone else who is not a participant in the online video meeting. The user could also turn of the video, which however may also be an awkward solution in terms of user experience.

As the status of the video feed changes in this exemplary embodiment, the video feed is modified. In this exemplary embodiment, the modified video feed <NUM> is transmitted to the online video meeting. The modified video feed <NUM> comprises modification to the video feed that focuses away from the user <NUM>. This way the other participants do not see the user clearly and thus do not see him speaking, at least not clearly. Additionally, in the modified video feed <NUM> the focus may be shifted to the background thereby drawing attention there. In some exemplary embodiments, the video feed may be obtained using more than one camera. For example, with two cameras, a view may be selected where the user is not seen speaking and selection of a view in which the user is not seen speaking may be used as the modification applied to the modified video feed. In this manner, the attention of other participants is not drawn to the user as he is speaking while the microphone is muted, and the other participants do not expect to hear the voice either.

The modifications applied to the video feed in the exemplary embodiment of <FIG> may also be applied to artificial backgrounds. In <FIG> an exemplary embodiment in which the status of the video feed changes is illustrated. In this exemplary embodiment, the user <NUM> participates in an online video meeting using a system as described above. The user <NUM> is rendered in a video feed <NUM> in which the background is an artificial background instead of the physical environment in which the user <NUM> is located at. The status of a microphone providing the audio to the video feed <NUM> is muted as illustrated in <NUM>. In this exemplary embodiment, the status of the microphone does not change although the status of the video feed changes. The status of the video feed in this exemplary embodiment comprises the status of the user <NUM>, which in this exemplary embodiment changes as the user speaks or becomes quiet after speaking. As the status of the video feed changes in this exemplary embodiment, the video feed is modified. In this exemplary embodiment, the modified video feed <NUM> is transmitted to the online video meeting. The modified video feed <NUM> comprises modification to the video feed that focuses away from the user <NUM>. Additionally, in the modified video feed <NUM> the focus may be shifted to the background thereby drawing attention there. In some exemplary embodiments, as the status of the video feed changes, the artificial background may algo change to for example another image while the user is out of focus.

During a video call, it may also happen that a user is in a video call and another person accidentally enters the background of the video feed. In this case the user is not muted and not having an outside call discussion as in in the previous exemplary embodiments. In such a case, the change in the amount of people recognized, by suitable software algorithms, from the video feed may be understood as the status of the video feed. Thus, if the amount of people recognized in the video feed changes, so does the status of the video feed. This is illustrated in the exemplary embodiment illustrated in <FIG>. In this exemplary embodiment, a user <NUM> is participating in a video call using a system such as the one described above. The video feed <NUM> transmitted to the online video meeting has the room the user is in as a background. In this exemplary embodiment it is recognized that another person <NUM> enters the captured video feed. As the number of people recognized in the captured video feed changes to two, it is determined that the status of the video feed changes as well.

Consequently, it is determined that the video feed is to be modified.

In this exemplary embodiment, in the modified video feed <NUM> the background is blurred causing the other person <NUM> to be blurred in the modified video feed <NUM> that is then transmitted to the online video meeting. This has the advantage of maintaining the attention to the user <NUM>. It also allows someone to enter the same space without being part of the video call accidentally. In some exemplary embodiments, there may be an indication when the other person <NUM> is blurred and the user <NUM> may be allowed to override the action of blurring the background if the appearance of the other person <NUM> was intentional.

Additionally, in some exemplary embodiments, settings of a camera aperture and/or artificial blurring may be dependent on the amount of activity in the background detected from the video feed while the user <NUM> is in the online video meeting.

<FIG> illustrates another exemplary embodiment in which the status of the video feed transmitted to an online video meeting is determined to change as a status of a microphone changes. In this exemplary embodiment, the modification of the video feed comprises applying a so-called Dolly-effect. The Dolly-effect may be understood as modifying the video feed such that the head of the user is maintained at constant size, but field of view and a camera distance are modified to create a noticeable effect.

In this exemplary embodiment, the user <NUM> is participating in an online video meeting. To attend the online video meeting, the user <NUM> uses a system that comprises a camera <NUM> in addition to a computer and a microphone. In this exemplary embodiment, the position of the camera and zoom level may be modified. That is, the angle of view as well as the field of view <NUM> of the camera <NUM> may be modified. As the user is muted, the video feed <NUM> is transmitted to the online video meeting. The user then unmutes the microphone and consequently, it is determined that the status of the video feed is changes as the status of the microphone is changed. Based on the change in the status of the video feed, it is determined that the video feed is to be modified. In this exemplary embodiment, the Dolly-effect is applied. The face of the user <NUM> remains at constant size, but in the modified video feed <NUM> transmitted to the online video meeting the elements in the background appear to move away from the user <NUM>. In this exemplary embodiment, the system changes simultaneously both the position of the camera and the level of zoom such that the head of the user <NUM> remains at constant size but the background of the user <NUM> changes in a noticeable manner. As the elements in the background consequently appear to move and change their size in ways that is counterintuitive for the human visual system, the other participants will notice a change in the modified video feed <NUM> that naturally draws their attention to the user <NUM>. Additionally, digital modification may be applied to the modified video feed <NUM> in combination with in-camera cropping and change of the field of view <NUM> to achieve this effect. The change of the field of view may be achieved for example by using electronically controlled lens zoom. Thus, the background effect may be done by adapting the virtual background or a real dolly zoom may also be performed by using for example rails to move the camera of a drone with a camera.

<FIG> illustrates yet another exemplary embodiment in which the status of the video feed is determined to change as a user <NUM> unmutes the microphone and a modified video feed is to be transmitted to an online video meeting the user is participating in. In this exemplary embodiment, based on the determined change in the status of the video feed, the video feed is modified by changing a camera that is used to capture the video feed.

In this exemplary embodiment, the system the user <NUM> uses to connect to the online video meeting comprises a computer with a built-in camera as well as an external camera <NUM>. The external camera <NUM> in this exemplary embodiment provides video feed from different angle than the built-in camera of the computer. In this exemplary embodiment, the user <NUM> is talking and therefore the built-in camera of the computer is used to provide the video feed <NUM> to the online video meeting. As the user then <NUM> mutes the microphone, in other words, it is determined that the status of the microphone changes and therefore it is determined that the status of the video feed also changes. Based on the change in the status of the video feed, it is determined that the video feed is to be modified. In this exemplary embodiment, the modified video feed <NUM> is modified by capturing the video feed using the external camera <NUM> instead of the built-in camera. In other words, the modification comprises changing the camera from which the video feed is captured.

The switching between the cameras, or alternatively, switching between viewpoints, provides an effect in which the user <NUM> appears more prominent or further away than previously in the video feed. Thus, when the built-in camera is used to provide the video feed, the user <NUM> the user may appear as more prominent than previously due to a change in the viewpoint caused by switching the camera and other participants may concentrate on the user's facial expressions etc. better. Then, as the user <NUM> is muted, the user <NUM> the angle from which the user <NUM> is captured may change thereby resulting the user appearing to be further away in the video feed. Thus, in the modified video feed <NUM> the user may appear as less distracting to other participants but still visible to them. This may allow for the user <NUM> also to move around in his environment, while other participants see that the user <NUM> is still present and able to participate if needed.

The exemplary embodiment described above may have benefits such as a user wanting to speak will get attention more easily in an online video meeting. Visual modification drives and supports the people to people interaction and participants of the online video meeting can more easily follow a lively discussion. Also, the focus of the participant may be drawn to what matters the most at the given time.

<FIG> illustrates an apparatus <NUM>, which may be a computing apparatus, according to an example embodiment. The apparatus <NUM> may be comprised in a device such as a device used to connect to an online video meeting. The apparatus <NUM> comprises a processor <NUM>. The processor <NUM> interprets computer program instructions and process data. The processor <NUM> may comprise one or more programmable processors. The processor <NUM> may comprise programmable hardware with embedded firmware and may, alternatively or additionally, comprise one or more application specific integrated circuits, ASICs.

The processor <NUM> is coupled to a memory <NUM>. The processor is configured to read and write data to and from the memory <NUM>. The memory <NUM> may comprise one or more memory units. The memory units may be volatile or non-volatile. It is to be noted that in some example embodiments there may be one or more units of non-volatile memory and one or more units of volatile memory or, alternatively, one or more units of non-volatile memory, or, alternatively, one or more units of volatile memory. Volatile memory may be for example RAM, DRAM or SDRAM. Non-volatile memory may be for example ROM, PROM, EEPROM, flash memory, optical storage or magnetic storage. In general, memories may be referred to as non-transitory computer readable media. The memory <NUM> stores computer readable instructions that are execute by the processor <NUM>. For example, non-volatile memory stores the computer readable instructions and the processor <NUM> executes the instructions using volatile memory for temporary storage of data and/or instructions.

The computer readable instructions may have been pre-stored to the memory <NUM> or, alternatively or additionally, they may be received, by the apparatus, via electromagnetic carrier signal and/or may be copied from a physical entity such as computer program product. Execution of the computer readable instructions causes the apparatus <NUM> to perform functionality described above.

The apparatus <NUM> further comprises, or is connected to, an input unit <NUM>. The input unit <NUM> comprises one or more interfaces for receiving a user input. The one or more interfaces may comprise for example one or more motion and/or orientation sensors, one or more cameras, one or more accelerometers, one or more microphones, one or more buttons and one or more touch detection units. Further, the input unit <NUM> may comprise an interface to which external devices may connect to.

The apparatus <NUM> also comprises an output unit <NUM>. The output unit comprises or is connected to one or more displays capable of rendering visual content such as a light emitting diode (LED) display, a liquid crystal display (LCD) and a liquid crystal on silicon (LCoS) display. The output unit <NUM> may comprise two displays to render stereoscopic visual content. One display to render content to the left eye and the other display to render content to the right eye. The output unit <NUM> may further comprise a transmission unit, such as one or more waveguides or one or more lenses, to transfer the rendered visual content to the user's field of view. The output unit <NUM> further comprises one or more audio outputs. The one or more audio outputs may be for example loudspeakers or a set of headphones.

The apparatus <NUM> may further comprise a connectivity unit <NUM>. The connectivity unit <NUM> enables wired and/or wireless connectivity to external networks. The connectivity unit <NUM> may comprise one or more antennas and one or more receivers that may be integrated to the apparatus <NUM> or the apparatus <NUM> may be connected to. The connectivity unit <NUM> may comprise an integrated circuit or a set of integrated circuits that provide the wireless communication capability for the apparatus <NUM>. Alternatively, the wireless connectivity may be a hardwired application specific integrated circuit, ASIC.

It is to be noted that the apparatus <NUM> may further comprise various component not illustrated in the <FIG>. The various components may be hardware component and/or software components.

<FIG> illustrates a flow chart according to an exemplary embodiment. In this exemplary embodiment a video feed is obtained as illustrated in S1. The video feed is obtained from a camera which may be a camera comprised in a device that may be a computing device such as a laptop, tablet computer or a mobile phone. Alternatively, the camera may be an external camera such as a mirrorless camera, a DSLR or a webcam. The video feed may be obtained by any suitable apparatus such as the apparatus described in <FIG>. It is also to be noted that a computing apparatus may also be understood as a computing device.

Next, in S2, the obtained video feed is transmitted to an online video meeting. For example, the apparatus that obtains the video feed may also be used to connect to the online video meeting. In S3, it is determined that there is a change of status of the video feed. The change of status may be determined for example based on recognizing a change in a status of a microphone used to capture audio for the video feed, recognizing a change of a status of a user or a detecting a change in a number of people present in the obtained video feed. The status of the microphone may be muted or unmuted. The status of the user may be speaking or silent for example.

In S4, based on the determined change of status, the video feed is modified. The modified video feed is thus modified in view of the video feed transmitted to the online video meeting prior to the determined change of status of the video feed. The modification may be a visual modification such as those described in the exemplary embodiments above. The modified video feed is then transmitted to the online video meeting as illustrated in S5.

Example embodiments described herein may be implemented using software, hardware, application logic or a combination of them. Also, if desired, different functionalities discussed herein may be performed in a different order, some functionalities may be performed concurrently, and, if desired, some of the above-mentioned functionalities may be combined.

Claim 1:
An apparatus comprising means for:
obtaining, from a camera, a video feed capturing a user;
transmitting the video feed to an online video meeting;
determining a change of status of the video feed, wherein determining the change of status comprises determining that the status of a microphone that is used to capture audio to the video feed, has changed from muted to unmuted;
based on the determined change of status, modifying the video feed, wherein modifying the video feed comprises blurring a background of the user; and
transmitting the modified video feed to the online video meeting.