Patent ID: 12254634

DETAILED DESCRIPTION

In the following description, a panoramic video conference system and method according to the present disclosure will be explained with reference to embodiments thereof. However, these embodiments are not intended to limit the present disclosure to any environment, applications, or implementations described in these embodiments. Therefore, description of these embodiments is only for purpose of illustration rather than to limit the present disclosure. It shall be appreciated that, in the following embodiments and the attached drawings, elements unrelated to the present disclosure are omitted from depiction. In addition, dimensions of individual elements and dimensional relationships among individual elements in the attached drawings are provided only for illustration but not to limit the scope of the present disclosure.

First, the application scenario of the present embodiment will be explained, and a schematic view is depicted inFIG.1. As shown inFIG.1, the panoramic video conference system1at least comprises a panoramic video generating apparatus3and a control apparatus2. In this scenario, the control apparatus2may be connected to the panoramic video generating apparatus3through a wired or wireless ways (e.g., a wireless communication network). The panoramic video generating apparatus3is used to continuously (e.g.,30frames per second) generate the panoramic video301corresponding to the conference. For example, the panoramic video generating apparatus3can be installed on a table in a conference room and generates the video comprising all participants in the conference.

In some embodiments, the panoramic video conference system1further comprises a microphone array4and a display5. The microphone array4and the display5can be connected to the control apparatus2by wired or wireless ways (e.g., wireless communication network).

It shall be appreciated that the panoramic video generating apparatus3is used to generate a 360-degree panoramic video301. The panoramic video generating apparatus3may include at least two cameras, and the panoramic video301may include a plurality of images (i.e., multiple frames). For example, the panoramic video generating apparatus3may be composed of three cameras with a field of view (FOV) of 120 degrees, so the video of the cameras can be combined to generate a 360-degree surrounded panoramic video301.

In the present disclosure, the panoramic video conference system1is a local side conference system, the panoramic video conference system1is connected with other remote conference systems (not shown). The panoramic video conference system1transmits the panoramic conference video to the remote conference system for online conference. The following paragraphs will focus on various functions of the local side conference system (i.e., the panoramic video conference system1). Those of ordinary skill in the art shall appreciate the implementation details of the connection between different conference systems. Therefore, the details will not be repeated herein.

In the present disclosure, in order to improve the quality of the online conference, the panoramic video conference system1mainly provides at least three types of functions, which are the privacy object hiding function, the speech source focusing function and the loop playback function. The following paragraphs will describe the details related to the present disclosure.

The schematic view of the structure of the control apparatus2in the first embodiment of the present disclosure is depicted inFIG.2. The control apparatus2comprises a storage21, a transceiver interface23and a processor25. The processor25is electrically connected to the storage21and the transceiver interface23. The storage21may be a memory, a Universal Serial Bus (USB) disk, a hard disk, a Compact Disk (CD), a mobile disk, or any other storage medium or circuit known to those of ordinary skill in the art and having the same functionality. The transceiver interface23is an interface capable of receiving and transmitting data or other interfaces capable of receiving and transmitting data and known to those of ordinary skill in the art. The transceiver interface23can receive data from sources such as external apparatuses, external web pages, external applications, and so on. The processor25may be any of various processors, Central Processing Units (CPUs), microprocessors, digital signal processors or other computing apparatuses known to those of ordinary skill in the art.

Next, the operations of the first embodiment of the present disclosure will be described below. Please refer toFIG.1andFIG.2. The first embodiment of the present disclosure relates to the privacy object hiding function provided by the panoramic video conference system1. It shall be appreciated that during the conference, some meeting participants, sensitive objects, and sundries in the conference room need to be hidden based on privacy rights or conference requirements. Therefore, in the present embodiment, the control apparatus2may identify objects shown in the panoramic video301and removes some of the objects from the panoramic video301(e.g., making the objects transparent).

Specifically, the control apparatus2analyzes the panoramic video301to identify a plurality of video objects in the panoramic video301. When the user selects a privacy mode through the control apparatus2, the control apparatus2may select a video object to be removed (i.e., to-be-removed video object) from the video objects accordingly. In some embodiments, the control apparatus2may also select a plurality of video objects to be removed from the video objects.

It shall be appreciated that the privacy mode may include a list of candidates of video objects to be removed, the list indicates video objects that need to be hidden. For example, the list can be selected from a plurality of preset templates, or generated by the user's dynamic selection after the control apparatus2identifies a plurality of video objects in the panoramic video301(e.g., manually checking conference participants and sensitive objects that need to be hidden).

In some embodiments, the control apparatus2further provides with an video identification model, and the control apparatus2identifies the video objects in the panoramic video301based on an edge detection algorithm and the video identification model. For example, the control apparatus2performs edge detection on the panoramic video301by using an edge detection algorithm to find out the objects in the video (i.e., the panoramic video301). Then, through the video identification model, the control apparatus2classifies the video data in this area and performs the feature comparison to identify the position data and identity data of objects and people in the video.

It shall be appreciated that the video identification model can be used to identify the video objects in the panoramic video301, and the video identification model can be implemented by a training process based on a plurality of object features and a plurality of conference member features through training of a neural network architecture. Those of ordinary skill in the art shall appreciate the corresponding operations based on above descriptions. Therefore, the details will not be repeated herein.

Next, in the present embodiment, the control apparatus2removes an video corresponding to the video object. Finally, the control apparatus2generates a panoramic conference video based on the panoramic video and a background filling video.

In some embodiments, the control apparatus2first calculates a plurality of color feature values of a plurality of adjacent blocks of the video object to be removed, and then the control apparatus2generates the background filling video based on the color feature values. For example, the control apparatus2can separate the foreground object and the background through a nearest neighbor algorithm (e.g., a k-nearest neighbors (KNN) algorithm), calculate the color feature value distributions of the removed blank area, and generate a background filling video to fill the blank area. In some embodiments, the control apparatus2may also perform a second area filling on the blank area based on a Sub-patch Texture Synthesis Algorithm, so as to make the fused video more natural.

For ease of understanding, taking an actual example as an example, please refer toFIG.3AandFIG.3Bat the same time.FIG.3Aillustrates an video303with a plurality of object features. After the control apparatus2analyzes the video303, the control apparatus2identifies that the video303comprises the video object OB1and the video object OB2. In the present example, the video object OB1and the video object OB2are both the video objects to be removed, the control apparatus2removes the video of the video object OB1and the video object OB2in the video303(i.e., the area framed by the video object object OB1and the video object OB2). As shown inFIG.3B, the control apparatus2generates the video305through the background filling video generated by the aforementioned method.

It shall be appreciated that, for the convenience of illustration, the video303and the video305are only one of the frames in the video. In the actual operations, the control apparatus2continuously and dynamically determines one or more frames in the video (e.g., based on 10 frames at the same time), and the operations can be adjusted according to the actual operation requirements of the control apparatus2. Those of ordinary skill in the art shall appreciate the implementation for multiple frames based on the above descriptions. Therefore, the details will not be repeated herein.

Next, the speech source focusing function provided by the panoramic video conference system1of the present disclosure will be described. During a conference, when a conference participant speaks, the panoramic video conference system1can focus the video on the speaker to improve the quality of the conference (e.g., generate a video focusing on the speaker).

Specifically, the panoramic video conference system1further comprises a microphone array4. The microphone array is connected to to the control apparatus2through a network connection. The microphone array4is configured to sense a sound of a conference room to generate a sound signal. The control apparatus2first receives the sound signal from the microphone array4. Next, the control apparatus2determines a speaking source (e.g., the speaker in the conference) based on the sound signal.

Subsequently, the control apparatus2compares a plurality of conference member features with the panoramic video corresponding to the speech source to determine whether the panoramic video corresponding to the speech source has a first conference member feature that matches the conference member features. The control apparatus2generats a focusing video corresponding to the speech source when it is determined that the panoramic video corresponding to the speech source has the first conference member feature. Finally, the control apparatus2generates the panoramic conference video based on the focusing video and the panoramic video (e.g., generates an upper divided video and a lower divided video, the upper divided video is the panoramic video, and the lower divided video is the focusing video).

For example, the control apparatus2may determine the source of the speech based on a direction of arrival (DOA) and beamforming technologies (i.e., a direction of arrival algorithm and a beamforming algorithm). It shall be appreciated that the control apparatus2determines whether the speech source is indeed made by the person participating in the conference by comparing whether the speech source matches one of the conference member features. Therefore, the operations can avoid misjudgment caused by noise in the environment or irrelevant actions, and improve the quality of the conference.

In addition, when the conference is in progress, the conference room is usually provided with a display for playing the remote video and sound. In order to avoid the situation that the focusing video is focused on the display due to the remote conference sound played on the display. Therefore, in some embodiments, the control apparatus2may limit the use of the “the speech source focusing function” to avoid misjudgment (i.e., the sound emitted by the display5must be actively excluded). Specifically, the panoramic video conference system1further comprises a display5, and the display5is configured to play a remote conference video and a remote conference sound. The control apparatus2generates an exclusion area based on a position of the display5. The control apparatus2compares the speech source with the exclusion area to determine whether the speech source comes from the exclusion area, and the focusing video corresponding to the speech source is not generated when the speech source comes from the exclusion area.

It shall be appreciated that the control apparatus2can confirm the position of the display5through the aforementioned object identification. In addition, in some embodiments, when the position of the display5moves, the control apparatus2can also dynamically adjust the exclusion area to avoid the misjudgment.

Next, the loop playback function provided by the panoramic video conference system1of the present disclosure will be described. The following paragraphs will describe the details related to the present disclosure. Before the conference starts or after the conference ends, the panoramic video conference system1can generate a close-up video of the person participating in the conference when the video detects the presence of the person participating in the conference, and plays the loop video to introduce the current attendees in the conference (e.g., the loop playback mode).

Specifically, the panoramic video generating apparatus3of the the panoramic video conference system1may further comprise a first camera and a second camera, the first camera is configured to generate a first video, the second camera is configured to generate a second video, and the panoramic video301is composed of the first video and the second video.

In the present embodiment, the control apparatus2can detect whether a first conference member with a first conference member feature appears in the first video, and generate a close-up video of the first conference member as the first video when the first conference member with the first conference member feature appears in the first video.

Furthermore, the control apparatus2can detect whether a second conference member with a second conference member feature appears in the first video, and generate a close-up video of the second conference member as the second video when the second conference member with the second conference member feature appears in the second video.

Therefore, before the conference starts or after the conference ends, the panoramic video conference system1can play the panoramic video301composed of the first video and the second video in a loop. It shall be appreciated that the present disclosure does not limit the number of cameras and video. Those of ordinary skill in the art shall appreciate the implementations with different numbers of cameras and video (i.e., greater than2) based on the above descriptions. Therefore, the details will not be repeated herein.

In some embodiments, the panoramic video conference system1may further comprise a cloud server. The panoramic video conference system1may transmit the conference data (e.g., the panoramic conference video and the related information) to the cloud server, and store the conference data in the cloud server as the historical conference records for the relevant member of the conference to query.

According to the above descriptions, the panoramic video conference system1provided by the present disclosure identify a plurality of video objects in the panoramic video by analyzing the panoramic video. In addition, the panoramic video conference system1provided by the present disclosure can select a to-be-removed video object from the video objects to remove from the panoramic video based on a privacy mode. Finally, the panoramic video conference system1provided by the present disclosure generates a panoramic conference video based on the panoramic video and a background filling video. The panoramic video conference system1provided by the present disclosure can selectively hide some video objects through the above-mentioned mechanism, so as to provide enterprises with more flexible privacy requirements. In addition, the panoramic video conference system1provided by the present disclosure provides a plurality of functions, and improves the service and quality of the online conference.

A second embodiment of the present disclosure is a panoramic video conference method and a flowchart thereof is depicted inFIG.4. The panoramic video conference method400is adapted for use in a panoramic video conference system (e.g., the panoramic video conference system1of the first embodiment). The panoramic video conference system comprises a panoramic video generating apparatus and a control apparatus (e.g., the panoramic video generating apparatus3and the control apparatus2of the first embodiment). The panoramic video conference method400is performed by the control apparatus. The panoramic video conference method400generates a panoramic conference video through the steps S401to S407.

In the step S401, the control apparatus analyzes a panoramic video received from the panoramic video generating apparatus to identify a plurality of video objects in the panoramic video. In the step S403, the control apparatus selects a video object to be removed from the video objects based on a privacy mode.

Next, in the step S405, the control apparatus removes a video corresponding to the video object. Finally, in the step S407, the control apparatus generates a panoramic conference video based on the panoramic video and a background filling video.

In some embodiments, the control apparatus further provides with an video identification model, and the panoramic video conference method400further comprises following steps: identifying the video objects in the panoramic video based on an edge detection algorithm and the video identification model, wherein the video identification model is implemented by a training process based on a plurality of object features and a plurality of conference member features.

In some embodiments, the the panoramic video conference system further comprises a microphone array (e.g., the microphone array4of the first embodiment). The microphone array is connected to the control apparatus through a network connection, and the microphone array is configured to sense a sound of a conference room to generate a sound signal. The panoramic video conference method400further comprises following steps: receiving the sound signal from the microphone array; determining a speech source based on the sound signal; comparing a plurality of conference member features with the panoramic video corresponding to the speech source to determine whether the panoramic video corresponding to the speech source has a first conference member feature that matches the conference member features; generating a focusing video corresponding to the speech source when it is determined that the panoramic video corresponding to the speech source has the first conference member feature; and generating the panoramic conference video based on the focusing video and the panoramic video.

In some embodiments, the panoramic video conference system further comprises a display (e.g., the display5of the first embodiment). The display5is configured to play a remote conference video and a remote conference sound. The panoramic video conference method400further comprises following steps: generating an exclusion area based on a position of the display; comparing the speech source with the exclusion area to determine whether the speech source comes from the exclusion area; and wherein the focusing video corresponding to the speech source is not generated when the speech source comes from the exclusion area.

In some embodiments, the the panoramic video conference system further comprises a first camera and a second camera. The first camera is configured to generate a first video, the second camera is configured to generate a second video, and the panoramic video is composed of the first video and the second video.

In some embodiments, the panoramic video conference method400further comprises following steps: detecting whether a first conference member with a first conference member feature appears in the first video; and generating a close-up video of the first conference member as the first video when the first conference member with the first conference member feature appears in the first video.

In some embodiments, the the panoramic video conference method400further comprises following steps: detecting whether a second conference member with a second conference member feature appears in the first video; and generating the close-up video of the second conference member as the second video when the second conference member with the second conference member feature appears in the second video.

In some embodiments, the the panoramic video conference method400further comprises following steps: calculating a plurality of color feature values of a plurality of adjacent blocks of the video object to be removed; and generating the background filling video based on the color feature values.

In addition to the aforesaid steps, the second embodiment can also execute all the operations and steps of the panoramic video conference system1set forth in the first embodiment, have the same functions, and deliver the same technical effects as the first embodiment. How the second embodiment executes these operations and steps, has the same functions, and delivers the same technical effects will be readily appreciated by those of ordinary skill in the art based on the explanation of the first embodiment. Therefore, the details will not be repeated herein.

It shall be appreciated that in the specification and the claims of the present disclosure, some words (e.g., the video, the conference member, the conference member feature, and the camera) are preceded by terms such as “first” or “second,” and these terms of “first” and “second” are only used to distinguish these different words. For example, the “first” and “second” in the first video and the second video are only used to indicate the video used in different operations.

According to the above descriptions, the panoramic video conference technology (at least includes the system and the method) provided by the present disclosure identify a plurality of video objects in the panoramic video by analyzing the panoramic video. In addition, the panoramic video conference technology provided by the present disclosure can select a to-be-removed video object from the video objects to remove from the panoramic video based on a privacy mode. Finally, the panoramic video conference technology provided by the present disclosure generates a panoramic conference video based on the panoramic video and a background filling video. The panoramic video conference technology provided by the present disclosure can selectively hide some video objects through the above-mentioned mechanism, so as to provide enterprises with more flexible privacy requirements. In addition, the panoramic video conference technology provided by the present disclosure provides a plurality of functions, and improves the service and quality of the online conference.

The above disclosure is related to the detailed technical contents and inventive features thereof. People skilled in this field may proceed with a variety of modifications and replacements based on the disclosures and suggestions of the invention as described without departing from the characteristics thereof. Nevertheless, although such modifications and replacements are not fully disclosed in the above descriptions, they have substantially been covered in the following claims as appended.

Although the present invention has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.

It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the present invention cover modifications and variations of this invention provided they fall within the scope of the following claims.