Patent ID: 11943074
Assignee: ZOOM VIDEO COMMUNICATIONS, INC.
Field: Audio-visual technology (Electrical engineering)
Classification: CPC H  G | IPC G  H

Claim 9:
10. An apparatus, comprising:
a memory; and
a processor configured to execute instructions stored in the memory to:
generate a real-time transcription of audio data from one or more participants temporally related to video data, wherein participants include a speaker participant and one or more audience participants;
determining, by a processing of the video data using a machine learning system trained for reaction detection, meanings of reactions of the one or more audience participants to the speaker participant during a video conference based on the real-time transcription of the video conference and the reactions,
wherein the video data visually represents the reactions;
wherein determining the meanings of the reactions of the reactions based on the real-time transcription of the video conference and the reactions includes determining a context associated with the speaker participant based on the real-time transcription of the video conference, and
wherein the context associated with the speaker participant relates to a content of the conference;

determine, by a server during the video conference, sentiment types of the one or more audience participants based on the determined meanings of the reactions, and maintain a count of the determined sentiment types aggregated as bins in a histogram;
determine, by the server during the video conference, an engagement level based on the most frequent bin in the histogram;
cause, by the server, an output of a real-time recommendation based on the engagement level at a device associated with the speaker participant during the video conference; and
provide, by the server using the machine learning system, a real-time recommendation to the speaker participant based on real-time recommendations that were previously effective, which are determined based on an aggregation of engagement levels, real-time recommendations, and speaker participant behaviors over multiple conferences.